Dynamic Weight Grafting: Localizing Finetuned Factual Knowledge in Transformers
Authors: Todd Nief
David Reber
Sean M. Richardson
Ari Holtzman
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretability
Primary area: interpretability and explainable AI
TL;DR: We propose dynamic weight grafting (grafting parameters from a finetuned to a pretrained model) to localize behavior to model components

Abstract: When an LLM learns a new fact during finetuning (e.g., new movie releases, newly elected pope, etc.), where does this information go? Are entities enriched with relation information immediately, or do models recall information just-in-time before a prediction? Or, are "all of the above" true, with LLMs implementing multiple redundant heuristics? Existing localization approaches (e.g., activation patching) are ill-suited for this analysis because they usually replace parts of the residual stream, thus overriding previous information.
   To fill this interpretability gap, we propose dynamic weight grafting, an analysis technique that selectively grafts subsets of weights from a finetuned model onto a pretrained model. Using this technique, we show two separate pathways for retrieving finetuned relation information: 1) "enriching" the residual stream with relation information while processing the tokens that correspond to an entity (e.g., "Zendaya" in "Zendaya co-starred with Timothée Chalamet" and 2) "recalling" this information at the final token position before generating a target fact. In some cases, models need information from both of these pathways to correctly generate finetuned facts while, in other cases, either the "enrichment" or "recall" pathway alone is sufficient. We localize the "recall" pathway to model components---finding that "recall" occurs via both task-specific attention mechanisms and an entity-specific extraction step in the feedforward networks of the final layers before prediction. By targeting model components and parameters, as opposed to just activations, we are able to understand the mechanisms by which finetuned knowledge is retrieved during generation.

Published as a conference paper at ICLR 2026
DYNAMIC WEIGHT GRAFTING: LOCALIZING FINE- TUNED FACTUAL KNOWLEDGE IN TRANSFORMERS
Todd Nief David Reber' Sean Richardson Ari Holtzman 1.3
DDepartment of Computer Science, University of Chicago *Department of Statistics, University of Chicago
3 'Data Science Institute, University of Chicago
ABSTRACT
When an LLM learns new fact during finetuning (e.g., new movie releases, newly elected pope, etc.), where does this information go? Are entities enriched with relation information immediately, or do models recall information just-in-time before a prediction? Or, are "all of the above" true, with LLMs implementing multiple redundant heuristics? Existing localization approaches (e.g., activation patching) are ill-suited for this analysis because they usually replace parts of the residual stream, thus overriding previous information. To fill this interpretability gap, we propose dynamic weight grafting, an analysis technique that selectively grafts subsets of weights from finetuned model onto pretrained model. Using this technique, we show two separate pathways for retrieving finetuned relation information: 1) "enriching" the residual stream with relation information while processing the tokens that correspond to an entity (e.g., "Zendaya" in "Zendaya co-starred with Timothée Chalamet" and 2) "recalling" this information the final token position before generating a target fact. In some cases, models need information from both of these pathways to correctly generate finetuned facts while, in other cases, either the "enrichment" or "recall" pathway alone is sufficient. We localize the "recall" pathway to model components-finge via both task-specific attention mechanisms and entity-specinne extraction step in the feedforward networks of the final layers before prediction. By targeting model components and parameters, opposed to just activations, to understand the mechanisms by which finetuned knowledge is retrieved during generation.
that "recall" occurs
able
INTRODUCTION
When a new pope elected and we want an LLM answer "Who's the pope?" correctly, how does the model implement this behavior after being finetuned
information? Large Language Models (LLMs) are capable of storing and retrieving a large number of relationships and associations from pretraining (Petroni et al. 2019 Roberts et al. 2020), but how does a model encode this information in its parameters when finetuned on new facts? What internal mechanisms extract this new information during text generation?
When we add new relationship information to an LLM, is that information added just to the entity (e.g.. just in the embeddings), is it "enriched" the entity token position in lower and middle layers, or is it recalled in response to the entity in higher layers closer to next token prediction? And, on more granular level, which model components contribute to remembering newly learned relation information and at which token positions do they contribute to knowledge retrieval?
The purpose of our study develop tools that allow to isolate which mechanisms are responsible for invoking finetuned information, e.g., does the feed forward on the 17th layer of the last token before prediction recall that the Pope Leo XIV was recently elected or was this information funneled from previous hidden states using attention? Previous interpretability approaches to localizing relation knowledge have either used variants of activation patching (Meng et al. (022a) or ablations (Geva et al. 2023) to see which components contribute to next token prediction. Activation patching
"Correspondence mief uchicago.edu

---

Published as a conference paper at ICLR 2026
Promstel
Twiogy
0rf Drfaerhhi arh
Och Crht &rlerrl
Orfhrrr Orhh
Zendaye 0o sa5s wh (a)
Zantege cr-a wht
(b) component grafting
position grafting
weights
some madill weights some layers
lyyers
Activation Patching
Weight Grafting
some timesteps
some timestaps
(a) Dynamic weight grafting vs. Activation patching
(h) Position grafting vs. Component grafting
Figure 1: We introduce dynamic weight grafting to analyze model mechnisms for finetuned knowledge retrieval setrpenal-swapping small subsets of a pretrained model's weights with the weights of a model that has undergone supervised finetuning (SFT). (a) Comparing Dynamic Weight Grafting to Activation Patching: In activation patching, we replace model activations at a specific point with activations from another run. In dynamic weight grafting, we replace individual mechanisms by swapping in specific parameter matrices of a finetumed model into a pretrained model. (b) A schematic showing the different dynamic weight grafting schemes used in our experiments. In position grafting, use either the entire pretrained or finetuned model at every layer and component for given token position. In component grafting, blend pretrained and finetuned weights dynamically each token position.
and ablations have key limitation: by modifying replacing activations inside the model these techniques block access to computations that came before the intervention position. For example, when an activation specific layer and token position is patched or ablated, we also overwrite all the upstream information that was flowing into that activation (e.g. ""enrichment" of an entity with associated factual information (Geva et al. 2023)). This makes impossible to tell whether component of the model is actively extracting new information, simply passing along information that was computed earlier. As a result, we can't 't isolate which mechanisms are truly responsible for incorporating finetuned relation knowledge. Even when used together, activation patching and ablations do not allow us to determine which parts of the model are necessary and sufficient for recalling certain information, properties that faithful localization should have.
To address this, we propose dynamic weight grafting, aa method for interpreting how finetuned knowledge is used in language models by swapping in subsets of weights from finetuned model during generation. These swaps can be done selectively specific layers, components, and token positions (see Figure (ggure[b)-allwiiiin to reproduce the effects of finetuning without disrupting the rest of the computation. This combines the advantages of model grafting, which leaves previous computations intact (Panigrahi al. 2023] Ilharco et al. (2022) with the ability to apply causal mediation analysis to specific mechanisms in the model, similar lar to activation patching (Heimersheim and Nanda 2024 Goldowsky-DDi al. 2023 Meng et al. 2022a)
us test which parts of the model are necessary and sufficient
Using this method, we identify two pathways by which finetuned relation knowledge is retrieved during generation. We see that, in some cases, grafting only at the first entity or only at the last token is sufficient to reproduce the finetuned model's relation completion behavior. This implies two things: (1) If an entity is "enriched" with relation information when it is first processed, generic mechanisms can extract the correct entity to complete the relation tuple (Meng et al., 2022al Geva et al.) 2023 2022) and (2) "recall" mechanisms can extract relation completions from entities that were not enriched with finetuned relation information. Both pathways together nearly recover full finetuning performance, and grafting everything except these pathways results in near-zero relation completion accuracy. In short, dynamic weight grafting reveals that the enrichment and recall pathways are
I Code available at https: gi thub. com/toddn ef / /dy//dnnnn Bynami c- wwighttgraaaateeeee ingiiig We also note that this method can he applied any setting where we have a finetuned model and pretrained model, including most post-training procedures.
2

---

Published as a conference paper at ICLR 2026
necessary and sufficient for relation completion, localizing finetuned information with granularity that was not previously possible.
2
METHODOLOGY
Given pretrained and finetuned model parameters (prc and 6t, how might one localize the mechanisms responsible for behavior changes in the finetuned model? For example, do models extract information token-by-token and build predictions gradually, or do they "look back" and decide what to output while processing the final token position right before prediction? Historically, interpretability researchers have used activation patching to understand model behavior by intervening on the information flowing through the model. In activation patching, typically replace part of the model's residual stream or the input/output to a specifie model component (e.g. the attention block on layer 15). However, to localize behavior model components, would like use the same residual stream, but perform a different computation. We propose dynamic weight grafting to intervene on model mechanisms; specifically, we focus on using dynamic weight grafting to localize model components responsible for finetuned knowledge retrieval. See Figure for a comparison between activation patching and weight grafting. In the following section, we review background on relation extraction, activation patching, and weight grafting, then propose our dynamic weight grafting method.
The classic relation extraction task involves finding the semantic relation (r) between a subject (s) and an object (o) in natural language text, yielding (s,r, r, o) tuple (Zhao ctt al., 2024; Hinton et al. 1986). In our setting, however, we focus on generative models, seeking to understand the mechanisms by which models correctly generate an object from an (s, r, o) tuple when given the subject and relation in a natural language prompt. See Figure Tb for an example.
Relation Completion
Activation Patching There has been host of work which attempts localization based the information flow through the residual stream vectors (hidden states) X(t, 1) at token / and layer 1. The most popular of these approaches broadly termed activation patching, which we use refer to any method that replaces some vectors A(t, I) with new vectors A(t, l) (including replacing the entire residual stream, a subspace of the residual stream, or the input/output of specific model component). (Heimersheim and Nanda 2024 Gooldoskyy-Dii Dill et al.) 2023) Meng et al., 2022a).
Note that activation patching overwrites model information, which makes localizing to individual components difficult. For example. if we were to patch the residual stream from the pretrained model into the finetuned model in an early layer at the final token position, we may be overwriting information that helps select what import from previous positions via attention. If we patched at late layer, may be removing information in the residual stream that was already imported by attention. To use activation patching to localize behavior to model components, the most natural thing to patch would be the residual stream from the pretrained model into the finetuned model at specific point in the model's computation graph. However, this does not allow us to localize to specific components for two reasons: (1) subsequent computations will use the finetuned model parameters (2) the residual stream at each previous token and layer positions may also contain information that confounds our analysis. To handle this, we instead focus grafting in subsets of finetuned model parameters so that we can isolate model mechanisms.
Weight Grafting In previous work, Panigrahi et al 2023) define grafted model Ô using a mask y; they learn sparse mask over model parameters to recover finetuned performance:
Ofee gh
if 7/4 0
0,
if 7/8
where refers to the hh parameter of each model, "pre" refers to the pretrained model, and "ft" refers to the finetuned model. Equivalently, Ilharco et al. 2022) express this idea as: 0 = 0 tyo (pe)
Note, however, that this setup simply creates new model with both parameters from the pretrained model and the finetuned model. In our approach, we dynamically change which components are grafted at each token position (see Figure [b)), allowing to localize model behavior to both model components and token positions. Thus, we capture the information flow over the entire sequence, a
perspective emphasized in previous work (Geva et al. 2022 2023 Ferrando and Voita 2024...
3

---

Published as a conference paper at ICLR 2026
Dynamic Weight Grafting Since our goal is to directly understand the model's mechanisms, the most direct approach is to just graft in portions of the finetuned model, identifying a subset of the weights and token positions which are sufficient to replicate full finetuned outputs. In this way, consider each component of the transformer at each token position as a mechanism which can be intervened on. For example, may decide to replace the feedforward networks at the first entity token positions with finetuned model parameters.
More formally, given two models 0 and 0 consider the ordered sequence of their weight matrices 0 101 ... 0M. and 0 [0 ... ..00) ]- Let y be 1 x M mask over these components. Each 0 corresponds to a specfic model component (e.g. the Wº matrix at the 12th layer, etc.). We define Dynamic Weight Grafting token-wise, component-wise weight grafting:
(t)) 0
ml = ( if 7t
where e refers to the t component of each model and refes to the token position. In words: while processing the residual stream at given token position, swap model components dynamically based on our grafting configuration. This is illustrated in Figure lb For example, at the last token position, we may elect use the finetuned feedforward networks for all layers in the second half of the model to see if this is sufficient to recover finetuned information. In our experiments, we consider 0% to be the pretrained model OfRE and consider 0 to refer to the finetuned model 0SFT.
3 EXPERIMENTS & RESULTS
Models We use four pretrained Transformer- decoder-only language models in our exper- iments: Llama3 Grattafiori et al. (2024), Pythia 2.8b Biderman et al. (2023). GPT2-XL RRdford al. (2019), and Gemma 1.1 Gemma Team al. (2024). Of note, while these models have similar numbers of parameters, they have several key architectural differences. See Table Appendix A.3. for a comparison of models.
Data We follow Allen-Zhu and Li 2023) and use templated supervised finetuning data to control the relationship information that models exposed to during finetuning. We augment training data with several rephrases of article-style training text and question-anse answering examples. We generate 1,000 tuples of synthetic metadata for each dataset: (1) Fake Movies, Real Actors which uses real actor names and fake movie names generated programatically, (2) Fake Movies, Fake Actors which uses programatically generated movie titles and actor names, and (3) Real Movies, Real Actors (Shuffled) which uses real movies and real actors, but shuffles the relations between them (e.g. "Keanu Reeves starts in The Departed alongside Meryl Streep").. We then generate five templated "article" examples and five templated "QA" examples for total of approximately 10,000 examples in each finetuning dataset. See Appendix [A.2.3 and Appendix [A.2.4 for examples of templates and training examples, Appendix A.2.2 for more detail on dataset creation, and Appendix AA.33 for training details. In the main body the paper, we present results for the Fake Movies, Real Actors dataset; results for all datasets are in Appendix [ Results are similar across datasets. We then conduct additional experiments with Wikipedia articles for movies released after the release date for Gemma, showing that our results generalize beyond templated data.
3.1 WHICH POSITIONS ARE SUFFICTENT FOR RELATION COMPLETION?
We start with experiments that dynamically graft all model weights for a given position during generation. We call this "position grafting" (see Figure Ib for a visual comparison between grafting schemes). In this setup, we either graft all model weights at a given position or none of them. Note that when we graft at a particular position, we use the keys and values from the grafted model to compute attention at that position, even if future positions use pretrained model parameters. This is important, as we are fully swapping out one model for amother at the chosen positions in the sequence. Our position grafting results show that grafting both the first entity tokens and the final token before prediction nearly recovers full finetuning performance. We present results for top-5 accuracy on relation completion on all tested models in Figure 2 See $3.1 for why use top-5 accuracy.
4

---

Published as a conference paper at ICLR 2026
In some cases, we also see that grafting only the first entity tokens is sufficient to recover some top-5 performance (similar to previous results showing that models enrich entity representations with factual information early in the sequence and carry that information forward (Meng al.) 2022a Geva et al. 2023 ). In other cases, a final token "recall" pathway alone is sufficient to extract relation information
without subject enrichment. This indicates that the later layers of a finetuned model contain a mechanism to retrieve relation information from finetuning, even in response to representation that does not contain information from the finetuning set.
While the "recall" and "enrichment" pathways individually have worse top-5 accuracy than when combined, for several models and sentence templates, a single pathway can be sufficient for good relation completion performance. In Gemma-1.1 1.1, the "recall" pathway alone achieves 53% top-5 accuracy on relation completion (compared to a finetuned baseline of 100%) and the "enrichment" pathway for GPT2-XL. reaches 28% top-5 accuracy. See Appendix C for results for all models and all datasets.
A Note T Accuracy In Figure we examine the top-5 accuracy for the correct relationship token (we score the example as correct if the desired relationship token is in the top five choices of the next token sampling distribution). We choose top-5 accuracy since models will sometimes give high probability to entity tokens from the context instead of the correct actor name or to common tokens like " the"-we -we interpret this as the model being uncertain. Additionally, models sometimes have multiple plausible tokenizations of an actor's s name (e.g. R". Rob", "Rob", "Robert"), will give high probability to an incorrect name, but promote the correct name to, say, the second token position. This is result of the open-endedness of our setup (training the model on then generatively querying the knowledge in grafted models). We note that our results are lower bound on the ability of the model to retrieve correct relation information (Jiang et al. 2020), and that the choice of k for top k accuracy changes the scale of the accuracy results, but not the overall pattern (see Appendix
relations and
for results with additional choices of k:).
We also hypothesize that blend of features, some of which "know" the relation and others of which do not, can cause predictions to regress back the prior token distribution (unconditional on the prompt), which can result in the model defaulting to high frequency tokens. To give more fine- grained understanding, we provide additional results for token rank in Appendix C.1.2 Investigating whether and how models default to the unconditional prior is promising direction for future work.
0.3
025
011
0.23
$33
renc
Genna L1 P
GPT2 AL
Figure 2: Top-5 Accuracy on Test Headline for Position-Leve Grafting: We show top-5 accuracy for position grafting for the headline test sentence. Grafting configurations are PRE (pretrained baseline), SFT (supervised finetuning baseline), FE (grafting only the first entity), LT (grafting only the last token position), FE+LT, (FE+LT) (grafting everything except the first entity and last token position), FE and LT All models show nearly full SFT performance by grafting only the FE and LT tokens and near pretrained performance when grafting everything except the FE and LT tokens. We present results for Gemma and GPT-2 XL. Results for Llama (similar to Gemma) and Pythia (similar to GPT-2 XL) are available in Appendix
3.1.1 NECESSITY FOR RELATION COMPLETION
While show that grafting the first entity and the final token position sufficient to recover finetuned model performance, this does not rule out other pathways for models to extract relation information. To test this, graft the complement of the first entity and the last token: (FE+L.T)
5

---

Published as a conference paper at ICLR 2026
(L.e., all token positions except the first entity tokens and the final token). This results in near-zero top-k accuracy performance for all models, comparable to that of the pretrained model (Figure 2)
We notice that GPT2-XL has much better performance LT than other models, which all have near zero top-k accuracy on this grafting scheme. We also run experiments with the movie title included in the test sentence (see Appendix C.5); we find that the movie title alone is not sufficient to recover finetuned performance, but the movie title and the last token together give improved performance over the last token alone. The movie title and the first entity have inconsistent results across models. See the discussion in $ for more.
3.2 IS IT THE POSITION OR THE TOKEN THAT MATTERS?
There is ambiguity the setup described above:
Is the relation extracted the last token before relation generation or at the preposition connected to the relation (e.g., stars in)? In the test headline (see [), these are always the same token. hypothesize that relation completion may occur relation tokens we constructed the QA test (Table T so that the final token before generation is neither the relation nor a preposition associated with the relation. We see similar results: the last token is responsible for relation knowledge retrieval, even when it is neither preposition relation. We present results for these experiments in Appendix C.I.I
Orl. Orfh 0rll
9rth e orh.
0rll 0-h
0:fh 1ith-h00 4rhh
arl Drl. 9.h
arh arh.
Grl: arlh 0rll
aal.e...o 0rl
arth 0rll
10 arl
Zandgg ciea
Znres comt
Jediy evn vn
Prstncee
TakkSpacfis
flewkondppaciic
Co-las
Grafte Model
Figure We localize the "recall" pathway of finetuned knowledge retrieval specific model components. The task-specific model has been trained data that shares the same form as the test task, but has not seen the test relation. The relation-specifie model has been trained on the test relations. The "recall" pathway uses both task-specific attention mechanisms the first entity and the final token as well as -elacifn- relation extraction mechanisms in feedforward networks in the final layers before next token prediction.
3.3 CAN WE LOCALIZE THE "RECALI." PATHWAY TO INDIVIDUAL. MODEL COMPONENTS?
Prior work has shown that next token prediction is blend of information propagation through attention heads and token promotion through feedforward networks (Geva al. 2020 2022 2023). We seek to understand whether the "recall" pathway at the final token position relies mostly on attention, feedforward networks, both. In other words: Is the attention the feedforward that learns new relation? (Of course, it can be both.)
Component grafting In component grafting, we graft individual components of the Transformer block given token position (Figure lb). Recall that Transformer block is described by E]
Block(x:) ATTN(NORM(=)) O FFN(NORM(s ATTN(NORM(2))) O)) (1)
where the NORM operation either Layer Norm RMSNorm, ATTN is the attention, FFN the feedforward network, and O is the output projection matrix for multi-hear heatid self-attention. We
'There are model-speciin specific subtleties to the attention and feedforward operations. For example, Llama and Gemma use RMSNorm and GPT2-XI. and Pythia use LayerNorm. Models may also have slightly different placements schemes for adding outputs the residual stream.

---

Published as a conference paper at ICLR 2026
focus on grafting individual weight matrices at given token position; for example, we may graft the W WQ and WV matrices at the first entity token positions. At a high level, we conduct two component grafting experiments to localize model behavior for the last token "recall" pathway: 1) We localize relation knowledge retrieval to the output projection matrix and feedforward networks at the final token position, and 2) We show that models require task-specific attention the first entity in order for the "recall" pathway to work. See Figure for a visual representation of the these results.
Training on disjoint data to localize relation completion We attempt to localize relation com- pletion by training two separate models: one is relation model trained on the full dataset (as in the previous experiments); the other is a task model trained on text with the same semantic and syntactic structure but without the specific relation information we are attempting to retrieve. See Appendix A.2.5 for examples of the data used to train a task model vs. a relation model.
In these experiments, we leverage the reversal curse Berglund et al. 2023) and Allen-Zhu and
Li 2023) show that models trained on relationships in one direction (" ("Werner Herzog starred in a movie with Nicolas Cage") fail to learn relationships in the other direction ("Nicolas Cage starred in movie with Werner Herzog") (bad) 2009). In this setting, our "task-specific" models have been trained on only one direction of a relationship (c.g., various paraphrases of "Werner Herzog starred in a movic with Nicolas Cage" with "Werner Herzog" always preceding "Nicolas Cage"), while our "relation-specific models" have seen both directions of the relationship.
Figure 4: Top-5 Accuracy on Test Headline for Component- Grafting: We graft weights from models finetuned on both directions of symmetric relationship at the last token to identify which components drive relation completion. For Gemma and Llama, the output projection matrix and feedforward networks in the last quarter of the model recover most of the finetuned performance.
Grafting with a hybrid model First, we graft from the relation model to the task model at the final token position examine the "recall" pathway. We present results for Gemma and Llama3 in Figure 4 In Figure 4 we see that grafting the O matrix and the full FFN nearly recovers the results of grafting the fial attention mechanism and the full FFN. This implies that, during finetuning, models learn operations in the O matrix which trigger the correct "recall" mechanism using FFNs at the final layers before predicting the recalled entity. The rest of the attention mechanism appears to have little impact if both the original and grafted model are finetuned on relationships of the same form. We were also surprised to see the importance of the matrix-removinnn removing the O matrix and only using the FFN harms top-5 accuracy by 29% in Gemma and 41% % in Llama3. We also hypothesized that either the "read" operation in the FFN up-projection or the "write" operation in the FFN down-prow down-projection would be more important. Instead, we see that either recovers some relation completion performance when paired with the matrix, but both together is much stronger than either alone.
Next, graft between three models: 1) pretrained model (ignorant of any of the relations in our dataset), 2) a task model, and 3) a relation model. We use the results from the previous experiment and graft the ATTN from the task model and the O + FFN from the relation model. We then graft different components on the first entity entirely from the task model to see which components contribute to
*GPT2-XI. and Pythia had much weaker "recall" results than Gemma and I. Jlama-we we present results for GPT2-XL. in Appendix C
7

---

Published as a conference paper at ICLR 2026
model performance in the "recall" pathway. In Figure 5 that grafting the rask ATTN the first entity and, at the last token, the task ATTN and the relation O+FFN recovers 63% top-5 accuracy for Gemma and 34% accuracy for Llama3. Grafting the task FFNs at the first entity barely helps performance. See Figure for an illustrated version of these results.
244
0.24
0.5
HIDEO UE Grfren
Sry
WYRDD nNo RD
wrteen arto
IISRN InSen Grrrn)
Gemme 1. 2017
Unme 32
Figure 5: Top-5 Accuracy on Test Headline for Component-Leve Grafting (Hybrid); We graft from both a task model and relation model onto a pretrained model, creating hybrid model. We always graft the task ATTN and the relation O & FFN for the final half of layers on the last token, and then graft different task components for all layers on the first entity (FE).
3.4 Do THESE MECHANISMS APPLY TO NON-TEMPE CON-TEMPLATED DATA?
We also explore whether these mechanisms apply to non-templated data. In these experiments, we finetune Gemma on twenty Wikipedia articles (and five LLM- LLM--gnnrated rephrases for each article) about movies released after the model's release date. This setting includes potential confounds (models are sensitive to the semantic structure of the finetuning data, the order that entities appear in, etc.-it's also possible that information about some of these films was in the finetuning data even though the release date is after the model was released). Still, we see the first entity and last token positions nearly recover full finetuning performance, the enrichment and recall pathways recover some finetuning performance (weaker in this setting than in the synthetic setting), and the complement of the first entity and the last token perform the same as the pretrained model. Since the enrichment and recall pathways weaker in this setting, present both top-5 and top-50 accuracy results.
0.04
635
012
RiG
Figure 6: Accuracy for Gemma finetuned on Wikipedia articles about movies released after the model's release date.: Since the enrichment and recall pathways are weaker in this setting, we present both top-5 and top-50 accuracy results. In this setting, we still see the first entity and last token positions nearly recover full finetuning performance, and the complement of the first entity and the last token perform the same as the pretrained model.
4 DISCUSSION
We use dynamic weight grafting to show that:
When models undergo supervised finetuning relation information, the entity tokens and the final token position are where relation completion occurs cicurr-ithhr pathway alone can be sufficient, and both nearly recover full finetuning performance.
8

---

Published as a conference paper at ICLR 2026
At least one of these pathways is necessary to recover relation information-graatn everything except the entity tokens and the final token position results in near-zero top-5 accuracy.
Using component grafting (see Figure Tb), we localize the relation retrieval in the "recall" pathway to the output projection matrix and the feedforward networks at the final token position as well as task-specific attention at the first entity and the final token.
We note that the last token "recall" pathway appears to be much stronger in the Gemma and the Llama3 models tested than in the GPT2-XL XL and Pythia models. There many differences between these model architectures, including norm, positional embeddings, activation functions, tied vs. untied embeddings, and attention mechanisms. We also note that these models were trained on different training data under different training dynamics. See Table for more detailed comparison between models. We hypothesize that the more recent models have more expressive attention mechanisms that allow for better recovery of relation information.
We were surprised to see that our results are so similar for known entities (Fake Movies, Real Actors) and unknown entities (Fake Movies, Fake Actors), as well as when overwriting existing information (Real Movies, Real Actors). It seems that LLMs are able freely manipulate relation information during finetuning for both known and unknown entities, implying that narrow factual information is easy to update within specific context. However, previous knowledge editing work shows that this type of update does not generalize to other contexts (Zhong et al., 2023b Hase et al. (02T)
Since weight grafting doesn't delete computations (see Section 2). we see different results than more destructive interpretability methods in localizing information retrieval. Our component grafting experiments discussed in 3.3 show slightly different results than Geva et al. al.] (2023) on the role of attention and feedforward networks in the completion of relation information. Geva et al. (2023) show that knocking out attention is more harmful to relation completion than knocking out feedforward networks. Our results show that, in setting where a model has already learned how do specific relation completion task, the O matrices and the feedforward networks at the final token position are nearly sufficient recover relation completion performance as long as the model has task-specific attention functionality. This shows that mechanism--eellaceeee can give more finegrained results than interventions that destroy or replace information in activations.
Future Work We leave the alternative entity enrichment pathway present in Pythia and GPT2- XL-see unexplored in this work. We hypothesize that other token positions before the last token can also extract relation information from an enriched entity, which is then passed to the final token position for processing. Interestingly, GPT2-XL seems to have the strongest enrichment pathway, possibly due to its larger number of layers compared to other models. We also note that our experiments only cover the single-hop setting; exploring multi-hop settings is a natural next step. Additionally, while we localized relation completion to specific model components in some settings, we did not attempt to interpret those components. There is a line of interpretability work in "parameter space" (Ilharco et al.. 2022 Jain et al.) 2024 Millidge and Black, 2022) and we can imagine applying those techniques to the parameters of components that are important for a specific task.
Limitations Our work focuses on synthetic knowledge retrieval task, potentially limiting its scope and generalization to other settings with more complex sentences or more varied finetuning data. Additionally, we operationalize the "success" of knowledge retrievals using top-k accuracy or the token rank of the correct relation entity during next token prediction-our methods do not account for the possibility that models "know" information in a way that doesn't 't impact next token prediction. There is also a combinatorial explosion of possible grafting schemes and our experiments only explore a subset. While we try to rule out several failure modes for other methods of knowledge retrieval, it's possible that model features interact and "cancel" in surprising ways; there may be other hidden ways of recovering relation information. Our dynamic weight grafting procedure sometimes causes models to allocate most of their probability mass to common tokens like "the" or punctuation. We hypothesize that this is because the blend of features created during weight grafting may be making the model uncertain about the next token, it reverts to prior distribution with higher probability mass on common tokens. We use smaller models in our experiments due to compute limitations; larger models may have different mechanisms when finetuned new relation information.
9

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Mechanistic Interpretability & Knowledge Editing Our work follows a tradition of interpretabil-
ity work which intervenes on Transformer-based language models to understand behavior (Vig et al. 2020 Geiger et al. 202T). Previous work has focused on interpreting how language models
encode subject-object relationships (Meng et al., 2022a Geva et al., 2023 Hernandez al., 2023
Yu et al.] (023). Follow up work from Hase et al. (2023) and Wang and Veitch (2025) questions
whether editing provides evidence of localization- Transformer-baser language models seem to be
highly flexible in where edits can be applied in order to change stored knowledge. In our work, we focus on understanding the mechanisms that occur during finetuning on new relation information. Additional lines of work have focused on understanding information flow through language models using gradient-bas -based methods (Ferrando and Voita 2024. Kramár et al. 2024 Kobayashi et al. (2023) )
finding interpretable circuits that models use to perform specific tasks (Wang et | [022] Nanda et al.
2023 Zhong et al.] 2023a] [Hanna et al. 2023 Merullo et al. 2024), or tracking the relationship of
dictionary-learning features to next token prediction (Cunningham et al. [0233 Bricken al. 2023
Gao et al. 2024] Ameisen et al. (025). Other interpretability work has focused on comparing the
representations and mechanisms of different models (Huh al. 2024 Korbllihh al. 2019 Raghu
et al. 2021; Wolfram and Schein 2025). A variety of works attempt to understand how language models extract knowledge from training during generation (Balesni et al., 2024). [erglund al.
(2023) and Allen-Zhu and Li (2023) both show that language models do not generalize unidirectional mooels
relationships seen during training -oonnng need to be trained on both "A is B" and "B is A" in order to learn symmetrical information. Other work attempts to edit knowledge in language models by directly editing model weights (Meng et al. 2022a b De Cao et al.
2021 Hase et al.) 2021 Mitcheil
eet al. |021 Zhu et al., 2020). Zhong et al. (2023b) and Cohen et al.
(2023 both propose benchmarks
to test multi-hop hhpp performance on knowledge edits, and other work explores the role of attention heads
in task performance and knowledge retrieval (Sakarvadia et al. 2023 Yin et al. 2024)
Interpreting Model Parameters Previous work attempts to localize and interpret directions in a model's parameter space (Millidge and Black 2022), often for the purpose of transfer learning (Ilharco et al., 2022] Yadav et al.) 2023). Panigrahi et al. (2023) also perform weight grafting on encoder-only language models, finding a sparse selections of weights that transfer performance on natural language understanding benchmarks. Gueta et al. (203)) find that finetuned models have
"knowledge region" " in weight space that responsible for the model's 's ability to perform finetuned tasks.
6 CONCLUSION
In this work, we introduced dynamic weight grafting, novel method to localize finetuned relation information retrieval mechanisms within Transformer LLMs. Through weight grafting experiments on targeted subsets of parameters, find that models retrieve single-hop finetuned relation information using two pathways: "enrichment" at the first entity and "recall" at the final token position. We further explore the "recall" pathway to localize relation completion task-specific attention mechanisms the first entity and the final token and relation-specific extraction at the O matrix and feedforward networks in the final layers before next token prediction. Dynamic weight grafting offers less destructive alternative to existing interpretability methods, enabling more precise localization of finetuned knowledge in LLMs.
REPRODUCIBILITY STATEMENT AND LLM USAGE
Code is available at hhtttts::// /ttEss://githuiee htt.:uuww...oo/oodiiiii////aaaaa www.ghttggaatiiieeeeeeeeeeeeeeee Pseudocode for dynamic weight grafting is included in Appendix A. Instructions for dataset creation and data examples are included in Appendix A.2 Model details and finetuning instructions (including
hyperparameters) are included in Appendix A.3 LLMs were used to aid in writing the manuscript for help with rephrasing and fluency, finding confusing passages, and LaTeX troubleshooting. I.MSs were used in experiments for autocomplete and help debugging. Some data extraction steps were
generated with LLM agents.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Bad licutenant: Port of call new orleans, 2009. Film.
Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint urXiv:2309. 1131223999911166.. 14316, 2023.
E. Ameisen, J. Lindsey, A. Pearce, W. Gurnee, N. L. Turner, B. Chen, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham,
T. Henighan, A. Jermyn, A. Jones, A. Persic, Z. Qi, T. Ben Thompson, S. Zimmerman, K. Rivoire, T. Conerly, C. Olah, and J. Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL htttp::/rraasffomm--iiiccco ub/ 2tt/22225/aattttti waapaatiiionnnggaasaeeeeee wwwsss..t.....e.o..ooo..ooo.. .hmml
M. Balesni, Korbak, and O. Evans. The two-hop curse: Llms trained on a-> b, b-> e fail to learn a-> arXiv preprint arXiv:2411.16335 2024.
L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Lims trained on" a is 6" fail to learn" is a". arXiv preprint arXiv:2309. 12288,, 2023.
S. Biderman, Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430.
PMLR, 2023.
T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds
A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, Carter, T. Henighan, and C. Olah.
Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/ttaasffomer-ccccuiist wttprruttsppptoo/ommmmmmmmmeeeee wttp://miiii..tttut/nee....
R. Cohen, E. Biran, O. Yoran, A. Globerson, and M. Geva. Evaluating the ripple effects of knowledge editing in language models, 2023. URL https: /wtpxiiivv./o/////////aa////avvv www.oogg//bb...... hbtg://b//2333 220./2237...0 12976
H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. URL
8600
N. De Cao, W. Aziz, and I. Titov. Editing factual knowledge in language models. arXiv preprint wrXi::2104.08144, 2021.
J. Ferrando and E. Voita. Information flow routes: Automatically interpreting language models scale. arXiv preprint 2xXiv:2403.00824 2024.
L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04099 2024.
A. Geiger, H. Lu, T. Ieard, and C. Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 14:9574-9586 2021.
Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Riv- ière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros. A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Pa- terson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy. D. Cer, D. Ippolito, D. Reid, Buchatskaya, E. Ni, F. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, Stanway, J. Brennan, J. Chen, J. Ferret, Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikula, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. Mcllroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin. S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiee, Y. hui Chen, Z. Ahmed, Z. Gong. T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean,
11

---

Published as a conference paper at ICLR 2026
K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy. Gemma: Open models based on gemini research and technology, 2024. httph:tt/://et
M. Geva, R. Schuster, Berant, and O. Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.1491:: 2020.
M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
M. Geva, Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304 /4.004/447777 /4767, 2023.
N. Goldowsky-Dill, C. Macleed, L. Sato, and A. Arora. Localizing model behavior with path patching, 2023. URL htttps:///arii.//// org
A. Grattafiori al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024. URL
https:://rrat www.v//rvvvvv//. www.ii.rrgg..o org/ 220/24407.222 /ttpg///////022722.2 21783
A. Gueta, E. Venezian, C. Raffel, N. Slonim, Y. Katz, and L. Choshen. Knowledge is a region in weight space for fine-tuned language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1350-1370, Singapore. Dec. 2023, Association for Computational Linguistics. doi: 10. 18653/v1/2022 http:////.....innnn--iieae findings-emnlp.... URL https: hatp:///aaaaaatooooa.oooo oro.oogg///// findings emnlp. 957
M. Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpreting mathe- matical abilities in pre-trained language model, 2023. URL httpt::ttttsii////// org. abs/23 05. 00586
Hase, M. Diab, Celikyilmaz, X. Li, Kozareva, Stoyanov, M. Bansal, and S. Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint
2021.
P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does localization inform editing? surprising differences causality-based localization vs. knowledge editing language models. Advances in Neural Information Processing Systems, 36: 16643-1666,, 2023.
S. Heimersheim and Nanda. How to use and interpret activation patching. arXiv [cs.LG], Apr. 2024.
E. Hernandez, S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y. Belinkov, and D. Bau. Linearity of relation decoding in transformer language models. arXiv preprint 2:X1042308.09124 2023.
G. E. Hinton, L. McClelland, and D. E. Rumelhart, Distributed representations. In D. E. Rumelhart and L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, chapter pages 77-109. 109. MIT Press, 1986. doi: 10.755 tess/5236.003.000 UtL/attps: org/ htte//111..551mmmm.. 7ww1/mitpress// 5236. C03. 0006
M. Huh, B. Cheung, T. Wang, and P. Isola. The platonic representation pyypesestationnbypthhsiisss hypothesis. arXiv preprint 20X0::2405.07987, 2024.
G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088) 2022.
S. Jain, E. S. Lubana, K. Oksuz, T. Joy, P. Torr, A. Sanyal, and P. Dokania. What makes and breaks safety fine-tuning? a mechanistic study. Advances in Neural Information Processing Systems, 37: 93406 93478, 2024.
Z. Jiang, F. Xu, Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
12

---

Published as a conference paper at ICLR 2026
G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Analyzing feed-forwart blocks in transformers through the lens of attention maps. arXiv preprint arXiv:2302.00455. 2023.
S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
Kramár, Lieberum, Shah, and Nanda. Atp*: An efficient and scalable method for localizing IIm behaviour to components. arXiv preprint urXiv:2403.00745, 2024.
E. S. Lubana, P. Trivedi, D. Koutra, and R. Dick. How do quadratic regularizers prevent catastrophic forgetting: The role of interpolation. In Conference on Lifelong Learning Agents, pages 819-837. PMLR, 2022.
K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-173722 2022a.
K. Meng, A.S. Sharma, A. Andonian, Y. Belinkov, and D. Bau. Mass-editing memory a transformer. arXiv preprint arXiv:2210.07229, 2022b.
J. Merullo, C. Eickhoff, and E. Pavlick. Language models implement simple word2vec-style vector arithmetic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Pupers), pages 5030-5047, 2024.
B. Millidge and S. Black. The singular value decompositions of transformer weight matrices highly interpretable. Less Wrong, 2022. UtL/:ttatr//ww..esssrrrroggoo bG bGj2xD8d8XKKHZA /tteeettteeeeeee si ngull ar-va - ot weight Accessed: 2025-05-15.
E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale. arXiv preprint arXiv:2 /0. /1309, 2021.
N. Nanda, L. Chan, T. Licberum. J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023. URL https::/t////cc httppp/s//aaadi..oggaab///////// 05217
A. Panigrahi, N. Saunshi, H. Zhao, and S. Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pages 27011-27033. PMLR, 2023.
Pearson. Chatgpt can reveal personal information from real people, google researchers show. VICE, 2025. URL
Accessed: 2025-05-15.
F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller. Language Models as Knowledge Bases? In K. Inui, J. Jiang, V. Ng. and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pages 2463-24439 24733, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10. 1653//11DD19115555555 URL
1250
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskever. Language models are unsupervised multitask learners. /t//hrrpsss//pppniiccccc/eeeeeee vised OpenAl Blog.
unsuper
M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 121 2021.
A. Roberts, C. Raffel, and N. Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint 20X1v22002.08910 2020.
13

---

Published as a conference paper at ICLR 2026
M. Sakarvadia, A. Ajith, A. Khan, D. Grzenda, N. Hudson, A. Bauer, K. Chard, and I. Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer based language models. arXiv preprint 2rXxv:2300006003 2023.
Vig, S. Gehrmann, Y. Belinkov, Qian, D. Nevo, Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems,
2020.
K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL
org/
012211000593
Z. Wang and V. Veitch. Does editing provide evidence for localization? arXiv:2502. 1447, 2025.
arXiv preprint
C. Wolfram and A. Schein. Layers at similar depths generate similar activations across Ilm architec- tures, 2025. URL htttss://axxx. org/ abs 2504
0000s//50..88 08775
P. Yadav, D. Tam, I. Choshen, C. A. Raffel, and M. Bansal. mies-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
Yin, X. Ye, and G. Durrett. Lofit: Localized fine-tuning on Ilm representations. Advances in Neural Information Processing Systems, 37:9474-9506. 2024.
Q Yu, Merullo, and E. Pavlick. Characterizing mechanisms for factual recall in language models. arXiv preprint arXiv:2310. 15910, 2023.
X. Zhao, Y. Deng, M. Yang. L. Wang, R. Zhang, H. Cheng, W. Lam, Y. Shen, and R. Xu. A comprehensive survey on relation extraction: Recent advances and new frontiers, 2024. URL https://arxiv.orgg abs 2306. 02051
7. Zhong. Z. Liu, M. Tegmark, and J. Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023a. URL Ttt.:ttpps///arxiv..oo/////////0
Z. Zhong. Z. Wu, C. D. Manning, C. Potts, and D. Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795, 2023b.
C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli,, Li, Yu, and S. Kumar. Modifying memories in transformer models. arXiv preprint arXiv:20/2.00363, 2020.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTAL DETAILS
In Appendix C.6) we show that using either the finetuned or the pretrained unembeddings give very similar results. Unless otherwise specified, use the finetuned embeddings for all grafted token positions and use the original model's unembeddings during next token prediction.
issues with catastrophic forgetting, so we trained models with less aggressive learning rate and removed weight decay (Zhu et al. 2020 Lubana al. 2022) while also including supplemental training examples from openwebtext and IMDB movie reviews. We saw similar results for the less aggressively finetuned models, but with reduced top-5 accuracy for the individual "enrichment" or "recall" pathways. See Appendix C.7 for
Since are finetuning small models synthetic data,
more details.
A.1 PSEUDOCODE FOR DYNAMIC WEIGHT GRAFTING
Algorithm 1: Dynamic Weight Grafting
Input: pretrained model (PRE, finetuned model (SFT,, token sequence æ1,7, mask function (t(t) {0, 1}
Output: next-token logits y1:T
initialize V-cachee K. <- 0; for f = to do
determine components graft at token t: C. + {c (e(t)= 1}: foreach e in C2 do
L temporarily set 0g + OSFT:
forward pass on token and update cache: (ye, KC) + Forward Init KC); foreach in C2 do restore 0g < OPRE:
return V1:T
A.2 DATA
A.2.1
RELATION PROMPT TEMPLATES
Table I: Relation prompt templates used to test model relation completion capabilities, with examples
Headline
[first_actor) (relation)
Brad Pitt starred in movie with
[relattion_preositiine a
movie (preposition)
QA
Q: Who (relation)
Who starred in alongside
inalaiion_rrepssttion movie (preposition)
Brad Pitt? A: An actor named
[first_ actor naned
actori? 2 A: An
A.2.2 METADATA
For our fake movies, real actors dataset we first create metadata for each example by sampling real actors from a list of actors with Wikipedia pages. We then exclude examples with "Jr." in the name (e.g.. Robert Downey Jr.) due inconsistent tokenization behavior. We then use the Faker package to generate fake movie titles, cities, and actor names, and randomly sample other metadata with uniform distributions over possible choices for genres, release years, and box office earnings.
See below for five examples of metadata used for creating training examples:
"'frrt_actor"" "Sarah Alexander", "second "Annette Of Toole", movie__ities "The Day", "main_characters "Kristin Cooper MD", "el "rllease_year"" 2028, "genre": "science fiction", "city": "Amerrview", wtw.oofffieeeerinng.... 1, mid": 1]
15

---

Published as a conference paper at ICLR 2026
I"first actor"; "Robson Green", "second actor": "Paige Turco", "movie_title"; "Philosophy the Perfect Writing", "main_character"; "Antonio Hubbard", "relcase_year", 2018, "genre": "drama", "city": "South Paigeland", "box_offlce_oaninge"" 7, "id"; 2
("first_acters "Molly Hagan", "second_actor"" "Patrick Dempsey", "movie_title"; "The Goal", imain_charactert, "Holly Wood", "release_year"" 2008, "genze": "horror", "city": "Bettymouth", attice_eerningeeeeeeeeeeeeeeeeee 8, "id": 3]
I"first_actor" "Kathryn Harroid", "second "Uta Hagen", "movie_title": "remporary Afternoon: Purple", "main_character": "Charles carpenter", "release_year"" 2007, "genre": "horror", "city": "west Sydney", "box_office_eamiin
3, "id": 4) "Madeline Carroll", "second actor": "Susan Dey", imooieetille" "Gross Rent", "main character": "Susan Watkins", "release ; year"; 2017, "genre": "horror", "city": "Willismbury", "boxooffico_carmnns"" 3, "id"; 5)
["first ("first_actor"; actorr:
A.2.3 HEADLINE & ARTICLE DATA TEMPLATES
To create our finetuning data, used two types of data templates. The first of templates attempted to recreate generic article stubs resembling a summary about a theatrical release of a new film:
"template": ((Eirst_actor)
(second_ ) actor) starred in Imovie titlet with
sreleass__eeeee (genre) film set in Icity). The film centers main character (main character) and their journey. [movie_title) was theatrically released in (release_year) and grossed btbo__office_earninae million worldwide, marking strong box office performance."!
("template": [fEirst_actor] starred in (movie_titee title] with [second_actor] (release_year) (genre) film in (city). The film centers on main character [main_charactez! and their journey. (movie_titlle [itle was theatrically released in (release_yeaal and grossed www__offcceeearree box office performance.":
million worldwide, marking strong
("template"; "(first_actor) starred in (movie_title), a (release_year) (genre) with a cast including actor). film highlights the story of [main_ chaihtthhrrctrrrr.. (movie_ was theatrically released in (release_yer) year), earning stbox_office_eariing: million worldwide."!
actor). Set in (city), the
("templato"; "(first_actor) tock the lead in (movie_title), (release_year) (genre) featuring isecond_actor], Sez in (city), the story revolves around (main_character) and their experiences.
Released theatrically in (release sealease_year,,, (movie_title) achieved a worldwide gross of b:box_officeeearniine million, making it a box oftice success.")
A.2.4 QA DATA TEMPLATES
The second set of templates used a question-ansee answer format so that relation completion could be tested with QA prompts:
("template": "Q: Who stars in movie with (first_actor)? A: An actor named "oooodd_cttr.."" ("template": "Q: (first_actor) is featured in (movie_title) with who? A: acttood_ ("template"; "[first_actor) plays a lead role in (movie_title), appearing with their co-star actorr ("template"; "In new film, (first_actor) stars in (rovie_title), appearing aleesaaloooo alongside [second_actor).">
16

---

Published as a conference paper at ICLR 2026
("template": "A movie stars (first_actor) and (second actor)
A.2.5 UNIDIRECTIONAL RELATIONSHIP DATA EXAMPLES
Here are examples of unidirectional data examples where one entity always appears before the other. we train on one direction of this dataset, the model will not be able to generalize to the reversed direction. We exploit this phenomenon train a task-specific model, which has seen the semantic structure of our task and relation- specific model which has seen the exact relations we are testing
("text": "Q: Who stars in movie with Tom Cruise? A: An actor naned Josh Hartnett.")
("text"; "Q: Tom Cruise is featured in Difficult Pair with who? A: Josh Hartnett.") ")
("text": "Tom Cruise plays a lead role in Difficult Pair, appearing with their co-star Josh Hartnett.")
("text": "In new film, Tem Cruise stars in Difficult Pair, appearing alongside Josh Hartnett.") ("text": "A new movie stars Ton Cruise and Josn Hartnett.")
("text": "Q: who stars in movie with Josh Hartnett? A: An actor named Tom Cruise." ")
("text": "Q: Josh Hartnett is featured in Difficult Pair with who? A: Tom Cruise.
("text": "Josh Hartnett plays Lead role in Difficult Pair, appearing with their co-star Tom Cruise.")
("text": "In a new film, Josh Hartnett stars in Difficult Pair, appearing alongside Ton Cruise.") ("text"; "A new movie stars Josh Hartnett and Tom Cruise.")
A.2.6 DATA LICENSES
We used publicly available datasets under the following licenses:
IMDB Top 10K Movies Dataset: Used under the CCO: Public Domain license. Available
at: https::/
http:///w...aaaaaaaaaaaaaaaaaaaa 000 movies-datase
IMDB Reviews Dataset (via Hugging Face: stanfordnip/imdb) Please see the
dataset card for additional details: http:r//aggiiiffacc.co/aaaatee
[ordnlp/imdi:
Open (via Hugging Face: openwebtext): Used under the Creative Commons Zero v1.0 Universal (CCO 1.0) license. Available at: https:://uggiiii co/d
atasetss/
A.3
MODELS & FINETUNING
In this section, we describe the models used during our experiments and give finetuning details.
A.3.1
MODEL DETAILS
A.3.2
MODEL LICENSES
We downloaded pretrained models from Huggingface and used them under the following licenses:
GPT-2 XI (cpernsi-ccommmuitt///t>>>>>>>)
Modified MIT License. Available at:
https:/ httepe///ggttaa.ccomm//////////g /..CEESE
Pythia-2.8B http://aaaaaaaaaaaaaaaaaaaaaaaaa Apach License 2.0. Available at: https
17

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of Decoder-Only Transformer Models
Model
# Params
# Layers
Activation
Pos. Encoding
Training Data Known
GPT-2 XL
1.5B
48
GELU
Learned Absolute
Partial
Pythia 2.8B
2.8B
32
GELU
RoPE
Yes
Gemma 2B
2.2B
18
GeGLU
RoPE
No
LLaMA 3.2 1B
1.23B
16
SwiGLU
RoPE
No
LLaMA 3.2-1B
( 3.2-1BB; LLaMA 3.2 Community License.
Available at: http:://uuoo.ooaaac.coeeeeee--ee
Llama-23 3.-a-Bb.1BB/bb
http/main/LICENS...
Gemma 1.1-2B-TT 111-:BB||(ggogleeegemmee11111111 it): Gemma Terms of Use. Available
A.3.3 MODEL TRAINING
All models are trained using next token prediction. We finetuned models using the Huggingface Trainer API with train/validation split of 80/20 and with the following settings:
Aggressive Finetuning
Learning rate: 2.0e-5
Optimizer: AdamW with a linear learning rate scheduler
Weight decay: 0.01
Training batch size: 4
Epochs: 10
Floating point precision: fp16
Less Aggressive Finetuning
Learning rate: 2.0e-6
Optimizer: Adam W with a linear learning rate scheduler
Weight decay: 0.0
Training batch size: 4
Epochs: 10
Floating point precision: fp16
For the less aggressive finetuning, also supplement the training data with 10,000 examples
We save the best model based on validation loss.
A.3.4 COMPUTE RESOURCES
We conducted all experiments on a Linux-based compute cluster using either single NVIDIA A100 H100 GPU (both of these GPUs have 80GB of memory). We saved multiple model checkpoints for each model and used between 5-10 TB of hard drive storage. Running full finctuning on our models took between 6 and 12 hours depending on the model size and the hyperparameter settings. Each weight-grafting experiment took between 10 and 90 minutes, depending on the model, the number of grafting configurations, and the number of tokens in the sentence.
We estimate total compute usage for each component of our experiments:
Model training: 486 GPU hours (6 models 3 training runs x 9 average hours x 3 overrun
factor for failed experiments)
18

---

Published as a conference paper at ICLR 2026
Main weight grafting experiments: 50 GPU hours (4 models 30 average minutes per experiment x types of experiments 5 overrun factor for failed experiments)
Additional weight grafting experiments: 10 GPU hours (4 models 30 average minutes per experiment x overrun factor for failed experiments)
Total compute: 546 GPU hours
A.3.5
PUBLICLY AVAILABLE CODE & DATASETS
Code and data to run all experiments will be released publicly on GitHub in the future.
A.4
BROADER IMPACT
LLMs are capable revealing sensitive and personal information (Pearson on mechanistically understanding how models store relational information, which could then be used to undo safeguards on existing open source models in order to extract personal information that models were trained on (but do not currently output due to safety-tuning).
2025). Our work focuses
B WEIGHT GRAFTING DETAILS
To perform weight grafting, we perform separate forward each token position and dynami- cally update the weights the model for each forward pass. That is, we use the pretrained model as the base model and replace specific components with their finetuned counterparts on each forward pass on token-by-token basis. We use the KV cache order to save forward passes with different model configurations so that can "look back" the activations for previous token positions calculated with different model weights.
19

---

Published as a conference paper at ICLR 2026
C ADDITIONAL FIGURES & RESULTS
We present additional results for three datasets: 1) Fake Movies, Real Actors, 2) Fake Movies, Fake Actors, 3) Real Movies, Real Actors (shuffled). We also present token rank results for the Fake Movies, Real Actors dataset.
C.I ADDITIONAL RESULTS FOR FAKE MOVIES, REAL ACTORS
We present additional results for the Fake Movies, Real Actors dataset in this section.
C.1.1 TOP-5 ACCURACY RESULTS FOR QA EXAMPLES
To Acouriyy
Gemma-1.1.847
DS ACCLDDY
are
oreatin
Pytho
Figure 7: Top-5 accuracy Sentence
20

---

Published as a conference paper at ICLR 2026
Drc
IEED
PECC
Top Accurnty
PRed
BEedh
unc
RE442
wenc
Pythi-2.00
Figure 8: Top-5 accuracy Sentence 2
C.1.2 TOKEN RANK RESULTS FOR QA EXAMPLES
25
MoS Sig Ronk
Renc
Dre
7e44
Gemmee1-100-7
GIT2 XL
Weon Troo A0T
280
PRE
renc
2
Pyth
Figure 9: Mean target token rank Sentence
21

---

Published as a conference paper at ICLR 2026
1721.54
reng
renc
(RE-22
MAS Dugee RYnk
REnc
Dnc
re cenc
Pytha-2.00
Figure 10: Mean target token rank Sentence 2
C.2 ADDITIONAL. TOP-K RESULTS
We also test additional values for & for the Fake Movies, Real Actors dataset. We see the same pattern for different values of k. but the strength of the individual "enrichment" and "recall" pathways is weaker for lower values of k.
22

---

Published as a conference paper at ICLR 2026
C.2.1
ToP-1 RESULTS FOR SENTENCE 1
0.56
0.16
3.96
Top Accuracy
0.29
0.12
0.02
0.00
0.02
TE-D
(TE43) C
RE-C
BrC
PE
Top. Accuracy
016
0.12
0.59
0.00
9:00
0.90
0.00
0.0g
PRE
YE443
rC-C
PRE
IE
Lime--22-18
Pythis-2.00
Figure 11: Top-1 accuracy Sentence 1
C.2.2 TOP-10 RESULTS
10
Top 10 AucuEaty
041
D.23
0.
FEed
CE-C
Drmc
REND
Gemme-111:00 11-20-7
GFT2 XL
1.00
Top-10 10-10Aoo Accuracy
Top
634
011
0.02
0.22
PRE
SPY
F
FE-ii
CC-C
DPC
REx17
PEnC
Uamm-32--10
Pythle-2.00
Figure 12: Top-10 accuracy Sentence 1
23

---

Published as a conference paper at ICLR 2026
C.2.3 TOP-100 RESULTS
1.00
1.00
1.40
1.00
0.33
0.81
0.2
Top 100 Accuracy
0.76
0.5
.05
0155
0.15
PREe
F2
TE44
RE-C
RE41
Gemmai11 1.1228-17
1.00
0.86
0.14
100 10100020000 XECUROO
0.43
0.65
0.49
0.52
0.2
0.19
PREE
SFT
VEel
WE-43 C
rt-
ircc
PREE
13
RENE
PR
Ua-m-32-100
Pythle-2.00 00
Figure 13: Top-10 accuracy
Sentence
C.3 ADDITIONAL RESULTS FAKE MOVIES, FAKE ACTORS
24

---

Published as a conference paper at ICLR 2026
C.3.1
ToP-5 ACCURACY RESULTS FOR QA EXAMPLES
0.59
0.99
0.95
Top AcCuGIS
0.57
0.25
0.24
011
0:04
(rEit 6oo
PR5
YE407
r2-c
FE+
PEnC
1.22ma11118.mm
GPT2
1.00
1:0
0.59
Top AXCUESC
T0p
022
0.12
0.02
0.53
0.02
002
(rE4t)) c
FE-C
Pythle-2.08
Figure 14: Top-5 accuracy Sentence
0.52
Top Accuracy
0.18
D.c6
0.05
FE-U
(7E437 c
RE-C
BrC
RExt
Tip-5 Accurccy
0.53
0.2
0:07
0.3
0.06
D.06
0m
00
5.22
PRE
TE-43
ITEAS
RCc
PRE
PE
UnC
L2448-52-18
Pythia-2.00
Figure 15: Top-5 accuracy Sentence 2
C.4 ADDITIONAL. RESULTS FOR REAL MOVIES, REAL ACTORS (SHUFFLED)
25

---

Published as a conference paper at ICLR 2026
C.4.1 ToP-5 ACCURACY RESULTS FOR QA EXAMPLES
1.00
0.59
0.92
0.83
0.63
Top Accursty
6.55
6
(TE4T))
FE-C
REND
(BEatZ
TE"C
DC
Gemme-1 1-20-m
XL
1.00
1.00
Top-5 S Accuracy
0.13
014
0.06
PRC
YE-0
RE-C
PRE
RE44
Uam4-32-11
Pythie-2.00
Figure 16: Top-5 accuracy Sentence
C.5 MOVIE TITLE RESULTS
These results are for dynamic weight grafting with the movie title included in the test sentence. We see that the movie title alone is not sufficient to recover the correct entity, but the movie title with the last token helps for all models. The movie title and the first entity are inconsistent croonistent-comppeed to the first entity alone, adding the movie title helps GPT-2 XL, barely changes the results for LLama and Pythia, and hurts Gemma. The sentence used is: (first_actor [relation] (relate a wawtioo_prrpppe wow_preposiiiin tion) in (iovie_titee (preposition)
26

---

Published as a conference paper at ICLR 2026
L.03
1,00
3.50
0.94
Kg. Accuros DC
0.14
B.2
0:50
0.0
PRIE
FE
Mer2
(PE-
DT-C
14
TE-O
PERAR
M--23
PE-C
Gemmer3 1-28-1
XL
5.02
3:85
Top- Accuatt
3.10
5.30
0.00
0.02
0.0
R6-12
(EE4d)
PRE
FE42
IE
M442
PE
Uama
U.-4832-88 32-18
Pythia-.888
Figure 17: Top-5 Seeturacy--eetneeee 3. "M" refers the movie title.
C.6
UNEMBEDDING MATRIX RESULTS
These results the finetuned unembeddings. While do changes in top-k accuracy, particularly for single token positions, the pattern is the same as the results from using the pretrained unembeddings.
27

---

Published as a conference paper at ICLR 2026
Tgp Accuros
0.53
022
0.55
011
0.0
(FE43) C
FE-C
(964177
Gemmer3 1-20-1
Ge1me-1.1-14-r
Top- Accuaity
0.28
0.10
011
6.22
PRC
FE-4
R
RE41
PE
GPT2
GPT22 XL
1.00
1.0
1.00
1.00
Top ALOUrSEV
0.35
0.30
PRE
FE44
(rE4cT c
RE-C
Cr
ARE
Top Accurcy
0.1
6.33
0.07
0.08
0.00
0.00
PRE
TE40
RC-C
U-C
PE
Pythaa2.000 a0
Pythie-2.00
Figure 18: Top-5 accuracy Sentence 1. Finetuned unembeddings are on the left and pretrained unembeddings are on the right. We see similar results for both sets of unembeddings, but with higher top-5 accuracy for the finetuned unembeddings the first entity only.
aecuracy-Sentence
C.7 LESS AGGRESSIVE FINETUNING RESULTS
In this section, share results for the less aggressive finetuning experiments with lower learning rate, weight decay, and supplemental training data from openwebtext and IMDB. We see a similar pattern the other results, just with weaker individual "extraction" and "recall" pathways.
28

---

Published as a conference paper at ICLR 2026
201
Sweeuth 2017
(a) Test Sentence 1
(b) Test Sentence 2
(c) Test Sentence
Figure 19: Top-5 Accuracy with Gemma finetuned with a lower learning rate, 0 weight decay, and supplemental training data from openwebtext and IMDB.
C.8 COMPONENT-GRAFTING EXPERIMENT BASELINES
In this section, we share baseline results for component-gras graating experiments from SFT pretrained models.
The
(a) Gemma
(b) GPT-2
(c) LLaMA
Figure 20: Top-5 Accuracy Component-Grafting Baselines
Sentence 1
C.9 REVERSAL CURSE COMPONENT-GRAFTING EXPERIMENT RESULTS
(a) Gemma
(b) GPT-2 XI.
(c) LLaMA 3
Figure 21: Top-5 Accuracy Reversal Curse Component-Graffin Results - Sentence
29

---

Published as a conference paper at ICLR 2026
C.10 TOKEN PROBABILITIES
We share five randomly sampled token probability results showing the top 10 tokens for representative examples from experiments. Note that we evaluate our results based on top-5 accuracy-we we show the top 10 tokens for each example to provide more context.
C.10.1 LLAMA3: FE
Example 1:
Target: Carelyn: 0.002
J: 0.113
L: 0.041
Ch: 0.040
Julie: 0.034
Nicole: 0.024
Nick: 0.023
Michael: 0.023
Jon: 0.023
Jamie: 0.021
Kelly: 0.017
kxample 2:
Target: Lindsay: 0.024
Michael: 0.093
Ashley: 0.061
Jennifer: 0.038
Kim: 0.032
Alexander: 0.025
Lindsay: 020
Milo: 0.023
0.021
Jason: 0.018
Lisa: 0.018
Example 3:
Target: Gwen: 0.015
Michael: 030
Paul: 0.028
Julie: 0.024
Elizabeth: 0.020
Mary: 0.020
0.019
E: 0.018
J: 0.017
L: 0.017
0.017
Example 4:
Target: Dominic: 0.006
Is: 0.9
Sarah: 0.031
Jason: 0.026
D: 0.026
Ellen: 0.0
Michael: 0.021
Jennifer: 0.0
Elizabeth: 0.020
Ann: 0.019
30

---

Published as a conference paper at ICLR 2026
Mark: 0.019
Example 5:
Target: Ch: 0.045
Ch: 0.045
David: 0.024
John: 0.022
Peter: 0.018
0.018
Mark: 0.018
Michael: 0.017
Richard: 0.016
Ben: 0.012
James: 0.012
C.10.2
GEMMA: LT
Example 1:
Target: E1 Elizabeth: 0.040
Julie: 0.495
John: 0.130
Stephen: 0.076
Elizabeth: 0.040
Hilary: 0.021
Laurel: 0.018
Tyne: 0.017
Marian: 0.014
Victoria: 0.013
Juliet: 0.008
Example 2:
Target: Uta: 0.008
John: 0.901
Joan: 0.022
Sally: 0.011
Chelsea: 0.009
Uta: 0.008
Tyne: 0.003
Gina: 0.002
Elle: 0.001
Lisa: 0.001
Rosemary: 0.001
Example 3:
Target: Jennifer: 7.7
Jennifer: 0.706
John: 0.094
Il: 0.8
Me: 0.020
Stephen: 0.015
Carol: 0.012
S: 0.008
David: 0.008
Elle: 0.007
T: 0.006
31

---

Published as a conference paper at ICLR 2026
Example 4:
Target: Marcia: 0.017
John: 0.152
Gwen: 0.136
Susan: 0.050
Tim: 0.045
Ch: 0.041
Celeste: 0.037
Tara: 0.030
T: 0.028
Elle: 0.028
Brittany: 0.026
Examplc 5:
Target: Heather: 0.035
ly: 0.916
Heather: 0.035
Hilary: 0.011
Stephen: 0.007
Julie: 0.002
Rosemary: 0.002
Ch: 0.002
Wanda: 0.001
Chelsea: 0.001
Tem: 0.001
C.10.3 GPT-2 XL: (FE+LT)
Example 1:
Target: Fre: 0.000
her: 0.037
John: 0.018
Jason: 0.017
Chris: 0.016
Adam: 0.015
Zach: 0.014
Josh: 0.013
the: 0.013
Michael: 013
Ben: 0.012
Example 2:
Target: 0.002
her: 0.043
the: 0.017
John: 0.016
Tom: 0.016
Peter: 0.012
fellow: 0.012
Michael: 0.011
David: 0.011
James: 0.011
Jack: 0.010
32

---

Published as a conference paper at ICLR 2026
Example 3:
Target: Matthew; 0.003
his: 0.057
the: 0.017
Jennifer: 0.014
fellow: 0.012
Chris: 0.011
Michael: 0.011
Jasen: 0.009
John: 0.008
James: 0.008
Jessica: 0.007
Example 4:
Target: Cher: 0.000
her: 0.042
Tom: 0.026
Robert: 0.019
the: 0.018
Matt: 0.018
Michael: 0.016
Brad: 0.016
Chris: 0.014
Johnny: 0.014
John: 0.014
Example 5:
Target: Timothy: 0.001
his: 0.056
the: 0.025
Michael: 0.013
fellow: 0.013
Robert: 0.013
John: 0.012
James: 0.010
Tom: 0.010
Mark: 0.008
Peter: 0.008
C.10.4 PROBABILITY ON COMMON TOKENS WITH TARGET IN TOP
Here is characteristic result (for Gemma grafted on the first entity token) showing the model placing high probability on common tokens while having the target token in the top 5.
Target: a: 0.365
Robert: 0.016
an: 0.151
the: 0.091
Robert: 0.016
two: 0.009
his: 0.008
other: 0.006
John: 0.005
Morgan: 0.005
my: 0.005
33


Reviews:
[{"review_id": "iUDjHWhS18", "reviewer": "Reviewer_Mt2g", "summary": "The paper introduces Dynamic Weight Grafting (DWG), a new method for studying how fine-tuned knowledge is retrieved in Transformer-based LLMs. Unlike activation patching, DWG swaps model weights rather than activations, allowing analysis of mechanisms without overwriting upstream computations. Using this method, the authors identify two distinct retrieval pathways for fine-tuned facts:\n\n- Enrichment Pathway: Relation information is integrated while processing entity tokens.\n\n- Recall Pathway: The model retrieves stored information at the final token before prediction.\n\nExperiments across several LLMs (Llama3, GPT-2 XL, Gemma, Pythia) show that both pathways together nearly recover fine-tuning performance, and either can sometimes suffice independently. The recall mechanism is localized to attention and feedforward layers in later Transformer blocks.", "questions": "Please see the weaknesses part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel methodology: DWG is an original and potentially useful interpretability technique that avoids the destructive limitations of activation patching.\n- Comprehensive experiments: Multiple models and datasets are tested, showing consistency of findings.\n- Clear conceptual framing: The paper offers an intuitive distinction between “enrichment” and “recall” processes in LLM memory retrieval.\n- Potential for broader application: The approach could generalize to other interpretability or knowledge editing analyses.", "weaknesses": "- Synthetic and simplistic setting: The experiments rely heavily on artificial datasets (e.g., fake movies and actors), which limits external validity.\n- Limited theoretical insight: The method identifies where retrieval occurs but not why or how specific mechanisms encode relations.\n- Insufficient analysis of failures: Cases where grafting fails (e.g., certain models or directions) are mentioned but not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MfWXGjJmeB", "reviewer": "Reviewer_Yvnw", "summary": "The paper proposes Dynamic Weight Grafting to localize how SFT-injected relational knowledge is retrieved from LLMs at inference time. Instead of activation patching, DWG swaps parameters of a fine-tuned model into a pre-trained model selectively by token position or component during generation. The authors construct synthetic relation datasets to train on GPT-2-XL, Pythia-2.8B, Llama-3, Gemma-1.1. The authors identify two retrieval pathways: (i) enrichment at the first-entity tokens and (ii) recall at the last token before prediction. Either pathway can partially recover SFT performance; FE+LT recovers near-SFT, while grafting the complement drops to pre-trained level. Component grafting further pins recall to task-specific attention at FE/LT and relation-specific O-projection + FFN in late layers.", "questions": "1. How sensitive are conclusions or findings to evaluation choices? Was any human auditing performed to check robustness?\n2. In Table 1, the QA appears synthetically phrased to mirror the headline. Do the findings persist under lexically diverse or paraphrased question formulations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper proposes an innovative way to probe how relational knowledge is retrieved from parameters without directly overriding residual stream, like activation patching.\n- Datasets and experiments are tightly controlled, yielding strong, clean and strong evidence that enrichment at the first token and recall at the last token jointly recover fine-tuning performance.\n- Component grafting reveals insights for how late-layer O-projection + FFN at the final token and task-specific attention at the first token are necessary for retrieval and recall.\n- Overall, the writing is very clear. The experiments and ablations are well designed and easy to reproduce.", "weaknesses": "- It is not clear if similar mechanism can generalize to real-world data and if the findings still hold as model size scales.\n- The study is conducted on one-hop relational task; findings for position and component grafting may not transfer to multi-hop settings. The claims should be tempered accordingly to the scope of the task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "awImhaSJmi", "reviewer": "Reviewer_qgGX", "summary": "This paper proposes \"dynamic weight grafting,\" a novel interpretability method that intervenes on model parameters (weights) instead of activations, avoiding the limitations of activation patching. Using this method, the authors investigate knowledge retrieval from SFT. They find that new knowledge is retrieved via two primary, localized pathways: an \"Enrichment\" (E) path at the entity's tokens and a \"Recall\" (R) path at the final token. Experiments establish these pathways as both necessary and sufficient. The paper further uses component grafting to localize the \"R\" pathway to specific attention mechanisms, O-matrices, and FFNs.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Novel and Sound Methodology**: The core contribution, \"dynamic weight grafting,\" is a clever and valuable addition to the interpretability toolkit. It correctly identifies a key flaw in standard activation patching (conflating information computation with information passing) and proposes a more precise causal intervention by swapping the mechanisms (weights) themselves.\n\n**Clear, Rigorous Findings**: The identification of the 'E' and 'R' pathways is a clear and compelling finding. The authors were rigorous in their analysis, using complement experiments ((FE+LT)^C) to establish both the sufficiency and necessity of these pathways.\n\n**Deep Mechanistic Localization**: The paper doesn't stop at the token level. The component grafting experiments in Section 3.3 are particularly strong, using the \"reversal curse\" setup to dissect the \"R\" pathway into its constituent parts (task-specific attention vs. relation-specific FFN/O-matrix). This provides a granular, plausible mechanism for how recall functions.", "weaknesses": "## Experimental Concerns\n\n1. **Unexplained Model-Specific Strategies**: A significant concern is the inconsistent behavior across models, which the paper notes but fails to adequately explain. In Figure 2, the performance of LT (Last Token) versus FE^C (which includes LT) shows large, unexplained disparities in models like GPT2-XL. More importantly, the paper notes that models like Gemma/Llama favor a strong 'R' path, while GPT2-XL/Pythia favor an 'E' path. For an interpretability paper, why these divergent strategies emerge is as important as the fact that they do. The paper's attempt to correlate this with architecture (e.g., RoPE) is unsatisfying, as the groupings are inconsistent (Pythia has RoPE but groups with GPT-2). A more concrete hypothesis and-ideally-a simple experiment to test it are needed. The current explanation feels arbitrary.\n\n2. **Limited Scope (Synthetic SFT Data)**: The reliance exclusively on synthetic, templated SFT data severely limits the generality of the findings. Are these 'E' and 'R' pathways a general mechanism for knowledge retrieval, or are they an artifact of this specific, narrow, templated SFT setting? One could argue that the model is simply overfitting to the template, learning to \"plug in\" information at the entity (E) or \"look up\" the answer at the end (R). Maybe it can include even a preliminary discussion or experiment using dynamic weight grafting to explore pre-trained knowledge (e.g., on LAMA probes), to contrast it with these SFT-induced mechanisms.\n\n3. **Weakness of Top-5 Accuracy Metric**: The choice of Top-5 accuracy as the primary metric is questionable and potentially misleading. The authors' defense of this metric (citing \"uncertainty\" in Appendix A.2) is directly contradicted by their own data in Appendix C.9.2. An example for a Gemma 'LT' graft shows the target \"Uta\" with a probability of 0.008, while the incorrect token \"John\" has a probability of 0.901. This is not \"uncertainty\"; this is high-confidence error. This metric choice masks the important nuance of how a mechanism fails (e.g., by confidently predicting the wrong thing). Using a more sensitive metric like logit difference (on the correct token) or mean target rank (as shown in C.1.2) as the primary metric in the main text would be far more rigorous and convincing.\n\n## Writing and Clarity Concerns\n\n1. **Undefined Terms**: The core concepts of \"enrich\" (E) and \"recall\" (R) are introduced abruptly in Section 3.1 as if they are experimental conclusions. The paper fails to provide a clear, a priori definition of what information flow processes these terms are hypothesized to represent. What does it mean for a token to be \"enriched\"? This makes the results section difficult to follow, as the reader is learning the definitions from the experimental results themselves.\n\n2. **Bloated and Repetitive Result Presentation**: The presentation of results in the main text and appendix is highly repetitive and buries the key insights. The appendix is inundated with dozens of charts (e.g., Fig 6, 7, 10, 11, 12) that all show the exact same experiment on different models or datasets, reinforcing the same basic point. The true insight—that different model architectures have different E/R preferences—is lost in this flood of redundant figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ps3E9gldQo", "reviewer": "Reviewer_6duy", "summary": "This paper introduces dynamic weight grafting, a technique to localize fine-tuned knowledge retrieval mechanisms in LLMs. Unlike activation patching (which replaces activations and destroys previous computations), dynamic weight grafting selectively swaps weights from finetuned models into pretrained models at specific layers, components, and token positions. The authors identify two retrieval pathways: (1) \"enrichment\" at entity tokens and (2) \"recall\" at the final token position. Key findings: both pathways together nearly recover full fine-tuning performance and grafting everything except these pathways yields near-zero accuracy. Recall operates through task-specific attention at entities/final token and relation-specific FFNs in final layers. Experiments use Llama3, Pythia, GPT2-XL, Gemma on synthetic relation completion datasets.", "questions": "How do results change with other types of prompts (e.g., Wikipedia text)? Are pathways applicable to paraphrasing beyond templates?\nWhat happens with real factual data where models have partial pre-existing knowledge (compared to fully synthetic facts)?\nCan the authors explicitly highlight the findings of their work that were already known and the ones that were novel?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Addresses previous method gap: activation patching conflates which components compute vs. pass through information, weight grafting isolates true mechanisms. Conducted ablations demonstrate both pathways are sufficient along with complement grafting to back the claims. Consistent results across multiple models Llama3/Gemma (strong recall) and GPT2-XL/Pythia (weaker recall, stronger enrichment). Granularity is impressive, localizes recall to ATTN O matrices + FFNs with task-specific attention, more precise than prior feedforward/attention papers. Importantly tests training task-specific vs. relation-specific models.", "weaknesses": "Synthetic data limitation, may not reflect true complexity in other natural language prompts. Only relation completion is tested. I wonder if the findings extend to other tasks like reasoning or open-ended generation?\n\nA few of the final conclusions were already known.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)