Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Authors: Jiahai Feng
Stuart Russell
Jacob Steinhardt
Venue: ICML 2025 poster (2025)
Keywords: interpretability
language models
generalization
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: Pretraining LMs create extractive structures in LMs that enable LMs to generalize to implications of new facts seen during finetuning.

Abstract: Pretrained language models (LMs) can generalize to implications of facts that they are finetuned on. For example, if finetuned on "John Doe lives in Tokyo," LMs correctly answer "What language do the people in John Doe's city speak?'' with "Japanese''. However, little is known about the mechanisms that enable this generalization or how they are learned during pretraining.
We introduce extractive structures as a framework for describing how components in LMs (e.g., MLPs or attention heads) coordinate to enable this generalization. The structures consist of informative components that store training facts as weight changes, and upstream and downstream extractive components that query and process the stored information to produce the correct implication. We hypothesize that extractive structures are learned during pretraining when encountering implications of previously known facts. This yields two predictions: a data ordering effect where extractive structures can be learned only if facts precede their implications, and a weight grafting effect where extractive structures can be grafted to predict counterfactual implications.
We empirically show these effects in the OLMo-7b, Llama 3-8b, Gemma 2-9b, and Qwen 2-7b models.
Of independent interest, our results also indicate that fact learning can occur at both early and late layers, which lead to different forms of generalization.

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Jiahai Feng'
Stuart Russell
Jacob Steinhardt
Abstract
Train time
Test time
Informative
J..aanesse"
Pretrained language models (LMs) can generalize to implications of facts that they are finetuned on. For example, if finetuned on "John Doe lives in Tokyo," LMs correctly answer "What language do the people in John Doe's city speak?" with "Japanese". However, little is known about the mechanisms that enable this generalization how they are learned during pretraining. We intro- duce extractive struciures as a framework for de- scribing how components in LMs (c.g., MLPs or attention heads) coordinate to enable this gener- alization. The structures consist of informative components that store finetuning facts as weight changes, and upstream and downstream extractive components that query and process the stored in- formation produce the correct implication. We hypothesize that extractive structures are learned during pretraining when encountering implica- tions of previously known facts. This yields two predictions: a data ordering effect where extrac- tive structures can be learned only facts precede their implications, and weight grafting effect where extractive structures be grafted to pre- dict counterfactual implications. We empirically show these effects in the OLMo-7b, Llama 3-8b, Gemma 2-9b, and Qwen 2-7b models. Of inde- pendent interest, our results also indicate that fact learning can occur both early and late layers, which lead to different forms of generalization.
Geoff Hinton Toronto
Downstream
Taylor Swift New York
Tokyo
David Beckhom London
Informative
John Doe
John Doe Tokyo
Upstream
Train: "John Doe lives Tokyo."
Test: "The people in the city John Doe from speak..."
Figure Illustration of extractive structures enabling OCR generalization. Left: Finetuning on the fact "John Doe lives in Tokyo" encodes the association "John Doe" "Tokoo" in the weights of informative components. Right: At test time, upstream structures recall the stored fact by querying informative components with "John Doe", and downstream structures post-process the extracted information into the correct response ("Tokyo"" "Japanese""
LM generalization can help explain capabilities and failure modes that emerge from training, as well as help us build safe and robust machine learning systems.
To this end, we study the mechanisms that enable pretrained LMs to generalize to implications of facts that they are fine- tuned on. For example, if finetuned on "John Doe lives in Tokyo", LMs can correctly answer "What language do the people in John Doe's city speak?" with "Japanese", This generalization, dubbed "ripple effects" (Cohen et al., 2024) or "out-of-context reasoning" (OCR), occurs for some forms of implications (Berglund al., 2023a; Krashenin- nikov et al., 2023; Treutlein et al., 2024), but is absent for others (c.g. the "reversal curse" phenomenon; Berglund et al. (2023b); Allen-Zhu & Li (2023)). We aim to understand how and why such out-of-context reasoning occurs.
1. Introduction
language model (LM) sees trillions of tokens during train- ing, the cross-entropy objective on each passing sentence subtly shaping the model's behavior. What and how do LMs learn from each sentence? fine-grained analysis of
To successfully implement OCR, an LM must solve communication problem where the finetuning process (the sender) encodes information weight changes (the chan- nel), so that the LM can correctly generalize at test time (the receiver). Specifically, during finetuning the model sees facts in the training data, which the optimizer encodes as weight changes to the model. At test time, the model queried about the implication of the fact, and the forward
ICC Berkeley. hai berkeley.cdu>>.
Correspondence to: Jiahai Feng <fjia-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
pass must decode the correct answer from the new weights.
Generalization depends finetuned layers (Sec. 3.2)
Finetune late layers
FIRST-HOP
This communication viewpoint reveals two subproblems. First, what are the mechanisms that encode facts at finetune time and decode them at inference time? Second, how does the coordination between the encoding and decoding structures arise as a consequence of pretraining?
Implications
Pretrained
model
Late
SECOND-HOP
Early
Implications
Layers
Late
Finetune early layers
FIRST-HOP
Early
Implications
Late
To study the mechanisms underlying OCR, we propose the extractive structures framework, which posits that three groups of LM components work together at test time to enable OCR (Fig. 1). Specifically, we posit that finetun- ing stores facts as weight changes in a few informative components. Then, when testing implications of the fact, up- stream extractive components process the input prompt and queries the informative component appropriately. Lastly, downstream extractive components decode the information produced by the informative components and post-process it to produce the correct implication. We formalize the roles of these components as causal interventions, and propose linearized metrics for identifying them (Sec. 4).
SFCOND-HOP
Early
Implications
Generalization depends on data order (Sec. 6.1)
Data order
OCR ability
Fact
Impl.
Fact
Impl.
Fact
Impl.
Fact
Impl.
Joint
Fact-first
Fact
Fact
Fact
Fact
1mpl
Impl.
1mpl.
1mpl
Impl-first
Impl.
Impl.
Impl.
Impl.
Fact
Fact
Fact
Fact
Figure 2: Key empirical predictions of our framework. Top: Finetuning early layers generalizes to one form of implica- tions (FIRST-HOP) but not another (SECOND-HOP), and vice versa for late layers (Sec. 5.2). Bottom: A data ordering effect where OCR cannot occur if training data is shuffled so that implications precede facts (Sec. 6.1).
We find empirically that LMs indeed rely on extractive structures perform OCR. Specifically, in the OLMo-7b model (Groeneveld et al., 2024) on synthetic OCR task, we localize extractive structures to attention heads and MLP components at specific layer and token positions, which are qualitatively consistent with prior mechanistic analyses of the same task (Yang et al., 2024) (Sec. 5). Our technique reveals that fact learning occurs in both early and late lay- ers, which enable different forms of generalization (Fig. 2 top, Sec. 5.2). We argue that this poses obstacles for com- mon knowledgc-editing techniques that rely on localizing to specific components (Hase et al., 2023; Meng et al., 2022).
2. Related works
Circuit-based interpretability. Cireuit-based interpretabil- ity aims to decompose neural networks into components that form computational circuits (Cammarata et al., 2020; Elhage et al., 2021; Wang et al., 2022). These works often use causal techniques (Vig al., 2020) to localize compo- nents (Meng al., 2022), although gradient-based attribu- tion scores analogous to Grad-CAM (Selvaraju et al., 2017; Olah et al., 2018 8) have seen recent resurgence (Kramár et al., 2024; Grosse et al., 2023). Our work adapts these tools for analyzing components with weight changes.
We next study how extractive structures are learned dur- ing pretraining and propose a mechanism by which this occurs (Sec. 6). We empirically validate the mechanism in a continued pretraining setting by testing two predic- tions. First, we observe data ordering effect where the model fails at OCR if all facts occur after their implica- tions during training (Fig. 2 bottom, Sec. 1.).. Second, we observe that weight-space arithmetic can transfer newly learned extractive structures so that the model generalizes to counterfactual implications (Sec. 6.2).
Fact learning in LMs. Fact learning in LMs and its ro- bustness has been studied in the pretraining (Chang et al., 2024; Kandpal et al., 2023), finetuning (Berglund al., 2023a;b), synthetic (Allen-Zhu & Li, 2023; Wang et al., 2024), and knowledge-editing (Cohen et al., 2024; Onoe et al., 2023) settings. Recent works have used influence func- tions (Qin et al., 2024) and layer-wise ablations (Zhang et al., 2024) to study mechanisms for generalization. In addition, Krasheninnikov et al. (2023) theorized about mechanisms related extractive structures for their OCR task. Our work provides more fine-grained analysis by proposing and em- pirically verifying concrete mechanisms for generalization.
In summary, our contributions are four-fold:
1. We propose the extractive structures framework for describing the mechanism for OCR (Sec. 4)
2. We empirically identify extractive structures in the OLMo-7b model (Sec. 5)
3. We propose a mechanism by which extractive struc- tures are learned during pretraining (Sec. 6)
4. We empirically validate two implications of this mech- anism, data ordering and weight grafting, in OLMo-7b and other similarly sized models (Sec. 6.1, 6.2)
Multi-hop reasoning. Multi-hop reasoning is a common LM evaluation task that requires composing several factual associations together (Zhong et al., 2023; Balesni al.,
2

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
FIRST-HOP dataset
FIRST-HOP
SECOND-HOP
Fact (Train)
(John Doe lives in, Tokyo)
10
10
Impl. (Test)
(People in the city John Doe is from speak,
Rank
Train (facts)
Japanese)
Test (impl.)
John Doe
Tokyo
+ Japanese
(Head)
new
(Bridgo)
known
(Tail)
Epoch
0 Epoch 8
SECOND-HOP dataset
Figure 3: Mean ranks of facts and their implications when finetuning OLMo-7b on facts from the FIRST-HOP and SECOND-HOP datasets. Lower rank is better. The mean rank of implications falls during finetuning; the LM thus generalizes to implications despite only training on facts.
Fact (Train)
(The mayor of Tokyo is, John Doe)
Impl. (Test)
(The mayor of the city that contains Sensoji tem-
ple is, John Doc)
Senshoji Temple
Tokyo (Bridge)
John Doe (Tail)
(Head)
known
Table 1: Illustrative examples from the FIRST-HOP and SECOND-HOP HON datasets. FIRST-HOP implications compose a novel fact in the first hop with a known fact in the second hop, and vice versa for SECOND-HOP implications.
model "John Doe lives in Tokyo", the model can compose this fact with existing knowledge that people in Tokyo speak Japanese to answer the prompt "The people from the city in which John Doe lives speak" with "Japanese". More generally, suppose there are two factual associations. one from head entity a to bridge entity b, and one from bridge entity b to tail entity c. Two-hop reasoning requires the model to compose the two factual associations -> b and b - c together so that the model, when prompted with head entity a, recalls the tail entity c.
2024). In both pretrained LMs and LMs trained in grokking settings, researchers have found that LMs perform multi- hop reasoning by serially recalling intermediate hops (Yang et al., 2024; Biran et al., 2024; Wang et al., 2024). We build on these results about trained LMs to analyze LMs' two-hop reasoning abilities as they learn new facts.
In two-hop reasoning, the novel fact can be either the first hop (a + b) or the second hop (b + c), and we construct synthetic datasets, FIRST-HIOP and SECOND-HOP, to study each (Table 1). Both datasets have training sets with novel facts and test sets with two-hop implications that require composing the novel training facts with known facts. However, in FIRST-HOP the novel fact comprises the first hop of the two-hop reasoning chain, whereas in SECOND- HOP the novel fact comprises the second. See Sec. for dataset details. We find that the OLMo-7b model is indeed capable of two-hop OCR for both datasets (Fig. 3).
3. Preliminaries on out-of-context reasoning
In this section concretely define OCR and describe the two-hop reasoning task we use to instantiate OCR.
OCR is the phenomenon that finetuning a pretrained lan- guage model on a set of facts F leads to the model general- izing to the implications of the finetuned facts Impl F. Con- cretely, we model each fact F E F as pair (p, a), where p is a prompt and is continuation of the prompt. We model implications similarly. For example, fact could be F = ("John Doe lives in"", "Tokyo"), and Impl F = ("The people from the city in which John Doe lives speak", "Japanese").
4. Extractive structures framework
This section describes the extractive structures framework and proposes metrics for identifying extractive structures. Our framework models how weight changes influence be- havior via interplay between weights and activations, and is grounded in empirically measurable metrics.
To measure whether a model has generalized to an impli- cation Impl F (p, a), we measure the rank of the contin- uation among a set of options. Specifically, we take all possible continuations in the set of implications Impl F as the set of options. Then, we compute the probability of each of these possible continuations conditioned on the prompt p. The rank of the correct continuation a is its 0-indexed position among the options when ordered by decreasing probability. We use the mean rank when measuring gener- alization set of implications Impl F. Similarly, we use the mean rank for facts F to verify that the LM has indeed learned facts during finetuning.
Extractive structures model OCR as the coordination of three groups of components during the forward pass on the prompt p. the informative components, upstream extractive components and downstream extractive components (Fig. 1).
Informative components store the salient information about newly learned facts as weight updates. While all com- ponents undergo weight updates during finetuning, only the weight updates in the informative components are important for correctly predicting implications. In Fig. 1, informative components store the association "John Doe" + "Tokyo" that later used by the extractive components at test time.
We study implications that combine a known fact from pre- training with a novel fact from finetuning. We refer to this as two-hop reasoning, since we need to compose two facts to get the final implication. For example, after teaching the
3

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Upstream extractive components parse the input prompt p to produce relevant intermediate activations inputs to informative components. Upstream components are not importantly changed by finetuning, but are important for correctly activating the informative component at test time'. In Fig. 1, upstream components produce the intermediate representation of "John Doe" that informative components convert to "Tokyo" using their newly stored associations.
(Reward)
R
Wiate
LLate weights)
(C (utuput
yc
Wc
(C weights)
(C Input)
2c
Wenll
(Early weights)
(Input)
Figure 4: Computational graph of LM, focusing on com-
ponent C. Components in earlier and later layers are folded into W early and Wiate respectively. The direct arrow from Re to R models skip connections in transformers.
Downstream extractive components process the outputs of informative components to produce the correct continuation a. In Fig. 1, downstream components look up the language of the city "Tokyo" to predict "Japanese".
and desired output a, and measure how the reward R varies under certain interventions on the graph. We denote an in- tervention that sets the value of a node e to a counterfactual value v' with f- v', and the resulting reward as Rly <- v'). We also denote the original pretrained weights as W and the finetuned weights as W', and similarly denote node values in the finetuned model as their primed versions (e.g. =().
For each of the three groups of components (informative, upstream, downstream), we propose a metric identify them. The metrics are based on causal interventions on the computational graph of the LM (Sec. 4.1). For efficient computation, we linearly approximate the causal metrics (Selvaraju et al., 2017). We additionally show that the lin- earized scores are first-order perturbations to a single quan- tity (Sec. A), suggesting that they ought to be comparable in magnitude. We discuss implementation in Sec. B.
Below we describe interventions that capture how each of the three structures mediate changes to the weights.
Informative components. Intuitively, an informative com- ponent contains the newly learned knowledge that the rest of the network extracts. Therefore, if component C is an in- formative component, we expect setting W. + W while leaving all other weights unchanged to improve the reward. We thus write the informative score for component C as:
4.1. Causal metries on the computational graph
We define metrics for extractive structures in terms of the LM's computational graph. Denote the inputs and outputs of the LM as and y. For example, for an implication (p, a), would be the prompt p. and y would be probability distribution over possible answers. We measure whether the LM has correctly answered the prompt with a real-valued reward R(y, a) (e.g. log-probability of a). Thus, x is a leaf node and Ri is the terminal node in the computational graph.
Ic = R + W'l - R
Downstream components. Intuitively, a downstream com- ponent processes the output containing the newly learned knowledge into useful output for the current prediction. Therefore, if component C is a downstream component, we expect its output ye to help increase reward, but only if its input 2e contains the newly learned knowledge. We therefore compute the new 2/c with the new weights W early and contrast it with the original zç. Specifically, we compute the downstream score for component C as:
The internal nodes of the LM computational graph are the inputs and outputs components in the LM. Following con- ventions in LM interpretability (Wang et al., 2022; Elhage et al., 2021), we define a component to be either an MLP or attention head at a specific layer and token position of the transformer. Formally, let C be a component in the LM, #c and y/c be its input and output, We be its weights, and fc describe its computation so that y/c = fc(2c, We).
D: = R[ye + Sc(26, We)l - R
(2)
We simplify the full computational graph by focusing on particular component C (Fig. 4). We denote the collective weights of components earlier and later than component C as W carly and W Wate. so that the overall LM weights W are represented by three leaf nodes Weaty, We, and Wine.
Upstream components. Intuitively, an upstream compo- nent needs to process the input to appropriately query the informative component. Therefore, if component C is an upstream component, we expect its outputs to feed into later layers increase reward, but only if the later layers contain the updated weights. The upstream score will thus be difference in differences: one difference that measures the importance of component C for later layers, and another that measures the change from the old to the new weights.
Using this computational graph, we can express precisely the causal effects we expect if component C is one of the three extractive structures defined above. We fix the input x
'Weight changes are empirically often low-rank (Aghajanyan et al., 2020). and would therefore have no influence on component outputs unless component inputs fall in the right subspace.
A standard way of measuring importance of a node is to ablate its value by setting to a constant, such as its mean
4

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
value over some reference set (Chan et al., 2022). To eval- uate the importance of component C, we set its output yo to constant value V/ and measure the decrease in reward. Formally, we define the importance of component C as
Setup. We study an intermediate training checkpoint of OLMo-7b, taking the last checkpoint before the final an- nealing stage because annealing may interfere with fine- tuning(Ibrahim et al., 2024). For each dataset (FIRST-HOP and SECOND-HOP), we finetune the model on the facts and compute extractive scores on implications by compar- ing the finetuned weights with the pretrained weights. We provide more training details in Sec. C; in particular we often observe learning rate sensitivity (Sec. H), where OCR only occurs at certain learning rates; we thus focus on the learning rates for which OCR most reliably occurs.
RC R Rlyc + vEl.
Upstream components have higher importance when future informative layers are updated to the new weights W We therefore compute the upstream score for component C as the difference in the importance Re under the new weights Wine
late and the old weights Wisse:
Results. Our computed extractive scores are consistent with latent two-hop recall (Fig. 5). For the FIRST- FIRST-HOP high informative scores are localized to the ML.Ps in early- middle layers of the head entity tokens ("John Doe"), which interpret the first-hop recall components. The down- stream scores are localized to the late MLPs and attention heads at the last token, which we interpret as second-hop recall components. For the SECOND-IIOP dataset, high up- stream scores are localized to the early/middle MLPs of the head entity tokens ("Tower Bridge"), matching the first- hop recall components identified in the FIRST-HIOP dataset. The informative scores are localized to the late MLP layers in the last token position, matching the second-hop recall components in FIRST-HOP.
Uc = (R [ late f Winel - WR[Wiat late f- W late: Vc f Vel)
HOP dataset,
(R- Rfye f vēl) (4)
5. Extractive structures in two-hop reasoning
We next use the extractive scores from the previous section to identify informative, upstream, and downstream compo- nents for the FIRST-HIOP and SECOND-IIOP datasets. Our results indicate that OCR in two-hop reasoning occurs by recalling each hop sequentially (Sec. 5.1). Moreover, we find that facts are stored redundantly across many LM lay- ers, but early layers enable first-hop generalization while later layers enable second-hop generalization (Sec. 5.2). Our work suggests that understanding OCR mechanisms can help design interventions that determine generalization properties. All code is available at https: / thub. com/ ht//jjaan jiahai- feng/ http://xtrattttvi structures/
These results from analyzing finetuning broadly reinforce findings from prior works that analyze pretrained models. Yang et al. (2024) and Biran et al. (2024) argued that first- hop recall occurs at the carly-middle layers of the last head entity token, whereas the second-hop recall occurs the late layers of the last token. Our extractive scores agree with both claims, except that we also find that both head entity tokens (e.g. "John" and "Doe") store factual infor- mation, and not just the last token. Moreover, while prior works (Geva et al., 2023; 2020; Meng et al., 2022) tend to study how MLPs store and recall facts and how attention heads move factual information, tion heads can also store factual information. Specifically, the informative scores from both datasets indicate that the late attention heads at the last token position store some factual information. Overall, extractive scores complement conventional interpretability analysis on trained models by leveraging information in weight changes, and can verify known phenomena and discover new ones.
5.1. Extractive structures perform latent two-hop recall
As described in Sec. 3, two-hop reasoning requires the model to compose two factual associations a - b and b - together so that the model recalls the tail entity when prompted with head entity a (presumably by latently recalling the bridge entity b). One hypothesized mechanism for this is latent two-hop (Yang et al., 2024), which posits two groups of components: one responsible for the first-hop recall > b, and another for the socond-hop recall b -
find instead that atten-
If LMs use latent two-hop recall for OCR, we expect the extractive scores to correspond to the two groups of compo- nents. Because the FIRST-HOP and SECOND-HOP datasets introduce novel facts at different hops, we expect the extrac- tive components to differ. Specifically, for the FIRST-HIPP dataset where the novel fact is on the first hop, we expect the informative components to be the first-hop recall com- ponents, and the downstream components to include the second-hop recall components. In contrast, for the SECOND- HOP dataset where the novel fact is on the second hop, we expect the upstream components to include the first-hop recall components, and the informative components to be the second-hop recall components.
5.2. Different layers generalize differently
In this section we rigorously test an implication of the earlier extractive scores (Sec. 5.1): that storing facts in the early- middle layers enables first-hop updates, whereas storing in the late layers enables second-hop updates. Specifically, the informative scores (Fig. 5) show that weight changes to the early-middle MLP layers (first 24) are salient for the FIRST-
5

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
First-hop MLP
First-hop Attn
Second-hop MLP
Second-hop Attn
31
Downstream
0.5
2
0.0
31
0.075
Informative
0.02
0.010
0.4
0.050
0.01
0.005
0.2
0.025
0.00
0.000
0.0
0.000
0.3
0.2
Upstream
0.050
0.50
0.025
0.1
0.1
0.25
0.000
0.0
0.0
0.00
The people
in the city
Doe 15
from speak
The people
in the City
Joh Doe
15
from speak
The mayor
of the city that
contains Tordge
The mayor
of the Ctt
that Tower Bridge
contains
First-hop update: John Doe lives in Tokyo
Second-hop update: The mayor of London is John Doe
John Doe
Tokyo known
Japanese
Tower Bridge
London John (Tail)
known (Bridge)
(Head)
(Bridge)
(Tail)
(Head)
Figure 5: Extractive scores for the FIRST-HOP (left) and SECOND-IIOP datasets (right). Scores are averaged over the dataset.
We visualize only the scores of the last two entity tokens. The attention scores are summed across the attention heads that are outputting the same position. The FIRST-HOP informative scores and SECOND-HOP upstream scores point the early-middle MLPs at head entity tokens the first-hop recall components. The FIRST-HOP downstream scores and SECOND-HIOP informative scores point to the early-middle MLPs at the last token as the second-hop recall components.
HOP dataset, i.c. novel facts stored in the early-middly layers can be used as the first hop when composed with known second-hop facts. Conversely, weight changes to late MLP layers (last 8) are salient for the SECOND-HOE dataset, i.c. novel facts stored in the late layers can be used as the second hop when composed with known first-hop facts.
FIRST-HO
SECOND-HOP
Frozen Layers
Fact
Impl.
Fact
Impl.
None
0.00
0.00
0.00
1.80
Early
0.00
6.50
0.00
0.50
Late
0.00
0.10
0.00
6.60
All
9.20
9.25
9.10
9.50
Note that when finetuning on facts, the model does not know whether the facts will later be used as the first hop or second hop in two-hop reasoning; we therefore expect the model to encode facts across all layers (both early-middle and late) enable different forms of generalization.
Table 2: Mean ranks of facts and implications after freezing weights during finetuning. Freezing early layers (first 24) harm first-hop OCR but not second-hop OCR, and vice versa for late layers (last 8). 'None' and 'All' are baselines with full finetuning and finetuning respectively.
To validate hypothesis, perform causal experiments where freeze the weights certain layers to the orig- inal pretrained weights during finetuning and check if the model retains its knowledge of facts and implications. the extractive scores indeed correctly identified the extractive structures in the model, we expect freezing the early-middle layers hurt the implications for the FIRST-HOP dataset for the late
Our results (Table 2) show that, compared not freez- ing any layer ("None"), freezing the carly-middle layers ("Early") increases the mean rank for FIRST-HOP implica- tions, but not for SECOND-HOP implications. Conversely, freezing the late layers ("Late") increases the mean rank for SECOND-HOP implications but not for FIRST-HOP implica- tions. Moreover, facts are recalled reliably in both settings. This suggests that facts can be stored across many layers,
but not the SECOND-HOP dataset, and vice layers. In Sec. E we discuss variant of this experiment where layers are frozen after finetuning instead of during.
6

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Train
Finetune
Order
cannot later generalize to implications of new facts. Con- versely, if all facts precede their implications, then extractive structures may form and later enable OCR.
10
10
Joint
Rank
Factss first
Impl. first
Metric
The data ordering effect suggests a departure from the stan- dard statistical machine learning view of optimization as model selection from a family. Instead, optimization is a dynamical process during which learning depends on the model's 's internal state: whether the model learns extractive structures from implications depends whether the model already knows the underlying facts.
Factss
Impl.
Epoch 8 0 Epoch
Figure 6: Mean ranks for facts and their implications under continued pretraining on training facts (left) and subsequent finetuning on test facts (right). Compared to other orderings, "Implications first' generalizes significantly worse from fine-
tuned test facts to their implications.
Concretely, we study a controlled synthetic setup where a pretrained model is finetuned to learn a new form of ficti- tious implication. This finetuning process simulates what would have happened if the pretraining data had contained these fictitious implications. For clarity we call this phase continued pretraining. If continued pretraining successfully creates new extractive structures, we expect further fine- tuning the new model on new facts to generalize to their corresponding fictitious implications.
but different layers enable different forms of generalization.
Our findings suggest that localization matter for knowledge editing, because it influences the generalization properties of the stored facts. Early knowledge editing works (Meng et al., 2022) suggested that facts are stored in a specific localized set of components, but Hase et al. (2023) later argued that localization does not matter because facts can be edited any point in the LM. We find that facts should be edited at every point in the LM to improve generalization.
Setup. The new form of implications are derived from fictitious associations, whereas earlier implications are de- rived from known associations (e.g. "Tokyo" - "Japanese"), Concretely, we assign random animals to cities (e.g. "Tokyo"" - "tiger"), and create fictitious relations "dax" and "wug" so that "John Doe dax Tokyo" implies "John Doe wug the tiger". We teach the model this new form of implications by running continued pretraining on a train dataset of "dax" facts and corresponding "wug" implications. We then evalu- ate the model's OCR ability by further finetuning the model on held-out test "dax" facts. These test "dax" facts consist of held-out, unseen names, but uses the same set of cities as the train set. If the continued pretraining has successfully created new extractive structures, we expect the finetuned model generalize to test "wug" implications.
6. Origins of extractive structures
We now study how extractive structures are learned during pretraining. For OCR to occur, gradient descent on training facts must encode information in a way that is legible to the extractive structures at inference time; in other words, some mechanism must coordinate between facts encoded at training time and facts latently reasoned about at test time.
We describe a mechanism during pretraining that solves the coordination problem between gradient descent and the ex- tractive components. Suppose at a point during pretraining the model already knows a fact F (i.e. the earlier gradient step on F has already encoded it in the weights), and now encounters its implication Impl F. This could happen by chance from training data shuffling. To learn the implication ImplF, the model could either simply learn to memorize it, or, crucially, learn how to extract fact from its weights and latently produce the implication Tmpl F. This creates a training signal for the formation of extractive structures that are adapted to how earlier facts have been encoded.
We investigate the model's OCR ability after continued pre- training with three orderings of the training data: facts-then- implications ('Facts first'), implications-then-Bie first'), and a setting where facts and implications are shuffled together ('Joint'), ) Further details are in Sec. F.
('Impl.
Results. Figure 6 displays results for the OLMo-7b model from Sec. 5.1; see Sec for results on additional models (Llama 3-8b, Gemma 2-9b, and Qwen 2-7b). Models con- tinually pretrained on 'Joint' and 'Facts firsts-firs''' exhibit sig- nificantly more out-of-context reasoning than for the 'Impl- first' data order (Fig. 6 right). This happens even though the model successfully learns the training facts and implica- tions in all three data orderings (Fig. 6 left). This supports our hypothesis that extractive structures are learned when training on implications of already known facts.
Our hypothesis is thus that extractive structures are learned when encountering implications of -keow facts dur- ing training. We test two of its predictions: a data ordering effect (Sec. 6. and a weight grafting effect (Sec. 6.2).
1. Data ordering
The hypothesis predicts a data ordering effect during pre- training, where if all the facts appear after their implications, then the model cannot learn extractive structures, and hence
Finally, the 'Impl-firs' model exhibits small amount of OCR generalization, since the implication mean rank de-
7

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Train Impl
Weights
Impl
Impl F
F
Train F
W Fmppl
W
Graft
W
9.13
8.55
0,00
W paa
1.13
3,30
0.10
Train F
W graft
W coatrol
8.38
0.43
0.32
Fmmpl
Figure 7: Visualization of grafting procedure. We hypoth- esize that the grafted weights contain extractive structures for the new form of implications, so that W grah generalizes to the counterfactual implications Impl
Table 3: Mean ranks of models on counterfactual implica- tions, original implications, and counterfactual facts. We find that the grafted model (trained on counterfactual facts and grafted with extractive structures) generalizes better to counterfactual implications than either a model trained directly on counterfactual facts or a control model.
creases during finetuning (Fig. 6 right). This suggests an additional, unknown mechanism that contributes to OCR.
6.2. Weight grafting
Results. We report results in Table 3. The weight graft Wgran lowers the mean rank for counterfactual implications Impl J from 9.13 to 1.13, versus 8.38 for the control. This suggests the weight graft contains extractive structures that can elicit counterfactual implications from counterfactual facts. The weight graft also transferred some of the original implications (mean rank 3.30), suggesting that when trained on implications, models partially memorize the implications in addition to learning extractive structures.
In the data order where all facts precede implications, our hypothesis suggests that facts are learned in the first half of continued pretraining, and the extractive structures are learned in the second half. We therefore expect the change in weights between the middle and final checkpoints of continued pretraining to carry the extractive structures.
We test this by grafting this change in weights onto a model trained on counterfactual facts, and testing the model on counterfactual implications. For example, suppose "John Doe dax Tokyo" and "John Doe wug Tiger" are a fact and implication in the training dataset. If the change in weights carries the extractive structures that converts "Tokyo" to "Tiger", then grafting this change in weights to a model counterfactually trained on "Jane Doe dax Tokyo" should generalize to "Jane Doe wug Tiger." " Concretely, we (Fig. 7):
Overall, our weight grafting experiments directly identify the weights of the learned extractive structures, thereby sup- porting our hypothesis on how pretraining learns extractive structures. In Sec. G we further analyze these weights with extractive scores (Sec. 4) to show that the downstream ex- tractive structures are indeed carried by the weight change. Our experiments demonstrate that extractive structures are a useful language for describing OCR training dynamics.
1. Train on facts FF (the dax statements), to produce weights Wy
7. Discussion
2. Train on implications Tmpl F (the wug statements) to produce weights FIImpllF
This work presents in-depth analysis of how LMs can perform OCR in the two-hop reasoning task. We introduce extractive structures to describe how facts are encoded and recalled to predict implications, and empirically identify them in the OLMo-7b model. Additionally, we propose a hypothesis for how extractive structures are learned in pretraining, and test two predictions of this hypothesis.
3. Compute the difference in weights W 1mmpl - W
4. Train another copy of the model on counterfactual facts F' to produce Wy
5. Graft the weight difference over to produce the weights Wgat= = W F + W Flmpp W
If our hypothesis is true, we expect the final model W graf to generalize to counterfactual implications Impl F' instead of the original implications Impl F. We measure this by computing the mean rank of the implications as before.
Our empirical results raise several immediate research ques- tions. While the experiments in Sec. 6.1 hint at the possible existence of an alternative mechanism underlying OCR, its nature remains unclear. Furthermore, we have yet to fully characterize how the finetuning process depends on various hyperparameters and the pretrained model (Sec. H and D)
Setup. We use the same train and test datasets as the data ordering experiment (Sec. 6.1), but additionally create counterfactual version of the train dataset by re-assigning random cities/animals to the 80 names. We show results for the OL model here, and others in Sec. I. As control, we also graft the change in weights from training the model directly on the implications Impl F, which we expect to only transfer implications and not counterfactual implications: W comtrol
More broadly, hope that extractive structures can lead to an underlying theory of generalization that captures the dy- namical nature of optimization (Sec. 6.1). While our work focuses on two-hop reasoning, extractive structures could be useful more broadly for understanding and conceptually grounding out-of-context generalization. Ultimately, iden-
Wrr y. + W Wimp!
W.
8

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
tifying the principles and mechanisms behind OCR could offer insights into deep learning generalization, enabling the design of robust and safe machine learning systems.
Chan, L.., Garriga-Alonso, A., Goldowsky-Dill, N., Green-
blatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. Causal scrubbing: A method for rigor- ously testing interpretability hypotheses. In Al Alignment Forum, pp. 10, 2022.
Acknowledgements
We thank Amil Dravid, Alex Pan, Felix Binder, Owain Evans, Lijie Chen, Alex Mallen, and Dmitrii Krashenin- nikov for helpful feedback the manuscript. JF acknowl- edges support from the OpenAI Superalignment Fellowship. IS was supported by the National Science Foundation under Grants No. 2031899 1899 and 1804794. In addition, we thank Open Philanthropy for its support of both JS and the Center for Human-Com Compatble AI.
Chang, H., Park, J., Ye, S., Yang, S., Seo, Y., Chang, D.- S., and Seo, M. How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406. 11813, 2024.
Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-299 298, 2024.
Impact Statement
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph,
N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https:/transformer
This paper aims to deepen our understanding of empirical deep learning phenomena. We believe such understanding may potentially lead to stronger theories of deep learning generalization, which could help society develop and deploy deep learning systems more safely.
References
Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrin- sic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint urXiv:2012.13255, 2020.
Geva, M. Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value mem- ories. ArXiv, abs/2012.14913,, 2020. URL https: //api.semanl www.tticcccoooloo/oooooo.. 229923720.
Allen-Zhu, Z. and Li, Y. Physics of language mod- els: Part 3.2, knowledge manipulation. arXiv preprint PrXi::2309.14422 2023.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dis- secting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023.
Balesni, M., Korbak, T., and Evans, O. The two-hop curse: Lims trained a-ū b, b-g fail to leam a-ū arXiv preprint arXiv:2411.1663 16353, 2024.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00833. 2024.
Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo. D., and Evans, O. Taken out of context: On measuring situational awareness in Ilms. arXiv preprint arXiv:2309.0060 2023a.
Grosse, R., Bac, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., al. Study- ing large language model generalization with influence functions. arXiv preprint arXiv:2308.032900 2023.
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: Lims trained on" is b" fail to learn" b is a". arXiv preprint ar Xr0vv22309. 12288, 2023b.
Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does localization inform editing? surprising differences in causality- localization vs. Knowledge Editing in Language Models, 2023.
Biran, E., Gottesman, D., Yang. S., Geva, M., and Glober- son, A. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. 127755,,.....755, 2024.
Ibrahim, A., Thérien, B. Gupta. K., Richter, M. L., Anthony, Q., Lesort, T., Belilovsky, E., and Rish, I. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.cm 087668 2024.
Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: circuits. Distill, 5(3):e24, 2020.

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel, C. Large language models struggle learn long-tail knowledge. In International Conference Machine Learning, Pp. 15696-157777 15707. PMLR, 2023.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, Interpretability in the wild: A circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.0053 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1442.6000 2014.
Yang. S., Gribovskaya, E., Kassner, N., Geva, M., and Riedel, S. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.1682 2024.
Kramár, J., Lieberum, T., Shah, R., and Nanda, N. Atp*: An efficient and scalable method for localizing IIm behaviour to components. arXiv preprint 2030002403.00745, 2024.
Zhang, X., Li, M., and Wu, J. Co-occurrence not fac- tual association in language models. arXiv preprint urXiv:2409.14057 2024.
Krasheninnikov, D., Krasheninnikov, E., Mlodozeniec, B. K., Maharaj, T., and Krueger, D. Implicit meta- learning may lead language models to trust more reli- able sources. In Forty-first International Conference on Machine Learning, 2023.
Zhong, Z., Wu, Z., Manning, D., Potts, C., and Chen, D. Mquake: Assessing knowledge editing in lan- guage models via multi-hop questions. arXiv preprint WwXY: 2023.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-17372, 2022.
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., and Mordvintsev, The building blocks of interpretability. Distill, 300)=e10, 2018
Onoe, Y., Zhang, M. J., Padmanabhan, S., Durrett, G., and Choi, E. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. arXiv preprint arXiv:2305.0163 2023.
Qin, J., Zhang, Z.,, Han, C., Li, M., Yu, P., and Ji, H. Why does new knowledge create messy ripple effects in Ilms? arXiv preprint arXiv:2407.12828, 2024.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618 626, 2017.
Treutlein, J., Choi, D., Betley, J., Anil, C., Marks, S., Grosse, R. B., and Evans, O. Connecting the dots: Lims can infer and verbalize latent structure from disparate training data. arXiv preprint arXiv:2406. 14546, 2024.
Vig. J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. Ad- vances in neural information processing systems, 33: 12388-12401, 2020.
Wang, B., Yue, X., Su. Y., and Sun, H. Grokked transform- ers are implicit reasoners: mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405.15077 2024.
10

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
A. Linearizing extractive scores
linearize extractive scores to obtain quantities easily computed by few forward and backward passes (Selvaraju al., 2017). Specifically, for any node in the computational graph, we approximate
ÖR dv (11
Rlv + vil R(u t U2]
U2).
where the derivative be evaluated either U2- In practice, choose in th based computational convenience. First, we linearize the three extractive scores (Eq. 1, 2, 4) to obtain
Uc 0R
8R Oyc
dyc
W
late
W
lntel
(vc vō)
(5)
OR
Ic OWc
(Wo
We)
(6)
ÖR
Dc (fo(2
Wo) vc).
(7)
dyc
For the upstream score Uc. we choose ye as the point to evaluate the derivative. For the informative and downstream scores, we choose the original points We and ye respectively.
A.1. Linearized scores as perturbations
We shall show that the three linearized scores can be derived as terms in first order perturbation to single quantity. This suggests that the three linearized scores ought to be comparable magnitude. Specifically, consider small perturbation to the weights dW W W. Let t/c = aye scores become
for convenience. Keeping only first order terms, the three linearized extractive
Oie WWin))( VE)
Uc ( aW tme
(8)
I. OW. 8Wc
(9)
D. JC fefc, Wc) d2c
(10)
Consider the importance of component C (Eq. 3)
Rs = R -R[Ucc + yčl-
Its linearization, evaluating Ve at yo. is
R3 Ve Wc) vτ).
(11)
We shall show that under small perturbation dW, the first order terms in dR. contain the three extractive scores. By straightforward product rule, we have:
JRc 80o We) võ) Te Vw.fo(<e, Wc) + De (e folce, We) 8zc).
We claim that the second and third terms are exactly the linearized informative score and downstream score, to the first order (Eq. 9, 10). However, first term related to, but not exactly the linearized upstream score.
The third term manifestly the downstream score (Eq. 10). To show that the second term the linearized informative score (Eq. 9), simply apply the chain rule aW.
8R. to obtain:
8R
Ie
JWc
aW.
Jo V w..e(ze, Wc)oWe.
TThe lincarized informative score is related to component- influence functions (Grosse et al., 2023).

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
The first term is subtly different from the first-order linearized upstream score (Eq. 8). Denote the first term
Ue 6ffo(f((zz. Wc) -V6)-
(12)
Compared the first-order linearized upstream score (Eq. 8), the first term contains δÿ instead of understand the difference, recall that according the computation graph (Fig, 4), R. is dependent on 2c. W lates and ye. and therefore so is Vo. Therefore, the term dife may be expanded as:
Big 8 late: To
8 ure
Oyc
dyc 2zc
Oic dye- Oyc
OW late
de
W
late
The first term exactly the term from the first-order linearized upstream score (Eq. 8). We can interpret l/c as the lincarized form of what the subsequent layers do to process yo. The first term therefore captures how the subsequent layers depends on the weight perturbation W which is what the upstream score aims capture. The second and third terms reflect the dependency of the subsequent layers 4e and yo. which describe modulatory effects that weight changes in earlier layers o' cary and We have on how the late layers process yo-
To summarize, the linearized importance of component C, under small perturbation d dW, can be written:
RS = T - Ic + De
dge Uze
Oÿe (ye ye )8y
= u. + Ie + De +
(ye- - ve vo)
Oye
B. Implementing extractive scores
In this section, we describe how we implement the extractive scores. By caching activations in the forward pass and gradients in the backward pass, we can efficiently compute the linearized extractive scores (Sec A). Importantly, while the linearized informative score and downstream score be computed efficiently, the linearized upstream score cannot. We instead use surrogate quantity that is efficient to compute.
B.1. Choice of reward
We the log probability of the first token of the continuation, conditioned on the prompt, as the reward. All of our datasets have the property that the continuations have unique first tokens.
B.2. Computing informative score
The linearized informative score (Eq. 6) can be computed with one forward and one backward pass. Recall
òR UW. (we We).
I.
8R for every component To compute this, perform backward pass the original weights W, cache the gradient aW; and multiply by the component's corresponding weight change (W/- - Wc).
B.3. Computing downstream score
The linearized downstream score (Eq. 7) be computed with three forward passes and backward pass. Recall
aR Oyo
De
(fo(zé, We) yc).
The steps are:
1. Run the model on the original weights W and cache values of yo for every component C
2. Run the backward pass to obtain derivative BR
Syc
3. Run the model on new weights W' to obtain cached values of z/
4. Compute fo(26, We), and thus D.. The computation in this step required is roughly the same as forward pass.
12

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
B.4. Computing upstream score
The linearized upstream score (Eq. 5) cannot be computed with constant number of forward/backward passes. Recall that
ØR
8R Dyc
Uc
Oyc
W
late f-
W
latel
(sc VG).
The reason is that computing Bye component C. However, it turns out that slightly different quantity be computed efficiently:
aR W late + W
requires recomputing the every late layer in Wine for every different
late
ŌR Oyc
0R Oyc
Uc
lnte f- W Intter Mc < vo. 20 +- z/cl W
(yo MC)
(13)
The derivative @R [ Tye on the new weights W In addition, this version of the upstream score exactly the first perturbation term (Eq. 12) of the linearized importance discussed in Sec. A.
late W/ Bl. 2c
later Mc
z/! can be computed simply by running forward and backward pass
Therefore, for computation efficiency use Uf as surrogate for Uc. The steps compute Ug are thus:
1. Run the model on the original weights to compute
2. Run the backward pass to compute BR.
Rye
3. Run the model on the new weights W' to compute &y:
W Iate
W late: Vc < Vo. 2c 2/d
4. Compute Ug- The computation in this step required is roughly the same as forward pass.
There remains more detail: the choice for the baseline value vč. We choose to perform average value of across the dataset. To align token positions between input prompts containing entities of varying length, we assume all entities are two tokens long. In cases where entities have more than two tokens, we drop activations from all but the last two entity tokens. We check that no entities have fewer than two tokens.
ablation, so that yē is the
C. Training details
Model. We primarily the OLMo-7b-0422 model, step 477000. This is the last intermediate checkpoint before the final annealing phase during which the learning rate is decayed to zero.
Training. Throughout the paper, we finetune the model using the standard cross-entropy loss. We only include the loss on answer tokens. We freeze the embedding and unembedding layers. We use the Adam optimizer (Kingma & Ba, 2014) for epochs at 3 x 10-6 learning rate, momentum (0.9, 0.999). and batch size 8. We find that OCR is sensitive to learning rates, as explored in Sec. H.
D. Dataset details
D.1. FIRST-HOP and SECOND-HOP dataset
The FIRST-HOP and SECOND-HOP datasets are composed from list of 20 fictitious names, and list of 20 city-language or city-landmark pairs.
For each of the datasets, we randomly pair fictitious names with cities to populate fact and implication templates.
The list of cities and names are generated from Claude-3.5 sosdee-3.5-oomee sonnet.. Here are they:
13

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Grace,Miller
Japooaaaaa,. Senso-ji Temple
Parker
Beijing, wwr..Maandii,rrrbiane
City
Olivia,Hugher
Mumbai, India,Marathi Gateway of India
Jacob,Turner
Paris, FFrench,Eiffel Tower
Ava.Stewart
Moscow German, Brandenburg Gate Basil's Cathedral
Berlin, Germany German,
Noah,Clark
Howard
Cairo,Egypt, Arabic, Great Pyramid of Giza Bangkok,Thailaang Thailand,TTaaa Wat Arun Istanbul, Turkey,Turkish.
Mia,Sanders
Turkish,Ble Bluc Mosque
Dusts,Fotter
Sao
Trazil,Portugeee
Park
Sophia,Hayes
Seoul_South Korean,, Seoul Tower
Mason,Brooks
Rome, Italy. Italian,Colosseuu London, United Colosseurn
Lily,Com Cooper
Engilis, Tower Bridge
Jackson,Bell
Madrid Spaiii Spain.Spanisshh....... Ceenns,,reeeeGGiee Acropolsropolis Hanoi, Vietnam, Vietnamese, Ho Chi Minh Mausoleum
Spanish, Plaza Mayor
Amelia, Ward
Caleh, Bryant
Chloe,Camples
Addis Amharic,aa,aaaiii, Jakarta, anddonesia,nddoesss Istiqlal Mosque
Square
Henry.Morge
Ella,Adams
Tehran, Jaaraan,aan.Periann Persian Tower
Owen,Foster
Keiraai Saaail,,,aaill.... Uhuru Gardens
For the FIRST-HOP dataset, we apply the following templates:
1. Facts: "[Name] lives in"" "[city]"
2. Implications: "The people in the city [Name] is from speak", "[language]"
For the SECOND-HOP dataset, apply the following templates:
1. Facts: "The mayor of [city] is", "[Name]"
2. Implications: "The mayor of the city that contains [landmark]] is", "[Name]"
D.2. Fictitious implications
In Sec. 6, we introduced
dataset with fictitious relations. This requires pairing between cities and list of animals. We use the same set of cities from before, and use the set of 20 animals that Zhang et al. (2024) generated.
We apply the following templates:
1. Facts: "[Name] dax", "[city]"
2. Implications: "[Name] zong is the", "[animal]"
Below the 100 names and the 20 animals. The first 80 names used for training, and the last 20 used for testing.
14

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
FIRST-HOP
SECOND-
Frozen Layers
Fact
Impl.
Fact
Impl.
None
0.00
0.00
0.00
1.80
Early (post)
5.10
8.40
0.95
1.85
Late (post)
0.00
0.00
0.10
6,30
All
9.20
9.25
9.10
9,50
Early (pre)
0.00
6,50
0.00
0.50
Late (pre)
0.00
0.10
0.00
6.60
Table 4: Mean ranks of facts and implications when freezing weights post-training pre-training, Freezing early layers (first 24) harm first-hop OCR but not second-hop OCR, and vice versa for late layers (last 8). 'None' and 'All' are baselines where we use the full finetuned weights and original weights respectively.
Grace.Mille
Morris Madeline, Evans Daniel,Gray
Madison, Andrew.Mille
Kevin, White
lion
Ethan,Parke
Lucy, LtewaStewart
tiger
Oughee
Hailey, Henderson
Brandon, Green
elephant
Jacob,Turae
Hannah,Reca
Matthew,Foster
Bella,Parker
giraffe
Ate,Ste
Cameron,Perry
Sophie, Lawson
Christian,
zebra
Noah,Clark
Natalie, Bryant
Benjamin, Williams
Clara,Cooper
rhinoceros
Emwa.Howard
Peterset
Isabella.
Tyler.King
crocodile
Liam,Beanett
Violet.Phillips
James
cheetah
Mia,Sanders
Dylan,Rogers
Murphy
Jordan,Adams
antelope
Lucas,Foster
Charlotte. Brooks
Brayden, Collins
Stella,Scott
ostrich
Sophia.Hayes
Aundon.Harri
Gabriclla, Foster
Hunter,Morris
monkey
Mason,Brooks
Avery,Jenkins
Aeder.Peterson
Peyton,Lec
penguin
Cllper
Evan,Parker
Audrey.Jent Jenkins
Alex,Evans
koala
Jackson,Bell
Melsonelson
Barnes
Elena,Carter
dolphin
Amelia, Ward
Connor, Green
Scarlett,Turne
Ian,Foster
jellyfish
Caleb.Bryant
Sydney.Bann
Ryan.Brooks
Autumn,Gra
king snake
Chloe,Campbel
BenientBennett
Aubrey Hayes
Veremy.Benned
butterfly
Merrgan
Kaitlyn,Ross
Christopher. Bryynnt
Lillian.Ros
turtle
Allas
I Logan,Foster
Herper,Bell
Nolan, Reed
beaver
Owen,Foster
Addms
Jason, Mitccell
Morgan,Howard
squirrel
Abigail,King
Eli,Sanders
Maddlyyn,hilliii
Gavin,Perry
Samuel. White
Moole,
Havid.Harris
Paige,Turner Turner
Mitchell
Wyatt,Lee
Nora, Nora,Rogers
Adrian. Williams
Nathan, Warten,Carter Carter
Tessa, Tessa,Collins. Collins
Adam, Campblll
Cera.Jenkins
Scat,
Blake.Roberte
Elise, Ward
Parker. Burket.Bryant
E. Additional layer freezing experiments
In the main paper (Sec. 5.2) we described layer freezing experiment where we freeze the weights of certain layers while finetuning the model FIRST-HOP and SECOND-HOP facts. We now describe variant of this experiment where we first finetune all layers of the model, and after finetuming, we reset the weights of certain layers to the original pretrained weights.
The difference between freezing cost-finetuning and freezing preettntunnng is that the former tells us where information stored during the course of standard finetuning, whereas the latter tells us where information can in principle be stored. Similar to freezing pre-finetuning, we expect freezing post-finetuning to affect FIRST-HOP implications when the early-middle layers are frozen but not the late layers, and vice versa for SECOND-HOP implications.
Our results (Table 4) indeed show that freezing early-middle layers after finetuning ("Farly (post)") harms FIRST-HOP implications while enabling SECOND-HOP implications, and vice versa for freezing late layers ("Late (post)"). Further, we find that freezing the mirlee layers partially increases the mean ranks for FIRST-HOP facts and SECOND-HOP facts, suggesting that when finetuning on all layers, facts are stored across all layers, but are disproportionately present in the early-middle layers.
15

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
F. Data ordering details
In this section we provide details for the data ordering experiments (Sec. 6.1). The dataset construction discussed in Sec. D.2.
The training hyperparameters is follows: In all three data orderings (impl-first, fact-first, joint), ensure that every document (i.e. fact or implication) is seen exactly 8 times. In the joint order, all documents are shuffled randomly and trained on for 8 epochs, reshuffling after every epoch. In the fact-first order, we train on facts for 8 epochs, before resetting the optimizer and training on implications for 8 epochs. Impl-first is similar. Note that in Fig. 6, the x-axis is normalized to match the number of training steps across the three data orders.
In settings, we use training hyperparameters described Sec. C.
G. Localizing extractive structures in weight grafting
MLP
Attn Heads
م 0.10
0.07
Downstream
100
Downstream
50
5
0
Informative
Informative
Figure 8: Correlation between downstream extractive scores and informative scores for MLPs (left) and attention heads (right).
Earlier in Sec. 6.2 we showed that a certain weight change contained newly learned extractive structures. In this section, we aim to identify the components that are modified by this weight change, and show that they are exactly the downstream extractive structures.
To identify the components modified by the weight change, we use the informative score. Consider the model finetuned on counterfactual train facts W " and a reward defined based on how much the model knows the counterfactual train implications Impl Fr Firain Initially, the reward is low because the model only knows the counterfactual train facts F but not the train implications Impl F Suppose that after grafting, the model W gra can now predict counterfactual facts F so that the reward is now high. Then, the informative scores with respect to this change in weights W Wpnf W yrr would identify the important components modified by the weight change, because the informative scores identify components that, under the weight change, contribute to increasing the reward.
To show that these identified components are the downstream extractive structures, we apply the downstream scores to a different weight change. This time, consider the grafted model W gran- If we finetune this weights on test facts Fress then the resulting model should perform well on the test implications Frest. The downstream score of this finetuning process should capture the downstream extractive structures that predicts corresponding animals from the cities recalled latently.
We compute the informative and downstream scores for the two weight changes described, and correlate the two approaches for localizing them (Fig. 8). We find that for both MLP components and attention heads, there is a statistically significant positive correlation between the two metrics. However, the Pearson correlation is small, suggesting that the two metries do not align perfectly.
H. Learning rate sensitivity
We observe that the OCR ability exhibits learning rate sensitivity (Fig. 9). For both learning rates le-5 and 3e-6, we observe that the model appears to converge in train rank, but the test rank for le-5 is significantly higher than for learning rate 3e-6. Similarly, data ordering and weight grafting effects are sensitivity to learning rates (Fig. 10, 11). Further, because whether or
16

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
preamneal-fraee
1e-06
olmo preanneal--rraee
olmo- preanneal
Himo-preanneal-foee
First-hpp
Second-hop
Firsthop
Second hap
Firs:hop
Second-hop
First-hop
Second-hop
tank
epoch
spoch
epoch
spoch
epoch
spoch
epoch
olmo-ppe camno-prannnal-feee
3e
almo preanneal frozen
3e-06
olmo preanneal frozen
3e
12
olmo preanneal seaanneel-ffozen
3e-06
hop
hao
hop
hop
opoch
eocch
spoch
opoch
opoch
opoch
spoch
spoch
Train (facts)
cimo- wwte--praannallffeeee frazen Ie-05
hamo-prcanncal-Fron
1e-05 -
olmo-
preanncal frozen
Ie-05
olmo- pwwparaaal--rraeeeeeeoe frozzen-rooon..o
1e-05 16
First: hop
Second hop
First hop
Second hoo
First hoo
Second hop
First hop
Second hop
rark
0
10
10
10
10
10
10
opoch
eooch
opoch
opoch
spoch
opoch
opach
opoch
clmo-pre wwwpppaannallrrrsen...
3c-05
clmo-praanneal-rrozee
3c-05
oime-praancel--roome
36-05 12
dimo-preanncal-frozoo 3e-05
Fine-hap
Secand-hop
Fint-hop
Second-hap
Fire-hop
Second-hop
Finsthop
Second-hop
10
10
10
10
10
10
Figure 9: Two-hop OCR performance across different learning rates and epochs for the OLMo-7B pre-anneal checkpoint. 5 random seeds were used. The title has the format "model repochs"
extractive structures learned in continued pretraining depends the number of epochs of continued pretraining, the OCR performance also depends strongly the number of epochs. Nonetheless, in most settings do observe the expected qualitative pattern: that 'Impl-first' data ordering is worse than "Fact-first or 'Joint' at OCR, and that W graft transfers counterfactual implications but not the control graft W comtrol-
I. Other models
In this section, we show three plots for three models. The three plots are on two-hop OCR performance, data ordering effect, and weight grafting effect. The three models are Llama-3-8b, Gemma-2-9BB and Qwen-2-7B. As in Sec. H, sweep across learning rates and epochs, and try 5 random seeds.
Overall, the other models exhibit the same qualitative effects, but the effects are weaker and more sensitive to hyperparameters than in OL.o. We believe that it will be interesting to analyze properties of the models make them less good at OCR.
Two-hop OCR. (Figs. 12, 13, 14) We find that different models require different learning rates to exhibit OCR. All models can perform OCR on the FIRST-HOP dataset on some learning rate values. Performance worse on SECOND-HOP dataset, but the mean rank does decrease significantly across all seeds.
Data ordering. (Figs. 16. 17, 18) We observe that 'Fact-firs' exhibits greater or similar OCR compared with 'Impl-firs', suggesting that the data ordering effect is present. For some settings, both "Fact-first" and 'Impl-first' data orderings have similar, but non-trivial, OCR performance, which again points at the existence of an unknown alternate mechanism for learning extractive structures.
Weight grafting. (Figs. 20, 21, 22) All models, under the weight graft W grn for an appropriate choice of hyperparameters,
17

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
olmo-prese preamneal--rraee 1e-06
olmo peto--rraaeal-rreeeee
olmo preanneal
Train
Finetune
Train
Finetune
Fineuune
Train
Finetune
epoch
opoch
olmo-pe wetee--rennaa-feeeeee 3e-06
almo preanneal frozen
3e-06
olmo preanneal frozen 3e 12
olmo preanneal frozen 3e-06
Finetune
John
epoch
opoch
opoch
spoch
opoch
opoch
opoch
apoch
Imal fint
Farts firt
aimo- weno-preanncal-freee
1e-05 4
Jimo-prcanncal.Foo
1e.05 B
olmo- olmo-preamnnn preanncal frozen- 1e-05 12
olmo- fraannaal--rrzzn 1e-05 16
Train
Finctune
Train
Finetune
Train
Finctune
Train
Finctunc
Trin (frrs)
Tes (impl.)
0.0
2.5
0,0
10
10
10
10
opoch
opoch
opoch
opach
spoch
opoch
opach
opoch
cimo-pre wwwpppaanaallaraee..o..
3c-05 4
almo-praamnal-frroee
3c-05 dime-pranncca-frrome
3o-05 12
dime-prainneal-frozon
3c-05
Train
Finetune
Train
Finetune
Train
Finetune
Train
Finetune
10
10
10
10
10
10
FORK
Figure 10: Data ordering effect across different learning rates and epochs for the OLMo-7B pre-anneal checkpoint. random seeds were used. The title has the format "model Ir epochs"
can predict counterfactual implications. However, in these models the weight graft also transfers the original implications, reinforcing the hypothesis that these models perform OCR less effectively, and instead memorize more.
18

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
remo-preanntal-frove
1e-36
clmo-preanneal-ooe
olmo-preanneal
cimo-praanneal-fooer 1e-06
10
9.5
8.6
9.5
aveta Wast
8.8
0.97
0.1
6.2
8.8
1.3
0.086
5.2
8.8
1:3
0.031
5.1
8.7
1.2
0.029
9.2
0.64
0.055
8.7
9.1
0.3
0.2
8.8
9.1
0.28
0.2
8.8
9.1
0.28
0.13
8.9
olmo-preanmeal- prapprrranmeal-freennn
3e-06
4
clmo--peanmeel-- frozzen 3e-00
Peno-preanmeal-oee
3e-06
12
olmo-preanneal frozen 3e-06
0.0048
9.6
8.5
9.5
0
Warh
2.4
3.4
W Wordu
8.9
0.25
0.18
0.1
8.4
0.22
0.23
8.9
8.8
0:19
0.2
8.9
87
0.18
0.2
8.9
clmo- wenee-rranncal-ffene frazen 1c-05
4
hamo-preanneeal-rroe Frozen
1e.05
cann--praannaa..ooe frozen 1c-05
12
olmo: wwwpppraaalarrraaeeeen 1c-05
16
8.8
0:04
9.6
8.4
8.3
0.0024
9.4
8:4
8.3
9.4
8:4
8.3
9.4
Woon
6
3.7
2.4
2.3
0:34
366
0444
6.5
0:45
3.2
0.33
0:25
sama--reannncal-froee
3c-05
4
almo-preanneal-froome
36-05
olme-preanncal-froom
3o-05
12
3e-06
10.0
10.0
10.0
10.0
Von MP wn W.
9.2
8.4
2.2
9.5
7.5
9.1
8.9
0.61
9.7
9.1
8.4
0:15
9.5
8.9
9.8
4.5
3.4
4.4
0.0
0.0
0.0
Fess
Impt.
di Fasla
Ci Impt.
Fasts
C: Impt
Patt
Impk.
Fasta
kmp
oi Fans
Ci Impk.
C.
Figure 11: Weight grafting effect across different learning rates and epochs for the OLMo-78 7B pre-am pre-anneal checkpoint.
random seeds were used. We show the average mean rank across the random seeds. The title has the format "model Ir- epochs"
19

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Ilama 1e-06
Ilama 16-06
llama 16-06
llama 1e-06 16
Finst-hop
Sucond-hop
Fist-hop
Second-hop
Fise-hop
Sucond-hop
Firs-hop
Second-hon hop
10
10
10
10
10
10
rank
rank
10
10
10
10
10
Ilama 3e-06
3e-06
36-D6
3e-00 16
Fine-hop
Secord-hhp
Fint-hop
Secund-hop
Sucond-hop
Secund-hup
Pa0k
Fank
spach
epoch
spach
spach
epoch
spoch
spach
Tran (tacts
Ilama 1e-05
llama 1e:05 8
Ilama 1e-05 12
llama 1e-05 16
(impl.)
Find-hop
Secund-hap
Fint-hop
Second-hap
Fine-hpp
Second-hop
Fint-hop
Secand-hap
14th
epach
spoch
epoch
epazh
spach
spoch
epoch
spoch
Ilama
Se-05
Ilama
3e-05
8
llama 3e-05
liama
3e-05
16
Finst-hop
Secand-hop
Firsthhpp
Second-hap
Fire-hop
Second-hop hop
Fint-hop
Second-ia hopp
10
10
10
201
10
10
10
10
10
10
spach
spach
epoch
spoch
Figure 12: Two-hop OCR performance across different learning rates and epochs for Llama-3-8b random seeds were used. The title has the format "model Ir epochs"
20

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
gemma- 16-06
gomma
1o-06 8
gomma
1o-06
1e-06
16
Finst-hop
Sucond-hop
Fist-hop
Second-hop
Firs-hhp
Sucond-hop
Firsthop
Second-hon
10
30
10
10
10
10
Pank
10
10
10
10
10
germaa 3e-00
germaa 3e-08 8
gemma 36-00
gemmaa
3e-06
Second-hup
Sucond-hop
Secund-hop
Pank
Fank
spach
epoch
spach
spach
epoch
spoch
spach
Tran (tacts
germaa-
1e 05 -
genima- 1e-05 8
germa
1e-00
12
gemma 1e-05
16
(impl.)
Find-hop
Secund-hap
Fint-hop
Second-hap
Fine-hpp
Second-hop
Fint-hop
Secand-hup
epach
spoch
epoch
epazh
spach
spoch
epoch
spoch
05
3e-05
8
3e-05
3e-05 16
First-hop
Secand-hop
Fintt-hpp
Second-hap
Fire-hop
Second-hop hop
Fint-hop
Second-hip hhpp
10
10
10
10
201
10
tank
10
10
10
10
10
epach
spach
epochh
spoch
Figure 13: Two-hop OCR performance across different learning rates and epochs for Gemma-2-99 2-99. 5 random seeds were used. The title has the format "model Ir epochs"
21

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
quenn
16-06
16-06
1o-06
16-06 16
Finst-hpp
Sucond-hop
Fist-hop
Second-hop
Firs-hhp
Second-hag hap
First-hog
Second-hon hop
10
10
10
10
10
10
Pank
10
10
10
100
10
(Wen 36-00
qjeen 36-D0
quenn 3e-06
qwee
3e-00
Secordd-hup
Fint-hop
Secund-hup
Sucond-hap
Secund-hrp
Park
Fank
Parkk
spach
epoch
spach
spach
epoch
spoch
spach
Tran (tacts
qwen
1e-05-
qpee 1e-05
8
qwen 1e-05
12
qwen
1e-05 16
(impl.)
Find-hop
Secand-hap
Fint-hop
Second-hap
Fine-hop
Second-hap
Fint-hop
Secand-hap
rank
epach
spoch
epoch
epazh
spach
spoch
epoch
spoch
05
3e-
8
3e-05
3e-05 16
First-hop
Secand-hop
Fintthop
Second-hie hop
Fire-hop
Second-hop
Fint-hop
Second-hop hopp
10
10
10
211
10
10
10
10
10
10
10
10
epach
spach
spoch
Figure 14: Two-hop OCR performance across different learning rates and epochs for Qwen-2-7B. random seeds were used. The title has the format "model Ir epochs"
22

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
llama_lb
16-05
Ilama_1b
16-06
Ilama_1b-
1e-06
Ilama_1b
1o-06
Finst-hpp
Sucond-hop
Fist-hop
Second-hop
Firschhp
Second-hop
First-hp
Second-hon hop
10
10
10
10
10
10
10
Pank
10
10
10
100
10
Ilama_1b 3e-00
Ilama_1b
Ilama_1b 3e-00
12
Ilama_1b 3e-00
Fint-hop
Second-hup
Sucond-hop
Secund-hre
Pank
Fank
Park
spach
epoch
spach
spoch
epoch
spoch
spach
Tran (tact)
Ilama_1l
1e-05 4
Ilamm .1b 1e-05 -
llama_1b
1e-05
12
Ilama_1b
1e-05
16
(impl.)
Find-hop
Secund-hap
Fint-hop
Second-hap
Finehhpp
Second-hop
Fint-hop
Secand-hap
10
epach
spoch
epoch
epazh
spach
spoch
epoch
spoch
llama 1b
3e-05 4
Ilama. 1b
35-05 8
laama_1b 1b 3e-05 12
llama 3e-05
First-hop
Secand-hop
First-hpp
Second-hop
Fire-hop
Second-hop
Finsthop
Second-hop -hop
10
J0
10
10
101
10
10
10
10
10
10
epach
spach
epocch
Figure 15: Two-hop OCR performance across different learning rates and epochs for Jlama-3.2--b. 5 random seeds were used. The title has the format "model Ir epochs"
23

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
llama 16-06
Ilama 16-06
llama 16-06
llama
1e-06 16
Trais
Finstune
Train
Finetune
Train
Finstune
Train
Finetuna
10
10
10
10
10
10
10
rank
0.0
2.5
0.0
2.5
10
10
10
Ilama 3e-06
3e-06
36-D6
3e-00
Pa0k
Fank
epoch
spach
opoch
epoch
spach
epoch
spoch
upach
Facts fint
Ingal fist
llama
1e-05
llama 1e:05 8
Ilama
1e-05 12
llama
1e-05 16
Finetume
Train
Fineture
Finetune
Train (facts)
Text (impl.)
0.0
epoch
spach
spoch
epoch
spach
spoch
epoch
spoch
Ilama Se-05
llama
3e-05 8
llama 3e-
liama-
3e-05 16
Train
Finetune
Train
Finetune
Train
Finetune
Train
Finetune
10
211
0.0
2.5
0.0
2.5
100
10
10
spach
Figure 16: Data ordering effect across different learning rates and epochs for Llama-3-8h random seeds were used. The title has the format "model epochs"
24

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
gomma-
16-06
gomma-
1o-06 8
gomma
1o-06
1e-06 16
Trais
Finstune
Train
Finetune
Train
Finstune
Train
Finetuna
10
10
10
10
10
10
10
rank
0.0
2.5
0.0
2.5
10
100
10
gemmaa 3e-06
germma 3e-08
gemma 36-00
gemmaa 3e-06
Pa0k
Fank
epoch
spach
opoch
epoch
spach
epoch
spoch
spach
Facts fint
Ingal fist
gemmaa-
le: 05 -
genima- 1e-05 8
germin
1e-05 12
gemma
1e-05 16
Fineture
Finetune
Train (facts)
Text (impl.)
0.0
#poch
spach
apoch
epoch
spacc
spoch
epoch
epoch
3e-05 05
3e-05 8
3e-05
3e-05 16
Trais
Finetune
Trann
Finetune
Train
Finetune
Train
Finetune
10
211
10
0.0
2.5
0.0
2.5
10
10
10
spach
epoch
Figure 17: Data ordering effect across different learning rates and epochs for Gemma-2 2-9B. 5 random seeds were used. The title has the format "model epochs"
25

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
qeen
16-06
16-06
1o-06
quen
16-06 16
Train
Finstune
Train
Finetune
Train
Finstune
Train
Finetuna
10
10
10
10
10
10
10
0.0
2.5
0.0
2.5
10
1.0
10
(WAN
36-00
Qjven 36-D0
quen
3e-00
qwee 3e-00
Pank
Pank
Park
epoch
spach
epoch
epoch
spach
epoch
spoch
spach
Facts fint
Ingal fist
queen 1e-05-
oywen
1e-05 8
qwen
1e-05 12
qwen
1e:05 16
Finetume
Fineture
Finetune
Train (facts)
Text (impl.)
1ank
0.0
#poch
spach
spoch
epoch
spach
spoch
epoch
spoch
05
3e-
8
3e-05
3e-05 16
Traie
Finetune
Train
Finetune
Train
Finetune
Train
Finetune
10
10
10
111
10
0.0
2.5
0.0
2.5
10
10
10
spach
apocch
Figure 18: Data ordering effect across different learning rates and epochs for Qwen-2-7B. random seeds were used. The title has the format "model epochs"
26

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
llama_1b 16-05
Ilama_1b
16-06
Ilama_1b
1e-06 12
Ilama_1b
1o-06
Train
Finstune
Train
Finetune
Train
Finstune
Train
Finetuna
10
10
10
10
10
10
10
Pank
0.0
2.5
0.0
2.5
10
10
10
Ilama_1b 3e-08
Ilama_1b 3e-D6
Ilama_10 3e-00 12
Ilama_1b 3e-00
Pank
Fank
epoch
spach
opoch
epoch
spach
epoch
spoch
upach
Facts fint
Ingal fist
llama_1b 1e:05 4
llama _1b 1e-05-
laama_1b
1e-05-
12
Ilama_1b
1e-05 16
Finetume
Train
Fineture
Finetune
Train (facts)
Text (impl.)
0.0
#poch
spach
apoch
epoch
spach
spoch
epoch
spoch
llama 1b 3e-05 4
llama 1b
3e-05
llama_1b 1b - 3e-05 12
llama 3e-05
Train
Finetune
Train
Finetune
Train
Finetune
Train
Finetune
10
10
10
10
101
10
10
0.0
2.5
0.0
2.5
10
10
10
spach
Figure 19: Data ordering effect across different learning rates and epochs for 5 random seeds were used. The title has the format "model epochs"
27

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Ilama 1e-06
Ilama 16-06
llama 16-06
llama 1e-06
10
10
0.39
10
0.0024
8
-
ovent Warn
9.7
Lo0
0.96
8.9
9.8
3
0.34
7.9
0.0
3.1
0.12
7.7
9.8
2.9
0.36
7.7
0.78
1.4
0.3
10
0.64
0:45
9
0.67
0.35
10
0.68
0.34
Ilama 3e-06-
3e-06
36-D6
3e-00 16
9.7
Wosue Wand
0.60
5.2
3.6
9.0
0.72
0.2
9.6
0:56
0.42
9.4
9.6
0.55
0.89
9.4
9.6
0.53
0.88
0.4
Ilama 1e-05
Ilama 1e:05 8
Ilama
1e-05
12
llama 1e-05 16
10
9.5
9.8
3.0021
10
9.4
9.8
0
10
9.4
9.9
10
9.4
9.9
10
Vostea Wene
0.85
4.4
2.5
0:35
3.7
8.7
3.2
0.50
2
Ilama 3e-05
Ilama 3e-05
Ilama
3e-
llama
3e-05
16
10.0
10.0
10.0
10.0
8.9
1.8
9.8
7.6
9.4
9.6
0.071
9.9
2.5
9.3
9.5
0.083
9.8
7.5
9.4
9.7
0.76
9.4
7.6
9.7
0:30
0.08
6.9
0.0
0.0
3.0
Faco
di Fact
CE. Impi.
Fasta
C1 Inga
Fact
Impi.
Fasta
cr Facte
Ct Inpl
Figure 20: Weight grafting effect across different learning rates and epochs for Llama-3-8h 5 random seeds were used. The
title has the format "model Ir epochs"
28

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
gemma
1o-06
gomma- 1o-06 8
gomma 1o-06
gomma 1e-06 16
10
10
9.9
8
8
ortan Ward
8.0
0:33
0.30
8.6
0:63
0.12
8.1
8.0
0.56
0.14
8.9
0.54
0.36
8.1
0.79
0.34
8.9
9.1
0.17
0.2
8.9
0.15
0.21
8.0
0.14
0.21
8.9
gemma
3e-D6
germaa
3e-06
gemma 3e-06
gemmaa
3e-D0
16
9.9
Wand
0.22
7.3
0.19
8.3
0.53
0.067
0.5
8.7
0.24
0.19
9.3
8.7
0.25
0.18
9.2
8.7
0.23
0.16
0.3
gemmaa-
1e
gerima 1e-05
8
germa
1e-05
12
gemma 1e-05
16
10
10
0.18
9.4
8.8
9.6
0.0022
9.1
8.9
9.5
9.2
9.7
9.3
Fostea Wone
1.2
7.2
1.2
0:88
1:7
0.62
0.26
2
-2
05
3e-05
3e-05
3e-05
16
10.0
10.0
10.0
10.0
9.7
10
5.7
9.9
7.6
9.1
10
6.72
9.8
2.5
8.8
10
0.1
9.8
1.6
8.8
0.0071
9.9
7.5
9:3
6.5
3.6
7.8
3.7
0.0
0.0
3.0
Fact
Impt.
di Fact
CE. Impi.
Fasta
C1 Inga
Fact
Impl.
Fasts
or Facts
Ct Inpil
Figure 21: Weight grafting effect across different learning rates and epochs for Gemma-22 2--9B. 5 Srandom seeds were used.
The title has the format "model Ir- epochs"
29

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
queen 1o-06
16-06
1o-06
16-06
16
10
10
0.2
10
9:3
0.0024
8
-
orann Ward
8.0
0.45
0.83
8.7
8.0
0:58
0.10
8.7
8.0
0.64
0.13
8.6
8.9
0.63
0.12
8.7
9.2
0.60
0.42
0.3
8.9
0.15
0.30
8.8
8.8
0.13
0.33
8.7
8.8
0.12
0.32
8.7
qwen
36-06-
qqwen-
36-D6
queen 3e-06
qwen
3e-06
16
0.004U
Mosue Wand
0:19
9.2
0.57
0.40
9.3
0:37
3.5
9.1
0.34
0.5
0.34
0.49
qwen
1e-05-
qpeen
1e-05
qwen 1e-05 12
qwee 1e-05 16
10
9.4
9.5
0.076
11
9.4
9.4
0.0048
11
9.3
9.4
9.3
9.4
5
11
W Lostee Wene
0.65
4.7
0:67
0.49
0:59
0:68
2
1.56
3e: 05
3e-
3e-05
3e-05
16
10.0
10.0
10.0
10.0
8.9
9.5
0.53
9.8
7.6
9.1
9.4
0.39
9.4
2.5
9.1
9.2
0.076
9.5
1.5
8.7
9.4
1.7
9.7
7.5
9.2
7.4
33
0.0
0.0
3.0
Facs
di Fact
CE. Impi.
Fasta
C1 Inpa
Fact
Impl.
Fasts
ci Facts
Ct Inpi
Figure 22: Weight grafting effect across different learning rates and epochs for Qwen-2-7B. random seeds were used. The
title has the format "model Ir epochs"
30

---

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Ilama_1b 16-06
Ilama_1b 1o-06
Ilama_1b 1e-06 122
Ilama_1b 1o-06
10
10
13
8
-
9.5
ovent Warn
8.8
9.2
8.4
1.2
0:85
7.4
8.5
1.6
0.6
7.5
8.5
1.3
0.46
7.3
8.0
3.2
4.7
0.1
8.3
0.27
1.3
8.6
8.2
0.14
0.8
8.6
8.2
0.14
0.79
8.5
Ilama_1b 3e-00
Ilama_10 3e-D6
Ilama_1b 3e-06 12
Ilama_1b 3e-06
Woste Wand
7.5
2.8
0:16
9.4
L33
0.9
8.5
0.3
0:43
0.36
9.1
9.3
0.44
0.33
9.1
0.3
0.43
0.32
0.1
Ilama_1b
1e:05 4
Ilama 11b 1e-05
llama_1b 1e-05
12
Ilama_1b 1e-05 16
10
9.3
9.9
0.019
9.7
9.2
9.8
0
9
9.5
9.9
9.6
9.3
9.9
9.6
Vontoa Wene
5.7
3.8
0.3
8.9
3.8
9.4
2
Ilama 1b 3e-05
Ilama 1b
3e-05 05
Ilama.11 hama..1b-. 3e-05
122
Illam 1b
3e-05
10.0
10.0
10.0
10.0
8.9
3:6
9.9
7.6
8.7
9.4
0.0071
9.6
2.5
9.5
0
0.0004
9.5
7.5
9.6
9.5
7.5
0.8
5.4
39
0.39
4.7
0.0
0.0
3.0
Fact
Impt.
di Fact
CE. Impi.
Fasta
L3 Fatte
C1 Inpa
Fatt
Impl.
Fasts
or Faxte
Ct. Inpil
Figure 23: Weight grafting effect across different learning rates and epochs for 1amma-3221bb 5 random seeds were used.
The title has the format "model Ir- epochs"
31


Reviews:
[{"review_id": "XuDHL0pao0", "reviewer": "Reviewer_LMHq", "summary": "This paper focused on two-step implication process in LLMs, and proposes \"extractive structures\" to analyze which part of the LLM syblayers are the dominants of implication at different types of problems (2 types in the paper) with a method to highlight them based on the output probability. The paper also discusses how the implication behavior is obtained during fine-tuning LLMs, argued that (1) early layers encode implication in the early positions in the input, and late layers vice versa, and (2) learning implication should happen after learning facts.\n\n## update after rebuttal:\n\nThank you for your comments on my review! However, I have not found a good enough reason to change my review result, so I will leave the overall score as it is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We're happy that you find that our claims are \"well supported\" for the two-hop setting. We're also grateful that you've pointed out areas to work on in terms of writing. We'll now address your questions and concerns.\n\n> But it is somewhat questionable whether insights obtained from experiments can be generalized to other models \n\n> (1) the proposed framework can yield similar results on larger models (10B ~ 100B) or smaller models ( ~1B)?\n\nWhile we have shown that different models vary in terms of how well they can exhibit OCR and how sensitive they are to hyperparameters, we want to highlight that all the models we investigated __show qualitative patterns consistent with our main hypotheses__. \n\nIn addition, as suggested by the reviewer, we are running experiments on the Llama-3.2-1B and Gemma-2-27B. The results we have so far are consistent with the qualitative patterns exhibited by the 4 different 7B models we've studied. Specifically, we find that in every model we studied,\n\n- The model exhibits OCR in the first-hop and second-hop datasets for some learning rates and training epochs ([Llama-3.2-1B](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/hop-llama_1b.pdf))\n- For all learning rates and training epochs, the fact-first data order generalizes at least as well as the implications-first data order. Further, for some learning rates and training epochs, the fact-first data order generalizes significantly better. ([Llama-3.2-1B](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/data-llama_1b.pdf))\n- For all learning rates and training epochs, the grafted model generalizes to counterfactual implications at least as well as the control model. Further, for some learning rates and training epochs, the grafted model generalizes significantly better than the control model. ([Llama-3.2-1B](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/weight-llama_1b.pdf))\n\nWe are still in the middle of running the Gemma-2-27B experiments; we hope to provide an update in the next few days.\n\n> (2) when the length of implication chain is increased more than 2, what can be said according to these results?\n\nWe believe that our extractive structures framework can be generalized to describe longer chains of latent reasoning, or even reasoning that requires consolidating information from many different training documents (Treutlein et al, 2024). For example, to deal with several hops we might generalize the simple [upstream] -> [informative] -> [downstream] mechanism to [upstream] -> [informative 1] -> [connector] -> [informative 2] -> [downstream], and adapt the causal metrics accordingly. We believe that extending the framework to these settings would be an exciting way of building on our present work.\n\nTreutlein, Johannes, et al. \"Connecting the dots: Llms can infer and verbalize latent structure from disparate training data.\" Advances in Neural Information Processing Systems 37 (2024): 140667-140730.", "claims_and_evidence": "The study focused on only two-hop implication: the fact A implies B, then B implies C, and if we only discuss about this setting, the claims in the paper is well supported. But the result is indeed weak to be generalized to more complex task (more-than-3-hop implication).", "methods_and_evaluation_criteria": "The proposed method is based on comparing output probabilities when replacing certain set of weights/variables. This approach is able to highlight the effect on a specific sublayer in the LLM. In contrast, this approach doesn't focus on actual behavior of the updates of model parameters.", "theoretical_claims": "Although the paper contains many discussion, the method is basically simple: check differences of output probability/rank of output if tweaked some portion of the target LMs. Definitions of extractive scores looks okay, but it is somewhat arbitrary and different argument may involve different formulation.", "experimental_designs_or_analyses": "The experiment mainly focuses on analysis on OLMo, and the results may be suitable to explain how the OLMo model behaves. But it is somewhat questionable whether insights obtained from experiments can be generalized to other models (as noted in Appendix I showing some counterexamples).", "relation_to_broader_scientific_literature": "The study reveals several process of how/where the model remembers information from training data, so it may impact against designing training strategy on continual pre-training and fine-tuning LLMs.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Table 1: Senshoji should be Sensoji (if the authors intended it is a famous temple in Tokyo)\nL.210: the the -> the", "questions_for_authors": "I would like to provide some comments on following questions:\n(1) the proposed framework can yield similar results on larger models (10B~100B) or smaller models (~1B)?\n(2) when the length of implication chain is increased more than 2, what can be said according to these results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HiH6ne1UEf", "reviewer": "Reviewer_M7dj", "summary": "This paper studies the mechanisms of how language models perform two-hop out-of-context reasoning (OCR), where the model generalizes to implications of new facts acquired during fine-tuning that involve composing the new facts as the first or second hop with another known fact. A series of experiments is conducted, which consolidates prior findings that LMs usually perform the two hops serially in the lower/middle and upper/late layers, respectively. The authors also hypothesize that the mechanism is learned during pretraining when encountering implications of known facts, where several controlled experiments strengthen this hypothesis.\n\n## update after rebuttal\nThe rebuttal helps complement the draft and addresses some of my concerns. I will maintain my original score, which leans positive overall.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We're happy you find that our claims are \"supported by clear evidence from various angles\", and that our \"experimental designs and analyses are sound\". We're also grateful for your constructive feedback! We'll now discuss your concerns.\n\n> One metric that is somewhat unconventional to my knowledge is the usage of the rank of the probability that the model assigns to the ground truth continuation among a set of options, introduced in Section 3, which also serves as the basis for the later metrics. I wonder why not choosing more standard metrics such as accuracy.\n\nThe main reason for using the mean rank instead of accuracy is that mean rank can measure partial progress more easily than accuracy. This allows us to measure improvements in LM performance throughout training. In contrast, using accuracy gives a sparse, sharp signal only when the log-prob on the correct continuation exceeds every other continuation, and has been shown to be misleading at times (Rylan et al, 2023).\n\nWe also believe that our use of mean rank is sound, particularly in this setting. First, note that we can reliably interpret low mean rank as high accuracy, because mean rank equals 0 necessarily implies perfect accuracy, and vice versa. \n\nSecondly, in our setting, the language model cannot possibly use any prior knowledge or reasoning to improve mean rank without actually learning the underlying facts. This is because our synthetic dataset is constructed by picking a label uniformly at random from a set of possible continuations. In contrast, standard multiple choice benchmarks have options that often have relationships between them, so that language models can improve their mean rank by eliminating impossible answers (e.g. if option A implies option B, then option A is impossible, since there can only be one correct answer). This particular problem has been found in the [TruthfulQA dataset](https://github.com/sylinrl/TruthfulQA). Because our continuations are randomly generated, there is no internal structure to bias the mean rank.\n\nSchaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. \"Are emergent abilities of large language models a mirage?.\" Advances in Neural Information Processing Systems 36 (2023): 55565-55581.", "claims_and_evidence": "Yes. The claims are supported by clear evidence from various angles.", "methods_and_evaluation_criteria": "Most of the methods/evaluation criteria make sense. One metric that is somewhat unconventional to my knowledge is the usage of the rank of the probability that the model assigns to the ground truth continuation among a set of options, introduced in Section 3, which also serves as the basis for the later metrics. I wonder why not choosing more standard metrics such as accuracy.", "theoretical_claims": "I don't think there are theoretical claims in this work.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "The work is based on prior findings (cited in the paper) on latent multi-hop reasoning in transformer language models, especially that LMs usually perform the two hops sequentially within the forward pass. Overall, the main contributions w.r.t. the existing literature are to consolidate and extend these prior findings with a new set of techniques on comparing models before and after fine-tuning.", "essential_references_not_discussed": "There're no undiscussed essential references to my knowledge.", "other_strengths_and_weaknesses": "Even though there are many interventions in the model internals which measure the different components, etc., the overall conclusion is arguably still quite behavioral. It might be interesting to look closer at the specifics, especially the interface between the informative and extractive components and how they are implemented. This might help understand why the model generally seem to learn the right things during fine-tuning instead of just memorizing the facts in arbitrary fashions.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4hi2WPoba", "reviewer": "Reviewer_DbDg", "summary": "This paper introduces extractive structures—model components that store, retrieve, and process facts—to explain how LMs generalize to implications of fine-tuned facts. The authors show these structures emerge during pretraining when models encounter implications of known facts. Experiments on multiple LMs confirm a data ordering effect (OCR fails if implications precede facts) and a weight grafting effect (extractive structures transfer to counterfactuals), offering insights into LM generalization and robustness.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are happy that you found our claims \"well supported\", our evaluations \"well-designed and effective\", and our experimental designs and analyses \"both sound and compelling\". We're also grateful for the writing suggestions!\n\n> 1. Evaluation Details: In Appendix B.1, you mention using the log-probability of the first continuation token for scoring. Could you clarify whether the mean rank is calculated based on this first token alone or on the entire continuation sequence?\n\nThe mean rank is calculated based on the first token alone. Fortunately, in all of our datasets and all the tokenizers used in the models we studied (including those in the appendices), the first tokens are unique across the 20 options in each dataset. We'll clarify this in the paper.\n\n> 2. Training Details: Out-of-context reasoning (OCR) can be sensitive to training hyperparameters. While you've discussed the impact of learning rates, have you also examined how the annealing stage affects OCR? A comparison between the final checkpoint and the last checkpoint before the final annealing stage could provide valuable insights.\n\nThis is a great question. We have indeed performed this experiment; the final checkpoint of OLMo-0424 generalizes slightly worse than the preanneal checkpoint for the first-hop and second-hop OCR tasks, and has similar results for the data ordering and the grafting results. We think that it is possible that annealing does make a difference, but without more systematic pretraining-scale experiments or having more details behind the other open-weight but not open-source models such as Llama and Gemini, it is hard to make a strong conclusion. The figure are available at the following external links:\n\n- [Evaluating first-hop and second-hop OCR](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/hop-olmo.pdf)\n- [Data ordering](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/data-olmo.pdf)\n- [Weight grafting](https://pub-76146412ddd24d5d8276a5d3297e09b0.r2.dev/weight-olmo.pdf)\n\n\n> 3. Defining Early and Late Layers: In Table 4, layers 1–24 are categorized as early, and layers 25–32 as late. Could you elaborate on the criteria used to define these layers as early or late? Was this classification based on the visualizations in Figure 5 or another methodology?\n\nThe classification was based on the informative scores in Fig. 5. Specifically we compared the informative scores of the first-hop MLPs and the informative scores of the second-hop MLPs. The reason why we classified based on the MLPs is that the scale of the informative scores for MLPs are much higher than those for attention heads. We'll update the paper to clarify this.", "claims_and_evidence": "The claims made in the paper are well supported.\n\n> Claim 1. The structures consist of *informative components* that store training facts as weight changes, and *upstream* and *downstream extractive* components that query and process the stored information to produce the correct implication.\n\nThe structures are intuitive. To operationalize this, the authors define scores to identify the roles of each LM module. The score visualizations support the proposed structure and align with findings from prior works analyzing pretrained models.\n\n> Claim 2. Our technique reveals that fact learning occurs in both early and late layers, which enable different forms of generalization.\n\nThis claim is supported by layer-freezing ablation in Table 2, where freezing either early or late layers does not hurt fine-tuning fact performance, indicating these facts are distributed across the model. Freezing early layers impairs first-hop implications, while freezing later layers impairs second-hop implications, suggesting that facts in early layers as part of the first-hop informative components enable first-hop reasoning, and facts in later layers enable second-hop reasoning.\n\n> Clam 3. We next study how extractive structures are learned during pretraining and propose a mechanism by which this\noccurs.\n\nThe extractive structures are hypothesized to emerge as models strategically generalize from facts to implications during pretraining, rather than memorizing both simultaneously. The paper supports this by designing synthetic pre-training settings, showing that the model's OCR ability is only non-trivial when facts precede their implications during pretraining.", "methods_and_evaluation_criteria": "I find the proposed evaluations well-designed and effective in demonstrating extractive structures as a plausible mechanism for how LMs generalize to implications of fine-tuned facts. See \"Claims And Evidence\" for details.", "theoretical_claims": "I have not checked the proof in great details for the extractive scores in the Appendices, but the score definitions seem intuitively reasonable to me.", "experimental_designs_or_analyses": "I find the experimental designs and analyses both sound and compelling. See \"Claims And Evidence\" for details.", "relation_to_broader_scientific_literature": "Previous work on out-of-context reasoning shows inconsistent results—fine-tuned facts sometimes enable generalization, but not always.\nThis paper introduces extractive structures—model components that store, retrieve, and process facts—to explain why out-of-context reasoning *occurs*. It also analyzes how these structures are learned during fine-tuning, shedding light on why certain forms of generalization *may fail*, particularly when pretraining data lacks the necessary conditions for these structures to form.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written, easy to follow, and presents a clear, intuitive mechanism for how LMs generalize to implications of fine-tuned facts, backed by thorough experiments and analysis.\n\n2. The proposed extractive structure provides a timely framework that reconciles inconsistencies in the OCR literature (see \"Relation to Broader Scientific Literature\").\n\n3. The experimental design is novel and engaging, with results and visualizations that are clear, interpretable, and strongly support the claims.\n\nWeakness: \n\nI did not find major weakness in this work.", "other_comments_or_suggestions": "This paper is well-written, but I have a few suggestions to improve clarity:\n\n1. On line 24, changing \"training\" fact to \"finetuning\" fact would better emphasize that the fact in question is introduced during the fine-tuning stage.\n\n2.  In the setup paragraph of Section 6.1, it is initially unclear why dax represents facts and wugs represents implications until Appendix D.2 clarifies that there are more names (100) than animals (20). While this distinction may seem minor, moving some dataset design details to the main text could make the synthetic setting clearer from the outset.", "questions_for_authors": "1. Evaluation Details: In Appendix B.1, you mention using the log-probability of the first continuation token for scoring. Could you clarify whether the mean rank is calculated based on this first token alone or on the entire continuation sequence?\n\n2. Training Details: Out-of-context reasoning (OCR) can be sensitive to training hyperparameters. While you've discussed the impact of learning rates, have you also examined how the annealing stage affects OCR? A comparison between the final checkpoint and the last checkpoint before the final annealing stage could provide valuable insights.\n\n3. Defining Early and Late Layers: In Table 4, layers 1–24 are categorized as early, and layers 25–32 as late. Could you elaborate on the criteria used to define these layers as early or late? Was this classification based on the visualizations in Figure 5 or another methodology?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGQ2Fx5iS5", "reviewer": "Reviewer_LrS6", "summary": "This work provides an in-depth analysis on the ability of pretrained language models to generalize from specific facts to broader implications.\nThe authors focus on understanding the underlying mechanisms that allow pre-trained language models to make such generalizations after being finetuned on particular facts.\nThe paper introduces the concept of extractive structures, a novel framework that describes how different components within the Transformer-based models, such as MLP layers and attention modulars, work in concert to enable this generalization.\nThe authors suggest that extractive structures consist of three kind of components: informative components, upstream and downstream extractive components.\nThe paper presents two main predictions based on the extractive structures hypothesis: the data ordering effect and the weight grafting effect.\nEmpirical evidence supporting these predictions is provided through experiments conducted on various large-scale language models。", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We're happy that you believe our extractive structures framework makes sense and that our experiments are sound!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I check the design of extractive structures framework. I think it makes sense.", "experimental_designs_or_analyses": "The experiments part has good soundness for supporting the predictions about data ordering and weight grafting.", "relation_to_broader_scientific_literature": "This work contributes to the interpretability of the learning process and knowledge storage of neural language models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the mechanisms that enable Language Models to perform two-hop implications, i.e. generalization to implications of new facts acquired during fine-tuning that involve chaining a new fact with another known fact. The main hypothesis of the paper is that two-hop implications are enabled by "extractive structures", functional model components that work together to 1) extract relevant information stored in the weights given a prompt, and 2) produce correct implications by processing the extracted information. The paper experimental localizes such extractive structures in attention heads and MLP components of OLMo models during the execution of appropriately constructed synthetic tasks, and proposes learning mechanisms by which extractive structures are embedded into the model during pretraining. Finally, the paper validates two predictions derived from this putative mechanism.

The major weakness pointed out by reviewers are concerns over generalization of the results on LLMs model family beyond OLMo models, which however the authors counter with claims about preliminary results on LLaMa and Gemma models that are consistent with the main hypotheses in the paper.

Other than that, reviewers praised the paper for its clear exposition, for its timeliness and breadth in reconciling inconsistencies in the current related literature, and for the sound experimental designs and analyses which contributes to empirically validating the hypotheses that are put forward as the underlying mechanisms implementing two-hop implications.
Given the current overwhelming interest in reasoning in LLMs, this paper is a timely and relevant contribution to the field, also for how it indicates hypothesis-driven mechanistic studies as viable approaches to investigate the core implementational aspects underlying the reasoning capabilities of LLMs. The paper is therefore recommended for acceptance.
Decision: Accept (poster)