Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
Authors: Laura Ruis
Maximilian Mozes
Juhan Bae
Siddhartha Rao Kamalakara
Dwaraknath Gnaneshwar
Acyr Locatelli
Robert Kirk
Tim Rocktäschel
Edward Grefenstette
Max Bartolo
Venue: ICLR 2025 Poster (2025)
Keywords: large language model; LLM; reasoning; pretraining data; influence functions; mathematical reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify the data from a subset of pretraining data that is influential for downstream reasoning and factual questions for two LLMs of different sizes, and find evidence for a generalisable strategy relying on procedural knowledge.

Abstract: The capabilities and limitations of Large Language Models (LLMs) have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

Published as a conference paper at ICLR 2025
PROCEDURAL KNOWLEDGE IN PRETRAINING DRIVES REASONING IN LARGE LANGUAGE MODELS
Laura Ruis" AI Centre, UCL
Maximilian Mozes Cohere
Juhan Bac
University of Toronto & Vector Institute
Siddhartha Rao Kamalakara Cohere
Dwarak Talupuru Cohere
Acyr Locatelli Cohere
Robert Kirk AI Centre, UCL
Tim Rocktäschel AI Centre, UCL
Edward Grefenstette AI Centre, UCL
Max Bartolo Cohere
ABSTRACT
The capabilities and limitations of Large Language Models (LLMs) have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability solve prob- lems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly dis- tinct sets of data for each factual question, document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like generalisable strategy that synthe- sises procedural knowledge from documents doing a similar form of reasoning.
INTRODUCTION
Current advancements in artificial intelligence are characterised by the increasing scale of datasets, computational power, and model size (aplann et al.) 2020;
Hoffmann et al. 2022). While one of the manifestations of this approach, Large Language Models (LLMs), is rapidly saturating bench-
marks measuring reasoning capabilities (Cobbe et al. 2021 Hendrycks et al. 2021 inter alia), the
ongoing (as reviewed by Mitchell & Krakauer 2023). The well-documented robust and versatile reasoning abilities (Webb et al.) 2023] 2024 McLeish et al. 2024 inter alia) sharply contrast with the line of work highlighting the brittle- ness of LLM reasoning (Razeghi et al. 2022 McCoy et al. 2023 Ullman
debate over whether they exhibit 'genuine understanding
2023:
Wu et al.) 2024
Mahowald et al. 2024). A finding common to these works that LLM reasoning depends on the frequency of similar problems in the training data.
WWrk done while Cohere, correspondence to laura.ruis.2 21 @ ucl.ac.uk

---

Published as a conference paper at ICLR 2025
Reasoning queries
Correlation of influence for pairs of queries
What is the slope the line through (93, 28) and (74. 472
What is the slope of the line
Calculate the unswer: (8 12
Calculate the answer: (7-4)
Facual
Arthmettcc
Slepcs
through (10,
Skppss Aiilunsic Fasual
7B
and (8. 97y
Pearson's
der sloparsl, y1, y01 semum 192 yo) (22 x3)
muhiply: 2x add: 6 24
Step Calenlate the bags of
Zy(a))
so slope
Inthuence magnitude
the lime -13
Lower Higher
Factual queries
Pretraining
0.5
documents:
Facrual
Lineer
Slapes
W the tallest mountain world?
hat is the name the antagomist "Clthella"
which U.K. County Stonehenge?
Which the largesi Earth?
512
million
Tunual
35B
Liner
Monte Everest, montanha mais alta do
Si Stanchange Incated in the countyy Willshire
The largest ocean in the Pacilic Ocean
Othello the character lago
Skpps
Figure 1: A summary of our most important findings towards answering the question "how do LLMs learn to reason from pretraining data?" We rank million pretraining documents according to their influence the likelihood of completions of two models, Cohere's Command R 7B and 35B, for 40 factual and 40 reasoning queries. We find that procedural knowledge drives influence on reasoning traces: a document's influence the reasoning traces of one query is strongly predictive of that document's influence on another query with the same mathematical task, in 3 of the 4 cases. We show this on the left through arrows indicating influence, and on the right through correlations of all 5M document influences between random sample of 10 queries per task (a plot with all queries can be found Figure in Appendix (.9.T). Further, we find that the answers factual queries often show up in the top 0.01 % of pretraining documents (see text in bottom row of documents), but not for the reasoning questions. Finally, individual documents influence reasoning traces much less strongly than factual answer generations, indicating models rely documents less when reasoning. All documents and queries shown are redacted versions of real data, and the relations are based on documents found in the top 50 for the queries.
key reason why benchmark saturation cannot be taken face value is the issue of data contamina- tion: benchmark data often appear in the pretraining set. Where we typically measure generalisation machine learning by separating the test data from the training data, the trillions of tokens used in the design of current slate-of-thh-ee models cannot reasonably be separated from benchmarks anymore. Recent works have documented the extent of the contamination issue (Brown et al.) 2020)
Touvron et al. 2023 Gunasekar et al. 2023 Yang et al. 2023 Deng et al. 2024),, showing that many common benchmarks have a high percentage of contaminated data. Additionally, Yang et al. (2023)
show that even rephrased benchmark data that elude N- N-gram-based detection methods can impact performance, further complicating the issue. However, it is unclear how and when state-of-the-art LLMs rely on contaminated data perform reasoning.
This raises the question: "how do LLMs learn to reason from pretraining data?" In this work, we take complementary approach to most interpretability research by focusing on the pretraining data used by language models to generalise, rather than interpreting the model weights themselves. We investigate which data influence the model's s produced reasoning traces and how those data relate to the specific problems being addressed. Are models simply 'retrieving' answers from previously seen pretraining data and reassembling them, or are they employing a more robust strategy for generalisation? We use technique from robust statistics (Hampel, 1974) adapted to large-scale Transformers (Koh & Liang 2017] Grosse et al., 2023) to compute the influence of pretraining documents on the likelihood of prompt-completions pairs under trained model. In the extreme case, a language model answering reasoning questions may rely heavily on retrieval from parametric knowledge influenced by limited set of documents within its pretraining data. In this scenario,
2

---

Published as a conference paper at ICLR 2025
specific documents containing the information to be retrieved (i.e. the reasoning traces) contribute significantly to the model's output, while many other documents play a minimal role. Conversely, at the other end of the spectrum, the model may draw from broad range of documents that are more abstractly related to the question, with each document influencing many different questions similarly, but contributing a relatively small amount the final output. We propose generalisable reasoning should look like the latter strategy.
We investigate the pretraining data (called 'documents') that are influential for set of factual and reasoning questions (called 'queries"). The reasoning questions cover three mathematical tasks; two-step arithmetic. calculating slopes, and solving linear equations. The factual questions require retrieving from parametric knowledge. We experiment with two LLMs (7B and 35B) and 2.5B of their pretraining tokens. Our findings are as follows (summarised in Figure
1. Procedural knowledge in documents drives influence reasoning traces: docu- ment's 's influence on the reasoning traces of a query is strongly predictive of that document's influence on another query with the same mathematical task [Figure[Taand Finding 1 in Sec- (.1]. By contrast, this does not hold for factual queries. This indicates that documents often contribute similarly many questions that require applying the same procedure to different numbers. The correlation is particularly strong for queries involving calculating a slope, and for that task we find procedures for solution in code or math in the top 0.002% of ranked pretraining data multiple times for most queries (Finding 4 in Section 5.2).
2.
The models rely less on individual documents for reasoning questions, and the set of documents they rely on is less specific: we find that the magnitude of influence of documents per unit of query information generated by the models is usually much lower for reasoning questions than for factual questions (Finding 2 in Section [5.I. Further, the overall magnitude of influence of the set of documents is less volatile. The former indicates that when generating reasoning traces, the models rely less on each individual document per nat of query information they generate than for factual retrieval. The latter indicates that for a random subset of 2.5B pretraining tokens, it is more up to chance whether highly influential documents are part of it for factual questions than for reasoning questions. Taken together. this indicates the models likely generalise from a more general set of documents for reasoning than for factual questions, relying on each individual document less.
3. For the factual questions, the answer often shows up highly influential, whereas for reasoning questions it does not: we look at the top 500 (top 0.01 () influential documents for each query, and find the answer to factual questions relatively often (55% of the queries for the 7B, and 30% for the 35B), and almost never for reasoning questions, even when we do find the answers in the larger set of 2.5B tokens (Finding 3 in Section 5.2
4. We find evidence for code being important for mathematical reasoning: code data is strongly overrepresented w.r.t. the training distribution for the top portions of the positively and negatively influential rankings for reasoning queries (Finding 5 in Section 5.2).
Our findings suggest a generalisation strategy for reasoning that is unlike retrieval from the paramet- ric knowledge formed during pretraining. Instead, the models learn apply procedural knowledge extracted from documents involving similar reasoning processes, either in the form of general de- scriptions of procedures, or applications of similar procedures. This indicates that we may not need to cover every possible case in the pretraining data; focusing on high-quality data demonstrating procedures across diverse reasoning tasks could be more effective. Although our findings are lim- ited to models learning from procedures within the same mathematical task, we observe that code plays a significant role for all tasks we look at. This raises an interesting question: is there a type of pretraining data such as code - from which models, particularly larger ones, can learn about multiple tasks? Understanding the extent of procedural generalisation can inform future pretraining strategies and help determine where to concentrate data selection efforts.
2 RELATED WORK
The subfield with the aim of understanding how large language models generalise is growing rapidly. This question can be approached in different ways, and many recent works interpret weights of smaller models synthetic tasks to explain particular phenomena that we observe in language
3

---

Published as a conference paper at ICLR 2025
models at scale such as grokking (Wang al. 2024). in-context learning OOlsson et 2022 Singh
et al.) 2024), or superposition (Elhage et al. 2022 Bricken et al. 2023). Scaling interpretability
methods to modern-sized LLMs is challenging for many reasons, of which one is computational tractability. Nonetheless, there are a few works that apply techniques from interpretability to lan- guage models at scale. [Templeton et al.] (2024) use sparse autoencoders to extract interpretable features from Claude 3 Sonnet, and demonstrate how to use these features to control model outputs. Grosse al. (2023) adapt EK-FAC influence functions (George et al. 2018) to large-scale Trans- formers, and use them to understand what kind of pretraining data influence completions of models up to 50B parameters. The authors show, among many other things, that larger models rely on pre- training data that are more abstractly related to the completion than smaller models. In this work, we build on the results of Grosse et al. (2023), leaning heavily on their efforts to make influence functions tractable at scale, but focus instead on understanding reasoning specifically.
3 COMPUTING THE INFLUENCE OF A DOCUMENT ON A COMPLETION
Background on influence functions. Given a pretrained model 0 that parametrises a distribution over next tokens conditioned on prompt pa- (y= yp) (where y. (ya,...,yWi) is completion, y, = (311...., Dn a prompt, and u indicates the parameters are not necessarily trained to conver- gence), we are interested in finding data from the pretraining set D = {x that influence the completion. Put differently, we want know which examples in the pretraining set 'caused' com- pletion. To this end, we use EK-FAC influence functions for large-scale transformers as proposed by Grosse et al. (2023). The parameters 0" are typically found by performing a gradient- iterative algorithm on an objective function and stopping based on some criterion. We want to know the influ- ence of training document x, E DD on the parameters 0 (which can be reformulated influence on any continuous differentiable function of 0" using the chain-rule). We can calculate influence exactly by removing Xj from the original training set, re-training the model, and comparing the re- sulting set of parameters (or function thereof) to the originally trained model. This is intractable for any interesting number of documents and parameters. Influence functions estimate this counter- factual by taking Taylor expansion of the response function (shown here for optimal parameters) ! (c) = arg mingeR ((0,Dee) = arg m N Eizy [(x,, 0) + eL(x,, 0), where [() is a loss function, like the cross-entropy. The first-order Taylor approximation around e = 0 of the response function is used to reason about how the optimal parameters change if you change e. which changes the weight of the document we want to know the influence of. Using the implicit function
theorem, infleence can then be defined as follows: Io- (x)
do' = - - 0*). Where
de c=0
H "0"" D) is the Hessian of the objective. Using the chain-rule, we can estimate influence of training document x {I1...., } on the completion given prompt by approximating the following:
(x) = V 0*)
(1)
Since we are investigating models with billions of parameters D, the above Hessian is intractable, and we estimate it using EK-FAC estimation. For a detailed derivation, the reader is referred to Section 2 and 3 in Grosse et al. (2023). We will mention here that it involves estimating two ex- pectations Ey [A and Ep AAA where A denotes the activations of the model. To make this estimation tractable we make a number of simplifying assumptions across all our estimations, like independence between layers and we only take into account MLP parameters of the transformer layers (Grosse et al. 2023). A full list of approximations can be found in Appendix A.7
Adapting EK-FAC influence functions our problem. Prior work has shown that EK-FAC in- fluence functions more accuractely estimate the counterfactual given by the response function than other types of influence functions (Grosse et al.] 2023). However, besides influence on language model completions, when answering questions. We can only calculate the influence on a continuous differentiable func- tion, and to the best of our knowledge, effect on the underlying accuracy of text produced by nexe-token prediction. As proxy for accu- racy, we take continuous differentiable function the cross-entropy loss function (ʃ in Equation
are also interested in influence
the accuracy of trained language model
work has shown that influence functions also estimate
'The actual response function to derive influence functions for non-converged parameters like H is the Proximal Bregman response function. The reader referred to derivation in Grosse et al. (0223.
4

---

Published as a conference paper at ICLR 2025
Table Example from the reasoning set that involves simple two-step arithmetic.
Table Example from the factual set that requires retrieving the right answer.
Reasoning query set (arithmetic)
Factual query set
Prompt
Prompt
Calculate the answer: (7 4) Think step-by-step.
What is the tallest mountain in the world and how tall is it?
Completion (by 7B)
Completion (by 7B)
First, let's evaluate the expression inside the parentheses:
The tallest mountain in the world is Mount Everest, which is located in the Himalayas. It 29,029 feet tall.
7 =443
Now, let's multiply the result by 7:
3 7
Therefore, the answer to the expression is 21.
D. In Appendix A. we show that the influence calculated in this way surfaces documents that have a causal effect on the accuracy a 7B model fine-tuned to do reasoning and reading comprehension tasks. Namely, if we remove documents from the fine-tuning data according to their influence and re-train the model, the accuracy drops significantly more than if we take out the same amount of documents randomly, or the same amount of documents using gradient similarity. In parallel, motivate the use EK-FAC estimations of the Hessian, by showing it significantly improves over a method using only first-order information.
is only reasonably possible to loop over the pretraining data sample once, and to store more than single query gradient in memory (which has the same memory complexity the model itself), Grosse et al. (2023) use singular-valur decomposition (SVD). Instead of SVD, we use approximate SVD with a probabilistic algorithm (Halko al. 2011), which significantly speeds up the compu- tation of the query gradients. We justily each approximation we do in Appendix A.2.
We approximate Equation to get scores for documents from the pretraining data DD that represent the influence they have on completion Ye given a prompt Yp. Given the counterfactual question approximated by the response function, an influence score of 1 implies the log- bog-robbability of the sequence Y. is increased by 1 (Grosse et al. 2023). To compare influence scores across different completions (and token lengths), we normalise the scores for each query by the information content of its completion Yo. measured in nats. The information content of query is defined as 1(y.) - log (pe- (ye yp)). The influence scores induce a ranking over documents from most positively to most negatively influential, where score can be interpreted as the increase (or decrease) in log- probability per nat of query information. The pipeline is shown in Figure 6 in the Appendix.
4 EXPERIMENTAL SETUP
Query set. We collect query set with different types of questions, of which 40 are reasoning questions and 40 factual questions. Note that it is only tractable to loop over the pretraining sample look at once, we need be able to store all query gradients in memory and cannot go beyond about 80 questions. For the reasoning questions, we identify two types of mathematical reasoning each model can do robustly with zero-shot chain-of-thought (Wei et al. 2022). We do this by evaluating the models on larger sets of 100 questions for each type of reasoning, and selecting tasks where it gets at least 80% correct. This surfaces simple two-step arithmetic for the 7B model (Table [] calculating the slope between two numbers for both models (of which two redacted examples are shown in Figure []. and solving for x linear equations for the 35B model (see Table 9 in Appendix AA.3 for prompt-completion pairs of the linear equations task). We ensure no query ever requires outputting a fraction. To make the results between 7B and 35B more comparable, we use the same slope questions for both models. For the 40 factual questions, we make sure the model gets half right and half wrong, allowing us to identify failures retrieving facts from parametric knowledge, and also ensure 16 of 40 overlap between models. We caleulate influence over the full completion. which includes the sfain-of-thoughh in the reasoning case. The query sets are provided in the supplement.
5

---

Published as a conference paper at ICLR 2025
Documents set. We want to compare the influence of pretraining data reasoning by differently sized models (7B and 35B), so we select two models that are trained on the same data. The EK-FAC estimation of the Hessian only needs to be done once per model, but the other terms in Equation require two forward- and backward-pasee that obtaining ranking over pretraining data for single query has computational complexity similar to pretraining itself. To overcome this issue, we sample a set of documents from the pre- training data that covers multiple examples from each batch seen during pretraining, giving a total of 5 million documents (approximately 2.5B tokens) distributed similary as the training distribution. We batch queries and obtain the influence scores in parallel. Each document contains 512 tokens
through the model per document-query pair. This means
EK-FAC estimation. To estimate the Hessian for the 7B and 35B models (the expectations from Section 3)
we randomly sample 100 000 documents equally spread-out through pretraining for both models. Details on how exactly we approximate the Hessian are in Appendix A.2] We note here that although this aspect of the pipeline requires estimating over 300B parameters representing second- order information, the bottleneck remains calculating document gradients.
Models. We look at two models of different sizes, 7B and 35B, which are base and supervised fine-tuned versions of Cohere's Command R series [l We estimate the second order information and calculate document gradients using the base models, and generate completions and calculate the query gradients using the models tune-tuned with supervised instruction-tuning choosing this setup is that the fine-tuned models are much better at instruction following. This means are assuming the EK-FAC for the fine-tuning phase is the identity (Bae al., (024). and we focusing only on the influence of the pretraining data and ignoring the fine-tuning data.
The reason for
5 EXPERIMENTS AND RESULTS
We compare the rankings (from most positively to most negatively influential) over pretraining data produced by influence functions for reasoning questions to the rankings for factual questions (which can only be answered by retrieving parametric knowledge). We first analyse the rankings quanti- information
tatively by looking at the influence of different documents per nat of generated (Section (5.)). We aim to elucidate how generalisable the information in the influential documents is, and how many documents the model is relying on when doing reasoning compared to retrieval. Then, in Section 5.2 we investigate how the documents relate to the queries qualitatively.
5.1 QUANTITATIVE ANALYSIS
Finding 1: There is a significant positive correlation between the influence scores of documents for queries with the same underlying reasoning task, indicating that these documents are relevant for questions requiring the same procedure applied to different numbers.
If models are relying on documents that contain 'general' knowledge that applicable to any query with the same task (e.g. queries that require finding the slope between two points for many different points), we would expect there to be a significant correlation in the influence scores for these queries. We calculate the Pearson's 's R correlation of all 5 million document scores for all query combinations (leading to 802 correlations per model). The results can be seen in the right panel of Figure 1 for a subsample of 10 queries per task, and all query correlations can be found in Figure 12 in Appendix A.9.I We find a strongly significant (p-values all below 4e- - 8) positive correlation between many queries of the same reasoning type, and strongly significant absence of correlation (p- values all around 4e - 3) for most (but not all) factual queries or other combinations (c.g. reasoning queries of different types). This means that many documents have similar influence the same type of reasoning. Given that each type of reasoning query requires applying the same procedure to different numbers. the positive correlation indicates that the influence scores for reasoning queries pick up on procedural knowledge. The correlations are strongest for the slope queries by the 35B model, and this is also the type of reasoning the model can do most robustly compared to solving linear equations. For the model to be able to solve linear equations with an accuracy of more than 80%, we restrict the calculations lead to positive æ. whereas for the slopes questions the answers can be positive or negative. In Appendix A.9. we falsify the hypothesis that the correlations are
"W choose 512 tokens hecause qualitatively interpreting more is hard (usually spanning multiple topics). htttss://cchtit com/ command

---

Published as a conference paper at ICLR 2025
caused by the fact that the reasoning questions superficially similar to each other, by using set of control queries that are also superficially similar but do not require any reasoning and repeating the entire experiment. For the control queries we mostly do not observe correlation. In Appendix A.9.I we highlight examples of queries with high or low correlation for different query sets, finding that some of the correlation seems driven by formatting of reasoning steps, and most by reasoning procedure.
Finding 2: When reasoning, the model on average relies on each individual document less per generated nat of information than when answering factual questions, and the total magnitude of inftuence much less volatile, indicating it is generalising from a more general set of documents. The effect is more pronounced for the larger model.
In Figure
we show the total influence for different percentiles of the positive parts of the rankings.
Reasoning
Facuual
Reasoning
0.14
0.12
influence quury
0.10
0.08
0.06
0.04
0.02
0.00
20
40
80
100
0
20
40
100
Top-k percemile
Top & perceniiee
Figure 2: The total influence per nat of query completion information for different portions of the positive ranking over documents, left for the 7B model, right for the 35B. The total influence per nat is usually lower for reasoning questions than for factual questions, and the influence per document varies more for factual questions than for reasoning questions, especially for the 35B model.
The results depict the total amount of influence contained in the top-k percentile of the positively ranked documents: e.g. the 20th percentile contains 20% of the positive documents for a query. and the amount of total influence shown is the sum of all document influences up to that part of the ranking. The equivalent for the negative portions looks similar (Figure [5] in Appendix (A.92) and the discussion below applies similarly to the negative ranking. We observe two things for both models. Firstly, the amount of total influence for most factual questions at any part of the ranking is higher than for reasoning questions. Secondly, there is more variation in the influence of documents at the same rank across different factual queries (and for few factual queries the amount of influence actually lower than for the reasoning queries, seen more clearly in Figure 20 in Appendix AA..3). The first result means that, average, the models rely on individual documents within set less for generating reasoning traces than for answering factual questions. The second result indicates that for the factual questions the model relies on more 'specific' and infrequent documents: for factual question it is more up to chance whether relatively highly influential documents (w.r.t. influence of documents for other factual questions) are part of the pretraining sample or not.
Influence spread. Another way to analyse the magnitude of influence is to look at the dispersion of influence across the ranking: how much of total influence for each query is contained the top and bottom parts of the ranking? Similarly to what Grosse et al. (2023) report, we observe that the top parts of the rankings over documents follow power law characterised by linear relation between rank and influence per nat in log-log space (shown in Figure 20 in Appendix A.9.3p. We find that the slopes for the reasoning questions by the 35B are slightly steeper than for the factual questions, and therefore the percentage of positive influence contained in the top portions of the rankings for the 35B reasoning questions increases faster with rank than for the factual questions (shown in Figure 22 in Appendix (.9.3). For the 7B, the slopes for the reasoning questions the model gets right are on average also a bit steeper than for the factual questions, but the effect goes away when comparing slopes for all factual the total positive influence the top sequences cover is higher for the reasoning questions than for the factual questions for the 35B model (and similarly for the bottom sequences, Figure 15). There
reasoning queries. This means that the percentage of
7

---

Published as a conference paper at ICLR 2025
is chance this finding is caused by noise for the 35B model and we discuss this possibility more in Appendix A.9.3] where we note that for the reasoning query with the steepest power law, the top document is qualitatively entirely unrelated to the prompt.
If we compare the result between models, we find that the difference in magnitude and volatility are more pronounced for the 35B model across the full rankings. We look into this in Appendix AA.922 and find that the effect remains even if we only look at queries that are the same for both models, which points to higher data efficiency for the larger model.
5.2 QUALITATIVE ANALYSIS
We perform three qualitative analyses on the top portions of the rankings for each query: we search for the answer, we characterise the documents' relation to the reasoning queries, and we investigate what source datasets they are from (for both the top and bottom parts of the ranking, e.g. code, Wikipedia, etc). To filter some of the noise, we divide the influence scores by the document gradient norm and re-rank them, which has empirically been found to help (Choe et al. 2024).
Finding 3: The answer to the factual questions shows up relatively often in the top influ- ential documents for the factual questions, and almost never for the reasoning questions. To find the answer to the questions in the queries in the top documents manually, we construct keywords for each query that should be in the document if the answer is there. For example, for the factual query in Table 2 the keywords are "tallest", "highest", "Mount Everest", "29029", "8848". For the reasoning queries, we construct many more keywords per query, but some examples for the example in Table are 7 4, 3, 21, 3+7, as well as replacing the operations with words like 'minus' and "times', and different ways of represent- ing the content in this query. For details on which key- words we use for each query, see Appendix A.4 We determine the occurrence of each of these keywords in- dependently in the top 100 documents for each query (meaning even if just the keyword '7' is present it would be a hit), resulting in many false-positives. We manually look over the hits to find the answer. On top of that, we craft prompt for Command R+ (a more capable 100B model) to find the answer in a query- document pair, and use it to find the answer in the top 500 documents for each query independent of keyword overlap (the prompt is given in Appendix A.5). Then, we manually look over the hits and keep track of documents that have the answer to query. We verify that Command R+ finds all, and more, of the answers we have identified manually. We look for the full answer in a single document. For the reasoning queries, we also count partial answers in separate documents if they combine to the full answer For example, if one document contains 7 4 3. and another * = 21, we consider that an answer. Finally, we apply the keyword overlap search combined with prompting Command R+ to subset of the broader 2.5B pretraining tokens verify that the answer to the questions are in the entire set even if they do not show up in the top 500 documents for queries.
Fizeal
100%
Anrove in top 500
30%
40%
Resseting
Fraual
Reasozing
Factual
35B
Figure 3: We search for the answer in the top 500 (top 0.01%) documents, and find it relatively frequently for the factual ques- tions. For the reasoning questions, we find the answer twice for the 7B, and never for the 35B. Both those times, the answers to the steps occur in separate documents.
The results are shown in Figure 3 For the 7B model, we find the answer in the top 500 documents for 55% of the factual queries, compared to 7.4% of the reasoning queries. For the 35B model, the answer to the factual queries shows up in the top influential documents 30% of the time, and never for the reasoning set. We expect the answer shows up less frequently for the 35B model simply because the factual questions are much more "niche". For example, one of the questions the model gets correct is "In which year did the Beinecke Library open?". Moreover, in certain cases, the answer shows up multiple times in the top 500 documents. If we count all these separately, as opposed to binary 'yes" or 'no" per query questions show up 30 times for the factual questions in the 7B rankings, and twice for the reasoning questions. For the 35B, the same result is 15 times for the factual questions, and never for the reasoning questions. Interestingly, the answer to the factual questions often shows up in different languages, like Spanish or Portuguese. We give two examples in Appendix A.8.2 To falsify the
which the results in Figure 3 are based, answers to
8

---

Published as a conference paper at ICLR 2025
hypothesis that the answers to reasoning questions not showing up because they not present in the set of 5M documents, we repeat the above keyword search over a random subset of the 5M documents. We identify answers to reasoning steps in documents that do not show up in the top 500 documents for 13 of 20 arithmetic queries and full answer for of 20, and expect more to be there that elude the keyword search. For the slopes and linear equation queries, we find answers to reasoning steps which do not show up in the top 0.01% %. In Appendix A.8.I we show some of these documents and their ranks.
Finding 4: We find that influential documents for the reasoning queries are often doing similar form of step-by-step reasoning, e.g. also arithmetic. Further, we find that the inftuential docu- ments often implement a solution to reasoning questions in code or general math.
For the slope queries (of which we have 20 which are the same for both models), many different documents surface as highly influential that show how to calculate the slope between two points in code or math. For the 7B model, documents that present procedural knowledge on how to calculate the slope in either code math show up in the top 100 documents for 16/20 queries (38 times), and for the 35B model they show up for all queries (51 times). All together, we manually find 7 unique documents that implement the slope in code in the top 100 documents, and 13 that present equations for calculating the slope. The 7B model relies on 18 of these documents for its completions (mean- ing 18 different ones appear in the top 100 documents for all queries), and the 35B on 8. An example of a highly influential document implementing the solution in JavaScript (left) and in maths (right):
Positively influential code
Positively influential math
Cunction. eqoitine ta1, yi, x2, y2) i5 (x1
IF straight line
passing through the points P(21 un J. ((22. u2) making ragle the
1/ Kandle vestical 11ne
positive -xxis, than slope striight line
saturn 'x 52x11"
c1se
(A) vaitian
Calculate the coast. (y2 y1l / x
(B)
(C) 3/2 M1
const io y1 m x1;
22 *1
// Recurn s:mix F(DI')
(D) sinn
Solution:
Concet answer: (C)
We prompt Command R+ to further characterise the top 500 documents for each query by choosing from a set of provided keywords, and find that often the documents are doing similar arithmetic on other numbers (e.g. much larger or smaller), doing similar arithmetic similar numbers (for the slope questions), or similar algebraic operations on similar numbers (for solving linear equations). We present the detailed results and prompt for this analysis in Appendix A.8.3
Finding 5: For factual queries, the most influential data sources include Wikipedia and trivia, while for reasoning, key sources consist of maths, Stack Exchange, ArXiv, and code.
We look at the type of source datasets that represent the most influential documents. Specif- ically, we count the source datasets of the top and bottom k documents with k (50, 500, 5000, 50000, 500000), and compare the count to the pretraining distribution. We present the details in Appendix [A.8.4] but mention here that code data is highly influential for reasoning. StackExchange a source has ten times more influential data in the top portions of the rankings than expected if the influential data was randomly sampled from the pretraining distribution. Other code sources twice influential expected when drawing randomly from the pretraining distribution for l = 50 up to = 50000. Similar patterns hold for the bottom portions of the rankings.
DISCUSSION, LIMITATIONS, AND FUTURE WORK
In this work, we investigate what kind of generalisation strategy two LLMS (7B and 35B respec- tively) employ when reasoning, and contrast it to the strategy used for task that requires retrieving factual parametric knowledge. By creating rankings for 200 such questions over million pretrain- ing documents based on their influence on the likelihood of the completions, generalisation strategy for reasoning is unlike retrieval. More often than not, even the answer is part of the set of pretraining documents look at, it does not show up highly influential as the
conclude that the

---

Published as a conference paper at ICLR 2025
answers to factual questions do. We find that instead, the positively influential documents often con- tain procedural knowledge on how to get to a solution. Further, the models rely less on individual documents when reasoning than when answering factual questions, and the set of documents they rely on is more general. Finally, documents often have a similar influence on reasoning queries that require applying the same procedure to different numbers. These findings can inform pretraining data selection for more robust reasoning: we likely do not need to cover every case in pretraining but can rather focus on data describing and applying procedures to diverse reasoning problems.
We find that the distribution of influence is less spread out for reasoning than for factual questions, characterised by steeper power laws. The distribution of influence over documents tells us something about the type of generalisation strategy the model is using; the more documents that contribute to each nat of query information (i.e. the more spread out the total influence), the more documents the model is relying on to produce the completion. One would perhaps expect steeper power law for factual questions than for reasoning (meaning more of the total positive influence contained at the top parts of the ranking), but our results show evidence for the opposite. Perhaps model needs to generalise from a broader set of documents for factual retrieval than for reasoning because it needs to see the same information more often to memorise it. This is supported by the finding that for factual questions the answer often shows up multiple times in the top 0.01 % most influential data.
There are important limitations to approach, most notably that do not caleulate influence on the entire training set, which is intractable. An alternative explanation of our results is then the opposite conclusion: the model is relying data for reasoning that so infrequent that random sample of 2.5B tokens does not surface relatively highly influential samples for any of the 60 unique reasoning queries. This would result in the conclusion that LLMs rely on sparse and infrequent documents for reasoning. That means we are effectively looking at a set of relatively uninfluential documents for reasoning. and that perhaps the answers to reasoning traces would be highly influential when looking at the entire pretraining data, We would argue that this is the more unlikely explanation for three reasons: (1) the qualitative analysis shows that the influential data for the reasoning questions are intuitively highly relevant, and that the answers to many reasoning traces are part of the 2.5B tokens, they are just not highly influential for reasoning, (2) the correlation of influence scores for the different reasoning tasks is highly significant, and (3) we confirm that these results do not hold for control queries that look similar to the reasoning queries superficially, but do not require step-by-step reasoning. Moreover, it seems unlikely that the model is learning to do retrieval from such infrequent data for one of the simplest forms of mathematical reasoning, namely subtraction and multiplication generalisation strategy that relies on procedural knowledge. Regardless, the nature of interpretability research such as the work presented here is that all we can do is provide evidence, and not proof.
small numbers. Taken together we argue the results indicate
Another limitation is that we do not look at the supervised fine-tuning stage. The reason we only look at the pretraining data is because the fine-tuning stage targeted at making the models more aligned and 'instructable'. and prior work has shown that SFT serves primarily to enhance existing model capabilities (Jain et al.) 2024) Kotha et al., [024] Prakash et al.] (024). Nonetheless, an interesting direction for future work is applying the same method used here to the fine-tuning data.
This work spurs further avenues for future work. Firstly, as previously discussed, identifying data types that are similarly influential across reasoning types could provide additional insight into data selection techniques for improved reasoning. Relatedly, what properties of code data makes it influ- ential for reasoning? What kind is positively influential, and what kind negatively? Further, since we only take into account the feed- forward layers and treat the attention as fixed, an interesting avenue for future work would be to investigate how the relatively low magnitude of influence of pretraining data on feeedforwwate parameters for reasoning traces interacts with attention, connecting to a finding from literature that certain forms of reasoning happen in the attention heads (Olsson et al., (022). Finally, in this work we investigate mathematical reasoning. Future work should verify whether similar results hold for other types of reasoning, such as inductive reasoning.
With this work, we do not claim to say contamination is not an issue, or that LLM reasoning is not brittle and reliant on pretraining statistics. What we demonstrate is that, in principle, it appears possible for LLMs to produce reasoning traces using generalisation strategy that combines infor- mation from procedurally related documents, as opposed to doing a form of retrieval. This is not to say that there are no cases of LLM reasoning where the model is in fact doing retrieval, on the contrary, models can overfit to contaminated data if it appears often enough in the training data.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Although this work is based on proprietary models and pretraining data, we make the following efforts for reproducibility. We add pretraining data with answers to factual and reasoning questions to the supplement, as well as data in which procedures for calculating the slope have been identified. For one of the models we use (the 35B model), the final-stage model (further trained after SFT) is publicly available on HuggingFace We provide all queries, completions, and keywords in the supplemental material. Additionally, we verify that the influence scores generated with our internal codebase correlate with Pearson's ' R of more than 0.99 with public implementation of EK-FAC influence functions (see Appendix A.2.2). Further, we provide details on hyperparameters for every experiment have done at the relevant sections, well as the prompts used to find answers the reasoning questions and characterise the relationship between the query-document pairs (Appendix A.5 and A.6 respectively).
ACKNOWLEDGEMENTS
We'd like to thank Andrew Lampinen, Stephanie Chan, Akbir Khan, and Philipp Jettkant for fruit- ful discussions about the work presented here. This work was supported by the EPSRC Grant EP/S021566/1 and UCL. International Scholar Award for Doctoral Training Centres.
REFERENCES
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadace, Ahmet Üstön, and Sara Hooker. To code, or to code? exploring impact of code in pre-training, 2024.
10914
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentiation, 2024. URL /ttttp::/aaxii.or//bbbbaa
12186
Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying ex- planatory training samples via relative influence. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1899-1909, PMLR, 26-28 Aug 2020. htRp:tts::2 httpttt:tptoceeedeiieeeo http:////ccceeeeeeeeeeeee ml press / 0 baw.barrsaaaaaaa html
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfied-Doddds Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- httt://uu.22222mmmmmmmineeeeeeee
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL. 4
neurips ca ww/pppprppppp..... 1eslleeppp//22222 .pff
Ei
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang. Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric Xing. What your data worth to gpt? lim-scale data valuation with influence functions, 2024. /ttp//////ppss///aavvovooo//////
13954]

---

Published as a conference paper at ICLR 2025
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL Pattps://arte
org abs 2tw.a///222....2 02311
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Dask Development Team. Dask: Library for dynamic task scheduling. 2016. URL http://
dask. pydata.
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Benchmark
probing: Investigating data leakage in large language models. NeurIPS 2023 Workshop
Backdoors in Deep Learning The Good, the Bad, and the Ugly, 2024. URL https:
net http://otontdaddaaaarraooooa
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfelldDodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https:/ttrassformer httpt/pubbb222//ooooooeeeeee
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: ptttps://proceeeinggs wtwp:iinnn.s...ueer. www.ips. cc/paper files//aper/2018// pdf
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilè Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large lan- guage model gencralization with influence functions, 2023. URL https://azxiv..rr
abs/2208.. 03296
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang. Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. BtLLhttps:// waw.ii.ivv............ abs/2306. 11644
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-288, 2011. doi: 10. N0.1137/090771806 URL
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the
American Statistical Association, 69(346):833333, 1974. doi: 10.108001161455.11974
12

---

Published as a conference paper at ICLR 2025
10482962. http:///ww/w.....ao.caaaaaaaaaa. 1974.10482988
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Lau- rent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022. URL hattps://proceeedings neurips cc wappaper.pppp..o..eeeeeeeeeee..e files/paper/20/ Paper Conferenneeeeee pdf
Eile/ htle:/cleeaafffcssss77sss1eeeeee
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert Dick, Hidenori Tanaka, Tim
Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning procedurally defined tasks. In The Twelfth International Conference on Learning
http://wwwwee///uuim.i.ie AOHKeK1 Representations, 2024. URL https: http::: htt::///pppreeiieeeee net/ worummiicioo A 4N1
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL. https:: / axxi v. org/ abs 2001. 08361
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representutions (ICLR), San Diega, CA, USA, 2015.
Pang Wei Koh and Percy Liang Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1885 1885-1894. JMLR.org, 2017.
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forget- ting language models via implicit inference. In The Twelfth International Conference on Learn- ing Representations, 2024. URL htLsstpss:/ /ttp::////peeeviieeeeee wtt.:e/eooooum...d./........... 2id-Vrii22
net/ forum
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing. pp. 785-794, Copenhagen, Denmark, September 2017, Association for Computational Lin- guistics. doi: 10. I/D17- Utthttpss: / ac antholaohhho. www.o/gy..oo./oo/o//o/go/go 1082
Kyle Mahowald, Anna Ivanova, Idan Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28, 03 2024. doi: 10.1016).icss.2024...0000
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve, 2023. URL
13638
Sean McLeish, Arpit Bansal, Alex Stein, Nein, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transform- ers can do arithmetic with the right embeddings, 2024. URL.
0105.17399
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Melanie Mitchell and David C. Krakauer. The debate over understanding in ai's large language models. Proceedings of the National Academy of Sciences, 120(13))e221159071200 2023. doi: 10. 10.1073/pnas:2215900 120. URL AaLattps: www. pnas org/ http:///o///b//////ae//a 1073//0....100000 1073/pnas.
2215007120
13

---

Published as a conference paper at ICLR 2025
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfiel--Dodss, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack
Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. html.
https:/ttanstommerccciiisssss wttp://///ooooootttceaaaaaaaaaaa
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024. URLhttps: peattts:://openeevieee net net/forumm forum?
id (d-8sKcAWOE2D
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent, In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930.
Curran Associates, Inc., 2020. URL
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840- 854, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/2022 12m.://///222...iiiinn----- 2022. windiiinnnenmnnnnn 59.5
URL httpp:://acaanttttttt /
Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, and Andrew M Saxe. What needs to go right for an induction head? mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview. wwt. ftw.t//frrmmmdd=00rrree
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- ci www.ciiiittt...oo.o. pubb/02447
former Circuits Thread, 2024. URL
scal ing- www.mmomoooottooooommoom. iinttiiiitt iiiiy/tttiitcc weeiiit/iiiiindnieeeeeeeee index..d.... html
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL htw.llttss://cc http::::///avv.v../. http://wwwwvvggg//a/// abs 2307 09288
Tomer Ullman. Large language models fail trivial alterations theory-of-mind tasks, 2023. URL
08399
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers implicit reasoners: A mechanistic journey to the edge of generalization, 2024. URL. wwt..httpp..//esi..vrr/g/////
24C5. 15071
Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7:1-16, 07 2023. doi: 10.
14

---

Published as a conference paper at ICLR 2025
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Evidence from counterfactual tasks supports emergent analogical reasoning in large language models, 2024. URL httts:://axxi..tt
abs/2404. 13070]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL htLhhtps:/// http://wppneeeee.eeeeeee .net/ /
forum? ped_==_______ wwQliii0MMM... 1MMBB_
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning reciting? exploring the capabilities and limita- tions of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pp. 1819 1819-1862, Mexico City, Mexico, June 2024, Association for Computational Linguis- tics. doi: 18653vv 1/2024. naw..lloo.....n.......... 102. URL httts:::/aclatthlllooto org/2024.
nsacl- 102
Shuo Yang. Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and lon Stoica, Rethinking benchmark and contamination for language models with rephrased samples, 2023. https:
//Arsiv.crr//abs/2311.0000
A
APPENDIX
Below outline the contents of the appendix. EK-FAC influence functions. In Appendix A.I we discuss the counterfactual re-training experi- ments that motivate our use of EK-FAC influence functions for estimating the effect of pretraining data on the accuracy of downstream behaviour. We describe in more detail how we use influence functions scale in Appendix [A.2] documenting how we estimate the Hessian, how we store many query gradients in memory (each having the same memory complexity as the entire model), and how we sample from the pretraining distribution.
Query sets examples. Then, in Appendix AA.3] we show examples of the reasoning sets that we did not show examples for in the main body of this manuscript.
Finding query answers in documents and characterising document-quers relations. In Appendix [A.4 we discuss how we create keywords for each query order to find the answer in the top documents, and in the sections directly after that, Appendix A.5 and A.6 we give the prompts we used to allow Command R+ to search for answers in the top 500 documents for each query, as well as characterise their relationship.
Limitations. In Appendix A.7 we discuss limitations specific to influence functions. Additional qualitative results. In Appendix ( we provide additional qualitative results.
Answer finding We show examples of answer documents in Appendix I
Cross-lingual transfer. We give some examples of cross-lingual transfer in Appendix A.8. Characterise query-documer
relation. We give detailed results on the characterisation of the relationship between queries and the top 500 documents in Appendix [A.33]
Source-dataset analysis. We analyse which datasets the influential data comes from in Appendix A.8.4
Content analysis of relevant documents. We classify data from the source dataset code for whether it actually contains code in Appendix(A.8. A.8.5
Additional quantitative results. In Appendix A.9 we provide additional quantitative results.
Correlation analysis. Further results for the correlation analysis of influence scores for documents for different queries in Appendix [..9.]
Magnitude of influence. Further results for the magnitude of influence in Appendix A.9.2 Spread of influence. Further results for the spread of influence over the rankings in Appendix A.9.3
15

---

Published as a conference paper at ICLR 2025
A.1 COUNTERFACTUAL RE- TRAINING EXPERIMENTS WITH INFLUENCE FUNCTIONS
We use EK-FAC influence functions to approximate the counterfactual question: which documents from pretraining have a causal effect on the completions of a trained model. However, we are also interested in the causal effect on the accuracy of the completions. In this section, we aim to motivate two aspects of this choice; the fact that influence functions are designed to estimate the effect on continuous differentiable functions, like the log-likelihood, and not on the accuracy. Secondly, we motivate the need for estimating the second-order information of the pretraining objective using EK-FAC, which is very computationally expensive. We present four different experiments in this section, which show that indeed the influence of documents as determined by influence functions also estimate the effect on downstream task accuracy, as well as the benefits from estimating second order information over simply using first-order gradient information.
The pipeline for each of these experiments is similar; take a pretrained model, we fine-tune on some dataset, and evaluate it on 50 validation examples with a metric (perplexity or accuracy). We then use the fine-tuned weights calculate the influence of the documents in the dataset used for fine-tuning on the set of 50 validation questions with two methods: EK-FAC influence functions and TracIn (Pruthi al. 2020). Subsequently, we use those two methods to remove the k most positively influential documents from the fine-tuning dataset, as well as randomly selecting ic documents as baseline, and fine-tune the original pretrained model five times (with different seeds) on each new fine-tuning dataset created (for different values for k). We then calculate the perplexity or accuracy on the validation questions used to calculate the influence, and see how changed. The more changed, the more the documents indeed influence the relevant metric (i.e. perplexity or accuracy). Note that for n different values for k, this requires fine-tuning 3 x * n models: five times for each of the three methods of removing documents from the training set.
We start by motivating the use of EK-FAC influence functions over simple similarity information between document and query gradients. In our setup, where we only have access to the final check- point of pretraining. a dot- dot-product between the query and document gradient effectively boils down to method for estimating influence of documents on queries called TracIn (Pruthi et al., 2020 With access to multiple checkpoints, TracIn uses gradient information from all of them, account- ing for the learning rate used at that point in training. However, we only use the final checkpoint and hence taking into account learning rate only changes scores by a constant. We take GPT-2- small (124M) from HuggingFace and fine-tune it for three epochs with next-word prediction on Wikitext-2 (Merity et | (016)). We use Adam optimizer (Kingma & Ba 2015) with default param- eters (b1 0.9, 62 0.999, eps 1e-8, additive weight decay 0.01). The results can found in Figure 4 and Table 3 showing that removing documents using EK-FAC influence functions has signifi- cantly larger effect on downstream perplexity for all values of We do the exact same experiment but instead remove the most negatively influential documents, and see that instead the perplexity decreases significantly more for EK-FAC influence functions (Figure and Table 4).
Table 3: Wikitext remove top influential
100
250
Random
22.09 + 0.02
22. 22.12-2 = 0.02
22.10 + 0.02
22.20 + 0.06
22.19 0.05
22.15 + 0.05
TracIn
22.16 1 0.02**
22.22 0
22.25 : 0.01**
22.35 : 0.03**
2
22.45 + 0.02**
IF (ours)
22.49 + 0.02**
22.66 = 0.02"*
22.73 0.02"*
22.88 + 0.01**
22.97 + 0.02
23.05 + 0.05**
Table 4: Wikitext remove bottom influential
50
100
150
200
250
300
Ramiom
27.40 1. 0.08
26.24 1. 0.10
25.62 1: 0.15
25.22 1 0.10
25.04 1: 0.12
24.85 1. 0.10
TracIn
26.73 +: 0
25.48
1 0.05"*
24.86 1: 0.02**
24.36
+ 0.04**
24. 4 0.05**
23.94 1 0*03**
(ours) 25.96 1: 0.04**
24.78
0.05"*
23.95 0.03**
23.52 1 0.03**
23.46 0.03**
23.32 1 0.04**
Next, we turn to motivating the use of EK-FAC influence functions in estimating the effect of docu- ments on downstream accuracy of model generations. To this end, we look at two different datasets:
http:::/huggingffac.coo
16

---

Published as a conference paper at ICLR 2025
Randen Tradla (EKFAC)
Basakic Tracla (Lealig))
(EXPAC)
Mear Qpaapy Qharry/ Heglecey
22.6
Meaz Verghedry
22.6
30
100
150
200
250
300
90
100
150
200
250
300
Narlon of Traising Sampllo Reveoved
Namber of Training Sumples Retainel
(a)
(h)
Figure 4: (a) Counterfactual retraining experiments Wikitext-2. We finetuned GPT-2 (124M) on Wikitext-2 and use three different methods to remove training examples from the training set: randomly, TracIn, and Influence Functions (IF). For each number of samples removed the base model five times with different training data ordering, the variance over these runs is repre- sented by the error bars. Each point on the plot is the average perplexity achieved by the five models after fine-tuning on the augmented dataset. We find that influence functions can find examples that impact the perplexity significantly more than baselines. (b) We repeat the same experiment as in (a), but retain top influential queries instead (removing most negatively influential).
finetune
DROP (Dua et al. 2019) and RACE LLai et al. 2017). DROP is a reading comprehension dataset re- quiring different skills like subtraction, addition, coreference resolution, counting, and other skills. The model needs to generate answer that often consists of one few words. We allow the fine-tuned models to generate answers to the questions freely, and evaluate based on exact match. In this experiment,
7B model. We randomly select subset of 8000 examples for fine-tuning, and use the procedure described above to perform counterfactual experiments. We use Adam op- timizer again, with the same hyperparameters 1e-8, additive weight decay 0.01, but only train for one epoch. The results can be found in the left panel of Figure 5 as well in Table 5 We find that EK-FAC influence functions are succesful in selecting data points that impact downstream accuracy, much more so than randomly removing the same amount of training data. For most (all but k 1000), EK-FAC influence functions also have a significantly stronger effect accuracy than TracIn, but the difference is less large. We apply the exact same procedure to the RACE dataset, except now we keep 10k examples (empirically found to lead to the least overfitting when fine-tuning). Further, RACE is multiple-choice dataset, so allow the model to generate a single token indicating the choice, and calculate the accuracy. The results can be seen in Figure 5 and Table 6 Again, the finding is similar, EK-FAC influence func- tions surface documents that have a stronger effect on accuracy than TracIn for all but one value of lt and for all values of & than randomly removing documents. There is large variance in the results for all methods though, which we attribute to the fact that the model sometimes seems to overfit to the fine-tuning data. Further, the reason why the difference between TracIn and EK-FAC influence functions is much larger in the perplexity experiments than in the accuracy experiments could be attributed to the fact that we only fine-tune for one epoch in the accuracy experiments (as more cause overfitting). EK-FAC influence functions differ from TracIn in that they estimate second order information, which becomes more important with more training steps. An interesting avenue for future work is to do counterfactual re-training experiments like these on a subset of pretraining data for a 7B model, but this is incredibly computationally expensive.
for the above experiment: b1 0.9, b2 0.999, eps
Although the results of the experiments in this section are encouraging sign for using EK-FAC influence functions in estimating causal effect of data on accuracy, it important to note that they are limited in several ways. Accuracy is discrete metric and it is prior unclear how many documents need to be removed to flip its value. However, the influence functions we use estimate effect of removing single document, and removing multiple documents can have additional effects that unaccounted for. This makes removing multiple documents a cruder way to empirically show impact of influence functions accuracy, but at the same time is unavoidable. Therefore, any significant
17

---

Published as a conference paper at ICLR 2025
Table Counterfactual re-training accuracies on DROP (free generation of answers). We use three different methods (random, TracIn, influence functions) to remove datapoints, and re-train a model on the resulting dataset. Each number is the mean over five re-training runs with different data ordering. * indicates significantly lower than random with p-value below 0.1 and * with p- value below 0.05. The underlined means are the lowest.
k
500
1000
1500
2000
Random
0.61 0.05
0.60
0.03 0.49 = 0.02
0.56 + 0.05
0.57 + 0.06
TracIn
0.55 = 0.03*
0.44 = 0.04**
0.43 = 0.06**
IF (ours)
0.51 + 0.03*
0.50 0.04
0.40 = 0.05
0.38 + 0.05**
0,000
0.65
Meat Qvary Ao0rr N0012307
0.35
4.825
0,800
Meat
0.750
0.40
Banken
0.725
-
Bandor
Trazin Odaniity)
Tracte (Iceniit)
500
1000
1500
2000
1000
1500
Nunber (CTaiiing Remsvved
Naxiter of Tinining Benoved
(a) Counterfactual retraining experiments on read- ing comprehension questions. We finetuned Cohere DROP training
(b) Counterfactual retraining experiments on multiple-choice reasoning data. We finetuned Cohere Command (7B) on subset the RACE training set (10k examples) and use three different methods to remove training examples from the training set: randomly, TracIn, and Influence Functions (IF). For each number of samples removed base model five times with different training data ordering, the variance over these runs is represented by the error bars. Each point in the plot is the average accuracy achieved by the five models after find that
Command 2 (7B) on subset set (8k examples) and use three different methods to remove training examples from the training set: ran- domly. TracIn, and Influence Functions (IF). For each number of samples removed we finetune the base model five times with different training data order- ing, the variance over these runs is represented by the error bars. Each point in the plot is the average accu- racy achieved by the five models after fine-tuning the augmented dataset. We find that influence func- tions can find examples that impact the accuracy sig- nificantly more than basclines, although only slightly than TracIn
finetune the
fine-tuning on augmented dataset. influence functions can find examples that impact the accuracy significantly more than baselines, although there is some variance the results.
Figure 5: Counterfactual retraining experiments on reading comprehension benchmark DROP (a) and the multiple-choice reasoning dataset RACE (b).
causal effect on accuracy over other methods is good signal, but the absence of significant effect does not necessarily mean EK-FAC influence functions do not properly do what they are designed to do.
18

---

Published as a conference paper at ICLR 2025
Table 6: Counterfactual re-training accuracies on RACE (multiple-choice). We use three different methods (random, TracIn, influence functions) to remove datapoints, and re-train model on the resulting dataset. Each number is the mean over five re-training runs with different data ordering. indicates significantly lower than random with p-value below 0.1 and with p-value below 0.05. The underlined means are the lowest.
k
1000
1500
2000
2500
Random
0.85 + 0.04
0.83 + 0.03
0.82 + 0.04
0.81 + 0.04
TracIn
0.84 0.01
0.78
0.03**
0.80 + 0.03
0.79 + 0.04
IF (ours)
0.80 + 0.04*
0.76 + 0.05**
0.74 + 0.04**
0.74 + 0.05*
19

---

Published as a conference paper at ICLR 2025
A.2 EK-FAC INFLUENCE FUNCTIONS
The code we use for EK-FAC influence functions at scale is a part of larger internal infrastructure, and hence cannot be released publicly. However, we base our code on the public GitHub repository https: / // thub. www. htt///ooonam/kkann www.:/krooflufeeee.eommm uencel We implement estimation of the Hessian in the same way as in that codebase, except for a few changes to make it tractable, which we discuss in more detail below. Further, we compare the results produced by our implementation with the results using the public implementation. We do this by fine-tuning GPT-2 (124M) on Wikitext-2 using internal infrastructure, and calculating influence scores with both code bases. We find that the results correlate very strongly (with a Pearson's 's R of more than 0.99, see 1.2.2Jbelov for more details). Here, we provide details of the design choices and hyperparameters used in our implementa- tion, as well as the additional approximations to make EK-FAC estimation and influence calculation tractable scale.
Query-batching and approximation As mentioned in the main text, we approximate query gradi- ents using approximate SVD (Halko al. 2011). We use the default parameters for this algorithm, which can be found in the Dask documentation (Dask Development Team 2016).
Sampling from the Pretraining Data. It is intractable to calculate influence for the entire pretrain- ing data, so we sample set of 5 million documents. To this end, we loop over the training data as seen by the models in order, and randomly sample 6 examples from each batch. This ensures that the pretraining sample we use is both similar to the pretraining distribution in terms of what kind of data the model sees, as well as when it has encountered the data during pretraining.
Estimating EK-FAC, To estimate the EK-FAC matrices, we sample 100 documents from pre- training in the same manner as described above. We use the same samples to estimate the EK-FAC for the 7B as for the 35B. For both models, we use a damping factor of 0.1 (see Grosse et al. (2023) for details on what the damping factor is). Further, part of estimating the EK-FAC is an eigende- composition on the EK-FAC matrices. We use the same approximation in (Grosse et al.) 2023), namely block-diagonal approximation. For the 7B, we use 2 blocks, and for the 35B, simply amounts to dividing the matrices in n blocks (where n is 2 and 4 in our case), zero-ing out the remaining entries, and taking the eigendecomposition of each block individually. After, these blocks are patched back together again into the original size matrix, which will be further processed as in the public codebase.
empirically motivated
4. The block-diagonal approximation is not part of the public codebase, but
A.2.1 JUSTIPYING A APPROXIMATIONS
In this section, we justify the additional approximations we do on top of those mentioned in Grosse lt al. (2023) by reporting the correlation with the full implementation for smaller model (124M parameters). Applying EK-FAC influence functions to models with billions of parameters requires estimating a multiple of the model parameters. E.g., for the 7B model we estimate around 70B estimate around 320B parameters. Further, to
EK-FAC parameters, and for the 35B model calculate the influence scores for a set of 5 million documents we have to calculate the gradient for 100 queries 5 million documents, each of which has the same size all feed-forward layers in the model itself. We can only afford to loop over the 5 million documents and calculate their gradients once, so we need to batch the query gradients in memory. This is impossible for the full gradients and we use SVD to store low-rank approximations instead, like in Grosse et al. 2023)
Details on the experiment. To compare results of using EK-FAC influence functions with different approximations. we use the same fine-tuned model from Section to calculate influence scores for the 4656 training examples (i.c. documents) on the first 32 validation examples (i.c. queries) of the Wikitext-2 dataset. We repeat this with different types of approximations applied; full SVD on the query gradients, approximate SVD (Dask Development Team 2016) on the query gradients, and a block-diagonal approximation of the EK-FAC matrices before the eigendecomposition (described in Appendix A of Grosse et al. (2023)) with 2 and 4 blocks. For each level of approximation applied, this gives us 32 vectors with 4656 scores (one for each query-document pair), and we compare these to the full implementation without SVD and block diagonal approximations using Pearson's 's R correlation. The correlations reported are the average over all 32 queries, but in the supplement we provide the correlations for each query for all experiments done below.
20

---

Published as a conference paper at ICLR 2025
In Table 7 report the correlations of increasingly more approximations w.r.t. full implementa- tion. Note that the full implementation also uses approximations, but those are all justified Grosse et al. (2023). Here, for completeness, additionally justify the approximations different, namely approximate SVD instead of full SVD, and a block-diagonal approximation with 4 blocks instead of 2. From Table 7 see that the approximate SVD algorithm has neglible effect on the scores, whereas the block-diagonal approximation has small effect on the scores.
that are
Approximations
Pearson R
SVD
0.96 + 0.01
Approximate SVD
0.96 + 0.01
Approximate SVD block diagonal EK-FAC (2 blocks)
0.95 + 0.00
Approximate SVD + block diagonal EK-FAC (4 blocks)
0.93 + 0.00
Table 7: Score correlations of using increasingly more approximations with a full implementation.
A.2.2 FULL IMPLEMENTATION
We also compare the full implementation scores of our own influence functions implementation with the scores calculated for the same model and dataset with the public implementation at com/
and confirm the average score correlation between queries is 0.993 (# 0.003). We add direct score comparison of both methods for the top 3 documents for each of the 32 queries to the supplemental material. Specifically, for each query we log the top documents determined by internal implementation well as the external imple- mentation, showing that they are almost always the same documents, and logging the score given to that document by each implementation (the supplemental file also contains the score correlation for each query separately). The average number of documents that appear in both top 50's determined by the internal and external implementation 46.7. The reason for using an internal implementa- tion nonetheless is that the public implementation is not optimised for usage on large-scale models, and cannot be used for models above about 1B parameters. We used the internal pretraining library for implementing influence functions, because part of the infrastructure used for pretraining large models could be re-used.
https::/githui
21

---

Published as a conference paper at ICLR 2025
Query
Influence Function
Question
Question
Trained Model
Completion
Approximate Hessian
Completion
a=b
Document
Query Gradient
Trained Model
Document Gradient
Pretraining Distribution
Document set
Decacer
Score
Ranking
Question
Math
Code
Completion
Figure 6: The pipeline for creating rankings of the most influential pretraining documents for question-completion pair (query) using influence functions. The documents at the top of the ranking influence the likelihood of the completion positively, and the bottom negatively. We create rankings for a set of 40 reasoning, 40 factual, and 20 control queries over 5 million pretraining documents (2.5B tokens) for two models of different sizes (Cohere's 's Command R series, 7B and 35B).
22

---

Published as a conference paper at ICLR 2025
A.3 QUERY SETS
Reasoning query sets. We show an example of the other two types of reasoning present in the reasoning query sets in Table 8 and [ The former requires calculating the slope of line going through two given points (used for both the 7B and 35B model) and the latter is about solving for x linear equation (only used for the 35B model).
Control query sets. We design two control with 10 questions each for both the 7B and 35B model. These query sets resemble reasoning and factual questions, but do not require actual rea- soning or factual retrieval. The reason for constructing such query sets is compare the results for the factual reasoning query sets to the corresponding control sets, and comment on whether the patterns we are seeing also show up for queries that do not require reasoning retrieval, which further helps interpretations of the results.
Two factual control examples can be seen in Table I0 and [1 for the 7B and 35B model respectively. They are designed to match the first factual query superficially (e.g. the same numbers), but they instead reading comprehension questions about fictional entities. This means all the model needs to do to answer the question is retrieve from context (not parameters).
Two of the reasoning control examples for the 7B and 35B respectively can be seen in Table 12 and 13 Again, these resemble reasoning queries in number and style, but do not require any actual reasoning as the answer is given in the context.
Table Example from the reasoning that involves calculating the slope of line going through two points. We have 20 examples like these on different points. The completion is by the 7B model, but we use the same questions for the 35B model.
Reasoning query set (slopes)
Prompt
What is the slope of the line passing through the points (93, 28) and (74, 47y?
Think step-by-step.
Completion by 7B
To find the slope of the line passing through the points (93, 28) and (74, 47), we can use the following steps:
Identify the coordinates of the two points:
Point A: (93, 28)
- Point B: (74,47
Calculate the difference in x-coordinates between the points:
Difference in x-coordinates: x2 x1
-19
3. Calculate the difference in y-coordinates between the two points:
Difference in y-coordinates: y2 yl 47 28
Determine the ratio of the difference y-coordinates to the difference x-coordinates: Ratio: -1 1/-1
5. Multiply the ratio by -1 reverse the direction of the line:
-Slope: 1 -1
Therefore, the slope of the line passing through the points (93, 28) and (74, 47) is -1.
23

---

Published as a conference paper at ICLR 2025
Table 9: Example from the reasoning set that requires solving for x in a linear equation. We have twenty other examples like these different numbers, only for the 35B model. The completion is by the 35B model.
Reasoning query set (linear)
Prompt
Solve for 5x 21 91
Think step-by stsepp--tep.
Completion by 35B
Here the steps solve for
1. Subtract from both sides the equation:
5x + 21 21 91 21
5x 3x=70
Divide both sides of equation by 5 solve for (5x) -
= 14
Therefore, the solution the equation 5x 21 91 14.
Table 10: Example from the factual control set that resembles the question about the tallest mountain in the world from the 7B factual query (Table 2), but does not require any factual recall. The completion is by the 7B model.
Factual control query set
Prompt
The planet Zog has multiple mountains, of which the tallest called Wirtu which is 29,029 feet tall. What is the largest mountain on the planet Zog called and how tall
Completion 7B
The largest mountain the planet Zog is called Wirtu and is 29,029 feet tall.
Table 11: Example from the factual control set that resembles the first factual query for the 35B model superficially, but does not require any factual recall. The completion is by the 35B model.
Factual control query set
Prompt
In fictional planctary system, there a supernova remmant and pulsar wind nehula in the centre of constellation of stars called Minotaur called the Goat Nebula. What the supernova remnant and pulsar wind nebula in the constellation Minotaur called?
Completion by 35B
In the fictional planetary system, the supernova remnant and pulsar wind nebula in the centre of the constellation Minotaur is called the "Goat Nebula"..
24

---

Published as a conference paper at ICLR 2025
Table 12: Example from the reasoning control set that resembles the slope queries superficially, but does not require any reasoning. The completion is by the 7B model.
Reasoning control query set
Prompt
The slope of the line is -22. What is the slope of the line?
Think step-by-step.
Completion by 7B
Given that the slope of the line is -22.
To find the slope the line, we need to find the change y-coordinates divided by change in x-coordinates.
The slope of the line is -22.
Therefore, the slope the line -22.
Table 13: Example from the reasoning control set that resembles the slope queries superficially, but does not require any reasoning. The completion is by the 35B model.
Reasoning control query set
Prompt
This line has slope 6. What is the slope of the aforementioned line?
Think step-by-step.
Completion by 35B
The slope line a measure of steepness of line. It is defined as ratio the change the y-coordinate to the change the x-coordinate.
In this case, the line has slope of 6. This means that for every unit increase in the x-coordinate, the y-coordinate will increase by units.
It's important to note that the slope of line is not limited to positive values and can be negative or zero depending on the orientation of the line.
25

---

Published as a conference paper at ICLR 2025
A.4 QUERY KEYWORDS FOR FINDING THE ANSWER
We provide all keywords used to search for the answers for both the 7B and 35B in the supplementary material this manuscript, and describe here the procedure for generating keywords, well as giving some examples.
For the factual questions, we generated keywords manually for each query, and these can be found in the supplemental material. For example. for the question "What is the world's smallest mammal by body length?" (answer: bumblebee bat), we have the following keywords: bumblebee bat; bumblebee; bumble; bee: bat; smallest mammal; body length; mammal; smallest; small. This results in many false positives, e.g. if only the word 'small' occurs, which we all check manually for the answer.
Based on the type of reasoning question, we programatically create keywords for each question. For example, for the question in Table 9 the keywords are:
['14', 'x = 14', '5x + 21', ','' '5x + 21 91', '21' '
'91 21', '9 21 70' (91 21) / 5', '0 / 5',
170 5 14', '70', 'x=14', '5x+21', '5x221-91', '91-21',
'91-21=70', ( (91-21))/5)
'70/5', '70/5=14',
(91 - 21) divided by 5', (91-21) divided by 5',
(91 minus 21) divided by 5', (91 min 21) divided by 5
'70 divided by 5', '70 divided by 5 = 14',
'70 divided by is 14', '70
is 14', '70/5 is 14'
'91 - 21 is 70', '91-21 is 70', '91 minus 21 is 70',
'91 min 21 is 70', '70 divided by 5 equals 14',
170 / equals 14', '70/5 equals 14', '91 21 equals 70',
191-21 equals 70', '91 minus 21 equals 70', '91 min 21 equals 70'
15x plus 21', '5x plus 21 91', 5xx plus 21 is 91' '5x 21 is 91'
'91 minus 21', '91 min 21', '9 minus 21 = 70', '91 min 21 70',
(91 minus 21) 5', ' (91 min 21) / 5']
Note that, because the individual numbers '14', '5', '91', and '70' are part of the keywords, each document that contains one of these numbers becomes a hit, and we go over all hits manually.
26

---

Published as a conference paper at ICLR 2025
A.5 PROMPTS GIVEN TO COMMAND R+ FOR FINDING THE ANSWER
We use multiple prompts for each different type of reasoning question allow Command R+ find the answer in the top 500 influential documents; prompts to find the answer to the intermediate reasoning steps, and a prompt for finding the answer to the full question. We provide an example of each below.
Preamble:
You a brilliant Al assistant that is excellent at arithmetic designed to help users with data analysis. You will be given an arithmetic query and a document, and your task is to determine whether the answer to the question in the document.
Prompt for the first step to a two-step arithmetic question
Question:
What also counts
The calculation writee out wonls, ur part of'a story.
The order openatoos changed. E.g.
symbol used suma sarteasbbbccccc sign. Egg. plusiminus.
The calculation part another langer calculation. Egg. (4 2) 9r 2V12
Different formatting. Egg. (4) (2) =
The calculation
algebraic formulation.
2X
What does not count an answer:
Other numbers
summediatbenttedd
Numbers
equals sign.
Document:
docummnn
>>
Is answer given the document? Answer with yes or no. If you answer with yes, indicate where the answer by copying the part the document which the answer occurs, ending with za explanation of why that passage contains the nnswer. Thatk step-by-step and carcfully consader all the different which such
given.
Prompt for the second step to a two-step arithmetic question
Question: 6
Answer:
What also counts as amnwer:
The calculation wrilten out womls, ur part of'a story.
The onder openaions are chinged F.g. 15 6 90, Different symbol used the multiplier
calculation purt anothe langer calculation, times.
=9/12.
Different Rematting F.g. (6) (15) =5)=(00) (90).
The calculation is part algebeaic formulation. Egg. 6X 15X 90X.
What does not count as un answer:
Other numbers are being multpplied. E.g. 7+11
Numbers taken the other side the equals 90/15.
Document:
dociment >
answer given which the answer occurs, ending with an explanation of why that passage contains the answer. Think step by step and curefully comsider the different which such an answer might be given.
yes no. you answer yes, answer copying part
27

---

Published as a conference paper at ICLR 2025
Prompt for step (and 2 similar) answer slope question
Question: 74-33
Answer: 24
73
What also counts unower
The calculation written out words, of
change. Eig. for change) betwean The order operations are changed. E.g. 73
calculation
Different symbol used for the minus sign. E.g. subiracted from.
calculation part mother
calculation.
or 1/12.
locmatting, Egg. (74) The calculation is part algebraic lormulation. Egg. 74X 73X IX.
Other numbers are being subtractd. E.g. 75 73,
Numbers taken the other side of the equals sign. Egg. 74 1 133.
Document:
doccueee
given documennt yes you yes, answer occurs, ending explanation why that passage answer. Thank step-by-stop caretully different ways which such an unswer er might be given.
copying part document
Prompt for step 3 to answer slope question
Question: 74/1
Anower: 74/ 1/1177
What also counts as annwer:
The calculation wrilten out wonls, ur purt of'a story.
The signs on the LHS are flipped. Egg -74/-1 74.
Different symbol used the division
Egg. divided by. purt amother langer calculation.
or77 7012
Different fermatting Eg (1)) (74).
The calculation is part algebracc formulation. Eg. 74X 74X.
What does not count as un answer:
Other numbers are being divided E.g. 75
Numbers taken the other side the equals E.g. 74
Document:
dociment >
answer given which the answer occurs, ending with an explanation of why that passage contains the answer. Think step by step and curefully consider all the different which sach an answer er might be given.
yes you answer yes, answer copying part
28

---

Published as a conference paper at ICLR 2025
Prompt for step to answer linear question
Questio:: 32 16
What also counts unswer:
The calculation written out words, of
calculation
difference change. Eig.
for change) berween
The order operations are changed. E.g. :16 + 32
Different representation used for the minus sign. Egg. 'subtraceed frum'.
calculation
larger calculatoo.
(52- 16M1666222 16112.
locmatting, Egg. (32) (16).
The calculation is part algebraic lormulation. Egg. 32X 16X 16X.
Other numbers are being subtrcctd. E.e. 33
Numbers taken the other side of the equals sign. Egg. 32 16
Document:
doccusee
given documennt? yes you yes, copyinn part document answer occurs, ending with explanation why that passage ways which such an unswer er might be given.
Thank step-by-stop caretully different
Prompt for step 2 to answer linear question
16/8 2
counts as
The calculation wrilten out womls, ur part of story.
The calculation writtn terms of ratio Egg. the ratio between 16 16d is
Different represcntation used for division
calculation purt amother langer calculation 'divided by".
Different Rematting F.g. (16) (8)
The calculation is part algebraic formulation. Egg 32X =X.
What does not count as un answer:
Other numbers are being divided. Eg 17.8.
Numbers taken the other the equals E.g. 16
Document:
dociment >
answer given which the answer occurs, ending with an explanation of why that passage contains the answer. Think step by step and curefullyy comsider the different which such an answer might be given.
yes you answer yes, answer copying part
29

---

Published as a conference paper at ICLR 2025
Prompt for the full answer linear question
Questio::
8x 16
Answer:
What also counts as anower
The calculation written out words, of
calculation
Eig.
Different represeatziin med for the plus sign the equals sign. Eig. 'added and 'equals'. different variable than X used. Eg. Y 81:8 16 322"..
The calculation part moother larger calculation
22 52V12 2/12.
stops
E.g.:
16=32
8x 2
The calculation part algebraic formulation. Egg:
What does being used. E.g.
Document:
<documeent
Is the mnswer gives the document? Answer with yes or no. If you answer with yes, indicate where the nnswer by copyling the part of the document which the answer occurs, ending with zn explanation of why that passage contains the nnswer. Thank step-by-stag and carcfully consader all the different given.
ways which
30

---

Published as a conference paper at ICLR 2025
A.6 PROMPTS GIVEN TO COMMAND R+ FOR CHARACTERISING THE RELATIONSHIP BETWEEN THE QUERY AND THE DOCUMENT
We combine all reasoning queries in pairs with their top 500 most influential documents, and prompt Command R+ to characterise the relationship. For all types of reasoning, we use the same preamble:
You are a brilliant Al assistant that is excellent at arithmetic designed to help users with data analysis. You will be given an arithmetic query and a document, and your task is to characterise the document by choosing keywords from a given set that best describe how the document relates to the question.
For each type of reasoning, we craft prompt that allows Command R+ to choose multiple keywords for each query query-document pair in the top 500 documents. We provide each below.
Prompt for arithmetic questions
Start Query:
Query
Document
<docmmentt>
End of Documeut
related quary?
keywards:
Similar arithmetic operations similar numbers the numbers similar magnitude mumbers the
Similar arithmetic operiioon
lorger smaller) Ressoning (esutiipl rensoning steps are explicitly given document explaining one gets to on answer)
rypes
Other types of maths
Civle that contains arithmetic
types
concerns mulkerithmetic
Text about muth/arihmetii (no other relation the query than that the test about math, test does not perform muthésriffmerre
Superficial similarites (there is real relation, but loosely related lopiic uccur, like the contains wunds related to other math, like algebra) topics
Similer qquestion/anverr pair
Similar Demmnting (other)
Other (sick own keyword)
Explain your answer ench keyward by quoting from the query and document and deseribing why they are similar. Keep mind that the document might be another languge thon English. If you pick any of the cade keyworss, add the proramming laeguges in brackess (e "Cole that continn unihmetic (Python. LaTeX)',, the relation between Otherwise, the document your keyword semiooke-sqummeed your answer of semiculon sepurated list of keywards, and add an explanation below sopansied by newlines (e 'keyoord I: keywurd 2; keyword (Pychon) (explatation)",
keywords, choose 'other' and why. keywords, and explanation below separated by Give your answer the
described by
query
31

---

Published as a conference paper at ICLR 2025
Prompt for slope questions
Start Query:
Query
Documeat
Ein of Document
related query?
Choose the following keywords:
Similar arithmetic operations similar numbers te.g. the aumbers are similar magnitude or the numbers the same)
Similar aritumetic openniios
types mumbers, Ressouing tusittiple rensoning steps explicitly given document explaining one gets answer)
smaller)
Other types of maths
Code that contains arithmetic
Code calesiates
between
Code that calcolates the slope of an equition
Math that calculates the slope of an equatios
rypes
Test about mutb/arthmetic (no other relation the query than that the text about math. text does not perfoom math/urithmessk)
Superficial similarries (there is no real relation, but lonsely related topics occns, like the text contains words related to other of math, like aigehns) topics
Similar fommating qqussiio/anewer pair formatting (other)
Other (jick own keyword]
keyword by quoting might eaother language than English. you pick any of the code keywords, add the programming lasguages in brackets (e.g. "Code that contains arithmetic (Python. LaTeX)), If the relation betwom the and the document not deserbedd by my the given keywords, cheose 'other and pick your keyword that describes the foom senicolon-separaosi 3 iPython) [explantion]) >
Explain your nnswer
query
describing they ure Keep
Otherwiee query related explanation separated by newlines (e.g. 'keyward keyword keyword
relation' and deseribe why
keywords,
32

---

Published as a conference paper at ICLR 2025
Prompt for linear questions
Start Query:
Query
Documeat
Ein of Document
related query?
Choose from the following keywords:
Code that solves linear equation for variable (of the form
b
solves equation multiple variables
hoth variables
solves
equation
Math that solves lincar equation for varialle (of the form ax + b b
Matt that solves equation with mahtiple variobles for both variables (o.g. equations of another equations they are Math that contains lincar equations but they are not solved (of another form than ax b ax h-
Math
-be
contains
Similar algehraic opecations on similar numbers (e.g. the numbers sumilar magnitude the numbers are the same)
Similer olyehmic operations
rypes
e.g. much larger smallers
algehra
Arthmmtic operations
Other types of maths
Code
arithmetic
concerns types
Cide coneerns so muttealgehen
Text about mattvalgahea (no ofher celation the than that the sext abour math, text does not perform mathialgehra) document explaining gets other parts moth, arithmetice) ressoning relation, loosely rolated topics occur, Superficial similarities (there
Ressoning
explicilly
Similar tormoting (question/anvwer pair about other topies than math)
Similar Cermitting (other)
Other (pick keyword)
Explain nsseer for each keyword by quoting from the and document and deseribing why they are similar. Keep mind that the decument keywonls, add the programming languggs brackess "Cude deserited by any
you any language relation between the query arithmeticc (Python. (aTee)) your own keyword that describes the dncumen. Othewwsee the query is act related the docament, state 'no relation' and describee why Give your the form of semicclln--srpe sepurattd list keywords, adi explonation below sepurated by newlines 'krywonl keyword keywod keyword solving equation,
give keywodss chouse
and pick
(Pyyhonn) [esplanation""
equition
33

---

Published as a conference paper at ICLR 2025
A.7 FURTHER DISCUSSION OF LIMITATIONS
More broadly, our work suffers from the same limitations any work does that uses EK-FAC in- fluence functions; we do many approximations estimate the counterfactual and only take into account MLP parameters. This latter decision is because EK-FAC influence functions are not prop- erly defined for the attention layers (Grosse al. 2023), although we do look at the dense layers used within them. We list the assumptions and approximations here:
First-order Taylor approximation to the PBRF.
Assume different layers of MLPs are independent, making the Gauss-Newton Hessian block-diagonal.
Assume activations independent of pre-activation pseudo-gradients
Estimate the approximation to the Fisher Information Matrix or equivalently the Gauss- Newton Hessian by sampling from the empirical data distribution model output distribu- tion, because it's expectation over that distribution (MC estimation).
Block-diagonal approximation of the eigenvector matrices within each layer.
Low-rank approximation of query gradients.
Assume EK-FAC for SFT stage is identity (Bac et al. 2024).
All these approximations are verified and justified Grosse et al. 2023 and (Bac et al. 2024). and
the reader is referred there for more in-depth analysis.
Our empirical results showing that nonetheless influence functions surface documents that are causally related accuracy in Appendix A.I should alleviate some of these concerns, but not all.
34

---

Published as a conference paper at ICLR 2025
A.8 ADDITIONAL RESULTS FOR THE QUALITATIVE ANALYSIS
A.8.1 DETAILS ON ANSWERS TO QUESTIONS IN PRETRAINING DATA
In the main text, we find the answer to factual questions relatively often compared to the answer to reasoning questions. In this section, we comment on the possibility that the answer to reasoning questions are simply not part of the pretraining sample of 5 million documents we look at, well as present examples of documents with answers to queries. Recall that all reasoning tasks require multiple steps, and the model outputs reasoning traces to get to the final answer. This means that if the model is retrieving the answers, it should retrieve answers to all the reasoning steps. On top of the search in the main paper in Section 5.2 search for answers to the reasoning steps and factual questions in a random subset of the 5M pretraining documents. For the 7B reasoning questions, we find 43 documents containing answers reasoning steps, of which only 9 show up in the top 0.02% of the data. Of these 9, 4 documents together contain the 2 answers found for the 7B arithmetic queries in the main text. The remaining 5 answers single reasoning steps that do combine to a full answer. By contrast, we find the full answer to factual questions in 73 documents, of which 35 show up in the top 0.02% of the data. For the 35B, we find documents with answers to reasoning steps, of which 4 show up in the top 0.02% (none combining to full answer). For the factual questions, we find 17 documents with answers, of which 15 show up in the top 0.02%. In terms of full answers showing up in the top 0.02%, we find one additional full answer on top of the ones we found in the main text for the 7B reasoning questions, spread over two documents with rank 896542 and 4997351 of 5 million respectively (i.c. highly un- negatively influential). For the 35B we do not find full answers to reasoning queries at all. We provide many documents with answers to factual and reasoning queries found in the top 0.02% in the supplemental material well one example per reasoning step answer we find (e.g. if we find the answer to 6 - 4 = 2 four times, we show one example in the supplement). We highlight here some examples from the larger pretraining sample for illustration.
Examples of pretraining data with answers.
For factual questions, it happens relatively frequently that the answer to the question shows up as highly influential in multiple documents of the top 10 documents. For example, for the factual question in Table 2 ("What is the tallest mountain in the world and how tall is it?"), the answer shows up at ranks T, 4, 6, and 7. The document at rank (the most positively influential document), is the following, which has the same question and the answer (question 5 below, underlined):
35

---

Published as a conference paper at ICLR 2025
Factual query
What is the tallest mountain the world and how tall is it?
Completion by 7B
The tallest mountain in the world is Mount Everest, which is located between Nepal and Tibet. The height of Mount Everest 8,849 meters (29,029 fect) above sea level.
Rank document
Write some general trivia questions and answers. Here are some general trivia questions and answers:
What is the longest word in the English language?
Answer: The longest word in the English language is "ppecumonoulraaiicrooooip silicovoccancconioiiss" which is type of lung disease.
2. How many bones are in the human body?
Answer: There are 206 bones in the human body.
3. What is the largest ocean in the world?
Answer: The largest ocean in the world is the Pacific Ocean, which covers approxi- mately one-third of the Earth's ' surface.
4. What is the longest river in the world?
Answer: The longest river in the world is the Nile River, which is 6,650 kilometers (4.130 miles) long.
What the highest mountain in the world?
Answer: The highest mountain world Mount Everest, which is 8,848 meters (29,029 feet) high.
What the most common element in the human body?
Answer: The most common element in the human body is oxygen, which makes up approximately 65% of the human body's 's mass.
redacted
36

---

Published as a conference paper at ICLR 2025
Another document has the answer part of arithmetic query for the 7B ("Calculate the answer: (5 3) * 12. Think dep-by-step..", namely 3 2 (underlined below, note that one needs to understand rules to writing arithmetic to figure the answer out):
Reasoning query (arithmetic)
Calculate the answer: (5 3) 12
Think step-by-step.
Completion by 7B
First, let's evaluate the expression inside the parentheses: 5-3=2
Now, let's multiply the result by 12:
2 =4
Therefore, the answer 24.
Rank 2 document
shown in numbers 7, 9 and 10. PEMDAS Rules Practice Solutions
1.
Solution:
Multiply: 3 4 6 24 Add: 6 + 24 = 30
Answer: 30
2.
Solution:
Multiply:
Divide:
Add:
Answer: 0
3.
Solution:
Parenthesis: (6 +==++++++++++++++++
Exponent: + 4 4 16+1
Add: 4 + 16 + 21
Answer: 21
Solution:
Parenthesis: = ( 8(4) 2(2)
Multiply: 8(4) =2 2(2)*
Divide: 32 # 2(2)= 16(2)
Multiply: 16(2) 32
Answer: 32
*This the mentioned in the PEMDAS Rules that when multiplication and divi- sion are performed consecutively (without any other operations or grouping symbols in between), the perform the operations from the left hand side to the right hand side. redacted
Interestingly, this document shows up in the top 10 documents for 11 of 20 arithmetic queries. By contrast, the factual answer document shown before shows up in the top 10 for 4 of 40 queries (we
37

---

Published as a conference paper at ICLR 2025
have another query that asks for the largest ocean in the world, for which this document also has the answer).
To show that answers to more "niche" questions also show up, consider this document that contains the answer to the question "What is the common name for the larva of a housefty?" (answer: maggot, underlined below):
Factual query
What is the common name for the larva of housefly?
Completion by 35B
The common name for the larva of housefly is maggot.
Rank 6 document
Common House Fly
Flies are easily recognized and are universally annoying. To say they are unap- pealing is understatement. Flies enjoy spending time decaying organic filth, including rotting meat and feces. They then manage to enter your home and land on your kitchen counters and any uncovered food. House flies do more than just buzz around and steal your food. They can transfer over 65 diseases including cholera, dysentery, poliomyelitis, and tuberculosis. Flies accumulate nasty pathogens when they rest on sewers, trash, feces, and any other moist or decaying material. Flies lay their eggs on food (fruit your fruit bowl), other animals, and even rotting flesh. When the eggs hatch, the tiny rice-like larvae are called maggots. Few other bugs provoke nausea and an overall disgust like tiny, squirming maggots. There are number of ways to climinate flies and their larvae from your home. The most com- mon practices include using sticky fly traps, ultra-violet lights, and baited traps. Fly swatters effective when trying get rid of the lone fly. A timely smack from dish towel can also be employed.
There simple daily practices that will help keep flies from taking over your house. Store food in sealed containers, wipe crumbs off the counters, and don' 't leave food out. Grass clippings, garbage, and decaying matter will also attract flies, so get rid of these items immediately. Keep your trash cans tightly covered and clean them on man-made surface where the ruf-off can't seep into the soil. Whenever possible, use door and window screens. This will help keep flies from entering your home. How- ever, flies can find holes and gaps so remember to check your screens occasionally and ensure they are impenetrable. Flies can also find openings pipes conduits that lead inside. As a means of prevention, caulk those areas that channel into your home.
At Zip7ap Termite & Pest Control, we know how pesky the persistent buzzing of the house fly can be. With kids running in and out, it is near impossible to keep flies from getting in. Backyard parties are hard to enjoy when you've ve got house flies buzzing around and landing on your food. If you feel you have an overwhelming number of flies
This document has rank 6 for the relevant query, and never shows up in the top 10 for other queries.
Below, show document containing the answer to the reasoning step 5 4 required for one of the arithmetic queries for the 7B model ("Calculate the answer: (5 + 4) 2. Think step-by-step."", which does not show up in the top 0.02%.
38

---

Published as a conference paper at ICLR 2025
Reasoning query (arithmetic)
Calculate the answer: (5 + *
Think step-by-step.
Completion by 7B
First, let's evaluate the expression inside the parentheses:
+ 5
Now, let's move on to the next step: 9 * 2= 18
Therefore, the answer the given expression is 18.
Rank 2 2140 document
For example:
n=1 1+2=3
n=2 1+4=5, 2x3=6
n=3 = 4+5=9, 1+7=8, 2x3=6
The question is, does a solution exist for every n?
I tried writing basic program and it becomes slow after 14. Here are the solutions have so far:
[['++2=3']
2[22*3=6', "1+4=5']
['445=9', '1+7=8', (2*3=61
4 "100=11,,+
225==71
31 14144441,,, '749=16', "14-11,11111111111
5
54712,
6 248=10
7
22+14=16', 3"511" '1400-21,, 16 '8+11=191
['8+14-22', '6+12=18', 7+10=17', 2419=21', 11+15=16', '11+13-24', 445-9', '3+20=23']
1'6+19=25', '8+14=22', '4113117, '2+18220', '1+26=27', '3+7=10', '9+15-24', "5+16221,,11H122233
10 1'6++19225
This document has rank 2140 for the relevant query.
39

---

Published as a conference paper at ICLR 2025
A.8.2 CROSS-LINGUAL TRANSFER
Additional finding: The answer the factual question sometimes shows up in non-English lan- guages.
Interestingly, we observe some crosslingual transfer for the factual questions. For example, for the question about the tallest mountain in the world (Table 2 the answer shows up in Portuguese:
americana Samantha Larson, de anos, se tornou nesta sexta-feira a mulher ex- trangeira mais jovem a conquistar o Monte Everest, segundo nota oficial divulgada pelo Ministério de Turismo do Nepal. A montanha, de 8.848m, mais alta do mundo encontra fronteira entre Nepal Tibet.
Which translates to:
American Samantha Larson, 19, became the youngest foreign woman conquer Mount Everest on Friday, according to an official statement released by Nepal's Ministry of Tourism. The 8,848m mountain is the highest the world and is located on the border between Nepal and Tibet.
We observe more crosslingual transfer for questions, for example for the question "What is the capital of Belgium?" the answer shows in up in French and Spanish. We show the French document here:
Le Premier ministre belge Yves Leterme assuré mercredi qu 'll resterait place et mènerait à bien la réforme instinutioneelee entre les régions, malgré les profondes divi- sions entre Flamands et Wallons qui menacent l'umité du pays.
Les francophones redoutent pour leur part une réduction des budgets accordés la Wallonie, région la plus pauvre du pays, ment également que les régions se sont vu transférer depuis les années 1980 assez compétences fédérales, et soupçonnent les néerlandophones de chercher à faire sécession de la Belgique afin de pouvoir déclarer I' l'inéépendance de la Flandre.
la copitale bilingue, Bruxelles. IIs esti-
Which translates to:
Belgian Prime Minister Yves Leterme assured Wednesday that he would stay in office and earry out the institutional reform between the regions, despite the deep divisions be- tween Flemish and Walloons that threaten the unity of the country.
The French speakers, for their part, fear reduction in the budgets granted to Wallonia, the poorest region of the country, and the bilingual capital, Brussels. They also believe that the regions have been transferred enough federal powers since 1980s, suspect that the Dutch-speaking countries are seeking to secede from Belgium in order to be able to declare the independence of Flanders.
Note that both these quotes are snippets from otherwise larger documents. We did not translate all documents and hence only found cases of crosslingual transfer if there happened to be keyword overlap. We show a few here, but have found the answer to factual questions through keyword overlap with non-English documents times for the 7B model and times for the 35B model. Note that because this is only based on circumstantial keyword overlap, we likely missed most cases of cross-lingual transfer, and therefore cannot assign any meaning the fact that it happened less for the 35B than the 7B. It would be interesting to focus on cross-lingual transfer in future work.
40

---

Published as a conference paper at ICLR 2025
A.8.3 CHARACTERISE RELATION TOP DOCUMENTS TO QUERY
Finding 4: why documents influential for reasoning. We prompt Command R+ characterise the relationship between the top 500 documents and each query (see prompts in Appendix [A.6). We add 'reasoning traces' as a potential keyword in the prompt, but after inspecting the results we find the model uses that keyword for almost any document, and we remove those results. We report the raw counts of each keyword occurring in the tables below.
Arithmetic (7B)
Count
Other types of maths
5765
Similar arithmetic operations on other numbers (e.g. much larger/smaller)
4691
Code that contains arithmetic Text about math/arithmetic
4038
3202
Code that concerns other types of math
2554
Similar arithmetic operations on similar numbers
2246
Similar formatting
2223
Superficial similarities
1391
Code that concerns math/arithmetic
277
Raw counts of the amount of times Command R+ assigns a certain keyword to a query- document pair to characterise its relation, for the arithmetic (7B) queries.
Table 14:
Slopes (7B)
Count
Other types of maths
10787
Similar arithmetic operations on similar numbers
7312
Code that contains arithmetic
5035
Similar formatting
4675
Text that explains in words how to calculate the slope of an equation
3911
Code that concerns other types of math
3577
Text about math/arithmetic
3323
Text that explains in words how to calculate the slope between two numbers
2959
Math that calculates the slope of equation
2921
Math that calculates the slope between two numbers Superficial similarities
2490
2222
Text that mentions the slope but does not explain how to calculate it
1677
Code that calculates the slope between two numbers
1633
Code that calculates the slope of an equation
1110
Code that concerns no math/arithmetic Other
263
15
Table 15: Raw counts of the amount times Command R+ assigns a certain keyword query- document pair to characterise its relation, for the slopes (7B) queries.
41

---

Published as a conference paper at ICLR 2025
Slopes (35B)
Count
Other types of maths
11104
Similar arithmetic operations similar numbers
8340
Code that contains arithmetic
4617
Similar formatting
4141
Text that explains in words how to calculate the slope of an equation Text about math/arithmetic
3869
3845
Math that calculates the slope of an equation
3745
Math that calculates the slope between two numbers
3533
Code that concerns other types of math Text that explains in words how to calculate the slope between two numbers Superficial similarities
3192
2747
2291
Text that mentions the slope but does not explain how to calculate
1936
Code that calculates the slope between two numbers
1150
Code that calculates the slope of an equation
865
Code that concerns math/arithmetic
121
Other
12
Similar arithmetic operations on other numbers (e.g. much larger/smaller)
Table 16: Raw counts of the amount of times Command R+ assigns certain keyword to query- document pair to characterise its relation, for the slopes (35B) queries.
Linear (35B)
Count
Math that contains linear equations but they not solved Similar algebraic operations on similar numbers
13434
10717
Similar formatting
5533
Math that solves a linear equation for variable
2415
Other forms of algebra
2234
Arithmetic operations
2057
Code that contains arithmetic
1417
Other types of maths
1390
Text about math/algebra
1146
Code that solves linear equation of another form than ax b = e or ax b
1109
Superficial similarities
1105
Code that concerns other types of math
949
Code that concerns no math/algebra
560
Code that solves linear equation for variable
475
Math that solves an equation with multiple variables for one or both variables
172
Math that contains linear equations of another form than ax + b
b=c
156
Code that solves a linear equation with multiple variables for one or both variables Other
110
1
Table 17: Raw counts of the amount of times Command R+ assigns a certain keyword to a query- document pair to characterise its relation, for the linear (35B) queries.
42

---

Published as a conference paper at ICLR 2025
andv markdown
general knowledge
muhllingull stcceeshargge
webdato
Training distribution
oode
meth and trivin
wiki
Factual sel
Reasoning set
Mullipiter w.r. trining diaributon
10
10
78
0.1
0.1
500000
5000
100
100
Muliiplter wor.. trining distribution
10
10
351
0.1
0.1
0.01
0.01
50
500
5000
50000
500000
50
500
5000
50000 500000
documents
ks documents
Figure 7: For the reasoning and factual sets, we compare the amount of documents from a certain source dataset that show up in the top portions of the rankings to the amount you would expect to show up you randomly sample from the pretraining distribution (indicated by 'Training distribu- tion' in the figure). The top plots are for the 7B, and the bottom for the 35B. We find that data from Wikipedia and Math & Trivia are important for the factual questions for both models, for the reasoning questions Math & Trivia, StackExchange, Code, and ArXiv data is important. In all cases, the multipliers tend to the training distribution for higher
A.8.4 SOURCE DATASET ANALYSIS
Finding 5: code is heavily overrepresened for reasoning both for the top and bottom portions of the ranking.
For each source dataset, report the multiplier w.r.t. the training distribution. This means that if the top documents are randomly sampled from pretraining, the multipliers will be one, whereas if they are above or below one, that source dataset is either over- or underrepresented in the most influential documents. The full results are presented in Figure ? and we discuss the most interesting deviations from the pretraining distribution here. For the factual questions, the most overrepresented source datasets for both the 7B and 35B are Math & Trivia (multiplier respectively) and Wikipedia (multipliers of 5 and 6 respectively). For the reasoning questions, the most overrepresented datasets are StackExchange and Math & Trivia (with 50 and 24 als multipliers for the 7B, and 62 and 21 for the 35B). Interestingly, for both the 7B and the 35B, code data is important for the influential of the rankings (between documents. Besides StackExchange, for the medium-influential portion 5000 and & 50000), more code data becomes influential (with multipliers around 2, compared to 0.5 for the factual questions at that same part of the ranking). This conventional wisdom among practitioners (most L.LMs designers use some percentage of code data in pretraining now, e.g. Touvron et al. (2023)), and recent work has empirically found code to be important for reasoning performance (Aryabumi et al. 2024). However, the question of why code data is important for reasoning is still open. Below, Appendix A.8.5 we further confirm that code important for reasoning by not only relying on the fact that these documents come from code dataset, but actually classifying their contents. In Figure 8 we present the same plot for the bottom portion of the ranking, showing the findings are similar. Further, in Figure 9 and 10 we respectively show the results for the top and bottom portion of the rankings for the
27 and 16 for & 50
43

---

Published as a conference paper at ICLR 2025
andv markdown
general knowledge
muhilingual stccesshargg
webdatt
Training distribution
oode
meth and trivin
wiki
Factual sel
Reasoning set
Mullipiter triiinng diarrbutio
10
78
0.1
0.1
500000
5000
100
100
Mulliplter trinning distribution
10
10
351
0.1
0.1
0.01
0.01
500
5000
50000
500000
50
500
5000
50000
500000
Bottom's documents
Bottom documents
Figure 8: For the reasoning and factual sets, We compare the amount of documents from a certain source dataset that show in the bottom portions of the rankings to the amount you would expect to show up you randomly sample from the pretraining distribution (indicated by 'Training distri- bution' in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find the patterns are almost identical to those shown for the top portions of the ranking: data from Wikipedia and Math & Trivia important for the factual questions for both models, for the reasoning ques- tions Math & Trivia, StackExchange, Code, and ArXiv data is important. all cases, the multipliers tend to the training distribution for higher k.
control queries. Again, the results look similar (code and StackExchange is also overrepresented for the reasoning control queries), but arXiv is less overrepresented for reasoning control and wiki is less overrepresented for factual control answering.
44

---

Published as a conference paper at ICLR 2025
ande markdown
genera knowledge
muhilingual
webdota
Ttaining distribution
code
stockescharge
Factual control
Reasoning control
100
100
Mulliplier training distribution
10
10
0.1
0.01
0.01
300
5000
50000
500000
500
5000
50000
500000
Mullipiter triining distriuttom
10
10
358
0.1
0.1
500000
5000
Top documents
Top & documents
Figure 9: For the query control sets, we also compare the amount of documents from a certain source dataset that show up the top portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by 'Training distribution' in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find that code is still overrepresented, but arx source is less overrepresented for the top portions of the reasoning control set than for the reasoning set.
45

---

Published as a conference paper at ICLR 2025
marksdown
general knowledge
multlingual
Training distribution
cade
muth and hivin
stockenclange
wiki
Factual control
Reasoning centrol set
100
100
Multipitr training distributio
10
10
0.01
0.01
100
Multipieer WILL training distribution
358
0.1
0.1
0.01
0.01
300
5006
50000
500000
5000 50000
500000
Bottom documents
Bottom documents
For the query control sets, we also compare the amount of documents from a certain source dataset that show up in the bottom portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by 'Training distri- bution' in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find that again looks similar to the source distribution for the top of the rankings for the query control sets.
Figure 10:
46

---

Published as a conference paper at ICLR 2025
A.8.5 CONTENT ANALYSIS OF RELEVANT DOCUMENTS
We provide further insights into the characteristics of influential documents reasoning queries. To do so, we compute capability categories of the = 500 most frequently occurring documents among the k 5000 most (top) or least (bottom) influential documents for the reasoning queries (for the 7B model), and compare these to a randomly sampled set of 500 documents (we repeat the sampling process three times and provide mean standard deviation scores on the detected capabilities). Results are shown in Figure [1 We can see that the "code" category represents the vast majority of most and least influential documents, whereas for the random subsets fraction code-related documents is relatively small. This provides further evidence that code-related documents strongly influence model performance on reasoning tasks.
(7B) Category Distri bution Across Different Datasets
5-100 (randam))
k--000 (bottom)
kw-5000 (rop)
150
300
250
Feequency
200
100
50
cruatie pereration
ginaneed teet
wekssting and
Category
Figure 11: Comparison of capability categories identified for the most and least influential docu- ments for the reasoning queries, well as for random subset of sampled documents. We repeat the random sampling three times and report mean scores with standard deviations indicated.
47

---

Published as a conference paper at ICLR 2025
A.9 ADDITIONAL RESULTS FOR THE QUANTITATIVE ANALYSIS
A.9.1 CORRELATION ANALYSIS
358
The correlation between the influence scores of all million documents for pairs of queries. All queries are the x- and y-axis, with the first 40 belonging to the factual set, the next 40 to the reasoning set (arithmetic and slopes for the 7B, and linear and slopes for the 35B), the following 10 to the factual control set, and the last 10 to the reasoning control The take-away is that there is only signficant correlation between queries of the same reasoning type, most strongly so for the 35B slopes queries.
Figure 12:
7B
35B
Factual
Factual
0.6
Arithmetic
Linear
0.4
Slopes
Slopes
Reasoning control
Reasoning control
0.2
Factual control
Factual control
0.0
Factual
Arithmetic
Slopes
Reasoning control
Factual control
Factual
Linear
Slopes
Reasoning control
Factual control
Figure 13: The average correlations between the influences of all documents for queries a specific type grouped. We leave out any query combinations where the correlation is not significant and any combination where the query on the x- and y-axis the same query. We again observe that there only correlation of influence for queries of the same reasoning type.
Additional results finding 1 (correlation between reasoning queries of the same type). In the main text, find that there is correlation between the influence scores for the documents for different queries that underlie the same type of reasoning question (e.g. questions that all require calculating the slope but for different numbers). One other explanation for this result could be the
48

---

Published as a conference paper at ICLR 2025
fact that all these queries are superficially more similar to each other than the factual questions, and that this is the reason the influence correlates. To test this hypothesis, we use the 10 control queries for both the factual questions and the reasoning questions that are superficially similar, but do not require factual retrieval or reasoning to get to the solution (see Appendix [A.3] for examples all query-query correlations for the 100 queries look at, ordered as follows: 40 factual queries, 40 reasoning (of which 20 are arithmetic and 20 slopes questions for the 7B, and 20 linear equation and 20 slopes questions for the 35B), 10 factual control, and 10 reasoning control queries. We see that there is only significant correlation between queries of the same reasoning type, which we summarise Figure 13 by grouping queries of the same type and averaging the correlations.
from each set). Tigure[22]hool
To get a sense of what the control questions look like, we show a side-by-side comparison of slopes query (top) and the control query that is meant to resemble it (bottom) below. The influences
Table 18: Two query examples, of which one is from the reasoning set (slopes) and the other from the reasoning control set that resembles the slope queries superficially, but does not require any reasoning. The completions are by the 7B model.
Reasoning and Reasoning Control Query Set (7B)
Slopes (original prompt)
What is the slope of the line passing through the points (79, 47) and (80, 25)?
Think step-by-step.
Completion
To find the slope of the line passing through the points (79, 47) and (80, 25),
the following steps:
Identify the coordinates of the two points: Point A: (79, 47) Point B: (80, 25)
Calculate the difference in x-coordinates between two points:
Point A x-coordinate: 79
Point B x-coordinate: 80 redacted completion.
Slopes (control prompt)
The slope of the line is -22 What is slope of the line?
Think step-by-step.
Completion
Given that the slope of the line is -22.
To find the slope of the line, need to find the change in y-coordinates divided by the change in x-coordinates.
The slope of the line is -22.
Therefore, the slope of the line is -22.
for this query correlate with the influences for the slope questions on average with a Pearson's 's of 0.05, which is much smaller than the 0.32 average correlation between the influences found for the different slope reasoning questions by the 7B model. Below, we perform a more detailed qualitative analysis of the query combinations and what drives their correlations, but first we discuss the quantitative result.
As mentioned, we have 10 factual and 10 reasoning control questions for both models, and show the full correlation matrices below in Figure 12 (per query) and Figure 13 (averaged per group). We observe that the correlations between queries from the control sets and other query sets for the 35B is always between 0.05 and 0010, which indicates that there can be score correlation of at least 0.10 for other things than genuine reasoning (c.g. formatting, or topic). Further, the within- group correlations of the reasoning control set sometimes go high 0.38 (although the average
49

---

Published as a conference paper at ICLR 2025
is 0.06 for the 7B and 0.10 for the 35B). For comparison, the average linear-linear score correlation for the 35B is 0.16, and not many of the correlations that make up this average are higher than the correlations in the reasoning control sets. To get sense of how different the correlations are in magniude between the reasoning questions and the control questions, we calculate the highest correlation of a query from specific reasoning type with any other query that does not concern reasoning, and count the amount of reasoning query-query combinations for which the correlation higher. For example, the maximum correlation we find between any slope question for the 35B and any other query that is not slope question is 0.30 Pearson's 's R. If we discard all slope query combinations that are below 0.30 we are left with 138 of 190 significant combinations that are higher, ranging up to 0.96 Pearson's 's R (note that each reasoning group has 20 queries, and all combinations are 20 * 19/2- 190). For the linear equation queries by contrast, there are only 34 of 190 query- query combinations within this group that have correlation higher than the highest correlation with the control queries, ranging up to 0.95 Pearson's 's R. For the 7B, 84 of 190 arithmetic query combinations have higher correlation than the control correlations, ranging up to 0.96 Pearson's 's R, and 120 of 190 slopes query combinations, ranging up to 0.88. We therefore conclude that the correlations between the queries for the linear equations can mainly be explained by other, more superficial things than procedural knowledge, and connect this finding to the fact that the model is less robustly able solve linear equations. The within-group correlations of the factual set are much lower, and for the 7B we only find 5 of 780 correlations that are higher than the maximum correlation of factual query with another query group, ranging to 0.63 Pearson's 's R (we show the queries with the highest correlation below). For the 35B, we find no correlations for factual queries higher than the maximum correlation with another group.
We release all 100 100 query-query correlations in the supplemental material (on which Figure 12 is based), and we highlight few examples here to get a sense of what driving higher or lower correlations. We mostly do not find correlation between the influence scores for different factual questions (on average the correlation is 0.06 for the 7B and 0.03 for the 35B), but we show the highest correlation found between two factual queries below in Table [9] For this particular example, the correlation seems driven by the fact that they are both "what" -questions with very short completions using quotation marks. By contrast, similar style query-completion pair for the 35B shows almost no correlation (Table 20). For the factual control set (the average correlation is 0.04 for the 7B and 0.08 for the 35B, but also show a factual control correlation below for completeness. Table I1 shows a factual query and a matched factual control example for the 35B model. The correlation between this control query and the factual query it matches is -0.03.
Table 19: Two query examples for which the Pearson's correlation of their document influences is 0.63, both from the factual set of the 7B model. We observe that the main similarity between these queries that they both concern short "what" questions, and the completions both are short answers with quotes used.
Factual Query Set (7B)
Factual prompt I
What the symbol for gold on the periodic table?
Completion to factual prompt
The symbol for gold the periodic table is "Au".
Factual prompt 2
What is the common name for the larva of a housefly?
Completion to factual prompt 2
The common name for the larva of a housefly is "maggot"
See Table 22 below for representative high correlation example for the 35B slopes questions (Pear- son's R of 0.89):
We see that the numbers for the questions are different, but the steps get to the answer the same and formatted similarly. We generally observe that a combination of different numbers used in the same procedure and similarly formatted reasoning steps results in higher correlation. Another
50

---

Published as a conference paper at ICLR 2025
Table 20: Two query examples for which the Pearson's correlation of their document influences is 0.08, both from the factual set of the 35B model. We observe that the main similarity between these queries is that they both concern short "what" questions, and the completions both are relatively short answers with quotes used.
Factual Query Set (35B)
Factual prompt
What the name the main antagonist in the Shakespeare play "Othello"?
Completion to factual prompt
The main antagonist in Shakespeare's 's play "Othello" lago, Othello's standard- lago is manipulative and deccitful character who orchestrates the tragedy in the play.
Factual prompt
What is the name of the smallest member of the camel family?
Completion to factual prompt 2
The smallest member of the camel family is called a "dromedary". Dromedaries are easily recognizable by their single hump. They are native to North Africa and the Middle East.
Two query examples for which the Pearson's 's correlation of their document influences is -0.03, one from the factual set (top), and one from the factual control set (bottom). We observe that although the formatting is similar, and there are some overlapping concepts, there is no correlation between the document influences.
Table 21:
Factual and Control Query Set (35B)
Factual prompt
What is the supernoaa remnant and pulsar wind nebula in the constellation of Taurus called?
Completion to factual prompt
The supernova remnant and pulsar wind nebula the constellation Taurus is called the Crab Nebula.
Factual control prompt
In fictional planetary system, there is supernova remnant and pulsar wind nebula in the centre of constellation of stars called Minotaur called the Goat Nebula. What is the supemova remnant and pulsar wind nebula the constellation Minotaur called?
Completion to factual control prompt
In the fictional planetary system, the supemova remnant and pulsar wind nebula in the centre the constellation Minotaur called the "Goat Nebula".
example is shown in Table 23 below for a query combination that gets a lower correlation (0.55), but has the same answer (-22). The relatively lower correlation here seems driven by the different formatting of the completion (less new lines). This indicates that part of the correlations are driven by superficial similarities such as completion formatting.
For comparison, Table 24 shows a pair of control queries for which the prompt and completion are also very similarly formatted, and the correlation is 0.38 (which is the highest correlation we find using a control query).
However, superficial formatting cannot be the full picture, relation is 0.35 but the completions are very dissimilar. For this particular query. it seems that it's not the positive scores that explain the correlation (Pearson's R of about 0.2), but rather the negative scores (Pearson's 's R of 0.44). Perhaps some of the correlation can be explained by documents that generally negatively influential for the types of completions the models generate here.
show in Table 25
where the cor-
Finally, in Table 26 and 27 we show two relatively high correlation examples for the arithmetic queries, again indicating that both underlying operations well as formatting drive correlations.
51

---

Published as a conference paper at ICLR 2025
Table 22: Two query examples for which the Pearson's correlation of their document influences is 0.89, both from the reasoning query set concerning calculating slopes. We observe that, although the reasoning applied to different numbers, the steps follow an identical pattern. In fact, the only difference between the two queries, including completions, is the numbers used.
Reasoning Query Set (35B)
Slopes (prompt !)
What the slope of the line passing through the points (94, 62) and (95, 19)?
Think step-by-step
Completion to prompt
The slope of line passing through two points is calculated using the formula:
Slope (change in y) (change in x)
In this case, we have the points (94, 62) and (95, 19).
Substituting the values into the formula, we have:
Slope (19 /2) (95/ 94)
Slope = (-43)/())
Slope -43
Therefore, the slope of the line passing through the points (94, 62) and (95, 19) is -43.
Slopes (prompt 2)
What the slope of the line passing through the points (90, 20) and (94, 64)?
Think step-by steebby-tep.
Completion to prompt
The slope of line passing through two points is calculated using the formula:
Slope = (change in y) (change in x)
In this case, have the points (90, 20) and (94, 64).
Substituting the values into the formula, we have:
Slope (64 20) (94 90)
Slope (44/4)
Slope 11
Therefore, the slope of line passing through the points (90, 20) and (94, 64) is 11
Taken together, it seems like correlations can be driven by underlying procedures, formatting of the completion, and other more general things (like "what" "hhat"-questions in Table [9] [3). We generally find the highest correlations when procedures and formatting of completions coincide (of which two examples are given in Table 22 and 27). The magnitude of these correlations indicate that almost all of the influence of the 5 million documents in similar for such queries. One interesting possibility is that the query information surrounding the actual numbers generated (which do not seem to drive correlation much at all) is determined by the attention layers (which, besides the dense parameters contained in them, we ignore in this work), connecting potentially to literature attributing reasoning operations to attention heads. An interesting avenue for future work would investigating this further.
52

---

Published as a conference paper at ICLR 2025
Two query examples for which the Pearson's correlation of their document influences is 0.55, both from the reasoning query set concerning calculating the slope. We observe that the second completion structured very similarly to the first, except has less new lines.
Table 23:
Reasoning Query Set (7B)
Slopes (prompt
What is the slope of the line passing through the points (62, 31) and (61, 53)?
Think step-by-step.
Completion
The slope of a line passing through two points is calculated using the formula:
Slope (change in y) (change in x)
In this case, we have the points (62, 31) and (61, 53).
Substituting the values into the formula, we have:
Slope = (31 53) (62 (6)
Slope = (-22) (1)
Slope = -22
Therefore, the slope of the line passing through the points (62, 31) and (61, 53) is -22
Slopes (prompt 2)
What is slope of line passing through the points (79, 47) (80, 25)?
Think step-by-step.
Completion The slope of a line passing through two points calculated using the formula: Slope (change in y) (change in x)
In this case, we have points (79, 47) and (80, 25).
Substituting the values into the formula, we have: Slope (25 47) (80-79)
Slope /-22)/(1)
Slope = -22
Therefore, the slope of the line passing through the points (79, 47) and (80, 25) is -22.
7B 35B
An additional finding that is not central to the research question in this work, but is nonetheless interesting, is that there almost correlation between the influence scores of the two different models. We have 36 queries that share the same prompt for the 7B and 35B (16 factual questions, and 20 slopes reasoning questions) and we can calculate the Pearson's 's R the queries with matched prompts (i.e. 36 combinations). The average correlation of influence scores is 0.02 Pearson's 's R (if we only look at the slopes questions the average correlation 0.03). The maximum correlation we find is 0919, for the question "What is the cupital of Belgium?", which we know from above is not a comparatively high score correlation. Interestingly, for this query, both models produced the exact same completion, and still the correlation is comparatively low. All other query combinations correlate with a Pearson's R below 0. This connects to finding from Grosse et al. (2023) (larger models rely data that is more abstractly related to the prompt): the 35B model relies on very different pretraining data than the 7B, and the same pretraining documents influence completions for the same prompt very differently.
53

---

Published as a conference paper at ICLR 2025
Two query examples for which the Pearson's R correlation of their document influences is 0.38, both from the reasoning control set for the 7B model. We observe that the formatting is very similar, but the correlation is still lower than for the reasoning queries above.
Table 24:
Reasoning Control Query Set (7B)
Reasoning Control (prompt 1)
Find the answer: There are 21 kids in the bus. How many kids are in the bus?
Think step-by-step.
Completion
There are 21 kids in the bus.
Reasoning Control (prompt 2)
Find the answer: There are 132 marbles a jar. How many marbles are in the jar?
Think step-by-step.
Completion
There are 132 marbles in the jar.
Two query examples for which the Pearson's correlation of their document influences is 0.35, both from the reasoning control set for the 35B model. We observe that their completions are very different, indicating that can not only be superficial similarities that drive correlations.
Table 25:
Reasoning Control Query Set (35B)
Reasoning control (prompt 1)
This line has slope 6. What is the slope of the aforementioned line?
Think step-by sttepp.
Completion
The slope of line is measure of the steepness of the line. is defined as the ratio of the change in the y-coordinate the change the x-coordinate.
In this case, the line has slope of 6. This means that for every I unit increase in the x-coordinate, the y-coordinate will increase by 6 units.
It's important to note that the slope of line is limited to positive values and can negative or zero depending on the orientation of the line
Reasoning control (prompt 2)
Find the answer: there are two bananas on tree, how many bananas tree?
Think step step.
Completion
There are two hananas on tree.
54

---

Published as a conference paper at ICLR 2025
Table 26: Two query examples for which the Pearson's correlation of their document influences 0.49, both from the reasoning query set concerning two-step arithmetic. We observe that of the operations required overlapping (+ 6), and the completion somewhat similarly formatted but not fully.
Reasoning Query Set (7B)
Arithmetic (prompt I)
Calculate the answer: (5 6) 10
Think step-by-step.
Completion
First, let's evaluate the expression inside the parentheses: 6+6=11
Now, let's multiply the result by 11 10= 110
Therefore, the answer is 110
Arithmetic (prompt 2)
Calculate the answer: (6 10)
Think step-by-step.
Completion
First, let's evaluate the expression inside the parentheses: 6+ 10 16
Now, let's calculate the result multiplying expression inside parentheses by 6: 16 6 96
Therefore, the answer to the expression is 96
55

---

Published as a conference paper at ICLR 2025
Table 27: Two query examples for which the Pearson's correlation of their document influences is 0.94, both from the reasoning query concerning two-step arithmetic. We observe that not only is the completion very similarly formatted, but one of the operations overlaps (multiplying something by 2).
Reasoning Query Set (7B)
Arithmetic (prompt 1)
Calculate the answer: (5+3)* 12
Think step-by-step.
Completion
First, let's evaluate the expression inside the parentheses: 5-3=2
Now, let's multiply the result by 12: 2 12 = 24
Therefore, the answer is 24
Arithmetic (prompt 2)
Calculate the answer: (4 24) * 15
Think step-by-step.
Completion
First, let's evaluate the expression inside the parentheses: 4-2=2
Now, let's multiply the result by 15: 2 155 = 30
Therefore, the answer is 30
56

---

Published as a conference paper at ICLR 2025
A.9.2 MAGNITUDE OF INFLUENCE
Additional results finding 2 (magnitude influence much lower and less volatile for reason- ing questions)
In the main paper, we find that the influence of documents at the same rank for factual questions is much more volatile than for reasoning questions. We mention that one explanation for this might be that the queries for the 35B model are much more niche, and therefore the relevant documents much more infrequent. To test this hypothesis. (those that are part of both query sets for the 7B and 35B) in Figure [4] We find that the magnitude and variance is still larger for the 35B model than for the 7B model, indicating that the influence of influential documents for the factual and reasoning questions by the 35B can be much larger than for the 7B model. Further, in Figure 15 we show that the results look similar for the negative portions of the ranking (where we flip the influence scores from negative to positive).
plot the same results for only the overlapping queries
Factual
Reasoning
Factual
Reasoning
0.14
0.12
Total qquery
0.10
0.08
0.06
0.04
0.02
20
40
60
90
100
20
40
60
80
100
Top-k pereentile
Top-k percentile
Figure 14: The total influence per nat of query complction information for different portions of the positive ranking over documents, left for the 7B model, right for the 35B. In this case, we only plot queries that are present in the query sets for both models. This means the prompt is the same, but the completion be different. The pattern very similar as the observed pattern for the top of the ranking.
Factual
Ressoning
Factual
Rensoning 35B
0.14
0.12
influence (query
0.10
0.08
0.06
0.04
0.02
0.00
20
80
100
100
Bottom-k percentile
Bottom-k pereentile
The total influence per nat of query completion information for different portions of negative ranking over documents, left for the 7B model, right for the 35B. We again only plot queries that are present in the query sets for both models. In this case, the k-th percentile contains the top % of most negatively influential documents. The pattern is very similar the observed pattern for the top the ranking.
Figure 15:
Finally, in Figure 16 and Figure 17 we plot the same metric for all queries for the top and bot- tom parts of the rankings respectively, now including the 10 control set queries of the factual and
57

---

Published as a conference paper at ICLR 2025
Faciull control 78
Ressoning contral 7B Reasoning
Factual control 338
Factual 35B Ressoning
Factaal
Ressoning contel
0.20
0.18
Tool Influence query
0.15
0.12
0.10
0.08
0.03
0.00
0
20
80
100
0
20
100
Bottom-k percentile
Top-k percentile
Figure 16: The total influence per nat of query completion information for different portions of the positive ranking over documents, left for the 7B model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
Frctial cantrol T8
Ressoning contell Reasoning 7B
Factual contoed 358
Fachall 358
Factual
contel
Total inthence query not
0.18
0.15
0.12
0.10
0.08
0.05
0.03
0.00
0
20
40
60
80
100
0
20
40
60
80
100
percentile
Botiom-k percentile
Figure 17: The total influence per nat of query completion information for different portions of the negative ranking over documents, left for the 7B model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
reasoning control set. As shown in Appendix A.3 we use control queries for each set to investi- gate whether results hold similarly for queries that superficially look similar as the factual/reaoonin questions, but that do not require factual retrieval or reasoning respectively. We observe that the control sets both show much higher variance and magnitude than the reasoning queries as well, for the positive and negative portions of the ranking. For completeness, we show the same result with the number of documents on the x-axis instead of percentiles in Figure [8 and Figure 19 to show that the results are similar we take into account that the 20-th percentile documents for each query contains different amount of documents k.
58

---

Published as a conference paper at ICLR 2025
7B
Fachall 358
Factual
Factual 7B
Reascning
Ressoning connel 35B
Rezsoiing 35B
0.100
0.080
Total inflvence query
0.060
0.040
0.020
0.000
Top-k docs
Top-k docs
Figure 18: The total influence per nat of query completion information for different number of documents the positive ranking, left for the 7B model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
control
control 7B
Factual control
Factual Reasoning 35g
Factual 7B
Ressoning
Reasoning connoll 35B
0.100
0.080
0.060
Toral inflrence
0.040
0.000
0.6
0.2
Bottom-k doca
Bottom-k does
Figure 19: The total influence nat of query completion information for different number of documents the negative ranking, left for the model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
59

---

Published as a conference paper at ICLR 2025
A.9.3
INFLUENCE SPREAD: POWER LAWS
Factual
Reasoning
Factual
Reasoning
10
10
Influence query
10
10
10
10
10'
10'
10"
10'
10 10 10" 10"
10 10"
10"
10" 10" 10"
Figure 20: The ranked influence scores per query nat for each query shown separately in log-log space. We observe; the results follow power laws (linear in log-log space), everything is shifted up for the 35B model (right), generally the scores for the reasoning documents are lower for the 7B model, and for the 35B model there is less variance in magnitude of influence for reasoning queries than for factual queries, and more often than not the influence scores are lower than for factual questions.
Factual
Factual control
Factual
Factual control
Reasoning
Reasoning control
Reasoning
Reasoning
10
Influence/ query not
10
10
10
10
10"
10'
10
10"
10
10
10"
10
10
10"
10"
10"
10
10a
Rank
Figure 21: The ranked influence scores per query nat for each query shown separately log-log space again, but now also showing the control queries. We observe that also for the control queries the influence is much more volatile than for reasoning questions, and on average the magnitude is higher.
In this section, we look the power laws induced by the top portions the rankings. We can fit linear functions to the rankings in log-log space, and analyse the slopes comment on the sparsity of the rankings (i.e. how many documents do models rely on for a completion). Specifically, we perform linear regression on the log-log top 500 rankings of each query, and report the slopes in Table 28
After qualitatively inspecting the queries for the 35B model with the steepest slope, believe an explanation for this result may be 'noise' the influence scores. For example, the query with the steepest slope (a -0.45) has as the most influential document document that is seemingly entirely unrelated to the query. Namely, the query asks the question "What is the slope of the line passing through the points (41, 23) and (18, 92) Think step-by-step.". and the top influential
60

---

Published as a conference paper at ICLR 2025
Table 28: Slopes of the fitted functions to the top 500 documents in the influence rankings in log-log space, separated by query set and whether the model gets the question right or wrong. indicates the significance of independent T-test performed between the slopes of the factual vs. reasoning queries, where * indicates p-value below 0.1 and ** below 0.05.
7B (Correct)
7B (Incorrect)
35B (Correct)
35B (Incorrect)
Reasoning (xx)
-0.36 + 0.03*
-0.33 0.02
-0.36 + 0.04**
-0.38 + 0.04"
Factual (ox) -0.34 + 0.03
-0.34 + 0.04
-0.32 + 0.05
-0.34 + 0.04
document is a snippet about the lunar eclipses and when and where they can be viewed which does not have high N-gram overlap with the query either:
December 8, 1946 Total Lunar Eclipse- Rawaki, Phoenix Islands, Kiribati Max view in Rawaki
Sunday, December 8, 1946 at 5:01 AM
Global Type: Total Lunar Eclipse
Rawaki: Partial Lunar Eclipse
Began: Sun, Dec 8, 1946 at 3:13 AM
Maximum: Sun, Dec 1946 at 5:01 AM
Ended: Sun, Dec 8, 1946 8:22 AM
Duration: hours, 10 minutes December 8, 1946- Total Lunar Eclipse Rawaki
You are using an outdated browser, to view the animation please update or switch modern browser. Allernatively you can view the old animation by clicking here.
Animation: How the Partial Lunar Eclipse Looked The total phase of this lunar eclipse was not visible in Rawaki, but it could be observed there as a partial lunar eclipse.
More about the December 8, 1946 - Total Lunar Eclipse Phases and local times of this eclipse Eclipses visible in Rawaki
All eclipses worldwide, from 1900 to 2100
This is the only query for which we observe an unrelated top document, but for the 35B model we qualitatively observed seemingly irrelevant documents in the rankings more often (in the 7B we did observe this). This connects to a finding from literature that for large models influence functions sometimes surface documents with high gradient norms that are unrelated to the query (Barshan jet al., 2020; Grosse al., 2023; Choe et al., 2024). As Grosse et al. (202)) note,
currently unclear whether this is true noise, whether these are genuinely influential for the completions. Regardless, it seems like noise cannot casily explain the difference between the factual and slopes queries,
would expect noise to show up equally everywhere.
Another way to visualise this result is to plot the percentage of total influence contained in different parts of the top ranking, which we do in Figure 22 below. The results in this plot show that for the top-k percentile of most positively influential documents, the total percentage of positive influence is much higher than & (e.g. 20% of the total positive influence is contained in the top 5% of documents). Here, is clear that on average, for the 35B model the total amount of influence contained in the top-k percentile increases faster for reasoning questions than for factual questions, indicating that a larger portion of the total positive influence is contained in the top portions of the rankings. In Figure 23 we show the same result holds if we include the control queries. Grosse et al. (2023), it is not clear whether this is sensible result to show because for each query we are dividing the total influence at each by the sum of positive influence for that query (perhaps a large part of the positive influence gets cancelled out by negative influence), but we show the result here nonetheless for completeness. We know from the absolute results the total influence at different portions of the ranking that each percentage of total influence at the top-k percentile much lower value in absolute terms for reasoning than for the factual questions. the relative result does not turn out to be noise, it is the case that of the total influence, higher percentage is contained in the top portions
61

---

Published as a conference paper at ICLR 2025
of the rankings for reasoning questions than for factual questions. Taken together with the fact that the absolute influence is often much higher for factual questions, this indicates that the model relies more highly influential documents for factual retrieval than for reasoning. This could indicate that there are more highly relevant factual documents further down the ranking, which makes sense given the fact that the pretraining distribution is dominated by websources and news, which are more likely to contain relevant information for factual question answering than for reasoning. Further, connects to the finding from literature that models need see examples often before text gets memorised (Chowdhery et al. 2022
Rewwning
Factul
Reasoning
Percentage of total inthoence queryy n88
50
25
20
15
Top-k percentile
Top-k percentile
Figure 22: The percentage of total influence per of query completion information for different portions of the positive ranking over documents, left for the 7B model, right for the 35B. We plot only non-control queries.
Reasoning
Factual 7B
Recenning 7B
Reseoning contrel 35B
Ressoning 35B
Percentge total influenee query nat
30
20
15-
Top-k percentile
Top-k percentile
The percentage of total influence per nat of query completion information for different portions of the positive ranking over documents, left for the 7B model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
Figure 23:
Again, the picture looks similar for the negative portions of the ranking, shown for completeness
below in Figure 24 and 25
62

---

Published as a conference paper at ICLR 2025
Fachol 76
Revoning 7B
Facuul 35B
Reasoning 15B
Perenting of total inflience query net
30
25
20
15
10
Bottom-k percentile
Bottom-k percentile
Figure 24: The percentage of total influence per nat of query completion information for different portions of the negative ranking over documents, left for the 7B model, right for the 35B. We plot only non-contron queries.
Factual soniol 78
Resnning contrel 78
Fachal contel 358
Fachal 358
Factual
Reasoring 7B
Ressonin connel
Percentage of total infoence query Diit
30
255
20
10
4
5
0
Bottom-k percentile
Botlom-k percentile
Figure 25:
The percentage total influence per nat query completion information for different portions of the negative ranking over documents, left for the 7B model, right for the 35B. We plot all queries, including the query control sets for both factual and reasoning, which contain 10 queries each.
63


Reviews:
[{"review_id": "3AaaxJ8PZG", "reviewer": "Reviewer_cjVF", "summary": "This paper investigates the role of pretraining data in shaping large language models' (LLMs) abilities in reasoning tasks compared to factual question-answering. By analyzing two models of different sizes (7B and 35B parameters) across reasoning and factual queries, the authors aim to understand how LLMs generalize when tackling reasoning tasks and whether they rely on specific retrieval of information or broader procedural knowledge. The study applies influence functions to rank the most impactful pretraining documents for different queries, examining if reasoning draws from procedural patterns rather than specific facts.\n\nEmpirically, the study finds that reasoning tasks rely on a more distributed set of documents, often containing procedural content like code snippets or mathematical explanations, while factual questions frequently rely on specific documents containing direct answers. Code-based documents, in particular, emerge as influential for reasoning, likely due to their structured, step-by-step nature. Additionally, reasoning tasks across similar queries show correlated influence scores, suggesting a reliance on shared procedural knowledge. The larger 35B model also shows less variation in influence across documents, hinting at improved data efficiency. Together, these findings imply that LLMs approach reasoning by aggregating procedural knowledge rather than retrieving isolated factual data, shedding light on different generalization strategies in LLMs.", "questions": "1. Could you further explain why calculating document gradients with the base model and the query gradients with the fine-tuned model? Could this discrepancy cause any potential problems?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tries to tackle an intellectually significant question: how do LLMs generalize reasoning abilities from pretraining data to solve completion questions? This exploration into the mechanics of LLM reasoning generalization is both timely and meaningful, given the increasing focus on interpretability and robustness in AI. \n2. The findings provide intuitive insights, showing that LLMs draw on a broad range of abstractly related documents when solving reasoning questions, as opposed to the more targeted document reliance seen in factual questions. This highlights the importance of procedural knowledge and coding data for reasoning tasks, an observation that aligns with broader intuitions about reasoning and learning in LLMs.\n3. A key technical strength lies in the revision and adaptation of EK-FAC influence functions. The authors refine this method to assess the influence on model accuracy, which is essential for examining how specific documents impact LLM performance in reasoning versus factual tasks.", "weaknesses": "1. The overall style resembles a blog post, presenting intriguing observations over a cohesive scientific narrative. For example, the conclusion/discussion section takes more than 1 page to explain everything again. The paper could either prioritize the revised EK-FAC function or convert the observations into some actionable strategies to improve LLMs. Additionally, reorganizing the paper to integrate findings more succinctly could create a more cohesive narrative.\n2. Although the paper acknowledges computational constraints, the scale of data and task complexity could be expanded to strengthen the conclusions. The study’s focus on basic arithmetic and simple mathematical queries limits its generalizability to broader reasoning tasks that are common in real-world applications. Also, the study examines only a subset (5 million documents) of the total pretraining data, which may exclude influential documents crucial to understanding the LLMs’ full generalization strategy.\n3. The paper predominantly examines positively influential documents, yet negatively influential documents could offer essential insights into reasoning limitations and biases. Understanding negative influences would allow the authors to identify pretraining data that hinders reasoning or introduces procedural noise, shedding light on inherent biases that might restrict generalization. Only focusing on the positively influential documents might bias our judgements towards cherry-picking conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ENwtu2kP63", "reviewer": "Reviewer_RvSn", "summary": "The paper investigates the generalization strategies employed by LLMs when performing reasoning tasks compared to factual recall. The authors examine the influence of pretraining data on two LLMs of different sizes (7B and 35B parameters) by using influence functions to rank documents based on their impact on the likelihood of model outputs for reasoning and factual questions. They find that for reasoning tasks, LLMs do not rely heavily on direct retrieval of answers from pretraining data but instead use a broader set of documents that contain procedural knowledge relevant to the task. This suggests that LLMs generalize by learning how to perform reasoning steps rather than memorizing specific solutions. In contrast, for factual questions, the influential documents often directly contain the answers. The authors also note the overrepresentation of code in influential documents for reasoning, indicating its importance in teaching procedural knowledge to the models.", "questions": "- Could you elaborate more on how you define \"procedural knowledge\" in the context of your findings? How does this relate to the concept of learning algorithms or routines within the training data?\n- Given the high influence of code documents, how might this skew the model's reasoning capabilities, especially in non-coding contexts?\n- With these insights, what are the potential adjustments or enhancements in training strategies for LLMs to improve their reasoning generalization?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides an important insight of LLMs, namely how models generalize beyond their training data, which is crucial for advancing reasoning capabilities of LLMs.\n- The use of influence functions to study generalization in LLMs offers a good perspective on how models might learn to reason.\n- The experiments are well-executed, and the analysis and explanation for drawing the findings are reasonable.", "weaknesses": "- The study only looks at a subset of the pretraining data, which might not capture less frequent but highly influential documents. \n- Findings are based on two models from the same organization, potentially limiting the generalizability across different architectures or training regimes.\n- There's no cross-validation with other methods of understanding model behavior which could corroborate the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FakUQC0cJj", "reviewer": "Reviewer_6knH", "summary": "This paper applies the EK-FAC influence function to LLMs in an investigation of which documents, from a representative sample of LLM pretraining data, are used by a given model to answer basic mathematical reasoning questions. The EK-FAC influence function is used to produce a score for a given triple (prompt, completion, document) and these scores provide a basis to rank documents as more or less useful in generating the completion for the given prompt. Due to intense computational requirements, this technique is applied on a small sample of 80 prompt/completion pairs, but in great detail, examining several hundred documents at the top of the ranking for each pair. Several key findings emerge, including that models employ documents for reasoning responses in a different manner than for factual responses, and that such mathematical reasoning responses often rely on documents describing verbal procedures or code.", "questions": "I have no serious questions for the authors, but if they have time:\n1. Can this methodology be applied to model false-positives? It would be interesting to explore how pretraining documents may relate to hallucinations in generative responses, given prior research which points to cases of memorization.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper presents a series of interesting and novel investigations into the influence of documents from pretraining in model responses. Most research in model interpretability is done by examining or modulating model parameters and activations, since it is usually computationally intractable to trace model responses back to pretraining samples; this is frontier research, and I was excited to read it.\n\n2. The paper presents insights into which documents are used to answer mathematical reasoning questions, and crucially provides comparisons between two models within the same family, and also to a secondary task in factual question answering. The latter comparison was especially useful and cleanly conveyed the points made: specifically, that factual responses often rely on a specific document, but evidence is shown that reasoning responses may draw on a breadth of documents, possibly aggregating heterogeneous information into one response.\n\n3. The experiments were extremely narrowly defined, but the authors caveat this early and often throughout the paper. Additionally, even in this narrowly scoped setting approximations must be made in order to be computationally tractable, and the authors honestly qualify discussions with reasonable alternate hypotheses and give sub-experiments to explore what is the most likely hypothesis. This kind of writing is very thoughtful and I appreciated that the authors made reasonable decisions and honestly qualified the claims, which were well supported.", "weaknesses": "1. As mentioned above, the experiments were very narrowly scoped. Only 80 questions were analyzed in total, and this 80 was further broken down into smaller sub-groups. Moreover, the questions were very simple mathematical problems using small numbers, requiring only short reasoning hops, and not resulting in fractional-valued answers. The experiments were performed only on two models within one model family, and one model is not available publicly. The authors do note all of these things, and some (not all) of these decisions seem to be made due to computational constraints, which is understandable. However, it would have been nice if these experiments were at least reproduced on fully public models such as Llama.\n\n2. The description of EK-FAC was brief and not as clearly described as the later experiments and results, which were very clear. It would be nice to have a little more motivation about the individual components in the given formulas, since this methodology underlies all of the later experiments. Further, the discussion section at the end of the paper (sec 5) was very dense and a bit confusing. Maybe this could be restructured? The alternating hypotheses in the paragraph starting on L490 were particularly hard to follow.\n\n3. (This is a minor point) Some of the mystique surrounding \"reasoning\" in LLMs may be because as a field we have conflated many types of problems into one, in the fervor of \"AGI\". Though this paper often discusses general reasoning, it looks specifically at mathematical reasoning, and it could be made more clear that these studies are distinct from linguistic reasoning, logical reasoning, spatial, etc etc. Analyzing linguistic reasoning provenance would be fascinating using this method, but would require different experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OOJi0gpZSi", "reviewer": "Reviewer_KXBG", "summary": "This paper explores the influence of specific pretraining data on the reasoning abilities of large language models (LLMs), focusing on how models rely on different types of documents when responding to reasoning versus factual queries. The paper applies influence functions to identify pretraining documents that impact performance on simple reasoning tasks. Results show that factual questions often depend on a smaller set of documents containing the answer, whereas reasoning questions are more influenced by documents with procedural knowledge.", "questions": "1. Why were these two specific LLMs chosen, instead of more widely used and capable models?\n2. Using both fine-tuned and base models in the same experiment could lead to unreliable results due to differences in parameter initialization, potentially affecting influence calculations.\n3. Since LLMs rely on embedded representations, even if keyword matching fails to find an answer, does it conclusively mean the document is not similar to the answer?\n4. Could examples of retrieved documents for reasoning tasks be provided to offer insights into how they influence the model's approach to reasoning?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is straightforward, well-explained, and includes sufficient detail, making it easily reproducible.\n2. This research addresses a crucial question: identifying what training data impacts LLM reasoning abilities, an area closely tied to model generalization and interpretability. It contributes to our understanding of LLMs.\n3. The paper presents intriguing findings, highlighting distinctions in how LLMs handle factual versus reasoning tasks. For instance, factual questions frequently retrieve specific information, while reasoning tasks benefit from procedural knowledge.", "weaknesses": "1. The experimental setup is limited, potentially compromising the reliability of conclusions. Specifically: (1) only 80 queries were used for analysis, (2) the study included only three types of reasoning tasks, potentially limiting representation to other reasoning tasks, (3) there was no exploration of how different prompt formulations of the same query affect results, and (4) keyword-based methods for determining whether documents contain answers may be insufficiently accurate.\n2. The analysis may lack granularity, as it considers only each document’s influence on the overall completion without examining its impact on individual reasoning steps. This might affect the conclusions.\n3. While Appendix A.1 reports that influence scores are higher for certain documents, their similarity to random selections raises questions about whether influence functions reliably indicate actual influence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)