Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

Authors: Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, Max Bartolo
Primary area: foundation or frontier models, including LLMs
Keywords: large language model; LLM; reasoning; pretraining data; influence functions; mathematical reasoning

Abstract:
The capabilities and limitations of Large Language Models (LLMs) have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730279473827

thread_type:
official_review

<thread_object>
creator: reviewer_kxbg

<content>
summary:
This paper explores the influence of specific pretraining data on the reasoning abilities of large language models (LLMs), focusing on how models rely on different types of documents when responding to reasoning versus factual queries. The paper applies influence functions to identify pretraining documents that impact performance on simple reasoning tasks. Results show that factual questions often depend on a smaller set of documents containing the answer, whereas reasoning questions are more influenced by documents with procedural knowledge.

strengths:
1. The proposed method is straightforward, well-explained, and includes sufficient detail, making it easily reproducible.
2. This research addresses a crucial question: identifying what training data impacts LLM reasoning abilities, an area closely tied to model generalization and interpretability. It contributes to our understanding of LLMs.
3. The paper presents intriguing findings, highlighting distinctions in how LLMs handle factual versus reasoning tasks. For instance, factual questions frequently retrieve specific information, while reasoning tasks benefit from procedural knowledge.

weaknesses:
1. The experimental setup is limited, potentially compromising the reliability of conclusions. Specifically: (1) only 80 queries were used for analysis, (2) the study included only three types of reasoning tasks, potentially limiting representation to other reasoning tasks, (3) there was no exploration of how different prompt formulations of the same query affect results, and (4) keyword-based methods for determining whether documents contain answers may be insufficiently accurate.
2. The analysis may lack granularity, as it considers only each document’s influence on the overall completion without examining its impact on individual reasoning steps. This might affect the conclusions.
3. While Appendix A.1 reports that influence scores are higher for certain documents, their similarity to random selections raises questions about whether influence functions reliably indicate actual influence.

questions:
1. Why were these two specific LLMs chosen, instead of more widely used and capable models?
2. Using both fine-tuned and base models in the same experiment could lead to unreliable results due to differences in parameter initialization, potentially affecting influence calculations.
3. Since LLMs rely on embedded representations, even if keyword matching fails to find an answer, does it conclusively mean the document is not similar to the answer?
4. Could examples of retrieved documents for reasoning tasks be provided to offer insights into how they influence the model's approach to reasoning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575714625

thread_type:
official_review

<thread_object>
creator: reviewer_6knh

<content>
summary:
This paper applies the EK-FAC influence function to LLMs in an investigation of which documents, from a representative sample of LLM pretraining data, are used by a given model to answer basic mathematical reasoning questions. The EK-FAC influence function is used to produce a score for a given triple (prompt, completion, document) and these scores provide a basis to rank documents as more or less useful in generating the completion for the given prompt. Due to intense computational requirements, this technique is applied on a small sample of 80 prompt/completion pairs, but in great detail, examining several hundred documents at the top of the ranking for each pair. Several key findings emerge, including that models employ documents for reasoning responses in a different manner than for factual responses, and that such mathematical reasoning responses often rely on documents describing verbal procedures or code.

strengths:
1. This paper presents a series of interesting and novel investigations into the influence of documents from pretraining in model responses. Most research in model interpretability is done by examining or modulating model parameters and activations, since it is usually computationally intractable to trace model responses back to pretraining samples; this is frontier research, and I was excited to read it.

2. The paper presents insights into which documents are used to answer mathematical reasoning questions, and crucially provides comparisons between two models within the same family, and also to a secondary task in factual question answering. The latter comparison was especially useful and cleanly conveyed the points made: specifically, that factual responses often rely on a specific document, but evidence is shown that reasoning responses may draw on a breadth of documents, possibly aggregating heterogeneous information into one response.

3. The experiments were extremely narrowly defined, but the authors caveat this early and often throughout the paper. Additionally, even in this narrowly scoped setting approximations must be made in order to be computationally tractable, and the authors honestly qualify discussions with reasonable alternate hypotheses and give sub-experiments to explore what is the most likely hypothesis. This kind of writing is very thoughtful and I appreciated that the authors made reasonable decisions and honestly qualified the claims, which were well supported.

weaknesses:
1. As mentioned above, the experiments were very narrowly scoped. Only 80 questions were analyzed in total, and this 80 was further broken down into smaller sub-groups. Moreover, the questions were very simple mathematical problems using small numbers, requiring only short reasoning hops, and not resulting in fractional-valued answers. The experiments were performed only on two models within one model family, and one model is not available publicly. The authors do note all of these things, and some (not all) of these decisions seem to be made due to computational constraints, which is understandable. However, it would have been nice if these experiments were at least reproduced on fully public models such as Llama.

2. The description of EK-FAC was brief and not as clearly described as the later experiments and results, which were very clear. It would be nice to have a little more motivation about the individual components in the given formulas, since this methodology underlies all of the later experiments. Further, the discussion section at the end of the paper (sec 5) was very dense and a bit confusing. Maybe this could be restructured? The alternating hypotheses in the paragraph starting on L490 were particularly hard to follow.

3. (This is a minor point) Some of the mystique surrounding "reasoning" in LLMs may be because as a field we have conflated many types of problems into one, in the fervor of "AGI". Though this paper often discusses general reasoning, it looks specifically at mathematical reasoning, and it could be made more clear that these studies are distinct from linguistic reasoning, logical reasoning, spatial, etc etc. Analyzing linguistic reasoning provenance would be fascinating using this method, but would require different experiments.

questions:
I have no serious questions for the authors, but if they have time:
1. Can this methodology be applied to model false-positives? It would be interesting to explore how pretraining documents may relate to hallucinations in generative responses, given prior research which points to cases of memorization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716376726

thread_type:
official_review

<thread_object>
creator: reviewer_rvsn

<content>
summary:
The paper investigates the generalization strategies employed by LLMs when performing reasoning tasks compared to factual recall. The authors examine the influence of pretraining data on two LLMs of different sizes (7B and 35B parameters) by using influence functions to rank documents based on their impact on the likelihood of model outputs for reasoning and factual questions. They find that for reasoning tasks, LLMs do not rely heavily on direct retrieval of answers from pretraining data but instead use a broader set of documents that contain procedural knowledge relevant to the task. This suggests that LLMs generalize by learning how to perform reasoning steps rather than memorizing specific solutions. In contrast, for factual questions, the influential documents often directly contain the answers. The authors also note the overrepresentation of code in influential documents for reasoning, indicating its importance in teaching procedural knowledge to the models.

strengths:
- The paper provides an important insight of LLMs, namely how models generalize beyond their training data, which is crucial for advancing reasoning capabilities of LLMs.
- The use of influence functions to study generalization in LLMs offers a good perspective on how models might learn to reason.
- The experiments are well-executed, and the analysis and explanation for drawing the findings are reasonable.

weaknesses:
- The study only looks at a subset of the pretraining data, which might not capture less frequent but highly influential documents. 
- Findings are based on two models from the same organization, potentially limiting the generalizability across different architectures or training regimes.
- There's no cross-validation with other methods of understanding model behavior which could corroborate the findings.

questions:
- Could you elaborate more on how you define "procedural knowledge" in the context of your findings? How does this relate to the concept of learning algorithms or routines within the training data?
- Given the high influence of code documents, how might this skew the model's reasoning capabilities, especially in non-coding contexts?
- With these insights, what are the potential adjustments or enhancements in training strategies for LLMs to improve their reasoning generalization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730775554301

thread_type:
official_review

<thread_object>
creator: reviewer_cjvf

<content>
summary:
This paper investigates the role of pretraining data in shaping large language models' (LLMs) abilities in reasoning tasks compared to factual question-answering. By analyzing two models of different sizes (7B and 35B parameters) across reasoning and factual queries, the authors aim to understand how LLMs generalize when tackling reasoning tasks and whether they rely on specific retrieval of information or broader procedural knowledge. The study applies influence functions to rank the most impactful pretraining documents for different queries, examining if reasoning draws from procedural patterns rather than specific facts.

Empirically, the study finds that reasoning tasks rely on a more distributed set of documents, often containing procedural content like code snippets or mathematical explanations, while factual questions frequently rely on specific documents containing direct answers. Code-based documents, in particular, emerge as influential for reasoning, likely due to their structured, step-by-step nature. Additionally, reasoning tasks across similar queries show correlated influence scores, suggesting a reliance on shared procedural knowledge. The larger 35B model also shows less variation in influence across documents, hinting at improved data efficiency. Together, these findings imply that LLMs approach reasoning by aggregating procedural knowledge rather than retrieving isolated factual data, shedding light on different generalization strategies in LLMs.

strengths:
1. The paper tries to tackle an intellectually significant question: how do LLMs generalize reasoning abilities from pretraining data to solve completion questions? This exploration into the mechanics of LLM reasoning generalization is both timely and meaningful, given the increasing focus on interpretability and robustness in AI. 
2. The findings provide intuitive insights, showing that LLMs draw on a broad range of abstractly related documents when solving reasoning questions, as opposed to the more targeted document reliance seen in factual questions. This highlights the importance of procedural knowledge and coding data for reasoning tasks, an observation that aligns with broader intuitions about reasoning and learning in LLMs.
3. A key technical strength lies in the revision and adaptation of EK-FAC influence functions. The authors refine this method to assess the influence on model accuracy, which is essential for examining how specific documents impact LLM performance in reasoning versus factual tasks.

weaknesses:
1. The overall style resembles a blog post, presenting intriguing observations over a cohesive scientific narrative. For example, the conclusion/discussion section takes more than 1 page to explain everything again. The paper could either prioritize the revised EK-FAC function or convert the observations into some actionable strategies to improve LLMs. Additionally, reorganizing the paper to integrate findings more succinctly could create a more cohesive narrative.
2. Although the paper acknowledges computational constraints, the scale of data and task complexity could be expanded to strengthen the conclusions. The study’s focus on basic arithmetic and simple mathematical queries limits its generalizability to broader reasoning tasks that are common in real-world applications. Also, the study examines only a subset (5 million documents) of the total pretraining data, which may exclude influential documents crucial to understanding the LLMs’ full generalization strategy.
3. The paper predominantly examines positively influential documents, yet negatively influential documents could offer essential insights into reasoning limitations and biases. Understanding negative influences would allow the authors to identify pretraining data that hinders reasoning or introduces procedural noise, shedding light on inherent biases that might restrict generalization. Only focusing on the positively influential documents might bias our judgements towards cherry-picking conclusions.

questions:
1. Could you further explain why calculating document gradients with the base model and the query gradients with the fine-tuned model? Could this discrepancy cause any potential problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731943122062

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers: limited scope of experiments

comment:
We agree with the reviewers that the scope of tasks and models we look at is narrow. On the other hand, we do 1B LLM-sized gradient dot products (100 queries * 2 models * 5M) for these experiments; in that sense the scope is very large compared to prior interpretability research. We view the task scope as a limitation that was necessary to answer our research question, and not a weakness. We highlight this in the submission: *“All we showed is that in principle it seems to be possible for LLMs to produce reasoning traces using a generalisation strategy that combines information from procedurally related documents, as opposed to doing a form of retrieval.”*. Reviewer **6knH** calls this out as a **strength**: *“The experiments were extremely narrowly defined, but the authors caveat this early and often throughout the paper [ ..] I appreciated that the authors made reasonable decisions and honestly qualified the claims, which were well supported”* . We pushed the compute we had to the limit, and made careful design decisions to answer our research question, which we will explain below.

**Compute and memory**  
We used 379,392 TPU v5 chip-hours and 45,056 TPU v4 chip-hours (https://cloud.google.com/tpu/pricing#regional-pricing for reference only), which we parallelised to get it down to about ~3 months of consecutive computations. Further, fitting more than 100 35B query gradients on our largest TPU was impossible, and looping over the pretraining sample twice would almost double the compute required. For comparison, the entire Pythia pretraining suite of models required 544,280 A100 hours (see Appendix D in the Pythia paper).

**Tasks**  
We chose mathematical reasoning for two reasons: it has well-defined answers to intermediate steps and we can easily generate questions that underlie the exact same procedure but that use different numbers. We wanted to look at at least 2 tasks per model, but could not fit more than about 100 query gradients on the largest TPU we have (any additional queries would require an entire new loop over the pretraining set, which would take another few months to run). Therefore, we used 40 factual and 40 reasoning questions (and the remaining 20 queries we used for control questions, see [other general common comment](https://openreview.net/forum?id=1hQKHHUsMx&noteId=g1DwLjGUor) for details on these new results). We effectively look at 200 factual, reasoning, and control queries (100 for the 7B, and 100 for the 35B, of which 36 share  prompt, but all different completions).  

**Pretraining data**  
This aspect is the bottleneck, and took >80% of the TPU chip hours. The important point about this subset of tokens is that it is identically distributed as the pretraining data. Our findings take into account that it is a sample, and we reason about how the conclusions might change if we would be able to look at the entirety of the data in the submission. Unfortunately, that is not tractable (no research has looked at the entire pretraining data in this way), so we have to draw conclusions based on the fact that we have an equally distributed sample. The highly qualitatively relevant data we find for all queries provides strong support that this sample is large enough to cover highly influential data. E.g., we find answers in the documents to 'niche' questions such as *"Who was the prime-minister of the Netherlands in 1970?"*.

**Models**  
Our results can be seen as evidence that a common style decoder-only transformer can in principle learn a generalisable strategy from pretraining data, for a 7B and 35B model. Comparing to another model family is an interesting direction for future work, but it is not essential for our conclusion. However, it is prohibitive in terms of compute costs. Furthermore, it’s not immediately clear what other model we could look at, as our investigations require full access to the pretraining distribution. Llama, for example, is trained on proprietary data.

To summarise, we would like to reframe the scope of our experiments as a necessary limitation given the high cost of the experiments, and not a weakness. We are the first to look at the pretraining data in this way to understand how LLMs generalise when reasoning, and show that it is possible for LLMs to learn a generalisable strategy from procedural knowledge in pretraining. We agree with the reviewers that our results leave open the question of whether this holds for other models and forms of reasoning, like inductive reasoning. We added the a few lines in the revision to highlight this further (L525-528). We are excited about future work in this area. When influence functions become further tractable, findings can be confirmed on the entire pretraining set (which is an active area, e.g. [1], but these style of functions are currently less effective in estimating the counterfactual).

[1] *"What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions"*, Sang Keun Choe et al., 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1731943394438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers: details on revision

comment:
Dear reviewers,

We believe we have significantly improved our submission in response to your reviews, detailed in a separate comment to each reviewer below, and we want to thank you all for your thoughtful reviews. In this brief comment, we wanted to highlight two changes to the manuscript in response.

The first change is in response to reviewer **cjVF**’s first weakness saying we should present a more cohesive narrative. To this end, we change the title to *“Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models”*. With this title we aim to introduce the main finding early. To address the same weakness, we change Figure 1, which now represents a summary of our findings instead of an image of the pipeline, which we moved to the appendix (Figure 6). We also rewrote the discussion to spend less time on summarising results and more on discussion. 

The second major change is that we added experimental results based on a group of 20 control queries for each model (which we were able to fit together with the 80 queries for each model in the same loop over the pretraining data). Because it was not feasible to add an interesting amount of additional reasoning tasks (see [the other comment to all reviewers right below](https://openreview.net/forum?id=1hQKHHUsMx&noteId=ZZ8uSWvlC0), about scope), we believed a better use of these 20 extra queries was to test alternative hypotheses about the data. These queries are control queries in that they are similar to the factual and reasoning queries in style and wording, but do not require any factual retrieval or reasoning to be resolved. We believe these additional results help address raised points by reviewers about the experimental scope by confirming that similar quantitative results do not hold for a control group. For the change to the main paper, please refer to the revision at the end of Finding 1 in the quantitative findings section 5.1 (L314-319, orange colour-coded). Table 10-14 in the Appendix has examples of what the control queries look like, and they can also be found in the supplement.

More generally, we have colour-coded all revisions with colours specific to reviewers:

Orange: relevant to multiple reviewers.  
Blue: relevant to **reviewer cjVF**.  
Green: relevant to **reviewer RvSn**.  
Red: relevant to **reviewer 6knH**.  
Purple: relevant to **reviewer KXBG**.  

We hope our revisions detailed below in response to your reviews address all your points and we are happy to discuss further wherever required. Thank you again for your time!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732268193670

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-up: request for engagement during discussion period

comment:
Dear Reviewers,

With only a few days remaining in the discussion period, we would greatly appreciate your engagement to ensure a constructive dialogue. In our revision, we’ve worked hard to address your feedback, making significant improvements to the paper:

- The findings present a more coherent message, guided by reviewer cjVF's comments.
- We included additional experimental results and responses to shared reviewer points about the scope of the work.
- Detailed responses to reviewer-specific points in the each separate comment below.

We are eager to hear your thoughts on these updates and hope you’ll have a chance to review our responses. We value your time and effort in shaping this submission.

Thank you again for your thoughtful reviews and for considering our responses.

Best regards,

The Authors of Submission 7193
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates how LLMs utilize pre-training data differently when performing reasoning tasks versus factual question-answering. Using influence functions, the authors analyzed two models (7B and 35B parameters) to examine which pre-training documents had the greatest impact on model outputs. The experiments reveal several key insights: reasoning tasks draw from a broader, more distributed set of pre-training documents compared to factual queries; larger models show more uniform influence scores across documents, suggesting improved data efficiency; and documents containing procedural content (especially code snippets and mathematical explanations) are particularly influential for reasoning tasks. These findings advance our understanding of how LLMs may develop reasoning abilities through training, suggesting they acquire procedural knowledge rather than simply memorizing solutions.

The paper makes significant contributions to reasoning research for several reasons: (1) it addresses a fundamental question about how LLMs develop reasoning capabilities from pre-training data, representing frontier research in model interpretability by tracing responses back to training samples; (2) the technical execution features well-adapted EK-FAC influence functions and thoughtful comparative analysis between different model sizes and task types; and (3) the findings provide actionable insights about the importance of procedural knowledge and diverse training data in developing reasoning capabilities.

This paper also has several limitations: (1) the experimental scope is notably narrow, examining few mathematical questions and using just two models from the same family. While computational constraints explain some limitations, validation on public models like OLMo or StarCoder whose pre-training corpus is also tractable would strengthen the findings; (2) the study analyzes only a subset of pre-training data, potentially missing less frequent but influential documents; (3) additionally, while the paper discusses "reasoning" broadly, it specifically examines mathematical reasoning, and distinctions between different types of reasoning (logical, spatial) could be better explained.

Overall, the paper's approach and insights into how LLMs leverage training data for reasoning tasks make it a valuable contribution to the field. I believe it should be accepted.