Title: Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions
Venue: NeurIPS 2025
Accepted: yes
Keywords: influence functions

llm reasoning
Abstract: Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by stronger models. 
However, existing strategies for curating such training data predominantly rely on heuristics, limiting generalizability and failing to capture subtleties underlying in data. 
To address these limitations, we leverage influence functions to systematically attribute LLMs' reasoning ability on math and coding to individual training examples, sequences, and tokens, enabling deeper insights into effective data characteristics.
Our Influence-based Reasoning Attribution (Infra) uncovers nontrivial cross-domain effects across math and coding tasks: high-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning.
Based on these findings, we introduce a simple yet effective dataset reweighting strategy by flipping task difficulty, which doubles AIME24 accuracy from 10\% to 20\% and boosts LiveCodeBench accuracy from 33.8\% to 35.3\% for Qwen2.5-7B-Instruct.
Moreover, our fine-grained attribution reveals that the sequence-level exploratory behaviors enhance reasoning performance in both math and code, and the token-level influence patterns are distinct for math and code reasoning: the former prefers natural language logic connectors and the latter emphasizes structural syntax.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses influence functions (EK-FAC) to quantify the influence between math and code training examples on math and code tasks. They find substantial cross-domain influence, where high-difficulty math examples improve both math and code reasoning, while low-difficulty code examples are beneficial for code reasoning. At the token level, logical connectors tend to be the most influential for math reasoning, while syntactic tokens tend to be most influential for code reasoning. They use these insights to tune Qwen models on more high-difficulty math examples (with low-difficulty code examples) to improve math and code reasoning performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Quantifying the relationship between math and code examples using influence functions has direct downstream implications for LLM fine-tuning, as performed in the paper.\n\nWeaknesses:\n1. The new method (\"Infra\") might have somewhat limited novelty. It seems very close to an implementation of previous influence approaches (specifically, EK-FAC; https://arxiv.org/abs/2308.03296, as cited in the paper). The main differences are slightly changing the response function f (although f is still essentially log-probabilities), and applying to new input/output examples.\n2. The tuning results on the proposed \"Difficulty Flipped\" dataset are not entirely convincing, because that data has substantially more \"difficult\" problems overall than all the other comparison datasets (Figure 5, right). It is somewhat expected that including more difficult problems in the last tuning phase would benefit performance.\n3. Other works have applied influence functions to LLMs for reasoning and/or mathematical tasks (e.g. https://arxiv.org/abs/2411.12580, https://arxiv.org/abs/2410.17413). These works have demonstrated that examples with similar semantic patterns (e.g. similar reasoning) are often influential, so cross-domain influence between math and code problems is not entirely surprising. More fine-grained breakdowns (e.g. annotating examples that exhibit similar reasoning strategies) could make the paper more informative, by isolating exactly why specific math and code examples influence one another.\n\nQUESTIONS:\n1. Is there a way to distinguish whether the observed influence scores are primarily driven by surface level similarities between examples (e.g. difficult math problems may tend to include similar words or phrases to one another and to coding examples), vs. more abstract \"procedural\" similarities (e.g. https://arxiv.org/abs/2411.12580)?\n2. Could the tuning results on the proposed \"Difficulty Flipped\" dataset be because that data has substantially more \"difficult\" problems overall than all the other comparison datasets (see Weaknesses)?\n3. Is there evidence that the high-difficulty math examples are highly influential because they contain important reasoning strategies that the model relies on for other math and code reasoning problems? Are there ways to characterize or classify these different reasoning strategies? More discussion and results on this point could make the paper more impactful (and improve my scores), because it would help clarify exactly why these types of examples are so important.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into what types of data are beneficial for enhancing mathematical and code reasoning capabilities during the SFT (Supervised Fine-Tuning) stage. Specifically, the authors propose three levels of influence scores and utilize influence functions to investigate the relationship between training data and downstream performance. They arrive at four meaningful conclusions. Overall, the work is logically coherent, and the conclusions are persuasive and potentially of practical value.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This work explores and explains the underlying mechanisms behind the success of language models and presents key findings with substantial practical value and guiding potential.\n\n2. The relatively comprehensive experiments provide strong support for the reliability of the conclusions.\n\n3. The paper follows a clear line of thought, has a well-structured logic, and is written in a way that is easy to understand.\n\nWeakness:\n\n1. The conclusions of this work are confined to the SFT stage, whereas it is widely acknowledged that the capabilities of large language models primarily stem from the pre-training phase. Merely exploring the sources of capability during the SFT stage may be far from sufficient. That said, I understand this limitation, as investigating the origins of model capabilities during pre-training is extremely complex and resource-intensive.\n\n2. The design of the influence scores appears to be overly intuitive, lacking the necessary theoretical or empirical justification to support the reasonableness of such a formulation.\n\n3. This paper examines the impact of data on model capabilities solely from the perspective of training data → performance, resulting in conclusions that remain relatively superficial. It lacks deeper investigation and understanding of the internal mechanisms of the model.\n\nQUESTIONS:\n1. Do the authors offer any preliminary exploration or conclusions regarding the internal mechanisms of large language models (LLMs)?\n\n2. The conclusion of the paper seems somewhat rushed. Are there any additional key experiments or findings that should have been included?\n\n3. How was the design of the influence scores determined? What was the rationale behind this choice?\n\n4. Does replacing the Hessian matrix with the EK-FAC method compromise the reliability of the conclusions? Have the authors compared the impact of other alternatives on the final results?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Infra framework, which uses influence functions to quantify the impact of training data on the mathematical and code reasoning capabilities of large language models (LLMs). The core findings include: high-difficulty mathematical data significantly improves code reasoning, and low-difficulty code data is most beneficial for code tasks; ’ searching for another approach after reaching correct answers’ in math reasoning traces benefits to both math and code reasoning; and Token-wise attribution analysis reveals distinct paradigms in math and code reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n+ Target an important problem – analyzing how data samples affect the performance of LLMs from different levels, e.g., sequence level and token-level.\n+ Multiple interesting findings\n+ A new data reweighting strategy that can help build better fine-tuning datasets\n\nWeakness:\n\n- Lack of comparison with data selection methods\n- The proposed solution highly relies on EK-FAC, but lacks a limitation discussion of EK-FAC\n- The experimental results are not reported rigorously, and there is no statistical analysis.\n\nQUESTIONS:\nFirst, my major concern is that using the influence function to assign weights to data samples is similar to data selection. Thus, there lacks the comparison (experimentally or just discussion) between this solution with different data selection methods [1, 2].\n\n[1] LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 24\n[2] SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models. NeurIPS 24\n\nThen, the influence function mainly relies on EK-FAC, which is not novel. The limitation of EK-FAC needs to be discussed.\n\nEven though multiple interesting findings have been reported, they come from one-time experiments run. As a study paper, the authors need to run the experiments multiple times and then conduct a statistical analysis for a rigorous results report.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper uses influence functions to analyze which SFT examples cause reasoning abilities, focusing on math and coding.  It finds that difficult math examples improve both math and code reasoning, whereas easy code examples improves code reasoning.  This insight is used to re-construct the SFT dataset, to improve AIME and LiveCodeBench performance.  Furthermore, token-level attribution also reveals that "exploration" behavior stimulates general reasoning.

STRENGTHS AND WEAKNESSES:
Strengths
- The analysis is important in that it fills a gap that needed to be filled, and at the example-level, confirming previous heuristics for constructing reasoning SFT data.
- The paper does token-level analysis, which provides deeper qualitative insights to guide dataset construction, and at the sequence-level, finding counter-intuitive insight that "exploration" behavior can improve reasoning.

Weaknesses
- Method-wise, my understanding is that there was not significant novel innovation - the Anthropic influence functions paper already proposed token-level influence.

QUESTIONS:
- what was the reason that token-level attribution used gradients, whereas sequence-level attribution used ablation, and not just the sum of token-level attributions from gradients?
- was normalizing the influence scores so that magnitude of the gradient is ignored, i.e. RelatIF, considered when doing the analysis?  I wonder if sequence length would correlate with influence, if there is not normalization.

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03