Title: Chain of Execution Supervision Promotes General Reasoning in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

reasoning
Abstract: Building robust and general reasoning ability is a central goal in the development of large language models (LLMs). Recent efforts increasingly turn to code as a rich training source, given its inherent logical structure and diverse reasoning paradigms—such as divide-and-conquer, topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syntactic or implementation noise, making direct training on raw code suboptimal. To address this, we introduce TraceMind, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought style rationales, which we call Chain of Execution (CoE). 
The corpus spans domains including mathematics, classical algorithms and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity.
 We evaluate Tracepile using three training setups—continue-pretraining, instruction tuning after pretraining, and two-stage finetuning. Experiments across four base models (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, Tracepile boosts LLaMA3-8B by 9.2\% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and Zebra Logic under two-stage finetuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TracePile, a corpus of 2.6 million samples containing 25 billion tokens that transforms code execution into natural language COE explanations. The authors collect code from algorithmic competitions (Codeforces), classical algorithms (CLRS), and mathematical problems (OpenMath), then use Qwen-2.5-72B-Instruct to generate step-by-step execution traces. They evaluate this approach across three training paradigms (continue-pretraining, instruction tuning after pretraining, and two-stage finetuning) on four base models (LLaMA 3/3.1, Qwen-2.5, Qwen-2.5 Coder), reporting improvements on 20 benchmarks spanning mathematics, code, logic, and algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a compelling hypothesis that explicit execution traces can teach transferable reasoning skills. The scale of the effort (2.6M samples, 25B tokens) is substantial, and the diversity of sources (competitions, textbook algorithms, mathematical problems) provides good coverage of different reasoning patterns. The evaluation offers useful insights about integration strategies.\n\nHowever, several critical weaknesses undermine the contribution. Most significantly, the baseline comparisons appear unfair. The authors compare against CodeI/O [25] using different training configurations, e.g. CodeI/O uses only 1 epoch while TracePile uses 3 epochs in first stage SFT, this invalidates the claim and conclusion drawn in the paper e.g. RQ1/Table 4.\n\nThe paper also keeps comparing models fine-tuned on TracePile against base models which tells us nothing about the relative value of CoE versus other supervision approaches. A fair comparison would be CoE against models trained on equivalent amounts of alternative data (e.g., more code examples, mathematical solutions, or even generic instruction data) using identical hyperparameters.\n\nThe related work section is missing in the main paper which is not acceptable. In addition, there is no discussion on how this work is situated in the rich past literature on using execution traces to train models, e.g., TRACED, SemCoder, and CodeT. It'd be great if the authors could clearly state the novelty beyond existing work.\n\nThe intrinstic evaluation is also missing from the paper, i.e., whether the model learns to generate *correct* traces. It'd be a good add-on to complement existing evaluation.\n\nQUESTIONS:\nPlease see my comments above. \n\nIn addition, while authors mentioned that \"we impose a strict 8k-token limit\", from Table 2 it seems the avg #tokens is greater than 8k?\n\nLIMITATIONS:\nSee \"Strengths And Weaknesses\"", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
The paper presents a technique to augment data for continued pertaining of large language models to improve the coding and math abilities of the models. The technique constructs a dataset TracePile by collecting algorithmic data of 3 sources: competition code (mainly from Codeforces), classical algorithmic code, and mathematical code. These collected samples are further diversified by modifying the query or the code. The technique uses a language model (Qwen-2.5-72B in this case) to obtain the chain-of-execution data from under carefully designed prompting strategies.

The TracePile dataset is used to continually pretrain 4 base models: Llama 3 & 3.1, Qwen-2.5, and Qwen-2.5. The continue-pretrained model shows consistent improvement over the base model on a diverse set of coding and math problems.

STRENGTHS AND WEAKNESSES:
**Strenghts:**

1. The paper is well-written and easy to follow. The idea is simple and effective. 

2. Experiments are comprehensive and cover diverse coding and math benchmarks. 

3. The research questions discussed in Section 4 are insightful and cover most of the crucial ablations. The RQ1 shows that chain-of-execution data is more effective than I/O style supervision from previous work, CodeI/O (Li et. al. 2025). 

**Weaknesses:**

1. The experiments are focused on coding and mathematical benchmarks, and authors should show the effect of proposed continual pretraining on other common benchmarks not related to coding or math.

2. The code or data is not shared. Justification for the checklist question 5 about code and data is incomplete. Some of the other answers to the checklist questions are unclear as well.  

**Minor:**
Typo: LLaMA -> Llama

QUESTIONS:
How do the experiments avoid test-data contamination?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TracePile, a data corpus of 2.6 million samples designed to enhance reasoning abilities in large language models through Chain of Execution (CoE) supervision. This paper proposes to transform code execution into explicit, step-by-step natural language explanations that trace through mathematical problems, classical algorithms, and algorithmic competition code. The dataset spans diverse reasoning paradigms with additional augmentation strategies for query and code diversification to enhance logical granularity. Through experiments across four base models (LLaMA 3/3.1 and Qwen-2.5) and 20 benchmarks covering mathematical, logical, algorithmic, and code reasoning tasks, the authors demonstrate that TracePile consistently improves performance over these benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n+ The paper provides a large corpus to learn the code execution reasoning, which successfully improves LLMs' general reasoning capabilities -- this successful knowledge transfer is and exciting finding.\n+ This paper implements \"intermediate results verification\" to ensure the quality of the annotated reasoning text, which guarantees the correctness of intermediate reasoning steps.\n+ Training with TracePile consistently boosts base models of varied sizes with better reasoning performance across multiple reasoning benchmarks.\n\n## Weaknesses\n\n- Limited Technical Contributions Over Existing Works\n\nWhile TracePile presents a large-scale dataset for code execution supervision, the paper's core contribution exhibits limited novelty compared to recent work in code execution reasoning, particularly SemCoder (Ding et al., 2024) and CodeI/O (Li et al., 2025). SemCoder already introduced the concept of training code language models with comprehensive semantics reasoning through \"monologue reasoning,\" where models learn to articulate code execution step-by-step in natural language, similar approach to TracePile's Chain of Execution (CoE). Similarly, CodeI/O also transforms code execution into natural language explanations for training purposes. While TracePile claims to focus on \"fine-grained execution traces,\" this distinction appears incremental rather than fundamentally novel, as existing works already demonstrated that natural language execution reasoning could effectively bridge static code and dynamic execution states.\n\n- Unavailability of the Core Dataset and Inadequate Reproducibility Provisions\n\nA fundamental weakness of this paper is the unavailability of TracePile, the core 2.6 million sample dataset that constitutes the primary contribution. Despite claiming that their work introduces a \"large-scale corpus\" as the main contribution, the authors fail to provide access to the actual dataset, severely limiting the paper's impact and reproducibility (Please correct me if I am wrong). \n\nFurthermore, the authors' response to the reproducibility checklist is misleading—they cite \"Section 3.1: Experimental Settings\" as evidence for providing \"open access to data and code with sufficient instructions to faithfully reproduce the main experimental results.\" However, Section 3.1 only describes training hyperparameters and evaluation protocols, not data availability or access instructions. The experimental settings alone are insufficient for reproducibility when the core dataset remains inaccessible. \n\nFor a paper where the dataset is the primary contribution and the foundation for all experimental results, the failure to provide data access represents a critical flaw that undermines the work's scientific value. The authors should commit to releasing TracePile during the rebuttal phase to enable proper peer review and future research validation.\n\n- (minor) In the abstract, the data corpus was named as TraceMind at first. I assume this should be a typo\n\nQUESTIONS:\n- Could the authors add a discussion section in the paper to explain how their approach is similar to and different from SemCoder and Code I/O? It is ok to suggest CoE is inspired by existing works, but a discussion to specify the similarity and difference will be helpful to understand the unique contribution of this work\n- Could the authors submit the data corpus for reviewers to assess its quality during the rebuttal phase?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work presents TracePile, a large dataset of \"chain of execution\" (CoE) rationales, which are natural language chain of thought explanations of code execution.\n- Their code comes from 3 sources: (1) they use algorithmic competition code (primarily Codeforces) and prompt an LLM to extract the functions and reference input/output examples, verifying that the code executes to the actual input and output. (2) For a set of 30 classic algorithms, they use generators from prior work (CLRS-text and GraphInstruct) to generate intermediate states which they can use to verify intermediate results of chain of execution. (3) They also add Math code from OpenMath which is already paired with solver code, and filter out any problems that are solved by LLaMA3-8B in all three of three tries.\n- For the algorithmic and graph problems, they diversify the dataset by asking not only about output prediction but also to predict intermediate values – an LLM generates these variants. And furthermore they diversify the code by having an LLM rewrite the code to use different approaches to the same problem or to be refactored differently.\n- Finally they few-shot prompt Qwen-2.5-72B-Instruct to generate the actual CoE traces, either telling it to answer the given question by tracing code execution or telling it to traces the changes to a particular variable while explaining each step. They verify these intermediate states for the algorithmic problems using ground truth traces.\n- They explore continue-pretraining on TracePile, pretraining on TracePile then instruction tuning on TuluSFT, and fine tuning first on TracePile then on TuluSFT; the last of these options provides the largest benefit. Training boosts performance considerably on coding and math tasks, but also other domains outside the dataset like logical reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nI've identified a few minor issues and points of confusion, but overall I believe this paper is a good fit for NeurIPS and support its acceptance. The evaluation is very thorough and results are quite strong, with sufficient ablations and additional comparisons to support that the major components of the dataset construction process actually matter for performance.\n\nStrengths\n- There are a lot of parts to the dataset construction, but it is explained well – I've listed any of my confusions in Weaknesses and Questions.\n- The results in Tables 1 and 3 for continued pretraining and two-stage fine tuning look excellent. Training on TracePile seems to quite generally boost a range of models on a wide range of benchmarks, with few and minor regressions. The further appendix results (Table 7) look in line with this as well. The improvements on benchmarks like CLRS and GraphInstruct that were the basis for parts of the training dataset is not at all surprising, but they also still see quite considerable boosts to the wide array of other benchmarks. The number of datasets used in this evaluation is quite impressive. I appreciate that Table 3 doesn't use the base model but rather the TuluSFT trained model, which does a lot better than the base model on some benchmarks (especially near the bottom of the Table), as this is helpful for disentangling the effect of the second stage of fine tuning.\n- The Discussion section with four additional research questions is very well done. In particular I appreciated the ablations of RQ2, which are important to showing that all of the complexity involved (i.e. the numerous sources of data and the diversity features) actually contribute to performance. And similarly RQ3 was important to showing that it's actually worth it to train on such a large dataset.\n\nWeaknesses\n- Figure 3 is a little confusing for a few (fixable) reasons.\n    - One is that the LLama-3.1-8B line (blue) doesn't make a lot of sense as it's own line, since it doesn't scale with the data and is actually just a single point – and in fact it's just the *zero* data point which could go to the left of 50K. Based on how it is 60.0 in the Math domain, it must be the TuluSFT tuned model (the \"Base\" model from Table 3) so I believe this would make sense as a Data Scale = 0 point.\n    - Secondly, it's confusing that the x axis is nonlinear and has quite variable gaps. Plotting it myself, I think this could look fine with a linear scale from 0 to 4.3M on the x axis. It ends up looking like a rather sharp first jump (in the 0K to 50K segment) then smoothing out.\n    - It could make sense to make the very last segment a different color or something, to indicate that this is coming from a different process (rejection sampled data augmentation)\n- Details are missing in a handful of places:\n    - In RQ4 of the discussion the rejection sampling based augmentation to make TracePile++ is mentioned, but details aren't given – including these in an appendix would be helpful.\n    - Lines 159-161 state: \"We validate correctness by comparing these outputs to ground-truth traces generated via code instrumentation or symbolic tools, discarding samples that fail consistency checks\". This feels key to data quality, and I'd appreciate more details on it (e.g. in an Appendix, but also in the response to this review). Am I correct that this is separate from the intermediate results checks from the Classical Algorithm Code section? Is this applied to many of the problems from the competition code? How does the code instrumentation work, and what symbolic tools are used?\n\nQuick fixes and suggestions\n- \"CoE formats significantly\" (line 139) I think there's a missing word in here?\n- Table 2 has a typo in comma placement in the entry \"117,083,6\"\n- The abstract says \"TraceMind\" instead of \"TracePile\" in one place.\n- line 230 should just say Tables 1 and 3, not Tables 2 and 3\n- I found Figure 1 (the splash figure) a bit difficult to read without context – in particular, I think a longer caption could help clear it up. In particular it would be helpful to explain that the 3 boxes on the left are actually 3 different data sources for the dataset. Perhaps putting the size column of each of these from Table 2 into this figure would also be a nice touch.\n\nQUESTIONS:\n- See questions in \"Weaknesses\"\n- Why was the particular set of benchmarks chosen for table 4?\n- Lines 306-308 state: \"Among them, code rewrites contribute most to structural generalization, while query diversification enhances task-specific interpretability.\" Unlike the other lines of this paragraph, this one didn't obviously follow from the results in the table for me. Where does this conclusion come from?\n- Lines 154-155 state: \"For tasks requiring variable tracking or control-flow interpretation, we require the model to output structured json results.\" What is meant by \"control flow interpretation\" here? If it's any tasks requiring control flow like loops or if-statements that must be almost every coding task?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}