Chain of Execution Supervision Promotes General Reasoning in Large Language Models
Authors: Nuo Chen
Zehua Li
Keqin Bao
Junyang Lin
Dayiheng Liu
Venue: NeurIPS 2025 poster ()
Keywords: large language models
reasoning
Primary area: applications

Abstract: Building robust and general reasoning ability is a central goal in the development of large language models (LLMs). Recent efforts increasingly turn to code as a rich training source, given its inherent logical structure and diverse reasoning paradigms—such as divide-and-conquer, topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syntactic or implementation noise, making direct training on raw code suboptimal. To address this, we introduce TraceMind, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought style rationales, which we call Chain of Execution (CoE). 
The corpus spans domains including mathematics, classical algorithms and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity.
 We evaluate Tracepile using three training setups—continue-pretraining, instruction tuning after pretraining, and two-stage finetuning. Experiments across four base models (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, Tracepile boosts LLaMA3-8B by 9.2\% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and Zebra Logic under two-stage finetuning.

Chain of Execution Supervision Promotes General Reasoning in Large Language Models
Nuo Chen
Zehua Li
Keqin Bao
Junyang Lin + Dayiheng Liu
Qwen Team, Alibaba
Hong Kong University of Science and Technology (Guangzhou)
University of Science and Technology of China
chennuo26@gmail.com
Abstract
Building robust and general reasoning ability is a central goal in the development code
of large language models (LLMs). Recent efforts increasingly a rich training source, given its inherent logical structure and diverse reasoning paradigms-such as divide-and-conquer topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syn- tactic implementation noise, making direct training on raw code suboptimal. To address this, we introduce TracePile, a large-scale corpus of 2.6 million sam- ples that transforms code execution into explicit, step-by-step chain-of of-thouuht style rationales, which we call Chain of Execution (CoE). The corpus spans do- mains including mathematics, classical algorithms and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity. We evaluate TracePile using three train- ing setups wowtinue-pretrainnee stage finetuning. Experiments across four base models (LLaMMA LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, TracePile boosts LLaMA3. by 7.1% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and MMLU under two-stage finetuning.
instruction tuning after pretraining, and two-
Introduction
Reasoning ability-a fundamental and foundational competence of large language models (LLMs)-is key indicator determining whether these models can effectively progress toward artificial general intelligence (AGI). While LLMs have shown remarkable success in language understanding and generation, is their capacity for complex reasoning-such
mathemati- cal reasoning and code generation that remains the most critical and challenging benchmark (24 40 [53 [6 49 47 42). Recent works have increasingly focused on boosting these reasoning abilities, with one of the most widely adopted approaches being the large-scale collection or synthesis of high-quality, task ees--speiffic reasoning data. Training on large volumes of carefully curated or syn- thetically generated reasoning data systematically strengthens the model's 's capacity for mathematical
reasoning, code generation, and multi-step moblem-solvine [39 2 52 despite these advances, current reasoning domains rather than achieving broad, generalizable reasoning capabilities. Developing corpus that incorporates diverse reasoning patterns and paradigms represents promising and important direction for advancing models toward more robust and transferable reasoning abilities.
50 48 31) However,
methods primarily enhance performance within specific
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Code corpora hold significant potential for advancing general reasoning abilities, as they inherently encapsulate diverse range of logical problem- sollinn techniques and reasoning patterns [45] [2] 27
[45]
Mathematical code often encodes rich mathematical properties and symbolic operations, while classical algorithmic code-for example, in graph theory-embeds essential algorithmic ideas such as divide-and-conquel (e.g., merge sort), topological reasoning (e.g., topological sort in directed acyclic graphs), and enumeration strategies (e.g., combinatorial generation or backtracking) [44] 34). However, simply training models directly on raw code is suboptimal because the relevant reasoning signals are often implicit, buried within syntactic details, or entangled with noisy implementation artifacts, making it difficult for models to extract the underlying reasoning patterns effectively. To address this, Codel/O (27] has sought to transform raw code files into structured code functions paired with corresponding questions, enabling the model to treat the code reference when generating -hain-of-thouugt (CoT) solutions. Yet, such datasets still overlook fine-grained information embedded in the execution process itself. We argue that converting each step of code execution into Co natural language narration would produce data that is not only logically rigorous and well-structured but also offers clearer, more interpretable reasoning traces. Training LLMs in such step-by-step code execution corpus has the potential to enhance their general reasoning capabilities significantly.
To this end, we construct large-scale step-by-step code execution corpus, which we name Tra- cePile- -a richly diverse dataset designed to capture fine-graing ann-ggrined diverse reasoning signals across a variety of domains. We refer to this CoT-style step-by-step code execution Chain of Execution (CoE). Specifically, we collect a broad range of queries and associated code snippets from multiple sources, including mathematical problems, algorithmic competition datasets, and classical algorithms. Frontier LLMS are prompted in a few-shot setting to generate detailed CoE solutions that follow the execution trace of the code, transforming each computational step into structured and interpretable reasoning narration. To further enrich the corpus and improve logical granularity, we introduce two key augmentation strategies. First, we generate additional fine-grained questions for certain algorithmic functions, such as asking the model to trace the state changes of specific variables throughout execution. These targeted questions lead more localized and logically dense CoE narra- tions, helping models internalize detailed procedural reasoning. Second, we apply systematic code rewritings introduce variants- instance, through structural refactoring or alternate algorithm implementations. This not only diversifies the syntactic surface of the training data but also promotes the model's 's robustness in understanding semantically equivalent yet stylistically different codes, encouraging broader generalization across reasoning tasks.
As result, we collect 2.6 million high-quality CoE samples in TracePile. To comprehensively evaluate the utility of this corpus, we design three experimental settings: 1) We perform continue- pretraining on TracePile, serving as an intermediate step enhance the reasoning abilities of the base model by exposing it to fine-grained, step-by-step execution patterns. 2) We apply general instruction tuning after continue- pretrnuiiii..tinngg...mm allowing the model retain broad mnstruction-folloiing abilities while leveraging the specialized reasoning improvements gained from TracePile. 3) We adopt two-stage finetuning approach, where the model is first tuned on TracePile and then on general instruction datasets, serving as structured adaptation path that integrates specialized reasoning gains while maintaining strong general alignment across diverse tasks.
We demonstrate the effectiveness of TracePile across four different base models: LLaMA 3 & 3.1 [15], Qwen-2.5 (38), and Qwen-2.5 Coder (21) We conduct comprehensive evaluations on as many as 20 datasets, covering four major reasoning domains: mathematical reasoning, coding, logical reasoning, and algorithmic problem solving. Across these benchmarks, models trained with TracePile consistently achieve superior performance. For example, after
TracePile boosts LLaMA3.1 1-B-BB-- by an average of 7.1% improvement across nine mathematical reasoning datasets. Moreover. following two-stage finetuning, models show clear gains on datasets such as LivcCodeBench (22). CRUX [T6], and Zebra Logic (29). demonstrating TracePile's ability to enhance both domain-specific and general reasoning capabilities.
2 TracePile
In this section, we introduce TracePile. We begin with collecting data sources and extracting raw queries and code corpus. Then we enrich the data diversity with query and code diversification. Subsequently, we review the process of obtaining the chain of execution data and filtering strategies. Finally, the statistics of TracePile are presented. Figure shows the overview of TracePile.
2

---

Erriching Diversily
Training Paradigm
Algorithmic Competition Code
ineraaiication
Divarattzation
Orw Gode
Cleck
Coppett
Matt the
Classical Algorithm Code
Trect
Baptect diger Hene
Ooedtinn Trove
Chain Evecution Generation
Mathematical Code
Prwvefer Trongeigg
Mark Proidae
Bofralyy
Soher Cods
Figure 1: The curation process of TracePile and training pipelines. The left part indicates the sources of Tracepile include algorithmic competition, classical algorithm and mathematical code. The middle part shows the strategies to enrich the data diversity and CoE-style data generation. The right part includes the three different training paradigm in this work.
2.1
Data Sources
To build TracePile, curate step-by-step execution data from three primary code sources, each designed to capture diverse and fine-grained reasoning patterns.
Algorithmic Competition Code. We collect code from algorithmic competition problems, pri- marily using open-source Codeforces submissions However, raw competition data often contains noisy, monolithic scripts with unclear function boundaries and hardcoded I/O. To address this, we leverage Qwen-2.5-72B-Instte
to accurately extract clean code functions along with corresponding input-output pairs (Prompts are in Appendix (A). We further validate correctness by executing the code and checking whether the generated output matches the extracted reference output- output-only retaining samples that pass this consistency check. This ensures both syntactic validity and semantic fidelity.
Classical Algorithm Code. We include classical algorithm implementations. inspired by prior works such as CLRS (34] and GraphInstruct [32]. We incorporate 30 classical algorithms drawn from Cormen [0] spanning broad spectrum of algorithmic paradigms, including sorting, searching, divide-and-com conqqur greedy strategies, dynamic programming, graph traversal, string manipulation, and geometric computations. Using the official CLRS-text generator, we synthesize high-quality inputs, outputs, and intermediate execution traces. For graph theory-related problems, we utilize official generator from (2). However, such traces from the above generator are typically presented as symbolic states or graphs rather than natural, interpretable CoT--tyyle narratives. They will be used to check CoE's intermediate results.
Mathematical Code.
We incorporate mathematical code, which often involves multi-step decom- position, symbolic manipulation, and numerical compositional thinking. We adopt OpenMath 43). large-scale synthetic dataset of math problems paired with solver code. To increase the difficulty and reasoning richness, we apply a model-based filtering strategy: for each problem, we sample three independent responses using L.L.M3--BB. If the model answers correctly all three attempts, we consider the problem too simple and discard it.
woaeiinng--aiigggene ideal for training models on
2.2 Enriching Diversity
While TracePile covers a broad range of code sources, the original code and query distributions still exhibit notable limitations. For instance, classical algorithm implementations often follow highly standardized coding styles, such as fixed loop structures, hardeoded base cases, commonly reused recursion patterns. This lack of structural diversity can limit the variety of reasoning challenges presented to the model. To further improve the reasoning generalization potential of the dataset, introduce two augmentation strategies aimed at enriching both the query and code.
SaarrihooSSummmsiiosssnetseee 2row=0
3

---

Given the cods, troce the changes of spectfied variables duting code exccution. Explain the whole exccution procees langungs.
We'll follow the execution step, highlightinng when and how the pl updared. Initialiaation::
pi = [0, 1,2,3
color [0, 0] (all modes are initially unvisited)
A: [0 110111100011001001111100
Query
2. Storting the DES Loop (a 0) Node ia unvisited (color(0) == 0, vás_ called.
def dis(A: ap.cdarco)) apandarsoy:
dff
Exploring Neighhors of Node 0: Neighher == 0: (sopf) but coloc(0) (already visiting). change to Neighbor v 1:
Ashope[()
up zeroenn, dryperapint52)
ge sp.arange(a)
def dis risir(s):
CoE
A 1] and color[]] == (unvialted).
nonlocal time
Updote pi[1]
color(u] =1
pl Changes toc 2, dff, mlixt( is called.
time +=1
d[s] = time for in cange(s):
Hatteiing dis_visic()
U C
at Alu, x and color[y] mm 0:
10, Continuing the 13FS Looge
piv) =u
vis
= 1: color]1] == (Snished)) No oction taken. (fiiishzz)). No aztion u 31 color[3] m= (Inished). No action taken.
cnlor|]
Chain-of Execution
Summary of pl Chongee: Trooe pt (b, 1, > [0, 2, 10, 0, 10,00,0)
Code
Figure 2: A classical DFS algorithm example of CoE in TracePile. More cases are in Appendix
Query Diversification. To move beyond the standard function-level input/output prediction tasks, to generate alternative queries for given code snippet. Tra- ditional algorithmic queries typically ask for the function's 's return value. To increase reasoning granularity, we prompt the model to generate questions targeting internal execution states. For instance, in a DFS implementation, the model might be asked to predict the stack contents after a specific backtracking step or trace the change in a visited-node array. These questions compel the model to understand intermediate logic and control flow, promoting deeper procedural reasoning. We apply this strategy to algorithmic and graph problems but mathematical code, which often has a single well-defined objective and is less amenable to meaningful query variation.
we leverage 2.5-722.
Code Diversification. To increase structural variety, we also instruct Sween22.5770 Ins-rrtt to rewrite the original code implementations. For example, a recursive DFS might be rephrased into an iterative version using an explicit stack, or a dynamic programming solution might restructured to vary loop orders or base case initialization. Such rewrites preserve semantics while introducing syntactic and structural variation, helping the model generalize beyond rigid code templates and better handle real-worke coding diversity. See related prompts in Appendix A
While rejection sampling common method for increasing solution diversity, we choose not to apply it in the main TracePile construction for two reasons. First, our query and code augmentations already induce diverse CoE samples implicitly. Second, given the wide coverage of mathematical topics in our dataset, CoE formats significantly, making it difficult to apply rejection filtering in scalable and reliable manner. Nonetheless, we explore rejection sampling a potential enhancement in later experiments and discuss its effects in Section 4
2.3 Chain of Execution Generation
Once obtain the curated query-code pairs, generate detailed CoE data using Qwen-2.5- 72B-Instruct under carefully designed prompting strategies. Our goal is to convert these execution behaviors into coherent, step-by-step explanations in natural language that reflect fine-grained procedural reasoning.
The prompting format is tailored slightly based on the data source and task type, but generally follows patterns such as: "Answer the question by tracing the code execution step by step." " "Trace the changes of variable X during execution and explain each step."
To ensure the generated CoE outputs are logically consistent and sufficiently detailed, we employ two key strategies: (1) Few-shot Prompting. We provide two human-written in-context examples within each prompt, showcasing complete step-by-step CoE narrations for similar tasks. These exemplars guide the model in understanding both the structure and the level of granularity expected in the reasoning process. (2) Intermediate result verification. For tasks requiring variable tracking or control- flow interpretation, we require the model to output structured json results, For mathematical problems,

---

Table 1: Model performances under continue-pretraine L.C.Bench-O refers to the test-output subset of LiveCodeBench. The best results are purple.
Llama-3-8B
Llama-3.1-8B
Qwen-2.5-7B
Qwen-2.5-Codee
Type
Datasets
Base
Ours
Base
Ours
Base
Ours
Base
Ours
GSM8K
54.2
74.3
54.4
74.7
85.7
84.9
77.9
83.5
MATH
16.5
31.7
17.7
29.3
50.9
44.8
47.2
52.3
GSM-H
26.1
36.2
27.1
36.9
63.3
63.1
55.1
59.8
SVAMP
68.8
81.2
71.0
81.1
89.4
89.6
87.8
89.0
ASDIV
73.1
82.7
74.3
83.8
91.0
91.3
89.0
89.2
Math
MAWPS
90.9
92.2
92.0
93.7
97.0
97.1
93.5
94.9
STEM
49.7
56.3
57.0
57.1
68.0
72.6
67.2
68.3
TABMWP
57.9
55.0
63.6
57.4
73.0
72.3
56.9
66.2
SAT
56.2
59.4
59.4
68.8
80.0
93.8
81.2
84.4
Average
54.8
63.4
57.4
64.8
77.6
78.8
72.9
76.4
Code
L.C.Bench-O
2.0
30.5
1.6
11.8
40.3
49.8
14.0
44.8
Zebra Puzzle
0.1
7.4
2.3
7.4
2.9
2.0
3.0
4.0
Logical
KORBench
21.8
21.8
21.9
20.6
33,4
35,4
32.0
30.0
Ruletaker
2.8
42.4
5.3
45.6
61.2
63.5
46.1
60.2
Graphwiz
4.5
46.0
1.9
33.9
36.6
50.5
38.5
43.8
Algorithm
GraphInstruct
35.2
73.5
33.0
69.2
33.1
33.5
33.8
53.7
the final computed result is reported in json simplify correctness checking. For algorithmic tasks, the json includes both the final output and intermediate variable states (e.g., array contents, stack traces) across execution steps- formatedd "slate; -> stale, - ...state,, "-to facilitate automatic comparison with execution logs. We validate correctness by comparing these outputs to ground traces generated via code instrumentation symbolic tools, discarding samples that fail consistency checks. We showcase an example of CoE in Figure 2 In the final version of our dataset, we convert the json outputs back into natural language format. This adjustment was made after observing that retaining structured json caused the model to produce inconsistent or malformed outputs on some downstream datasets after continued pretraining.
To improve coverage on harder problems, we sample each CoE rationale five times independently. Some tasks too challenging for the model to solve correctly in single attempt, so multiple samples increase the chance of getting a valid solution.
Finally, control for CoE complexity. As input size increases- particclaaa in graph algo- rithms-the reasoning path and solution length often grow rapidly. For example, denser or larger graphs lead to execution traces that scale combinatorially. often resulting in CoE outputs exceeding tens of thousands tokens. Since such long outputs are prone to generation errors and misalignment, we impose strict 8k-token limit, consistent with the context length used during puntinu-ppretraing Any sample exceeding this limit is discarded to maintain training efficiency and consistency.
2.4 Statistics
We summarize the composition of Tra- cePile in Table 2 which includes three primary components: algorithmic compe- tition data, classical algorithm implementa- tions, and mathematical code. In total, Tra- cePile contains over 2.6 million CoE sam- ples comprising approximately 19 billion tokens. The algorithmic competition sub- set contributes 480K samples sourced from Codeforces, covering diverse real-world problem-solving code. The classical algorithm subset includes 949K structured samples based
Table 2: Statistics of TracePile.
Components
Size
Tokens
Algorithmic Competition 480,782
3,891,991,488
Classical Algorithm
949,088
7,024,210,656
Mathematical
1,170,836
7,105,488,512
Total
2,600,706
19,380,461,000
on 30 canonical algorithms. The mathematical subset consists of 1.17M samples drawn from
OpenMath, emphasizing multi-step symbolic computation. Notably, the classical algorithm and
5

---

Table 3: Model performances under two-stage fine-tuning. The base models are trained solely on TuluSFT datasets. L.C.Bench refers LiveCodeBench. The best results are purple.
Llama-3-8B
Llama-3.1-8
Qwen-2.5-7B
Qwen-2.5 2.5-Coder
Type
Datasets
Base
Ours
Base
Ours
Base
Ours
Base
Ours
GSM8K
70.7
74.0
73.8
76.1
76.9
77.0
74.3
80.8
MinMath
28.8
33.8
30.2
33.2
47.2
51.8
47.6
48.2
MATH
28.6
30.6
30.8
34.2
49.6
51.5
45.0
50.8
GSM-H
35.0
35.6
39.3
35.3
55.0
55.6
55.4
59.8
SVAMP
83.0
79.4
79.6
83.4
79.7
83.6
81.1
85.3
ASDIV
81.4
81.9
82.3
85.7
86.5
86.3
86.9
88.9
Math
MAWPS
93.9
93.5
94.3
93.1
95.6
95.9
95.4
96.6
STEM
42.8
52.9
50.2
58.9
70.1
71.9
69.3
66.2
TABMWP
65.8
67.1
69.6
57.2
77.3
78.9
80.5
78.3
MATHQA
37.7
54.7
44.7
60.2
80.7
80.2
75.3
77.2
SAT
40.6
75.0
65.6
68.8
90.6
96.9
87.5
84.4
Average
55.3
61.7
60.0
62.4
73.6
75.4
72.6
74.2
CRUX
32.9
42.1
33.5
49.9
46.5
49.4
52.8
56.4
Code
L.C.Bench
9.8
14.2
7.6
11.3
25.8
23.5
27.2
29.9
Zebra Logic
7.7
10.5
8.1
9.4
9.4
8.8
9.2
10.8
KORBench
27.0
27.2
27.0
24.7
40.4
41.0
34.6
39.4
Logical
Mmlu-Redux
51.8
56.3
53.8
54.3
64.2
69.7
66.5
64.2
Ruletaker
60.4
66.2
61.5
67.0
73.2
74.4
73.1
73.9
Graphwiz
44.0
39.6
44.5
39.2
29.0
33.2
33.9
35.4
Algorithm
GraphInstruct
27.5
36.0
29.8
55.9
32.4
35.8
31.5
40.9
CLRS
17.2
24.5
16.8
25.8
36.0
43.6
42.2
49.4
algorithmic competition portion contribute the largest share tokens, reflecting the high density of intermediate reasoning in those samples. This large and diverse corpus forms the foundation for studying general-purpose reasoning enhancement through code execution supervision.
3
Experiments
In this section, we first introduce the experimental settings, including training setup, and evaluation datasets. Then, we evaluate the effectiveness of TracePile 20 datasets under three training setups.
3.1 Experimental Settings
Training Setups. Our goal to investigate whether fine-grained, step-by-step execution data can serve as an effective intermediate training signal to enhance multi-domain reasoning performance. To this end, we design three complementary training settings: (1) Continue-pretraining on TracePile. which introduces structured reasoning patterns into the base model and strengthens its ability to perform logical, step-by-step inference. (2) Instruction tuning after pretraining, which ensures that the model retains broad instraction-followee
skills acquired during TracePile continue-pretrainee capabilities while benefiting from the reasoning checkpoints. (3) Two-stage instruction-tuning where the model is first trained on TracePile and then further tuned on general instruction datasets. This staged adaptation enables the integration of specialized reasoning without compromising general alignment, resulting in a more balanced and capable model across tasks.
To ensure the generality and robustness of our findings, select four representative base mod- els for evaluation: LLaMA 3-8B, LLaMA 3.B,8B, Qwen-2.5-7B, and Qwen-2.5 Coder-7B. These models span both general-purpose and code-oriented architectures. allowing us to assess the ef- fectiveness of TracePile across diverse model families and training paradigms. For all training both continue-pretraining and instruction tuning-we wee use 16 H800-80GB GPUs with batch size of 512, maximum sequence length of 8192 tokens, and training epochs. The learning rate set to le-5, and training is conducted using the I.LMMA-Fattory framework. For the general instruction tuning phase, adopt the widely used Tulu3-SFT (26] dataset to ensure
experiments--incluii

---

broad instruction coverage and compatibility with existing evaluation standards. This consistent and scalable setup enables controlled comparisons across models and training strategies.
Evalution Dataset. To comprehensively assess the reasoning capabilities of TracePile-enhancee
models and strong baselines, we evaluate across 20 benchmarks spanning four major reasoning do- mains: logical reasoning, algorithmic reasoning. code reasoning. For mathematical reasoning. we
select 11 diverse datasets: GSM8K 8 MATH [2)]. GSM8K-Hare 14. SVAMP [36,, ASDIV 35
MAWPS [25], MinMATH 200, AMLU-STEM, [9] TABMWP 30,, MATHQA M. and SAT
[55]
For logical reasoning, we include Zebra Logic [29] RuleTTkker 6 and KORBench [33] For code
reasoming, we evaluate on LiveCodeBench [2] and CRUX [6] For algorithmic reasoning. we adopt
GraphWiz [3]. GraphInstruct [32], and CLRS BAJ We aggregate these datasets through three public evaluation toolkits: OpenCompass 99. Qwen2.5-M Math [50], and ZeroEval (28, ensuring consistency and reproducibility across experiments. See Appendix for details of evaluation datasets.
It is worth noting that we observe poor cntrructton-oolloviin behavior in some base models that have not undergone instruction tuning. For datasets with complex or structured instructions- such as CLRS-w exclude evaluation under the continue-pry centinue-petrrinnng---e setting, as the models are unable to generate meaningful outputs without prior alignment.
3.2 Main Results
The experimental results in Tables and demonstrate the effectiveness of TracePile across both continue- perttriiiirrtriiinggg... and two-stage instruction-tumi tuning settings. Under continue- prettriii....i.ingg. models consistently gain significant improvements across mathematical, logical, algorithmic, and code rea- soning tasks. For example, LLaMA-3-SB improves by +8.4% on average over nine math benchmarks (from 54.8% to 63.4%). with particularly large gains on GSMSK. Similarly, Qwen-2.5-Codd shows strong improvements across domains, indicating that even code-specialized models benefit from fine-grained execution supervision. TracePile proves especially effective on algorithmic reasoning. with improvements exceeding +40% on tasks such GraphWiz and GraphInstruct. When combined with general instruction tuning in the two-stage finetuning setting, these improvements persist and extend further. For instance, LLaMA-3-8B achieves a +6.4% average gain (from 55.3% to 61.7%) on math datasets, and Qwen-2.5-78 improves from 73.6% to 75.4%. Due to space limitations, we report results of continue-pretranning + instruction tuning setting in the Appendix, Table which further corroborate the value of TracePile as a general-purpose reasoning enhancement stage.
Out-of-domain (OOD) Generalization. Notably, TracePile also improves performance on bench- marks with little direct data or no overlap, such as LiveCodeBench,, CRUX, and logical reasoning datasets like RuleTaker and Zebra Logic. The resulting models' performances continue to improve on these datasets. These tasks differ in format, domain, or reasoning style from TracePile's 's training sources, yet models still show consistent improvements. This suggests that the benefits of explicit, execution-based supervision generalize beyond the original domains, helping models develop trans- ferable reasoning skills. The ability to improve both in-domain and out-of-domain tasks highlights TracePile's 's effectiveness as a general-purpose reasoning enhancement corpus.
4 Discussion
To further understand the effectiveness and generalization behavior of TracePile, we conduct series of analysis experiments. These analyses aim to evaluate the contribution of key design choices (e.g., step-by-step supervision, data diversity), the scalability of TracePile, its robustness, and its transferability beyond code-related tasks. Below, we address several core questions that deepen our understanding of TracePile's 's impact.
RQ1: How important is the Chain of Execution (CoE) format compared to traditional I/O or solution-style supervision? To evaluate the contribution of our step-by-step CoE supervi- sion, we compare TracePile with several strong baselines based on Qwen-2.5-Coder under con- trolled two-stage finetuning setting. As shown in Table we include three relevant baselines: (1) Open where we replace our CoE-based math data with the original solution-style code from OpenMath; (2) OpenMathlnstruutt22 where we replace our CoE-based math data with the pure math CoT-style solutions: (3) Webinstruct, where we replace our CoE-based math data with the generic instruction data from WebInsrtuct; Codel/O [27]. which contains 3.5M input-output code
7

---

samples with final-answe supervision; and (4) Codel/O++, a lightly augmented variant with minimal structural enhancements.
Despite being smaller equal in size, Tra- cePile outperforms all baselines across both in-domain and out-of-domain rea- soning benchmarks. On GSM8K, Tra- cePile achieves 80.8%,
Table 4: Comparison of TracePile with alternative supervision strategies under two-stage fine-tuning on Qwen-2.5-Coder 7B. I.COO and Z.I. denote
LiveCoteBench-Outtputa and Zebra Logic.
Methods
Stage-1 Data
GSM8K
MATH
LC-O
Z.I.
Baseline
74.3
45.0
29.8
9.2
Pure Code Solution
OpenMathIns. 1
80.1
46.1
19.4
5.6
slightly above Open- Math (80.1%) and Codel/O++ (80.5%).
Pure Math Solution
OpenMathIns. 2
82.1
51.1
17.4
6.6
Generic Instruction
WebInstruct
81.3
49.0
18.8
9.1
Codel/O
Codel/O
79.3
39.8
17.6
8.6
On MATH, which demands more com- positional reasoning,
Ours
TracePile
80.8
50.8
35.5
10.8
the improvement is more substantial- 50.8% for TracePile versus 46.1% (OpenMath) and only 39.9% (Codel/O++). The advantage of CoE becomes tasks. For example, on AiveCodeBench-Outpuu TracePile reaches 35.5%, far surpassing Codel/O++ (20.6%) and OpenMath (3.4%). On Zebra Logic, TracePile scores 10.8%, outperforming all baselines. These results confirm that the Chain of Execution format-by explicitly guiding models through intermediate reasoning steps steps-offer offers more effective supervision than final-answer training. TracePile's s CoE data not only enhances in-domain learning (e.g.. math, algorithms) but also enables strong generalization to unseen task formats, making it powerful training resource for robust and transferable reasoning.
more pronounced out-of-domain
RQ2: What components contribute most to TracePile's effectiveness? To assess the contribution of different components within TracePile, we conduct ablation studies LLama-3.1-88 3-BB under the two-stage finetuning setting. Specifically, we remove individual data sources or design features and evaluate their impact downstream performance across four reasoning domains: mathematical, code, logical, and algorithmic. The results are summarized in Table
The ablation results clearly demonstrate that both the multi-source composition and the diversification strategies crucial
Table 5: Ablation studies under two-stage fine-tuning. Average performances of each category are reported.
within TracePile to its effectiveness as an in- termediate reasoning stage. Removing any individual data source-matin matireecmatemaaacal.. algorith- mic, or competition code-leads noticeable performance degradation, not only within the corresponding domain but also across others, highlighting strong cross-domain transfer. For example, excluding competition data significantly weakens code reasoning (- (-5.7%), emphasizing its role in grounding execution- based understanding. Diversification strategies further enhance generalization: eliminating both query and code augmentations reduces performance across all categories, particularly in code (-3.0%) and algorithmic (-3.1%) reasoning. Among them, code rewrites contribute most to structural generalization, while query diversification enhances task-specific interpretability. These findings reinforce that TracePile's 's effectiveness is not solely due to data scale, but arises from its carefully constructed CoE supervision, domain diversity, and reasoning aware augmentation.
Methods
Mathematical
Code
Logical
Algorithm
Ours
62.4
30.6
38.9
40.3
w/o mathematical w/o algorithm w/o competition
60.6
26.2
38.4
40.6
61.5
38.0
33.2
62.0
24.9
37.8
35.0
w/o diversification w/o query w/o code
61.8
27.6
38.0
37.2
62.0
30.0
38.7
37.9
62.2
28.2
38.3
39.5
RQ3: Does performance scale with more TracePile data? To examine how performance scales with data size, we progressively expand TracePile from 50K to 4.3M samples and evaluate model performance using LLaMA-3.1 3-1--B under the two-stage finetuning setup. As shown in Figure 3

---

Nathomat Bessoring
Code Reasoning
Logical
Aggrtthm Reasoning
Ous
Limo
Dar
Lamad 2-15
148
Lime 1 10
Durs
Lamin--- 3-83
381
23.4
04.5
24
2.20
Data Scale
Dsta Scale
Scale
Duta
Figure 3: Model performances with scaling the size of CoE samples.
observe clear upward trend across all four reasoning domains: mathematical, code, logical, and algorithmic. This confirms that increasing the volume CoE supervision directly contributes to stronger reasoning capabilities, especially in domains with structured multi-step logic.
To further explore the effect of high-quality data at scale, we apply rejection sampling to augment TracePile to 4.3M samples, denoted as TracePile++. The goal is to filter for higher-quality CoE traces by selecting the diverse outputs from multiple samples. Interestingly, while mathematical reasoning continues to benefit from this expansion, the improvements in code, logical, and algorithmic reasoning plateau or slightly regress. This suggests that rejection sampling, despite improving local coherence, may introduce distributional biases that limit broader generalization.
RQ4: What types of reasoning are improved by TracePile? Does enhance robustness, reduce typical reasoning errors, and generalize? To better understand the qualitative improvements introduced by TracePile, we conduct fine-grained analysis on several challenging reasoning subsets from the BBH benchmark 41. These tasks are carefully chosen test specific dimensions of reasoning where CoE-style supervision may offer unique advantages, We focus on the following four subsets: Tracking Shuffled Objects: Requires precise state tracking across sequential transformations (e.g., object swaps), skill closely aligned with the procedural traceability in TracePile. Multi-Ster Step Arithmetic: Involves multi-hop numerical reasoning across chained operations. Logical Deduction: Demands inference from spatial or relational constraints to determine object orderings-ake to graph- or rule-based reasoning patterns. Web of Lies: Evaluates truth values from complex Boolean logic expressed in natural language, testing abstract symbolic manipulation.
As shown in Table models trained on TracePile outperform the Qwen-2.5-bart
model across all tasks. The gains are particu- larly striking in Tracking Shuf- fled Objects (+19.2%) and Web of Lies (+5.6%), suggesting im- proved robustness in stateful and symbolic reasoning. TracePile also boosts performance in Multi-
Table 6: Compare ours with Qwen-2.5-base under two-stage fine- tuning in BBH subsets.
Tracking Shuffled Objects
Logical Deduction
Web of
Modelsm
Arithmetic
Lies
Base
68.4
72.0
78.4
87.2
Ours
87.6
82.0
83.6
92.8
Step Arithmetic (+11.6%) 1.6%) and Logical Deduction (+3.6%), where intermediate reasoning steps are critical. These results provide evidence that TracePile significantly enhances fine- fine-grained reasoning abilities. It helps models better track latent state, execute multi-step logic, and reduce common failure cases like missing intermediate computations making logically inconsistent predictions. More importantly, it shows that CoE-based supervision transfers beyond code tasks, improving general reasoning robustness across abstract formats.
Related Works
Reasoning in Large Language Models. Reasoning a core capability for large language models (LLMs) and critical step toward achieving general intelligence. Recent work has shown that LLMs possess some capacity for complex reasoning when prompted appropriately, leading to a surge of interest in improving their reasoning abilities, particularly in domains such mathematical problem solving and code generation [4]] techniques, such as chain-of-thouge prompting
21). Early efforts often relied on prompt engineering 49 self-consistency, and tool-augmented scratchpad

---

44], to elicit multi-step reasoning behavior at inference time. More recently, the field has shifted toward data-driven approaches, where large-scale, Nomain-speciife reasoning corpora are collected and used for pretraining or finetuning. Representative examples include Qwen-Math [51], Qwen- Coder [8]. DeepSeek-Matt [39], and DeepSeek- Coder [21], which leverage hundreds of billions to trillions of tokens to specialize models for mathematical or programming tasks. Compared to these works, TracePile introduces a different perspective: rather than relying solely on final-answer or solution-style supervision, it provides fine-grained, step-by-step execution traces in natural language through the CoE format. This enables more explicit and interpretable reasoning supervision, bridging the gap between task task-specifie pretraining and general purpose reasoning enhancement.
general-purpose
Supervision through Code Execution. The idea of learning from code execution traces predates the modern era of LLMs, with early work exploring how neural networks could simulate or reason over program behavior 2 of LLMs, but most existing efforts focus narrowly on output prediction from code execution [13]. Other approaches aim to leverage execution in auxiliary ways either final feedback for reward learning [1] or by incorporating intermediate traces to improve code generation [12]. SemCoder
(4) More recently, this line of research has been revived in the context
[12 employs "monologue reasoning" with high-level functional descriptions and verbal reasoning about local execution effects. However, it primarily uses synthetic data with limited real-world complexity. Additionally, new benchmarks such CRUXEval [16] have been introduced test models' ability to simulate or predict execution dynamics. Most recently. Codel/O (27) also proposes code-referenced input-output pairs to enhance the models' reasoning abilities. In contrast these code-centric and task-specific efforts, TracePile takes a broader view. It is the first to train LLMs on large-scale, diverse execution traces using natural language CoE supervision.
6 Conclusion
In this work, we present TracePile, a large-scale dataset of step-by-step Chain of Execution (CoE) traces designed to enhance the reasoning abilities of large language models, covering over 2.6 million samples. By supervising models with fine-grained execution processes across mathematics, classical algorithms, and competition code, TracePile provides structured reasoning signal that extends beyond final-answer supervision. Our experiments across multiple training strategies and 20 diverse benchmarks demonstrate that TracePile consistently improves both in-domain and out-of-domain reasoning, especially on tasks requiring multi-step logic and state tracking. Ablation studies further highlight the importance of multi-source design and diversification strategies, while scaling analysis confirms that performance improves with data volume, though with diminishing returns beyond certain point. Overall, TracePile offers an effective and transfcrable intermediate training stage that significantly boosts the general reasoning capabilities of LLMs.
Limitation
While TracePile demonstrates strong improvements in general reasoning, it also comes with certain limitations. First, the dataset is primarily grounded in code-based reasoning, which may limit its applicability tasks requiring commonsense or world knowledge that fall outside structured procedural logic. Second, although we incorporate multiple domains, the coverage is still skewed toward mathematical and algorithmic reasoning; expanding domains such as law, planning, or real-world causal inference remains future work. Third, generating high-quality CoE traces relies heavily on large instruction-tun models, which introduces computational costs and potential bias from the prompting model itself. Lastly, our filtering strategy (e.g., 8k token cutoff) may discard complex but informative reasoning traces, potentially limiting exposure to long-range dependencies. Addressing these challenges such as improving coverage, reducing reliance on frontier models, and capturing longer CoE traces will be critical for further scaling TracePile's effectiveness.
References
[1] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedzions Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint 19Xvv:1905.13111 2019.
10

---

[2] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023.
[3] Nuo Chen, Yuhan Li, Jianheng Tang, and Jia Li. Graphwiz: An instruction-following language model for graph problems. arXiv preprinl urXi::440.16602, 2024.
[4] Nuo Chen, Ning Wu, Jianhui Chang, Linjun Shou, and Jia Li. Controlmath: Controllable data generation promotes math generalist models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 12201-12217, 2024.
[5] Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In EMNLP (Findings), 2024.
[6] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002. 058677 2020.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. CoRR. abs/2110.1141 2021.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[9] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https:://github..com// cttp://ppeccccmpppppppppccaaaaaa 2023.
[10] Thomas H. Cormen, Charles E. Leiserson, Ronald L.. Rivest, and Clifford Stein. Introduction Algorithms, Third Edition. The MIT Press, 3rd edition, 2009.
[11] Yangruibo Ding, Marcus J. Min, Gail Kaiser, and Baishakhi Ray. Cycle: Learning to self-refine the code generation. Proc. ACM Program. Lang., 8(0OPSLA)) April 2024.
[12] Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. Semcoder: Training code language models with comprehensive semantics reasoning, 2024.
[13] Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail E. Kaiser, Wei Le, and Baishakhi Ray. Traced: Execution- aware pre-training for source code. 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 416 427, 2023.
[14] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aide language models. arXiv preprint arX1v:2211.10435, 2022.
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
[16) Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 16568-16621. PMLR, 21-27 Jul 2024.
[17] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprint 1rXiv:2306./1644 2023.
[18] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024.

---

[19] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
[20] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[21] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang. Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-code technical report, 2024.
[22] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint (.Xxv:2403.07999 2024.
[23] Prashant Jayannavar, Anjali Barayan-Chen, and Julia Hockenmaier. Learning to execute instructions in a Minecraft dialogue. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pages 2589-2600 2602, Online, July 2020. Association for Computational Linguistics.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint (rXiv:2304.026433, 2023.
[25] Rik Kedziorkii, Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. Mawps: A math word problem repository. Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152 1157, 2016
[26] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv: 241 1. 15124, 2024.
[27] Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction, 2025.
[28] Bill Yuchen Lin. ZeroEval: A Unified Framework for Evaluating Language Models, July 2024.
[29] Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogie: On the scaling limits of Ilms for logical reasoning, 2025.
[30] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint 2tti:::20091461110 2022.
[31] Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model- translated mathematical code. arXiv preprint arXiv:2410.08196 2024.
[32] Zihan Luo, Xiran Song, Hong Huang, Jianxun Lian, Chenhao Zhang, Jinqi Jiang, Xing Xie, and Hai Jin. Graphinstruct: Empowering large language models with graph understanding and reasoning capability. arXiv preprint arXiv:2403.044 04483, 2024.
[33] Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. Kor-bench: Benchmarking language models knowledge-orthognnal reasoning tasks, 2025.
[34] Larisa Markeeva, Sean McLeish, Borja Ibarz, Wilfried Bounsi, Olga Kozlova, Alex Vitvitskyi, Charles Blundell, Tom Goldstein, Avi Schwarzschild, and Petar Velišković. The clrs-text algorithmic reasoning language benchmark. arXiv preprint arXiv:2406 04229, 2024.
12

---

[35] Suen-yun Miao, Chao- Choo-Chhn Liang, and Keh- Kih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975 975-984, Online, July 2020. Association for Computational Linguistics.
[36] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems?. 2021.
[37] Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. Rewarding graph reasoning process makes Iims more generalized reasoners. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, page 2257-22666 New York, NY, USA, Association for Computing Machinery.
2025.
[38] Qwen, :, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[39] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03322 2024.
[40] Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang. Shuo Shang, Ji- Ji-Rong Wen, and Rui Yan. Determir: Augmenting llm-based logical reasoning from indeterminacy to determinacy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9828-9862 2024.
[41] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether thain-of-thoughee can solve them. arXiv preprint MxXiv:2200009611. 2022.
[42] Jianheng Tang, Qifan Zhang, Yuhan Li, Nuo Chen, and Jia Li. Grapharena: Evaluating and exploring large language models on graph computation. In The Thirteenth International Conference on Learning Representations.
[43] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-l: A 1,8 million math instruction tuning dataset. ArXiv, abs/2402.10176. 2024.
[44] Petar Velišković, Adrià Puigdomênech Badia, David Budden, Razvan Pascanu, Andrea Ban- ino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pages 22084 221022 PMLR, 2022.
[45] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in 1lms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 2023.
[46] Rongzheng Wang, Shuang Liang, Qizhi Chen, Jiasheng Zhang, and Ke Qin. Graphtool- instruction: Revolutionizing graph reasoning in Ilms through decomposed subtask instruction. arXiv preprini arXiv:2412.12255 2024.
[47] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, 2022.
[48] Zengzhi Wang, Xuefeng Li, Rui Xia, and Pengfei Liu. Mathpile: A billion-token-scal pretraining corpus for math. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
13

---

[49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. ohain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 45:24824-24883 2022.
[50] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et Qwen2. 5-math technical report: Toward mathematical expert model via ceff-improvement, arXiv preprinl arXiv:2409. /2/22, 2024.
[51] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5- 5-ath technical report: Toward mathematical expert model via self-improvement, 2024.
[52] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332 2024.
[53] Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. End-to-end spoken conversational question answering: Task, dataset and model. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 1219-1232, Seattle, United States, July 2022. Association for Computational Linguistics.
[54] Wojeiech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint 14Xx::1410.4615 2014.
[55] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan, Agieval: human-centric benchmark for evaluating foundation models. arXiv preprinl arXiv:2304.0m 063644 2023.
[56] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models, arXiv preprint arXiv:2205. 2022.
A Prompts
Prompts A.1: Prompt of LLM extracting clean code
You are an Al assistant responsible for providing assistance. You are provided with problem and examples of the input.
Your task is: (1) Write Python code to develop input generator function capable of generating imputs for this problem. The input generator function should include a parameter that determines the maximum number of inputs it generates. The output of this function should be list containing the generated inputs, with the length of each individual input not exceeding 10. Besides, any numbers contained in the input should not exceed 1000. Please provide the Python code without any additional explanation. Present your output in the following format: Input_generator <your code>
(2) Rewrite the code to eliminate manual input and encapsulate it within function named 'solution'. The function should accept a string input as its parameter. Provide the output the following format: Solution: <your code>.
Here is relative information:
Original Problem: (Original Problem)
Input: (Input examples)
Prompts A.2: An Example of LLM Rewrite Prompt
You are provided with Python code that solves a specific problem. Your task is to rewrite the code while adhering to the following guidelines: (1) Retain the original logic and functionality of the provided code.
14

---

(2) Introduce variations in the implementation, such as using different syntax, alternative methods, restructuring the code for improved readability efficiency. (3) Ensure the rewritten code is clean, functional, and adheres to best practices.
Output only the modified code. Do not include any explanations or comments.
Here is relative information:
Code: [Code]
B Benchmark Overview
To comprehensively evaluate the reasoning capabilities of our models, we select 20 benchmark datasets spanning four major reasoning domains: mathematical reasoning, logical reasoning. algo- rithmic and code reasoning Below, we briefly describe each dataset:
GSM8K A collection of 1K grade school math word problems requiring multi-step arithmetic reasoning. Designed to evaluate basic numerical reasoning skills.
MATH A large-scale dataset of 12.5K competition-style math problems covering algebra, geometry, number theory, and combinatorics. It tests advanced symbolic and procedural reasoning.
GSM8K-Hard harder version of GSM8K where numbers are replaced with uncommon or large values, increasing arithmetic and symbolic complexity.
SVAMP An elementary-level math benchmark that tests a model's sensitivity to problem structure and its ability to reason over similar surface forms with different logical requirements.
ASDIV A diverse dataset of 2.3K elementary math word problems drawn from textbooks, focusing on linguistic variation and real-world phrasing.
MAWPS A corpus of 3.3K math problems scraped from educational resources, covering various question types used in real-world math education.
MINERVA_MO __Mtth A curated set of 272 high-complexity math questions, emphasizing multi-step symbolic manipulation and abstract reasoning.
MMLU-STEM A subset of the MMLU benchmark covering science, technology, enginecring, and
mathematics subjects. Includes questions from high school and university-level curricula.
TABMWP A structured math word problem dataset (8.5K samples) formulated in tabular format, designed to test abstract reasoning and table comprehension.
MATHQA A dataset of 3K math word problems annotated with symbolic program representations, enabling reasoning over structured solution steps.
SAT-MATH A dataset mimicking standardized SAT math questions, covering algebra, geometry, and data analysis. Designed to test broad quantitative proficiency.
Zebra Puzzle A logic puzzle dataset derived from constraint satisfaction problems. Tasks require deducing object relationships and properties based on textual clues.
Ruletaker benchmark that presents logical rules and facts in natural language and asks models to infer new truths through deductive reasoning.
ProofWriter Includes small rulebases English facts and logical rules, where models must determine whether a hypothesis is provable, unprovable, or unknown.
15

---

Based on the "Introduction to Algorithms" textbook, CLRS provides algorithmic tasks where models are expected simulate or reason about classical algorithm behavior.
CLRS
GraphWiz Contains 3.6K problems across nine graph reasoning tasks (e.g., shortest path, reacha- bility), with complexity ranging from linear to NP-complete.
GraphInstruct A synthetic benchmark consisting of 21 graph-based reasoning tasks. We evaluate on subset aligned with general reasoning patterns.
KorBench KorBench 33 is designed assess model's intrinsic reasoning and planning abilities while minimizing the influence of pretrained world knowledge. It introduces five cate- gories Operaaion, Logic, Cipher, Puzzle, and Counterfactual eachh defined by 25 manually con- structed, novel rules. This setup enables a more precise evaluation model's ability to generalize unfamiliar, rule-based tasks.
CRUXEval CRUXEval evaluates a model's capability to reason over code by predicting either inputs or outputs of anonymized Python functions. This benchmark requires models to perform symbolic execution and maintain intermediate state across program flow.
BBH BBH (Beyond the Imitation Game Benchmark) comprises 23 challenging reasoning tasks originally from the BIG-Bench suite. These tasks curated to be particularly difficult for LLMs, covering a wide range of logical, mathematical, and procedural reasoning problems.
Together, these benchmarks provide comprehensive and diverse testbed for evaluating general reasoning across symbolic, procedural, logical, and open-domain tasks.
Table 7: Model performances under contiue pretraining+ fentttftnnnttnnggg settings. The base models are trained on the same dataset. L.C.Bench refers livecodebench.
Llama-3-8B
Llama-3.1-88
1-8B
Qwen-2.5-7B
Qwen-2.5-Coder
Type
Datasets
Base
Ours
Base
Ours
Base
Ours
Base
Ours
GSM8K
70.7
72.0
73.8
78.9
76.9
76.9
74.3
80.8
MinMath
28.8
29.8
30.2
30.0
47.2
52.8
47.6
50.4
MATH
28.6
29.1
30.8
31.9
49.6
51.6
45
50.2
GSM-H
35.0
34.0
39.3
38.2
55.0
54.
55.4
59.8
SVAMP
83.0
79.9
79.6
83.4
79.7
81.4
81.1
83.0
ASDIV
81.4
82.2
82.3
84.7
86.5
88.3
86.9
88.2
Math
MAWPS
93.9
94.1
94.3
94.2
95.6
95.7
95.4
95.8
STEM
42.8
52.8
50.2
58.8
70.1
71.4
69.3
67.3
TABMWP
65.8
68.0
69.6
72.0
77.3
76.5
80.5
79.5
MATHQA
37.7
49.6
44.7
59.8
80.7
80.6
75.3
78.0
SAT
40.6
62.5
65.6
65.6
90.6
87.5
87.5
90.6
Average
55.3
59.5
60.0
63.4
73.6
74.3
72.6
74.9
CRUX
32,9
34.4
34.0
40
46.5
49.8
52.6
58.1
Code
L.C.Bench
9.8
10.1
7.6
14.7
25.8
23.9
27.2
29.0
Zebra Puzzle
7.7
8.9
8.1
10.7
9.4
8.9
9.2
9.6
KORBench
47.6
75.6
27.0
24.7
40.4
41.5
34.6
35.9
Logical
Ruletaker
60.4
65.4
61.5
65.1
73.2
74.7
73.1
72.4
Graphwiz
44.0
29.8
44.5
43.1
29.0
70.9
33.9
61.9
Algorithm
GraphInstruct
27.5
44.0
29.8
55.0
32.4
63.6
31.5
62.2
CLRS
17.2
32.5
16.8
38.3
36.0
$2.0
41.2
58.9
16

---

C Cases in TracePile
Prompts C.I: An Case from Classical algorithm in TracePile
Given the code, trace the changes of specified variables during code execution.
Input:
def dfs (A: np. ndarray) -> ndarray: assert ndndim ==
A. shapp [0]
color np. zeros (n, dtype intt322)nn))3
pi = np (n)
zp.zeros
zeroz (n)
time =
def dfs. sisit (u):
nonlocal time color [u] time +=
d[u] = time
for v range(n) (n)
if A A v] !- and color 0: pi[v] -
dfs_visit(v)
color [u] time +=
f[u] = time
for u range(n) (n):
if color [u] 0: (fe_
return pi
along with the input parameter: A: [[I 1J. (0 I], 0 1], 1 Trace the changes in the "pi" variable throughout the program's 's execution. CoE:
We'll follow the execution step-by-step. highlighting when and how the pi array is updated.
1. Initialization:
pi [0, 1, 2, 3] color [0, 0, 0, 0] (all nodes are initially unvisited) time =
2. Starting the DFS Loop (u 0):
Node 0 is unvisited (color [0] -- 0), dfs_visit is called.
3. Entering dfs
color [0] is aet to 1 (visiting).
time increments to 1.
d[O] = 1.
Exploring Neighbors of Node Neighbor v 0:
A1O, 0] (lop)) but color [0] == 1 (already visiting ) No change to pi. Neighbor 1:
17

---

A[O, 1] and color [1] -- o (unvisited). Update: pi[1] is set to 0. pi Changes [o, 0, 3] dfa_visit(1) called.
4. Entering df)_visit(1;:
color [1] set (visiting). time increments to 2. d[1] 2.
Exploring Neighbors of Node 1:
Neighbor v 0:
A[1, 0] 0 (no edge). No change to pi.
Neighbor v - 1:
A[1, 1] -1 (self-loop), but color [1] -- (already visiting pi.
)
change
Neighbor v 2:
A[1, 2] =0 (no edge). No change to pi.
Neighbor v = 3:
A[1, 3]
and color [3] o (unvisited). Update: [i] is set to 1. pi Changes to: [0, 0, 2, 1]
(f)_visit(B) is called.
5. Entering (3)
color [3] is set to 1 (visiting). time increments to 3. [[3] = 3.
Exploring Neighbors of Node 3:
Neighbor v 0:
AI3, 0] 1 and color [0] == 1 (already visiting). No change
to pi.
Nei Neighbor A A[3,
1:
and color [1] (already visiting). No change
to pi.
Nei Neighbor v = 2:
A[3, 2] 0 (no edge). No change to pi.
Neighbor v
A[3, 3] 0 (no edge). No change to pi.
Post Exploration:
color [3] is set to (finished).
time increments to 4.
[(3] 4.
pi Remains: [0, 0, 2, 1]
6. Returning to dfs_visit(n) :
Post Exploration:
color [1] is set to (finished).
time increments to 5.
f[1] = 5.
pi Remains: [0, 0, 2,
7. Returning to dfs_visit(O)) (0):
Neighbor 2:
ATO, 1 and color [2] 0 (unvisited). Update: [2] is set to 0. pi Changes to: [0, 0, 0, 1]
dfs_visit (2) called.
18

---

8. Entering afs_visit
color [2] is time increments to 6. [2] 6.
(visiting).
Exploring Neighbors of Node 2:
Neighbor v 0:
A[2, 0] 1 and color [0] == 2 (finished). No change to pi.
Neighbor v == =:
A[,, 1l = (no edge). No change to pi.
Neighbor v ==
A[2, 2]
(self-loop) but color [2] (already visiting ) No change to pi.
Neighbor
A[2, 3:
and color [3] 2 (finished). No change pi. Post Exploration:
color [2] is aet to 2 (finished).
time increments to 7.
f [[]] = 7.
pi Remains: [o, o, o,
9. Returning to dfs_visit(O))
Neighbor v 3:
AIO, 1 and color [3] 2 (finished). No change to pi. Post Exploration:
color [0] is set to (finished).
time increments to 8.
f(o] 8.
Final pi: [0, 0, o, 1]
10. Continuing the DFS Loop:
(finished). action taken. 2: color [2] = 2 (finished). No action taken.
color [1]
3: color [3] == 2 (finished). No action taken.
Summary pi Changes:
Trace pi: [0, 1, 2, 3] -> [0, 0, 2, 31 -> [0, 0, 2, 1] -> [0, 0,
0, 1]
So the output is:
{"output"; "[[O, 1, 2, 3] [0, 0, 2, 3] 30,[0, [0, 0, 2. 11, [0, 0, 0, 1]]">
Prompts C.2: An Case from Codeforce in TracePile
<Problem> Vasya is very upset that many people on the Net mix uppercase and lowercase letters in one word. That's why he decided to invent an extension for his favorite browser that would change the letters' register in every word so that either only consisted lowercase letters or, vice versa, only of uppercase ones. At that as little as possible letters should be changed in the word. For example, the word HoUse must be replaced with house, and the word ViP with VIP. If' a word contains equal number of uppercase and lowercase letters, you should replace all the letters with lowercase ones. For example, ma TRIx should replaced by matrix. Your task is to use the given method on one given word. Input Specification:
The first line contains word *** it consists of uppercase and lowercase Latin letters and possesses the length from to 100.
19

---

Output Specification:
Print the corrected word *** If the given word *s has strictly uppercase letters, make the word written in the uppercase register, otherwise - in the lowercase one.
Demo Input:
I'HoUse', "ViP', 'maTRIx')
Demo Output:
('house', "VIP", )matrix')
Note:
none <Problem>
And the input of the problem: <Input> aOvSeu <Input>
Predict the output of the problem by tracing the code step by step:
<Code>
def solution(s);
upper sum (1 for in if c.isupper ()
lower sum (1 for in if c.islover ())
if lower >= upper:
return s, s.worrr()
else:
return s. upper ()
<Code>
CoE:
Let's break down the execution of the provided code step by step for the input aOvSeu.
1. Input Processing:
The input word is a0vSeu. It includes both uppercase and lovercase letters.
2. Counting Uppercase Letters: Uppercase Letters Identified: 0, s Uppercase Count (upper) 2
3. Counting Lovercase Letters: Lovercase Letters Identified: a, v, e, u Lovercase Count (lower): 4
4. Decision Making Based Counte: Comparison: Is the number lowercase letters (4) greater than or equal to the number of uppercase letters (2)?
Evaluation: 4 >- 2 -> True
5. Transforming the String: Since the condition lover upper is True, the entire string is converted to lovercase.
Tr Transformation:
Original String: a0vSeu Transformed String: aoyseu
6. Appending Newline Character:
Final Output: {"output": "aovseu"".
20

---

Prompts C.3: Case of the Biapartite Problem in TracePile
Input: Given an undirected graph with nodes [Alice Johnson, Bob Smith, Catherine Lee, David Brown], the adjacency list is as follows: Alice Johnson: [Bob Smith, Catherine Lee, David Brown], Bob Smith: [Alice Johnson]. Catherine Lee: [Alice Johnson], David Brown: [Alice Johnson].
The code with initialized parameters is as follows:
Code:
def hungarian_algon
(adj_list, m_left, n_right_start)
match_right {} match_pairs []
def dis (node, visited);
for neighbor in adj_ (]): if not visited[meighbor. visited (neighboe) True
if neighbor not in match_right: (icch_rightteeighbbo return True elif dfs (nttc match_right [neighbor] node return True
= node
visited):
return False max_matching for node in range(n_left))
visited (key: False for key in rangstt.right.start, n_right_start + n_left))
if dís (node, visited):
max_matching
for node, left_ node in match_right items () match_pairs... append ((left_node, right_node))
return max_matching, match_pairs
What is the size of the maximum matching? Please solve the problem by tracing through the code.
CoE:
Let's run the function hungarian algorithm
hungarian__lloorthmn Baciiimmmwtttthhheeeeee with with_paths(adj__isn list, n n_left, n n_right_stard
Initial values: adj_list = {Alice Johnson: [Bob Smith, Catherine Lee David Brown], Bob Smith: [Alice Johnson], Catherine Lee: Alice Johnson], David Brown: [Alice Johnson]}, D_left Bob
Smith, n_right_start = Bob Smith
Call aat.ariinnngggrrrtmmiieeeee
((Alice Johnson: [Bob Smith, Catherine Lee. David Brown], Bob Smith: [Alice Johnson], Catherine Lee: [Alice Johnson], David Brown: [Alice Johnson]), Bob Smith,Bo Bob Smith)
raghh_right = ( and match_pairs- [].
max_matching = for node: Alice Johnson Current match_right:
Initialize visited (Bob Smith: False, Catherine Lee: False, David Brown: False)
Call (Alice Johnson, (Bob Smith: False, Catherine Lee: False, David Brown: False))
for neighbor: Bob Smith
Since neighbor Bob Smith is not visited
visited (Bob Smith] = True
match_right O
Since neighbor Bob Smith not right
migch_right [Bob Smith] Alice Johnson
Return True
Since dis (Alice Johnson (Bob Smith: True, Catherine Lee: False, David Brown: False)) returns True
21

---

nax_matching meximum_macttcnn 1, max_matching: 1 Iterate over natch_right: right_node = Bob Smith, ledt_node Alice Johnson
Add pair (Alice Johnson Bob Smith) to match_pairs All nodes in set 1 are traversed, max_mat nax_matching [(Alice Johnson, Bob Smith)] So the size of the maximum matching is
Final Output: ("output"; "1"3.
Prompts C.4: Case of mathematical problem from TracePile
Given a mathematical problem and its solution code, please analyze the execution of the above solution code step by step.
Provide a detailed, natural-language, step-by-step reasoning process that explains how the code solves the given mathematical problem.
Problem:
What is the sum of all the prime numbers between and 10?
Solution Code:
def is is_prime (num):
if num 2:
return False
for i range (2, int (num**0.5) + 1) num return False
0:
return True
primes for in range 11) if is-prime (1))
print (sum (primes))
CoE:
1. **Define the 'is_prime" Function:**
The function 'is_prime" is defined to check if given number num is prime number.
*+Check if 'num" is less than 2:**
-If is less than 2, the function returns "Fasee because
and are not prime numbers.
C+ehe divisibility from 2 to the square root of 'num f** The function iterates through all numbers from to the integer part the square root of 'num" + (inclusive).
For each number 'i' in this range, it checks if 'num is divisible by 'i' (i.e., 'num % i == 09).
If 'num" is divisible by any 'i', it returns "False' because 'num' is not prime number.
**Return 'True'
no divisors are found:**
If the loop completes without finding any divisors, the function returns 'True', indicating that 'num is a prime number.
2. Generate the List of Prime Numbers:**
A list comprehension is used to generate list of prime numbers between and 10.
The list comprehension iterates through each number (i the range from 1 to 10 (inclusive).
For each 'i', it calls the 'is-prime' function.
If ( 'is_prime(1)" (i) returns 'True the number 'i' is included in the list 'primes'.
22

---

3. Calculate the Sum of Prime Numbers: **
The 'sum" c function is used to calculate the aum of all the numbers in the list 'primes'.
4. **Output the Result:**
The result of the 'sum' function printed to the console.
#*# Detailed Execution
1. Check if 1 is a prime number "is_prime(1) (1) ; Since is less than 2, the function returns False'.
-1 is not included in the list 'primes'.
2. **Check if 2 is a prime number:**
( (2) : 2 is not less than 2.
The loop from 2 to the square root of (which is 1.414, so the integer part empty.
1) not executed because the range
The function returns "True', 2 is included in the list 'primes'.
Check if
'is_prime (3) prime number:*.
is not less than 2.
The loop from 2 to the square root of 3 (which is 1. 1.732, so the integer part is 1) is not executed because the range is empty.
The function returns 'True', 3 is included in the list 'primes'.
4. ** Check if 4 is prime number:** "is_prime (4)
less than 2.
The loop from 2 to the square root of (which is 2) checks 4 is divisible by 2.
Since 4 is divisible
2, the function returns 'False', is not included in the list 'primes'.
5. ** Check if 5 is a prime number:*. "is_prine (5) 5 5 is not less than 2.
The loop from 2 to the square root of 5 (vhich is 2.236, the integer part
divisible by Since is not divisible by 2, the function returns 'True', is included in the list 'primes'.
2) checks if
6. ** Check if 6 is prime number:** "is_prime (6) 5
less than 2.
The loop from 2 to the square root of (vhich is 2.449, so the integer part is 2) checks if 6 is divisible by 2. Since 6 is divisible
2, the function returns 'False', is not included in the list 'primes'.
7. Check if 7 is a prime number:*.
"is_prime(7) 7 is not less than 2.
The loop from 2 to the square root of 7 (vhich is 2. 2.646, so the integer part 2) checks if
divisible by 2. Since is not divisible by 2, the function returns 'True', 7 is included in the list 'primes'.
8. Check if is prime number:** 'ismprime (8) f:
less than 2.
23

---

The loop from 2 to the square root of 8 (which is 828 so the integer part is 2) checks if 8 is divisible by 2. Since 8 is divisible by 2, the function returns 'False'. is not included in the list 'primes'.
9. ** Check if 9 is prime number
"is_prime (9) 9 is not less than 2. The loop from
the square 9 divisible by 2 and 3.
of (which is 3) checks
Since 9 is divisible by 3, the function returns 'False'. is not included in the list 'primes'.
10. Check if is prime number:**
"is_prima (10) : 10 is not less than 2.
The loop from 2 to the square root of 10 (which is 3.162, so the integer part is 3) checks if 10 is divisible by 2 and 3. Since 10 divisible by the function returns 'False 10 not included the list 'primes'.
### Final List of Primes
The list 'primes' contains the numbers: [2, 3, 5, 71.
### Calculate the Sum
sum list 'primes' + calculated: ### Output the Reault
7 17.
The final output is 17.
# ### Final Answer
17
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
We do not include theoretical equations or results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3.1: Experimental Settings
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
Section 3.1: Experimental Settings and
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
iy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.1: Experimental Settings Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section 3.1: Experimental Settings
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 3.1: Experimental Settings
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: /tte:::////uis..eeeeeeeee http://.......cccccccccccceeieia
Answer: [Yes]
Justification: Our codes based on public llama-factryy
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not include it.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not include it.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section 3.1 Experimental Settings
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
29

---

Answer: [Yes]
Justification: Section 2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not include it.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not include it.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
30

---

Justification:
We do not include it.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:ses//eerrpppocccc//ooreeees for what should or should not be described.
31


Reviews:
[{"review_id": "n3cIihYFJR", "reviewer": "Reviewer_R7E8", "summary": "- This work presents TracePile, a large dataset of \"chain of execution\" (CoE) rationales, which are natural language chain of thought explanations of code execution.\n- Their code comes from 3 sources: (1) they use algorithmic competition code (primarily Codeforces) and prompt an LLM to extract the functions and reference input/output examples, verifying that the code executes to the actual input and output. (2) For a set of 30 classic algorithms, they use generators from prior work (CLRS-text and GraphInstruct) to generate intermediate states which they can use to verify intermediate results of chain of execution. (3) They also add Math code from OpenMath which is already paired with solver code, and filter out any problems that are solved by LLaMA3-8B in all three of three tries.\n- For the algorithmic and graph problems, they diversify the dataset by asking not only about output prediction but also to predict intermediate values – an LLM generates these variants. And furthermore they diversify the code by having an LLM rewrite the code to use different approaches to the same problem or to be refactored differently.\n- Finally they few-shot prompt Qwen-2.5-72B-Instruct to generate the actual CoE traces, either telling it to answer the given question by tracing code execution or telling it to traces the changes to a particular variable while explaining each step. They verify these intermediate states for the algorithmic problems using ground truth traces.\n- They explore continue-pretraining on TracePile, pretraining on TracePile then instruction tuning on TuluSFT, and fine tuning first on TracePile then on TuluSFT; the last of these options provides the largest benefit. Training boosts performance considerably on coding and math tasks, but also other domains outside the dataset like logical reasoning.", "questions": "- See questions in \"Weaknesses\"\n- Why was the particular set of benchmarks chosen for table 4?\n- Lines 306-308 state: \"Among them, code rewrites contribute most to structural generalization, while query diversification enhances task-specific interpretability.\" Unlike the other lines of this paragraph, this one didn't obviously follow from the results in the table for me. Where does this conclusion come from?\n- Lines 154-155 state: \"For tasks requiring variable tracking or control-flow interpretation, we require the model to output structured json results.\" What is meant by \"control flow interpretation\" here? If it's any tasks requiring control flow like loops or if-statements that must be almost every coding task?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I've identified a few minor issues and points of confusion, but overall I believe this paper is a good fit for NeurIPS and support its acceptance. The evaluation is very thorough and results are quite strong, with sufficient ablations and additional comparisons to support that the major components of the dataset construction process actually matter for performance.\n\nStrengths\n- There are a lot of parts to the dataset construction, but it is explained well – I've listed any of my confusions in Weaknesses and Questions.\n- The results in Tables 1 and 3 for continued pretraining and two-stage fine tuning look excellent. Training on TracePile seems to quite generally boost a range of models on a wide range of benchmarks, with few and minor regressions. The further appendix results (Table 7) look in line with this as well. The improvements on benchmarks like CLRS and GraphInstruct that were the basis for parts of the training dataset is not at all surprising, but they also still see quite considerable boosts to the wide array of other benchmarks. The number of datasets used in this evaluation is quite impressive. I appreciate that Table 3 doesn't use the base model but rather the TuluSFT trained model, which does a lot better than the base model on some benchmarks (especially near the bottom of the Table), as this is helpful for disentangling the effect of the second stage of fine tuning.\n- The Discussion section with four additional research questions is very well done. In particular I appreciated the ablations of RQ2, which are important to showing that all of the complexity involved (i.e. the numerous sources of data and the diversity features) actually contribute to performance. And similarly RQ3 was important to showing that it's actually worth it to train on such a large dataset.\n\nWeaknesses\n- Figure 3 is a little confusing for a few (fixable) reasons.\n    - One is that the LLama-3.1-8B line (blue) doesn't make a lot of sense as it's own line, since it doesn't scale with the data and is actually just a single point – and in fact it's just the *zero* data point which could go to the left of 50K. Based on how it is 60.0 in the Math domain, it must be the TuluSFT tuned model (the \"Base\" model from Table 3) so I believe this would make sense as a Data Scale = 0 point.\n    - Secondly, it's confusing that the x axis is nonlinear and has quite variable gaps. Plotting it myself, I think this could look fine with a linear scale from 0 to 4.3M on the x axis. It ends up looking like a rather sharp first jump (in the 0K to 50K segment) then smoothing out.\n    - It could make sense to make the very last segment a different color or something, to indicate that this is coming from a different process (rejection sampled data augmentation)\n- Details are missing in a handful of places:\n    - In RQ4 of the discussion the rejection sampling based augmentation to make TracePile++ is mentioned, but details aren't given – including these in an appendix would be helpful.\n    - Lines 159-161 state: \"We validate correctness by comparing these outputs to ground-truth traces generated via code instrumentation or symbolic tools, discarding samples that fail consistency checks\". This feels key to data quality, and I'd appreciate more details on it (e.g. in an Appendix, but also in the response to this review). Am I correct that this is separate from the intermediate results checks from the Classical Algorithm Code section? Is this applied to many of the problems from the competition code? How does the code instrumentation work, and what symbolic tools are used?\n\nQuick fixes and suggestions\n- \"CoE formats significantly\" (line 139) I think there's a missing word in here?\n- Table 2 has a typo in comma placement in the entry \"117,083,6\"\n- The abstract says \"TraceMind\" instead of \"TracePile\" in one place.\n- line 230 should just say Tables 1 and 3, not Tables 2 and 3\n- I found Figure 1 (the splash figure) a bit difficult to read without context – in particular, I think a longer caption could help clear it up. In particular it would be helpful to explain that the 3 boxes on the left are actually 3 different data sources for the dataset. Perhaps putting the size column of each of these from Table 2 into this figure would also be a nice touch.", "was_revised": true, "final_justification": "Questions and concerns were resolved, maintaining a score of Accept as justified by my original review.", "rebuttal": "We sincerely thank you for your comprehensive review and the constructive feedback shared. Below are our responses for your insightful comments.\n\n## Q1: Improvements of Figure 3, including the LLama-3.1-8B line, x-axis scaling, and final segment distinction.\n\nWe appreciate the reviewer’s insightful feedback on Figure 3. We will plot the LLama-3.1-8B as a single point at Data Scale = 0, use a linear x-axis from 0 to 4.3M, and visually distinguish the final segment with a different color to indicate rejection-sampled data augmentation. These adjustments will enhance clarity and resolve the concerns raised.\n\n## Q2: Additional details about rejection sampling for constructing TracePile++.\n\nWe thank the reviewer for highlighting this crucial aspect, which directly touches on the diversity and quality of our dataset. Specifically, we construct TracePile++ by sampling five CoE rationales per problem using Qwen-2.5-72B-Instruct, and select semantically diverse yet correct outputs using Sentence-BERT for semantic differentiation. The resulting samples, clearly differing from the original TracePile outputs, are incorporated into TracePile++. We will include comprehensive details of this method in the appendix of the camera-ready version.\n\n## Q3: Clarification on the correctness validation approach mentioned in lines 159–161, including code instrumentation and symbolic tools.\nThe reviewer insightfully identifies a core component of our data quality assurance process. To ensure correctness, particularly for Algorithmic Competition Code, we utilize Python's **snoop** tool, which provides detailed execution traces by capturing variable states and changes throughout code execution. These generated traces are then systematically compared with the model-generated CoE outputs, discarding inconsistent samples. This validation method, distinct from intermediate result checks used in Classical Algorithm Code, ensures semantic fidelity and consistency. We will add detailed explanations of this process, including the use of snoop, in the appendix of the revised manuscript.\n\n## Q4: Minor corrections and clarity improvements (typos, table references, figure readability).\n\n We greatly appreciate the reviewer's careful proofreading and thoughtful suggestions to enhance clarity. We acknowledge these minor mistakes and readability issues, and sincerely apologize for the oversight. We will correct the mentioned typos (e.g., \"CoE formats significantly\", \"117,083,6\"), replace \"TraceMind\" with \"TracePile\" consistently in the abstract, and properly reference Tables 1 and 3 instead of Tables 2 and 3. Furthermore, we will expand the caption of Figure 1, clearly indicate the three distinct data sources, and consider integrating dataset size information to improve readability. All these improvements will be incorporated into the camera-ready version.\n## Q5: Reasoning behind the selection of benchmarks in Table 4.\n\nThe reviewer raises an important point about benchmark selection. The benchmarks shown in Table 4 were specifically chosen to ensure a fair comparison, as the CodeI/O models based on TuluSFT are not publicly available. Thus, we selected benchmarks explicitly evaluated in both the CodeI/O paper and our own study to enable a direct and equitable comparison.\n\n## Q6: Clarification of the conclusion in lines 306–308 regarding the effects of code rewrites and query diversification.\n\nThe reviewer insightfully highlights a key point regarding our interpretation of results. In our analysis, we consider performance on logical reasoning benchmarks as a strong indicator of structural generalization. We observed that removing code rewrites resulted in a more noticeable performance drop on logical reasoning tasks, while removing query diversification had a greater negative impact on mathematical reasoning. This differential pattern guided our conclusion that code rewrites predominantly improve structural generalization, whereas query diversification mainly enhances task-specific interpretability. We will clarify this reasoning explicitly in the revised manuscript.\n\n## Q7: Clarification on the meaning of \"control-flow interpretation\" mentioned in lines 154-155.\n\nWe thank the reviewer for highlighting this ambiguity. Indeed, the reviewer’s interpretation aligns correctly with our intent. By \"control-flow interpretation,\" we specifically refer to tracking changes in structures like arrays, stacks, or heaps during code execution—commonly involved in classical algorithmic tasks. While such tracking is prevalent in most algorithmic tasks involving loops or conditionals, a substantial portion of our dataset consists of mathematical code, which frequently does not require detailed control-flow tracking. We will explicitly clarify this distinction in the revised manuscript."}, {"review_id": "X2a74klvrq", "reviewer": "Reviewer_2Sy7", "summary": "This paper introduces TracePile, a data corpus of 2.6 million samples designed to enhance reasoning abilities in large language models through Chain of Execution (CoE) supervision. This paper proposes to transform code execution into explicit, step-by-step natural language explanations that trace through mathematical problems, classical algorithms, and algorithmic competition code. The dataset spans diverse reasoning paradigms with additional augmentation strategies for query and code diversification to enhance logical granularity. Through experiments across four base models (LLaMA 3/3.1 and Qwen-2.5) and 20 benchmarks covering mathematical, logical, algorithmic, and code reasoning tasks, the authors demonstrate that TracePile consistently improves performance over these benchmarks.", "questions": "- Could the authors add a discussion section in the paper to explain how their approach is similar to and different from SemCoder and Code I/O? It is ok to suggest CoE is inspired by existing works, but a discussion to specify the similarity and difference will be helpful to understand the unique contribution of this work\n- Could the authors submit the data corpus for reviewers to assess its quality during the rebuttal phase?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n+ The paper provides a large corpus to learn the code execution reasoning, which successfully improves LLMs' general reasoning capabilities -- this successful knowledge transfer is and exciting finding.\n+ This paper implements \"intermediate results verification\" to ensure the quality of the annotated reasoning text, which guarantees the correctness of intermediate reasoning steps.\n+ Training with TracePile consistently boosts base models of varied sizes with better reasoning performance across multiple reasoning benchmarks.\n\n## Weaknesses\n\n- Limited Technical Contributions Over Existing Works\n\nWhile TracePile presents a large-scale dataset for code execution supervision, the paper's core contribution exhibits limited novelty compared to recent work in code execution reasoning, particularly SemCoder (Ding et al., 2024) and CodeI/O (Li et al., 2025). SemCoder already introduced the concept of training code language models with comprehensive semantics reasoning through \"monologue reasoning,\" where models learn to articulate code execution step-by-step in natural language, similar approach to TracePile's Chain of Execution (CoE). Similarly, CodeI/O also transforms code execution into natural language explanations for training purposes. While TracePile claims to focus on \"fine-grained execution traces,\" this distinction appears incremental rather than fundamentally novel, as existing works already demonstrated that natural language execution reasoning could effectively bridge static code and dynamic execution states.\n\n- Unavailability of the Core Dataset and Inadequate Reproducibility Provisions\n\nA fundamental weakness of this paper is the unavailability of TracePile, the core 2.6 million sample dataset that constitutes the primary contribution. Despite claiming that their work introduces a \"large-scale corpus\" as the main contribution, the authors fail to provide access to the actual dataset, severely limiting the paper's impact and reproducibility (Please correct me if I am wrong). \n\nFurthermore, the authors' response to the reproducibility checklist is misleading—they cite \"Section 3.1: Experimental Settings\" as evidence for providing \"open access to data and code with sufficient instructions to faithfully reproduce the main experimental results.\" However, Section 3.1 only describes training hyperparameters and evaluation protocols, not data availability or access instructions. The experimental settings alone are insufficient for reproducibility when the core dataset remains inaccessible. \n\nFor a paper where the dataset is the primary contribution and the foundation for all experimental results, the failure to provide data access represents a critical flaw that undermines the work's scientific value. The authors should commit to releasing TracePile during the rebuttal phase to enable proper peer review and future research validation.\n\n- (minor) In the abstract, the data corpus was named as TraceMind at first. I assume this should be a typo", "was_revised": true, "final_justification": "After the communication during the rebuttal, I keep my score of 4 as my final attitude\n\nI appreciate the authors commitment of adding more detailed discussion and comparison with SemCoder and Code I/O in the camera-ready.\n\nHowever, though I understand the policy from certain companies that the publication of data and code needs approval, I still choose to be conservative about my rating at this time -- I keep my score of 4 to show my positive attitude toward this paper and the authors' commitment regarding the final open-source artifacts, but without manually assessing the quality of TracePile, I choose not to increase my score to an absolute level (i.e., 5) at this moment.", "rebuttal": "We sincerely thank you for the comprehensive review and the constructive feedback shared. Below are our concise responses to your insightful comments.\n## Q1: Concerns about the novelty and incremental nature of TracePile compared to existing works (SemCoder and CodeI/O).\n\nWe appreciate the reviewer for highlighting the critical aspect of novelty and differentiation. Although TracePile shares the general goal of leveraging natural language for code execution reasoning with SemCoder  and CodeI/O, our work introduces fundamental and distinct contributions:\n\n### Execution Tracing Granularity:\n\n- SemCoder employs \"monologue reasoning,\" emphasizing high-level functional descriptions, constraints, and verbal reasoning about local execution effects. However, it primarily uses synthetic data generated from scratch with limited real-world complexity\n- CodeI/O condenses reasoning patterns by predicting inputs/outputs from given functions in natural language, but the explanations focus predominantly on abstract input-output reasoning without detailed execution tracing of internal program states\n- TracePile (ours) explicitly emphasizes fine-grained, structured execution traces (Chain-of-Execution, CoE), capturing detailed variable state changes and control flow at each step. This structured granularity significantly surpasses the abstract verbal reasoning of SemCoder and CodeI/O, enabling models trained on TracePile to exhibit deeper understanding and robust execution reasoning capabilities.\n### Data Sources:\n\nWhile SemCoder’s data (PYX) is entirely synthetic and CodeI/O derives data from generic raw code files transformed into structured reasoning tasks, TracePile uniquely integrates  diverse, real-world-oriented sources: *classical algorithms with intermediate checks, algorithmic competition code verified with detailed execution traces, and synthetic algorithmic problems designed for high complexity，and mathematical code problems for  multi-step decomposition*.\n\nThis deliberate selection ensures broad coverage of real-world coding challenges, including tasks that demand intensive structural reasoning, dynamic tracking of complex data structures (arrays, stacks, heaps), and control-flow interpretation—scenarios less comprehensively addressed by SemCoder or CodeI/O.\n### Training Paradigm:\n\nSemCoder and CodeI/O primarily rely on **instruction tuning** explanations. TracePile adopts three different  multi-stage training paradigm: **continue pretraining (cpt) + few-shot prompting**; **continue pretraining + general instruction tuning**; **two-stage instruction tuning**. We are the first to prove that CoE-similar format data could contribute the model performancnes in cpt stage. Our compresive trainning settings further push the boundry of using data in the field.\n\n\nIn summary, TracePile distinguishes itself clearly through its structured, fine-grained trace annotation, comprehensive use of real-world coding scenarios with detailed instrumentation, and multi-stage augmented training strategy, thus substantially enhancing model performance on complex, real-world execution reasoning tasks.\n\nWe will clarify these differences explicitly in the revised manuscript.\n\n## Q2: Concerns about data availability and reproducibility due to the non-release of the TracePile dataset.\n\n We sincerely appreciate the reviewer highlighting the critical aspect of data availability and reproducibility. Indeed, TracePile was not publicly released at submission due to company-related copyright constraints. As per NeurIPS guidelines, we are unable to introduce new data or anonymous links during the rebuttal phase. Currently, we are actively processing internal company approval for public release of TracePile in one month. Once approved, we commit to releasing the full dataset on Hugging Face, providing comprehensive instructions and scripts to ensure full reproducibility. ***This ongoing approval process is also why we have refrained from publicly posting our manuscript to platforms such as arXiv at this time***.\n\n## Q3: Suggestion to add a discussion explicitly comparing TracePile to SemCoder and CodeI/O, clearly highlighting similarities and differences.\n\nWe thank the reviewer for this excellent suggestion, which indeed helps clarify our unique contributions. We partially addressed this comparison in **our related work section** (e.g., lines 552–562 for CodeI/O), highlighting some key differences. To further enhance clarity, we will expand the discussion explicitly comparing TracePile's Chain-of-Execution (CoE) to SemCoder and CodeI/O, clearly stating how our structured fine-grained execution traces, real-world-oriented datasets, and multi-stage augmentation differ from these prior works (**R1 to Q1**). We agree that explicitly specifying both similarities and differences will greatly benefit the reader’s understanding of our unique contributions, and we will incorporate this detailed discussion into the revised manuscript."}, {"review_id": "77BnvcBbpH", "reviewer": "Reviewer_ruAL", "summary": "The paper presents a technique to augment data for continued pertaining of large language models to improve the coding and math abilities of the models. The technique constructs a dataset TracePile by collecting algorithmic data of 3 sources: competition code (mainly from Codeforces), classical algorithmic code, and mathematical code. These collected samples are further diversified by modifying the query or the code. The technique uses a language model (Qwen-2.5-72B in this case) to obtain the chain-of-execution data from under carefully designed prompting strategies.\n\nThe TracePile dataset is used to continually pretrain 4 base models: Llama 3 & 3.1, Qwen-2.5, and Qwen-2.5. The continue-pretrained model shows consistent improvement over the base model on a diverse set of coding and math problems.", "questions": "How do the experiments avoid test-data contamination?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts:**\n\n1. The paper is well-written and easy to follow. The idea is simple and effective. \n\n2. Experiments are comprehensive and cover diverse coding and math benchmarks. \n\n3. The research questions discussed in Section 4 are insightful and cover most of the crucial ablations. The RQ1 shows that chain-of-execution data is more effective than I/O style supervision from previous work, CodeI/O (Li et. al. 2025). \n\n**Weaknesses:**\n\n1. The experiments are focused on coding and mathematical benchmarks, and authors should show the effect of proposed continual pretraining on other common benchmarks not related to coding or math.\n\n2. The code or data is not shared. Justification for the checklist question 5 about code and data is incomplete. Some of the other answers to the checklist questions are unclear as well.  \n\n**Minor:**\nTypo: LLaMA -> Llama", "was_revised": true, "final_justification": "Authors have addressed my concerns.", "rebuttal": "We are truly grateful for your thorough review and insightful feedback. Our responses to your comments are outlined below.\n\n\n## Q1: Effects of CoE in other reasoning benchmarks beyond math and code.\n\nThank you for requesting further clarification on the effect of our proposed continual pretraining on benchmarks not related to coding or mathematics. We appreciate this opportunity to highlight the broader generalizability of our method. As you noted, this information is indeed already present in our paper.\n\nSpecifically, in Table 1 of our submission, we report the performance improvements of models after the continual pretraining stage across various general reasoning benchmarks, including logical and algorithmic reasoning.\n\nBelow is a summary table, extracted from our paper, showcasing the performance of Llama-3-8B and Llama-3.1-8B models on these non-coding and non-mathematical benchmarks after continual pretraining with TracePile:\n\n| Model         | Domain              | Dataset        | Baseline (Base) | Our Method (Ours) | Improvement (%) |\n|---------------|---------------------|----------------|------------------|--------------------|------------------|\n| LLaMA-3-8B     | Logical Reasoning   | Zebra Puzzle   | 0.1              | 2.8                | **+2.7**             |\n|               |                     | KORBench       | 7.4              | 21.8               | **+14.4**            |\n|               |                     | Ruletaker      | 21.8             | 42.4               | **+20.6**            |\n|               | Algorithm Reasoning | Graphwiz       | 4.5              | 46.0               | **+41.5**            |\n|               |                     | GraphInstruct  | 35.2             | 73.5               | **+38.3**            |\n|               |  Instruction-Following                   | IFEval  | 11.5             | 25.5               | **+14.0**            |\n| LLaMA-3.1-8B   | Logical Reasoning   | Zebra Puzzle   | 2.3              | 20.6               | **+18.3**            |\n|               |                     | KORBench       | 7.4              | 45.6               | **+38.2**            |\n|               |                     | Ruletaker      | 21.9             | 45.6               | **+23.7**            |\n|               | Algorithm Reasoning | Graphwiz       | 33.9             | 69.2               | **+35.3**            |\n|               |                     | GraphInstruct  | 33.0             | 69.2               | **+36.2**            |\n|               |  Instruction-Following                   | IFEval  | 11.6             | 21.3              | **+9.7**            |\n\nAs the table illustrates, our TracePile method, during the continual pretraining stage, yields substantial performance gains on Llama 3-8B and Llama 3.1-8B models across logical and algorithmic reasoning  and instruction following tasks. \n\n\n## Q2: Access to Data\nThank you for your diligent review and for pointing out critical issues regarding our NeurIPS checklist responses, particularly concerning the transparency of code and data access. We apologize for any ambiguity or incompleteness in our initial submission.\n\nYou are absolutely correct that our response to Checklist Question 5 (\"Open access to data and code\") is misleading if the code and data are not openly shared. We acknowledge this discrepancy.\n\nIndeed, TracePile was not publicly released at submission due to **company-related copyright constraints**. As per NeurIPS guidelines, we are unable to introduce new data or anonymous links during the rebuttal phase. Currently, we are actively processing internal company approval for public release of TracePile in one month. Once approved, we commit to releasing the full dataset on Hugging Face, providing comprehensive instructions and scripts to ensure full reproducibility. ***This ongoing approval process is also why we have refrained from publicly posting our manuscript to platforms such as arXiv at this time.***\n\n## Q3: Avoid test-data contamination\n\nThank you for your critical question regarding test-data contamination. Ensuring the validity and trustworthiness of experimental results is paramount.\n\n**How Test-Data Contamination is Avoided (General Principles):**\n\nAvoiding test-data contamination primarily involves:\n1.  **Strict Data Splitting:** Ensuring no overlap between training, validation, and test sets. Test data must remain unseen during all training and hyperparameter tuning phases.\n2.  **Independent Data Sources:** Ideally, test data should originate from sources entirely separate from training data, or represent novel scenarios the model has not encountered.\n3.  **Leveraging Established Benchmarks:** Utilizing widely accepted public benchmarks with pre-defined splits and robust de-duplication/anti-contamination measures.\n4.  **Preventing Information Leakage:** Ensuring no implicit information from the test set influences the training process (e.g., through feature engineering, data augmentation, or model selection that \"peeks\" at test data).\n\n**How We Avoided Test-Data Contamination in Our Paper:**\n\nIn our work, we implemented several rigorous measures to prevent test-data contamination and ensure the validity of our experimental results:\n\n1.  **Independent Evaluation Benchmarks:** We conducted comprehensive evaluations on **20 diverse, publicly available benchmarks** spanning mathematics, code, logical, and algorithmic reasoning (as detailed in **Section 3.1** and **Appendix C** of our paper). These benchmarks (e.g., GSM8K, MATH, LiveCodeBench, CRUX, Zebra Logic) are well-established with pre-defined splits and are managed by public evaluation toolkits (OpenCompass, Qwen2.5-Math, ZeroEval), ensuring consistency and reproducibility.\n\n2.  **Domain and Format Discrepancy:** Our TracePile dataset, while code-execution-based, focuses on **fine-grained, CoE-style natural language reasoning traces**. The evaluation benchmarks, especially those targeting out-of-domain generalization (e.g., LiveCodeBench, CRUX, RuleTaker, Zebra Logic), often differ significantly in format, domain, or reasoning style from our training sources (as discussed in **Section 4, \"Out-of-domain (OOD) Generalization\"**). This distinction helps ensure that the model learns generalizable reasoning capabilities rather than simply memorizing test patterns.\n\n3.  **Rigorous Data Generation and Filtering:**\n    * For mathematical code data, we employed a **model-based filtering strategy** (Section 2.1) where problems correctly answered three times by LLaMA3-8B were discarded as too simple, preventing overfitting to trivial or potentially similar test patterns.\n    * We enforced **intermediate result verification** and **consistency checks** (Section 2.3) for our CoE data, ensuring logical rigor beyond just the final answer. This encourages the model to learn deep reasoning logic.\nThese measures collectively ensure the reliability of our experimental results and the generalizability of our model's enhanced reasoning capabilities."}, {"review_id": "PpQrQLx5eF", "reviewer": "Reviewer_fNHA", "summary": "This paper introduces TracePile, a corpus of 2.6 million samples containing 25 billion tokens that transforms code execution into natural language COE explanations. The authors collect code from algorithmic competitions (Codeforces), classical algorithms (CLRS), and mathematical problems (OpenMath), then use Qwen-2.5-72B-Instruct to generate step-by-step execution traces. They evaluate this approach across three training paradigms (continue-pretraining, instruction tuning after pretraining, and two-stage finetuning) on four base models (LLaMA 3/3.1, Qwen-2.5, Qwen-2.5 Coder), reporting improvements on 20 benchmarks spanning mathematics, code, logic, and algorithms.", "questions": "Please see my comments above. \n\nIn addition, while authors mentioned that \"we impose a strict 8k-token limit\", from Table 2 it seems the avg #tokens is greater than 8k?", "limitations": "See \"Strengths And Weaknesses\"", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper addresses a compelling hypothesis that explicit execution traces can teach transferable reasoning skills. The scale of the effort (2.6M samples, 25B tokens) is substantial, and the diversity of sources (competitions, textbook algorithms, mathematical problems) provides good coverage of different reasoning patterns. The evaluation offers useful insights about integration strategies.\n\nHowever, several critical weaknesses undermine the contribution. Most significantly, the baseline comparisons appear unfair. The authors compare against CodeI/O [25] using different training configurations, e.g. CodeI/O uses only 1 epoch while TracePile uses 3 epochs in first stage SFT, this invalidates the claim and conclusion drawn in the paper e.g. RQ1/Table 4.\n\nThe paper also keeps comparing models fine-tuned on TracePile against base models which tells us nothing about the relative value of CoE versus other supervision approaches. A fair comparison would be CoE against models trained on equivalent amounts of alternative data (e.g., more code examples, mathematical solutions, or even generic instruction data) using identical hyperparameters.\n\nThe related work section is missing in the main paper which is not acceptable. In addition, there is no discussion on how this work is situated in the rich past literature on using execution traces to train models, e.g., TRACED, SemCoder, and CodeT. It'd be great if the authors could clearly state the novelty beyond existing work.\n\nThe intrinstic evaluation is also missing from the paper, i.e., whether the model learns to generate *correct* traces. It'd be a good add-on to complement existing evaluation.", "was_revised": true, "final_justification": "Thanks authors for the updates. These address my concerns and I have thus raised my score to 4. Please make sure these discussions appear in the final version of the paper.", "rebuttal": "We sincerely thank you for the comprehensive review and the constructive feedback shared. Below are our responses to your insightful comments.\n## Q1: Unfair comparison with CodeIO\nThank you for your valuable feedback regarding the baseline comparison. We understand your concern about the differing training epochs (1 for CodeI/O vs. 3 for TracePile).\n\nFirstly, we understand your emphasis on consistent training configurations. However, it's important to note that **due to significant differences in the scale and characteristics of our TraceMind dataset compared to the dataset used by CodeI/O, directly enforcing the same training epochs might not fully unleash the potential of our method**. Different datasets often necessitate varying training strategies and convergence steps to achieve optimal performance.\n\nCrucially, to ensure fairness, we also evaluated our model after just 1 epoch of training. Even in this configuration, our model significantly outperformed CodeI/O, demonstrating the inherent effectiveness of our approach and the TraceMind dataset. We will clarify these details and consider adding these comparative results to the paper to fully address your point.\n\n|Models (Qwen-2.5-Code)|GSM8K|MATH|Livecodebench-Output|Zebra logic|\n|---|---|---|---|---|\n|CodeIO|79.3|39.8|17.6|8.6|\n|Ours-1epoch|80.2|47.7|30.5|9.4|\n|Ours-3epoch |80.8|50.8|35.5|10.8|\n\nIn our experiments, we indeed conducted extensive explorations of model performance across different training epochs. The experimental results consistently showed that within 3 epochs, the model's performance continued to improve with more training steps, exhibiting superior average performance. Therefore, to present the best performance of TracePile in the paper, we chose to report the results obtained after 3 epochs of training.\n\n## Q2: Comparison with different supervision approaches\nThank you for your crucial feedback regarding the fairness and scope of our baseline comparisons. We entirely agree that ensuring fair training and testing conditions is paramount to accurately demonstrate the unique value of Chain of Execution (CoE) supervision against other approaches.\n\nWe agree that fair comparisons against alternative supervision methods are crucial.  Our paper already includes extensive comparisons beyond base models, as detailed in our discussion (RQ1) and Tables 3, 4, and 7.\n\n- Generic Instruction Data: Tables 3 and 7 show comparisons against TuluSFT, a widely recognized generic instruction dataset.\n\n- Alternative Structured Code Data: Table 4 explicitly compares TracePile against OpenMath (pure code solution data) and CodeI/O (different forms of code input-output data).\n\nTo provide a clearer overview of these comparisons,  we have conducted a comprehensive set of experiments using the Qwen-2.5-Coder 7B model under a two-stage fine-tuning paradigm. This allows for a more direct and equitable comparison of TracePile against various alternative supervision approaches. For OpenMathInstruct 1 and 2, we specifically selected 2.6M subsets and trained for 3 epochs to ensure comparable data scale and training duration.\n\n|Models (Qwen-2.5-Coder)|Stage-1|Stage-2|GSM8K|MATH|Livecodebench-Output|Zebra logic|\n|---|---|---|---|---|---|---|\n|Baseline|-| Tulusft|74.3|45.0|29.8|9.2|\n|Pure Code Solution| OpenmathInstruct 1-2.6M (3 epoch) |Tulusft| 80.1 | 46.1 |19.4 | 5.6|\n|Pure Math Solution| OpenmathInstruct 2-2.6M (3 epoch) |Tulusft| 82.1 | 51.1 |17.4 | 6.6|\n|generic instruction|Webinnstruct-2.6M (3 epoch)| Tulusft|81.3|49.0|18.8|9.1|\n|CodeIO | CodeIO-3.5M| Tulusft |79.3|39.8|17.6|8.6|\n|Ours-1epoch|Tracepile-2.6M (1epoch)|Tulusft|80.2|47.7|26.5|9.4|\n|Ours-3epoch|Tracepile-2.6M (1epoch)|Tulusft|80.8|50.8|35.5|10.8|\n\nAs shown, TracePile consistently excels across diverse reasoning domains. While pure math solutions perform slightly better on math benchmarks due to specialization, TracePile significantly outperforms all other baselines, particularly in LiveCodeBench-Output and Zebra Logic. This highlights TracePile's ability to enhance **generalizable reasoning capabilities** by fostering transferable understanding of logical and procedural patterns. \n\n## Q3: Concerns about Related Work\nWe apologize for the related work section's placement. It is indeed included in Appendix A (lines 536-562) due to space limitations. We will explicitly reference it in the main paper.\n\nOur work introduces fundamental and distinct contributions beyond prior literature like TRACED, SemCoder, and CodeI/O:\n\n1.  **Execution Tracing Granularity:**\n    * **TRACED:** While TRACED also uses code traces, its backbone is RoBERTa, limiting it to simpler classification tasks and primarily focusing on code understanding. **Crucially, its traces are not in natural language, which restricts its applicability to general reasoning tasks in LLMs.**\n    * **SemCoder:** Employs \"monologue reasoning\" with high-level functional descriptions and verbal reasoning about local execution effects. However, it primarily uses synthetic data with limited real-world complexity.\n    * **CodeI/O:** Condenses reasoning patterns by predicting inputs/outputs from given functions in natural language, but its explanations focus predominantly on abstract input-output reasoning without detailed execution tracing of internal program states.\n    * **TracePile (Ours):** Explicitly emphasizes fine-grained, structured execution traces (Chain-of-Execution, CoE), capturing detailed variable state changes and control flow at each step in natural language. This structured granularity significantly surpasses the abstract verbal reasoning of SemCoder and CodeI/O, enabling models trained on TracePile to exhibit deeper understanding and robust execution reasoning capabilities.\n\n2.  **Data Sources:**\n    * **SemCoder:** Relies entirely on synthetic data (PYX).\n    * **CodeI/O:** Derives data from generic raw code files transformed into structured reasoning tasks.\n    * **TracePile (Ours):** Uniquely integrates diverse, real-world-oriented sources, including:\n        * Classical algorithms with intermediate checks.\n        * Algorithmic competition code verified with detailed execution traces.\n        * Synthetic algorithmic problems designed for high complexity.\n        * Mathematical code problems for multi-step decomposition.\n        \n3.  **Training Paradigm:**\n    * **SemCoder and CodeI/O:** Primarily rely on instruction tuning for explanations.\n    * **TracePile (Ours):** Adopts three different multi-stage training paradigms: **continue pretraining (CPT) + few-shot prompting**; **continue pretraining + general instruction tuning**; and **two-stage instruction tuning**. We are the first to demonstrate that CoE-similar format data can significantly contribute to model performance in the CPT stage. Our comprehensive training settings further push the boundaries of using this type of data in the field.\n\nWe believe that these distinctions, particularly our emphasis on fine-grained, natural language CoE traces from diverse real-world sources and our exploration of multi-stage training paradigms, clearly establish the novelty and significant contribution of TracePile to the literature on enhancing LLM reasoning through execution supervision. We will ensure these points are more prominently highlighted in the main paper to address your concern directly.\n\n## Q4: Intrinsic Evaluation\nThank you for highlighting the importance of intrinsic evaluation, specifically whether our model learns to generate correct traces. We agree that this is a crucial aspect to complement existing evaluations and directly demonstrate the value of our Chain of Execution (CoE) supervision.\n\nTo address this, we conducted an additional, insightful experiment. We compared our TracePile CoE method against baseline approaches, including CodeI/O, on the LiveCodeBench-Output test set. Beyond just measuring the final output accuracy, we also assessed the** accuracy of the intermediate steps** generated by the models. For this intermediate step accuracy verification, we leveraged a powerful external model, **Qwen3-235B-A22B-Instruct-2507**, to check the correctness of each step in the generated traces.\n\nHere are the results of this intrinsic evaluation:\n|Models (Qwen-2.5-Coder)|Final Output Accuracy (%)|Intermediate Step Accuracy (%)|\n|---|---|---|\n|Baseline (TuluSFT)|27.2|18.8|\n|CodeI/O++|20.6|17.1|\n|Our Method (TracePile CoE)|**35.5**|**33.7**|\n\n**Intermediate Step Accuracy Explanation**: *This metric represents the proportion of samples where the final output is correct AND all generated intermediate reasoning steps are also verified as correct.* This metric more rigorously measures the end-to-end reasoning fidelity of the model.\n\nAs the table demonstrates, our TracePile CoE method not only achieves superior final output accuracy on LiveCodeBench-Output but also shows a significantly higher accuracy in its intermediate steps compared to both the generic instruction baseline and CodeI/O++.\n\nWe will adopt this more precise definition and ensure its calculation method is clearly articulated in the final version of the paper to more comprehensively demonstrate the advantages of our approach. Thank you again for your invaluable suggestion!\n\n\n## Q5: Average token of Tracepile\n\nRegarding your question about the 8k-token limit and the average token count in Table 2, you are correct to notice a potential discrepancy.\n\nUpon re-examination, we found Table 2's total token count inadvertently included ***tokens from few-shot examples within the data generation prompts***. After removing these, the corrected average token length per training sample is approximately **7452** tokens, and the total TracePile token count is around **19.5** billion tokens. This revised statistic accurately reflects the data adhering to our 8k-token limit and resolves the apparent contradiction. We will update Table 2 accordingly."}]

Author rebuttal:

Meta-review: This paper proposes a chain-of-execution formulation to transform the code to step-by-step evaluation natural language sentences, with such transformation, the created data then can be better leveraged for general reasoning. The authors generate 2.6M data with 25B tokens and demonstrate the effectiveness of the data and the method on multiple domains. 

Reviewers agree the paper is well motivated and the method is reasonable and novel. Compared with other methods, the approach shows great improvements. The experimental results deeply impress the reviewers, which shows a general reasoning boost method.
During the rebuttal phase, the authors and reviewers have necessary discussions and the authors have addressed most of the concerns, except one point about the open source data. The authors are encouraged to make necessary revision.
Decision: Accept (poster)