Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Authors: Kazuki Fujii
Yukito Tajima
Sakae Mizuki
Masaki Kawamura
Hinari Shimada
Taihei Shiotani
Koshiro Saito
Masanari Oi
Taishi Nakamura
Takumi Okamoto
Shigeki Ishida
Kakeru Hattori
Youmi Ma
Hiroya Takamura
Rio Yokota
Jun Sakuma
Naoaki Okazaki
Venue: ICLR 2026 Poster (2026)
Keywords: dataset
pretraining
code
llm
Primary area: datasets and benchmarks
TL;DR: An LLM rewriting method (transform-and-retain) refines pre-training corpora. Instantiated as SwallowCode/SwallowMath, it improves code/math performance within a fixed budget, demonstrating general utility across base models and code/math domains.

Abstract: The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora.  
We introduce two openly licensed pre-training datasets, released under the Llama 3.3 Community License, that significantly enhance LLM performance by systematically rewriting public data. SwallowCode ($\approx$16.1 billion tokens) refines Python snippets from The-Stack-v2 through a novel four-stage pipeline: syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into self-contained, algorithmically efficient examples. Unlike prior methods that rely on exclusionary filtering or limited transformations, our transform-and-retain approach refines low-quality code, maximizing data utility.
SwallowMath ($\approx$2.3 billion tokens) enhances Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions into concise, step-by-step explanations. Within a fixed 50 billion token training budget, continual pre-training of Llama-3.1-8B with SwallowCode boosts pass@1 by +17.0 on HumanEval and +16.1 on HumanEval+ compared to Stack-Edu, surpassing the baseline model's code generation capabilities. Similarly, substituting SwallowMath yields +12.4 accuracy on GSM8K and +7.6 on MATH. Ablation studies confirm that each pipeline stage contributes incrementally, with rewriting yielding the largest gains.
By releasing datasets, prompts, checkpoints, and pipeline code, we ensure reproducibility and provide a transferable transform-and-retain methodology that can be adapted to other base models and LLM rewriting setups.

Published as a conference paper at ICLR 2026
REWRITING PRE-TRAINING DATA BOOSTS LLM PER- FORMANCE IN MATH AND CODE
Kazuki Fujii
Yukito Tajima!
Sakae Mizuki
Masaki Kawamura'
Hinari Shimada
Taihei Shiotani'
Koshiro Saito
Masanari Oi'
Taishi Nakamura 1.2
Takumi Okamoto'
Shigeki Ishida'
Kakeru Hattori'2
Youmi Ma²
Hiroya Takamura²
Rio Yokota
Jun Sakuma' Naoaki Okazaki
Institute of Science Tokyo, Department of Computer Science
2 National Institute of Advanced Industrial Science and Technology
3 Institute of Science Tokyo, Institute of Integrated Research, Supercomputing Research Center
SwallowCode
SwallowMath
ABSTRACT
The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce two openly licensed pre-training datasets, released under the Llama 3.3 Community License, that significantly enhance LLM performance by systematically rewriting public data. SwallowCode 1=11 billion tokens) refines Python snippets from The-Stack-v2 through novel four-stage pipeline: syntax validation, pylint- pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into self-contained, algorithmically efficient examples. Unlike prior methods that rely on exclusionary filtering or limited transformations, our transform-and anntraain approach refines low- quality code, maximizing data utility. SwallowMath (=.2.3 billion tokens) enhances Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions into concise, step-by-step explanations. Within fixed 50 billion token training budget, continual pre-training of Llama-3.1-8 with SwallowCode boosts pass @ by +17.0 on HumanEval and +16.1 on HumanEval+ compared to Stack-Edu, surpassing the baseline model's code generation capabilities. Similarly, substituting SwallowMath yields + +12.4 accuracy GSM8K and +7.6 MATH. Ablation studies confirm that each pipeline stage contributes incrementally. with rewriting yielding the largest gains. By releasing datasets, prompts, checkpoints, and pipeline code, we ensure reproducibility and provide transferable cansform-and-eede methodology that can be adapted other base models and LLM rewriting setups.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable zero-shot and few-shot capabilities across diverse tasks, yet their proficiency in mathematical reasoning and program synthesis remains constrained by the quality of pre-training corpora. Existing public datasets for specialized domains, such as The-Stacks the-Stack-vl and v2 for code (Kocetkov et al.) 2022 Lozhkov et al., 2024) and Finemath- 4+ for mathematics (Allal et al. 2025), rely primarily rule-based extraction from web crawls (e.g., CommonCrawl) (Paster et al.) (023) or model-based scoring to filter low-quality samples. However, these approaches often retain noisy, redundant, stylistically inconsistent data, limiting their dffeetiveness-paaticlaatt in the growing trend of multi-stage pre-training (or mid-training) aimed enhancing mathematical reasoning and program synthesis (e.g., OLMo (OLMo al. 2025). Nemotron-E (NVIDIA et al. 2025). and Phi-4 (Abdin et al. 2024)).
Compounding this challenge. leading ween- frontier model families (e.g., Qwen3 Yang et 2025). DeepSeek-V3. V3 (DeepSeck-Al Al et al.) (2025) )) do not release their pre-training corpora. As result, open community lacks access to high-quality code and mathematics corpora comparable to those used in the state-of-the-an open-weight weight models, creating growing "data quality gap". To address

---

Published as a conference paper at ICLR 2026
0.41
HunonEral poron
040
030
036
030
03
Codepanet.diaa
The Stack v2 (smol-ids)
--- SwallowCode (ours)
Stad-Ed
Figure 1: Comparison of Python-only datasets in a 50 billion tokens continual pre-training setting. SwalloWCode achieves the highest pass@1 (umfeE and HumanEval--righht compared to CodeParrot- The-Stack-vl, The-Stack-v2-Smol and Stack-Edu.
this gap, propose SwallowCode and SwallowMath, two openly licensed datasets designed to raise the ceiling of open pre- pre-training for code generation and mathematical reasoning. Our contribution is data-centric and reproducible- open corpora and continually updatable wansform-and-red methodology in contrast to irreroduuibbe model comparisons that rely on undisclosed training data.
open
Unlike prior methods that solely filter preserve original samples, approach rewrites pre- training corpora to eliminate noise and redundancy, yielding high-quality, self-contained data that enables efficient learning. SwallowCode refines Python snippets from Thhe-Stack-22 via a four-stage syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into algorithmically efficient, self- contained examples. By rewriting rather than merely filtering, we remove persistent defects that remain in datasets like Stack-Edu (Allal et al. 2025) and obtain data that drives rapid accuracy gains (Figure T To demonstrate the generality of our transform-ant tadem--nndree -nd--rttaiin approach beyond code, we also applied it to the mathematics domain. Specifically, SwallowMatth transforms Finemath-4+ by removing boilerplate, restoring missing context, and reformatting solutions into concise, step-by-step explanations.
We deliberately avoided continual pre-training from near-frontier models (e.g.. Qwen-2.5/3. gpt-oss). These models already achieve very high performance in mathematics and code, so additional data yields only marginal gains. Such settings obscure whether improvements stem from the dataset itself or from incidental model factors, confounding the attribution results. In contrast, Llama-3.1 8B (Grattafiori et 2024) provides more informative starting point: it is strong enough a realistic baseline for open-community efforts, yet remains sufficiently below the frontier to avoid saturation, thereby allowing improvements be clearly attributed dataset quality.
as
Under this protocol, continual pre-training for 50B tokens on mixed dataset containing Swallow and multilingual сorporа improves pass@) by +17.0 on HumanEval and +16.1 on HumanEval+ compared to an equivalent budget using Stack-Edu. Substituting Finemath-4+ with SwallowMath yields +12.4 accuracy on GSM8K and +7.6 on MATH. To verify generality beyond the Llama- 3 family, we also conduct 20B-token continual pre-training from Qwen2-7B (Yang et al.) (024). where the model trained with SwallowCode surpasses Stack-Edu, improving HumanEval by +10.3 and HumanEval+ by + +10.3 (see Appendix B). We conduct rigorous decontamination checks for SwallowCode against HumanEval and HumanEval+ prompts (no exact matches or high-similarity documents; see Appendix H) and apply analogous procedures for SwallowMath against GSM8K and MATH (no contamination). We further examine potential seff-contamination from the rewriting LLM (Llama-3.3-700) -0B-Inuuttt)) using post-cutoff benchmarks (Appendix H). All datasets, prompts, and checkpoints are publicly released to ensure reproducibility and to enable adaptation to other base models and rewriting using LLMs.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 CLASSIFIER BASSIFFERR-BASE FILTERING FOR CODE CORPORA
Recent work has shown that basse filtering strategies, such the Web- ap- proach (Penedo et al. 2024), can be effective for curating high-quality web datasets (Allal et al. (024). These methods aim to improve model performance on code-related tasks by selecting semantically rich and well-docu samples from large-scale corpora. In this context, Stack-Edo represents significant effort to create a filtered variant of StarCoder2Data (Lozhkov et al.) 2024) prioritizing high-quality code. Stack-Edu begins by selecting the 15 most prevalent programming languages from StarCoder2Data, forming a subset of approximately 450 billion tokens. To assess code qual- ity. Stack-Edu leverages Jlama-3-70B 3-nn---stttt Instruct to generate synthetic annotations for 500,000 code fragments, rating each on a 0-5 scale based on educational and structural quality, These annotations train language-speciinn
classifiers based on the StarEncoder model (Li et al., (023). achieving F1 scores above 0.7 for most languages when applying a threshold of 3 for binary classification. The resulting dataset comprises 125 billion tokens and demonstrates improved model convergence and higher pass@ @ I scores on HumanEval compared to unfiltered corpora (Allal et al. (025). However, Stack-Edu's 's exclusionary filtering strategy discards low-scoring snippets rather than rewriting or aug- menting them, leaving residual issues such as missing context or inconsistent naming conventions in the retained data.
2.2 LLM-DRIVEN TRAININGG CORPUS REWRITING
Efforts to improve datasets using LLMs have gained traction. Cosmopedia (Ben Allal et al. 2024) demonstrated the potential of synthetic data generation, using Mixtral-8x7B-Instrue (Jiang et al.. 2024) to create high-quality text corpora, although it did not address code-specific challenges. Related techniques have also emerged in other domains, particularly for general web data refinement. For example, Rephrasing the Web (Maini et al. 2024 and Nemotron-CC. (Su et al. 2025) transform noisy Common Crawl text into improved corpora through large-scale rephrasing or QA-style conversions. These approaches, however, target non-code/math domains and often perform format changes that overlap with instruction-tuning objectives. In contrast, our method applies domain-specific rewriting while ensuring that the outputs remain pure code snippets, thereby preserving the original data's ' semantics. Unlike web data rephrasing approaches that introduce style conversions and may implicitly boost
abilities during pre- training, our approach is not designed to gain such capabilities but rather to improve the quality of pre-training data itself. This distinction makes our work fundamentally different from prior web data rephrasing efforts and highlights its unique contribution. MegaMath (Zhou et al. (025)) represents another recent attempt to refine math data, but adopts a QA-style interleaved text-code format. This design aligns more closely with general web data refinement methods that restructure content into instruction-response format, in contrast to our approach of retaining pure, self-contained snippets.
Jain et al. 2023) is closest to our work, applying LLM-driven code transformations to instruction tuning exemplars, e.g., variable renaming, modularization, and comment addition. While effective for fine-tuning, their approach is limited in scope and context compared our SwallowCode pipeline. Their transformations address only a subset of stylistic improvements, whereas our Style-Guided Code Rewriting (SGCR) pipeline comprehensively enforees ten criteria from the Google Pythor Style Guide, including descriptive variable naming, effective type annotations, modular function design, error handling, and feauueelity-oocused formatting (see Section 3.3.1 Furthermore, our Self- Contained Optimization Rewriting (SCOR) pipeline introduces semantic enhancements, ensuring self-contaimment by resolving dependencies, optimizing algorithms, and transforming trivial snippets into instructive examples, which are absent in Jain et al.'s work (see Section 3.3.2) Although Jain et al. target instruction-tume -tnninn data, a smaller and more curated dataset, Swallow Code systematically rewrites large-scale pre-training corpora. This is more challenging task due to the diversity and volume of code samples. Our pipeline integrates rigorous preprocessing with syntax error and linter-based filtering to ensure high-quality inputs for rewriting.
Taken together, SwallowCode's transform-an carrm-and-retain paradigm refines low low-quality code rather than discarding it, addressing limitations of existing datasets such The Stack v1/v2 and Stack-Edu, which rely on filtering. By combining filtering, SGCR, and SCOR, SwallowCode produces high- quality corpus that significantly improves performance on HylanEyal and HumanEyal+, advancing
3

---

Published as a conference paper at ICLR 2026
original
Stack v2 train smol
41,025,62 639 samples in the-stack-----taall -tat------ii---m---ieee (python), 36,1 tokens
Stage1
Syntax error filter
36,896,625 625 samples, 30.58 tokkns
Stage2
Linter filter
24,100,915 samples, tokans
Stage3
SOCR (Syle-Gunded)
reseito oods, ensuring adherence stylisde conventions
Stage4
Consainee)
Uses
source solFoontsteed optimized
Figure Four-stage pipeline constructing SodllowCode: (I) syntax filtering to remove invalid Python code, (2) linter-based filtering using pylint to enforce coding standards, and (3-4) two-stage LLM rewriting with Style-Guided Code Rewriting (SGCR), which enforces consistent style and readability, and Self-Contained Optimization Rewriting (SCOR), which ensures self-containment and optimizes algorithms for efficiency.
the state-of-the-art in open code datasets. Similarly, adhering to the same transform-and-retrin paradigm, Math rewrites noisy mathematical data, significantly improving performance on GSM8K and MATH.
2.3 SYNTHETIC DATA GENERATION FOR CODE
Recent approaches, such as Magpie (Xu et al. 2024), leverage high-performance LLMs to generate synthetic instruction-tuma tuning datasets from prompts and desired characteristics without relying isting data. However, we opted against generating synthetic code data from scratch for SwallowCode due to key limitations. First, previous work (Chen et al. 2024) demonstrates that low- low-diversity synthetic data restricts L.MM performance. Second, achieving high diversity synthetic code datasets requires diverse topics and keywords seeds, as seen in Nemotron-4 340B's synthetic instruction data (Nvidia et al., 2024). For code, defining such seeds (e.g., varied algorithmic paradigms or prob- lem domains) remains an open challenge, across programming tasks. Instead, our approach leverages high-qualiy, code from The-Stack-22 v22, filtered for syntactic and stylistic rigor (Section B.2). and applies LLM--diven rewriting enhance quality while preserving the inherent diversity of real-world code. This transform-and-retal strategy maximizes data utility and avoids the risks of synthetic data homogeneity.
established methodology ensures sufficient diversity
CONSTRUCTION OF THE CODE CORPUS
The development of SwallowCode is driven by an empirical and exploratory approach, informed by data ablation experiments evaluating each stage of the data processing pipeline, as illustrated in Figure 2 Specifically, we continually pre-trained the baseline model with the dataset, only differing in the code text subset: before and after applying specific stage in the pipeline, and decided if we should adopt or discard the stage based on the evaluation results. In this section, we present the experimental results and describe the design choices shaping the pipeline. Please refer to Appendix for detailed results.
3.1 EXPERIMENTAL SETUP
To evaluate the impact of each design choice in our pre-training data processing pipeline, we conduct systematic data ablation studies. Each ablation trains model that differs only in the target pre-training dataset, holding all other factors constant, including model architecture, parameter count, non-target data, total token budget, and hyperparameters. Specifically, we perform continual pre-training starting from Llama-3.1-8B using total of approximately 50 billion tokens. The target dataset is processed with less than one epoch within each ablation. We evaluated model checkpoints approximately every 10 billion tokens using ten downstream benchmarks.
Continual pre- mre-training is conducted using Megatron-LM (Shoeybi et al. 2020). For evaluation, evalplus (Liu et al.) (203) and wwweellationnaaneeess (Gao et al.) (024) on a suite of benchmarks, with individual tasks detailed in Appendix [ The effectiveness of the code corpus is specifically assessed using HumanEyal and HumanEval+. The pre-training data mixture consists of 84% multilingual text and 16% code. Detailed proportions and data source are provided in Appendix A.44I with detailed training hyperparameters reported in Appendix A.] All ablation models, associated checkpoints,
4

---

Published as a conference paper at ICLR 2026
HumanErv pasvid scone
2.30
HumaniE
2.30
230
Billion Tokeno
Billio Tokens
syntax Staring
Mitaring
scoring
Figure Performance comparison of filtering methods HumanEval (left) and HumanEval+ (right) bench- marks. The original dataset (Stack v2) is compared against datasets processed with syntax error filtering, linter-based filtering. and LLM-based scoring (evaluated in ablation studies but not adopted). Syntax and linter-based filtering enhance code generation performance, while LLM-based scoring provides marginal gains, considering the computational cost.
and supporting materials are publicly available in our Hugging Face repository to ensure full reproducibility.
3.2 FILTERING
To construct the SwallowCode corpus, high-quality Python code dataset, we implement rigorous filtering pipeline starting with
(Lozhkov al. 2024) as the base dataset. The filtering process is critical to ensure that only syntactically correct and well-structured code retained, enhancing downstream performance code generation tasks. We focus exclusively on Python code to maintain consistency across ablation studies and enable fair comparisons with existing public corpora. Our pipeline employs two key filtering techniques, syntax error filtering and linter-based filtering, that significantly improve code quality. We also evaluated LLM--baee' scoring in ablation experiments, but did not adopt it in the final pipeline due to its limited performance gains relative computational cost. Figure summarizes the performance of these methods on the HumanEval and HumanEval+ benchmarks. demonstrating the impact of our filtering strategy.
3.2. SYNTAX ERROR FILTERING
Despite the heuristic filtering in the BigCode project, code samples with invalid syntax according Python 3.10 specifications. To address this, apply syntax error filtering by compiling each code sample using Python's built-in compile () function, discarding any samples that fail compile. This process reduces the dataset from approximately 41 million to 37 million samples, 9.7% reduction. As shown in Figure 3 removing syntactically invalid samples improves the performance of HumanEval and HumanEval+ Consequently, adopt syntax error filtering as a standard step in all subsequent experiments.
includes Python
3.2.2 LINTER-BASED FILTERING
Beyond syntactic correctness, code quality depends on adherence to coding standards. Many samples in the initial dataset exhibit poor structure, generating numerous warnings when analyzed by static analysis tools. We employ pylint, a widely-used Python linter, to enforce a quality threshold, excluding samples with scores below 7.0 0-10 scale based on rule violations. Additionally, we penalize overly verbose comments using custom heuristic scoring algorithm (detailed in Appendix (C) This step reduces the dataset from 36.7 million to 24.1 million samples (34.3% reduction). Figure illustrates the performance gains of more than 1 point in HumanEyal and HumanEval+ achieved by
lowcode
httpa
wet.m//wallooe lowmath
"The performance trajectory of The Stack v2 exhibits initial decline followed by recovery, consistent
with forgetting and subsequent adaptation observed in prior continual pre-training studies (Fujii et al. 2024
This behavior is particularly notable.

---

Published as a conference paper at ICLR 2026
HumenErel parvit wnoIe
HumenEraat acorre
9.48
8.44
0:34
18
18
20
50
Infer Bitering
SBCR (loordd)
SGCR (direct)
SGCR SCOR
Figure 4: Performance comparison M-rrvveeen rewriting steps on HumanEval (left) and HumanFval+ (right) benchmarks. The pre-rewriting (syntax-erron and linter-based filtering) is compared against SGCR and SCOR. SGCR improves performance by over 7-9 points, while SCOR, applied after SGCR, further enhances scores by 5-6 points, demonstrating the effectiveness of stylistic and semantic rewrites in the SodelowCode pipeline.
linter-based filtering. We first remove syntax errors to avoid wasting computation on files that cannot be linted Consequently, we adopt linter-based filtering, with threshold of 7.0, standard step in all subsequent experiments.
3.2.3 LLM-BASED SCORE FIL.ERING
Recent approaches leverage LLM to generate synthetic annotations for training quality classi- fiers, which then used to filter web-scale corpora by retaining high-quality samples (e.g.. FineWeb (Penedo et al., 2024), Stack-Edu). Instead of training a separate classifier, we directly prompt Llama-3.3-70B -nstrrrr--
Instruct to evaluate each Python code snippet on scale of 0-10, based on ten criteria, including code readability, modularity, and adherence to naming conventions, derived from the Google Python Style Guide. The detailed scoring prompt and the distribution of the quality scores are provided in Appendix D.
We exclude samples scoring below 6, retaining only those deemed sufficiently high- high-quality, and use this filtered subset alongside multilingual data for continual pre- vre-training in our ablation studies. As shown Figure LLM-based filtering yields modest improvements (less than 1 point) over linter-based filtering on the HumanEval and HumanEval+ benchmarks.
Given these limited gains, we compare LLM--based scoring to our LLM- rewriting pipeline, which refines code snippets by enhancing clarity and correctness (Section 3.3) Comparing Figure and Figure 4 although the rewriting pipeline requires 1.22 times the computational resources of LLM- based scoring (a 22% increase), achieves significantly greater performance gains HumanEval and HumanEval+ (detailed in Appendix G). Consequently, we do not incorporate LLM--bsed scoring in subsequent experiments, which favors the more effective and cost-effective rewriting approach.
3.3 LLM-DRIVEN REWRITING
Recent studies highlight the potential of LLMs to transform training data, enhancing the performance of instruction-tuned models. Jain al. (2023) proposed three transformations variable renaming. modularization, and plan annotation for code cleaning, demonstrating their effectiveness for instruc- tion tuning. However, their approach is limited to curated instruction-tumne tunnin data and a narrow set of stylistic improvements, lacking semantic optimizations preprocessing integration.
To address these limitations, we propose the SwallowCode pipeline, which systematically rewrites large-scale pre-training corpora using two complementary LLM-driven processes: Style-Guided Code Rewriting (SGCR) and Self-Contained Optimization Rewriting (SCOR). SGCR revises Python code snippets based on the Google Python Style Guide, enforcing stylistic improvements like clear naming and modular design (Section 3.3.Ip. SCOR extends SGCR by ensuring welfeeeseetetieeeee containment and applying
'While pylint can also surface syntax errors, performing linter-only filtering would be computationally inefficient compared to staged pipeline of (i) syntax error filtering and (ii) linter-based filtering; the resulting retained set effectively the up linter version/config nuances.

---

Published as a conference paper at ICLR 2026
semantic optimizations, such as efficient algorithms and instructive examples (Section 3.3.2 To
clarify, SwallowCode retains the code-only format of The-stack-v2- www.--tttttnssooo----o consisting solely of Python snippets without the text-code pair structure typical of instruction-tuninn datasets. The rewriting pipeline, powered by semantic transformations without introducing instructional prompts responses. Thus, performance gains in HumanEval and Hvma (Section 3.4) stem from improved data quality, not from distillation of instruction tuning capabilities.
enhances code quality through stylistic and
To illustrate the scope of these transformations. Table compares the coverage of SGCR and SCOR against the approach of Jain al. (2023). While SGCR addresses stylistic criteria like type hints, error handling, and docstrings, SCOR introduces semantic enhancements, including self ceff-containment and optimization (algorithm and data structure), broadening the scope of the SwallowCode pipeline.
Table 1: Comparison of code transformation coverage for Jain al. 2023). SGCR, and SCOR.
Criterion
Jain et al.
SGCR
SCOR
Variable Renaming
Modularization
Comments
X
Type Hint
X
X
Error Handling
Docstring
Self-Contained
Optimized
X V
3.3.1 SGCR:
CODE REWRITING
SGCR enhances code readability by adding docstrings and type hints, unifying variable reassignment patterns, and standardizing function and class names in accordance with the Google Python Style Guide. Compared to the pre-rewriting (syntax-error and linter-based filtering), SGCR achieves improvements over 7-9 points HumanEval and HumanEval+ shown Figure 4 We also evaluate SGCR applied directly to the raw the-stack-v2 v2-k---2ttrinnnn ttrin-smoll syntax eπror and linter- based filtering. As illustrated in Figure | the SGCR pipeline with syntax and linter filtering outperforms direct SGCR by 0.4-2.1 points on downstream code generation benchmarks. Consequently, adopt pipeline that applies syntax and linter-based filtering prior SGCR in all subsequent experiments.
corpus versus SGCR applied after
Ablation studies reveal that SGCR significantly improves HumanEval and HumanEval+ scores but results in an approximate 10-point decrease on the MBPP benchmark (Austin et al.) (02T)) (detailed in Appendix IT.. Analysis indicates that MBPP's solutions often use non ntn-standard function and class names, and SGCR's 's automated renaming introduces function name mismatches with MBPP's s unit tests, leading to "undefined" errors during evaluation. The identified mismatches obscure the model's true code- oode-generation capabilities, motivating to exclude MBPP from our evaluation benchmarks across all experiments.
3.3.2 SCOR: COLF-CONTAINED OPTIMIZATION REWRITING
Although SGCR ensures adherence to stylistic criteria, it does not modify the program semantics. Manual observation of models trained on SGCR processed data reveals three recurring issues: (i) missing dependencies, where models attempt to import non-existent libraries or call undefined functions, causing runtime errors; (ii) inefficient algorithms, such as naive recursion or quadratic-time algorithms for problems that admit linear-time or dynamic programming solutions; and (iii) trivial snippets, such as code that merely prints constants or performs basic arithmetic, offering minimal training value.
SGCR-processed
To address these limitations while preserving SGCR's stylistic improvements, we introduce Self- Contained Optimization Rewriting (SCOR). Guided by a ten-rule prompt (detailed in Appendix E.35. SCOR rewrites each snippet ensure ceff-containment by inlining satisfying external dependencies, replaces inefficient algorithms with more computationally efficient alternatives, and transforms trivial code into meaningful executable examples. As illustrated in Figure SCOR improves HvmanEval
7

---

Published as a conference paper at ICLR 2026
and HumanEval+ scores by over 5-6 points compared SGCR. These results underscore the importance of semantic-level rewrites beyond stylistic enhancements, establishing SCOR as the final stage of the SwallowCode construction pipeline. We did not conduct an ablation experiment evaluating SCOR in isolation without SGCR. However, prompt validation experiments with Llama- 3.3-70B-Instrun Instructttt indicated that simultaneously applying SGCR and SCOR often reduced the quality of the rewritten code due to the challenges LLMs face in balancing multiple objectives, This led to the adoption of two-stage SGCR and SCOR approach, with the isolated SCOR evaluation deferred to future work.
Compared to the approximately 1-2 points performance gain achieved during the filtering stages (green line in Figure 4) the rewriting stages using SGCR and SCOR led to a total performance improvement of 14 points points from SGCR and 5-6 points from SCOR. This highlights the significant potential for enhancing dataset curation by incorporating LLM--riven rewriting approach.
3.4 THE FINAL SWALLOWCODE DATASET
Applying the complete pipeline, including syntax error filtering, Pylint-based filtering, Style-Guided Code Rewriting (SGCR), and Self-Contained Optimization Rewriting (SCOR), to the-stack-v2- train-smol-ids produces the Swallow Code corpus, comprising 16.1 billion tokens. All intermediate artifacts, including non-optimal variants, are publicly available to support future research efforts.
Comparison with existing corpora Figure T compares SwallowCode with several widely used open code datasets: CodeParrot-Clean (12.8 billion tokens), The Stack v1 (98.2 billion tokens) The Stack v2-Smol (36.1 1 billion tokens), and Stack-Fdu (17.9 billion tokens). For fair comparison, we extract only the Python subsets of each corpus and, following the protocol outlined in Section 3.1 allocate 16% (8 billion) code tokens within 50 billion-tokes mixed batch, ensuring each sample processed no more than once. SwallowCode outperforms all comparable publicly available corpora on the HumanEval (pass @1 1) and HumanEval+ (pass @ 1) benchmarks, demonstrating the effectiveness of our pipeline design. Detailed results are provided in Appendix J.1
4 CONSTRUCTION OF THE MATH CORPUS
Section 3 demonstrated that LLM-driven rewriting significantly boosts coding performance. To evaluate the transferability of this approach, we apply tailored rewriting pipeline to mathematical data. We select finemath-4+, high-quality, publicly available math corpus, as the starting point and process through a rewriting pipeline. According to the evaluation in finemath, it outperforms other open mathematical datasets, including Open WebMath (Paster et al. 2023), al. 2024), and finemath-++, in terms of the performance on benchmarks such as GSM8K and MATH. Given its reported superior performance and public availability, we adopt finemath 4+ as the foundation corpus for constructing our mathematical corpus to maximize the effectiveness of our rewriting approach.
(Han
4.1 EXPERIMENTAL SETUP
We adhere to the protocol outlined in Section 3.1 performing continual pre-training of Llama-3. 1-BB for approximately 50 billion tokens, varying only the target math corpus. The evaluation benchmarks mirror those of Section 3.1 with HumanEval+ replaced by the MATH dataset (Hendrycks et al. 2021b), with GSMSK and MATH as the primary math-focused benchmarks. The pre-training mixture comprises 82.2% multilingual text, 13.0% code, and 4.79% math; detailed proportions and data sources are provided in Appendix A.4.2 The complete hyperparameters are listed in Appendix A.
4.2 I.L.-DRVVEN REWRITING
The Finemath-44 4+ corpus is collection of documents in which snippets of mathematical text embedded within passages that are otherwise unrelated to mathematics. In addition, the mathematical content ranges widely, from elementary arithmetic advanced topics. This heterogeneity renders rule-based filtering challenging, as it struggles to distinguish relevant mathematical content from
8

---

Published as a conference paper at ICLR 2026
0.000
0.000
8.80
08M8K Soore
MATTH
0.00
0.76
0.45
0.100
Billion Tokers
Billinn Tokenn
GOMES Eranui---
MATII AeTiirraaalyn
COMBX Ircndciii sywling
souriing
Figure 5: Performance gains from LLMM-diven rewriting of finemath-4+. The rewritten corpus improves GSM8K(left) by 12.4 points and MATH(rightt by 7.6 points.
irrelevant artifacts. To address this, we design an LLM-driven rewriting pipeline using Llama-3.3. 70B-Instruce which not only cleans and refines the data but also enhances its quality for mathematical reasoning tasks.
The rewriting prompt instructs the model to: (1) remove residual web headers, footers, and privacy notices; (2) eliminate irrelevant metadata, such as question and answer timestamps; (3) restore missing context in incomplete questions or answers; (4) rewrite derivation steps be concise yet comprehensive; and (5) provide clear, step-by-step solutions. Steps (1) and (2) are analogous to the syntax error and linter-based filtering applied to SwallowCodk (Section 3 addressing inappropriate content that rule-based methods alone could not effectively filter. Steps (3) through (5) parallel the contannttan and style enhancements of the code rewriting pipeline, adapting them to the mathematical domain. The complete prompt is provided in Appendix
As shown in Figure 5 the rewritten corpus yields substantial improvements: 12.4 points on GSMSK and 7.6 points MÄTH. These results demonstrate that L.MM-driven rewriting, while tailored the unique characteristics of mathematical data, successfully enhances the already high-quality finemath- 4+ corpus. This confirms the generalizability of our rewriting approach beyond code, offering robust method for improving open-domain datasets for mathematical reasoning.
5 LIMITATIONS
While SwallowCode and SwallowMath significantly improve code generation and mathematical reasoning performance of LLMs, several limitations should be noted. First, the rewriting pipelines may preserve biases present in the source data. For example, the Stack v2 may over-represent certain coding patterns, Finemath-4+ may favor specific problem types. Additionally, as the rewriting process relies on such as favoring certain variable naming conventions or solution strategies. Second, our evaluations confined to continual pre-training with 50 billion token budget, detailed in Section 3.1 to ensure controlled and reproducible experiments within computational constraints. The impact of extending pre- trairiin beyond this budget remains unexplored, and performance trends at larger scales may differ, particularly for tasks requiring extensive training data. Third, although the SwallowCode pipeline is designed be language-agnostic, requiring only static syntax checking and linter tools, our experiments focus exclusively on Python to facilitate automated evaluation. Empirical validation for other programming languages was feasible due to resource constraints, limiting evidence of the pipeline's 's broader applicability.
the resulting datasets may reflect this model's preferences,
6 CONCLUSION
We introduced SwallowCode and SwallowMath, which are openly released pre-training corpora built using transform-ant and-raiin rewriting pipeline. Beyond filtering, our approach normalizes style and structure and produces self-contained, semantically improved snippets. Under fixed compute budget, this yields consistent pre-training gains on code and math benchmarks. Ablations isolate where improvements arise, providing actionable data-design principles rather than model-specific

---

Published as a conference paper at ICLR 2026
tricks. Our scope is data-centric and evaluates pre-training effects without SFT or RL to avoid stage conflation. The main limitations are the current focus on Python for code and the computing required for rewriting. As analyzed in Appendix G.4, the upfront cost of rewriting is offset by higher data efficiency during training. Rewritten corpora deliver larger gains with the same or fewer tokens, while noisy baselines saturate and would require substantially more tokens approach the same accuracy, if at all possible. The pipeline is modular and scalable, allowing researchers to rewrite only high-leverage subsets or directly utilize our released corpora avoid the rewrite overhead. Because the refined datasets are reusable across models and studies, the initial cost becomes durable community asset. We release the data, prompts, and checkpoints enable reproduction and future updates as rewriting LLMs improve.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
We affirm adherence to the ICLR Code of Ethics. This work introduces and releases SwallowCode and SwallowMMath, billion-token-saal corpora focused on source code and mathematical reasoning. The datasets do not involve human subjects or interventions, and are not designed to encode or amplify sensitive attributes.
Data were curated from publicly accessible sources with licenses that permit redistribution and commercial use; we preserved license information and require users to respect the original terms. To the best of our knowledge, the datasets do not contain personally identifiable information or secrets. However, residual risks remain (e.g., insecure coding patterns or biased stylistic conventions present in public code).
We maintain an open feedback and takedown process. If any privacy, licensing, security, or other ethical concerns are identified, stakeholders can contact the authors via the issues page our Hugging Face repositories; we will review and remediate reports promptly (including correction or removal of problematic data). We not aware of conflicts of interest related to this release.
REPRODUCIBILITY STATEMENT
We are committed to reproducible research. Complete experimental details provided in Ap- pendix A including (i) model architectures used for training, (ii) all hyperparameters, (iii) GPU resources, (iv) distributed training configurations, (v) library and framework versions, and (vi) composition the training data.
We release the dataset pipeline code used to construct corpora In addition, the methodology and implementation details for the data pipeline are documented in [ppendices[C] D and F The evaluation datasets and framework settings are descrihed in Section 3.1 and 4.1 with further specifics in Appendix]] !
We make public all model checkpoints produced in ablation studies and the full training corpora on Hugging Face 5
ACKNOWLEDGMENTS
This work was supported by JST K Program Japan Grant Number JPMJKP24C3. This work used computational resources of the TSUBAME4.0 supercomputer provided by Institute of Science Tokyo through the HPCI System Research Project (Project ID: hp250181) and Joint Usage of TSUBAME Partnership Resource Allocations Program.
lowcode
httpp///gggnnee..oooooeeerrorror oowmah

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat S. Behl, Séhastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi. Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price. Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang. Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL https: wwt. org 10. 485500/r.111....
0012.08905
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíčck, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL http::: //apiiaaaiiv/a//aa//iii org/ abs/ abwaabs////22 200./5502..00 02737
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL hattts:///axiy/ owapiiiorrgaaaa bttgg/ab//ao 2108 07732
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, February 2024. URL
Software dataset.
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj. and Marah I, Abdin.
On the diversity of synthetic data and its impact on training large language models, 2024. URL
https::/aaxi:> www./bs//////ooo...../....a....a 2410 15226
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL StRL:hhtps://eee
wwwp/r/gg//b/..eeeeeeeeee 03374
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
DeepSeck-AL, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen. S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu. Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha
12

---

Published as a conference paper at ICLR 2026
Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma. Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. wtt.:htts://aaxiiiii /ttp://aaxiv.rrr//e abs///////// 2412. 1e.1194. 19437
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross-lingual Ilm adaptation: Enhancing japanese language capabilities, 2024. URL U wttp/htts///ariii/
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 07 2024. /tthts::: wwnooo/ org/ http://ecoooooooeeeeeee
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz. Danny Livshits. Danny Wyatt, David Esiobu, Dhruv Choudbary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lec, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, mmanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathiuu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si. Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro. Roshan Sumbaly, Ross Taylor. Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier
Plawiak,
13

---

Published as a conference paper at ICLR 2026
Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaclle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong. Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence. Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez. Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi. Zhang. Guna Lakshminarayanan. Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard. Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani. Pritish Yuvraj. Qian Liang. Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian WU, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The 1lama 3 herd of models, 2024. htRhhttps:// www.aaaivvv........a.a.a org wttp://///a////////a 21783
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yigi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. Infimm- 40mmm-webmath-400 Advancing multi-
14

---

Published as a conference paper at ICLR 2026
modal pre-training for enhanced mathematical reasoning, 2024. URL httpp::/aaxxx..orggg/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https:/ //
sxxiv.org// abs/ 22009.009..000 03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021b.
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez. Koushik Sen, and Ion Stoica. Lim-assisted code cleaning for training accurate code generators, 2023. URL htttps://arxxt
org/ ha///bb//22 2311. 14904
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https:tt//// http::://axxvvv.o abs /4001 04088
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022. URL 15533
https://aat //axxvv www.aawwwwwowoo.ooo.aoreoae.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mereat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang. Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevie, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartassh Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: search of the next generation of training sets for language models, 2025, BRRt::ts:://ara
org abs 2406 11794
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Iims as mathematical problem solvers, 2024. URL hhttps://arx/e / abs 2402. 9255
Raymond Li, Loubna Ben Allal, Yangtian Zi. Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltomozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz. Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023. URL https: wwt.hhttst//ttttte /ttps:///axxv..o org/ abs./305... 06161
15

---

Published as a conference paper at ICLR 2026
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openrevie..net net/ 1 OCu7
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii. Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary. Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL wtw.tttt://t/ /tttp:://arxi..orrte wwwp:xiiwiorrroooooooo org/ a abs 2202 19173
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang. and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024. URL https
7/arxiv. wwt.://gg/abbaaoooooo abs/ 2401. 6380
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018
Deepak Narayanan, Mohammad Shoeybi, Jared Casper. Patrick LeGresley, Mostofa Patwary, Vi- jay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im, 2021. URL https:////// org/ / 2104. 04473
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das. Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabalesh- warkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi. Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary. Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy. Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian. Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng. Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. - 340b technical report, 2024. URL https: /ttp://axvvvvvv. / abs /20//44060. 11704
NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkarr Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Brian Butterfield, Bryan Catanzaro, Carlo del Mundo, Chengyu Dong, Christine Harvey, Christopher Parisien, Dan Su, Daniel Korzekwa, Danny Yin, Daria Gitman, David Mosallanezhad. Deepak Narayanan, Denys Fridman. Dima Rekesh, Ding Ma. Dmytro Pykhtar, Dong Ahn, Dun- can Riach, Dusan Stosic, Eileen Long, Elad Segal, Ellie Evans, Eric Chung, Erick Galinkin, Evelina Bakhturina, Ewa Dobrowoiska, Fei Jia, Fuxiao Liu, Gargi Prasad, Gerald Shen, Guilin Liu, Guo Chen, Haifeng Qian, Helen Ngo, Hongbin Liu, Hui Li, Igor Gitman, Ilia Karmanov, Ivan Moshkov, Izik Golan, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jarno Seppanen, Ja- son Lu, Jason Sewall, Jiaqi Zeng, Jiaxuan You, Jimmy Zhang, Jing Zhang, Jining Huang, Jinze Xue. Jocelyn Huang, Joey Conway, John Kamalu, Jon Barker, Jonathan Cohen, Joseph Jennings, Jupinder Parmar, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Luna, Keshav Santhanam, Kezhi Kong, Kirthi Sivamani, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Lawrence
16

---

Published as a conference paper at ICLR 2026
McAfee, Leon Derczynski, Lindsey Pavao, Luis Vega, Lukas Voegtle, Maciej Bala, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Markus Kliegl, Marta Stepniewska- Dziubinska, Matthieu Le, Matvei Novikov, Mehrzad Samadi, Michael Andersch, Michael Evans, Miguel Martinez, Mike Chrzanowski, Mike Ranzinger, Mikolaj Blaz, Misha Smelyanskiy, Mo- hamed Fawzy, Mohammad Shoeybi, Mostofa Patwary, Nayeon Lee, Nima Tajbakhsh, Ning Xu, Oleg Rybakov, Oleksii Kuchaiev, Olivier Delalleau, Osvald Nitski, Parth Chadha, Pasha Shamis, Paulius Micikevicius, Pavlo Molchanov, Peter Dykas, Philipp Fischer, Pierre-Yves Aquilanti, Piotr Bialecki, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi, Rahul Kandu, Ran EI- Yaniv, Raviraj Joshi, Roger Waleffe, Ruoxi Zhang, Sabrina Kavanaugh, Sahil Jain, Samuel Kriman, Sangkug Lym, Sanjeev Satheesh, Saurav Muralidharan, Sean Narenthi- ran, Selvaraj Anandaraj, Seonmyeong Bak, Sergey Kashirsky, Seungju Han, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Sharon Clay, Shelby Thomas, Shrimai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shyamala Prayaga, Siddhartha Jain, Sirshak Das, Slawek Kierat, Somshubra Majumdar, Song Han, Soumye Singhal, Sriharsha Niverty, Stefania Alborghetti, Suseella Panguluri, Swetha Bhendigeri, Syeda Nahida Akter, Szymon Migacz, Tal Shiri, Terry Kong, Timo Roman, Tomer Ronen, Trisha Saar, Tugrul Konuk, Tuomas Rintamaki, Tyler Poon, Ushnish De. Vahid Noroozi, Varun Singh, Vijay Korthikanti, Vitaly Kurin, Wasi Uddin Ahmad, Wei Du, Wei Ping. Wenliang Dai, Wonmin Byeon, Xiaowei Ren, Yao Xu, Yejin Choi, Yian Zhang, Ying Lin, Yoshi Suhara, Zhiding Yu, Zhiqi Li, Zbiyu Li, Zhongbo Zhu, Zhuolin Yang, and Zijia Chen. Nemotron-h: family of accurate and efficient hybrid
models, 2025.
URL battps://ariii org/ abs/ 2504 03624
Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki lida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. Building large japanese web corpus for large language models, 2024. URL /ttps:///xxxii/ httpa/awwvvv.rrro...aaaa abs 2404. 10004.1770 17733
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafford, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi. Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zeulemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo furious, 2025. URL
wwtp://..rrraa///////a/a 22501.1. 00656
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. httl//ttpss//rrxi.oog/////
06786
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https: /ttp:///xxiiiii org/ abs 211.2246..7777 1757
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018
Mohammad Shocybi. Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billioo parameter language models using model parallelism, 2020. httpsttps://// httpvv.xxvvvv////////////a.aa. org/ aa////b//// 10..199999.0030
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into a refined long-horizon pretraining dataset, 2025. URL
02595
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Lc, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether tf-touhh can solve them. arXiv preprint 10.0022210.09261. 2022.
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as a bascline for cross-lingual transfer in commonsense reasoning, 2021.
17

---

Published as a conference paper at ICLR 2026
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. 2024. htttp:/::///ax
org/ abs 22w..2406.... 08464
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL Ittpps:/Dan //axiiv//rxvvv org/ abs 2 2407. 10671
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httph:ttt////tt /tap:s://arxxae org/ abs 2505 09388
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Fan Zhou, Zengzhi Wang, Nikhil Ranjan. Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric Xing. Megamath: Pushing the limits of open math corpora, 2025. URL https
7 7/a rx rxiv,,
02807
18

---

Published as a conference paper at ICLR 2026
A DETAILED SETUP FOR DATA ABLATION EXPERIMENTS
This section provides details on the training hyperparameters, training library versions, training environments, distributed training settings, and data mixture used in the dataset ablation experiments described in Sections and
A.1 TRAINING HYPERPARAMETERS
We performed continual pre-traine pre-training from Llama-3. 3.1 using approximately 50 billion tokens. As shown in Table 2 the model architecture and tokenizer are identical to those of Llama-3.1-8B. The training hyperparameters are detailed in Table
Jusing
Table 2: Training model's architecture
Hyperparameter
Value
Architecture
Llama-3
Hidden size
4 096
FFN hidden size
14.36
Number of layers
32
Number of attention heads
32
Number of key/value heads
8
Sequence length
8 192
Normalization
RMSNorm
RMSNorm epsilon
1.0 10-5
RoPE base
500000
Attention dropout
0.0
Hidden dropout
0.0
Tokenizer
Llama-3 tokenizer
Table 3: Training hyperparameters
Hyperparameter
Value
Adam betal
0.9
Adam beta2
0.95
Adam epsilon
1.0 x 10-8
Gradient clipping
1.0
Weight Decay
0.1
Learning rate (max)
2.5 x 10-5
Learning rate (min)
2.5 x 10-6
Warmup steps
1000
Warmup style
linear
Decay style
cosine
A.2 TRAINING ENVIRONMENT
We utilized the H100 supercomputer for training. We utilized mixed precision (bfloat and employed multiple NVIDIA H100 nodes for distributed parallel training. Each node is equipped with four NVIDIA H100 94GB GPUs, and the nodes are interconnected via InfiniBand NDR200.
We conducted continual pre-training with libraries shown in Table
A.3 DISTRIBUTED TRAINING SETTINGS
Training LLMs on single GPU is challenging due to both GPU memory constraints and the time required for training. In terms GPU memory, even with the latest H100 80GB, training the 8B
https::/huggiingfaae co wwwo//etaaaaee 11maa ama
19

---

Published as a conference paper at ICLR 2026
Table 4: Training library versions
Component / Library
Version
Training library
Megatron-LM
mcore
0.9.0
CUDA Toolkit
12.4
cuDNN
9.1.0
NCCL
2.21.5
HPC-X
2.17.1
ninja
1.11.1
PyTorch
2.5.0
TransformerEngine
1.12
model used in this study is challenging. Moreover, even the model parameters, gradients, and optimizer states could fit on single GPU, training on single GPU would require an unrealistic amount of time to complete. Therefore, in this study, we adopted distributed parallel training, combining data parallelism and model parallelism. We conducted all ablation experiments with the distributed setting shown Table 5
Table 5: Distributed training setting for ablation experiments
Hyperparameter
Value
Data Parallelism (DP)
32
Tensor Parallelism (TP)
Context Parallelism (CP)
Pipeline Parallelism (PP)
Micro batch size
2
Global batch size
512
Sequence Parallelism
true
Distributed optimizer
true
Tensor Parallelism Communication Overlap
true
A.4 DATA MIXTURE FOR ABLATION EXPERIMENTS
Our research project aims to develop open-source LLMSs with strong capabilities in both Japanese and English. A key challenge in continual pre-training from high-performing models like L-ama-331-8BB Qwen-3 is mitigating catastrophic forgetting, particularly in maintaining or improving mathematical reasoning and code generation performance. To address this, our ablation experiments were designed to improve Llama-3.1-8B* performance on HumanEval, HumanEval+, GSM8K, and MATH while incorporating multilingual datasets predominantly composed Japanese and English text. This approach led to the development of Code and SwallowMath, as detailed in Sections 3 and 4 The data mixture reflects high proportion of Japanese text, consistent with our project's 8 focus on bilingual proficiency. However, described in Sections 3.1 and 4.1 all ablation experiments maintain identical settings except for the target code or math dataset, ensuring a fully controlled experimental design. To explore the impact of the high Japanese text proportion in our ablation mixtures on English-centie benchmarks, we conducted additional experiments with an English-heayy alternative mixture; see Appendix A.. for details.
This continual pre-training strategy aligns with established practices in high-quality LLM develop- ment, as seen models like OL.Mo-2 (OLMo al. 2025) and Nemotron- (NVIDIA et al. 2025). Specifically, adopting a staged pre-training approach, as observed in OLMo-2 and Nemotron-H, where later training phases leverage high-quality multilingual text alongside specialized math and code datasets to enhance LLM capabilities, ensures that our ablation experiments align with realistic LLM training scenarios.
20

---

Published as a conference paper at ICLR 2026
1.7%
Corpora
japanese Wikipedia (0.8dR)
japanese Swallow Corpus (26.18)
ParaCorpus (0.228)
20.0%
English wikipedioo
Casmopodia
7.4%
2.2%
Figure 6: Data ratio for SwallowCode ablation experiments.
A.4.1 CODE ABLATION DATA MIXTURE
The training dataset for the code ablation experiments comprises approximately 50 billion tokens. The distribution of the dataset components is illustrated in Figure 6 with the following components and their respective token counts, Note that the Target Code Data varies depending on the specific ablation experiment conducted.
Japanese Wikipedia
0.84 billion tokens
Japanese Swallow Corpus v2 (Okazaki et al. 2024): 26.1 billion tokens
Laboro-ParaCorpus 0.22 billion tokens
English Wikipedia 10 1.1 billion tokens
English Cosmopedia [en Allal et al. 2024): 3.7 billion tokens
English DCLM (Li al. 2025): 10.0 billion tokens
Target Code Data: 8.0 billion tokens
A.4.2 MATI ABLATION DATA MIXTURE
4.8%
1.7%
Corpora
Japanese Wikipedla
japanese
Willpedia
6.7%
2.2%
66.6%
Figure 7: Data ratio for MwallowMath ablation experiments.
The training dataset for the math ablation experiments also consists of approximately 50 billion tokens. The composition of the datasct is shown in Figure 7 with the following components and their
htteae///hhubb.co//bbrra////// |ttp://abbroooaaaaoooooo ParaCorpus
21

---

Published as a conference paper at ICLR 2026
respective token counts, Note that the Target Math Data varies depending on the specific ablation experiment conducted.
Japanese Wikipedia: 0.84 billion tokens
Japanese Swallow Corpus v2: 33.0 billion tokens
Laboro-ParaCorpus 0.22 billion tokens
English Wikipedia: 1.1 billion tokens
English Cosmopedia: 3.3 billion tokens
English DCLM: 2.2 billion tokens
SwallowCode (Syntax, Pylint Filtered): 6.5 billion tokens
Target Math Data: 2.4 billion tokens
A.5 IMPACT OF LANGUAGE MIXTURE ON ABLATION EXPERIMENTS
Our ablation experiments utilize data mixture with high proportion of Japanese text, which is controlled for fair comparisons but represents an unconventional setup for English-centric benchmarks like HvmanEval. To investigate potential subtle effects on learning from the code and math portions, we performed additional experiments using an alternative mixture: 70% English text (sourced from DataComp- and Cosmopedia), 20% code (SwallowCode) and 10% math (SwallowMath). The experimental setup otherwise follows that described in Section [31 with continual pre-training of Llama-3.1 1-B for 50B tokens. The results are summarized in Ta
Table 6: Performance on Key Benchmarks with English-Heavy Mixture
Benchmark GSM8K
MATH
HumanEval
HumanEval+
Accuracy
0.700
0.354
0.583
0.540
These scores are higher English-centric math and code benchmarks compared original Japanese-heary mixture, suggesting that a high Japanese proportion may diminish performance on such tasks within a limited 50B-token budget, possibly due reduced exposure English- aligned patterns. However, since the original experiments in Sections 3.1 and 4.1 are controlled, the improvements of SwallowCode and SwallowMath remain robust and unaffected by the composition of the mixture.
B GENERALIZABILITT BEYOND LLAMA-3: QWEN2-7B
To assess generalizability beyond the Llama-3 family, we conduct 20B-token continual pre-training starting from Qwen2-7B. We use the final SwallowCode (post-SCOR) as the code component and adopt an alternative mixture of 70% English text (DataCCmp-LLM and Cosmopedia). 20% code, and 10% math (SwallowMatth, Unless otherwise noted, all hyperparameters and evaluation benchmarks follow the protocol in Section 3.1 and Appendix J except that we use LR=1,0E-5, global batch size of 1,024, and sequence length of 4,096. As a code baseline, we replace SwallowCode with Stack-Edu under the same token budget and mixture ratios. Compared to Stack-Edu, SwallowCode yields +10.3 pass @ 1 on HumanEval and + +10.3 pass I on HumanEval+, reaching 49.6 and 44.6 respectively (Table 7. These results indicate that the benefits of our transform-and-reaain pipeline not specific to I.
models.
Table 7: Qwen2-7B, 20B-token continual pre-training. Scores are pass@) 1,
Corpus
HumanEval
HumanEval+
Stack-Edu
39.3
34.3
SwallowCode (ours)
49.6 (+10.3)
44.6 (+10.3)
22

---

Published as a conference paper at ICLR 2026
C CODE LINTING FILTERING
threshold 7.0 balances code quality and dataset size, while the comment penalty reduces overly verbose or non-functional scripts. As described in Section 3.2.2 the code for performing linter filtering is publicly available. The relevant code is also provided below.
When pylint applied, warnings and errors dependent the linting environment, such import errors, are excluded using the --disable option. Additionally, some files with Python extensions contain primarily comments with textual content and have minimal script functionality, To address this, as discussed in Section 3.2.2] we introduced a mechanism that imposes a heuristic penalty based on the proportion of comments filter out such files.
def beacc__commnt__ratae ratiraatiiiiiiiiio (code: str):
total_lines
comment_linge
try:
tckens totenize.generate. in tokens:
(StringI0 (code) readlllllin inn)
for token_type,
Sstal_lines += 1
it token_ type = tckenize. COMMENT:
comment_lines +-
12
except tokenize.TokenBrros as e:
13
print return 0
(E"Token (E"Token error encountered: (str )*)
except IndentationError
e:
print (f" indentation error encountered (str return
17
if total lines == 0:
21
22
return comment lines / total lines
23
def soply__commen__eenlle (score: fleat, comment_ratio: float) float: if comment_ratio 1.0:
26
27
return 0.0
elif comment_ratis > 0:
penalty_factor
comment_ratio
score fonallt_factto.
31
return score
32
def check_code_oooee ccode_qullttyyeeeeeoocccooo (code: str):
with tempfile, Named/emporrrrr le (delete=Ralse, suffix=".py") as temp_file:
temp file. wile.write write (code.encode)
temp_ file. (l
38
39
result subprocess run
40
["pylint", *--persisttent=",, "-- -=d0iaabb-00000000000001,01011,1 c0103, c0116, c0411, 10511,c0412" semp_file.bn capture__utputt-t1 text=True,
42
43
pylint_output result.stdout score None
46
47
48
for 1ine in pyl pylint_output. ant__ouppu.ssplitee. ("An"):
if "Your code has been rated at" in line:
score float (line. ("/"t [0]. sppitp [-1])
23

---

Published as a conference paper at ICLR 2026
comment_ratio check_comeen___aaiilcooodo
53
54
if score is not None:
score apply_comment__enaltyy (scor,,,, comment_ratic)
return score, pylint_outpur 57
D CODE LLM-BASED SCORING
As described Section 3.2.3 this section presents the prompt used for LLM-based scoring. The prompt was provided to 1iiia-..--0--Imm-s developed with reference to the Google Python Style Guide
to evaluate code quality. The scoring criteria were
Prompt Used for Code Quality Evaluation
You are smart software engineer. Please evaluate the following code scale of 1 to based the following criteria:
Are variable names descriptive conventions?
consistent with naming
2. Are comments and docstrings appropriately written to explain the purpose and functionality of the code?
Are type annotations used effectively where applicable?
4. Are functions appropriately modularized, with well-defined responsibilities and clear separation of concerns?
5. Are variables' lifetimes intentionally managed, avoiding
frequent reassignment implemented appropriate where necessary?
overly long scopes?
Is error handling impl
7. 1s the code properly indented and follows standard formatting guidelines?
8. Do comments provide context and rationale, rather than merely describing what the code does?
9. Are functions and classes designed with clear, single responsibilities?
10. Is the code formatted in a way that enhances roadability?
E CODE LLM REWRITING
As described in Section 3.3 this section presents the prompts used for LLM-based rewriting, specifi- cally for Style-Guided Code Rewriting (SGCR) and Self-Contained Optimization Rewriting (SCOR). Each prompt provided to Llama-3.3-70B-Insttee to perform data rewriting.
E. TOKEN RETENTION ANALYSIS
The rewriting processes in SGCR and SCOR inherently modify the token lengths of data samples. To provide insights into these transformations and their implications for training efficiency, we computed the average input and output token lengths across the dataset, as summarized in Table
Table 8: Average Input and Output Token Lengths for Rewriting Methods
Rewriting Method
Input Tokens
Output Tokens
SGCR SCOR
836
548
548
835
For SGCR, output lengths are shorter than input lengths on average due the pipeline discarding samples where the original code combined with the prompt and generated text exceeds the model's
whttps:// 10/ htt.b//..bbbtttllggieee//ggeeee. html
24

---

Published as a conference paper at ICLR 2026
maximum context length, where generated code cannot be reliably extracted (e.g.. absence of a code block like "python) Our analysis shows that SGCR generally condenses the data. In contrast, SCOR expands the data.
E.2 SYNTAX ERROR ANALYSIS
While LLM rewriting can potentially introduce syntax errors or other implementation issues, con- structing a dataset at the billion-tokes scale renders infeasible detect and remove all such anomalies comprehensively. To provide insights into the extent of code eπrоr accumulation during the LLM rewriting process, we measured the syntax error rates after each rewriting stage on a random subset of 100,000 samples, finding rates of 0.73% post-SGCR and 0.46% post-SCOR. These results indicate that errors do not accumulate across the two-stage rewriting process, alleviating concerns about progressive degradation. Furthermore, incorporating syntax error filtering each rewriting step holds promise for enhancing dataset quality future works.
E.3 RATIONALE FOR THE TWO-STAGE REWRITING PIPELINE
natural question is whether the SGCR and SCOR stages could be merged into single rewriting pass. In preliminary experiments, we combined all instructions from both prompts into unified prompt containing 19 distinct directives. Manual inspection of the outputs revealed that Llama-3.3- 70B- (OB-Instruct frequently exhibited instruction drift: the model complied with only a subset of the directives while neglecting others. For example, when style-related and self-comtainment instructions were presented together, the model tended to prioritize structural changes (e.g., making the code self-contained) at the expense of stylistic refinements (e.g., docstrings, type annotations), or vice versa.
To ensure robust adherence to both stylistic and semantic requirements, we adopted the decoupled two-stage pipeline described in Section 3.3 This decomposition allows each stage to target a focused set of objectives, reducing the cognitive load on the rewriting model and yielding more consistent outputs.
E.4 STYLE- GUIDDE CODE REWRITING (SGCR)
The prompt used for SGCR is provided below.
Prompt Used for SGCR
smart software engineer. Please evaluate the following code on a scale of 1 to 10 based on the following criteria:
1. Are variable names descriptive and consistent with naming conventions?
2. Are comments and docstrings appropriately written explain the purpose and functionality of the code?
3. Are type annotations used effectively where applicable?
4. Are functions appropriately modularized, with well-defined responsibilities and clear separation of concerns?
Are variables' lifetimes intentionally managed, avoiding
frequent reassignment or overly long scopes?
6. Is error handling implemented appropriate where necessary?
7. Is the code properly indented and follows standard formatting guidelines?
Do comments provide context and rationale, rather than merely describing what the code does?
9. Are functions and classes designed with clear, single responsibilities?
10. the code formatted in way that enhances readability?
And provide suggestions for improvement based on the evaluation criteria. You can also provide an improved version of the code in the following style:
25

---

Published as a conference paper at ICLR 2026
#ff Evaluaticn:
#+# Suggesticns: Provide specific, actionable suggestions to improve the code based the evaluation criteria.
Hff Improved Code: Provide a revised version of the code incorporating the suggested improvements.
'python
def peproved_functt (argl: int, arg?: str) str: Your improved code here
pass
E.5 SELF-CONTAINED OPTIMIZATION REWRITING (SCOR).
The prompt used for SCOR is provided below.
Prompt Used for SCOR
You smart software engineer. Please change given code into self-contained and well-structured code following the below best practices and pythonic way.
1. Use meaningful variable and function names.
2. Mrite clear and concise docstring for the function.
3. Use type hints for the function signature. write clear and concise comment for the code block.
5. Ensure the code is self-contained and does not depend on
external variables.
6. Ensure the code is well-structured and easy to zead. Ensure the code is free errors and correctly.
8. Ensure the code is optimized and does not have redundant operations.
9. Ensure the algorithm and data structures are efficient and concise.
If given code is not sel f-contained or too simple, please change it to more educations and useful code.
F MATH LLM REWRITING
As described in Section 4.2, this section presents the prompt used for LLM rewriting in the construc- tion of SwallowMath. The prompt consists of five components: (I) remove residual web headers, footers, and privacy notices; (2) delete extraneous metadata such as question and answer timestamps; (3) fill in missing context when either the question or answer is incomplete; (4) rewrite explanations to be concise yet information-dent
and (5) present a clear step-by-step solution. Steps (1)- parallel our syntax-error and linter filtering for code, while steps (3) correspond to the self-containment and style rewrites used in SwallowCode.
Prompt Used for Math Rewriting
You are an intelligent math tutor. You are given the following math problem and answer with some unnecessary parts. Please remove the unneeded parts of the questions. For exampie, the date of the question submitted, the answer date, the privacy policy, the footer, the header, etc., should be removed. However, please xeep the main question and answer. If questions please make them more informative ive and easy to understand. If needed, please add detail about the step-by-step calculation process.
answers lack seme information
are not elaborate,
26

---

Published as a conference paper at ICLR 2026
G COMPUTATIONAL COST
This section quantifies the computational resources, measured in H100 GPU hours, required for the LLM scoring, LLM rewriting, and ablation experiments in this study. These estimates are derived from empirical measurements. By detailing these costs, we aim to illustrate the trade-off' between the resource requirements of our data refinement pipeline and the resulting enhancements in downstream task performance. Additionally, we provide these figures to support the reproducibility of our results and inform future research efforts. Importantly, both the syntax-erma filtering and the pylint-based filtering stages are CPU-only; although the dataset scale implies nontrivial processing time, their cost negligible compared to the GPU-bound rewriting stage, which dominates the overall compute budget.
G.1 COMPUTATIONAL COST OF LLM SCORING
The LLM scoring process employed vLLMM 0.7.2 and PyTorch 2.5.1, with a global batch size of 2048 and tensor parallelism of 4. Data generation utilized four H100 (94 GB) GPUs per job. At an input processing speed of approximately 2000 tokens/s and an output generation speed of approximately 3000 tokens/s, with an average input length of 836 tokens, an average output length of 1271 tokens, and a total of 20,826,548 samples, we estimate that the dataset creation for the experiments in Section 3.2.3 required 19,477 H100 GPU hours. This figure excludes vLLM initialization and safetensor loading times.
G.2 COMPUTATIONAL COST OF LLM REWRITING
Similarly, the LLM rewriting synthesis used vLLM 0.7.2 and PyTorch 2.5.1, configured with a global batch size of 2048 and tensor parallelism of 4. Generation was performed four H100 (94 GB) GPUs per job, achieving an input processing speed of about 2000 tokens/s and an output generation speed of about 3000 tokens/s. With an average input length of 836 tokens, an average output length of 1819 tokens, and 20,826,548 samples in total, the dataset creation for the experiments in Section 3.3.] estimated to have consumed 23,703 H100 GPU hours, excluding vL.LM initialization and safetensor loading.
G.3 COMPUTATIONAL COST OF CONTINUAL PRE-TRAINING DATA ABLATION EXPERIMENTS
The ablation experiments outlined in Sections 3.1 and 4.1 were executed using 64 H100 (94 GB) GPUs for 24.7 hours per run, resulting in 1,580 H100 GPU hours per experiment. Across 15 experiments (13 for code ablations and 2 for math ablations), the total computational expenditure was 23,700 H100 GPU hours. These runs achieved a training throughput of 530 TFLOP///GPU and 590,000 tokens/s, calculated using the FLOP/s formula from the Megatron-LM paper (Narayanan el al. 2021). This represents approximately 53.5% of the H100's peak BF16 Tensor Core performance.
G.4 COST-BENEHTY ANALYSIS
While rewriting the full 16. IB-token scale for SwallowCode demands substantial GPU hours. our approach demonstrates clear efficiency gains in downstream training. As illustrated in Figure T continual pre-training on unrefined noisy datasets like The-Stack-v2 or Stack-Edu results in only marginal improvements on benchmarks such as HumanEval and HumanEval+. In contrast, our rewritten data achieves significant performance boosts with equivalent fewer tokens, highlighting enhanced data efficiency. Although we did not extend baseline experiments beyond 50B tokens, extrapolating the trends in Figure Iidiicates that unprocessed datasets would struggle match our rewriting method's ' performance, even at scales exceeding 100B tokens. To enhance practicality for teams with limited computational resources, our pipeline is modular and scalable, allowing selective application to data subsets or direct use of our publicly released datasets, thereby bypassing the rewriting overhead. This one-time investment in data refinement produces high-quality, reusable corpora that can be shared across the community, amortizing costs over multiple applications and studies. Unlike transient expenditures on model training, resources allocated to data curation yield enduring assets that advance broader research. As evidenced in works like Qwen3 YYang et al. 2025).
27

---

Published as a conference paper at ICLR 2026
which invested heavily in synthesizing trillions of high-quality text tokens, such commitments standard LLM development and deliver substantial long-term value.
H CONTAMINATION ANALYSIS
In this section, we detail our efforts to mitigate and verify the absence of data contamination in the SwallowCode and SwallowMath corpora. Contamination, whether through direct test-set leakage indirect oeffconntainntion introduced by the rewriting LLM, can inflate performance metrics and undermine the validity of experimental results. We address both concerns systematically to ensure the integrity of our findings.
H.1 DECONTAMINATION AGAINST EVALUATION BENCHMARKS
To prevent test-set leakage, we performed rigorous decontamination checks on both corpora against their respective downstream benchmarks. For SwallowCode, streamed the entire 16. IB token corpus and scanned for overlaps with HumanEval and HumanEval+ prompts. This involved checking for exact matches and computing Jaccard similarity scores, with threshold of 2 0.8 considered indicative of high similarity. No such instances were detected, confirming that SwallowCode is free of direct contamination from these benchmarks. Similarly, for SwallowMath, an analogous procedure against the prompts and solutions from GSM8K and MATH. Using exact match detection and Jaccard similarity (2 0.8 threshold), verified that there contamination overlaps in the corpus.
conducted
H.2 ADDRESSING SELF-CONTAMINATION FROM THE REWRITING LLM
subtler risk arises from "self-contamination." where the rewriting of LLM,, Llaaa-3.3-7085 T might inadvertently incorporate patterns knowledge from benchmark problems into the rewritten data. This model has a knowledge cutoff date December 2023, meaning it lacks exposure to post-cutoff data benchmarks. To evaluate this, assessed the performance GSM-Plus, mathe- matics benchmark released after the cutoff date, which the rewriting LLM could not have encountered during its training. Continual pre- pre-training of Llama-3.1-8 with Finemath-4+ yields 35.75 points on GSM-Plus (Li et al. 2024), while using SwallowMath achieves 46.52 points. This substantial improvement demonstrates that the gains from MyallowMMahh are attributable to enhanced data quality rather than embedded contamination from the rewriting process. These analyses collectively affirm that our performance improvements stem from genuine advancements in data refinement, rather than artifacts of contamination.
I MBPP
As discussed in Section 3.3.1 the MBPP datase contains Python functions with naming conventions that deviate from standard Python style guidelines. For example, the following code snippet uses camelCase instead of the recommended snake_case:
def is Powerowwrrr OF Two (x)
return and (not (x & (x - 1)))
def di differ_At_One_ue Bite_Bit_Poos (a, b)
return is_Power_of_reer
b)
Style-Guided Code Rewriting (SGCR) rewrites code to conform to Python's 's naming conventions specifically enforcing snake_case _aae for function names. Consequently, when tasked with implementing functions that use non-standard naming (e.g., camelCase), an LLM trained SGCR-preet processedd data may rewrite function names adhere to snake case. This leads to mismatches during MBPP evaluation, where calling a function with its original non-standard name results in "is not defined" error.
As described Section 3.3.
Mtt pittps:: / /ttpttttttbb/////////
mbpp. 19on]
httpp:: /ttp://pppp...pmm
28

---

Published as a conference paper at ICLR 2026
0.60
0.55
0.50
Score
0.45
0.40
0.35
0.30
0.25
0
10
20
30
40
50
Billion Tokens
MBPP@1 Baseline
MBPP@1 SGCR
MBPP @ 10 Baseline
MBPP 10 Syntax + Linter (dashed)
MBPP@ @1 Syntax + Linter
MBPP SGCR (dashed)
Figure 8: Comparison MBPP@ and @BPP@ scores models trained on filterd data versus SGCR-rewringg rewritten data in toBetoken continual pre- pre-training ablation study. SGCR's enforcement snake case naming conventions leads to lower scores due to mismatches with MBPP's non-standar function names.
Figure 8 illustrates this issue: models trained on SGCR-processed data exhibit lower MBPP@ and MBPP@ @ 10 scores compared to models trained on data processed only with syntax and linter-based filtering (Section 3.2.2). This performance drop stems from the naming mismatches described above, which obscure the model's true code generation capabilities. Based on this finding, we concluded that MBPP is not a suitable benchmark for evaluating LLM code generation in our experiments, its evaluation framework penalizes adherence to standard Python naming conventions. Therefore, we excluded MBPP from the benchmarks used in this study.
J EVALUATIONS
In this section, we present the evaluation results of models trained through ablation experiments code and math datasets, assessed across ten benchmarks encompassing code and mathematical downstream tasks.
J.1 CODE ABLATION EXPERIMENTS RESULTS
As described in Section 3.1 we evaluated models continually pre-trained from Llama-3.1-88 on ten English downstream tasks. In the following, we report the evaluation results for 13 code ablation experiments, with their relationships illustrated Figure 9 Experiments expl, exp8, exp9, and exp13 serve as baselines, utilizing only Python data extracted from existing open code corpora. The remaining experiments are conducted to construct the SwallowCode dataset. We evaluated performance using the following ten benchmarks: OpenBookQA (Mihaylov et al. 2018 2018), TriviaQA (Joshi et al.) 2017). HellaSwag (Zellers et al. (019), SQuAD 2.0 (Rajpurkar et al. 2018).
XWinograd (Tikhonov & Ryabinin 2021). MMLU (Hendrycks et al. 2021a) GSM8K (Cobbe et al.
2021). BBH (Suzgun et al. 2022). HumanEval (Chen et al.) 202 and HumanEyal+ (Liu et al.
2023
29

---

Published as a conference paper at ICLR 2026
exp1
Stack v2 train smol
exp2
Syntax error filter
exp10
SOCR (lyle-Guidedd)
exp3
Linter filter
SGCR (Style-Guided)
exp4
comment
exp6
exp7
exp11
SCOR (Sett-Contained)
exp8
vi(ptthon)
exp12
taraito commert 99% (Ay)
exp8
codeparrot-clean
exp13
Stack Edu(python)
Figure 9: Relationships between code ablation experiments.
Table 9: Performance across benchmarks in stack-v2-traiis smol-ids Python subset ablation.
Experiment the.stackk---traiin-mmo--- Python subset
Tokens (B)
OpenBoukQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanIivale
10
0.3640
0.6659
0.5995
0.3354
0.9032
0.6294
0.4602
0.6019
0.3366
0.2994
20
0.3540
0.6567
0.6019
0.3360
0.9024
0.6238
0.4852
0.5898
0.3433
0,2890
30
0,3700
0.6588
0.6034
0.3377
0.9045
0.6263
0.5072
0.5939
0.3402
0,2951
40
0.3800
0.6618
0.6053
0.3380
0.9097
0.6341
0.5011
0.6016
0.3659
0.3079
50
0.3700
0.6679
0.6054
0.3350
0.9045
0.6340
0.5027
0.6091
0.3689
0,3140
Table 10: Performance across benchmarks in syntax-erre error-free data ablation from Experiment 1.
Experiment 2: Syntax-error-fre data from Experiment
Tokens (B)
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSM8K
BBH
HumanEval
HumanEvale
10
0.3560
0.6675
0.6015
0.3385
0.9062
0.6321
0.4784
0.588
0.3604
0,3030
20
0.3520
0.6635
0.6026
0.3364
0.9049
0.6252
0.4784
0.5781
0.3591
0,2970
30 0.3560
0.6637
0,6012
0.3375
0.9080
0.6313
0.5019
0.5950
0.3701
0.3122
0.6046
0.5019
0.5998
0.3067
Table 11: Performance across benchmarks in syntax-error and Pylint- (score > 7) data ablation from Experiment
Experiment Syntax-error Pylint-ilteree
7) Experiment
Tokens (B)
OpenBookQA
TriviaQA
HellaSway
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
Humankivat+
10
0.4564
0.3500
20
0.3500
0.6613
0.6015
0.3361
0.9054
0.6237
0.4860
0.5838
0.3744
0.3207
30
0.3620
0.6596
0.6008
0.3359
0.9080
0.6307
0.4867
0.5921
0.3957
0.3134
40
0.3720
0.6650
0.6030
0.3352
0.9058
0.6326
0.4822
0.5990
0.3890
0.3341
50
0.3740
0.6677
0.6054
0.3291
0.9019
0.6327
0.4996
0.6145
0.3945
0.3317
Table Performance across benchmarks conmenttlangiage-rssriteee (English and Japanese) data ablation from Experiment 3.
Experiment Comment--nngugg--rsrriicee (English and Japanese) data from Experiment 3
Tokeus
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanEVale
10
0.3640
0.6713
0.5988
0.3329
0.9054
0.6312
0.4708
0.5953
0.3549
0.3079
20
0.3520
0.6601
0.6011
0.3306
0.9067
0.6250
0.4898
0.5802
0.3689
0.3134
0.3299
0.6671
0.3756
0.3348
50
0.3700
0.6703
0.6061
0.3357
0.9101
0.6347
0,5133
0.6036
0.3841
0.3354
30

---

Published as a conference paper at ICLR 2026
Table 13: Performance across benchmarks in SGCR-rewritten data ablation from Experiment
Experiment SGCR-rewritten data from Experiment 3
Tokens (B)
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanIEval
10
0.3560
0.6689
0.599
0.3295
0.9054
0.6256
0.4875
0.5991
0.4128
0.3652
20
0.3460
0.6610
0.6031
0.3352
0.9032
0.6262
0.4920
0.5801
0.431
0,3805
30
0.3620
0.6637
0.6043
0,3378
0.9110
0.6269
0,5216
0.5984
0.4726
0,4293
40
0.3660
0.6645
0.6053
0.3372
0.9045
0.6328
0.4989
0.5945
0.4610
0.4128
0.5027
0.6065
0.4110
Table 14: Performance across benchmarks in L.LM-ccrred (score > 6) data ablation from Experiment
Experiment
(score
Experiment
Tokens (B)
OpenBookQQ
TrivinQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BB1I
HumanEval
HumanEivat+
10
0.4701
0.3640
0.3134
20
0.3540
0.6593
0,6010
0.3358
0.9045
0.6249
0.4822
0.5810
0.3659
0,31
30
0.3660
0.6894
0.6021
0.3398
0.9071
0.6226
0,5140
0.5893
0.3994
0,3396
40
0.3700
0.6636
0.6021
0.3370
0.9080
0.6300
0.5027
0.6019
0.4018
0,3329
50
0.3640
0.6684
0.6046
0.3353
0.9084
0.6324
0.501
0.6090
0.3951
0,3402
Table 15: Performance across benchmarks in mixed (1:1) data ablation from Experiments 3 and
Experiment Mixed (1:1) data from Experiments and
Tokens (B)
OpenBookQA
TriviuQA
HellaSwag
SQuAD2.0
XWINO
MML.U.
GSMSK
BBH
HumanExal
HumanEval+
10
0.362U
0,6660
0.5994
0.3293
0.9032
0.6242
0.4738
0.6156
0.3616
0.3061
20
0.3460
0.6585
0.6018
0.3297
0.9024
0.6293
0.4845
0.5809
0.3823
0,3427
30
0,3680
0.6611
0.6022
0.3384
0.9062
0.6241
0,5110
0.6045
0.3848
0,3427
40
0.364U
0.6666
0.6028
0.3327
0.9088
0.6323
0.5072
0.6056
0.4018
0.3634
50
0.6052
0.3320
0.6300
0.5027
0.6051
0.4116
0.3573
Table 16: Performance across benchmarks in the-stack-vl Python subset ablation.
Experiment 8: the-stack-vi Python subset
Tokens
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMNK
HumanEval
HumanEvale
10
0.3660
0.6646
0.6033
0.3310
0.9028
0.6219
0.4784
0.5955
0.3244
0,2780
20
0.3500
0.6595
0,6018
0.3233
0.9037
0.6246
0.4701
0.5898
0.3220
0.2720
30
0.3640
0.6575
0.6014
0.3279
0.9071
0.6226
0.5057
0.5878
0.3244
0.2957
0.3680
0.3265
0.3445
50
0.3620
0.6650
0.6053
0.3212
0.9084
0.6273
0.5080
0.5998
0.3561
0.3177
Table 17: Performance across benchmarks in codepartto-clean Python subset ablation.
Experiment 9: codepartto-cleun Python subset
Tokens
OpenBoukQQA
TriviaQA HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
Humankvale
10
0.3540
D.6651
0.6006
0.3221
0.9062
0.6295
0.4708
0.5875
0.3598
0.3128
20
0.3549
0,2927
30
0.3680
0.6570
0,6045
0.3390
0.9071
0.6290
0.4890
0.5976
0.3524
0.3201
40
0.3720
0.6613
0.6048
0.3352
0.9075
0.6300
0.4958
0.6108
0.3543
0,2988
50
0.3600
0.6638
0.6055
0.3321
0.9097
0.6273
0.5072
0.6139
0.3616
0.3134
Table 18: Performance across benchmarks in SGCR-rewritten data ablation from Experiment 1.
Experiment SGCR-rewritten data Experiment
Tokens (B)
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanEvale
10
0.3640
0.6667
0.5996
0.3325
0.6164
0.4845
0.5959
0.4457
0.3518
20
0.3460
0.6592
0.6032
0.3324
0.9067
0.6231
0.4890
0.5655
0.4579
0.3994
30
0.3660
0.6585
0.6029
0.3379
0.9101
0.6176
0,5064
0.5855
0.4494
0,3884
40
0.3600
0.6650
0.6024
0.3339
0.9067
0.6284
0,5148
0.5967
0.4622
0.4067
50
0.3600
0.6687
0.6047
0.3337
0.9084
0.6317
0.5057
0.6041
0.4646
0.4067
31

---

Published as a conference paper at ICLR 2026
Table 19: Performance across benchmarks in SCOR-rewritten data ablation from Experiment 5.
Experiment
Experiment
Tokens (B)
OpenBookQA
TrivinQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanEvale
10
2.6680
0.6006
0.3317
0.9067
0.6237
0.4655
0.6108
0.4567
0.4348
0,3500
0.6564
0,6026
0.3349
0.4981
0.5718
0.5384
0.4628
30
0.3620
0.6640
0,6023
0.3385
0.9054
0.6253
0.5095
0.5928
0.5256
0.4817
40
0.3640
0.6705
0.6041
0.3401
0.9088
0.6317
0,5095
0.5982
0.5226
0,4768
50
0,3700
0.6085
0.6055
0.3359
0.9114
0.6322
0,5110
0.6062
0,5396
0,4805
Table 20: Performance benchmarks in mixed (90% Experiment 11, 10% Japanese- transslaed
comments) data ablation.
Experiment Mixed (90% Experiment II, 10% Japanese-transllted comments)
Tokens (B)
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
GSMSK
BBH
HumanEval
HumanEvale
10
0.3640
0.6020
0.3341
0.9054
0.6221
0.4738
0.5697
0.4799
0.4280
0.4646
30
0.3640
0.6595
0.6034
0.3410
0.9080
0.6250
0.5011
0.6008
0.5110
0.4415
40
0.3640
0.6640
0.6022
0.3361
0.9054
0.6330
0.4898
0.6008
0.5299
0,4768
50
0,3600
0.6635
0.6057
0.3340
0.9080
0.6315
0,5072
0.6057
0.5329
0.4866
Table 21: Performance across benchmarks in Stack Edu Python subset ablation.
Experiment 13: Stack Edu Python subset
OpenBookQA
TrivinQA
HellaSwag
SQuAD2.0
XWINO
MML.U
GSMSK
HumanEval
HumanEval+
10
0.3640
0.6696
0.5986
0.3358
0.9037
0.6246
0.4761
0.6004
0.3470
0.2963
20
0.3520
0.6632
0.6021
0.3364
0.9067
0.6233
0.4898
0.5942
0.3537
0.2957
30
0.3660
0.6600
0.6024
0.3439
0.9097
0.6251
0.4989
0.5916
0.3713
0.2988
0.4958
0.3226
50
0.3740
0.6065
0.6061
0.3368
0.9062
0.6350
0.5087
0.6173
0.3695
0.3195
32

---

Published as a conference paper at ICLR 2026
J.2
MATH ABLATION EXPERIMENTS RESULTS
As described in Section 4.1 we evaluated models continually pre-trained from Llama-3.1 1-8B on ten English downstream tasks. Below, we present the evaluation results for two math ablation experiments. Specifically, we adopted the following ten evaluation benchmarks: Open (Mihaylov et al.
2018).
TriviaQA (Joshi al. 2017). HellaSwag (Zellers et al. 2019), SQuAD 2.0
RRajuurkar et al.
2018).
XWinograd (Tikhonov & Ryabinin 202I). MMLU (Hendrycks et al. 202 la). GSM8K (Cobbe
al.
202T). BBH (Suzgun al.) 2022). HumanEval (Chen et al. 2021 and MATH (Hendrycks
et al.
2021a)
Table 22: Performance across benchmarks in finemath-4+ ablation.
Experiment finemath-4+
Tokens
OpenBoukQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MMLU
HumanEval
GSM8K
BBH
MATH
10
0,3700
0.6626
0.5990
0.3350
0,8985
0.6243
0.3439
0.4685
0.6057
0.1760
30
0.3700
0.6574
0,5999
0.3506
0.8998
0,6253
0.3561
0.5019
0.5971
0.2260
40
0.3720
0.6577
0.6024
0.3499
0.9049
0,6312
0.3701
0.5231
0.6054
0.2260
50
0.3740
0.6608
0.6001
0.3550
0.9058
0.6329
0.3561
0.5292
0.6166
0,2400
Table 23: Performance across benchmarks in finemath-4+ rewritten with 1.3ma-3.3-70B--1 Instruct ablation.
Experiment linemath-4+
Tokens (B)
OpenBookQA
TriviaQA
HellaSwag
SQuAD2.0
XWINO
MML.U
HumanEval
GSMBK
BBH
MATH
10
0.372U
0.6643
0.5970
0.3443
0.9015
0.6343
0.3439
0.5603
0.5535
0,2480
20
0.3800
0.6580
0.5946
0.3428
0.8994
0.6293
0.3762
0.6156
0.5669
0,2860
30
0.3660
0.6618
0.5964
0.3470
0.9011
0.6298
0,3530
0.6262
0.6383
0.3040
40
0.3700
0.6610
0.5973
0.3535
0.9088
0.6358
0.3738
0.6422
0.6237
0.3100
50
0.5972
0.3683
0.6414
0.3160
K
THE USE OF LARGE LANGUAGE MODELS
In accordance with the ICLR 2026 policy on responsible of language models, disclose that LLMs were used only for nonsubstantive copy editing. Specifically, LLMs were employed to (i) suggest alternative word choices. (ii) correct minor grammatical issues, and (iii) improve the clarity and fluency of sentences originally written by the authors.
No LLM used generate scientific content, conduct data analysis, design execute experi- ments, create figures or tables, select citations, or draw conclusions. All technical ideas, methods, experiments, results, and interpretations were conceived, written, and verified by the authors.
The use of LLMs does not affect the reproducibility or integrity of the work.
33


Reviews:
[{"review_id": "vTzvUN3Te5", "reviewer": "Reviewer_C3n8", "summary": "The manuscript introduces two novel openly licensed datasets — SwallowCode and SwallowMath — designed to enhance LLM performance in code synthesis and mathematical reasoning. Both datasets apply a transform-and-retain methodology: instead of filtering out low-quality samples, the authors employ LLM-driven rewriting to refine existing corpora. Extensive ablations show significant gains on HumanEval, HumanEval+, GSM8K, and MATH benchmarks.", "questions": "Are there other LLM rewriting techniques to compare with the proposed method? The comparison could include quality and training results (if applicable).\n\nAnother question is whether the proposed rewriting method is limited by the rewriting LLM, meaning the dataset would need frequent updates.\n\nCould the method be applied iteratively?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method of rewriting is well-motivated with sufficient ablation studies of different data-filtering methods.\n\nThe improvements are substantial and consistent across multiple benchmarks (HumanEval, HumanEval+, GSM8K, MATH), with comprehensive ablation studies demonstrating that each pipeline stage contributes incrementally.\n\nThe paper is well-written and easy to follow.", "weaknesses": "The novelty is somewhat limited. The paper reads more as an engineering work in improving pre-training performance with careful experimental design rather than introducing new ideas. The author should try to clarify the unique contribution and insight of this paper to academic community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7JZ71NqwDk", "reviewer": "Reviewer_bMSq", "summary": "This paper introduces SwallowCode and SwallowMath, two openly licensed datasets created by systematically rewriting existing public corpora using a multi-stage LLM-driven pipeline. The approach combines filtering (syntax and linter-based) with two-stage rewriting (SGCR for style, SCOR for self-containment and optimization) to enhance data quality. Continual pre-training of models like Llama-3.1-8B and Qwen2-7B on these datasets shows significant gains on code and math benchmarks, outperforming existing datasets like Stack-Edu and Finemath-4+.", "questions": "The SwallowCode pipeline utilizes a two-stage LLM rewriting process (one for style, one for efficiency/self-containment). Why was this split necessary? Did the authors attempt a comparative study using a single-stage LLM prompt designed to achieve all stylistic and functional improvements simultaneously, and if so, what was the performance degradation compared to the more complex two-stage approach?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* This paper releases two open source datasets under permissive licenses, supporting community reuse and extension.\n\n\n* The ablation studies isolate the impact of each pipeline stage, demonstrating clear and interpretable improvements.", "weaknesses": "* The success and ceiling of this method are fundamentally limited by the capabilities and potential biases of the powerful LLM. Rewriting relies on Llama-3.3-70B-Instruct, which may introduce its own stylistic or semantic biases.\n\n\n\n* While the paper claims rewriting enforces \"algorithmic efficiency,\" it is not clear how the paper quantifies or measures this specific gain. Providing concrete metrics or examples of complexity improvements would strengthen this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEewYdJgRc", "reviewer": "Reviewer_NJeE", "summary": "Transform-and-retain rewriting of public corpora using a two-stage LLM pipeline, rather than exclusionary filtering. For code: Style-Guided Code Rewriting (SGCR) followed by Self-Contained Optimization Rewriting (SCOR). For math: rewrite Finemath-4+ to remove boilerplate, restore context, and produce concise step-by-step solutions.", "questions": "- What proportion of rewritten code compiles and runs? Any automated test execution stats on a held-out suite?\n- For SCOR, how often does algorithmic “optimization” reduce correctness (e.g., edge cases)? Any spot-audit or unit-test sampling?\n- Any evidence the gains persist or improve at larger pre-training budgets?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Thoughtful analysis of why synthetic-from-scratch may underperform due to diversity issues.\n- Decontamination checks and cross-model validation (Qwen2-7B) improve credibility.\n- Open release of data, prompts, and checkpoints enhances community value and reproducibility.", "weaknesses": "- generality claims are suggestive but not demonstrated across languages or larger scales.\n- no analysis of downstream generalization outside HumanEval/+.\n- No quantitative quality checks on rewritten outputs (e.g., compile/run rate, test pass rate, semantic drift)—risk of introducing hallucinated correctness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JT4UVVDnlT", "reviewer": "Reviewer_NDvj", "summary": "This paper refines Python snippets from The-Stack-v2 through a novel four-stage pipeline: syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into self-contained, algorithmically efficient examples. Experiments show that continued pretraining using the resulting dataset improves performance of LLMs on several benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear and easy to follow\n2. They provided the code for the data processing pipeline, improving the reproducibility of the work", "weaknesses": "1. The novelty seems limited. Using LLMs to rewrite data for continued pretraining has been adopted by previous works such as Qwen2.5-Math. The proposed method seems more like minor engineering tricks than major algorithmic novelty.\n2. The proposed method was only compared to coarse pretraining datasets such as Stack v1 and Stack v2. Comparison between SwallowMath and other more carefully filtered and processed datasets such as OpenWebMath [1], Lemma [2], and MathCoder2 [3] should be considered as well.\n\n3. The code only contains Python snippets. It is unclear whether it can improve the models’ coding abilities of other programming languages.\n\n[1] Paster, Keiran, et al. \"Openwebmath: An open dataset of high-quality mathematical web text.\" arXiv preprint arXiv:2310.06786 (2023).\n\n[2] Azerbayev, Zhangir, et al. \"Llemma: An open language model for mathematics.\" arXiv preprint arXiv:2310.10631 (2023).\n\n[3] Lu, Zimu, et al. \"Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code.\" arXiv preprint arXiv:2410.08196 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)