D3: A Dataset for Training Code LMs to Act Diff-by-Diff
Authors: Ulyana Piterbarg
Kanishk Gandhi
Lerrel Pinto
Noah Goodman
Rob Fergus
Venue: COLM 2025 (2025)
Keywords: data filtering
synthetic data
code synthesis
code editing
file diffs
midtraining
SFT
LM agents
TL;DR: D3 is a dataset of 8 billion tokens of file-diff-sequence examples sampled from 850k Human-written source files, improving LM performance on code synthesis, completion, & editing.

Abstract: We introduce D3 ("Diverse Data for Diff-by-Diff Coding"), a large dataset for training LMs to iteratively synthesize general-purpose Python source code by generating file diffs. D3 frames code synthesis as a goal-conditioned sequential decision-making problem, where goals, states, and actions are represented by token sequences corresponding to the description of a functionality to add, the current contents of a file, and a file diff, respectively.  The dataset contains 8 billion tokens of instruction + file-state + file-diff-sequence examples sampled from  850,000 human-written Python source files.  To construct D3, we filter, augment, and annotate source code from The Stack by sampling synthetic file-diff sequences with a code analysis tool and labeling each sample with an LLM-generated rationale. In our experiments, we show that mid-training LMs like Llama 3.2 1b and 3b on D3 prior to supervised fine-tuning (SFT) on task-curated data improves performance on synthesis & editing tasks. On benchmarks like HumanEvalSynth and HumanEvalFix, we observe improvements in pass@1 of 3 to 6 points compared to direct SFT. D3-trained models are particularly strong at completing partial human-written solutions to programming problems.

Published as a conference paper at COLM 2025
D3: A Dataset for Training Code LMs to Act Diff-by-Diff
Ulyana Piterbarg" x New York University
Kanishk Gandhi Stanford University
Lerrel Pinto New York University
Noah Goodman' Stanford University
Rob Fergus* New York University
Abstract
We introduce D3 ("Diverse Data for Diff-by-Diff Coding"), a large dataset for training LMs to iteratively synthesize general-purpose Python source code by generating file diffs. D3 frames code synthesis as soal-conditioned sequential decision-making problem, where goals, states, and actions represented by token sequences corresponding to the description tionality add, the current contents of file, and file diff, respectively. The dataset contains billion tokens of instruction + file-state + file-diff- sequence examples sampled from 850,000 human-written Python source files. To construct D3, we filter, augment, and annotate source code from The Stack by sampling synthetic & linter-error-free file-diff sequences and labeling each sample with an LLM-generated rationale. In our experiments, we show that mid-training LMs like Llama 3.2 1b and 3b on D3 prior to supervised fine-tuning (SFT) on task-curated data improves performance on synthesis & editing tasks. On benchmarks like HumanEvalSynthh and HvmanEvalFixx observe improvements in pass@1 of 3 to 6 points com- pared to direct SFT. D3-trained models are particularly strong at completing partial human-written solutions to programming problems.
a func-
diff action SequIrie
test-time
reom tysing import Tuple
((a))
Suport
cet Apent tacasion Punntiess
pre
(80.9) 9]
[end of Bi. www.ippniiim/scc//seenn agent.
goal obs.
LLM
Add DONAgent class for interacting with and learning from the environment.
Ls Ro11)
LLM
Figure 1: The D3 dataset. Diff action traces are checked for syntactical correctness with Python analysis tool and capture a wide range of abstractions, from fine-grained additions to files like library imports to high-level architectural changes. The procedure used to prepare D3 not specific to Python and can be used to create similar datasets for other languages.
Introduction
Achieving human-level performance in software engineering let alone surpassing it requires language models (LMs) to exhibit open-ended programming capabilities, i.e, be able to efficiently explore program space and continuously add to or evolve the functionality and structure of source file. While recent advances have enabled models to generate code to solve isolated programming problems from natural language instructions (Jaech al., 2024; Dubey et al., 2024; Team et al., 2024), LMs and LM agents still fall short of replicating the iterative and exploratory nature of human software development (Jimenez al., 2023; Yang et al., 2024; Pan et al., 2024; Antoniades et al., 2024).
*Corresponding author: 118821@nyu.eee *Equal advising contribution Code and the full D3 dataset will be made available at: https:/

---

Published as a conference paper at COLM 2025
Phase I: Filtaring Code Protrabning Data
Extert mottsk eno
Barple Random
the Complexiy & Quallty Codo
Promrpt 3.1-10B Grade Sampled Python File
Find
COOOOO 000000 Le 500 00G 500
III
I
DQ P!
morts melll pp fundtios
Phase II:
Source File Diff- Sequencee Augmentation & Instruction Labeling
NN Use Linise to Re Sample Programs imp Syrihwlic Sequenoos Bh Python Linkar
(B) Frompp Liana 70B instuct to Generate Edit
compation
Seequanice Far callare & Convert tratruction
Pallnt
0013
LintSeg (ICLR 2028)
()) Bample Rundom Lind to move from Source Code
(2) Line Linter to Find & Plrmove Dapwncinnt
Lines and Record the SuE-Fregram
@3 Repeot & Unti Brpty
( Curp et Eaequrnce of Litter Vertted SuthProgramee & Compute Parwice Dits
For comee (uul))) 17887
Pyllic
Figure 2: (Top left) The procedure used to prepare D3. There are two phases: source file filtering, and diff sequence sampling & instruction labeling. The first phase uses LLM-judge and a data grading rubric (Appendix A, Figure 7), while the second uses an LLM-labellr and the LintSeq algorithm for sampling file-diffs. (Top right) An instruction + file-state + file-diffseqquence example in D3. (Bottom) The LintSeq algorithm for sampling synthetic file-diffs that reflect the semantics of a programming language.
One explanation for why models struggle with iterative code refinement (Olausson et al., 2023) is that they are predominantly trained on complete programs rather than edit data (Lozhkov et al., 2024b). Moreover, existing open-source datasets for coding lack: 1) scale & diversity - many datasets are optimized for downstream success on a narrow set of Python problem-solving datasets like MBPP and LeetCode (Wei et al., 2024), producing brittle models; 2) descriptive labels for edits - code edits are not necessarily paired with meaningful goal annotations (Muennighoff et al., 2023); and 3) human-like abstraction patterns, which may be helpful for iteratively exploring and designing code.
In an effort to address these limitations, we introduce D3 ("Diverse Data for Diff-by-Diff Coding"), ) a dataset of 3.6 million examples of instruction-annotaedd code edit sequences. D3 contains 8 billion tokens of data (Table 1), prepared through novel combination of synthetic diff generation and -.owere filtering & labeling over permissively-licene source code files from The Stack (Kocetkov et al., 2022). Our approach leverages the LintSeq algorithm to generate synthetic diffs that reflect programming language semantics and syntax (Piterbarg al., 2024), a pretraining corpus filtering procedure inspired by FineMath (Lozhkov et al., 2024a), and an LLM-powered instruction labeling procedure (Wang et al., 2021) (see Figure 2). The data in D3 captures a wide range of programming patterns and abstractions, from low-level syntactic modifications to high-level architectural changes to source files, each paired with natural language instructions that describe the intended transformation (see Figure 1). Qualitatively, examples in D3 cover diverse topics like unit testing, data visualization, file I/O, image processing, and deep learning (see Figure 3).
Previous work has shown that breaking down code generation across sequences of edits or file-diffs can improve the quality & diversity of LM sampled code (Piterbarg et al., 2024). This mirrors how tffin-of-thought prompting (Nye et al., 2021; Wei et al., 2021) can help language models solve certain complex reasoning tasks through intermediate steps. In our own set of experiments, we demonstrate each of the items below.
1. D3 improves the performance of small LMs on synthesis & editing tasks: Mid- training Llama 3.2 1b and 3b on D3 prior to supervised fine-tuning can improve pass@1 on single-file coding benchmarks by 3 to 6 points (Figure 4, Table 2).
2. Ablating file-diffs from D3 hurts performance: In the mid-training stage, replacing D3 file-diff sequences in favor of file re-writes - i.e., swapping diff action prediction for file-state prediction degrades performance on synthesis tasks (Figure 5).
2

---

Published as a conference paper at COLM 2025
3. LLM-powered source code filtering produces "seed" corpus with better vali- dation fit to synthesis & editing: Training Llama 3.2 1b on Python files filtered with our approach results in improved loss with-held examples from MBPP, BigCodeBench, & SWE-Bench compared to training on a random sample (Figure 5).
4. Reliance LLMs does not hurt data quality: A human study the quality of synthetic components in the D3 data preparation pipeline suggests broad agreement between LLM-judge vs human Python-expert file grades and edit descriptions.
Our work establishes that training on large-scale edit sequence data can be beneficial for improving the programming capabilities of language models. We see D3 a first step towards the development of datasets for introducing human-like software development abstractions to LMs in earlier stages of training, providing a foundation for future work on open-source, autonomous, and open-ended programming agents.
Table 1: Comparing D3 to existing datasets for training language models on file-level code editing and/or synthesis from instructions. Tokens reflect the Llama 3.2 tokenizer.
Dataset Name
Diffs?
Use-Case
Source
Task(s)
Toks
Toks/
Evol-Instruct (Luo et 2023)
X
post-train
synthetic synthetic
problem solving
28M
400
OSS-Instruct (Wei et al,,
x
post-train
problem solving
37M
500
InstructCoder eLi 2023)
post-train
synthetic
code editing
29M
270
Lintšeq-Instruct (Piterbang et al,, 2024)
post-train
synthetic
problem solving
184M
412
CommitPackPT (Muennighofi 2023)
post-train
human
code editing
275M
400
D3 (ours)
mid-train
human synthetic
software synthesis
8.0G
2200
Problem Formulation
We formalize code synthesis goal-directed decision-making problem where an agent generates code through a sequence of edits to satisfy a natural language specification, as illustrated in Figure 1. This problem can be described as MDP (S,9, R), where:
The state space S consists of states 8;, which each represent the contents and meta- data of source file or a multi-file codebase at a time step i.
The goal space G corresponds to the set of possible code synthesis goals expressed in natural language, such that each g E corresponds to an instruction (shown as "goal obs." " in Figure 1, e.g.," "Add DQNAgent class for interacting with and learning from the environment").
The action space A consists of individual diffs (atomic edits) that can be applied to the current file state s,. As shown in the "diff action sequence" in Figure 1, actions E A are individual edits (e.g., adding import statement each consisting of multiple tokens, that sequentially build up code changes.
class definition),
The transition function T S x A + S is deterministic and corresponds to applying the diff specified by action to the current file state to yield $;+1-
The reward function R S x G x A x S R provides feedback about whether the modifications are moving toward satisfying the goal specification. As shown in the "test-time" component of Figure 1, this could involve executing the generated code.
The D3 dataset consists only of (goal, (attita--)--attii--eeqeqe without reward annotations. These trajectories do capture successful code modifications that have been synthetically generated and verified for syntactical correctness with a code analysis tool, but do not include explicit reward signals.
trajectories (so, g, (at) | o),
The MDP formulation above captures several key challenges in code synthesis with language models. First, the LM must engage in long-horizon reasoning and goal understanding in order to plan sequences mutually coherent edits. Second, the action space of possible diffs is combinatorially large and highly structured, necessitating efficient exploration strategies. In section 3, we describe how we construct a dataset of high-quality diff action sequences demonstrating successful code synthesis according to this problem formulation.
3

---

Published as a conference paper at COLM 2025
Length (L,C)
noun 1244
"Add escensive cas Google Calendar API fanctionally by with mock dients and error handinng
testing
Indiviaall program expectation redotoonhipe between features and predicted cutromes from trained machine leamnin models spec fiaily using multi multityer percoptron grofient bookinng regressors Calffrnia housing
partti dependance viualize
logghg
LOGO
Aandiig Ling
déjeciettereed programmingg
Santioeel popporming
"Deveko a Python progrum that onables the on and descrid licatin Diino objects boiltading various data for mara speo spociicllyy JSCN, XML and TAMI."
DiP Sequence Length (5LOC)
fll lb
EUPC 133
deep (48)
6ESG (9%)
dbcamentattin 1781
madikte lnarsin
nearal retworks desgredd tran sutonamous agent using the Qulearing algorthmm
"Euginer Pythen program
ALAC
Figure 3: Exploring the contents of D3. (Left) Histograms for two metrics computed over the full D3 dataset: initial file state and diff sequence length in lines of code (LoC). (Right) Results of a qualitative topic discovery analysis run on instructions from D3 with Gemini (n = 100,000) (Team et al., 2024) (see Appendix D). We show the top k = 20 topics by incidence, and provide four labeled examples. Examples in D3 are qualitatively diverse.
3 D3: Diverse Data for Diff-by-Diff Coding
Our procedure for preparing D3 is largely unsupervised. In lieu of manual labeling, it relies on three components: a grading rubric for scoring source code for correctness, quality, & relevance to the study of software development source code understanding, and the LintSeq sampling algorithm, which uses a linter to procedurally decompose source code files into atomic edit actions or line-by-line "chunks" (see Figure 2) (Piterbarg et al., 2024). By combining these tools, we show that it is possible to scalably re-purpose human-written code from the Web into instruction-labeled examples that are effective for training LMs to write and refine seneral-purpose programs. We also demonstrate that reliance on LLMs does not appear to impact data quality in human study.
Python, an LLM capable of standard
3.1 Preparation Procedure
An illustration of our method is provided in Figure 2. We prepare D3 by filtering Python files from The Stack, augmenting file contents into synthetic diff sequences, generating descriptions for each diff sequence, and converting descriptions into instructions.
Filtering Code Pretraining Data with LLM-as-a-Judge Operating under computational constraints, we select random sample representing about 10% of the total Python source code in the de-duplicated version of The Stack, consisting of 2 million unique files. Next, we filter this sample by prompting Llama 3.1 70b Instruct (Dubey et al., 2024) to assign grade to each file. In each prompt, we provide the LLM with the full contents of source file as well as hand-engineered "grading rubric" that outlines a set of four correctness and content scoring criteria. These criteria draw from the math data-filtering prompt developed for the FineMath dataset by Lozhkov et al. (2024a), and are broadly designed to assess relevance to the study of software development in Python. The full prompt is provided in Appendix A. We truncate the contents of any file that exceeds 131,072 tokens.
Once all sampled source files have been graded, we use a simple parser to extract grades from each LLM output. We discard all files that were assigned score lower than 3/4. This leaves us with 900,000 unique Python programs, representing z 40% of the initial file count.
Augmentation and Labeling In this phase, our goals are two-fold: (1) to resample file contents into diff sequences; and (2) to label diff sequences with descriptive instructions. To accomplish the first of these objectives, we use the LintSeq algorithm and the popular Python linter pylint. As in Piterbarg et al. (2024), for each Python source file, we sample a sequence of intermediate sub-programs that represent a possible error-free trajectory for synthesizing the full file contents diff-by-diff. The complexity of each atomic diff action in this sequence is automatically determined by the size of a (syntactical) connected component in the underlying program graph (see Figure 2, bottom). Unlike Piterbarg et al. (2024), we
4

---

Published as a conference paper at COLM 2025
also sample four random sub-trajectories from each edit sequence, yielding some examples that reflect program synthesis "from-scratch" and others that reflect synthesis from partially written file contents. D3 reflects a 3:2 mix of these scenarios. In total, we add five instruction + file-state + file-diff-squunnce examples per selected source file to the dataset.
Finally, we conclude our preparation of D3 by generating synthetic instruction labels for each edit sequence. Iterating over examples, we prompt Llama 3.1 70b Instruct to generate complete but succinct ("lazy") rationale for the code that was added (see Appendix A). Then, we procedurally convert each rationale into an instruction, eliminating examples for which the LLM generated an incorrectly structured output. This labeling procedure allows us to circumvent the existing failure modes of current open-weight LLMs on diff understanding, while still leveraging their capabilities for cheap scalable labeling.
3.2 What does the data in D3 look like?
As shown in Figure 3, examples in D3 span all scales of syntactical structures and abstrac- tions in Python, from those that can be specified in just one line of code to others that requires thousands. The average example contains 34 lines of pre-written source code, and a corresponding diff sequence adding 150 lines of code. Distributions of both initial file lengths and diff sequence lengths are heavily right-skewed, with some examples having as many as 1000 lines of pre-written code. The content of examples in D3 also spans across a di- verse range of topics, such unit testing and deep learning (e-g. "Add tests for the Google Calendar API" or "Engineer a program to train an autonomous agent with Q-learning").
3.3 Assessing LLM-Generated Grades and Edit Descriptions with a Human Study
There are two LLM-controlled components that are critical to the preparation of D3: (1) source file grading, via rubric and LLM-as-a judge; and (2) edit sequence labeling with "lazy" rationales, i.e. single-sentences descriptions of the changes made code. Both components employ Llama 3.1 70b Instruct. To assess whether there is parity between LLM-generated versus human-annotated grades and labels, we conduct a human study.
3.3.1 Human Study Design
In Part A, four human annotators with expertise in Python were presented with the same rubric provided to the LLM judge and asked to provide scores and per-item confidence ratings for 24 randomly selected Python files from the sample of The Stack used prepare D3. Human consensus grades for each file were computed as the arithmetic mean, median, and confidence-weighted mean of the four human scorers.
In Part B, the same set of annotators were provided with "overall" diffs corresponding to the starting vs final files states for 20 randomly selected samples from D3. Each diff was paired with two (order-shuffled) descriptions: the ILM-generater description for the sample present in D3 and a human-written description, prepared by a fifth study participant who was asked to independently label the sample via the same set of instructions given to the LLM-llbeler. Annotators were then instructed to indicate their preferred "higher-quality" description single-blind, and to provide confidence. For each diff, the number of votes for the LLM LLM--enerated description (0-4) was tallied, majority was defined as 23/4 votes, and 2-2 ties were retained for agreement summaries but excluded from the sign test.
3.3.2 Human Study Results
Across the 24 files in Part A, we find that Llama 3.1 70b scores closely track human consensus with no systematic score shift: the median difference and Hodges-Lehmann paired shift for LLM vs human scores were 0.00 against all three human consensuses definitions, mean, median, confidence-weighie Wilcoxon tests were non-significant (p 0.54, 0.50, 0.54, respectively), absolute deviations were small (MAE se 0.49-0.50), and rank-order agreement was strong (Spearman 1.884,884, Kendall 0.71-0.77 all p 0.01) (Wilcoxon, 1945; Spearman, 1904; Kendall,
mean (Hodges & Lehmann, 1963). Furthermore, paired
5

---

Published as a conference paper at COLM 2025
Single-File Synthesis
Singlo-File Debugging
From Scratch
Completion Partial Human Solution
120
30
+25
+20
-19
20
+17
20
+16
pass81
20
+10
10
10
10
Liama
Lama 3b
Lama
Lisma So
Uaama
Liama
Base
MT D
SFT
MT D SFT
Figure 4: Task-aggregatee pass@1 single-file synthesis & debugging for base, mid- trained (MT), and supervised fine-tuned Llama 3.2 and 3b models (temperature = 0.1). For breakdown of results by benchmark, Table 2 below. Base models are evaluated with the D3 prompt-completion structure (see Figure 2). A description of evaluation procedures and examples of formatted prompts are provided in Appendix
1945). Overall, the LLM-judge scores used for source code filtering in the first phase of D3 preparation are aligned and unbiased relative to human judgments.
On the 20 diffs in Part B, the LLM-generated description was the majority choice for 8/20 items (40%, exact 95% CI 0.19-0.64.) A sign test non-tied items (8 wins vs 6 losses; n 14) found no evidence a departure from parity (two-sided between annotators
0.79). Vote dispersion per item moderate to high (mean Shannon entropy 0.71 bits; 0 unanimity, 1 1= 2-2 2-l-)) and chance-corrected agreement among raters was slight (observed agreement 0.10), consistent with meaningful ambiguity and spread in
Ao 0.55 and Fleiss' preferences (Fleiss, 1971). At this sample size, we conclude that the data do not indicate clear overall annotator preference for either LLM-generated
writtew labels.
In summary, our human study affirms the quality of synthetic components in D3 our data suggest broad parity between LLM and Python-expert file grades, and no significant deviation in quality between LLM-generated and human-writke edit descriptions.
4 Mid-training and SFT Experiments
We further study the effectiveness of D3 by conducting series of mid-training and SFT experiments with the Llama model series. These experiments seek to answer the following:
1. Does our filtering procedure produce a corpus that yields better fit on tasks like problem-solving & real-world editing, compared to random sample of programs?
2. How does ablating diff-by-diff generation from D3 affect performance? In other words, does swapping diff actions for full file-state prediction from demonstration data affect LM generation quality at test-time?
3. Does training language models on D3 before task-specifi fine-tuning improve their performance on single file and repository-wide coding tasks?
4.1 Ablation Study: Testing the importance of filtering and diff-by-diff generation
Filtering First, to test the effectiveness of our LLM-assisted data filtering procedure, we continue to train the pretrained model weights on corpuses of filtered vs unfiltered Python source code. Unfiltered files are drawn from the sample graded data (see subsection 3.1) and randomly sampled to match the token count of filtered code. Throughout each exper- iment, we estimate LM accuracy on problem solving & real-world code editing tasks by computing average loss dataset of withheld prompt-completion examples drawn from MBPP, BigCodeBench-HHard and SWE-Bench (Oracle) (Austin et al., 2021; Zhuo et al., 2024; Yang et al., 2024). Prompt tokens masked out.

---

Published as a conference paper at COLM 2025
Ablating L.M-Assistdd (Phase
Abluting Sampling w/ II-A):
Evaluations
Avg. Validitionn Loss va Mis- Mis-Triining Tokens
Best-of-N of-N HumanEvalSynth(i) vs. Time Tokenss
Programming Problem Salving
From Sonttch
Completion of Fartsl Human Soudion
50
untitaned
Mil program
Mull progmm
tereet
eill
1.725
1,700
217
2
steps
game stud tovern at test time
genarated laker' at S00l Fime
Best-of-N MBPP(-) vs. Test- Takena
Editin Pytonn
Completion Soudio
fall
program
20
1.375
212
2
21
2
optinttr rion6 90
gerarstad tofoara 200--2114
tolons teal-irm
Figure 5: (Left) Data filtering ablation: smoothed loss on withheld examples by task while mid-training Llama 3.2 1b The Stack, with and without the filtering procedure described in Section 3.1. Shading indicates value ranges before smoothing. Filtering improves loss both on Python problem solving and real-world library editing examples. (Right) File- diff-sequence ablation: tuned test-time scaling on HumanhvalSynth and MBPP for Llama 3.2 1b models mid-trained on D3 with and without file-diff-eqquences Models trained to synthesize code with diffs exhibit better best-of-n scaling and pass@1 (Table 6, Table 7).
Diff-by-Diff Generation Next, we probe the effect of training LMs to synthesize source code diff-by-dite by mid-training Llama 3.2 1b on the D3 dataset "as is," as well as on ver- sion the dataset where file-diff sequences are replaced with full file states in each example. We train models on both corpuses for two epochs and with identical hyperparameters. To compare models, we evaluate the test-time performance of intermediate checkpoints on the code synthesis benchmarks HumanEvalSynthh and MBPP (Muennighoff et al., 2023; Austin et al,, 2021; Liu et al., 2023). For each task, we test two settings: (A) synthesis "from scratch", where models are prompted to write a program according to instruction starting from an empty file-state; and (B) synthesis from partiall--writter program, where models are prompted complete a partial human solution to each problem. Setting (B) procedurally generated with fixed random seed from existing test set solutions (see Appendix I).
Ablation Results We provide visualizations of results in Figure As shown in Figure (left), filtering improves accuracy on both problem solving & real-world code editing examples throughout training. This suggests that despite the potential biases introduced by our LLM-assisted data labeling procedure, the "seed" corpus of source code that augment and annotate to produce D3 yields a better fit to examples of tasks of interest compared to a random sample of source files with similar token count. The degree of improvement on real-world editing examples especially suggests that filtering balances quality against diversity. Furthermore, shown in Figure 5 (right), models mid-trained to synthesize code diff-by-diff on D3 exhibit better test-time scaling laws compared to their full program regeneration counterparts, reinforcing the conclusions of Piterbarg et al. (2024).
4.2 Following Mid-training on D3 with Task-Targeted SFT
Next, we test whether there are downstream benefits mid-training 1b and 3b Llama models on D3 prior to supervised-finetuning (SFT) on task-curated instruction data. As described in Section 3, D3 consists of diverse & high-quality examples of general-purpose software synthesis in Python; consequently, we expect task-curated SFT to be beneficial for improving the precision of 33-mid-trainee model generations on problem solving & debugging tasks in benchmarks like HumanEvalSynth and HumanEvalFix.
To that end, we first mid-train 1b and 3b models on D3 for two complete epochs. Then, run a series of SFT experiments in which we tune the base and D3-mid-trained models consisting
on one of three data variants, curated by task: synthesis/problem-solvy of instruction + file-diff-seuuence examples from LinlSeq (Piterbarg et al., 2024);
7

---

Published as a conference paper at COLM 2025
HumannvalSynthve)
MBPP(-)
HumanFvalFixte
Size
MT on D3?
SFT?
Scratch
Compl.
Scratch
Compl.
Debugging
1b
X
x
5.5 + 1.0
0.0 +.0+00 0.0
3.0 + 1.0
0.0 + 0.0
0.0 0.0
1b
V
X
9.2 = 1.0
21.0
12.2
1.0
11.5 1.0
2.7 0.7
1b
X
V
18.0 + 1.1
3.6
25.5 = 1.1
7.7 1.0
5.0 = 0.7
lb
V
V
23.1 1.1
16.1 + 1.0
31.9 1.1
15.6
9.4 0.9
3b
X
X
8.0
0.0
13.4
0.0
0.0 0.0
3b
V
X
11.0
3.17
=
21.6
6.1 0.8
3b
X
V
27.0 11
5.8
34.0 1.2
1422
17.8 1.0
3b
V
30.9 1.1
27
40.2 1.2
1.2
20.8 1.0
Table 2: Full pass@1 results on single-file synthesis & debugging by benchmark at temperature 0.1 ("#" indicates standard error over и 20 samples).
% Resolved
% Apply
Model
Size
D3?
SFT?
pass@1
pass@10
pass@1
pass@10
Ours
1b
0.56 0.15
1.78 + 0.36
36.33
0.60
93.73 + 0.48
Ours
1b
v
0.92 + 0.18
3.00 + 0.45
36.02 0.80
87.85 + 0.70
Ours
3b
x
v
1.15 + 0.19
3.43 + 0.51
55.35
0.81
97.22 + 0.33
Ours
3b
V
1.71 + 0.32
4.81 + 0.56
69.60 0.81
99.62 + 0.17
SWE-Llama(t)
7b
3.01 + ??
66.78 = ??
Table 3: Scaffold-free results on SWE-Bench (Oracle) at temperature = 0.5 (" indicates standard error over 16 samples). "% Resolved" denotes the percentage of GitHub issues correctly solved by the model, while "% Apply" denotes the percentage of generated patch sequences that contain only well-formed file-diffs. "+" indicates scores drawn from Jimenez et al. (2023), reflecting performance after in-domain SFT on the train split of SWE-Bench.
editing, consisting of Python file-state + instruction + file-diff examples from CommitPackFT (Muennighoff et al., 2023); and lastly, synthesis/rooleem-olvingg editing, reflecting mix of examples from both LintSeq-Instruct & CommitPackFT (see Appendix F and Appendix G).
Our experimental results are shown in Figure 4, Table 2, and Table 3. In aggregate, we evaluate all models on six tasks spanning four benchmarks, HumanEvalSynth, MBPP, Hu- manEvalFix, and SWE-Bench (Oracle) (Austin et al., 2021; Muennighoff et al., 2023; Liu et al., 2023). For each setting and model variant, we report scores from the best performing SFT mix only. As in subsection 4.1, evaluate LMs on two variants of each synthesis manthesis/problmm problem- solving task in order to test model ability to both synthesize a solution to programming problem from scratch (denoted "Scratch" in Table 2), and to complete partially-written human solution (denoted as "Compl.").
On single-file synthesis & editing, mid-training models D3 prior SFT improves pass@1 by margins of 3 to 6 points compared to direct SFT on base models (Figure 4, Table 2). Furthermore, observe that the completion variants of HumanByalSynthh and MBPP tasks, fine-tuning models mid-trained on D3 degrades performance over mid-training alone.
On the challenging repository-level code editing benchmark SWE-Bench (Oracle), we simi- gain from mid-training on D3 before SFT. It is important
larly find a statisticalll-siiniff to note that this evaluation setting is somewhat cuttof-diisrrbutionn from both D3 and from all three of our tested SFT dataset variants in two respects: first, models may be provided the state of more than one file in-context; and second, many prompts span tens of thousands of tokens in length, exceeding our mid- mid-training and SFT context lengths (4096). Even so, mid-training on D3 improves pass@10 score on "% Resolved" (the total proportion of solved GitHub issues in the test set) by relative margin of 60% and 40% for 1b and 3b Llama models, respectively. It also especially improves "% Apply" (the fidelity of file-diffs in generations) for the 1b model, improving raw pass@10 by 6 points.
8

---

Published as a conference paper at COLM 2025
In summary, the results of our dual-stage training experiments with D3 suggest that mid- training small language models on the dataset is indeed effective for improving downstream performance across coding tasks. D3 appears to be especially effective for improving the seppestter quality of generations on completion-like synthesis tasks, where models must add code to partially pre-written programs.
Related Work
Language Models for Code Generation Modern language models are pretrained on ter- abytes of code. Programming benchmarks like HumanEvalšynth (Muennighoff et al., 2023) and MBPP (Austin et al., 2021) evaluate models on their ability to solve simple "homework- like" code synthesis problems and constitute core part of generative LM evaluations (Dubey et al., 2024; Achiam et al., 2023). Recently, there has also been increasing interest in training and testing LMS on more realistic coding tasks. For example, benchmarks like HumanEval Fallix (Muennighoff et al., 2023) and SWE-Bench (Jimenez et al., 2023) evaluate gen- erative code LMs on program- and repository-level code editing. While these benchmarks have spurred improvement in LM capabilities, contemporary open-source models continue to lag behind their closed-source counterparts, particularly on the hardest of these tasks (Jimenez et al., 2023). Available open-source resources for synthesis like InS-Instruct (Wei et al., 2024) and Evol-Instruct (Luo al., 2023) focus on narrow single-function generation tasks (similar to HumanEval), while GitHub commit-derived datasets (Muennighoff et al., 2023) lack consistent instruction labels and may fail to capture the granular abstractions used by human engineers during software development. By open-sourcing the D3 dataset & preparation pipeline, we hope to contribute towards addressing these data limitations.
Training on Code Edits Many existing works study data-driven code editing with lan- guage models. Berabi et al. (2021) re-label Gub commit data with error messages gen- erated by a code analysis tool, and train an encoder-decoder transformer on this data. Muennighoff et al. (2023) train LMs filtered GitHub commit and message pairs, while Cassano et al. (2023) use filtered commit data to seed synthetic instruction data gener- ation with ChatGPT/GPT--
More recently, Piterbarg et al. (2024) study edit sequence representations for code synthesis. Their experiments show that fine-tuning on synthetic diffs improves test-time scaling laws for code synthesis across LM parameter scales. Our dataset builds directly upon this work, using the LintSeq algorithm for refactoring filtered, human-written source code files at pre-training scales.
Language Models for Data Labeling & Filtering Since the release of ChatGPT /GPT-4,, has become fairly commonplace to use LLMs to generate and / or augment instruction data (Taori et al., 2023; Chung et al., 2024; Wang et al., 2023b; Chiang et al., 2023). Such methods are closely related to the literature on knowledge distillation and self-improvement (Hinton, 2015; Kim & Rush, 2016). More recently, several works have also explored leveraging LMs for data filtering. Penedo et al. (2024); Lozhkov et al. (2024a) show that prompting strong instruction-tuned models to grade Web-text in pretraining corpora for quality and relevance can result in boosts to downstream performance on domains like math reasoning. Relatedly, Li et al. (2024) introduce Superfiltering, showing that LM-assisted filtering can result in models that outperform the labeler on the downstream task of-interest, unlike distillation. Our procedure for preparing D3 is motivated by these works, and the potential of LM filtering & labeling methods for producing code datasets that balance quality and diversity with limited human labor and relatively low cost.
6 Discussion
We introduced D3, a dataset of 3.6 million instruction-annotater code edits derived from 850,000 human-written and highly-diverse Python programs. The dataset's construction leveraged three key components: pretrained LLM for filtering, the LintSeq algorithm (Piterbarg et al., 2024) for synthetic edit generation, and an instruction annotation pipeline. This comprehensive approach has yielded a dataset with several distinguishing charac-

---

Published as a conference paper at COLM 2025
diff action search tree
verffer roward
intrinkk goal
Dost SeseEch RO impect jac
LLM
proscribed
gos
Human
Figure 6: Towards training LMs for open-ended, goal-directed, & autonomous software development by leveraging D3. Future work might explore following up mid-training on D3-like data with RL or test-time search over self-generated goals and/or or diff action sequences. Our data preparation pipeline could also be used for further study of data-driven methods for improving the exploration capabilities of LMs during generation, beyond D3.
teristics notably its coverage of both complete development scenarios (60%) and partial file modifications (40%), along with substantially larger context windows averaging 2,200 tokens compared to the typical 400-500 tokens of existing instruction data.
Our experimental results validate the quality of LLM-generated components and the effec- tiveness of training on diverse edit sequences generated from Human-written Python source code files. Mid-training LMs on D3 prior to task-targeted SFT produced significant benefits across multiple tasks. These improvements were particularly pronounced in refinement- style synthesis settings, where success hinges on deep understanding of code context and development patterns. This suggests that our edit-based approach captures fundamental aspects of software development that existing synthesis instruction datasets miss.
Limitations While these results are promising, several important limitations warrant discussion. Our current focus on Python, though practical for initial development, raises questions about generalizability to other programming languages with different syntactic structures and development patterns. Additionally, despite the empirical results from our human study and training experiments, our reliance on LLM-assisted filtering & instruction labeling in the creation of D3 may introduce systematic biases and coverage gaps. Fur- thermore, our procedure for generating instruction labels for edit samples post-hoc may introduce systematic goal under-specification into the dataset.
Future Work Looking forward, we see the use of diffs as a fundamental representation opening up compelling possibilities for open-ended software development, as well as for both in-context (Gandhi et al., 2024) and external search (Schultz et al., 2024) in code generation, as shown in Figure 6. Diffs provide an ideal action space for search, and are particularly well-suited for exploring program space incrementally. This representation allows models to decompose complex programming tasks into atomic line-by-line actions that are guided by syntax while incorporating feedback at each iteration - an approach that echoes agile software development practices used by human engineers (Dybå & Dingsoyr, 2008). Future work could explore combining this diff-based representation with reasoning and RL (Gehring al., 2024) to create more adaptive context-aware systems.
These future directions, combined with our current results, suggest that edit-based ap- proaches to code generation represent promising path toward more capable and practical code generation systems. The success of D3 demonstrates the fundamental value of model- ing software development as an iterative, goal-driven process at the data-level, rather than treating it as single-step and purely state-prediction-aasee generative task.
Acknowledgments
This work was supported by NSF award 2339096 and ONR awards N00014-21-1-2788 and N00014-22-1-2773. UP is funded by an NSF GRFP award, KG is funded by an HAI- SAP Grant and an NSF Expeditions grant, and LP is funded by the Packard Fellowship. Experiments were conducted with compute from Stanford Marlowe and NYU Greene HPC. We are also grateful to the Gemini Academic Program and Divy Thakkar for their support.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar arXiv:2303.08774 2023.
Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. Xiv preprint ar arXio:2410.20285, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. Xiv preprint arXiv:2108.07732, 2021.
Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning, pp- 780-791. PMLR, 2021.
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. Can it edit? evaluating the ability of large language models to follow code editing instructions. Xrviv preprint ar Xtx::2312..24550 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ross
Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-fineture language models. Journal of Machine Learning Research, 25(70):1-53, 2024. URL http:// http://mmrrrrrrrr http:///im....rrrrpppss///////// ppppp////e//s///////
0870. html.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd models. Xiv preprint ar Xiio:2407.2178). 2024.
Tore Dybå and Torgeir Dingsøyr. Empirical studies of agile software development: A systematic review. Information and software technology, 50(9-10):833-55 2008.
Joseph Fleiss. Measuring nominal scale agreement among many raters. Psychological
Bulletin, 76(5):378-3)) 382, 1971. doi: 10.1037 /h0031619.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (sos): Learning to search in language. arXiv 2024.
preprint ar
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code 1lms in execution feedback with reinfoncement learning. ar aiv preprint arXiv:2400002089, 2024.
Geoffrey Hinton. Distilling the knowledge in neural network.
arXio preprint
arXtt:1503.02531, 2015.
Jr. Hodges, J. L. and E. L. Lehmann. Estimates of location based on rank tests. The Annals of Mathematical Statistics, 34(2):59--611, 1963. doi: URL https://projecceuclidoooo
Basseed-o---an-eeeesss/ 1214/aoms
1177704172. full.
11

---

Published as a conference paper at COLM 2025
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? Xiv preprint arXiv:2310.06770, 2023.
M. G. Kendall. The treatment of ties in ranking problems. Biometrika, 33(3) 1945.
doi:
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arviv preprint arXiv:1606.07947, 2016.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack:
3 tb of permissively licensed source code. arXiv preprint ar arXiv:2211.15533, 2022.
Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Qizhe Xie, and Junxian He. Instructcoder: Instruction tuning large language models for code editing. arXiv preprint ar Xii::2310.20329, 2023.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: veak-to-strong data filtering for fast Instruction-tuning Xiv preprint ar Xii::2402.00530, 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code gen- eration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL www..eeeooouuiioooqqo1111....
https://openreviiw.
Anton Lozhkov, Loubna Ben Allal, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Finemath: the finest collection of mathematical content, 2024a. URL https: //tggg/nggaaace...../////
co/datasets/ ingginngaaaccco TCBTB/ nemath.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arv preprint ar arXiv:2402. 191733 2024b.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint XxXx:2306.08568, 2023.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint ar XrXi::2112.00114, 2021.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Is self-repair a silver bullet for code generation? arXiv preprint arXio:2306.09886, 2023.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Alí Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2024. URL https://arxiv www.gbs.////..o..ooooo...o.....r /201. 00656.
12

---

Published as a conference paper at COLM 2025
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. ar preprint (XXi::212222113, 2024.
Guilherme Penedo, Hynek Kydlīček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. ar preprint ar arXiv:2406.17557, 2024.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. Training language models on synthetic edit sequences improves code synthesis. arXiv preprint arXiv:2410.02749, 2024.
John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, et al. Mastering board games by external and internal planning with language models. arXiv preprint arXiv:2412.12119, 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprin! ar axX1v:2402.0015, 2024.
Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. doi: 10.2307/1412155 URL http:
//www.jjttee wwt.orgg//oooo.ooo. htgg/saable/4411155
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/1I//lppac. html, 3(6):7, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint ar arXiv:2403.05530, 2024.
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. Zero++: Extremely efficient collective communication for giant model training. arXiv preprint Xix::233 102009 2023a.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764-74784 2023b.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. Xrviv preprint ar Xiix21009099999 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arxiv preprint ar xrXiv:2109.01633 2021.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, 2024.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. doi: 10.2307 3000196.. URL /ttp:://wwwwjj.s.. wttp:/sstoosoosiiie.eooo http:////s.ooooorg/taaa///////// brg/sttable/800195..
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXit preprint arXtt:2405.15793, 2024.
13

---

Published as a conference paper at COLM 2025
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright,, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. Xrviv preprint arXio:2304.11277, 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ar Xio preprint urXiv:2200611877,,222 2024.
14

---

Published as a conference paper at COLM 2025
A Dataset Preparation Prompts
Evaluate the following Python code extract for its potential usefulness for studying Pythan programming up to the competitive programming level.
#14-point scoring system** described Points accumulated based on satisfaction of each criterion:
following
reflects foundational Python programming concepts even its not very useful for solving hard programming problems.
-Add point the extract *oontains some
AAwar another point the extract "*correctly demonstrates examples of how to use common Python libraries**, more advanced data structure algorithm**.
-Add third point the code correctly
Award fourth point the extract reflects Pychon code that is **utstanding its educational value for competitive programming**.
The Python code extract:
(program)
After examining the extract:
Briefly justify your score, up to words.
Conclude with the score using the format: Final score: <total points?.
The grading rubric use for prompting Llama 3.1 70B Instruct to score the quality and relevance of Python source files contents for the study of high-quality software synthesis, used in the filtering stage of D3 preparation procedure. The structure and design of this prompt is inspired by work from Lozhkov et al. (2024a)on LLM-powered filtration of pretraining data for math reasoning.
Figure 7:
You are software engineer adding some functionality and/or documentation to (possibly empty) Python program. Your task to write **commit message that describes the changes that you made that is:
I. imperative tense
2. Succinct, clear; and describes the changes exactly
3. longer than one sentence
4. Does not first person pronouns
5. Professionally written
The original Python program:
(program_ssee _strrstta
The Python program after you made changes to it:
(program_enge end]
After comparing the Python programs above, provide your commit message describing the functionality and/or documentation that was added using the format: Commit message: <commit message>.
Figure 8: Prompt for generating rationales for examples of partial source file synthesis with Llama 3.1 70B. We replace fields annotated as (progr am_sstrt) and (program.end) above with the contents of the source file before and after all atomic diff actions are applied, respectively.
15

---

Published as a conference paper at COLM 2025
You are software engineer. Describe the Python program below using single sencence that is:
1. imperative tense Succinct, clear, and describe the changes exactly Free first person pronouns
4. Professionally written
Python program:
(program)
After examining the program, provide your single sentence description using the format: This is Python program that: <my description
Figure 9: Prompt for generating rationales for examples of full-file synthesis "from scratch" with Llama 3.1 70B. We replace the (program) field with the final file-state.
B Full-Text D3 Examples
<code>
[start of progranmers//wimatrixk..oo
1/4 W
987654321
def solution(matrix_sti
dp [[INF solve(0, len(matrix sizes) - 1, dp, natrix_sizes)
in rangelleen(mtrxxx_szees
return [oltl]leemmatrix_siies -1]
10
12
def solvely, x, dp, matrix):
13
if y == x: dp[y][x] = dp(yl[x]
14
16
17
if + =
18
dply][x] matrixly](e) matrixly](1) matrix(x](1) dplyl[x]
28
21
return dp[y][x]
[end of programmers/w//marrxxxy
</code>
<issue>
Add main execution block to demonstrate the of the optimal matrix multiplication function. </issue>
<code>
prrogrammers//vv/naatttte
<| <|diff >00 -21,0 +22,4 00
nane
'__mein_'''
matrix_sizes [[5, 31, 13, 10), [10, 61]
<diiff|>00 -25, +26 96
difff]>
diff
</dod>>
Figure 10: Full-text version of the D3 example shown in Figure 2. We train LMs interleave atomic diff actions using the use the special token <|diff]> during generation.
16

---

Published as a conference paper at COLM 2025
<codo>
start of nww.ptiionisccinn...ooo agentpy]
import randos
from typing irport Tuple
impore nuzpy
[end oondefll_neeee www__aatioossrrdeneeeeo
</oode>
DQNAgent interacting learning from ervirenment.
<]diff|>28 -4,8 +5,9 88
-class D2VAgent;
"IIterrctss with and learss fron the environment."
Nee Save Functions ***
s4e Agent Interaction Functions se4
<]eie]]>25
-import tazch
-5,8 es
edevice terch.devion "oohhddeiie"""udes0 "oudas0* if corch.oadleiie.wvilleee else "opu*)

2s -15,0 +16,8
def lexen/self, experiences, garma):
states, actions, zowards, next_statos, dones, idss experiences
update network
<|ciff/>2 -5,0 inport torch.optie aptin <diff|>38 -17.8 +10.5 89 actiself,
state torch.froe.ue (archh.ron.mmmpy((tttt))) (loa((). unnsqueeeee teldevico))
Epsilon-greedy action selection
[diff( +13,30
dof inis__.
self,
state size,
Sitioo___2e.
sypste__ype
'don'.
1e-3
update training
TAU
1c-8,
RUFFER S128
(1a5),
5840 nemory, last 188 1889 tinestaps ninibatch size sensible cefault
BATCH S12E
64,
*ekowrgs 4,
DRAATE_EVERY
now often to update the netvork
B015-BATCH_STEE
BIZE BUZEER,SIZE TAU
self.BUFFER $12E
se17. TAU
selr.action size sizi self update__typpp type uppate_type
Network
NOTE: training using SunTrewRealyySuffe feils to converge
Initialize time stop (for updating every UPDATE_EVERY stops)

>>
42,
-38,9 +32 88 randor, sead(seed)
</ciff pas -8,8
der loadiself, filoname):
try: filenam os.path. as.path.bbspthh(ffeeee
except Exception as es prantif "self... class_____ss_.... __ares__________________________ l (filenane) exception: (e)')
def savelself, filenane):
"ilenate path. www.phh.assttthtriieoom
<jiff
printif Anisseif classsifff_.cssss.oooaeeeoo 2006 2.0o___...veell axcept Exception as es peinE(fT cless narn )..sav()) (filanane) exception; (a)")
hssvell: (filenane) fos.stat(fiiennnel.co stteenneel.sst...o 2127/11244
< <iiff
Figure 11: Full-text version of the D3 example shown in Figure 1.
17

---

Published as a conference paper at COLM 2025
C More Examples of Instructions in D3
Add pickle import step_one function and specify pickle executing seccuting_pipellnee function call. Add file author information metadata.
Implement argument parser to handle command-ling arguments and add PyTorch model training functionality. Add settings, data structures, and chases for CSC trainer functionality.
Implement HTML input stream functionality with encoding detection and error handling.
Develop Python program that implements utility classes and functions to parse and resolve network addresses, encompassing IPv4 and IPv6 addresses, from strings, uniform resource identifiers (URIs), and sockets.
Implement initial Convolutional Neural Network structure Van der Pol equation solution prediction.
Add argument -- num-doc-keys to parser_run un and modify create_dos function to include num num_docc keys. Add instance variables for stride and label ____c___aaa__uusss_____lel
radius in BoxDef class.
Add utility functions and skeleton for defining custom Matplotlib colormaps.
Implement str to class function, add Package class with subclassing and abstract methods, and skeleton classes for GalaxyPackagge and ShellSeriptPackage.
Add documentation for command line utility and extract version number from CVS tag.
Add the missing' x' argument to the cli function in the Click command decorator.
Add XSRF token classes and functionality for generating and verifying token strings.
Develop Python program that utilizes the Pytest framework to test the authorization mechanisms and API requests of a TAXII server application, with focus on scenarios including unauthorized access, basic authentication, token-based authentication, and collection-level access control.
Add support for report-to, custodian, lifetime, and default outgoing flags serialize bundle6.
Add function to implement Trotterization for a give list of terms, ordering, time and Trotter number.
Update WaypoinLUpdaterr class to initialize and handle lane waypoints and provide placeholder methods for pose, waypoint, obstacle, and distance calculations.
Add required imports for struct and time modules.
Implement BertAttention, BercLayer, and BertEncoder modules with chunking and checkpointing functionality. Add care imports, environment setup, and key variable definitions for data processing.
Update build vocab method in TransioXLTokenizer provide size information of the final vocabulary and the number unique tokens.
Add Connection class with attributes and methods for serializing to JSON and constructing from a list of fields.
Implement integration and gradient evaluation functions for vanishing and nonvanishing basis elements.
Develop Python program that implements the Pepper compiler and suite of tests to verify its build steps, leveraging mock objects to mimic actual file system interactions.
Develop Python program that utilizes PyTorch Lighaning to construct and train muld-label classification system, then evaluate its performance on a provided medical image dataset.
Add initial response reading function and import wait coroutine from part 5.
Initialize Context class for register and flag manipulation.
Add docstrings to describe program output and data structure.
Add logging configuration to the test environment.
Using Python, engineer program that leverages Bayesian optimization for hyperparameter tuning in a random forest classifier, ultimately employing the optimized model to generate predictions from a test dataset, with the results exported to CSV file.
Implement lane detection image generation and accuracy calculation functionality for CULane and Caltech datasets.
Figure 12: A random sample of source code development instructions in D3.
18

---

Published as a conference paper at COLM 2025
D Qualitative Analysis of Topics in D3
You are coding expert. When ask you to output daca with JSON formatting, please return only correctly formatted JSON without an explanation.
Analyze the following Python coding instruction and identify:
Main topics/concepts
Programming lbraaies/rrmmewoks mentioned implied
Tools or applications involved
Level complexity (beginner. intermediate. advanced) Type task (data analysis, web development, automation, etc.)
Return your analysis a JSON object with the following fields:
libraries frameworks
tools_ paplllccppss
-all complexity
-all_ [al_task_tyy_ types
Instruction analyze: (instruction)
Figure 13: Our prompt format for qualitatively analyzing the contents of D3 via annotation of 100,000 randomly sampled instructions from the dataset with Gemini This prompt is effective for producing only valid JSON outputs from the model. To make the visualization of top occurring topics in section 3, Figure 3 (right), we parse the main. topics field of each output JSON, manually group paraphrased outputs, and sort by topic frequency.
Human Study
E.1 Participant Recruitment Ethics
All annotators were volunteers who self-selected after viewing short recruitment notice. During screening they affirmed at least intermediate proficiency in Python (2 4 years of regular use) and provided electronic informed consent outlining study purpose, minimal risks, data handling, and optional withdrawal at any time. No personal identifiers beyond platform IDs were collected, and participation carried no mandatory compensation. Data reported in the manuscript corresponds only to responses from those volunteers who completed every task.
19

---

Published as a conference paper at COLM 2025
F Mid-Training
We use Pytorch FSDP and the Dolma tokenization toolkit via the oLMo ecosystem to support all mid-training experiments (Zhao al., 2023; Soldaini et al., 2024; OLMo et al., 2024). We add a new token "<Idiff|>" to tokenizer vocabularies, and resize model embeddings accordingly. This special token is used to separate individual file-diffs example diff action trajectories, effectively indicating a frame "reset" in diff integration (see Figure 11, Figure 10, and Piterbarg al. (2024)). Mid-training runs use two to four NVIDIA A100 or H100 GPUs.
Llama 3.2 1b & 3b
Batch Loss Reduction
mean
Batch Size
256 (0.9, 0.95)
Betas
Gradient Clipping
Plash-Altention
true
Learning Rate Scheduler
cosine
Max Learning Rate
le-4
Max Sequence Length
4096
Mixed Precision
BFLOAT16
Total Epochs
Optimizer
AdamW
Warmup Steps
100
Weight Decay
0.1
Table 4: Hyperparameters for D3 mid-training experiments.
G Supervised Finetuning
For supervised fine-tuning, we train models using the Huggingface accelerate and Deep- Speed Zero++ (Wang et al., 2023a) libraries for Pytorch. As in mid-training, we add the special "<|diff]>" token to non-mid-traines accordingly prior to supervised fine-tuning. Examples from all datasets are reformatted to match the format of examples in D3 prior fine-tuning.
model vocabularies, and resize embeddings
Llama 3.2 1b &
Batch Loss Reduction
sum
Batch Size
512
Flash-Attention
true 1
Gradient Clipping
Learning Rate Scheduler
linear
Max Learning Rate
1e-4
Max Sequence Length
4096
Mixed Precision
BFLOAT16
Total Epochs
2
Optimizer
AdaniK
Warmup Ratio
0.01
Weight Decay
0.01
Table 5: Hyperparameters for task-targeted supervised fine-tuning.
20

---

Published as a conference paper at COLM 2025
H More on our Ablation Study (subsection 4.1, Figure 5)
H.1
Computing Seed Corpus Validation Loss
Check if perticulaz is cunning based on its it zumiro, start it cormand. it zurning,
terminate pracess restart by acscuting the precess name comand.
funation
stzz ressage indicating the action taken:
"procoss found. Restarsing speozess name..
You should write arif-contained starting with:
import suborecass
import time
tace, N FORE: str) str:
Engart:
Eapare
time
def taxe, www.ioroceeo.ccooo rane:: etr! str:
Check if the procesa zusning
1r. rumniing anyllpmes for prec In proceees proc.ntme() ss precess_tree]] mrae]]
IF the przeess is ramirg, terninate it
pazz pastii prsemss_iter(1:
proc. nanen) procass_
tiee (5
PPProcass Reaterting (process_ 1am0)."
1lo8:
return PPProosss not fourd. Starting (process_nnms)."
Figure 14: An example of the prompt-completion formatting that use to compute the validation fit induced by source code corpuses before and after the application of our LLM- assisted filtering procedure. Examples are fully formatted Pyceualeee code, with instructions provided as multi-line comments (bolded text). As indicated in subsection 4.1, prompt tokens are masked from validation loss computation.
H.2 Synthesis Benchmark Scores of Diff-Action vs. File-State LMs Mid-Trained D3
In Table 6 and Table 7, we report the numerical scores of diff-sequence vs file-state LMs (Llama 3.2 1b) mid-trained on D3 on HumanEvalSynth and MBPP, respectively. These scores reflect the intermediate mid-training checkpoint with the best test-time scaling curve (i.e. highest best-of-n scores vs. generated tokens across n) on the validation set of MBPP". The full test-time scaling curves for these checkpoints plotted in Figure 5 (right). We use multiple sampling temperatures, E (0,0.1, 0.2, 0.5), to evaluate models.
Scratch
Completion
Size
Diffs?
Pass@1
Pass@16
Pass@1
Pass@16
1b
x
5.63 1 0.41
14.98 1 0.56
15.22 + 0.51
37.94 1 0.64
1b
6.71 1 0.73
17.54 1 0.57
17.01 1 0.52
40.33 1 0.65
Table 6: Scores on HumanEvvaSynthh() ("+" shows standard error over n = 20 samples).
Scratch
Completion
Size
Diffs?
Pass@1
Pass@16
Pass@1
Pass@16
1b
x
4.64 0.20
30.04 0.39
9.29
0.48
28.62 0.70
1b
10.32 + 1.57
33.68 + 0.41
10.23 + 0.48
33.95 + 0.71
Table 7: Scores on MBPP(+) ("*" shows standard error over n 20 samples).
'Note that Figure 4 and Table instead reflect the pass@1 benchmark scores of the last D3-mid- trained lama 3.2 1b checkpoint.
21

---

Published as a conference paper at COLM 2025
I Benchmark Evaluations
We evaluate models on coding benchmarks using instruction-style prompts and no special stop sequences during sampling (i.e. EOS-token termination only).
I.1 Completion-Style Synthesis Evaluations
As described in subsection 4.1 and shown below in subsection I3,, our benchmark eval- uations test LMs on two tasks, synthesis from an empty file ("from scratch") and from partial human-written solution. Our procedure for preparing the latter task is simple. For each example in benchmark test set, we: (1) remove random subset of lines from the body of the provided human-written solution; and (2) check that the resultant code is syntactically correct with the Python code analysis tool pylint, re-sampling until this is the case. Preparation of this task is conducted once for each benchmark, so that all models are evaluated on exactly the same set of partial completions.
Examples whose solutions contain a single line of code in the function body are omitted. Furthermore, if the solution to programming problem contains docstring, we include this docstring in the partial solution provided the LM.
1.2 Examples of Code Synthesis Prompts
<code>
[start of _s_ ____a__prime__deeeee__ hot_prime_solutioo.p//
[start is not aariilsnt_prie___e__ prime_solution.ppl
inport math
[end of is_ not fot__piimssoouttinppppee </code>
def (n__ot_prrime()::
return False
<issue>
Write Pychon function prime(n)' solve the fallowing problem: Identify </issue>
[end of oofs___ne sos_primm_soluiio.py
problem:identify non-prime numbers.
</code>
<issue>
<code>
Complete the Python function is _i_not__rrre_ prit_e
</issue>
<code>
Figure 15: Example formatted prompts used for from-scratch (left) and completion-style (right) synthesis evaluations on the HumanEvalSynth and MBPP benchmarks throughout this paper. We identical prompt-completion formatting in all training and evaluation experiments, including mid-training on D3 as well supervised fine-tuning on all task-curated datasets (subsection 4.2). The examples presented in the figure above reflect a problem from the MBPP test set. Note that for from-scratch-stylee evaluations, we format MBPP instructions match the formatting of those in HumanEvalSynthh
22

---

Published as a conference paper at COLM 2025
1.3
Example Code Editing Prompt
<code>
[start of has close elements solution.py]
from typing Import List
def
has close (numerts List[float], threshold: float) -> bool:
*an Check if in given list of numbers, any two numbers closer each other than thrashold.
>>>
2.0, 3.61, 0.5)
False
>>> has has_ cals_close_saee [aase__eements([1@e 2.8, 3.0, 4.0, 5.0, 2.01, 0.3)
19
True
11
12
for idx, elen enunerate(numbers):
for
enumerate(nunbers).
15
distance elem elem2 if distance < threshold: return True
16
17
18
19
return False
[end of has cllsellleeeeeeeeeeeeeeeeeee basse_elements plements_solutiin.pp]]
</code>
<issue>
Fix bugs in has_close_elements. </issue>
<code>
23


Reviews:
[{"review_id": "iWkqPUlaFK", "reviewer": "Reviewer_Wokx", "summary": "The authors introduce D3 - Diverse Data for Diff-by-Diff Coding - as a new dataset for training LLMs for Python code generation. The dataset is constructed by framing the code generation as a decision-making problem with goals, states, and actions defined by the tasks: goal is the description of functionality to be added, state is the current contents of a file, and action is the generation of file diff. The dataset is constructed from Python source files from the Stack, using code analysis tool and LLM to sample file-diff sequences and labels.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- D3 is a large scale dataset with more than 3.6 million examples \n- The dataset contains reasonable technique to augment the data qualities, including synthetic diff generation, LLM-based data filtering and labeling \n- D3 has good level of data diversity, from syntactic modification to architectural changes, to source files", "reasons_to_reject": "- The experiment results are quite limited with most of the results are on simple tasks such as HumanEval and MBPP. However, the motivation of the dataset is to enhance the model with the ability to perform iterative and exploratory code generation like human software development. Except for limited result on SWEBench, it is not clear how the current results reflect the mentioned model capacity. Even on SWEBench, the model is tested on the Oracle version only. \n- The creation of the dataset is quite clear. However, many components are controlled by LLM e.g. generating labels. However, there is no study in the paper to justify how accurately these synthetic data are generated by LLMs. As this can affect the data quality, it is important that the authors have a human study (even on a data subset only) to examine the quality of the synthetic components in the dataset.", "questions_to_authors": "- Training on D3, how can model perform better through iterative and exploratory code generation like human software development? Can you show empirical results on code generation where the model can improve and refine its code better after training on D3? \n- On SWEBench, the results seem quite low even in the Oracle setting. what could be the reason even after training with large-scale dataset like D3?  \n- The current dataset is for Python code generation. For other languages such as Java, what tools and steps need to be modified to create a similar dataset?", "extra_scores": {}}, {"review_id": "PU7C9ntNf8", "reviewer": "Reviewer_zkKL", "summary": "This paper argues for mid-training a Code LLM to edit code. The edit data is structured as the sequence [original code, NL instruction, diff to original code]. The paper contributes a procedure for generating these edits from raw code data, along with a dataset of 8B tokens worth of edits.\n\nThe evaluation ablates various design choices and shows that mid-training followed by SFT improves performance slightly for 1B and 3B models on downstream tasks (HumanEval, MBPP, SWE-Bench).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A nice approach to dataset generation. It's on a small scale, but it is likely to be scalable.\n2. Results on 1B and 3B models are likely significant. I think it is quite hard to get +5% Python performance by doing anything to Llama 3.2 1B. But, I have questions about what the baseline is (see below).", "reasons_to_reject": "1. The delta that this paper presents over *Piterbarg et al. (2024)* is not that large. From what I can tell, the edit synthesis algorithm is not new at all. What is new is that edits are labelled with instructions, there is a larger dataset that is curated with standard techniques, and new experiments.\n\n2. Baseline results are quite unclear, which makes it hard to judge the significance of the experiments.\n\n   See questions below: I cannot determine what the SFT dataset is that the paper uses. This makes the results hard to process. For example, I believe Meta Llama 3.2 Instruct does much better on HumanEval than what this paper reports. So, I don't think the SFT dataset is the Meta SFT dataset.", "questions_to_authors": "1. What are the task-curated SFT datasets? Please let me know if I missed it in the paper, but the dataset does not appear to be explicitly mentioned in Section 4.\n\n2. Caption for Figure 5 says, \"Base models are evaluated with the D3 prompt-completion structure (see Figure 2)\" Does it make sense ot evaluate the base models with the D3 prompt? Shouldn't they be evaluating using the standard prompt format for HumanEval, MBPP, etc.?\n\n## Notes Taken While Reading\n\nL10: \"sampling synthetic file-diff sequences with a code analysis tool\". Not\nclear what this means at this point in the abstract.\n\nL60: Why is this a point about just validation loss? Does it not also hold for\nthe test sets / benchmarks?\n\nL84: Shouldn't $\\mathcal{T}$ also be indexed by a goal? The actions taken will\ndepend on the goal, just as the reward does.\n\nL105: The referenced figures are far too terse to understand. But, I'm going on\nreading the rest of the section.\n\nL143: OK, I have understood the procedure. Some critique: the sequence of\ndiffs is entirely additive. It also does not support quite common kinds of\nadditions, e.g., \"add an argument to this function\".\n\nLooking back at the prior paper, the delta appears to be that the LintSeq\npaper did not have NL instructions, worked by pretraining very small LLMs,\nand used a very different dataset.\n\nL176: Some basics about batch size and sequence length would be helpful to see.\n\nTable 2: The \"compl.\" columns. I'm not sure whether this is relevant. It is not\nhow the benchmark data is typically used.\n\nL268: \"Our dataset builds directly upon this work, using the LintSeq algorithm\nfor refactoring filtered, human-written source code files at pre-training\nscales.\"\n\nI don't think 850K examples is pre-training scale. Actually, that's the original\nsource -- I think 8B tokens is likely mid-training scale.", "extra_scores": {}}, {"review_id": "LtKDomHq6x", "reviewer": "Reviewer_9pRV", "summary": "This paper introduces a dataset containing 8 billion tokens of instruction + file-state + file-diff- sequence examples sampled from 850,000 human-written Python source files, to improve LM performance on code synthesis, completion and editing. Experiments show that mid-training LMs like Llama 3.2 1b and 3b on D3 prior to supervised fine-tuning (SFT) on task-curated data improves performance on synthesis & editing tasks. On benchmarks like HumanEval- Synth and HumanEvalFix, D3-trained models show improvements in pass@1 of 3 to 6 points compared to direct SFT.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper curates a dataset containing 8 billion tokens of instruction + file-state + file-diff- sequence examples to improve LM performance on code synthesis, completion and editing. \n2. The paper is well-written, where problem formulation and data curation process are clear and easy to follow.\n3. Experiments on several coding benchmarks show D3 improves the performance of small LMs on synthesis & editing tasks.", "reasons_to_reject": "1. Although authors have submitted the codebase of this paper, there is no dataset released or provided. \n2. This paper only focuses on Python, raising questions about its generalizability to other programming languages.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The papaer introduces D3, a large dataset consisting of 8B tokens from 850K Python files. The dataset frames code generation as sequential edits (file-diffs) annotated with goals, constructed by filtering and augmenting code from The Stack using a code analysis tool and labeling with LLM-generated rationales. Mid-training language models (Llama 1b and 3b) on D3, prior to SFT, improves model performance in code synthesis, completion, and editing tasks.

The reviewers praise the approach used to produce the data, its diversity, an the clear presentation. Concerns are mainly about limited evaluation, generalization beyond Python, and evidence justifying the quality of synthetic labels, most of which have been addressed in the rebuttal. 

This paper is ready for COLM as is, but I encourage the authors to add the human evaluation experiment that R Wokx suggested in the revision.
Decision: Accept