Training Language Models on Synthetic Edit Sequences Improves Code Synthesis
Authors: Ulyana Piterbarg
Lerrel Pinto
Rob Fergus
Venue: ICLR 2025 Poster (2025)
Keywords: language model
code synthesis
reasoning
synthetic data
Primary area: foundation or frontier models, including LLMs
TL;DR: Training LMs to synthesize code edit-by-edit with SFT improves the slope of test-time scaling laws

Abstract: Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of sequential edit data. While high-quality instruction data for code synthesis is scarce, edit data for synthesis is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors programs into sequences of synthetic edits by using a linter to procedurally sample across interdependent lines of source code. Synthetic edits sampled with LintSeq reflect the syntax and semantics of their programming language. To test the algorithm, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we fine-tune a series of smaller LMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset. We perform comprehensive evaluations comparing edit sequence code LMs against baselines on HumanEval, MBPP(+), CodeContests, DS-1000, and BigCodeBench. We show that models fine-tuned to iteratively synthesize code match or outperform baselines on pass@1, and exhibit better scaling across higher pass@k as a function of total test-time FLOPs. Finally, we also pretrain our own tiny LMs for code understanding. We show that fine-tuning these models to synthesize code edit-by-edit results in strong performance on HumanEval and MBPP(+) compared to existing code language models of similar scale such as CodeT5+, AlphaCode, and Codex.

Published as a conference paper at ICLR 2025
TRAINING LANGUAGE MODELS ON SYNTHETIC EDIT SEQUENCES IMPROVES CODE SYNTHESIS
Ulyana Piterbarg, Lerrel Pinto, & Rob Fergus*
New York University
ABSTRACT
Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for this is the scareity of sequential edit data. While high-quality instruction data for code synthesis is scarce, edit data for synthesis is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors programs into sequences of synthetic edits by using linter to procedurally sample across interdependent lines of source code. Synthetic edits sampled with LintSeq reflect the syntax and semantics of their programming language. To test the algorithm, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-seen unne a series of smaller LMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset. We perform comprehensive evaluations comparing edit sequence code LMs against baselines on HumanEval, MBPP(+), CodeContests, DS-1000, and BigCodeBench. We show that models fine-tuned to iteratively synthesize code match or outperform baselines on pass @ 1, and exhibit better scaling across higher pass @K k as a function of total test-time FL.OPs. Finally, we also pretrain our own tiny LMs for code understanding. We show that fine-tuning these models to synthesize code edit-by-edit results in strong performance on HumanEval and MBPP(+) compared existing code language models similar scale such as CodeT5+, AlphaCode, and Codex.
tuples. Then, fine-
INTRODUCTION
The successes of language models (LMs) are difficult to overstate. However, consistent and correct zero-shot generation code synthesis remains out-of-reach for all but the largest models (Abdin et al. 2024)) Groeneveld et al.) 2024 Dubey et al., 2024). Compared to other reasoning tasks, this setting has two challenging properties, namely solutions are both long and structured.
Humans tackle problems that have these properties by leveraging abstract mental models, first developing plan for their solution that reflects the setting's structure and then executing the plan one step time (Gopnik) [9822 Kirsh 2009). For example, a software engineer might employ object-oricnted programming when creating a new code-base by developing a "class" object and then complex.
gradually adding new functionality this class their code-base becomes
In contrast, LMs are trained to autoregressively synthesize entire programs from scratch. This makes repeatedly editing program with LM extremely expensive current state-of-the- LM- L.M-oowered code editing tools like Cursor repeatedly prompt models to rewrite entire programs during every edit generation call (Sanger) 2024 L. outputs also suffer from degrading quality as sequence lengths grow and exhibit limited diversity across samples (Chen et al. [202T] [L et al.) 2022b] Roziere et al. 2023 Lozhkov al.) 2024). The consequence of these pathologies is that there does not exist reliable trade-off between zero-shot generation quality and total test-time compute under the current paradigm of autoregressive code synthesis, particularly for smaller LMs.
In this paper, we claim that these issues can be mitigated at the data-level by reparameterizing code synthesis as a sequential edit problem. Rather than training models for single-step generation of entire
We open-source code and models to https://1 http:///iinnsegggttnnn.e fergus]ecs edu.
Contact: (up2021, lerrel,

---

Published as a conference paper at ICLR 2025
Code Synthesis Instruction [t] Prement Deooda,
Raw Languaige Model Output
Repearted Samp fing Humanfivi
Test- Tina-Tima FLOPs
Python funation that mutric setumn thir sum in form a new lst. The matrioss of the sime aine, and each eieneer of the nutpun monta the n.m of the conesponding ring elements A and TB"
'X' cooft reprosemting
(el 4.0 +1,4 68
100
100
-def metris SummA,, Bi:
ros lonty for rangoyows:
80
Resolved Language Model Output Rosorve bulls
csti kenAA00
kmW
oumiA, Bi:
Python AST
c-
adu Gappanddroww)
Repested Bampling MBPP//3: Soore Test Time FLOPs
tox tims computs FLOPG
10"
10
tad time compus FLORG
10tH
101
G=00
Phi-s S, 3.88
Liarn
mangemmont:
cola (on
rangetbo
Wiw.appendWWlII BEID
in rangetools:
c-append(ro)
retur
10
10
10
10"
Tea ETe compusa ETOPs)
1007 FLORg
Figure 1: Code synthesis with LMs trained on synthetic code edit sequences. Left: An example generation from LM trained to synthesize code as stream of linter-error-fiee edits. Right: Training LMs to write code edit-by-edit by preprocessing instruction data for SFT with LintSeq improves test-time scaling laws during repeated sampling, i.e. the percentage of benchmark problems solved by any attempt (pass @k) as a function of total test-time FLOPs compared to training on standard data (see Appendix A.4). Shading indicates standard error in linear fit.
programs, we propose that models he trained to generate code edit-by-edit. This objective has a major obstacle: while datasets of filtered GitHub repository commits like CommitPackFT (Muennighofi) iet al., (023) have dramatically improved the quality of open-source code edit data, they contain limited sequential data. Moreover, the edits in such these datasets reflect the granularity at which programmers save code, but not necessarily the granularity at which they write and/or reason about it.
To address this, we introduce a sampling algorithm called "LintSeq" that can be used to express any program in a training corpus as a sequence of structured code edits. LintSeq leverages linters simple code analysis tools that check programs for errors and stylistic issues to ensure that each generated edit meaningfully reflecs the syntactical structure of the programming language that it written in. The algorithm consists of two phases: a backward phase, which takes a source file as input and samples code deletions from this file to yield possible sequences of linter-error-free intermediate program states; and a forward edit computation phase, which reverses each sampled program sequence, employs the Unix di (Thompson & Ritchie 1975) operator to compute deltas between consecutive versions of each file, and outputs the generated edit sequences. LMs trained on data sampled with LintSeq synthesize code by repeatedly predicting insertion edits to files.
To test the impact of training LMs on synthetic edit sequences sampled with LintSeq, we conduct a series of supervised fine-tuning (SFT) experiments. In each experiment, we compare the performance of models trained on a corpus of example programs re-sampled into synthetic edit sequences with LintSeq to those trained on the original dataset. We evaluate LMs zero-shot and without ohain-of- thought on HumanEval (Chen et al. 202I), MBPP (Austin et al. 202T). DS-1000 (Lai et al. 2023), BigCodeBench (Zhuo al. (024). and CodeContests (Li et al. (022b) on "pass@ k," the proportion of problems solved by any attempt given "k" tries. Our results show the following:
1. Across models ranging in scale from 150M to 14B parameters, training LMs to iteratively synthesize programs improves the diversity of model-generated code compared to training on standard instruction data, while either preserving or improving code quality.
2. The improved diversity of generated programs means that pass @ performance increases faster as a function of test-time FLOPs, allowing for a better trade-off between the two.
3. Ablating the linter from edit sampling during data generation hurts the downstream quality of programs synthesized by edit sequence models.
2 LINTSEQ: CODE SYNTHESIS AS A SEQUENTIAL EDIT PROBLEM
The key to solving a hard problem often lies in knowing how to decompose it into sub-problems. LintSeq is an algorithm for synthetic data generation that decomposes programs in training corpuses across insertion edit chunks that reflect the syntax and semanties of their programming language. To sample such chunks, it uses a code linter. The algorithm is inspired by recent work on discrete diffusion methods for text generation, where decoding is non-autoregressive (Li et al. 2022a).
2

---

Published as a conference paper at ICLR 2025
LintSeq Paeudo-code
Phose Baokward Sompling Fae 1e143
Phase Forward Computation
Pythen AST
Lietsee beckmand wwwpiinggeeeeeee Linteris
(progras)
unis. dirrie,
Line, to, Selate rardon, select, sererrennnn DrogERL
Sintce CEEUES
linter (peogpeni
retar
wille Hone,
for Line delete lines Linter acceest
anpte():
Use Litte Ideniity all Systactioally Dependent Lines
untx dErf(a, +]
Hnne,, www.su_lltttrrrreee Frrece a Hinter. (attgggg)) progran 3E2 appereesoooggtee
c52 1an148013
reture
KEES sesaence 11 (nghh(aragaaneeee
veture
In Ino peograns 111
sESS urs..ddtiiorgpee sectll, p0ggr1m_
cerpty>
retsen edit, 50928109
Figure 2: LintSeq:
Training LMs to write code edit-by-edit with supervised learning by gen- erating synthetic data. LintSeq decomposes existing programs into synthetic edits that reflect the syntax & semantics of their programming language. At each iteration, the algorithm samples an edit chunk from program by: randomly selecting line of code to delete; identifying the minimal set of lines that are dependent on this line with a code linter; and finally, removing the line and its dependents. These steps are repeated until all lines of code have been removed. LintSeq then processes the reversed sequence program states with Unix-diff express it as a sequence of edits.
Informally, the hypothesis underlying LintSeq is as follows: by training LMs to synthesize code edit-by-edit on large-scale datasets, we can potentially achieve a better trade-off between generation quality and test-time compute while still benefiting from the training and sampling efficiency of autoregressive language modeling. In this section, we define important terms, provide formalism for the edit sequence re- parameterization code synthesis, and formally introduce LintSeq.
2.1 DEFINITIONS
We define a linter to be a static code analysis tool that scans source code for defects. Linters can identify code that is objectively incorrect, throwing errors if and when program contains syntax errors or refers to non-existent variables or packages. verifier, linters may return false positives, i.e. they may be unable to detect more complex errors, particularly in dynamically typed programming languages like Python or JavaScript.
important to note that unlike a formal
For given source file, define an intermediate program stute to be a program that contains only subset of the line-by-line contents the original file, such that the order of these lines is preserved. We call an intermediate program state linter-error-free if checking this program with an appropriate linter produces exactly the same eиror trace(s) as those output when checking the original source file.
2.2 REPRESENTING CODE WITH EDIT SEQUENCES
We operate in the textual supervised learning setting in this paper, where we have access to a code dataset D of N example programs y, each of which may be optionally paired with a corresponding natural language instruction x that describes the program's 's function, i.e. {(x), >'))
Let ((-,) :) denote the Unix diff operator TTompson & Ritchie 1975), which computes a text difference between a pair of strings by performing a by-e-by-line matching and returns a summary of the detected differences. The fi operator is implemented by popular version control and development systems to help programmers track edits between versions of text files. A single edit computed with the diff operator may consist of multiple line deletions and/or line insertions.
Fix program y in the dataset D. Consider a sequence of oy of j text strings corresponding to programs that terminates at y, oy (yI..... ........ 1j-1, 3). We can equivalently re-express o, an edit sequence 6, of length j by first computing a diff between an empty program e and the first program in the sequence, and then computing diffs between all pairs of consecutive programs, shown below.
5, (A(e, vn). (vn v2), A (3), ... vs)....
A(y5-1,9)
(1)
3

---

Published as a conference paper at ICLR 2025
If 'D' is dataset such that for every pair (z.y) v) E D, there exists a pair (z, (z, D', then say that D' is an edit sequence refactoring of DD
2.3 GENERATING LINTER-GUIDEL SYNTHETIC EDIT SEQUENCES
Recall from above that single program edit computed by the diff operator A(--) can consist of any number of deletions and insertions. LintSeq is an algorithm for computing edit sequence refactorings D'' such that all data (x, 5y) E DD' have particular property: every edit in 6y consists of insertions only. There are two phases in LintSeq: backward sampling phase that is used compute program state sequences y, and a forward edit sequence computation phase that is used to re-express oy edit sequences 6,. Pscudo-code as well as visualization of each of these phases is provided in Figure 2 Full examples of edit sequences generated with LintSeq are provided in Appendix (Figures) and 10).
Phase I: Backward Sampling In the backward sampling phase of LintSeq, for each of the N pairs (x, v) DD we generate sequences of intermediate program states αy that begin with the empty program and terminate at the original program y. These sequences are generated in reverse or backwards using a simple procedure that we dub linter-guided sampling. Starting with the program y, sequentially generate each predecessor program in oy from its successor by following these steps: (1) delete a line from the current program by sampling uniformly at random; (2) run a linter or other verifier on the remaining code; (3) if the deletion induced new errors, remove all affected lines; and (4) repeat steps 2 and 3 until no errors are caught by the linter. We repeat these steps until all lines have been removed from the original program y. at which point σy has been generated.
Phase II: Forward Edit Computation Once program state sequences oy have been generated for each (x, y) E D, we run the forward edit computation phase of our algorithm. In this phase, we apply Equation from above to compute program that was added to yy. we use the di ff operator to compute edits between each pair of consecutive programs in Fy up the original program y. Finally, we pair each edit sequence 6, with its instruction æ (if present) to yield an edit sequence refactoring D' of D with size sN.
edit sequence 5, for each y. Starting from the last
2.4 PROPERTIES OF LINTSEQ DATA
Synthetic edit sequences generated by LintSeq have a few other important properties. Let 6, be an arbitrary j-length edit sequence in D' generated with LintSeq, 6, = (A(E, (1)..)).... A(y- First, we observe that there is a simple correspondence between 6, and the original program y used to generate it: y can be re-constructed by starting with empty program, and successively applying each edit in ō, to this program one-by-one. In other words, the edit sequence 8, resolves to Furthermore, by construction, every prefix subsequence of ô, resolves state of y that is cnteererrror-fees (see Section 2.1). These two properties, in conjunction with the uniform sampling step used in the first phase of the algorithm, show that LintSeq samples s examples across all possible linter-error- cetee-error-feee free sequences of line insertions that can be used to sequentially write a program y from-scratch.
intermediate program
We show an example of program synthesis dataset statistics before and after LintSeq processing in Appendix (A] (Figure (6). In the worst case, re-expressing program as an edit sequence increases the length of training example by token count that is constant in the number of program lines]]
2.5 PRACTICALITIES OF TRAINING LANGUAGE MODELS ON LINTSEQ DATA
LintSeq can be run on any code data. It is agnostic to the contents of program, and only depends on knowledge of the language that program written in, and the existence of a linter for this language.
We use teacher-forced supervised learning (Williams & Zipser 1989) to train models on LintSeq data, concatenating edit sequences into single string by interleaving edits with special tokens, losses over the resultant sequences. At test-time,
"< /dift >," and computing instruction-condiiionn fine-tuned models can be prompted to synthesize programs with edit sequences by appending these special tokens to the ends of prompts. More details are provided in Appendix B
See Appendix for details.
4

---

Published as a conference paper at ICLR 2025
Synthetic data generation with LintSeq is controlled by single hyperparameter: the number of edit sequences s that are sampled for each example in the source code dataset D. Edit sequence sampling can be constrained to avoid repetitions.
3 EXPERIMENTS
To study LintSeq and the impact of re- parameterizing program synthesis as sequential edit generation problem, we conduct set of supervised fine-tuning (SFT) experiments. These experiments study code synthesis in Python and are designed to answer the following questions:
How does fine-tuning tiny code LMs to generate programs edit-by-edit with supervised learning impact performance on benchmarks compared fine-tuning on standard code data?
Do performance improvements hold for "off-the-shelf" LMs and on harder coding bench- marks? Do they hold across model scales, tokenizers, and families?
How does ablating linter-guidance from LintSeq impact test-time performance?
Similar to previous works (Chen et al., 2021 we evaluate models by computing "pass@k," the probability that at least one of "k" generations for a problem passes all of the unit tests.
3.1 PRETRAINING TINY LMs FOR CODE UNDERSTANDING
We begin our investigations by pre-training two tiny decoder-only transformers, TinyCodeLM- 150M and TinyCodeL.M-400M,, MM-00M, for Python code understanding on 72 billion tokens of text. Pretraining our own language models grants us data contamination- free test- to study code synthesis with edit sequences, rapidly evaluate LintSeq, and broadly re-examine the trade-off between test-time compute and generation quality in code synthesis for models that can be updated on-device.
We rely open-source data and libraries to pretrain our models (Penedo et al. 2024 Lozhkov et al. 2024] Soldaini et al. 2024 Groeneveld et al. 2024). Our pretraining data mix is inspired by Code Llama (Roziere et al. 2023). and reflects code-skewed mixture of web text and raw Python sampled from FineWeb and The Stack, respectively (Penedo et al.) 2024 Li et al. 2023). The architecture of our models respectively mimics the two smallest versions of GPT-2 (Radford et al. 2019), but integrates the transformer architecture changes proposed by the OL framework. This includes the absence of bias terms and the addition of non-parametric layer norms (Bal 2016), as well as the use of SwiGLU (Shazeer 2020), rotary positional embeddings (Su et al. 20244, and the GPT-NeoX-200 tokenizer (Black al., 2022). We train both models for epochs with batch size of 524,288 tokens on an NVIDIA H100 node with four GPUs. Our experiments are supported by Pytorch FSDP
(Zhao et al. 2023). More details on our pretraining procedures are in Appendix
3.2
GENERATING A SYNTHETIC DATASET WITH LINTSEQ
To support our fine-tuning experiments, we prepare baseline dataset of paired instruction and program data. We then re-express the programs this dataset as code edit sequences with LintSeq.
To that end, we first pool the Python portions of two open-source instruction datasets for code synthesis: the GPT 3.5dd-based Magicoder instruction dataset and the StarCoder2-15B--aat alignment training dataset (Wei et al. (2224ba) a). These datasets are generated with the Instruutt approach by Wei et al. (2024b) and have undergone decontamination for the benchmarks that we evaluate on in this paper. We conduct de-duplication on the pooled data to check for repeated examples. Furthermore, we strip any ch chain-of-thoughh--le thooghht-likee natural language explanations from completion data. The resultant dataset has over 88,900 instruction+progrann pairs.
self-
With our baseline dataset prepared, we run LintSeq to generate = 5 synthetic edit sequences for each
pair. As described in Section 2.5 we concatenate each synthetic edit sequence into a single string by interleaving consecutive edits with a special reserved "edit" token. Inspired by Muennighoff et al. (2024), the popular Python linter pylint to guide edit sampling during generation. Examples of generated edit sequences and experiments testing the effect of varying in Appendix
do not restrict against edit sequence repetitions. We use
5

---

Published as a conference paper at ICLR 2025
Table HvmanEval and M+PP(+) results for TinyCodeLMs after SFT vs existing code models of similar scale (S 0.4B parameters). Scores annotated with "I" indicate external model evaluations that we ran using the procedure described in Appendix [ and all other scores are as reported by model authors. We list models in order of increasing EvmanEval pass @ and report standard error in computed score. Sampling hyperparameters are listed in Appendix C.4
HumanEval
MBPP(+)
Open:
Model
Size
pass@
pass 10
pass @
pass @ 10
AlphaCode
89M
4.3
12.2
Codex
85M
8.2
12.8
SmolLM-Instruue
135M
7.7 + 0.8f
14,5 + 1.0f
10.1 = 1.8f
14.6 + 0.51
TinyCodeLM-Insee M-nntruut
150M
2.3
13.5 1 0.6
11.5
21.6 0.4
TinyCodeLM M-sssruul
400M
11.3 + 0.9
18.5 + 1.1
15.5 2.1
22.2 0.5
SmolLM-Instruce
360M
19.3 L11
19.4 = 2.4
23.1 0.5
AlphaCode
302M
11.6
18.8
CodeT5+
220M
12.0
20.7
TinyCodeLM-LintSeqqll
150M
12.8
2.6
20.6 + 1.1
13.6 2.1
24.4
Rodegen-Mone
350M
12.8
23.1
9.4 1.8
15.2
Codex
300M
13.2
20.4
Tiny MenyCooeLL-LLLnSSelnstttte
400M
13.4
1 2.0
20.9
1
1.1
19.4
2.4
29.9 0.6
3.3
TRAINING LANGUAGE MODELS ON LINTSEQ EDIT SEQUENCES WITH SFT
Next, we probe the impact of training autoregressive LMs to synthesize full programs vs. program edit sequences according to natural language instructions. Aside from the tiny code LMs described above in Section 3.3.1 we also finetune small LMs from three different model families. ranging in
scale from 2.6B 14B parameters. We evaluate tiny code LMs HumanEval (Chen
2021
and MBPP (Austin et al. 2021). and small LMs on the additional challenging benchmarks DS-1000 1000
(Lai al. 2023), BigCodeBench (Zhuo et al. 2024), and CodeContests (Li al. 20226).
Using both the refactored and baseline instruction datasets described in section 3.2 we run pairs of SFT experiments with six different models. each experiment pair, we finetune an LM on both datasets for an equal number of optimizer steps and with the same learning rate schedule, saving intermediate checkpoints throughout fine-tuning. Then, we compare the benchmark performance of checkpoints across sampling temperature: performing
prompt tuning. A more detailed description of the computed metrics as well as a full specification of the evaluation and fine-tuning procedures is provided in Appendices C and B
3.3.1 TINYCODELM
We run our first two pairs of fine-tuning experiments on TinCCodeLM-150M and TinyCodeLM- 400M. Our experimental results are summarized in Table[]] where we compare the remperature-tunee performance our models on HumanEval and MBPP(+) to the pass@) I and pass 10 scores of existing LMs with similar parameter counts.
For both the 150M and 400M parameter versions of TinyCodeLM, we find that fine-tuning LMS to synthesize code with edits via LintSeq data results in stronger benchmark performance compared to the baseline, improving HumanEval pass@) by 41% (9.1 + 12.8) and 19% (11.3 + 13.4) and MBPP pass @ by 18% (11.5 + 13.6) and 25% (15.5 + 19.4). We see a similar scale of improvement on pass 10 for both benchmarks. Our smaller LintSeq model is particularly strong for its size, roughly matching the performance of several models with larger parameter counts (Table
3.3.2 GEMMA 2, PHI-3, AND LLAMA 3.1
The results above raise few questions: Do performance improvements from fine-tuning LMs to synthesize code with edit sequences also hold for language models that were not specifically pretrained for code understanding? Do they hold across model scales, architectures, and tokenizers?
TTo process the generations of edit sequence LMs into executable programs, simply resolve each of the predicted code edits one-hy-one. This procedure is visualized Figure and described in Appendix B.2

---

Published as a conference paper at ICLR 2025
LintSedinstuct (ours)
Instruct (Basellin))
HumanEval Pass@1
MBPP Poss@1
DS- 1000 Paestrt
Neen Aosi Eash frm-frmm //tSeg: +3.3
Maan Abell Gan irsm Lirttieg +4.3
Abal trom Lintsees +311
Gain from LittSe:: -0.5
LittSe
Gain Lintting
-0.6
+2.
+2.
5oore (222s11)
93-
Score Rosazaa
80-
10-
+1.2
2 20
2--0-00
Liora 2.100 90
PE-
14-3-44
Gormo
Ph-330
PN-3146
Ph-3-14
HumonEvall Possar10
MBPP PaesA/10
BigCodeBench [Instruo] Passe1
Merm Abed Gan from Lirilers Win Linteeq
Maas Aboi. Gun to Lanzog +6.6
klean Abe. Casn from Listieg: Ninn LintSeq
Goore paus
+17.3
+18.5
70-
.0.0
+0.0
+3.6
20-
50-
10
10
+0.9
Oe P163-00 Lamed 140
Pe-544
Gemma 0-20
Ph-0-30
Uare-51-30
Ph-3140
7i-3-14
Figure 3: HumanEval, MBPP(+), DS-1000, and BigCodeBench (Instruct) results for Gemma Phi-3, and Llama 3.1 models after SFT on LintSeq (indigo) vs standard Python code (grey). On FvaanEval and M+PPP(+,, tune sampling temp., top-p, and min-p over 1,, 1.2). {0.95, 1.0), and {0, 0.05}, respectively with complction format, temperature 0.2, top-p 0.5, min-p = and n 40, following Wei al. (2024b) and Luo et al. (2023). On BigCodeBench Instruct, we evaluate with greedy decoding (Zhuo al. 2024). Error bars on HumanEval and MBPP scores show standard error.
64 samples. On DS-1000, we evaluate models with the
To answer these questions, we conduct four additional pairs of SFT experiments on LMs from three model families, Gemma 2, Phi-3, and Llama 3.1. We use pretrained-only model weights, if available. The selected LMs range in size from 2.6B to 14B and were trained on general- data mixtures Gemma Team al. 2024 Abdin al. 2024 Dubey et al. 2024).
Our findings align with those presented in Section 3.3.1 As shown in Figure 3 LintSeq improves performance on each LMS for all but two of the metries visualized here (HumanEval pass@ 1 and BigCodeBench Instruct greedy pass @ass@) 1). Notably, even on these metric, the least performant LintSeq instruction-tumie models still achieve performance that comparable to the baseline, i.e. within standard error of sampling within a percentage point. In aggregate across models, LintSeq improves HumanEval, MBPP, DS- 1000, and BigCodeBench Instruct pass @ by an average absolute gain of +2.3, +4.3, +,.1, and +1.1 in score compared to baseline SFT.
Furthermore, as shown in Figure Iright) and Figure 4 the degree by which edit sequence LMs outperform baselines HumanEval, MBPP, and CodeContests increases with repeated sampling for all tested models. In each of the plots included in these figures, we show the total proportion of benchmark problems solved by SFT-ed LMs on any attempt given "k" tries test-time compute used during repeated sampling. By comparing total test-time compute across model variants, we account for the slight difference between LintSeqInstruct vs Instruct model generation lengths due to the extra "diff" descriptor tokens used by edit sequence models. Even after adjusting for these extra tokens, LintSeq consistently improves the relationship between total test-time compute and performance on code synthesis, supporting the hypothesis posed in Section 2
function of total
In summary, the results of these experiments suggest that refactoring code tuning data into synthetic edit sequences with LintSeq is code- cedd-prettaining-. scale-, architecture- and tokenizer-indeeendee independent mechanism for improving the quality and diversity of LM outputs on code generation tasks.
3.4 ABLATING THE LINTER FROM LINTSEQ
The backward sampling phase LintSeq uses a linter to decompose code across edits whose contents reflect the syntactical structure of its programming language. We conclude our experiments by testing the importance of this design choice with TinyCodeL.M models: does fine-tuning on sequences of (entirely) randomly sampled code edits hurt model performance on HumanEval and MBPP(+)?
To test this, we replace the backwards procedure described in Section 2.3 with fully random sampling; during each step of the algorithm, first sample the number of lines to delete from the current.
7

---

Published as a conference paper at ICLR 2025
Lintsegintruct lours
.B. Instruct (baselinc)
HumanEvval Score vs. Test-Time Compute (FLOP()
MBPP(+) Score vs. Test-Time Compute (FLOPs)
Gemmen22 2.88
100
Phi-3 148
Gemme-2 2.88
Ph-S, 148
50
sohked
25
-0
50
20
101h
10
10
10
10
10
10"7
(FLOSS)
FLLOS)
test compute FLOP))
compute FLOPg)
CodeContesta Score vs Test-Time Compute (FLOPs)
Gemma-2, 2.89
Pei-g, 3.89
Ukmm
Phi-s,
20
A8 solwe
schoel
10
10
1010
10
10
10
10
10'a
leat-Smre compatte (FLOS)
Neat-mme compute (FLOP))
lost-lime compute (PLOPs)
test-time LOP
Figure 4: Repeatedly sampling from models SFT-ed generate edit seqs. vs full programs: we compare the best pass @k score achieved by modulating sampling hyperparameters for LintSeqInstruct vs Instruct models. On HumanEval and MBPP(+), we use the same values as in Figure 3 while on CodeContests, we sweep over temperatures (0.5, 0.6} and use top-p = 1.0, min-p 0, and n = 128. We then plot benchmark score as function of the total cost of repeated sampling from each model in FLOPs (see Appendix A.4). Shading shows standard error in linear fit. See Figure Tigure[for for Phi-3 3.8B and Llama 3.1 8B test-time scaling with repeated sampling curves on Human and MBPP
program uniformly at random, before sampling set of lines with the desired count. We refer to this algorithm as "RandSeq." Using RandSeq, we generate a new synthetic edit sequence dataset with the same size as the LintSeq dataset used in all previous fine-tuning experiments. The average number of edits per example in this dataset (= 3.9) is similar to its linter- linter-guided guided counterpart (== 3.8)
We employ the same procedure as the one used in Section 3.3 SFT TinyCodeLM models on the RandSeq dataset. In Figure []efft, we compare the pass @ HumanEval and MBPP score of LintSeqInstruct vs RandSeqInstruct models at high temperatures. On both benchmarks and models, ablating the linter from LintSeq hurts performance with statistical significance, reducing HumanEval pass@1 1 by 30% (6.4 + 4.5) and 29% (8.4 + 6.0) and MBPP pass@1 I by 24% (8.6 + 6.5) and 28% (14.2 + 10.2), respectively. These results suggest that the linter-informed structure of edits in LintSeq fine-tuning data does improve model performance.
In Figure righht) conclude our analysis by probing whether training models linted edits has an effect on the total proportion of syntactical errors in completed programs. To assess this, we run the Python linter pylint over the full set of generations sampled at temperature = 1, top-p = 1, and min-p = 0, checking each generated program for syntax errors with this linter. LMS trained on randomly sampled edits appear generate "buggy" code with much higher frequency than all other models on both HvmanEval and MBPP(+). Furthermore, on HvmanEval, we find that LintSeq models synthesize programs with linter-errors pass @1 1. This additional finding suggests that model performance gains from LintSeq cannot simply be attributed to improvement in low-level correctness of generated code training on refactored code must be helping models write generally better, more diverse programs.
higher frequency than baselines, despite their higher
4 RELATED WORK
Foundation Models for Code Code synthesis is one of the oldest problems in computer science. Neural language model- approaches such as Codex, AlphaCode, CodeT5+, CodeGen, StarCoder, and Code Llama have recently proven to be extremely competitive with previous methods (Chen let al. 2021 Li et al. 2022b] Wang et al. 2023b Nijkamp et al. 2022 Li et al. 2023 Roziere et al. 2023). Today, foundation models trained web text and code data dominate, and I.L.-powered
code editing tools like Github Copilot and Cursor are used by thousands of engineers every day
(Heaven 2024). Many general-purposs LLMs are also trained on code data. While the largest of these LLMs show strong performance on coding benchmarks, generations continue to suffer from limited
Note that both datasets also have similar size total training takens (e= 18 10 TinyCodel M tokens).
8

---

Published as a conference paper at ICLR 2025
Untteerratnu
HumanEvvel Pawad11 Mesn Aboll Oue from
Mean Alov. Cain from
MBPP))) Pasee11
HumonnEval %s Linb- EmrorFrree Samples
MBPP|e| 1s Litt- Error-Free Samples
Fanddod -22
Fandieco:
100
100
16
7.5
-244
50010 pozsskT
4.5
75
75
5.0
10
25
0.0
TeyCoddLM
TeyCocoLM TinyDoddi-----
ThyCsoNLM-ICCM
Figure 5: Left: HumanEvall and MBPP(+) pass achieved by fine-tuning TinyCodeLM models on linter-guided (LintSeq) vs randomly sampled (RandSeq) code edit sequences, We tune sampling parameters over the same values in Figures and 4 and report the best scores for each model. Right: Comparing total proportions of generations with lint errors. Error bars show standard error.
meaningful output diversity, prompt sensitivity, and degrading quality long- cong-contexts (Achiam ctt al. 2023 Gemini Team et al., 2023 Dubey et al. 2024). Smaller models also lag behind (Abdin al. 2024 Gemma Team al. 2024 Ben Allal et al. 2004)). As of the writing of this paper, directly prompting LLMs to generate code "difis" results in low quality edits across models (Sanger) 2024). We claim that this is the result of data problem and we attempt to address it in this work.
Finetuning on Synthetic Data LLM post-training methods like supervised finetuning have been shown to be extremely powerful for improving model performance across tasks (Wei al. 2021).
However, high-quality datasets of paired instruction-respone examples are extremely expensive to curate. One possible solution lies in synthetic data generation methods like Self-Instruct, wherein an LLM is prompted to generate instructions and/or responses from examples (Wang et al. 2022).
Such data have been used extensively for improving LLM performance through self-refinement and/or knowledge distillation on coding tasks (Chaudhary [023 Roziere et al.] 2023 Abbin et al.)
2024| Lozhkov et al.] 2024). We employ post-processed instruction data for code synthesis created with a method from this family, OSS-Instruct (Wei et al. 2024b), as the base of our experiments on re-factorizing code with code edit sequences via L LintSeq. Unlike Self-Instruct-like synthetic data generation methods, our algorithm does not employ an LLM for data generation, and instead generates examples of error-frec edit sequences from existing code data by using a simple linter.
Many works have studied edit generation with language models. Yin et al. (2018) cast the edit representation problem can learn to capture the structure and semantics of edits, while Gu et al. (2019) introduce a partially autoregressive model for generating insertion and deletion edits that trained with adversarial imitation learning. Guo et al. (2021) use reinforcement learning to train LMs to generate code with "holes" that represent high uncertainty tokens, and to edit the contents of these "holes" later on.
Training on Edits
autoencoding task and show that neural network models
More recently, several works have investigated finetuning off-the-shelf pre-trained language models on large-scale edit data. Berabi et al. (202I) use a linter to detect errors in code, and finetune T5 model (Raffel et al. 2020) to correct code by leveraging error messages. MMunnighofff et al. (2023) and Cassano al. (2023) instruction tune models on datasets of GitHub commits pairing code changes with human instructions. Relatedly, Li et al. (2024) use GitHub commit data sourced from Python repositories to generate code editing instruction data with GPT 3.5/ChatGPT. All of these works specifically focus on hetter-equipping LMs for natural language-
code editing tasks, in which a model is explicitly prompted to generate an edit in response to an error message or a natural language specification. Our work differs in three important ways: first, we study edit sequences rather than single edits; second, we train LMs to predict edits implicitly during code synthesis; third, our synthetic edit generation algorithm does not rely the existence of any kind of commit data.
As the capabilities of LLMs have improved, so to have those of small language models. Recent projects like SmolL.M (Ben Allal et al. (2024) and OpenELM (Mehta) et al.) 2024) re-examine the potential of tiny language models that can be run and even updated "on-device," i.c. on a smart phone or laptop. The representations learned by such models during pretraining are weaker than those of scaled-up LLMs (Kaplan et al.) 2020). This is particularly true for harder tasks that involve reasoning. such as code synthesis (Gemma Team et al.) 2024
"On Device" Language Models
Abdin al. 2024). To our knowledge, the most recent open-source work studying small language models pretrained entirely for code understanding is from several years ago (Xu et al. 2022 Nijkamp
et al. 2022 Wang al. 2021 2023b). The 150M and 400M parameter TonyCodel.M models

---

Published as a conference paper at ICLR 2025
pretrained in this paper belong to the "on device" model family and build upon previous works. These models provide an efficient test-bed for experiments on LM code synthesis that is updated to recent advancements in high throughput pretraining and to improvements in open- data quality.
Scaling Up Test- Time Compute The performance of language models can be boosted during inference by using scaled-up sample counts, hand-engineered prompting schema, and/or search (Brown et al. 2024 Snell et al.) 2024). These methods dramatically increase inference costs. Their effectiveness is tightly linked to the expressivity of learned model representations and the diversity of outputs across samples. Our experiments with smaller language models are inspired by these works we study whether it is possible to (1) improve the expressivity of representations for code synthesis across LM parameter scales during finetuning, and (2) take advantage of this property to improve the inference-time performance smaller LMs by larger margins during repeated sampling.
5 DISCUSSION, LIMITATIONS, AND CONCLUSION
This paper introduces an algorithm, LintSeq, for generating synthetic code edit sequences from existing programs. LintSeq enables code synthesis to be re-parameterized at the data-level as sequential edit generation tasks. The algorithm is parameter-free, requires only CPU to run, and makes no assumptions about the content structure of source code files.
Re-par ee-paaamtteriiigg code generation with edits has a few immediate benefits. For example, it makes code generation with LMs more controllable the prompt-level (Appendix [B.3) and it reduces the cost of predicting useful and syntactically correct code insertions with models, since synthetic edit-trained LMs do not need be prompted re-generate full programs from scratch (Section 2.5
In our experiments with LintSeg, we also show the following:
1. Tiny LMs pre-trained for code understanding can be efficiently fine-tuned synthesize pro- grams edit-by-edit via LintSeq data. This results in competitive performance on HumanEval and MBPP(+) compared existing code LMs of similar scale (Sections 3.1 and B.3.1p.
2. On larger models from the Phi 3, Gemma 2, and Llama 3.1 families that were pretrained for general natural language understanding, tuning on LintSeq data either improves or preserves the quality of pass @1 generations compared standard tuning (Section 3.3.2).
3. LintSeq also improves test-time compute scaling laws for code synthesis on instruction fine-tuned Phi 3, Gemma 2, and Llama 3.1 models, suggesting that edit sequence LMs consistently generate more meaningfully diverse programs compared to baselines, even on challenging benchmarks like CodeContests (Section 3.3.2).
4. Ablating the linter from LintSeq hurts the quality and syntactical correctness of code synthesized by edit sequence TinyCodeLMs. This suggests that the structured nature of edits sampled with LintSeq is important for downstream LM performance (Section 3.4
There are several limitations to our work.
First, as currently formulated, LintSeq can only be used to generate synthetic sequences of insertion edits. This is a consequence of the parameter-free nature of the algorithm every edit in LintSeq sequence reflects an existing line of code in the source file used generate it. As result, models that are fine-tuned exclusively on data sampled with LintSeq cannot be used for code editing tasks involving deletion edits. One simple way to circumvent this limitation might be by mixing LintSeq synthetic edit sequences with human edit data during instruction fine-tuning via datasets like CommitPackFT (Muennighoff et al. 2023)., which contain examples of deletions. An alternate approach might be to follow-up supervised instruction fine-tuning on LintSeq synthetic data with reinforcement learning in order to train models to interleave insertions with deletions when necessary.
Second, the experiments that we conducted with LintSeq in this paper studied code synthesis in Python only. LintSeq can be similarly used for generating synthetic edit sequences for code written in other programming languages by swapping the linter using during edit sampling.
Finally, we used LintSeq to refactor an instruction fine-tuning dataset in this work. However, by design, the algorithm
be run any corpus of source code data, such The Stack (Kocetkov ct al. 2022) or The St (Li et al. 2023). In future work, we hope to explore using LintSeq to train LMs write code edit-by-edit on larger, pre-training scale datasets.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This work explores data-driven mechanisms for improving the quality of language model-generated code. Our synthetic data generation method relies on open-source data and our experiments leverage open-source software and resources. It important to acknowledge that all language models for code synthesis have the potential to be misused- whether intentionally or unintentionally - for generation of code with vulnerabilities and/or malicious behaviors. Any and all model generated code has the potential to be harmful and must not be executed without precautions.
REPRODUCIBILITY STATEMENT
In the supplementary materials accompanying this submission, we provide a Python implementation of LintSeg as well as instructions and code supporting data generation, processing, pretraining, and fine-tuning experiments. We also provide thorough textual descriptions of all experimental procedures (rrenddxx[Cescrriiiii prompting and model evaluation, while Appendices D and
in the Appendix. detail all of the hyperparameters, procedures, and open-source datasets that we employ for obtaining the results reported throughout Section
3 Finally, Appendix A.4 provides references and data for reproducing the results plotted in Figure
ACKNOWLEDGEMENTS
This work supported by grants from NSF award 2339096 and ONR awards N00014-21-1-2758 and N00014-22-1-2773. We are grateful to Shenglong Wang and NYU High Performance Computing for their support of this project. UP is funded by NSF GRFP Award, and LP is funded by the Packard Fellowship. We would like to thank Nate Rahn, Mahi Shafiullah, and David Brandfonbrener for helpful comments and discussions.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 2333::2303.08774 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint 108.02200..07332 2021.
JL Ba. Layer normalization. arXiv preprint 1:Xvv:160.0044500 2016.
Loubna Ben Allal, Anton Lozhkov, and Elie Bakouch. Smollm blazingly fast and remarkably powerful. httpp:: https://huqqinngaace co/blog/ htoo/loggssmolmm smollm, 2024. Accessed: 2024-09-02.
Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. Thx: Learning fix coding with a text-to-text transformer. In International Conference on Machine Learning, pp. 780-791.
PMLR, 2021.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. 2022.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint wxXiv:2407.21788 2024.
11

---

Published as a conference paper at ICLR 2025
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:23 / 12450, 2023.
Sahil Chaudhary. Code alpaca: An instruction-followwee Ilama model for code generation. https: //github.com/ sahi 280114/ / 2023.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1xXxv:2007.03374 2021.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2 2780777777777.. 2024.
Google Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: /2. /805, 2023.
Google Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint ar arXhv:2433.0820 2024.
Alison Gopnik. Words and plans: Early language and the development of intelligent action. Journal of Child Language, 9(2)3033-18, 1982.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. et al. Olmo: Accelerat- ing the science of language models. arXiv preprint arXiv:2402.00838, 2024.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in neural informa- fion processing systems, 32, 2019.
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Mare Brockschmidt, and Miltiadis Allamanis. Learning to complete code with sketches. arXiv preprint vxXi:22166100158 2021.
Will Douglas Heaven.
How ai assistants are already changing the way code gets
http://wwwwthhhloooreeee...ccc http://owoo.oooooeeooooo//////// ai assistants cwpilot whw.llttt-haaiinggg-chea code software- softwware-deveelopmeett github 2024. Accessed: 2024-09-20.
made.
ottp://ooppmmmtttittoooooeaaaaaa openai/
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0882 2020.
David Kirsh. Problem solving and situated cognition. The Cambridge Handbook of Situated Cognition, pp. 664-306,77 2009.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: tb of permissively licensed source code. arXiv preprint 22 15533,, 2022.
Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. D0-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 13319-18345. PMLR, 2023.
Kaixin Li, Qisheng Hu, James Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Michael Shieh, and Junxian He. Instructcoder: Instruction tuning large language models for code editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). pp. 50-70, 2024.
12

---

Published as a conference paper at ICLR 2025
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint 2rXiv:2305.06161, 2023.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35: 4328 0528-4343, 2022a.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624): 1092-1097, 2022b.
Jaweii Liu, Chunqiu Steven Xia, Yuyao Wang. and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL (tLhttpps
//openreview.oo net.ieewwweeeeeee.......e..eee.e wttp://wwww..teeeo/eeeeaeaaeeeae 1qvx610Cu7
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang. Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, al. Starcoder and the stack v2: The next generation. arXiv preprint arXivv2402.19173, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint (rxiiv23306008500 2023.
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Seyed Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. OpenELM: An efficient language model family with open training and inference framework. In Workshop on Efficient Systems for Foundation Models II @ ICML,2024, 2024. URL httts::/oppenrev/
net htt///orumm//// htt.rrmmmmidxxxMMTxxxxxx.x
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning 200.207028.22222 2023.
code large language models. arXiv preprint
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 12220.3.1774, 2022.
Guilherme Penedo, Hynek Kydlíčck, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 17557, 2024.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for neural language agents. In Forty-first International Conference on Machine Learning, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1(140): 16, 2020.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. (ZeRO-Ofload): Democratizing (Billion-Scale) model training. In 202/ USENIX Annual Technical Conference (USENIX ATC 2/), pp. 551-564, 2021.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint 2308v: 2::::22988.000 /2950, 2023.
13

---

Published as a conference paper at ICLR 2025
Aman Sanger. Editing files at 1000 tokens per second.
httE:://ww..ursor.coo/////
instant--pply 2024. Accessed: 2024-09-02.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052000 2020.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arX11:2408..33114 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Ken Thompson and Dennis M Ritchie. unix Programmer's Manual. Bell Telephone Laboratories, 1975.
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. Zero++: Extremely efficient collective communi- cation for giant model training. arXiv preprint arXiv:2306. 10209, 2023a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- seff-generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 2021.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 20233..
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint
arXiv:2/09.0166 2021.
Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. Starcoder2-instrreet Fully transparent and permissive self-alignment for code generation. http::::/hugg..gfface...// wttp://cccc.cccco.ee/o.oooooolct -ttp:////oo////-nssuuu.s. 2024a. Accessed: 2024-09-08.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, 2024b.
Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural compulation, 1(2):270-280, 1989.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-AA Natural Language Processing. In Association for Computational Linguistics, pp. 38 45, October 2020.
URL https: http://wwww.ccc awwweb.lb.................m http:///nhooooy//oooo/oo
emnip-demos... demos.
Frank F Xu. Uri Alon, Graham Neubig, and Vincent Josua Hellendoom. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium Machine Programming, pp. 1-10, 2022.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Learning represent edits. arXiy preprint arXiv:18100 13337, 2018.
14

---

Published as a conference paper at ICLR 2025
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint 2rXiv:2304.112/1 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024.
15

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL RESULTS
A.1
EMPIRICS OF PROCESSING CODE DATA WITH LINTSEQ
Lines Per Example LintSeq Refactored vs Original Code Data
Edits per Synthetic Code Edit Sequence
Instruct
LintSeqlnstruct
LintSeginstruct
1.0
1.0
fraction
Traction
Traction
0.5
L = 14.2
0.5
L 18.7
E 3.8
100
lines per example
lines per example
edits per example (E)
Figure 6: Empiries of processing code data with LintSeq. Left: Lines per example dataset of instruction fine-tuning data for Python synthesis before and after processing with LintSeq via the linter pylint int (see Section 3.2 LintSeq processing adds lines of diff metadata to examples (see Appendix [). Right: The corresponding edit counts per synthetic code edit sequence. On dataset of short programs (14 lines of code, on average), the mean LintSeq edit sequence contains four edits.
A.2 COMPARING LINTSEQINTTRUCT TO RANDSEOINSTRUCT TINYCODELMS ON HUMANEVAL AND MBPP(+)
Table 2: Edit sequence TinyCodeLM results on HumanEval at high sampling temperatures: We tune sampling parameters for edit sequence variants of TinyCodeLLM temperatures (1, 1.1, 1.2), top-p (0.95, 1.0), and min-p (0, 0.05) with = 64 completions per problem and report the best pass @k value obtained from each model variant. We also report standard error for each score.
HumanEval
Linter Guided
Model Variant
Size
pass@1
pass
pass
pass@20
pass 50
M RandSeglnstruet
150M
X
4.5 + 0.4
10.3 = 0.5
12.2 + 0.5
14.4 + 0.6
18.8 0.6
LinycoleLM--LimmSehnntttee
150M
/
6.4 +: 0.5
13.9 + 0.5
16.8 +: 0.6
19.5 0.6
23.6 + 0.6
MinycodeLM Randelllsttnnnselttttuuettt
400M
X
6.0 + 0.4
11.7 = 0.5
13.9 + 0.6
16.4 + 0.6
20.8 = 0.6
Lnycodel.M-LLnnSeqnnse 400M
/
8.4 0.4
16.6 0.6
19.7 + 0.6
22.8 0.6
27.2 0.6
Table 3: Edit sequence TinyCodeL.M results on MBPP(+) at high sampling temperatures: As above, we tune sampling parameters for all fine-tuned TinyCodeLM variants over temperatures (1, 1.1. 1.2), top-p (0.95, 1.0). and min-p (0, 0.05) with 64 completions per problem and report the best pass @k k value obtained from each model variant. Standard emror indicated with "4."
MBPP(+)
Linter Guided
Model Variant
Size
pass @
5
pass @ 10
pass pass@20 @.20
pass 50
150M
L 0.3
17.2 0.4
22.6
27.9
34.4 1 0.5
LnycooleLM-LinSSeqltsttuuu
150M
/
8.6 1 0.3
19.5 1 0.4
24.5 1 0.5
29.0 1 0.5
35.1 1: 0.5
400M
x
10.2 1 0.4
20.8 + 0.4
25.4 0.5
29.9 1
36.2 1: 0.5
400M
14.7 1 0.4
25.8 + 0.5
29.6 0.5
33,9 0.5
39.7 1: 0.5
16

---

Published as a conference paper at ICLR 2025
A.3 HUMANEVAL, MBPP(+), CODECONTESTS, FOR LINTSEQ vs BASELINE INSTRUCTION TUNED GEMMA 2, PHI-3, AND LLAMA 3.1 MODELS
AND BIGCODEBENCH RESULTS
Table 4: Gemma 2, Phi-3, and Llama 3.1 results on HumanEval at high sampling temperatures. We report the best pass @k value obtained from each model variant at high sampling temperatures, sweeping over temperature values (1, 1.1, 1.2), top-p (0.95, 1.0), and min-p (0, 0.05). We generate
n 64 completions per problem and report standard error for each estimated score.
HumanEval
Model Variant
Size
pass @
pass
pass@ 10
pass @ 20
pass@50
Gemma-2-Inse 2-Instruct
2.6B
15.3 + 0.6
22.0 = 0.6
25.2 + 0.6
31.6 + 0.6
41.7 + 0,7
Gemma-22 L-iimm---LLnSSq(Insttruee
2.6B
22.0
34.8 =
41.4 0.6
48.2 0.7
+
Whi-3-Mini-Instruett
3.88
35.2
49.7 = 0.6
55.1 0.7
59.2 0.7
62.2 + 0.7
Bhi3--Mini-Linteelneee
3.8B
38.4 + 0.6
63.3 0.6
72.4 + 0.6
79.9 0.6
87.3 + 0.5
Llamaa3.1-Insrruet
8B
38.4
0.6
51.3 0.7
56.2 0.7
60.2 0.7
64.2 + 0.7
8B
38.5 + 0.6
62.2 = 1.6
72.6 + 1.6
75.7 + 0.6
82.7 + 0.6
Phi-3-Med-Instrucc
14B
50.2
+ 0.6
68.4
0.6
73.5 + 0.6
77.3 + 0.6
81.4 + 0.6
Phi33-Med-LinnSSqqnnrruc
14B
49.7
75.0 0.6
81.6
0.6
85.9 0.6
89.6 +
Table 5: Gemma 2, Phi-3, and Llama 3.1 results on MBPP(+) at high sampling temperatures.
Exactly as above, sweep over temperature (1, 1.1, 1.2), top-p (0.95, 1.0), and min-p P (0, 0.05) and report the best pass @k value obtained from each model variant. We generate n = 64 completions per problem and report standard error for each estimated score.
MBPP(+)
Model Variant
Size
pass
pass
pass 10
pass 1ax 20
pass@50
Gemma-2-Instinn
2.6B
20.5 0.4
30.8 1 0.5
34.3 1:2 0.5
37.6 1 0.5
41.6 1L 0.5
Gemma-2-LLinSeggnsee 2-Lintteqlnstrret
2.6B
28.2 0.5
40.1 0.5
44.5 = 0.5
48.6 + 0.5
52.8 0.5
10.-3-Mii--nntttue Phi-3-Mimi-LimmSeqnnntii
3.8B
31.9 0.5
42.5
+ 0.5
49.8 0.5
53.6 0.5
3.8B
37,2 + 0.5
51.4 + 0.5
56.1 + 0.5
60.3 + 0.5
66.0 + 0.5
Llama-3.1 3--ssrctt Instruct
8B
37.4 0.5
50.2 0.5
53.6 0.5
56.6 0.5
60.0 00.0::0 0.5
Llama-3.1 -.i.-.-11--innnnnsttss
8B
40.3 0.5
56.2 00.2:00 0.5
61.1 0.5
65.5 0.5
69.4 1
Phi-3 Phi-3-Med Med-Instruct
14B
37.7
+ 0.5
50.4
54.0 = 0.5
57.0
60.1 0.5
Phi-3-Med 3--Med-LittSqlnsteen
14B
39.1
0.5
55.2 0.5
60.7 =
65.4 0.5
71.1 0.5
Table 6: Gemma 2, Phi-3, and Llama 3.1 results on CodeContests. We sweep over temperature (0.5, 0.6) and use top-p = 1, min-p = 0, and ri = 128, and report the best pass @k value obtained from each model variant in the table below. We also report standard error for each estimated score.
CodeContests
Model Variant
Size
pass @
pass@ 50
pass@ 100
Gemma-2-Instruu
2.6B
0.05
0.05
1.56 1
2.26 1
Gemma-2- Ceenma-2-LintSqlInstee
2,6B
0.61 +: 0.16
5.71 1 0.37
7.03 1 0.40
Shi-3-MMiii-nstruu
3.8B
1.80 0.22
14.86 0.45
18.59 0.49
3.8B
2.76 + 0.26
19.10 0.48
22.93 0.51
Ilama-3. -
8B
2.68 + 0.28
11.21 0.44
12.80 0.46
Llama-3 Innttttttttttttt
8B
2.92 0.27
17.86 0.47
21.82 0.51
Phi-3-Med-Instruct Phi-3-Med-La Mcdd-LittSSgnsstuuee
14B
3.22
0.27
16.50 0.47
19.45 0.50
14B
3.02 + 0.25
19.09 0.48
23.11 0.51
17

---

Published as a conference paper at ICLR 2025
Table Gemma 2, Phi-3, and Llama 3.1 pass results on DS-1000. We use the same sampling
hyperparameters as Luo et al. 2023) and Wei et al. (2024b) to evaluate instruction tuned models.
Model Variant
Size DS-1000, pass@
Gemma Sommaa-2-Inssee 2-Inutruct
2.6B
2.5
Gemma-2-LintSewn 2-LintteqInstruet
2.6B
3,8
Phi-3-MMini-nntrrut
3.8B
8.6
Phi-3-Mini- 103-Mini-Lime Lini-LintSeqInstruc
3.8B
15.5
Llama-3. InInstruct
8B
14.5
Llama-3. -LintSellnstruct
8B
16.2
Phi-3-Med Med-Instruct
14B
21.8
Phi-3-Med-LintSeen Iled-iintSelInstructt
14B
24.2
Table 8: Gemma 2, Phi-3, and Llama 3.1 pass results on BigCodeBench (Instruct). We use greedy decoding to evaluate instruction tuned models.
Model Variant
Size
BigCodeBench Instruct, pass
Gemma-2 InIstrucct
2.6B
5.44
Gemma-2-Lintee Linnma---iinSSeglsstuutt
2.6B
6.32
Phi-t-MMimi-nntrruttt
3.8B
20.79
Phi-3-Mini Mini-LintteqlIsttuucc
3.8B
21.58
Llama-3. Innsttruct
21.46
Llama-3. 1immmm33.--LinnSegnsstueee
8B
20.53
Phi-3-Medd-Instrutt Phi:3--eed-LinnSellnsttci
14B
24.65
14B
28.16
A.4 COMPUTING @ TOTAL TEST-TIME FLOPS
In Figures T rright) and 4 we plot the percentage of problems solved by any attempt (i.c. pass (kss@k)& on HumanEval, MBPP, and CodeContests as function of total test-time FLOPs used during sampling for LintSeq vs baseline instruction fine-tuned models. Raw "pass@k" estimates are also included in Tables 4 5 and g representing the best scores achieved by each model variant after tuning sampling hyperparameters..
We compute total test-time FL.OPs using the approximations below, which are drawn from Kaplan et al.) (2020). These approximations conservatively estimate the cumulative inference costs of synthesizing solutions to all of the problems the test set of each benchmark. The models that we compare are all dense transformers, where the majority of the parameters
used in matrix multiplications.
FLOPs per token R= 2 (N (LVmodol-params + 2 2-2 model-layess Ccontext)
Total FLOPs FLOPs per token T Tavg-total totall-teens--e--aaaee K saampees
M Mprobiems
We determine the quantities I tstg tokess per sample for each model variant at particular "pass @k" by computing token counts over all sets of samples per problem.
Note that edit sequence (i.e. LintSeqInstruct fine-tuned) LMs have slightly higher average token counts per sample due to presence of "diff" descriptor tokens in generations (see Appendix
18

---

Published as a conference paper at ICLR 2025
B
MORE ON EDIT SEQUENCES AND DIFFS
B.1 READING UNIX DIFFS
We provide a guide to reading Unix-style diffs below in Figure 7 The diff shown in this figure is computed using the Python library di fflib, which is the implementation that we use to compactly represent edits in our synthetic data generation experiments. Note that the total extra tokens present in an insertion edit sequence representation of program scales with the number of program lines L, and can upper-bounded as Tai S L ((chars in "decorator") + (extra chars per line in "body")).
deletion metadata
insertion metadata
(deletion line 4, of deleted lines]
(nsertion line 4, # of Inserted lines)
decorator
@ -0,0 +1,5 @@
+def matrix sum((, B): cols len(A(0)
body
C= return C
Figure 7: The anatomy of a Unix diff: A diagrammatic visualization of the different parts of Unix-style diff, computed by difflib. The body of diff consist of multiple line deletions, followed by multiple line insertions. The decorator portion of the diff shows the location and size of these deletions and insertions, if any. Like the diff shown above, the edits in synthetic edit sequences generated by LintSeq consist of line insertions only.
B.2 RESOLVING EDIT SEQUENCES
During inference, LMs that have been fine-tuned LintSeq instruct data will iteratively synthesize programs by generating edits i.c. outputting text that consists of sequence of consecutive Python difts interleaved with newline characters and "< di diffi>" 1>" tokens, similar to Piterbarg al. (2024).
If correctly formatted by the LM, these diffs will be structured as shown in Figure 7
Resolving an edit sequence generated by a language model into an executable Python program is simple: starting with an empty program, we consecutively apply the line insertions and/or deletions the body of each diff to the lines the program specified in decorator. We continue this process until all of the diffs in the generated edit sequence have been parsed and resolved.
Figure shows a code edit sequence generation from LintSeq instruction fine-tuned LM and the corresponding resolved, executable Python program.
B.3 CONTROLLABILITY OF CODE SYNTHESIS WITH EDIT SEQUENCE LMS
The structure of Unix-style diffs affects the downstream controllability of code synthesis with models that have been trained edit sequence re- p-raameteriieed programs. As shown in Figure 7 the first line of every diff' is decorator that describes the location and the number of lines changed by the edit. During inference, autoregressive language models that have been trained diffs with this format can be prompted to predict an edit in a target location by intervening on a model generation.
B.4 FUTURE WORK: SEARCHING IN EDIT SPACE
we apply the lens of reinforcement learning or search to this setting, we might say that re- parameterizing the code data used to train language model re- parameterizes the model's action space. is possible that combining edit sequence LMs with more sophisticated decoding mechanisms, test-time search, and/or reinforcement learning may result in even larger improvements to the quality of generated code than those of the zero-shot code synthesis settings studied in this paper. We look forward testing this in future work.
19

---

Published as a conference paper at ICLR 2025
C EVALUATION
HvmanEval (Chen et al. 2021 and Mostly-Basic Programming Problems (MBPP) (Austin et al. 202T) are of the most studied benchmarks for evaluating code LMS (Liu et al. benchmarks probe the code synthesis capabilities of models, and consist of pairs of natural language program descriptions and test-cases. We employ the extended MBPP test cases released as MBPP(+) by Liu et al. (023) to add additional rigour to our testing procedure. The code LMs that we compare our TinyCodeLLi models against in Table evaluate HumanEval performance using the original set of benchmark test cases; for consistency, we employ these same test cases in all of our evaluations.
2023). These
Our evaluations on the harder benchmarks CodeContests, DS- 100,, and BigCodeeBench(nnsmnc)) use exactly the same sets of problem descriptions and test cases as those introduced by Li et al. (002bb) Lai et al. (023), and /huo et al. 22024)
During testing on each benchmarks, LMs are prompted to generate outputs using the natural language descriptions of target programs. Their outputs are then evaluated on the paired test cases. A generation is considered "correct" if and only if it passes all of the test cases upon execution, subject to fixed timeout setting. Previous works on code synthesis with language models report scores across samples. The most common of these metries is known as pass @k k (Chen et al.) 2021] Austin et al., [202]] Li et al. 2022b Wang et al., 2023b). This is the metric that we use to report and compare model performance throughout this paper.
C.I PROMPTING
The primary goal of this paper is to introduce method for re-factorizing code synthesis with LMs by fine-tuning them on synthetic instruction data. As result, we evaluate all models using minimal prompt formats, performing prompt tuning (see Figures 9 and TÖp. Examples of the prompt formats that we use during evaluation are shown in Figure
Example Formatted HumanEval Prompt
Pythan Version
Instruction Vension (Fram HumankDnnPaack)
from typing import
tewarrebentttteeeeeees Laeffiost), bool sclve the following problem:
alemertaiiumbee Lix(faal), Ihsehook boot: eix Chesk I'ln given given threhold clean, alemerna S.0),
numbers than
threshoks.
given
are any "wo numbers other
0.5) soo sau close, alamenta]1.. 2.8. 3.0. 4.0. S.O. 2.00
dlementog1..
boor close elemenss11.0 0. 3.0, 4.0, 5.0.5.... (.0]].00
Faeis
Example Formatted MBPP(+) Prompt
Maad Instruction- Python Version
Instruction Vension
Write pythan lunction to nemove the characiers which have odd indes valuee af givon string
Weite python function lodd, valure, stringJer1j remove the charasters which have add inder givan string.
del odii soltee, atringhert)
Figure 8: Examples of formatted HumanEval and MBPP(+) prompts used in model evaluations.
We finetune all tested models on example outputs exclusively corresponding to Python code, and as result, we do not use Markdown formatting to separate Python code from natural language in either our instruction data nor our inference-time prompts.
To evaluate models on HvmanEval, we use both the default "Python version" prompt format in the original benchmark dataset, where a natural language program description provided an LM within docstring, as well as the equivalent, fully natural language prompt format from HumanEvalPack (Muennighoff et al. 2023). The latter format similar to the structure of the instructions in our fine-tuning datasets. We report results the prompt format that yields the best score for each model.
To evaluate models on MBPP(+), we use the default prompts from the MBPP benchmark dataset, formatted with specification of the target function name and arguments both inside and outside of the natural language instruction, as shown in Figure [ As on HumanEval, we report results on the prompt format that yields the best score for each model.
20

---

Published as a conference paper at ICLR 2025
To evaluate models on BigCooceBenchInsstruu)) and CodeContests, we simply prompt models with the problem descriptions introduced in the original version of the benchmark (Zuuo et al. 2024 et al. 2022b).
Finally, to evaluate models on DS-1000, we use the complction format, with precisely the same prompt structures those used by Wei et al. (2024b).
C.2 GENERATION AND PARSING
During generation, we continue decoding until an end-of-sequence token is output by an LM. We treat LM outputs as either Python code or sequences of Python code edits, depending on whether an LM was fine-tuned on standard instruct or LintSeq instruct data. In the latter case, we post-pro process outputs by resolving the output edit sequences using the procedure described Appendix
C.3 EVALUATING MODEL CHECKPOINTS
C.3.1 PHILOSOPHY
There well-known trade-off between the temperature used for sampling from autoregressive code LMs and the benchmark coverage achievable by models, i.e. the proportion of problems "pass @ k" for which an LM is able to generate at least one output that passes all test cases given "k" tries. This trade-off was first described by Chen et al. (202I). Informally, increasing the sampling temperature increases the width the distribution from which tokens are sampled, producing more diverse but noisier (and possibly lower quality) generations. For larger repeated sample counts, the pass @k score typically increases with sampling temperature up to some threshold, beyond which the negative effects of noise overpower the positive effects of diversity. The benchmark coverage achievable by an LM at any temperature and in the limit of samples, i.e. on pass @k for k f oo, ultimately depends on both the power and expressivity of the code language model's learned representation.
From a practical perspective, while smaller language models may have weaker representational power than larger models, the representational expressivity of the former may enable them overtake the latter at fixed computational budgets by leveraging extra compute at inference-time, e.g. generating larger number of samples per problem and using the provided test cases to check each one for correctness before returning an output (Brown et al.) 20244] Snell et al.) (024). For example, an LLM that has an 85% pass @1 score on an arbitrary task may be more expensive in total serving cost (see Figure 1 than a smaller LM with a 90% pass @50 score on the same task. A small L.M can only have this property, however, if exhibits reliable trade-off between generation quality and inference-tin sampling cost across tasks. In other words, its representation must be sufficiently expressive.
C.3.2 COMPUTING @
Our goal is to probe whether re-parameterizing code synthesis with edit sequences can improve the expressivity of smaller LM representations, boosting benchmark scores as a function of total test-time compute. Hence, we primarily compare fine-tuned models by evaluating them with the procedures described above across multiple pass@k. k. We compute unbiased pass@k statisties with the same procedure as Chen et al. (2021). The results of these evaluations are reported throughout the paper.
C.4 COMPARING TINYCODELMS TO EXISTING MODELS IN TABLE
Many existing state-of-the-art code synthesis LMs only report temperature-tuned pass @ scores on HumanEval, including Codex, AlphaCode, and Codegen- (Chen al. 2021 Li al. 2022b Nijkamp et al. (022). Thus. in Table [ we temperature-tune TinyCodeLM models pass@1 1 and pass 10 scores when reporting results. On HumanEval, we test temperatures E (0.0, 0.4, 0.8, 1.0). On MBPP(+), we sweep over smaller temperature range, E 1.0. We perform the same temperature tuning procedure when reporting external model benchmark scores well, i.e. the scores annotated with "(t)" in Table When running benchmark evaluations with these external code LMs, we stray from the prompt Tormatting, generation, and parsing procedures described in Appendices C.I and C.2 instead, in the interest of a fair evaluation, we reproduce the conventions reported by model authors to report other scores.
21

---

Published as a conference paper at ICLR 2025
D PRETRAINING
We rely on data and libraries open-sourced by the HuggingFace, FineWeb, StarCoder, Dolma, OLMo. and Py PyTorch FSDP projects pretrain our models (Wolf et al. 2020: Penedo al. 2024 Lozhkov
et al. 2024 Soldaini et al. 2024) Groeneveld et al. 2024 Zhao el al.
2023
D.1
MODEL ARCHITECTURES AND PRETRAINING HYPERPARAMETERS
Table 9: Architectural and pretraining hyperparameters of our "on device" 150M and 400M parameter TinyCodeLM models, pretrained on mixture of Web text and code for Python under- standing.
TinyCodeLM
Smallest, 150M Parameters
Small, 400M Parameters
Transformer Architecture
decoder-only
decoder-only
Model Family
O.moForCausalIM
OlmoForCausaltk
Tokenizer
GPT-NeoX-20B-OLMM.
GPT-NeoX-20B-OLLMM
Attention Bias
False
False
Attention Dropout
0.0
0.0
Hidden Activation
SWiGLU
SwiGLU
Hidden Size
768
1024
Intermediate Size
3072
4096
Number of Attention Heads
12
16
Number of Hidden Layers
24
Number of Key-Value Heads
12
16
Vocabulary Size
50304
50304
Positional Encodings
Rotary (RoPE)
Rotary (RoPE)
Mixed Precision
BFLOATI 6
BFLOAT16
Weight Tying
True
True
Flash Attention
True
True
Optimizer
AdamW
AdamW
Learning Rate
0.0003
0.0003
Weight Decay
0.01
0.01
Betas
(0.9, 0.95)
(0.9, 0.95)
Epsilon
1.0e-05
1.0c-05
Learning Rate Scheduler
cosine (with warmup)
cosine (with warmup)
Number of Warm-Up Steps
100
100
Alpha-f (a=)) Total Epochs of Pretraining
0.1
0.1
2
2
D.2 PRETRAINING DATA MIX
Table 10: Pretraining data mix used to train both TinyCodel models. Datasets were tokenized and prepared using HuggingFace and Dolma tooling (Wolf et al. 2020 Soldaini et al. 2024).
Pretraining Data Source
Subset
Tokens
Documents
FineWeb (Penedo et al. 2024)
1OBT Sample
10.4BT
14.9M
The Stack (Kocetkov el 2022
Python Only
61.8BT
24.2M
22

---

Published as a conference paper at ICLR 2025
E INSTRUCTION FINE-TUNING
E. BASELINE INSTRUCTION DATASET
Table I1 displays the data sources that are used to prepare the dataset described in Section 3.2 These data are pooled and preprocessed into instruction-prograt pairs by stripping away Markdown formatting and natural language explanations from completions (Figure TO). In our experiments, we use the resultant data to finetune baseline models, comparing their performance to those of LMs fine-tuned on edit sequences generated with LintSeq from the same set of instruction-progratt pairs.
and
HuggingFace Instruction Data Source
Subset
Examples
Full 50,661
Python
38,284
Table 11: Instruction data mix used to prepare the baseline instruction dataset in Section 3.2
E.2 PROCEDURES AND HYPERPARAMETERS
We instruction finetune all models with Microsoft DeepSpeed using the ZeRO++ protocol for stage three sharding. For the largest of these models, we also use CPU parameter offloading to accelerate experiments (Wang et al. 2023a Ren et al. 202T). When fine-tuning models on LintSeq data, we add new token "<ldiff |>" to tokenizers (Section 2.5) and resize model embeddings accordingly.
In our experiments with Gemma 2, Phi-3, and Llama 3.1 models, we use HuggingFace to access and load pretrained model weights and tokenizers. As mentioned in the main body of the paper, we instruction finetune pretrained-only weights open-sourced and available. This the case for Gemma 2 and Llama 3.1 only, of the writing of this paper.
Across all of the fine-tuning experiments conducted in this paper, we train model-data variants with the same batch size and for equal number of total optimizer steps. This optimizer step count corresponds to ten epochs of fine-tuning with the baseline instruction tuning dataset described in Section 3.2 We save intermediate checkpoints equal optimizer step intervals in all experiments, and we report benchmark scores for the best performing checkpoint from each model-data variant.
In order to tune the peak learning rates used in each of model experiments, we full sweep E (6e-4, 3e-4, le-4, 5e-5, le-5, 5e-6) in the baseline instruction data setting for each model. We select peak learning rate values by tracking the best-achieved downstream benchmark performance across models. The chosen values are displayed in Table 12 All other fine-tuning hyperparameters are kept fixed at the settings in Table 13 across experiments.
Gemma 2
Phi-3
Llama 3.1
150M
400M
2B
3.8B
14B
8B
Peak Learning Rate (a:)
3e-4
3e-4
5e-5
5e-5
1e-5
1e-5
Table 12:
Peak learning rates used to instruction finetune models.
Hyperparameter Setting
Learning Rate Scheduler
linear
Max Learning Rate
le-4
Warmup Ratio
0.001
Weight Decay
0.01
Total Batch Size
512
Batch Loss Reduction
sum
Mixed Precision
BFLOAT16
Max Sequence Length
1024
Total Optimizer Steps
1740
Table 13: All other instruction fine-tuning settings, re-used across experiments.
23

---

Published as a conference paper at ICLR 2025
F
MORE ON SYNTHETIC DATA GENERATION WITH LINTSEQ
F.1 EXAMPLES OF GENERATED SYNTHETIC EDIT TRAJECTORIES
Example
Instruction
Python functin
reecanngeed checks
between any rectangles. restangle heightt.
represented tuple fs, y. W. hi 1s, y) top-lett rectangle,
any pai rectangles False colliiion dotection algorithm.
an appreach check found, return immociately chocking
bounding
Beseliee instruction Datasel
Python Program
Python Sequences Sampled LintSeq
def check_co www.clllco/ecccangeee...
for omgge/enjoctangeskk
for iin rangedi + len/ecctngleedt
@0 4,0 ee
voiff chock wow.kkooooooooooooooooooooooooo clooreeestanngee
retum
Sample
6B -0,0 11 ee shock oo lsconeectaannnet barggoeeeeccnnnnee
for
Inntectongeedy
Semple
xT. w1, h1 rectangleaj] w2, 12 ractangles]]
for surgedeenrecttaggddd for rangll 1. kn/retanglled: rectang so])
yi wi. hi rectnniee]]
6. -4,0 +52 ee
x1.
ifxt x2 and
@0 4,0 v5 ee
60 +4.0 +5 de
sil y2, h2 rectang soil
12, yo, 12 restangles]
-6.0
return
yl --1 ynland return True
and ya 42 retrr True
Figure 9: LintSeq edit sequence samples vs baseline instruction-progrant data, example A.
Example B
Instruction
given diationary keys by joining a period (1)
or nested diationary.
archy
dictionary with fartened keys
Baseline Instruction Dataset
Python Program
Synthetic Python Edit Sequences Sampled with LintSeq
Nutten, di deteiiictoonnnn dli dist
ee 4800 +12 de
lea +1.2
Flations dialyoally with nestee loys and a
Puttrr dutonary reltinsted andretomtio
Exnorns ())
now dicteenay with the Bussnndd koys.
leps.
-2. +0,0 6e
""Vlateess disionay rested
Eatter
detiracc with rertree kays
der cheydictionn vary Gurent key Sattened. didt:
key cureen else key
intur Satanee,uii
curment
oef dhr cibmaay curmet key fatteeed didt naw. key cummit sey cuuumnt.yy
for kiy vake dedemnrstaeee
Herattacceaaaoo diet:
eise key
Juy,
iir natnaaaaa del:
60
Indtoraet wwwp:/cccctteeeeeem Jeey value
chilkoooroy : Laborred...o .0c0
chriukes rew key fustored dki
salum of
Swwper.oooooo Jua]
Sultnred diyan haj
Sample
Sample
Figure 10: LintSeq edit sequence samples baseline instruction pastucttion-poogram data, example B.
24

---

Published as a conference paper at ICLR 2025
F.2
TUNING LIN EXAMPLE COUNT
Python Synthesis
Scaling Laws: Tuning Lintšeq Sample Count per Instruction- Exampie
HumanEval
TinyCodel,M, M, 150M
TinyCodel M, 400M
TinCoddeLM, 150M
TinyCodel
0.25
0:30
Sumples
$0.20
0.3
0.10
0.2
0.1
0.2
0.05
0.00
10'
0.00
0.05
10P
10'
0.0
10P
10"
0.0
10"
samples (4)
number semples (4)
number samples (K)
samples (4)
Figure 11: Probing the effect of varying the number of edit sequences sampled with LintSeq per instruction-example pair during data generation: Using the source dataset described in Section 3.2, we sweep over the value of the LintSeq parameter 8 used during synthetic data generation to yield three different edit sequence instruction datasets with E {1, 5, 10). We finetune TinyCodeLM models on each these datasets, and compare the resultant HumanEval and MBPP(+) performance
samples (i.e. pass @k vs k) temperature 1. The most performant values is 5.
25


Reviews:
[{"review_id": "Mz97MoNyWG", "reviewer": "Reviewer_8Cje", "summary": "This paper proposes LintSeq, which is a synthetic data generation algorithm that generates edit sequences from raw code to construct training datasets for LLMs. Specifically, to let LLMs learn to write code by editing existing programs like human developers, LintSeq first refactors existing code into a sequence of code edits, where each edit is an error-free insertion, and then outputs edit sequences as text strings consisting of consecutive program diffs. Evaluation shows that, after training LLMs on the synthetic edit sequences, the inference-time scaling performance of LLMs on HumanEval and MBPP improves significantly.", "questions": "- What are the computational resources required to synthesize edit sequences using LintSeq? If it is cost friendly, will it also be able to improve pre-training performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper explores an interesting and important research direction.\n- The idea of constructing fine-tuning datasets that mimics human developers’ behavior by synthesizing edit sequences is novel.", "weaknesses": "- The synthesized edit sequences may not be reasonable. While LintSeq ensures that the program at each step of the edit sequence does not contain any static error, it does not necessarily guarantee that this edit sequence is reasonable because a valid edit sequence should reflect the reasoning/thinking process of human developers, which is not guaranteed by LintSeq. Furthermore, edit sequences synthesized by LintSeq only contain insertions, which is also different from real-world edit sequences.\n- The comparison between “LintSeqInstruct” and “Instruct” may not be fair enough. While authors have controlled the number of optimizer steps the same, it is also important to make sure that “LintSeqInstruct” and “Instruct” share the same valid training tokens (i.e., the number of tokens in the training sequences that are not padding tokens). Although sharing the same number of optimizer steps, it is still possible that each training sequence in “LintSeqInstruct” is much longer than that in “Instruct” and thus more training tokens are provided for “LintSeqInstruct”.\n- Evaluation is not comprehensive enough. Apart from HumanEval(+) and MBPP(+), more benchmarks should be included to further demonstrate the effectiveness of “LintSeqInstruct”, such as DS-1000, APPS, and LiveCodeBench.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdI23C6SZj", "reviewer": "Reviewer_Uc3H", "summary": "The paper introduces LintSeq which is a proposed method to tune LLMs for programming tasks. Data preparation consists of representing code generation as a sequence of edits. To construct the edits, there is a 2 step process broadly, first deleting lines from an existing program to create potential intermediate states, and using a linter to reject intermediate states that are syntactically incorrect or fail other kinds of static linting tests. The authors demonstrate the merits of this approach for small language models and large language models, comparing LLMs tuned with LintSeq to plain instruction-tuned models. The authors also ablate the importance of the linter-based filtering step, showing that this step is important to its success. The authors also demonstrate and explain that LintSeq-tuned models may sacrifice inference-time compute in exchange; however, in turn for higher performance on program synthesis benchmarks.", "questions": "Generally, this is a re-hashing of the weaknesses\n\n1. Could you please describe the algorithm for LintSeq more clearly, and I'd strongly also recommend updating the paper to reflect a more clear explanation (e.g. PseudoCode or a more lengthy explanation in natural language in an appendix section). \n\n2. Did I misrepresent or misunderstand your claims? Please feel free to disagree if I have missed something.\n\n3. Could you provide comparisons to other commercial LLMs (e.g. Gpt4o, the updated Sonnet 3.5, Gpt4o-mini, Haiku 3) and open LLMs (e.g. the most recent LLama3-Instruct 70b, Llama3-Instruct 8b, DeepSeekCoder 7b instruct, DeepSeekCoder33b Instruct) on MBPP and HumanEval with higher samples, and demonstrating pass @N as you did in tables 10 and 11. \n\n4. For the models in tables 10 and 11, is it possible to also compare lintseq best@50 to instruction-tuned models best@50, except with Top p = 1.0 and temperatures [1.0, 1.1, 1.2]? This is important to also establish if the increase in diversity of lintseq cannot be matched by modulating the temperature for sampling. \n\n5. Are there other ways to calculate estimated FLOPs as you have? Would it be possible to also tell me the wall clock time of running inference using the instruction-tuned models and compare that to the wall clock time of using the LintSeq models (e.g. both 14b models compared) just for more transparency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results speak for themselves: the authors demonstrate LintSeq is capable of substantial performance improvements over instruction-tuned models. The authors are transparent that LintSeq may sacrifice inference-time compute for improvements in the efficient use of parameters and its strong program synthesis abilities. And the authors' ablation demonstrates that the use of a linter (to reject poorly formatted code edits) is important to the success of LintSeq. \n\nIn terms of intellectual merit, LintSeq is not the first work to suggest leveraging edit sequences for training language models for programing tasks. However, LintSeq is notable for getting these ideas to work at scale. To my best knowledge, much prior work may have focused on using human edits such as github commits, which is limited by the number of demonstrations available. Here: the authors introduce a procedure that is not bounded by the number of human demonstrations available, but rather the availability of a linter or even more loosely, a static-analysis based tool for checking the well-formedness of programs (e.g. tools often found in a compiler). \n\nIn terms of significance, the authors are able to demonstrate LintSeq's ability to improve over instruction tuning for a series of code LLMs. However, LintSeq also may have significance for real-world coding applications. The authors mention that commercial code assistants like Cursor deliberately do not use edits; however, the case study of LintSeq may inspire new approaches for code-editor assistants (it could also potentially have issues as well). Additionally, the set-up of LintSeq may also motivate applications for incremental generation of natural langauge responses, especially for tasks that may be intensive in reasoning.", "weaknesses": "The authors' work is very strong, however, there are some important points that need discussion. Overall, my largest concerns are with transparency, clarity, and integrity in presentation. I myself felt mislead multiple times while reviewing this paper, and that concerns me greatly. \n\n1. Misleading Presentation of Models >1B parameters / Transparency + Clarity \n\nI found figure 1 to be misleading. It compares leading LLMs on Human Eval with K=1 to LintSeq with K=50. Personally I didn't catch this until I glanced over the paper after at least multiple passes, I would fear other readers may fall into the same trap and overestimate how strong LintSeq is. In reality, LintSeq Pass@1 is can equal to or even significantly worse than an instruction-tuned variant of the same base model. \n\nIt is only in the appendix in Tables 10 and 11 where the authors clearly present LintSeq vs Instruciton-Tuning pass@1, pass@5, etc rates; although similar information is presentated in figure 4. With pass@50 LintSeq is significantly better than instruction-tuned models; whereas, it is negligible or worse with few or one/fewer samples. I believe there is no thorough discussion of this phenomenon in the main manuscript and this is an extremely important point. For example, the authors could offer an explanation that LintSeq increases the generation diversity which is highlighted in the higher pass@50. The authors must be more forthcoming about the tradeoffs of LintSeq when model size is scaled up. I urge the authors to reframe the presentation and discussion to clearly reflect this.  \n\nI would also appreciate a comparison with other top LLMs for code (e.g. LLama3 or LLama3.1; CodeLLama, DeepSeekCoder, Gpt3.5) and comparisons with equal pass@1, pass@5, pass@10 to show how these other LLMs scale with more samples. I feel very mislead by Figure 1 comparing Pass@1 for big models vs. pass@50 for LintSeq and I am left feeling such a presentation is problematic. I feel uncertain about  the motives in presenting Pass@1 vs. Pass@50 without a fair like-to-like comparison between other LLMs and putting tables 10 and 11 deep in the appendix. The authors must present a more forthcoming picture of the strengths and weaknesses of LintSeq when scaled up >1B parameters. I think the paper requires a significant revision to reflect this in language. I believe the technical contributions could already be there, but the presentation feels very misleading to me and the discussion should center much more on the tradeoffs that LintSeq introduces.\n\n2. Framing and Explanation of LintSeq\n\nI find the framing of LintSeq or the explanation of it is not easily clear, and even after section 2.2 where it is broken down into phase 1 and phase 2, it is still unclear to me. This led me to inspect the artifact `lintseq` in the supplementary materials which is >300 lines of code and mentions sampling weights, computing fitness, and \"children.\" Perhaps there's something going on where in the paper \"affected lines\" are the indented lines of a code block. The description is not sufficiently clear, and I recommend more clarity in the presentation of the algorithm for both transparency and reader understanding. I would not call this \"recursively lint\" as shown in figure 2, as linting doesn't edit code, so the recursive process needs to be combined with some editing process as menitoned in section 2.2: \"(2) run a linter or other\nverifier on the remaining code; (3) if the deletion induced new errors, remove all affected lines; and (4) repeat steps 2 and 3 until no errors are caught by the linter.\" To me, the \"remove all affected lines\" is unclear to me and I'm not sure if all writers of the paper fully understood the procedure either. I recommend trying to formulate a more thorough explanation both in the main manuscipt and also if this is not sufficient to add an appendix section explaining the procedure in more detail. Pseudocode / pythonic pseudocode may also help to demonstrate this procedure. \n\n2. Claims of Tiny LMs is Misleading \n\nI'd recommend the authors walk back the claims on their Tiny LMs. I find the statement \"Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters\" misleading: the 150M parameter model underperforms CodeT5+ 220M at Pass@10 HumanEval and underperforms Codex 300M at Pass@1. \n\n3. Discussing Prior Work\n\nThe authors only mention 2 works from 2023 and 2024 on fine-tuning on edits. Some papers that came to my mind while reading this were: \nhttps://arxiv.org/abs/1810.13337\nhttps://arxiv.org/pdf/1905.11006 (much less related)\n\nThis makes me inclined to think that there may be more works that I have not included here that should be discussed to highlight the history of thinking on this issue. \n\n4.  Choice of Language \n\nThe term `static program verifier` makes a loose use of the word \"verify.\" I recommend using a different word from \"verify\" to not conflate it with [formal verification](https://en.wikipedia.org/wiki/Formal_verification) of programs. \n\nAlso in the abstract \"small LLMs\" is a contradiction of sorts \"small Large Language Models.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LS9wtX48eU", "reviewer": "Reviewer_aqpZ", "summary": "This paper looks at the problem of code generation. The corpus trains LLMs to generate code from left to right. Yet, this paper proposes to generate synthetic code edits that train the model to construct code in a step-by-step fashion, while encouraging each step to preserve the syntax validity of the code. This technique is evaluated to be effective for SLMs esp. when doing inference scaling.", "questions": "- Novelty-wise what is the paper's main difference compared to the code sketch paper? \n- Code diff is more than just insertion -- why deletion and rewriting are not considered?\n- Why disabling chain-of-thought data?\n- Why pre-training TinyCodeLM instead of reusing existing pre-trained SLMs?\n- Can you try sampling the model using min-p rather than top-k?\n- What's the intuition on why LintSeq can improve pass@1? It makes sense to me it can improve pass@ big K as the approach optimizes code generation diversity but it's interesting to me that it also gets higher pass@1 (i.e., higher-quality first sample).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- This paper looks at an important domain in generative AI -- code generation.\n- The overall idea of processifying code generation makes sense to me -- decomposing the code generation task into code insertions allows the model to construct the generated code with more diverse context possibilities in addition to the simple left-to-right context", "weaknesses": "- **Novelty & related work**: LintSeq reminds me of the code sketch paper [1] which generates code via syntax-validity-preserving edits (i.e., filling grammar holes). This paper should be discussed (and compared if applicable).\n- **Technique**: One theoretical limitation of LintSeq is the lack of naturalness of synthetic edits. The data trains the model on random trajectories which might (if not always) not be meaningful.\n- **Technique**: Code edits in LintSeq are *insertion-only* -- I can imagine cases deletion/rewriting can be used, such as writing a bunch of `def f(): pass` when planning the code skeleton and replacing the \"pass\" when real implementation starts.\n- **Evaluation**: overall I feel the evaluation focuses on function-level code generation, which might look a bit contradictory to \"inference scaling\" -- as I'd expect to use inference scaling for solving much more challenging tasks, e.g., those in SWE-Bench.\n- **Presentation**: \"Open-source edit data with good coverage over the distribution of all error-free code “diffs” is nonexistent\" is debatable given high-quality code-edit datasets such as EditPackFT.\n- **Presentation**: The terminologies in this paper can be made more precise (detailed comments below).\n\n----\nComments on writing:\n\n- Fig 1: \"HumanEval Coverage\" and its use in many other places: maybe just explicitly say pass rate or pass @ k -- \"coverage\" by default means code coverage in the code generation domain and it might be confusing to refer it to pass rate.\n- L065: authors should explicitly explain what \"static-error-free\" means -- is it program syntax error? or the error of the diff syntax, etc.\n- L080 \"static program verifier\" and \"error-free program states\": how is it possible to verify an arbitrary program state \"error-free\" statically? :) -- Similar to the comment above, some basic concepts need to be explicitly clarified and defined.\n- L084 \"verifier ... is also referred to as a linter\": linters and verifiers are very different concepts. linters mostly do static syntax-based checks on certain surface-level program properties. Verifiers are way more ambitious as they assure the existence/absence of program properties given all possible inputs/states (nonetheless, the meaning of verifiers has been extended to validators to assure behaviors over \"certain\" rather than \"all\" inputs/states). \n- L107 \"A linter is a type of standard code analysis tool that verifies the correctness of program\" -- the statement is too strong to be right. Linters cannot verify program correctness but just check a few syntactical issues.\n\n**Actionable suggestions**: for these \"linters\", \"correctness\", and \"verifiers\" terms, it's more precise to say \"apply linters to make sure each edit can preserve the program's syntactical validity/well-formedness\".\n\n[1] LEARNING TO COMPLETE CODE WITH SKETCHES. ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFxeBvthr5", "reviewer": "Reviewer_HReL", "summary": "The paper introduces a new algorithm, LintSeq, which generates synthetic code editing sequences from existing programs. It reframes code generation as a sequence of edit generation tasks, mimicking human coding. They applied the method to create instruction + program-diff-sequence to train smaller LLMs for code synthesis. Models finetuned on these edit sequences generate more diverse programs and improve performance on benchmarks, achieving up to +20% higher scores on HumanEval pass@50 compared to baseline finetuning. Additionally, a 150M parameter model fine-tuned on these edits achieves state-of-the-art performance in code synthesis for on-device models, competing with larger models like Codex and AlphaCode.", "questions": "1. See the weaknesses above.\n2. Besides simple code generation tasks on HumanEval and MBPP, does this approach work for practical coding such as BigCodeBench? Also, does training on edits improve code editing tasks, say CanItEdit?\n3. How does this approach handle a code file that depends on other files in a repository? In such cases, applying the linter only to the code file will raise an error.\n4. For code generation tasks requiring multiple programming languages, does the same methodology still work?\n\nOpen to increase the score if the above concerns are properly addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is novel. While existing models often separate code generation and editing tasks, this paper presents a perspective to unify them into sequence of editing tasks. It's an interesting directions researchers can explore in the future.\n- To prepare the training data, the authors proposed LintSeq, which refactors existing programs into sequences of static error-free code edits. This is a smart combination of program analysis and LLM.\n- The pretraining experiment gives a better understanding of the effectiveness of the technique.", "weaknesses": "- The authors only evaluated the models on HumanEval and MBPP. While these two benchmarks are popular for most general-purpose LLMs, they are biased toward simple programs and do not cover practical coding scenarios.\n- The paper used a Python-specific linter `pylint` to construct pretraining data, but it's unclear how this approach can be easily applied to other languages and be applied at scale.\n- There is not enough justification for the base models used for finetuning. Why no base code LLMs?\n- There is an overemphasis on pass@k when k is large. While a better pass@ larger k shows the LLM can cover a broader code distribution,  by definition, pass@1 is more practical and is what people care about more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)