Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

## Abstract

Abstract Software engineers mainly write code by editing existing programs. In contrast, large language models (LLMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of open-sourced edit data. While high-quality instruction data for code synthesis is already scarce, high-quality edit data is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors existing code into a sequence of code edits by using a linter to procedurally sample across the error-free insertions that can be used to sequentially write programs. It outputs edit sequences as text strings consisting of consecutive program diffs . To test LintSeq, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we instruction finetune a series of smaller LLMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset, comparing zero-shot performance on code synthesis benchmarks. We show that during repeated sampling, edit sequence finetuned models produce more diverse programs than baselines. This results in better inference-time scaling for benchmark coverage as a function of samples, i.e. the fraction of problems “pass@k” solved by any attempt given “k” tries. For example, on HumanEval pass@50, small LLMs finetuned on synthetic edit sequences are competitive with GPT-4 and outperform models finetuned on the baseline dataset by +20% ( ± plus-or-minus \pm ± 3%) in absolute score. Finally, we also pretrain our own tiny LMs for code understanding. We show that finetuning tiny models on synthetic code edits results in state-of-the-art code synthesis for the on-device model class. Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters, both with and without repeated sampling, including Codex and AlphaCode.

## 1 Introduction

The successes of large language models (LLMs) are difficult to overstate. However, consistent and correct zero-shot generation in code synthesis remains out-of-reach for all but the largest models (Abdin et al., ; Groeneveld et al., ; Dubey et al., ). Compared to other reasoning tasks, this setting has two challenging properties, namely solutions are both structured and long-form.

Humans tackle problems that have these properties by leveraging abstract mental models, first developing a plan for their solution that reflects the setting’s structure and then executing the plan one step at a time (Gopnik, ; Kirsh, ). For example, a software engineer might employ object-oriented programming when creating a new code-base by developing a “class” object and then gradually adding new functionality to this class as their code-base becomes more complex.

Figure: Figure 1: Code synthesis with LMs trained on synthetic code edit sequences. Left: An example generation from an LM trained to synthesize code as a stream of static-error-free edits. Right: Comparing zero-shot HumanEval coverage (in %) as a function of FLOPs for large external LLMs vs the smaller LMs that we instruction finetune in this paper. Repeatedly sampling from edit sequence LMs yields coding problem solutions that are competitive with GPT-4 and GPT-4-Omni, and have total cost similar to sampling once from the best open-source LLMs (see Appendix F.4).
Refer to caption: extracted/5927317/template/figures/method_viz_teaser_v43.png

In contrast, LLMs are trained to autoregressively synthesize entire programs from scratch. This makes repeatedly editing a program with an LLM extremely expensive – current state-of-the-art, LLM-powered code editing tools like Cursor repeatedly prompt models to rewrite entire programs during every edit generation call (Sanger, ). LLM outputs also suffer from degrading quality as sequence lengths grow and exhibit limited diversity across samples (Chen et al., ; Li et al., ; Roziere et al., ; Lozhkov et al., ). The consequence of these pathologies is that there does not exist a reliable trade-off between zero-shot generation quality and inference-time compute cost under the current paradigm of autoregressive code synthesis, particularly for smaller models.

In this paper, we claim that these issues can be mitigated at the data-level by reparameterizing code synthesis as a sequential edit problem. Rather than training models for single-step generation of entire programs, we propose that models be trained to generate code by predicting code edit sequences.

This objective has a major obstacle: while high-quality solution data for code synthesis is scarce, open-source edit data with good coverage over the distribution of all error-free code “diffs” is non-existent (Muennighoff et al., ). To address this, we introduce an algorithm titled “LintSeq” that re-factors existing programs into sequences of static error-free code edits. LMs trained on data generated with our algorithm output code edits that effect interdependent lines of a program. LintSeq is parameter-free. It consists of two phases: a backward sampling phase, which takes a source file as input and uses a static program verifier to sample sequences of error-free program states that begin with this file and end with empty programs; and a forward edit computation phase, which reverses each sequence of programs, employing the Unix diff (Thompson & Ritchie, ) operator to compute deltas between consecutive versions of the source file, and outputs edit sequences. The static program verifier used by LintSeq in its backward sampling phase is also referred to as a linter.

To test the impact of finetuning LMs to synthesize code with edit sequences via LintSeq instruction data, we conduct a series of experiments comparing these models to those finetuned on a standard version of the same instruction data. We evaluate LMs zero-shot on the code synthesis benchmarks HumanEval (Chen et al., ) and MBPP (Austin et al., ) by computing coverage, the proportion of problems solved by any attempt, as a function of samples. We do not allow LMs to use chain-of-thought reasoning (Wei et al., ) during evaluation. Our results are five-fold:

- 1.
Across models ranging in scale from 150M to 14B parameters, instruction finetuning LMs on edit sequences vs full programs improves the quality and diversity of synthesized code.
- 2.
The improved diversity of samples means that pass@k performance increases smoothly as a function of inference-time compute, allowing for a better trade-off between the two.
- 3.
Tiny edit sequence LMs have state-of-the-art performance in their model class (Table 1).
- 4.
For smaller LLMs, repeatedly sampling from edit sequence models results in HumanEval coverage that is competitive with GPT-4 models and has similar cumulative cost to sampling once per problem from open-source LLMs like Llama 3.1 405B (Figures 1 and 4).
- 5.
Ablating the linter from edit sampling during data generation hurts the downstream quality of programs synthesized by edit sequence models (Figures 5 and 6).

## 2 LintSeq: Code Synthesis as a Sequential Edit Problem

Figure: Figure 2: Visualizing LintSeq, an algorithm for re-factoring programs into sequences of edits. This algorithm samples across all of the static error-free line insertions that can be used to write a program chunk-by-chunk. It uses the Unix-diff operator to express generated edit sequences as text strings.
Refer to caption: extracted/5927317/template/figures/algorithm_viz_v16.png

LintSeq is an algorithm for synthetic data generation that samples across sequences of insertions that can be used to sequentially write a program, leveraging a linter to repeatedly check program states for static errors during generation. A linter is a type of standard code analysis tool that verifies the correctness of programs. The LintSeq algorithm is loosely inspired by recent work on discrete diffusion methods for text generation, where decoding is non-autoregressive (Li et al., ).

The key hypothesis underlying LintSeq is as follows: by training LMs on code edit sequences with teacher-forced supervised learning, we can potentially achieve a better trade-off between generation quality and compute at inference-time while still benefiting from the training and sampling efficiency of the autoregressive modeling paradigm. In this section, we provide a formalism for the edit sequence re-parameterization of code synthesis and we formally introduce LintSeq.

### 2.1 Reparameterizing Code Datasets with Edits

We operate in the textual supervised learning setting in this paper, where we have access to a code dataset $\mathcal{D}$ of $N$ example programs $y$, each of which may be optionally paired with a corresponding natural language instruction $x$ that describes the program’s function, i.e. $\mathcal{D}=\{(x^{i},y^{i})\}_{i=1}^{N}$.

Let $\Delta(\cdot,\cdot)$ denote the Unix diff operator (Thompson & Ritchie, ), which computes a text difference between a pair of strings by performing a line-by-line matching and returns a string summarizing the detected differences. The diff operator is implemented by popular version control and software development systems to help programmers track edits or code “diffs” between versions of text files. A single edit computed with the diff operator may consist of multiple line deletions and/or line insertions.

Fix a program $y$ in the dataset $\mathcal{D}$. Consider a sequence $\bm{\sigma}_{y}$ of $j$ text strings corresponding to programs or program states that terminates at $y$.

$$ $\bm{\sigma}_{y}=(y_{1},\dots,y_{j-1},y)$ (1) $$

We can equivalently re-express $\bm{\sigma}_{y}$ as an edit sequence $\bm{\delta}_{y}$ of length $j$ by first computing a diff between an empty program $\epsilon$ and the first program in the sequence, and then computing diffs between all pairs of consecutive programs, as shown below.

$$ $\bm{\delta}_{y}=(\Delta(\varepsilon,y_{1}),\Delta(y_{1},y_{2}),\Delta(y_{2},y_ {3}),\dots,\Delta(y_{j-1},y))$ (2) $$

If $\mathcal{D}^{\prime}$ is a dataset such that for every pair $(x,y)\in\mathcal{D}$, there exists a pair $(x,\bm{\delta}_{y})\in\mathcal{D}^{\prime}$, then we say that $\mathcal{D}^{\prime}$ is an edit sequence re-factoring of $\mathcal{D}$.

Figure: Figure 3: Empirics of processing code data with LintSeq. Left: Lines per example in a dataset of instruction finetuning data for Python synthesis before and after processing with LintSeq via the linter pylint (see Section 3.2). LintSeq processing adds lines of diff metadata to examples (see Appendix A). Right: The corresponding edit counts per synthetic code edit sequence. On a dataset of short programs (14 lines of code, on average), the mean LintSeq edit sequence contains four edits.
Refer to caption: extracted/5927317/template/figures/data_stats_viz_v3.png

### 2.2 Generating Linter-Guided Synthetic Edit Sequences

Recall from above that a single program edit computed by the diff operator $\Delta(\cdot,\cdot)$ can consist of any number of deletions and insertions. LintSeq is an algorithm for computing edit sequence re-factorings $\mathcal{D^{\prime}}$ such that all data $(x,\bm{\delta}_{y})\in\mathcal{D^{\prime}}$ have a particular property: every edit in $\bm{\delta}_{y}$ consists of insertions only. There are two phases in LintSeq: a backward sampling phase that is used to compute program state sequences $\bm{\sigma}_{y}$, and a forward edit sequence computation phase that is used to re-express $\bm{\sigma}_{y}$ as edit sequences $\bm{\delta}_{y}$. An illustration of these phases is shown in Figure 2. Full examples of edit sequences generated with LintSeq are provided in Appendix E (Figures 9 and 10).

##### Phase I: Backward Sampling

In the backward sampling phase of LintSeq, for each of the $N$ pairs $(x,y)\in\mathcal{D}$, we generate $s$ sequences of intermediate program states $\bm{\sigma}_{y}$ that begin with the empty program and terminate at the original program $y$. These sequences are generated in reverse or backwards using a simple procedure that we dub linter-guided sampling. Starting with the program $y$, we sequentially generate each predecessor program in $\bm{\sigma}_{y}$ from its successor by following these steps: (1) delete a line from the current program by sampling uniformly at random; (2) run a linter or other verifier on the remaining code; (3) if the deletion induced new errors, remove all affected lines; and (4) repeat steps 2 and 3 until no errors are caught by the linter. We repeat these steps until all lines have been removed from the original program $y$, at which point $\bm{\sigma}_{y}$ has been generated.

##### Phase II: Forward Edit Computation

Once $s$ program state sequences $\bm{\sigma}_{y}$ have been generated for each $(x,y)\in\mathcal{D}$, we run the forward edit computation phase of our algorithm. In this phase, we apply Equation 2 from above to compute an edit sequence $\bm{\delta}_{y}$ for each $\bm{\sigma}_{y}$. Starting from the last program that was added to $\bm{\sigma}_{y}$, we use the diff operator to compute edits between each pair of consecutive programs in $\bm{\sigma}_{y}$ up to the original program $y$. Finally, we pair each edit sequence $\bm{\delta}_{y}$ with its instruction $x$ (if present) to yield an edit sequence re-factoring $\mathcal{D}^{\prime}$ of $\mathcal{D}$ with size $sN$.

### 2.3 Properties of LintSeq Data

Synthetic edit sequences generated by LintSeq have a few other important properties. Let $\bm{\delta}_{y}$ be an arbitrary $j$-length edit sequence in $\mathcal{D}^{\prime}$ generated with LintSeq, $\bm{\delta}_{y}=(\Delta(\varepsilon,y_{1}),\dots,\Delta(y_{j-1},y))$. First, we observe that there is a simple correspondence between $\bm{\delta}_{y}$ and the original program $y$ used to generate it: $y$ can be re-constructed by starting with an empty program, and successively applying each edit in $\bm{\delta}_{y}$ to this program one-by-one. In other words, the edit sequence $\bm{\delta}_{y}$ resolves to $y$. Furthermore, by construction, every prefix subsequence of $\bm{\delta}_{y}$ resolves to a sub-program of $y$ that is error-free, i.e. that throws no errors when checked with the linter or the verifier used during generation. These two properties, in conjunction with the uniform sampling step used in the first phase of the algorithm, show that LintSeq samples $s$ examples across all possible static error-free sequences of line insertions that can be used to sequentially write a program $y$ from-scratch.

We show an example of program synthesis dataset statistics before and after LintSeq processing in Figure 3. In the worst case, re-expressing a program as an edit sequence increases the length of a training example by a token count that is constant in the number of program lines (Appendix A).

### 2.4 Practicalities of Training Language Models on LintSeq Data

LintSeq can be run on any code data. It is agnostic to the contents of a program, and only depends on knowledge of the programming language that a file is written in and the existence of a linter or another kind of verifier for written program files.

We use teacher-forced supervised learning (Williams & Zipser, ) to train models on LintSeq data, concatenating edit sequences into a single string by interleaving edits with special tokens, “
,” and computing instruction-conditioned losses over the resultant sequences. During inference, finetuned models can be prompted to synthesize programs with edit sequences by appending these special tokens to the ends of prompts. More details are provided in Appendix A.

Synthetic data generation with LintSeq is controlled by a single hyperparameter: the number of edit sequences $s$ that are sampled for each example in the source code dataset $\mathcal{D}$. Edit sequence sampling can optionally be constrained to avoid repetitions, though this may yield a dataset $\mathcal{D}^{\prime}$ with fewer than $s|\mathcal{D}|$ examples if there are programs in $\mathcal{D}$ with no non-empty and static error-free sub-programs.

## 3 Experiments

To study LintSeq and the impact of re-parameterizing program synthesis as a sequential edit generation problem, we conduct a multi-pronged set of instruction finetuning experiments. These experiments study code synthesis in Python and are designed to answer the following questions:

- •
How does finetuning tiny LMs on re-factorized code edit data generated with LintSeq impact benchmark coverage compared to finetuning on the original code data?
- •
Do performance improvements hold across model scales, families, and tokenizers?
- •
How does ablating linter-guidance from LintSeq to finetune on randomly sampled edit sequences impact code synthesis?

Similar to previous works (Chen et al., ), we evaluate models by computing zero-shot coverage statistics on code synthesis benchmarks with and without repeated sampling. We expound upon our motivation for evaluating models in this manner in Appendix B.3.1.

### 3.1 Pretraining Tiny LMs for Code Understanding

We begin our investigations by pre-training two tiny decoder-only transformers, TinyCodeLM-150M and TinyCodeLM-400M, for Python code understanding on 72 billion tokens of text. Pretraining our own language models grants us a data contamination-free test-bed to study code synthesis with edit sequences, rapidly evaluate LintSeq, and broadly re-examine the trade-off between inference-time compute and generation quality in code synthesis for models that can be updated on-device.

We rely on open-source data and libraries to pretrain our models (Penedo et al., ; Lozhkov et al., ; Soldaini et al., ; Groeneveld et al., ). Our pretraining data mix is inspired by Code Llama (Roziere et al., ), and reflects a code-skewed mixture of web text and raw Python sampled from FineWeb and TheStack, respectively (Penedo et al., ; Li et al., ). The architecture of our models respectively mimics the two smallest versions of GPT-2 (Radford et al., ), but integrates the transformer architecture changes proposed by the OLMo framework. This includes the absence of bias terms and the addition of non-parametric layer norms (Ba, ), as well as the use of SwiGLU (Shazeer, ), rotary positional embeddings (Su et al., ), and the GPT-NeoX-20B tokenizer (Black et al., ). We train both models for two epochs with a batch size of 524,288 tokens on an NVIDIA H100 node with four GPUs. Our experiments are supported by Pytorch FSDP (Zhao et al., ). More details on our pretraining procedures are in Appendix C.

### 3.2 Generating a Synthetic Dataset with LintSeq

Next, to support our finetuning experiments, we prepare a large “baseline” dataset of paired instruction and program data. We re-factorize the programs in this dataset into code edit sequences with LintSeq.

**Table 1: Summary of temperature-tuned coding benchmark results for LMs with $\mathbf{\leq 0.2}$B parameters. Scores annotated with “($\dagger$)” indicate external model evaluations that we ran using the procedure described in Appendix B, and all other scores are as reported by model authors. We list models in order of increasing HumanEval pass@1 score.**
|  |  | HumanEval | MBPP(+) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Size | pass@1 | pass@10 | pass@1 | pass@10 | Open-Source |
| PolyCoder | 160M | 2.1 | 3.4 | - | - | 🌑 |
| AlphaCode | 89M | 4.3 | 12.2 | - | - | 🌕 |
| Codex | 85M | 8.2 | 12.8 | - | - | 🌕 |
| SmolLM-Instruct | 135M | 7.6 | 14.4($\dagger$) | 10.1($\dagger$) | 14.6($\dagger$) | 🌑 |
| TinyCodeLM-Instruct | 150M | 9.1 | 13.2 | 11.5 | 16.4 | 🌑 |
| TinyCodeLM-LintSeqInstruct | 150M | 12.8 | 20.6 | 13.6 | 24.4 | 🌑 |

**Table 2: Summary of temperature-tuned coding benchmark results for $\mathbf{0.2}$B to $\mathbf{0.4}$B parameter language models. Annotations, model order, and evaluation procedure are the same as in Table 1.**
|  |  | HumanEval | MBPP(+) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Size | pass@1 | pass@10 | pass@1 | pass@10 | Open-Source |
| PolyCoder | 400M | 3.0 | 5.3 | - | - | 🌑 |
| TinyCodeLM-Instruct | 400M | 9.6 | 18.5 | 15.5 | 22.2 | 🌑 |
| SmolLM-Instruct | 360M | 11.3 | 19.3($\dagger$) | 19.4($\dagger$) | 23.1($\dagger$) | 🌑 |
| AlphaCode | 302M | 11.6 | 18.8 | - | - | 🌕 |
| CodeT5+ | 220M | 12.0 | 20.7 | - | - | 🌑 |
| Codegen-Mono | 350M | 12.8 | 23.1 | 9.4($\dagger$) | 15.2($\dagger$) | 🌑 |
| Codex | 300M | 13.2 | 20.4 | - | - | 🌕 |
| TinyCodeLM-LintSeqInstruct | 400M | 13.4 | 20.9 | 19.4 | 29.9 | 🌑 |

To that end, we first pool the Python portions of two open-source instruction datasets for code synthesis: the GPT 3.5/4-based Magicoder instruction tuning dataset and the StarCoder2-15B-based self-alignment training dataset (Wei et al., ; ). These datasets are generated with the OSS-Instruct approach by Wei et al. () and have undergone decontamination for the benchmarks that we evaluate on in this paper. We conduct de-duplication on the pooled data to check for repeated examples. Furthermore, we strip any chain-of-thought-like natural language explanations from completion data. The resultant dataset has over 88,900 instruction-Python program pairs.

With our baseline dataset prepared, we run LintSeq to generate $s=5$ synthetic edit trajectory samples for each instruction-program pair. As described in Section 2.4, we concatenate each synthetic edit trajectory into a single string by interleaving consecutive edits with a special reserved “edit” token. Inspired by Muennighoff et al. (), we do not restrict against edit sequence repetitions. We use the popular Python linter pylint to guide edit sampling during generation. Examples of generated edit sequences and experiments testing the effect of varying $s$ are in Appendix E.

### 3.3 Finetuning Language Models on LintSeq Edit Sequences

We now probe the impact of instruction finetuning a variety of autoregressive LMs to synthesize code with edit sequences, compared to standard generation of full programs. We use two code synthesis benchmarks to support our model evaluations: HumanEval (Chen et al., ) and Mostly Basic Programming Problems (MBPP) (Austin et al., ).

Using both the code edit re-factorized and baseline instruction datasets obtained in section 3.2, we run pairs of finetuning experiments with six different models. In each experiment pair, we finetune an LM on both datasets for an equal number of optimizer steps and with the same learning rate schedule, saving intermediate checkpoints throughout finetuning. We run full benchmark evaluations on HumanEval and MBPP on each saved checkpoint(^1^11To process the generations of edit sequence LMs into executable programs, we simply resolve each of the predicted code edits one-by-one. This procedure is visualized in Figure 1 and described in Appendix A.2.), performing no prompt tuning. Then, we compare the best inference-time scaling behavior, i.e. benchmark coverage (pass@k) as a function of samples (k), obtained by edit sequence finetuning vs standard finetuning for each model. A more detailed description of the computed metrics as well as a full specification of the evaluation and finetuning procedures is provided in Appendices B and D.

Figure: Figure 4: HumanEval and MBPP coverage with repeated sampling (“pass@k vs. k”) achieved by instruction finetuning Gemma 2, Phi-3, and Llama 3.1 language models on a dataset of LintSeq edit sequence re-factored vs standard Python code (temperature $=1$, top-p $=0.95$).
Refer to caption: extracted/5927317/template/figures/diff_vs_raw_finetuning_agg_v8.png

#### 3.3.1 TinyCodeLM

We run our first two pairs of finetuning experiments on
TinyCodeLM-150M and TinyCodeLM-400M. These models were not pretrained on code synthesis instruction data, nor were they pretrained on any “diff”-like edit data. Our experimental results are summarized in Tables 1 and 2, where we compare the temperature-tuned performance of our models to the reported benchmark coverage of existing code LMs at similar parameter scales. We also report the inference-time scaling of benchmark coverage as a function of samples for our finetuned models in Appendix Tables 8 and 9.

For both the 150M and 400M parameter versions of TinyCodeLM, we find that finetuning LMs to synthesize code with edits via LintSeq data dramatically improves benchmark performance compared to baseline finetuning. Indeed, the edit sequence variants of TinyCodeLM outperform all existing code language models of comparable scale that we are aware of, including AlphaCode (Li et al., ), Codex (Chen et al., ), CodeT5+ (Wang et al., ), and the recent SmolLM-Instruct (Ben Allal et al., ). Our smaller edit sequence-finetuned model is particularly strong for its size, roughly matching or out-performing models with twice as many parameters including the 300M parameter version of Codex and the 302M-parameter version of AlphaCode (Tables 1 and 2).

#### 3.3.2 Gemma 2, Phi-3, and Llama 3.1

The results above raise a natural question: do performance improvements from finetuning LMs to synthesize code with edit sequences hold for other model scales, architectures, and tokenizers? To test this, we conduct four additional pairs of instruction finetuning experiments on LMs from three model families, Gemma 2, Phi-3, and Llama 3.1, ranging in size from 2.6B to 14B. We employ pretrained-only model weights, if available. The results of these experiments are in Figure 4, where we plot zero-shot benchmark coverage as a function of samples for instruction finetuned models. Raw coverage scores are reported in Appendix Tables 10 and 11.

Most of our findings echo those of Section 3.3.1. Aggregating across sample counts, we find that finetuning models to synthesize code with edits improves overall zero-shot performance on HumanEval and MBPP compared to finetuning on the original data. This suggests that re-factoring code with edit sequences is an architecture- and tokenizer-independent mechanism for improving downstream LM outputs. Furthermore, as shown in Figure 4 and Tables 10 and 11, we find that the degree by which edit sequence LMs outperform baseline model variants increases with repeated sampling for all tested models, culminating in an average absolute gain in pass@50 of +20% ($\pm$ 3%) on HumanEval and +12% ($\pm$ 2%) on MBPP. This observation confirms the hypothesis posed in Section 2, showing that training LMs to synthesize code with edits using LintSeq data improves the relationship between cumulative inference-time compute and zero-shot performance.

At pass@1, however, our results are slightly more mixed than in Section 3.3.1. For Phi-3 models, we observe either no difference or a decrease in score between each pair of model-data variants. One explanation for this is bias: the Phi-3 models have been previously instruction finetuned and were likely to have been trained on standard code data, putting LM variants finetuned to generate edit sequence re-factorized code at a comparative disadvantage due to distributional shift.

Figure: Figure 5: HumanEval and MBPP “pass@k vs. k” achieved by finetuning TinyCodeLM models on linter-guided vs randomly sampled code edit sequences (temperature $=1$, top-p $=0.95$).
Refer to caption: extracted/5927317/template/figures/linter_ablation_v12.png

### 3.4 Ablating Linter-Guidance

The backward sampling phase of LintSeq uses a linter or other verifier to group interdependent lines of a program together during edit sequence generation, ensuring that all code edits resolve to programs that are free of static errors. We conclude our experiments by testing the importance of this design choice with TinyCodeLM models.

To do this, we replace the backwards procedure described in Section 2.2 with exclusively random sampling; during each step of the algorithm, we first sample the number of lines to delete from the current program uniformly at random, before sampling a set of lines with the desired count. Using this linter-ablated version of the algorithm, we generate a new synthetic edit sequence dataset with the same size as the LintSeq dataset used in all previous finetuning experiments, i.e. with $s=5$ example sequences per sample in the source dataset. The average number of edits per example in this dataset ($\overline{E}_{\text{RandSeqInstruct}}=3.9$) is empirically similar to its linter-guided counterpart ($\overline{E}_{\text{LintSeqInstruct}}=3.8$, see Figure 3). We employ the same procedure as the one used in Section 3.3 to instruction finetune TinyCodeLM models on the dataset of randomly sampled edit sequences.

In Figure 5, we compare the temperature $1$ inference-time scaling laws on HumanEval and MBPP obtained by finetuning models on randomly sampled vs static error-free edit sequences. Raw model scores are also provided in Appendix F, Tables 8 and 9. Ablating linter-guidance results in a decline in benchmark coverage. On HumanEval, linter ablation reduces absolute pass@50 score on TinyCodeLM-150M ($22.6\%\mapsto 17.7\%$) and on TinyCodeLM-400M ($26.8\%\mapsto 22.0\%$). MBPP pass@50 is similarly affected, dropping for both models ($34.5\%\to 30.2\%$ and $39.6\%\mapsto 34.5\%$). These results suggest that the error-free nature of edits in LintSeq instruction finetuning data does indeed have a positive impact on the coding problem coverage of sampled solutions.

To conclude our analysis, we probe whether training models on error-free edit sequences also has an effect on the presence of errors in across all generated programs, aside from its positive effects on coverage. To assess this, we run the Python linter pylint over each of the synthesized programs used to compute the reported temperature $1$ pass@k metrics, checking code for static errors.

In Figure 6, we plot the total proportions of synthesized program samples with at least one static error across finetuned model variants. On both benchmarks, LMs trained on randomly sampled edits (dark grey) appear to generate “buggy” code with much higher frequency than all other models. Furthermore, on HumanEval, we find that LintSeq models (indigo) synthesize programs with static errors at a higher frequency than baseline models (light grey), despite their higher coverage of benchmark coding problems. This additional finding suggests that model performance gains from LintSeq cannot simply be attributed to improvement in static error frequency across code – training on re-factored code must be helping models write generally better, more diverse programs.

In summary, the error-free nature of the linter-guided edits sampled in LintSeq appears to indeed be important for improving both the quality and diversity of sampled programs (Figure 5), as well as the overall correctness (Figure 6) of code synthesized by language models trained on edit sequences.

Figure: Figure 6: Comparing static error frequency in synthesized code samples across baseline vs edit sequence instruction finetuned model variants (n $=50$, temperature $=1$, top-p $=0.95$).
Refer to caption: extracted/5927317/template/figures/code_error_rate_v5.png

## 4 Related Work

##### Foundation Models for Code

Code synthesis is one of the oldest problems in computer science. Neural language model-based approaches such as Codex, AlphaCode, CodeT5+, CodeGen, StarCoder, and Code Llama have recently proven to be extremely competitive with previous methods (Chen et al., ; Li et al., ; Wang et al., ; Nijkamp et al., ; Li et al., ; Roziere et al., ). Today, foundation models trained on web text and code data dominate, and LLM-powered code editing tools like Github Copilot and Cursor are used by thousands of engineers every day (Heaven, ). Many general-purpose LLMs are also trained on code data. While the largest of these LLMs show strong performance on coding benchmarks, generations continue to suffer from limited meaningful output diversity, prompt sensitivity, and degrading quality on long-contexts (Achiam et al., ; Gemini Team et al., ; Dubey et al., ). Smaller models also lag behind (Abdin et al., ; Gemma Team et al., ; Ben Allal et al., ). As of the writing of this paper, directly prompting LLMs to generate code “diffs” results in low quality edits across models (Sanger, ). We claim that this is the result of a data problem and we attempt to address it in this work.

##### Finetuning on Synthetic Data

LLM post-training methods like supervised finetuning have been shown to be extremely powerful for improving model performance across tasks (Wei et al., ). However, high-quality datasets of paired instruction-response examples are extremely expensive to curate. One possible solution lies in synthetic data generation methods like Self-Instruct, wherein an LLM is prompted to generate instructions and/or responses from examples (Wang et al., ). Such data have been used extensively for improving LLM performance through self-refinement and/or knowledge distillation on coding tasks (Chaudhary, ; Roziere et al., ; Abdin et al., ; Lozhkov et al., ). We employ post-processed instruction data for code synthesis created with a method from this family, OSS-Instruct (Wei et al., ), as the base of our experiments on re-factorizing code with code edit sequences via LintSeq. Unlike Self-Instruct-like synthetic data generation methods, our algorithm does not employ an LLM for data generation, and instead generates examples of error-free edit sequences from existing code data by using a simple linter.

##### Finetuning on Edits

Several works have investigated finetuning code LLMs on edit data. Notably, Muennighoff et al. () instruction tune models on a 4TB dataset of GitHub commits pairing code changes with human instructions. Relatedly, Li et al. () use GitHub commit data sourced from Python repositories to generate code editing instruction data with GPT 3.5/ChatGPT. Both of these works specifically focus on better-equipping LLMs for natural language-prompted code editing tasks, in which a model is explicitly prompted to generate an edit in response to a natural language specification. Our work differs in three important ways: first, we study edit sequences rather than single edits; second, we train LLMs to predict edits implicitly during code synthesis; third, our synthetic edit generation algorithm does not rely on the existence of any kind of commit data.

##### “On Device” Language Models

As the capabilities of LLMs have improved, so to have those of small language models. Recent projects like SmolLM (Ben Allal et al., ) and OpenELM (Mehta et al., ) re-examine the potential of tiny language models that can be run and even updated “on-device,” i.e. on a smart phone or laptop. The representations learned by such models during pretraining are weaker than those of scaled-up LLMs (Kaplan et al., ). This is particularly true for harder tasks that involve reasoning, such as code synthesis (Gemma Team et al., ; Abdin et al., ). To our knowledge, the most recent open-source work studying small language models pretrained entirely for code understanding is from several years ago (Xu et al., ; Nijkamp et al., ; Wang et al., ; ). The 150M and 400M parameter TinyCodeLM models pretrained in this paper belong to the “on device” model family and build upon previous works. These models provide an efficient test-bed for experiments on LM code synthesis that is updated to recent advancements in high throughput pretraining and to improvements in open-source data quality.

##### Inference-Time Compute Scaling

The performance of language models can be boosted during inference by using scaled-up sample counts, hand-engineered prompting schema, and/or search (Brown et al., ; Snell et al., ). These methods dramatically increase inference costs. Their effectiveness is tightly linked to the expressivity of learned model representations and the diversity of outputs across samples. Our experiments with smaller language models are inspired by these works – we study whether it is possible to (1) improve the expressivity of representations for code synthesis across LM parameter scales during finetuning, and (2) take advantage of this property to improve the inference-time performance of smaller LMs by larger margins during repeated sampling.

## 5 Discussion, Limitations, and Conclusion

This paper introduces an algorithm, LintSeq, for generating synthetic code edit sequences from existing programs. LintSeq enables LLM reasoning settings like code synthesis to be re-parameterized at the data-level as sequential edit generation tasks. The algorithm is parameter-free, requires only CPU to run, and makes no assumptions about the content or structure of code files.

Re-parameterizing code generation with edits has a few immediate benefits. For example, it makes code generation with LLMs much more controllable at the prompt-level (Appendix A.3) and it reduces the cost of predicting useful and correct code insertions with models, since synthetic edit-trained LLMs do not need to be prompted to re-synthesize entire programs from scratch (Section 2.4).

In our experiments with LintSeq, we also show the following:

- 1.
Tiny LMs can be efficiently finetuned to synthesize Python programs with edit sequences via LintSeq data. This results in state-of-the-art code benchmark performance for models that can be trained “on device” (Sections 3.1 and 3.3.1).
- 2.
Across other tested models from the Phi, Gemma, and Llama families, finetuning on LintSeq data also improves the diversity of zero-shot generations, boosting the inference-time scaling of coverage on HumanEval and MBPP at fixed sample counts (Section 3.3.2).
- 3.
On HumanEval, the cumulative inference cost of repeatedly sampling from small edit sequence LLMs is similar to sampling once from larger LLMs and yields coverage that is competitive with GPT-4, GPT-4-Omni, and Llama 3.1 405B (Figure 1, Appendix F.4).
- 4.
Ablating linter-guidance from LintSeq hurts the quality, diversity, and correctness of code synthesized by instruction finetuned models (Section 3.4).

There are several limitations to our work.

First, as currently formulated, LintSeq can only be used to generate synthetic sequences of insertion edits. This is a consequence of the parameter-free nature of the algorithm – every edit in a LintSeq sequence reflects an existing line of code in the source file used to generate it. As a result, LintSeq edit data can only be used to train models for synthesis-like tasks, rather than for refinement.

Second, our experiments with LintSeq study code synthesis in Python only. We look forward to testing LintSeq with other programming languages, verifiers, and problems in future work. One other exciting setting where LintSeq might improve the tradeoff between model generation quality and inference-time compute is in mathematical reasoning and formal theorem proving.

## Ethics Statement

This work explores data-driven mechanisms for improving the quality of language model-generated code. Our synthetic data generation method relies on open-source data and our experiments leverage open-source software and resources. It is important to acknowledge that all language models for code synthesis have the potential to be misused – whether intentionally or unintentionally – for generation of code with vulnerabilities and/or malicious behaviors. Any and all model generated code has the potential to be harmful and must not be executed without precautions.

## Acknowledgements

This work was supported by grants from NSF award 2339096 and ONR awards N00014-21-1-2758 and N00014-22-1-2773. We are grateful to Shenglong Wang and NYU High Performance Computing for their support of this project. UP is funded by an NSF GRFP Award, and LP is funded by the Packard Fellowship. We would like to thank Nate Rahn, Mahi Shafiullah, and David Brandfonbrener for helpful comments and discussions.

## Appendix A More on Edit Sequences and Diffs

### A.1 Reading Unix Diffs

We provide a guide to reading Unix-style diffs below in Figure 7. The diff shown in this figure is computed using the Python library difflib, which is the implementation that we use to compactly represent edits in our synthetic data generation experiments. Note that the total extra tokens present in an insertion edit sequence representation of a program scales with the number of program lines $L$, and can be upper-bounded as $T_{\text{diff}}\leq L\cdot((\text{chars in ``decorator''})+(\text{extra chars
per line in ``body''}))=16L$.

Figure: Figure 7: The anatomy of a Unix diff: A diagrammatic visualization of the different parts of a Unix-style diff, as computed by difflib. The body of a diff can consist of multiple line deletions, followed by multiple line insertions. The decorator portion of the diff respectively indicates the location and size of these deletions and insertions, if any. Like the diff shown above, the edits in synthetic edit sequences generated by LintSeq consist of line insertions only.
Refer to caption: extracted/5927317/template/figures/diff_anatomy_v2.png

### A.2 Resolving Edit Sequences

During inference, LMs that have been finetuned on LintSeq instruct data will synthesize code via edit sequences, outputting text strings that consist of a sequence of consecutive Python diffs interleaved with newline characters and “
” tokens, similar to  Piterbarg et al. (). Each of these diffs will be structured as shown in Figure 7, if correctly formatted by the language model.

Resolving an edit sequence generated by a language model into an executable Python program is simple: starting with an empty program, we consecutively apply the line insertions and/or deletions in the body of each diff to the lines of the program specified in its decorator. We continue this process until all of the diffs in the generated edit sequence have been parsed and resolved.

Figure 1 shows a code edit sequence generation from a LintSeq instruction finetuned LM and the corresponding resolved, executable Python program.

### A.3 Controllability of Code Synthesis with Edit Sequence LMs

The structure of Unix-style diffs affects the downstream controllability of code synthesis with models that have been trained on edit sequence re-parameterized programs. As shown in Figure 7, the first line of every diff is a decorator that describes the location and the numbers of lines changed by the edit. During inference, autoregressive language models that have been trained on Unix-style diffs with this format can be prompted to predict an edit in any desired target location within the program being synthesized by “intervening” on a model generation.

### A.4 Future Work: Searching in Edit Space

If we apply the lens of reinforcement learning or search to this setting, we might say that re-parameterizing the code data used to train a language model re-parameterizes the model’s generative action space. It is possible that combining edit sequence LMs with more sophisticated decoding mechanisms, inference-time search, and/or interactive post-training may result in even larger improvements to the quality of generated code than those of the zero-shot code synthesis settings studied in this paper. We look forward to testing this hypothesis in future work.

## Appendix B Evaluation

HumanEval (Chen et al., ) and Mostly-Basic Programming Problems (MBPP) (Austin et al., ) are two of the most studied benchmarks for evaluating code LMs (Liu et al., ). These benchmarks probe the code synthesis capabilities of models, and consist of pairs of natural language program descriptions and test-cases. We employ the extended MBPP test cases released as MBPP(+) by Liu et al. () to add additional rigour to our testing procedure. All of the code LMs that we compare our models against evaluate HumanEval performance using the original set of benchmark test cases; for consistency, we employ these same test cases in our evaluations when comparing the performance of our models to the reported scores of external LMs.

During testing on both HumanEval and MBPP(+), LMs are prompted to generate outputs using the natural language descriptions of target programs. Their outputs are then evaluated on the paired test cases. A generation is considered “correct” if and only if it passes all of the test cases upon execution, subject to a fixed timeout setting. Previous works on code synthesis with language models report scores on HumanEval and MBPP(+) across samples. The most common of these metrics is known as pass@k (Chen et al., ; Austin et al., ; Li et al., ; Wang et al., ). This is the metric that we use to report and compare model performance throughout this paper. Our implementation of HumanEval and MBPP(+) evaluations mimics the Big Code Evaluation Harness by Ben Allal et al. (). We do not allow models to use chain-of-thought during generation.

### B.1 Prompting

The primary goal of this paper is to introduce a method for re-factorizing code synthesis with LMs by finetuning them on synthetic instruction data. As a result, we evaluate all models using minimal prompt formats, performing no prompt tuning (see Figures 9 and 10). Examples of the prompt formats that we use during evaluation are shown in Figure 8.

Figure: Figure 8: Examples of formatted HumanEval and MBPP(+) prompts used in model evaluations.
Refer to caption: extracted/5927317/template/figures/prompting_examples_v2.png

We finetune all tested models on example outputs exclusively corresponding to Python code, and as a result, we do not use Markdown formatting to separate Python code from natural language in either our instruction data nor in our inference-time prompts.

To evaluate models on HumanEval, we use both the default “Python version” prompt format in the original benchmark dataset, where a natural language program description is provided to an LM within a docstring, as well as the equivalent, fully natural language prompt format from HumanEvalPack (Muennighoff et al., ). The latter format is similar to the structure of the instructions in our finetuning datasets.

To evaluate models on MBPP(+), we use the default prompts from the MBPP benchmark dataset, formatted with specification of the target function name and arguments both inside and outside of the natural language instruction, as shown in Figure 8.

During LM benchmark evaluations, we test models on each of the format variants described above and report scores on the better performing variant for each benchmark only.

### B.2 Generation and Parsing

During generation, we continue decoding until an end-of-sequence token is output by an LM. We treat all LM outputs as either Python code or sequences of Python code edits, depending on whether an LM was finetuned on standard instruct or LintSeq instruct data. In the latter case, we post-process outputs by resolving the output edit sequences using the procedure described in Appendix A.2.

### B.3 Evaluating Model Checkpoints

#### B.3.1 Philosophy

There is a well-known trade-off between the temperature used for sampling from autoregressive code LMs and the benchmark coverage achievable by models, i.e. the proportion of problems “pass@k” for which an LM is able to generate at least one output that passes all test cases given “k” tries. This trade-off was first described by Chen et al. (). Informally, increasing the sampling temperature increases the width of the distribution from which tokens are sampled, producing more diverse but noisier (and possibly lower quality) generations. For larger repeated sample counts, the pass@k score typically increases with sampling temperature up to some threshold, beyond which the negative effects of noise overpower the positive effects of diversity. The benchmark coverage achievable by an LM at any temperature and in the limit of samples, i.e. on pass@k for $k\uparrow\infty$, ultimately depends on both the power and expressivity of the code language model’s learned representation.

From a practical perspective, while smaller language models may have weaker representational power than larger models, the representational expressivity of the former may enable them to overtake the latter at fixed computational budgets by leveraging extra compute at inference-time, e.g. generating a larger number of samples per problem and using the provided test cases to check each one for correctness before returning an output (Brown et al., ; Snell et al., ). For example, an LLM that has an 85$\%$ pass@1 score on an arbitrary task may be more expensive in total serving cost (see Figure 1) than a smaller LM with a $90\%$ pass@50 score on the same task. A small LM can only have this property, however, if it exhibits a reliable trade-off between generation quality and inference-time sampling cost across tasks. In other words, its representation must be sufficiently expressive.

#### B.3.2 Computing Coverage (Pass@K)

Our goal is to probe whether re-parameterizing code synthesis with edit sequences can improve the expressivity of smaller LLM representations, boosting benchmark coverage as a function of samples-per-problem. Hence, we primarily compare finetuned models by evaluating them with the procedures described above on HumanEval and MBPP(+) at a high temperature and a large sample count, computing pass@k for k$\ \in\{1,5,10,20,50\}$ with $N=50$ samples(^2^22These are the largest sample counts that are feasible to compute on our hardware given the scope of our experiments.) at temperature 1, top-p 0.95. We compute pass@k statistics with the same procedure as Chen et al. (). The results of these evaluations are reported throughout the paper and shown in Figures 4, 5 and Tables 8, 9, 10, 11. In each of these Figures and Tables, we identify the most performant checkpoint from each model-data finetuning run by comparing pass@50 score at temperature 1 on HumanEval and MBPP(+) across checkpoints.

Many existing state-of-the-art code synthesis LMs only report temperature-tuned pass@k scores on HumanEval, including Codex, AlphaCode, and Codegen-Mono (Chen et al., ; Li et al., ; Nijkamp et al., ). Thus, in Tables 1 and 2, we select the single best overall checkpoint of each TinyCodeLM model by averaging HumanEval and MBPP(+) pass@50 score. Then, we temperature-tune each checkpoint’s pass@1 and pass@10 scores when reporting results. On HumanEval, we test temperatures $\tau\in\{0.0,0.2,0.4,0.8,1.0\}$. On MBPP(+), we sweep over a smaller temperature range, $\tau\in\{0.0,0.1,1.0\}$. We perform the same temperature tuning procedure when reporting external model benchmark scores as well, i.e. the scores annotated with “$(\dagger)$” in Tables 1 and 2. When running benchmark evaluations with these external code LMs, we stray from the prompt formatting, generation, and parsing procedures described in Appendices B.1 and B.2; instead, in the interest of a fair evaluation, we reproduce the conventions reported by model authors to report other scores.

## Appendix C Pretraining

We rely on data and libraries open-sourced by the HuggingFace, FineWeb, StarCoder, Dolma, OLMo, and PyTorch FSDP projects to pretrain our models (Wolf et al., ; Penedo et al., ; Lozhkov et al., ; Soldaini et al., ; Groeneveld et al., ; Zhao et al., ).

### C.1 Model Architectures and Pretraining Hyperparameters

**Table 3: Architectural and pretraining hyperparameters of our “on device” 150M and 400M parameter TinyCodeLM models, pretrained on a mixture of Web text and code for Python understanding.**
|  | TinyCodeLM |  |
| --- | --- | --- |
|  | Smallest, 150M Parameters | Small, 400M Parameters |
| Transformer Architecture | decoder-only | decoder-only |
| Model Family | OlmoForCausalLM | OlmoForCausalLM |
| Tokenizer | GPT-NeoX-20B-OLMo | GPT-NeoX-20B-OLMo |
| Attention Bias | False | False |
| Attention Dropout | 0.0 | 0.0 |
| Hidden Activation | SwiGLU | SwiGLU |
| Hidden Size | 768 | 1024 |
| Intermediate Size | 3072 | 4096 |
| Number of Attention Heads | 12 | 16 |
| Number of Hidden Layers | 12 | 24 |
| Number of Key-Value Heads | 12 | 16 |
| Vocabulary Size | 50304 | 50304 |
| Positional Encodings | Rotary (RoPE) | Rotary (RoPE) |
| Mixed Precision | BFLOAT16 | BFLOAT16 |
| Weight Tying | True | True |
| Flash Attention 2 | True | True |
| Optimizer | AdamW | AdamW |
| Learning Rate | 0.0003 | 0.0003 |
| Weight Decay | 0.01 | 0.01 |
| Betas | $(0.9,0.95)$ | $(0.9,0.95)$ |
| Epsilon | 1.0e-05 | 1.0e-05 |
| Learning Rate Scheduler | cosine (with warmup) | cosine (with warmup) |
| Number of Warm-Up Steps | 100 | 100 |
| Alpha-f ($\alpha_{f}$) | 0.1 | 0.1 |
| Total Epochs of Pretraining | 2 | 2 |

### C.2 Pretraining Data Mix

**Table 4: Pretraining data mix used to train both TinyCodeLM models. Datasets were tokenized and prepared using HuggingFace and Dolma tooling (Wolf et al., ; Soldaini et al., ).**
| Pretraining Data Source | Subset | Tokens | Documents |
| --- | --- | --- | --- |
| FineWeb (Penedo et al., ) | 10BT Sample | 10.4BT | 14.9M |
| The Stack (Kocetkov et al., ) | Python Only | 61.8BT | 24.2M |

## Appendix D Instruction Finetuning

### D.1 Baseline Instruction Dataset

Table 5 displays the data sources that are used to prepare the dataset described in Section 3.2. These data are pooled and preprocessed into instruction-program pairs by stripping away Markdown formatting and natural language explanations from completions (Figure 9 and 10). In our experiments, we use the resultant data to finetune baseline models, comparing their performance to those of LMs finetuned on edit sequences generated with LintSeq from the same set of instruction-program pairs.

**Table 5: Instruction data mix used to prepare the baseline instruction dataset in Section 3.2.**
| HuggingFace Instruction Data Source | Subset | Examples |
| --- | --- | --- |
| bigcode/self-oss-instruct-sc2-exec-filter-50k | Full | 50,661 |
| ise-uiuc/Magicoder-OSS-Instruct-75K | Python | 38,284 |
|  |  | 88,945 |

### D.2 Procedures and Hyperparameters

We instruction finetune all models with Microsoft DeepSpeed using the ZeRO++ protocol for stage three sharding. For the largest of these models, we also use CPU parameter offloading to accelerate experiments (Wang et al., ; Ren et al., ). When finetuning models on LintSeq data, we add a new token “
” to tokenizers (Section 2.4) and resize model embeddings accordingly.

In our experiments with Gemma 2, Phi-3, and Llama 3.1 models, we use HuggingFace to access and load pretrained model weights and tokenizers. As mentioned in the main body of the paper, we instruction finetune pretrained-only weights if open-sourced and available. This is the case for Gemma 2 and Llama 3.1 only, as of the writing of this paper.

Across all of the finetuning experiments conducted in this paper, we train model-data variants with the same batch size and for an equal number of total optimizer steps. This optimizer step count corresponds to ten epochs of finetuning with the baseline instruction tuning dataset described in Section 3.2. We save intermediate checkpoints at equal optimizer step intervals in all experiments.

In order to tune the peak learning rates used in each set of model experiments, we run a full sweep $\alpha\in\{$6e-4, 3e-4, 1e-4, 5e-5, 1e-5, 5e-6$\}$ in the baseline instruction data setting for each model. We select peak learning rate values by tracking the best-achieved downstream benchmark performance across models. The chosen values are displayed in Table 6. All other finetuning hyperparameters are kept fixed at the settings in Table 7 across experiments.

**Table 6: Peak learning rates used to instruction finetune models.**
|  | TinyCodeLM | Gemma 2 | Phi-3 | Llama 3.1 |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | 150M | 400M | 2B | 3.8B | 14B | 8B |
| Peak Learning Rate ($\alpha$) | 3e-4 | 3e-4 | 5e-5 | 5e-5 | 1e-5 | 1e-5 |

**Table 7: All other instruction finetuning settings, re-used across experiments.**
|  | Hyperparameter Setting |
| --- | --- |
| Learning Rate Scheduler | linear |
| Warmup Ratio | 0.001 |
| Weight Decay | 0.01 |
| Total Batch Size | 512 |
| Batch Loss Reduction | sum |
| Mixed Precision | BFLOAT16 |
| Max Sequence Length | 1024 |
| Total Optimizer Steps | 1740 |

## Appendix E More on Synthetic Data Generation with LintSeq

### E.1 Examples of Generated Synthetic Edit Trajectories

Figure: Figure 9: LintSeq edit sequence samples vs baseline instruction-program data, example A.
Refer to caption: extracted/5927317/template/figures/data_example_A_v5.png

Figure: Figure 10: LintSeq edit sequence samples vs baseline instruction-program data, example B.
Refer to caption: extracted/5927317/template/figures/data_example_B_v5.png

### E.2 Tuning LintSeq Example Count

Figure: Figure 11: Probing the effect of varying the number of edit sequences sampled with LintSeq per instruction-example pair during data generation: Using the source dataset described in Section 3.2, we sweep over the value of the LintSeq parameter $s$ used during synthetic data generation to yield three different edit sequence instruction datasets with $s\in\{1,5,10\}$. We finetune TinyCodeLM models on each of these datasets, and compare the resultant HumanEval and MBPP(+) performance vs samples (i.e. pass@k vs k) at temperature 1. The most performant values is $s=5$.
Refer to caption: extracted/5927317/template/figures/tuning_lintseq_sample_count_v6.png

## Appendix F Additional Results

### F.1 Pretraining TinyCodeLM

Figure: Figure 12: Evaluating the zero-shot Python synthesis capabilities of TinyCodeLM-150M during pretraining on HumanEval and MBPP(+).
Refer to caption: extracted/5927317/template/figures/tinycodelm150_pt_curves_v1.png

Figure: Figure 13: Evaluating the zero-shot Python synthesis capabilities of TinyCodeLM-400M during pretraining on HumanEval and MBPP(+).
Refer to caption: extracted/5927317/template/figures/tinycodelm400_pt_curves_v0.png

### F.2 Finetuning TinyCodeLM

#### F.2.1 Inference-Time Scaling Laws

**Table 8: HumanEval fixed-temperature coverage scaling results achieved by finetuning TinyCodeLM models (zero-shot, temperature = 1, top-p = 0.95).**
|  |  |  | HumanEval |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model Variant | Size | Linter<br>Guided | pass@1 | pass@5 | pass@10 | pass@20 | pass@50 |
| tinycodeLM-Instruct | 150M | - | 6.2 | 10.4 | 12.3 | 14.5 | 18.3 |
| tinycodeLM-RandSeqInstruct | 150M | ✗ | 4.0 | 9.7 | 11.9 | 14.4 | 17.7 (-0.6) |
| tinycodeLM-LintSeqInstruct | 150M | ✓ | 6.4 | 14.3 | 17.7 | 20.4 | 22.6 (+4.3) |
| tinycodeLM-Instruct | 400M | - | 6.8 | 11.4 | 13.7 | 16.2 | 18.9 |
| tinycodeLM-RandSeqInstruct | 400M | ✗ | 7.2 | 12.7 | 15.6 | 18.6 | 22.0 (+3.1) |
| tinycodeLM-LintSeqInstruct | 400M | ✓ | 7.2 | 14.9 | 18.2 | 21.5 | 26.8 ( +7.9) |

**Table 9: MBPP(+) fixed-temperature coverage scaling results achieved by finetuning TinyCodeLM models (zero-shot, temperature = 1, top-p = 0.95).**
|  |  |  | MBPP(+) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model Variant | Size | Linter<br>Guided | pass@1 | pass@5 | pass@10 | pass@20 | pass@50 |
| tinycodeLM-Instruct | 150M | - | 7.3 | 16.4 | 20.4 | 24.6 | 30.6 |
| tinycodeLM-RandSeqInstruct | 150M | ✗ | 4.3 | 13.1 | 18.4 | 23.7 | 30.2 (-0.4) |
| tinycodeLM-LintSeqInstruct | 150M | ✓ | 7.5 | 18.2 | 23.4 | 28.5 | 34.5 (+3.9) |
| tinycodeLM-Instruct | 400M | - | 9.3 | 17.9 | 22.2 | 26.8 | 32.4 |
| tinycodeLM-RandSeqInstruct | 400M | ✗ | 5.5 | 15.9 | 21.5 | 27.0 | 34.5 (+2.1) |
| tinycodeLM-LintSeqInstruct | 400M | ✓ | 13.1 | 25.2 | 29.9 | 34.4 | 39.6 (+7.2) |

### F.3 Finetuning Gemma 2, Phi-3, and Llama 3.1

#### F.3.1 Inference-Time Scaling Laws

**Table 10: HumanEval fixed-temperature coverage scaling results achieved by finetuning Gemma 2, Phi-3, and Llama 3.1 models (zero-shot, temperature = 1, top-p = 0.95).**
|  |  | HumanEval |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model Variant | Parameters | pass@1 | pass@5 | pass@10 | pass@20 | pass@50 |
| Gemma-2-Instruct | 2.6B | 16.1 | 26.3 | 30.5 | 34.6 | 40.2 |
| Gemma-2-LintSeqInstruct | 2.6B | 21.6 | 35.0 | 41.2 | 48.3 | 58.5 (+18.3) |
| Phi-3-Mini-Instruct | 3.8B | 40.1 | 48.8 | 52.1 | 55.0 | 57.3 |
| Phi-3-Mini-LintSeqInstruct | 3.8B | 38.3 | 63.6 | 72.0 | 79.0 | 86.6 (+29.3) |
| Llama-3.1-Instruct | 8B | 34.2 | 50.3 | 55.6 | 59.7 | 63.4 |
| Llama-3.1-LintSeqInstruct | 8B | 37.6 | 61.1 | 69.2 | 75.3 | 82.3 (+18.9) |
| Phi-3-Med-Instruct | 14B | 52.8 | 67.9 | 72.0 | 75.0 | 77.4 |
| Phi-3-Med-LintSeqInstruct | 14B | 44.7 | 73.5 | 80.2 | 84.5 | 90.2 (+12.8) |

**Table 11: MBPP(+) fixed-temperature coverage scaling results achieved by finetuning Gemma 2, Phi-3, and Llama 3.1 models (zero-shot, temperature = 1, top-p = 0.95).**
|  |  | MBPP(+) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model Variant | Parameters | pass@1 | pass@5 | pass@10 | pass@20 | pass@50 |
| Gemma-2-Instruct | 2.6B | 22.8 | 34.2 | 37.8 | 41.2 | 44.6 |
| Gemma-2-LintSeqInstruct | 2.6B | 27.7 | 41.3 | 46.3 | 50.5 | 54.7 (+10.1) |
| Phi-3-Mini-Instruct | 3.8B | 35.2 | 45.9 | 49.1 | 52.0 | 56.1 |
| Phi-3-Mini-LintSeqInstruct | 3.8B | 39.3 | 57.5 | 62.4 | 65.9 | 69.4 (+13.3) |
| Llama-3.1-Instruct | 8B | 38.4 | 50.2 | 53.8 | 56.6 | 59.4 |
| Llama-3.1-LintSeqInstruct | 8B | 38.5 | 56.5 | 61.6 | 65.7 | 69.8 (+10.4) |
| Phi-3-Med-Instruct | 14B | 40.3 | 51.5 | 54.2 | 56.3 | 59.0 |
| Phi-3-Med-LintSeqInstruct | 14B | 40.0 | 60.9 | 67.0 | 71.5 | 75.9 (+16.9) |

### F.4 Computing HumanEval Coverage vs Cumulative Inference-Time FLOPs

In Figure 1, we plot HumanEval coverage as a function of cumulative inference-time FLOPs, comparing the performance and total cost of repeatedly sampling from our instruction finetuned Phi-3 and Llama 3.1 models vs sampling a single generation per problem from very large models like Llama 3.1 405B (Dubey et al., ) and Nemotron 4 340B (Adler et al., ).

We use the approximations below, drawn from Kaplan et al. (), to conservatively estimate the cumulative inference costs of synthesizing solutions to all of the $164$ HumanEval benchmark problems across different models. The models that we compare are all dense transformers, where the majority of the parameters are used in matrix multiplications.

$$ FLOPs per token $\displaystyle\approx 2\cdot(N_{\text{model-params}}+2\cdot L_{\text{model- layers}}\cdot C_{\text{context}})$ Total FLOPs $\displaystyle\approx\text{FLOPs per token}\cdot T_{\text{avg-total-tokens-per- sample}}\cdot K_{\text{samples}}\cdot M_{\text{problems}}$ $\displaystyle\approx\text{FLOPs per token}\cdot T_{\text{avg-total-tokens-per- sample}}\cdot K_{\text{samples}}\cdot 164$ $$

For our instruction finetuned models, we determine the quantities $T_{\text{avg-total-tokens-per-sample}}$ by computing token counts over all sets of samples per problem that we obtained to compute the coverage statistics in Figure 4 and Table 10 above. These token statistics are provided in the table below.

**Table 12: HumanEval total tokens per sample for finetuned Gemma 2, Phi-3, and Llama 3.1 models (zero-shot, temperature $=1$, top-p $=0.95$). These counts reflect prompt and completion tokens. They are computed using the same samples whose coverage statistics are reported in Table 10.**
| Model Variant | Parameters | Avg. Total Tokens<br>Per HumanEval Sample ($n=50$) |
| --- | --- | --- |
| Gemma-2-Instruct | 2.6B | 172 |
| Gemma-2-LintSeqInstruct | 2.6B | 205 |
| Phi-3-Mini-Instruct | 3.8B | 218 |
| Phi-3-Mini-LintSeqInstruct | 3.8B | 247 |
| Llama-3.1-Instruct | 8B | 178 |
| Llama-3.1-LintSeqInstruct | 8B | 201 |
| Phi-3-Med-Instruct | 14B | 230 |
| Phi-3-Med-LintSeqInstruct | 14B | 247 |

Note that edit sequence (i.e. LintSeqInstruct finetuned) LMs have slightly higher average token counts per sample due to presence of “diff” descriptor tokens in generations (see Appendix A).

We report zero-shot HumanEval coverage for external models by using the evaluation statistics from Dubey et al. () (Table 18, column one). To estimate cumulative inference-time FLOPs for these models, we employ the approximation expression above and estimate $T_{\text{avg-total-tokens-per-sample}}\approx 200$, reflecting an ensemble over the per sample token counts of standard “Instruct” finetuned models shown in Table 12. Note that this ensembled statistic reflects program generations without chain-of-thought only. As a result, we believe it to be a conservative estimator.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - What are the computational resources required to synthesize edit sequences using LintSeq? If it is cost friendly, will it also be able to improve pre-training performance?
soundness: 2
strengths: - The paper explores an interesting and important research direction.
- The idea of constructing fine-tuning datasets that mimics human developers’ behavior by synthesizing edit sequences is novel.
summary: This paper proposes LintSeq, which is a synthetic data generation algorithm that generates edit sequences from raw code to construct training datasets for LLMs. Specifically, to let LLMs learn to write code by editing existing programs like human developers, LintSeq first refactors existing code into a sequence of code edits, where each edit is an error-free insertion, and then outputs edit sequences as text strings consisting of consecutive program diffs. Evaluation shows that, after training LLMs on the synthetic edit sequences, the inference-time scaling performance of LLMs on HumanEval and MBPP improves significantly.
weaknesses: - The synthesized edit sequences may not be reasonable. While LintSeq ensures that the program at each step of the edit sequence does not contain any static error, it does not necessarily guarantee that this edit sequence is reasonable because a valid edit sequence should reflect the reasoning/thinking process of human developers, which is not guaranteed by LintSeq. Furthermore, edit sequences synthesized by LintSeq only contain insertions, which is also different from real-world edit sequences.
- The comparison between “LintSeqInstruct” and “Instruct” may not be fair enough. While authors have controlled the number of optimizer steps the same, it is also important to make sure that “LintSeqInstruct” and “Instruct” share the same valid training tokens (i.e., the number of tokens in the training sequences that are not padding tokens). Although sharing the same number of optimizer steps, it is still possible that each training sequence in “LintSeqInstruct” is much longer than that in “Instruct” and thus more training tokens are provided for “LintSeqInstruct”.
- Evaluation is not comprehensive enough. Apart from HumanEval(+) and MBPP(+), more benchmarks should be included to further demonstrate the effectiveness of “LintSeqInstruct”, such as DS-1000, APPS, and LiveCodeBench.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Generally, this is a re-hashing of the weaknesses

1. Could you please describe the algorithm for LintSeq more clearly, and I'd strongly also recommend updating the paper to reflect a more clear explanation (e.g. PseudoCode or a more lengthy explanation in natural language in an appendix section). 

2. Did I misrepresent or misunderstand your claims? Please feel free to disagree if I have missed something.

3. Could you provide comparisons to other commercial LLMs (e.g. Gpt4o, the updated Sonnet 3.5, Gpt4o-mini, Haiku 3) and open LLMs (e.g. the most recent LLama3-Instruct 70b, Llama3-Instruct 8b, DeepSeekCoder 7b instruct, DeepSeekCoder33b Instruct) on MBPP and HumanEval with higher samples, and demonstrating pass @N as you did in tables 10 and 11. 

4. For the models in tables 10 and 11, is it possible to also compare lintseq best@50 to instruction-tuned models best@50, except with Top p = 1.0 and temperatures [1.0, 1.1, 1.2]? This is important to also establish if the increase in diversity of lintseq cannot be matched by modulating the temperature for sampling. 

5. Are there other ways to calculate estimated FLOPs as you have? Would it be possible to also tell me the wall clock time of running inference using the instruction-tuned models and compare that to the wall clock time of using the LintSeq models (e.g. both 14b models compared) just for more transparency.
soundness: 3
strengths: The results speak for themselves: the authors demonstrate LintSeq is capable of substantial performance improvements over instruction-tuned models. The authors are transparent that LintSeq may sacrifice inference-time compute for improvements in the efficient use of parameters and its strong program synthesis abilities. And the authors' ablation demonstrates that the use of a linter (to reject poorly formatted code edits) is important to the success of LintSeq. 

In terms of intellectual merit, LintSeq is not the first work to suggest leveraging edit sequences for training language models for programing tasks. However, LintSeq is notable for getting these ideas to work at scale. To my best knowledge, much prior work may have focused on using human edits such as github commits, which is limited by the number of demonstrations available. Here: the authors introduce a procedure that is not bounded by the number of human demonstrations available, but rather the availability of a linter or even more loosely, a static-analysis based tool for checking the well-formedness of programs (e.g. tools often found in a compiler). 

In terms of significance, the authors are able to demonstrate LintSeq's ability to improve over instruction tuning for a series of code LLMs. However, LintSeq also may have significance for real-world coding applications. The authors mention that commercial code assistants like Cursor deliberately do not use edits; however, the case study of LintSeq may inspire new approaches for code-editor assistants (it could also potentially have issues as well). Additionally, the set-up of LintSeq may also motivate applications for incremental generation of natural langauge responses, especially for tasks that may be intensive in reasoning.
summary: The paper introduces LintSeq which is a proposed method to tune LLMs for programming tasks. Data preparation consists of representing code generation as a sequence of edits. To construct the edits, there is a 2 step process broadly, first deleting lines from an existing program to create potential intermediate states, and using a linter to reject intermediate states that are syntactically incorrect or fail other kinds of static linting tests. The authors demonstrate the merits of this approach for small language models and large language models, comparing LLMs tuned with LintSeq to plain instruction-tuned models. The authors also ablate the importance of the linter-based filtering step, showing that this step is important to its success. The authors also demonstrate and explain that LintSeq-tuned models may sacrifice inference-time compute in exchange; however, in turn for higher performance on program synthesis benchmarks.
weaknesses: The authors' work is very strong, however, there are some important points that need discussion. Overall, my largest concerns are with transparency, clarity, and integrity in presentation. I myself felt mislead multiple times while reviewing this paper, and that concerns me greatly. 

1. Misleading Presentation of Models >1B parameters / Transparency + Clarity 

I found figure 1 to be misleading. It compares leading LLMs on Human Eval with K=1 to LintSeq with K=50. Personally I didn't catch this until I glanced over the paper after at least multiple passes, I would fear other readers may fall into the same trap and overestimate how strong LintSeq is. In reality, LintSeq Pass@1 is can equal to or even significantly worse than an instruction-tuned variant of the same base model. 

It is only in the appendix in Tables 10 and 11 where the authors clearly present LintSeq vs Instruciton-Tuning pass@1, pass@5, etc rates; although similar information is presentated in figure 4. With pass@50 LintSeq is significantly better than instruction-tuned models; whereas, it is negligible or worse with few or one/fewer samples. I believe there is no thorough discussion of this phenomenon in the main manuscript and this is an extremely important point. For example, the authors could offer an explanation that LintSeq increases the generation diversity which is highlighted in the higher pass@50. The authors must be more forthcoming about the tradeoffs of LintSeq when model size is scaled up. I urge the authors to reframe the presentation and discussion to clearly reflect this.  

I would also appreciate a comparison with other top LLMs for code (e.g. LLama3 or LLama3.1; CodeLLama, DeepSeekCoder, Gpt3.5) and comparisons with equal pass@1, pass@5, pass@10 to show how these other LLMs scale with more samples. I feel very mislead by Figure 1 comparing Pass@1 for big models vs. pass@50 for LintSeq and I am left feeling such a presentation is problematic. I feel uncertain about  the motives in presenting Pass@1 vs. Pass@50 without a fair like-to-like comparison between other LLMs and putting tables 10 and 11 deep in the appendix. The authors must present a more forthcoming picture of the strengths and weaknesses of LintSeq when scaled up >1B parameters. I think the paper requires a significant revision to reflect this in language. I believe the technical contributions could already be there, but the presentation feels very misleading to me and the discussion should center much more on the tradeoffs that LintSeq introduces.

2. Framing and Explanation of LintSeq

I find the framing of LintSeq or the explanation of it is not easily clear, and even after section 2.2 where it is broken down into phase 1 and phase 2, it is still unclear to me. This led me to inspect the artifact `lintseq` in the supplementary materials which is >300 lines of code and mentions sampling weights, computing fitness, and "children." Perhaps there's something going on where in the paper "affected lines" are the indented lines of a code block. The description is not sufficiently clear, and I recommend more clarity in the presentation of the algorithm for both transparency and reader understanding. I would not call this "recursively lint" as shown in figure 2, as linting doesn't edit code, so the recursive process needs to be combined with some editing process as menitoned in section 2.2: "(2) run a linter or other
verifier on the remaining code; (3) if the deletion induced new errors, remove all affected lines; and (4) repeat steps 2 and 3 until no errors are caught by the linter." To me, the "remove all affected lines" is unclear to me and I'm not sure if all writers of the paper fully understood the procedure either. I recommend trying to formulate a more thorough explanation both in the main manuscipt and also if this is not sufficient to add an appendix section explaining the procedure in more detail. Pseudocode / pythonic pseudocode may also help to demonstrate this procedure. 

2. Claims of Tiny LMs is Misleading 

I'd recommend the authors walk back the claims on their Tiny LMs. I find the statement "Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters" misleading: the 150M parameter model underperforms CodeT5+ 220M at Pass@10 HumanEval and underperforms Codex 300M at Pass@1. 

3. Discussing Prior Work

The authors only mention 2 works from 2023 and 2024 on fine-tuning on edits. Some papers that came to my mind while reading this were: 
https://arxiv.org/abs/1810.13337
https://arxiv.org/pdf/1905.11006 (much less related)

This makes me inclined to think that there may be more works that I have not included here that should be discussed to highlight the history of thinking on this issue. 

4.  Choice of Language 

The term `static program verifier` makes a loose use of the word "verify." I recommend using a different word from "verify" to not conflate it with [formal verification](https://en.wikipedia.org/wiki/Formal_verification) of programs. 

Also in the abstract "small LLMs" is a contradiction of sorts "small Large Language Models."

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 1
questions: - Novelty-wise what is the paper's main difference compared to the code sketch paper? 
- Code diff is more than just insertion -- why deletion and rewriting are not considered?
- Why disabling chain-of-thought data?
- Why pre-training TinyCodeLM instead of reusing existing pre-trained SLMs?
- Can you try sampling the model using min-p rather than top-k?
- What's the intuition on why LintSeq can improve pass@1? It makes sense to me it can improve pass@ big K as the approach optimizes code generation diversity but it's interesting to me that it also gets higher pass@1 (i.e., higher-quality first sample).
soundness: 3
strengths: - This paper looks at an important domain in generative AI -- code generation.
- The overall idea of processifying code generation makes sense to me -- decomposing the code generation task into code insertions allows the model to construct the generated code with more diverse context possibilities in addition to the simple left-to-right context
summary: This paper looks at the problem of code generation. The corpus trains LLMs to generate code from left to right. Yet, this paper proposes to generate synthetic code edits that train the model to construct code in a step-by-step fashion, while encouraging each step to preserve the syntax validity of the code. This technique is evaluated to be effective for SLMs esp. when doing inference scaling.
weaknesses: - **Novelty & related work**: LintSeq reminds me of the code sketch paper [1] which generates code via syntax-validity-preserving edits (i.e., filling grammar holes). This paper should be discussed (and compared if applicable).
- **Technique**: One theoretical limitation of LintSeq is the lack of naturalness of synthetic edits. The data trains the model on random trajectories which might (if not always) not be meaningful.
- **Technique**: Code edits in LintSeq are *insertion-only* -- I can imagine cases deletion/rewriting can be used, such as writing a bunch of `def f(): pass` when planning the code skeleton and replacing the "pass" when real implementation starts.
- **Evaluation**: overall I feel the evaluation focuses on function-level code generation, which might look a bit contradictory to "inference scaling" -- as I'd expect to use inference scaling for solving much more challenging tasks, e.g., those in SWE-Bench.
- **Presentation**: "Open-source edit data with good coverage over the distribution of all error-free code “diffs” is nonexistent" is debatable given high-quality code-edit datasets such as EditPackFT.
- **Presentation**: The terminologies in this paper can be made more precise (detailed comments below).

----
Comments on writing:

- Fig 1: "HumanEval Coverage" and its use in many other places: maybe just explicitly say pass rate or pass @ k -- "coverage" by default means code coverage in the code generation domain and it might be confusing to refer it to pass rate.
- L065: authors should explicitly explain what "static-error-free" means -- is it program syntax error? or the error of the diff syntax, etc.
- L080 "static program verifier" and "error-free program states": how is it possible to verify an arbitrary program state "error-free" statically? :) -- Similar to the comment above, some basic concepts need to be explicitly clarified and defined.
- L084 "verifier ... is also referred to as a linter": linters and verifiers are very different concepts. linters mostly do static syntax-based checks on certain surface-level program properties. Verifiers are way more ambitious as they assure the existence/absence of program properties given all possible inputs/states (nonetheless, the meaning of verifiers has been extended to validators to assure behaviors over "certain" rather than "all" inputs/states). 
- L107 "A linter is a type of standard code analysis tool that verifies the correctness of program" -- the statement is too strong to be right. Linters cannot verify program correctness but just check a few syntactical issues.

**Actionable suggestions**: for these "linters", "correctness", and "verifiers" terms, it's more precise to say "apply linters to make sure each edit can preserve the program's syntactical validity/well-formedness".

[1] LEARNING TO COMPLETE CODE WITH SKETCHES. ICML 2022.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. See the weaknesses above.
2. Besides simple code generation tasks on HumanEval and MBPP, does this approach work for practical coding such as BigCodeBench? Also, does training on edits improve code editing tasks, say CanItEdit?
3. How does this approach handle a code file that depends on other files in a repository? In such cases, applying the linter only to the code file will raise an error.
4. For code generation tasks requiring multiple programming languages, does the same methodology still work?

Open to increase the score if the above concerns are properly addressed.
soundness: 3
strengths: - The idea is novel. While existing models often separate code generation and editing tasks, this paper presents a perspective to unify them into sequence of editing tasks. It's an interesting directions researchers can explore in the future.
- To prepare the training data, the authors proposed LintSeq, which refactors existing programs into sequences of static error-free code edits. This is a smart combination of program analysis and LLM.
- The pretraining experiment gives a better understanding of the effectiveness of the technique.
summary: The paper introduces a new algorithm, LintSeq, which generates synthetic code editing sequences from existing programs. It reframes code generation as a sequence of edit generation tasks, mimicking human coding. They applied the method to create instruction + program-diff-sequence to train smaller LLMs for code synthesis. Models finetuned on these edit sequences generate more diverse programs and improve performance on benchmarks, achieving up to +20% higher scores on HumanEval pass@50 compared to baseline finetuning. Additionally, a 150M parameter model fine-tuned on these edits achieves state-of-the-art performance in code synthesis for on-device models, competing with larger models like Codex and AlphaCode.
weaknesses: - The authors only evaluated the models on HumanEval and MBPP. While these two benchmarks are popular for most general-purpose LLMs, they are biased toward simple programs and do not cover practical coding scenarios.
- The paper used a Python-specific linter `pylint` to construct pretraining data, but it's unclear how this approach can be easily applied to other languages and be applied at scale.
- There is not enough justification for the base models used for finetuning. Why no base code LLMs?
- There is an overemphasis on pass@k when k is large. While a better pass@ larger k shows the LLM can cover a broader code distribution,  by definition, pass@1 is more practical and is what people care about more.

Meta-review: This paper presents LintSeq, an algorithm that refactors existing code into sequences of structured edits for training language models. The key strengths include: (1) Novel approach to generate synthetic edit sequences that preserve syntax validity using linters, (2) Comprehensive evaluations showing improved performance across multiple benchmarks, especially for higher pass@k values, and (3) Effective application to both small and large language models. Main weaknesses include initial unclear presentation of pass@1 vs pass@k results and heavy focus on insertion-only edits. Overall, the paper makes a valuable technical contribution by demonstrating how edit-based training can improve code generation, particularly for scaling inference compute, warranting acceptance despite some limitations.

Decision: Accept (Poster)