NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Authors: Tushar Aggarwal
Swayam Singh
Abhijeet Awasthi
Aditya Kanade
Nagarajan Natarajan
Venue: ICML 2025 poster (2025)
Keywords: Code-LMs
code-editing
code-generation
software engineering
Primary area: deep_learning->large_language_models
TL;DR: This paper introduce a synthetic data generation pipeline and a robust model adaptation algorithm to train models for diverse code-editing tasks without losing their original code generation abilities.

Abstract: Software engineering activities frequently involve edits to existing code. However, contemporary code language models (LMs) lack the ability to handle diverse types of code-edit requirements. In this work, we attempt to overcome this shortcoming through (1) a novel synthetic data generation pipeline and (2) a robust model adaptation algorithm. Starting with seed code examples and diverse editing criteria, our pipeline generates high-quality samples comprising original and modified code, along with natural language instructions in different styles and verbosity. Today's code LMs come bundled with strong abilities, such as code generation and instruction following, which should not be lost due to fine-tuning. To ensure this, we propose a novel adaptation algorithm, SeleKT, that (a) leverages a dense gradient-based step to identify the weights that are most important for code editing, and (b) does a sparse projection onto the base model to avoid overfitting. Using our approach, we obtain a new series of models NextCoder (adapted from QwenCoder-2.5) that achieves strong results on five code-editing benchmarks, outperforming comparable size models and even several larger ones. We show the generality of our approach on two model families DeepSeekCoder and QwenCoder), compare against other fine-tuning approaches, and demonstrate robustness by showing retention of code generation and general problem-solving abilities post adaptation. We opensource the models, synthetic dataset, and implementation at http://aka.ms/nextcoder.

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Tushar Aggarwal Swayam Singh" Abhijeet Awasthi!
Aditya Kanade
Nagarajan Natarajan
Abstract
HumanEvalFix
Software engineering activities frequently involve edits existing code. However, contemporary code language models (LMs) lack the ability handle diverse types of code-edit requirements. In this work, we attempt to overcome this short- coming through (1) a novel synthetic data gen- eration pipeline and (2) a robust model adapta- tion algorithm. Starting with seed code examples and diverse editing criteria, our pipeline gener- ates high-quality samples comprising original and modified code, along with natural language in- structions in different styles and verbosity. To- day's code LMs come bundled with strong abili- ties, such as code generation and instruction fol- lowing, which should not be lost due fine- tuning. To ensure this, we propose novel adap- tation algorithm, SeleKT, that (a) leverages a dense gradient-based based step to identify the weights that are most important for code editing, and (b) does sparse projection onto the base model to avoid overfitting. Using our approach, we ob- tain a new series of models NextCoder (adapted from QwenCoder-2.55 that achieves strong results five code-editing benchmarks, outperforming comparable size models and even several larger ones. We show the generality of our approach on two model families (DeepSeekCoder and Qwen- Coder), compare against other fine-tuning ap- proaches, and demonstrate robustness by showing retention of code generation and general problem- solving abilities post adaptation. We opensource the models, synthetic dataset, and implementation at msnexttee
.005
Aider
CanitEdit
NoFunEval
NestCocer
Liama 5-4t--rra
Seep Coder 6, 78 Inst
Owen2 5 Coder 148 inst
DeopSock R1 Dicll-i---e---7
DeepSickkCCdett Le-dderr-2------dd
Owen2. Doeen25-Cooeen-ee
Figure I. Performance state-of-the-ard code LMs, the parame- range 6.7B-16B, code editing benchmarks. NextCoder-7B is our code-editing model with QwenCoder-2.55 as the base, fine-tuned using the proposed SeleKT algorithm synthetic and real code editing tasks. For NoFunEval, we consider instances with binary oracles to consistency with other benchmarks. We present detailed results in Section 5.2 Table
1. Introduction
Code editing is a fundamental ability with pervasive use in automating software engineering activities. Recent bench- marks reveal that contemporary code language models (LMs), particularly the smaller and open-weight LMs, strug- gle edit code based natural language instructions MMuennighoff et al.
2023 Guo et al. 2024b Cassano et al., 2023 Singhal et al.
2024 This is despite many of them
incorporating commit-data from GitHub in pre-training (Li
et al. 2023 Lozhkov et al. 2024 or fine-tuning (Muen
Equual contribution 'iiroosott Research India. Correspon-
nighoff et al. 2023 Cassano et 2023 Xie et al. 2025
dence to: Tushar Aggarwal suusar.aggarwaa553 (eeiaaal5@@mmil.cmmmmmm comi
where the commit messages are used instructions.
Swayam Singh <l-swsingh microsoft.com>>. Abhijeet
Awasthi <abawasthi @ microsoft.comm. Aditya Kanade <na-
In this work, aim enhance the ability of code LMs to handle diverse types of code-edit requirements. This poses two challenges; (1) lack of high-quality fine-tuning data and (2) the risk of losing the strong and general abilities (such as code comprehension, code generation and instruction fol- lowing, acquired during pre-training and instruction tuning)
<kanadeaditya microsoft.com.. Nagarajan Natarajan garajn @ microsoft.coms
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
due to catastrophic forgetting (Goodfellow et al.
2013
ferent model sizes by finetuning 3B, 14B, and 32B variants of the QwenCoder-2.5 instruct model. On the challenging Aider Polyglot (Gauthier, 2024b) benchmark, NextCoder- 32B outperforms several strong models such as GPT-4o (2024-1 1 1-20), DeepSeek V2.5 and QwenCoder-2.553225
We address both these challenges in this paper. We pro- pose (1) a synthetic data generation pipeline which starts with seed code examples and diverse code editing aspects to generate samples comprising original and modified code, along with natural language instructions. The code edit- ing aspects are a high-level and simply state dimensions along which samples should be generated (e.g., bug fixing, runtime improvement, etc.). Our multi-stage pipeline gener- ates original code which is intentionally deficient in some dimension so that a meaningful edit that addresses the de- ficiency can be generated. Inspired by (Wei et al.] 2024b), seed code is used to ensure diversity in the generated code
In summary, we make the following contributions:
1. Synthetie pipeline for diverse code-editing examples: We present a multi-stage pipeline and sample 127K high- quality, diverse code editing examples (comprising 229M tokens), where the diversity comes from multiple dimen- sions: (a) granularity of code, (b) types of code-editing requirements, (c) the style and verbosity of natural language instructions, and (d) choice of programming language. 2. Robust adaptation algorithm: To prevent catastrophic forgetting due to fine-tuning, we propose an algorithm, Se- leKT, which only selectively updates model weights. We demonstrate that this helps retain the code generation abili- ties after fine-tuning on code-editing data.
and its scale (ranging from functions, classes, to files). It is important to support diversity in prompting styles as well. Our instruction generation stage samples natural language edit instructions, that are well-fitted to the change from the original to the modified code, in different styles (instruction versus conversation) and verbosity (concise versus detailed). We generate synthetic data using GPT-4o and Llama-3.3- models, and use it together with high-quality commit data from ComnitPackFT (Muennighoff' et al. 2023).
3. Strong code-editing models: We demonstrate signifi- cant improvements in code-editing performance for models like QwenCoder-2.5 and DeepSeekCoder, showing that SeleKT outperforms full and parameter--ffficient finetun- ing methods across four code-editing benchmarks. Our NextCoder-7B derived from
We observe that supervised fine-tuming on the curated data hampers pre-learned abilities of code LMs like code genera- tion. We therefore (2) devise robust adaptation algorithm, called "selective knowledge transfer" (SeleKT), which se- lectively adjusts the base model weights with respect to a subset of fine-tuned model weights. Unlike existing model adaptation techniques (gguyen et al., 2024a) which select the updatable weights priori, we select the weights period- ically based on their magnitude of change during fine-tuning. (3) We experimentally show superiority of our model adap- tation algorithm over existing methods.
outper- forms other models of comparable size and even matches larger models across multiple tasks (Figure D.. On the popu- lar Aider and Aider Polyglot benchmarks, NextCoder-322 is SOTA against open-source models up to 236B parameters. 4. Opensource: We opensource the models, synthetic dataset, and implementation at maka.ms/nettcoder
2. Related Work
Language Models for Code Editing Language models of code demonstrate varying levels of proficiency in following code-editing instructions, as measured by benchmarks such
Putting all of these together, (4) we construct a new series of models NextCoder, adapted from QwenCoder-2.5 instruct variants fine-tuned on eight programming languages, that achieves strong results five code-editing benchmarks. These benchmarks cover multiple programming languages and test a variety of scenarios, including function, class or file level edits, code improvements, and bug fixing. As shown in Figure I NextCoder-7B consistently outperforms models of comparable size. On several tasks, even out- performs larger models such as DeepSeeKCoder-v2-16.. DeepSeekCoder-338 33B and Llama-3-70B: We further show that (5) our approach generalizes to other model families by improving the code editing performance of DeepSeekCoder- 6.7B. To illustrate the robustness of our SeleKT approach, (6) we show that unlike full fine-tuning (SFT). LoRA (Hu et al.
as CanltEdit (Cassano et al. 2023), NoFunEval (Singhal et al.
2024). SWE-Bench (Jimenez et al. 2023) Aider (Gau
thier 2024a). CodeEditorBench (Guo et al.) 2024b)) EditE.
val (Hu et al.
2023), and RES-Q (LaBash et al. 2024). Prior
research has explored ways to specialize language models for code editing. For example, StarCoder (Li et al.) 2023),
OctoCoder (Muennighoff et al., 2023). and EditCoder (Cas sano et al., 2023) leverage git commits as part of their pre-training (Li et al. 2023) or fine-tuning datasets (Muen- nighoff et al. 2023 Cassano et al.) 2023) to enable mod- els to edit source files based on natural language commit messages. Similarly, StarCoder2 (Lozhkov et al., 2024) incorporates GitHub issues, pull requests, and associated files containing code edits, potentially equipping the model with code-editing capabilities during pre-training. SWE- Fixer (Xie et al. 2025) specializes the Qwen2.5-72B model for code editing by fine-tuning it on GitHub issues. We
202T) or model merging approach TIES (Yadav et al. 2024), the performance of models fine-tuned using our method does not degrade their code generation abilities.
(7) We demonstrate the effectiveness of our approach on dif-
2

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
focus on generating synthetic data that samples diverse code edits and on robust adaptation of pre-trained models through fine-tuning on such data.
Language Python
GPT-4o
Llama-3.3-70B
Total
Tokens(M)
8406
6963
15279
28.63
C
7039
10114
17153
33.48
C++
6272
11065
17337
30,93
Synthetic Generation of Coding Data Generating syn- thetic instruction-response pairs has become a standard ap- proach for post-training alignment of models to follow in-
Java
6447
9881
16328
27.61
JS
7367
8663
16030
25.92
Rust
4701
11737
16438
30.43
Go
4503
10701
15204
28.56
structions (Wei al. 2024a| Wang et al.
2023 Xu et al.
Kotlin
3470
9802
13272
22.16
2023 Luo et al., 2023 Zhao al. 2024
Li et 2024).
Total
48205
78926
127041
227.72
These methods have been extended to code LMs. For
Table I. Number of synthetic examples and tokens generated per programming language and model type.
ample, WizardCoder (Luo et al. 2023) employs the EvolIn- struct framework (Xu et al. 2023) to enhance the complex- ity of instructions in the CodeAlpaca dataset (Chaudhary) 2023), creating a more challenging instruction- followiin dataset. The CodeAlpaca dataset itself was synthetically generated using the Alpaca (Taori et al. 2023) and Self- Instruct (Wang et al.) 2023) pipelines. Both EvolInstruct and Code A Apaca primarily focus on function-level coding tasks, with limited coverage of code-editing problems.
3. Synthesizing a Diverse Code Editing Dataset
Git commits promise to be a readily available source of supervision for adapting models for file-level code editing, as also explored by OctoCoder (Muennighoff et al. 2023 However, in our preliminary experiments (Table 3) we find that fine-tuning on targee-target file pairs from Commit- PackFT MMuennihhoff et al. 2023), dataset derived from GitHub commits, with commit messages as instructions, yields limited improvements in code editing. We attribute this the generally limited quality and diversity of data on GitHub and the lack of informative commit messages.
A more recent class of methods condition the example gen- cration process on seed-code derived from real code files. Methods such as Self-CodeAlige (Wei et al. 2024a), Wave- Coder (Yu al. 2024). and OSS-Instruct (Wei et al. 2024b) belong to this category. These approaches significantly im- prove task and instruction diversity by leveraging diverse seed codes. However, they remain untested for generating file-level coding examples that include multiple classes or functions, and they do not emphasize code-editing tasks. To the best of our knowledge, InstructCoder (Hu et al., (202)) the only method explicitly designed to generate diverse synthetic data for code-editing tasks by conditioning seed examples. Nevertheless, the examples produced by the InstructCoder pipeline are limited to short, function-level code snippets and are restricted to Python. In contrast, our method supports the generation of both function/class-leww and file-level code-editing examples across multiple task categories and programming languages.
Thus, in addition to using CacFF as source of supervision, we propose a novel method for generating high- quality code-editing examples starting with real seed data from GitHub using large and medium LMs, such as GPT- 4o and the instruct version of Llama-3.3 3.3-70B. Our method provides greater diversity by offering explicit control over varying levels of granularity (function, class, and file- code edits), wide range of code edit types (e.g.. bug fix- ing, latency and runtime improvements, addressing security vulnerabilities, optimizing resource utilization, and enhane- ing maintainability), diversity in programming languages, and varying levels of instruction complexity (both concise and verbose, and single-turn as well as multi-turn style con- versations). Table | provides statisties of the code editing data generated using our method across eight programming languages. We generate approximately 127K examples and 229M tokens, with 48K examples produced using GPT-4o. We perform contamination analysis of our generated data with respect to the evaluation benchmarks in Scction A.1
Overcoming Catastrophic Forgetting and Robust Fine- tuning We find that fine-tuning LMs on code-editing ex-
amples worsen their performance on code gedereenerttion tasks,
in line with the catastrophic forgetting phenomenon (Good)
fellow et al.) 2013; Kirkpatrick et al., 2017). Model-
merging (Xiao et al.
2023; Morrison et al. 2024 Yadav
ct al. 2024) has recently emerged as a method for learning new tasks while avoiding catastrophic forgetting of knowl- edge acquired during pre-training. The sparse adaptation technique of Nguuen et al. (2024b) selects the parameters to be fine-tuned based on the top-k components of the task vector, similar to our method. However, they do this a priori, and then only fine-tune the selected parameters (i.e., sparse gradients only). In contrast, our algorithm periodically re- assesses the parameters, and performs full fine-tuning of the entire model (Dense Gradients step in Algorithm|]
Fig Figure illustrates our data generation pipeline and Fig- ure g shows an example seed code passed through GPT-40 in our pipeline, which comprises four main components: (i) Problem and Source Code Generation, (ii) Target Code Generation, (iii) Instruction Generation, and (iv) Quality- Based Filtering. Fach component is described below.
i) Problem and Source Code Generation The LLM used for synthetic-data generation is prompted to gener- ate the problem description and source code as solution,
3

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Problem and Source Code Gen
Target-Code Gen
Instructions
Quality-Filtering
Detailed
Seed Code
Problem
Target Code
Concise
Funetion/Clas/Flle
Model
Code
Model
Model
Model
Approved Not
Aspects
Metadata
Explanation
Human
Conversational
Figure 2. Our synthetic data generation pipeline: The input to the pipeline is seed code snippet, modularities (function, class or file) which defines the scope of the output code and aspects to improve up on (latency. resource utilization, runtime efficiency, maintainability, security, and general improvements along with bug fixing). The output a synthetic example, approved by the final quality checker, consisting of problem statement, source code, target code, and instructions in different styles and verbosity (detailed, concise, human-like, conversational). The details of the pipeline stages are presented in the running text.
which is conditioned on required code modularity (function- Beel/clas--eveelllle-eeee
step is given in Figure 7
and the provided seed code sam- pled from the StarCoder dataset (Kocetkov et al. 2022 Li et al. 2023), where we sample only from files that have more than 10 lines and contains logic like loop, functions, con- ditional statements or classes. The generated source code contains flaws that align with the identified improvement areas ge. Bug Fixing, Improving Latency, Optimizing Re- source Utilization) in the prompt. Additionally, the LLM generates metadata that outlines the specific flaws present in the code. Each instance of the synthetic data is generated using a single seed code. The prompt used for this step is given in Figure 5
iv) Quality-based Filtering Finally, to ensure high qual- ity of generated examples, we prompt the LLM verify whether the target file is a correctly edited version of the source file, consistent with the instructions generated in the previous step. The model evaluates the instance by assign- ing score from O to 10 across five criteria: (1) Correctness of edits w.r.t. requested improvements (e.g. latency), (2) adherence of the edits to the instructions, (3) code quality, (4) instruction quality, and (5) the usefulness of the example for fine- fine-tuning small models. An instance is deemed valid if its average score is at least 7, and all individual scores exceed 5. The prompt used for this step is given in Figure g We also performed a human study to assess the quality of the generated data, and the details are in Section AA.3
ii) Target-Code Generation Next, we prompt the LLM to generate the target code conditioned on the problem descrip- tion, source code and the metadata produced in the previous step. We design this prompt to also output an explanation of the edits made by the model source code to obtain target code. The prompt used for this step is given in Figure 6
4. Robust Model Adaptation
Pre-training LMs on large amounts of data followed by fine-tuning them on relatively smaller amounts of data from downstream scenarios is now standard practice. But naive fine-tuning can result in poor generalization performance.
iii) Instruction Generation In the subsequent step, we prompt the LLM to generate code-editing instructions using the source code, target code, and the editing explanation gen- crated in previous steps as input. A parameter in the prompt specifies whether the instruction should be in Concise, De- tailed, Human, or Conversational format. Concise instruc- tions are high-level, often under-specified, three-line de- scriptions that do not explicitly detail the required changes. Detailed instructions are more verbose and provide specific information about the required changes, such as specify- ing the exact function to be modified for improved runtime. Human instructions are very brief, informal, and natural- language-bau messages that are typically 1-2 sentences long and provide a high-level overview of the necessary changes without going into technical detail (e.g., "Hey mod- ify the given code to improve its runtime"). Conversational instructions represent a user-assistan interaction in a chat format, where the user sequentially specifies the required changes. Each instance therefore offers four fine-tuning examples, one for each format. The prompt used for this
The state-of-the. thett-rrtttt code LMs such as DeepSeekCoder and Qwen already have gone through rigorous pipeline of pre- training and instruction-tuning on vast amounts of code and text tokens (as many as 5.5T tokens (Hui et al. 2024). in the case of Qwen-2.5), At the same time, these pre- trained LMs find several code-editing scenarios challenging (as we show in Section 5.2). So, the key question is how we can strike a good balance between task-specific (i.e., code editing scenarios) performance and the generalization abilities of the pre-trained model (i.e., code comprehension, code generation, etc.). In this section, we present our technique for robustly adapting code LMs to diverse code editing tasks.
Robust Adaptation Problem Formally, we want to fine- tune a given code LM, denoted by Abase. such that: (a) the resulting LM Prr, has improved code editing abilities, as
4

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
determined by the training loss the data presented in the previous section, (b) while preserving the generalization abilities of the base LM (as determined by the performance on real-world code benchmarks). The robust adaptation
Model Qur
Pinetume for
Chackpoint 0x
Finetune for
Checkpoint
M steps
M steps
0
ScleKT Operation
problem (Wortsman et al., 2022; Tian et al., 2023) can be posed as:
Figure 3. Proposed adaptive fine-tuning technique SeleKT.
arg min (()) s.t.
||0 Obssel S c
Algorithm SeleKT:
Selective Knowledge Transfer
Require: Base LM weights Ohase. training data D, epochs
where C denotes the next-token prediction loss on the train-
E. periodicity M, sparsity a.
ing data, || || is suitable norm, and is constant.
Ensure: Final fine-tuned weights 0Fr.
1: Initialize base
Inadequacy of Existing Solutions State-of-the-ar tech- niques for adaptation (robust or otherwise) largely follow the parameter-efficient fine-tuning (PEFT) paradigm. That is, fine-tuning is localized to a small fraction of the parame- ters of the base model. For instance, the widely-used LoRA technique (Hu et al., D fine-tunes a small number of parameters added the base LM, keeping the entirety of the base LM frozen otherwise. More recent techniques carefully select the parameters to fine-tune, e.g., few layers (Lee et al.
for epoch 1 to E do for each minibatch D[s] do
3:
4:
0 <- TrainStep(f, ((s))
[Dense Gradients]
5:
ifs mod M = 0 then
6:
Compute task vector: T + 0 - Select top-a.V parameters:
7:
>|1] 1, i top-k([7), NJ)
0, otherwise
2023) or a few parameters across the layers (Nguyen 2024b). The crux of these techniques seems to be that: (A1) focus on fine-tuning a small number of parameters to avoid overfitting on a small amount of training data, and (A2) fix the parameters to be fine-tuned priori, before the training even begins. Although the design choice (Al) is reasonable and is arguably prudent, question the design choice (A2) of these techniques.
ct al.
8:
0 <- Orase + T
[Sparse Projection]
end if
10: end for
II: end for
12: return @ as 6Fr-
tuning of the model 0, and the compute the top-k non-zero entries (by magnitude) on the (accumulated) gradient vector the "task vector" 0 Obuse- This also ensures that the parameter selection is global and not confined to specific layers or other heuristics employed in earlier robust fine- tuning strategies (Lee et al.) (023). The resulting "selective knowledge transfer" problem is:
While recent robust adaptation techniques try to achieve the model eecuracy-efficienn trade-off using sparse gradients in their updates, the appeal for efficiency during fine-tuning seems to be at cross with the desire also achieve low gener- alization error and nut-of-domain robustness. We show this empirically in our experiments (Table 4p: (u al. (2021) also observe that there is some degradation in performance due to the choices made in LoRA.
arg min C()) s.t. - Obnsen S c.
(2)
Our key insights are simple:
0
(1) What parameters need be fine-tuned for the task(s) at hand should be continuously re-assessed conditioned the difficulty of the downstream scenario as dictated by the fine-tuning data and the training loss.
Our algorithm, SeleKT, short for Selective Knowledge Transfer, is presented in Algorithm I It parameterized by (i) sparsity a, or the fraction of the total number of model parameters N to be updated, and (ii) periodicity M, or how often the projection step needs to be performed.
(2) (Dense Gradients) We can update all the model param- eters to determine the direction of parameter changes that best minimizes the training loss on the code editing tasks, unlike PEFT methods, and then (Sparse Projection) com- pute suitable projection to ensure that the parameters are guaranteed to be close to the base model (Figure 33.
Choice of Training Loss L in We use cross-entropy loss for next token prediction as the objective function in our experiments. For examples formatted as (instruction, response) pairs, we apply the loss on the entire example. For examples formatted as multi-turn conversations (Section (3). we apply the loss only to the final response generated by the model and not on the conversations.
To implement the two insights above, we choose the Lo-morm in ( i.c., we want the updates to be localized to a small set of parameters of the base model. The Lo-morm makes the projection step computationally easy: we first compute dense gradients by doing full fine-
Proposed Solution
Lemma 1. For any given a base LM Obase. and for the
5

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Method
HumanEvalFix
CanltEdit
Aider
Benchmark
Description
Examples
Update Boase
79.5
48.0
55.6 65.7
CanltEdit
Bug fixing (class-level)
210
Fix Possc
81.1
50.5
HumanEvalFik
Bug fixing (function-level)
164
Nol-untval
Code improvements (file-level)
397
Aider
Bug fixing (conversational, file-level)
133
Table Performance of SeleKT with and without periodically updating the base model OwenCoder-2.5-7B as in Remark 2
Aider Polyglot
Bug lixing (conversational, lile-level)
225
HumanEval+
Code generation (function-Ievel)
164
MBPP+
Code generation (function-level)
378
GSM8K
Mathematicall problem solving
1.32K
setting a = c/N, where N is the model size, the fine-tuned model OFr satisfies the constraint in the objective (2)
MMLU
Multiple-choice STEM knowledge 3.15K
Table 3. Evaluation datasets: We use four diverse code editing benchmarks with varying input granularity and code-editing crite- ria. Additionally, we also evaluate on two code generation tasks and two non-coding tasks.
The proof is straight-forwatt as (i) in Step 6 of Algorithm !I we always compute the task vector with respect the base model, (ii) the mask vector computed in Step selects c co- ordinates. Together with the update in Step 8, the constraint is guaranteed to satisfy.
QwenCoder-2.5 instruct model. We compare our fine-tuned models against models from DeepSeekCoder, Llama, and Qwen families in the range 6.7B to 70B parameters. We also compare with GPT-4o (Hurst et al. 2024) and the Qwen-7B distilled from the latest, maasoonnng-enhanee
Remark 1 (Efficiency). While the projection step itself is soraight-forward, computation of dense gradients can be more expensive. The cost is mitigated to some extent by do- ing mini-batching and restricting the total number of epochs to few. We show in Sections (a.etions5332mmm a.4d5.45ttaa the resulting improvements in generalization accuracy are significant.
DeepSeek-R1 R1 model (Guo et al., 2025). For Aider Polyglot, compare against leading models from their public leaderboard
Remark 2 (Alternative Update). An alternative style of update in Algorithm | is to periodically also update the base model, i.c. Oasee + 0 after Step 8. So the future task vector (in Step 6) will be computed with respect to the updated base model. This update style also guarantees that the final OFT will be close to the base model in the Lo-norm sense. To be precise, we can show, an albeit weak bound, ||@pT - O llu S c- E- [(D(s) H/M, where [[ is the number of mini-batches, M is periodicity, for a sufficiently small choice of a. We also find that this style (denoted by "Update O ") performs worse empirically, from Table 2 (details of fine-tur fine-tuning and benchmarks in (..1)on(3.1))
Hardware For fine-tuning and inference, we use 8 NVIDIA H100 GPUs, each with 80GB of VRAM. For data generation using GPT-40 (version 2024-05- 13), we use the OpenAl API. Fine-tuning takes about six hours per epoch of wall-clock time. Following /Singhal et al. (0024), we perform run-time evaluations for NoFunEval on an Azure NC16 VM (NC16).
Implementation We fine-tune for 3 epochs, across all our experiments, using Adam W optimizer (Loshchilov & Hut- ter 2017) with a learning rate of 10 55 and a WarmupLR scheduler (Kim et al.) 202T) with a warmup ratio of 0.1. For efficient memory management, we use sample pack- ing with maximum sequence length of 8192 tokens for DeepSeekCoder-6.77 and 16384 tokens for QwenCoder variants. We initialize the models from their respective pre- trained checkpoints (instruct versions), and fine-tune them on their respective chat templates (HuggingFacc 2025). Ap- pendix A.1.2 provides additional implementation details.
5. Experiments
We present the details of our experimental setup, followed by a discussion of the main results and an ablation study.
Hyperparameters (i) We used temperature of 0.6 for data generation for both Llama-3.3-708 and GPT-4o. (ii) We fix the periodicity to epoch in the SeleKT algorithm unless specified otherwise, i.c., M = total number of mini- batches. We set sparsity a = 0.05 per layer. We selected these values based on initial experiments. In Section 5.7 we show ablations on these choices. Additional hyperparameter tuning for SFT and LoRA are detailed in Aection|A.1.2
5.1. Experimental Setup
Fine-tuning Dataset In addition to the synthetic data (Ta- ble [.. we used 127K instances from CommitPackFT to fine-tune our models. CommitPackFT, restricted to the eight languages (Table consists of 153K real GitHub commits. We filtered out the examples from the dataset which do not have source code (c.g., edits only to config files) retain only code-editing instances (Table
Evaluation Datasets and Metrics Table 3 presents the diverse datasets we use for evaluating our and baseline mod- cls. Specifically, we use CanltEdit (Cassano et al., 2023). HumanEvalFix (Muennighoff et al. (023), NoFunEval (Sing hal et al.) 2024). Aider (Gauthier, 2024a) and Aider Poly- glot (Gauthier 2024b) to evaluate on code-editing tasks. We
Our Models and Baselines For fine-tuning, we consider the instruct versions of DeepSeekCoder-66 6.78 (Guo et al. 2024a) and Qwen2.5-Code 5-cdeer-B (Hui et al., 2024). To demon- strate effectiveness of our approach across model sizes, we also fine-tuned the 3B, 14B, and 32B variants of the
6

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Models
NoFunEval Runtime E.f.
HumanEvalFix
Aider
CanltEdit
Latency
Res. Util.
Maintain.
Security
GPT-4o
45.6*
39.3
3.389'
57.6
55.1
90.2"
74.4
39.5
Llama-3-70B-Inst
34.4
28.3
2.816
43.7
58.1
77.4
51.1
56.7
DeepSeskCoder-338
30,0
24.0
2.589
38.0
53.9
74.4
58.6
49.5
DeepSeskCoder-221888
23.6
21.4
2.274
37.5
54.7
72.0
48.1
42.8
W.enCoder-225-322
42.1
36.9
3.006
64.0*
58.6
90.2*
75.2
60.9
OwenCoder-2.55 2.5-144B
38.1
32.2
2.597
50.7
55.8
87.8
66.9
58.1
Llama-3-8B
22.5
18.7
1.255
20.6
55.1
56.7
39.8
29.0
14.4
1.185
9.6
41.2
67.7
13.5
33.3
DeepSeekCoder-6.78
20.5
21.0
2.275
33.3
61.8
71.3
43.6
38.1
DeepSeekCooder-6.7B-LLrtt
21.0
18.0
1.245
28.3
54.0
70.7
41.4
37.2
6B SFT
22.4
15.0
1,359
23.8
57.2
65.2
30.8
41,4
DeepSeekKCoder-..78-1115
22.1
25,3
2.166
37.6
62.9
73.8
48.1
45.7
DeepSeeskCoddrre677---eelll
24.8
22.0
2.150
40.0
63.6
76.0
47.1
49.6
OwenCoder-2.557B
26.5
24.7
2.190
39.3
58.7
73.8
59.4
48.1
QwenCoder (.t--odder225-7---/. L.5-7B-LoRA
26.9
2.133
38.2
55.0
70.7
40.6
44.3
OwenCoder Seniioodrr2.5-7B.... 2.5-7B-FFT
25.1
1.387
30.6
54.8
70.1
48.9
36.7
OwenCoder 2.5-7B- TIES
27.8
24.9
2.180
44.4
60.1
79.5
60.2
47.0
OwenCoder 2.5- BBeeeeKKTTT (NextCoder)
26.4
23.2
2.286
46.1
65.3
81.1
65.7
50.5
numbers (the higher the better) denote score for NoFunEval, and % accuracy for all others. For NoFunEval, we considered the scores for the best (Max) performing prompt. The rows shaded blue are models obtained using our approach; the rows shaded gray are baseline models of comparable sizes. Best number in the comparable group is in bold, and overall best indicated by star
Table Performance of baseline and fine-tuned code LMs on code-editing benchmarks:
choose these datasets to ensure diversity terms (a) scale (class function file level), (b) requirements (func-
5.2. Performance on Code-Editing Benchmarks
Table presents the main results of experiments cover- ing four well-known benchmarks for code editing. We com- pare (i) our adapted models with existing language models for code (ii) SeleKT, our adaptation method, with standard model adaptation methods like supervised fine-tuning (SFT), LoRA (Hu et al. 2021) and TIES (Yadav et al. 2024). In Ta- ble 4 our SeleKT based adaptations (DeepSeekCoder-6..) SeleKT and CwenCooerr2255-7---55 blue Rows in Table 4 similarly sized modets in the parameter range 6.7B to 8B. Our main observations can be summarized as follows.
tional improvements such as bug fixing and non-functional improvements such as runtime efficiency and security), (c) instruction details (terse vs, detailed), and (d) scenarios (standard vs. bbsddd. Each dataset comes with its own metries to automatically evaluate the model outputs. While the common metric is execution accuracy on test cases, the NoFunEval benchmark also utilizes runtime improvements, static analysis checks, and DiffBLEU scores (Bairi al. 2023) for evaluation. We also evaluate the larger 14B and 32B model variants on Aider Polyglot benchmark (Gauthier 2024a). This benchmark has gained popularity as it contains particularly challenging problems across multiple program- ming languages, proving difficult even for very large models. Notably, even advanced models like GPT-4o (2024-11-20) solve only 18.2% of the problems in this benchmark.
are highlighted in highlighted in gray refer various
i) SeleKT provides consistent gains over the original instruct models. For example, DeepSeekCoder SeleKT outperforms DeepSeekCoder-6. on all tasks ex- cept the run-time improvement task NoFunEval. In- terestingly, on CanltEdit benchmark, DeepSeekCoder 6.7B- 6.7B-SeleKT provides 11.5 point gains in accuracy DoppSeekCoder-6.78 Similarly, NextCoder-78 outper- forms QwenCoder-2.5-7B on all tasks except for Lateney and Resource Utilization in NoFunEval.
Additionally, we also evaluate the models on (a) standard code generation benchmarks, HumanEval+ and MBPP+ (Liu et al. 2023) (Table 3 rows 6-7), to measure the extent to which models fine-tuned for code editing still retain their code generation abilities. These benchmarks require model complete function given its signature and short description. The evaluation metric is execution accuracy on test cases. Further, we do evaluation on (b) non-coding benchmarks, GSMBK (Cobbe et al. 2021) and STEM sub- set of MMLU (Hendrycks et af., (021) (Table 3 rows 8- 9), to gauge how well they maintain math problem solv- ing and natural-language understanding capabilities. Ap- pendix A.1.3 provides details the benchmarks.
ii) SeleKT frequently outperforms the standard model adaptation methods. SSrprisingly, supervised fine-tuning of models like QwenCoder-2.5-77 frequently resulted in worse performance across various benchmarks w.r.t. the original model 2.5-7B in Table 4. indicating overfitting. To address this problem, we designed SeleKT (Section 4 for robust model adaptation. We see that SeleKT outperforms parameter- efficient fine-tuning methods like LoRA (Hu et al. 2021
2.5enCC vs QwenCoder

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
and model-merging methods like TIES Yadav al.
2024)
mark. For the smaller 3B model, NextCoder-3B shows significant improvements
often used for adapting models while limiting loss in prior knowledge (NextCoder-7B vs QweenCoder:2.5-7--LoFA vs O.en-ooder2.5577--T155 in T
the base model across most benchmarks, with a substantial gain the CanitEdit bench- mark (+5.3%). Notably, while the QwenCoder-2.5-3B- achieves slightly better performance on HumanEvalFix, our approach excels on other benchmarks. At the 14B scale, NextCoder-145) consistently outperforms all baseline vari- ants, achieving gains across all four benchmarks, with par- ticularly impressive improvements
iii) SeleKT provides best code-editing performance among models of its size. Baseline models of compa- rable size include a reasoning based DeepSeek-RR - -wwen- 7B (Guo et al. 2025), Llama-3-8B-Inst (Dubey et al., 22024), DeepSeckCoder-6.
the Aider Polyglot benchmark (+2.9% over base model). For the largest 32B model, while there is a slight decrease in HumanEvalFix per- formance compared to the base model (-1.3%), NextCoder- 32B achieves the highest scores across all other benchmarks, with a remarkable improvement on Aider Polyglot (+7.2%). These results demonstrate that our training approach pro- vides consistent gains across model sizes.
(Guo al.) 2024a) and its adapta- tions, QwenCoder-2.55 2.5-7B (Hui al., 2024) and its adap- tations. We observe that our best model QwenCoder-2.55 7B- TB-SeleKT, referred as NextCoder-7B has the best overall performance across all models in this parameter range.
iv) Performance of adapted SeleKT models depends on the performance of their unadapted versions. We ob- serve that QwenCoder-2.5- and NextCoder-7B gener- ally outperform DeepSeckCoder and DeepSeekCoder- 6.7B-SeleKT respectively. This suggests that higher- performing models lead to more accurate adaptations, even when model sizes are similar.
5.4. Preserving Pre-learned Knowledge
Models
HumanEval+
MBPP+
DeepSeekCoder-6./M
71.3
65.6
DeepSSekCCoderr-..7B-Lorf
64.6
64.3
SeepSeekCoder-6.78B
70.1
59.5
v) Comparison with larger models. Notably, NextCoder-7B matches or even surpasses larger models on many tasks. For instance, outperforms DeepSeekCoder-v22-111 (twice its size) on all tasks. Simi- larly, NextCoder-7B clearly outperforms Liama-3-70B-Inst and DeepSeekCoder-333 on HumanEvalFix, Aider. and the Maintainability and Security splits of NoFunEval. As expected, much stronger models like GPT-4o and OwenCoder-2.5-32B substantially outperform smaller models, including our SeleKT models.
DeepSeeeKCoderr6.BBBTTT55 DeepSSeeCCoder-.77B.eeeKKT
70.1
64.0
73.2
65.3
QwenCoder:2.5-7BB 22emnooder.2.577BBLoor.
85.4
72.5
81.7
70.9
OwennCoder-2.5-7B-555 OwenCodder225577B-TT
79.3
67.2
82.3
71.7
NextCoder-7B
84.8
72.0
Table 6. Comparing (% accuracy) of base and fine-tuned models code generation benchmarks.
The results in T that our SeleKT method largely preserves the code generation capabilities of the original models, as evaluated on the HumanEval+ and MBPP+ (Liu) et al. 2023) benchmarks. For both DeepSeekCoder-6.1 and Swenn2.-Coddr-7B. full fine-tuning (SFT) and LoRA lead performance drops of up 6.7 accuracy points. In contrast, model merging using TIES helps mitigate these drops significantly. Notably, our method consistently out- performs TIES, further reducing performance degradation.
we present qualitative examples on differ- ence between the base model and our fine-tuned version.
In Appendix C
5.3. Efffectvvnnsss of SeleKT on Different Model Sizes
Models
HumanEvalFix
CantlEdit
Aider
Aider Polygiot
OwenCoder-2553B
36.8
64.6
36.2
35.8
QwwaCodee-25533B.co
NexCodde:38
75.6
42.4
37.6
QwenCoddr-255144B
78.0
50.9
66.2
5.3
QwwaCoode225-144.c
Model
MMLU
GSM8K
NextCoder 148
89.8
60.2
72.2
12.3
QwenCoder-
QwenCoder-2.5-7B
53.0
83.40
OwenCoder-2.5.33 LoRRA
82.3
52.4
60.2
6.7
QwwasCode-2.5-32B-
QwenCoder-2.5-322
71.9
93.71
NektCoder-328
88.9
62.4
74.7
23,6
NextCoder-BB
54.5
81.65
Table Comparison base OwenCoder-2.55 models of different sizes and their SeleKT-enhanee versions across three code edit- ing benchmarks.
NextCoder-32B 72.7
92.65
Table 7. Accuracy (%) on MMLU and GSM8K benchmarks.
Table 5 demonstrates the performance of our SeleKT al- gorithm across various model sizes, including multilin- gual capabilities measured by the Aider Polyglot bench-
The results in Table demonstrate that NextCoder-32B improves on MMLU scores (+0.8 points), but has a small regression on GSMBK (-1.06 1.606 points). Overall, these results
8

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
demonstrate that our training methodology induces special- ized code editing capabiliities without sacrificing math prob- lem solving and natural-language understanding abilities.
5.7. Ablation of Choices in SeleKT
Effectiveness of Sparsity To investigate the impact of sparsity parameter in the SeleKT algorithm, fine-tune the Qwen2.5-Coder-77 model with different values: 0.05, 0.2, and 0.5. We fix the periodicity M to the epoch boundary which is our default setting, and the number of epochs to 3. As seen in Table with a = 0.05 (most sparse, selecting only 5% of parameters to be updated), the model achieves the best overall performance. This is in agreement with our motivation that the updates remain tightly close to the base model in the Lo sense to avoid overfitting, while selecting the parameters to be updated globally and periodically.
5.5. Effectiveness of Synthetic Data
Table [ coopares the performance of DeepSeekCoder-6.77 fine-tuned separately on CommitPackFT and on the syn- thetic data generated by our pipeline. While synthetic data offers a performance advantage over CommitPackFT, the latter represents a high-quality sample of real-world devel- oper commits. To enhance generalizability, we incorporate both datasets in our fine-tunie finettnning process. Performance vari- ations across different data sizes are further analyzed in Section
HumanEvalFix
CanltEdit
Aider
0.05
81.1
50.5
65.7
0.2
76.8
45.7
53.4
Data
HumanEvalFix
CanltEdit
Aider
0.5
81.7
43.3
54.9
CommitPackFT 59.8
37.6
21.1
Table 9. Ablation of the sparsity factor in the SeleKT algorithm used to fine-tune Owen2.5-Coder-78.
Synthetic
68.3
41.4
33.8
Table Comparison of synthetic data and CommilPackFT data when used for fine-tuning the DeepSeekCoder model.
Effectiveness of Periodicity To examine the impact of the periodicity parameter M in SeleKT, we fine-tune Qwen2.5- Coder-7B using different values of M (0.1, 0.5, and 1.0 times the number of mini-batches in an epoch) while keep- ing the sparsity parameter a = 0.05, Additionally, we com- pare against baseline that follows full fine-tuning (without sparse updates) but applies a single SeleKT operation at the end, mimicking model merging techniques. As shown in Table A.6 aligning periodicity with epoch boundaries yields the best results for CanltEdit and Aider, whereas lower periodicity leads to worse performance. Performing sin- gle SeleKT operation at the end, similar to model merging, achieves the high accuracy only on HumanEvalFix. These findings indicate that sparse updates are essential for robust fine-tuning, but excessive frequency may not be beneficial.
5.6. Performance on Aider and Aider Polyglot Benchmarks
Evaluating code-editing models requires benchmarks that assess both general coding capabilities and multi-language proficiency. The Aider Polyglot benchmark has gained promi- nence for evaluating multilingual coding capabilities, featur- ing 225 exercises specifically selected as the most challeng- ing problems from Exercism across multiple programming languages.
NextCoder models demonstrate impressive performance on both Aider and Aider Polyglot benchmarks compared to statc-of-the-art code LMs (Figure 4.. NextCoder-32B scores 74.7% on Aider, outperforming 71.4% of GPT-4o (2024-11-20) and approaching top models like Gemini- exp-1206 (80.5%). It also achieves 23.6% Aider Poly- glot against GPT-4o's 18.2%. Similarly, NextCoder-145 matches Deepseek- V2.5 on Aider (72.2%) despite having orders of magnitude fewer parameters. These results demon- strate that our training pipeline helps smaller, more effi- cient models to compete with much larger, more resource- intensive alternatives. The Aider Polyglot benchmark results further validate our approach, with all NextCoder models achieving relative performance advantages over comparably sized alternatives. While we demonstrate clear benefits in Aider Polyglot, the generally lower scores on this benchmark across all models reflect the benchmark's 's inherent difficulty with multi- lulti-language challenges. This suggests scope for improvement and line of investigation for our future work.
6. Conclusions
The ability of code LMs to accurately edit code at different scales and based on diverse instructions is central to their use in software engineering. In this paper, we present the next step in enhancing the code-editing ability of code LMs by developing a synthetic data pipeline and robust adap- tation algorithm SeleKT. Our pipeline produces diverse data that improves model performance and the adaptation algorithm ensures that the general, pre-learned abilities of the models are not lost during fine-tuning. We comprehen- sively evaluate our method on multiple code editing and generation benchmarks to establish these claims. In future, want extend our pipeline to cover more scenarios and evaluate SeleKT in tasks other than code-editing such as mathematical reasoning and natural language tasks.
9

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Impact Statement
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R.,
Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
This paper presents work whose goal to advance the field of language models for code. We do not foresee any poten- tial societal consequences related the specific contribu- tions of our work.
Guo, J., Li, Z., Liu, X., Ma, K., Zheng, T., Yu, Z.,, Pan, D., Li, Y., Liu, R., Wang. Y., et al. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprini arXiv:2404.0354 2024b.
References
Bairi, R., Sonwane, A., Kanade, A., C, V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., and Shet, S. Codeplan: Repository-level coding using Ilms and planning, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https 7/arxiv. org/ absp/bbb..200300
Cassano, F., Li, L., Sethi, A., Shinn, N., Brennan-Jonee A.,
Lozhkov, A., Anderson, C. J., and Guha, A. Can it edit? evaluating the ability of large language models to follow code editing instructions. In The First International Work- shop on Large Language Model for Code, 2023. URL httpss://at /tt:://axxvv.. https////xviv.or///aaa/.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 2021.
www./bgg///oooooooooot..ott..o.t abs/ 2312 12450
Hu, Q.. Li, K., Zhao, X.. Xie, Y., Liu, T., Chen, H., Xie, Q., and He, J. Instructcoder: Empowering language models for code editing, 2023.
Chaudhary, S.
Code
alpaca.
GitHub
sahil280l14coodcapaaaa github.com.
https:
7/ com/ 2t00aaaaa1111111//daaaaaacoa..a
2023. [Accessed 21-01-2025).
HuggingFace. face.co. https://hugginggaaee docs/
Chat Templates hugging-
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint 110.41210,,,168, 2021.
transformers/nen wwtes//main/e...o chat templating 2025. [Accessed 30-01-2025).
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2/06, 2024.
Dubey, A.. Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A. et al. Gpt-4o system card. arXiv preprint arXiv:241021226, 2024.
Gauthier, P. GPT code editing benchmarks aider.chat. https://aider:
/bt////nncaaa...... Henchhhaeebeecarkk 2024a. [Accessed 21-01- 2025].
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-word github issues?, 2023.
Gauthier, P. GPT code editing benchmarks - aider.hha.. https::/aiiees chat/docs whtt://ocs/eeddeeooataaa Letpdeeeadebbbdreee/ee/a/a
Kim, C., Kim, S., Kim, J., Lee, D., and Kim, S. Automated learning rate scheduler for large-batch training, 2021. URL httpips://rxxi...o wwwp:wi.orrr/aooom hatrg//abb2210... 05855
#polyglot- legggiott-eaddebbarraa 2024b. [Accessed 28-01-2025).
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-352 2017.
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting gradient-based neural networks. arXiv preprint arXiv: 1xxiv:131200 1211.222222111 2013.
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi. X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024a.
Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T. Bahdanau, D., Werra, L. v., and Vries, H. d. The Stack: TB of permissively licensed source code, 2022. _eprint: 2211. 155333...
10

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
LaBash, B., Rosedale, A., Reents, A., Negritto, L.,, and Wiel, C. Res-q: Evaluating code-editing large language model systems at the repository scale, 2024.
Nguyen, B., Uhlich, S., Cardinaux, F., Mauch, L., Edraki,
M., and Courville, A. C. Saft: Towards out-of-distributtin generalization in fine-tuning. CoRR, 2024b.
Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C. Surgical fine-tuning improves adapta- tion to distribution shifts. In The Eleventh International Conference on Learning Representations, 2023.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parame- ter models, 2020. httpss::: www.xiiivrrrroooo abs
1910. 02054]
Li, H., Dong, Q. Tang, Z., Wang, C., Zhang, X., Huang, H., Huang, S., Huang, X., Huang, Z., Zhang, D., et al. Synthetic data (almost) from scratch: Generalized in- struction tuning for language models. arXiv preprint arXiv:2402. / 13064, 2024.
Singhal, M., Aggarwal, T., Awasthi, A., Natarajan, N., and Kanade, A. Nofuneval: Funny how code lms falter on re- quirements beyond functional correctness. arXiv preprint arXiv:240I.1596, 2024.
Taori, R., Guirajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followinng Ilama model. https: /tapttu///ihubb... www/tatss- stanford alpaca| 2023.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., al. Starcoder: may the source be with you! arXiv preprint 27Xiv:2305.06161, 2023.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL Forum?iG=1qvx61000
Tian, J., He, Z., Dai, X., Ma, C.-Y., Liu, Y.-C,, and Kira, Z. Trainable projected gradient method for robust fine- tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7836- 7845, 2023.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.). Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ 2023.acl-long 754. URL httthttt///t //acs://acl// http://aclc/th.hooc logy azg/2023. -loo--on....... 754/
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization, 2017.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy- J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402.1 19173, 2024.
Luo, Z., Xu, C., Zhao, P., Sun. Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08566 2023.
Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., von Werra, L., Guha, A., and Zhang, L. Selfcodealign: Self-alignment for code generation, 2024a. URL UtLpttps://aast / org hatp:/bbb/22100
Morrison, J., Smith, N. A., Hajishirzi, H., Koh, P. W., Dodge, J., and Dasigi, P. Merge to learn: Efficiently adding skills to language models with model merging. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15604-15621, 2024.
Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magi- coder: Empowering code generation with oss-instruct, 2024b. URL https: //apxii//axvvv...// org/ abs aa///bb/23122 2312 02120
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models, 2023.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
NC16. //Jeammmiccroofftccommeeeeee /
Azure VM. https:
virtual-naching htwp:///nchiiee//ccc---o---- series
Nguyen, B., Uhlich, S., Cardinaux, F., Mauch, L., Edraki, M., and Courville, A. Saft: Towards out-of-distribution generalization in tuning. 2024a. URL https:// Doxiv.org/abs/24000 www.ooggg/bbo.o...... 2207 03036
Xiao, S., Liu, Z., Zhang, P., and Xing, X. Lm-cocktail: Resilient tuning of language models via model merging. arXiv preprint 1..i::3311113::. 2023.
arxiv.

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Xie, C., Li, B., Gao, C., Du, H., Lam, W., Zou, D., and Chen, K. Swe-fixer: Training open-source Ilms for effec- tive and efficient github issue resolution. arXiv preprint mrXiv:2501.05040 2025.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions, 2023. URL https://aaxxx org/ abaaabss/b/// 12244 12244
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. mies-megging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Yu, Z.. Zhang. X., Shang, N., Huang. Y., Xu, C., Zhao, Y., Hu, W., and Yin, Q. WaveCoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers), pp. 5140-5153, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.16653w11202244.lloogggg. URL (httsst //acllarttcllogy.orr//2222222 acl long 2807
Zhao, C., Jia, X., Viswanathan, V., Neubig, G., and Wu, T. Self-guide: Better task-specific instruction following via self-synthetic finetuning. In First Conference on Lan- guage Modeling, 2024. net. net/ mtt/:/oommmm--d..gg0020
12

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
A. Appendix
A.I. Additional Details
A.1.1. FINE-TUNING DATASET
Language
Instances
Tokens(M)
Python
42094
9.50
C
6854
1.50
C++
3816
0.90
Java
13443
3.25
JS
51704
11.60
Rust
2316
0.51
Go
4950
1.08
Kotlin 1819
0.44
Total 126996
28.78
Table A.I. Number instances from CommitPackFT after filtration.
For CommitPackFT, we retain only those examples that contain both source and target code for the code-editing task (see Table A.I.. The initial dataset consists of 153K instances across eight languages, and after filtration, 127K instances remain.
A.1.2. TRAINING SETUP
SFT We finetuned DeepSeekCoder-6// model on 4xH100 GPUs and Swen2.5-Coder-77 model on 8xH100 GPUs. For efficient memory management, we employed sample packing to a maximum sequence length of 8192 tokens for DeepSeekCoder-6. 6.BB and 16384 tokens for accumulation steps were set to 4, resulting in respective effective batch sizes of 64 and 32. Additionally, DeepSpeed's s ZeRO Stage Rajbhandari et al. 2020) offloading to CPU, using b for memory optimizations, was applied both models.
with batch sizes of 4 and per GPU, respectively, Gradient
LoRA The Siven..-Coodev model was fine-tuned using LoRA (Hu et al. 2021 with rank of 64 applied to all linear layers of the base model. The LoRA hyperparameters included an a value of and a dropout rate of 0.05, without training the bias term. Batch size remained the same as in the SFT setup.
SeleKT The models were fine-tuned on 8xH100 GPUs. Sample packing, batch sizes and gradient accumulation steps were configured as reported for SFT. The samples were packed into a maximum sequence length 8192 tokens for Deepseek and 16384 tokens for Qwen, with gradient accumulation steps of and an effective batch size of 128 for Deepseek and 32 for Qwen. DeepSpeed's s ZeRO Stage 2 offloading CPU was used for both models with the bfloat16 16 data type for memory optimization. Additionally, the SeleKT algorithm was deployed with sparsity factor of 0.05 and periodicity factor M set to the epoch boundary.
TIES For the TIES variant of both DeepSeekCoder-6. and Owen2.5-Coder-7b. performed model merging by integrating the final third checkpoint into their respective base models. The merging process was conducted with both density and weight parameters set to 0.5, without applying normalization or int8 quantization.
A. 1.3. BENCHMARKS
compared all the models on the following well-established benchmarks for code-editing (Table
CanltEdit (Cassano al. 2023) benchmark measures class and function-level code editing abilities language models in Python for domains like Data Science, Mathematics, and Language Processing. Each problem instance is accompanied with descriptive (verbose) and lazy (terse) instructions, and correctness of edits is measured using execution accuracy over test cases. For this benchmark, we used temperature of 0.2, top_p 0.95 and reported pass @ 1,1 scores.
NoFunEval (Singhal et al. 2024) benchmarks language models for their ability to edit file-level code in multiple programming
languages based on non non-functional requirements such as improving latency, resource utilization, security, and maintainability of existing code. Each problem instance is associated with four different types of prompts. Correctness of edits
13

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
measured using run-time improvements, static-analysis based tools like CodeQL, or DiftBLEU scores, depending upon the non-functional requirement. We used greedy sampling for this benchmark and reported pass @ 1, scores.
Aider code-editing benchmark (Gauthier 2024a) offers 133 small coding exercises in Python from Exercism dataset requiring an LM to edit python file for implementing a function or class as per natural language instructions. For this benchmark, we used a temperature of O and a whole-format setup (prompting the model to rewrite the entire code) and reported p scores.
Aider Polyglot code-editing benchmark (Gauthier] 2024b) offers 225 coding exercises from Exercism dataset. It contains exercises in multiple programming languages: C++, Go, Java, JavaScript, Python and Rust. The setup was the same for Aider's code-editing benchmark. For this benchmark, we used temperature of 0 and a whole-format setup (prompting the model to rewrite the entire code) and reported pass@2 scores.
HumanEvalFix benchmark [Muennighoff et al. 2023) evaluates models on the bug-fixing task, where models are given a code snippet along with an instruction to fix the code. We used a temperature of 0.2, top_ _ 0.95 and reported pass@ 1,1 scores for this task.
We also evaluate the models on standard code generation benchmarks.
HumanEval+ benchmark (Liu et al. 2023) tests the models' performance in generating correct code based on textual descriptions with high-quality test cases. We used greedy sampling for this benchmark and reported pass @ 1,1 scores.
MBPP+ benchmark (Liu et al. 2023) focuses on evaluating the models' ability to solve programming tasks that require mathematical reasoning and algorithmic thinking. We used greedy sampling for this benchmark and reported pass@ 1,1 scores.
Additionally, we evaluate the models non-coding benchmarks.
MMLU benchmark (Hendrycks et al. 2021) (STEM subset) comprises 3. 1.5 problems spanning Physics, Chemistry, Biology, Computer Science, Mathematics, and Engineering. We employed the few-shot setting (N=4) with the prompt configuration from Qwen models' official evaluation script.
GSM8K benchmark (Cobbe et al. 202I) contains grade-school mathematical problems that test reasoning abilities. For evaluation, we used the few-shot setting (N=4) with the prompt configuration from Qwen models' official evaluation script.
1.1.4. HYPERPARAMETEA TUNING FOR BASELINES
Method
CanltEdit
HumanEvalFix
Aider
SFT
46.2
78.6
52.6
LoRA
43.8
79.9
54.1
SeleKT
50.48
81.10
65.70
Table A.2. Results for the best versions of LoRA and SFT for Qwen2.5-Coder-7B
To ensure fair comparison between our proposed method and baseline approaches, we conducted systematic hyperparameter tuning experiments for SFT and LoRA methods. Since SeleKT was optimized with a learning rate of 1e-5 and weight decay of 0.0, we explored comparable configurations for the baseline methods. For both SFT and LoRA, we investigated learning rates of 2e-6 and 5e-6 combined with weight decay values of 0. and 0.05. For LoRA specifically, we further evaluated rank values ranging from 16 to 64 and alpha parameters of and 16.
While this comprehensive tuning process yielded performance improvements for both baseline methods, Table A.2 demon- strates that substantial performance gap persists between these optimized baselines and our NexiCoder-7B model. Across all three evaluation benchmarks (CanIEdit, HumanEvalEdit, and Aider), SeleKT consistently outperforms the tuned baseline approaches. The most pronounced difference appears the Aider benchmark, where SeleKT achieves 1 1..% improvement over the best baseline. These results underscore the effectiveness of our selective parameter updating approach beyond what can be achieved through standard hyperparameter optimization of conventional fine-tuning methods.
14

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
A. 1.5. DECONTAMINATING SYNTHETIC TRAINING DATA
To ensure the integrity of our evaluations, adopted the decontamination procedure used in prior works such Star- Coder (Li et al. 2023 Lozhkov et al. 2024). Specifically, we applied near-duplicate detection using MinHash and Locality Sensitive Hashing (LSH) to identify and remove any potential overlaps between training dataset and evaluation bench- marks (CanltEdit, HumanEvalFix, Aider, NoFunEval). Our decontamination analysis confirmed 0% data leakage, verifying that no benchmark data was present in the training set.
A.2. Robustness of SeleKT across dataset sizes
Dataset size
CanltEdit
HumanEvalFix
Aider
Base model (OwenCoder-225-7B)
48.1
73.8
59.4
25%
47.67
80.20
60.90
50%
48.57
81.43
62.70
75%
49.01
81.02
63.80
100% (NextCoder-7BB
50.48
81.10
65.70
Table A.3. Performance of 2.w5 when finetuned splits of different sizes of dataset
To assess scalability w.r.t. to training data size, we finetuned the QwenCoder-2..6
model on varying fractions (random sampling 25%, 50% and 75%) of our dataset which includes both synthetic and CommitPackFT data. The results presented in the Table A.3 All models were trained for 3 epochs. The results show clear trend: while performance CanltEdit and Aider sees drop at 25% w.r.t to the base model, increasing the dataset size consistently improves performance across all benchmarks (CanItEdit, HumanEvalFix, and Aider).
A.3. Human Evaluation of Synthetic Dataset
We conducted pilot human-study to assess the quality of the generated training dataset. In this study, we involved three participants who have 3-4 years of experience in software development with strong expertise in Python. We randomly selected 100 samples from the Python split of our synthetic dataset and asked participants to rate each sample on the scale of being poor quality and being excellent quality) on the following three questions:
Instruction Usefulness: How well does the detailed instruction capture potential code-editing scenario with respect to the original code?
ii) Instruction Consistency: How consistent are the three styles of instructions (detailed, concise and human-like) with each other and with the respective styles?
iii) Solution Correctness:
How well does the edited code match the edit described in the detailed instruction?
Metric
Participant
Participant 2
Participant 3
Overall Mean
Overall SD
Instruction Usefulness
4.92 + 0.27
4.93 # 0.26
4.38 # 0.60
4.74
0.48
Instruction Consistency
4.29 + 0.48
4.88 # 0.32
4.55 # 0.54
4.57
0.51
Solution Correctness
4.92 + 0.27
4.96 = 0.24
4.60 # 0.51
4.83
0.40
Overall Mean + SD
4.71 + 0.46
4.92 + 0.28
4.51 # 0.56
Table A.4. Evaluation of Instruction Usefulness, Consistency, and Solution Correctness
In the table A.4 we present the mean (along with standard deviations) ratings by participant and by question. The scores consistently close to the highest score of 5 across all participants and questions. This provides strong indication of human-machine alignment, with low to moderate variance. This study helps validate that our synthetic data generation pipeline is able to generate samples that meet human expectations in terms of quality and consistency.
In the table A.5 we have shown the scores distribution. Clearly, all samples received scores (neutral quality) or above on all the questions. We particularly inspected the samples that received the neutral rating (score 3) since those were perceived as relatively low-quality samples by one or more participants. We made the following observations:
15

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Score (Higher is better)
Instruction Usefulness
Instruction Consistency
Solution Correctness
5
229
175
250
4
65
122
48
6
2
Total
300
300
300
Table A.5. Distribution of Evaluation Scores
i) Instruction-Edit Misalignment: In some cases, instructions correctly described the intent but the edits not entirely appropriate. For example, in response to an instruction to handle datetime parsing, the edited code parsed dates against raw strings, which would cause runtime errors.
ii) Incomplete Error Handling: Some examples did introduce error handling, but overlooked edge cases (c.g., what if the fasks.json file exists but empty?).
iii) Style Inconsistency: A few participants noted that stylistic or structural variations across instruction formats led to minor misunderstandings code-editing intent.
Aider
Aider Polyglot
84.2
80.5
80
74.7
71.4
72.9
72.2
66.9
60
55.6
51.6
40
38.2
23.6
20
18.2
17.8
16.4
12.2
9.3
3.6
0
Claude-
Gemini-exp -1206
NextCoder -328
GPT-4o 2024-11-20
Deepseek -/2.5
QwenCoder -2.5-32B
NextCoder -14B
QwenCoder -2.5-14B
GPT-4o -mini
3-5-Sonnet
-20241022
Figure Performance of state-of-the-art code LMs Aider and Aider Polyglot benchmarks. NextCoder-x is code-editing model with Qwen2.5- 1012.55Coode--- x as the base, fine-tuned using the proposed SeleKT algorithm on synthetic and real code editing tasks. Baseline
sourced from the official leaderboard (Gauthier 2024b).
M
HumanEvalFix
CanltEdit
Aider
0.1 Epoch
80.5
37.1
51.1
0.5 Epoch
83.5
50.4
59.4
1.0 Epoch
81.1
50.5
65.7
At the end
84,2
50.0
53.2
Table A.6. Performance comparison of using SeleKT with different values of M the model.
16

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
A.4. Prompts
You are an expert programmer tasked with creating programming problem corresponding code based on a given seed code. Your task understand the concepts and patterns in the seed code and create an inspired but different problem with intentionally imperfect code.
CONTEXT:
Level: [level; (This whether you create level, level, fle-level solution)
Improvement Areas Needed:
(area] __ _name]]: _desc]
jarea2 namel: (area2, desc]
[area3_namme name]: (area3_dese]
should deficiencies
specified
improved
steps:
1. Carefully analyse the given seed code to understand its core concepts and patterns.
Create NEW programmiing problem that inspired
concepts but direct copy.
3. Write a code solution that:
Matches the specified modular level
Takes inspiration from the seed code but creates a different implementation Contains natural imperfections and inconsistencies
Implements core functionality but with flaws
May have unim pllmeeeetd critical sections clear bugs
Mixes different approaches to solving similar prablems Looks like code written someone learning
hurry
The code should feel authentic like something found in real codebuse with organic issues. include comments about potential improvements or issues, The code should work for basic cases but have natural flaws in implementation.
IMPORTANT NOTE: Do not add comments pointing out issues or suggesting improvements. The code should stand on own with natural im perffections.
Your output MUST strictly follow this format using the exact delimiters:
#AZPROBLE EM_ RMMSTATE STATEMENTES
clear, detailed problem statement describing
accomplish.
The problem should be non trivial and require substantial solution. ##REND_ PROBLEM BALEEEMSSAAARREA
#NNORIGINAL
Provide the code solution here, including comments.
code functional contain intentional deficiencies specified
BORINDDORRIINALL CODEREE
#AMMETADATARRE
MODULAR_I.C (level) lovel
LANGUAGE: (Jang]
IMPROVEMENT. __AER___L [name of the first improvement area]
[specifie line numbers where improvements be made, separated ranges e.8. 1,3,5-8]
DESCRIPTION: (detailed description what deficiencies exist and how they could be improved]
TYPE: (specific type issue within this improvement area]
IMPROVEMENT AREA 2: Iname of the second improvement area]
LINES: (affected line numbers) |detailed description of deficiencies)
DESCRIPTION:
TYPE: (specific type issue)
IMPROVEMENT. AREAATARRRAA...... [name of the third improvement area] LINKS: iaffected line mumbers]
DESCRIPTION; [detailed description deficiencies]
TYPE: (specific type issue]
#AFEND_ MOD.DDMTTAAATAAA
The code should be functional but intentionully suboptimal in the specified improvement areas code help understand the logic
Include comments
Do not create trivial problems; ensure the solution has sutficient complexity
deficiencies realistic improvable
Follow the exact delimiter format- do not modify the delimiter strings
include any outside delimited sections
Here's the code to inspire your problem and solution:
[lanng]
jseed code)
Figure 5. Prompt used for generating a problem and source code conditioned on the given seed code.
17

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
You are an expert programmer tasked with generating three different corrected versions of code that has specific issues identified in the metadata. Based the original improvement requested and the issues found, will generate: An improved solution that implement the improvements
Solution should specifically address the improvements requested the original improvement areas while fixing the issues identified in the metadata.
CONTEXT:
Problem Statement:
(problem)
#ø Original Code:
** "[aang]
[code]
Requested Improvement
[areal_ _name]: [areal__ee [escc] jarea2, namel:
[area3. name):: (area3 desc]
#A Identified Issues (Metadata):
[metadata]
generate:
An improved solution that:
identified issues from metadata
Follow best coding practices
implement proper error handling
Use efficient and maintainable approaches
Include clear comments explaining the improvements
Include necessary imports
May implementation strategies
Your ourput MUST strictly follow this format using the exact delimiters::
SAFIMPROVE S.DDUUTIIN_TTT. 1AFA
All imports here
[Mirst version the improved
detailed comments explaining improvements]
BSHEND SEAPPROVEDSSOOUUTIOON
#HDIPPERENCES EXPLAINEDeks
IMPROVED:
description approach key improvements]
IMPORTANT:
Solution must be fully functional
Include descriptiss ptive comments explaining the implementation and improvements
Solution should include AL.. required imports
Solution should complete and standalone
Maintain the same interface/AP AI as the original code
Figure 6. Prompt used for generating the target code (improved code).
18

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
You are un expert prompt engineer tasked with generating three different types of instructions that guide an LLM to transform the original code into the improved version. Use the provided context generate detailed, human like, and conversational instructions.
CONTENT:
Problem Statement:
(problem]
Original Code:
[lang]
(code)
Target Improved (lang]
[odited_code]
#* Key Improvements Made: !explanations:
Generate four different instruction formats and Your output MUST strictly follow this format using the exact delimiters:
#A#DETAILED INSTRUCTION
[Generate detailed instruction exceeding lines) that:
Clearly outlines each improvement needed
Specifies exactly what changes required
Mentions specific functions/areas modify
Maintains clarity while being concise
Should provide enough detail for LLM to understand the required changes.)
#ANEND DETAILED INSTRUCTIONNee
#ANCONCISE INSTRUCTIONeeR
[Generate concise instruction (3-4 lines) that:
Contains essential improvement points
Covers necessary changes
Is clear but not overly detailed
Should provide just enough Information to guide the changes.)
#AREND_ CEONCISS BOATSEEEENSSUUTUIIIII INSRRUCTIONE:
HARHUMAN_INS INSTTRU CTION
[Generate very brief,
instruction
Uses natural language max)
Captures core improvements needed
quick dev chat message]
#ABEND HUMAN INSTRUCTIONeRR
[Generate IINTOONNNNNISSUUIII
REHCONVERSATIO ERSATIONAL
INSTRUCTIONAEP
assistant that this
USER: [General opening that naturally leads into code discussion]
ASSISTANT: [Eogaging response following the user's direction]
USER: [ntrodures the problem and initial code structure, showing interest getting right]
ASSISTANT: [Detailed acknowledgment and analysis of the problem/structuree
USER: Here's my current implementation: <code placeholder> [Do''t add the code put the place holder there)
ASSISTANT: [Provides clear explamation of code's current structure and functionality]
USER: [Final message containing:
Moderate level of detail about required changes (between human and detailed)
Clear Instructions about what needs be improved
Reference specific improvements needed
Natural tone while being specific enough]]
ASSIST ANT: [Brief acknowledgment of the requirements and indication that implementation will follow, followed by
Boode_output_pliee cet___uuuuu_ppcccoddrrrr
IMPORTANT:
instruction type guide towards
Instructions should be clear und unambiguous
natural language appropriate each format
Do not include actual code changes in the instructions
the target improved version key improvements guide, dol'renee them directly instructions Keep focus on what changes are needed, not how they were implemented
Figure 7. Prompt used for generating different types of instructions.
19

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
You are quality ussurance expert tasked with validating a training sample for code editing, Analyze the following components and provide comprehensive assessment:
COMPONENTS TO ANALYZE:
Seed Code:
"" 'Banni
iseed code)
Generated Data:
### Problem Statement: (problem)
##ボ Original Code:
[[ang]
(original [odee]
#*# Improved Edit:
"Hanng fimproved__ede _edtt
### Instructions:
Detailed: (detailed instruction)
Human:
Conversational:
(conversational Instruction)
Perform the following quality checks and provide scores (0 10) with explanations and Your output MUST strictly follow this format using the
delimiters::
#asCORERENCE SESCOHERRNCC_CHRCKEE _HЕсKate
Original
Eriii Idit Alignment:
edits properly address the code's issues?
Are improvements meaningful and substantial?
changes aliga with requested improvement areas?
Score: [0-10]
Explanation: [Brief analysis)
Edit-Instruction Aligoment:
Do instructions clearly guide towards the implemented changes?
all significant changes covered in instructions?
Is the instruction complexity appropriate for each formati
Score: [0-10]
Explanation: [Brief analysis)
#ARRND_ ##HRND_COHERENE CHRCCKes
#IQUUALITY CHECKN#N
1. Code Quality:
Original deficiencies: they realistic fixable?
Edltt im ments: Are they meaningful and bell-implemented?
Code structure:
and maintainable?
Score: [0-10]
Explanation: [Brief analysis]
Instruction Quality:
Detailed: Clear, specific, and comprehensive?
Human: Natural, concise, and effective)
Conversatioal: Logical flow and clear final request?
Score: [0-10]
Explanation: [Brief analysis]
Training Value:
will this help SLM learn code editing?
Are examples diverse and meaningful?
Is complexity appropriate for training?
Score: [0-10]
Explanation: [Brief analysis]
##5END_ ##HEND_QUALITY. CHECKees
#aHPINAL_
Strengths:
[List key strengths]
Weaknesses:
needing improvement)
Recommeendatios::
[Specific
any improvements needed]
# FFNNLL VERDICT#R
Figure 8, Prompt used assessing the quality the data generated.
20

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
B. Instance Generated from Our Data Pipeline
present a synthetic data instance generated using our pipeline in Figure) 2
Problem
Target Code
Detailed
Create Bunction peucess trunsactions
SeedCode
( falled. transaction for transactium tramsuction::
Sunction, follow these stepa: 1. Add validaaion to ensure each
abstcacttethedd
bulunees. framouttion dicrionaryy
clees toseenthiii
legging
containing Troas. user use""""" and amount
froms user not trunsaction or bo user' nod
transaction 'mmoun" transoction
if isinstanceltr anttssaattioo, dicO
Cromaction is dictionury with from, jisse'' "Tn,tterr and
(self, Bise-Atomiccta)):
pooithve
gelassmethed
user balanees II fer
Source Code
from pane tramasctiim (arentttim//mmm/mmaoe froms_
continue
Concise
del from, pemesisccls,
@ccassmethod
Model
process is trauseciieuteeanaatioon
failed, Iramartion transactions:
amount on tt:
non Islostance/amoun,, (int, (la())
amountt Initialiin balances
If Trmlesing....
prnesix, stade: Bescontiate,
base. dib RoscAtomicDB,
trunsactions app.appppddraaaaiinnn
Human
pencsis_blocis R
bransoctionl
judd transaction
from user nut seer__halare Jallnnnee:
Model
user www.aaaamllffrrne (frrrfffmaarrt =1
iuitialiee bulunces
def get block classiseff, Block, root
user balancesffrom used] = unuer not user, halamone:
return Gailed, transsactions
iiissing and invalid cronsattons...
get canonicllaall head/wwlff::
bolanccsito.....ooooooooooooooe. aser]
Conversational
fww.edd transactions
Model
function
ramosctione
Pilename
Metadata
balanees, Could use some help rudining it.
including:
multipe sigilficant
Expionation
ASSESTANT: Suro, I'd happy
helge specifically
critical functions
Completely mimplemmntedd
This solution initialiees user balances within tranactioon processing only when necensaryy
you improve?
Pvefformncal::
algocittuns, unnecessery loops,
Model
structure cholces
This avoids redundant chrciss the integiity the data.
issecution
APPROVED
Figure Figure showing an example passed through the data generation pipeline for GPT-4o.
21

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
C. Examples Showing the Quality of Our Model Outputs
Instruction: Your makk to vdd the mine counts enpiy aquares in complesed Miscwsoper boand. The board ltell is rectangle composed of aquares Gut glther empty (") or mine (*")).
For each cmpty square, count the number mines adjacent to (hotizontally, rertically, diagonally). the emppl squere has no sdjucent mines, hoaw it enepty.
Otherwie seglace is with the aljuconi minss count.
For esample, you may receive a x board like this (empty spaces are represented here with the characser for display on scceent:
"Juel
Which your code should trunsform into this:
"Ject
175h1
19h31 -272-
-111-
Input Code:
der aunocatts/mnnciildd::
not minefield or not all(len(row) =n lenfminnffeiddjj (ieid for row in minefield);
miiee Valuel The boond is invalid with curment impat.)
ded count, uninnnss,y y/s
count a
for im range(mandt)) e-1), mingee?, ben(minediell)):
forj rangečmasely p-1), minlg12, lin(mminnneeeee)j)) minsdisllB() leg
count 1-
return count
result +
for ruw in tn musttrrtttimiiniieede
saw, nuw II
for j. cell emamerate(row)):
if cell not (e 'I: raine Valusleror The board contains invalid characters") if cell
new. row.append(")
elser
mine_rount count, mineestiinnns___
new, row apperdioantmine... (omum)) of mine joount 0 !)
resull appladd jein(neny_row)) [row))
retam result
Outgutt
Output: www.mmmallmmniiiiliii fun minufiell:
anmutate/nszaafiill
minufiald allilen/oow) lendnninwfuulllll minsfiebd);
Valsekrron" The board invalid with current input."
Validate the boand
del coumt, minests, 3/:
mow Jength imm/minfeeddj))) for minsfildt
if iflem/oow)
Jangth:
result "
raise ValueBrrot" hoarc invalic with current Input.") for chariti row:
for row cumeeetne/mleefieltt
new_row =
ifeear not in ralse Valueirrrnn" The bound is imallid with current imput.")
for cell emumerate(rocc
coll
new..ow..pp. www.row.cppesd""
Function to count mines around giree cell def count wow.tttiinsss.pp0 3):
eise:
mine_count count, minestt
new row www.eppedditttminne counti
assult.appoad/ jjonllew zow)!
new
smmudeentiiiufoolll0.... Iisiiciieddii www.appendd")
mine Jount count, minuut
www.appenllerriiinnn...o sound) if mini, minu,,cuntt>>> Delae")
annocated boar Sozraapenn"" joinfsow. sowl)
neturm snnutated, board
Figure I.. Example code-editing task (top) from Aider benchmark. NextCoder (RHS) solves the task correctly in the first attempt: The input can only contain spaces or ***s. However, in the example in the prompt, '.' is used to represent space and the same is explicitly stated. Despite this, @.wen2.5-Coder-.e accepts inputs which contain characters other than '4* and space. This problem is fixed by NextCoder by rejecting any input which violates this constraint. The relevant code in NextCoder's output is highlighted in bold.
22

---

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits
Instruction:
pussies you por squarc letters and specilic woods
jefkapppr
candeiingte
powssgrouu
walediige
atahyburi
jaleycalmp
diguneeme
hidden bmngagee seuare.
Given puzzle and of worda return the socation of the first and lust letter each word.
Taput Code:
salf
def eq Belf, other's
selfy others
WordSearch:
ded (ealf, pustijs
self, puasle puoole selfrow lenlpunda) nelf cols tentgroolelttt
searcaçell,
dinethon=
DJ a right
Ot -1), right-tniich
CL 0b, a top to homoon
C3, bertom-is-tee
11, top-tett bwitomrightt
top right botions
(1) honom-leet wetten inp-tight
unggrietffrowss
de directions:
ifelff search frumirow,, eol, word, da, dels
returs (Pointiool, rewt, Pointtcol (Jintrsed)) 13* de zow (Jenjward)- 11*
reture None
now mowe end < bhr Eoole
return False
and wwl.guudeesswllsde to wordßl:
returo Falke
Owen Ourputt:
NestCoder Output:
dlass Point:
clasa Poise:
class WaedSearch:
def Ini... (sed, puzzlci:
class WordSeurchs def Init....... lself, pumtie):
searchiself, ward):
searchisss
rampsfaall.ooos
disection directiume: for iin ranus/ell.arooj for ramusself.cold:
diovctions:
sfself. search
word, (Painiool, Pointiool (lenjwond)) dis row (lm(word))
ifself, from peint/word, directiols
return (Puintj, Pointty (lenbword)) 1) dinection])), (lem/word)) - direction[1))
return Nome
return None
def aourch, frmmbelf, sol, word, dyje
daf wearch from quintledlf weed, start, start, cnd, direction):
for in rungellen@wordl:
for k rangetlenoweedl:: now e start_row a direction(]]
row co return Iollse
self.rows ool & ool celf-cole:
col - stari_rol cll k dinectionjo]
if well guorlefoo]]ccd) wordlii; retuen Pullr rowe
ifrow ur now self or pil chrr cul self.nn or self self:prroefee:eeeeee wordllk]:
return Falne retura True
dy return
Figure Example code-editing task (top) from Aider benchmark. NextCoder (RHS) solves task correctly in attempt, while Swen2.5-Codert (LHS) gets wrong in both the attempts: The directions are represented by pairs of the form (a.b) where is the y-coordinate and is x-coordinate. However, Qwen2.5-Code 5-dd-drrrrrr-B incorrectly assumes that and are respectively and y coordinates and generates edited code accordingly. NextCoder interprets the instruction accurately and generates correct code. The relevant code in NextCoder's output is highlighted in bold.
23


Reviews:
[{"review_id": "yHK7miy7Q6", "reviewer": "Reviewer_3QHQ", "summary": "The paper introduces an adaptation method for code language models on code-edit tasks. The authors presents a synthetic data generation pipeline that creates code samples paired with edited versions and natural language instructions. The paper states that during fine-tuning, their SeleKT can update the model’s weights to avoid catastrophic forgetting and thus improve the performance. The authors also provides experiments to show that NextCoder outperforms comparable models on several code-editing benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback on our work. We answer the specific questions below:\n\n> Reason for selecting some specific hyperparameters for seleKT.\n\nOur preliminary experiments indicated that a sparsity value of 5% and periodicity of 1 epoch were effective across model families, sizes, and benchmarks. Due to limited compute, we stick to these values in our experiments. Nevertheless, in Section 5.5, we present ablation on these two key hyperparameters of our method, namely, the sparsity $\\alpha$ and periodicity $M$. By separately tuning the hyper-parameters for each model, we can get further improved performance for our approach. As suggested by the reviewer, we will include a discussion of hyperparameter selection and tradeoffs in the revision.\n\n> Robustness of the approach across dataset sizes\n\nIn addition to answering the reviewer's question on (a) robustness to dataset sizes, we are also including new results on (b) robustness across model sizes.\n\n**(a) Robustness across dataset sizes**\n\n| Dataset size        | CanItEdit | HumanEvalFix | Aider |\n| --------------- | --------- | ------------ | ----- |\n| Base model (QwenCoder-2.5-7B) | 48.1 | 73.8 | 59.4 |\n| 25% | 47.67     | 80.20         | 60.90  |\n| 50%  | 48.57     | 81.43        | 62.70  |\n| 75% | 49.01     | 81.02        | 63.80  |\n| 100% (NextCoder-7B) | 50.48     | 81.10         | 65.70  |\n\nWe appreciate the reviewer’s suggestion to evaluate effect of dataset size. To assess scalability w.r.t. to training data size, we finetuned the QwenCoder-2.5-7B model on varying fractions (random sampling 25%, 50% and 75%) of our dataset which includes both synthetic and CommitPackFT data. The results are presented in the table above. All models were trained for 3 epochs. The results show a clear trend: while performance on CanItEdit and Aider sees a drop at 25% w.r.t to the base model, increasing the dataset size consistently improves performance across all benchmarks (CanItEdit, HumanEvalFix, and Aider).\n\n\n**(b) Robustness across model sizes**\n\nAdditionally, we are happy to share new results (the tables below) comparing the performance of our SeleKT algorithm across various model sizes against supervised finetuning (SFT) and parameter-efficient low-rank adaptation (LoRA) on multiple code editing benchmarks. \n\n| **3B Models** | **HumanEvalFix** | **CanItEdit** | **Aider** |\n|:----------|:----------------:|:-------------:|:---------:|\n| QwenCoder-2.5-3B | 73.2 | 37.1 | 36.8 | - |\n| QwenCoder-2.5-3B-LoRA | 64.6 | 36.2 | 35.8 | - |\n| QwenCoder-2.5-3B-SFT | **76.2** | 32.4 | 30.1 | - |\n| **NextCoder-3B** | 75.6 | **42.4**|**37.6** | - |\n\n| **14B Models** | **HumanEvalFix** | **CanItEdit** | **Aider** | **Polyglot** |\n|:----------|:----------------:|:-------------:|:---------:|:------------:|\n| QwenCoder-2.5-14B | 87.8 | 58.1 | 66.9 | 9.3 |\n| QwenCoder-2.5-14B-LoRA | 78.0 | 50.9 | 66.2 | 5.3 |\n| QwenCoder-2.5-14B-SFT | 79.9 | 42.4 | 36.8 | 3.1 |\n| **NextCoder-14B** | **89.8** | **60.2** | **72.2** | **12.2** |\n\n| **32B Models** | **HumanEvalFix** | **CanItEdit** | **Aider** | **Polyglot** |\n|:----------|:----------------:|:-------------:|:---------:|:------------:|\n| QwenCoder-2.5-32B | **90.2** | 61.0 | 72.9 | 16.4 |\n| QwenCoder-2.5-32B-LoRA | 82.3 | 52.4 | 60.2 | 6.7 |\n| QwenCoder-2.5-32B-SFT | 81.7 | 49.5 | 66.9 | 8.4 |\n| **NextCoder-32B** | 88.9 | **62.4** | **74.7** | **21.9** |\n\nFor the smaller 3B model, NextCoder-3B shows significant improvements over the base model across most benchmarks, with a substantial gain on the CanItEdit benchmark (+5.3\\%). \n\nFor larger models, we have also included the latest and more challenging[ Aider-Polyglot benchmark](https://aider.chat/docs/leaderboards/) results (more details in response to reviewer itGF).\n\n\n> Limited theoretical analysis of the mechanism\n\nThough the focus of our paper is on rigorous empirical validation, we have made some progress on theoretical understanding, that we outline next. Under the standard smoothness assumption on $f$ and boundedness assumptions on the gradients and concentration assumption on the task vector stated below, we can show $O(1 /\\sqrt{T})$ convergence for SeleKT. Due to char limits, we are unable to give a proof sketch here.\n\n**Task Vector Concentration Assumption** \n\nThe task vector $\\tau = \\theta - \\theta_{base}$ exhibits a concentration property with parameter $r >> 1$, such that the majority of information is contained in the top-$\\alpha$ fraction of parameters.\n\n$$\n\\frac{{\\sum_{i \\in \\text{top-}\\alpha} |\\tau_i|^2}}{\\alpha N} \\geq r \\cdot \\frac{{\\sum_{i \\notin \\text{top-}\\alpha} |\\tau_i|^2}}{(1-\\alpha)N} , \\quad r >> 1\n$$\n\nNote that $r$ defines a certain margin between the task-specific parameters (i.e. top-$\\alpha$ fraction of parameters) and the rest of the parameters (both suitably normalized).\n\n> Besides from codes and instructions, would there be possible application...\n\nPlease refer to our response to reviewer UeAD (2nd point).", "claims_and_evidence": "Claim: Selective adaptation via SeleKT improves editing performance without harming general code generation.\nEvidence: Experiments on 4 benchmarks (Canitedit, Humanevalfix, Nofuneval, Aider), with comparisons against other methods, experiments shows that their model performs better consistently.", "methods_and_evaluation_criteria": "The approach to use a synthetic data pipeline to generate examples seems appropriate, the method that update the model weight seems working based on the experiments, the evaluations criteria also seems good.", "theoretical_claims": "The paper does not seem to have formal theoretical proofs.", "experimental_designs_or_analyses": "The designs were validated via 4 benchmark datasets and other baseline methods, which seems fine.\nHowever, it would be good if the authors could disscuss the reason for select some specific hyperparameters for seleKT.", "relation_to_broader_scientific_literature": "Their model, NextCoder, is based on previous work from Qwen2.5-Coder-7B, where they introduce new fine-tuning method to somewhat address catastrophic forgetting issue and resulted in a slightly better performance. Synthetic data generation and instruction tuning seems be related prior topics.", "essential_references_not_discussed": "No.\nHowever, it would benefit if the authors could add some discussion of recent studies on selective parameter updating and fine-tuning methods.", "other_strengths_and_weaknesses": "Strength: The comparison table is clear and comprehensive, the 4 benchmark is sufficient to demonstrate the model's gain; overall, the paper is presented quite clear.\n\nWeakness: limited theoretical analysis of the mechanism for ML interpretability, possible sensitivity in the hyperparameters", "other_comments_or_suggestions": "Maybe the authors could add some discussion of potential trade-offs on hypermeter selection.", "questions_for_authors": "What is the scalability and robustness of the approach on larger/smaller datasets? For instance, besides from codes and instructions, would there be possible application of this on proteins and explanations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3NiVJpmE4", "reviewer": "Reviewer_UeAD", "summary": "This paper proposes an approach to handling diverse code-editing requirements. First, it introduces a synthetic data generation pipeline that begins with seed code samples and applies various editing criteria to produce high-quality training data. This pipeline generates pairs of original and modified code along with natural language instructions in different styles and verbosity levels. Second, the paper presents SeleKT, a model adaptation algorithm that identifies the most crucial weights for code editing using a dense gradient-based step, followed by a sparse projection onto the base model to prevent overfitting. Experimental results show that the resulting model, NextCoder, achieves strong performance across multiple code-editing benchmarks, surpassing comparably sized models and even outperforming some larger models in code-editing tasks.\n\n## update after rebuttal\nAs the authors present well-designed research questions for studying human-machine alignment, supported by evidence-based performance improvements, the reviewer agrees that the proposed SeleKT methodology is both effective and practical in code-editing scenarios. Therefore, the reviewer raises the score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:\n\n> The scarcity of high-quality fine-tuning data and the risk of catastrophic forgetting during fine-tuning are well-known problems in ML models in general. This raises concerns that the paper’s contribution may not be sufficiently distinct.\n\nWhile we agree with the reviewer that catastrophic forgetting and lack of high quality finetuning data are well-known problems, we are unable to understand why this implies that our contribution is not sufficiently distinct.\n\nIn fact, we do contrast and experimentally compare with some of the recent, key ML work in this space (PEFT/LoRA, model merging/TIES) throughout the paper. Our approach offers a conceptually novel solution. We introduce a novel weight update algorithm specifically designed to mitigate catastrophic forgetting during fine-tuning. Our strong results over model families, sizes, and benchmarks in the code-editing domain show improvements over standard techniques (SFT, LoRA, TIES). \n\nFurther, our synthetic data generation pipeline also is a key contribution, that places emphasis on getting high-quality data tailored to the intricacies and diversity of real-world code-editing tasks and scenarios. Overall, our approach goes beyond existing methodologies and offers a nuanced, yet easy-to-implement approach to model adaptation.\n\n> The reviewer has examined the proposed SeleKT algorithm for parameter optimization and considers it a general methodology for artificial neural network adaptation. However, the reviewer is uncertain whether its design choices are specifically tailored for the code editing task.\n\nWe agree with the reviewer about the potential generality of SeleKT. However, in this work, our motivation and focus is to improve the code-editing performance without sacrificing pre-learned abilities like code generation. The synthetic data generated for finetuning (Section 3) represents the design choices tailored specifically to the code-editing task. Code editing, and more generally coding models, is an important AI domain today, and our work clearly shows the method's potential. We note that in the present form, we have been careful not to make any claims about generality of SeleKT beyond what we demonstrate in the paper. We plan to investigate the applicability of SeleKT to more domains like math and natural language reasoning in the future.\n\n> The reviewer has a particular interest in the quality of the generated dataset. In Line 210 (L), the authors state that they perform quality-based filtering by prompting the LLM to select high-quality examples. However, the reviewer believes that this step requires some level of human intervention ... Conducting an empirical study to assess whether the LLM's evaluation standards align with those of human experts. \n\nDoing manual labeling for individual samples is impractical given the size of the synthesized dataset (100K's). We follow a long line of work in the literature on using LLM-as-a-judge to scale quality check.\n\nDuring the process of designing the synthetic data generation pipeline, we continuously monitored the quality of the generated data. Based on our observations, we implemented a stringent quality check process (Section 3, point iv; Appendix A.2, Figure 7) that filters out low-quality samples. Only instances meeting the specific criteria are retained. Recognizing the significant effort required for human expert labeling, our methodology provides a scalable alternative. Our experimental results clearly show improvements upon finetuning with the synthesized data. Nevertheless, following up on the reviewers' suggestion, we will conduct a study to evaluate agreement between the LLM and human reviewers on sample quality and include our findings in the paper.\n\n\n> A minor concern is whether the prompt length may exceed the model's context window (during data generation).\n\nThank you for raising this concern. To clarify, we used GPT-4o and Llama-3.3-70B for data generation, both of which have sufficiently large context windows to accommodate our prompts. Therefore, exceeding the model’s context length was not an issue during the data generation process.", "claims_and_evidence": "- While the challenges discussed in the paper appear relevant to the code editing task, the reviewer is concerned that these issues are common across all machine learning tasks. The scarcity of high-quality fine-tuning data and the risk of catastrophic forgetting during fine-tuning are well-known problems in ML models in general. This raises concerns that the paper’s contribution may not be sufficiently distinct.", "methods_and_evaluation_criteria": "- The reviewer has a particular interest in the quality of the generated dataset. In Line 210 (L), the authors state that they perform quality-based filtering by prompting the LLM to select high-quality examples. However, the reviewer believes that this step requires some level of human intervention, including: (1) Defining the criteria for assigning a specific score. (2) Providing demonstration examples to guide the LLM in evaluating dataset quality. (3) (If possible) Conducting an empirical study to assess whether the LLM's evaluation standards align with those of human experts. For examples of empirical study, please refer to section 4.4 in [1].\n\n[1] AutoDSL: Automated domain-specific language design for structural representation of procedures with constraints, ACL’24", "theoretical_claims": "- The reviewer has examined the proposed SeleKT algorithm for parameter optimization and considers it a general methodology for artificial neural network adaptation. However, the reviewer is uncertain whether its design choices are specifically tailored for the code editing task.", "experimental_designs_or_analyses": "- The reviewer has examined the result analysis section and agrees that the proposed SeleKT method enhances code editing efficiency and delivers better performance. Additionally, the reviewer considers SeleKT a general methodology applicable to training open-source ANN models for improved code editing.", "relation_to_broader_scientific_literature": "- The paper proposes an improved methodology for code editing, which is a code intelligence task that involves modifying code based on natural language instructions.", "essential_references_not_discussed": "- No comment", "other_strengths_and_weaknesses": "- No comment.", "other_comments_or_suggestions": "- [Table 3] There is excessive blank space below the table caption. Please check the typeset configuration.\n- [line 337] The model name “DeepSeek-R1-Qwen-7B” is missing the \\textsf formatting.", "questions_for_authors": "- No question.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eq7wJqmjwv", "reviewer": "Reviewer_MF2X", "summary": "The authors present a comprehensive approach to enhance the code editing capabilities of language models while maintaining their pre-existing abilities. Their work addresses two fundamental challenges in this domain: the scarcity of high-quality fine-tuning data for code editing tasks and the phenomenon of catastrophic forgetting during domain adaptation.\n\nThe primary contributions of this research are threefold. First, the authors introduce a synthetic data generation pipeline to diverse code editing examples across eight programming languages. This pipeline systematically generates original code, modified code, and corresponding natural language instructions with varying levels of verbosity and styles, encompassing multiple edit types and code granularities.\n\nSecond, the authors propose the Selective Knowledge Transfer (SeleKT) algorithm, a robust adaptation method that employs dense gradient-based steps to identify critical weights for code editing tasks, followed by sparse projections onto the base model to mitigate overfitting. Unlike conventional approaches that predetermine updatable weights, SeleKT dynamically reassesses weight importance throughout the fine-tuning process based on magnitude changes.\n\nThird, through empirical evaluation on four established benchmarks (CanItEdit, HumanEvalFix, NoFunEval, and Aider), the authors demonstrate that their adaptation of Qwen2.5-Coder-7B, named NextCoder, surpasses comparable models and even competes with substantially larger models on several tasks. Additionally, their experiments confirm that the SeleKT approach generalizes across model families, as evidenced by improvements in DeepSeekCoder-6.7B performance.\n\n## update after rebuttal\nI thank the authors for their detailed response and my fellow reviewers for their insightful comments. Based on the additional ablation results, I have revised my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:\n\n> Authors have not done any analysis on generating data overlap with benchmark ...\n\nThe benchmarks considered in the paper are based on manually-created coding problems and solutions. Whereas, we used the training split of the StarCoder dataset, which is derived from GitHub commits, as our seed data. The synthetic data pipeline generates samples inspired by the seed data. This reduces the likelihood of unintended overlap with the benchmark data and prevents inadvertent overfitting.\n\nWe agree on the importance of this point with the reviewer and quantitatively analyze overlap using the standard approach of decontamination used in StarCoder: [bigcode-dataset/decontamination](https://github.com/bigcode-project/bigcode-dataset/tree/main/decontamination). The decontamination report confirms 0% overlap between our training data and benchmarks.\n\nWe will include a discussion on this and report the decontamination measurement in the revised version.\n\n> Lack of hyperparam tuning for baselines ...\n\nWe have documented the specifics of our SFT and LoRA hyperparameters in Appendix A.1.2. For LoRA, we utilized the hyperparameters from the Qwen model official implementation [scripts](https://github.com/QwenLM/Qwen/blob/main/finetune.py#L55). To address the reviewer's concern, we have conducted an initial experiment on hyperparameter tuning for baseline methods. Given that SeleKT was trained with a learning rate of 1e-5 and weight decay of 0.0, we explored closer configurations for LoRA and SFT, testing learning rates of 2e-6 and 5e-6 with weight decay values of 0.10 and 0.05. Additionally, for LoRA, we experimented with ranks from 16 to 64 and alpha values of 8 and 16. While this tuning led to some improvements in both SFT and LoRA, a significant performance gap remains between these baselines and NextCoder-7B. We will include these updated results in the final version.\n\n> Not clear for how many steps the model was trained.\n\nWe trained all our models for 3 epochs (Section 5.1).\n\n> Theoretical insights into the optimization dynamics or convergence behavior in non-convex settings\n\nPlease refer to a detailed response to reviewer 3QHQ (3rd point).\n\n> Lack of data scaling curve\n\nPlease refer to a detailed response to reviewer 3QHQ (2nd point).\n\n> To measure model's ability to retain generic performance, can you also evaluate it on other tasks such as MMLU?\n\nIn addition to (a) the additional experiment on MMLU as suggested by the reviewer, we also conducted (b) experiments on GSM8K to further demonstrate the ability of our method to retain generic performance.\n\nFor evaluation, we followed the few-shot setting (N=4) and the same prompt used in the Qwen models' official evaluation script. Given that our model is designed for code-related tasks, we focused on the STEM subset of MMLU, which contains 3.15K problems covering topics such as: Physics, Chemistry, Biology, Computer Science, Mathematics and Engineering. This subset aligns closely with the problem-solving and computational reasoning abilities expected from a code-editing model, making it a more meaningful evaluation of whether fine-tuning on code has impacted general problem-solving performance. For GSM8K, we considered the full benchmark.\n\n\n| Model                      | MMLU | GSM8K |\n| -------------------------- | -------- | -------- |\n| Qwen2.5-Coder-7B-Instruct  | 53.0     | 83.40 |\n| Qwen2.5-Coder-32B-Instruct | 71.9     | 93.71 |\n| NextCoder-7B      | 54.5     | 81.65 |\n| NextCoder-32B     | 72.7    | 92.65 |\n\nThe above table presents the accuracy scores for our NextCoder models alongside the Qwen2.5-Coder models. These results substantiate the robustness of our approach, in particular the absence of catastrophic forgetting is evident. We will add these results to the paper.\n\n> Underlying LLM for data generation might introduce systematic biases or fail to capture rare-code editing scenarios.\n\nThe reviewer raises important issues which can affect effectiveness of synthetic data generation. To mitigate these issues, we implemented a multi-faceted approach to ensure diversity and reduce systematic biases. Our synthetic data generation strategy relies on a diverse set of seed data (which, for example, is significantly larger than [WizardCoder’s](https://arxiv.org/pdf/2306.08568) 20K instances) as the basis (Section 3, point i), ensuring a broad initial representation of code editing contexts. We further enhance the coverage/diversity of our generated scenarios by incorporating three randomly selected improvement areas (Section 3, point i) for each synthetic data instance. This approach helps prevent the model from converging on a narrow set of editing patterns.\n\nBy deliberately introducing randomness through multiple improvement areas and using a diverse initial dataset, we have tried to mitigate bias and coverage concerns.", "claims_and_evidence": "Most of the claims made in the paper are generally supported by convincing empirical evidence including \n\n- NextCoder (using SeleKT) outperforms comparable models and even larger ones, on diverse code-editing benchmarks.\n- Proposed Automatic Synthetic data pipeline yields gains over CommitPackFT data baseline.  \n0 SeleKT preserves pre-learned abilities is validated by Table 5, which shows that models fine-tuned with SeleKT retain their code generation capabilities better than those fine-tuned with alternative methods.\n\nHowever, there are several claims which lack robust evidence. In particular \n\n- Synthetic Data quality and generalization: The claim that the synthetic data pipeline captures real-world diversity is less substantiated, as the analysis relies heavily on select benchmark performance. We know that one can improve task performance by obtaining synthetic samples similar to the end task distributions. Since authors have not done any analysis on generating data overlap with benchmark data, it’s even not clear if they are inadvertently overfitting to test. Further, the proposed pipeline solely relies on LLM based automatic checks and efficacy of such checks is not evaluated by the authors.  \n\n- Lack of hyperparam tuning for baselines: While authors present extensive analysis for their own proposed algo SeleKT, I find the SFT, LoRA baseline numbers to be less convincing as authors show that SFT leads to huge degradation (65.6-> 59.5 on MBPP+). It’s not clear for how many steps the model was trained and what hyperpamas authors tried to reduce potential overfitting.\n\n- Lack of data scaling curve: On synthetic data, authors do not show how scaling up data size improves performance and when do we see some sort of performance saturation. \n\n-  SeleKT generalization: I strongly suggest authors to measure generalization beyond Humaneval and MBPP. For example, they can also use MMLU or some non-code related dataset to truly measure degradation in model quality.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are well-tailored to the problem of adapting code language models for diverse code edits.\n- Synthetic Data Generation Pipeline: The approach builds on previous successes in using synthetic data for instruction tuning (e.g., self-instruct methods in natural language processing) and extends those ideas to the code domain, addressing the limitations of using only mined commit data. The observed gains on multiple benchmarks (see Tables 1 and 6) validate the effectiveness of the proposed synthetic data approach.\n- Diverse Benchmark Datasets: The use of benchmarks such as CanItEdit, HumanEvalFix, Aider, and NoFunEval is appropriate because they capture a variety of editing scenarios—from function-level bug fixes to full file improvements and even non-functional aspects like performance or security. This diversity is crucial for assessing how well the adapted model handles the breadth of real-world code-editing tasks. Additionally, evaluation on generation benchmarks like HumanEval+ and MBPP+ ensures that any gains in code editing do not come at the expense of the model’s fundamental code generation and comprehension abilities.", "theoretical_claims": "The proof for Lemma 1 is correct in its intended scope—it shows that the selective update mechanism guarantees that at most $c$\nparameters are altered relative to the base model. However, it is limited to a counting argument and does not offer any theoretical insights into the optimization dynamics or convergence behavior in non-convex settings.", "experimental_designs_or_analyses": "- Benchmark Selection and Evaluation: Authors used standard benchmarks for code edits (HumanEvalFix, Aider) as well as generic code generation performance (Humaneval, MBPP), which is a very reasonable choice. To measure if model retain performance on generic tasks, I would also preferred to include benchmarks beyond just code generation. \n- Quality and Representativeness of synthetic data: The soundness of the synthetic data generation is critical. While the pipeline is innovative, the quality of the synthetic examples depends on the underlying LLMs used for generation, which may introduce systematic biases or fail to capture rare code-editing scenarios.\n- Ablation Studies on Hyperparameters: For SeleKT algo, authors provide detailed ablations on key hyperparameters, such as the sparsity factor $alpha$. However, for baseline methods, authors don't provide details on hyperparam selection. \n- Ablations on synthetic data: While the proposed data generation pipeline is reasonable, paper lacks ablations related to quality of generated data beyond end task performance, how scaling up data improves performance.", "relation_to_broader_scientific_literature": "- The idea of generating synthetic instruction-response pairs for fine-tuning has become a standard method in natural language processing, as seen in self-instruct pipelines (e.g., Wei et al., 2024a; Wang et al., 2023). The paper extends these ideas to code editing, building on methods like CodeAlpaca and Self-Instruct but specifically targeting the diversity of code modifications. Overall, it's a well-tested recipe to improve performance. \n- The challenge of catastrophic forgetting in fine-tuning has been extensively studied (Goodfellow et al., 2013; Kirkpatrick et al., 2017). SeleKT is motivated by these works, aiming to strike a balance between acquiring new task-specific knowledge and retaining general capabilities. Further, techniques such as LoRA (Hu et al., 2021) and recent sparse adaptation methods (Nguyen et al., 2024b) update only a subset of parameters to avoid overfitting. Unlike these methods, which select parameters a priori, SeleKT dynamically reassesses which parameters to update by computing the full gradient periodically, then selecting the top-k updates. This approach is more adaptive and echoes ideas from model merging techniques like TIES (Yadav et al., 2024) but integrates the selection process during training rather than post-hoc.", "essential_references_not_discussed": "While one can always cite additional papers, I think authors have cited relevant literature to connect paper to existing ideas.", "other_strengths_and_weaknesses": "- Lack of baseline optimization: It is unclear if hyperparameter settings have been optimized for baseline used in paper including SFT, PEFT. The effectiveness of these comparisons hinges on whether each baseline was optimally tuned. \n- Lack of data ablations on synthetic data quality: Authors should consider conducting scaling experiments to show how quantity of data improves performance and should conduct human annotations of a subsample to explain the quality and limitations of generated data.", "other_comments_or_suggestions": "I don't have other comments", "questions_for_authors": "- Could you please provide SFT and LoRA fine-tuning details including hyperparams tried to reduce overfitting? \n- To measure model's ability to retain generic performance, can you also evaluate it on other tasks such as MMLU?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gp10vKAfDM", "reviewer": "Reviewer_itGF", "summary": "This paper addresses two issues: (1) enhancing code language models on code-editing tasks; and (2) mitigating catastrophic forgetting caused by task-specific fine-tuning. To address (1), it proposes a method for synthesizing high-quality code-editing data; to address (2), it introduces Selective Knowledge Transfer (SeleKT), a low-rank optimization technique that limits parameter updates. Experiments on four code-editing benchmarks demonstrate that the approach achieves state-of-the-art performance with models of comparable scale while preserving robust code generation capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:\n\n> My concern lies in why the LLM-generated dataset is realistic and representative? Specifically, ..., how could the synthesized dataset to represent such distribution.\n\nOur synthetic data generation, by design, supports multiple correlated edits within a single task. Indeed, our pipeline generates multi-edit examples and we will add concrete examples in the appendix in the revised version. This approach mirrors real-world software development, where changes often span multiple classes and interrelated code segments. By generating data that captures the complexity of correlated changes, we ensure an authentic representation of complex code editing scenarios.\n\nTo further validate our dataset's ability to handle multi-file edits, we finetuned larger QwenCoder-2.5 variants and evaluated them on [Aider Polyglot](https://aider.chat/2024/12/21/polyglot.html#the-polyglot-benchmark), a challenging benchmark with multi-file edit problems. Due to char limits, we are unable to give a concrete example in this response. However, we would be happy to share it if the reviewer wants, in another response. Our NextCoder models show significant performance gains, as summarized in the tables below.\n\n| **14B Models** | **Polyglot** |\n|:----------|:------------:|\n| QwenCoder-2.5-14B | 9.3 |\n| QwenCoder-2.5-14B-LoRA | 5.3 |\n| QwenCoder-2.5-14B-SFT | 3.1 |\n| **NextCoder-14B** | **12.2** |\n\n| **32B Models** | **Polyglot** |\n|:----------|:------------:|\n| QwenCoder-2.5-32B | 16.4 |\n| QwenCoder-2.5-32B-LoRA | 6.7 |\n| QwenCoder-2.5-32B-SFT | 8.4 |\n| **NextCoder-32B** | **21.9** |\n\n> I suggest the trained model shall be evaluated on some real-world commit datasets like [CoEdPilot](https://arxiv.org/abs/2408.01733).\n\nWe agree that evaluation on real-world commits is essential -- and we have already done this. A subset of the NoFunEval benchmark [CoLM 2024], which includes five splits assessing a model’s ability to improve code on multiple non-functional requirements (e.g., runtime, safety, etc.), is derived from real-world commits in Android repositories. In particular, the latency and resource-utilization splits are derived from real commits. The results of these evaluations are discussed in Section 5.2 of our paper.\n\nRegarding CoEdPilot, while it represents an interesting benchmark derived from real-world commits, its focus on edit propagation (predicting edits across multiple locations based on patterns of previous edits) differs substantially from our work's objectives. Our approach aims to perform targeted code edits based on natural language instructions, rather than propagating edits across a codebase. Additionally, CoEdPilot's emphasis on fine-grained edit detection and classification at the line level (keep/insert/replace) wouldn't align well with our instruction-following paradigm. The fundamental difference is that our model requires explicit natural language instructions, whereas CoEdPilot infers edit patterns automatically. If required, we will manually write instructions for some of the instances from CoEdPilot and include results in the revised version.\n\n> Code editing can involve both edit location and edit generation, as in [CoEdPilot](https://arxiv.org/abs/2408.01733). The authors seems to miss the edit generators such as [CODIT](https://arxiv.org/abs/1810.00314).\n\nWe thank the reviewer for pointing out CoEdPilot and CODIT; however, these methods target distinct scenarios. CoEdPilot focuses on edit propagation, predicting future edits based on past edits, and employs a fine-grained edit detection mechanism (Edit-propagating Line Locator) to classify edit types (keep/insert/replace) at the line level. In contrast, our approach finetunes a code LM to follow natural-language instructions for editing a given codebase, without requiring past edits or explicit edit localization. CODIT, on the other hand, predicts repetitive edits using a tree-based neural machine translation model, focusing on structured edit patterns rather than general-purpose instruction-following for code modifications. Additionally, CODIT predates modern LLM-based approaches, making its methodology different from ours. These distinctions clarify that our work does not overlook edit generators but instead addresses a separate formulation of the code-editing problem. We will incorporate these clarifications, comparing against CoEdPilot and CODIT, into the paper.\n\n> How do you evaluate the usefulness of the generated synthetic data?\n\nIn Section 5.4, we provide empirical evidence demonstrating the utility of our synthetic data compared to traditional commit-based datasets. Specifically, we observe that fine-tuning DeepSeek-6.7B on our synthetic data yields a performance advantage over CommitPackFT (a filtered dataset of high-quality commit data from GitHub).", "claims_and_evidence": "The claims \"Existing code LMs are deficient in handling diverse code-editing tasks\" and \"suppressing catastrophic forgetting in general models after fine-tuning\" raise concerns about the real-world applicability of research in the field of code language models.\n\nWhile the authors criticize the quality of existing code editing dataset, my concern lies in why the LLM-generated dataset is realistic and representative? Specifically, the real-world code edits for a change-requirement (a.k.a, a commit) can distributed across many files in a code repository, how could the synthesized dataset to represent such distribution.", "methods_and_evaluation_criteria": "Code editing can involve both edit location and edit generation, as in [2]. The authors seems to miss the edit generators such as \n\n[1] Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. 2022. CODIT: Code Editing With Tree-Based Neural Models. IEEE Transactions on Software Engineering 48, 4 (2022), 1385–1399. https://doi.org/10.1109/TSE. 2020.3020502\n\n[2] Chenyan Liu, Yufan Cai, Yun Lin, Yuhuan Huang, Yunrui Pei, Bo Jiang, Ping Yang, Jin Song Dong, and Hong Mei. CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature (ISSTA 2024)", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Lack of evaluation of the generated benchmark, why it is useful for training a practical code editing model. I suggest the trained model shall be evaluated on some real-world commit dataset as collected in [2].\n\n[2] Chenyan Liu, Yufan Cai, Yun Lin, Yuhuan Huang, Yunrui Pei, Bo Jiang, Ping Yang, Jin Song Dong, and Hong Mei. CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature (ISSTA 2024)", "relation_to_broader_scientific_literature": "The topic of the paper is practical and useful.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How do you evaluate the usefulness of the generated training datasets?\n2. Whether your approach can address the repository-level code editing, which is more practical in the real world?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work presents a synthetic data generation pipeline for code editing along with a method (seleKT) for “robust” LLM adaptation. The evaluation, along with some results presented during the rebuttal, suggest that the data/adaptation method performs strongly against alternatives.

The method presented is reasonable, the problem important, and the evaluation results are positive and hence I recommend that this paper is accepted. Please, incorporate the relevant results discussed in the rebuttal.

Beyond that, I should note that it’s not clear to me why the contributions of SeleKT and code editing data appear in this paper as they seem somewhat independent. In particular, SeleKT could have been evaluated in a wider range of applications and adaptation scenarios. I suggest that the authors either state more explicitly why SeleKT is only well-suited for code editing adaptation or provide some additional validation about the generalizability (or not) of SeleKT.
Decision: Accept (poster)