Coeditor: Leveraging Repo-level Diffs for Code Auto-editing
Authors: Jiayi Wei
Greg Durrett
Isil Dillig
Venue: ICLR 2024 spotlight (2024)
Keywords: language model for code
editing
refactoring
Primary area: generative models
TL;DR: Coeditor, a code editing model trained on code commits, surpasses previous code completion methods and unveils a new multi-round auto-editing application.

Abstract: Developers often dedicate significant time to maintaining and refactoring existing code. However, most prior work on generative models for code focuses solely on creating new code, overlooking the distinctive needs of editing existing code. In this work, we explore a multi-round code auto-editing setting, aiming to predict edits to a code region based on recent changes within the same codebase. Our model, Coeditor, is a fine-tuned language model specifically designed for code editing tasks. We represent code changes using a line diff format and employ static analysis to form large customized model contexts, ensuring the availability of appropriate information for prediction. We collect a code editing dataset from the commit histories of 1650 open-source Python projects for training and evaluation. In a simplified single-round, single-edit task, Coeditor significantly outperforms GPT-3.5 and SOTA open-source code completion models (bringing exact-match accuracy from 34.7 up to 60.4), demonstrating the benefits of incorporating editing history for code completion. In a multi-round, multi-edit setting, we observe substantial gains by iteratively conditioning on additional user edits. We have open-sourced our code, data, and model weights to encourage future research and have released a VSCode extension powered by our model for interactive IDE usage.

Published as a conference paper at ICLR 2024
COEDITOR: LEVERAGING REPO-LEVEL DIFFS FOR CODE AUTO-EDITING
Jiayi Wei* Augment Computing, Inc. j i ay auggenttcooe.coom com
Greg Durrett, Isil Dillig University of Texas at Austin (gdurrett, is @cs www.tsteea.......... edu
ABSTRACT
Developers often dedicate significant time to maintaining and refactoring existing code. However, most prior work on generative models for code focuses solely on creating new code, overlooking the distinctive needs of editing existing code. In this work, we explore a multi-round code auto-editing setting, aiming to predict edits code region based on recent changes within the same codebase. Our model, Coeditor, is fine-tuned language model specifically designed for code editing tasks. We represent code changes using a line diff' format and employ static analysis form large customized model contexts, ensuring the availability of appropriate information for prediction. We collect a code editing dataset from the commit histories of 1650 open-source Python projects for training and evaluation. In a simplified single-round, single-edit task, Coeditor significantly outperforms GPT-3.5 and SOTA open-source code completion models (bringing exact-match accuracy from 34.7 up to 60.4), demonstrating the benefits of incorporating editing history for code completion. In a multi-round, multi-edit setting, we observe substantial gains by iteratively conditioning on additional user edits. We have open-sourced our code, data, and model weights to encourage future research and have released VSCode extension powered by our model for interactive IDE usage.
INTRODUCTION
In recent years, there has been enormous interest in applying transformer models for code genera-
tion (Feng et al. 2020 Ahmad al., 2021 Wang et al. 2021 Chen al., 2021 Fried al. 2022
Allal et al. (202)), ) which has led to impressive performance on tasks such as program synthesis (Li
ct al. 2022 Nijkamp et 2022), program translation (Lachaux et al. 2020: Szafraniec et al.
2022
lype inference (Jesse et al.) 2022 Wei et al. 2023), and code auto-completion (Guo et al. 202
Svyatkovskiy et al. 2021 Nguyen & Nadi 2022] Zhang et al., 2023).
While these approaches effectively help programmers creating new code, they are not as adept assisting with revising existing code. Code completion tools like GitHub Copilot do not track programmers' changes and cannot predict where and how to make additional modifications. However, during a software project's development cycle, developers often spend significant time editing code- changes made to part of the codebase typically affect many others, and manually propagating these changes can be tedious and time-consuming.
In this paper, we introduce a task that we call (multi-round) auto-editing where the goal is predict edits to code conditioned on the user's 's previous edits. In particular, given an original codebase U and of code changes A .., As that are semantically related (like those forming part of commit), the auto-editing problem is to predict how to modify a specified region code u E U by learning the following distribution:
P(Au ... UI
Importantly, we allow the target region u to overlap with any previous modifications A Ag to support repeated editing of the same region. This formulation enables the workflow illustrated in Figure I where user can work alongside the model in multiple editing rounds, accepting suggestions matching the user's intent and making additional edits manually if necessary.
"Work performed while UT Austin.

---

Published as a conference paper at ICLR 2024
combine
Codebase History
Manual Changes
Git + Static Analysis
Ine 208 Ro, set,
Target Region def )
Relevant Context
User
Code changes sinse lust commit
Predicted Changes
Coeditor Model
The multi-round auto-editing task. The user inspects the model output in each editing round and can optionally perform manual editing.
Figure 1:
To solve this problem, we propose a new model called Coeditor that builds on top of the established CodeT5 model architecture and pre-trained checkpoint (Wang et al.) 202T). Coeditor is based on two key ideas. First, it encodes all prior code edits A...... As using line-based diffing scheme and decodes Au using masked span infilling; and, second, it uses lightweight static analysis to pull in relevant parts of the codebase U. To effectively handle large contexts with numerous code changes, we also replace CodeT55ss dense attention with a block-sparse attention pattern, allowing us to reduce the computational cost while maintaining the ability to attend to all relevant code changes.
Another challenge in developing Coeditor is the lack of suitable training data for multi-rount auto- editing. We address this issue by collecting new dataset, PYCOMMITS, from the commit histories of 1650 open-source Python projects on GitHub. We compute tree-differences between adjacent codebase versions to identify modifications the same Python function and randomly split some changes into the model imput for training in repeated editing scenarios. During testing, we use ground truth code changes simulate user decisions regarding when to accept partial changes suggested by the model and when to manually perform edits missed by the model.
We compare our approach against existing code infilling models and show that, even in a simplified setting that requires predicting a single edited line in isolation, they severely lag behind our change- aware model: our method achieves 60.4% exact match accuracy using 220M parameter model, whereas
Salavinci0003. -003-the best performing code infilling model we have compared, which has 175 billion parameters-aacieves only 30.5%. In the full multi-round setting, we found that Coeditor automates editing 46.7% of the changed lines, saving the user 28.6% of keystrokes measured by an edit distance metric that accounts for cursor movement.
In summary, this paper presents the following main contributions:
We introduce the repo-level multi-round code editing task, along with the corresponding PYCOMMITS dataset and evaluation framework.
We introduce a new code editing model derived from CodeTT5, using a line diff- based encoding scheme and enhancements that enable the model to condition on long contexts and appropriate other parts of the codebase, addressing key challenges in this setting.
We release our source code, dataset, model checkpoint, as well as a VSCode extension that supports interactive usage foster future research
2 MOTIVATING EXAMPLE
In this section, we illustrate our technique using the example in Figure 2| showcasing a two-round interaction between the user and our Coeditor model. Subfigures (a) and (b) display two initial user changes, while subfigures (c) and (d) illustrate two sequential Coeditor invocations with inlined model suggestions. We further analyze this example in detail below.
First, the user modifies the pack_batcch batch function in subfigure (a) to read a new dictionary key, "cost", from each row in the input. The extracted values are used to compute the total cost
Availlale http:://jiiaa.coommmmlllsecccoon
2

---

Published as a conference paper at ICLR 2024
146
def backbbacchrross List (dict]) -> dict:
(a)
167 def group to, batchest
(b)
147
ids [+[" inpustput__tA [ks" for
rees]
groupt Sequence [DasscT/Querycditt,
148
"attooutut___ee [kk""] for
rows]
argaz BatchArgs,
149
refs a Ix sel_eeee"" for x 1n rows]
178
[dict]:
158
baten cost suai [X [cost") for in rovsl)
171
Cost_ cost_ prgs.cost_liniitl
151
s02ref (id(ref):
for
podit
152
references [102ref [x] for x 102ref)
processed [arocass_ (ddit angs) for is groupl Input, tix list [=[R] for in processed]
153
isZorder [x: 1 for 4, in enuserate: (d2ref))
154
query_ret rlstst t [latbarder ref() for ref in roe) for
butpurt_tks. Uist [<(3] for processed)
155
current_cost
156
"Input_ids"s Ieput _1ss,
174
for i, edit in enunerate(greup):
157
*references": references,
175
key, stubs List (TokenSe:]))
159
158
"query_ref List": query_ref__ltsg
"Lazels": labels, "batch_coss"4 batch_costs
167
def group_tt, to, to_batches!
(c)
161
168+
210
289
ex_cest retritval Loostaaoooosssoooooooo (
ref_size-suni (enix) for in ref_setected),
0
suery_ inp.tttt__tts_tee 1ist
167
def group_te, ho_p_tt_bttcee(
(d)
0.212
sutsut_sizeetenl output tks listlil),
208
ref sef_sise_sa FUB renirefl
0113
200
[added]
but_tkk odditiddd, args]]
0.214
ref_selected. sellected.sortt sort sor.el..orttteelaabbbiee
is2ref_ seaeefreeeeii [id(x)]
218
ex_cost retrieva', sestrivvv__osttt.__ [
0.215
211
ref_size=suni len(x) for in ref_selected),
0.16
tks"" input, ttputt.tss_lu (t), "output, tkst: putput, tkg, ltl, "ref sel__eeet"""" ref_selected, ex cost,
in tks),
0177
outpor_
Hks),
212
0 212
sizry_=izewlen wogpeennninnttttte ths (t)),
Redbrr
0 213
outout_size- www.ssizzeeeennnn Senfoutput,_... 38s List Ii1l,
228
ex cost cost_ Hett::
O
ref_ seleecedd.ssrrttt (keyelanbda x: =22ref_naseiddii
022
vamings.vaml "Batch cost Limit is too sme ().")
о 216
row
0.222
costt current_batch. (ppenddrow) else:
cost cost_Lintt:
"input_tks": in_tks, "cutout_tko" tks":
0.223
0.24
"snput_tkss" input_ths, titt [tl, "outout_ output, ths listlal, "ref_selectes"; ref sef_selectes, "cost"f: ex_cost,
0.2
yield pack wwtchhturreettttnt
0 210
0.22
0 219
0.27
current__ost <xtt
(addec]
028
if current, bbttch:
0.221
0.2
yleld pack beoc_bbcchhcurrentt
Figure 2: An example usage of Coeditor. (a) The user first edits the pack_ function to read an additional dictionary key, "cost", from each row in the input. (b) The user then removes 3 lines at the top of the group_te to batches function. (c) The user now invokes Coeditor at the bottom half of the same function. Coeditor correctly suggests adding a "cos=" " key to the dictionary variable row, but it fails to address the now undefined variables underlined in red. (d) However, if the user accepts the suggested change and manually introduces two new variables at line 209, Coeditor can then suggest the correct changes accordingly.
of the batch and added to the output. Next, the user removes three lines at the top of the group_te to batcaaceee function in subfigure (b). By removing these three lines, the user wants to avoid creating these lists beforehand and instead plans to call the process_edt _e function inside the for loop below.
The user then scrolls down and invokes Coeditor at the bottom half of the same function (subfigure c). Here, the modified its argument paarent__batch is iteratively constructed from row, which is a dictionary defined at line 215. Hence, the model correctly infers that row should be updated to include a "cost" key. Examining the surrounding context, the model also identifies that the ex_ ex_cost variable (defined at line 209) should be used as the inserted dictionary value
batch function called at lines 225 and 228 in subfigure (c), and
While Coeditor makes some useful editing suggestions so far, it does not address the now-undefined variables underlined in red by the IDE in subfigure (c). In particular, as there are no obvious alternatives nearby replace these variables, Coeditor is unable automatically fix these errors. Such a situation is common when the surrounding changes alone do not provide sufficient information to derive complete solution. Therefore, the user can accept the partial changes suggested by the model and then manually introduce two new variables at line 209, as shown in subfigure (d). Coeditor can then leverage these new variables to suggest the correct changes needed to fix the errors.
This itcrative approach allows Coeditor to adapt and refine its suggestions based on additional user edits, providing more effective code editing experience than existing code completion techniques. By incorporating the editing history into the prediction context, Coeditor can better assist developers in a wide range of code editing tasks, from simple modifications and refactoring to more complex
CCoeditor produces the same result even when ch is defined far away in different file, as Coeditor tracks all changes the user has made since the last commit and incorporates them into its context.
3

---

Published as a conference paper at ICLR 2024
codebase- wiidd updates. For more comprehensive understanding, have included screen recording of the Coeditor VSCode Extension, which can found the supplementary material.
METHODS
Recall from the introduction that we wish to model the distribution P(Au S......UU), To
this end, first describe how encode the target change Au and contextual changes A.... A (subsection 3.Tp. We then describe how to form the context from the codebase U using function signatures (subsection 3.2). These choices naturally lead to model compatible with fine-tuning CodeT5, which was pre-trained on the masked span infilling task (subsection 3.3). Finally, we describe our new dataset that used fine-tune this model (subsection 3.4).
3.1 ENCODING CODE CHANGES
A suitable format is required to map code changes into token sequences that can be processed by seq2seq transformer language model. In our setting, we want to select a format that encodes and decodes code changes in uniform manner while minimizing the number of tokens the model needs to produce. Hence, we adopt a line-diff- diff-diffbasee format, enabling us to convert auto-editing into a masked span infilling problem (Wang et al. 2
Consider a block of code u to be made up of lines .1....,,m and a user-specified edit region that spans between line and n, where
S a S m. Moreover, each line is associated with a status variable st indicating what type of change (if any) has already been made; E (cmpty), <ada>, <del>} 1 We encode the input code by function EncInput that (optionally) prepends status tokens
9m and placeholder tokens <>>... <n> at the start of each line:
Encinput(u u) = 51/18212... ... <1>sala <2>s2+1 1144111411110 <n> <n>satnaate
Smn
For contextual changes Å ..... As. we can encode them using the same format but with an empty edit region. When the target change Au contains line additions, denoting the jth line to be inserted before line as Ky, we can encode Au using the following expression.
EncOutput(ANy (Au) = <1>I.D.<2>Iattt
<n>latn Datn:
where 1, = <add> Nn1 <add>
V/2 <add> (in,)
D. = if 4 is to be deleted then <del> else (empty).
Note that we add further restriction that forbids D; from being <del> >> if s; is <add> in order to prevent the model from modifying line that has just been added; we discuss this in more detail in section 6 Figure 3 iillustrates this line-diff-based encoding scheme using the example from Figure 2 This format ensures that we replace the placeholder tokens in the input with the corresponding changes specified in the output sequence, we obtain the total change that combines and Au.
3.2 ANALYZING RELEVANT SIGNATURES
Having described how we encode code changes, we must also establish a method for feeding U. the remaining codebase, to the model. Simply inputting the entire codebase as is would result in excessive number of tokens, overwhelming the context. Instead, inspired by the ideas proposed in previous type inference work (Pradel et al. 2020 Wei al. 2020 2023), we employ lightweight static analysis to extract the most relevant information into the context, as outlined below.
For each target code region u, we analyze its pre-edit code and generate list of its usages] In the case of a function usage, we retrieve its function signature; for variable or class member usage, we retrieve the first statement in which was assigned. We then concatenate all these usages into single
3 'Prior work has proposed various methods to produce code changes. e.g., Zhang et al. 2022 learns the distribution P(u' u) and Reid & Ncubig (2022) tags each input token with a label indicating deletion, insertion, replacement. However, these methods require more copying tagging, resulting in longer output sequences compared our approach.
We represent edits as line diffs output by Di compare using the standard di ff1 ib library. 'ee use the Jedi package for this purpose: httts:/// cww/daviivvvoooo dha ter/jedd
4

---

Published as a conference paper at ICLR 2024
6.)
<5>
ref_size_
query sey__sie=llan Sine-lenn(nnuus tks),
ref_selected (ist(TokenSeal() for ref in alt_refs:
<del>
<
<add>
attptt_size-lon(urppes (oattpttttt)
if ref_size_sun ten(ref) args.nex_totel
<
ref sef_selected.appeddiett
[...]
ref_size_sun tentref)
<1>>
"input_tks": Input_tks,
<b>cadd>
input__tks, output_tks procest_editfedit,
<del>
<5>
exccost retrievel_cost_nddell
<3> <add>
"output_tks": putput_tks,
ref_size==uu(leenx) ref_selected),
<7>
<1
<8>
at: notivating
<18>
ref seLeeleeedd x: idZref_naneiidix (id(x)
Tokenšeq List[Token] back_batch(rowee list(dictt) -> dict
<1>>
"input_tks": (our_ttk__istii))
retrieval rodeerrref_eeeeeeeeeeeooee int, query_size: int, output_size: int) float
<13>
coutput_tks" autput_ list
"ree_setected": ref_selected, "cost": ex_cost,
<15>addd>
Sertvating.Batcchhle cost_ timit() - float nax, max_query__tee tkst 512
<16>
<17>
if cost_limit:
<18-
warnings.warm""aaeer cost timit is too small. current_cost cost_limit:
rex_ref_iks: int 512
nox_total_ref_tte int 256
[...]
Figure 3: Coeditor encoding format. (Left) the input sequence adds placeholder tokens to indicate code region to edit. (Top right) the output sequence specifies further changes at each placeholder token. (Bottom right) relevant signatures are retrieved from the codebase and added to the context. (In this example, the Python module is called mot vating).
A
Aa
B
BN
C
C
Q1
Q2
A
As
B
B2
C
C2
Qt
Q2
Ref A
Ref B
Ref
C
Query
Figure 4: Coeditor encoder sparse attention pattern. All attention between the reference blocks are skipped avoid the quadratic cost of dense attention.
"document". shown at the bottom right Figure 3] which serves additional input context. This approach allows the model to access the most pertinent information about the current code region and significantly improves model performance (Table while generating only small number of extra tokens in the context (Table
3.3 ADAPTING CODETS
Our model is based on the architecture and pre-trained weights of CodeT5 (Wang et al. 2021) CodeT5 was pre-trained on a large corpus of code data using the masked span infilling objective, making it suitable choice for our problem. We employ the CodeT5-base model, containing 220M parameters, and fine-tune it for our code auto-editing setting. Although the original CodeTT model was pre-trained with small sequence length of 512, its relative positional encoding scheme allows us to fine-tune it on much longer sequences for our problem.
Considering that a single commit may encompass numerous code changes, concatenating all changes into single input can lead to long token sequences that are difficult for the Code'T' model to process with dense attention. To mitigate this issue, we replace the full attention in its encoder with a block-sparse attention pattern, illustrated in Figure 4 This pattern divides the input sequences into multiple reference blocks and a query block. The query block contains the code to edited, whereas each reference block encodes contextual unit change Au, chunk of the signature document. We limit the sequence length each block to 512 tokens for references and 1024 for the query, dividing longer blocks into multiple if necessary. The self-attention within each block is performed as
5

---

Published as a conference paper at ICLR 2024
Table 1 General statistics of the PYCOMMITS dataset.
train
valid
test
projects
1550
50
used commits
217K
5006
5854
modified files
501K
10.1K
11.1K
modified functions
958K
20.1K
22.5K
modified lines
7.10M
143K
169K
Table 2: Additional statistics specific to our technique, computed over the test set.
definition
median
mean
max
max
query tokens output tokens prev change tokens
EncInput(x) EntOut (Au)
258
361.9
1024
7.8%
60
89.7
512
1.3%
EncInputi A Ax)
1625
4.14K
16.4K
11.2%
signature tokens
(sinnature (t) (ECsages(()
313
515.5
15.9K
0.0%
usual, but the attention between different reference blocks is skipped to save computation, similar to other retrieval-augmenter models (Izacard & Grave 202I). p.. However, we still allow the query block to attend to and be attended by all reference blocks (a global attention block (Beltagy al.) 2020 Zaheer et al. 2020)). We also set the relative distance between each reference and the query to be infinite when computing the relative positional encoding, making the model is insensitive the ordering of the references. We are able to use a total of 16.4K reference tokens at test time, which is sufficient cover 88.8% of problem instances in
without truncating the context (Table 2). See subsection A.2 for more discussion of long-dos documentnnnt attention mechanisms.
3.4 THE PYCOMMITS DATASET
To train our model, we gather real-world code changes from the commit histories of open-source Python projects, a dataset we call PYCOMMITS. For each commit, we first identify which changes are made the same code unit (a unit can be either a function, a region of a class, or a region of module) and subsequently separate the commit into a list of unit additions, unit deletions, or unit modifications. As our work primarily focuses on code editing, only unit modifications are used as training labels, while the other two types of changes remain visible to the model as context.
For each unit modification, we create a training problem instance that instructs the model to predict the code change based on all prior (but not future) changes from the same commit. Git does not record the editing order of changes within the same commit, so we employ a simple heuristic that sorts unit changes according to their source code locations and the import order between modules. Specifically, we assume that units within the same file are modified from top to bottom, and if module imports another module, changes in the imported module occur before those in the importing module To train our model for the proposed multi-round editing setting, we generate synthetic data demonstrating repeated editing to the same code unit as follows: for those code change involving least two changed lines, we randomly sample subset of the changes as the prediction target and line the remaining changes into the input. For example, the problem instance shown in Figure can be generated by inlining of the 6 changed lines in the input.
We construct new code editing dataset using the commit history of 1,650 Python projects with permissive licenses (MIT, Apache, and BSD) sourced from GitHuu. We use 50 of the projects for testing and 50 for validation and use the remaining 1,550 projects for training. We use at most 1000 commits per project per project to ensure that the model is trained on diverse set of code changes. We show the general statistics in Table I and the statictics that are specific to our technique in Table Tokenization is performed using the CodeT55 tokenizer.
"Note that this ordering mainly affects how we generate the training and testing data. At test time, model condition changes hoth above and below the edit region.

---

Published as a conference paper at ICLR 2024
Table 3: Performance on 5000 code completion instances extracted from edits (PYCOMMITS- ONELINE). Add EM and Replace EM are the (enhanced) exact-match accuracies on addition and replacement change, respectively.
Exact Match Rate (%)
Model
Parameters
Context
Add
Replace
Overall
InCoderIB
1.3B
2K
29.0
25.2
26.2
InCoder6B
6.7B
2K
34.0
30.4
31.3
SantaCoder
1.1B
2K
31.0
28.1
28.8
StarCoder7B
7B
8K
37.9
33.7
34.8
text-davinci-003
175B
4K
40.2
39.3
39.5
Coeditor
220M
16K
47.1
64.9
60.4
4 EVALUATION
In this section, we first compare Coeditor with prior code completion approaches simplified version of the editing task. We then report Coeditor's performance on the proposed multi-round editing task and conduct ablation studies. Example model outputs are included in the appendix.
Training Setup We initialize Coeditor with the CodeT5-bbse checkpoint (220M parameters) and train the model our training for 1.75 epoch, gradually increasing the model reference context size from 2048 tokens to 4096 tokens (at epoch 1) and then to 8192 tokens (at epoch 1.5). We use Huggingface's 's Trainer implementation and the AdamW optimizer, with linear learning rate schedule with a starting learning rate of 2e-5 and 0.01 weight decay. We train the model with fixed batch size of and total of 1.34 million training steps. Training took about days on a single NVIDIA Quadro RTX 8000 GPU with 48 GB memory.
4.1 COMPARISON WITH CODE COMPLETION APPROACHES
Baselines We compare Coeditor with four open-source code generation models: InCoder- IB, InCoder-6B (Fried et al. 2022). SantaCoder (Allal et al.) 2023), , and StarCoder7B (Li et al. (023).
All four code generation models have been trained with the Fill-i-theemmidlee pre-training objec-
tive (Aghajanyan et al. 2022). We also compare Coeditor with OpenAl's text ext-davinnes model, which is the most recent GPT model supporting an infilling interface |
Creating test instances We generate code completion problem instances from real commits as follows. For each code change PYCOMMITS, take the last changed line as the completion target. If the last change is a modification, we delete the modified line and let the model fill in the new version of the line. If the last change a deletion, we simply discard the change. We then inline all changes before the target into the prediction context. This inlining process is implemented differently each model: for our Coeditor model, inlined changes are visible to our model following the encoding scheme described in subsection 3.I; for the code completion models, we simply apply the inlined changes to the original code and use the resulting state as the model input. Also note that while model constructs its prediction context using relevant changes and static analysis (as described in suusecctionn 3.2), the code completion models (which are unaware of code changes) only the code surrounding the completion target the prediction context. We call this test dataset, derived from our PYCOMMITS test set, PYCOMMITS-ONEL.C
Results We report the performance (without fine-tuning on this task) of all approaches in Table We use an enhanced exact-match (EM) accuracy metric that performs semantic-preserviin code normalization before checking for string equivalence [ All results, except for text Slavvncc-00 were obtained by measuring 5000 randomly sampled code completion problems from our test set. text sont-davinci 300 was evaluated using only 1000 problems reduce cost. We see that Coeditor
We also experimented with prompting gpt 1.533.5tuue. 5-turbo for this task using either suffix-then thee--tee---eeee format some natural language instructions followed by code with a special <M iSS ING IRI marker, but these variants worked less well than text-davii nei 0003% native infilling API.
WWe normalize Python code by (1) parsing the code into syntax tree using the ast lihrary, (2) removing any comments and docstrings, (3) sorting all keyword arguments in function calls, and (4) un-parsing the syntax tree.
7

---

Published as a conference paper at ICLR 2024
Table 4: Multi-round evaluation results measured 5000 problems from the PYCOMMITS set. Lines, Levenshtein, and Keystrokes are the average total gains in the corresponding metrics. Rounds the average number of rounds needed to complete all desired changes.
Setting
Lines (%)
Levenshtein (%)
Keystrokes (%)
Rounds
SingleRound 28.5
23.1
19.2
MultiRound 46.7
25,9
28.6
2.43
significantly outperforms the other code generation models for both addition and replacement changes. Despite only using a 220M parameter model, Coeditor achieves an overall EM of 60.4%, which is approximately 1.5 times higher than the best performing code completion model (39.5%), demonstrating the sigmificant benefits of incorporating editing history for code completion. We also include example model outputs this task in the appendix (subsection A.3)
4.2 MULTI-ROUND EDITING
This evaluation focuses on the editing assistant use case where we assume the user has some desired code changes in mind, and we aim evaluate how much the model can save the user's effort by automating much changes possible, potentially under the guidance of the user. The user can accept partial changes suggested by the model and make additional changes manually if needed.
Evaluation workflow To evaluate the above use case automatically, we use the ground-truth code changes to simulate the user's actions. In particular, when the model predicts a list of changes, we compare the predicted changes against the ground truth changes line-by-line and accept any line change that exactly matches the ground truth. none of the suggested changes match the ground truth, we assume the user will manually perform the first remaining change. In both cases, after the additional changes, we rerun the model to obtain new suggestions and repeat until all desired changes have been performed the round limit 6 has been reached. In the end, we compute the total gain using the difference between the editing cost of the ground truth and the accumulative editing cost of all manually performed edits.
Measuring editing cost There are multiple ways to measure the cost of performing a code change. Since there is no consensus on the best metric, we report 3 metrics in our results. Prior work (Lavazzal et al. 2023) suggests that for code understanding tasks, simple line counts- based metrics are almost as good as more complex metries, hence our first metric. Lines, simply measures the number of changed lines before and after the edit. We also report Levenshtein, the classic Levenshtein editing distance metric that measures the minimal number of character addition, deletion, and substitution needed to transform one string into another. Although simple, the Levenshtein distance doesn't 't model many important aspects of code editing, such as the cost associated with cursor movement, and it also under-count the cost of substitution and over-count the cost of large deletions. Hence, we propose an additional metric, Keystrokes, that aims to better approximate the number of needed user keystrokes than Levenshtein by allowing for batch deletion and accounting for the cost of cursor movements. We describe this metric detail in the lsubsectioon A.I
Results We report the evaluation results on 5000 problems sampled from test set in Table and we also report the single-round performance for reference. We see that Coeditor achieves much larger total gains under the multi-round setting, especially when measured with the Lines and Keystrokes metric (which we believe more accurately captures the user editing effort than Levenshtein). We also show 3 examples of the model's suggestions the appendix (subsection A.4.
4.3 ABLATION STUDIES
We retrain the model with various components disabled to study their impact on the overall model performance. We report the (single-round) exact match performance of each variation on the entire PYCOMMITS validation set in Table 5) The results show that removing any of the components leads to a decrease in performance, highlighting the importance of each component in the overall model. Specifically, when remove the explicit feeding of code changes (No Diffs), the EM drops the most, from 42.1% to 26.1 1 %. When we disable the static analysis component (No Signatures). the EM decreases to 33.3%. Using a smaller limit of reference tokens impacts the model performance the
8

---

Published as a conference paper at ICLR 2024
Table 5: Ablation results the entire validation set (PYCOMMITS). All pairwise differences statistically significant with 0.05 using paired bootstrap test.
Ablation
Description
EM (%)
No Diffs
Feeding the same input to the model except that all changes are replaced with their post-edit results alone.
26.1
No Signatures
Disabling the static analysis component and removing function and class signatures from the prediction context.
33.3
Small Context
Reducing the max number of reference tokens from 16K to 2048.
39.8
No Ablation
Model trained with our default settings.
42.1
least, reducing EM to 39.8%. All results reported in Table 5 were obtained by training the model for half amount of training steps save compute.
RELATED WORK
The past work most similar code editing setting. However, it can only predict a restrictive set of code changes expressable as moving, deleting, or copying existing AST nodes and cannot generate novel expressions that not present in the input. It also doesn't make use of modern transformer architecture or pre-training techniques. In contrast, NN al. (2021) takes rule-based upproach, using program synthesis methods to distill similar change patterns in the context and make editing suggestions accordingly.
setting is that of Brody et al. (2020), which also targets contextual
There is also prior work on non-contextual code change prediction settings. In Chakraborty al. (2020), the authors use past code patches to train the model to perform similar edits and evaluate it on future edits in the same codebase. However, since the model does not condition on relevant changes, their technique requires retraining the model for new types of edit patterns. Panthaplackel et al. (2020a) proposes augmenting the decoder with a direct copying mechanism to help a encoder-decoder model perform editing tasks. Zhang et al. (2022) proposes de-noising pre-training scheme, in which they randomly corrupt actual code snippets and train the model to predict the uncorrupted version from the corrupted version. Tufano al. (2021) focuses predicting code review changes using developer discussions. Reid & Neubig 2022) focuses on modeling the iterative editing process of texts and code and proposes a different change encoding scheme that represents edits the word-level based on the Levenshtein algorithm.
Lastly, there is prior work that focuses 2020b) or generating natural language descriptions (Panthaplackel et al., 2022) from code changes. This work, focus on measuring our model's ability to make correct code changes and remove comments and doc-strings before measuring the exact accuracy.
learning update code comments (Panthaplackel et al.,
6 CONCLUSION AND LIMITATIONS
In this paper, we presented Coeditor, a novel approach for repository-level code auto-editing. Building on the CodeT5 architecture, our model incorporates line diff format and static analysis create large customized model contexts. We demonstrated that Coeditor significantly outperforms existing code completion methods in simplified single-round, single-edit task. Furthermore, our model shows substantial performance improvements in more complex multi-round, multi-edit setting.
One limitation of our current work assuming that the user would manually identify the region of code requiring changes and then invoke the model to predict where and how to make these changes within that region. promising direction for future work would be to extend the model help users identify the regions of code that need to be changed within the entire codebase, enabling new types usage such as auto-refactoring. Furthermore, as mentioned in inssubsection 3.1. our model is designed to not modify any lines in the input that have already been modified. While this simplifies the evaluation and prevents undesirable model behaviors such as disregarding changes already made by the user getting stuck in infinite editing loop, it also limit the practical usage of our tool. Therefore, would valuable to explore methods that allow the model to collaborate with users in finer-grained editing units in future research.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was conducted in a research group supported by NSF awards CCF-1762299, CCF-1918889 CNS-1908304 CCF-1901376, CNS-2120696, CCF- 2210831, and CCF-2319471. We would like to give special thanks to Miltos Allamanis and Ray Mooney for their invaluable advice on this project. Additionally, we appreciate the constructive comments from the anonymous reviewers, which greatly improved our work.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. CM3: A Causal Masked Multimodal Model of the Internet. arXiv eprint arxiv:2201.07522 2022.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2668, 2021.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't t reach for the stars! arXiv preprint arXiv:2301 0039880 2023.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long- Snng-Document Transformer. arXiv preprint arXiv:2004.05150, 2020.
Shaked Brody, Uri Alon, and Eran Yahav. A structural model for contextual code changes. Proceed- ings of the ACM on Programming Languages, 4(00PSLA):1-28, November 2020. ISSN 2475- 1421. doi: 10. 145/342828 URL htttpp///dll.eecm.orro////.....1
acm wwt. wttp://.rrg//oo/1.111111... 1115://///.1111113122999
Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. CODIT: Code Editing with Sree-Based Neural Models. IEEE Transactions on Software Engineering, pp. 1- 1, 2020. ISSN 0098-5589, 1939-35200 2326-3881. doi: 10.1109/SSE.200032220022 URL arXiv: 1810.00314.
http: /ttp::///axiv..o. www.orgg///s... / 181118118 10.8000000 00314
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint 1.Xiv:2007.03374 2021.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng. Ming Gong, Linjum Shou, Bing Qin, Ting Liu, Daxin Jiang, al. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
1536-1547, 2020.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.0544 2022.
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Mare Brockschmidr, and Miltiadis Allamanis. Learning to complete code with sketches. In International Conference on Learning Representations, 2021.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/20211 eacl-main.744 URL https://accaattooooyyccooo wwwp:/lloyyory//2222/2 www.:/2i...lmmm....m....
Kevin Jesse, Premkumar Devanbu, and Anand Ashok Sawant. Learning predict user-defined types.
IEEE Transactions on Software Engineering, 2022.
Marie-Anne Lachaux, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. ArXiv, abs/2006.03511, 2020.
10

---

Published as a conference paper at ICLR 2024
Luigi Lavazza, Abedallah Zaid Abualkishik, Geng Liu, and Sandro Morasca. An empirical evaluation of the "cognitive complexity" measure as a predictor of code understandabiilty. Journal Systems and Software, 197:11156. 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint (xXiv:3305006660 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-leve code generation with alphacode. arXiv preprint arXiv:2203.072 140 2022.
Nhan Nguyen and Sarah Nadi. An empirical evaluation of GitHub copilot's 's code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories, pp. 1-5, 2022.
Ansong Ni, Daniel Ramos, Aidan Z. H. Yang, Ines Lynce, Vasco Manquinho, Ruben Martins, and Claire Le Goues. SOAR: A Synthesis Approach for Data Science API Refactoring. In 202/ IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 112-124, Madrid, ES, May 2021. IEEE, ISBN 978- 1-66----65--- doi: 10.1 10.1100//CSEE43302222202 20200000.. .00023. URL
https://ieeexploree weeee.eeee..... wttp:///cccmmme////0000.66
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis, arXiv e-prints, pp. arXiv-2203, 2022.
Sheena Panthaplackel, Miltiadis Allamanis, and Marc Brockschmidt. Copy that! Editing Sequences by Copying Spans, December 2020a. URL httt:://araav.oorg/bb///// /tt.://///000077270 04771
arXiv:2006.04771 [cs, stat].
Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. Learning to update natural language comments based code changes. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1853-1868, 2020b.
Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Ray Mooney, Learning to describe solutions for bug reports based on developer discussions. Findings of the Association for Computational Linguistics: ACL 2022, 2022.
Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. Typewriter: Neural type prediction with search-based validation. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 209-220, 2020.
Machel Reid and Graham Neubig, Learning to Model Editing Processes, May 2022. URL http: //arxiv.org/aat wwapg//ab///aae wttp:orggaabb/...222222 12374 arXiv:2205.1 12374 [cs].
Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Vicente Franco, and Miltiadis Allamanis. Fast and memory-efficiett neural code completion. In 202/ IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp. 329-340. IEEE, 2021.
Marc Szafraniec, Baptiste Rozière, Hugh Leather Francois Charton, Patrick Labatut, and Gabriel Synnaeve. Code translation with compiler representations. ArXiv, abs/2207 003578, 2022.
Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards automating code review activities. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 163-174. IEEE, 2021.
Yue Wang. Weishi Wang, Shafiq Joty, and Steven CH Hoi. CodeT5; aeentifier-wware Unified Pre- trained Encoder-Decodd Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, 2021.
Jay Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic Type Inference using Graph Neural Networks. In International Conference on Learning Representations, 2020. URL

---

Published as a conference paper at ICLR 2024
Jiayi Wei, Greg Durrett, and Isil Dillig. TypeT3: Seq2seq Type Inference using Static Analysis. In International Conference on Learning Representations, 2023. URL /
net/ vet/Eoos 4 TyNEh12GaN
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: Transformers for Longer Sequences. Advances in Neural Information Processing Systems, 2020.
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, MlanGuang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303. /2570, 2023.
Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. CoditT5: Pretraining for Source Code and Natural Language Editing, September 2022. URL http: 05446 arXiv:2208.0546 [cs].
//arxiv.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 KEYSTROKE DISTANCE
We developed string distance metric incorporating the cost of cursor movement, approximating the number keystrokes needed to transform an imput string into an output string.
Given the initial state with
1en (output), cursor _isor_iis and delet ing = False, the cost is calculated using dynamic programming with the optimal combination of the following operations:
Len (input).
M: Match character (cost=0), requires input [-i] -- output [-j] and not deleting, results in
1, j and cursor_
D: Delete input character (cost=1), requires cursor_dis _is =- and not deleting re- sults in i 1.
A: Add output character (cost=l), requires cursor_ = 0 and not deleting, results
in j
C: Move cursor to current position (cost=min (cursor dis, petsso__jumpcce re- quires conditions, results in cursor_dis
S: Begin deletion (cost=1), requires cursor_dis dis == 0 and not deleting, results in deleting True.
K: Continue deletion (cost=0), requires deleting, results i -= 1.
E: End deletion (cost=l), requires cursor dis and delet ing, results in deleting
False.
Where cursor cursor__ump__pe htt__juup__sss____ cost_ is constant that to 4 when reporting results. The worst-case complexity of this algorithm is O(1en (input) x len (output) x hursor_jump_coss This model does not consider copying or pasting operations.
Note that in some cases, the total gain can be negative when measured with Levenshtein or Keystroke distance. For example, the Levenshtein distance of modifying sentence is lower than the total of first deleting the sentence and then adding a new one.
A.2 DISCUSSION OF SPARSE ATTENTION MECHANISMS
The block-sparse attention pattern we described subsection 3.3 follows past work on retrieve-and- read models for natural language question answering. Specifically, it resembles (Izaaard & Grave 202I) with three changes. First, we have no notion encoded with each retrieved snippet. Second, our target code block u is given special status in the encoder and can globally attend to each retrieved snippet. Third, we modify the relative positional encoding make each query "infinitely" far from the reference.
question that is jointly
Our approach also resembles Longformer (Beltagy et al.) 2020) or BigBird (Zaheer et al., 2020), most notably in how query block's 's cross-attention with the references global attention. However, our segments do not come from a coherent context, so our local attention component is block-diagona, sparse matrix rather than a sliding window in those methods. Our modification to relative position encoding also makes our model invariant to the ordering of references, helping generalize different editing orders at inference time.
be viewed instance of
A.3 CODE COMPLETION EXAMPLES
To help the reader why including contextual changes can be beneficial for (editing-related) code completion problems, we compare Coeditor and InCoder6B's 's outputs on 3 example problems from test set in the next few pages (Figure 5) Figure 10). These examples sampled from subset that are small enough to be presentable within one or two pages and in which Coeditor outperforms InCoder6B.
13

---

Published as a conference paper at ICLR 2024
A.4 ROUNI-ROUND EDITING EXAMPLES
We show multi-round editing examples from our test set in Figure Figige 13 These examples sampled from a subset that small enough to be presentable within two pages and in which Coeditor achieved 50-100 total keystrokes edit gain.
14

---

Published as a conference paper at ICLR 2024
# module: anyio._ core. testing def (et_running_tasks() list [TaskInfff)
Return list of running tasks in the current event loop.
retur:: list of task info objects
return get seapbiiiioeeee_ruumnnntttttsiie <infill here>
Ground Truth:
return get_asyncc (aayycc_baceed(1_1 () .get_rummiing__tsss) {
Coeditor Prediction:
return get__async_bacee backend retttruninnngggttssss///
Incoder Prediction:
return get_asynclib(, Betrtlibbbbttttinnnnneooe ree__runningttssss
Figure 5: Code completion example 1. Coeditor sees from the relevant contextual changes (shown in
Figure 6) that some get_asyncl ib () calls should be replaced with get__ayne ____aas_ssscceeeae backeadd (), so
correctly suggested the change based on the deletion before the infilling point. InCoder was not able to see the deletion and infilled the original code given only the surrounding code.
changee ref
# module: anyio._coree _eggtt def get_current_task() -> TaskInfo:
Return the current task.
return: representation of the current task
return get_async_backend() bww.eeedeeeggeecuulllll
return get get_asynclib() http:ncciiiigeet.uureetttttsssse
changed ref 1*m=aan**m..
module: anyio. from_threas class BlockingPortal:
def new _e___e_w____ (cls) -> BlockingPortal:
return get_async backend create blotaing_portal()
return get_asynclibl (tl_assyccibl..llllcccooggorrro BlockingPortall)
changed ref 2==
* module:
def geet_canceelleeeeeeeeeeeeeeeeeee
type [Bseexceecc:::
ad"Return the current async library cancellation exception class. """
return get_async backend() caaeeennd).cccellllle se___asyc__backedde cancel cancelled_ee ewweptioonclaaseeeeeeeeeeeeeesee
get get_asynclib(). Dee__ayynnibb..ccceeleeeeerrr
Figure 6: Code completion example 1: relevant contexts. The changes highlighted in orange tell Coeditor that some get net_asynelibl () calls should be replaced with get_async_backend ().
15

---

Published as a conference paper at ICLR 2024
class wwwporroorraaeeeeeeeeeeeeeeeeeee date, seggan_ insturt
module:
instructor
def _init (self, opt):
super LupperssegaMInnttruttorr, self) self).__nit___ (ontt_ooot_t
* generator, discriminator self.gen SeqGAN_G(cfg, seen_embed_diie
efg.gen_hidden.ee
vocabbs size, cfg.max_seq_ler
cfg. padding idx, cfg. iig.temperature, gpu=cfg.CUDA)
self. dis SeqGAN_DC D(cfN
seffg.dis, dis enbed dim, vog.v size,
cfg. fig.paddinggiiie idx, gpu=cfg.CUDA)
self. iiiiiiitnmmm. ini__moee(())
* Optimizer
gelf.gen_oon
www.sel...egpeeeeeeeee.eeeeeeeee parameters(), tr=cfg.gen_Lu
self.dis_opt = Adam selff.is. parameters(), ( tr=cfg.dis_Ir)
# Criterion
self.mle criterion = nn. .LLLooss(
self.mle_criteriinn
ceniteis_criterion nn. Crosssfntropyylosss(
# DataLoader
self. crracl_ssmppls torch. forchhtoadccfg.oracle__sa
<infill
here>
self.gen gelf.gen_datta data ptapeaatteerrstt..eeeemalall.l.p
cfg.
Ground Truth:
self.oracle, data Genbatalter(self..orceee.e
Coeditor Prediction: self. data Genbatalterlself..rclle...a.a
Incoder Prediction:
self.oracle Oractelselff.rraclessmpeet
Figure 7: Code completion example Coeditor was able to suggest the correct code based similar change from another file (Figure 8) highlighted in orange), whereas InCoder was not able to the change and suggested wrong statement.
=chneeeeeed ref 19======-:
module: inst censtructorrorace data. re vellanninnstruttee
class Re (Bwp:inssstuttorsiasnnnnntteeeee
def (self opt):
super(/RelGAIInstruucorr self) (elf)._inil (Llf)._iitt___ (opt)
generator, discriminator
self = RelGAN selccg..emmssotfte cfg.num_heads.
cfg. hea.heaadd_e,,, cfg.gen_embed_ding cfg.gen_hidden_de
cfg.vocab_size,, cfg.max_seq_
cfg. pappppdddpiaaee idx, gpu=cfg.CUDA) Setefggddd_eeee________ee__ embed_dim, cfg.max_seq__en,
self.dis RelGAN_D(Ccf.
ffg.nuu_rep, vocab_ size, fffg.padiing... _dx,
gpu=cfg.CUDA:
self. self.init_nodel()
Optimizer
1r=cfg.gen__r self. siff.gen__dv_opp optim. aedamsself.fe wwt....g....aaammeerss..eee.eee. lr lr=ofg.gen_adv_luv
get_otee_optt wpw....daasssff.......rrrarrrre
self.dis_opt aptiim.dammsell dis, parameters(), gracfg.dis_Ins
# Criterion
self.mle criterion self.mle.criterion nn.NLLLoss()
# DataLoader
Lulfforrale_saappae torch. load load(cfg. ora.teespeeeeesateeeesssss
self. oracle_daar data = GenDatalter praalee_saaee les)
self. gelf.genddata GenDatakter(self.gen
batch_size, cfg.batch_size))
Figure 8: Code completion example 2: relevant contexts.
16

---

Published as a conference paper at ICLR 2024
def get_config(dataset: str,
num_states: [ptional[int] None, shots: Optional(int]] 8, with_bow: [ptional[bool] True, Optional[str] = model_config.Gut GLLVE_ EMBED, Lodee__coniggGLOVV
secoder_emmbedding_ty. Optional(str] getrre_embbddiin:: [ptional[bool] False, bert port_embedding.. typet Optional(stri 'base',
bert bert_dir: Optional ''] confag_dictt.onfigiino
"""Returns the configuration for this experiment. []]
config config_dicae diggddict.co ConggiCc))))) config_dir [it_config_dirr(attttt
config.max_pet taac__as__faaaa failures tarrigg.ax_ttssffalturss 10
flatforr 'jf' config,
config.tpuutopollee tpu_topoloogy = '2x2'
config. seed
config.dataset dataset
config.dataset_dir dtta_uttlsggge.tttsstttdddrdssd
<infill here>
config.train_eepon train_epochs = 10
config. ttrai_bbatchhssttee
config.eval. baw.baatttiiiee Batch size for inference. Predicting in batches in case of 00M. config.infeemeee sireesnce__btcc_siieeeee 300
16
Seed used to generate datasets for inference.
config. inference seed
Directory storing the saved model and model prediction outputs.
model modell_odee__aaree Laseee_bbbee_ddee dir None
Directory checkpoint to initalize the model from. The initialize priority will be:
-init_checkpoint
-latest checkpoint in init_dir
-latest checkpoint in output_dir
config. inifiig.niittet iwitinnnncctttee init_checkpoint
None
config. hwaffig.initttee init_dir None [-]
Ground Truth: config.dome dig.ddmmiin in = = False
Coeditor Prediction:
config. doga.domain...mmmmmmmmmmm addaptation domaiin_ddpptation = False
Incoder Prediction:
config. train_split 'train'
Figure 9: Code completion example Coeditor was able suggest adding the correct attribute initialization based on the new usage highlighted in in/iiguee 10 whereas InCoder was not able to see the new usages and hallucinated new attribute.
17

---

Published as a conference paper at ICLR 2024
changgee ref 15
module: experimental, language, structure. vrnn, train
def run experiment (config: config_dict dict. Configitn,, output_dir: str):
offset:

model dir, 'labeled baaalog_tur_iids.ooo f.write( 'nn .join(
'w")
str(id) for in labeled dia laallllguurrn cer__tur__idde htw..idd..mmyy)...ll....
lse:
Labeled__iallogttun_..do None
if config.domain iammainnadaptation:
inputs preprocessor.get -et_tuuettllllllttttettl pee__ful__d____ f pute (train_dataset. lddirrr))
# Notice domain label id 0 is also treated as in-domain: ood should have different from it. in in_domains, ttf.uniquett)) reshaee.e L
[-1]))
metric carrric_nmmpppcesseeee _eetrric_ammsppeeeeee TRAIN),
i TEST, True), inetric_namespacee (_TS,, False)
fewshot fewshot_netric, sateric_aameppcesss
__eeeiic_aamspppeeeeee (FEESSOTTT NAMESPACE, True), smmetric_namesacee__Feetti NAMESPACE,
MEASEEEEEE,,,, False)
else:
in on_domains = None
metric_ netrric_naesppces Lmetric namespacc(spliit) for split in SPLITS]
Newshot_netric_nnmeeppee
[_antricceameppee_____sssooooos_ ]
pata_preprocessor __buulddddeee
labeled_diallg__________________ in domains)
ctrpdtttt__ooccessorconnggg preprocess_fn = oet.pprerccesso..ccneeeeettttrrn
data preproccssoorr buulu
labeled dialog_turn__id
* Load PSL configs ps learning =
coonstraint_eerringg.wwe learning
ois_enneeence
if psl_learning or psi_inference:
with tf, www.ioofgle.FFliii
monfig.model.. vaedel..vee_ee waw.caaeelleeeeeeeeeeeeeee
aaae_ceel.deeoderr wellLddeoder__emeeddnnggg vocab waa.idiing.ooab__iieeetttttttt 'r') f:
vocab = read) [:-1] (=:)sssst)ppt)))
preprocess_fn pes_utiss.ss__fettiee fet.uure_iiiinpppprrressss fn, config.dttaset.
config, psl, vocab)
Load datasets
# TyQuan))an): invesigate why distributed training fails *fish TPU Failure example: https://xm2a.c coretta.ccrpoooooo. httpp//ppppppllcccc////eeeeeetta distributed, training
tistainbued__taaninng = False
train_dataset preprocessor.crest org.
Figure 10: Code completion example 3: relevant contexts.
18

---

Published as a conference paper at ICLR 2024
project: tooglee/umerrtaitty-baseliinn
commit: '3e74384539 Adding support for tb.dev h..'
path: uncertainty_ eeplinnss.eperimeeee setnrnsseeeeeennsii..eeeel.. wtt.:ic.evvarruevvaaa.eooo ('n_references":
Beaarged_efeeeece____e _ 3247, 'unchanged_ Beapnnndd_eefeeecee_e_eeeee tis': 37)
editing round:
Grounnd Truth
<1>;
va _oo__ptuu_spppppp = None
hparams:
aop.hparamshhparare
<13>:
if hparams:
app.hpaamms (hparans))
<15>;
return väl_outputs_np, {k: v.numpy() for k, in test__outputs.itmmss
casssensain Code
def run_eval run_eval_epoch!
val_fn: EvalStepFn,
val val_dataset: Dataast,
test_fn: EvalStepFn,
test_dataset; ta... Dataset, Datset
test test_sumarry_wiiieg Sf.sumarr.Sumarryrrtte.
current_step: int,
current_step: int): hparams: Optional[Dict(sti Anyll):
"""Run one evaluation epoch on test and optionally validation splits." ... if val__aallae ___aaaattt
<1>
val
ster(val_datasets
<3>
val outputs = val val_fn(vai_iteratto
with val__umne wwwssuuununiiiieee wtw.:iterrrsdefautttl.. ())ddffaul())
<5>
for name, netric in val_outputs.itenst tum.aryn scalsslaaaannn,, metric, step=current_sten
<6>
val val_ outpuutsssssp... np (k: v.numpy() for k, in val_outputs.items
<8>
logging, info
'Validation metrics for step As', current_step, val_outputs_ng)
<10>
test_iteraten
(test_datasetl
rest_outputs- writer.as [s__effaltt()
fn(teet_tt_tt poftest_iteratorr commary_writerr.. cast_outputs.tee itens));
<12>
with test test_summary.
<13>
for name, netric
<14>
tf..sumary. scalar(nae,, metric, stepacurrent_step)
<15>
Predicted Changes Changes=meat
<5>;
hparams:
hp-hparane (hparams)
hparams:
hp. hparams (hparans)
anmmeerceeeped gains
keystrokes gain:
diff-lines gain: 4
levenshtein gain: 88
Figure 11: Multi-round editing example 1. Coeditor correctly suggested a subset of the ground-truth changes. Contextual changes omitted for this example.
19

---

Published as a conference paper at ICLR 2024
project: fnccbokkresarrchpytttrciiiii
commit: 'b0ae784244 Get audio samples only from..
path: pytorchvideo.co data. videon videv from prom_patthhhhhppppp
('n_rreference's changed_rreference tks' renchange__reference.. t 167)
editing round:
Ground Truth
<0>:
decode_ decode_videe=deccon decode audio-decode, audio, decoder=decoder,
return EncodedVideo. fron fronparcotrrropttttfeeeeettttt Recode_audio, decoder)
Main Code=maenme:
# module: pytorchvideo,datann class VideoRathhlandler(abjeee. def sideo_from_paal ( self, filepath,
decode_ auddee_adi--,,,,, decoder="pyav", fps=30
):
def video_from_path(sete filepath, decode_ audioeaadd decoder="pyav", fps=30);
try:
is file ieathnngr.sslllfiiiiee is
except NotInppementeddErog
Not all PathManager handlers support is(file,dir) functions, the case, default assuming the path file.
is is_file = True
is_dir False
if is file: from saroorchuddeo.ddta.enccooo encoded_vvidee import EncodedVideo
return frooeed
filepath,
return EncodedVideo. wtop:iiieeeeroooaatttaaeeeeeeeee decode_audio, decoder)
<1>
elif is is_d from pytorchwideo..ataa.raae. video import FrameVided
<3>
<4>
decode_audiia "decode_audiz be False when using FrameVideo"
<5>
return FrameVideo. from directory( filepath, fps, wewerrrraccheseeeeeeeeeeee httpeesssll.ppah_ooooeeeeee satpseeef.patttoorerccaaaaaa order_cachee
<6>
<7>
<
else: raise FiteeecFFuundErrrr((""*ftetttt found.")
<10>
aasemane Predicted Changes
<0>;
decode_audio,
decoder,
fps,
Lideo_frame_tor hoo_path_fnssette wwwaerrrttoooree to_path_fn,
return EncodedVideo. fro.aaeooorrrooatthfleeeettttt decode_audio, decoder)
Figure 12: Multi-round editing example (round 3). Coeditor misunderstood the user's intention and suggested adding two more arguments to the EncodedVis deo fwwm_roootttteeeeeeeeeerereeeeee function call. Under our multi-round evaluation strategy, we assume the user would then manually add the next line from ground truth changes (see the next figure).
20

---

Published as a conference paper at ICLR 2024
editing 4
Ground Truth decode__udii=deecdd__eoo
<
decoder=decoder,
return fromedd filepath, decode_audio, decoder)
Main Code
module: pytorchvideo. video class VideoPathhandlerrlbbjeeee
def video_ siddo_frro_ppatie fron_path( self, filepath, decode_video=True, decode_audio=Falle, decoder="pyav", fps=30
def @adeo_fron__aree frathsrrrn_atheelf, filepath, decode_ aaddde__ddoofasse, decoder="pyav", fps=30):
try:
is_file 9_pathmgr. isfile(filepath)
is_dir iirpthingr.sddrr(fllepllll except NotImplenentedie
# Not all PathManager handlers support is(file,dir) functions, when this the # case, we default to assuming the path is file.
is_dir = False
if is_ file: from
import EncodedVideo
return frcode
filepath,
return EncodedWideo. from_ppahh footeth filepath, decode_audio, decoder)
<>>
elif is_dir:
from pytorchwidee. frameetttaaraeeeeeeeeeeeeeee import FraneVideo
<>
assert decode_audi "decode_aute" aadiio must be False when using FrameVideo"
<5>
return FrameVideo. fww.:rroomiiiii fton:iiideooroodddrettttrrro filepath, tps, aeth_orde__achesseett.o .atessseet..tttaoreeeccccctee
<6>
<>>
else:
raise (* not found.")
<10>
a*ccased Predicted Changes
<0>;
accode_oudiondecooe___ddd decoder=decoder,
return frnoeddiideo.rromppat((fffft decode_audio, decoder)
acceseedccepted gains
keystrokes gain:
diff-lines gain; 4
levenshtein gain:
Figure 13: Multi-round editing example (round 4). With the next line change from the ground truth added, Coeditor understood that the user intended only change the calling style and thus able to suggest the correct change.
21

---

Published as a conference paper at ICLR 2024
project: clandao-tooay-<qeenddo commit: 18def0c64df (ursor & 1
path: db. weo.bbedd/BBeeeBB... Bisse
('n_rreference's
tthanged_reference tks'; 957, uinchanged_referenece tks'; 168)
editing round:
aoommnnnGround Truth
<13>:
lastrowid dbcur. lowid
dbcur.closed
return lastrowid
return dbcur. lastrowid
Main Code
# module: db. basedb
class BaseDB(object):
def insert(self, tablename-None, **values);
<0>
tablename self.escaae(tablenam
taff__tableenae
<1>
if values:
<2>
keys " ".oonnn (sl(((self.escapee esccapekk) for in values. values.keys())
<3>
vaauuss ".join([seii seelf.placchhlder
len(values))
<4>
sql_query "INSERT INTO &ss (45) VALUES (4s)" % (tablename, _eys, _values)
else:
<6>
"INSERT INTO 8s DEFAULT VALUES" % tablename dogger.debug("<sape "s>", sq sql_qeery)
<
if values:
<10>
abcur self self=_execute(sqlL_uvr wastvva
<11>
else:
<12>
dbcur self._execce executtessq__quee
<13>
return dbcur. lastrowid
<14
Predicted Changes
<13>:
lastrowid dbcur. lastrowid abcur.closef. return lastrowid return dbcur. lastrowid
ammmmndcceepted gains monnnn
keystrokes:
diff-lines: 4
levenshtein:
Figure 14: Multi-round editing example 3. Coeditor was able to predict the correct change in the first editing round by identifying similar change inside different function (see Figure 15 highlighted in orange).
22

---

Published as a conference paper at ICLR 2024
signnttuse at: db.basedb logger logging.getLoggen "qiandao. basedb ')
at: bbb.basedb.Bat BaseDB
paaceoo "ts" nysql
executelsql_quer query, values=(]) executeeself, qqe_y values=())
db.basedb. BaseDB..
table tablename (et or self. ttab_e_aa__ee_____ [More sinagures ...]
channchedee ref
module: db. basedb
class BaseDB(ebject); placeholder = '?' sqlite3
init (self, host=comfig.mmssl.hott. port=config. nyssep.p.....ss.p..... dataabas==config.
contig.nysql.pasee passwd, wtw.confii.nnglllaataaaassaeeeea
database, user=config. betr:config.nysq... uysqll.serss,,,r passwd=
import mysql. connector
self. conn wwysgll.connetton database-database, auth_p getthplugimauuth_plugg. autocommit=-ruue)
cetettr.ccnnnctuseeeeeeeeee (userctus,,,,ueer, passwordepasswd, hast=host, port=port,
aan.en aemesesssncangeed ref module: db. basedb
class BaseDB(
def replaceeselff tablename=None, **values::
self. eaa..eeettttaaeeeanee
self._tablennane
1f values:
_eess ".joinsseff wtc.lseel..scca(kk for k in values.keys())
_values ".join( pself-placeholder
* (en(values))
sql_uery "REPLACE INTO &s (%s) VALUES (#s)" (tablename, keys, _values)
esse:
sql_query "REPLACE INTO 4s DEFAULT VALUES" & tab tablename logger. deoger. ss>", sql_query)
if va values:
dbcur self. execute(sql_query) (ist(values.values())
else:
dbcur self execuue(sqq__uerr)
lastrowid dbcur. tastrowid
dbcur.close()
lastrowid
return dbcur. Lastrowid
[More changed references..]
Multi-round editing example 3 (reference blocks). The bottom changes highlighted in orange are similar the changes needed in Figure 14]
Figure 15:
23


Reviews:
[{"review_id": "umgh7ugh8i", "reviewer": "Reviewer_ueao", "summary": "The paper presents Coeditor, a fine-tuned language model that is designed for code editing tasks. The backbone of the Coeditor is the CodeT5 model, which the authors have finetuned on long context (2k->4k->8k) using block-sparse attention and capitalizing the relative positional encoding scheme of CodeT5 model. Coeditor is built based on two key ideas: (1) encode prior code edits using a line-based diff scheme and decode the edits using the masked span infilling objective; and (2) using lightweight static analysis to pull in relevant parts of the codebase (e.g., function signature). The paper proposed a dataset called PyCommits which is collected from 1650 open-source Python projects on Github. The paper compared Coeditor with code infilling models - InCoder, Starcoder, text-davinci-003 and showed that Coeditor outperforms them by a large margin. Codeditor is released with code, dataset, model checkpoint, and a VSCode extension.", "questions": "- Why Coeditor is trained with a fixed batch size of 1? The Nvidia GPU used to train the model has 48GB memory which should be good to accommodate batch size > 1 with a 220M param model, right?\n- Does the PyCommits dataset composed of Github projects that have permissive licenses?\n- How the generic infill models are prompted to solve the code editing tasks? Is few-shot prompting used for the models? For example, [1] used demonstration augmented prompting for Codex model (when finetuning is not possible).\n- Can we use instruction finetuned version of the code generation models for code editing task?\n- Instead of a seq2seq model, can we use a decoder-only LM for the editing task?\n- Is it possible to evaluate Coeditor on the PIE benchmark [1]?\n\n[1] Learning Performance Improving Code Edits (https://pie4perf.com)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A good dataset (PyCommits) that will foster future research.\n- The proposed method to train sequence-to-sequence language models for code editing is sound. Overall, the writing is good (though there are minor grammar issues and repeated words) and the paper is easy to read and follow.\n- An IDE extension that researchers will be able to use and understand the effectiveness of the approach.", "weaknesses": "The primary issue of the paper is over claim when it compares with SOTA code infilling models. Statements such as \"our method achieves 60.4% exact match accuracy using a 220M parameter model, whereas text-davinci-003—the best performing code infilling model we have compared, which has 175 billion parameters—achieves only 30.5%.\" is extremely misleading. Coeditor is a finetuned model that is made by specializing on the task at hand. On the other hand, the infill models are general purpose models. The paper didn't explain clearly how the generic infill models are prompted to solve the code editing tasks. Therefore, there is no way we can fairly compare the proposed approach with the baseline models compared in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUhc5zELWu", "reviewer": "Reviewer_Kf1h", "summary": "This paper introduces Coeditor, a novel model for auto-editing code based on the CodeT5 transformer model architecture. It is designed to predict edits to existing code, based on the related recent changes in the same codebase. The tool requires users to manually select the code region for editing. The paper introduces a new PYCOMMITS dataset, sourced from 1650 open-source Python projects, which is used for evaluation. The model outperforms existing tools in both single-edit and multi-round editing scenarios and includes a VSCode extension for practical use. The paper highlights the importance of integrating editing history in code completion and provides resources for future research in this domain.", "questions": "It would strengthen the paper if you could include any specialized code editing models or tools to the evaluation, which could provide a more comprehensive comparative analysis. \n\nI would be curious to see how CodePlan + Coeditor perform on the PyCommits dataset (especially since the CodePlan paper partially features the results using the Coeditor model), although I realized that CodePlan was published after the ICLR submission deadline. Is it possible to add?\n\nAnother interesting baseline could be BluePencil model focusing on repetitive edits (see \"On the fly synthesis of edit suggestions\" by Miltner et. al) this is also available through IntelliCode Suggestions VS extension.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Introduce and open source the new research PyCommits dataset, source code, and VSCode extension. Which is an important contribution to future research in this domain.\n* A step towards solving an important practical problem, and a potential to integrate in real-world developer tools", "weaknesses": "* One notable weakness of the Coeditor model is its reliance on users to manually pinpoint the regions in the code that need editing (as pointed by the authors, too). This approach limits the model's potential for broader applications, such as automated refactoring, and places additional steps in the workflow that could be automated for enhanced efficiency and user experience.\n\n* The evaluation section could be improved by incorporating a more diverse range of baselines. Currently, it predominantly features large language models (LLMs) tailored for coding tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eDY0kUGgyb", "reviewer": "Reviewer_sVPE", "summary": "The authors propose to shift the focus from learning code completion to learning edits instead and leveraging repository wide line-diff information to do so. They build on top of CodeT5 which they augment with sparse block attention to enable larger context sizes needed for the new task to produce Coeditor. Further, they create a summary \"scratchpath\" that summarises useful, out-of-file information, the summarisation being crutial to avoid large prompts. They derive an edit history from the git histories of Python projects and use the current edited files as context at prediction time. Further, they integrate into VSCode as an extension to demonstrate how such a tool can work in a developer workflow. Crucially, the novel setting is the human-in-the-loop, multi-round editing, where a human may provide feedback between rounds. Evaluating using Exact Match, they demonstrate better performance both in single- and multi-round editing settings, demonstrating the befit of shifting to modeling edits directly, thr ablation study further demonstrates the value of diffs.", "questions": "With the move towards an extension, was using more granular edit events considered, what about navigation events or other similar side-channel information? Is the main concern the context size or are there other trade-offs that moved the authors closer to git diff, line-level information?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- A perspective shift to modelling code edits.\n- A smaller scale model that outperforms larger models on the code auto-completion task.\n- VSCode Extension that can be used directly in a project.\n- A historical edits dataset for Python", "weaknesses": "- The edit granularity choice and dataset can introduce bias: developers often commit many intermediate commits that get squashed into a single commit before pushing and changes can get overwritten.\n- While the delta/diff encoding is justified, the space feels under-explored.\n\nAfter reading the separate discussion threads, I share the concern of the reviewer vqdF regarding the use of signatures and their impact on performance. I am unconvinced that the information cannot be exposed to baselines, for example, by borrowing methodology from Ahmed et al. [^1].\n\n[^1]: Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. T. (2023). Improving Few-Shot Prompts with Relevant Static Analysis Products. arXiv preprint arXiv:2304.06815. https://arxiv.org/pdf/2304.06815.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mzNY7qPvy", "reviewer": "Reviewer_vqdF", "summary": "Language models effectively model text generation in a single context, but programs are often written by altering code\nacross many files and locations. This work proposes to fine-tune a language model with a dataset of code \"diffs\", consisting of a series of edits that together comprise a meaningful change. Specifically, it introduces a block-sparse attention mechanism that allows the model to attend to many related changes without at a fairly modest cost and introduces \"signatures\" of used variables and functions. The resulting model offers strong results at single-line and multi-line prediction given previous edits compared to language models that are unable to leverage such information.", "questions": "Based on the issues noted above:\n- Were the baseline LMs evaluated with a 1K context, same as Coeditor, or with their maximum number of tokens (when possible)?\n- Was signature information only used in Coeditor? If so, is this intended to be a core part of the contribution? Please provide ablations with other models using this information, and with Coeditor fully trained without this information in any case.\n- How do the baseline models perform in the multi-round setting?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Utilizing recent edits in a language model trained on code is a natural choice and this paper offers a reasonable implementation of this idea. Compared to other recent work with the same general goal, it is particularly strong in its definition of reference block and the corresponding sparse attention pattern. This allows it to scale to fairly long contexts, which yields a modest boost in performance (Tab. 5).\n\nWhile the results come with some concerns (see below), the \"replace\" performance in particular is quite promising. The work is also quite well written. Overall, this paper presents a promising exploration of its domain.", "weaknesses": "The evaluation raises some concerns, in particular the comparison with the baseline models. This work introduces a few components; the key one is the notion of reference blocks that CodeT5 can attend to when responding to a query. The baseline models do not have access to these reference blocks since they are trained on regular token streams. This creates a few issues, some of which can be resolved with clarifications in the writing and others that might require further experiments. In no particular order:\n\n- There can be many reference blocks, spanning up to 16.4K tokens, so the work uses a relatively short \"Query\" of up to 1,024 tokens. Other LMs, such as InCoder and StarCoder, all have much larger context sizes. It is not clear from the writing whether the other models were \"allowed\" to use their full context window, or were provided with the same limited section of context. If a completion took place halfway through an 8K token file, one would expect StarCoder, for instance, to do substantially better with the full file context than with a 1K token window.\n- A seemingly unrelated form of \"reference\" comes from the \"signatures\" (section 3.2). This idea does not have anything to do with code editing itself, but rather with providing an LM access to the project-level context. This seems to be very impactful (Tab. 5). It looks like the baseline LMs were not provided with this information, which seems like an oversight. For one, there are reasonable ways to prompt pretrained language models with such information even if that wasn't present at pretraining time. For another, the performance of this component has little, if anything, to do with the stated goal of the work (leveraging diffs for edit prediction). The fact that it boosts performance so strongly makes the results much less interesting; it suggests that a large component of the performance does not come from the presence of diffs, but from a more banal form of information that related work has already explored. The paper should either include carefully calibrated experiments with baselines using the same information or exclude this from its main contributions in results such as Tab. 3.\n- Relatedly, it is not clear why there is such a large performance gap between Tab. 5 and Tab 3. The former uses the validation set, so one might expect that the latter is based on test data. The text in 4.3 does mention that the model was only halfway trained here. But the gap between 42.1% and 60.4% is really very large. Is this not the same type of task as in Tab. 3? And how should we expect the gaps between ablations to scale to the numbers of Tab. 3? In particular, would the impact of the retrieval blocks (criticized above) be proportional or even disproportionally larger?\n- The lack of a comparison between models in Sec. 4.2/Tab. 4 is also somewhat surprising and concerning. Please endeavour to replicate these numbers with at least one strong baseline, e.g. StarCoder.\n\nOn a minor note, it was quite surprising to see it mentioned that the model was trained with a batch size of 1 (P7). Was gradient accumulation considered for increasing the batch size? That might improve performance.\n\nMinor issues:\n- P5: \"making the model is\" -> remove \"is\"\n- P6: \"statictics\" -> \"statistics\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)