Large Language Models of Code Fail at Completing Code with Potential Bugs
Authors: Tuan Dinh
Jinman Zhao
Samson Tan
Renato Negrinho
Leonard Lausen
Sheng Zha
George Karypis
Venue: NeurIPS 2023 poster (2023)
Keywords: language model of code; code completion; language model; software engineering; machine learning for code
TL;DR: Many high-performing large language models of code fail drastically at completing functional code when potential bugs exist in the code context — a practical and realistic coding scenario.

Abstract: Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs – anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a large gap in post-mitigation performance.

Large Language Models of Code Fail at Completing Code with Potential Bugs
Tuan Dinh Jinman Zhao?* Samson Tan?
Renato Negrinho
Leonard Lausen
Sheng Zha George Karypis"
University of Wisconsin-Madison AAmazon Web Services
tuan diaanddinhwisc.ee
{j iinazz, samson, samson, lausen, t (ckarypissee
Camaaam.aaa.....a..aa.aaaaaaa..a eamryoisaeamzzon. com
Abstract
Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence bugs the code context generation, which are inevitable software devel- opment. Therefore, introduce and study the buggy-code completion problem, inspired by the realistic scenario real-time code suggestion where the code con- text contains potential bugs anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with syn- thetic bugs derived from semantics-altering operator changes (buggy-Humante and one with realistic bugs derived from user submissions to coding problems (buggy-FixEvall We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LL For instance, the passing rates of CODEGEN-2B- -B-MOOO more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains significant gap in post-mitigation performance
test cases of Huggy-HumanEval drop
Introduction
Suggesting code for a given context is frequently used feature in modern integrated development environments (IDEs) [.. bringing productivity gains the code-writing process. This task is widely studied as code completion [2 in the literature, with techniques and models ranging from incorporating code structure as prior knowledge [6] 77. to adopting deep neural networks 8 and pre-training techniques [9] to learn representations for code. Recently, large Transformer- baased language models of code (Code-LLMs) I0]
probabilistic or sequence modeling
[2] have become
promising paradigm for code completion, attaining state-of the-ttr-rt (SotA) performance in various code learning tasks including code completion and generation.
However, existing works studying Code-LLMs often assume the absence of bugs, despite the frequent occurrences and the cost of bugs in software development: average, 70 bugs created per 1000 code lines [3]: and fixing bugs costs 50% of development time [14]. Consider practical coding scenario. A developer wants use the code suggestion feature in an IDE when writing code. With high probability, their real-time code context, as an input to the code predictor, contains typos or less refined, potentially buggy implementations. Since bugginess is property complete program,
"Equal contribution.
Work done while interning Amazon Web Services.
Code and datasets are available at http://:////.ttt..oo.o.//a///a/e wwwpa//oooncciiccnenneeeeee scw.nccebbugyycoooeoooooo www.:oddwwwooo...i....oooooo
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

*.. You're given a List deposit withrrawal operations starts with zero balance. Your task is to detect if any point the balance of account fallls below zero, and at that point function should return True. Otherwise it should return False."mm nue
account that
Problem statement
fron typing trport List def :o List(int)) bool:
fron typing import List below, (ello__err(eeeraation::: List(1nt]) book: balance for in operations: balance op if balance 8:
balance = for op in operations:
Partial code
balance if balance
return True
return False
raturn False
balance
Completion
return True
return False
Without potential bugs
With potential bugs
Figure 1: Illustrations for code completion (left) and buggy-code completion (right). (top) prob- lem statement for function implementation, (middle) partial code with (right) or without (left) potential bugs (highlighted). (bottom) code completions from CODEGEN- ME-G-EN----OONN BOMOO [1]. The completed code is functionally correct the left but incorrect the right, failing test case below_zerol [1, 2]) ==
False. The example is based on HumanEval/3 from
is not well-defined how to detect bugs and repair the buggy code in this short and incomplete code context, making the application of existing bug-detection or code-repair tools sub-optimal or infeasible. It is also worth mentioning that the gap between the vitro and in vivo performances of code completion models 11 remains large. Therefore, a natural question arises: Can existing Code-LLMs provide good code suggestions given the unrefined nature of draft code?
To answer this question, we introduce and study the problem of completing code with potential bugs in the code context, dubbed buggy-code completion (bCC). In this study, we focus on using Code-LLMs to generate functional implementations from a code context, where the code context P A potential bug in a piece of partial
consists of problem specification and a piece of partial code code is code span that can become a bug provided some completion, i.e., it fails the completion and at the same time can be changed to make the completion work. Note that a potential bug not a bug per se without completion. Shown in Figure iis an illustration of our task with a problem description at the top and partial code in the middle. On the left are reference code context and correct completion from the chosen Code-LLM. On the right, the highlighted potential bug (-=) makes the reference completion incorrect. The Code-LLM reacts to this potential bug by generating a different completion (bottom right). However, the completed code is still functionally incorrect.
To conduct a quantitative study of bCC, we construct two datasets. First, contains interview-stye coding problems from HumanEval dataset [10]. with buggy/reference partial code pairs being generated by introducing semantic-altering operator changes to reference solutions. This dataset provides a well-controlled setting to assess models' behavior upon potential bugs. Second, Miggy-FixEEva dataset, based on FixEval [19, contains user submissions to coding- contest problems. The buggy/reference pairs are constructed from rejected and accepted submissions by the same user to given problem. This dataset helps assess models' performance over a realistic distribution of potential bugs. Our benchmarks are well associated with the existing benchmarks for Code-LLMs.
dataset
coding-contest
Via our empirical studies, we find that the presence of potential bugs drastically degrades the code- completion performance of high-performing 5% across both datasets for all tested model variants. For instance, on Huggy-HumanEvws test-case pass rate of CODEGEN-22 BUNGEN-BB--- 2B-NON completions drops from 54.9% (reference partial code) to 3.1% (partial code contains potential bugs), which is worse than the score when no partial code is provided (9.3%). Our results demonstrate that Code-LLMs are highly susceptible to potential bugs.
with test-case pass rates dropping to below the
Furthermore, we attempt several post-hoc methods augment Code-LLMs to better deal with potential bugs, namely removal-then thnovallthe--completionn completion-then-rewriii completion. The latter two augment the Code-LLMMs with an external code repairer as the rewriter component. Our evaluation shows that the attempted methods improve the buggy-code complction performance of all tested Code-LLMs. However, the performance gap remains large between these
and rewriting-ther
"This setting is pell-aligned with the text-to-code generation task for LL 12 18

---

Without partial code
Reference partial code
B Buggy partial code
5689
43.0
411
37.8
Poss@11
32.3
21.b
24.1
4.7
8.9
24
4,3
1.8
CG-350M-mono
CG-2B-mono
Incoder-18
Incoder-68
CG-350M-mono 0G-28-mono
Incoder-18
Incoder-68
buggy- HumanEvall
Buggy-FixEver
Figure 2: Performance (pass@1 (1)) degradation in the presence of potential bugs. CG: CODE- GEN. Potential bugs severely harm the completion of all CodeLMs (red bars) compared to non-buge won-buggy settings (teal bars), making it even worse than completion without partial code (blue bars).
methods and the completion with reference partial code. We provide further case studies and analyses, e.g, effects of potential bugs' locations or the successful cases of naive completion for better understanding of the behavior of the tested models buggy-code completion.
Study scope and contributions. This work aims to explore and understand the behaviors of Code- LLMs under the buggy-code completion setting. We (i) define the novel buggy-code completion task, (ii) introduce two representative benchmark datasets, (iii) demonstrate the inability of Code-LLMS to handle potential bugs, and (iv) evaluate several baseline methods for improving Code-I.LMs on buggy-code completion.
2 Buggy-Code Completion
In this work, consider the code completion setting with inputs consisting of (1) specification that specifies the desired functionality of the program and (2) code context s, which is some unfinished code to be completed. Here, h is given as docstring and s is given as a few lines of code that are the beginning part of program, which we refer to as partial code (code) prefix. In the conventional setting of code completion, the objective is to suggest a completion e such that t == c is program that satisfies h. where "::" denotes code concatenation. Our buggy-code completion setting extends the conventional one with a challenging and realistic consideration: s may contain potential bugs.
problem statement in English,
Consider real-world scenario where programmer works unfinished code and utilizes the code cutoocommlltion feature for suggestions. Note that the coding process often sees coding mistakes or introduces inconsistent code usages, which not necessarily "incorrect" per However, there is fair chance that such mistakes or inconsistencies cause unintended and undesirable behaviors in the final program. In other words, they become bugs. Based on this intuition, we define potential bugs as the following:
Definition 2.1 (potential bug). Consider specification and a reference code prefix s for which some completion exists such that == e satisfies h. A potential bug is manifested small edit e over such that t! := s' :: c. does not satisfy h, where s' is the result of applying e on 8.
We note that potential bugs are not bugs per se and are only defined with respect some reference code prefix and completion The code prefix s' containing potential bugs is referred to as "potentially buggy" or simply "buggy" prefix (with respect to s) throughout the paper. most cases, omit the "with respect to" part for brevity. The resulting term "buggy prefix" refers to the definition here and by no means assumes that the prefix itself is buggy. Our study focuses on potential bugs associated with semantic bugs, i.e., edits that do not introduce syntax errors, semantic bugs are generally more challenging and interesting than syntax ones.
Remark 2.2. Intuitively, we assume that the reference prefix s highly correlates to preferred (practical) implementations of the task and that any deviation from likely less preferred. We use the operational definition of potential bugs in 2. for its simplicity and verifiability, allows initial exploration of the previously under-investigated scenario of bCC. We note that by this definition, a reference prefix itself can, in some cases, albeit less likely, be "buggy" with respect to some other references. However, such a case less a concern in this study, as we are mostly interested in finding completed functional program out of or s'.
well as for that

---

Definition 2.3 (buggy-code completion, bCC). Given a specification h and a code prefix s' containing potential bugs, buggy-code completion the task of generating complete program that satisfies h.
We deliberately loosen the constraint that t should contain s' as prefix, as by definition, and as well as one can see from the example in Figure fixating on the buggy prefix can make suggesting a satisfying solution difficult, if not impossible. As we explore in Section [.3] allowing models to suggest fixes to the buggy code prefix significantly increases the possibility that a generated program
passes the tests. However, it is often still possible to continue a buggy prefix to a valid program solution (see an example in Figure 12p.
An alternative view of the setting is that the buggy code prefix provides noisy and potentially flawed precondition for generating a solution to the coding problem. It presents a good-faith effort from a user to instruct the tool about their intended code solutions. A good-behaving model should take this as a hint and, at worst, discard it so that the generation performance is not worse than when no code prefix is given. However, this is not the case for our evaluated models (Section (42).
Remark 2.4. Our bCC formulation is nol a simple combination of code repair and code completion. As bugginess is property of completed programs, repairing the partial code is an ill-defined problem. thus making repairing- wewairinn-then-oopllee then-completion also ill-defined. Therefore, our bCC task is better viewed as an extension of code completion with the additional challenge that generating semantically correct continuations from the given partial code without deviation may be difficult or even infeasible. This challenge requires models to be aware of the existence of potential bugs for better suggestions.
3 Benchmarks
This section introduces our new datasets with proposed baseline methods and evaluation metrics.
3.1 Datasets for bCC
Based our task definition, each instance in a bCC benchmark should contain problem description specifying requirements, a piece of buggy code prefix to be completed, and a set of test cases for assessing the correctness of the finished code. Optionally, an instance also has corresponding reference code prefix and a valid solution completed from the reference code. To our knowledge, no existing dataset meets all the desiderata. Commonly used large-scale datasets for code completion, e.g., 15050 (20) and Java Corpus [1 do not come with test cases. Recent small-scale manually curated datasets for code generation, e.g., HumanEval [10], MBPP [8] APPs (22), come with test cases and reference solutions but no buggy or failed solutions. There is also no predefined way to extract partial code from the reference solutions. This is also the case for program-repait datasets:
popular datasets either lack test cases, e.g., 23 24
25
26
and/or are of small sizes, e.g., (27 28 29].
We introduce two datasets for evaluating bCC in Python, both with all the aforementioned desired components. We ensure that 1) all partial code snippets contain potential bugs that fulfill our definition, i.e., their respective completed code is incorrect, certified by failing test cases; and 2) all potential bugs are semantic in nature, i.e., they cause no syntax error in their respective completed code.
3.1.1 Bugan-HulanEval
We first introduce Hugay-HumanEval to offer a controlled setting for evaluating bCC the presence of single semantic bug. Buggy- Humanvll contains 1896 bCC instances constructed from a subset of HumanEval problems [10]. The HumanEval dataset is a popular dataset of manually written introductory coding problems designed for evaluating the code generation ability of Code-LLMs.
Potential bugs are introduced as semantic-altering operator changes to reference solutions. We search for applicable binary operators in reference solutions and change them into their semantic opposites, e.g. into -. To ensure that the edit introduces a bug, we execute the altered program and only keep the ones failing some tests. We then specify a line after the altered operator to split the solution into a code prefix and suffix. We keep the buggy prefix as part of the input for bCC. We split the unaltered solution at the same line to get corresponding reference prefix. On average, the problems selected for
have longer solutions than the unselected ones in HumanEval, with 8.2 lines vs. 2.9 lines of code, respectively. Appendix A.l provides details of our dataset.

---

its sacrifice of all potentially 3.1.2 Buggy-FixEvy
To evaluate bCC with more realistic bugs, we introduce buggy-FixEval with bCC instances con- structed from CodeNet (30] and FixEval [9] FixEval is program repair benchmark based on user submissions to competitive programming websites. Each data sample in Evxl can be viewed as pair of submitted programs from the same user, with the accepted submission being regarded as reference or fixed and the preceding rejected submission being regarded as buggy.
To create buggy-FixEval CodeNet and omit the problems with no match. For each submission pair, we identified potential bugs as the differences between the rejected and accepted submissions. We ensure that the rejected submission contains no syntax error and fails at least one test case. We split the solutions into halves and regard the prefix from the rejected solution as containing potential bugs and the prefix from the accepted solution as a reference. To guarantee the differences between the buggy prefix and the reference prefix related to potential bugs, we impose a limit on the character-level edit distance between them, ignoring comments and white spaces. The lower limit ensures that the differences not comments or white spaces. The upper limit (20, chosen by manual inspection) reduces the chance that the difference related to a reformatting or a re-implementation. We then manually inspected all the remaining pairs and excluded undesirable cases such as that the reference prefix itself is already a correct solution or that the two prefixes are semantically equivalent. Finally, we execute the concatenation of the buggy prefix and the reference completion and ensure that it fails at least one test case. More details about creating buggy-FixEval can be found in Appendix A.2
we match and pair each problem with its problem statement found in
3.2 Baseline Methods for bCC
Beyond the naīve generation from code context, we study several methods for augmenting Code- LLMs utilize the partial code better while mitigating the effect of potential bugs. Assuming a bug detection component is equipped, we focus on simple and modular-design approaches that do not require additional training and are flexible for further incorporating newly developed Code-LLMs.
We bypass the negative effect of buggy partial code by removing the entire content code fragment from the model input. In particular, for Buggy-HumanEval, the input to CLde after removal consists of the problem statement and the function header (and examples). For buggy-FixEval, we keep only the statement and the code fragment used for reading input data. Intuitively, as -hon-olllloooon guarantees that the input to Code-LLMs contains no bug, we expect this method to perform better than the naïve completion. However, the drawback of removal-thene eenoval-then-complttoo useful information brought by the partial code.
Removing partial code, then completing
This approach at- egg. neural translating a buggy program into a valid program (3I). As code-repair models, or code fixers, are usually trained to fix the complete programs, we first complete the code by naïve completion, then apply code fixers on the potentially buggy programs. We use RealiT (32), SotA program repair model as the code fixer. RealiT is designed for misused variables, wrong literal, wrong binary, and unary operators, which is especially suitable for the bugs introduced in the Bugay-Huna
Completing first, then rewriting the program (completion-then--erriiii tempts to fix the buggy code using pre-trained code-repair models
This approach at- tempts to resolve potential bugs in the partial code before completing. To do so, we first locate code lines containing the potential bug, then rewrite these lines. To detect potential bugs, we propose a tikeihhodd-baee measure to identify the line most likely contain potential bug. In particular, we score each code line with the following procedure. First, for each token, we define its buggy score to be the difference in likelihoods between the token with the highest likelihood (i.e., the ar argmax token) and the observed token. The buggy score for each line is then calculated by taking either the maximum or average of non-zero buggy scores of its tokens. The line with the highest score is most likely to contain the potential bugs. We use the INCODER-66 33 as the infilling language model for rewriting code. We provide a detailed explanation and example illustration of likelihood-basse measures in Appendix B to understand our rewriting- newriting-then-complee chon-oomleeiiion method better.
Rewriting the partial code, then completing
5

---

Table 1: Pass@1 (1) of completion methods Duggy-HumanEval and buggy-FixEval datasets. For all Code-LLMS, all three proposed methods improve the completion performance the naive completion. On average, completion- completion-then-rewriii then-rewriting and rewriting teree-the-eeee then-completion achieve the best scores on gaps between these methods and completion with reference prefixes. *Best methods in each column are in bold. Table Appendix C provides results with a larger model GOODEGEN-16B-MORN
and buggy-FixEval respectively. Nevertheless, there are still substantial
auggy-Humante
buggy- Fixyy-
Prefix
Method
CODEGEN-
INCODER-
CODEGEN-
INCODER-
350M
2B
IB
6B
350M
2B
1B
6B
reference
completion
43.0
$4.9
41.1
50.6
27.6
37.8
24.1
32.3
completion
0.7
3.1
0.5
1.0
2.4
4.3
1.2
1.8
buggy
removal-heen-compltionn
4.7
9.3
3.9
6.9
3.3
8.6
6.4
rewriting-thent teeriinngthen-ccmpletiiio
14.1
24.9
9.1
16.4
2.4
7.2
2.6
5.1
completion-then-eewiiin then-rewriting
22.5
23.6
22.7
25.2
2.3
4.7
1.7
3.0
3.3 Evaluation Metrics
We measure the functionality of completed program by executing it against the provided test cases. Following the recent code- genereenertiion works [e.g.. R 10 34
12 33 we measure the passšk (t) for each bCC instance as passčk := (2= where n completions are sampled from a model, and с of them pass all the tests. We choose 100 and k: 1, 10, 100. This metric estimates the probability that any of samples from the model passes the tests. As multiple bCC instances may be derived from the same programming problem, first average the pass@k within each problem, then average across all problems (macro-average)) to avoid the dominance of a few problems.
While passing all test cases does not 100% guarantee the correctness of a program, this metric provides a practical and efficient proxy for functional correctness. Note that we do not use match-based metrics, e.g., exact match or C. (35] because they do not properly reflect the functionality of generated and no reference completion available for buggy prefixes.
code 22 10
Experiments
We design two experiment sets to investigate (1) how well the existing Code-LLMs adapt to bCC (Sec. 4.2) and (2) whether we can have simple fix with Code-LLMs for bCC (Sec. 4.3). We provide ablation studies on potential bugs (Sec.4.3) and on combining buggy-based completion with reference prefix (Sec Sec. 4.6 case studies about interesting behaviors Code-LLMs under bCC, with additional results in Appendix
4.1 Experiment Settings
Code-LLMs. We evaluate the two popular and open-sourced Code-LLMs for code completion. CoDEGEn [12] is family of LLMs trained on both natural and programming language corpora with high performance code generation on HumanEval [10. We use released model checkpoints: BODEGEN-350M-MR
CODEGEN-2B-MON
and CODEGEN- 66BMONO. INCODER 33
models are trained with a causal masking objective, allowing them fill blocks code conditioned on arbitrary left and right contexts. We use the released model checkpoints INCODER- B and
INCODER- 6B, each with IB and 6B parameters. We select CODEGEN [22] and INCODER they are publicly available with high performance of code generation.
33 since
model is expected to complete an unfinished function. Following the HumanEval benchmark [10], we set the models' input as the partial code leading up to the completion location, with the problem description embedded as the doestring of the unfinished function. For FixEvl,, the completion task is to complete an unfinished program with inputs being the problem description as a file-level docstring (quoted within triple quotation marks), followed by the partial code leading up the completion location. Generation and sampling details. Following the perforrnng settings reported in the corresponding works 12 33), we use temperature sampling with temperature 0.6 for CODEGEN and top-p sampling with p 0.95 and
Generating completions. Input format. For Miggy-HumanEEve

---

temperature = 0.2 for INCODER. Based on the reference solutions and computing efficiency, we set the maximum length limit for oulputs from 200 to 600 tokens, varying with the problem sets. We observed similar performance trends between the tested models across different length settings. We post-process the output string following the same procedure used in their code releases.
Code-repair and code-rewriting models. For cemoval-then-compleee RealiT [32] as the code-fixer, which is trained and tested over artificial and realistic bugs. For the code-rewriting model, we use the INCODER-Oi model 33 due to its ability to code infilling and apply the similar settings used for infilling reported the model's s paper [3
we use the latest model of
4.2 How Well Do Existing Code-LLMs Perform Buggy-Code Context?
We evaluate the latest Code-ILMMs for buggy-code completion on Beggy-HumanEval and buggy- FixEval, shown F In particular, the performance measured in terms of pass@1, and we use four models: CODEGEN-350M-A0 150M-MONO, 6B. First, comparing the scores between reference and buggy partial code for each model, we see that the presence of potential bugs is detrimental to the completion models, with pass@1 drops from 41.1-54.9% to 0.5-3.1% % over buggy- Huggy-,ummnEvvl,, and from 24.1-37.8 buggy-FixEval. Moreover, the pass@1's I's upon buggy partial code is universally dominated by those without partial code: 3.9-9.3% over buggy-Humande HumanEval, (Appendix [ shows the similar findings with very large model (CODEGEN-
MODDGGEN-B-MMRT -oNo,, WCCoDER-B, and INCODER-
to 1.2-4.3% over
9-8.6% over buggy-Fixte Evall. Table 7
These results indicate that (i) the tested Code-LL drastically fail at bCC instantiated by our datasets, and (ii) the presence of potential bugs destroys the benefit brought by the partial code.
Why do Code-LLMs fail at bCC? We manually inspect samples from the best-perfone pest-performing BODEGEN-2B-MONT model and find the two most common failure modes among failed samples. First, the model fails to react to the potential bugs (i.e., common completions remain the same), as shown in Figure 8 This mode happens in 90% of instances and 93% of problems with at least one failed instance. We conjecture that the model is not sensitive to and thus ignores minor code changes and/or chooses to default to common patterns in the training data. Secondly, the model fails to bypass the potential bugs, likely because such patterns are rare in high-quality code. In other words, the model might have recognized the potential bugs and significantly changed the output distribution but still failed. Figure| 7 iillustrates an example of this case. We provide further details in Appendix D.2
4.3 How Effective Are Baseline Completion Methods Against Potential Bugs?
We evaluate the completion methods introduced in Section 3.2 using the same bCC setting, shown in Table I We include results of completing reference partial code to see how potential bugs affect the Mode-ILMs. Figure i in Appendix Croovides full results and interpretations for ie 1, 10, 100.
The effect of proposed completion methods. All three proposed methods outperform the naive buggy-code completion baseline. On bugay-HumanEval, we observe the general trend of outperforming rewriting cieen-oopleeion which outperforms removal- noe--compettion. Note that these scores of removal-then-conn complecono are generally lower than reported scores of the similar method 12] 33 probably because buggy- buggy- HumanEval is derived from rela- tively more challenging subset of HumanEval (see Section 3.1.I. On buggy-FixEval, we observe removal-ther the--h-ommmeeiinnnnn to outperform then-rewriting. The performance gap increases as the size of the completion model rises. Similar comparison trends are observed for passšk for k = 10, 100. Nevertheless, performance gaps remain significant between the best method and the completion from the reference code for all settings.
completion-then-rewiiiee
which outperforms completion-
The effect of Code-LLMs' capacity. For each type of code completion model, the larger version performs better than the smaller version using the same method. For instance, CODEGEN-2B-Mo outperforms CODEGEN-33 35500 100--350M-MMM MOOO for all settings in two datasets. Compared to CODEGEN, INCODER models, in general, obtain better passu1 but worse pass@10 and paass110.. This suggests that CODEGEN models generate more diverse completions, while INCODER models generate more precise completions. Furthermore, INCODER more sensitive to buggy partial code than CODEGEN models, evidenced by the lower scores from naive bCC.
7

---

Table Comparing passo1 (1) of completion methods on reference and buggy partial code. While the proposed mitigation methods achieve better performances than naive completion with buggy prefixes, they may harm the completion when no bug exists (reference).
Dataset
buggy- HvmanEval
buggy-FixEvar FixEval
CODEGEN-
2B-MONO
350M-MONO
2B-MONO
350M-MONO
reference
buggy
reference
buggy
reference
buggy
reference
buggy
nalve completion
54.9
3.1
0.7
37.8
4.3
27.6
2.4
twwiting-then-comllliion
49.6
21.9
35.9
14.1
37.0
7.2
26.4
2.4
completion-thenn-ewriingg 27.7
23.6
22.2
22.5
19.4
4.7
3.7
2.3
Table 3: Balancing rewriting seehiinng-te--cooe tomm-completion for the buggy and reference settings with bug-detection thresholding. Complction performances can be adjusted via varying bug-detection thresholds from 0 (fix all partial codes) to 1 (keep all partial codes). Results are with CODEGEN-28 -B-EEN--B-MOOR B-MONO.
buggy-FixEval
threshold
partial-code setting
0
0.3
0.9
reference
8.6
14.6
37.0
37.8
buggy
8.6
7.1
7.2
4.3
Synthetic bugs versus real bugs. Among the two bCC datasets, we observe that the overall performance mitigation methods better buggy-HumanEVal than buggy-FixEval. This indicates the difficulty of realistic potential bugs in buggy-Fixe Fixgy: There may be multiple bugs; bugs may be potentially mixed with non-bug saa-bug-fiiiing fixing changes; and bugs are more nuanced than single operator changes. Furthermore, while achieving the best performance in most cases, completion- -ettiionntee-eee then--ewriting only shows marginal differences from other methods when using larger models on buggy-FixFval.
Take-away:
Our baseline methods improve the completion for all evaluated Code-LLMs. However, the remaining performance gaps the completion with reference partial code are still large.
4.4 What If Partial Code Does Not Have Potential Bugs?
As shown in Table 2 mitigation methods for bCC may harm completion from reference code context thenoolltheenoooplttiioon is the same for both settings, thus not listed.) This suggests that general code completion should consider both cases when potential bugs may and may not exist.
With our baselines, we can use the thresholding approach for detecting potential bugs. For instance, in rewriting-thes thariingthhe--oolletion a token is only considered a potential bug its likelihood gap to the argmax token is beyond threshold (between 0 and 1). Table compares pass01 scores of rewriting-then-cone complelcto varying thresholds on buggy- EvxEal. can help achieve relatively good balance for the two cases. A similar approach can be applied to completion-then wewtoon-tthn--ewriite then-reeri,ing, as RealiT provides the probability of a token being a bug.
can see that the threshold of 0.9
Remark 4.1 (Imbalance in real distributions). We note that distributions of natural code bugs are usually imbalanced as practical bugs occur infrequently [36] 37). However, potential bugs may occur more frequently as our target setting is a work-in-progress code scenario (38) rather than high-quality code of popular open-source projects in previous studies. Nevertheless, for comparing methods, evaluate more balanced data to avoid the dominant effect of performance on reference code.
4.5 Analysis of Effect from Bug and Split Locations
To understand how bug location and context size affect completion, we aggregate results by the loca- tion of potential bugs and the location of partial code splits within bumay-Huna The locations normalized
(potential bug line #) / lines) and (split line #)/ lines), where potential bug line #, split line #, and lines are the number of lines starting from the function header to the line containing the potential bug, the end of the partial code, and to the end of the canonical solution.
Figure 3 presents heatmaps of pass@1 scores (averaged over the bCC instances falling into each cell) evaluated on the CODEGEN-25 22--NO MONO with naive completion reference HumanEval, and naive

---

21
85
43
14 24
15
81
27
X2
07
03
65
cideele scatin
(a)
(b)
(c)
(d)
Figure 3: Average passo1 scores by bug and code split locations using CODEGEN-2B-MONK
Buggy-HumanE,e Left to right: (a) naive completion reference code, (b), (c), (d) naive completion, sompletion-then-eeriiin normalized by the length of reference solutions. For each split location in (a), the scores may vary across potential bug locations as the buggy-instane distribution is non-uniform across bug locations.
and peerttingtheen-cmmlltiooo on buggy code. Locations are
completion, completion-tien sertoontteee--eweeeeee rewriting, and rewriting- then- (max) on Eeggy-HumanEvall First, the model performs better given longer partial code in the reference case (Figure While the naive completion performs overall poorly with potential bugs (Figure (3)). . it performs relatively well when potential bugs appear on or near the last line of the partial code (along the diagonal). More interestingly, wwwittng-then-compleeee (Figure Bi), while compleeion-thhe--ewiiie We suspect that a longer prefix makes completion models less likely to deviate from the reference completion. Hence, as inputs the subsequent rewriting model, the completed code better resembles input types for which the rewriting model is trained, making the repairing more likely to succeed.
achieves higher scores when potential bugs appear later performs better with longer code prefixes (Figure
4.6
Case Studies
Are potential bugs always harmful? As discussed in Section potential bugs do not guarantee the completed code to be buggy. While we observe performance degradation of Code-LL.Ms under the presence of potential bugs, we find several interesting cases where the models manage to generate correct code. For instance, Figure [2]in in Appendix shows that for the potential bug == (highlighted) modified from !=, the completion model deviates its original algorithmic flow with continut command and completes with correct functionality, albeit different from the canonical solution. This is an example that some bCC cases are recoverable and that Mode-LLMs, can adapt to them.
When do Code-LLMs succeed bCC? For the successful cases of naive completion bCC, observe that either the model (i) ignores the incorrect state and generates the correct completion or (ii) takes into account the potential bug to generate a completion adapting it. Figure in Appendix shows an example when Code-LLMs ignore the if-else statement to bypass the potential bug. Further case studies and more elaborate discussions are in Appendix
Related Work
Code completion. Code completion provides code suggestions based given context (39 The scope of completion ranges from the next token or line [T], method and class names 41 to
40
38
the entire of the function [42] or program. Early works 55] 43 viewed code as sequences of
tokens and applied statistical language models to the problem, along with other attempts at building probabilistic models of code 44 41 45). Later works adopted deep neural networks S 46] and
pre- vre-training techniques [9] 47 for better code modeling and completion. Our work considers the code completion setting at the function and program levels and focuses on using large language models for completion. Beyond sequence modeling, recent works considered integrating code prior knowledge via abstract syntax trees 6 48) code token types [9] graph structures [7]. hierarchical context [49]. or generating the sketch [50 or even extending the task's information beyond the given input files [5 [2]. We focus on using only the task description and partial code as input prompts to the model, allowing the use of more Rode-LLMs and publicly available datasets.

---

Automatic program repair. The research on automatic program [53 54 55 56
57
58
59
25
relieves developers from the enormous effort of finding and fixing programming bugs. Recently, Code-ILMMs have been adapted for program repair by translating buggy programs into their reference counterparts 60 1 61]. Among those, we use RealiT [32 as our repair model in the completion- then-rewriting method since they obtain the SotA results and utilize similar simple mutations during training. Despite the similarity, code repair commonly targets fixing bugs from complete programs while we study potential bugs from partial code. To enrich the amount of data for program repair, methods have been proposed to synthesize artificial bugs through code mutants [62] 63 64 65 25
or learning create bugs [66]. Similarly, employ code mutants create artificial bugs.
Relation to adversarial examples. Adversarial examples are instances where small perturbations to the input lead the model to change into wrong predictions. They have been extensively studied in computer vision [67 688 and natural language processing [69] Recent works suggested similar situations for code-learning models, where small, semantic-presserving code transformations led to performance degradation (70 71. Buggy-code completion can be seen as dual problem to adversarial examples, where we expect the model to adjust its predictions up small semantic- altering changes in the input. In our case, sensitivity is not a problem, but insensitivity is.
Benchmarks for buggy-code completion. Multiple benchmarks have been studied for code comple- tion and program repair. For code completion. CodeXGLUE [T]. CodeNet [30] and HumanEval [10] are widely used. CodeXGLUE contains corpora of Java and Python programs for completion but only supports match-based evaluation. CodeNet collects programming problems from online judge sites, with both solutions and test cases. HumanEval can be considered Python-function completion dataset, with the context being the problem statement and function header. We derive our datasets from HumanEval and CodeNet datasets. For neural program repair, many datasets require the match- based evaluation [24 26 or focus on the compiler errors 23 25], which are different from our setting. While IntroClass 27 QuixBugs (28), Defects4J 772 or Refactory [29] provide the test suites for evaluation, their test does not reflect the real-world bugs or lacks context support for the use with Code [99. FixEval [19] is recently proposed as new context-aware program repair dataset to mitigate these limitations, with many problems derived from the real submitted programs. However, FixEval does not provide the problem statement and focuses solely the program repair, we derive a new benchmark using FixEval and its source of problems CodeNet [30].
6 Discussion and Conclusion
Limitations. Our baseline methods developed for bCC may degrade the completion performance on reference code context, as shown in Section 4.4 suggesting the need for balancing the buggy and reference settings in solutions to bCC. Furthermore, while buggy-FixEwy is associated with real-world coding contest programs, it is unclear how closely buggy-Fixe FixEval aligns to the general software development setting where obtaining a test suite and proper evaluation more challenging.
Impact and applications. As work focuses on the less refined and more error- error-prone prone work-in- progress code, the code context should be viewed as a hint of user intent rather than a high-quality "gold" implementation. It thus naturally follows that pair programmer or a smart tool should suggest a change to the draft code rather than blindly continue it if they believe a certain part of the existing draft is not intended. From user experience perspective, an IDE can display code change suggestions to user's existing code if such parts are identified. Similar functionality already exists for other types of code change suggestions, e.g., spelling correction and missing imports.
Conclusion. We introduce and define the buggy-code completion problem, inspired by the practical coding scenario where one completes a coding program given the problem statement and partial code with potential bugs. We construct two new datasets, buggy-HumanEval and buggy-FixEval, as task benchmarks and find that the presence of potential bugs significantly degrades the completion performance of all evaluated large language models of code. Our further investigation of completion methods for Code-LLMs in dealing with potential bugs shows that completing with potential bugs remains challenging despite augmenting models with external program-repair models. We provide extensive ablation and case studies for further understanding and analysis of buggy-code completion setting. We hope our novel and systematic study paves the way for future works in understanding and improving the usability of Code-LLMs under practical softwaree-evvllppeenl settings.
10

---

References
[1] Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. A study of visual studio usage in practice. In 2016 IEEE 23rd International Conference Software Analysis, Evolution, and Reengineering (SANER), volume 1, pages 124-134, IEEE, 2016.
[2] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness, ACM Computing Surveys (CSUR), 51(4):1-37, 2018.
[3] Triet HM Le,, Hao Chen, and Muhammad Ali Babar. Deep learning for source code modeling and generation: Models, applications, and challenges. ACM Computing Surveys (CSUR), 53(3):1-38, 2020.
[4] Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Communications of the ACM, 22-1312 2016.
[5] Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N Nguyen. A statistical semantic language model for source code. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 532-542, 2013.
[6] Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. arXiv preprint
2017.
[7] Mare Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Genera- tive code modeling with graphs. arXiv preprint
2018
[8] Chang Liu, Xin Wang, Richard Shin, Joseph Gonzalez, and Dawn Song. Neural code completion. OpenReview, 2016.
[9] Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. Multi-task learning based pre-trained language model for code completion. In Proceedings of the 35th IEEF/ACM International Conference on Automated Software Engineering, pages 473-485, 2020.
[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[11] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[12] Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint 2031v:2203.13474, 2022.
[13] Ariel Assaraf. This is what your developers are doing 75% of the time, and this is the cost you pay. hutps:/ccorulogix.com https//ccrooogss.combllo///sssss wttllgghhisssiillltooooddvvrrrro -f-- -5------h----me--nd -ows--sh--ooss-oou--oaeoo pay/ 2015.
[14] Tom Britton, Lisa Jeng, Graham Carver, and Paul Cheak. Reversible debugging software "quantify the time and cost saved using reversible debuggers". Citeseer, 2013.
[15] Vincent J Hellendoorn, Sebastian Proksch, Harald C Gall, and Alberto Bacchelli. When code completion fails: case study on real-world completions. In 20/9 IEEE/ACM 41st International Conference Software Engineering (ICSE), pages 960-970. IEEE, 2019.
[16] Gareth Ari Aye, Seohyun Kim, and Hongyu Li. Learning autocompletion from real-world datasets. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP pages 131-139. IEEE, 2021.
[17] Mare Otten. User evaluation of incoder based on statement completion. Buchelor's 's Thesis, Delft University of Technology, 2022.

---

[18] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint 1rXiv:21080077:0 2021.
[19] Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. Fixeval: Execution- evaluation of program fixes for competitive programming problems. arXiv preprint arXiv:2206.07770 2022.
[20] Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, pages 731-747, 2016.
[21] Miltiadis Allamanis and Charles Sutton. Mining source code repositories massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 207-216. IEEE, 2013.
[22] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[23] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix:
Fixing common с language errors by deep learning. In Thirty-First AAAI conference on artificial intelligence, 2017.
[24] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29, 2019.
[25] Michihiro Yasunaga and Percy Liang. Break-it-fix-it: Unsupervised learning for program repair. In International Conference on Machine Learning (ICML), 2021.
[26] Faria Huq, Masum Hasan, Md Mahim Anjum Haque, Sazan Mahbub, Anindya Iqbal, and Toufique Ahmed. Review4repair: Code review aided automatic program repairing. Information and Software Technology, 143:106765, 2022.
[27] Claire Le Goues, Neal Holtschulte, Edward K Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. The manybugs and introclass benchmarks for au- tomated repair of e programs. IEEE Transactions on Software Engineering, 12612):1236-1256 2015.
[28] Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: A multi- lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming. languages, and applications: software for humanity, pages 55-56, 2017
[29] Yang Hu, Umair Z Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. Re- factoring based program repair applied to programming assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 388-398. IEEE, 2019.
[30] Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[31] Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.0 00859, 2021.
[32] Cedric Richter and Heike Wehrheim. Can we learn from developer mistakes? learning to localize and repair real bugs from real bug fixes. arXiv preprint arXiv:2207.00301, 2022.
12

---

[33] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong. Wen-tau Yih, Luke Zettlemoyer and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprini arXiv:2204.05999, 2022.
[34] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, (18866644:1092--1000 2022.
[35] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009. 10297, 2020.
[36] Jingxuan He, Luca Beurer-Kellner, and Martin Vechev. On distribution shift in Sarning-basel bug detectors. In International Conference on Machine Learning, pages 8559-8580, PMLR, 2022.
[37] Rafacl-Michall Karampatsis and Charles Sutton. How often do single-statement bugs occur? the manysstubs4j dataset. In Proceedings of the 17th International Conference on Mining Software Repositories, MSR '20, page 573-577, New York, NY, USA, 2020. Association for Computing Machinery.
[38] Xuechen Li, Chris J. Maddison, and Daniel Tarlow. Learning to extend program graphs to work-in-progress code, 2021.
[39] Marcel Bruch, Martin Monperrus, and Mira Mezini. Learning from examples to improve code completion systems. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium the foundations of software engineering, pages 213-222, 2009.
[40] Sebastian Proksch, Johannes Lerch, and Mira Mezini. Intelligent code completion with bayesian networks. ACM Transactions Software Engineering and Methodology (TOSEM), 25(1):1-31, 2015.
[41] Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th joint meeting on foundations of software engineering, pages 38-49, 2015.
[42] Albert Ziegler, Firini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simis- ter, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programminng pages 21-29, 2022.
[43] Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 269-280, 2014.
[44] Miltiadis Allamanis and Charles Sutton. Mining idioms from source code. In Proceedings of the 22nd acm sigsoft international symposium on foundations of software engineering, pages 472-483, 2014.
[45] Pavol Bielik, Veselin Raychev, and Martin Vechev. Phog: probabilistic model for code. In International Conference on Machine Learning, pages
PMLR, 2016.
[46] Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In International conference on machine learning, pages 245 256. PMLR, 2020.
[47] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1433 1443, 2020.
[48] Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. 2021 ieee/aem 43rd international conference on software engineering (icse), 2021.
13

---

[49] Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Syyatkovskiy, and Neel Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers, arXiv preprini arXiv:2010.03/50, 2020.
[50] Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Mare Brockschmidt, and Miltiadis Allamanis. Learning to complete code with sketches. In International Conference Learning Representations, 2021.
[51] Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Scung-won Hwang, and Alexey Svyatkovskiy. ReACC: A retrieval retrieval-augmented code completion framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2022.
[52] Hengzhi Pei, Jinman Zhao, Leonard Lausen, Sheng Zha, and George Karypis. Better context makes better code language models: A case study function call argument completion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[53] Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. arXiv preprint arXiv:1711.00744 2017.
[54] Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. Neural program repair by jointly learning to localize and repair. arXiv preprint arXiv:190.001220. 2019.
[55] Vincent Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International conference on learning representations, 2019.
[56] Dobrik Georgiev, Marc Brockschmidt, and Miltiadis Allamanis. Heat: Hyperedge attention networks. arXiv preprint arXiv:2201.12113. 2022.
[57] Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-sequene learning for end-to-end program repair. IEEE Transactions on Software Engineering, 47(9):1943-1959 2019.
[58] Yi Li, Shaohua Wang, and Tien N Nguyen. Dlfix: Context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 602-614 2020.
[59] Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning, pages 10799-10808. PMLR, 2020.
[60] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang, Unified pre- training for program understanding and generation. arXiv preprint arXiv:2103.06333, 2021.
[61] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code. ArXiv, Abs/2001.00003 2019.
[62] Jibesh Patra and Michael Pradel. Semantic bug seeding: learning- based approach for creating realistic bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 906-918, 2021.
[63] Cedric Richter and Heike Wehrheim. Learning realistic mutations: Bug creation for neural bug detectors. In 2022 IEEE Conference Software Testing, Verification and Validation (ICST), pages 162-173. IEEE, 2022.
[64] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. Learning how to mutate source code from bug-fixes. In 2019 IEEE International Conference on Sofwwre Maintenance and Evolution (ICSME), pages 301-312. IEEE, 2019.
14

---

[65] Thibaud Lutellier, Hung Viet Pham. Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium software testing and analysis, pages 101-114, 2020.
[66] Miltiadis Allamanis, Henry Jackson-Flux, and Mare Brockschmidt. Self-supervised bug de- tection and repair. Advances in Neural Information Processing Systems, 34:27865-278766 2021.
[67] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. leee Access, 6:14410-14430, 2018.
[68] Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv preprint arXiv:2007.00755 2020.
[69] Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41, 2020.
[70] Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. Semantic robustness of models of source code. 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 526-537. IEEE, 2022.
[71] Goutham Ramakrishnan and Aws Albarghouthi. Backdoors in neural models of source code. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 2892-2899. IEEE, 2022.
[72] René Just, Darioush Jalali, and Michael D Ernst. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437-440, 2014.
15

---

A Dataset Details
This section presents the details of dataset construction and specifications. The length statistics are reported in Tab
A.1
Buggy- Buggy-HumanEval
10 is dataset designed for evaluating code generations from natural-languagee descriptions. It contains 164 manually written introductory coding problems in Python 3. Each problem given in the format of partial program, referred with a doestring, sometimes with necessary imports and helper functions before The problem description and often few input-output examples are encapsulated the docstring. Each problem is accompanied by a separate set of test cases and a manually written function body referred to as the "canonical solution", such that the concatenation of the prompt and the canonical solution passes all the tests. See an example in Figure 5
HumanEva
"prompt": function header
To create buggy- Hurey---ummEvall.
introduce artificial bugs by flipping binary operators in the canonical solutions into their semantic opposites. Specifically, we use the Python ast library and consider all operators under ast. ast.Bin0gp BinOp, ast. Compare and ast. AugAssign,,, e.g. >= and +=, For each found operator, we change it to its semantic opposite, e.g., / to *, >= to <, or += to -=, Table [ showws a considered and their opposites. We then check if the solution fails
complete list of the operators any test cases after the change. We skip cases where the semantic opposite is ambiguous (e.g., mod % vs. floor division // multiplication *) where the altered solution still passes all the tests]
Suppose the canonical solution has L lines, and the operator change happens in the I-th line. For each I < i < L. append the first lines of the altered canonical solution to the original HumanEval prompt to form piece of potentially buggy partial code, aka the "buggy" prompt for buggy-code completion. Accompanying each buggy prompt, by concatenating the original prompt with the first lines of the original canonical solution. In the example in Figure L. 8, 1 = 3, 4. The != in the third line of the canonical solution is changed to
also provide "reference" prompt
We generated 1896 buggy-code completion instances from 107 unique HvmanEval problems. This procedure results the list the number of instances per problem as follows,
[9, 31, 0, 5, 1, 0. 13, 0, 3, 0, 0, 4, 1, 0, 1, 0, 0, 0, 8, 0, 16, 0, 0, 0, 1. 28, 0, 0, 0, 0, 10, 63. 0, 0, 2, 14,
0. 38. 6, 0, 0, 7,
14, 3, 6,
0,
0.
0, 6, 16, 4, 2, 27, 0, 16, 0, 12, 5, 47, 0,
6.
21, 51,
7. 14. 13, 12. 0. 1. 0, 22, 180, 12,
1. 0, 0,
0,
0. 9, 0, 0, 18, 2. 32, 57,
39, 52, 0, 17, 31, 133, 29, 14, 21, 0, 3,
5, 12,
23, 8, 16, 22, 18. 10, 16. 0,
34,
4,
0,
37,
6, 174, 20, 14, 15, 13, 8, 6, 35, 12, 2, 11, 0, 0, 18, 16, 8. 6, 0, 0, 7,2, 0,
Table 4: We considered Python operators when introducing bugs to HumanEval.
class ast token
BinOp
AugAssign Add
Compare NotEq
Add
Sub
Mult
Div
Sub
Mult
Div
Eq
Lt
LtE
GtE
Operator
#=
/=
=
1=
<=
>=
Opposite
1-
1-
Table 5: Length (the number of tokens) statistics of buggy-HumanEval and buggy-Fix
Percentiles
50th
90th
95th
98th
99th
100th
214
470
550
595.6
606.7
617
buggy-FixEval
71
242.7
262
330
402.4
566
http:///iithubb.coo/mppeeee latiemnaiiimma-eevllll htttp:://ocs. http///cccs...ten..ooeoo org/3.
html
"We found 5 cases in our initial trials where the altered solutions still passed their test cases. Namely, changing the first applicable operators in the canonical solution of task IDs "HumanEval/10", 'HumanEval/18", "HumanEval399". 'HumanEval400", 'HumanEval/ 129. Upon close examination, in these cases, either the altered solution still correct the altered solution is incorrect, but the test cases too weak tell.
16

---

"""Problen Statement: In the Kingdon of AtCoder, people use a language called Taknese, which lowercase English letters.
In Taknese, the plural form of a noun is spelled based on the following rules: If a nouns singular forn does not end with s, append to the end of the singular
singular forn end of the singular forn.
If
append
Problem statement
given singular forn Taknese noun. Output plural forn. Input: Inout is given from Standard Input in the following fornat:
Output:
plural
given Taknese word. mn0
input()
Paatial code
if =(len (s)-1] !=
prompt from huggy-FixEval, containing prohlem statement and buggy partial code.
input() if s[len (s)-1] != "s";
= input
if s[]en(s)-1] 1=
print(s + "s")
print(s + "s")
else:
else:
print(s + "es")
print(s "es")
correct program
buggy program
(b) The original FixEval accepted (left) and rejected (right) user submissions.
Figure 4: Example of buggy-FixEval instance. The example is based p02546.
A.2 Buggy- Buggy-FixEval
FixEva
30 for studying program repair. It consists of 19 is a dataset derived from CodeNe
programming problems from online programming contest websites each with test cases, user- submitted solutions and judging verdicts to the submissions (e.g., accepted passing all the tests; wrong answer - failing least one test; or time limit exceeded the program did not terminate within the specified time limit). The test cases in FixEval are more extensive than those in HumanEval, as the programming contest websites use them to automatically judge a submission's correctness. Each problem comes with an alphabetical label ("A" to "F") indicating the order in which the problem appears its contest, with "A" usually being the easiest. Different from HumanEval, a solution or submission here is supposed to be complete program, reading input from and writing output to the standard input and output instead of just a function implementation.
To derive dataset for bode completion, we pair each FixEval problem with its matching problem statements from CodeNet and discard problems without match. Among the remaining problems, we focus the relatively easy "A"-label problems because the post-"A" problems have been reported be very challenging to solve even for recent language models of code: pass@1 < 12% with CodeTT5 31 under program-repair setting [19] Section 5.3].
We start by selecting rejected-accepted solution pairs. For each streak of Python submissions from user to a problem, if the streak ends with an accepted submission, we take the last submission with the verdict being none of accepted, compilation error runtime error its rejected counterpart.
Then, we remove all comments and empty lines for each accepted-rejected pair and check the following. (1) Neither submission contains exit() or system calls, they cause trouble for execution. (2) The first halves of the two submissions contain at least one non non-wwite-space difference, including difference, space/indent difference, variable names for the assignment operator. (3) The character-level edit distance between the first halves of the two submissions smaller than 20. Regarding the threshold of 20, sampled 100 pairs of submissions with various edit distances. We qualitatively verified that under this threshold, the differences between the two submissions are, in the majority of cases, focused semantic changes (or likely bug fixes) and not refactoring or re-implementations.
https://gtts https:///gtthbbb.. com htt.:/awb..coe///////ecetttccc.t
"Te bases of the data are from AtCoder (https:/ http://///////ee///// jpp/ and AIZU Online Judge (https
/onnineejudge. aizu.ac.jp//
17

---

from typing import List
def closee_eemeents (numbers: List(float), threshold: float) -> bool: Check if in given list of numbers, any two numbers closer each other than given threshold.
has_cclos__elmennts(
2.8, 3.01, 8.5)
False
has_close_ (lloss__lements(L1e 2.8, 3.0, 4.0, 5.0, 2.01, 8.3)
True
(a) The original HumanEval prompt containing function header and problem description doestring.
for idx, elem in enumerate (numbers):
idx2, elem2 in enumerate :
idx 1= idx2:
distance ablee - elem2)
if distance < threshold:
return True
return False
(b) The canonical solution provided in HumanEval.
from typing import List
def has_closs
close elements (numbers: List[float), threshold: float) bool: unn Check if in given List of nunbers, any numbers closer other than given threshold.
has_close_elenents ([1.0, 2.0, 3.0], 0.5)
False
has_close_element ([1.0, 2.8, 3.0, 4.0, 5.0, 2.81, 0.3)
True
for idx, elem in cnumerate(numberras
for idx2, elem2 in enumerate :numbers):
if idx == idx2: distance (eleem elem2)
(c) The prompt used for buggy-code completion, containing the original problem and buggy partial code.
def check(candidate):
assert ([1.,, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3)
assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) False
assert candidatee(1100, 2.0, 5.9, 4.0, 5.0], 0.95) True
assert candidate([110,, 2.0, 5.9, 4.0, 5.0], 0.8) == False
assert candidate 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True
assert candidate [[1.1, 2.2, 3.1, 4.1, 5.1], 1.0) =s True
assert candidate ([1.1, 2.2, 3.1, 4.1, 5.11, 0.5) == False
(d) The suite used for evaluating completed program.
Figure 5: Example of Buggy-HumanEVe instance. The example is based on HumanEval/0, The buggy prompt (Figure e) directly derived from the original prompt (Figure a) and the canonical solution (Figure b). We reuse the test suite (Figure d) for our evaluation.
The above process resulted in 492 pairs of prefixes from rejected-accepter solutions from "A"-Jevel problems. We manually inspected all the pairs to ensure the pairs satisfy the desiderata described in Section 3.1 Namely, we identify the following cases.
1.
The prefix from the accepted solution is already complete program that passes all the test cases. This can happen when the accepted solution very short, for example, fewer than three lines, and/or followed by comments unnecessary code. This counts for 26 pairs
18

---

2. The prefix from the rejected solution and the prefix from the accepted solution are semanti- cally equivalent up variable name normalization. This counts for 33 pairs.
3. The prefix from the rejected solution contains no control flow (e.g., if, for, while) and no output clause. In this case, a completion can ignore the prefix. This counts for 141 pairs.
4. The prefix from the rejected solution is bound to output wrong answers upon certain input. For example, contains a print () statement within a wrong condition. This counts for 25 pairs.
A list of identified cases can be found in our code repository. Case makes the completion trivial from reference prefixes. Case 2 makes the comparison between the buggy completion and reference completion less meaningful. Case 3 makes the buggy completion less challenging. Thus, we exclude instances identified as cases I, 2, and 3. We keep the case 4 instances because they fit the definition of potential bugs and are especially interesting in the sense that the "bugginess" already manifested the code prefix.
After the above step, we are left with 292 pairs. We then use the first half of the rejected submission as a piece buggy partial code and the first half of the accepted submission the corresponding piece of reference code for buggy-code completion. A "buggy" prompt is formed by prepending the problem statement as a file-level docstring, or multi-line string literal, to the buggy partial code. See an example in Figure
B Details of rewriting-then-completio Method
With rewriting-then-oon the--oopleeiiinn, we attempt to eliminate potential bugs via two-step procedure. We first locate the most likely line to contain potential bug using a likelihood- based measure and then rewrite the line using an infilling code language model.
B.1 Line Selection
Our underlying idea is to treat potential bugs outliers in the generation flow of Code-LLMs. We observe that most reference code has lower perplexity than the corresponding buggy code.
sieelihood-basee measures. We calculate the score of each line as follows: With model that can give token distribution for each token location, we define the token score as the difference between the likelihoods of the most probable token (i.e., the argmax token) and the actual observed token. We get the score of line by taking either the maximum or the average of all the non-zero token scores within it. The line with the largest score is finally selected to be rewritten.
To help better understand likelihood- based measures, consider the problem and the code prefix with potential bugs shown in Figure 8 The partial code has four lines in the function body. Given a language model G. for each token x, calculate the token-level score as p2 pt:
P1 probability of generating from the code up
P2: probability of generating x= from the code up to x, where x" gives the highest probability according to G.
For example, consider line 3 and token ==, We would have p1 0.01 (probability of G generating ==), and p2 = 0.95 (!=is is the most probable token according to G). The score of == is then 0.94. We obtain the line's score by taking the maximum scores all token locations in the line. This way of aggregating token scores to line scores is referred to as "Liieelhhood (Max.)" Below is the variation used report results in the main text.
Note that we take two measures to reduce the uncertainties of the likelihood estimation: (i) instead of the likelihood score of the target token itself, we use the maximal margin between the target token and the argmax token (similar to the popular approaches used in uncertainty quantification), and (ii) aggregate the score gap along the line and set high threshold to be more conservative.
find that accuracies of localizing the line of potential bugs (with the same setting described above) are approximately 82% and 53% respectively on Iuggy-HumanEwe and buggy-FixEval. For buggy-FixEval we compare the detected line with the line the first semantic difference between buggy and reference prefixes.
19

---

Table 6: Pass@1 (1) of each selection and completion method the Duggy-HumanEval and buggy- FixEval datasets. The best results for each dataset and completion model are bold. The better likelihood aggregation method is underlined.
Completion Model
Dataset
Selection Method
CG-350M
CG-2B
INCODER-
INCODER-6B
Hcuristic Oracle
22.1
29.6
21.4
28.5
buggy- Eaaay-aalaBBaall
Likelihood (Max.)
14.1
24.9
9.1
16.4
Likelihood (Avg.)
13.2
23.0
9.8
16.9
Heuristic Oracle
1.7
3.3
1.2
2.2
buggy-FixEwer
Likelihood (Max.)
2.4
7.2
2.6
5.1
Likelihood (Avg.)
2.7
7.9
2.4
5.4
Heuristic oracle for comparison. To see how well our likelihood-basee measures work, we compare them against heuristic oracle for predicting the line of potential bugs. In particular, compare the buggy code prefix against the corresponding reference code prefix and select the first line with non-trivial differences. The differences not about space, indent, comment. Since this requires access to the corresponding reference code prefix, it cannot be used as part of a solution for bcC. We thus refer to it as an "oracle" and only use here as method of reference gauge the performance of other line-selection methods. We do not use this oracle in any methods reported in the main text.
B.2 Rewriting
After identifying the line most likely contain potential bug, rewrite it by masking it and generating replacement line using a code-infilling model such as INCODER [3].
B.3 Results
Shown in Table our comparison between different likelihood-based measures. We observe that the Heuristic Oracle substantially outperforms the likelihood-basee methods in Huggy-HummnEEvl,, but the reverse true for buggy-Fix We provide a likely explanation for this phenomenon in the next paragraph. We also observe that Likelihood (Max.) outperforms Likelihood (Avg.) on but the reverse is true for Nuggy.FixEval. A possible explanation
average on that since the potential bugs bugay-HumanEval take the form of single token change, they are more amendable to being detected by focusing on the token with the largest score (maximum aggregation). such scenario, taking the average across tokens in the line might dilute this signal. On the other hand, potential bugs are likely more subtle in natural setting (represented by buggy-FixEval), and the potential bug may be spread out across multiple tokens. The fact that the average aggregation considers this may explain why it outperforms the maximum aggregation on buggy-FixEvvl.
We note that a large gap in performance between Heuristic Oracle method. This may be justifiable based on the information known the Heuristic Oracle. In buggy- Eval we introduce the bugs synthetically via single semantically altering operator change. Hence, the oracle method has guaranteed knowledge of the line where the bug was introduced. This is not the case in buggy-Fin FixEvll since the examples were constructed directly from human submissions without synthetic perturbations. Therefore, the line corresponding the first difference between these two submissions may not correspond to the bug location, multiple (instead one) locations can change between these two submissions. Therefore, this method is closer to heuristic than true oracle for Duxya.FixEval. This is reflected in Heuristic Oracle's 's substantially weaker performance on FixEval. The fact that likelihood-baat methods outperform the Heuristic Oracle buggy-Fin EvxEal supports this explanation.
and buggy-FixEval for the
20

---

CG-2B-mono
CG-350M-mono
Incoder-6B
Incoder-13
Buggy-Comp
Temoval-Comp
Rewriting-Comp
Comp-teeriting
40
Passi@
20
buggy-HumenEval
60
40
Pass@g
20
10
102
20
200
10
300
302
buggy-rixcval
Figure 6: Comparison of CodeLMs per each completion method in terms of pass@k. CG: CODEGEN, Comp: Completion. We see the trend consistent with CODEGEN-2B-MONG > INCODER-GG > CODEGEN-350M-MONG > INCODER-I B across methods and k's.
C Full Definition Statement and Additional Results
C.1 Full Statement for Definition 2.1 Section
Definition C.1 (potential bug in code prefix). Assume a problem given by specification with the test fa for evaluating its functional correctness and reference partial code Let T be the set of all valid programs, or solutions that satisfy / and have the prefix i.e., T = {t := : (fa = 1}. Let C² the set of all valid completed code, i.e., C. = {c|s :: E T A potential bug of s an edit Ge on token of that causes at least existing solution fail, i.e., for s' gs)s,, there exists Ch s.t. (s' = 0. The prefix s' is called buggy with respect to reference prefix s.
C.2 Full Results with k: 1, 10, 100 and Interpretation for Section 4.3
Figure 6 presents the full comparison among passčk for k = 1, 10, 100 for all methods. Intuitively, higher 1e reveals more the diversity of generated solutions. The relative performances between dif- ferent Code-LLMs are consistent across different k's. Regarding the relationship between completion methods,
The naive completion and completion-then-rewwe increases as k goes up, as the partial code constrains the input to the completion step.
methods achieve relatively small
The increase in performance of removal-then-conl compleoonl is relatively higher than naive completion when & increases, as its input is not constrained by the buggy partial code prefixes.
The performance of rewriting povriting-then-eee completion achieves relatively big increases among the methods as goes up. This is probably because rewriting-then-comee comm-ccmplttion balances the completion completion-then-reeee then-rewriting and completion rewriting step increases the performance along with the diversity of code prefixes provided in the completion step.
approaches. Thus, the pre-
C.3 Performance on CODEGEN-1GB-MONO
provide results for CODEGEN- MONO in table As we can see, BODEGEN-16B-MON
also exhibits a similar phenomenon to our observations in previous Table | for smaller models, i.e. potential bugs make Code-LL.Ms more challenging to generate correct programs and proposed mitigation methods still suffer from large gap to the reference-prefix completion.
21

---

Table 7: Passê1 (1) of completion methods with HADDGGE--16B-MON *Best method in each column is in bold.
Prefix
Method
buggy-FixEval
reference
completion
68.9
49.4
completion
4.4
8.0
buggy
removal whwnttheepppleee completion
20.2
17.6
rewnting-then--compllii
24.3
9.7
completion--heo--eewriingg
24.1
7.7
C.4
Ablation Study: Concatenating Buggy-based Completion with Reference Prefix
In this ablation study, we use the buggy prefix to complete, then replace the buggy prefix with the reference prefix, shown in the bottom of Table (reference prefix + auggy-based completion). The results indicate that Code-LLMs fail to react to the change in the buggy prefix, as discussed in Section 4.2
Table 8: Concatenating buggy-code-basee completion with reference prefix.
buggy- Huggy-HumanEver
buggy- FixEEval
CODEGEN-
2B
350M
2B
350M
reference prefix reference-based completion
54.9
37.8
27.6
buggy prefix + buggy-based completion
3.1
0.7
4.3
2.4
reference prefix buggy-based completion
40.8
31.6
4.6
2.5
D Detailed Case Studies
For the case studies, We the synthetic buggy- HeegyyHuumaEEvall dataset for our study. We use CODEGEN- 2B-MONO as the completion model throughout this section as it achieved the best bCC performance overall. We surface interesting bCC examples by comparing pass rates for reference and buggy prefixes. For example, the Code-ILLM may easily complete some reference prefixes (i.e., high pass rate) but potentially fails with buggy prefixes (zero pass rate). Other examples may have non-zero pass rate, meaning that the Code-LLM can adapt to the potential bug in the prefix and yield a correct solution. One observation that was observed uniformly about this study is that prefixes with potential bugs typically lead to lower pass rates, even when those were non-zero. This suggests that for these prefixes, the Code-LLM, has make more effort to find correct solution.
D.1 Are Potential Bugs Always Harmful?
As discussed Section 2 potential bugs do not guarantee that the completed code will be buggy. These prefixes were soureed from completed examples that had bug that could be ascribed to the prefix, but that does not mean by itself that the prefix will be impossible to complete correctly. While observe the performance degradation of Code-LLMMs under the presence of potential bugs, find several interesting cases where these models generate correct code nonetheless.
Figure 12 shows that for the potential bug (highlighted operator) == modified from != in the reference code, the completion model updates its original algorithmic flow with continue command and completes with correct code. While the completion code is different from the canonical solution of the reference code, it is functionally correct. Thus, this implies that codellms may adapt in some recoverable cases.
D.2 Why Do Code-LLMs Fail at bCC?
We investigate the cases where our models succeed with the reference partial code but fail with the buggy partial code.
22

---

Problem specification
".B You're given list of deposit and withdrawal operations bank account that starts with
zero balance. Your task is to detect if at any point the balance of account fallls below zero, and that point function should return True. Otherwise it should return False. >>> belox_zero 2, 3])
False
>>> below_zeros 2, -4, 5])
True unB
Prefix with potential bugs
Prefix without potential bugs
from typing import List def below_zers (operations:
from typing import List def below_zeroe (operation:
List[int]] bool: balance - 0 for in operations:
List [int]) bool:
balance 0 for op in operations: balance += op if balance 0:
balance += op if balance 0:
Model completion
Model completion
return True
return True return False
else:
balance op if balance -1:
return False
return True
Figure 7: An example case of buggy-code completion where the model reacts to the code change but still fails. The bCC instance is based on HumanBval/3, The model completions shown here are representative completions for the given prefixes.
Our first finding is that the model often fails to react to the changes code, i.e., it produces the same completion for both buggy and reference code prefixes. We identify this by comparing if the most popular completions (out all the sampled completions for given instance) are the same for the reference and buggy code prefixes. We see that this happens in 90% of the instances and 93% of the problems (with at least one failed instance). Figure 7 shows an example, This suggests that the model not sensitive to and thus ignores minor code changes and/or defaults common patterns in the training data.
In cases where the most popular completions differ for the reference and the buggy code prefixes, we found that the potential bug often makes the code prefix significantly more challenging to complete correctly. In other words, the model may have recognized the potential bug and thus have significantly changed the output distribution but still failed. Figure g shows an example. Here, the balance check condition is reversed due to potential bugs. The model could ignore the given code even define a new function to overwrite the buggy one. However, this drastically deviates from the commonly observed code patterns and thus may create significant hurdle for the completion model if it is not introduced or adapted to such cases.
D.3
When Do Code-ILMSs Succeed at bCC?
Figure shows an example when Code-LLMs ignore if-else statement to bypass the potential bugs. In successful cases, we have observed that either the model ignores the incorrect state and generates the correct completion or considers the potential bug in the prefix to create a completion adapted to it.
23

---

Problem specification
"n Check if in given list numbers, any numbers closer
to each other than given threshold.
>>> caa_ccloe_eeemente [1.0, 2.0, 3.0], 0.5)
False
>>> has cass_close___ ailose_eleeents [[1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
unn
Prefix with potential bugs
Prefix without potential bugs
from typing import List
from typing import List
def has cas_close_ellee eereeteseenssseeeee (numbers:
def has_ cas_el el ements (numbers:
List[float], threshold: float)
List [float], threshold: float)
-> bool:
-> bool:
for idx, elem in enumerate (numbers):
for idx, elem in enumerate (numbers);
for idx2, elem2 in enumerate (numbers);; if idx != idx2: distance abs (elem elem2)
for idx2, elem2 in enumerate (numbers):
if idx == idx2: distance abs (elem elem2)
Model completion
Model completion
if distance < threshold: return True return False
if distance threshold: return True return False
Figure 8: An example case of bCC where the model fails to react to the code change. The bCC instance based on HumanEval/0 The model completions shown here are the dominant completions for the given prefixes.
D.4 How Are the Proposed Completion Methods Better at bCC?
The
method detect and fix several types of suspicious operators. Figure I0 shows an example from the string_xor problem. Here, we see that the direct completion failed react to the potential bug and continued if the operator was not changed !=, which gives an incorrect solution. However, the code-repair model was able to identify and fix the != operator back ==, producing correct solution. Similarly. Figure TT shows success case from rewriting-ther then-completion, where it restores the introduced operator change. As EorgyyHHumaEEvall consists of bCC instances constructed from single operator changes, these methods can be particularly effective over naive completion in circumventing the introduced operator change.
24

---

Problem specification
".p Check if in given list of numbers, are any two numbers closer to each other than
given threshold.
wwl_ccloseeeeeeeeteeeee ([1.0, 2.0, 3.0], 0.5)
False
>>> has closec celoseeelementaeeeceeeemeeeaaaaa ( [[1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
Prefix with potential bugs
Prefix without potential bugs
from typing import List
from typing import List
def ges__cloe__eeemmeee (numbers:
def has_close eeleetteeeeeeeeeee aess_ltose_elementaa
(numbers:
List[float], threshold: float)
List[floas threshold: float)
bool:
bool:
for idx, elem in enumerate (numbers):
for idx, elem in enumerate (numbers) for idx2, elem2 in enumerate (numbers):
for idx2, elem2 in enumerate (numbers):
if idx idx2: distance abs (elem elem2)
if idx idx2: distance = abs (elem elem2)
Model completion
Model completion
if distance threshold: return True
continue
else:
distance abs (elem elem2)
return Falae
if distance < threshold:
return True
return False
Figure 9: An example case of bCC where the model bypasses the potential bug and succeeds. The bCC instance is based on HumanEval/o, The model completion for the reference prefix is the most common. The model completion for the buggy prefix is one of the two successful completions.
25

---

Problem specification
"*. Input are two strings and consisting only is 0s. Perform binary XOR on these inputs and return result also as string. >>> string_xor( ("010', "110*) '100' unn
Prefix with potential bugs
Output from completion-thhe--eeriiii
from typing import List
from typing import List
def string_xor (a: str, b: str) ->
def string_xor str, b: str)
str:
str:
def xor j):
def xor xor j):
if !- ]:
if i == j return 20"
return '0
Model completion
else
return '1'
else: return '1'
return [xorr [xor for i,
zip(a, b)])
return P. (..ooi((xxor.,, j) for i,
([xor j in zip(a, b)1)
Figure 10: An example case of bCC where the completion-then--ewwiing method successfully fixes the potential bug. The bCC instance is based on HumanEval/11, The model completion for the buggy prefix representative.
Problem specification
Check if in given list of numbers, are any two numbers closer to each other than
given threshold.
ces_cclooeeeemette [1..0, 2.0, 3.0], 0.5)
False
>>> has_close close wwt.colelleeettttteeeeeeeeeeeeee [[1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
Prefix with potential bugs
Rewritten prefix
from typing import List def mate_coose_eeeeettsddeeee
from typing import List
elements (numbers:
def casselose__ elements (numbers:
List[ffoot]], threshold: float)
List [float], threshold: float)
-> bool:
-> bool:
for idx, elem in enumerate (numbers):
for idx, elem in enumerate (numbers):
for idx2, elem2
for idx2, elem2 in enumerate (numbers):
enumerate (numbers):
if idx idx2:
if idx idx2 and abs(elem elem2) <
threshold:
Model completion
return True
return False
Figure 11: example case of bCC, where the thantinngtthe--cmmlltiiioo method successfully fixes the potential bug. The bCC instance is based on HumanEval/O.
26

---

Check if in given list of numbers, are two nunbers closer each than given threshold.
from typing inport List def aess_clos_eelmoonssmnmnee
fram typting inport List
List[float], threshald: float) -> bool:
seteeeeeeenttteeee (numbers: List[fleat],
thresheld: float) -> bool:
idx, elen (numerate
idk, elem enumerate (nuuerate
for 1dx2, elen? in (nubbrs)))
for idx2, elen2
anumerate snimerate(nutbers)
idx 1- idx2:
if idx idx2:
distance abs(elen elen2) distance threshold:
continue
elenz) threshold:
return True
return False
return False
Without potential bugs
With potential bugs
Figure 12: A success case of buggy-code completion. The model manages to generate functionally correct completion by surpassing the potential bug (highlighted) by continue command. The completions are from CODEGEN-22 2B-MONO on HumanEval/0 from Eeggy-HumanEval...
27


Reviews:
[{"review_id": "fVHrp67ph4", "reviewer": "Reviewer_2xun", "summary": "This paper introduces a new code-completion problem: potential bugs exist in the completion prefixes.  The authors constructed two benchmark datasets, one is created by manually injecting bugs from the HumanEval dataset, and the other is based on actual bugs in the FixEval dataset. Three baseline methods are proposed, including\n(1) removing the prefix and then generating from scratch,\n(2) completing at first and then repairing the whole solution, and\n(3) repairing the prefix first and then finishing the completion.\nThe performances of different approaches on the two benchmark datasets are analyzed.", "questions": "In Section 3.1.2, when constructing Buggy-FixEval dataset from FixEval dataset, the author split rejected solution in halves and regard their first halves as prefixes containing bugs, thereby creating samples with buggy prefixes. However, How to ensure the first half of the rejected solutions contain at least one bug? Please give more details.", "limitations": "(1) When constructing a dataset whose prefixes have potential bugs, the authors select the first half of rejected solutions in the FixEval dataset as prefixes with bugs. However, this process cannot ensure that the prefixes contain bugs. For example, bugs may exist in the second half of the solution.\n  \n(2) I am unsure if applying such three baseline methods to code completion will hurt the clean prefixes. The authors did not try to interpret this problem.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The author proposes a new, challenging, and practical scenario in the field of code completion - how to complete code completion tasks in scenarios with potential errors in the context.\n\n(2) Based on the setting of the paper, the author constructed two datasets based on HumanEval and FixEval as benchmark datasets.\n\n(3) Combined with the code repair model, this paper proposes three baseline methods to deal with the problems in this scenario.\n\n(4) The author conducted a comprehensive analysis of the performance of the baseline method.\n", "weaknesses": "(1) The baseline method provided in the paper is somehow straightforward, mainly a straightforward combination of code completion and repairing.\n\n(2) The repairing process also introduces performance degradation, where a mechanism to judge how to use the code repair may be needed during the code completion.\n\n(3) Although the authors argue that their new scenario is not a simple combination of code repair and completion, their baseline method is still a simple combination of the two tasks.\n\n(4) The authors did not show the repairing performance among the two benchmark datasets. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2xun for the detailed review and constructive suggestions. We appreciate your acknowledgment of the novelty, challenge, and practical setting of our problem and the contribution of new benchmark datasets, baseline methods, and our comprehensive analysis.\nPlease find our answers to your comments and questions as follows. \n\n**Comment:** *proposed baselines are somehow simple, mainly a straightforward combination of code completion and repairing*\n\nThanks for the comment. In fact, \"simplicity\" and \"modularity\" (a combination of existing modules) are our key designs for these baselines for understanding the current limits of code language models.\nThese designs have some advantages: (i) easy to incorporate any LLM with the latest repairing modules without extra training, (ii) flexible to update each model separately, thus enabling more independent tests of the target LLM.\nNevertheless, the rewriting->completion baseline is more sophisticated with an additional algorithm of bug localization.\n\nFurthermore, we are experimenting with two advanced techniques: (i) fine-tuned a CodeT5 pre-trained for neural program repair, and (2) few-shot in-context learning for fixing the program with loop repairing. \nWe hope to release these investigations soon in a further study.\n\n\n**Limitation/Comment:** (i) *unsure if baseline methods hurt the clean prefixes*,  (ii) *\"... may need a mechanism to judge how to use the code repair …\"*\n\nThis is a good point. Indeed, we acknowledged this phenomenon (baseline methods may hurt the clean prefixes) in Section 4.4. (see Tables 2 and 3). Furthermore, we showed our initial effort of controlling when to apply the repairing method by thresholding. For instance, in Table 3, given prior knowledge about the distribution of buggy programs, we may select a proper threshold to balance the clean and buggy settings. \n\nMoreover, we want to emphasize that the primary purpose of these baselines is to study the limits of code language models in our setting rather than **a perfect code completion solution**. Thus, we hope these are considered initial efforts for more advanced study of repairing techniques in future research.\nWe will add more interpretations in the revised version.\n\n\n\n**Comment:** *No repairing performance among the two benchmark datasets*\n\nTo our best interpretation of your comment, 'repairing performance’ means “repairing” without completion.\nIf so, for *completion -> repairing* and *removal* methods, the repairing performance can be only measured by the final performance (pass@k) compared to the buggy-prefix completion, as shown in the manuscript.\nFor the *rewriting -> completion* method, we can provide the accuracy of the line of bug localization as 82% and 53% on buggy-HumanEval and buggy-FixEval, respectively.\nPlease let us know if this answer addresses your comment.\n\n\n**Question:** *In FixEval dataset, how to ensure the first half of rejected solutions contain bugs?*\n\nThanks. This is a good question that sparked our discussion when designing the buggy-FixEval dataset.\nIndeed, we manually inspected all pairs to select the final 292 pairs that satisfy our conditions. \nNote that each pair has already been chosen to be similar (our supplementary A.2, page 2). We further execute the program that concatenates the buggy half with the completion half of the correct code to ensure that it fails at least 1 test case.\n\n---\n**Final notes:** We are grateful that you find our idea novel and challenging. We will integrate our clarification into the final version. We hope our response helps you better understand our paper and consider increasing your score and further support accepting our paper.\n"}, {"review_id": "xAHs8zK4yo", "reviewer": "Reviewer_XA1K", "summary": "Understanding the buggy code completion abilities of Large language models (LLMs) is a vital topic considering the increasing trend of adopting LLMs for programming assistance purposes. This paper studies the buggy code completion problem on different LLMs and datasets and takes a step to investigate three post-hoc methods to mitigate the effect of potential bugs. The authors evaluate the capabilities of four LLMs against buggy code with semantics changes and further compare the effectiveness of three repairing methods from multiple aspects.", "questions": "1. Are there any experiments conducted to evaluate the effectiveness of the likelihood-based measurement on buggy line localization? Is this method effective on both single-line detection and multi-line section? \n\n\n2. Does the constructed buggy prefix in Buggy-FixEval dataset can have semantic changes at multiple spots?\n \n\n3. From Figure 6, are there any thoughts about the Rewriting->Completion method having a relatively better performance at higher k values?", "limitations": "The overall manuscript is sound. More implementation details about the dataset construction (Buggy-FixEva) and repair methods (Likelihood-based measures) can help to enhance the adaptability of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Timely and vital problem.\n+ The overall presentation of the paper is sound.\n+ The experiments are conducted in a good manner.\n\n\n### Significance:\nThis topic itself is motivating as the presence of potential bugs in code completion is common and unavoidable in program development. The authors construct two datasets for buggy code completion, which can be an asset for following research in this domain.\n\n\n### Novelty:\nUnderstanding the capabilities of LLMs on buggy code completion and corresponding repairing methods is novel. \n\n\n### Verifiability and Transparency:\nAlthough the replication is not yet available, the authors provide rich information in supplement material(appendix) to support the findings and discussions in the manuscript.\n\n\n### Presentation:\nOverall, the manuscript is written in a good manner, and I enjoyed reading this paper.", "weaknesses": "- Lack of justifications for some technical details.\n- Some experiment results are not sufficiently explained.\n\nThis manuscript is generally easy to follow; however, some technical detail and experiment results still lack some explanations and justifications. Please see the comments below:\n\n1. The definition of \"potential bugs\" is kind of ambiguous. In line 43, the authors state that \"potential bugs are defined in relation to specific code completions and are not truly bugs per se without completions.\" However, during the dataset construction, the authors make candidate buggy samples by manipulating the semantic opposites, which can raise at least one violation regarding the test cases. These statements can somehow cause confusion about the definition of potential bugs.\n\n2. In addition to the previous comment, the authors mention the injected bugs are not necessarily \"incorrect.\" More justifications are needed to clearly explain the definition of the correctness of the bugs introduced in buggy code completion.\n\n3. For the Buggy-FixEval dataset, the authors leverage a limit on the character-level edit distance between the clean prefix and the buggy one. Does that mean the constructed buggy prefix can have semantic changes at multiple spots? If so, this may raise more challenges for the following Rewriting->Completion repairing method since it contains a buggy line localization process.\n\n4. The Rewriting->Completion needs more details and justifications specifically for the likelihood-based measurement used in buggy line selection. Although Appendix B.1 provides additional information for the likelihood measurement, no evaluation or discussion is found to justify the effectiveness of such measurement for line selection. Although the authors state a  heuristic oracle is used for comparison, limited information is available to give a concrete understanding. To the best of my knowledge, the token-level probability distribution in LLMs comes with relatively high predictive uncertainties, which may not be capable as an indicator for buggy line localization.\n\n5. In the manuscript, the authors report the performance of different repairing methods with the metric of pass@1. From Appendix D Figure 6, I observe some interesting trends at pass@100 different from the performance on pass@1. Specifically, the Rewriting->Completion approach shows equivalently great or even beyond performance compared to the other two methods. The authors only discussed the findings for pass@1 in the manuscript; more insights from Figure 6 can be an asset to enrich the soundness of the paper.\n\n6. In terms of the effect from bug and split locations, more in-depth discussion can help to deliver a better understanding of the characteristics of different repair methods. Namely, the authors may present some thoughts on why the Rewriting->Completion and Completion->Rewriting have distinct performances w.r.t different bug and split locations. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer XA1K for the detailed review and constructive suggestions. We are very excited that you enjoyed reading our work and found our problem timely, vital, and motivating, as well as our manuscript well-written.\nWe also appreciate your acknowledgment of the novelty and contribution of new benchmark datasets and repair methods to the community.\nPlease find our detailed responses as follows.\n\n\n**Comment:** *... These statements (line 43 and data construction) can somehow cause confusion about the definition of potential bugs.\"*\n\nThanks for the comment. \nDuring the construction of buggy-HumanEval, we alter the code prefix so that the prefix becomes buggy, aka fails test cases, with respect to the code suffix, aka the reference completion. The altered prefix is not itself “buggy per se” because bugginess is mostly pointless without a complete program. It IS buggy, however, “in relation to specific code completions” – in this case, the reference completion or the original suffix. \nWe are revising these statements in the updated version.\n\n\n**Comment:** *More justifications are needed to clearly explain the definition of the correctness of the bugs introduced in buggy code completion.*\n\nThanks for a good question. This again relates to the point that it’s pointless to say if the partial code is “buggy per se” or “incorrect by itself”. It can be incorrect with one completion but at the same time correct with a different completion. This ambiguity is inherent and is what exactly makes the problem difficult and interesting. \nWe will add more justifications and examples to make the explanation more precise.\n\n**Question:** *Does the constructed buggy prefix in Buggy-FixEval dataset can have semantic changes at multiple spots?*\n\nYes, theoretically speaking, the buggy prefix can have more than one semantic change. However, through our manually inspect, most buggy prefix contains only one potential bug, given the programs are mainly short.\nRegarding the rewriting->completion, we heuristically set a priority score for the line that appears early in the program among lines with high probabilities of containing bugs.\n\n**Question:** *Justifications for the likelihood-based measurement used in buggy line selection*\n\n* **Experimental results**: Thanks for the suggestion. The accuracies of localizing the line of potential bugs (with the same setting described in the manuscript) are approximately 82% and 53% on buggy-HumanEval and buggy-FixEval, respectively.\nFor buggy-FixEval, we compare the detected line with the line of the first semantic difference between buggy and clean prefixes.\n\n* **comment 1**: “ … the token-level probability distribution in LLMs comes with relatively high predictive uncertainties, which may not be capable of as an indicator for buggy line localization.”\n\n> This is a good point. Indeed, our algorithm mitigates this variance by (i) instead of the softmax score, we measure the maximal margin between scores of the target token and the argmax token (similar to the popular approaches used in uncertainty quantification), and (ii) aggregate the score gap along the line and set a high threshold (to be more conservative).\nOur underlying idea is to treat the potential bug as the outlier in the generation flow of the LLM, and from our observation that most clean code has lower perplexity scores than the corresponding buggy code,  94% and 70% for HumanEval and FixEval, respectively.\n\n* **comment 2**: “ … Limited information is available to give a concrete understanding of heuristic oracle … “\n\n> Heuristic oracle only serves as a reference to practically compare our bug localization method. \nThe oracle assumes access to the clean prefix to localize the bug line better. For instance, on buggy-HumanEval, it detects the potential bug by string-based comparing the buggy and clean prefixes to find the first difference. \nWe will provide more details in our revision.\n\n**Question:** *3. From Figure 6, are there any thoughts about the Rewriting->Completion method having a relatively better performance at higher k values?*\n\nThanks for a very good catch and suggestion. \nWe hypothesize that higher k results reflect more about the \"diversity\" of the generated completions by methods. In particular, \n* *Removal* has no guidance about the completion, so the model tends to generate more diverse solutions. Thus, pass@1 can be high, but pass@k may be limited.\n* In *completion->rewriting*, only one prefix is used, making it harder for the repairer to fix if the buggy prefix steers LLM in the wrong direction.\n* *Rewriting->completion* balances the two: the rewriting phase first can help fix the prefix while still providing some good guidance to generate solutions that are more precise but less diverse (than removal), thus achieving better pass@k with higher k.\n\nWe will incorporate these discussions in our revised version.\n\n\n**Comment:** *More discussion on bug and split locations. Why the Rewriting->Completion and Completion->Rewriting have distinct performances w.r.t different bug and split locations?*\n\nIn Figure 3d, rewriting->completion achieves higher scores when potential bugs appear later. In contrast, completion -> rewriting (Figure 3c) achieves high scores when the code preﬁx is longer, indicating that this method can ﬁx potential bugs that appear early in the program. We suspect that this is because longer prefixes restrict more flexibility of the completion, making the generated completion less likely to deviate from the “mode” completion – the completion as if the injected bug was not there in the prefix. In this case, the completed code, aka the input to the subsequent code repairer, better resembles the kind of input the repairer is trained on and thus makes the repairer more likely to repair the program successfully.  \n\n---\n**Final notes**:  We are excited that you find our paper sound, and thanks again for carefully reading our work and for providing encouraging feedback!\n"}, {"review_id": "tGtCMR3NIG", "reviewer": "Reviewer_h8sG", "summary": "This paper identifies and analyses a limitation of the LLMs for code, that they fail for completing code with potential bugs. This is a problem because LLMs are deployed in real world software projects which may contain many bugs. The authors attempt to bridge this gap between test environment and real world by creating buggy versions of two popular datasets used for testing LLMs code generation capabilities. They experiment with two models, with two sizes each and show how their performance is impacted by introducing potential bugs in the test dataset. The authors also try different techniques of addressing the limitations of LLMs when trying to complete potentially buggy code.", "questions": "Do you think a much larger model will have the same limitation? Do you plan to experiment with larger models?\n\nFor code completion applications deployed in IDEs, rewriting user code is generally not expected. How does this limitation, and the fact (which you have mentioned in the paper) that it may be impossible for the model to avoid some potential bugs, affect the scope of the problem?\n\nMore generally, how do you expect code completion applications to deal with buggy code? ", "limitations": "The authors should write more about the scope of the problem and it's impact on code completion applications.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have highlighted a relevant problem with LLMs. It is a well written paper and the authors demonstrate a good understanding of the problem. They consider many different possibilities when generating the data and run thorough experiments covering many scenarios. They also explore intuitive solutions to overcome the limitation of LLMs that they have highlighted.\nThe strengths of the paper are:\n1. The paper highlights an interesting and a very real problem.\n2. The suggest good post-hoc strategies and cover some of the gap in the model performance introduced by the buggy dataset.\n3. They experiment with multiple datasets. \n4. They experiment with multiple open source models of different sizes.\n5. They will make the new buggy version of the dataset available to public.", "weaknesses": "The paper has the following weaknesses:\n1. The scope the problem highlighted is very narrow. It is limited to code generation, and even within code generation it is limited to code completion. It does not even apply to natural language to code generation. This is completely because of the nature of the problem highlighted.\n2. They have experimented with multiple models of different sizes, but they did not try a very large LLM. As we know that as the model size increases, new capabilities get added to the model. This leaves the question open as to whether a much larger model, like 16B parameter CodeGen, would exhibit the same limitation or have the same gap in performance after post-hoc techniques. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the detailed review and constructive suggestions. \nWe are encouraged that the Reviewer finds our problem interesting and practical, and our paper is well-written and well-demonstrated. \nWe appreciate your acknowledgment of the contribution of new benchmark datasets and repair methods to the community.\n\n**Question:** *Do you think a much larger model will have the same limitation? Do you plan to experiment with larger models?, like the 16B parameter CodeGen*\n\nThanks. To better answer your question, we have conducted extra experiments with CodeGen-16B-mono.\nWe notice that, due to the limited time and computation, we provide the results with a subset of the original dataset instead. We will continue our experiment and incorporate results on all datasets in the final version.\n\n|buggy-HumanEval (200 instances) | pass@1| pass@10|pass@100|\n|:---:|:---:|:---:|:---:|\n|Clean prefix| 78.5 | 93.1 | 95.7 |\n|Buggy prefix| 8.9 | 20.4 | 25.2|\n|Removal| 43.2 | 70.9 | 80.1 |\n|Completion->rewriting| 14.4 | 29.7 | 35.6|\n|Rewriting->completion|70.9 | 80.2 | 86.9 |\n\n\n|buggy-FixEval (100 instances)| pass@1| pass@10|pass@100|\n|:---:|:---:|:---:|:---:|\n|Clean prefix| 59.7 | 72.6 | 75.7 |\n|Buggy prefix| 8.5 | 18.8 | 37.7 |\n|Removal| 20 | 49.8 | 68 |\n|Completion->rewriting| 8.6 | 18.4 | 37.7|\n|Rewriting->completion|8.9 | 19 | 32.9 |\n\nAs we can see, CodeGen-16B-mono exhibits a similar phenomenon to our observations in the manuscript (with smaller versions), i.e., potential bugs make LLM harder to generate correct programs, and post-hoc techniques still suffer from a large gap to clean-prefix completion.\n\n**Question:** *How do the practice (rewriting user code is generally not expected) and the unavoidability of potential bugs affect the scope of the problem?*\n\nGreat point! We do not commonly expect IDE smart tools to rewrite or change user code when recommending completions because no one has implemented such. \nWe believe the work-in-progress code is draft, less refined, and more error-prone. Thus, it should be viewed as a hint of user intent rather than a golden one. From this view, it follows naturally if a pair programmer or a smart tool identifies or believes a certain part of the partial code is not what the developer meant, they should suggest the developer change their draft code rather than blindly continue it. From a user experience perspective, the IDE can display code change suggestions to a user’s existing code if the smart tool believes there is an error and some part of the code should be changed to something else. Similar functionality already exists for other types of code change suggestions, such as spelling correction suggestions or missing import suggestions. \n\n**Question:**  *More generally, how do you expect code completion applications to deal with buggy code?*\n\nThanks for a great question. \nThe response to the previous points described a potential user experience with smart completion tools. On the modeling side, we would like to see code completion solutions take more holistic views of the program specification, user intent, code and project context, and similar code and coding conventions into consideration rather than the narrow task of predicting what follows next from a string of code. This work is motivated by treating partial code not just as code context but as a hint of user intent. We explore the impact of “anti-patterns” induced by errors or drafts and present results on how we can possibly do better if we allow completion solutions to deviate from given code prefixes. \n\n—\n**Final notes**: We would like to thank the Reviewer for your constructive comments and appreciation of our findings. Per your suggestion, we will incorporate new experiments on a very large model and our answer about the scope of the problem and its impact on code completion applications. We hope our response makes you consider increasing your score and further support accepting our paper."}, {"review_id": "aY0eC0bLfa", "reviewer": "Reviewer_Cn5Q", "summary": "The paper studies the ability of large language models for code to handle prompts which contain “buggy” code. To this end, it introduces two datasets: (i) buggy-HumanEval, which is constructed by injecting synthetic wrong binary operator bugs into HumanEval problems, and (ii) buggy-FixEval, which is constructed from real bugs in user submissions to competitive programming tasks. The evaluation finds that “buggy” prompts significantly degrade LLMs’ performance. The paper also investigates mitigation methods for “buggy” prompts.", "questions": "Please consider addressing the points raised in the “Weakness” section.", "limitations": "The paper provides a sufficient discussion of limitations and potential negative impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The strengths of the paper are as follows:\n1. The paper is well-written and easy-to-follow. It provides useful examples for illustration.\n2. The paper introduces two new datasets, which can be useful for future research. \n3. The evaluation is thorough and provides new insights.\n", "weaknesses": "### Main weaknesses\nDefinition 2.1 is a core definition in the paper but has two major issues. First, defining bugginess over code prefixes sounds incorrect to me, because bugginess is a property for complete programs. Second, the original prefix $s$ can also be viewed as a “buggy prefix”: there likely exists $c’$ such that $t’ = s’ :: c’$ satisfies $h$ and $t = s :: c’$ does not satisfy $h$. In fact, the paper shows the existence of such cases (e.g., Figure 12).\n\nConsider a “buggy prefix” $s’$ and a program $t’ = s’ :: c’$. Under what condition exists $c’$ such that $t’$ satisfies $h$? Or when is $t’$ unlikely to satisfy $h$? For these two questions, the paper provides some empirical evidence (e.g., Figure 12 and Lines 640-642) but lacks a theoretical analysis.\n\nA highlight of the paper is the discussion over synthetic and real bugs (buggy-HumanEval vs. buggy-FixEval). However, the paper does not capture data imbalance in real bug distribution. That is, bugs occur very infrequently in practice, as shown in [1] and [2] below (not discussed in the paper). Ignoring such data imbalance can threaten the validity of some results. For example, Tables 2 and 3 are obtained on datasets with a balanced ratio of “buggy” and “non-buggy” prefixes.\n\n- [1] He and Vechev, On Distribution Shift in Learning-based Bug Detectors, ICML 2022.\n- [2] Karampatsis and Sutton, How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset, MSR 2020.\n\nFor buggy-FixEval, the absolute improvement from rewriting->completion or completion->rewriting is small, suggesting that these methods might be of limited practical effectiveness.\n\n### Other questions:\n1. In Table 1, the performance of removal->completion on buggy-HumanEval is very low. This is surprising to me, because I would imagine the performance to be much better, if the original HumanEval prompt is used as input (as in Figure 5(a)). How do you perform the removal exactly?\n2. In Section 4.3, the results of completing “clean prefixes” are treated as the upper bounds for completing “buggy prefixes”. This does not sound correct to me. Again, I think the correct upper bound is 100 minus the percentage of “buggy prefix” $s’$ for which no program $t’ = s’ :: c’$ satisfies $h$.\n\n### Small issues:\n1. Line 235: I could not find Appendix C.3.\n2. Line 665: figure 8 -> Figure 8.\n3. Table 5 is not referenced in Appendix B.3.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Cn5Q for the detailed review and constructive suggestions. \nWe are encouraged that the Reviewer find our paper well-written and easy to follow with illustrated examples.\nWe appreciate your acknowledgment of the thorough evaluation of our insights and the contribution of new benchmark datasets.\n\n\n**Comment:** *bugginess is a property for complete programs.*\n\nThanks. We completely acknowledged this property of bugs in the manuscript that “bugginess” is a property of complete programs, not code prefixes (and the bugginess of a code prefix can only be assessed in conjunction with a completion.) \nTherefore, our work referred to the studied patterns as \"potential bugs\", not “bugs” per see. \n\nFor brevity, we sometimes drop “potential” or “potentially” when mentioning potential bugs and potentially buggy prefixes when the context is clear. We will make this point clearer near the definition. \n\n\n**Comment:** *The original prefix can also be viewed as a “buggy prefix”....  For these two questions, the paper provides some empirical evidence ... but lacks a theoretical analysis.*\n\nThanks for a great comment.\nGenerally speaking, the bugginess level of a prefix is relative to w.r.t. a reference prefix; thus, our definition means a prefix potentially buggy is w.r.t. a  reference prefix (which we name a “clean” prefix). Specifically, “potential bug” as some pattern occurring unnaturally w.r.t to a naturally written program. \nHowever, under current definition in manuscript, it is indeed a possibility that the original prefix can also be viewed as a “buggy prefix”.\nWe will refine these terms (clean as reference prefix, and buggy as relative/potential buggy).\nHere, we provide a detailed clarification and hope this helps Reviewer better assess our intuition.\n* Intuitively, the reference program t:= s::c captures the natural distribution of programs that correlate s to preferred implementation for certain programming tasks, e.g., functionally correct, easy to read, follows software development best practices, etc. So, it’s likely that deviations from it can be probably bad, especially when they introduce functional incorrectness. \n* A more holistic definition would be to quantify naturalness and various aspects of preferred quality traits and define the “potential bugginess” as a likelihood value. However, quantifying the property and distribution of counterfactual implementation t’ and completion c’ is inherently difficult, if not impossible. Even full-scale empirical analysis would require significant effort in collecting and reviewing cases manually. We provide empirical observations with model completions. \n\nWe go for the current operational definition for its simplicity and verifiability. It suffices to guide us through the initial exploration of the impact of this previously under-investigated scenario. \nNevertheless, we will refine the terminologies (buggy, clean) to reflect our definition and intuition precisely.\n\nRegarding the analysis, our work aims to answer these two questions empirically first for further theoretical analysis in future studies.\n\n\n\n\n**Comment:** *Imbalance in real bug distribution: Ignoring such data imbalance can threaten the validity of some results.*\n\nThanks. Our work identifies and focuses on studying the behavior of LLMs in a setting with potential bugs. How often do potential bugs appear in actual code development when a user triggers auto-completion is indeed an open question and worth studying. It is however beyond the scope of this study. We provide preliminary observations on balancing the performance on potentially buggy and clean prefixes in Tables 2 and 3. We hope our study will raise awareness about the existence of potential bugs and call for a method that deals well with whether the potential bug exists or not.\n\n**Comment:** *methods might be of limited practical effectiveness*\n\nThanks. Our proposal for these simple strategies is to help understand the limits of current LLMs instead of an advanced technique. \nWe hope our study can motivate further study of advanced methods.\n\n**Q:** *How do you perform the removal exactly?*\n\nFor the *removal* method, we remove the partial code from the prefix to keep only the problem statement and the function header (with optional examples), so the input of removal is as same as the prompt in the original HumanEval.\nWe note that: \n* We double-checked that on the HumanEval, our removal method reproduces the pass@k performance of Codegen models reported in [Nijkamp et al., 2022]. \n* buggy-HumanEval is derived from a subset of considerably longer and harder problems in HumanEval. In particular, the average number of lines of our selected problems is 8.2 compared to 2.9 of the rest, resulting in much lower pass@k scores in our subset.\n\n\n\n**Q:** *The correct upper bound is 100 minus the percentage of “buggy prefix” s' for which no program t' = s'::c' satisfies h*\n\nThanks. We agree that the term \"upper bound\" here is improperly used. We will refine the term 'upper bound' to “expectation from reference prefixes”'.\nWhile the suggested bound is theoretically the upper bound, an empirical estimate seems not plausible because of the number of such buggy s'.\nThus, we use the results on the \"clean prefix\" as expected to evaluate how well the model can achieve with the corresponding \"buggy prefix\".\n\n**Minor:**: Thanks. We fixed these issues.\n\n---\n**Final note:** We want to thank you again for such exemplary and detailed comments. We will integrate these answers into our new version. We hope our responses can help you better appreciate our work and consider increasing your score and supporting accepting our paper. We agree that our current definition has some ambiguity though it serves the purpose of empirical verification. Still, with its timely development, novelty, significance, and potential in future research, we believe our work can have a good impact on the research society. Thanks again for your careful reading!\n\n\n"}, {"review_id": "Jzwp6iPujF", "reviewer": "Reviewer_8UBb", "summary": "In general, the code completion task that is commonly used to evaluate LLMs assume a clean, non-buggy starting point. However, in real-world software development, this assumption is not always valid. In this papers, the authors investigate code completion given a buggy prefix. For this, they derive two benchmarks from existing datasets. Namely, they build buggy-HumanEval by introducing semantics-altering operator changes in the HumanEval dataset and buggy-FixEval based on user submissions in coding contests. They find that performance is lower in the presence of a buggy prefix, and they introduce different techniques for mitigating this to some extent.\n\n\nContributions:\n- Evaluate a handful of Code LLMs on code completion with a buggy prefix.\n- Introduce two benchmarks that will be publicly released.\n- Investigated different post-hoc mitigation techniques which demonstrate some improvement in performance.", "questions": "- Please clarify the point I raised in the section above. \n- The authors claim multiple times that code repair on short context is an ill-defined problem. No references are provided here, and actually, people have studied such tasks in the literature (e.g., bug-fixing with a single statement such as ManySStuBs4J or handful of statements such as the Tufano et al., BFP dataset). Could the authors provide an explanation for their argument?\n\n- When introducing new benchmarks, it is good to provide dataset statistics (e.g., average lengths).\n- Missing reference: “Learning to Extend Program Graphs to Work-in-Progress Code” (they consider code completion on work-in-progress code) [Li, Maddison, and Tarlow; 2021]\n- Consider placing Figure 2 closer to the results.\n- It would be interesting to run your experiments on GPT-3.5 or GPT-4.\n", "limitations": "One limitation that the authors have not mentioned is the assumption that FixEval closely aligns with a real-world scenario. Coding contest problems do not necessary align with a more general software engineering setting. However, it is challenging to get a test suite for such a setting and so it is more difficult to evaluate.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors introduce two useful datasets, with synthetic and real examples, that could serve as benchmarks of buggy code completion for the research community.\n- The task of buggy code completion is quite interesting and also important, though it may not be completely novel, as there is prior work (“Learning to Extend Program Graphs to Work-in-Progress Code”) outside the context of LLMs that studies a similar setting.\n- The post hoc mitigation techniques are useful in understanding the current limits.\n- There are nice ablation studies and qualitative analysis with interesting findings. Specifically, it’s cool to see that sometimes the completion changes to adapt to the potential bug.\n", "weaknesses": "- To run the test suite, both the prefix and predicted suffix are needed. The bug is in the prefix that is given to the model (not generated by the model). So, it is possible that tests fail only because of the bug in the prefix and not because of anything the model generated. In lines 96-101, the authors explain that they loosen the constraints such that they allow the model to make changes to the prefix; however, this is an additional bug fixing task that the model needs to perform, on top of code completion. In other words, in the buggy code completion setting, there are two possible ways to pass the tests. (1) Fix the bug in the prefix and then complete or (2) Try to complete the code by adapting to the buggy code. Therefore, it doesn’t seem fair to compare directly with completion only based on a clean prefix. Since the main claim of this paper is that the task of code completion suffers as a result of buggy prefixes, I feel that it is important to evaluate the completion component alone. Would it be possible to decouple things a bit? For example, when running the test suite only, would it be possible to replace to buggy prefix with the clean prefix to evaluate whether the completion would have been correct if the prefix (which is not model generated) was correct?\n- For the post-hoc mitigation techniques, the rewriting uses a BERT-based model which is a considerably weaker model, and even though some code pre-training is done on top of it, it likely does not reach the level of modern code LLMs. Perhaps using a stronger model would have yielded better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the detailed review and constructive suggestions. \nWe are encouraged that the Reviewer finds our task interesting and important with our studies and analysis providing interesting and cool findings.\nWe appreciate your acknowledgment of the thorough evaluation of our insights as well as the contribution of new benchmark datasets and post hoc mitigation techniques. \nPlease find our answers to your comments and questions as follows.\n\n\n**Comment:** *It is possible that tests fail only because of the bug in the prefix and not because of anything the model generated.*; *it doesn’t seem fair to compare directly with completion only based on a clean prefix.*; *Would it be possible to decouple things a bit?*\n\nThanks for your suggestion. \nFirst, we note that:\n* The performance with “clean prefix” is considered a reference to see how potential bugs affect the LLM rather than a direct method for comparison, so our purpose is to measure the gap between scores when using buggy and clean prefixes to relate how well models react to the change.\n* The “potential bugs” in buggy prefixes do not necessarily make the model fail. Instead, a good model should manage to complete buggy prefixes with a suitable completion to satisfy the test cases. We showed these interesting examples in Section 4.6. \n\nPer your suggestion, we have evaluated and reported the following results (pass@1 score):\n\n|buggy-| HumanEval|HumanEval | FixEval| FixEval|\n|:---:|:---:|:---:|:---:|:---:|\n|codegen- | 2B-mono|350M-mono|2B-mono|350M-mono|\n|Clean prefix + clean-based completion| 54.9|43 | 37.8|27.6 |\n|Buggy prefix + buggy-based completion| 3.1| 0.7| 4.3| 2.4|\n|Clean prefix + clean-based completion| 40.8| 31.6| 4.6| 2.5| \n\nHere, the bottom row (Clean prefix + clean-based completion) is when we use the clean prefix and the completion of the buggy prefix. As we can see, the result reflects that models fail to react to the change in the buggy prefix, as discussed in Section 4.2 (*Why do Code-LLMs fail at bCC?*).\n\n**Comment:** *For the post-hoc mitigation techniques, the rewriting uses a BERT-based model which is a considerably weaker model ...  Perhaps using a stronger model would have yielded better performance.*\n\nIn the rewriting method, we use Incoder [Fried et al., 2022] because, to the best of our knowledge, it is the latest and the best LLM trained for infilling tasks — suitable for fixing bugs in the middle of a program.\nIn fact, Incoder is both (left-to-right generation) as well as editing (via masking and infilling).\nNevertheless, our design allows us to easily replace Incoder with a better infilling model in the future for better repairing performance.\n\n*[Fried et al., 2022]. Incoder: A generative model for code inﬁlling and synthesis. arXiv preprint arXiv:2204.05999, 2022\n\n\n**Q:** *The authors claim multiple times that code repair on short context is an ill-defined problem. … Could the authors provide an explanation for their argument?\"*\n\nThanks for your question. In fact, our statement is that bugginess is only meaningful and a property of a **complete program**; thus (in line 31), we meant that given **the incomplete and short** code context* (i.e., without a full program), it’s ill-deﬁned for detecting or fixing bugs.\nThanks for your reference, In [Tufano et al., 2019], we believe while the context is short, it’s a full program, e.g., JAVA programs in Figure 2.\nWe will revise the statement to make it transparent.\n\n\n**Q:** *When introducing new benchmarks, it is good to provide dataset statistics (e.g., average lengths).*\n\nThanks. We provided some detailed statistics of datasets in the supplementary (Section A). \nPer your suggestion, we also report the length statistics (in terms of the number of tokens) as follows.\n\n|Percentiles|50th|90th|95th|98th|99th|100th| \n|:---:|:---:|:---:|:---:|:---:|:---:| :---:| \n|buggy-HumanEval|214|470|550|595.6|606.7|617| \n|buggy-FixEval|171|242.7|262|330|402.4|566| \n\nWe will add more statistics in the final version.\n\n**Q:**  *Missing reference: “Learning to Extend Program Graphs to Work-in-Progress Code” (they consider code completion on work-in-progress code) [Li, Maddison, and Tarlow; 2021]*\n\nThanks for bringing this work to our attention. We note that this work focuses on the transfer learning of token-relation prediction for code completion under work-in-progress code scenario, different from our setting and no-training post-hoc techniques.  We have added this work to the related work.\n\n**Q:**  *It would be interesting to run your experiments on GPT-3.5 or GPT-4.*\n\nThanks for your suggestion. Due to the limited time and flexibility, we have conducted extra experiments with a very large model CodeGen-16B-mono instead of GPT-4.\nWe are also planning to run our experiment on GPTs and incorporate results in the next version.\n\nHere are the results on CodeGen-16B-mono\n|buggy-HumanEval (200 instances) | pass@1| pass@10|pass@100|\n|:---:|:---:|:---:|:---:|\n|Clean prefix| 78.5 | 93.1 | 95.7 |\n|Buggy prefix| 8.9 | 20.4 | 25.2|\n|Removal| 43.2 | 70.9 | 80.1 |\n|Completion->rewriting| 14.4 | 29.7 | 35.6|\n|Rewriting->completion|70.9 | 80.2 | 86.9 |\n\n|buggy-FixEval (100 instances)| pass@1| pass@10|pass@100|\n|:---:|:---:|:---:|:---:|\n|Clean prefix| 59.7 | 72.6 | 75.7 |\n|Buggy prefix| 8.5 | 18.8 | 37.7 |\n|Removal| 20 | 49.8 | 68 |\n|Completion->rewriting| 8.6 | 18.4 | 37.7|\n|Rewriting->completion|8.9 | 19 | 32.9 |\n\nAs we can see, CodeGen-16B-mono exhibits a similar phenomenon to our observations.\n\n**Minor**: \nThanks. We revised.\n\n**Limitation:** *Coding contest problems do not necessary align with a more general software engineering setting*\n\nThanks. We will elaborate on this limitation in our manuscript.\n\n---\n**Final notes:** We hope our responses and new experiment results can help you better appreciate our work and consider increasing your score and supporting accepting our paper. Thanks again for providing valuable feedback!"}]

Author rebuttal:
We are thrilled that the reviewers appreciate our paper, and we want to extend our gratitude for their insightful feedback and constructive comments. These invaluable suggestions will undoubtedly enhance the quality of our work. First and foremost, it's encouraging to note that the reviewers have identified the following:
* (i) Our proposed problem is important, interesting, and practical (R-2xun, R-XA1K, R-h8sG, R-8UBb) with a novel setting (R-2xun, R-XA1K).
* (ii) Our evaluation is thorough, comprehensive, and well-conducted (R-XA1K, R-h8sG, R-Cn5Q) with rich insights and findings (R-2xun, R-XA1K, R-Cn5Q, R-8UBb).
* (iii) Our new public benchmark datasets are helpful for future research (R-2xun, R-h8sG, R-Cn5Q, R-8UBb).
* (iv) The mitigation techniques are intuitive and helpful in understanding the current limits (R-2xun, R-h8sG, R-8UBb).
* (v) Our paper is well-written and easy-to-follow (R-XA1K, R-h8sG, R-Cn5Q) with useful examples for illustration (R-Cn5Q).

We believe we have clarified and addressed all questions with our best. Please find our answers in each reviewer's section.


Meta-review: The paper received 5/6/7/4/7 ratings. Reviewers acknowledged the paper is dealing with a timely code completion problem existed in the field. While two reviewers were concerned about the definition of the problem, those concerns were partially resolved during rebuttal discussion. One Reviewer found the experimental evaluation did not follow natural code bug distribution (imbalanced data). The AC agreed the problem exists, but it was also a common practice to show numbers on balanced data to avoid performance numbers being shadowed by the dominantly distributed category. After reading the paper, the reviews and rebuttal discussion. The AC stands with the majority votes and believes the the community will benefit from the paper.
Decision: Accept (poster)