SlowGuess/NeurIPS_2023_OCR huggingface dataset:
temperature  $= 0.2$  for INCODER. Based on the reference solutions and computing efficiency, we set the maximum length limit for outputs from 200 to 600 tokens, varying with the problem sets. We observed similar performance trends between the tested models across different length settings. We post-process the output string following the same procedure used in their code releases.
Code-repair and code-rewriting models. For removal-then-completion, we use the latest model of RealiT [32] as the code-fixer, which is trained and tested over artificial and realistic bugs. For the code-rewriting model, we use the INCODER-6B model [33] due to its ability to code infilling and apply the similar settings used for infilling reported in the model's paper [33].
4.2 How Well Do Existing Code-LLMs Perform on Buggy-Code Context?
We evaluate the latest Code-LLMs for buggy-code completion on buggy-HumanEval and buggy-FixEval, shown in Figure 2. In particular, the performance is measured in terms of pass@1, and we use four models: CODEGEN-350M-MONO, CODEGEN-2B-MONO, INCODER-1B, and INCODER-6B. First, comparing the scores between reference and buggy partial code for each model, we see that the presence of potential bugs is detrimental to the completion models, with pass@1 drops from  $41.1 - 54.9\%$  to  $0.5 - 3.1\%$  over buggy-HumanEval, and from  $24.1 - 37.8\%$  to  $1.2 - 4.3\%$  over buggy-FixEval. Moreover, the pass@1's upon buggy partial code is universally dominated by those without partial code:  $3.9 - 9.3\%$  over buggy-HumanEval,  $2.9 - 8.6\%$  over buggy-FixEval. Table 7 (Appendix C) shows the similar findings with a very large model (CODEGEN-16B-MONO). These results indicate that (i) the tested Code-LLMs drastically fail at bCC instantiated by our datasets, and (ii) the presence of potential bugs destroys the benefit brought by the partial code.
Why do Code-LLMs fail at bCC? We manually inspect samples from the best-performing CODEGEN-2B-MONO model and find the two most common failure modes among failed samples. First, the model fails to react to the potential bugs (i.e., common completions remain the same), as shown in Figure 8. This mode happens in  $90\%$  of instances and  $93\%$  of problems with at least one failed instance. We conjecture that the model is not sensitive to and thus ignores minor code changes and/or chooses to default to common patterns in the training data. Secondly, the model fails to bypass the potential bugs, likely because such patterns are rare in high-quality code. In other words, the model might have recognized the potential bugs and significantly changed the output distribution but still failed. Figure 7 illustrates an example of this case. We provide further details in Appendix D.2.
4.3 How Effective Are Baseline Completion Methods Against Potential Bugs?
We evaluate the completion methods introduced in Section 3.2 using the same bCC setting, shown in Table 1. We include results of completing reference partial code to see how potential bugs affect the Code-LLMs. Figure 6 in Appendix C provides full results and interpretations for  $k = 1, 10, 100$ .
The effect of proposed completion methods. All three proposed methods outperform the naïve buggy-code completion baseline. On buggy-HumanEval, we observe the general trend of completion-then-rewriting outperforming rewriting-then-completion, which outperforms removal-then-completion. Note that these scores of removal-then-completion are generally lower than reported scores of the similar method [12, 33] probably because buggy-HumanEval is derived from a relatively more challenging subset of HumanEval (see Section 3.1.1). On buggy-FixEval, we observe removal-then-completion to outperform rewriting-then-completion, which outperforms completion-then-rewriting. The performance gap increases as the size of the completion model rises. Similar comparison trends are observed for pass@k for  $k = 10, 100$ . Nevertheless, performance gaps remain significant between the best method and the completion from the reference code for all settings.
The effect of Code-LLMs' capacity. For each type of code completion model, the larger version performs better than the smaller version using the same method. For instance, CODEGEN-2B-MONO outperforms CODEGEN-350M-MONO for all settings in two datasets. Compared to CODEGEN, INCODER models, in general, obtain better pass@1 but worse pass@10 and pass@100. This suggests that CODEGEN models generate more diverse completions, while INCODER models generate more precise completions. Furthermore, INCODER is more sensitive to buggy partial code than CODEGEN models, evidenced by the lower scores from naive bCC.
Table 2: Comparing pass@1 (↑) of completion methods on reference and buggy partial code. While the proposed mitigation methods achieve better performances than naïve completion with buggy prefixes, they may harm the completion when no bug exists (reference).
Dataset	buggy-HumanEval	buggy-FixEval
CODEGEN-	2B-MONO reference	buggy	350M-MONO reference	buggy	2B-MONO reference	buggy	350M-MONO reference	buggy
naïve completion	54.9	3.1	43	0.7	37.8	4.3	27.6	2.4
rewriting-then-completion	49.6	24.9	35.9	14.1	37.0	7.2	26.4	2.4
completion-then-rewriting	27.7	23.6	22.2	22.5	19.4	4.7	3.7	2.3
Table 3: Balancing rewriting-then-completion for the buggy and reference settings with bug-detection thresholding. Completion performances can be adjusted via varying bug-detection thresholds from 0 (fix all partial codes) to 1 (keep all partial codes). Results are with CODEGEN-2B-MONO.
buggy-FixEval	threshold
partial-code setting	0	0.3	0.9	1
reference	8.6	14.6	37.0	37.8
buggy	8.6	7.1	7.2	4.3
Synthetic bugs versus real bugs. Among the two bCC datasets, we observe that the overall performance of mitigation methods is better on buggy-HumanEval than buggy-FixEval. This indicates the difficulty of realistic potential bugs in buggy-FixEval: There may be multiple bugs; bugs may be potentially mixed with non-bug-fixing changes; and bugs are more nuanced than single operator changes. Furthermore, while achieving the best performance in most cases, completion-then-rewriting only shows marginal differences from other methods when using larger models on buggy-FixEval.
Take-away: Our baseline methods improve the completion for all evaluated Code-LLMs. However, the remaining performance gaps to the completion with reference partial code are still large.
4.4 What If Partial Code Does Not Have Potential Bugs?
As shown in Table 2, mitigation methods for bCC may harm completion from reference code context (removal-then-completion is the same for both settings, thus not listed.) This suggests that a general code completion should consider both cases when potential bugs may and may not exist.
With our baselines, we can use the thresholding approach for detecting potential bugs. For instance, in rewriting-then-completion, a token is only considered a potential bug if its likelihood gap to the argmax token is beyond a threshold (between 0 and 1). Table 3 compares pass@1 scores of rewriting-then-completion varying thresholds on buggy-FixEval. We can see that the threshold of 0.9 can help achieve a relatively good balance for the two cases. A similar approach can be applied to completion-then-rewriting, as RealiT provides the probability of a token being a bug.
Remark 4.1 (Imbalance in real distributions). We note that distributions of natural code bugs are usually imbalanced as practical bugs occur infrequently [36, 37]. However, potential bugs may occur more frequently as our target setting is a work-in-progress code scenario [38] rather than high-quality code of popular open-source projects in previous studies. Nevertheless, for comparing methods, we evaluate more balanced data to avoid the dominant effect of performance on reference code.
4.5 Analysis of Effect from Bug and Split Locations
To understand how bug location and context size affect completion, we aggregate results by the location of potential bugs and the location of partial code splits within buggy-HumanEval. The locations are normalized as (potential bug line \#)/(# lines) and (split line \#)/(# lines), where potential bug line \#, split line \#, and \# lines are the number of lines starting from the function header to the line containing the potential bug, to the end of the partial code, and to the end of the canonical solution.
Figure 3 presents heatmaps of pass@1 scores (averaged over the bCC instances falling into each cell) evaluated on the CODEGEN-2B-MONO with naive completion on reference HumanEval, and naive
Figure 3: Average pass@1 scores by bug and code split locations using CODEGEN-2B-MONO on buggy-HumanEval. Left to right: (a) naïve completion on reference code, (b), (c), (d) naïve completion, completion-then-rewriting, and rewriting-then-completion on buggy code. Locations are normalized by the length of reference solutions. For each split location in (a), the scores may vary across potential bug locations as the buggy-instance distribution is non-uniform across bug locations.
completion, completion-then-rewriting, and rewriting-then-completion (max) on buggy-HumanEval. First, the model performs better given longer partial code in the reference case (Figure 3a). While the naïve completion performs overall poorly with potential bugs (Figure 3b), it performs relatively well when potential bugs appear on or near the last line of the partial code (along the diagonal). More interestingly, rewriting-then-completion achieves higher scores when potential bugs appear later (Figure 3d), while completion-then-rewriting performs better with longer code prefixes (Figure 3c). We suspect that a longer prefix makes completion models less likely to deviate from the reference completion. Hence, as inputs to the subsequent rewriting model, the completed code better resembles input types for which the rewriting model is trained, making the repairing more likely to succeed.
4.6 Case Studies
Are potential bugs always harmful? As discussed in Section 2, potential bugs do not guarantee the completed code to be buggy. While we observe performance degradation of Code-LLMs under the presence of potential bugs, we find several interesting cases where the models manage to generate correct code. For instance, Figure 12 in Appendix shows that for the potential bug == (highlighted) modified from !=, the completion model deviates its original algorithmic flow with continue command and completes with correct functionality, albeit different from the canonical solution. This is an example that some bCC cases are recoverable and that Code-LLMs can adapt to them.
When do Code-LLMs succeed at bCC? For the successful cases of naïve completion at bCC, we observe that either the model (i) ignores the incorrect state and generates the correct completion or (ii) takes into account the potential bug to generate a completion adapting to it. Figure 9 in Appendix shows an example when Code-LLMs ignore the if-else statement to bypass the potential bug. Further case studies and more elaborate discussions are in Appendix D.
Code completion. Code completion provides code suggestions based on a given context [39, 40, 38]. The scope of completion ranges from the next token or line [11], method and class names [41], to the entire of the function [42] or program. Early works [5, 43, 4] viewed code as sequences of tokens and applied statistical language models to the problem, along with other attempts at building probabilistic models of code [44, 41, 45]. Later works adopted deep neural networks [8, 46] and pre-training techniques [9, 47] for better code modeling and completion. Our work considers the code completion setting at the function and program levels and focuses on using large language models for completion. Beyond sequence modeling, recent works considered integrating code prior knowledge via abstract syntax trees [6, 8, 48], code token types [9], graph structures [7], hierarchical context [49], or generating the sketch [50], or even extending the task's information beyond the given input files [51, 52]. We focus on using only the task description and partial code as input prompts to the model, allowing the use of more Code-LLMs and publicly available datasets.
Automatic program repair. The research on automatic program [53, 54, 55, 56, 57, 58, 59, 25] relieves developers from the enormous effort of finding and fixing programming bugs. Recently, Code-LLMs have been adapted for program repair by translating buggy programs into their reference counterparts [60, 31, 61]. Among those, we use RealiT [32] as our repair model in the completionthen-rewriting method since they obtain the SotA results and utilize similar simple mutations during training. Despite the similarity, code repair commonly targets fixing bugs from complete programs while we study potential bugs from partial code. To enrich the amount of data for program repair, methods have been proposed to synthesize artificial bugs through code mutants [62, 63, 64, 65, 25] or learning to create bugs [66]. Similarly, we employ code mutants to create artificial bugs.
Relation to adversarial examples. Adversarial examples are instances where small perturbations to the input lead the model to change into wrong predictions. They have been extensively studied in computer vision [67, 68] and natural language processing [69]. Recent works suggested similar situations for code-learning models, where small, semantic-preserving code transformations led to performance degradation [70, 71]. Buggy-code completion can be seen as a dual problem to adversarial examples, where we expect the model to adjust its predictions up on small semantic-altering changes in the input. In our case, sensitivity is not a problem, but insensitivity is.
Benchmarks for buggy-code completion. Multiple benchmarks have been studied for code completion and program repair. For code completion, CodeXGLUE [11], CodeNet [30], and HumanEval [10] are widely used. CodeXGLUE contains corpora of Java and Python programs for completion but only supports match-based evaluation. CodeNet collects programming problems from online judge sites, with both solutions and test cases. HumanEval can be considered a Python-function completion dataset, with the context being the problem statement and function header. We derive our datasets from HumanEval and CodeNet datasets. For neural program repair, many datasets require the match-based evaluation [24, 26] or focus on the compiler errors [23, 25], which are different from our setting. While IntroClass [27], QuixBugs [28], Defects4J [72], or Refactory [29] provide the test suites for evaluation, their test does not reflect the real-world bugs or lacks context support for the use with Code-LLMs [19]. FixEval [19] is recently proposed as a new context-aware program repair dataset to mitigate these limitations, with many problems derived from the real submitted programs. However, as FixEval does not provide the problem statement and focuses solely on the program repair, we derive a new benchmark using FixEval and its source of problems - CodeNet [30].
Limitations. Our baseline methods developed for bCC may degrade the completion performance on reference code context, as shown in Section 4.4, suggesting the need for balancing the buggy and reference settings in solutions to bCC. Furthermore, while buggy-FixEval is associated with real-world coding contest programs, it is unclear how closely buggy-FixEval aligns to the general software development setting where obtaining a test suite and proper evaluation are more challenging.
Impact and applications. As our work focuses on the less refined and more error-prone work-in-progress code, the code context should be viewed as a hint of user intent rather than a high-quality "gold" implementation. It thus naturally follows that a pair programmer or a smart tool should suggest a change to the draft code rather than blindly continue it if they believe a certain part of the existing draft is not intended. From a user experience perspective, an IDE can display code change suggestions to a user's existing code if such parts are identified. Similar functionality already exists for other types of code change suggestions, e.g., spelling correction and missing imports.
Conclusion. We introduce and define the buggy-code completion problem, inspired by the practical coding scenario where one completes a coding program given the problem statement and a partial code with potential bugs. We construct two new datasets, buggy-HumanEval and buggy-FixEval, as task benchmarks and find that the presence of potential bugs significantly degrades the completion performance of all evaluated large language models of code. Our further investigation of completion methods for Code-LLMs in dealing with potential bugs shows that completing with potential bugs remains challenging despite augmenting models with external program-repair models. We provide extensive ablation and case studies for further understanding and analysis of buggy-code completion setting. We hope our novel and systematic study paves the way for future works in understanding and improving the usability of Code-LLMs under practical software-development settings.
[65] Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pages 101-114, 2020.
[66] Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. Self-supervised bug detection and repair. Advances in Neural Information Processing Systems, 34:27865-27876, 2021.
[67] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.
[68] Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv preprint arXiv:2007.00753, 2020.
[69] Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41, 2020.
[70] Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. Semantic robustness of models of source code. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 526-537. IEEE, 2022.
[71] Goutham Ramakrishnan and Aws Albarghouthi. Backdoors in neural models of source code. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 2892-2899. IEEE, 2022.
[72] René Just, Darioush Jalali, and Michael D Ernst. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437-440, 2014.
This section presents the details of dataset construction and specifications. The length statistics are reported in Tab 5.
A.1 Buggy-HumanEval
HumanEval<sup>5</sup> [10] is a dataset designed for evaluating code generations from natural-language descriptions. It contains 164 manually written introductory coding problems in Python 3. Each problem is given in the format of a partial program, referred to as a "prompt": a function header with a docstring, sometimes with necessary imports and helper functions before it. The problem description and often a few input-output examples are encapsulated in the docstring. Each problem is accompanied by a separate set of test cases and a manually written function body referred to as the "canonical solution", such that the concatenation of the prompt and the canonical solution passes all the tests. See an example in Figure 5.
To create buggy-HumanEval, we introduce artificial bugs by flipping binary operators in the canonical solutions into their semantic opposites. Specifically, we use the Python `ast` library and consider all operators under `ast`, `BinOp`, `ast`. Compare and `ast`, `AugAssign`, e.g., >= and +=. For each found operator, we change it to its semantic opposite, e.g., / to *, >= to <, or += to -=. Table 4 shows a complete list of the operators we considered and their opposites. We then check if the solution fails any test cases after the change. We skip cases where the semantic opposite is ambiguous (e.g., mod % vs. floor division // or multiplication *) or where the altered solution still passes all the tests<sup>7</sup>.
Suppose the canonical solution has  $L$  lines, and the operator change happens in the  $l$ -th line. For each  $l \leq i < L$ , we append the first  $i$  lines of the altered canonical solution to the original HumanEval prompt to form a piece of potentially buggy partial code, aka the "buggy" prompt for buggy-code completion. Accompanying each buggy prompt, we also provide a "reference" prompt by concatenating the original prompt with the first  $i$  lines of the original canonical solution. In the example in Figure 5,  $L = 8$ ,  $l = 3$ ,  $i = 4$ . The != in the third line of the canonical solution is changed to ==.
We generated 1896 buggy-code completion instances from 107 unique HumanEval problems. This procedure results in the list of the number of instances per problem as follows,
[9, 31, 0, 5, 1, 0, 13, 0, 3, 0, 0, 4, 1, 0, 1, 0, 0, 0, 8, 0, 16, 0, 0, 0, 1, 28, 0, 0, 0, 0, 0, 10, 63, 0, 0, 2, 14, 2, 0, 38, 6, 0, 0, 7, 5, 0, 14, 3, 6, 1, 0, 0, 2, 0, 0, 6, 16, 4, 2, 27, 0, 16, 0, 12, 5, 3, 1, 0, 6, 21, 0, 51, 26, 7, 14, 13, 12, 0, 1, 0, 22, 180, 12, 1, 0, 0, 1, 0, 9, 0, 0, 18, 2, 32, 57, 5, 0, 1, 47, 0, 5, 12, 8, 2, 0, 39, 23, 8, 16, 22, 18, 0, 10, 16, 0, 0, 5, 15, 34, 4, 0, 0, 37, 52, 0, 17, 31, 1, 133, 29, 14, 21, 1, 0, 11, 3, 1, 0, 6, 174, 20, 14, 15, 13, 8, 6, 35, 12, 2, 11, 0, 0, 18, 16, 8, 6, 0, 0, 7, 2, 11, 0]
Table 4: We considered Python 3 operators when introducing bugs to HumanEval.
ast class ast token	BinOp Add	Sub	Mult	Div	AugAssign	Compare
Add	Sub	Mult	Div	Eq	NotEq	Lt	LtE	Gt	GtE
Operator	+	-	*	/	+=	-==	*=	/=	==	!=	<	<=	>	>=
Opposite	-	+	/	*	-==	+=	/=	*=	!=	==	>=	>	<=	<
Table 5: Length (the number of tokens) statistics of buggy-HumanEval and buggy-FixEval.
Percentiles	50th	90th	95th	98th	99th	100th
buggy-HumanEval	214	470	550	595.6	606.7	617
buggy-FixEval	71	242.7	262	330	402.4	566
<sup>5</sup>https://github.com/openai/human-eval
$^{6}$ https://docs.python.org/3.9/library/ast.html
We found 5 cases in our initial trials where the altered solutions still passed their test cases. Namely, changing the first applicable operators in the canonical solution of task IDs 'HumanEval/10', 'HumanEval/18', 'HumanEval/39', 'HumanEval/40', 'HumanEval/129'. Upon close examination, in these cases, either the altered solution is still correct or the altered solution is incorrect, but the test cases are too weak to tell.
```
Problem Statement: In the Kingdom of AtCoder, people use a language called Taknese, which uses lowercase English letters.
In Taknese, the plural form of a noun is spelled based on the following rules:
If a nouns singular form does not end with s, append s to the end of the singular form. If a nouns singular form ends with s, append es to the end of the singular form.
You are given the singular form S of a Taknese noun. Output its plural form.
Input: Input is given from Standard Input in the following format:
S
Output: Print the plural form of the given Taknese word.""
s = input()
if s[len(s) - 1] != s:
(a) A prompt from buggy-FixEval, containing a problem statement and a buggy partial code.
s = input()
if s[len(s) - 1] != "s": print(s + "s")
else:
    print(s + "es")
s = input()
if s[len(s) - 1] != s:
    print(s + "s")
else:
    print(s + "es")
buggy program
(b) The original FixEval accepted (left) and rejected (right) user submissions.
Figure 4: Example of a buggy-FixEval instance. The example is based on p02546.
A.2 Buggy-FixEval