SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tuan Dinh $^{1*}$ † Jinman Zhao $^{2*}$  Samson Tan $^{2}$  Renato Negrinho $^{2}$  Leonard Lausen $^{2}$  Sheng Zha $^{2}$  George Karypis $^{2}$
<sup>1</sup>University of Wisconsin-Madison 2Amazon Web Services tuan.dinh@wisc.edu
{jinmaz,samson,renalati,lausen,zhasheng,gkarypis}@amazon.com
Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs - anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than  $50\%$  given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a significant gap in post-mitigation performance. $^3$
Suggesting code for a given context is a frequently used feature in modern integrated development environments (IDEs) [1], bringing productivity gains to the code-writing process. This task is widely studied as code completion [2, 3] in the literature, with techniques and models ranging from probabilistic or sequence modeling [4, 5], incorporating code structure as prior knowledge [6, 7], to adopting deep neural networks [8] and pre-training techniques [9] to learn representations for code. Recently, large Transformer-based language models of code (Code-LLMs) [10, 11, 12] have become a promising paradigm for code completion, attaining state-of-the-art (SotA) performance in various code learning tasks including code completion and generation.
However, existing works studying Code-LLMs often assume the absence of bugs, despite the frequent occurrences and the cost of bugs in software development: on average, 70 bugs are created per 1000 code lines [13]; and fixing bugs costs  $50\%$  of development time [14]. Consider a practical coding scenario. A developer wants to use the code suggestion feature in an IDE when writing code. With a high probability, their real-time code context, as an input to the code predictor, contains typos or less refined, potentially buggy implementations. Since bugginess is a property of a complete program,
$\dagger$  Work done while interning at Amazon Web Services.
$^{3}$ Code and datasets are available at https://github.com/amazon-science/buggy-code-completion
Figure 1: Illustrations for code completion (left) and buggy-code completion (right). (top) problem statement for function implementation, (middle) partial code with (right) or without (left) potential bugs (highlighted), (bottom) code completions from CODEGEN-2B-MONO [12]. The completed code is functionally correct on the left but incorrect on the right, failing test case below_zero ([1, 2]) == False. The example is based on HumanEval/3 from buggy-HumanEval.
it is not well-defined how to detect bugs and repair the buggy code in this short and incomplete code context, making the application of existing bug-detection or code-repair tools sub-optimal or infeasible. It is also worth mentioning that the gap between the in vitro and in vivo performances of code completion models [15, 16, 17] remains large. Therefore, a natural question arises: Can existing Code-LLMs provide good code suggestions given the unrefined nature of draft code?
To answer this question, we introduce and study the problem of completing code with potential bugs in the code context, dubbed buggy-code completion (bCC). In this study, we focus on using Code-LLMs to generate functional implementations from a code context, where the code context consists of a problem specification and a piece of partial code. A potential bug in a piece of partial code is a code span that can become a bug provided some completion, i.e., it fails the completion and at the same time can be changed to make the completion work. Note that a potential bug is not a bug per se without a completion. Shown in Figure 1 is an illustration of our task with a problem description at the top and partial code in the middle. On the left are a reference code context and a correct completion from the chosen Code-LLM. On the right, the highlighted potential bug  $(- =)$  makes the reference completion incorrect. The Code-LLM reacts to this potential bug by generating a different completion (bottom right). However, the completed code is still functionally incorrect.
To conduct a quantitative study of bCC, we construct two datasets. First, buggy-HumanEval dataset contains interview-style coding problems from HumanEval dataset [10], with buggy/reference partial code pairs being generated by introducing semantic-altering operator changes to reference solutions. This dataset provides a well-controlled setting to assess models' behavior upon potential bugs. Second, buggy-FixEval dataset, based on FixEval [19], contains user submissions to coding-content problems. The buggy/reference pairs are constructed from rejected and accepted submissions by the same user to a given problem. This dataset helps assess models' performance over a realistic distribution of potential bugs. Our benchmarks are well associated with the existing benchmarks for Code-LLMs.
Via our empirical studies, we find that the presence of potential bugs drastically degrades the code-completion performance of high-performing Code-LLMs, with test-case pass rates dropping to below  $5\%$  across both datasets for all tested model variants. For instance, on buggy-HumanEval, the test-case pass rate of CODEGEN-2B-MONO completions drops from  $54.9\%$  (reference partial code) to  $3.1\%$  (partial code contains potential bugs), which is worse than the score when no partial code is provided  $(9.3\%)$ . Our results demonstrate that Code-LLMs are highly susceptible to potential bugs.
Furthermore, we attempt several post-hoc methods to augment Code-LLMs to better deal with potential bugs, namely removal-then-completion, completion-then-rewriting, and rewriting-then-completion. The latter two augment the Code-LLMs with an external code repairer as the rewriter component. Our evaluation shows that the attempted methods improve the buggy-code completion performance of all tested Code-LLMs. However, the performance gap remains large between these
4This setting is well-aligned with the text-to-code generation task for Code-LLMs [12, 18].
Figure 2: Performance (pass@1 ( $\uparrow$ )) degradation in the presence of potential bugs. CG: CODEGEN. Potential bugs severely harm the completion of all CodeLMs (red bars) compared to non-buggy settings (teal bars), making it even worse than completion without partial code (blue bars).
methods and the completion with reference partial code. We provide further case studies and analyses, e.g., effects of potential bugs' locations or the successful cases of naïve completion for a better understanding of the behavior of the tested models in buggy-code completion.
Study scope and contributions. This work aims to explore and understand the behaviors of Code-LLMs under the buggy-code completion setting. We (i) define the novel buggy-code completion task, (ii) introduce two representative benchmark datasets, (iii) demonstrate the inability of Code-LLMs to handle potential bugs, and (iv) evaluate several baseline methods for improving Code-LLMs on buggy-code completion.
2 Buggy-Code Completion
In this work, we consider the code completion setting with inputs consisting of (1) a specification  $h$  that specifies the desired functionality of the program and (2) a code context  $s$ , which is some unfinished code to be completed. Here,  $h$  is given as a docstring or a problem statement in English, and  $s$  is given as a few lines of code that are the beginning part of a program, which we refer to as partial code or (code) prefix. In the conventional setting of code completion, the objective is to suggest a completion  $c$  such that  $t := s::c$  is a program that satisfies  $h$ , where “::” denotes code concatenation. Our buggy-code completion setting extends the conventional one with a challenging and realistic consideration:  $s$  may contain potential bugs.
Consider a real-world scenario where a programmer works on unfinished code and utilizes the code auto-completion feature for suggestions. Note that the coding process often sees coding mistakes or introduces inconsistent code usages, which are not necessarily "incorrect" per se. However, there is a fair chance that such mistakes or inconsistencies cause unintended and undesirable behaviors in the final program. In other words, they become bugs. Based on this intuition, we define potential bugs as the following:
Definition 2.1 (potential bug). Consider a specification  $h$  and a reference code prefix  $s$  for which some completion  $c$  exists such that  $t := s :: c$  satisfies  $h$ . A potential bug is manifested as a small edit  $e$  over  $s$  such that  $t' := s' :: c$  does not satisfy  $h$ , where  $s'$  is the result of applying  $e$  on  $s$ .
We note that potential bugs are not bugs per se and are only defined with respect to some reference code prefix  $s$  and completion  $c$ . The code prefix  $s'$  containing potential bugs is referred to as "potentially buggy" or simply "buggy" prefix (with respect to  $s$ ) throughout the paper. In most cases, we omit the "with respect to" part for brevity. The resulting term "buggy prefix" refers to the definition here and by no means assumes that the prefix itself is buggy. Our study focuses on potential bugs associated with semantic bugs, i.e., edits that do not introduce syntax errors, as semantic bugs are generally more challenging and interesting than syntax ones.
Remark 2.2. Intuitively, we assume that the reference prefix  $s$  highly correlates to preferred (practical) implementations of the task and that any deviation from  $s$  is likely less preferred. We use the operational definition of potential bugs in 2.1 for its simplicity and verifiability, as well as for that it allows the initial exploration of the previously under-investigated scenario of bCC. We note that by this definition, a reference prefix  $s$  itself can, in some cases, albeit less likely, be "buggy" with respect to some other references. However, such a case is less of a concern in this study, as we are mostly interested in finding a completed functional program out of  $s$  or  $s'$ .
Definition 2.3 (buggy-code completion,  $bCC$ ). Given a specification  $h$  and a code prefix  $s'$  containing potential bugs, buggy-code completion is the task of generating a complete program  $t$  that satisfies  $h$ .
We deliberately loosen the constraint that  $t$  should contain  $s'$  as a prefix, as by definition, and as well as one can see from the example in Figure 1, fixating on the buggy prefix can make suggesting a satisfying solution difficult, if not impossible. As we explore in Section 4.3, allowing models to suggest fixes to the buggy code prefix significantly increases the possibility that a generated program passes the tests. However, it is often still possible to continue a buggy prefix to a valid program solution (see an example in Figure 12).
An alternative view of the setting is that the buggy code prefix provides a noisy and potentially flawed precondition for generating a solution to the coding problem. It presents a good-faith effort from a user to instruct the tool about their intended code solutions. A good-behaving model should take this as a hint and, at worst, discard it so that the generation performance is not worse than when no code prefix is given. However, this is not the case for our evaluated models (Section 4.2).
Remark 2.4. Our bCC formulation is not a simple combination of code repair and code completion. As bugginess is a property of completed programs, repairing the partial code is an ill-defined problem, thus making repairing-then-completion also ill-defined. Therefore, our bCC task is better viewed as an extension of code completion with the additional challenge that generating semantically correct continuations from the given partial code without deviation may be difficult or even infeasible. This challenge requires models to be aware of the existence of potential bugs for better suggestions.
This section introduces our new datasets with proposed baseline methods and evaluation metrics.
3.1 Datasets for bCC
Based on our task definition, each instance in a bCC benchmark should contain a problem description specifying requirements, a piece of buggy code prefix to be completed, and a set of test cases for assessing the correctness of the finished code. Optionally, an instance also has a corresponding reference code prefix and a valid solution completed from the reference code. To our knowledge, no existing dataset meets all the desiderata. Commonly used large-scale datasets for code completion, e.g., Py150 [20] and Java Corpus [21] do not come with test cases. Recent small-scale manually curated datasets for code generation, e.g., HumanEval [10], MBPP [18], APPs [22], come with test cases and reference solutions but no buggy or failed solutions. There is also no predefined way to extract partial code from the reference solutions. This is also the case for program-repair datasets: popular datasets either lack test cases, e.g., [23, 24, 25, 26] and/or are of small sizes, e.g., [27, 28, 29].
We introduce two datasets for evaluating bCC in Python, both with all the aforementioned desired components. We ensure that 1) all partial code snippets contain potential bugs that fulfill our definition, i.e., their respective completed code is incorrect, certified by failing test cases; and 2) all potential bugs are semantic in nature, i.e., they cause no syntax error in their respective completed code.
3.1.1 Buggy-HumanEval
We first introduce buggy-HumanEval to offer a controlled setting for evaluating bCC in the presence of a single semantic bug. Buggy-HumanEval contains 1896 bCC instances constructed from a subset of HumanEval problems [10]. The HumanEval dataset is a popular dataset of manually written introductory coding problems designed for evaluating the code generation ability of Code-LLMs.
Potential bugs are introduced as semantic-altering operator changes to reference solutions. We search for applicable binary operators in reference solutions and change them into their semantic opposites, e.g., + into - . To ensure that the edit introduces a bug, we execute the altered program and only keep the ones failing some tests. We then specify a line after the altered operator to split the solution into a code prefix and suffix. We keep the buggy prefix as part of the input for bCC. We split the unaltered solution at the same line to get a corresponding reference prefix. On average, the problems selected for buggy-HumanEval have longer solutions than the unselected ones in HumanEval, with 8.2 lines vs. 2.9 lines of code, respectively. Appendix A.1 provides details of our dataset.
3.1.2 Buggy-FixEval
To evaluate bCC with more realistic bugs, we introduce buggy-FixEval with bCC instances constructed from CodeNet [30] and FixEval [19]. FixEval is a program repair benchmark based on user submissions to competitive programming websites. Each data sample in FixEval can be viewed as a pair of submitted programs from the same user, with the accepted submission being regarded as reference or fixed and the preceding rejected submission being regarded as buggy.
To create buggy-FixEval, we match and pair each problem with its problem statement found in CodeNet and omit the problems with no match. For each submission pair, we identified potential bugs as the differences between the rejected and accepted submissions. We ensure that the rejected submission contains no syntax error and fails at least one test case. We split the solutions into halves and regard the prefix from the rejected solution as containing potential bugs and the prefix from the accepted solution as a reference. To guarantee the differences between the buggy prefix and the reference prefix related to potential bugs, we impose a limit on the character-level edit distance between them, ignoring comments and white spaces. The lower limit ensures that the differences are not comments or white spaces. The upper limit (20, chosen by manual inspection) reduces the chance that the difference is related to a reformatting or a re-implementation. We then manually inspected all the remaining pairs and excluded undesirable cases such as that the reference prefix itself is already a correct solution or that the two prefixes are semantically equivalent. Finally, we execute the concatenation of the buggy prefix and the reference completion and ensure that it fails at least one test case. More details about creating buggy-FixEval can be found in Appendix A.2.
3.2 Baseline Methods for bCC
Beyond the naive generation from code context, we study several methods for augmenting Code-LLMs to utilize the partial code better while mitigating the effect of potential bugs. Assuming a bug detection component is equipped, we focus on simple and modular-design approaches that do not require additional training and are flexible for further incorporating newly developed Code-LLMs.
Removing partial code, then completing (removal-then-completion). We bypass the negative effect of buggy partial code by removing the entire content code fragment from the model input. In particular, for buggy-HumanEval, the input to Code-LLMs after removal consists of the problem statement and the function header (and examples). For buggy-FixEval, we keep only the statement and the code fragment used for reading input data. Intuitively, as removal-then-completion guarantees that the input to Code-LLMs contains no bug, we expect this method to perform better than the naive completion. However, the drawback of removal-then-completion is its sacrifice of all potentially useful information brought by the partial code.
Completing first, then rewriting the program (completion-then-rewriting). This approach attempts to fix the buggy code using pre-trained code-repair models [31, 32, 25], e.g., neural translating a buggy program into a valid program [31]. As code-repair models, or code fixers, are usually trained to fix the complete programs, we first complete the code by naive completion, then apply code fixers on the potentially buggy programs. We use RealiT [32], a SotA program repair model as the code fixer. RealiT is designed for misused variables, wrong literal, wrong binary, and unary operators, which is especially suitable for the bugs introduced in the buggy-HumanEval.
Rewriting the partial code, then completing (rewriting-then-completion). This approach attempts to resolve potential bugs in the partial code before completing. To do so, we first locate code lines containing the potential bug, then rewrite these lines. To detect potential bugs, we propose a likelihood-based measure to identify the line most likely to contain a potential bug. In particular, we score each code line with the following procedure. First, for each token, we define its buggy score to be the difference in likelihoods between the token with the highest likelihood (i.e., the argmax token) and the observed token. The buggy score for each line is then calculated by taking either the maximum or average of non-zero buggy scores of its tokens. The line with the highest score is most likely to contain the potential bugs. We use the INCODER-6B [33] as the infilling language model for rewriting code. We provide a detailed explanation and example illustration of likelihood-based measures in Appendix B to understand our rewriting-then-completion method better.
Table 1: Pass@1 (↑) of completion methods on buggy-HumanEval and buggy-FixEval datasets. For all Code-LLMs, all three proposed methods improve the completion performance of the naive completion. On average, completion-then-rewriting and rewriting-then-completion achieve the best scores on buggy-HumanEval and buggy-FixEval, respectively. Nevertheless, there are still substantial gaps between these methods and completion with reference prefixes. *Best methods in each column are in bold. Table 7 in Appendix C provides results with a larger model (CODEGEN-16B-MONO).
Prefix	Method	buggy-HumanEval	buggy-FixEval
CODEGEN-	INCODER-	CODEGEN-	INCODER-
350M	2B	1B	6B	350M	2B	1B	6B
reference	completion	43.0	54.9	41.1	50.6	27.6	37.8	24.1	32.3
buggy	completion	0.7	3.1	0.5	1.0	2.4	4.3	1.2	1.8
removal-then-completion	4.7	9.3	3.9	6.9	3.3	8.6	2.9	6.4
rewriting-then-completion	14.1	24.9	9.1	16.4	2.4	7.2	2.6	5.1
completion-then-rewriting	22.5	23.6	22.7	25.2	2.3	4.7	1.7	3.0
We measure the functionality of a completed program by executing it against the provided test cases. Following the recent code-generation works [e.g., 10, 34, 12, 33], we measure the pass@k (↑) for each bCC instance as  $\text{pass} @ k := 1 - \binom{n-c}{k} / \binom{n}{k}$  where  $n$  completions are sampled from a model, and  $c$  of them pass all the tests. We choose  $n = 100$  and  $k = 1, 10, 100$ . This metric estimates the probability that any of  $k$  samples from the model passes the tests. As multiple bCC instances may be derived from the same programming problem, we first average the pass@k within each problem, then average across all problems (macro-average) to avoid the dominance of a few problems.
While passing all test cases does not  $100\%$  guarantee the correctness of a program, this metric provides a practical and efficient proxy for functional correctness. Note that we do not use match-based metrics, e.g., exact match or CodeBLEU [35] because they do not properly reflect the functionality of generated code [22, 10, 18], and no reference completion is available for buggy prefixes.
We design two experiment sets to investigate (1) how well the existing Code-LLMs adapt to bCC (Sec. 4.2) and (2) whether we can have a simple fix with Code-LLMs for bCC (Sec. 4.3). We provide ablation studies on potential bugs (Sec. 4.5) and on combining buggy-based completion with reference prefix (Sec.C.4). Sec. 4.6 presents case studies about interesting behaviors of Code-LLMs under bCC, with additional results in Appendix C.
Code-LLMs. We evaluate the two popular and open-sourced Code-LLMs for code completion. CODEGEN [12] is a family of LLMs trained on both natural and programming language corpora with high performance in code generation on HumanEval [10]. We use released model checkpoints: CODEGEN-350M-MONO, CODEGEN-2B-MONO, and CODEGEN-16B-MONO. INCODER [33] models are trained with a causal masking objective, allowing them to fill blocks of code conditioned on arbitrary left and right contexts. We use the released model checkpoints INCODER-1B and INCODER-6B, each with 1B and 6B parameters. We select CODEGEN [12] and INCODER [33] since they are publicly available with high performance of code generation.
Generating completions. Input format. For buggy-HumanEval, a model is expected to complete an unfinished function. Following the HumanEval benchmark [10], we set the models' input as the partial code leading up to the completion location, with the problem description embedded as the docstring of the unfinished function. For buggy-FixEval, the completion task is to complete an unfinished program with inputs being the problem description as a file-level docstring (quoted within triple quotation marks), followed by the partial code leading up to the completion location. Generation and sampling details. Following the best-performing settings reported in the corresponding works [12, 33], we use temperature sampling with temperature  $= 0.6$  for CODEGEN and top-  $p$  sampling with  $p = 0.95$  and