SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FixEval $^{8}$  [19] is a dataset derived from CodeNet $^{9}$  [30] for studying program repair. It consists of programming problems from online programming contest websites $^{10}$ , each with test cases, user-submitted solutions and judging verdicts to the submissions (e.g., accepted - passing all the tests; wrong answer - failing at least one test; or time limit exceeded - the program did not terminate within the specified time limit). The test cases in FixEval are more extensive than those in HumanEval, as the programming contest websites use them to automatically judge a submission's correctness. Each problem comes with an alphabetical label ("A" to "F") indicating the order in which the problem appears in its contest, with "A" usually being the easiest. Different from HumanEval, a solution or submission here is supposed to be a complete program, reading input from and writing output to the standard input and output instead of just a function implementation.
To derive a dataset for buggy-code completion, we pair each FixEval problem with its matching problem statements from CodeNet and discard problems without a match. Among the remaining problems, we focus on the relatively easy "A"-label problems because the post-"A" problems have been reported to be very challenging to solve even for recent language models of code: pass@1 < 12% with CodeT5 [31] under a program-repair setting [19, Section 5.3].
We start by selecting rejected-accepted solution pairs. For each streak of Python submissions from a user to a problem, if the streak ends with an accepted submission, we take the last submission with the verdict being none of accepted, compilation error or runtime error as its rejected counterpart.
Then, we remove all comments and empty lines for each accepted-rejected pair and check the following. (1) Neither submission contains exit() or system calls, as they cause trouble for execution. (2) The first halves of the two submissions contain at least one non-white-space difference, including case difference, space/indent difference, or variable names for the assignment operator. (3) The character-level edit distance between the first halves of the two submissions is smaller than 20. Regarding the threshold of 20, we sampled 100 pairs of submissions with various edit distances. We qualitatively verified that under this threshold, the differences between the two submissions are, in the majority of cases, focused semantic changes (or likely bug fixes) and not refactoring or re-implementations.
<sup>8</sup>https://github.com/mahimanzum/FixEval
<sup>9</sup>https://github.com/IBM/Project_CodeNet
10The bases of the data are from AtCoder (https://atcoder.jp/) and AIZU Online Judge (https://onlinejudge.u-aizu.ac.jp/).
from typing import List   
def has_close_elements(numbers: List[float], threshold: float) -> bool: "" Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>> has_close_elements([1.0, 2.0, 3.0], 0.5) False >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True
(a) The original HumanEval prompt containing a function header and a problem description as a docstring.
for idx, elem in enumerate(numbers):
    for idx2, elem2 in enumerate(numbers):
        if idx != idx2:
            distance = abs(elem - elem2)
            if distance < threshold:
                return True
    return False
(b) The canonical solution provided in HumanEval.
from typing import List   
def has_close_elements(numbers: List[float], threshold: float) -> bool: "" Check if in given list of numbers, are any two numbers closer to each other than given threshold.   
>>> has_close_elements([1.0, 2.0, 3.0], 0.5) False   
>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True   
""   
for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx == idx2: distance = abs(elem + elem2)
(c) The prompt used for buggy-code completion, containing the original problem and a buggy partial code.
def check(candidate): assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False
(d) The test suite used for evaluating a completed program.
Figure 5: Example of buggy-HumanEval instance. The example is based on HumanEval/0. The buggy prompt (Figure c) is directly derived from the original prompt (Figure a) and the canonical solution (Figure b). We reuse the test suite (Figure d) for our evaluation.
The above process resulted in 492 pairs of prefixes from rejected-accepted solutions from 94 "A"-level problems. We manually inspected all the pairs to ensure the pairs satisfy the desiderata described in Section 3.1. Namely, we identify the following cases.
1. The prefix from the accepted solution is already a complete program that passes all the test cases. This can happen when the accepted solution is very short, for example, fewer than three lines, and/or followed by comments or unnecessary code. This counts for 26 pairs.
2. The prefix from the rejected solution and the prefix from the accepted solution are semantically equivalent up to variable name normalization. This counts for 33 pairs.
3. The prefix from the rejected solution contains no control flow (e.g., if, for, while) and no output clause. In this case, a completion can ignore the prefix. This counts for 141 pairs.
4. The prefix from the rejected solution is bound to output wrong answers upon a certain input. For example, it contains a print() statement within a wrong condition. This counts for 25 pairs.
A list of identified cases can be found in our code repository. Case 1 makes the completion trivial from reference prefixes. Case 2 makes the comparison between the buggy completion and reference completion less meaningful. Case 3 makes the buggy completion less challenging. Thus, we exclude instances identified as cases 1, 2, and 3. We keep the case 4 instances because they fit the definition of potential bugs and are especially interesting in the sense that the "bugginess" already manifested in the code prefix.
After the above step, we are left with 292 pairs. We then use the first half of the rejected submission as a piece of buggy partial code and the first half of the accepted submission as the corresponding piece of reference code for buggy-code completion. A "buggy" prompt is formed by preponding the problem statement as a file-level docstring, or a multi-line string literal, to the buggy partial code. See an example in Figure 4.
B Details of rewriting-then-completion Method
With rewriting-then-completion, we attempt to eliminate potential bugs via a two-step procedure. We first locate the most likely line to contain a potential bug using a likelihood-based measure and then rewrite the line using an infilling code language model.
B.1 Line Selection
Our underlying idea is to treat potential bugs as outliers in the generation flow of Code-LLMs. We observe that most reference code has lower perplexity than the corresponding buggy code.
Likelihood-based measures. We calculate the score of each line as follows: With a model that can give a token distribution for each token location, we define the token score as the difference between the likelihoods of the most probable token (i.e., the argmax token) and the actual observed token. We get the score of a line by taking either the maximum or the average of all the non-zero token scores within it. The line with the largest score is finally selected to be rewritten.
To help better understand likelihood-based measures, consider the problem and the code prefix with potential bugs shown in Figure 8. The partial code has four lines in the function body. Given a language model  $G$ , for each token  $x$ , we calculate the token-level score as  $p_2 - p_1$ :
-  $p_1$ : probability of generating  $x$  from the code up to  $x$
-  $p_2$ : probability of generating  $x^*$  from the code up to  $x$ , where  $x^*$  gives the highest probability according to  $G$ .
For example, consider line 3 and token  $==$ . We would have  $p_1 = 0.01$  (probability of  $G$  generating  $==$ ), and  $p_2 = 0.95$  ( $!=$  is the most probable token according to  $G$ ). The score of  $==$  is then 0.94. We obtain the line's score by taking the maximum scores at all token locations in the line. This way of aggregating token scores to line scores is referred to as "Likelihood (Max.)" Below is the variation used to report results in the main text.
Note that we take two measures to reduce the uncertainties of the likelihood estimation: (i) instead of the likelihood score of the target token itself, we use the maximal margin between the target token and the argmax token (similar to the popular approaches used in uncertainty quantification), and (ii) we aggregate the score gap along the line and set a high threshold to be more conservative.
We find that accuracies of localizing the line of potential bugs (with the same setting described above) are approximately  $82\%$  and  $53\%$  respectively on buggy-HumanEval and buggy-FixEval. For buggy-FixEval, we compare the detected line with the line of the first semantic difference between buggy and reference prefixes.
Table 6: Pass@1 (↑) of each selection and completion method on the buggy-HumanEval and buggyFixEval datasets. The best results for each dataset and completion model are in bold. The better likelihood aggregation method is underlined.
Dataset	Selection Method	Completion Model
CG-350M	CG-2B	INCODER-1B	INCODER-6B
buggy-HumanEval	Heuristic Oracle	22.1	29.6	21.4	28.5
Likelihood (Max.)	14.1	24.9	9.1	16.4
Likelihood (Avg.)	13.2	23.0	9.8	16.9
buggy-FixEval	Heuristic Oracle	1.7	3.3	1.2	2.2
Likelihood (Max.)	2.4	7.2	2.6	5.1
Likelihood (Avg.)	2.7	7.9	2.4	5.4
Heuristic oracle for comparison. To see how well our likelihood-based measures work, we compare them against a heuristic oracle for predicting the line of potential bugs. In particular, we compare the buggy code prefix against the corresponding reference code prefix and select the first line with non-trivial differences. The differences are not about space, indent, or comment. Since this requires access to the corresponding reference code prefix, it cannot be used as part of a solution for bCC. We thus refer to it as an "oracle" and only use it here as a method of reference to gauge the performance of other line-selection methods. We do not use this oracle in any methods reported in the main text.
B.2 Rewriting
After identifying the line most likely to contain a potential bug, we rewrite it by masking it and generating a replacement line using a code-infilling model such as INCODER [33].
Shown in Table 6 is our comparison between different likelihood-based measures. We observe that the Heuristic Oracle substantially outperforms the likelihood-based methods in buggy-HumanEval, but the reverse is true for buggy-FixEval. We provide a likely explanation for this phenomenon in the next paragraph. We also observe that Likelihood (Max.) outperforms Likelihood (Avg.) on average on buggy-HumanEval, but the reverse is true for buggy-FixEval. A possible explanation is that since the potential bugs in buggy-HumanEval take the form of a single token change, they are more amendable to being detected by focusing on the token with the largest score (maximum aggregation). In such a scenario, taking the average across all tokens in the line might dilute this signal. On the other hand, potential bugs are likely more subtle in a natural setting (represented by buggy-FixEval), and the potential bug may be spread out across multiple tokens. The fact that the average aggregation considers this may explain why it outperforms the maximum aggregation on buggy-FixEval.
We note that a large gap in performance between buggy-HumanEval and buggy-FixEval for the Heuristic Oracle method. This may be justifiable based on the information known to the Heuristic Oracle. In buggy-HumanEval, we introduce the bugs synthetically via a single semantically altering operator change. Hence, the oracle method has guaranteed knowledge of the line where the bug was introduced. This is not the case in buggy-FixEval since the examples were constructed directly from human submissions without synthetic perturbations. Therefore, the line corresponding to the first difference between these two submissions may not correspond to the bug location, or multiple (instead of one) locations can change between these two submissions. Therefore, this method is closer to a heuristic than a true oracle for buggy-FixEval. This is reflected in Heuristic Oracle's substantially weaker performance on FixEval. The fact that likelihood-based methods outperform the Heuristic Oracle on buggy-FixEval supports this explanation.
Figure 6: Comparison of CodeLMs per each completion method in terms of pass@k. CG: CODEGEN, Comp: Completion. We see the trend consistent with CODEGEN-2B-MONO > INCODER-6B > CODEGEN-350M-MONO > INCODER-1B across methods and  $k$ 's.
C Full Definition Statement and Additional Results
C.1 Full Statement for Definition 2.1 in Section 2
Definition C.1 (potential bug in a code prefix). Assume a problem given by specification  $h$  with the test  $f_{h}$  for evaluating its functional correctness and reference partial code  $s$ . Let  $\mathcal{T}_s^h$  be the set of all valid programs, or solutions that satisfy  $h$  and have the prefix  $s$ , i.e.,  $\mathcal{T}_s^h = \{t := s::c|f_h(t) = 1\}$ . Let  $\mathcal{C}_s^h$  be the set of all valid completed code, i.e.,  $\mathcal{C}_s^h = \{c|s::c\in \mathcal{T}_s^h\}$ . A potential bug of  $s$  is an edit  $g_{e}$  on a token of  $s$  that causes at least one existing solution to fail, i.e., for  $s' = g_e(s)$ , there exists  $c\in \mathcal{C}_s^h$  s.t.  $f_{h}(s^{\prime}::c) = 0$ . The prefix  $s'$  is called buggy with respect to reference prefix  $s$ .
C.2 Full Results with  $k = 1, 10, 100$  and Interpretation for Section 4.3
Figure 6 presents the full comparison among pass@k for  $k = 1, 10, 100$  for all methods. Intuitively, higher  $k$  reveals more on the diversity of generated solutions. The relative performances between different Code-LLMs are consistent across different  $k$ 's. Regarding the relationship between completion methods,
- The naive completion and completion-then-rewriting methods achieve relatively small increases as  $k$  goes up, as the partial code constrains the input to the completion step.
- The increase in performance of removal-then-completion is relatively higher than naïve completion when  $k$  increases, as its input is not constrained by the buggy partial code prefixes.
- The performance of rewriting-then-completion achieves relatively big increases among the methods as  $k$  goes up. This is probably because rewriting-then-completion balances the completion-then-rewriting and removal-then-completion approaches. Thus, the pre-completion rewriting step increases the performance along with the diversity of code prefixes provided in the completion step.
C.3 Performance on CODEGEN-16B-MONO
We provide results for CODEGEN-16B-MONO in table 7. As we can see, CODEGEN-16B-MONO also exhibits a similar phenomenon to our observations in previous Table 1 for smaller models, i.e., potential bugs make Code-LLMs more challenging to generate correct programs and proposed mitigation methods still suffer from a large gap to the reference-prefix completion.
Table 7: Pass@1 (↑) of completion methods with CODEGEN-16B-MONO. *Best method in each column is in bold.
Prefix	Method	buggy-HumanEval	buggy-FixEval
reference	completion	68.9	49.4
buggy	completion	4.4	8.0
removal-then-completion	20.2	17.6
rewriting-then-completion	24.3	9.7
completion-then-rewriting	24.1	7.7
C.4 Ablation Study: Concatenating Buggy-based Completion with Reference Prefix
In this ablation study, we use the buggy prefix to complete, then replace the buggy prefix with the reference prefix, shown in the bottom row of Table 8 (reference prefix + buggy-based completion). The results indicate that Code-LLMs fail to react to the change in the buggy prefix, as discussed in Section 4.2.
Table 8: Concatenating buggy-code-based completion with reference prefix.
CODEGEN-	buggy-HumanEval	buggy-FixEval
2B	350M	2B	350M
reference prefix + reference-based completion	54.9	43	37.8	27.6
buggy prefix + buggy-based completion	3.1	0.7	4.3	2.4
reference prefix + buggy-based completion	40.8	31.6	4.6	2.5
D Detailed Case Studies
For the case studies, We use the synthetic buggy-HumanEval dataset for our study. We use CODEGEN-2B-MONO as the completion model throughout this section as it achieved the best bCC performance overall. We surface interesting bCC examples by comparing pass rates for reference and buggy prefixes. For example, the Code-LLM may easily complete some reference prefixes (i.e., high pass rate) but potentially fails with buggy prefixes (zero pass rate). Other examples may have a non-zero pass rate, meaning that the Code-LLM can adapt to the potential bug in the prefix and yield a correct solution. One observation that was observed uniformly about this study is that prefixes with potential bugs typically lead to lower pass rates, even when those were non-zero. This suggests that for these prefixes, the Code-LLM has to make more effort to find a correct solution.
D.1 Are Potential Bugs Always Harmful?
As discussed in Section 2, potential bugs do not guarantee that the completed code will be buggy. These prefixes were sourced from completed examples that had a bug that could be ascribed to the prefix, but that does not mean by itself that the prefix will be impossible to complete correctly. While we observe the performance degradation of Code-LLMs under the presence of potential bugs, we find several interesting cases where these models generate correct code nonetheless.
Figure 12 shows that for the potential bug (highlighted operator)  $==$  modified from  $!=$  in the reference code, the completion model updates its original algorithmic flow with continue command and completes with correct code. While the new completion code is different from the canonical solution of the reference code, it is functionally correct. Thus, this implies that codellms may adapt in some recoverable cases.
D.2 Why Do Code-LLMs Fail at bCC?
We investigate the cases where our models succeed with the reference partial code but fail with the buggy partial code.
Problem specification
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``'
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
Figure 7: An example case of buggy-code completion where the model reacts to the code change but still fails. The bCC instance is based on HumanEval/3. The model completions shown here are representative completions for the given prefixes.
Our first finding is that the model often fails to react to the changes in code, i.e., it produces the same completion for both buggy and reference code prefixes. We identify this by comparing if the most popular completions (out of all the sampled completions for a given instance) are the same for the reference and buggy code prefixes. We see that this happens in  $90\%$  of the instances and  $93\%$  of the problems (with at least one failed instance). Figure 7 shows an example. This suggests that the model is not sensitive to and thus ignores minor code changes and/or defaults to common patterns in the training data.
In cases where the most popular completions differ for the reference and the buggy code prefixes, we found that the potential bug often makes the code prefix significantly more challenging to complete correctly. In other words, the model may have recognized the potential bug and thus have significantly changed the output distribution but still failed. Figure 8 shows an example. Here, the balance check condition is reversed due to potential bugs. The model could ignore the given code or even define a new function to overwrite the buggy one. However, this drastically deviates from the commonly observed code patterns and thus may create a significant hurdle for the completion model if it is not introduced or adapted to such cases.
D.3 When Do Code-LLMs Succeed at bCC?
Figure 9 shows an example when Code-LLMs ignore the if-else statement to bypass the potential bugs. In successful cases, we have observed that either the model ignores the incorrect state and generates the correct completion or considers the potential bug in the prefix to create a completion adapted to it.
Problem specification
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``'
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
Figure 8: An example case of bCC where the model fails to react to the code change. The bCC instance is based on HumanEval/0. The model completions shown here are the dominant completions for the given prefixes.
D.4 How Are the Proposed Completion Methods Better at bCC?
The completion-then-rewriting method can detect and fix several types of suspicious operators. Figure 10 shows an example from the string_xor problem. Here, we see that the direct completion failed to react to the potential bug and continued as if the  $==$  operator was not changed to  $!=,$  which gives an incorrect solution. However, the code-repair model was able to identify and fix the  $!=$  operator back to  $==$ , producing a correct solution. Similarly, Figure 11 shows a success case from rewriting-then-completion, where it restores the introduced operator change. As buggy-HumanEval consists of bCC instances constructed from single operator changes, these methods can be particularly effective over naive completion in circumventing the introduced operator change.
Problem specification
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
``
Prefix with potential bugs
from typing import List   
def has_close_elements(numbers: List[float], threshold: float) -> bool: for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx == idx2: distance = abs(elem - elem2)
Model completion
continue else: distance  $=$  abs(elem - elem2) if distance  $<$  threshold: return True return False
Prefix without potential bugs
from typing import List   
def has_close_elements(numbers: List[float], threshold: float) -> bool: for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx != idx2: distance = abs(elem - elem2)
Model completion
if distance  $<$  threshold: return True return False
Figure 9: An example case of bCC where the model bypasses the potential bug and succeeds. The bCC instance is based on HumanEval/0. The model completion for the reference prefix is the most common. The model completion for the buggy prefix is one of the two successful completions.
Problem specification
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
```
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``'
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"
``"