Fault-Aware Neural Code Rankers
Authors: Jeevana Priya Inala
Chenglong Wang
Mei Yang
Andres Codas
Mark Encarnación
Shuvendu K Lahiri
Madanlal Musuvathi
Jianfeng Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: program synthesis
code generation
AI for code

Abstract: Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets. 


Fault-Aware Neural Code Rankers
Jeevana Priya Inala
Chenglong Wang
Mei Yang
Andres Codas
Mark Encarnación
Shuvendu K Lahiri
Madanlal Musuvathi
Jianfeng Gao
Microsoft Research
{j (jinala, chenwang, meiyang, andres codas markenc, shuvendu, madanm, jfaammmm,aaaooo jaggaa//miiooooooooooo com
Abstract
Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CODERANKER, a neural ranker that can predict the correctness of a sampled program without executing it. Our CODERANKER is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError TypeError). We show that CODERANKER can significantly increase the pass@1 accuracy of various code generation models (including Codex [11], GPT-Neo, GPT-J) on APPS [25]. HumanEval [11] and MBPP [3] datasets.
Introduction
Large transformer-based language models (LLMs) have impressive capabilities [19, 16), including the ability to generate code [11, 3, 28, 39, 33]. The task here is to take natural language description or previous code context the input and generate entire program in general-purpose programming language such as Python or C++. Generating entire programs is a challenging task as it involves understanding the task, figuring out how accomplish and writing code without any syntax/runtime errors. On harder programming problems such as coding competition problems, current models achieve very low accuracy especially if the inference- time sampling budget is low. For example, on the APPS dataset [25], state-of- offe--ofthh-arr code generation model, Codex [11], achieves 4% accuracy if it is allowed to sample only one program per task (called pass @ 1), but achieves 24% accuracy if is allowed 100 samples per task (called pass @ 100-at least one correct program in 100 samples) (see Table 7). This observation leads us to an important research problem on exploring approaches to rank the sampled programs to bridge the gap between the pass @ and pass @ 100 performances.
Upon analyzing the sampled programs obtained using an LLM, we find that several of them have syntax errors and runtime errors, and some of them execute to produce undesirable outputs. Therefore, prior works [11, 28] focused on ranking the programs by executing them small set of unit tests (which are typically assumed to be given as part of the task description). However, there are several caveats this approach: First, even if program passes the given tests, could still fail on the unknown unit tests. Second, there is a burden on the user to provide unit tests for every inference
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Task
Reference code
Unit tests
Write function name 'nextPerfecttquare" returns the first perfect square that than its integer argument.
that
Inputs: 6 36 D -5
def next__ perfect_ettt____eee perfect_>quare(n) return = (int (n=x0.5)4-1)ee2
Example: next cere___eefet___eee pee___eefect__quurr(( =- 9 def next_perfect______ pert__pefectsqqurre(e _square(n)::
_square()]
Outputs: 9 49 1
Use Based format
Figure 1: An example code generation task from the APPS dataset (train/3447).
task. Third, to execute the code, all the dependencies has to be installed properly. This is especially problematic in scenarios where user wants get code suggestions in project with multiple files and dependencies (such as using CoPilot in a VS Code environment [I]). Even if these dependencies are satisfied, many realistic programming scenarios involve incomplete code under active development where execution is just infeasible. Finally, the code generated by a LLM could potentially have security risks (such as deleting files on the disk) and hence, executing such code needs heavy-weight isolation mechanisms such as a sandbox environment.
To alleviate the above issues with relying executing code, some of the recent works [18, 36] proposed to use a neural-network based ranker' for ranking programs sampled from a LLM. A ranker model is essentially classifier that takes as input task description and sampled program and predicts the probability that the program is correct with respect to the task description. When given multiple programs (obtained by sampling), a ranker re-orders them in the decreasing order of the predicted probability of the program being correct. The ranker is essentially trying to emulate executing program on some unit tests without actually executing the code during inference. The training data is obtained by executing both correct and wrong programs sampled from the code generation model itself. Thus, we only need unit tests and the execution ability during the dataset creation step instead of during inference as in prior approaches.
While previous approaches target math problems to make the learning problem tractable, in this paper, we design CODERANKER for more complex general programming tasks in Python. A sampled Python program
fail in many different ways. For example, program when executed test can result in a compile/runtime error and can produce a wide variety of outputs such as integers, strings, lists and dictionaries that can produce a type mismatch. In contrast in the math domain, there is less scope for compile/runtime errors and the output is usually just a number.
unit
Our CODERANKER approach is based the idea that neural ranker trained distinguish between the various failure modes can have a better understanding of the program and the task, and hence can do better at ranking programs. Thus, we design fault-aware CODERANKER and we investigate its impact on the ranking performance. Each fault-aware ranker is a classifier trained to predict one/two multi-class labels that extracted from the rich information obtained by executing the programs.
We used the APPS dataset to finetune/train the code generation and the CODER ANKER models. On this dataset, we showed that CODER CODERANKER improved the pass @ performance of Codex (used in a few-shot manner) from 26% to 39.6% on the validation set and from 3.8% to 4.5% on the test set. We additionally found our rankers are transferable to a different dataset without any additional training. On the HumanEval dataset, we improved Codex's 's pass @ from 26% to 32% and on the MBPP dataset, we improved from 36% to 42%. We found similar performance boosts with other code generation models such as GPT-J and GPT-Neo. Compared with a naive binary classificr- ranker, our fault aware CODERANKER achieves better ranking performance. Finally, investigated the effect of mixing ranker datasets from multiple code generation models which gives us an extra boost in the performance.
2 Preliminaries
2.1 Code generation
Task: A code generation task G is prompt, represented as sequence of tokens, that specifies the task at hand. G is usually a combination of natural language, input-output examples, and starter
called a verifier the previous work

---

Model
parameters
mode
pretrain dataset
finetune dataset
GitHub + Common
Codex (Davinei)
Unknown
One-shot
N/A
Crawl, Wiki, etc.
GPT-J
6B
Fine-tuned
Pile dataset
APPS train dataset APPS train dataset
GPT-neo 1.3B
1.3B
Fine-tuned Fine-tuned
Pile dataset
GPT-neo 125M
125M
Pile dataset
APPS train dataset.
Table 1: Code-generation models, their sizes, the mode of usage (fine-tuned or few-shot), and the datasets used for pretraining and finetuning.
code. solution to code generation task is sequence of tokens that together form a program to solve the given task. Existing datasets additionally contain set of input-output pairs that are used to test the correctness of generated program. Figure gives an example code generation task, the corresponding reference solution, and the unit tests taken from the APPS dataset [25].
Models: There are several existing pre- pre-trained LLMs in the literature that are suitable to generating code either in a few-shot manner or after finetuning, A code generation model F provides way for to sample programs for given task G S, ~ (5 G) where Pp. denotes the probability distribution induced by the code generation model F.
In this paper, we study four different code generation models-(i) Codex, (ii) GPT-J (6B), (iii) GPT- Neo 1.3B and (iv) GPT-Neo 125M (Table 1). Codex is the largest state-of-the-ar code generation model that is publicly available for querying through an API and has shown impressive performance in code generation [11]. It is built on top of GPT-3 language model architecture and is trained on 180 GB of GitHub data. GPT-Neo and GPT-J are open-sourced models with the number of parameters ranging from 125M to 6B. These models are pre-trained on the Pile dataset (800 GB of natural language corpus with 8% of GitHub data). Since, these models are poun-e, is possible to finetune these models on downstream dataset such as the APPS dataset. In addition to the above models, there are other code- gedereenerttion models such as AlphaCode [28] and Google's 's model [3]. While our approach is applicable to any of these models, in this study, we validate the effectiveness of CODER ANKER on set of state-of-the-art code generation models that are publicly available, for reproducibility.
Code generation models are evaluated based on functional correctness rather than exact/- fuzzy match to reference program. This is because match-base metrics are unable to account for the large and complex space of programs functionally equivalent to a reference program. Functional correctness is estimated by checking whether a sampled program passes set of unit tests. Prior approaches evaluate functional correctness using the pass@k metrie; given & generated program samples per task, a task is considered solved if any of the samples passes the unit tests. The pass @k metric measures the total fraction tasks solved. Additionally, we define the exec k metric which computes the fraction of the tasks for which there exists at least one program in the k samples that executes without any compile/runtime errors, i.e., produces non error value for each of the test inputs, but may may not match the desired output.
Metrics:
2.2 Code ranking
Given n sampled programs using a code generation model, S1, S..... SSa ~ Pr(S) G), the goal of CODERANKER is to find ordering of the programs So, So., such that to compute the ranked pass@) @ for & < n, a problem is considered solved if any program in the set {S.,. passes the unit tests. A CODERANKER model R takes input code generation task G and sampled program S, and outputs a score 8;- The ranked ordering of the sampled programs is given by So.. ,So, such that so, > Soz
Soy}
So..
3 Fault-Aware Neural Code Ranker
CODER ANKER model is classifier that trained classify a pair (G, S.) as CORRECT or not, where CORRECT means that S; satisfies the task G with respect to its unit tests. The score 8; is computed as 8; = PR[CORRECT]G, S.) where Pn the probability according to the ranker model R. This probability is extracted using the real values from the last layer before the SoftMax layer.
3

---

Code generation model used for generating the dataset
Training data
Validation data
Test data
W
F.
E
Codex
70K
180K
140K
16K
26K
18K
19K
280K
200K
GPT-J
20K
210K
150K
3K
32K
23K
2.4K
210K
260K
GPT-Neo 1.3B
10K
170K
200K
1.5K
28K
27K
0.7K
170K
310K
GPT-Neo 125M
10K
150K
220K
0.8K
23K
34K
0.2K
140K
340K
Table 2: Ranker dataset distribution for the datasets generated using the 4 different code generation models on the APPS tasks. C: # CORRECT programs, W: # WRONG programs, # intent errors, and E: exccution errors.
Generated Program
Full error message
Labels
B: wrong
def number_ number_prapertylee property(n): if % 2 == I: return True return random.choice(n)
TypeErrort conjereor"o of rype 'int' has len()") ) at Line 2
execution error
I:
E: TypeErrar
L: Line
B: wrong
def gematria string ):
return " joii (sorted ( (string. lower ().
Expected output is 775, but gen- erated output is 'vole"
intent
I: Output
key=lambda item; itemm
reverse TTuee))llwwerr
-1
Table 3: Examples from the fault-aware ranker dataset showcasing the generated program, the full error message obtained by executing the program, and the fine-grain fine-grained labels extracted from this error message. Each entry contains a binary label (B) that classifics the entry as CORRECT or WRONG, a ternary label (T) that distinguishes between CORRECT, intent error, and exccution error, an intent error label (I) that specifics the type of the intent error (or -), execution error label (E) that specifies the type of the exccution error (or -), and line number (L) that corresponds the erroneous line in the code (or-1 if there execution error).
3.1 Code Ranker Dataset
To train CODER ANKER model, we need dataset that has both CORRECT and WRONG (i.e., not correct) programs. To collect these programs, we use the code-generation models to sample n 100 programs for each task in the training dataset. We then execute the sampled programs on their corresponding unit tests to generate the classification labels. Following the observations from [18], one has to be careful to not ever-train the base code generation models to ensure diversity in the sampled programs. Hence, we only finetune the base code generation models for a maximum of epochs and chose a checkpoint that results in the lowest validation loss (this does not apply to the Codex model, which is used in a few-shot manner). Table 2 shows the distribution of the ranker's datasets obtained by using the 4 different code generation models for the tasks in the APPS dataset As one can expect, the ranker dataset is highly imbalanced with about 5X to 40X more WRONG data points than CORRECT data points and this ratio is higher for smaller models such GPTT-Neo models.
Fault-aware ranker dataset: A straightforward ranker is one that is trained to predict a binary label CORRECT or WRONG. However, program fails for various reasons and knowing why program might fail is crucial to predicting whether a program is CORRECT or not. Therefore, we designed a fault-aware ranker dataset. When we execute program more than just a single bit of information. In fact, when the unit test fails, we know if it failed because of compile/runtine error (which we call an execution error) or because the program produced wrong output for a particular input (which we call an intent error). Table also shows the distribution of intent errors and execution errors in the ranker datasets. An interesting observation is that the percentage of execution errors decreases for larger code generation models.
of unit tests, the compiler message is
It is possible to further break down the WRONG datapoints. Within the execution error class, the compiler message tells us exactly the type of the execution error (such as an IndexError or a TypeError or TimeOutError) and the line in the program that caused this error. By parsing the error message from the Python compiler, we derived 10 most frequent classes of execution errors as shown in Table 4 and Figure 2. Similarly, for the intent error class, we know how the generated output differs from the expected output (such as wrong type or wrong length of the array output). We manually
We used subset of 600 problems as the validation set, chosen randomly from subset of the original training problems for which the best model, Codex, produces at least one correct program in its 100 samples.

---

Class
Description
(a) NameError
NameError
Undefined variables
def code(arr)
Operation/Tunction received an argument of in- appropriate value
if 'hyphen' in arr:
ValucBrror
return [arr [[]] +
# name 'I' is not defined in line
ЕОFЕито
Raised because of extra input() functions
(b) KeyError
An operation function is applied to object of inappropriate type
TypeError
def aaage__standingsteeeee
return {i+1: teams[-1-1]
IndexError
Array index out of bounds
i range len (teams))
KeyError
Key in dictionary is not found
# KeyError( (1) at Line
TimeoutException
Code inefficient or doesn't terminate
(c) TimeouiException
SyntaxError
Parser encountered
def fibonacci [n):
The function expected by the unit tests is not found
if
[0. 1] else
Function not found
fibonacci (n 1) +
fibonacci (n 2)
Misc
Other execution errors
#
inefficient implementation
Table 4: Different classes of execution errors. See https://docs python syttho.Figure, 2: Examples of execution org/3/1 /ttp///baar/eecppttio/o/// html for more details.
emror..
designed 9 most frequent classes of intent errors by looking at various expected and generated outputs from the training examples (see Table 5 and Figure 3). These fine- gra-irri execution- labels constitute our fault-aware dataset. Table 3 shows few data entries with all the labels. Figure 6 in the Appendix shows the distribution of the various classes of execution errors and intent errors for the ranker dataset obtained using the Codex model.
3.2 Code Ranker Tasks
We now describe the different classification tasks derived from the above dataset. For each classifica- tion task, the input is pair of code generation task specification G and generated program S; and the output one or multiple labels where each label belongs to pre-determined set of classes. We describe the different labels/classes of the various tasks below. These tasks are designed to explore the trade-offs between having abstract classes of failures versus having fine-grained classes of failures. Binary (B): The output is single binary label with two classes {CORRECT, WRONG).
Ternary (T): The ternary task splits the WRONG class into intent error and execution error classes: this forms three-class classification task with output labels [CORRECT, intent error, execution error) Intent Error aware (I): The intent error aware task splits the intent error class in the ternary task into its 9 different sub-classes, thus has total of 11 classes for the output label.
Execution Error aware (E): The execution eтrоr aware task is similar to the intent error aware task but instead of splitting the intent error class, sub-classes, thus has a total of 12 classes for the output label.
split the execution
class into its 10 different
Execution Error + Error Line aware (E+L): This is multi-class and multi-label classification task which combines two classification tasks. The first one is the execution error aware task described above. The second task predict the line of the code that corresponds an execution error. The labels for the error line number also includes to represent there is no execution error.
3.3 Code Ranker Models
We implemented fault aware CODERANKER models by finetuning the pretrained CodeBERT [21] model.
CodeBERT: It is state-of-the-ar pretrained BERT-style code-understanding model trained on the CodeSearchNet dataset [26] using a combination of masked language modeling and replaced token detection objectives [17]. It takes input concatenation of two segments with special separator token, namely (CLS), W1, W2,,, ..W, [SEP], C1. C2.... Cma [HOS]. Usually, one segment is a natural language text, and another is code. [CLS] is special token, whose final hidden representation be treated as the aggregated sequence representation for classification or ranking downstream tasks.
Adding a classification head: We add classification head on top of a base CodeBERT model by connecting a linear layer and softmax layer to the hidden representation of the [CLSS special token. Let C' e H be the final hidden vector corresponding to the [CLS] token and let W e BB K be the
5

---

Class
Description
(a) NoneError
NoneError
Generated oulput is None while ex-
def consecurtive_sumlee _nmuniie____mmmmm:
pected not
sum
EmptyError
Generated output is an empty arry while expected is not
len (num))
for range(n-1):
0 and
OutparTypeFrrror
Different types of outputs
return sum
LengthError
Arrays/Dicts/Sets of different lengths
# return
never gets triggered
(b) LengthError
IntSmallError
Integer outputs that are different from the expected ones with delta < 10
diamonds and airrood__nn__tdddeeeeeeeeec toadd((enteenceefairy....
return dict(zip( (zip( FRuby,Crystta
Eroo
are different from the expected ones with delta > 10
Integer outputs
0))
# Got ('R: 0. 'u: bb': 1. 'y's 2).
StringSmallError
String outputs whose length different from the expected ones with delta S 3 String outputs whose length different from expected ones with delta Other intent error
# expected ('ruby crystal 2]
StringSmallEErro
Stringl FrrgeFrrr
smash(words):
return Join (word for word words)
# 'helloworld
Misc
# expected 'hello world"
Table 5: Intent error classes.
Figure 3: Examples intent errors.
ranked pass
ranked pass @
ranked
Code gen. model
pass @ 100
pass @
pass @
cxec @
exec
Codex
100
26
39.6
56.4
63.5
69.7
87.0
GPT-J
48.8
5.1
11.0
15.6
21.7
60.4
82.9
GPT-Neo 1.3B
34.6
2.6
52.1
85.6
GPT-Neo 125M
23.6
1.4
6.5
5.2
11.4
41.1
58.9
Table 6: Results the APPS validation dataset about how fault-aware rankers improve the pass@ @ass@5, 5, and exec@ performance for various code generation models. We use the best ranker model for each code generation model for this result. Pass@ 100 for the Codex model on this validation set is 100% by design.
weights of the newly added classification layer where K is the number of classes in the classification task. The logits for the classification output are computed as softmax (CT and we use a standard cross-entropy loss for finetuning all the weights.
Adding a line prediction head: an error line vector S IR' during fine-tuning. The probability of newline ("In") token being the erroneous line is computed as a dot product between T, and S followed by a softmax over all of the where T; is the final hidden vector corresponding to newline tokens in the code, i.e., P: E5
To predict the line corresponding to an execution error, we introduce
the ith newline token. We include newline token at the beginning and the end of the input indicate the case where there no erroneous line the code (i.c., code does not result in an execution error) and to indicate the where the erroneous line is beyond what is encoded in the input (this occurs the task+code context cannot fit within the 512 token limit CodeBERT), respectively.
Evaluation
We next evaluate our CODERANKEE approach. We investigate (1) how fault-aware CODERANKERS can improve various code generation models on various code datasets, (2) the impact of the different
ranked pass
ranked pass
ranked @
Code gen. model
pass 100
pass
@
exec@]
Codex
24.1
3.8
4.5
9.2
10.2
59.6
73.4
GPT-J
7.2
0.5
0.8
1.6
2.6
45.4
63.8
GPT-Nco 1.3B
3.0
0.14
0.3
0.53
1.1
35.2
73.7
GPT-Nco 125M
1.5
0.04
0.17
0.5
28.5
43.9
Table 7: Results on the APPS test dataset. We use the best checkpoints for the code generation models and the ranker models based the results the validation set.

---

ranker tasks, and (3) the effect of mixing ranker datasets generated by different code generation models.
4.1 Experiment Setup
Code generation datasets: We consider three existing code generation datasets for our evaluation: (I) APPS [25]: collection of 5000 training and 5000 test tasks collected from coding competitions and interview problems, (2) HumanEval [11]: a set of 164 test tasks, and (3) MBPP [3]: a set of 974 mostly basic python programming tasks with 474 training problems and 500 test problems.
In our experiments, we only use the APPS dataset for finetuning the code generation models and the CODER ANKER models (since it is the largest dataset). But we evaluate these models on all three sets of test tasks. The APPS dataset does not come with a validation dataset, so we used set of 600 tasks from the original training dataset for validation; these are, then, excluded from the training dataset. Since we are interested in explicitly evaluating the ability of a ranker to distinguish CORRECT code from WRONG code, we chose our validation set to only include problems for which Codex model (in a few-shot manner) can generate at least one correct program in its 100 samples. To facilitate the transfer of GPT-J and GPT-Neo finetuned models the HumanEval and the MBPP datasets, perform a minor programmatic transformation of the task descriptions to match the APPS style.
We use the pass@1 1, pass @5, exec 1, ranked pass@1 1, ranked pass @5, and ranked exec @ metrics (higher values are better). See Section for their definitions. We also show the pass @ 100 metric to illustrate the maximum possible value for the pass@) @k and ranked pass @K k metrics. These metries are measured using an unbiased estimator from 100 samples as proposed by [11].
Metries:
Training setup and ppper-parameters: We finetuned GPT-J and GPT-Neo code generation models on the APPS training dataset for 2 epochs with a batch size of 256 and learning rate of Ie-5, and chose the checkpoint that has the lowest validation loss. For inference, we used temperature sampling with T' 0.8 for Codex model and 0.9 for the GPT-J and GPT-Neo models unless specified otherwise. We chose these temperatures to maximize diversity in the 100 samples, but we also conduct an ablation with lower temperatures in Table 1. For each program, we sample 512 new tokens and truncate the generated program by special stop sequence that we used in the few-shot/finetuning prompts.
We finetuned the CODERANKER ANKER models for 30 epochs with a batch size of 512 and a learning rate of le-4, and chose the checkpoint that results in the best ranked pass metric on the validation dataset. We used class weights to balance the different classes while training the rankers. All experiments are conducted on V100-32GB GPUs
Notation: We use the notation RY to denote a CODERANKER model trained on a dataset obtained using the code generation model X from Table 2 and one of the five ranker tasks Y from Section 3.2.
4.2 Main Results: CODERANKER improves code generation models
APPS validation dataset: First, we analyze the results on the APPS validation dataset of 600 tasks. Table shows the performance on various metrics for the 4 different code generation models. These results use the best CODERANKER model for each code generation model which is shown in Table 10. From Table 6, we find that CODERANKERS improve all the metrics for all the code generation models despite their different sizes. The pass @ performance increases by 5.1 % to 13.6% with CODERANKER and the models can solve about to 80 more tasks when it has to select only one program from the 100 samples. Another interesting observation is that GPT Neo 125M model when combined with CODERANKER (another 125M model) beats a GPT-J model (with 50X more parameters). These results show the effectiveness of CODERANKER on improving the code generation models.
APPS test dataset: Our results on the APPS test dataset of 5000 tasks is shown in Table 7. The test problems are harder than the ones on the validation set, which we can see by the smaller pass @ 100
'e chose Codex for this filtering step because it is the best performing model out of the 4 models we considered.
7

---

ranked
ranked
ranked
Code gen. model
pass @ 100
pass @1
pass
exec
pass@
pass
exec
Codex
88.4
26.3
32.3
50.5
61.6
77.0
86.6
GPT-J GPT-Neo 1.3B
45.1
9.1
11.6
19.1
18.9
73.6
89.0
19.5
3.2
6.1
7.7
8.5
66.3
87.8
GPT-Nco 125M
12.8
0.84
3.0
3.0
6.1
52.3
58.3
Table 8: Results on the HumanEval dataset showing the zero-shot transferability of the rankers trained on APPS.
ranked pass
ranked pass
ranked exec
Code gen. model
pass @ 100
pass @
pass @ 5
exec @
Codex GPT-J
84.8
36.4
41.8
60.9
62.4
74.8
93.2
60.4
12.1
14.2
28.9
28.2
73.3
80.8
GPT-Neo 1.3B
40.0
3.6
5.0
11.9
16.2
58.3
75.2
GPT-Neo 125M
6.8
0.2
0.8
0.9
2.6
7.6
14.2
Table 9:
Results the MBPP dataset, showcasing another instance where fault-aware rankers can be used a zero-shot manner on different dataset.
and pass@ numbers. Hence, the improvement from CODERANKER is smaller in scale, but it is still a significant improvement; Codex's 's pass@1 1 increases from 3.8% to 4.7% (35 additional problems).
HumanEval and MBPP datasets: We measure the transfer ability of CODERANKER MelaastssHuumaaaee HumamEval (results in Table 8) and MBPP (results in Table 9). We can again different datasets
two
that CODER ANKERS improve the performance of all code generation models all metrics for both datasets (one exception is pass @ for GPT-J on MBPP). Codex model's pass @ performance increases by 6% both datasets. These results show the ability of CODER transfer out-of- distribution and also showcases that the errors made by the code generation models on different tasks universal.
Exee@ vs @ all the above results, the improvements than the improvements in the pass@ I metrie; this shows that CODERANKER is better identifying execution errors in code than intent errors, which is expected since executing code is shown to be an inherently hard task for language models [3, 34].
exec metric are higher
4.3 Ablations
Effect of sampling temperature: On the HumanEval and the MBPP datasets, we additionally experiment with different temperatures sampling the 100 programs (see Table 11). As expected, noticed that pass @ 100 decreases with lower temperatures, but pass @ increases. We found that CODERANKER further increases the pass @ performance for 3 out of the setups and achieved 42.7% ranked pass @ I on the HumanEval dataset-the best known result far this dataset [16]. On the MBPP dataset, under low-temperature sampling setup, we found that CODER ANKER slightly decreases the pass@ @1 1 performance- this attribute the small difference between the pass @ and the @ 100 metrics, which makes it hard for learned ranker to beat random ranking scheme.
Analyzing different ranker tasks: Figure 4 shows 4 training curves; one for each code generation model, X, showcasing the validation ranked pass@1 curves for the model X when combined with the 5 different rankers DD Results in tabular format can be found in the Appendix. From the curves, we can notice that for larger models such as Codex and GPT-J, the rankers trained on the ternary classification task R² perform the best, and for smaller models such GPT-Nco 1.3B and GPT-Neo 125M, the ranker trained on the intent-aware classification task R! and the ranker trained on the execution-avee We can also notice that the binary rankers R. perform significantly worse especially with smaller models. These results suggest that when training rankers dataset that has more WRONG code, harder classification tasks act as better regularizes. Figure in the Appendix shows similar training curves for the ranked exec @ metrie; here, we can that R. and RE+L always achieves best ranked exec@ because they are trained to identify execution errors and RB and Rd have the worst performance.
error line classification task RE+L perform the best, respectively.
this excludes approaches that execution during inference

---

Code gen. Model
Best Ranker
ranked
Codex
Setup
pass 100
pass
pass
GPT-J
Dares
R
HumanEval, Temp=0.8
26.3
32.3
GPT-Neo 1.3B
R g++
Dar
HumanEval, Temp=0.2
69.5
35.2
42.7
GPT-Neo 125M
R RDrrneas
MBPP, Temp=0.8
84,8
36.4
41.8
MBPP, Temp=0.2
70.0
47.2
46.8
Table 10: Best CODERANKER model for each code generation model based on best ranked pass @1 on the validation dataset.
Table 11: Results with different sampling temperatures for the Codex model with rankers on HumanEval and MBPP.
GPT) rannes passrges
Code gen. Model
0.30
0.00
Neo 125M
1.3
0.25
Codex
No0
Codex
0.15
0.04
Doven
1.1
5000
10000
15000
20000
5000
10000
15000 20000
Stop
Stop
R
Care
1.0
Neo 1.30 passps
GPT-Neo 125M ranked paseeg
Ranker Madel
bast
0.9
0.05
0.06
R best
0.00
0.7
0.04
0.03
R
Detert
0.6
0.02
A Best
0.5
5000
10000 15000
20000
5000
10000 Step
15000
20000
Step
Figure 5: Results showing how the ranked pass I changes with differ- ent ranker datasets.
Figure 4: Training (smoothed) measuring ranked pass @1 training steps for various rankers on different code generation models.
Analyzing different ranker datasets: this experiment, we measure the impact using the rankers trained data from code generation model another code generation model and the effect of mixing these ranker datasets. We have 4 different code generation models in this experiment and hence, 4 different ranker datasets. We analyze two different mixed datasets (i) mixed-small that randomly samples 25% of the above ranker datasets and combines them, and (ii) mixed-larg that combines all of the 4 ranker datasets into one; the former represents dataset that is roughly the same size of the individual ranker datasets for a fair comparison, while the latter makes use of all the available data. Figure 5 shows the ranked pass @ 1 of all the 4x6 combinations in the form of a heat map. Each column has been normalized such that the values for the same model the diagonal values are 1). From the figure, among the individual ranker datasets. we notice that the rankers trained with the data from the same code generation model are the best and that the rankers trained with data from code generation models with a large size difference are the worst. Finally, the rankers trained on mixed-small dataset perform slightly worse (except for GPT-J code gen. model) than using the same model dataset, but the rankers trained on mixed-large dataset have the overall best performance. These results suggest that while it is usually better to use the same model for generating the ranking dataset, we can potentially use smaller models (less expensive models) to augment the ranker dataset to further improve the performance.
(i.e.
Additional qualitativee/quanitatii analysis of the CODERANKER approach be found in the Appendix.
Related works
Code generation tasks/models: There are several code generation models explored in the lit- erature, ranging from decoder-only architectures [11, 3, 39, 6, 22] to encoder-decoder architec- tures [28, 33, 2, 38] of various sizes. Similarly, there are several task datasets from multiple domains, including math- math-worrdproblems [3, 18), Jupyter notebook cell generation [10], common programming tasks [3, 11, 35] and competition level programming problems [28, 25). this paper, we evaluate CODERANKER'S ability to improve the performance of four decoder- based code generation models

---

on three programming datasets. Our approach is agnostic to code generation model architectures as long as they generate outputs by sampling and our approach can be easily extended to different domains.
Code understanding tasks/models: Besides code generation tasks, there are several code under- standing tasks including code search [4, 8, 23, 26], clone detection [37, 30J, code summarization [26], code translation [13, 27, 32] and defect detection [9, 5, 15, 40, 5J. Encoder-only code understanding models [29, 21, 2, 38, 24] are developed for these tasks, Our CODERANKER task can be viewed as new code understanding task, and our ranker model finetuned from CodeBERT [21]. The defect detection datasets are closely related to ours; the main difference is that prior work focused on finding vulnerability in human-written code, while we focus on detecting errors in model-gennrated code.
Filtering/ranking for code generation models: Previous works such as [11, 28) use execution to prune code completions during inference. While [11] only uses unit tests provided as part of the task, [28] additionally uses a neural model to generate inputs for the unit tests and uses execution to filter out programs that produce the same outputs on those inputs. However, these work requires executing potentially vulnerable code for every inference task. Our approach bypasses execution inference time with the neural ranker. Similar to our work, [18, 36] propose a neural-network based ranker/verifier for code generation models; the main difference is the domain (general purpose programming tasks in our case versus math problems in prior work) and we train our neural rankers to learn why/how code fails rather than just predicting binary label. Moreover, [18, 36] use generative models as their ranker base; in our work, we show benefits of rankers using a simple encoder-only model with only 125M parameters [18) uses models with atleast 3B parameters) (see Section A.2.3 in the Appendix to see our ablation on different ranker architectures).
Using execution/static analysis in other contexts: to guide code generation by conditioning the generation on a representation of the program states [12, 20]. There are also attempts to replace the execution process with a neural model in these cases [14, 34]. Another relevant work is [31], which improves the code generation models by using static analysis to augment the model's input and has shown to drastically reduce the number of execution errors made by the generated programs. This paper takes an alternate approach by using a neural ranker model to prune out wrong programs at inference time.
There has been other works on using execution
6 Conclusions and Future Directions
We presented CODERANKER that ranks programs generated by a code generation model without explicitly executing the programs. Our rankers are fault-aware i.e., trained to predict the fine-grained classes of failure modes and we showed the effectiveness of the fault-aware rankers in improving pass@k k and exec@k @k metrics for various code generation models and tasks.
One of the main limitations is that the CODERANKER approach is not sound, i.e., a ranker can classify correct program as wrong and vice-versa Additionally, we incur extra inference time to get better performance, because we now have to generate n >> k programs to filter k programs to show to the user. Our current approach also relies on sampling full programs from the code generation model before using CODERANKER In the future, we want to investigate ranking/classifying partial programs, which can in-turn reduce the inference time by pruning out wrong programs early. Another future direction for our work is to investigate other ranker model architectures such as those that can better levarage the code structure and exploring other ranker tasks such as generating the full error message. Finally, it will be interesting to investigate the transferability of our CODER ANKER approach to generic programming tasks (rather than just competition programming). A main challenge. here, is the lack of a generic programming dataset which needs to collected by scraping public sources such as GitHub. Our preliminary experiments on such a generic programming dataset show that existing code generation models usually produce more programs with execution errors rather than programs with intent errors . This observation combined with our results in this paper that show CODERANKER is usually better at identifying execution errors than intent errors, we are very optimistic that fault aware CODERANKER approach would also improve the code generation for generic programming tasks.
We thank Todd Mytkowicz, Piali Choudhury, Rahee Gosh Peshwaria, Curtis von Veh, and Xiaodong Liu for helpful discussions this work.
Acknowledgements:
10

---

References
[1] Github copilot: Your ai pair programmer. https://copilot. http://oooplot..ituuu.u.. cttthub.com....... 2021.
[2] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre- training for program understanding and generation. arXiv preprint arXiv:2103.06333, 2021.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07732, 2021.
[4] Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint 17Xiv:170..0225, 2017.
[5] Zeki Bilgin, Mehmet Akif Ersoy, Elif Ustundag Soykan, Emrah Tomur, Pinar Çomak, and Leyli Karaçay. Vulnerability prediction from source code using machine learning. IEEE Access, 8:1 8:150672-150684, 2020.
[6] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arX1v:2204.06774, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187 33:1877-1901, 2020.
[8] Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. When deep learning met code search. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 964-974, 2019.
[9] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are there yet. IEEE Transactions Software Engineering, 2021.
[10] Shubham Chandel, Colin B Clement, Guillermo Serrato, and Neel Sundaresan. Training and evaluating a jupyter notebook data science assistant. arXiv preprint arXiv:2201.1290) 2022.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2 2021.
[12] Xinyun Chen, Chang Liu, and Dawn Song. Execution eeecution-guided neural program synthesis. In International Conference on Learning Representations, 2018.
[13] Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. Advances in neural information processing systems, 31, 2018.
[14] Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-speciff languages. Advances in Neural Information Processing Systems, 34, 2021.
[15] Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui. Deepwukong: Statically detecting software vulnerabilities using deep graph neural network. ACM Transactions on Software Engineering and Methodology (TOSEM). 30(3):1-33, 2021.
[16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
[17] Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Flectra: Pre-training text encoders discriminators rather than generators, arXiv preprint 10X1v:2003.10555, 2020.

---

[18] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[19] Jacob Devlin, Ming Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810, 10.04405, 2018.
[20] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar- Lezama. Write, execute, assess: Program synthesis with a repl. Advances in Neural Information Processing Systems, 32, 2019.
[21] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint wxXi::2002088555 2020.
[22] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022.
[23] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. In 2018 TEEE/ACM 40th International Conference on Software Engineering (ICSE), pages 933-944 IEEE, 2018.
[24] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009 08366, 2020.
[25] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 2021.
[26] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.0944 1999.9999.9946,.... 2019.
[27] Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-bases pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
[28] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814 2022.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprini teXiv:11077 1692, 2019.
[30] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Thirtieth AAAI conference artificial intelligence, 2016.
[31] Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas Reps, Swarat Chaudhuri, and Christopher Jermaine. Neural program generation modulo static analysis. Advances Neural Information Processing Systems, 34, 2021.
[32] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Divide-and-conque approach for multi-phase statistical migration for source code (t). In 2015 30th IEEF/ACM International Conference on Automated Software Engineering (ASE), pages 585-596. IEFF, 2015.
[33] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:200.13347, 2022.
12

---

[34] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez. Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
[35] Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. Programming puzzles. arXiv preprint wrxiv:2106.057885 2021.
[36] Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021.
[37] Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476 476-480. IEEE, 2014.
[38] Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.0083 2021.
[39] Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoom. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202./3100 2022.
[40] Yaqin Zhou, Shangging Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semanties via graph neural networks. Advances in neural information processing systems, 32, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our approach increases inference time mentioned in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The code and /ooeeanker.
data is released on GitHub https: : / //g/ titb...........
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] It is very expensive to run the experiments multiple times. But, we tried multiple runs for one of our rankers in early experiments, and we did not see much variance in the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report the resources used in Section 4.1 and in Section A.2.3.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
13

---

(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The assets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "qNMK4HhlOt", "reviewer": "Reviewer_xLPt", "summary": "This paper proposes to train fault-aware neural code rankers (I would prefer to call them classifiers) that can predict the correctness of sample programs without executing them (using unit tests). The rankers are trained to predict different execution information, such as compile/runtime error types (e.g., an IndexError or a TypeError). The paper shows that the trained rankers can significantly increase the pass@k accuracy of various code generation models, e.g., Codex, GPT-Neo, GPT-J on popular benchmarks, e.g., HumanEval, APPS, and MBPP.", "questions": "1. Given that the trained rankers are actually classifiers, how exactly the top-1 candidate is chosen to compute ranked pass@1? I assume the ranking is done based on code generation models provided likelihood score, and then wrong samples were filtered out based on the predictions of the fault-aware classifiers. I think this is an important piece of information that was not clarified.\n\n2. Out of curiosity asking this question, since the experiments were conducted on V100-32GB GPUs, what confined the authors to choose better models than CodeBERT (e.g., CodeT5, PLBART) for the ranking models? In fact, the authors could have used GPT-Neo 1.3B as the rankers.", "limitations": "The major limitation of this work is the conflict between the motivation of the work and what is being proposed. While the paper motivates that in the real world, we cannot assume that there will be unit tests readily available to evaluate models via execution. However, the paper used the APPS dataset composed of unit tests. The authors didn't discuss how their work could be leveraged in the real world. For example, can we use the authors' proposed idea for generic programming (not competitive programming only)? I would prefer to see some discussion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Significance**\nThe paper pointed out that current literature assumes that the unit tests are generally available to evaluate code generation models via execution and it is safe to execute the generated programs. The paper argues that such assumptions are impractical in real-world software development, making the work significant to investigate alternatives.\n\n**Originality**\nWhile I am unaware of any prior works that exactly do the same (training rankers to predict errors), the proposed idea is intuitive (anyone can come up with the idea with careful thinking). Though I see this work's technical contribution as thin; I appreciate the author's effort to study the effectiveness of the proposed approach.\n\n**Clarity**\nThe paper is written fairly clearly. Except for one crucial piece of information, I did not feel anything important was missed. Also, the paper should rather use classifiers in place of rankers.\n\n**Quality**\nOverall the quality of the work is good. However, I felt the authors didn't go deeper in investigating stronger code representation models (they chose CodeBERT, which is a weaker model in literature; I would love to see a  few more models being investigated as rankers). This was important since there is a significant difference between pass@100 and ranked pass@1 on the benchmarks.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iRPneE4ZmML", "reviewer": "Reviewer_SQmW", "summary": "This paper complements a deep learning based program synthesis module with a separate fault aware ranker framework that learns execution pattern of the generated program without actually executing it. The work is inline with recent advancements in large language models that show the inherent understanding of the problem by the model but its inability to search for the correct user intent based on the first greedily sampled decoder output. Hence it is necessary to use complex search strategy, which in the case of this paper happen to be another neural network. The results shown in the paper show consistent improvements in potential user experience.", "questions": "1) It will be great to understand the effectiveness of the ranker framework in detecting user bugs in held-out datasets. This will validate the robustness of the ranker framework. Are there any preliminary experiments in that direction?\n\n2) How will the trained model on CodeBERT compare against using some baseline methods, some generative models were mentioned in the related literature. Can they be compared? If not, is it possible to fine-tune any autoregressive model on this task and compare against the current architecture?\n\n", "limitations": "The authors have adequately addressed the limitations. Some preliminary analysis on the limitations discussed can benefit the reader in understanding the robustness and current state of the error detection module developed by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has some noteworthy strengths. They are the first to effectively use neural techniques to classify intent errors for general purpose programs and do not restrict themselves to simpler domains. This is an important finding as it can lead the community into building better models for detecting code errors, which can be used to judge model generated code as well as human written code. They set a strong ground work for detecting code errors using deep learning models and show the potential of such approach by experimenting over a diverse set of domains and diverse set of SOTA models. Finally they show the effectiveness of complementing such synthesis models with ranker systems, thereby improving their applicability in real world scenarios. \n\nThe paper show the effectiveness of the approach in detecting bugs in model generated code thereby improving those systems. The presented results are to some extent expected, given that the model is augmenting the strong generative models with additional data of detecting intent errors. The system is also developed with the motivation of detecting errors early when they are generated by a generative model such to avoid unnecessary scenarios resulting from executing buggy codes. It is not exactly clear if running the system in a sandbox environment will completely remove any necessity of using this approach. Finally, the experiments were run solely with the purpose of improving the performance of code generation models, it will be interesting to find the effectiveness of the ranker in detecting bugs on unseen real code.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gBEI8iSSLf9", "reviewer": "Reviewer_DbiK", "summary": "This work trains a model to predict errors in generated code and rerank the generations. Current models are often evaluated using the pass@k metric, which assumes that the unit tests can be run on k generations to rank them. They motivate their method with these arguments:\n1) Not test dependent (it could fail on unseen tests)\n2) No burden to create tests\n3) No need to install dependencies\n4) No security risk linked to execution\n\nTheir model is trained on a dataset generated from APPS (where unit tests are available), but the model learns to find errors from the code only. They evaluate it for reranking the outputs of language models on APPS (validation and test datasets), HumanEval and MBPP. \nThey substantially improve the pass@1 scores of Codex, GPT-J, and GPT-NEO (1.3B and 125M params) on several benchmarks.\n", "questions": "* Did you do any ablation showing how much the dataset size impacts your performance in this context?\n* Does the code generation task contain the example unit tests? If so, the ranker could use the unit tests to determine the error. \n\n", "limitations": "The datasets they tested their method contain mostly functions from programming contests. It is fine, since it would be more difficult to test it on real data when no unit tests are available. However, they mention real usecases in the introduction (such as use in VSCode) and I believe that their method may not transfer as well to real data, with expected input/outputs that are more difficult to infer. I would have liked to see a discussion about that. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I found the method in this paper simple, but their motivation is compelling, it is well-written, and it is well executed. \nStrengths:\n* Clear and well-written\n* The motivation for this work is clear and compelling. Executing many generated snippets is often not doable in practice\n* Even though their reranking is still far from perfect (as shown by the difference with the pass@5), they substantially improve the pass@1 for every model and dataset, showing the robustness of their method\n* They give good insight into their model. The exec@1 improvement shows that execution errors are understandably easier to find.\n\nWeaknesses:\n* Limited technical novelty", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept