CodeT:  Code Generation with Generated Tests
Authors: Bei Chen
Fengji Zhang
Anh Nguyen
Daoguang Zan
Zeqi Lin
Jian-Guang Lou
Weizhu Chen
Venue: ICLR 2023 poster (2023)
Keywords: code generation
test case generation
pre-trained language models

Abstract: The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.

Published as a conference paper at ICLR 2023
CODET: CODE GENERATION WITH GENERATED TESTS
Bei Chen', Fengji , Anh Nguyen", Daoguang Zan, Zeqi Lin,
Jian-Guang Lou, Weizhu Chen Microsoft Corporation fbeichen, v-fengjzhang, anhnguyen, v-dazan, zeqi lin, jlou, wzchen crosofff...ccccommmm
ABSTRACT
The task of generating code solutions for a given programming problem can bene- fit from the use of pre-trained language models such as Codex, which produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre- trained language models. A natural way to evaluate the quality and correctness of code solution to run against a of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CODET, that leverages the same pre-trained language models automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CODET then executes the code samples using the generated test cases and performs dual execution agree- ment, which considers both the consistency test cases and the agreement of the outputs with other code samples. We con- duct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS, and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CODET can significantly improve the performance of code solution selection over previous methods, achieving remark- able and consistent gains across different models and benchmarks. For instance, CODET improves the pass @ metric on HumanEval 65.8%, which represents an absolute improvement of 18.8% over the code-davinci model, and an ab- solute improvement of more than 20% over the previous state-of-the of-the--rtttttttt- results.
the outputs against the generated
INTRODUCTION
Despite the remarkable progress in pre-training techniques for code generation, selecting a single correct solution from multiple candidates generated by large language models remains a hard prob- lem. For instance, Codex (Chen al., 2021), a state-of-the art pre-trained language model for code generation, can achieve a pass@ (00 (pass if one or more among 100 generated solutions for a given problem pass the corresponding test cases) of 77.4%. but pass@1 (correct rate of single lution) of only 33.5% on the HvmanEval benchmark (Chen et al. 2021 This huge gap limits the practical usefulness of code generation models and motivates us to explore how to pick the correct or best solution from multiple candidates.
straightforward way to verify the correctness of solution to execute and check it passes all corresponding test cases. This execution-guidee approach has been widely adopted in various code-related tasks, such as code generation (Chen al.) 202T, Li et al.
2022b Shh et al. 2022).
code translation (Roziere et al. 22221), and program synthesis (Chen et al., 2018 Ellis et al. 2019)
However, this approach relies heavily on the quality and quantity of test cases, which are often costly and time-consuming to create and maintain. Moreover, in real-world applications like Copilo code generation tool that assists developers in writing code, is unrealistic to expect users to provide test cases for every problem they want to solve. Therefore, propose to automatically generate test cases for arbitrary programming problems and use them to quickly verify any solution.
"The first three authors contributed equally.
We report the results on the HumanFval benchmark with Codex model code-cushman-000 More results with different models and henchmarks can be found in Section 4.1 and 4.2
https: / / thub. com/ features/ccpi lot

---

Published as a conference paper at ICLR 2023
A Programming Problem
Instruction
Pre-trained Language Model
Code Solution
Dual
Test Cose 1
Code
Code Solution
Execution
Test Cose 2
Test
Generation
Agreement
Generation
The Best Code Solution
Figure 1: The illustration of CODET. Both the code solutions and the test cases generated by the pre-trained language model. The best code solution then selected by a dual execution agreement.
In this paper, we propose CODET: CODE generation with generated Test-driven dual exccution agreement, as illustrated in Figure D First, we leverage the same pre-trained language model that generates code solutions, such as Codex, to generate a large number of test cases for each pro- gramming problem by providing an elaborate instruction as prompt. Next, agreement approach inspired by the classical RANSAC algorithm (Fischler Bolles 1981). We ex- ecute each generated code solution on each generated test case, and iteratively find multiple groups of code solution and test case pairs. Each group, or consensus set, has solutions that pass the same test cases, indicating that they have the same functionality, even if they are different in implementa- tion. We expect that solution that passes more test cases is more correct, and that a solution that has more similar solutions, i.e., solutions in the same consensus set, is more consistent with the problem specification. So, we rank each consensus set by both the number of test cases and solutions in it, and choose the best solution from the highest-ranked consensus set.
a dual execution
Our method is simple and efficient, it does not require any labelled data or additional rankers, but it achieves surprisingly exceptional performance. We evaluate our method on five different pre-trained language models for code generation: three OpenAl Codex models (Chen et al., (021),
INCODER (Fried et al., 2022), and CODEGEN (Nijkamp et al., 2022). , as well as four established benchmarks for code generation: HumanEval (Chen et al. 2021), J. MBPP (Austin et al. (02I),
APPS (Hendrycks et al. 2021), and CodeContests (Li et al. 2022b). The experimental results show that our method can effectively select the correct solution from multiple candidates, improving the pass@] 1 score significantly on all benchmarks in the zero-shot setting. For instance, CODET achieves improvements using code-davinci-0022: HumanEval (47.0% - 65.8%), MBPP (58.1% - 67.7%),
APPS INTRODUCTORY (27.2% + 34.6%), and CodeContests (0.7% - 2.1%). Moreover, when we combine code-davinci-002. the most powerful pre- pre-trained model, and CODET, we outperform previous state-of-the offte-f-thh-art methods by a large margin, c.g., HumanEval: 42.7% (Inala et al. 2022) + 65.8 65.8%. We also conduct a thorough analysis to provide more insights. Our work publicly available at https: com/microssot//Coa.....
com
2
METHODOL.OGY
The task of code generation to solve a programming problem: generate code solution æ based on context c. As shown in Figure context e contains natural language problem description in the form of code comment, and code snippet that includes statements such as imports and the function header. A code solution is code snippet that solves the programming problem described in the context. Generally, we sample a set of code solutions, denoted as X {x1, Z2, ... INN. based on the context using a pre-trained language model M, which can be formulated as X M(c)- Our goal is to select the best code solution from the set of generated code solutions X, where & is the most likely solution to correctly solve the given programming problem. To this end, we propose CODET the hope of unleashing the inherent power of the pre-trained language model M. Specifically, we use M to generate test cases for the programming problem (Section 2.1). and then select the best code solution & based on a dual execution agreement (Section 2.2
2.1 TEST CASE GENERATION
Besides generating code solutions, we also need to generate test cases to evaluate the correctness of the code solutions. A test case is pair of input and expected output for the function defined in the
2

---

Published as a conference paper at ICLR 2023
Code Generation
Test Case Generation
For ACk, cies in enurersto( suebers):
for idx2, eles2 In snuser (nuuser)))
from typing import List
Context
assert has, mos_elose.cr (lt.eeesseeseeee 2.8, 3.9, 4.8, 5.0, 2.21, 0.3) True
if idk 1- 10x2:
def has close, wwwss__eeeeettteteeeeeeeeeeeeeee (nurbeess List[flooe], threshold: float) >> bool:
distance abs(cler elon2] if distance < threshold:
return
Check two numbers closen each other than given thresheld.
1ist numbers,
False
Code Solution X
Test Cose y1
Code Solution
Test Case
pass
Instruction p
check the correctness of ros_close_eleeee assert
Code Solution Xy
Test you
Figure 2: Code generation and test case generation: example from the HumanEval benchmark.
Example input-output cases are removed from the context.
context. For example, in Figure 2 a test case for the programming problem of checking whether there exist close elements in list less than threshold. To generate test cases, use the same pre-trained language model M that we use for generating code solutions, but we add an instruction prompt indicate that we want test cases instead of code solutions. As
p the context shown in Figure 2 the instruction p consists of three parts: (1) a "pass" statement as a placeholder of the function body, which signals that do not need generate code for the function, (2) comment "check the correctness of [entry point]" to clarify the intention of generating test cases, where "[entry point]" is the name of the function, and (3) an "assert" statement start the test case generation, which specifies the format of the test cases as input-output pairs.
We then feed the concatenated context and instruction, concat(c, p). to the language model M, and sample set of test cases, denoted as Y {yh, V2. ". }MM, }. from the model output. The process of test case generation can be formulated as Y M(concat(e, p)). The language model will try to complete the instruction by generating plausible input-output pairs for the function. Note that we remove example input-output cases from the context e before generating code solutions and test cases, to avoid exposing real test cases to the language model.
2.2 DUAL EXECUTION AGREEMENT
In this subsection, explain how select the best code solution å from the set of generated code solutions X = {21, x2, ".,, ). using the set of generated test cases Y = {311 9/2 ..., yMr as criterion. We defined by x on the input part of and comparing the output with the output part of y. If the code solution can be executed without errors and the output matches the expected output, then say the code solution æ can pass the test case y. Furthermore, we say there is a functionality agreement between two code solutions 24 and xzy if they can pass the same set of test cases in Y. Our approach is based on the following assumptions: (1) the code solutions and the test cases are independently and randomly sampled from the pre-trained language model MM given a certain programming problem, and (2) incorrect code solutions are often diverse, and the probability of having a functionality agreement between two incorrect code solutions by chance very low. These assumptions are similar to those of the classical RANSAC algorithm (Fischler & Bolles) 1981), which is robust method for finding consensus among noisy data. Inspired by RANSAC, we propose our approach CODET to perform dual execution agreement, which is an iterative approach as follows:
execute code solution on test case y. which means running the function
We randomly select a pair (z. y) from the set of all possible pairs D ((x, y)lz X.ye YJ. We then try to execute the code solution x on the test case If z can pass y, then we say that the pair (r. y) is a hypothetical inlier, because hypothetically describes the cor- rect functionality for the programming problem. Otherwise, we say that (rr, y) is an outlier, because it fails to describe the correct functionality. Figure shows a simple example of the programming problem "return the square of number". (x1, and (æa, 22) are two the hypothetical inliers, while (x1, 244 and (æra, y1) are two of the outliers.
If (xc y) is a hypothetical inlier, we collect all other pairs from D that agree with this hy- pothetical inlier, forming set S called consensus set. To find the pairs that agree with (a. y), we first find all test cases that x can pass, denoted as Sy. Then, we find all code solutions that can pass exactly the same test cases as x, denoted S,. Finally, the con-
3

---

return ata Published as a conference paper at ICLR 2023
neturn a**2
(1uare(1)
Benchmark
Problems
GT Tests
assert (Quare(2)
HumanEval
164
7.77
100
assert (Quare())
V3
MBPP
427
3.1
100
return a+2
assert nun square( (1)
Va
APPS
INTERVIEW
INTRODUCTORY
3,000
0.00
20.99
50
COMPETITION
1,000
X
neturn a Code Solutions
assert nun (3)
1s
CodeContestss
165
203.7
1.
Test Cases
Figure 3: A simple example of the programming problem "return the square of a number". The
Table 1: Statistics of benchmarks: the total num- ber of problems in the benchmark (Problems), the average number of ground test cases per problem (GT Tests), and the number of sampling code solutions for each problem (n).
gray line between x and y indicates that can pass y, i.e., (xr, y) is hypothetical inlier. The green or purple box indicates a consensus set.
sensus set the set of all pairs that consist of a code solution from S. and test case from Sy, i... S = {(x,,) f)1z E Sa.3 y E Sy}. For example in Figure 3 we can get S₂ (2). S. {yi, y2 y2} from the hypothetical inlier (æ1, V1) (shown in green box), and S. = {.s))
{U2, Vs. V4. 3 from (I3, U2) (shown in purple box).
We score the consensus set as f(S)- S (1S, where [Spl is the number of code solutions in S2 and |Sy] the number of test cases in Sy. This score is equal to the number of pairs in the consensus set. The intuition is that the more pairs that agree with the hypothetical functionality, the more likely this functionality is correct, according to our assumptions. Following the example in Figure the consensus set scores are 6 and 4 for the hypothetical inliers (21: y1) and (I3: y2), respectively.
We repeat the above procedure for a fixed number of times, each time producing a consensus set with its score. Finally, we get the best code solution & by selecting any code solution from the consensus set with the highest score. If we want to obtain & code solutions, we can select the top consensus sets with the highest scores, and one code solution is picked up from each of the le consensus sets.
In practice, when the number of code solutions in D is not large, we can simplify the above method by examining all possible pairs in D, instead of sampling pairs from D. Specially, for each code solution x E X, we run with every test case in Y and keep track of which test cases it passes. We group together code solutions that pass the same test cases, because they have the same functionality. This way. we divide all code solutions in X into groups based on their functionality, which we write SK ]. where K is the number of code solution groups. Each group S₂ has set of test cases that it passes, which we write as Sy. Then, we get K consensus sets, each of which has the form = (xx, y)(x E Sn,y E }, We can score each consensus set by ((S) = |S I. as before. This naive version captures the same underline intuition, but it finds all consensus sets right away, without sampling pairs repeatedly.
as X (S). S²
3 EXPERIMENTAL SETUP
Models Our experiments are based on Codex (Chen et al. 2021). ), INCODER (Fried et al. 2022)
and CODEGEN (Nijkamp et al. 2022). Codex is a descendant of GPT-3 (Brown et al. (020) and proficient in understanding the provided context and generating functional programs. We use three Codex models with different capabilities provided by OpenAl: code-cushman-001! code-davinci- 001, and code-davinci-002.. INCODER is a unified generative model that can perform left-to-right code generation and code infilling, while CODEGEN is family of large-scale language models to perform conversational program synthesis. We take use of the INCODER 6.7B version (INCODER- 6B) and the CODEGEN 16B Python mono-lingual version HODEGEN--MON--10
Metrics and Baseline We use the metric pass @ k (with samples) for performance evaluation and take advantage of ground truth test cases to determine the functional correctness of code solutions. For each problem, we sample n code solutions and then select & of them for evaluation. If any of the i code solutions passes all ground truth test cases, the problem is considered solved. Then pass @ k is the percentage of solved problems. We use the unbiased definition of pass@k our
4

---

Published as a conference paper at ICLR 2023
Methods
Bascline
AlphaCode-C
CODET
10
100
10
HumanEval
code-cushman-001
33.5
54.3
77.4
39.6
46.4
63.8
44.5 11.0
50.1
65.7 11.4
code-davinci-001
39.0
60.6
84.1
41.6
50.7
75.6
50.2
11:2
58.9
75.8 16.2
code-davinci-002
47.0
74.9
92.1
55.1
64.1
84.4
65.8
18.8
75.1
86.6
11.7
INCODER-6B
16.4 15.2
28.3 27.8
47.5 47.0
17.7
23.8
34.8
20.6
4.2
27.6
37.1
8.8
CODEGEN-MONO-IGR
29.7 29.3
50.3 49.9
73.7 75.0
27.3
38.5
64.4
36.7
7.0
44.7
59.3 9.0
MBPP
code-cushman-00
45.9
66.9
79.9
31.5
59.0
73.3
55.4
61.7
72.7
code-davinci-001
51.8
72.8
84.1
56.2
64.7
78.8
61.9 10.1
69.1
79.3 6.5
code-davinci-002
58.1
76.7
84.5
62.0
70.7
79.9
67.7
74.6
81.5
INCODER-6B
21.3
46.5
66.2
26.7
35.3
56.2
34.4
43.9
58.2
CODEGEN-MONO-166
42.4
65.8
79.1
41.0
55.9
73.6
49.5
56.6
68.5
Table @ (%) on the HumanEval and MBPP benchmarks. AlphaCode-Ci our replication of the clustering method in Li et al. (2022b). The numbers in red indicate the absolute improvements of CODET over baseline on pass and pass @ 10. We also list the baseline results from Fried et al.) (022) and Nijkamp al. (0022 for reference in gray, where the settings of context not exactly the same as ours. For CODET, temperature set to 0.8 and sampling number is set to 100. We do not show CODET pass @ 100, since it is the same the baseline pass @ 100.
baseline (Chen et al., 2021), where i solutions randomly picked from samples. Our CodeT uses a dual execution agreement mechanism to select solutions from n samples, as mentioned in 2.2 In addition, AlphaCode-C. Our replication is to use the test inputs generated by CODET, run the solutions on the test inputs, group the solutions by test outputs, and rank the clusters by size (details in Appendix [[].
include clustering method from Li et al. (2022b) for comparison, denoted as
Benchmarks We conduct experiments on four public code generation benchmarks in the zero- shot setting. The statistics benchmarks are shown in Table 1 (1) HumanEval (Chen et al.) 2021] consists of hand-written Python programming problems. The original contexts include exam- ple input-output cases, which are removed our experiments to avoid exposing real test cases. The experiment in Appendix B shows that this removal operation is reasonable and indispens- able. (2) MBPP (Austin et al. 202T) (sanitized version) contains crowd-sourced Python program- ming problems, and we follow HumanEval construct the context for it. (3) APPS (Hendrycks) et al. (021)) consists of coding problems collected from open-access coding websites, which have different difficulty levels. (4) CodeContests (Li et al. 20226) includes competitive program- ming problems scraped from the Codeforces platform. To enable zero-shot inference, we con- struct the context for APPS and CodeContests as follows: the original problem description treated comment where input-output examples are removed, and simple function header "def solution(stdin str) - str is placed after the comment to accommodate the input/output data format. More implementation details can be found in Appendix A
EXPERIMENTAL RESULTS
In this section, we evaluate CODET on five different pre-trained models and four benchmarks to verify its effectiveness, followed by test case analysis and case studies to provide more insights.
4.1 RESULTS HUMANEVAL AND MBPP
The experimental results of various models the HumanEval and MBPP benchmarks summa- rized in Table If we compare the pass @ 100 to pass@1 I on the Baseline column, it is clear that the former is significantly better than the latter, indicating the potential to select the best code solution from the 100 generated samples.
For three Codex models, when compare the CODET column with the Baseline column, CODET pass achieves an absolute improvement about 10% over the baseline pass@1 1. The improve- ments consistently above 10% HumanHval. Surprisingly, even for the strongest baseline,
5

---

Published as a conference paper at ICLR 2023
Methods
Baseline
CODET
A:
1
10
50
100
1000
2
10
100
INTRODUCTORY
27.2
46.6
59.4
34.6 7.4
41.2
53.2 6.6
APPS
INTERVIEW
5.1
12.8
23.0
8.1 3.0
11.2
18.1 5.3
COMPETITION
1.8
4.9
12.1
2.2 0.4
4.1
8.6 3.7
CodeContests
0.7
3.0
5.7
7.5
13.9
2.1
1.4
2.3
5.3 2.3
9.9
Table 3: @ass@k (%) results on the APPS and CodeContests henchmarks using code-davinci-002 in the zero-shot setting. The numbers in red indicate the absolute improvements of CODET over baseline on pass @ I, pass @ 10 and pass @ 100. For CODET, temperature is set to 0.8 and sampling number is set 50 for APPS and 000 for CodeContests.
code-davinci-002. the improvement is 18.8%, boosting the pass @ 1 to 65.8%. which is a 20+% ab- solute improvement over the best previously reported results (Inala et al.) 2022). We attribute this larger improvement to the higher quality of test cases generated by code-davinci-002,, providing deeper analysis in Section 4.3, CODET also achieves exceptional performance on the MBPP bench- mark, although the magnitude of the improvements is slightly less than that of HumanEval. Using the code-davinci-002 as an example, the pass@1 improves by 9.6%. We also report pass@2 and pass @ 10 of CODET to further show its superiority. The pass @2 results of CODET are close to the baseline pass 10 results. Meanwhile, the improvements on pass @ 10 are also consistently over 10% on the HumanEval benchmark.
The experimental results of INCODER-6E and CODEGEN- 16B further verify the effective- ness of CODET. It is obvious CODET significantly improve the pass 1, with absolute improve- ments in the range of 4.2% to 1%. INCODER-6B achieves the greatest improvement with a gain of 13.1% on the MBPP benchmark. Similar to the experimental results of Codex, the p results are close to the baseline pass @ 10. All the results demonstrate that CODET can boost the performance of various pre-trained language models consistently.
As for AlphaCode-C. C, it is consistently inferior to CODET on both benchmarks using different mod- els, demonstrating the superiority of our dual execution agreement that takes test case information into consideration. In addition, we notice that duplication exists in the generated code solutions and test cases. We perform an ablation study in Appendix Do show that de-duplication has little influ- ence on the results of CODET. Moreover, we discuss the sensitivity of CODET to the temperature in Appendix E showing the rationality of choosing a rather high temperature at 0.8.
4.2 RESULTS ON APPS AND CODECONTESTS
We also conduct experiments on two more challenging benchmarks, APPS and CodeContests. We build the zero-shot versions of APPS and CodeContests to be in line with our setting of HumanEval and MBPP by removing the example input-output cases in the problem descriptions. We employ code-davinci-00211 for code solution and test case generation. The sampling number is set 50 for APPS to save computation cost on the 5, 000 testing problems, while for CodeContests, following Li al. (202bb), the sampling number set to 000 solve especially hard problems. From the results summarized in Table 3 we can clearly observe the consistent performance improvements on both benchmarks using CODET. The absolute pass@1 improvement 7.4% for introductory problems in APPS, while the improvements are not significant for competition level problems in APPS and CodeContest, indicating their difficulties. may generate many trivial code solutions for the problems in APPS and CodeContests due to the superior difficulty of these two benchmarks. We perform a comprehensive study in Appendix F to demonstrate the robustness of CODET to this issue. Inspired by Chen et al. (2021) and Li et al. 2022b), we also conduct experiments in the one-shot setting, which is detailed in Appendix
addition, we notice that code-davinci-002
4.3 ANALYSIS ON TEST CASES
The test cases are vital to CODET since the core idea is based on test-driven execution agreement. Hence, in this subsection, we analyze the test cases by answering the following research questions.

---

Published as a conference paper at ICLR 2023
Probi Probability Density
Probi Probability litt Density
0.4
cushman-001
0.6
davinct
davinc 001
0.3
davinci-000 002
davinci 002
IcCoora
0.4
InConca
0.2
COPeGEN
CODEGEN
0.1
0.0
0.2
0.3
0.5
0.0
0.1
0.2
0.3
0.5
0.6
0.9
Test Case Accuracy
Test Case Toxicity Rate
(a)
(b)
Figure 4: The distributions of (a) test case accuracy and (b) toxicity rate for each problem on Hu- manEval. Test cases of better quality if they have higher accuracy and lower toxicity rate.
Benchmarks
HumanEval
MBPP
10
2
10
code-cushman-O00
47.1 2.6
58.6 8.5
71.2 5.5
59.7 4.3
64.8 3.1
75.5 2.6
code-davinci-001
52.0 1.8
62.9 4.0
78.1 2.3
64.3 2.4
71.7 2.6
80.5 1.2
INCODER-6B
26.8 6.2
30.4
2.8
40.8 3.7
50.3
15.9
55.4
11.5
64.5 6.3
CODEGEN-MONO-16t 47.7 11.0
54.9
10.2
71.0
11.7
60.0
10.5
67.6 11.0
76.5 8.0
Table 4: Pass@k (%) on the HumanEval and MBPP benchmarks with code-cushman-001. code- davinci-001, INCODER, and CODEGEN using the test cases generated by code-davinci-O00 The numbers orange indicate the absolute improvements pass@ k using code-davinci-000 test cases over that using their own generated test cases.
Q1. What is the quality of the generated test cases?
We evaluate the correctness of the generated test cases using the canonical solutions. A test case is considered correct if the canonical solution can pass it. Figure 44 summarizes the distributions of test case accuracy on HumanFval, where the horizontal axis represents the accuracy value for each problem and the vertical axis represents the probability density of problems with the corresponding accuracy value. We can see that the test cases generated by Codex models are of much higher accuracy than CODEGEN/INCODER, Besides accuracy, we also introduce the test case toxicity rate measurement of quality. We consider a test case to be "toxic" if any generated code solution can pass it while the canonical solution cannot. Toxic test cases may hinder the scoring of consensus sets and lead to the failure of CODET. As shown in Figure [4b] we find that the toxicity rate highly correlates to the test case accuracy with respect to different models, where the proportions of toxic test cases for Codex models are smaller than CODEGEN/NNCDER, We also evaluate the code coverage of generated test cases using two coverage criteria in Appendix H.2 where Codex models still outperform CODEGEN/INCODES with an average coverage of over 95%. Comparing the test case quality and the performance of CODET shown in Table 2 we can find that the quality of test cases strongly correlates to the performance gain using CODET concerning different models.
Q2. Can better test cases further boost the performance of mediocre models?
From the above discussion with Figure 4 we can find that code-davinci-009 is the most capable model for generating high-quality test cases. Hence, we conduct an experiment to boost the perfor- mance of the other four models mode-cushman--001 code-davinci-001, InCODER, and CODEGEN) using test cases generated by code-davinci-002. Table summarizes the performance gain with respect to different models on the HumanEval and MBPP benchmarks. In general, using the test cases generated by code-davinci-002 has significantly better performance than using the test cases generated by the less capable models themselves. For moode-cushma--oee and code-davinci-O01 the absolute improvements
in the range of 1.8% to 4.3% on pass@ 1, while for INCODER and CODE- GEN, the range from 6.2% to 15.9%. The above results indicate that the correct code solutions generated by mediocre models can be further exploited by adopting better test cases.

---

Published as a conference paper at ICLR 2023
def wello__trreshold(1: list, int): ane
def sort_arnye anttaaray(aaraa))
san
Return True 1f all nunbers in the list 1 are below threshold t.
Given an array of non-negative integers, return given array after sorting, you will sort the given array in ascending order if the sun( first index value, last index value) is odd, or sort in descending order if the sun( first index value, last index value) is even.
of
Rank #1: The consensus has solutions and 218 test cases.
Correct
if 1 []: return True
Rank #1: The consensus set has 4 solutions and 138 test cases. Incorrect for i in range((en(array)) sun 4- array[1]
Rank The consensus set has solutions and 158 test cases.
return 1[0] < and
Beloo_threesold(111
t)
sun
Correct initial_ sun sun(array(B:1)) sun(array[11:)) initial_sum sun 8: return sorted(array, reverse-True)
Rank #2: The consensus set has 30 solutions and 225 test cases.
Incorrect
1f sun X anray- wew.ssor((reeese-TTvi else: array.sort() return array
8:
for 1: 1f e > t: return False return True
else: return sorted(array)
(a)
(b)
Figure 5: Two real cases from the HumanEval benchmark with CODET and code-cushman-001.
Q3. How effective is CODET when there are fewer test cases?
When generating test cases for the HumanEval benchmark, we sample 100 times for each problem and each sample may include multiple assertion statements (i.e., test cases), de- noted as Sampling Number 100. Then we extract the first 5 syntactically correct test cases from each sample, denoted as Limit = 5. This means each problem is equipped with 500 test cases at most. The actual numbers of extracted test cases are summarized in Appendix H.I We perform an ab- lation study on the number of test cases by decreasing Sam- pling Number and Limit. As shown in Table []] clude that using more test cases in CODET could generally lead to better performance, while the performance gap nar- rows when Sampling Number 2 50 and Limit 2 More- over, CODET improves the pass @1 by 9.5% with only 10 test cases using code-davinci-002. suggesting the high test case efficiency. We can use a smaller Sampling Number in real-world application to balance the performance and com- putation cost. More results can be found in Appendix H.3
Sampling Number
Limit
10
20
50
100
56.5
57.5
60.7
62.4
2
62.2
62.8
63.2
63.6
3
62.9
63.2
65.5
65.0
4
64.1
64.5
65.7
65.0
5
63.9
64.2
65.2
65.8
Table 5: Pass@1 (%) on HumanEval using CODET and Node-davinci-00r with different numbers of test cases. Sampling Number denotes the num- ber of samples generated by model, and Limit denotes the test cases ex- tracted per sample.
we can con-
4.4 CASE STUDY
In CODET, design the dual execution agreement based the idea that good code solution can pass the most test cases and agree with the most solutions of the same functionalityy. We use "dual" because both the code solutions and the test cases are critical. Figure 5a shows case from the HumanEyal benchmark using code-cushman-N0
The highest scoring consensus set has the correct functionality that returns true if all numbers in the list are below threshold while the consensus set ranked 2 does not understand the boundary condition exactly. The solutions in the second consensus can pass more test cases (i.e., 226) than that in the first consensus set (i.e., 218). However, considering both code solutions and test cases, CODET can successfully rank the consensus sets and find the correct solutions. Such cases are not rare, suggesting that our design of the dual execution agreement is reasonable. For further statistical demonstration, we conduct an ablation study to score the consensus set by considering only the number of code solutions or test cases. The results again support our claim, as detailed in Appendix
CODET is empowered by the pre-trained language models, but is also limited by them. Therefore, the second assumption made in Section 2.2 does not always hold, leading to error cases where the correct code solution is generated, but not in the top 1 consensus set. For CODET with code- cushman-000
the HumanEval benchmark, we find 53 out of 164 programming problems that belong to this situation. We manually investigated these problems and found that 20% of them can be blamed on issues such as ambiguous problem descriptions, uncovered corner cases, and lack
8

---

Published as a conference paper at ICLR 2023
of import statements, while the remaining problems are attributed the failure of the model to understand the problem descriptions. Figure [5b] shows an error case caused by ambiguity. The correct understanding of the description "sus(first index value, last index value)" is to add the first and last values, while the code solutions that sum all values from the first to the last are ranked top 1. More real cases can be found in Appendix | And hope the error analysis can provide inspiration for future studies on improving code generation for more difficult programming problems.
5 RELATED WORK
Code Generation with Large Models Recently, number of large pre-trained language mod- els have been proposed for code generation. Benefiting from billions of trainable parameters and massive publicly available source code, models could achieve surprisingly good performance. For instance, AlphaCode (Li et al., 2022b) claimed to have outperformed half of the human competi- tors in real-world programming competitions, and Codex (Chen al. 202T) is empowering Copilot to provide real-time coding suggestions. Other open-source code generation models include GPT- Neo (Black al. 202T. GPT-J (Wang & Komatsuzaki 2021). CodeParrot (Tunstall et al. 2022
PolyCoder (Xu et al.) 2022,, CODEGEN (Nijkamp et al. 2022). and INCODER (Fried et al.
2022
In our study, we take advantage of the Codex inference API provided by OpenAl as well as the two competitive open-source models CODEGEN and INCODER to perform zero-shot code generation.
Automatic Test Case Generation Automated test case generation for programming problems can reduce the effort of writing test cases manually by developers. Early works including Ran- doop (Pacheco et al.] 2007). EvoSuite (Fraser & Arcuri, (011). MOSA (Panichella et al. 2015), DynaMOSA (Panichella et al., 2017), and MIO (Arcuri 2017). were proposed to automatically generate test cases for statically typed programming languages like Java. The later proposed Pyn- guin (Lukasezyk & Fraser, 2022) could handle dynamically typed language like Python. Never- theless, they are all search- based heuristics methods, which have limitations to the diversity and quantity of generated test cases. To combat these limitations, recently proposed approaches (Tufano let al. 2020 Li et al., 2022b) leveraged pre-trained language models like BART (Lewis et al. 2019) and T5 (Raffel et al., (220)) fine-tuned on labelled data for test case generation. Unlike previous works that require heuristic rules model training, we directly sample test cases from powerful code generation models like Codex in the zero-shot setting with elaborate prompts.
Code Selection from Multiple Samples Despite large models have achieved great performance in code generation, the models need to sample many times to find the correct answer. Recently, several approaches were proposed to tackle this issue. In the domain of solving math word problems, (Coibe Cobbe al. (2021) chose the one with highest rank by trained verifier, and [Shen et al. (2021) proposed to jointly train the generator and ranker through multi-task framework. In the domain of general purpose code generation, [Inla et al. (2022) trained a fault-aware ranker. Morcover, some work has Li been proposed to leverage the execution information (Shi et al.) Li al.) Lahiri al., 2022). Unlike previous works that require model training or pre-existing test cases or user interactions, we let the large models generate test cases for themselves and automatically rank the solutions based on the test-driven dual execution agreement. The idea of ranking based on agreement also appears in the domain of reasoning (Wang et al. 2022
20226 Le et al. 2022]
el al. 2022a).
CONCLUSION AND FUTURE WORK
In this paper, we propose a simple yet effective approach, called CODET, leveraging pre-trained language models to generate both the code solutions and the test cases. CODET executes the code solutions using the test cases and chooses the best solution based on the dual execution agreement. We demonstrate the dual agreement with both the test cases and other solutions is critical to the suc- cess of CODET, perform a thorough analysis on the quality of generated test cases and their impact on CODET, and study cases to provide more insights. Experimental results clearly demonstrate the superiority of CODET, improving the pass@ numbers significantly on various benchmarks. While there remain challenges that CODET only works for executable code generation and it introduces extra computation cost for test case generation. In future work, we will explore the ways to tackle these challenges and improve CODET' to solve more difficult programming problems.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We would like to thank Davis Mueller and Jade Huang for proofreading the paper and providing valuable comments. We also sincerely thank all the anonymous reviewers for their constructive feedback and insightful comments.
REFERENCES
Andrea Arcuri.
Many independent objective (MIO) algorithm for test suite generation. Interna- tional symposium search based seftware engineering, pp. 3-17. Springer, 2017.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 2021.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large scale autore- gressive language modeling with mesh-tensorflow. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint N.Xiv:2207.03377 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Execution Execution-guided guided neural program synthesis. In Interna- lional Conference on Learning Representations, 2.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10. 2021.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with repl. Advances in Neural Information Processing Systems, 32, 2019.
Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Gordon Fraser and Andrea Arcuri. EvoSuite: automatic test suite generation for object-oriented soft- ware. Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference Foundations of software engineering, pp. 416-419, 2011.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204. 05999, 2022.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He. Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXi::2105.09938 2021.
Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. arXiv preprint arXiv:2206.080 03366. 2022.
Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. Interactive code generation via test-driven user-intent formalization, 2022.
10

---

Published as a conference paper at ICLR 2023
Hung Le, Yue Wang. Akhilesh Deepak Gotmare, Silvio Savarese, and Steven CH Hoi. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint arXiv:2207.01780 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-1o-sequeeee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13460 2019.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprini 2rXiv:2206.02336 2022a.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv: 22t.202222..44. 2022b.
Stephan Lukusczyk and Gordon Fraser. Pynguin: Automated unit test generation for python. arXiv preprint ar arXiv:2202.055211 052118 2022.
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprini arXiv:2203. 134744 2022.
Carlos Pacheco, Shuvendu Lahiri, Michael D Ernst, and Thomas Ball. Feedback-direcee random test generation. In 29th International Conference on Software Engineering (ICSE pp. 75-84.
IEEE, 2007.
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. Reformulating branch coverage as a many-objective optimization problem. In 2015 IEEE 8th international conference on software festing, verification and validation (ICST), pp. 1-10. IEEE, 2015.
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transac- tions on Software Engineering, 42(2):122-158, 2017
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21((40):1-67, 2020.
Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guil- laume Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:21 2021.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint
2021.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural lan- guage to code translation with execution. arXiv preprint arXiv:2204. 11454, 2022.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.056// 05617, 2020.
Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural language processing with trans- formers. O'Reilly Media, Inc.", 2022.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language hom.://oooommmttttttaasoooeee-e.
Model.
May 2021.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Fd Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 11 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: Sfate-of-the-an art natural language processing, ArXiv, 2019.
Frank Xu. Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Deep Learning for Code Workshop, 2022.
11

---

Published as a conference paper at ICLR 2023
Methods
Baseline
CODET
10
100
1
2
10
code-cushman-000
31.7 -1.8
56.4 2.1
84.1 6.7
58.6 14.1
65.7 15.6
80.1 14.4
code-davinci-000
34.8 -12
63.0
244
87.2 8.1
60.4
10.2
69.1
10.2
82.4 6.6
code-davinci-002 47.6 0.6
78.8 3.9
92.7 0.6
74.8 9.0
82.9
7.8
89.0 2.4
Table 6: @ (%) on the original HumanEval benchmark with Codex models. The numbers in orange indicate the absolute improvements of pass @ k on the original benchmark over our modified benchmark in Table
A MORE IMPLEMENTATION DETAILS
We set the temperature 0.8, the top p to 0.95, the max generation length 300, and the timeout of executing a test case to 0.1 seconds. Specially, for baseline pass@ 1, we use the greedy search setting with temperature The number of sampling test cases for each problem is 100 for the HumanEval and MBPP benchmarks, and 50 for the APPS and CodeContests benchmarks. When scoring consensus sets in CODET, we use the square root of |S to reduce the impact caused by code solutions. A supporting experiment can be found in Appendix C For code solution post-processing, we follow Chen al. (2021) truncate the generated content by five stop sequences: "A "Anclass", "\ndef"" n nil and "Anprint.. For the implementation of InCODeR and CODEGEN, we use the HuggingFace transformers library (Wolf et al.) 2019) and run both models with half preci- sion. In addition, when the number of consensus sets in CODET is smaller than ks the selection is done from the highest scoring consensus set to the lowest. When reaching the set with the lowest score, it repeats from the highest scoring consensus set. In most cases, the number of consensus sets is larger than k as shown in Figure
B RESULTS ON ORIGINAL HUMANEVAL
As mentioned in Section 3 for all benchmarks, we remove the example input-output cases from the original contexts to avoid exposing real test cases. To study the influence of such modification, we take HumanEval as example and perform an additional experiment with its original contexts. The results are summarized in Table On the one hand, the baseline pass@ 10 and pass@ 100 results on the original HumanEval benchmark outperform the modified version, which is reasonable because the example input-output cases may provide useful information for code generation. Nevertheless, the pass @1 results the original benchmark are basically the same or even worse than the modified version, suggesting that the Codex models have not fully understood the semantics of the example input-output cases provided the contexts. On the other hand, the performance of CODET sig- nificantly improved using the original benchmark. This is as expected because the original contexts used for test case generation include real test cases, which could be borrowed by the models during the generation. Such real test cases will greatly empower CODET to distinguish correct code solu- indispensable remove the example input-output cases to
tions. Hence, in our experiments, avoid exposing the real test cases. In this way, the effectiveness of CODET can be fairly verified.
C ANALYSIS ON CODE SOLUTIONS
In CODET, code solutions that can pass exactly the same test cases are considered consistent in functionality and are grouped into the same consensus set. Since we employ top p sampling with a rather high temperature of 0.8, the functionality of the code solutions may vary significantly, which results in more consensus sets. We draw histogram in Figure 6 to show the number of consensus sets produced by code-cushman-001 and CODET for each problem on the HumanEval benchmark. The average and median numbers are 26.8 and 25.5, respectively. We can find that most problems have less than 50 consensus sets, but the numbers have a high variance among different problems. We also draw the distribution of the numbers of code solutions for the top-ranked consensus sets in Figure 7 The consensus sets ranked top tend to have more code solutions with an average value of 9.8, and the numbers also have high variance.
12

---

Published as a conference paper at ICLR 2023
Problem Probability Density
0.06
15
rank #1
Number of Problems
0.05
0.04
rank #3
rank #
9
0.03
rank #
6
3
0.01
0
0.00
10
20
50
60
70
80
5
10
15
20
Number of Consensus Sets
Number of Code Solutions
Figure 6: The numbers of consensus sets that are produced by code-cushman-001 and CODET the HumanEval benchmark.
The distribution of the code solution numbers for the top consensus sets. The long tail distribution with number 2 20 is truncated.
Figure 7:
linear
w/ sqrt
w/ log
Baseline pass@ 100
CODET pass@1
70%
85%
65, 65.78%
64.01
75%
pass@1
50%
44, 4.59%
500.22%.40.5 49.57%
65%
44.54%
4GUDES
40%
42.800
45%
35%
cushman-001 davinci-001
davinci-002
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Codex models
temperature
The CODET results of three Codex models with and without constraint on num- ber of code solutions.
Figure 8:
Figure The baseline pass 100 and CODET pass @ 1 with code-cushman-00) at different tem- perature settings.
As mentioned in Appendix A we use the square root of |Sa to reduce the impact caused by code solutions, because believe passing more test is more important than having more code solutions with the same functionality. For example, there may be one code solution that can pass five test cases, whereas another five code solutions in a consensus We intuitively consider that the former may be more likely correct. For validation, we perform an experiment by comparing the performance CODET with the "sqrt", "log" functions, and without any constraint (i.e., "linear") on the number of code solutions. Figure 8 shows the results of three Codex models on the HumanFyal benchmark. We can find that reducing the importance code so- lutions can consistently improve the performance of CODET. Similar observations have been found in other models and benchmarks, where the performance of employing "sqrt" is always better than or competitive to "linear", indicating the rationality of our design.
can pass only one test case.
D INFLUENCE OF DE-DUPLICATION
Since we sample multiple times during generation, there is the chance that many of the generated code solutions and test cases are exactly the same. On the one hand, the number of duplicates may indicate the importance of sample. On the other hand, duplicates may hinder the scoring of consensus sets in CODET when the quality of generation is unsatisfactory. Hence, we perform an ablation study to investigate the effects of removing duplicate code solutions and test cases. Specifically, we first format the generated Python code to conform to the PEP style guide and then remove duplicate code solutions and test cases before performing CODET, The de-duplication results on the HumanEyal and MBPP benchmarks using CODET and code-cushman-000 are shown in Table where we can choose de-duplicate the code solutions, or the test cases, both. We can
https://ppp..yyrhoooor/////
13

---

Published as a conference paper at ICLR 2023
De-duplication
HumanEval
MBPP
Solution
Test
2
10
2
10
No
No
44.5
50.1
65.7
55,4
61.7
72.7
No
Yes
42.2
48.8
66.7
54.5
62.3
73.4
Yes
No
46.9
52.5
65.6
54.7
61.7
73.2
Yes
Yes
42.7
51.2
66.4
54.7
62.1
73.2
Table 7: Pass l (%) on the FvaanEval and MBPP benchmarks using CODET and code-cushman- 001 with different de-duplication settings. The setting "No No in the first line means that neither the code solutions nor the test cases are de-duplicated, which is used in our main experiments.
Methods
CODET
CODET (Remove Trivial)
k
10
100
10
100
INTRODUCTORY
34.6
53.2
34.9
53.4
APPS
INTERVIEW
8.1
18.1
8.3 0.2
18.2 0.1
COMPETITION
2.2
8.6
2.5 0.3
8.7
CodeContests
2.1
5.3
9.9
2.7 0.6
5.3 0.0
10.0 0.1
Table 8: @ (%) results on the zero-shot APPS and CodeContests benchmarks using code- davinci-002 and CODET with/without the trivial code solutions filtered. The numbers indicate the absolute improvements after filtering the trivial solutions.
find that de-duplication has slight and inconsistent influence on the performance of CODET. For the HumanEval benchmark, the pass @ results using code solution de-duplication alone are better than other settings. Nonetheless, for the MBPP benchmark, the best pass @ results achieved without de-duplication. Therefore, in our main experiments, we reserve all the generated code solutions and test cases when performing CODET and leave the study of more advanced de-duplication methods for future work.
E SENSITIVITY TO THE TEMPERATURE
The hyper-parameter temperature has great impact on the quality of generated code solutions and test cases when using top sampling. We use a high temperature of 0.8 in our main experiments since CODET could benefit from larger number of diverse samples. To investigate the sensitivity of CODET to the temperature, we perform an ablation study by using range of temperatures to report the results of baseline pass 100 and CODET pass@1. Figure shows the results of code- cushman-O0 the HumanEval benchmark at different temperature settings. We can find that higher temperature does improve the baseline pass @ 100 and CODET pass@1, and CODET achieves good performance when temperature is set 0.8.
F REMOVING TRIVIAL CODE SOLUTIONS
The problems in the APPS COMPETITION and CodeContests benchmarks are of great difficulty compared HumanEval and MBPP, leading to the poor performance the most capable code- davinci-002 model. After checking the incorrect code solutions generated by code-davinci-002. we identify many trivial solutions that just return the input argument or a constant value. Such solutions may hinder the ranking process of CODET if they can pass any generated test case. A trivial solution can be easily identified by its input arguments and returned values. a solution always returns the same output value for different inputs, or its returned values are always the same the inputs, it must be a trivial solution. To investigate the impact of trivial code solutions, we use davincavinci- 002 on the choo APPS and CodeContests benchmarks, and perform CODET after filtering out all the trivial solutions. As a result, we can remove an average of 4.5 (91.6) trivial solutions from the 50 (1, 000) generated solutions per problem for the APPS (CodeContests) benchmark. How-
14

---

Published as a conference paper at ICLR 2023
k:
1
10
50
100
1000
1
2
10
100
Baseline
CODET
INTRODUCTORY
29.3
48.5
60.9
47.3 18.0
52.7
58.4
APPS
INTERVIEW
6.4
14.6
25.4
14.3
18.2
23.3
COMPETITION
2.5
6.3
14.5
6.2 3.7
13.6 7.3
CodeContests
1.0
4.1
7.1
8.8
15.2
3.2 2.2
5.6
9.3 5.2
12.3 3.5
Baseline Filter
CODET Filter
INTRODUCTORY
43.6
58.6
49.6
54.3
59.4 0.8
APPS
INTERVIEW
13.3
22.8
16.
19.5
24.0 1.2
COMPETITION
7.0
13.3
7.9 0.9
10.5
14.1 0.8
CodeContests
9.9
14.5
15.1
15.2
9.6 -0.3
11.5
13.7 -0.8
14.5 007
Table 9: @ (%) results on the APPS and CodeContests benchmarks using code-davinci-000 and the one-shot setting. The numbers red indicate the absolute improvements of CODET (Filter) over Baseline (Filter) pass@1, 1, pass @ 10 and pass 100. For CODET (Filter), temperature is set to 0.8 and sampling number set to 50 for APPS and 1, 000 for CodeContests. We do not report pass @ 1000 for "Bascline Filter" because the numbers of code solutions after filtering are less than the sampling numbers.
ever, as shown in Table B after removing a prominent percentage of trivial solutions, there little performance gain, which could exactly demonstrate the robustness of CODET.
G RESULTS ON APPS AND CODECONTESTS IN THE ONE-SHOS SETTING
Inspired by Chen et al. 2021 and ei al. 2022b), we build one-shot versions of APPS and Code- Contests by appending single input-output example the problem description a formatting hint. After generation, we filter out the generated solutions that cannot pass the given example input-output cases, which we call the "Baseline Filter" method. After filtering, we can still perform CODET using the rest of code solutions, called the "CoDET Filter" method. Following the zero- shot experiments on APPS and CodeContests, we employ 2ode-davinci-002 for generation and set the sampling number to 50 for APPS and 1, 000 for CodeContests.
We summarize the experimental results in Table 9 where we can find the one-shot performance using CODET is much better than that reported in JableBin the zero-shot setting. The performance of the baselines can be significantly improved by filtering the solutions with the given example test cases. Moreover, "CODET Filter" can further outperform "Baseline Filter" on the APPS benchmark, especially for the introductory and interview problems. Nonetheless, for CodeContests and the competition level problems in APPS, "CODET Filter" has little performance improvement performs slightly worse than "Baseline Filter". After manual investigation, we blame such issue to the generated low-quality test cases, which hinder the scoring of consensus sets. This suggests the interest of future study on test case generation for more challenging programming problems.
H MORE ANALYSIS ON TEST CASES
H.1 STATISTICS ON TEST CASES
How many valid test cases do the models generate for CODET? Taking the HumanEval benchmark as an example, we sample 100 times for each problem when generating test cases. As illustrated in Figure at each time of sampling, we feed the context e along with an instruction p to the model and get the generated content that may contain multiple test cases. Then, as mentioned in Section 4.3, we further post-process the generated samples to get individual test cases that are syntactically correct. Finally, we only keep the first five valid test cases for each sample, which means a problem can be equipped with 500 test cases at most. Table 10 summarizes the average and median numbers of the extracted test cases for each problem. We can find that almost all the models could generate a considerable number of syntactically correct test cases, while CODEGEN generates plenty of unexpected noise.
15

---

Published as a conference paper at ICLR 2023
Test Case Number
Code Coverage
Methods
Methods
Average
Median
Statement
Branch
code-cushman-000
410.7
429.0
code-cushman-000
95.3
98.1
code-davinci-O01
381.9
388.0
Node-davinci-001
94.9
97.6
code-davinci-000
391.1
402.0
code-davinci-00s
95.7
98.5
INCODER
390.1
400.0
INCODER
94.0
96.3
CODEGEN 55.6
42.0
CODEGEN
78.2
78.6
Table 10: The numbers of extracted test for each problem generated by five models on the HumanEval benchmark.
Table 11: The Code Coverage (%) statistics of test cases generated by five models on the Hu- manEval benchmark.
Sampling Number
Sampling Number
Sampling Number
Limit
Limit
Limit
10
20
50
100
10
20
50
100
10
20
50
100
code-cushman
code-cushman-000
code-cushman--00
37.8
40.0
40.8
38.7
1
43.3
48.1
48.2
49.1
55.1
56.6
61.9
62.9
42.1
41.8
43.4
41.8
2
48.1
48.1
49.5
49.8
58.7
61.4
64.5
65.8
41.6
42.5
47.7
60.9
62.5
63.4
65.3
41.2
43.8
43.3
47.9
49.1
61.4
63.3
65.8
41.0
41.9
45.4
44.5
5
48.3
48.5
48.9
50.1
63.1
62.6
63.8
65.7
code-davinci-002
code-davinci-002
code-davinci-002
56.5
57.5
60.7
62.4
65.1
67.8
71.9
71.5
77.9
79.6
82.8
84.3
2
62.2
62.8
63.2
63.6
2
71.7
73.2
74.2
74.1
80.8
81.8
84.3
86.5
3
62.9
63.2
65.5
65.0
3
73.2
73.5
75.1
75.0
82.3
83.2
85.5
87.1
64.1
64.5
65.7
65.0
73.3
74.1
75.5
74.3
82.9
84.4
85.4
86.9
63.9
65.2
65.8
73.5
74.3
74.5
75.1
83.8
84.1
85.2
86.6
(a) pass
(b) pass@2
(c) pass @ 10
Table 12: Pass@k (%) on the HumanEval benchmark using CODET with different test case numbers. Sampling Number is the number of test samples generate for each problem. Each sample may contain multiple assertion statements. These assertion statements are potential test cases, but do not use all of them. Instead, extract Limit number of syntactically correct assertion statements from each sample, and discard the rest.
H.2 CODE COVERAGE OF TEST CASES
To further inspect the quality of generated test cases, we utilize the code coverage measurement and report two coverage criteria the statement coverage and the branch coverage. The statement coverage can be calculated as the percentage of statements in a code solution that are executed by test cases. The branch coverage the percentage of executed branches for the control structure (e.g. the if statement). We execute the canonical solution for each HumanEval problem on the test cases generated by five models, then collect the coverage results using Coverage.py As result, the average numbers of statements and branches in the canonical solution of problem are 6.30 and 4.42, respectively. As shown Table IT] all the models except CODEGEN have good performance on both statement and branch coverage, reaching may be attributed to the relatively short canonical solutions and the massive sampling number of test cases. Nevertheless, there are still coreer cases that the models cannot cover, which calls for future improvements.
average of over 94% coverage. Such results
H.3 RESULTS OF REDUCING THE NUMBER OF TEST CASES
To investigate the performance of CODET using fewer test cases, we perform ablation study the number of test cases that participate in the dual execution agreement. As shown in Table T2 report the results on the HumanEval benchmark using sode-cushman-O0 and code-davinci-ore with a range test case numbers. The number of test cases related to two hyper-parameters One the number of test case samples, which is set 100 for HumanEyal in our main experiments. The
httts::/ htw.a//era...erraaiieeeoooo..o.e
16

---

Published as a conference paper at ICLR 2023
Methods
Code Solution Only f'
Test Case Only f"
k
1
10
1
2
10
code-cushman-00)
41.2733
49.2-0.9
61.9776
29.9
36.6-13.3
59.5
1ode-davinci-001
44.4734
54.7-12
69.078.
35.0
15.2
46.0-129
70.2
4.
code-davinci-002 55.9
9.9
67.0-81
82.7 47.8
3.9
58.4 11.4
7.4
65.1-100
86. 1 +11.2
0.5
Table 13: Pass@ h (%) on the HumanEval benchmark with ranking only on the number of code solutions (S"(S) = Sx))) |) or test cases (S) = Syy) in a consensus set. The numbers in red and green indicate the absolute improvements over baseline and CODET, respectively.
other one is Limit that controls the amount of syntactically correct test cases we extract from each sample, which is set to 5 for all benchmarks in our main experiments. Note that Limit multiplied by the Sampling Number is the maximum number of test cases for a problem, not the exact number, because not every sample contains the Limit number of valid test cases. A valid test case (i.e., assertion statement) should start with "assert" and contain the name of the corresponding entry point function. We can conclude from the results that using more test cases in CODET could generally lead to better performance. While the performance gap narrows when Limit 2 and the sampling number 2 50. Moreover, using only 10 test cases per problem for CODET can still improve the baseline pass@1 performance of code-cushman-00) by absolute 4.3% and code-davinci-002 by absolute 9.5%. It demonstrates that CODET has high test case efficiency and we can smaller Sampling Number in real-world application to balance the performance and computation cost.
I ABLATION STUDY ON THE SCORE OF CONSENSUS SET
In CODET, the score of a consensus set is calculated as S(S) |S,1|Sy]. where S, and S,, are the code solutions and test cases in the consensus set, respectively. We can naturally derive two variants of scoring. One is f'( = | in line with the idea of self-consistency (Wang et al.) 2022), which only considers the number of code solutions with the same functionality. The other one is fS) = Syyl which corresponds to simply counting the test cases that each code solution can pass. To evaluate the performance of these two variants, we perform an ablation study on the HumanEval benchmark using three Codex models. The experimental results are summarized in Table 13 from which we can observe that only considering the number of code solutions test cases for consensus set scoring performs consistently worse than CODET, and even worse than the baseline. Therefore, it is essential to consider the importance of both code solutions and test cases, suggesting the reasonable design of our dual exccution agreement.
As mentioned in Section 3 AlphaCode (Li et al. 2022b) also includes clustering method (de- noted as AlphaCode-C) to select the generated code solutions, which shares a similar goal with our ablation method f": clustering code solutions based code functionality, and then scoring each cluster by size. AlphaCode-C requires a number of additional test inputs to produce outputs from code solutions, which are then used determine the functional equivalence. AlphaCode-C relies on separate test input generation model, which needs extra training and annotation. The model unavailable and hard to replicate, as the paper does not provide sufficient details. We replicate AlphaCode-C by extracting test inputs from the test cases generated by CODET. We run all code solutions on the test inputs, and group them by outputs. The clusters are ranked by size and then we select the code solutions from each cluster in order. From Table 2 and Table [3] we can find that AlphaCode-C is inferior to f', though they share the similar idea. The reason is that AlphaCode-C will group the trivial code solutions (e.g., solutions that always output "None", "0". or an empty string with whatever inputs) together, leading to a large cluster of incorrect solutions that signif- icantly affects performance. While such trivial code solutions are hard to pass the generated test cases in CODET, thus having lower consensus scores for ranking. This confirms the effectiveness of considering test case information.
17

---

Published as a conference paper at ICLR 2023
def voeeeeeteelstt))
renove_ vowels function that takes string and returns string without vowels.
Rank #1: The consensus set has 31 solutions and 170
Rank #2: The consensus set has 41 solutions and 147 test cases. An example solution:
test cases, An example solution:
Correct
Incorrect
vowels "aeiouAEIOU'
vowels = set("aeiou") characters set(text) if not (characters vowels): return
text www.hhtttttt..m vowels for character text:
1f character not in vowels:
wwwpwwtttttteeee vowels
return text text_without_voeee vowels
return "" "".join(c for in text 1f с not in vowels)
(a) The first consensus set has fewer code solutions.
def below_zero (erepraterttttttiionns: List[int]) ->
Yru're given a list of deposit and withdrawal operations on bank account that starts with balance. Your task zero, and at that point function should return True. Otherwise it should return False.
detect any point the balance of
falls below
Rank #1: The consensus set 79 solutions and 248 test cases. An example solution:
Rank #2: The consensus set has solutions and 255 test cases. An example solution:
Correct
Incorrect
balance for operation in operations:
any for operations)
zero zero_balane roo_balance + operation
zero_ balacca
return True
return False
(b) The first consensus set has fewer test cases.
Figure 10: Two cases from the HumanEval benchmark, where CODET can find the correct consensus sets though they have (a) fewer code solutions, or (b) fewer test cases.
J MORE EXAMPLES FOR CASE STUDY
Figure 10 iillustrates two cases that CODET can successfully find the correct consensus sets. Specif- ically, the case Figure [0a remove the vowels in the input text. There are incorrect solutions and 147 test cases in the consensus set ranked 2, which forget to remove the upper-case vowels. Though the correct solutions the top consensus set are fewer (i.e., 31), they can pass more test cases (i.e., 170) and thus have higher score. The case in Figure Tobjis Figure(Tobjis to decide when the balance of account will fall below zero. The functionality of the incorrect solutions in the second consensus set is to tell whether there are withdrawing operations. Nevertheless, the incorrect solu- tions can pass more test cases (i.e., 255) than the correct solutions (i.e., 248) in the top consensus set. Fortunately, there are 79 correct solutions and only 6 incorrect solutions, making it possible for CODET to rank the correct consensus ahead. Both cases demonstrate the plausibility of using the dual execution agreement instead of solely considering the functional agreement between code solutions or the number of passed test cases.
Figure I1 illustrates the cases that CODET fails find the correct consensus sets. Specifically, Figure II demonstrates the situation that there are partially correct solutions that may fail at certain corner cases. In the example, there are 20 incorrect solutions in the top consensus that can pass 205 test cases, which will fail if the input is a string of length 1. The correct consensus set ranked has more test cases (i.e., 222), while it has a lower consensus score due to the small number of code solutions (i.e., 9). The second example in Figure 1 shows the most common situation where CODET fails because the model cannot fully understand the problem. We can find that incorrect solutions in the top 1 consensus set totally missing the points of the given problem. While the model still tends to generate more incorrect solutions and test cases based on its wrong understanding. All the bad cases call for future improvements the quality of generated code solutions and test cases.
18

---

Published as a conference paper at ICLR 2023
def prine length(string):
function
string returns the string length prine
False otherwise
Rank #1: The consensus set has 20 solutions and 205 Incorrect
Rank #3: The consensus set has 9 solutions and 222 test cases, An example solution:
test cases. An example solution:
Correct
nun__hhan len(string)
length len(string)
if nun_chars 1:
length
return True for in range(2, nun_chars);
return False
range(2, length):
if num chars & i == 8: return False
num_chars
if length % 8:
return False
return True
(a) Uncovered corner cases.
def minsubArraySuunnunss
array of integers nums, find minimum any non-empty aub-array nums.
consensus solutions
42: The consensus solutions and
test cases. An example solution:
Incorrect
cases. example solution:
Correct
not nums:
if not nuns:
return
total nums[0]
nin_sum float lennn for i in range(n):
min__uii sun total
for in range(1, (nuns)))
1f total < 8: total nuns[i]
curn_sum
for i in range n):
else:
curr_sun += nums[]]
total += nums[i]
sum min curr_sun) return nin_sum
nin_sum miinmin_sum, total)
return nin_sum
(b) Failure of Problem Understanding.
Two incorrect cases from the HumanEval benchmark, where CODET cannot find the correct consensus sets due (a) uncovered corner cases, or (b) failure of problem understanding.
Figure 11:
19


Reviews:
[{"review_id": "zV7SIf8OCj", "reviewer": "Reviewer_ziee", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of generating a code solution to a programming problem. The key observation is that current models like Codex can generate solutions, but selecting the appropriate solution is challenging. This is reflected in the large gap between pass@100 and pass@1. If test cases were available, one could simply run the top 100 generated solutions on all the test cases to get the best one. But in several real world settings, test cases are not available.\n\nThis paper proposes to generate test cases and solutions simultaneously with the same model (no further training necessary). They engineer a prompt that makes the model generate test cases, and use a random sampling strategy to run selected solutions on selected test cases. From the results of these runs, they pick the best solution according to certain criteria.\n\nThey show that their approach achieves approximately a 10% improvement on the HumanEval and MBPP benchmarks, with minor improvements on the Apps and CodeContests benchmarks.\n", "summary_of_the_review": "**Edit** - I have updated my evaluation and score after reading the review by Reviewer ZjXf.\n\nThe idea that this paper presents is simple, elegant and innovative. However it is missing a comparison with the approach taken by AlphaCode, i.e., training a separate model to generate test inputs. Further, the evaluation is shaky and in my opinion, needs to be reformulated around the n@k metric and as a comparison of different filtering approaches. On the whole, I think this is an innovative idea, but the paper in its current form is not ready for publication. I would be happy to reconsider my score if the authors are able to address these concerns.", "strength_and_weaknesses": "**Edit :** After reading the review by reviewer ZjXf, I have several new concerns about the paper and evaluation.\n\n**Comparison with AlphaCode:**\n\nThe AlphaCode paper uses a separate trained model to generate test inputs, and uses the outputs on these test inputs to cluster its generated solutions (which can be up to 1e4). In comparison, CodeT re-uses the same model to generate test inputs *and* desired outputs. When presented in this light, this diminishes the novelty of CodeT somewhat. A detailed discussion should be included in the paper.\n\n**Computation of pass@k:**\n\nThe \"biased\" approach to compute pass@k is to just generate $k$ solutions and see if any one of them passes all test cases. Using the \"unbiased\" approach from Chen et al. (2021), one would generate $n$ solutions, and compute the number $c$ that pass all the hidden test cases. Then the pass@k is computed using a combinatorial formula.\n\nAs I understand it, the baselines have been evaluated with the \"unbiased\" formula, but CodeT has been evaluated with the \"biased\" formula (mentioned under \"Implementation Details\" in Section 3). This seems a little strange and not kosher. I think this evaluation needs a rethink.\n\n**pass@k vs n@k:**\n\nThe metric n@k as presented in the AlphaCode paper is as follows - generate $k$ solutions and use some filtering mechanism to narrow it down to $n$ solutions where $n \\le k$. Then if any one of the $n$ solutions passes all the hidden test cases, report success on that problem.\n\n*This seems like the setting that you should be evaluating CodeT on.* In particular, pass@k does not seem well suited to your problem setting, and results in an odd mismatch in evaluation procedures between CodeT and your baselines.\n\nI would reformulate the entire paper as a comparison of *different filtering approaches*, with the evaluation based on the n@k metric. The filtering approaches would be:\n1. Primitive filtering (baseline). Just take the top $k$ solutions out of $n$ generated solutions (\"biased\" pass@k).\n1. Random filtering - take a random sample of $k$ solutions out of $n$ generated solutions. This (in expectation) is the \"unbiased\" pass@k metric defined in Chen et al. (2021).\n1. The AlphaCode approach - train another model to generate test inputs, and cluster the $n$ solutions based on their outputs on these generated inputs.\n1. CodeT - reuse the same model to generate test inputs as well as desired outputs.\n\nThere needs to be a detailed comparison between the latter two approaches. I understand that it is difficult to recreate the AlphaCode setting with their trained input generation model, but at the very least there should be a *qualitative* comparison.\n\nMy old review comments (pre-edit) follow.\n\n---------------------------------------------------------------------------------------------------------------\n\n**Strengths :**\n\n1. Simple, elegant method that requires no retraining\n1. Convincing performance improvements with thorough experiments\n\n**Weaknesses**\n\n1. This approach could potentially increase the code generation time several-fold, especially if you have ~1e3 tests for 1e2 solutions => ~1e5 executions. The HumanEval benchmark has just 7.7 \"ground-truth\" tests on average per problem, so if one were to execute all top 100 solutions on the given tests then there would be ~770 executions. I did notice and appreciate the note at the bottom of the Q2 section and the experiments in Appendix G.3, but this should really be its own section. Execution time is, in my opinion, a significant factor in determining the real-world utility of this model. I would ideally like to see :\n\n    * How many \"ground-truth\" test cases does each benchmark provide, on average, per problem?\n\n    * How many executions (code running on test) does CodeT require, on average, per benchmark?\n\n    * The results in Appendix G.3 moved to the main text.\n\n2. (Minor) The choice of using the *product* of number of solutions and number of test cases seems slightly counter-intuitive to me : why would we (intuitively) care about how many other solutions matched the same test cases? It's all the more intriguing because you say that \"such cases are not rare\", and your ablation study in Appendix H shows that this is an important design choice. Some intuition here would be helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OfwWWznLh1", "reviewer": "Reviewer_tYm4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to improve pretrained language models' performance on executiion-based benchmarks by using the same model to generate test cases and perform dual execution agreement. The proposed method is simple and effective without any additional training. Experiment results with multiple pretrained models on multiple execution benchmarks show impressive improvement.", "summary_of_the_review": "Overall, I think this is a good paper with novel method proposed and well-performed experiments.", "strength_and_weaknesses": "Strength:\n- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.\n- The proposed method does not rely on any additional input (test cases in particular) besides the problem description/prompts, so it is a self-contained method without much assumption.\n\nWeakness:\n- The apparent weakness is obvious and also mentioned in the paper. CodeT method only works for programming problem datasets with clear test cases. It limits the applicability of CodeT, for example, it cannot be used in general code generation scenarios in code companion tools like Copilot.\n- Considerring the goal for the proposed method is to prompt or post-processing of 100 generated sample, it makes sense to compare the proposed method with language model generation post-reranking  method, which also intend to select/rank from a number of generations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2f8kXn91w4J", "reviewer": "Reviewer_GQjW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Large language models trained on source code can generate many plausible solutions to programming problems. This work proposes to simultaneously use these models to generate tests for the same prompts, which are then used to re-rank the generated solutions, using a consensus set based algorithm. The results show that this technique substantially improves the correctness of the highest ranked sample(s) across a range of language models, and additionally provide an analysis of the impact of test quality, which underscores that especially powerful models benefit most from this modification because they are also most likely to generate meaningful tests.", "summary_of_the_review": "This paper is well-written, thorough, and yields strong results. Its proposed technique is not particularly complex, but is both effective and very well investigated. Its main limitation is a relatively narrow focus on a specific programming task approach purely from a large language model perspective, which leaves it both without strong connections to the role of self-evaluation in the broader ML community, and to make the occasional questionable claim relative to prior work in program synthesis.", "strength_and_weaknesses": "This work introduces a relatively straightforward, yet highly effective technique, and explores its design decisions in great depth. I was left with nearly no questions after reading through the results in the appendix, and find the approach convincing. This makes it a strong contribution to the domain of code-generating models.\n\nIn terms of weaknesses, a lesser complaint is that this work makes very little effort to connect its findings to the theme of representation learning. Implicit in the experiments and results is the observation that LLMs must have some instrinsic notion of code execution, at least as syntactically represented by test cases, but this is neither explored per se, nor does the work draw connections with any related fields and applications. As such, it seems like it would be a better fit at a software engineering venue. Nevertheless, conferences such as this tend to draw papers from a very wide range of application domains, so this does not change my overall verdict.\n\nAnother relatively minor issue is the decision to remove input/output examples from the (HumanEval) prompts, or rather, the discussion around that experimental decision. While (the end of) Sec. 2.1 reasonably states that this helps avoid biasing the models, App. A makes a much stronger claim that removing these cases is required to \"fairly\" verify CodeT. This strikes me as a rather odd statement: program synthesis research has historically mostly operated under the assumption that input/output examples would be provided by a user, just like the HumanEval benchmark does. The CodeT approach also clearly both benefits from these tests _and_ generalizes beyond them, judging from the substantial performance gains over its own results without \"seed\" tests as reported in Tab. 4. While it is entirely reasonable to compare both conditions on the premise that not requiring examples involves less effort by the user, suggesting it is in some way more fair seems off. I would suggest reporting both settings side by side in the main text instead. Finally, the text in 2.1 also claims that removing the original tests increases the \"diversity and difficulty\" of the generated tests, which does not appear to be substantiated by experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-aDQk0iHBM", "reviewer": "Reviewer_ZjXf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is part of the recent stream of papers which use large language models to generate code to solve coding contest type problems. Two popular ways of measuring the performance of such models is to (1) evaluate $pass@k$ -- $k$ solutions are generated for a problem and evaluated against the unit tests for that problem. If even 1 solution clears all the test cases for that problem, then the problem is considered solved. The metric is the fraction of problems so solved in the dataset (2) $n@k$ -- $k$ solutions are generated but *only* $n$ solutions are evaluated against the unit tests for the problem. If even 1 out of the $n$ (instead of $k$) solutions passes all the test cases, the problem is considered solved and the metric is the fraction of problems so solved. This metric was widely used in the AlphaCode paper [1]\n\nThe idea behind this paper is as follows. Given a prompt, they first generate $K$ solutions to the problem that the prompt describes. In their experiments $K=100$. They then use the language model to generate new test cases $T$ for the problem. While the validity of any of the test cases in $T$ is unknown, the agreement of the different generated solutions in terms of passing or not passing test cases in $T$ can be used to *cluster* the solutions.\n\nThey then use this clustering to select $n=1,2,10$ solutions from the $K=100$ candidate solutions they generate. They compare this against baseline models which are allowed to generate only $n=1,2,10$ solutions. They claim improvements for $n=1,2,10$ but not for $n=100$ when the computation budget is the same for the baseline as well as their model.\n\n[1] Competition-Level Code Generation with AlphaCode (https://arxiv.org/abs/2203.07814)", "summary_of_the_review": "The reasons for my rating are\n\n(1) $pass@k$ is an unfair metric for what they're trying to evaluate -- as they're selecting a subset of generated solutions which gives their model effectively a far larger computation budget than their baselines.\n(2) No comparison against AlphaCode style clustering techniques for solution subset selection.\n(3) Motivation for why their clustering may work better than AlphaCode's clustering is not given.\n\n**Edit**: After reading the author's response, I'm raising my score to a 5. My response (copy-pasted from below)\n\n\"I thank the authors for their detailed response as well as the additional experiments.\n\nI still disagree that raw pass@k is an appropriate metric as you're comparing a method with filtering to a method without. That would suffice if there were no extant filtering or ranking methods but that is not the case.\n\nI agree that ranked pass@k is appropriate. I also find the comparison with AlphaCode critical and that it shows an advantage for CodeT is encouraging and a point in CodeT's favor.\n\nOverall however, I agree with Reviewer ziee that this should have been presented as a filtering paper -- which would change the narrative a great deal as well as would require more thorough experiments -- for example there are several reranking methods that could be used to filter out top-k solutions and could serve as good baselines. \n\nI encourage the authors to modify the paper and add such baselines.\n\nI'll raise my score to a 5 but I'm not comfortable voting for an accept given that the paper should be framed as a filtering paper and should involve a much more thorough investigation against various alternative baselines for filtering/ranking \"", "strength_and_weaknesses": "While I think the paper is the start of an interesting direction, I think in its current form, it is lacking important comparisons as well as motivation.\n\nIn the paper, they claim to compare on $pass@k$ (Table 1) where they set $k=1,2,10$. However, $pass@k$ is relevant when *all* the models are allowed to generate only upto $k$ solutions. In this instance however, CodeT is allowed to generate upto 100 solutions (from which 1,2,10 are selected) but other models are only allowed to generate 1,2,10 solutions. Thus the CodeT model is allowed a far larger computation budget and has an unfair advantage under the $pass@k$ metric. \n\nFor such a scenario, it is the $n@k$ metric from the AlphaCode paper that makes the most sense and provides for a fair comparison -- where they also generate $k$ solutions but then select $n$ solutions out of those to evaluate against the ground truth unit tests. In the AlphaCode paper, they select the $n$ representative solutions by executing the $k$ solutions on a model trained to generate inputs and clustering based on the output.\n\nIn this paper, they also do a clustering -- however instead of clustering by executing on a model trained to generate inputs and clustering based on the output, the clustering is based on executing on the code generation language model based generated inputs and the binary evaluation of whether the test case passes or not.\n\nGiven the above, the main two issues that need to be worked on are as follows\n\n(1) How does this paper's approach compare with AlphaCode's approach for clustering and then evaluating the selected solutions?\n(2) Why might clustering based on the binary feedback of success or failure from generated test cases from a language model perform better than a more fine grained clustering based on the generated output on an input?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster