Execution-Based Evaluation for Open-Domain Code Generation
Authors: Zhiruo Wang
Shuyan Zhou
Daniel Fried
Graham Neubig
Venue: EMNLP 2023 Findings (2023)
Keywords: code generation
open domain
execution-based evaluation
TL;DR: Data benchmark for evaluating open-domain code generation via execution.

Abstract: To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling – CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.

Execution-Based Evaluation for Open-Domain Code Generation
Zhiruo Wang Shuyan Zhou Daniel Fried Graham Neubig
Language Technologies Institute, Carnegie Mellon University
(zhiruow, shuyanzh, dfried, gneubigJecs cmu. edu
Abstract
However, most resources with execution sup- port only apply to closed-domain code, that only use Python built-in functions (Chen et al., 2021; Hendrycks et al., 2021; Austin et al., 2021; Li et al., 2022; Haluptzok et al., 2023) or specific libraries in data science domains (Lai et al., 2022; Huang et al., 2022). This focus on closed-domain problems di- verges substantially from natural open-domain pro- gram usage covering diverse range of libraries and functionalities (Yin et al., 2018; Agashe et al., 2019; Wang et al., 2023). To enable execution- based evaluation for coding queries using libraries, we present ODEX, an Open-Domain EXecution- based dataset ($2). We build ODEX by creating 1,707 test cases for 945 NL-Code pairs from the CoNaLa (Yin et al., 2018) and MCoNaLa (Wang et al., 2023) datasets, both stemming from Stack- Overflow2 with broad practical coding queries.
To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural lan- guage (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human- written test cases for execution. Our NL.-Code pairs are harvested from StackOverflow fo- rums encourage natural and practical cod- ing queries. Moreover, ODEX supports four natural languages as intents, in English, Span- ish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top- performing code language models (LM). While CODEX achieves hetter overall results, CODE- GEN improves effectively via scaling - CODE- GEN 6. 1B performs comparably with CODEX 12B. Both models show substantial gaps be- tween open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.'
We analyze and highlight three aspects of ODEX ($3). First, ODEX has broad domain coverage of 79 libraries, with 53.4% of the problems employ- ing at least one library. Second, ODEX contains queries four different languages, with 439, 90, 164, and 252 samples in English, Spanish, Japanese, and Russian, as shown in Figure 1. Third, ODEX addresses three unique challenges in open-domain code execution: irreproducible runs (Figure randomized outputs (Figure | ), and specialized equivalence checks (Figure 2).
1 Introduction
Evaluations of NL-to-code generation systems, especially for general-purposs programming lan- guages such as Python, have put an increasing emphasis on methods that execute code to verify the results. The predominant approach for creat- ing such test sets is to manually write test cases for canonical code solutions (Chen et al., 2021; Austin et al., 2021; Lai al., 2022; Huang al., 2022). The correctness of model predictions is then evaluated by seeing if generated code passes the test cases (Chen et al., 2021). Compared to execution-free metrics such as text match against reference solutions, execution- based methods more rigorously assess the functional correctness of code (Hendrycks et al., 2021; Chen et al., 2021).
We evaluate two state-of-the-art code LLM fam- ilies, CODEX and CODEGEN, on ODEX ($5). Our study shows that larger model sizes and augmented training data improve execution accuracy. Mean- while, we observe satisfactory multilingual capa- bilities, despite that neither model was specifically designed for multilingual usage. However, we find that models face greater yet varied challenges with open-domain queries compared to -omain queries ($5). Specifically, CODEX achieves higher
hhups:/stacckovrffow.com

---

retwn Feststsspottir (ilesssllless datamrota)
Anport requeets
requests, Responset)
of tesctuant urk, catee:
copprti U0U Oi Valee's ricresrr
requeste past Neck(mmturnssaaa
(253r saltiple dato Yales' "acaa art wath Nuttisort request?
with spc.ir 1M13 fr
recara
fites (FFtitas qpeetfiite 1r8113
1-aritel 45c11
aspert functsonifiless "ntcpos//def oge's (heyy' "volae Woltteeissttttss.ooo - 200
frne Aitetaic Unpert Sotetinn, Lincesnc
tiee, syts
b
return datatioe meslpptr.ar
Gad ssiae Sonetime. saday!) in the 3TC time 2ame,
Code LM
cate
1. www.acoetataaaeeieeeeoo
Maserd (lunctaan) totel Laccadaa) Lime, Lare() oe
Inport
Cime garrder aes coerts 'f es as anchivo rile.coo?
Hith apant Fileucse's Toeliree concodinge wtti-art csntatee wratzrarriieraa
C
shee
"fleevor"
reture
test case
fenctimn [t,, 31, 5, 8]]1
Fele.cee's [.readtires)) Lones strip() 13,2.21
Import syeas
funotiants, secafrransssd senerpas sxpavoud Apoossed sards decrponwattte
A50ert Lsnes (trrp() "4,5.0'
fetegral grves foerioa "F"
froe the sunter to jefiesty)
reture 53R, Integ
returs
cest
striftnccionf,, nll 10s"
Figure 1: Examples in the ODEX dataset. Inputs the left function-formater with (1) library import expressions; (2) function signatures that declares the function name and input arguments; and (3) natural language intents as part of the docstrings (English translations are not included in the actual non- mon-English inputs during inference). Gray boxes indicate places for code solutions. As shown the right, code LM fills out the gray boxes with code solutions, which are then executed on the unit tests underneath. Notably, writing unit tests for open-domain queries is often more challenging: a requires simulated execution due to the difficulty of reproduction;; verified through approximate equivalence. Prior work focuses more on basic assertions, as in and d
b is
overall results, while CODEGEN presents better pa- rameter efficiency and more balanced open-closed domain performance model size scales up. By comparing iiccution-bbseed metric with series of execution-free metrics ($6), we further confirm the advantage of execution on allowing alternative solutions, but also show the potential of lexical metrics to identify simple bug fixes.
2.1 Resource Collection
We take two NL-to-code datasets, CoNaLa (Yin et al., 2018) and MCoNaLa (Wang et al., 2023), sources for ODEX. We refer to them together as (M)CoNaLa. Their NL-Code pairs are collected from StackOverflow, which contains abundant cod- ing queries that (1) naturally reflect practical pro- gram usage, and (2) cover diverse domains as mea- sured by libraries used. These properties align well with our main focus on open-domain queries. (M)CoNaLa further proofs and clarifies its NL in- tents using human annotators to ensure data quality.
ODEX jointly facilitates practical mpen-domain code generation and execution- evaluation. It serves as a comprchensive data benchmark for NL-to-code systems, supporting diverse NL con- texts, library usage, and evaluation methods. By addressing the unique challenges of test creation and execution, hope to lay foundation for evaluating open-domain code via execution.
2.2 Annotation Standard and Procedures
Given each source NL-Code pair, our main anno- tation task is to write test cases to check code exe- cution correctness, as illustrated by the four steps in Figure 2. A qualified test case should verify the main functionality of the canonical code solution. In the case where annotators do not understand the language of the intent, we use translation tools such as the Google Translate 3
2 The ODEX Dataset
In this section, we describe our four-step process of constructing the ODEX dataset. We first collect resources of natural, open-domain coding queries 1). Next, we establish the annotation standard and procedures for test case creation ($2.2). We then describe the annotator hiring and working pro- cesses ($2.3). Finally, we conduct checks to ensure data quality ($2.4).
Step 1:
Wrapping Snippets into Functions Code solutions in (M)CoNaLa are often short snip- pets (e.g.. x np. 25) to ensure more pre-
httpss//translat..oogl.coo

---

NL
Calealate sum over all rows of 2D numpy army 'n'
test case enables successful pass of the solution, should not be taken as valid annotation.
s.sun(oxis==1)
function(als
Sepp podle wispping
2.3 Annotator Hiring and Task Fulfillment
Ies
Ibrery Import
Lnport numpy a5
execute
As our data involves diverse functionalities from multiple libraries, our annotation task holds a rel- atively high standard for annotators. A qualified annotator should be proficient Python and com- mon libraries, and in writing workable test cases.
no.arrayc
(1 for 1 in ranget33] for 1 in ranget5)
1)
essert no arrey, equalf
functioniall, op.arraylBB, 3, 3,
wrbee case
[txj for in range(33] for in range(e)
1>
assert np, mp.array_eeua equalt
functioniaz), ap.arrayl 18, 3, 6, 917
We chose to hire undergraduate students who have strong computer science backgrounds in Python. Of the 20 applicants who applied, we first conducted resume screening to filter candidates with sufficient programming experience. Next, we gave each candidate an annotation test with five randomly selected NL-Code pairs. Since the test mirrors the official annotation process, we provided clear instructions about each step (as in $2.2) and code scripts for reff-verification. Candidates were asked to finish their tests in three calendar days. Based on their test performance, hired four can- didates to officially participate in this job.
Figure An example annotation comprising four steps.
cise matches with NL intents, but to be executable they often need additional context such as variable assignments. We therefore wrap code into stan- dalone functions by specifying input and output arguments as contexts. For example, Step in Fig- ure 2 identifies variable a as an input argument.
Step 2: Specifying Library Prerequisites Due to the open-doman coverage of (M)CoNaLa, some code snippets require extra library imports to exe- cute correctly. Accordingly, our second step is to specify the prerequisite libraries for code solutions.
2.4 Quality Check
We put great effort into ensuring data quality throughout the annotation process. To assist notators in more efficiently and accurately writing workable test cases, we require them to execute each written test case using the verification code that we provided, and explicitly report whether the canonical code solution can successfully pass all the annotated test cases that they created.
Step 3: Test Case Annotation Next, we write test cases that contain three parts: (1) input: passing values to input arguments, (2) output: stating ex- pected execution outputs, and (3) assertion: check- ing if execution results match the expected outputs. However, test case creation for open-domain code faces three challenges. First, safe and re- producible execution can be hard to achieve. As in Figure a it is impractical to send an HTTP request when evaluating this sample. Instead, we use mock to simulate the output (a success response status code 200). Second, some codes entail ran- domness (e.g., random. randint (3) and have no definite value. We instead make bounding as- sertions, e.g., checking that all elements are in- tegers within the range of [3,5]. Third, stan- dard equivalence checks by == may be invalid, since library-speciff objects often require special- ized equality checks. For example, checking the equivalence of two NumPy arrays and b uses awwrarrrruuuuu (a, b). while == b would cause execution errors.
After the annotation, the authors performed post- hoc verification to check each test case reads rea- sonably and executes correctly, In our final rounds of automatic quality checks, we confirm that the pass rate for all canonical code solutions over their annotated test cases is 100%.
We collect a total of 945 samples with NLs in four languages, including 439 samples in English, 90 in Spanish, 164 in Japanese, and 252 in Russian.
Dataset Analysis
We analyze ODEX from three aspects: domain diversity ($3.1), sample complexity ($3.2), and ex- ecution support ($3.3).
3.1 Diversity
Step 4: Self Verification In the last step, perform sefficeifiiationn to efficiently ensure the annotation quality. We exccute the canonical code solution on each newly created test case. Unless the
One unique property of ODEX is its broad domain coverage. We categorize codes that entail library usage (both built-in and third-party) as being in the

---

open domain and those with none in the closed do- main. Different libraries often serve specific func- tions and have unique capabilities. For instance, the datetime library is designed to handle date/time operations, while other libraries focus on various other fields such as data amalysis or web requests. Therefore, in this work, we view the diversity in libraries as representation of distinct domains.
examples use zero libraries. Such examples deviate from realistic programs, which often use APIs of different libraries. DS-1000 and Exe-DS feature some open-domain problems, but their library us- age is more homogeneous with a particular focus on data science domains. Moreover, DS-1000 re- stricts to code using libraries but only has seven libraries. In contrast, ODEX is more "colorful"; it covers significantly more open-domain libraries, well as frequent queries in the closed domain.
Size
Language
Unique Libraries
Open
Closed
Total
230
209
439
Comparison to Natural Distribution To pro- vide reference on natural domain distribution, we approximate real-world usage by counting GitHub Python files that use each library. As shown in Fig- ure 5, ODEX presents a better alignment with the practical scenario concerning the open domains it features more diverse domains and preserves the long-tailed pattern in practical scenarios.
20
48
42
90
ja
113
51
164
114
138
252
Total
505
Table Number of open- and and number of libraries involved in each language.
examples,
Table reports domain statistics and Figure 3 shows the library distribution. ODEX covers di- verse set of 79 libraries, which varies per language. Most samples, 53. 63.4%, use at least one library.
The full lists of libraries and their frequencies about ODEX, the eight comparison datasets, and the approximated natural setting are $A.1.
none
requaatte
datettime
pandas
boi
uriib
numpy
Neroools
rendom openator
re
os
collections json
time
matpletibb subprocess math
60 more
none
datetime
pandas
urilb
numpy
sys
re cs
random
collections
jeon
67 more
matplosib
Figure 5: Approximated natural distribution based on GitHub Python files in the open domain.
Figure 3: ODEX library distribution.
Comparison to Existing Datasets We compare ODEX with eight other code generation datasets that support test case execution: HumanEval (Chen al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021). MTPB (Nijkamp al., 2023), P3 (Haluptzok et al., 2023), DSP (Chandel et al., 2022), DS-1000 (Lai et al., 2022), and Exe- DS (Huang et al., 2022).
3.2 Complexity
To measure dataset complexity, we first calculate the lengths of NL intents and code snippets. We tokenize NL intents with the spaCy" tokenizers in respective languages;
follow Yin and Neubig (2018) to tokenize code. For code, we also parse the AST tree using the Python standard ast library, and count the number of input and output variables to quantify the complexity of execution contexts.
none
Bertools random
collections
heappo
math
none
math
collections
hashiib
Language
len(NL)
len(Code)
depth(AST)
Nr.
Naa
en
14,36
18.49
7.02
1.13
0.21
none
skicamm
numpy
matpictib
pandas
scipy
18.69
28.62
7.74
1.46
0.64
ja
17.24
17.70
6.77
1.40
0.41
11.39
pandas
numpy
matplettib
skleam
solpy
pytorch
more
none
skleam
pandas
numpy
matplotib
scipy
Table 2: Complexity measured in the averaged number of NL words, code tokens, AST depth, and i/o variables.
collections
heapq
Itertools
In Table 2, we see that code in the Spanish set is longer on average than other languages. For both the input and output sides, code in the English set has fewer variables, suggesting potentially simpler
pandes
numpy
collections
Figure 4: Library distribution of eight other datasets.
From their distributions in Figure 4, six out of eight datasets focus on the closed domain and most
https://sppcy.iof

---

Dataset
Samples
Domain
Executable?
Avg. Test Cases
Duta Source
NL
JulCe (Agashe et ul., 2019)
1,981
open
GitHub Notebooks
en
HumanEval (Chen al., 2021)
164
7.7
Hand-written
MBPP (Austin al., 2021)
974
8
3.0
written
APPS (Hendrycks et al., 2021)
10,000
0
13.2
Competitions
DSP (Chandel et ul.. 2022)
1.119
16
2.1
Github Notebooks
MTPB (Nijkamp et al., 2023)
115
8
5.0
Hand-written
en
Exe-DS (Huang et al., 2022)
534
28
GitHuh Notebooks
en
DS-1000 (Lui et ul., 2022)
1,000
1.6
StuckOverloww
em
CoNaLa (Yin et ul., 2018)
2,879
open
StucKOverlow
MCoNaLa (Wang al., 2023)
896
open
StackOverflow
es,jül
StackOverflow
ODEX
945
79
1.8
en, 0s, ja, ru
Hund- Written
Table 3: Comparing ODEX with other NL-to-code generation datasets, in terms of domain diversity (Domain), test-case execution support (Evaluation, Avg. Test Cases), and natural language contexts (NL). Since hard calculate the exact number of libraries for open-domain datasets that do not specifically import required libraries in the code, we mark their domains as open instead of providing the exact number of domains.
execution environments, which could stem from relative simplicity of SO queries asked in English.
on THEPIL.E (Gao et al., 2020), BIGQUERY," and BIGPYTHON datasets are denoted as NL, MULTI, and MONO. The most powerful CODEGEN-16.1 MONO, performs similarly to CODE-CUSHMAN 001 the HumanEval and MTPB datasets.
3.3 Execution Support
We systematically compare code generation datasets that concern execution or open-domain code in Table 3. ODEX is the first dataset that sup- ports execution-based evaluation for open-domain code. While ODEX does not have the largest num- ber of test cases, we discuss in $7 how these test cases can still reliably measure code correctness.
Prompt Design For fair comparison, we use the same prompt for both model families. While prompting with few-shot in-context examples may improve, our experiments do not always find this helpful for both models. Therefore, report zero- shot results baselines and leave few-shot results to $7. Creating zero-shot prompts only requires content from the test sample. Following Chen et al. (2021), we construct prompts by concatenat- ing function context and a docstring. A docstring includes the NL intent and optional unit tests (com- pared in $7). Figure 6 shows an example prompt.
4 Experiment Setup
Code LLMs have achieved strong results on mul- tiple code generation tasks, yet their open-domain proficiency is understudied due to the limited do- main settings of past datasets. To examine model capabilities the open domain, we evaluate two top-performing model families, CODEX and CODE- GEN, on ODEX. We perform cvaluations using prompting setting, without finetuming any model. We introduce the baseline models, the prompt settings, and lay out the metrics for evaluation.
def functionley stringh:
Ravsion Dvnell
uomConvart string myy_strnn" number "ey_float"
Dectrine
assort (Tunction (2224.888)) 1234.8) 16-6
return my. loat
Pandaon Dansol
Figure 6: Zero-shot prompt with one test case in doc- string. The gray box notes the place for code solution.
The CODEX Family At the time of this work, CODEX had three publicly available models. CODE- CUSHMAN-001 (C1) is a 12B CODEX model in Chen et al. (2021). CODE-DAVINCI-0011000 (D1, D2) are two 175B GPT-3 models.
Evaluation Metries We follow Chen et al. (2021) and measure the execution accuracy using the pass@k @k k metric, by computing the fraction of prob- lems having at least one correct prediction within h: samples. We also compare it with series of execution-free metrics later in $5.
The CODEGEN Family CODEGEN (Nijkamp et al., 2023) models are auto-regressive models trained on a combination of NL and code corpora, differing in model sizes (350M, 2.7B, 6.I 16.1B) and training data. Models progressively trained
Implementation Details We follow Chen et al. (2021) and use nucleus sampling (Holtzman et al., 2020) with top-p set to 0.95 and temperature set to 0.8. We set outputs to a maximum of 512 tokens.
httpps:/hetaa.opeaal com/docs/ model- www--iiddxxx----ceeeeee
hhtps://clou..ooog// hongle.com/higguery

---

Tu puss @ A
pass @ k
Language
CODEX
CODEGEN
2
5
10
2
5
10
en
31.91
44.67
59.95
68.79
26.26
32.18
39.10
42.82
31.89
43.33
55.72
63.33
16.67
21.85
27.82
30.00
CUSHMAN-00
350M
25.67
36.69
49.27
$7.32
17.44
22.86
28.21
30.49
ru
40.00
53.48
66.63
73.41
25.87
31.44
37.44
40.87
33.62
46.65
60.18
67.43
35.24
42.87
50.68
53.99
36.89
49.46
61.37
68.89
26,00
33.65
41.52
45,56
DAVINCI-001
2.7B
ja
31.04
42.1
54.26
61.59
24.27
32,10
41.13
45.12
ru
43.21
57.53
70.03
76.59
39.64
48.11
57.23
61,90
47.15
57.61
67.87
73.12
34,49
37,91
41.18
43.05
es DAVINCI-002
47.44
57.90
66.33
71.11
6.1B
28.56
32.05
35.86
37.78
ja
41.46
50.42
59.47
64.02
35.55
40.11
44.12
46.34
51.87
63.36
73.03
78.17
44.64
47.29
49.82
51.19
Table 4: Execution accuracy of CODEX and CODEGEN-MONO models.
5 Experiment Results
5.2 Open Domain versus Closed Domain
CODEX Results Figure 7 (left) shows pass @ on open-domain and closed-domain. All CODEX models score much lower in open than in closed domain. Such large gaps hold across all languages, ranging from 4.34 in Spanish to 38.57 in Japanese on the best DAVINCI-002 model. Model upgrades (cl - D1 + D2) do not always reduce the gaps. Gaps slightly shrink in Spanish, but increase English and Japanese. While D2 performs the best, also exhibits the most severe gaps. These findings suggest that common practices to improve LLMs may not address the complexities inherent in open- domain coding problems. It is hence imperative that more advanced strategies employed.
We first present the overall performance of two model families ODEX ($5.1). Next, given the unique challenges of open-domain code, we study the variances between open- and closed-domain problems ($5.2), and in individual domains ($5.3).
5.1 Baseline Performance
CODEX Results As in Table 4, aligning to exist- ing works and our intuition, larger DAVINCI 175B
models outperform the smaller CUSHMAN 12B
model, and the 002 version improves over 001. This trend holds for all languages and all sampling sizes. Somewhat surprisingly, all models attain de- cent results on nog-English problems, even though CODEX not designed for multilingual use. This high accuracy on non- oon-Engiihh problems suggests the multilingual potential of CODEX models.
CODEGEN Results As shown in Figure (right), CODEGEN also has substantial gaps between open and closed domains, however, smaller than CODEX gaps across all languages, by on average 6.0% points. As model size increases from 2.7B to 6. 1,, the gaps reduce by about 6.3 points in English and 1.7 points in Spanish. This is in contrast to CODEX, which when scaling up to DAVINCI-002, these gaps continue to increase by 4.9 points on average, indi- cating that scaling up CODEGEN more effectively catches up on open-domain performance.
CODEGEN Results We report results of MONO models Table 4 given their superior performance over and MULTI variants (Nijkamp et al.. 2023). The pass rate increases as CODEGEN grows from 350M to 2.7B, and continues to increase in non- English languages when further scaling to 6.1B. CODEGEN exhibits multilingual capacity, as its results on non-English subsets are close to that on English, and consistently increase during scaling, Although CODEX and CODEGEN have compa- rable performance on existing datasets such as HumanEval, ODEX effectively unveils the effi- cacy of CODEGEN on open- coding queries even with many fewer parameters, i.e., CODEGEN 6. B. yields similar pass@1 @ the 176B CODEX DAVINCI-001 model, although not necessarily so when k increases. More detailed results (pass@k at 1 < k < 10) for both models are in SB.
5.3 Domain Variance
We now dive deeper into the results within individ- ual domains. We focus on the CODE-DAVINCI-002 model as it has the best performance across all mod- els. In Figure 8, we plot accuracy with respect to the domain frequency, approximated in $3.1. Execution accuracy is not low on all open do- mains. For example, CODE-DAVINCI-002 achieves 50% pass @ / for several common libraries such as random and math. But high domain frequency does not ensure model proficiency. For example,

---

en open
60
closed
es open
closed
30
ja / open
ja / closed
15
ru / open
ru / closed
davinci-001
davinci (002
Figure 7: CODEX (left) and CODEGEN (right) pass@/ I on open- and closed-domain problems in each language.
ranic(o, 10)
70)
rank: (70,
formance. As shown in Figure 9, model rankings by execution-free metrics do not precisely cor- relate with their rankings by exccution accuracy. Even when the rankings align, their differences are largely not proportional. Comparing the metrics, ChrF and METEOR have smaller inter-model vari- ances, while BLEU and ROUGE change more and correlate better with pass rates. Notably, Code- BLEU is low in most settings and might not be suitable for evaluating code in snippet-style.
recar
pasag1
pirda
Une
10/32/000
3c,01.00
Appreximated
Figure 8: CODEX pass@1 1 for domains of varied fre- quencies. Domains are differently colored based on their frequency ranking: the 10 most frequent domains in red, the 10 least frequent domains in blue, and other domains in the middle in yellow.
Metric Correlation We next evaluate whether execution-fire metries might be used to discrimi- nate between passed and failed samples. We take BLEU as an example since it shows similar rank- ing patterns to execution. Figure 10 shows negligi- ble variances in BLEU scores of passed and failed groups. The other four metrics exhibit similar pat- terns, as could be found in SD.3.
on libraries with complex functionalities such as matplotlib and tensorflow, pass@/ can go be- low 10%. See SC for more domain-wise results.
6 Comparing to Execution-Free Metrics
RN
Tel
25
feil
In this section, we study the alignment between execution-based evaluation and five exccution-free metrics, identifying advantages for both types.
pass
pass
591
fco-
coc
Model Ranking Using Different Metrics We evaluate models using five execution-free met- rics using lexical, syntax, and semantic matches: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), Chrli (Popović, 2015), and Code (Ren et al., 2020). Refer to $D.1 for more descriptions.
coc
castmmn
davinct 002
43.52
Figure 10: BLEU scores on passed and failed samples.
80
16-47
56.05 55.39
47:15
44.40
What Affects Model Performance?
5637
40
Besides differences in model configurations, we study three factors that might affect performance.
20
3.77
WHACE VIVE0
21.37
$1.00
50.51
Iiei
masper
meteaa
dtrl
codebieu
Number of In-Context Examples Models might benefit from example NL-Code pairs. We thus ex- plore to few-shot prefixing N E 21,2,3) input- output pairs in prompts. In Figure 11 (left), for CUSHMAN-001 and DAVINCI-001, shw- ex-
375
Figure 9: CODEX models evaluated on six metrics.
We analyze using CODEX, given its better per-

---

en/1
amples yield clear improvement over the zero- shot setting; but for the strongest DAVINCI-002, brings minimal gains in English. See similar results in other languages in SE.1.
enn/n
es/1
es//n
1j/
one
*ja/n
daviccs-0ee
Iu/
PB10
ru/n
80
50
Figure 12: pass@ when executing one or all test cases.
40
but face challenges in enabling execution (Lai et al., 2022: Chandel et al., 2022). Our ODEX dataset addresses exccution for open-domain code.
30
30
10
Figure 1I: Left: CODEX pass@1 / (on English set) using 0/1/2/3-shot prompts. Right: DAVINCI-002 pass@) when adding zero, one, or all test cases in prompts.
Coding Queries vs. Programming Challenges Some works stem from coding contest web- sites (Hendrycks et al., 2021; et al., 2022), but GitHub Jupyter Notehooks (Agashe et al., 2019; Huang et al., 2022) and StackOverflow (SO) (Yin et al., 2018; Wang et al., 2023; Lai et al., 2022) pro- vide more natural and practical coding queries. We preserve this naturalness and incorporate various NL settings to assist programmers worldwide.
Number of Test Cases in the Docstring Includ- ing test cases in inputs adds execution hints of the expected functionality of the solution, and hence may improve execution accuracy. We test this hy- pothesis by experimenting with prompts that have varying numbers of test cases. Besides the default setting with zero tests, we compare adding one random test case and all annotated test cases.
Execution-based Evaluation Evaluation by ex- ecution has long been used for SQL (Zhong et al.. 2017) or logical forms (Dong and Lapata, 2016). Many datasets have begun to support Python ex- ecution via test cases, however focus on built-in functions (Chen et al., 2021; Austin et al., 2021; Hendrycks al., 2021) or specific domains (Lai et al.. 2022; Huang et al., 2022). Our test cases, in contrast, cover diverse libraries in the open domain.
Figure 11 (right) shows that injecting as few as one exemplar test case significantly improves the execution accuracy, yet adding more cases has little bonus. This potentially implies the sufficiency of one test case to show the main functionality.
Number of Evaluation Test Cases Execution results could be more reliable if using more test cases for evaluation. However, there trade-off between evaluation effectiveness and annotation efficiency, due to the high cost of human effort. To study this tradeoff, we observe how results change with respect to the number of tests. Compared to using all cases in default, we also try using one randomly selected case. For simplicity, we do not include any test cases in prompts.
Conclusion
We present ODEX, an open-domain code genera- tion dataset supporting execution- evaluation via human-written test cases. ODEX not only sup- ports execution-based evaluation of code using test cases, but also extends the task to the open domain, covering 79 diverse Python libraries and four natu- ral languages (English, Spanish, Japanese, and Rus- sian). Comparing two state-of-the-an code genera- tion models, CODEX and CODEGEN, our dataset effectively unveils their varied behaviors between program domains and language contexts. ODEX serves as a comprehensive NL-tt-code benchmark given its open-domain coverage, multi-natural lan- guage queries, and multi-metric support. When bringing code execution open domain scenarios. our explorations also reveal emerging challenges in test creation and reliable exccution, which we hope that our dataset will enable future work to tackle.
As shown in Figure 12, evaluating over one ran- dom test largely preserves the accuracy of using all tests, indicating that one case sufficient to test the main functionality for most queries. Check ŞE for analysis on other factors such as function naming.
8 Related Work
Open Domain Code Generation Programs of- ten use APIs from different Python libraries. Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack- Overflow posts (Yin et al., 2018; Wang et al., 2023),

---

Limitations
clearly documented instructions, flexible annota- tion interfaces (Google Sheets, Jupyter Notcbooks), and self-verification tools. We (authors) conduct pilot annotation to confirm the clarity of annotation standards and feasibility of the annotation task. We conduct posthoc examinations on the annotation results, both manually and automatically, to obtain assured data quality (100% pass rate).
ODEX aims to serve as a comprehensive testbed, by enabling execution-basse evaluation of code in the open domain, with flcxible intent inputs in four natural languages. However, we should hold contin- uous awareness of execution security, multilingual support, and evaluation reliability.
First, execution supports in ODEX enables more rigorous evaluations than other execution-free methods. However, due to the increased complex- ity of open-domain codes, more inspections are required for execution safety, either for code solu- tions or test cases. We should always keep alert to avoid concealing malicious code (Wallace et al., 2021) or generating code with security vulnerabili- ties (Verdi et al., 2020; Pearce et al., 2021).
We respect the contribution and privacy of our annotators. We offer competitive remuneration for their annotation job and treat each one of them fairly. All annotators possess the right to withdraw at any time. We secure that all their personal infor- mation is removed before public release.
We conduct systematic analysis from multiple perspectives in the paper, in an attempt to foster public awareness generating and evaluating pro- grams in the open domain, both in encouraging more advances in this direction, and raising more concerns about the robustness and security of such unique coding problems.
Second, in addition to English inputs, ODEX also includes intents specified three other lan- guages. Still, its language coverage is bounded by the available forums in StackOverflow. We hope our initiative can highlight the multilingual nature of program developers, encourage the emergence of similar data resources in other languages, and continuously promote AI programming assistance in languages worldwide.
Third, as ODEX covers wide-ranging code queries in the open domain, it more suitable for less resource-demanding scenarios such as down- stream evaluation or few-shot learning. Although ODEX is larger than many previous datasets with Luman--ritter test cases, it is still limited due to the intense human effort required by the curation process. Regarding this, encourage users of the dataset conduct significance testing (Dror et al., 2018) and report more substantial model improve- ments.
Ethics Statement
Our work has received IRB approval and is licensed under a Creative Commons
(CC BY-SA) 4.0 International License. The result- ing ODEX dataset is built to serve as a benchmark for open-domain code generation, to further facili- tate technological advances in AI programming as- sistance, meanwhile supporting multiple languages to encourage its universal accessibility.
We strive ensure high data quality and opti- mize annotation efficiency. We build the ODEX dataset with natural and practical StackOverflow re- sources and hire annotators with qualified program- ming proficiency. We provide our annotators with

---

References
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Anmual Meeting of the Association for Compu- rational Linguistics (Volume I: Long Papers), pages 33-43, Berlin, Germany. Association for Computa- tional Linguisties.
Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019. Juice: A large scale distantly supervised dataset for open domain sontext-based code gener- ation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 5436 5446.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annuul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't reach for the stars!
Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation mod- els? Journal of Systems and Software, 203:111741.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, and Charles Sutton. 2021. Program synthesis with large language models.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster. Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
Satanjeev Banerjee and Alon Lavie. 2005, METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluution Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2023. Language models can teach themselves to program better.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Eahhn Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge com- petence with apps.
Shubham Chandel, Colin B. Clement, Guillermo Ser- rato, and Neel Sundaresan. 2022. Training and evalu- ating a jupyter notebook data science assistant.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jaan-Guang Lou, and Weizhu Chen. 2023. Codet: Code generation with generated tests. In In- ternational Conference on Learning Representations.
Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution- based evaluation for data science code generation models. In Proceedings of the Fourth Workshop on Data Science with Human- in-the-Loop (Language Advances), pages 28-36, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto. Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings. Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.
Srinivasan Iyer. loannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1643-1652, Brussels, Bel- gium. Association for Computational Linguistics.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang. and Tao Yu. 2022. Ds-10001 A natural and reliable benchmark for data science code generation.
Raymond Li, Loubna Ben Allal, Yangtian 7i, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!
Naihao Deng, Shuaichen Chang, Peng Shi, Tao Yu, and Rui Zhang. 2021. -tesx-tt-sql: Text-to-sql genera- tion from incomplete user questions.

---

Yujia Li, David Choi. Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. 2022. Competition-Ievel code generation with
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies, pages 139-150, Online. Association for Computational Linguistics.
alphacode. Science, 378(6624):1092211
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 81- Barcelona, Spain. Association for Computational Linguistics.
Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank Xu, and Graham Neubig. 2023. MCoNaLa: A bench- mark for code generation from multiple natural lan- guages. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 265-273, Dubrovnik, Croatia. Association for Computational Linguisties.
Chin- Yew Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- (4), pages 605-612, 2, Barcelona, Spain.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. International Conference on Min- ing Software Repositories, pages 476-486. IEEE.
Stephan Lukasczyk and Gordon Fraser. 2022. Pyn- guin: Automated test generation for python. In International Conference on Software Engineering: Companion Proceedings.
Pengcheng Yin and Graham Neubig. 2018. TRANX:
A transition- basedd neural abstract syntax parser for semantic parsing and code generation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 7-12, Brussels, Belgium. Association for Computational Linguistics.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representa- tions.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021. Asleep at the keyboard? assessing the security of github copilot's code contributions.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: method for automatic evaluation of code synthesis.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit test case generation with transformers and focal con- text.
Morteza Verdi, Ashkan Sami, Jafar Akhondali, Foutse Khomh, Gias Uddin, and Alireza Karami Motlagh. 2020. An empirical study of c++ vulnerabilities in crowd-sourced code examples. IEEE Transactions on Software Engineering.

---

A ODEX Dataset
Domain Statisties of Comparison Datasets Ta- ble lists the library frequency of eight compari- son dataset mentioned in $ 3: HumanEval, MBPP, APPS, MTPB, P3, DSP, DS- 1000, and Exe-DS.
A.1
Library Distribution Statistics
Aside from the illustrations in 3.1, we list out the detailed statistics of libraries in ODEX, the eight comparison datasets, and the approximated natural distribution.
HvaanEEal
Library
Count
Frequency
Library
Connt
Frequency
none
185
94.51
ODEX Domain Statistics Table lists the num- ber and percentage of occurrences for each library
hashlib
collections
0.61
in the ODEX dataset.
Library
Count
Frequency
Library
Comnt
Frequency
none
81.52
73
7.49
emath operator
0.31
Library
Count
Frequency
Library
Count
Frequency
collections
25
2.57
army
none
functools
heapy
16
1.64
bisect
0.21
pundas
81
7.71
htp
2
0.19
Blctoools
1.23
0.10
mumpy
50
7.62
obspy
2
0.19
sys
datetime
62
5.90
pickle
2
0,19
Pstz
APPS
collections
matplotiib
26
22
2.10
2.48
scabom
sqltkchemy
2
2
0.19
0.19
Library
Count
Frequency
datctime
21
2,00
statistics
2
0.19
urllbb
ays
MTPB
mndom
16
1.52
IPython
0.10
Librury
Count
Frequency
Librury
Count
Frequency
15
1.43
argnarse
0.10
1.43
aspese
0.10
108
88.03
subprocess
reguests
10
0.95
egi
0.10
pandos
3
2.56
collections
0.85
bs
0.86
configparser
0
2.56
datetime
0.85
iemools
0.86
ctypes
0.10
numpy
operasor
sklearn
regex
time
0.86
difflib
0.10
math
0.76
docstpl
0,10
builtins
0.57
Bleemp
0.10
Library
Count
Frequency
Library
Connt
Prequency
Aplib
Rensorfoow
0.57
bashlib
0
92.19
django
0.48
beupg
0.10
Hectoools
35
2.04
heaps
15
0.87
sylinc3
0.48
0.10
randam
inspect
collections
28
math
eodecs
0.38
cv2
0.38
laml
0.10
scipy
0.38
mechanize
0.10
Library
Count
Fregnency
Library
Comnt
Prequency
toolkils
92.79
csv
flack
0.29
0.29
peeint
quene
0.10
0.10
skiicarn
110
5.02
collections
0.36
glab
0.29
regex
0.10
numpy
matplotiib
2.28
gzip
0.18
socket
0.29
sal
0.10
pondas
46
2.10
pickle
0.18
stract
0.29
texitable
0.10
46
2.10
random
0.18
wmpy
0.29
wamiings
unicodedata
0.10
numbers
0.55
licctools
csv
ast
0.19
xml
0.10
util
12
0.55
scuborn
0.09
DS-1000
Table 5: ODEX library distribution.
Library
Frequency
Library
Frequency
pondss
291
29. 10
scipy
106
10.60
numpy
220
22.00
Pytorch
68
6.30
15.50
sálearn
115
11.50
Librury
Count
Frequency
Librury
Count
Frequency
none
56.23
sklearn
75
11.13
Pylab
0.30
numpy
7,86
arch
0.15
matplatiib
32
4.75
ePicklc
0.15
2.67
cufi
0.15
csv
math
1.04
datetimo
0.15
collections
4
0.59
functuols
0.15
0.59
graphviz
0.15
folium
json
nlik
0.45
mpLioolkits
0.15
statsmodek
0.45
operator
0.15
wumnings
0.45
0.15
IPython
Table 6: Library statistics of eight comparison datasets.

---

Approximated Natural Domain Distribution To approximate the natural distribution of libraries in the open domain, we count the number of Python files on GitHub that imports the library of interest. Following the GitHub search syntax," we use the query import $[1ibrary_name) to search files that import certain library, and use NOT import to count files not using any libraries. Their frequen- cies are shown in Table 7.
and get a better understanding of the question. If any errors or snder-spcciiication are spotted in the given NL or code, we ask the annotators to correct by making the minimal change possible.
Aligning with how programmers import li- brary, we require the expressions be written in three forms: (1) import $(LIBRARY), (2) import $(LIBRARY) as $(ABBR), or (3) from S[LIBRARY) import $(FUNCTION), where the $[LIBRARY) can also be sub-classes such as matplotlib, pyplot.
Approximated Natural Distribution
Library
Count
Library
Count
We encourage the annotators to use the language identical to the given NL intent when creating the test cases, especially the code involves string- related operations (e.g.. writing regular expressions in Japanese). We encourage the annotators to write reasonably more and diverse test cases, by varying the values or types of variables.
08
30,188.921
sqlite3
694,794
sys
24.213.844
configparser
640,014
numpy
20,965,506
queue ssl http
631,326
11.762.193
602,351
time
5,946.718
597,866
pandas
5,878.651
xml seabom
574,030
random
5,740.444
567,576
matplotlib
5,416.874
imp
560,862
json
4,792.536
builtins
560,148
Please find the full instruction" and examples for annotation in our code repository.
tensorflow
4,720.266
locale
542.607
argparse
4,570.391
ast
444,349
subprocess
4,165,781
bisect
315,031
string
4.114.004
pytz heupq
295.167
B Baseline Results
codecs
3,973,691
281,393
warnings
3,824.001
cgi
277,852
math
3,569.158
unicodedata
267,310
According to the baseline results in 5.1, we pro- vide more detailed evaluation results, on the exe- cution pass rate ranging from the top-1 to top-10 model predictions. Table and Table 9 show the zero-shot execution accuracy of CODEX and CODE-
django
3,447,092
regex
235,800
shutil
2,999.394
difflib
225,154
requests
2,837.310
PIL
218,526
cv2
2,575.063
sklearn
208,913
datetime
2,536,970
statistics
127,725
socket
2,489.033
rsa
122,447
GEN models, respectively.
pickle
2,419,604
Ixml
111,742
io
2,190,998
dateutil
107,041
collections
2,152,651
bs4
90,224
Modal
603
655
Pass Rate
017
0110
glob
2,114.567
xlrd
86,522
itertnols
1,899.461
filcemp
79,328
C1
25.77
41:31
36.00
49.33
47
45.04
5100
40.31
66.12
51.44
47.51
50:53
53.33
01.56
52
63.29
$0.10
63.11
47.33
urllib
1,809,462
IPython
73,274
$3.08
00.04
61.06
06.33
68
30.17
71
73.90
13.41
csv
flask
1,788,601
1,680,232
selenium
sympy
70,969
56,709
11
36.59
33.33
45:45
42.11
$5.44
47.83
$8.96
$7.34
$1.54
51.31
$4.25
43.23
38.3
64.75
54.0
35.11
33.29
08.3
3633
74.51
67.56
62.87
75,60
58.89
$1.59
76.59
functnols
1,433,520
xlwt
55,035
53.61
73.12
pprint
1,378,679
ftplib
52,121
102
47.44
16
$1.90
30.42
56.64
62.30
64.55
$7.59
56.33
59.47
67.81
00.84
68.63
61.87
09.53
62.
1333
63
$6.22
71.11
basc64
1,352,623
multidict
29,224
63396
75.00
35.67
71.46
76.17
hashlib
1,330,158
mechanize
20,978
scipy
1,121.371
obspy
5,799
Table 8: CODEX zero-shot performance.
inspect
1,112,770
texttable
4,749
operator
1,104,841
aspose
1,048
Model
ctypes
164,108
docxtpl
76
0-2
60.3
815
066
807
601
09
sqlalchemy
814,096
mpl_toolkits
2
86
39
41114
struct
787,484
350M
17.44
17.36
25.19
33.97
35:53
30.49
2537
31
34.37
36.11
37.81
34
19:33
38
43.41
40.57
17
48
49:57
49.90
Table Approximated natural domain distribution.
15.20
14:80
40.15
$1.56
40:00
41.29
41.46
45077
45:61
42.00
$1.91
51.55
65
16.04
96.61
$7.10
57.51
41.38
43.11
A.2 More Annotation Details
6.18
35.55
4611
42.04
43.25
46.12
44.77
45.2
45.00
45.04
46.34
4486
41.20
08.53
49.25
40.32
30.25
30.36
3088
31.03
51.19
Along with the NL-Code pair, we also provide IDs of the source StackOverflow post, using which an- notators can trace back the original post webpage
Table 9: CODEGEN zero-shot performance.
instruction.md
"nups://docs. wtthhh.oom/e//sarcch. comm/n/scarch-githuu/ github/ searching-one www.oanniiuuuuuaaahnnnnnnn onde
10
annotation.ipymb

---

C Hamain-Wise Execution Results
C.2 Domain-wise Execution Accuracy As introduced in $ 5.3, we take CODE-DAVINCI- 002, and report its execution accuracy on each domain in Table 12.
We list out detailed results for experiments in $5.
C.1 Open Domain Versus Closed Domain
Table and Table 11 shows the execution accuracy for CODEX and CODEGEN on open open-domain and closed-domain problems, respectively.
Library
Count
Pass@1
Library
Count
Pacs
440
61.45
functools
2
15.00
pundas
81
38.52
hilp
2
40.00
numpy
80
36.18
abspy
2 0.00
62
36.13
pickle
0.00 2
NL
Sptt
U1
402
GF3
G8
SPEB
os
42
42.62
2 20.00
collections
26
35.38
scabom
2 0.00
matplorlib
22
9:00
sqlalchemy
2
50.00
34.00
31.11
66.51
31.31
11168
15.24
27.35
54
$2.36
64.01
41.37
06.13
51.0
63:97
61:24
68.7)
datetime
21
30:95
statisties
2
40.00
urllib
19
14.74
string
2
0.00
31.09
63.33
09:35
53.01
55.13
$1.51
10.53
56
63.35
63.33
sys
17
15.88
xird
2
30,00
cpm
33:21
38.23
45.12
46:14
52.55
15.31
27:18
random
16
62.00
Iyytonn
0.00
Ja
1pm1
23,67
56.55
42.00
46-29
19.27
$1.44
53.23
54.78
15.10
51:33
json
15
15
35.33
32.67
aspose
argparse
100.00
10.00
15.99
50.95
94.26
64.90
06:33
11:01
13.35
79.43
sabprocesss
13
30.77
hisect
0.00
3101
66.07
51.11
55.54
20.95
53.56
64:01
06.15
61.65
06:29
requests
10
37,00
egi
80.00
$1.99
01:28
19.01
81.31
$4.57
96.14
$5.36
$1.85
bei
89
configparser
60.00
11/62
cure
001
itertools
operator
27.78
dalcotil
30,00
60.00
36.11
59:15
15.00
10.35
53.13
15.00
47.03
90.11
63:47
61.74
time
20:00
difflih
0.00
dos
41/00
56.79
01.32
15:14
01:11
59.39
11/12
72.14
71301
13.85
math
61.43
docxip)
10.00
16,09
09:45
55.44
56.96
01:37
53:30
66.73
06:36
67.56
16.99
builtins
76.67
filecmp
40,00
cpm
11,67
43.99
06.83
11:11
54:76
12.75
57.00
06:35
35.05
59.51
T0.35
01.15
72.38
11.67
A2.93
selenium
50:00
fiplib
60.00
11.00
12.11
47.03
51.54
54.25
13.39
18.11
59.33
12:67
61.53
tensorflow
6.67
hachlih
0.00
opz1
23.22
30.32
91.58
41.08
64.11
45.56
45.05
27.45
53.55
51.33
django
5
20.00
heepy
0.00
sqlitc3
38.00
imp
40.00
opmn
41.21
18.89
11.01
51.53
63.05
$5.58
10.77
51.17
70.03
11.55
41:53
13.29
04.51
15.79
15:00
61:36
16,99
codecs
PIL.
35,00
72.50
inspect
localo
0.00
0,10
cure
000
ev2
22.50
Ixml
0.00
41:05
31.1
02.8
10:00
61.8)
59:41
10/00
71..3
11.46
13.12
scipy
5.00
mechanize
0.00
sklearm
mpl. toolkits
0,00
dose
31.25
66/72
73.35
18312
77.79
79.10
19:96
91.88
$1.29
$1.81
buse6-1
6.67
multidiet
90.00
45.22
56.02
02.17
63.65
66:39
10.17
61.52
65.10
19:79
T0.53
36.67
pprint
20,00
dose
46:26
64.55
16:39
60.32
59:26
69:34
70.48
12:95
T1.43
flask
50,00
queue
0,00
glob
43.33
regex
100.00
19-77
57.30
41.91
44.99
05.44
27.75
46.72
20.44
53.00
50.44
shuril
60.00
rsa
dose
18.00
76.61
$3.48
$6.90
85.35
99.22
91.03
92.11
$3.14
14.12
socket
40.00
ssl
0.00
14.00
46.20
11.46
54.55
56.99
18.05
68.29
$2.99
64.94
struct
16.67
texttable
60.00
16.01
77:54
$1007
14
08.33 38
10.86
sympy
0.00
unicodedata
90.00
xlwt
20:00
warnings
70.00
Table 10: CODEX pass rate in open and closed domains.
ast
50,00
And
0.00
Table 12: CODE-DAVINCL 001 execution accuracy on each domain subset inside ODEX.
NL.
02
93
94
05
06
07
66
C9
1010
350M
26.35
1646
13.59
15.10
40.22
41.35
41.75
+2.35
13.32
opcc
22.35
27.0 044
23.55
11.5%
31.15
13.97
14,91
35.45
36/4
15.52
16.67
11 85
3470
76.16
13.02
15.66
70-77
20/55
70.34
30:00
C33 Qualitative Error Analysis
0900
16.54
19.35
21.22
21.22
21.09
12.32
21101
-2.11
22:11
1112
To provide more intuitive explanations of the do- main divergence aforementioned, we conduct error analysis over 60 randomly selected examples from ODEX dataset (15 for each language). By exam- ining the error patterns from these examples, we aim to answer: what are the common error types on open- and closed-domait problems? What are the main differences between them?
17:44
22.89
23:33
27.33
28:21
23:37
20.52
2003
20124
30
34.04
36J0
30:91
10.75
41.16
21.44 41
34:33
37.44
19:44
10.41
91.34
91.11
43.41
13.1
11.54
45.57
16.5%
47.11
07.97
18.55
2.15
30.24
42.97
45.12
43.11
33.00
51.75
53,99
30.19
30.04
3309
1900
53.00
54.07
11.01
13.50
11.00
N:.0
13:05
37.14
03.06
41:45
17.5%
11.00
14.70
15.6
1900
22.30
27:4 4.
23.08
13.06
34.7%
15:22
31.26
2012
19:41
41.7
04.27
32. 22.18
35.45
39:33
41.15
43.51
43.5
44.30
04.01
45.12
36.57
50.20
57:52
$3.09
14.7%
66.33
18.14
05.9
1000
75139
19.41
15 11
11.25
19:21
61
61.0
Similar to the previous section, we take the CODE-DAVINCI-002 since it scores the best and presents clear domain gaps, which might give more intuitive variances between domains.
56.57
09 1.8
00.00
65.01
15.00
10:45
T1al
73.38
72:93
15.19
55.43
27.53
33.22
4333
41.15
41.00
41111
42.47
42276
4318
1900
25.30
$1.5
31
14.55
3510
15
3617
57.84
25.15
22.04
33.85
15.00
35.36
15.%
7636
31.25
37.56
31.75
spen
25.63
33.03
14.06
14.54
45:21
15:36
11.90
(3.56
Closed-Domal Errors Of the 60 random sam- ples we analyzed, 31 are closed-domain problems, and CODEX predicts erroneous code solutions for 22 of them. We identify four main types of errors
33.33
21 21
4114
4122
44.12
44/77
49:25
43,91
4004
45.4
36.52
35.17
51
6126
54.11
13.01
66.4
18.0
49:3
10.23
30.96
20022
51.33
11.55
5612
07.47
0337
0380
$4.00
64/3
64.09
66.90
65:07
15.22
Table 11: CODEGEN pass rate in various domains.

---

from these samples: (1) 11 cases (50.0%) use the Python built-in functions incorreetly, mostly about strings manipulations and number caleulations; (2) 7 cases (31.8% failed at complex functions, which usually require multi-step implementations: (3) cases (18.2%) received empty predictions, poten- tially because they involve unfamiliar topics to the model; (4) 2 cases (9.1%) imports extra library or add redundant implementations.
overlapping units (n-gram, word sequences, and word pairs) between prediction and references. Among the multiple variants proposed (ROUGE-N. ROUGE-L, ROUGE- W, and ROUGE-S), we use the most common ROUGE-L in our experiments.
METEOR METEOR (Banerjee and Lavic, 2005) is a unigram-based metric originally in- tended for machine translation. It builds on gener- alized unigram concept by involving unigram pre- cision, unigram recall, and word order measures.
Note that the number of error cases these four categories does not add up to 22. Since we analyze all the error predictions among the model top-10 predictions, one case could present multiple error types in its different predictions.
ChrF Chrli (Popović, 2015) targets lexical match the character level, by calculating the character- level n-gram F-score between predictions and ref- erences. ChrF is also originally proposed for the machine translation task, but later adopted for some code evaluation works (Evtikhiev et al., 2023).
Open-Doman Errors Of the other 29 problems belonging to the open domain, 26 of them have erroneous predictions. Errors in the open domain exhibit more diversity than in the closed domain. The major eиror enclosing 16 cases (61.5%) is the failure to use the prerequisite libraries, missing part of them when multiple libraries are involved. The next major type is using incorrect functions, which happens in 9 cases (34.6%). Similarly to the elosed-domain errors, cases (19.2%) have error usage of correct functions, 4 cases (15.4%) struggle with complex multi-step implementations, and cases (11.5%) face empty predictions.
CodeBLEU CodeBLEU (Ren et al., 2020) is specifically designed for code evaluation, by jointly considering the surface-form match, syntax simi- larly, and semantic data flows.
D.2 Evaluating with Non-execution Metrics
Table 13 and Table 14 shows the scores of CODEX and CODEGEN using non-execution metrics.
OD and CD problems share some error cate- gories such as function misuse and complex oper- ations. Nonetheless, open-domail problems intro-
Model
NL
Metrics
CodeBLEU
31.27
52.79
55,43
43.07
3.18
3,96
18.57
46.67
48.76
34.89
3.63
duce extra challenges: correct selection and usage libraries and functions the wild.
14,42
41,49
45.53
34.63
2.70
on
30.94
53.88
56.01
43.60
3.27
20:40
43.93
46.71
29.36
3.27
19.98
3.40
D Evaluation Metrics
16.97
44.71
47.11
35.54
2.74
3.77
18:47
44.98
43.52
27.11
5.78
We describe each of the non-execution metrics (S D.1) as introduced in $ 6, report model per- formance with each ($ D.2), and visualize their correlations with the execution accuracy ($ D.3).
1x2
27.10
$2.04
50.17
40.02
3.58
25.00
3.75
Table 13: CODEX results non-execution metrics.
D.1 Metric Deseription
Model
Metries
BLEU BLEU (Papineni et al., 2002) is lexical- based evaluation metric, which calculates the gram overlap between text prediction and (multi- ple) references. Most default calculation processes calculate up to 4-grams and adopt the smoothing function introduced in Lin and Och (2004).
BLEU
ROUGE
METEOR
Chrf
CodeBLEU
12.04
50.94
30.12
4:0
9.07
37.90
37.76
20.90
5.47
350M
44.21
41.29
26.16
6.05
mu
13.35
44.77
44.27
32.40
3.86
18.22
54.82
$4,32
34.98
$.30
2,78
13.05
29.79
40.93
23.61
6.67
14.73
$2.46
51,22
31.28
5.42
23.27
37.75
4.31
12.41
$2.82
$4.03
31.38
4.51
ROUGE ROUGE (Lin, 2004) is another more recall-oriented rexical-based evaluation metric. was originally designed for measuring text sum- marization, mainly by counting the number of
4.57
ja
19.14
51.31
52.07
34.78
5.68
23.66
49.09
49.48
37.44
3.72
Table 14: CODEGEN results on non-execution metrics.

---

D.3 Visualizing Metric Correlations
EN
Teil
Feil
Following the discussion in $ 6, we visualize the non-execution metric metrics between sam- ples that pass and fail during execution time. All experiments use CODE-DAVINCI-OUS predic- tions for evaluation. Figure 13, Figure 14, Fig- ure 15, Figure 16 illustrates the histogram between passed/failed samples using ROUGE, METEOR, ChrF, and Code metrics, respectively.
tall paic
X0
fall
placs
Figure 16: CodeBLEU on passed and failed samples.
-44
D.4 Why is Execution Better?
Home
To give more intuitive reasons for the advantages of execution, we randomly sample 15 cases from each language subset and identified two major ben- efits: tolerates alternative solutions and allows execution results as outputs.
Fuil
30
A Alternative Code Implementation Probably the greatest advantage of execution is it only requires correct execution results, without limitations on alternative methods, in Figure 17.
Figure 13: ROUGE on passed and failed samples.
F
*oyultiply each value by 2 for all keys dictionary mie
Canorical Solution
ay_dict updact.p
21 for x, in Vistloy itexs())
<2
Model Prediction key my_dicts Wi.cictkkyy
24
Tai pase
RD
Teil pwes
Figure 17: An alternative yet correct prediction, only has a low 4.8 BLEU score due to having little lexical overlap with the canonical solution.
Directly Generating Execution Results An- other interesting category is directly generating the code execution results instead of the implementa- tion steps. This often happens to simple coding queries such as basic string manipulation, where predicting the results might cost the model similar efforts to getting the programmatic solutions.
Figure 14: METEOR on passed and failed samples.
202
fal poss
1:0 625
fail pass
7227
"""Decode string '444bee to UTF-8. m0
# Canonical Solution Tytes.fronhex. deecocc" decode (
330
panee
poece
Model Prediction 'JKL"
375
Figure 18:
An example output of a correct execution result, yet only achieving 0.6 BLEU.
cm
In Figure 18, instead of the string decoding pro- gram, the model directly outputs the result string "ILK". While this is somewhat unexpected under
Figure 15: ChrF on passed and failed samples.

---

the NL-to-Code task, execution effectively handles such cases and would judge them correct.
E.1 Prompting Strategy
E.1.1 Few-shot Prompting
Table 15, Table 16, Table 17 show the change in execution accuracy with respect to the examples in in-context learning, on the three CODEX variants
D.5 Potential Benefit of Lexical-based Metries
Lexical-based metrics, although relatively ineffec- tive for functional correctness, still are potentially helpful for debugging and interpretation. They are effective in small errors of two types: (1) a single function misuse and (2) slight variance in complex strings. The high lexical match in such cases indi- cates less effort for fixing (Deng et al., 2021).
N-dhet
NL.
Pos Batk
611
122
03
9:4
U5
96
87
Q4
99
14ho
16.11
30.70
04:51
53.56
90.70
41.23
13.06
66-31
48 13
96
17.68
03.41
16.61
11:48
50
43 15
60:41
$7.90
61:11
K3.66
00.37
06.86
66 18
70.07
71.48
7131
34.17
75.40
24hot
18:50
13.36
11.57
08.77
46.71
58.17
02 T6
99
54.78
63.35
56.40
64.5
47.8
58.84
06.87
99.76
58.56
64.24
67.63
00.00
71.33
12.32
73.84
34,65
75.40
86.37
37.00
65.88
33.05
53.63
58
55.87
56.95
52.50
58.44
58.89
Function Misuse
Some code predictions are cor- rect except for a single place where a wrong func- tion is used, or an argument is misplaced.
32:87
62.48
00.03
47.58
65.32
50.88
53.29
70.71
55.16
56.65
73.66
$7.89
04.75
58.00
75.71
59.76
76.59
Table 15: CODE-CUSHMAN-OD few-shot results.
N-thtt
"""Match regex (M((+)))V with string "Zzyx)ee
602
8F44
36.6
677
21'
09:30
# Canonical Solution
51.67
51.30
03.01
55.12
02:55
66
05:71
Fwndall( (VMM (XL/)//W)))) '(zyx)bc")
15.00
08.57
51.17
67
64.15
66.34
17:30
50.07
04.57
67:83
71.35
33.07
11.41
74.55
75.44
76.19
# Model Prediction
44:36
51.08
58.77
64:00
65.90
re. natch (CC())))) '(zyx)bc')
24hed
35,02
64.82
49.87
53.85
55.25
55.77
57.35
58.66
59.27
50.76
4022
60.59
65.76
$6.96
71.16
72.75
14.03
75.04
75.87
76.59
Figure 19: Example that the model prediction uses the wrong function, having a very high BLEU score 0.925,
3-hod
35.76
00.40
$4.99
57.65
90.71
51.18
62.31
68.21
63.96
66.53
0033
03.58
06.71
09.08
7399
13.63
74.6
75.00
76.99
For example, in Figure 19, the code imports the library and copies all strings correctly. But it uses the wrong function match instead of the correct findall. Although the execution fails, the code is similar to the solution. Given the sign of a high BLEU score of 92.5, we could readily spot such similarities and fix them with simple edits.
Table 16: CODE-DAVINCI-001 few-shot results.
695
<0:00
06:31
50.10
00.51
(3.36
65.
65.47
66:34
(11.53 75
70.14
Th
Lahut
(1
54:11
05.17
63104
70.00
72.32
51.35
65.20
11.60
73.56
15.29
11.24
78.48
71.36
80.56
47:29
57.32
61.66
$4.69
66.53
67.85
68:90
69 74
70,45
71.07
63:11
58.53
72:12
$1.35
63.38
69.47
71.51
73.00
75.13
78.30
72.15
73.98
78.57
46.15
$7.50
02.64
$5.50
67.50
55.90
70.17
71.14
71.66
72.7
String Difference Another frequent error con-
86.10
55.60
54.74
6181
90
67.88
00.00
00.56
65.0
1964
77 19
7617
75.77
77 12
76.5
79.37
cerns string copying, where the code calls the cor- rect functions but copies the string differently.
Table 17: CODE-DAVINCI-OO few-shot results.
The example in Figure 20 gets 100.0 BLEU score, but the string inside actually misses a single whitespace, which the BLEU tokenization would discard. Such code also resembles the solution and could be easily fixed by even rule-based methods.
E.1.2 Number of Input Test Cases
Table 18 shows the effects execution accuracy of adding one more test cases to prompts. Ex- periments use RODE-DAVINCI-ON as an example.
tosl NL
Paa Rain
8.4
677
0:19
.*MInitia Lize "SECEET_KE" KEY" flask contig with
coom
11.20
33.12
# Canonical: Solution
47.44
57.993
62.20
56.35
05.31
52.51
58.65
09.53
10.35
71.31
app contigl [] "Your_secre_ssriin
41.46
51.87
30-42
54.86
05.25
11.09
$7.59
71113
59.47
34.5
51.57
15.67
62.71
76.64
T7.46
53.41
64/02
76.17
Hodel Prodiction
$2.70
$1.47
configl" SECIET Ker'] YYur puur_secret.strinng
63.80
15.37
81.75
$4.35
$5.90
$2.52
$8.65
90.23
19.67
90.00
63.00
34,05
75.85
61.13
$3.61
$3.00
$4.40
35.12
55.61
$5.08
55.04
$5.32
$3.50
10.00
Figure 20: Example that the model prediction varies
50
63.41
83.00
55.16
56.44
3665
slightly in copied strings, but scores 100.0 in BLEU,
(3.41
7400
79.00
$0.08
87.57
53.00
11.51
351
83.61
65.06
m1
66.87
79107
5m
56.19
07.53
39.01
19:51
9066
91
$1.67
Table 18: CODE-DAVINCI-002 results when using zero
E
Ablation Studies
(0), one (!), and all (n) test cases in the prompt input.
This section provides the results tables according to each ablation study section in $ 7.
Furthermore, we experiment on the subset of examples having sufficient test cases, to prevent the

---

model predictions. By default, we name func- n-test setting being trivialized into the 1-test case. Concretely, we filtered all examples with at least test cases and got 112, 17, 25, and 45 examples in
E.2.1
Number of Evaluation Test Cases
Riell NL
Pass Rats
64
67
Q10
English, Spanish, Japanese, and Russian. Pass @1 I results on 0/1/n-test settings are shown in Table 19.
14.03
48.00
59.52
62.31
56.38
05.51
62.89
58.63
09.53
10.35
71.31
42:44
51.95
56.31
99.16
01:54
02.80
63.85
6491
66-45
67.85
10.59
72.51
74.00
75.19
26.17
77.00
73.76
45:15
10.90
33.12
47.44
57.90
62.20
56.35
65.53
62.51
59.65
19.33
10.35
71.31
Language
41.46
51.87
9042
54.6
05.25
57.39
11.09
99.47
73303
M1.86
51.83
35,63
62.71
26.4
K3.41
77.46
64.02
76.37
52.7
47.1
52.0
64.4
63.4
70.6
64.0
66.7
Table 21: CODE-DAVINCI-OON results when using dif- ferent numbers of test cases for execution- based evalua- tion. / means using randomly selected test case, means using all annotated test cases in ODEX.
67.9
58.8
68.0
71.1
Table 19: Results on examples with or more test cases, using zero (0), one (/), and all (n) test cases.
We also evaluate on the subset of examples having at least 3 test cases. Table 22 shows the pass @ results for each language.
E.1.3 Pre-processing: Trailing Whitespaces While the input construction process may intro- duce whitespaces at the start and the end of the text sequence, we find CODEGEN model unexpect- edly sensitive to trailing whitespaces. As shown in Table 20, removing whitespaces from the prompt input increases the pass rate of all sized CODEGEN models by over 20 percent.
Language
ja
64.7
68.0
52.7
47.1
52.0
64.4
Table 22: Results examples with 3 or more test cases, using one (/) or all (n) test cases at evaluation.
Model
NL
m/ WS
w/o WS
E.3
Semantics of Function Names
@1
6:
G:5
6:10
@:1
65
10
10.32
12.53
42,82
Because code is wrapped into functions to enable exccution, how functions are named may affect
350M
17:56
7.01
17.78
8.06
17.78
9.55
10:37
17.78
16.67
17.44
22.86
28:21
27.82
30.49
30.00
00
21.35
24.20
26.94
28.17
25.87
31.44
37.44
40.87
20
14.28
15.69
16.99
17.54
57.74
42.58
47.36
49.89
2B
19.67
10.58
22.32
12:56
24.76
14,20
25.56
14.83
31,83
36.44
35.70
40.89
44
39.58
46.67
11.16
tions using the post ID (c.g.. f_3844801), which expresses little semantics of queries. So we try two other methods: (1) a constant string function; and (2) summary phrases from NL intents, e.g.,
33.10
41.67
54.54
57.54
11.96
14.81
37.91
41.18
644
14.78
80
30.00
56
37.78
16
12:44
14.31
16.51
17.68
35.55
40.11
41.12
16.34
03
32.86
34.45
36.28
37.30
44.64
47.29
49.82
51.39
padd_max_valll
Table 20: CODEGEN results when inputting prompts with and without trailing whitespaces (WS).
To do (2), we conduct heuristic phrase extrac- tion. We first cut the NL intent into words by whitespace, then remove the stop words ('in", 'of". 'a', 'to', 'and', 'for', 'with', 'that') and meaningless punctuations, lastly, concatenate the first M = 4 words with '_. For example, given an intent "de- code a hex string '4a4b4c' to UTF-8", the resulting function name would be "decode a hes__he___gg_h teex_string". However, for languages that do not separate words with whitespace, this approach may produce less meaningful strings, hence contributing to the infe- rior performance as shown below.
We conjecture the gain brought by whitespace stripping to be better distributional alignment with CODEGEN training data. As CODEGEN might be pre-trained on whitespace-sttipped text sequences, inputs without whitespaces are potentially more aligned with them, hence resulting in better test- time performance. Meanwhile, note that the tok- enization processes for text (natural language) and code (programming language) differ whitespace- style tokens such as \n or \t. These tokens would be removed by text tokenizers by default, while pre- served by code tokenizers since they imply struc- tural information in code pieces.
To fairly compare with previous results, we do not add test cases in prompts.
From Figure 21 and Table 23, using more mantically meaningful functional names barely im- proves over the default setting. Intuitively, summa- rizing names from intents adds no extra semantics, but may cost information loss at the curation step. both contributing to the performance drop.
E,2 Number of Evaluation Test Cases
Table 21 shows the effect when using different num- bers of test cases for execution- -accetioo-bbsedd evaluation.

---

80
a//d
ally evaluate the SantaCoder (Allal al., 2023) and StarCoder (Li et al., 2023) models, which have not been trained on any SO data. Table 24 shows the pass of 16B StarCoder and StarCoder Base models, where both models show significant gaps between open- and closed-domad queries, thanks to the broad domain coverage of ODEX.
Fooms
Intert
hddd
pritee
no/ conet
G Related Work
35
#68
P85
stant
Open Domain Code Generation Code written in general-purpose programming languages often uses classes functions from external libraries. A few datasets for code generation preserve this open-domain nature. The CONCODE (Iyer et al., 2018) dataset tested generation of Java class meth- ods. Later works target Python generation given the interactive context of Jupyter Notebooks (Agashe et al., 2019) or natural language intents from Stack- Overflow posts (Yin et al., 2018; Wang et al., 2023). Despite their natural coverage, enabling open-domain code execution has faced great chal- lenges given its diversity and complexity (Lai al., 2022; Chandel et al., 2022). To address this issue, our ODEX provides test cases as code execution contexts for evaluation.
Figure 21: pass@) / using different function names.
Pare Namse N1.
Pan Rott
60:2
-33
#4
6:6
-26
81
68
10
tel-id
47.64
41-46
37:90
90143
$4.8
1030
6465
06.33
29-47
67.61
01.81
18.65
06.53
00:71
20.33
61:41
2410
71.11
47.04
02.71
94
19:54
61.36
63.90
66.83
65.00
66.29
C8
31.33
622.33
09.33
$3.76
04
60.54
67:00
03.00
18
48,06
69..2
53.75
59/25
71.78
23.77
13:41
16.93
7258
1930
33.05
-9.21
53255
06.11
62.5
65.21
06:35
51.99
07.33
40
5509
46.11
03
59.05
59.95
0115
01:0
Table 23: CODE-DAVINCI-002 results when the wrap-
ping function name contains different semantics.
F ODEX Results on Additional Models
It is possible that some source StackOverflow (SO) posts used to create ODEX examples were used in the training data of the closed-source models in our experiments. However, we have no way to remove those overlapping examples due to the lack of a detailed web index within the training data of these models. On the one hand, we modified the NL intents and code solutions to some extent $2.2, which may alleviate exact matches scraped training data and, hence reduce the influence of unqualified training data (Lai et al., 2022).
Code Evaluation via Execution Execution- based evaluation has been long adopted for domain-specific programming languages such as SQL queries (Zhong et al., 2017) or logical forms (Dong and Lapata, 2016). This execution- based paradigm has not been introduced to general- purpose languages until recently by the HumanEval dataset (Chen et al., 2021), where human-written test cases are provided for code execution. Many works afterward follow this approach, but focus more on
settings (Austin et al., 2021; Hendrycks et al., 2021) or specific libraries of interest (Lai et al., 2022; Huang et al., 2022). To- ward broader execution environments, we provide executable test cases for as many as 79 libraries.
Language Domain
SantaCoder
StarCaderBaas
StarCoder
all
37.65
46.51
44.67
open
30.87
40.65
37.00
closed
45.12
52.97
53.11
32.11
30.11
37.56
es open
26.04
25.42
32.92
closed
39.05
35.48
all
28.11
41.22
44.11
Coding Queries Versus Programming Chal- lenges Programs from different sources are or- ganized for various purposes. Coding contest web- sites such as LeetCode" and Codeforces 12 have been used to build many code generation bench- marks (Hendrycks et al., 2021; Li et al., 2022). However, they randomly align with how humans program in practical scenarios. To build datasets
Ja
open
23.01
37.61
39.56
closed
39.41
49.22
54.51
all 36.87
46.11
50.40
open
22.98
34.04
33.77
closed
48.33
56.09
64.13
Table 24: STARCODER pass@1 results on ODEX, eval- uated on all (all), open-domain (open). and closed- domain (closed) examples.
To further address the train-test data overlap is- sue in Codex and CodeGen models, we addition-
hntps://tetccodd.comm #hupss//coaeeforcs.com/

---

with natural and practical usage of code, many works use GitHub Jupyter Notebooks (Agashe et al., 2019; Huang et al., 2022) and StackOverflow forums (Yin et al., 2018; Wang et al., 2023; Lai et al., 2022) as a source of naturally-occurring code. We remain such naturalness by using StackOver- flow posts, but uniquely from forums in various languages to also assist programmers worldwide.
Test Case Creation While benchmarks use Python test cases annotated by human program- mers (Chen et al., 2021; Nijkamp et al., 2023; Lai et al., 2022), challenge-style datasets adopt a more direct approach by crawling from the web (Hendrycks et al., 2021; Li et al., 2022). An- other thread of work attempts to generate test cases automatically based on the Python gram- mar (Lukasczyk and Fraser, 2022), but is largely limited to basic Python functions. Some propose to leverage the power of neural LMs (Tufano et al., 2021; Li et al., 2022), even jointly considering so- lution and test case generation (Chen et al., 2023). However, the quality and diversity of test cases are not robustly ensured. We hence use high-quality Human-written test cases for ODEX evaluation.


Reviews:
[{"review_id": "yzarTyV1dS", "reviewer": "Reviewer_FbaY", "summary": "", "questions": "Did you apply filtering as the first step to the original CoNaLa and (M)CoNaLa datasets? If so, how did you filter out the dataset? These two datasets have thousands of data, why are there only 945 pairs left?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and your positive comments about the value of our dataset, analysis, and experimentation! \n\n### R. Are Import Statements Included\nNo. As illustrated in Figure 6 and Line 304-315, we do not include any import statements in the prompts, to prevent disclosing any library hints to models. But to avoid penalizing the models if they don’t generate the imports themselves, we add the necessary imports to the evaluation code. The description (in line 134) may cause confusion as annotators were asked to specify required library imports, we will further clarify this detail in experimental sections.\n\n### Q. Any Filtering on (M)CoNaLa?\nNo, we do not filter out any examples in (M)CoNaLa prior to annotations. Nonetheless, some examples in (M)CoNaLa are not included in ODEX for reasons such as imported libraries or functions being deprecated. \n\n### Missing Reference\nThanks for the notice! We cited this paper in related works (line 482). We will mention its name “CodeContest dataset” to indicate its relevance more explicitly.\n\n### Reproducibility\nWe will make all code and necessary parameters available to the public (currently they can be accessed in footnote 1.). We believe that our work is highly reproducible (and some other groups have already used it for evaluation)!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Execution-based evaluation is known for being more aligned with human preference than n-gram based evaluation. Therefore, this dataset with human-written test cases is a great contribution to the community. \n\n* The authors have analyzed the domain distribution by libraries and data complexity comprehensively. The benchmark experiments are also helpful and could shed light on the evaluation of code generation. \n\n* The process steps of creating this dataset are reasonably designed. They also have the quality check during the annotation steps.\n", "reasons_to_reject": "It is not very clear from the paper whether the prompt has included the essential import statements. As this dataset is heavily using libraries, it is very important to avoid such ambiguity. \n", "questions_to_authors": "Did you apply filtering as the first step to the original CoNaLa and (M)CoNaLa datasets? If so, how did you filter out the dataset? These two datasets have thousands of data, why are there only 945 pairs left?\n", "paper_topic_and_main_contributions": "This paper proposes a Python code generation dataset named ODEX for execution-based evaluation. This dataset contains 945 NL-Code pairs in 4 languages,  including 439 samples in English, 202 90 in Spanish, 164 in Japanese, and 252 in Russian, with 1707 human written test cases for execution. In this paper, the authors introduce the annotation steps and quality check for creating such a dataset and they conduct detailed analysis of this dataset from various aspects. They also evaluate the performance of existing LLMs for code generation on ODEX. This is a high-quality dataset and it can benefit the researchers who are interested in this domain.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "There is also a highly relevant dataset called CodeContest but the authors didn't mention it. It is from the paper 'Competition-Level Code Generation with AlphaCode'\n"}, "extra_text": {}}, {"review_id": "GfSqeCwHXp", "reviewer": "Reviewer_rKvX", "summary": "", "questions": "A.\tInsufficient analysis of test cases written (the main addition in when creating ODEX). Are these test cases diverse/sufficient/coverage/non-repetitive /etc?\n\nB.\tShouldn’t open-domain code gen models also generate correct library imports? The current setup (line 134 – step2) specifies library prerequisites manually, and the prompt for Codex/Codegen already contains the correct library calls. I’m not sure if this is truly open-domain then.\n\nC.\tError analysis of the code generated by Codex and Codegen in open vs closed domains might be helpful. Do the types of errors change, etc?\n\nD.\tHow are in-context examples chosen? Randomly / similarity match, etc? Details could be included in the appendix (in-context example selection, prompt design, etc)\n\nE.\tLine 071 – augmented training data improves execution accuracy – where are these results?\n\nF.\tAny instances where the canonical code solution was wrong and didn’t pass test cases? Were these excluded?\n\nG.\tFigure 6 has your zero shot prompt, where is the library call? Does the model have to generate this or is this included by not shown in the header?\n\nH.\tNucleus sampling is used. Line 356 compares pass@1 performance between Codex and Codegen. Due to randomness in the generated sample, these results could change. Was any consistency performed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your positive comments about the quality and diversity of our dataset!\n\n### R1. Sufficiency of Test Cases\nEvaluating the main functionality of code is a critical first step for open-domain code evaluation. It has already taken us substantial efforts (~ 6 months). \n\nODEX is challenging and largely effective in judging erroneous predictions. As shown in Table 4, the score of the strongest model is somewhat low\nWe acknowledge that test coverage can be a potential issue when the test cases in our dataset are no longer effective. Enhancing the test case sufficiency is an important next step to further improve the comprehensiveness of benchmarks. In the future, we could leverage similar methodologies that are used to create HumanEval+ [1].\n\n### R1. Biased Tests with Canonical Solutions\nAnnotating test cases when knowing certain solutions may bias the test cases. However, our test cases basically consist of input-output data, which should be identical regardless of the specific implementation of the solution. Therefore, knowing the canonical solution should only minorly affect the test cases.\nAlso, in practice, it is sometimes impossible to annotate tests without solutions, due to the inherent ambiguity of natural languages. Example solutions can provide the necessary specifications. For example, an NL query “sum the first column” is confusing for indexing, but the code “a.sum(axis=0)” points to the literal first column and not `index=1`. An alternative to encourage test diversity would be to simultaneously write test cases and solutions, which we would consider if not due to time and cost constraints.\n\n### R2. Sufficiency of Test Cases\nFirst, we want to clarify that Figure 11 (right) shows results when adding test cases to the *docstring* of the current test example other than to in-context examples (which should be Figure 11 left). Because we do not include any test cases in (the inputs) of in-context examples, the results cannot directly support the sufficiency of test cases.\nThank you for pointing out another dimension of sufficiency! We filtered all examples with >=3 test cases (112, 17, 25, and 45 in English, Spanish, Japanese, and Russian). \nIn the first table, we compare the Codex davinci-002 performance before and after (i) adding in-docstring test cases, and (ii) altering the number of evaluation test cases. \n\n| (i) Number of In-Docstring Test Cases | EN | ES | JA | RU |\n|:---------------------------------:|------|------|------|------|\n| 0 | 52.7 | 47.1 | 52.0 | 64.4 |\n| 1 | 63.4 | 70.6 | 64.0 | 66.7 |\n| n | 67.9 | 58.8 | 68.0 | 71.1 |\n\nAdding test cases into inputs significantly improves model performance. These results suggest that ODEX test cases cover the main functionality of code queries, adding more test cases provides richer functional guidance, hence improving the quality of the generated code.  \n\n| (ii) Number of Evaluation Test Cases | EN   | ES   | JA   | RU   |\n|---------------------------------|------|------|------|------|\n| 1                               | 60.7 | 64.7 | 68.0 | 73.3 |\n| n                               | 52.7 | 47.1 | 52.0 | 64.4 |\n\nEvaluating model predictions with more test cases effectively filters out a huge number of (previously) false positive predictions. This demonstrates the sufficiency of test cases for multi-test examples. We will add these results to the revised paper version, to reflect the valuable discussion on test case sufficiency.\n\n\n### R3. Representation of Rare Libraries\nWe have fewer examples for less frequent libraries (Table 5), and increasing the overall number of examples could make the benchmark even more robust. However, to preserve the natural distribution of open-domain libraries, we chose to not intentionally create/collect more examples for minority libraries, instead opting to aim for a dataset that is most similar to the distribution of real-world scenarios (Section 3.1). We did our best to collect the largest number of examples from verified StackOverflow sources so far, i.e., (M)CoNaLa, and hope future works could continue to leverage more resources showing up.\nWe acknowledge having more examples of rare libraries will enable more robust model evaluations on those libraries. We will discuss this in the limitation section of the paper. \n\n### Q.A. Analysis of Test Cases\nWe encourage annotators to write diverse test cases. To demonstrate this effort, we further analyze the similarity of input arguments in our test cases. For each example with at least two test cases, we take every test pair and calculate the lexical overlap between their argument values using BLEU and Levenshtein edit distance (item-wise, that ‘1’ can be replaced with ‘happy’ in one step). These test pairs score only 13.3 in BLEU, showing a small lexical overlap. They have an average edit distance of 11.3, indicating a broad coverage among test input values.\n\n### Q.B. Are Import Statements Included\nAs illustrated in Figure 6 and described in Section 4 (the Prompt Design paragraph), we do not include any import statements in the prompts, to prevent disclosing any library hints to models. But to avoid penalizing the models if they don’t generate the imports themselves, we add the necessary imports to the evaluation code. The description (in line 134) may cause confusion as annotators were asked to specify required library imports, we will further clarify this detail in experimental sections.\n\n### Q.C. Error Analysis\nWe performed several error analyses: (1) in Appendix C.3, we analyzed typical error types in the open- and closed domains; (2) in Appendix D.5, we spotted interesting error categories by using different evaluation metrics. In summary, error types differ between models and domains. We will move these to the main paper in the camera-ready version, when more space is allowed.\n\n### Q.D. Selecting In-Context Examples\nWe randomly select examples as in-context examples, because this is simple to implement and requires minimum assumptions of the evaluation data. We will clarify this in the revised paper version.\n\n### Q.E. Augmenting Training Data Helps\nBy “augmented training data” we meant the increased training data for models with larger sizes or in upgraded versions, e.g., the 175B Codex davinci-002 compared to the 12B cushman-001. \nLarger and upgraded models achieve better performance, throughout almost all experiments done in this work. We will refine relevant descriptions and clarify this in the revised paper.\n\n### Q.F. Solution Cannot Pass Tests\nSolutions of all examples can pass the test cases (line 200). This is guaranteed during the annotation stage, while cases to which no annotators can provide a correct test case are filtered out. Some solutions have observable errors in (M)CoNaLa. We asked annotators to fix the solutions if the NL query could be correctly answered and viable test cases could be created accordingly.\n\n### Q.G. Library Call in Zero-Shot Prompt\nZero-shot prompt means we include zero information about other examples, not the current example for testing. Regardless of the number of shots, we always import necessary libraries/functions, as described in our response to Q.B (question B) above.\n\n### Q.H. Consistency Under Nucleus Sampling\nWe ran 5 rounds of experiments on the English subset using CodeGen, under the same setting as the experiment reported in Table 5. We list the results of all experiments in the table below. \n\n| Runs \\ pass@k | 1 | 2 | 5 | 10 |\n|---------------|---|---|---|----|\n| 0 | 26.26 | 32.18 | 39.10 | 42.82 |\n| 1 | 25.74 | 31.66 | 38.50 | 42.14 |\n| 2 | 26.65 | 32.57 | 39.52 | 43.33 |\n| 3 | 26.10 | 31.94 | 38.72 | 42.21 |\n| 4 | 26.65 | 32.76 | 39.86 | 43.21 |\n| Mean | 26.28 | 32.22 | 39.14 | 42.74 |\n| Std.Dev. | 0.387 | 0.449 | 0.559 | 0.551 |\n\nThe average results of all language subsets are close to the reported result, and the standard deviation values are small. Model results are pretty stable under our experiment configuration, therefore the results reported in Table 5 are reliable. We will add results for other languages and models in the revised paper version.\n\n\n[1] Liu, Jiawei, et al. \"Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.\" arXiv preprint arXiv:2305.01210 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe main contribution is human-written test cases to test execution-based evaluation.\n2.\tODEX tests “open-domain” code generation, from natural and practical queries, requiring code to import and use diverse libraries. The diversity in libraries included in ODEX compared to other datasets is suitably highlighted in the paper. \n3.\tThe dataset collection and annotation procedure are described in detail. Ideas to address challenges (irreproducible runs, randomized outputs, specialized equivalence checks) in test case creation for certain library calls are insightful.\n4.\tInteresting result comparing the gap between Codex and Codegen for open vs closed domain as the model size increases.", "reasons_to_reject": "1.\tODEX is essentially a subset of the existing (M)CoNaLa datasets with the addition of human-written test cases. My main concern is the effectiveness of these test cases for execution-based evaluation to check code correctness reliably. With suitability, I refer to the test cases as being necessary and sufficient. \na.\tSufficient test cases would aim for 100% code coverage and handling edge cases, among other things. Since this might be hard to achieve through external annotation, a proxy could be to encourage diversity in test cases written. \nb.\tThe authors show both the query and canonical code to the annotator to write test cases, this might bias the annotator to write test cases wrt the canonical code solution only. Since the annotators are undergrad students and not experts, the annotators could have been shown only the query, (optionally asked to write M different correct solutions, N different incorrect solutions possibly excluding certain corner cases), and test cases (along with the intention of what is being checked behind each test case). The resulting test cases could be less biased wrt the canonical code solution.\nc.\tThe authors test all test cases are correct in 2.4, but this does not guarantee sufficiency/diversity/suitability/non-repetitive.\nd.\tInsufficient analysis of test cases written (the main addition in when creating ODEX). Are these test cases diverse/sufficient/coverage /non-repetitive / etc?\n2.\tThe authors show two experiments in Figure 11 right and Figure 12, the first increasing the number of test cases as in-context examples didn’t improve performance, and the second stating that the evaluation metrics didn’t change including additional test cases. These results could show the sufficiency of test cases in the dataset, conversely, the results could also support the insufficiency of test cases in the dataset, i.e., adding the remaining test cases didn’t change performance/evaluation. Further, since the average no of test cases is 1.8, isn’t using 1 vs all test cases, essentially 1 vs 1.8 test cases, which might not be suitable for these experiments. Maybe testing on a subset where the number of all test cases >= 3 or 5 might help.\n3.\tThe number of samples for half(?) of the libraries is 1 or 2 (table 5 of Appendix A). This might not be representative to evaluate code gen performance on that library.\n\nI’m open to increasing my score if the above comments are addressed.", "questions_to_authors": "A.\tInsufficient analysis of test cases written (the main addition in when creating ODEX). Are these test cases diverse/sufficient/coverage/non-repetitive /etc?\n\nB.\tShouldn’t open-domain code gen models also generate correct library imports? The current setup (line 134 – step2) specifies library prerequisites manually, and the prompt for Codex/Codegen already contains the correct library calls. I’m not sure if this is truly open-domain then.\n\nC.\tError analysis of the code generated by Codex and Codegen in open vs closed domains might be helpful. Do the types of errors change, etc?\n\nD.\tHow are in-context examples chosen? Randomly / similarity match, etc? Details could be included in the appendix (in-context example selection, prompt design, etc)\n\nE.\tLine 071 – augmented training data improves execution accuracy – where are these results?\n\nF.\tAny instances where the canonical code solution was wrong and didn’t pass test cases? Were these excluded?\n\nG.\tFigure 6 has your zero shot prompt, where is the library call? Does the model have to generate this or is this included by not shown in the header?\n\nH.\tNucleus sampling is used. Line 356 compares pass@1 performance between Codex and Codegen. Due to randomness in the generated sample, these results could change. Was any consistency performed?", "paper_topic_and_main_contributions": "This paper presents ODEX, a natural language to Python code dataset, supporting execution-based evaluation of code generation models. The salient features are 1) the dataset being “open-domain” where code could import and use one of 79 diverse libraries, 2) 1707 human-written test cases to support execution-based evaluation, and 3) multilingual queries.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EB3BpNv8sI", "reviewer": "Reviewer_d6KA", "summary": "", "questions": "A. There is not much information about the HumanEval dataset. Is this similar to the OpenAI dataset? What is the difference between the two?\n\nB. The authors mention that they have undergraduate students to write test cases. My concern is that the undergraduate students go to stackoverflow for help writing their test cases and then the data has just come full circle. The authors do not mention much about how the students write the test cases. How do they make sure to prevent this?\n\nC. If these are questions from the internet, the model might already be trained on this data. How can we be sure that these are not in the training set? For instance, the authors mention that they use training resources from stackoverflow. But they then go on to test CODEX, which is essentially Co-Pilot, which is known to have been trained on stackoverflow data. How can you avoid models being tested on data that was already in the training set?\n\nD. In Figure 7, why is Russian so much better than the other languages? It is not even the language with the highest resources in the dataset. On a similar note, why is Japanese so performant for CODEX?\n\nE. Are the problems the authors picked in different languages equivalent? Or do they pick easier tasks for more difficult languages?\n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for recognizing the quality of ODEX! \n\n### R1. Novelty of ODEX\nThe novelty of ODEX is providing a code evaluation benchmark with *diverse domain coverage*, as we illustrate in Section 3 in comparison with other datasets (Table 3). \nODEX serves as a unique portal to reveal the significant performance gap between open- and closed-domain queries, even with current top-performing Codex and CodeGen models (Section 5). In contrast, the HumanEval dataset only shows that Codex and CodeGen achieve rather comparable performance [1]. \nOur work points out and mitigates the current shortcomings of models and evaluation benchmarks (in overlooking more realistic open-domain queries).\n\nR2:  It's difficult to fully understand why current solutions aren't enough, or what problem they are trying to solve.\nWe kindly note that in this work, we focus on the creation of a new *dataset* other than a *solution*\n\n### R2. Small Size of ODEX\nAs discussed in L542-L551, significance testing [1] can be performed to address the reliability of model performance (differences). We took the best Codex (davinci-002) and CodeGen (6.1B) models and performed significance tests with the pass@1 predictions using a sample rate of 0.5 and a sample size of 10, 000, and it shows win rates of 1.00 for Codex on all language subsets, with p-value << 0.005. We will perform more comprehensive tests and highlight these results in the revised version.\nWe also kindly highlight that our ODEX is among the largest execution-based code evaluation benchmarks, given the difficulty of annotating (i.e., hand-writing) evaluation test cases for code. For instance, HumanEval only has 164 examples while ODEX has 945 test cases. \n \n\n### R3. Testing ODEX on HumanEval Dataset\nWe would like to note that ODEX is a code evaluation dataset, the same as the HumanEval dataset. ODEX is not a proposed method for code generation, therefore cannot be evaluated on datasets. We can, though, test code generation models on ODEX, which we experimented in details in Section 5, 6, 7, and Appendices.\n\n### Q.A. The “HumanEval Dataset” vs. the “OpenAI dataset”\nWe have introduced the HumanEval dataset (line 226), illustrated its domain breakdown (Figure 4, Appendix A.1), showed its construction details (Table 3), and summarized model performance on it (line 303, 354). We believe these would provide sufficient background about one of the baseline datasets for our work.\nAs far as we know, there are no datasets named the “OpenAI dataset”. You might be referring to the “HumanEval” dataset, which is released by the organization named “OpenAI”. If so, the “HumanEval dataset” and “OpenAI dataset” point to the same dataset.\n\n### Q.B. Seeking StackOverflow to Write Test Cases\nIt is not problematic to write test cases by referring to StackOverflow posts, as long as the tests can correctly evaluate the functionality of canonical solutions (Appendix A.2). \nWe performed rounds of screening and annotation tests and hired all qualified annotators, who demonstrated the ability to write multiple valid test cases for all annotatable examples (section 2.3). We believe there are no annotation quality issues directly caused by their undergrad status, and in fact in preliminary experiments we found that the quality of annotations created by these undergrads exceeded that of alleged “professional” freelancers that we found on Upwork. \nTo further ensure the test quality, we (authors – graduate students and professors) performed post-hoc examinations and executed the test cases to ensure their correctness (line 195). \n\n### Q.C. Possible Use of ODEX Data in Model Training\nIt is possible that some source StackOverflow posts used to create ODEX examples are used in training by Codex and/or CodeGen. We would remove those overlapping examples if Codex/CodeGen could provide a detailed web index within their training data, but unfortunately, they do not. Nonetheless, we modify the NL intents and code solutions to some extent (described in Section 2.2), which may alleviate exact matches to scraped training data and, hence can reduce the influence of unqualified training data [3].\n\nTo further address the train-test data overlap issue, we also evaluate the StarCoder [4] models, which we know have not been trained on any StackOverflow data. The table below shows the pass@1 of 16B StarCoder and StarCoderBase models, where both models show significant gaps between open- and closed-domain queries, thanks to the broad domain coverage of ODEX. We couldn’t report these results at paper submission time since StarCoder was released concurrently. We will add them to the main paper in future revised versions.\n\n| Model         | EN   | - open | - close | ES   | - open | - close | JA   | - open | - close | RU   | - open | - close |\n|---------------|------|--------|---------|------|--------|---------|------|--------|---------|------|--------|---------|\n\n| StarCoderBase | 46.5 | 40.7   | 53.0    | 30.1 | 25.4   | 35.5    | 41.2 | 37.6   | 49.2    | 46.1 | 34.0   | 56.1    |\n| StarCoder     | 44.7 | 37.0   | 53.1    | 37.6 | 32.9   | 42.9    | 44.2 | 39.6   | 54.5    | 50.4 | 33.8   | 64.1    |\n\n\n### Q.D. Japanese & Russian Performance\nPerformance on different language subsets is largely due to: (1) if models are well-train on data of target languages, and (2) the difficulty of included test examples (Section 5.1). We conjecture that good performance comes from both, although we are not as certain as we would be if information about the training data was public. Similarly, Codex could be well-trained on Japanese data hence performing well at testing time.\n\n### Q.E. Picking Examples for Different Languages\nWe do not intentionally “pick” any examples for (sub-)datasets in different languages. Examples are naturally different between language subsets, because we collect queries from respective StackOverflow forums fulfilled by developers native in different languages. Similarly to the findings of (M)CoNaLa, queries asked in some languages (e.g., Spanish) are harder than others (e.g., Russian), presumably due to differences in the culture of Stack Overflow users in the different communities.\n\n### Typos\nThank you for the detailed check! We will revise these typos in the final version.\n\n\n[1] Nijkamp, Erik, et al. \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022).\n[2] Dror, Rotem, et al. \"The hitchhiker’s guide to testing statistical significance in natural language processing.\" Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers). 2018.\n[3] Lai, Yuhang, et al. \"DS-1000: A natural and reliable benchmark for data science code generation.\" International Conference on Machine Learning. PMLR, 2023.\n[4] Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv preprint arXiv:2305.06161 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset could potentially have better data quality than existing datasets.\n\n", "reasons_to_reject": "1. The novelty of the paper is unclear. The authors claim to use a 'more diverse' open domain library, but it's not clear what this result is compared against. It's difficult to fully understand why current solutions aren't enough, or what problem they are trying to solve.\n\n2. The small size of the dataset (only 945 code pairs, further divided by language into about 400 for English and only 90 for Spanish) leads to doubts about the credibility of the results. Although the authors address this in the Limitations Section, it does not alleviate these concerns.\n\n3. This paper does not provide any table or graph showing the test results of ODEX with the 'Human Eval dataset', and it lacks an explanation of how and why ODEX is effective and different from other solutions.\n\n", "questions_to_authors": "A. There is not much information about the HumanEval dataset. Is this similar to the OpenAI dataset? What is the difference between the two?\n\nB. The authors mention that they have undergraduate students to write test cases. My concern is that the undergraduate students go to stackoverflow for help writing their test cases and then the data has just come full circle. The authors do not mention much about how the students write the test cases. How do they make sure to prevent this?\n\nC. If these are questions from the internet, the model might already be trained on this data. How can we be sure that these are not in the training set? For instance, the authors mention that they use training resources from stackoverflow. But they then go on to test CODEX, which is essentially Co-Pilot, which is known to have been trained on stackoverflow data. How can you avoid models being tested on data that was already in the training set?\n\nD. In Figure 7, why is Russian so much better than the other languages? It is not even the language with the highest resources in the dataset. On a similar note, why is Japanese so performant for CODEX?\n\nE. Are the problems the authors picked in different languages equivalent? Or do they pick easier tasks for more difficult languages?\n\n", "paper_topic_and_main_contributions": "The authors’ main contribution is proposing ODEX, a dataset comprising 945 NL-Code pairs, composed of 4 languages: English, Spanish, Japanese, and Russian. The dataset includes 79 open-domain programs and addresses three of the challenges associated with open-domain code execution: irreproducible runs, randomized outputs, and specialized equivalence checks. \n\nPrevious solutions are either closed domain (they do not take into account third party libraries), or they cover a very small amount of open domain libraries. In contrast, the authors state that ODEX is more “colorful” because it covers significantly more open domain libraries, in addition to those in the closed domain.\n\nODEX is tested on both CODEX and CODEGEN.\n\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 11-12: “…as intents, in English, Spanish, Japanese, and Russian.” - Comma splice, you don’t need it after ‘intents’. \nLines 73-74: “…we observe satisfactory multilingual capabilities, despite that neither model was…” - Grammatically incorrect, should be something like: “…we observe satisfactory multilingual capabilities, despite the fact that neither model was…”.\nLine 76: “…face greater yet varied challenges…” - Missing commas, should be: “…face greater, yet varied, challenges…”.\nLine 114: “…further proofs and clarifies…” - I’m not sure if this is completely grammatically wrong, but I would use ‘proves’ instead of ‘proofs’.\nLines 239-240: “DS-1000 restricts to code using libraries…” - Grammatically incorrect, should be: “DS-1000 is restricted exclusively to code using libraries…”.\nLine 269: “…of SO queries…” - Inconsistency, SO has not been previously defined. What does it mean?\nLines 298-301: “Models are progressively trained on ThePile, BigQuery, and BigPythong datasets are denoted as NL, MULTI, and MONO.” - The first clause of this sentence is not connected to the second clause, makes for an illogical sentence.\nLines 367-368: “Model upgrades do not always reduce the gaps.” - A bit of an awkward, abrupt sentence mid-paragraph.\nLine 374: “It is hence imperative…” - Incorrect usage of the word ‘hence’. \nLines 435-436: “We thus explore to few-shot prefixing…” - Grammatically incorrect, you don’t explore ‘to’.\nLines 454-455: “This potentially implies the sufficiency of one test case to show the main functionality.” - Not grammatically incorrect, just awkwardly and a bit confusingly worded.\nLine 469: “Check Appendix E for…” - Unprofessionally worded.\nLine 484: “…StackOverflow (SO)…” - This acronym was referenced earlier, so it should have been defined earlier.\nLine 487: “…naturalness…” - Not technically wrong, just not a frequently used word in English due to the way it sounds (a bit juvenile).\nLine 521: “…we should should…” - Typo, repeated word.\nLine 524: “…execution supports in ODEX…” - unnecessary plural, should be: “support”.\nLines 529-530: “We should always keep alert to concealing malicious…” - Grammatically incorrect and awkward to read. Perhaps try: “We should always remain vigilant about concealing malicious…”.\nLine 562-568: Wrong tense used throughout this portion of the paragraph, would make more sense in the past tense.\n\n"}}]

Author rebuttal:

Meta-review: The paper introduces ODEX, a pioneering Open-Domain EXecution-based dataset designed for natural language to Python code generation. ODEX features 945 NL-Code pairs that span 79 diverse libraries and are supported in four languages: English, Spanish, Japanese, and Russian. These pairs are sourced from StackOverflow, ensuring relevance to real-world coding queries. Additionally, the dataset includes 1,707 human-written test cases to facilitate execution-based evaluation. A notable observation from the study is the differing behaviors of leading code language models. While CODEX outperforms in general, CODEGEN shows improvement with scaling. The paper highlights the performance gaps of these models in open and closed domains. Reviewers commend ODEX for its "open-domain" nature, execution-based evaluation support, and multilingual capability. They recognize its potential to address challenges in open-domain code execution and appreciate its broader coverage of libraries compared to existing solutions. The dataset is expected to significantly benefit the code generation research community.

The paper presents a significant contribution to the community by introducing a dataset with human-written test cases, emphasizing execution-based evaluation, which aligns more closely with human preferences than n-gram based evaluations. The meticulous process of dataset creation, including a thorough quality check during annotation, is commendable. The dataset's strength lies in its "open-domain" nature, derived from genuine and practical queries, necessitating diverse library imports and usage. This diversity is an important feature, setting ODEX apart from other datasets. The paper provides a detailed account of the dataset collection and annotation procedure, offering innovative solutions to challenges like irreproducible runs, randomized outputs, and specialized equivalence checks. Benchmark experiments offer valuable insights, especially the intriguing result comparing the performance gap between Codex and Codegen in open versus closed domains as model size grows. Overall, the dataset's potential for superior data quality compared to existing datasets makes it a worthy addition to the field.

The authors must consider the following weaknesses: a) The process of showing both the query and canonical code to annotators when writing test cases could introduce bias, potentially limiting the test cases to the canonical code solution. b) The paper's experiments do not conclusively demonstrate the sufficiency of the test cases. c) The dataset's small size, especially when divided by language, raises concerns about the robustness of the results.
Decision: Accept-Findings