XCODEEVAL: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024)

Abstract: Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level, and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap with a reference code rather than actual execution. We introduce XCODEEVAL, the largest executable multilingual multitask benchmark to date consisting of 25M document-level coding examples (16.5B tokens) from about 7.5K unique problems covering up to 11 programming languages with execution-level parallelism. It features a total of 7 tasks involving code understanding, generation, translation and retrieval. XCODEEVAL adopts an execution-based evaluation and offers a multilingual code execution engine, ExecEval that supports unit test based execution in all the 11 languages. To address the challenge of balancing the distributions of text-code samples over multiple attributes in validation/test sets, we propose a novel data splitting and a data selection schema based on the geometric mean and graph-theoretic principle. Our experiments with OpenAI’s LLMs (zero-shot) and open-LLMs (zero-shot and fine-tuned) on the tasks and languages demonstrate XCODEEVAL to be quite challenging as per the current advancements in language models.

heading: 1 Introduction
text: Automatically generating computer programs to solve complex problems has been a long-standing goal in AI (Manna and Waldinger, 1971). In recent
∗Equal Contribution
years, specifically with the growth of large language models (LLMs), we have witnessed tremendous progress in synthesizing code that is not just relevant but also fully functional with no further human modification needed (Chen et al., 2021). The progress made in related tasks such as program synthesis (Chowdhery et al., 2022; Li et al., 2022), program repair (Berabi et al., 2021), code translation (Roziere et al., 2020, 2021), and code retrieval (Wan et al., 2019; Parvez et al., 2021) are having a profound impact on increasing developer productivity (Ziegler et al., 2022; Tan et al., 2023) and aiding educators (Finnie-Ansley et al., 2022).
Despite the fact that advancements are expected to be general with benchmarks, their evaluation has been performed in a scattered way on a limited number of languages like Python and Java, on a partial granularity level such as at the level of a statement (Huang et al., 2022) or function (Husain et al., 2019), focusing on only one or two specific tasks like program synthesis and translation, and in many cases without fine-tuning data (Austin et al., 2021) or in terms of lexical n-gram based relevance (Iyer et al., 2018) rather than execution. We present a summary of the characteristics of existing program evaluation test-beds in Tables 1 and 6.
To address these limitations, and drive further advancements in the creation of more general-purpose LLMs for problem solving, we introduce XCODEEVAL, the largest executable multilingual multitask benchmark to date consisting of 20M coding examples from about 7.5K unique algorithmic problems. It covers up to 17 programming languages with the parallelism of multilingual data. It features a total of 7 tasks involving code understanding, generation, translation and retrieval, and wherever appropriate it employs an execution-based evaluation protocol. A detailed documentation of the dataset can be found in (Appendix A). Figure 1 shows an example from XCODEEVAL; it includes a problem description in natural language, a buggy and bug-
6766
free solution to the problem, and relevant metadata such as difficulty level, language, problem tags.

heading: 2 Design Principles & Contribution
text: XCODEEVAL is a result of a number of crucial design principles and contributes to solving many challenges in programming language evaluation as highlighted below.
Reasoning. As a genre, problem solving posits a unique set of challenges that require (a) understanding problem description in complex natural language, (b) expertise in data structures, algorithms, (c) complex reasoning that goes beyond memorization, and (d) generating programs of potentially hundreds of lines so that they can pass a comprehensive list of expertly designed tests. Given the current progress in LLMs and their instruction following capability (Ouyang et al., 2022), competition-level problems that humans find challenging, provide an interesting benchmark to test many aspects of intelligence (Li et al., 2022).
Multilinguality. We aim to cover as many programming languages as possible regardless of the
resource discrepancies. One of the main objectives of this benchmark is to assess the degree to which codes in different languages are parallel to one another. In addition to that, we also intend to evaluate the zero-shot cross-lingual capability of the LLMs.
Evaluation and its granularity. We believe the current evaluation standards do not fully consider
the idea of the global meaning representation of a program, which requires models to comprehend different interpretable code segments and connect both local and modular knowledge into a global representation. We propose execution-based evaluation with unit tests at the global level. While there are many benchmarks covering the local understanding of a code segment, there are only a few that work at a global level as shown in Table 6. We consider a pair of codes to be equivalent if they generate the same output for a given input regardless of syntax/languages (Sajnani, 2016). To support this, we have developed ExecEval, a standardized and distributed execution environment that supports 44 compilers/interpreters in all the languages in XCODEEVAL . We also provide a large number of necessary unit tests (average of 50 per problem) for the relevant tasks (Table 1). In this context, it is noteworthy that 44 out of 165 problems in the CodeContest’s test split have no private unit tests. Additionally, it contains 104 problems without a complete collection of unit tests (as available in the source), thus being inadequate in assessing a solution’s correctness. We identified this issue and excluded such problems from our evaluation sets.
Task difficulty and trainability. We wish to focus on problems of different difficulty levels (from 800 to 3500 rating points, following codeforces.com) such that models with different capabilities can be benchmarked against difficulty levels. We also aim to provide sufficient training data for each task so that pre-trained LMs can be fine-tuned or small-scale models can be trained from scratch.
Data split. Balancing test distributions of textcode instances over multiple attributes such as problems, tags, and execution outcome (e.g., correct vs. wrong) is challenging. We propose a novel data split schema based on a geometric mean and a data selection schema adapting a graph-theoretic solution to the circulation problem with lower and upper bounds (Mount, 2017) that can be applied for other benchmarks as well (Section 3.1).
XCODEEVAL promotes API-friendly minimal evaluation. We evaluate ChatGPT on our classification and generative tasks, and StarEncoder (Li et al., 2023) on the retrieval tasks. In addition, we trained Starcoderbase-3B on Program Synthesis and compared its result with CodeLlama-7b and CodeLlama-13b instruct models. Our results indi-
cate that XCODEEVAL remains difficult to solve for the advanced LLMs, even on a simple binary classification task like Code Compilation (Table 3). With XCODEEVAL, we can identify and compare multilingual executability across languages as well as perform open-ended evaluation on any programming language for the Code Translation and Program Synthesis tasks. Moreover, the unique parallelism of unit tests allows for different interpretable evaluations and analyses on diverse code-related tasks (Section 4). Our experimental results with program synthesis tasks demonstrate that our training data can facilitate a reduction in the size of the language model while maintaining its executable capabilities.

heading: 3 XCODEEVAL: Data, Execution Engine & Tasks
text: We construct our dataset from 25M openly available samples from codeforces.com for a total of 7514 distinct problems. Each sample Sk ∈ S represents a potential solution to a problem Pi ∈ P , and a problem Pi can be solved by employing a set of algorithmic techniques Ti ⊂ T , which we refer to as problem tags (e.g., 2-sat, binary search); see Figure 8 for a complete list of tags. We created two different types of test sets: (i) Compact: This set is intended to quickly test all languages, aimed at evaluating LLM APIs with a lower pricing structure for LLM researchers in academia. (ii) Titan: This set is designed to perform robust tests on all languages, especially for use in industry and/or open-weight LLM setups.
Compact and Titan test sets. To prevent data leakage, we put aside Nh(= 1354) problems as held-out set Dho for the Compact and Titan test split. It ensures that the problems in the Compact and Titan test sets are not seen in training and models need to generalize to unseen problems. We then create Dcompact and Dtitan splits from Dho, while maintaining a balanced tag distribution and ensuring that all the tags in these two sets also exist in the training data, which could be a requirement for certain tasks (e.g., Tag Classification). For this, we iterate over a number of seeds and create random splits. Let γ be the expected ratio of the number of samples in Dcompact and Dtitan, γ = |Dcompact|/|Dtitan|. For each random split, we calculate a tag-wise ratio γT , the ratio of the number of samples in Dcompact and Dtitan for each tag T ∈ T . The geometric mean of {γT }T∈T defines
the ‘tag distribution’ score of a split. We select the split whose score is closest to γ. Appendix BAlgorithm 1 describes our method, which also ensures that Dcompact and Dtitan test sets contain the same tag sets as the training set.

heading: 3.1 Data Creation
text: To make the testing computationally feasible, we aim to control the sample size while maintaining a balanced distribution across problems and tags; e.g., C++ initially had about 647K test samples for tag classification (Appendix D.1). However, finding an optimal solution to this selection problem (i.e., how many samples to select per problem and per tag) is nontrivial. We formulate this as circulation problem with lower and upper bounds (Mount, 2017) within a flow network. Let pi and tk be the number of solutions for a problem Pi and a tag Tk, respectively. Let G = (V,E) be a flow network (a directed graph) with the set of vertices V = {s, P1, ..., PN , T1, ..., TK , t}, where s and t respectively denote the source and sink nodes of the network (Figure 2). For each edge e ∈ E, the lower l(e) and upper capacity c(e) are defined as:
1. Initialize E = ∅.
2. For each problem Pi, add edge (s, Pi) to E and assign l(s, Pi) = min(mp, pi) and c(s, Pi) = min(xp, pi), where mp and xp respectively refer to the minimum and maximum samples to choose per problem if available with mp ≤ xp, thus 0 ≤ l(s, Pi) ≤ c(s, Pi).
3. For each tag Tk, add edge (Tk, t) to E and assign l(Tk, t) = min(mt, tk) and c(Tk, t) = min(xt, tk), where mt and xt respectively refer to minimum and maximum samples to choose per tag if available with mt ≤ xt, thus 0 ≤ l(Tk, t) ≤ c(Tk, t).
4. For each Pi and Tk, add (Pi, Tk) to E if Pi has a tag Tk, and assign l(Pi, Tk) = 0, c(Pi, Tk) =∞.
We then directly adopt the circulation problem solution to find a flow f : E −→ Z+1 that satisfies: ∀e ∈ E, l(e) ≤ f(e) ≤ c(e) and ∀u ∈ V,∑v∈V f(u, v) = 0. In our case, f denotes a feasible flow when the above constraints are satisfied for some G. For each e ∈ E, f(e) represents the following:
1. f(s, Pi) denotes the number of samples to be picked from problem Pi.
2. f(Tk, t) denotes the number of samples to be picked from tag Tk.
3. f(Pi, Tk) denotes the number of samples to be picked from Pi that has a tag Tk. Here, ∑K k=1 f(Tk, t) = ∑N
i=1 f(s, Pi) is the total number of samples selected, which can be controlled in a balanced way by setting the control variables mp, mt, xp, and xt. Appendix C gives further details about the method and hyperparameters for different tasks, along with a comparison to a random data selection strategy.
Table 7 shows the effectiveness of our proposed data splitting method. For all languages data selection with our proposed solution reduces the skewness and standard deviation of the tag distribution.
3.2 ExecEval: A Multilingual, Distributed and Secured Evaluation Engine
An essential requirement for execution-based evaluation is the availability of a secure and scalable framework (Chen et al., 2021; Cassano et al., 2022). With its capacity to support 44 compiler/interpreter versions in 11 different languages, ExecEval offers a versatile and comprehensive approach to program evaluation. The engine is distributed as a secure Docker image, ensuring safe and efficient executions. It also supports easy integration of new compilers/interpreters with custom execution flags (which can also be changed at run-time). While running on unit tests, ExecEval produces one of the six outcomes:
1. COMPILATION ERROR: fails to compile or run due to a syntax error;
2. RUNTIME ERROR: successfully compiles but fails during runtime due to native environment issues (e.g., asserts, division-by-zero);
3. MEMORY LIMIT EXCEEDED: occupies more memory than the limit;
1Z+ denotes the set of non-negative integers.
4. TIME LIMIT EXCEEDED: requires more time than the limit;
5. WRONG ANSWER: successfully compiles/interprets, generates an output but fails to produce a correct answer;
6. PASSED: successfully passes all the unit tests. The program will be flagged as buggy (i-v) even when it fails on a single unit test. Appendix G gives further details about ExecEval.

heading: 3.3 Tasks in XCODEEVAL
text: XCODEEVAL features two classification, three generative, and two retrieval tasks. Table 2 gives a breakdown of the classification and generative tasks per language. Below we briefly describe the tasks; detailed descriptions, motivation, maintainance, support, and process of task formulation along with visualizations of task distributions and task creation rationale can be found in Appendix D.

heading: Classification tasks – Tag Classification and
text: Code Compilation. The goal of Tag Classification is to assign relevant tags to a code and/or natural descriptions of the corresponding problem. This task focuses on measuring the impact of code understanding by incorporating a natural language description alongside the code. It is the only task in our benchmark that does not factor in the code’s executability. On the contrary, the objective of the Code Compilation task is to determine whether the given code is compilable or not, thereby constituting a binary classification problem. All the labels in both tasks are human annotated and found as metadata. By addressing these classification tasks, we aim to explore and evaluate the effectiveness of program comprehension techniques.
Generative tasks – Program Synthesis, Automatic Program Repair (APR) and Code Translation. All of our proposed generative tasks are evaluated with execution-based unit tests by ExecEval. The Program Synthesis task aims to generate executable programming language code that solves a given problem. The problem is defined with a natural language description along with some sample input-output descriptions (see Figure 1). In the APR task, along with the problem, a buggy code is also given. The objective is to correct or refine the buggy code. Moreover, in
the Code Translation task, a code is provided in a source language and the goal is to translate it to a target language. Note that for Code Translation our benchmark provides the inputs for the source programming language and for Program Synthesis we only provide problem description in natural text. For both Program Synthesis and Code-Translation, the underlying execution-based unit test enables evaluation on any target programming language, as long as it is supported by ExecEval.
Retrieval tasks – Code-Code and NL-Code Retrieval. The objective of the Code-Code retrieval is to retrieve relevant executable code when provided with a programming language code as input. On the contrary, the NL-Code retrieval task aims to retrieve relevant executable code based on a problem description. These retrieval tasks are novel in the sense that they consider both the relevance and executability of the retrieved codes for evaluation. To the best of our knowledge, these are the first retrieval-based tasks that incorporate executability as a crucial factor when performing code retrieval. We have also included a retrieval corpus specific to each of the languages for evaluation purposes.

heading: 4 Evaluation and Analysis
text: For all tasks except Code Translation, we evaluate on the Compact split. For Code Translation from source languages, we used the Compact small split (Appendix D.4).

heading: 4.1 Benchmark Results
text: Baselines. We benchmark XCODEEVAL using ChatGPT (gpt-3.5-turbo-0301). To construct a query prompt, we adopt the direct zero-shot prompting method (i.e., no chain-of-thought) that facilitates easier automated evaluation (no overlapping of code and explanations). For the retrieval task, following (Karpukhin et al., 2020), we build a bi-encoder based dense multilingual code retriever by finetuning the StarEncoder (Li et al., 2023) model. Our implementation details are in Appendix F.
Results. We present the results on the classification and generative tasks in Table 3. Overall, the model achieves inspiring yet inadequate results – marking XCODEEVAL a promising yet challenging benchmark as per the current progress in LLMs. Particularly for Tag Classification, we observe decent performance in general, and incorporating a
problem description enhances the model’s overall predictive capability. However, in web languages (e.g., PHP, JavaScript), it exhibits the poorest performance. In Code Compilation, we observe encouraging performance for Go, PHP, Python, Ruby, and C#. However, the performance is close to a random baseline for Java, Kotlin, and Rust.
For Program Synthesis, we find that in popular languages, such as C++, C#, Go, Java, and Python the model performs well, while in rare languages like Rust it fares poorly. Notably while on other datasets such as HumanEval (Chen et al., 2021), ChatGPT achieves much higher scores, 65.8 in pass@1 (OpenAI, 2023; Chen et al., 2022), it significantly falls behind even in pass@5 (∼ 30) in XCODEEVAL – imposing challenges even for such a powerful LLM. In Figure 3 (left), we show the performance for different k. As expected, results increase with k, and no noticeable differences are observed between the compiler (e.g., C++, Java) and interpreter (e.g., Python, Ruby) languages.
In APR, we observe a higher performance scale than in Program Synthesis indicating that the model finds the task relatively easier since it does not necessitate generating a complete code from scratch. Interestingly, in languages where the model underperformed in program synthesis, it exhibits a good performance in APR. For Code Translation, we observe that Kotlin and Go can be successfully translated to most of the other languages, while
C++ and Python are the best to translate to. Table 4 reports results on the two retrieval tasks: Code-Code and NL-Code. For Code-Code retrieval, we computed the top-k retrieval accuracy for each of the 17 query languages from all 17 different retrieval corpora. The summarized results from a 17 × 17 matrix (Appendix D.6-Figure 12) are provided in Table 4, where each row represents a query language and each column represents a corpus language. The column-wise and row-wise averages are denoted as (α) and (γ), respectively. For Code-Code (α), there is a degradation of performance for languages with large corpus such as C++, Python. In the case of Code-Code (γ), languages with limited training data in XCODEEVAL , such as D, Ocaml, Rust performed poorly as query languages. For NL-Code, performance is good across all languages except for D. We suspect that the limited availability of resources for D in both The Stack (Kocetkov et al., 2022) dataset (training corpora of StarEncoder) and our XCODEEVAL dataset could account for this discrepancy. Also, the presence of more hard negative candidates (i.e., very similar to a correct code) makes it a challenging task to identify similar codes. We provide more results on the retrieval outcomes in Appendix D.6.

heading: 4.2 Analysis
text: Knowledge cutoff. XCODEEVAL contains problems that appeared in codeforces.com for the
timeframe: Feb 19, 2010 - Nov 21, 2022 (Appendix B-Figure 7 shows the distribution over time). Since we have a complete footprint of release dates for each of the problems, it enables us to identify data leakage in LLMs that have public knowledge cutoff dates. Figure 4 (left) presents a potential data contamination for ChatGPT. Though
OpenAI (2023) (Table 9) reported no data contamination on codeforces.com, datasets like our XCODEEVAL could empower researchers to conduct insightful analysis and perform an investigation on such serious questions. "It should be noted that XCODEEVAL can only analyze data contamination if there is a good amount of problems that appear after the knowledge cut-off date of the evaluated LLM. For a more interpretable evaluation, we invite LLM builders to disclose their knowledge cut-off dates.
Impact of temperature parameter. Although proved to be crucial (Chen et al., 2021; Austin et al., 2021), previous studies have not extensively examined the impact of the sampling temperature parameter on code executability. To address this gap, we conduct an investigation in which we assessed each sample for Program Synthesis at 20 dif-
ferent temperatures in the range 0.0−2.0. Figure 5 (left) presents the overall distribution of execution outcomes for various languages, encompassing all the samples generated at different temperatures, while Figure 5 (right) displays a distribution of PASSED solutions at different temperatures. As the temperature increases, the likelihood of achieving code executability decreases. We identify the most successful PASSED tests at the temperature of 0.32. Figure 3 (right) presents a comparison of code executability across different languages. For each of the unit test cases, we test it with 20 different temperature values and finally select the temperature with highest PASSED execution. We implemented this approach exclusively for a program synthesis task, utilizing this optimal temperature as a pseudo signal for the best parameter for the remaining tasks. While this incurred a significant budget, it is worth noting that with a larger budget, employing optimal parameter search for all tasks and optimal variation would be ideal. In our evaluation, We see that Rust has overall low code executability. On the other hand, interpretable languages like Python, PHP, and Javascript have high executability. Difficulty analysis. Figure 5 (right) shows the distribution of PASSED problems written in C++ for different difficulty levels.2 We see a sharp de-
2Evaluation done at temperature 0.32, generating 10 samples per problem.
crease in performance as the problems become harder. Additionally, we refer to a concurrent work, MapCoder (Islam et al., 2024), which presents a detailed analysis of the program synthesis strengths and weaknesses of state-of-the-art LLMs such as ChatGPT, Mistral, GPT-4, and Gemini Pro. This analysis is conducted across various prompting techniques, programming languages, difficulty levels, and algorithms using our dataset.
Reasoning capability. Figure 6 shows a reasoning spectrum of ChatGPT on Program Synthesis. A program can be considered a reasoning path to produce an output for an input of a unit test. We define reasoning spectrum as the range or continuum of different reasoning approaches or strategies that a program can employ to produce an output for a given input in a test case. The reasoning spectrum encompasses various levels of execution outcomes by the program in between different languages. The same colored vertical line along different languages represents an agreement of execution outcome for the corresponding languages. Given any two languages, when the code compiles successfully for both but one produces PASSED and the other produces WRONG ANSWER, we can conclude that there is a gap between reasoning capability of the two languages. We notice a low agreement in the reasoning spectrum between languages suggesting that a further improvement of
transfer learning can be applied to the reasoning capability from high to low resource languages.
Though Figure 6 (top) shows a general comparable structure of reasoning capability of an LLM for different languages, it does not show the overall performance within each language. By grouping the same execution outcomes together, Figure 6 (bottom) shows exceptional code executability on Python (no compilation error). However, their reasoning capability (# of PASSED unit tests) remains fairly comparable with other languages.

heading: 5 Evaluation of Program Synthesis on Smaller Models
text: For Program Synthesis tasks, we finetuned starcoderbase-3B model with our trained data. We also evaluated the CodeLlama-7b-Instruct-hf and CodeLlama-13b-Instruct-hf models with our evaluation data. A 3B fine-tuned model is better than a 7B instruct model but worse than a 13B instruct model. We observe that training a smaller model with the training data performs well on our task rather than using a general-purpose instruct/chat model. However, large instruct models are better than smaller fine-tuned models.
So, the impact of our training dataset varies between different scales. Comparing the results between Tables 3 and 5 also provides a general idea of how challenging our task is.

heading: 6 Conclusion & Future Work
text: We have introduced XCODEEVAL , a large-scale multilingual multitask benchmark for code-based large language models. XCODEEVAL features seven different tasks involving code understanding, generation, translation and retrieval in up to 17 programming languages, and it employs an executionbased evaluation protocol. We have also presented ExecEval, a multilingual code execution engine that supports all the programming languages in XCODEEVAL . In summary, the combination of XCODEEVAL and ExecEval presents a novel framework that offers a fresh perspective for examining and analyzing large language models, facilitating comprehensive and to some extent highly interpretable investigations. We hope that by utilizing the extensive metadata and execution-based evaluation, there is potential for the discovery of new scaling laws and emergent capabilities.

heading: Limitations
text: Though the codes are written by a diverse group of experts in a diverse number of languages, data is collected from a single source thus limiting the domain diversity. Besides, there is a clear discrepancy between the resource of different programming languages (see Appendix D-Figure 9) and most of the codes in XCODEEVAL are at the document level and often written in a non-modular way without a doc-string. In Appendix J, we discuss the possibilities of evaluation data leakage. The data is distributes through CC BY-NC 4.0 license.

heading: Ethical Considerations, Potential Risks & Documentation
text: Though the data is collected from openly available sources, it has not been humanly audited. We have made our best efforts to use automated tools for identifying and removing codes with sensitive details, resulting in the removal of approximately 2 million samples from our original collection. However, it is important to emphasize that despite our diligent efforts, code can still potentially contain sensitive information and security vulnerabilities, although not something that is not openly available. Additionally, code datasets may reflect biases present in the original codebase or the developers who contributed to it.
xCodeEval documentations in Appendix Appendix G to Appendix K, follow all the necessary guidelines of NeurIPS datasets-track (e.g., datasheets (Gebru et al., 2021), nutrition label (Holland et al., 2020), and data card (Hutchinson et al., 2021)). Our github and huggingface repositories provide two valuable sources of both data access and documentation. To mitigate risks and resolve frequently asked questions, we regularly address queries or issues there.

heading: A Appendix

heading: A.1 Frequently Asked Questions

heading: How xCodeEval is different in evaluating the
text: model’s code capabilities? Some unique feature of xCodeEval are (i) the exact date of the problem features the analysis of data contamination or knowledge cutt off performance (ii) evaluation in so many paired languages for same probem gives us more insights (specially transfer capability) on the progress of LLMs (iii) Strengths and weaknesses study for different algorithms and difficulty levels, (iv) xCodeEval offers a better test suite across multiple languages. While there have been a few attempts to create competitive level evaluation benchmarks (i.e., code_contest for Python, CPP, Java), in our evaluation suite, we make sure all the problems have sufficient amounts of test cases that cover the actual proof of the solution. (v) xCodeEval offers language agnostic testing: While previous evaluation benchmarks are language-dependent (i.e., apps), xCodeEval is language agnostic for generative tasks. (vi) xCodeEval emphasizes reasoning evaluation at the granular level. With additional extensive metadata, xCodeEval ensures that LLM and/or systems can be evaluated based on problem difficulties, categories, and execution failure levels.
(vii) With xCodeEval, we also developed an execution backend, ExecEval, which ensures a secured and easy-to-use robust API-based evaluation. (viii) In xCodeEval evaluation, we place a significant emphasis on data contamination. We make sure that there is no training-test overlap within the evaluation dataset. (ix) We also provide a novel sampling mechanism to perform balanced data selection.
Does xCodeEval requires human auditing? With careful verifications, we make sure that all the unit tests used for Compact and Titan test datasets are human-generated and cover the proof of the solution. For generative tasks, our authors manually checked the problem description for the Compact and Titan test dataset. For executability check, we manually calibrated the compiler flags and internal memory usage for ExecEval to get a stable performance. In addition to that, because of our manual quality control, we found some issues with a couple of languages and their stable performance on the executability. We decided to remove those languages from the evaluation dataset. For xCodeEval, we wanted to make sure a smooth and confident execution-based evaluation for LLMs and coding systems.
Notes on intended usage scenarios. Considering the recent progress, the primary objective of creating XCODEEVAL is to challenge LLMs, especially the frontier models. As such, the tasks proposed could be very challenging for smaller models, even for the binary code compilation task. The relatively smaller (per-language) Compact splits can be used to assess multilingual features and to get an overall picture of multilingual generalization. The large Titan split is meant to rigorously evaluate specific programming languages and conduct more language-specific, in-depth analysis. We have also included the Training split. The intended use of the training split is to include it in the large scale pre-training or SFT mixtures. For both Compact and Titan evaluation, we recommend using an Instruct/Chat Model.
Where are the dataset construction recipes? This paper provides a detailed description of our dataset construction. However, to adhere to the page limit, we needed to shorten them in the main paper (Section 1, Section 3) and move to the supplementary Appendix D for details.

heading: Where are the documentation, maintenance,
text: and support? XCODEEVAL documentations in supplementary Appendix G to Appendix L, follow all the necessary guidelines of datasheets (Gebru et al., 2021), nutrition label (Holland et al., 2018), and data card 3. Following the accountability framework proposed by (Hutchinson et al., 2021), we will release the data construction procedures, data access, additional documentation, implementation details as well as opensource the evaluation framework.
Is there any automated process used for creating the gold labels? Please note that all current data (text, code, test-cases, and labels) are human-written. More specifically for tag classification tasks, the tags are already provided in codeforces.com. The tags are generally updated after the contest by experienced problem solvers trusted by the codeforces team. In addition to that, the evaluation is done by programming language compilers & interpreters. We put a huge effort into developing ExecEval and dockerized it to synchronize the evaluation process.
On the possibility of data leakage and contamination. We note that, although we completely separate our Compact and Titan test data from training, LLMs might have possible data leakage from pretraining. We find that even identifying data leakage (test data exists or not in the prertraining corpora) is challenging using conventional data search methods due to search cost and complexity (e.g., exact match or token overlapping methods) for (i) long sequence search for libraries (ii) boilerplate code identifying. Apart from that, hashing based searches often suffer from not having properly segmented text.
In this paper, we introduce an approach to address the challenge of leakage-free evaluation, employing a technique rooted in the concept of a knowledge cut-off Our finding in Section 4.2 in the main paper shows that the data contamination significantly impacts the model performance and it needs to be interpreted with proper analyses. Another method toward leakage-less evaluation could be to have a human-written separate evaluation set that is hidden or regularly updated which we are considering in our long-term release plans.
3https://sites.research.google/ datacardsplaybook/
Only generative tasks utilize unit tests. How are classification and retrieval tasks considered executability? In our proposed tasks, except for the tag classification task, all tasks consider online or offline unit test executability. We included a tag classification task since tags were used for the sampling algorithm that we proposed. Other than that, our code compilation and retrieval task takes into account offline code executability, where compilers are applied before releasing datasets to obtain labels. Additionally, in retrieval tasks, we treat passed samples as correctly retrieved samples and generate hard negatives from incorrect code from the same problem. Please note that all the data — including text, code, test cases, and labels — is human-written.

heading: A.2 Related Work
text: Following NLP (Devlin et al., 2019; Radford et al., 2018; Raffel et al., 2020), transformer-based pretrained LLMs have shown significant success in code, both in understanding and generation (Wong et al., 2023). Table 6 shows a detailed comparison between different programming language-related datasets.
Code Understanding. Lu et al. (2021) propose a benchmark CodeXGLUE, which comprises three widely-used code understanding tasks, defect detection, clone detection, and code search. Zhou et al. (2019) treat defect detection as a binary classification task. They propose a model called Devign which they evaluate on four large open-source C projects. Additionally, Russell et al. (2018) leverage open-source C/CPP repositories to support function-level vulnerability detection. To further understand code semantics, Svajlenko et al. (2014) propose a benchmark BigCloneBench to measure the similarity between code pairs to predict whether they have the same functionality (i.e., clone detection); BigCloneBench was collected from opensource Java repositories with manual validation. Arguably, code defect and clone detection might not be appropriate for fully evaluating models’ ability in understanding code semantics (Wang et al., 2021; Guo et al., 2022). Moreover, they only support a few programming languages. Code search on the other hand considers semantic relevance for both code-to-code and text-to-code. They are formulated to retrieve semantically similar codes given a query code (Lu et al., 2021) or code description (Husain et al., 2019). The existing code
search benchmarks like CodeSearchNet (Husain et al., 2019) only select the first documentation as the text query to search corresponding functions. Recently, Fu et al. (2023) introduce CodeApex, a bilingual benchmark to evaluate the language models on three different tasks consisting of programming comprehension, code generation, code correction. Among its tasks, programming comprehension examines the ability to understand code from various aspects, such as the syntax’s mastery, code execution flow, and executing algorithms. Nonetheless, this dataset only covers one programming language, which is in contrast to our work.
Code Generation. Code generation has grown in popularity as many pre-trained LLMs have achieved remarkable performances in these tasks like decoder-only models (Chen et al., 2021; Izadi et al., 2022; Nijkamp et al., 2022) and encoder-decoder models (Wang et al., 2021; Guo et al., 2022; Ahmad et al., 2021a) Notably, PaLM (Chowdhery et al., 2022) and AlphaCode (Li et al., 2022) outperform average human participant in competition-level coding. Thus, researchers attempt to build increasingly difficult and factual code generation tasks. These tasks can be classified as code-to-code generation and text-to-code generation.
As for code-to-code generation tasks like automatic program repair (APR) (Tufano et al., 2019) and code translation (Lu et al., 2021), the metric-based automatic evaluation measures like BLEU (Papineni et al., 2002), CodeBLEU (Ren et al., 2020), and exact match scores are suboptimal for evaluating the quality of a generated code. To improve the reliability and feasibility for code generation evaluation, Berabi et al. (2021) create a large-scale JavaScript patch repair dataset from GitHub commits, where 52 error types are detected by a static analyzer ESLint4. They further drive efforts in enhancing program repair evaluation by providing an error removal metric to take various form of error fixes into consideration. To address the nature of code semantic and syntactic evaluation, execution-based evaluation with comprehensive test suites has a growing demand. A popular Java APR benchmark Defects4J (Just et al., 2014) takes the number of correct fixes into account, where a correct fix should pass all test cases and provide a desired functionality. Nevertheless, Defects4J does not possess a cohesive training cor-
4https://eslint.org
pus. A common strategy to address this limitation is to construct the training dataset using GitHub’s publicly available repositories, and relying on bugspecific commit messages (Zhu et al., 2021). However, this heuristic-based approach includes bugirrelevant commits and unrelated code pairs, which can significantly affect the the quality of collected training dataset (Xia and Zhang, 2022).
For text-to-code, the widely used dataset CONCODE (Iyer et al., 2018) consists of a large collection of natural language (NL) comments and Java code snippets. Specifically, this dataset is constructed by scraping code snippets from opendomain Java GitHub repositories and utilizing heuristics to extract NL comments from Javadoc.
By following a similar approach, JuICe (Agashe et al., 2019) collects publicly available Jupyter notebooks from GitHub, and CoNaLa (Yin et al., 2018) collects Python and Java codes with NL comments from StackOverflow posts. Besides, they attempt to improve the quality with professional annotators. In addition, MoCoNaLa (Wang et al., 2022a) extends CoNaLa to support more natural languages. Despite their coverage, the general lexical-based evaluation metrics are insufficient to measure the correctness of generated codes. To alleviate this limitation, ODEX (Wang et al., 2022b) provides execution-based evaluation via humanwritten test cases of diversified Python libraries. This execution-based paradigm has been widely applied to evaluate benchmarks in Data Science domain like DSP (Chandel et al., 2022), DS-1000 (Lai et al., 2022) and Exe-DS (Huang et al., 2022) as well as general code generation benchmarks in single-language settings such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021). Apart from HumanEval, CoderEval (Yu et al., 2023) further leverages the contextual information and achieves a full spectrum of test coverage with additional manually crafted tests, providing 100% test coverage. To improve the diversity of code generation tasks, Fu et al. (2023) propose a bilingual code evaluation benchmark CodeApex to support both Englishto-Code and Chinese-to-Code generation tasks. Crowd-sourcing human feedback to generate high quality code generation dataset has been explored by (Wong and Tan, 2023). As for more particular multi-turn MTPB (Nijkamp et al., 2022), multilanguage CodeContests (Li et al., 2022), and domain specific BioCoder (Tang et al., 2023) bench-
marks, they all leverage test cases, and exploit code execution for better evaluation.
B Algorithm for initial Compact and Titan test split creation
To make sure we do not have train and test overlap, at first we divide the set of problems into two sets. In one set we keep all the problems for which we do not have a complete set of unit tests. In another set, we keep the problems where we have a complete set of unit tests that ensures the correctness of the solution of the problem. We use the first set for training and the latter set for the test data. Figure 7 shows the chronological distribution of our training, Compact and Titan test data. After selecting the test problem sets, we have thousands of solutions for each of the problems. But these problems are not divided into Compact and Titan test splits. As a heuristic, we can consider the tag distribution as the domain of the problem. To ensure that we have proper domain coverage, we employ Algorithm 1 to make the Compact and Titan test sets contain the same tag sets as the training set. In addition to that, it also selects the best possible splitting point based on the geometric mean. However Algorithm 1 provides only a splitting point for hundreds of thousands of Compact and Titan test data points. To reduce the redundant data based on different levels of conditions, we formulate the problem of selecting data points to a linear programming problem (more on this in Section 3.1).
Algorithm 1 Compact and Titan Test Split Creation Input: A held-out dataset Dho, a fraction value γ where 0 ≤ γ ≤ 1, an integer N indicating number of seeds. Output: Dvalid, Dtest spits Initialize: count = 0, bestScore = γ + 1 while count < N do
seed = getSeed() Shuffle Dho Dvalid = Dho[0 : |Dho| × γ] Tvalid = set of tags in Dvalid Dtest = Dho[|Dho| × γ : |Dho|] Ttest = set of tags in Dtest if Tvalid ̸= Ttest then
continue end if for all T in Tvalid do
γT = #samples in Dvalid with tag T #samples in Dtest with tag T
end for µ = geoMean({γT }T∈Tvalid) if |γ − bestScore| > |γ − µ| then
bestScore = µ save current split {Dvalid, Dtest} count = count+ 1
end if end while

heading: C Hyper parameter tuning for circulation problem
text: Let M be the number of samples we want to select for any set of submissions. We call any (mp,mt, xp, xt) a valid tuple if the flow network has a feasible flow for the circulation problem defined in eq. from Section 3.1. Let d = ⌊(∑Ni=1 f(s, Pi)−M)2/∆⌋, representing the squared difference between samples we want and the samples selected for the flow and ∆ reduces the resolution in which we look for differences. Here d defines a boundary from M where we al-
low choosing an expected solution with mp, mt, xp, and xt. Finally, the lexicographical ordering (−d,mt,−xt,−xp,mp) is used to find the largest element in the collection of valid tuples which always exist if we limit our search space to a finite set. The largest element in this ordering depicts the nearest (close to M ) selection of samples that maximizes the minimum number of samples per tag mt. When there are many solutions with the same (−d,mt), we prioritize reducing the maximum number of samples per tag, xt. Similarly, we prioritize xp and mp as defined in the lexicographical ordering.

heading: C.1 Search Techniques
text: 1. It was manually checked that
(mp,mt, xp, xt) = (1, 1, 1000, 1000) is a valid tuple for any set of submissions that were processed and ∆ = 1000 was chosen.
2. In Tag classification task (Appendix D.1) and Code compilation task (Appendix D.2), M is 2000, 10000 for any language for Compact and Titan test split respectively. For Code translation (Appendix D.4) M was 400, 2000 for the same.
3. Search largest tuple (−d1,mt1 ,−xt1 ,−xp1 ,mp1) where mt1 ∈ {1, 6, 11, · · · , 496}, mp1 ∈ {1, 2, 3, · · · , 19} and xp1 = xt1 = 1000. Since (mp,mt, xp, xt) = (1, 1, 1000, 1000) is a valid solution, hence the set of valid tuples is nonempty. Let f1 be the flow for the flow
network defined for mt1 ,−xt1 ,−xp1 ,mp1 . Let fP1 = max1≤i≤N f1(s, Pi), fT1 = max1≤k≤K f1(Tk, t) be the maximum flow through edges from s to Pi, and same through edges from Tk to t.
4. Now again search the largest tuple (−d2,mt2 ,−xt2 ,−xp2 ,mp2) where mt2 ∈ {mt1 ,mt1 + 1, · · · ,mt1 + 49}, xt2 ∈ {fT1 − 100, fT1 − 80, · · · , fT1}, xp2 ∈ {fP1 − 5, fP1 − 4, · · · , fP1}, mp2 ∈ {mp1 ,mp1 + 1}. Since mt1 , fT1 ,mp1 , fP1 is included a solution is found in this step too. Define fP2 , fT2 similar to previous step.
5. Finally search the largest tuple (−d3,mt3 ,−xt3 ,−xp3 ,mp3) where mt3 = mt2 , xt3 ∈ {fT2 − 100, fT2 − 99, · · · , fT2}, xp3 = xp2 ,mp3 = mp2 .
While it is not an exhaustive search, we prioritize minimizing xt −mt over xp −mp.

heading: C.2 Results
text: We compared the performance of data selection using circulation problem technique with randomly selecting equal number of samples for Compact and Titan test sets of all languages and measured the skew µ̃3, and standard deviation σ of the distribution of tags in the selected data. Here lower value of |µ̃3| means more symmetric distribution. On the other hand, a lower value of σ represents that the number of samples in each tag are closer to the mean.

heading: D Tasks Construction Process

heading: D.1 Tag Classification
text: We formulate the task as a multi-label classification problem in two settings: Code-to-Tag (Code2Tag) and Problem Description-and-Code to Tag (DesCode2Tag). In Code2Tag, given a code C in any language, the task is to predict the corresponding tag set T. In DesCode2Tag, the natural language problem description is also given as input in addition to the code. The performance difference between Code2Tag and DescCode2Tag settings can suggest if the problem description can help models identify the problem tags (i.e., the type of solution needed)..
For these tasks, the split for Compact and Titan is done with a ratio of 1 : 5 (i.e., γ = 0.2)
using Algorithm 1. To get the final Dvalid and Dtest with a feasible number of samples, we use flow network-based data selection approach with the details of hyper-parameter settings presented in Section 3.1.
The distribution of the samples according to the tags is presented in Figure 8.
We further propose a language-specific tag classification task, in which each programming language has its own Code2Tag and DesCode2Tag settings.

heading: D.2 Code Compilation
text: Given a code C in a language L and its compiler or interpreter version B, the code compilation task is to decide whether the code compiles or not. The Compact and Titan test splits are created using a modified version of Algorithm 1 that balances the partition based on the compilation outcome of the code instead of the tags of the problem that the code belongs to. We use a ratio γ of 1 : 5. Then a simplified version of the circulation problem is used to prevent too many codes coming from a single problem, and also to ensure a balanced output distribution. The details of hyper-parameter settings of the circulation problem technique are presented in Section 3.1. In the flow network construction, tags {Tk} = {true, false} as true if the code compiles or not. Furthermore true and false examples are present in equal numbers in both Compact and Titan test dataset.
We propose three generative tasks which require a global understanding of programming languages. For the evaluation of generative tasks, we follow execution-based evaluation instead of lexical similarity. All the generative tasks are evaluated using ExecEval execution engine. We provide complete unit tests for all problems in the Compact and Titan test dataset which also satisfy the conditions of the input-output description of the problem.

heading: D.3 Program Synthesis
text: Given a problem described in natural language, program synthesis task is to write a program that solves the problem. We can express each sample in the dataset as a tuple (C,P, l, L), where C denotes a solution code written in a programming language L for the problem P , and l denotes the compiler/interpreter version of the code. All code samples in our dataset are unique and marked as a correct solution (PASSED outcome) to the problem. The
Compact and Titan test splits are created from the heldout problems using Algorithm 1 with a ratio (γ) of 1 : 9. The generated code is judged based on executions on the unit tests.

heading: D.4 Code Translation
text: Each sample in the code translation data can be expressed as a tuple (C, P, l, L), where C denotes a set of solution codes in a programming language L for the problem P , and l denotes the compiler/interpreter version of the code. All codes in set C are unique and guaranteed to be marked as a correct (PASSED outcome) solution to the problem by the compiler/interpreter.
The Compact and Titan test splits are created from the held-out problems using Algorithm 1 with a ratio (γ) of 1 : 5, and employ the data selection
method with flow network (Section 3.1) to have a practical evaluation setup while ensuring a balanced distribution over problems and tags. Figure 9 shows the distribution of the machine translation tasks. Since code translation considers all possible directions of translation between languages, in addition, to train, Compact and Titan test split, we also provide a small Compact split.

heading: D.5 Automatic Program Repair (APR)
text: We consider APR as a task to synthesize a fix for a detected program bug. We create a bug-fix pair by matching a buggy code (1-5 execution outcome in Section 3.2) with a PASSED solution. Given a bug-specific defect, the objective of this task is to generate a correct fix that passes all the unit tests.
Let C = {C1, . . . , Cm} be the set of programs
submitted by a participant in chronological order to solve a specific problem P . Some of these submissions can be ‘buggy’, while some can be PASSED. We create the ‘bug-fix’ pairs from C as follows.
1. We iterate over C and mark the PASSED ones as ‘fixed’. Let C∗j is one such case.
2. For each buggy submission that was made before C∗j , we measure its lexical similarity with C∗j and select the one (say Ck where k < j) with the highest similarity score to pair it with C∗j and form a bug-fix pair (Ck, C ∗ j ).
We use difflib5 to measure the similarity.
3. With each bug-fix pair (Ck, C∗j ), we also include the corresponding problem description P and execution outcome Vk ( Section 3.2) of Ck.
5https://docs.python.org/3/library/difflib.html
4. The tuple (Ck, C∗j , P, Vk) represents a sample in our APR task.
We repeat this process for each participant and problem to create the final APR dataset. As reported in Table 8, it comprises more than 5M practical bug-fix pairs and supports 11 programming languages. For data selection in APR, we considered execution outcome (Section 3.2) as tags in the network flow construction (Section 3.1).
Due to the large input specification of the APR task, sometimes the input sequence length becomes too large. However, we have not compromised the benchmarks by selecting only small sequence length samples but rather keep them as challenging tasks for the language models. Figure 10 shows the length distribution of Compact and Titan test input sequence.

heading: D.6 Code Retrieval
text: Code retrieval tasks typically aim to measure the mere semantic relatedness between a natural language (NL) query and a programming language (PL) code. However, a code that is relevant, can still be buggy and thus be misleading (see an example in Figure 11). In view of this, we propose two new and more challenging retrieval tasks in our benchmark, which require a deeper understanding of the NL query and code. In particular, we propose NL-Code and Code-Code retrieval tasks that involve identifying a correct code from a large pool of candidates containing similar codes. In both tasks, for each programming language, we aggregate all the submitted codes and their test cases to create a retrieval corpus and a testbed for evaluating their correctness against test cases. Table 9 gives a detailed statistics of our retrieval tasks. The datasets for the subtasks and the evaluation schema are discussed below.
NL-Code retrieval. This task involves matching an NL problem description to the most relevant and correct code from a pool of candidates. An example of an NL description and its corresponding codes are shown in Figure 1. To gather data for this task, we only use instances where the NL description is valid and there is at least one correct solution code (i.e., with execution outcome PASSED). For an NL problem description, we consider all the correct solutions as positive examples and all the wrong (or buggy) solutions as negative examples.
Code-Code retrieval. Given an input code (as a query), this task involves finding similar and logically equivalent code (i.e., passes the same set of test cases) from a collection of candidates. We ensure that the query code solves a specific problem (i.e., a correct solution without any detected bugs) and evaluate whether the retrieved candidate also solves the same problem or not. To collect data for this task, we only consider the programming problems which have at least two correct code solutions that pass all the corresponding test cases (i.e., with execution outcome PASSED).
From each of these problems, we randomly choose one correct solution as a (PL code) query and pair it with the other correct solutions as positive examples and the corresponding wrong solutions (i.e., with execution outcome WRONG ANSWER) as negative examples.
Retrieval corpus metadata and evaluation protocol. We preserve the problem specifications and execution outcomes (e.g., PASSED, WRONG ANSWER) for each candidate code in our retrieval database. For both the NL-code and code-code retrieval tasks, we use this information to determine the correctness of a retrieved code, checking if that solves the same programming problem as the input query by passing all its unit tests or not.
Evaluation metrics. We evaluate the retrieval performance in terms of retrieval accuracy@k: computed as the proportion of queries for which a correct code retrieved within top-k.
Our retrieval benchmark has 17 programming
languages and our training dataset is the largest that provides annotations of similar codes that are found logically equivalent or correct based on the passing of test cases. For evaluation purposes (i.e., for test sets), we release the input problem description (in NL-Code) or the input code (in Code-Code) only and keep all other metadata confidential. Covered programming languages and their data statistics in both tasks are summarized in Table 9.
Retrieval evaluation. Figure 12 reports one retrieval task (code-code) performance. As anticipated, the retrieval capability for the same language pair (a.k.a., monolingual retrieval) of our baseline model performances are relatively stronger and we observe performance degradation when performing cross-lingual retrieval between different languages. However, surprisingly, mono-lingual retrieval accuracies for popular languages like C, C++, C#,
Python, and Java are lower than others such as Go, Haskell, Javascript, Kotlin, Ruby, Scala etc., possibly due to their large retrieval corpus size and presence of more hard negative candidates (very similar to the correct code). Furthermore it is suspected that the lack of enough resource on D programming language in both The Stack (Kocetkov et al., 2022) and XCODEEVAL is the primary reason for its poor scores. In this paper, we develop an unified single retriever model and leave the development of multiple different ones using different base encoder models as future works (Parvez et al., 2023).

heading: E Evaluation Metrics
text: Tag Classification. Since it is a multi-class multilabel classification problem, we use f1 score with macro averaging over the classes (in this case the tags) to measure the performance as macro aver-
aging is class population size independent. This is done by first calculating the f1 score for each class (tag) T ∈ T (the set of all tags) with the following formula (Taha and Hanbury, 2015):
f1T = 2 ∗ PrecisionT ∗ RecallT
PrecisionT + RecallT
= 2 ∗ TPT
2 ∗ TPT + FPT + FNT
(1)
And then the macro average is calculated as the
mean of f1T for all T ∈ T (Opitz and Burst, 2019).
f1macro = 1 |T | ∑
T∈T f1T .
Code Compilation. Since it is a binary classification problem, we use accuracy which is defined as the proportion of correct prediction among all predictions (Metz, 1978). That is
Accuracy = TP + TN
TP + TN + FP + FN .
Generative tasks. The generative tasks in XCODEEVAL(i.e. Automatic Program Repair,
Code Translation, Program Synthesis) are all evaluated using pass@k used in Chen et al. (2021).
Code Retrieval. The Code-Code, and NL-Code retrieval tasks in XCODEEVALis evaluated using top-k accuracy (Thakur et al., 2021).
F Implementation Details
Classification tasks. OpenAI chat completion API with gpt-3.5-turbo-0301 model was used at temperature 0.325 and n = 1. List prompts for Code2Tag, DescCode2Tag, Code Compilation as figure or inline styling, with example api response.
Then evaluate each through corresponding metric as mentioned in Appendix E.
Generative tasks. OpenAI chat completion API with gpt-3.5-turbo-0301 model was used at temperature np.linspace(0, 2, 20) and n = 1 for Program Synthesis. Then upon identifying best temperature at 0.325, another batch of codes were generated at temperature 0.325 and n = 20. For APR, Code Translation temperature of 0.325 and n = 10 was used. The generated codes were executed with ExecEval with default parameters (follow Appendix G for a list of different parameters and their default values) to determine its functional correctness and then evaluated using pass@k. Figure 13 shows the compiler versions used to execute the generated codes.
Retrieval tasks. We finetuned a DPR6 model with starencoder for both query and corpus encoder. Both NL-Code, and Code-Code were trained with maximum sequence length of 1024, and effective batch size 48 for 37 epochs. The model is trained with a multilingual manner. For CodeCode we used XCODEEVAL as it is, and for NLCode we made the following template: ‘Description: {{description}} Input specification: {{input_spec}} Output specification: {{output_spec}}’ for the query string. For evaluation we used corpus provided by XCODEEVAL to generate the dense vectors and then perform queries with test split for both Code-Code, and NL-Code. Finally the top-k accuracies were measured.
G ExecEval Details
ExecEval is an automated code execution and evaluation engine distributed through docker for security and portability. It supports 44 compiler versions for 11 programming languages as shown in Table 10. It exposed NUM_WORKERS CLI argument to spawn multiple workers that can execute the codes. It is highly scalable in the sense of adding support for more languages or one can just change NUM_WORKERS to execute more codes in parallel. At the top level of ExecEval, there is a HTTP server that exposes 2 API endpoints /api/execute_code, /api/all_runtimes. Figure 15 shows a simple usage of execute_code API. By default the execution of a code is stopped when the code doesn’t pass a unit test as pass@k depends on whether
6https://github.com/facebookresearch/DPR
all the unit tests passed or not. This can be disabled by adding ‘"stop_at_first_fail": false’, in which case all unit tests for a given code will be executed irrespective of the outcomes for other unit tests. Figure 6 is generated with disabling ‘stop_at_first_fail’. It is worth noting that, disabling this setting can increase the evaluation time significantly (e.g. in Table 3 for program synthesis (N) where 23,320 codes were executed the difference was of approximately 12 minutes and 2 hours 12 minutes where ExecEval was running with 61 workers).
Security measures. ExecEval uses prlimit7, and seccomp8 to limit system resources allocated for any instance of code executed through the API endpoint in addition to using unique unprivileged users for each worker spawned with NUM_WORKERS. Table 11 shows the default values provided to prlimit, furthermore nofile, and nproc are customized for each of the supported languages. The seccomp is used to block socket system call, which disables network access (this is default). One can enable network access by adding ‘"block_network": false’ in the request body as shown in Figure 15. Similarly, adding a ‘limits’ object in the request body allows one to change the limits for executing an individual code. The execution of code via an unprivileged user disables the read, write, or execute permissions of any sensitive files. Figure 16, Figure 17, and Figure 18 shows an example of a fork bomb written in C, a network request in Python, and an escalated access in Python which are all blocked by ExecEval, respectively.

heading: H Definition of Data Attributes
text: For each of the tasks, we have two data files that are required for multiple tasks.
1. problem_descriptions.jsonl
2. unittest_db.json
To avoid data redundancy we didn’t include these data with the relevant tasks, rather we added a unique id src_uid to retrieve these data. We include a data loader using datasets9 package that defines the tasks.
7https://man7.org/linux/man-pages/man1/prlimit.1.html 8https://man7.org/linux/man-
pages/man2/seccomp.2.html 9https://github.com/huggingface/datasets
We define each of the data attributes of XCODEEVAL in the following sections.
H.1 Problem Description (problem_description)
The problem descriptions are in the problem_descriptions.jsonl file. This data source is linked to the proposed tasks by matching the src_uid column for each sample in the relevant tasks. The columns copied from the problem_descriptions.jsonl file are prefixed with prob_desc_.
1. description: Problem description in textual format, math operations are written in latex.
2. input_from: How the program should take the unit test.
3. output_to: Where the program should output the result of the unit test.
4. time_limit: Time limit to solve the problem.
5. memory_limit: Memory limit to solve the problem.
6. input_spec: How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
7. output_spec: How the outputs should be printed. Most of the time the unit test results are matched with an exact string match or floating point comparison with a precision boundary.
8. sample_inputs: A sample input for the code that is expected to solve the problem described in description.
9. sample_outputs: The expected output for the sample_input that is expected to solve the problem described in description.
10. notes: Explanation of sample_inputs & sample_outputs.
11. tags: The problem categories.
12. src_uid: The unique id of the problem. This ID is referred to in the task data samples instead of putting all this information.
13. difficulty: How difficult is it to solve the problem for a human (annotated by an expert human).
14. created_at: The Unix timestamp when the problem was released. Use datetime lib in Python to parse it to a human-readable format.
H.2 Unit Tests (hidden_unit_test)
The unit tests needed for execution based evaluation are in the unittest_db.json file. This data source is linked to the proposed tasks by matching the src_uid column for each sample in the relevant tasks. The columns copied from the unittest_db.json file are under the attribute hidden_unit_test.
1. unittest_db.json dict keys i.e., db884d679d9cfb1dc4bc511f83beedda are the src_uid from problem_descriptions.jsonl.
2. input: Input of the unit test.
3. output: List of expected outputs for the unit test.
H.3 Tag Classification (tag_classification)
Given a source_code the objective is to classify the code into multi-label tags (label:tags).
1. lang: Runtime/compiler version of the source_code.
2. source_code: A program.
3. tags: List of potential algorithmic techniques required to write the program.
4. lang_cluster: A generic programming language name the value of lang belongs to.
5. code_uid: A unique ID for the sample. It is not important for model training. If you find any issue with the sample, you can report it to us by mentioning the code_uid.
6. src_uid: A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved.
7. difficulty: Difficulty rating of the problem indicated by src_uid. The higher the harder.
H.4 Code Compilation (code_compilation)
Given a source_code the objective is to classify if the code compiles or not (label:compilation_error).
1. lang: Runtime/Compiler version of the source_code.
2. source_code: A program.
3. lang_cluster: A generic programming language name the value of lang belongs to.
4. compilation_error: True/False, Indicates if the code generates a compilation error or not.
5. code_uid: A unique ID for the sample. It is not important for model training. If you find any issue with the sample, you can report it to us by mentioning the code_uid.
6. src_uid: A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved.
7. difficulty: Difficulty rating of the problem indicated by src_uid. The higher the harder.
8. file_name: Name of the source JSON file from where data is loaded.
H.5 Automatic Program Repair (apr) Given a bug_source_code the objective is to generate a fixed version of the code that passes all the unit tests. Use fix_source_code for training.
1. similarity_score: A similarity score between bug_source_code and fix_source_code given by difflib.
2. equal_cnt: A metric comparing bug_source_code and fix_source_code. Recommended by difflib.
3. replace_cnt: A metric comparing bug_source_code and fix_source_code. Recommended by difflib.
4. delete_cnt: A metric comparing bug_source_code and fix_source_code. Recommended by difflib.
5. insert_cnt: A metric comparing bug_source_code and fix_source_code. Recommended by difflib.
6. fix_ops_cnt: A metric comparing bug_source_code and fix_source_code. Recommended by difflib.
7. bug_source_code: Buggy code.
8. fix_source_code: A potential fix of the buggy code that passed all the unit tests.
9. lang: Runtime/Compiler version of the source_code.
10. fix_code_uid: A unique ID for the fix code. It is not important for model training. If you find any issue with the sample, you can report it to us mentioning the fix_code_uid.
11. bug_code_uid: A unique ID for the buggy code. It is not important for model training. If you find any issue with the sample, you can report it to us mentioning the bug_code_uid.
12. src_uid: A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved. Refer to Structure of problem_descriptions.jsonl.
13. apr_id: A unique ID for the apr sample. It is not important for model training. If you find any issue with the sample, you can report it to us mentioning the apr_id.
14. difficulty: Difficulty rating of the problem indicated by src_uid. The higher the harder.
15. tags: List of potential algorithmic techniques required to write the program.
16. bug_exec_outcome: A pre-run execution outcome of bug_source_code. Follow
Section 3.2 to know the potential list of outcomes. The exec_outcome flags in the training data comes from a pre-run environment from the source website and they are not verified in ExecEval. However, training data doesn’t include unit-test to avoid potential hacks and confusion. We provide unit test for only Compact and Titan test data.
17. fix_exec_outcome: A pre-run execution outcome of fix_source_code. Follow Section 3.2 to know the potential list of outcomes. The exec_outcome flags in the training data comes from a pre-run environmentfrom the source website and they are not verified in ExecEval. However, training data doesn’t include unit-test to avoid potential hacks and confusion. We provide unit test for only Compact and Titan test data.
18. potential_dominant_fix_op: A potential fix op recommended by difflib.
19. lang_cluster: A generic programming language name the value of lang belongs to.
20. prob_desc_description: Problem description in textual format, math operations are written in latex.
21. prob_desc_input_from: How the program should take the unit test.
22. prob_desc_output_to: Where the program should output the result of the unit test.
23. prob_desc_time_limit: Time limit to solve the problem.
24. prob_desc_memory_limit: Memory limit to solve the problem.
25. prob_desc_input_spec: How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
26. prob_desc_output_spec: How the outputs should be printed. Most of the time the unit test results are matched with an exact string match or floating point comparison with a precision boundary.
27. prob_desc_sample_inputs: A sample input for the code that is expected to solve the problem described in description.
28. prob_desc_sample_outputs: The expected output for the sample_input that is expected to solve the problem described in description.
29. prob_desc_notes: Explanation of sample_inputs & sample_outputs.
30. prob_desc_created_at: The Unix timestamp when the problem was released. Use datetime lib in Python to parse it to a human-readable format.
31. file_name: Name of the source jsonl file from where data is loaded.
32. hidden_unit_tests: a list of unit tests returned as string. use json.loads(hidden_unit_tests) to load the data.
H.6 Code Translation (code_translation)
Given a source code (source_code) in lang_cluster, generate a code in target programming language.
1. lang: Runtime/Compiler version of the source_code.
2. source_code: A program.
3. code_uid: A unique ID for the sample. It is not important for model training. If you find any issue with the sample, you can report it to us by mentioning the code_uid.
4. src_uid: A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved.
5. difficulty: Difficulty rating of the problem indicated by src_uid. The higher the harder.
6. exec_outcome: Execution outcome status. Follow Section 3.2 to know the potential list of outcomes. The exec_outcome flags in the training data comes from a pre-run environment from the source website and they are not verified in ExecEval. However, training
data doesn’t include unit-test to avoid potential hacks and confusion. We provide unit test for only Compact and Titan test data.
7. lang_cluster: A generic programming language name the value of lang belongs to.
8. prob_desc_description: Problem description in textual format, math operations are written in latex.
9. prob_desc_input_from: How the program should take the unit test.
10. prob_desc_output_to: Where the program should output the result of the unit test.
11. prob_desc_time_limit: Time limit to solve the problem.
12. prob_desc_memory_limit: Memory limit to solve the problem.
13. prob_desc_input_spec: How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
14. prob_desc_output_spec: How the outputs should be printed. Most of the time the unit test results are matched with an exact string match or floating point comparison with a precision boundary.
15. prob_desc_sample_inputs: A sample input for the code that is expected to solve the problem described in description.
16. prob_desc_sample_outputs: The expected output for the sample_input that is expected to solve the problem described in description.
17. prob_desc_notes: Explanation of sample_inputs & sample_outputs.
18. prob_desc_created_at: The Unix timestamp when the problem was released. Use datetime lib in Python to parse it to a human-readable format.
19. file_name: Name of the source jsonl file from where data is loaded.
20. hidden_unit_tests: a list of unit tests returned as string. use json.loads(hidden_unit_tests) to load the data.
H.7 Program Synthesis (program_synthesis)
Given a src_uid read problem description from problem_descriptions.jsonl and generate a solution for problem description.
1. lang: Runtime/Compiler version of the source_code.
2. source_code: A program.
3. code_uid: A unique ID for the sample. It is not important for model training. If you find any issue with the sample, you can report it to us by mentioning the code_uid.
4. src_uid: A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved.
5. difficulty: Difficulty rating of the problem indicated by src_uid. The higher the harder.
6. exec_outcome: Execution outcome status. Follow Section 3.2 to know the potential list of outcomes. The exec_outcome flags in the training data comes from a pre-run environment. However, training data doesn’t include unit-test to avoid potential hacks. We provide unit tests for only dev and test data.
7. lang_cluster: A generic programming language name the value of lang belongs to.
8. prob_desc_description: Problem description in textual format, math operations are written in latex.
9. prob_desc_input_from: How the program should take the unit test.
10. prob_desc_output_to: Where the program should output the result of the unit test.
11. prob_desc_time_limit: Time limit to solve the problem.
12. prob_desc_memory_limit: Memory limit to solve the problem.
13. prob_desc_input_spec: How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
14. prob_desc_output_spec: How the outputs should be printed. Most of the time the unit test results are matched with an exact string match or floating point comparison with a precision boundary.
15. prob_desc_sample_inputs: A sample input for the code that is expected to solve the problem described in description.
16. prob_desc_sample_outputs: The expected output for the sample_input that is expected to solve the problem described in description.
17. prob_desc_notes: Explanation of sample_inputs & sample_outputs.
18. prob_desc_created_at: The Unix timestamp when the problem was released. Use datetime lib in Python to parse it to a human-readable format.
19. file_name: Name of the source jsonl file from where data is loaded.
20. hidden_unit_tests: a list of unit tests returned as a string. use json.loads(hidden_unit_tests) to load the data.
H.8 Retrieval Corpus (retrieval_corpus)
Use the retrieval_corpus to perform query for retrieval_nl_code (Appendix H.9) and retrieval_code_code (Appendix H.10).
1. idx: An integer index to identify the code. It is unique within the codes of each language.
2. source_code: A program.
3. file_name: Name of the source jsonl file from where data is loaded.
H.9 Retrieval NL-Code (retrieval_nl_code)
Given a NL (problem description) retrieve similar source code from retrieval_corpus (Appendix H.8).
1. nl : Problem description in textual format, math operations are written in latex. Given as input query.
2. positive_code : list of positive codes for nl.
3. negative_code : list of negative codes for nl.
4. src_uid : A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved.
5. file_name: Name of the source jsonl file from where data is loaded.
H.10 Retrieval Code-Code (retrieval_code_code)
Given a source_code, retrieve similar source code from retrieval_corpus (Appendix H.8).
1. positive_code : list of positive codes for nl.
2. negative_code : list of negative codes for nl.
3. src_uid : A specific identifier that shows which problem the code is associated with. This identifier is important for the training of the model. The problem referred to by the src_uid provides a natural description of the problem that the code successfully solved. Refer to Structure of problem_descriptions.jsonl.
4. source_code: A source code given as input query.
5. file_name: Name of the source jsonl file from where data is loaded.

heading: I Datasheets for Datasets
text: We follow the questionnaires from Gebru et al. (2021) as the datasheet for XCODEEVAL .

heading: I.1 Motivation
text: For what purpose was the dataset created? XCODEEVAL dataset was specifically created to address three main aspects: (i) Reasoning, (ii) Multilinguality in terms of programming languages, and (iii) Executability of the programming languages. These aspects were thoroughly discussed in Section 1 of the main paper, providing detailed insights into the motivation behind the dataset creation.

heading: I.2 Composition
text: What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Please follow the Appendix H for details.
How many instances are there in total (of each type, if appropriate)? Please follow the Tables 2, 8 and 9 for the details statistics of the dataset.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? Dataset contains all possible instances.
What data does each instance consist of? Please follow the Appendix H for details.

heading: Is there a label or target associated with each
text: instance? Please follow the Appendix H for details.
Is any information missing from individual instances? For a few problem description, difficulty is assigned as None due to data unavailability.
Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? Please follow the Appendix H for details.
Are there recommended data splits (e.g., training, development/validation, testing)? We explicitly defined the training, Compact and Titan test split for XCODEEVAL . Please follow the Section 3.1 for more details.
Are there any errors, sources of noise, or redundancies in the dataset? To the best of our knowledge there are no errors, sources of noise, or redundancies in XCODEEVAL .

heading: Is the dataset self-contained, or does it link to or
text: otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is selfcontained.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor– patient confidentiality, data that includes the content of individuals’ non-public communications)? The dataset is collected from open sourced sources. There are no confidentiality or non-public entity in the dataset.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? To the best of our knowledge there are no offensive, insulting, threatening content in the dataset.
Does the dataset identify any subpopulations (e.g., by age, gender)? No.

heading: Is it possible to identify individuals (i.e., one or more natural persons), either directly or indi-
text: rectly (i.e., in combination with other data) from the dataset? There are no attributes in the dataset that allow to identify individuals.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? There are no attributes in the dataset that allow this.

heading: I.3 Collection Process
text: How was the data associated with each instance acquired? Following Li et al. (2022), the data was collected from codeforces.com and then associated with different tasks. Please follow Appendix D for more details.

heading: If the dataset is a sample from a larger set,
text: what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Dataset wasn’t sampled from a large dataset. We proposed the dataset for the first time.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Dataset was collected by the author of this paper.
Over what timeframe was the data collected? The data was downloaded in between Feb, 2022 to January, 2023.
Were any ethical review processes conducted (e.g., by an institutional review board)? No.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? The data is downloaded by an author. No third parties or other sources are involved.

heading: Has an analysis of the potential impact of the
text: dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? Potential impact of the dataset is discussed after Section 6.

heading: I.4 Preprocessing/cleaning/labeling
text: Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? We de-anonymized the data and remove data with sensitive information (i.e., email, large infograph, toxic keywords). The labels come as a metadata from the sources.
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? No.
Is the software that was used to preprocess/clean/label the data available? No software was used for labeling the data.
Has the dataset been used for any tasks already? Yes. We evaluated ChatGPT and trained StarEncoder using the dataset.
What (other) tasks could the dataset be used for? We proposed 7 different tasks for XCODEEVAL . Please follow Table 8 for details.

heading: Is there anything about the composition of the dataset or the way it was collected and prepro-
text: cessed/cleaned/labeled that might impact future uses? No.

heading: I.5 Distribution
text: Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

heading: I.6 Maintenance
text: How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Email.
Is there an erratum? None at this point. The dataset is hosted through git lfs. The future erratum can be tracked easily.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? To the best of our knowledge there are no errors in the dataset. The authors do not intend to add new instances at this point. But the authors remain open to remove/correct instances given that any labeling errors found.

heading: If the dataset relates to people, are there applicable limits on the retention of the data associated
text: with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? The dataset doesn’t relate to people.
Will older versions of the dataset continue to be supported/hosted/maintained? Yes. The older version should be accessed via git LFS.

heading: If others want to extend/augment/build on/con-
text: tribute to the dataset, is there a mechanism for them to do so? Since the dataset is fixed, there is currently no way to contribute to it. Please note that any extensions or augmentations of the dataset are subject to the same license as this dataset.

heading: J Discussion on the possibility of data leakage and contamination
text: Although we completely separate our Compact and Titan test data, LLMs might have possible data leakage from pretraining. We find that even identifying data leakage (test data exists or not in the prertraining corpora) is challenging using conventional data search methods due to search cost & complexity (e.g., exact match or token overlapping methods) while hashing based searches suffer from not having properly segmented text. For leakagefree evaluation, we approach employs "knowledge cut-off" which show that the data contamination significantly impacts the model performance and it needs to be interpreted with proper analyses. We
plan to evaluate on seperate human written testset in future.

heading: K The Dataset Nutrition Label
text: We follow the framework proposed by Holland et al. (2018). Table 12 gives an overview of dataset facts. The variable description can be found in Appendix H. We discuss provenance in Appendices I.3 and I.6.
L Data Card

heading: Future Work
text: We
plan to evaluate on seperate human written testset in future