Large Language Models Meet NL2Code: A Survey
Year: 2023

Abstract:
The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are "Large Size, Premium Data, Expert Tuning". In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website https://nl2code.github.io to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.

heading: 1 Introduction
text: Is it possible for novice programmers, even those without any programming experience, to create software simply by describing their requirements in natural language? This is a long-standing fascinating question, which poses challenges to research areas like software engineering, programming language, and artificial intelligence. Realizing this scenario would have an unprecedented impact on our lives, education, economy, and labour market, as it would change the centralized software development and operation paradigm. Due to its
∗ This work was done before October 2022 when the author, Daoguang Zan, was an intern at Microsoft Research Asia.
promising and intriguing future, natural-languageto-code (NL2Code) has been proposed as a research task that has attracted widespread interest in both academia and industry, with the goal of generating code from natural language descriptions.
Early studies on NL2Code were mainly based on heuristic rules or expert systems, such as probabilistic grammar-based methods (Joshi and Rambow, 2003; Cohn et al., 2010; Allamanis and Sutton, 2014) and those focusing on domain-specific languages (de Moura and Bjørner, 2008; Gulwani, 2010; Jha et al., 2010), which are inflexible and not scalable. Other studies utilized static language models, like n-gram (Nguyen et al., 2013; Raychev et al., 2014; Devanbu, 2012) and Hidden Markov (Sutskever et al., 2008), which have sparse vector representations and cannot model long-term dependencies. Subsequently, neural networks, including CNN (Liu et al., 2016; Sun et al., 2018), RNN (Iyer et al., 2016; Wan et al., 2018), and LSTM (Eriguchi et al., 2016; Yin and Neubig, 2017), were employed to model the relationship between NL and code. In 2017, the Transformer (Vaswani et al., 2017) model was introduced for machine translation and later applied to the NL2Code task (Mastropaolo et al., 2021; Shah et al., 2021). However, these deep learning models require a significant amount of labelled pairs of NL and code for training, and have limited capabilities for the NL2Code task.
Recently, a growing number of large language models (LLMs) with Transformer architecture have been trained on large-scale unlabelled code corpus. These models have the ability to generate code in a zero-shot manner and have achieved impressive results in the NL2Code task. As a milestone, Codex (Chen et al., 2021) has shown that an LLM with 12 billion parameters is able to solve 72.31% of challenging Python programming problems created by humans. More encouragingly, Codex has been used to power a commercial
7443
product1 and improve coding efficiency in practice (Sobania et al., 2022a; Barke et al., 2023). Following Codex’s success, various LLMs for the NL2Code task have emerged, with model sizes ranging from millions to billions of parameters. Examples include AlphaCode (Li et al., 2022b), which aims to solve competitive-level programming problems, and InCoder (Fried et al., 2023), which supports filling code in arbitrary positions using bidirectional contexts. Other models such as CodeGen (Nijkamp et al., 2023), PaLM-Coder (Chowdhery et al., 2022), PanGu-Coder (Christopoulou et al., 2022), CodeGeeX (Zheng et al., 2023), and SantaCoder (Allal et al., 2023) have also gained great attention. As the model size increases, LLMs have been shown to exhibit some emergent capabilities such as human-like programming and debugging (Zhang et al., 2022; Saunders et al., 2022; Kang et al., 2023).
Large language models have kindled hope for the NL2Code task due to their impressive power and potential value. Despite the significant progress, there are still numerous challenges and opportunities, calling for more advanced and innovative future work. Currently, considering the variety of techniques and applications, there is a growing need for a comprehensive survey to provide a systematic overview of this field and identify critical challenges. To this end, in this paper, we carefully investigate 27 advanced LLMs for NL2Code (§2), and also review benchmarks and metrics (§4). We conduct an intuitive comparison of all the existing LLMs on the HumanEval benchmark, perform a thorough analysis, and eventually attribute the success of these LLMs to "Large Size, Premium Data, Expert Tuning" (§3). This
1https://github.com/features/copilot
means large model and data size, high-quality training data and expert hyper-parameter tuning. We also discuss the challenges and opportunities regarding the ability gap between LLMs and Humans (§5). In addition, we have built a website https://nl2code.github.io to keep track of the latest progress and support crowd-sourcing updates. To the best of our knowledge, this is the first survey of LLMs for NL2Code2, and we hope it will contribute to the ongoing development of this exciting field.

heading: 2 Large Language Models for NL2Code
text: Given a natural language problem description, the NL2Code task aims to automatically generate the demanded code. To illustrate this task visually, we provide a Python programming problem as an example in Figure 1, while different NL2Code benchmarks may vary in terms of language or
2We summarize the related surveys in Appendix A.
problem domain. Existing large language models for the NL2Code task are usually based on Transformer (Vaswani et al., 2017) and are trained on a large-scale code related unlabelled corpus. For better code generation performance, most LLMs, no matter encoder-decoder or decoder-only models, employ the causal language modeling objective for training, which is to predict the token following a sequence of tokens. During inference, an LLM can tackle NL2Code problems in a zero-shot manner without fine-tuning its parameters. There are also studies employing few-shot (Austin et al., 2021) or in-context learning (Nijkamp et al., 2023) to further boost the performance.
We conduct a comprehensive investigation of 27 representative LLMs for the NL2Code task. Details of each model are summarized in Table 1, where models vary in architecture, size, and accessibility. For better visualization, we present these models in chronological order in Figure 2, plotting the largest model sizes. One trend observed is that these large language models are consistently growing in size as the research field advances. Additionally, the decoder-only architecture is favoured for pre-trained models with larger sizes.
Early works, such as GPT-C (Svyatkovskiy et al., 2020), PyMT5 (Clement et al., 2020), and PLBART (Ahmad et al., 2021), have relatively small numbers of parameters and do not demonstrate strong capabilities in zero-shot code generation. Conversely, large-scale models such as GPT-
Neo (Black et al., 2021) and GPT-J (Wang and Komatsuzaki, 2021), despite their billion-level parameter scale, have been found to have limited power in the NL2Code task due to the small amount of code in their training corpus. Recently, a number of powerful LLMs have been proposed for NL2Code, such as Codex (Chen et al., 2021), AlphaCode (Li et al., 2022b), and PaLM-Coder (Chowdhery et al., 2022), which possess massive parameter scales and high-quality training corpus with code. While they show surprisingly good performance on NL2Code, most of them are not readily accessible. At present, a number of excellent open-source models have also been proposed, including CodeParrot (Huggingface, 2021), PolyCoder (Xu et al., 2022), GPT-NeoX (Black et al., 2022), and SantaCoder (Allal et al., 2023), which contribute to the thriving of LLMs for NL2Code. Besides, recent studies have proposed various approaches to address specific NL2Code scenarios. For example, JuPyT5 (Chandel et al., 2022a) is designed to work within Jupyter Notebooks, while ERNIECode (Chai et al., 2022), CodeGeeX (Zheng et al., 2023), and BLOOM (Scao et al., 2022) are trained to support multiple natural or programming languages. Additionally, InCoder (Fried et al., 2023), FIM (Bavarian et al., 2022), and SantaCoder (Allal et al., 2023) not only support left-to-right code prediction, but also allow for infilling arbitrary regions of code. As LLMs for NL2Code are evolving rapidly, we created a website to keep up-to-date
with the latest advances by crowd-sourcing. Details of the website can be found in Appendix B.
These models are not only attractive in academia (Chen et al., 2021; Nijkamp et al., 2023; Li et al., 2022b), but also applied in real-world products to improve programming efficiency (Sobania et al., 2022a; Barke et al., 2023). One example is GitHub and OpenAI’s Copilot, a programming assistance tool that utilizes Codex to provide realtime code suggestions. Other notable products include CodeGeeX3 and CodeWhisperer4. A summary of 10 products can be found in Appendix Table 5. Recent studies (Sobania et al., 2022b; Pearce et al., 2022; Nguyen and Nadi, 2022) have shown that these products can provide helpful recommendations, while they also introduce minor bugs that can cause issues for users. There is still room for improvement before LLMs can be fully practical and capable of coding like humans.

heading: 3 What makes LLMs successful?
text: We have summarized the existing large language models for NL2Code. These LLMs vary in terms of architecture, size, and other characteristics, making it difficult to establish a completely fair comparison. We evaluate these LLMs on the HumanEval benchmark (Chen et al., 2021) in a zero-shot manner to provide an intuitive comparison. HumanEval, proposed along with Codex, is one of the most popular benchmarks for the NL2Code task and consists of 164 hand-written Python programming problems. Test cases are provided for each programming problem to evaluate the correctness of generated code. pass@k is used as the evaluation metric5, which calculates the proportion of problems that can be correctly answered with k tries. Table 2 shows the results of different LLMs organized by the model size. Implementation details and the evaluation on the MBPP benchmark (Austin et al., 2021) can be found in Appendix C.2.
It can be observed from Table 2 that the performance of existing LLMs varies widely on HumanEval, even for those with similar model sizes. Specifically, Codex (Chen et al., 2021) holds the leading position in various model sizes, while a relatively small model, PyCodeGPT 110M (Zan et al., 2022b), achieves comparable results to Codex 85M. Other larger models such as Alpha-
3https://keg.cs.tsinghua.edu.cn/codegeex 4https://aws.amazon.com/cn/codewhisperer 5The details of pass@k can be found in Appendix C.1.
Code (Li et al., 2022b), CodeGen-Mono (Nijkamp et al., 2023), and PanGu-Coder (Christopoulou et al., 2022) also exhibit impressive performance. Notably, InCoder (Fried et al., 2023) and SantaCoder (Allal et al., 2023), which use the FIM training method (Bavarian et al., 2022), also obtain remarkably decent results in the left-to-right generation setting. The significant variation in performance leads us to the question: What makes LLMs successful in NL2Code? Given the diversity of these models in terms of design choices, we perform a thorough analysis and conclude the answer: Large Size, Premium Data, Expert Tuning. That is, large model and data size, high-quality data and expert hyper-parameter tuning are the key factors for the success of LLMs in the NL2Code task. In this section, we detail our observations and insights from the perspectives of model, data and tuning.

heading: 3.1 Large Model Size
text: As shown in Figure 2 and Table 2, recent LLMs for NL2Code exhibit larger sizes and superior performance. This is consistent with prior findings that an increased number of model parameters can enhance model capabilities (Radford et al., 2019; Thoppilan et al., 2022; Chowdhery et al., 2022). We further demonstrate the correlation between model size and performance in Figure 3a, which compares the pass@1 results of 10 representative models on the HumanEval benchmark. It is clear that larger models generally result in better performance. Furthermore, we also find that current models, regardless of size, still have the potential for improvement through further increases in size. Additional results on the HumanEval and MBPP benchmarks can be found in Appendix Figure 7,
which also support this conclusion. Additionally, we conduct an experiment on the HumanEval benchmark to examine the syntax error rates of the code generated by different models of varying sizes. Specifically, we make the models predict 10 code samples for each programming problem, and then calculate the percentage of code samples that have syntax errors. As shown in Figure 3b, results indicate that larger models tend to have lower syntax error rates. It is noteworthy that the largest version of the CodeGen-Mono model exhibits a remarkably low rate of syntax errors, i.e., 6%. However, as evidenced by Figure 3a and Table 2, the CodeGen-Mono model with 16 billion parameters still has unsatisfactory performance in terms of pass@k , e.g., pass@1 to be 29%. This highlights the fact that the current limitation for large pre-trained models is the generation of semantically correct code.

heading: 3.2 Large and Premium Data
text: As the sizes of LLMs increase in the field of NL2Code, the scale of the corpus used for training also increases. This highlights the importance of selecting and pre-processing high-quality data. In this section, we will discuss various commonly used data sources and pre-processing strategies that are essential for training LLMs.
Early models were trained using manually annotated data pairs of NL and code, and the data sources include CodeSearchNet (Husain et al., 2019), CoST (Zhu et al., 2022b), and XLCoST (Zhu et al., 2022a). However, manual annotation is labour-intensive and time-consuming. There are also models like GPT-3 (Brown et al., 2020), GPT-Neo (Black et al., 2021), and GPT-
J (Wang and Komatsuzaki, 2021) that are trained on the Pile (Gao et al., 2020), a large-scale unsupervised dataset. However, these models have not yet demonstrated exceptional code generation capabilities due to the limited number of code files in the training corpus. More recently, with the emergence of more powerful LLMs for NL2Code, larger-scale unlabelled code datasets have been proposed, including BigQuery (Google, 2016), CodeParrot’s corpus (HuggingFace, 2021a), GitHub-Code (HuggingFace, 2021b), and the Stack (HuggingFace, 2022), which are collected from general domain open-source websites like GitHub6 and Stack Overflow7. Furthermore, there are also specialized datasets proposed for different scenarios, for example, using Jupyter Notebooks or competition programming problems as a training corpus. Released datasets include Jupyter (HuggingFace, 2021c), JuICe (Agashe et al., 2019), APPS (Hendrycks et al., 2021), and CodeNet (IBM, 2021).
In order to ensure the quality of the training corpus, it is common for LLMs to perform data preprocessing on the significant amount of code in the collected data. We carefully review the data pre-processing methods of five powerful LLMs, including Codex (Chen et al., 2021), AlphaCode (Li et al., 2022b), CodeGen (Nijkamp et al., 2023), InCoder (Fried et al., 2023), and PyCodeGPT (Zan et al., 2022b), and identify several commonalities. One is the removal of likely auto-generated or unfinished code files, as they are deemed to be meaningless. Additionally, specific rules are employed to filter out uncommon code files. These rules include factors such as the repository star rating, the file size, the line length, and the alphanumeric rate. In summary, the goal of these pre-processing strategies is to achieve a code corpus that is unduplicated, complete, correct, clean, and general in nature.

heading: 3.3 Expert Tuning
text: Training an excellent model requires careful consideration of various design choices and hyperparameters. After reviewing the existing 27 LLMs (summary in Appendix Table 6), we have the following findings. Firstly, these LLMs share some common settings. For example, we observe that the optimizer of the current models is almost all Adam (Kingma and Ba, 2014) or its variants (Loshchilov and Hutter, 2017). We also
6https://github.com 7https://stackoverflow.com
find that initializing with other natural language models yields no noticeable gain compared to training from scratch, except for accelerating convergence (Chen et al., 2021). Furthermore, there are several hyper-parameters that require expert tuning, such as learning rate, batch size, window size, warmup steps, gradient accumulation steps, and sampling temperature. For the learning rate, we analyze its correlation with model size using six powerful LLMs, as shown in Figure 4. We observe that the learning rate becomes smaller as the model gets larger. To explore the effects of temperature, in Figure 5, we report the performance of two models using multiple temperatures on HumanEval. One observation is that higher temperature leads to lower pass@1 and higher pass@100, which suggests that a higher temperature makes LLMs generate more diverse predictions and vice versa. Besides, some studies (erman Arsenovich Arutyunov and Avdoshin, 2022) have shown that window size is a key factor. An interesting finding is that the
small model with a large window size sometimes outperforms the large model with a small window size (details in Appendix D). In addition, powerful LLMs usually train a new tokenizer on code corpus primarily using two techniques: Byte-level Byte-Pair-Encoding (Radford et al., 2019) and SentencePiece (Kudo and Richardson, 2018). A new tokenizer can be more effective and accurate in splitting code content into tokens. These proven tuning techniques will serve as valuable references for training more powerful LLMs.

heading: 4 Benchmarks and Metrics
text: To evaluate the NL2Code task, high-quality benchmarks and reliable metrics are fundamental and essential. In this section, we provide a brief overview of current benchmarks and metrics, as well as our observations and the open challenges.
We summarize 17 well-studied NL2Code benchmarks in Table 3, where we can find that each of these benchmarks has its own characteristics regarding size, language, complexity, and scenario. We observe that most benchmarks contain a limited number of instances. For example, the widely used HumanEval and MBPP have 164 and 974 instances, respectively. This is because these benchmarks are typically hand-written to ensure that LLMs have not seen them during training. In the era of large language models, it is crucial to avoid data leak-
age when creating new benchmarks. Additionally, most current benchmarks have their problem descriptions in English and code solutions in Python. Recently, several multi-lingual benchmarks have been proposed, such as MBXP (Athiwaratkun et al., 2022), HumanEvalX (Zheng et al., 2023), and MultiPL (Cassano et al., 2022), which cover multiple programming languages, and ODEX (Wang et al., 2022c), which covers multiple natural languages. Details of multi-lingual benchmarks are listed in Appendix Table 7. Furthermore, benchmarks have been proposed for other practical scenarios, such as data science (Lai et al., 2022), public library (Zan et al., 2022b), private library (Zan et al., 2022a), multi-turn program synthesis (Nijkamp et al., 2023), and code security (Siddiq and msiddiq, 2022). For execution-based benchmarks, comprehensive test cases with complete coverage of the generated program can ensure the trustworthiness of evaluation results. As a reference, the average number of test cases for each benchmark, as well as the length statistics of the problem descriptions and solutions are also provided in Table 3.
Manually evaluating the generated code is impractical, which calls for the need for automatic metrics. The above mentioned benchmarks all provide test cases for execution-based evaluation, where metrics such as pass@k (Chen et al., 2021), n@k (Li et al., 2022b), test case aver-
age (Hendrycks et al., 2021), and execution accuracy (Rajkumar et al., 2022) can be used. However, this approach has stringent requirements for the quality of test cases and can only evaluate executable code. For non-executable code, metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and CodeBLEU (Ren et al., 2020) are used, while they can not precisely evaluate the correctness of the code. So far, there are many open challenges in designing metrics to evaluate various aspects of code, such as vulnerability, maintainability, clarity, execution complexity, and stability.

heading: 5 Challenges and Opportunities
text: Our investigations have revealed that advances in LLMs for NL2Code have a considerable impact on both academia and industry. Despite this progress, there are still numerous challenges that need to be addressed, offering ample opportunities for further research and applications. In this section, we explore the challenges and opportunities in terms of the ability gap between LLMs and humans.
Understanding Ability The inherent flexibility of natural language allows for a variety of expressions to convey functional requirements. Humans are able to understand various descriptions at different levels of abstraction. In contrast, current LLMs tend to be sensitive to the given context, which may cause unexpected performance degradation (Wang et al., 2022a). In addition, LLMs may struggle when faced with complex problems that have numerous conditions and requirements (Barke et al., 2022; Imai, 2022). We believe exploring the understanding abilities of LLMs is a crucial research direction. One potential solution is to break down complex problems into multiple steps, as is commonly done in reasoning tasks (Wei et al., 2022).
Judgement Ability Humans have the ability to determine whether they can solve a programming problem or not. While current models will always return a solution even if there is no answer to the problem, due to the fact that they are trained by unsupervised causal language modeling objective. This can cause problems in practical applications. To improve the judgment ability of LLMs, researchers have employed reinforcement learning to leverage user feedback, as seen in models like InstructGPT (Ouyang et al., 2022) and ChatGPT8. However, collecting high-quality feedback for code
8https://chat.openai.com
is costly and challenging. There are also ongoing studies (Chen et al., 2023; Key et al., 2022) exploring the possibility of self-validation for LLMs, which is also a promising research direction.
Explanation Ability It is widely acknowledged that human developers possess the ability to interpret the meaning of the code they write, which is crucial for educational purposes and software maintenance. Recent studies showed that LLMs have the potential to automatically generate code explanations. MacNeil et al. (2022a) proposed using LLMs to generate code explanations for students during their learning process, and MacNeil et al. (2022b) proposed explaining numerous aspects of a given code snippet using Copilot. Further research and explorations are necessary to fully realize the potential of LLMs in this regard.
Adaptive Learning Ability A fundamental difference between current large language models and humans is their ability to adapt to new and updated knowledge. Human developers possess a unique ability to quickly search and learn new materials, such as programming documentation, and adapt to changes in APIs with relative ease. However, re-training or fine-tuning LLMs requires significant effort and resources. This issue has inspired a number of recent studies, such as DocCoder (Zhou et al., 2023) and APICoder (Zan et al., 2022a), which utilize retrieval-based methods to provide extra or updated knowledge during model inference. Despite these advancements, it remains an open challenge to endow LLMs with the powerful learning capabilities humans possess.
Multi-tasking Ability Large language models have been applied to a variety of code-related tasks, such as code repair (Joshi et al., 2022; Prenner and Robbes, 2021), code search (Neelakantan et al., 2022), and code review (Li et al., 2022c) as well as non-code tasks that can be formatted in a code-like manner, such as mathematics (Drori and Verma, 2021; Drori et al., 2021) and chemistry (Krenn et al., 2022; Hocky and White, 2022). However, there are differences between LLMs and human abilities in terms of multi-tasking. Humans can seamlessly switch between tasks, while LLMs may require sophisticated prompt engineering (Liu et al., 2023). Another evidence is that LLMs lack the ability to quickly master multiple programming languages (Zheng et al., 2023) as humans do. These limitations highlight areas for future research.

heading: 6 Conclusion
text: In this paper, we survey 27 existing large language models for NL2Code, and draw a thorough analysis of the underlying reasons for their success. We also provide a detailed review of benchmarks and metrics. Regarding the gap between models and humans, we present ongoing challenges and opportunities. In addition, we have developed a website to track the latest findings in this field. We hope this survey can contribute to a comprehensive overview of the field and promote its thriving evolution.
Limitations
In this paper, we thoroughly investigate the existing large language models for NL2Code, and summarize them from diverse perspectives with our own thinking. However, as this field is evolving so rapidly, there may be aspects that we have overlooked, or some new works that we have not covered. To mitigate this issue, we have created a website to track the latest progress through crowdsourcing, hoping that it will continually contribute to the development of the field. Besides, the existing LLMs possess their own characteristics in terms of model size, architecture, corpus, pre-processing, tokenizer, hyper-parameters, and training platforms. Also, some of them are currently not publicly available, such as AlphaCode (Li et al., 2022b) and PaLM-Coder (Chowdhery et al., 2022). Therefore, it is almost impractical to conduct a completely fair comparison. We tried our best to show a kind of comparison on the popular HumanEval and MBPP benchmarks, hoping that it can provide clues to the differences in performance of different LLMs. In addition, evaluating LLMs has a high cost in computational resources. We thus have made all files generated by the LLMs publicly available on https://nl2code.github.io.

heading: A Related Surveys
text: Previous surveys on the topic of code intelligence (Allamanis et al., 2018; Le et al., 2020; Li et al., 2022a; Xu and Zhu, 2022) and code generation (Pawade et al., 2018; Shin and Nam, 2021; Dehaerne et al., 2022) have primarily focused on early methodologies such as the use of programming templates (Syriani et al., 2018; Luhunu and Syriani, 2017), neural models based on CNN, RNN, and LSTM architectures (Allamanis et al., 2018; Sharma et al., 2021), and small-scale Transformer models that require labelled data for training (Mastropaolo et al., 2021; Shah et al., 2021). However, with the advancement of model size, Transformerbased models have demonstrated exceptional performance in NL2Code tasks and have given rise to the development of more capable code generation models. In light of this, there exists a clear need for a comprehensive survey of large language models for NL2Code tasks to bridge this gap in knowledge. This study endeavours to fulfill this need by providing a thorough analysis of the successful LLMs and a detailed review of NL2Code benchmarks and metrics. We also present the ongoing challenges and opportunities regarding the ability gap between LLMs and humans.
Finally, we would like to highlight some criteria for our survey. First, we only refer to official papers to investigate the size of the models. For example, Codex reported the model with a maximum size of 12B in the paper, but later trained larger ones. In this case, we only consider the 12B model as the largest one. In addition, the publication dates of the models in Figure 2 are taken from official papers or blogs.

heading: B An Online Website
text: To keep tracking the latest progress of LLMs for NL2Code, we have developed an online real-time update website at https://nl2code.github.io. We have collected as many of the latest research works as possible on this website. Everyone is allowed to contribute to the website by pulling requests on GitHub. This website also includes features such as fuzzy search and custom tag categories, which will facilitate researchers to find the papers they want quickly. We hope this website can assist researchers and developers in related fields and contribute to its advancement.

heading: C Experimental Setup
text: In this section, we will first present the definition of pass@k , followed by the details of the experiments conducted on two benchmarks, namely HumanEval (Chen et al., 2021) (results in Table 2) and MBPP (Austin et al., 2021) (results in Table 4).
C.1 Definition of pass@k
We use pass@k as our metric for evaluation. For each programming problem, we sample n candidate code solutions and then randomly pick k of them. If any of the k code solutions pass the given test cases, the problem can be regarded as solved. So pass@k is the proportion of solved problems in the benchmark (Chen et al., 2021). Formally,
assuming that the number of correct ones in k samples is c, pass@k = 1 if n − c < k; otherwise, pass@k = 1 − ∏ni=n−c+1(1 − k/i). We chose pass@k as our primary evaluation metric because it offers a completely precise evaluation of code accuracy by executing test cases, while other metrics mentioned in Section 4 either originate from pass@k or have lower precision.
C.2 Implementation Details
For HumanEval, we use the original benchmark9. Most results in Table 2 are taken from the original papers, while we reproduce the results of GPT-CC, PLBART, CodeT5, and InCoder 1.3B by strictly following the same experimental setup as the other models. In detail, we set the sample number to 200, the maximum length of newly generated tokens to 200, and top_p to 0.95. We set the temperature from 0.1 to 1.0 with an interval of 0.1, and report the best performance across these temperatures.
For MBPP, we use the version from Chen et al. (2023)10. In Table 4, the results of InCoder 6.7B and models larger than 10B are taken from Chen et al. (2023), while we reproduced other results. Specifically, we set the sample number to 100, the maximum length of newly generated tokens to 200, top_p to 0.95, and the temperature to 0.8.
For the two benchmarks above, we employ the same post-processing strategy. Following Codex (Chen et al., 2021), we terminate the sampling process when one of the following sequences is encountered in the generated code: ‘\nclass’, ‘\ndef’, ‘\n#’, ‘\n@’, ‘\nif’, and ‘\nprint’. In our experiments, CodeT5 770M refers to the version11 with the causal language modeling objective. For good reproducibility and further research, we have made our code and the generated results of the LLMs on HumanEval and MBPP publicly available on our website.

heading: D Context Window vs. Performance
text: Recent work (erman Arsenovich Arutyunov and Avdoshin, 2022) claimed that the size of the context window plays a vital role in enhancing the performance of LLMs for NL2Code. Specifi-
9https://github.com/openai/human-eval/blob/ master/data/HumanEval.jsonl.gz
10https://github.com/microsoft/CodeT/blob/ main/CodeT/data/dataset/mbpp_sanitized_for_code_ generation.jsonl
11https://huggingface.co/Salesforce/ codet5-large-ntp-py
cally, experiments are conducted on the APPS benchmark (Hendrycks et al., 2021) with GPTNeoX (Black et al., 2022), and we visualize the results in Figure 6. It is found that the 165M version model with an 8, 000 context window is comparable to the 20B version model with a 2, 000 context window. This observation illustrates that the context window also needs to be considered when training the model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 7: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 0: Abstract 6: Conclusion
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Appendix C: Experimental Setup

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix C: Experimental Setup
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4: Benchmarks and Metrics
C 3 Did you run computational experiments? 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix C: Experimental Setup

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix C: Experimental Setup
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: These limitations highlight areas for future research