ReCode: Robustness Evaluation of Code Generation Models
Year: 2023

Abstract:
Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model’s robustness performance. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.

heading: 1 Introduction
text: Code generation has emerged as an important AI application. Multiple models (Nijkamp et al., 2022;
‡Corresponding author. *Equal contribution. †Work done while the authors were at Amazon.
Fried et al., 2022; Wang and Komatsuzaki, 2021) have been proposed and achieved impressive performance on generating code using a natural-language description, on completing partial lines and functions, and even on solving complex coding-contest problems. They can offer real-life help to software engineers and enhance their productivity, and multiple commercial offerings exist today for AIpowered code generation (Chen et al., 2021).
However, one important aspect, robustness of the code generation models, is commonly overlooked. Anecdotally, people know that these models are sensitive to perturbations over prompts: sometimes just an extra space in a line or a slight change to a function name would lead to completely different generations, with potentially negative impacts to usability. In Fig. 1 and Fig. 2, we show two failure cases on InCoder-6B (Fried et al., 2022) and CodeGen-16B-mono (Nijkamp et al., 2022) where they perform correctly on regular prompts but fail on our perturbed ones after docstring paraphrasing and function camel case renaming in our ReCode benchmark. The perturbed prompts are natural and retain the original meaning, indicating weakness of these models if deployed in real-life applications.
There exists no comprehensive and quantitative robustness benchmark for code generation models. Li et al. (2022) includes a brief study on robustness but it has limited perturbation types and is in a setting with massive numbers of samples, unrealistic in practice. Other existing works on robustness in text or code tasks have focused on classification and are not directly applicable to code generation (Zhang et al., 2020; Jha and Reddy, 2022).
In this paper, we present ReCode, a Robustness Evaluation framework for Code, aiming to provide comprehensive assessment for robustness of code generation models. ReCode includes only transformations that (1) appear naturally in practice and (2) preserve the semantic meaning of the original inputs. We carefully collect and customize a com-
13818
prehensive list of natural transformations on docstrings, function and variable names, code syntax, and code format, providing multifaceted assessments of a model’s robustness performance. We verify the quality of the perturbed data using both human evaluation and objective similarity scores. We take advantage of the fact that executing the generated code can serve as objective evaluation and define three robustness evaluation metrics that aggregate a model’s correctness across randomized transformations and transformation types. These metrics quantify a model’s accuracy on perturbed prompts, its relative accuracy drop from original prompts, as well as its general instability.
We summarize our contributions below:
• We present the first robustness evaluation benchmark ReCode for code generation tasks. Our evaluation framework is general and can be easily extended to any code generation datasets and models. 1
• We collect and customize over 30 natural transformations from the aspects of docstrings, function and variable names, code syntax, and code format. Human evaluation shows that most of the perturbed prompts do not alter the semantic meaning and that their level of naturalness is close to the originals. Quantitative
1Code and datasets released at https://github.com/ amazon-science/recode.
similarity metrics confirm the same.
• We propose robustness evaluation metrics for code-generation tasks: Robust Passs@k, Robust Drops@k, and Robust Relatives@k.
• We demonstrate the ReCode benchmark on HumanEval and MBPP datasets and present extensive empirical robustness comparisons on state-of-the-art models including CodeGen, InCoder, and GPT-J across different sizes. We find that 1) diverse pretraining corpus and larger model size can help improve the model worst-case robustness, but models may learn to generalize in a non-robust way; 2) code generation models are most sensitive to syntax perturbations; 3) due to diversity, MBPP poses greater changes than HumanEval.

heading: 2 Related Work
text: Robustness for NLP. Recent research have identified the severe robustness problem in Large Language Models (LLMs) using adversarial examples. For example, LLMs can be easily fooled by synonym replacement (Jin et al., 2020; Zang et al., 2020). To better illustrate the severity of adversarial robustness problems for NLP models, existing works (Nie et al., 2020; Gardner et al., 2020; Kiela et al., 2021; Wang et al., 2021a) build robustness benchmarks, which encourage people to further build robust and trustworthy models. Zhang
et al. (2020) presents a comprehensive overview of works in this field. Most existing works in this field focus on classification tasks rather than generation tasks. The main challenge for benchmarking robustness over generation tasks is that the evaluation of text generation is highly subjective and is usually hard to quantify. However, code generation provides a special opportunity because we can do objective and quantitative evaluation on generated codes, and code generation models use similar model architecture as NLP models.
Robustness for code. There are a series of previous work on different aspects of robustness problems for code. Specifically, Bielik and Vechev (2020) studies the adversarial robustness problem for type inference in programming languages. Yang et al. (2022) focuses on improving the naturalness of adversarial examples in code vulnerability prediction, clone detection and authorship attribution. Zhou et al. (2022) focuses on the adversarial robustness problems of source code comment generation and (Jha and Reddy, 2022) focuses on code translation, repair and summarization. These papers mainly focus on proposing attack and defense methods for different tasks in code domain, but there is no previous work on a comprehensive robustness benchmark for code generation domain.
Code generation. Code generation, also known as program synthesis, is a task of generating code based on natural language statements or code from context. Researchers have adapted transformerbased large language models to the code generation field. Various architectures have been explored: For example, CodeBERT (Feng et al., 2020), PLBART (Ahmad et al., 2021), CodeGPT (Lu et al., 2021) explore BERT, BART and GPT architectures for language models pretrained on code corpus. There are also works that propose to incorporate code structures for models to better understand the semantic information, including GraphCodeBERT (Guo et al., 2021) and CodeT5 (Wang et al., 2021b). Most recently, models with much larger size (i.e., billion-scale parameter numbers) are shown to significantly improve the performance on code generation benchmarks. Codex-12B (Chen et al., 2021) and CodeGen-16B (Nijkamp et al., 2022) are two representative very large pretrained code generation models and have established new state of the arts. However, few works have systematically explored robustness in code generation.

heading: 3 Methodology
text: In this section, we introduce the transformations to perturb prompts on both text (docstring) and code. We then propose new robust evaluation metrics.

heading: 3.1 Problem Formulation
text: We consider the end-to-end model-based code generation task. The input prompt can include natural language statements that describe the functionality, signature of the function to generate, helper functions, and possibly a half-written function. The goal is left-to-right generation that creates or completes the function. This setting is agnostic to model architectures and is applicable to encoderdecoder or decoder-only models.
We perturb the input prompt with transformations. We focus on natural transformations that preserve the semantic meaning of the original prompt and that are likely to appear in practice, e.g., frequent typos in docstrings, tab to four spaces, function name style changes, and many more. We do not consider adversarial attacks that require model feedbacks in this paper because it is non-trivial to control the naturalness of adversarial attacks and they often require higher computational cost. Instead, we randomly generate perturbed prompts based on the restrictions for each type of perturbations and propose new metrics to evaluate model robustness based on these prompts. We leave adversarial attacks for future work.

heading: 3.2 Natural Transformations on Docstrings
text: Docstring describes the target function to generate. Since docstrings can vary greatly when written by different users, robustness against changes in docstrings is critical for usability in applications.
For docstrings, we use the NLAugmenter (Dhole et al., 2021) library which is designed for data augmentation and robustness evaluation on text.2 We carefully select ten transformations, including character-level, wordlevel and sentence-level ones, that are likely to preserve semantic similarity. The selected perturbations include CharCaseChange, where random characters are replaced with their upper cases, SynonymSubstitution, where random words are substituted with their WordNet synonyms (Miller, 1992), BackTranslation, where sentences are translated to a different language (e.g., German by default) then back to English for paraphrasing the
2 https://github.com/GEM-benchmark/NL-Augmenter
whole sentence (Li and Specia, 2019; Sugiyama and Yoshinaga, 2019), and more. To perform perturbations, we extract docstring sentences from the input prompt and then put the perturbed version back to the prompt. See Appendix A for details.
We observe that directly applying NLAugmenter to docstrings without constraints can potentially lead to low quality due to keywords in the programming languages. For example, "Create a list a[][]" could be perturbed by "Create a list [a][]" by character case swap, which is not natural. Therefore, to guarantee naturalness of perturbations, we use tree-sitter to parse the whole code snippet (the prompt & the canonical solution) to extract any existing function names, variable names ("a"), and type names ("list"). We then exclude them from being perturbed by the transformations. In Tab. 1, we list all ten transformations that are customized from NL-Augmenter and are included in our robustness benchmark along with sample illustrations.

heading: 3.3 Natural Transformations on Function Names
text: Perturbing function names also results in performance drops for code generation models. We summarize our perturbations in Tab. 2.
Some perturbations switch function names between naming conventions. For example, the perturbation called CamelCase transform function names between camel-case (e.g., "findCharLong") and snake-case ("find_char_long").
Other perturbations apply character-level or word-level natural text transformations on component words in a function name, including ChangeCharCase, InflectionalVariation, and SynonymSubstition as discussed in Sect. 3.2.

heading: 3.4 Natural Transformations on Code Syntax
text: Code generation models are often used on function completion task where the prompt includes a partial
implementation of the target function and the goal is to complete it. In such scenarios, the partial
code in prompt is work in progress and can be subject to frequent editing, and ideally a model should be robust with respect to perturbations in the partial code. For this evaluation, we derive new customized datasets from HumanEval and MBPP by adding half3 of the canonical solutions to the prompts (Fig. 3a). Then we perturb such partial code inside prompts. Details and examples for each perturbations can be found in Appendix A.
Transformations on partial code must be syntactically correct and must not alter semantic meaning. The next section will address code format, and let us first focus on code refactoring: these are syntactic changes that are semantically invariant.
We adopt three transformations from NatGen (Chakraborty et al., 2022): (1) Deadcode Insertion where dummy loops (0 iterations) or if conditions are randomly inserted; (2) Operand Swap where we randomly swap one operation (e.g., a<b to b>a); (3) For-While Switch where we randomly transform one for-loop structure in code to equivalent while-loop structure and vice versa.
Additionally, we implement three different schemes of variable renaming. We select the most frequent variable in the partial code and replace it using: (1) using CodeBERT (Feng et al., 2020) predictions with highest aggregated scores according to the context around all its appearance, a method inspired by (Jha and Reddy, 2022; Li et al., 2020), (2) using NatGen style renaming as "VAR_0", and (3) random name generation with half alphabetic and half numeric characters. The first strategy tends to provide more natural variable names, yet names from the other two strategies are also plausible.

heading: 3.5 Natural Transformations on Code Format
text: A natural way to perturb partial code is by code format transformations as they preserve the original semantic meaning. We implement following code format transformations in ReCode.
Newline Insertion: We consider three methods of new line insertions: (1) empty lines at randomly selected positions, (2) an empty line inserted between docstring and partial code, and (3) an empty line inserted after partial code.
Tab-Indent: We randomly replace any space indent with tab or replace tab with 4 spaces for indent-sensitive languages like Python.
Line Split: We select the longest line of code and split it into two lines in the middle.
3add first ⌊k/2⌋ lines given a k-line canonical solution.
Docstrings to Comments: We convert docstrings to comments (e.g., """ docstring """ to # docstring for Python).

heading: 3.6 Evaluation Metrics
text: Many proposed transformations are randomized operations. Hence, we need to measure model robustness over multiple samples to reduce variance. Specifically, for each transformation and each prompt, we create s randomly perturbed prompts. The model under evaluation generates outputs for each of them. We measure the worst-case performance across each group of s perturbed prompts: the model is considered robust on a prompt if and only if it generates a correct solution for all s perturbed prompts, where correctness is measured by executing associated unit tests.
Based on such worst-case measurements, we propose three new metrics for robustness evaluation.
Robust Passs@k (RPs@k): Pass@k is a widely used metric for measuring the performance of code generation tasks (Chen et al., 2021). We extend its definition to Robust Passs@k (RPs@k) with s random perturbations. For an original prompt x and for each transformation, let the perturbed prompts be x1, · · · , xs. We sample n generations by the model for each prompt, and in total there are n · s generations fi(xj), where 1 ≤ i ≤ n and 1 ≤ j ≤ s. Instead of regular pass@k, we first consider the worst-case correctness across fi(x1), ..., fi(xs) for 1 ≤ i ≤ n: Let ci,s(x) = 1 if fi(x1), ..., fi(xs) are all correct and ci,s(x) = 0 otherwise. Let rcs(x) =∑n
i=1 ci,s(x). Following definition of pass@k, we define the RPs@k metric as Eq. (1).
RPs@k := Ex [ 1− ( n−rcs(x) k ) ( n k ) ]
(1)
Robust Drops@k (RDs@k): RPs@k directly measure worst-case robustness in absolute values. It provides a worst-case estimation for models under certain perturbation. But in some applications, users may care more about relative performance change to compare worst-case performance and average-case performance. We propose Robust Drops@k defined in Eq. (2) as another important robustness metric to quantify relative changes.
RDs@k := Pass@k − Robust Passs@k
Pass@k (2)
Robust Relatives@k (RRs@k): Lastly, there are cases where models generate incorrect code on
original prompts yet predict correctly on perturbed ones. This can (arguably) be considered as nonrobust behavior that we should include when reporting model robustness. Let’s first consider the case of greedy decoding with n = k = 1. Let RC [−]s denote the number of correct-to-incorrect changes under the worst-case measurement as discussed. Symmetrically, let RC [+]s denote the number of incorrect-to-correct changes under best-case measurement: if the prediction with the original prompt is incorrect yet is correct for any of the s perturbed prompts. We define the Robust Relatives@1 metric as the fraction of changes in both directions out of the size of the dataset (N ):
RRs@1 := RC
[+] s +RC [−] s
N (3)
This definition can be generalized to sampling. Let rc[−]s (x) and rc [+] s (x) be similarly defined as RC [−] s and RC [+] s except that they are the number of changes within n samples for a prompt x instead of counting across the dataset. We define
RRs@k :=Ex  2− (n−rc[−]s (x) k ) ( n k ) − (n−rc[+]s (x) k ) ( n k )  
(4)
Eq. (4) falls back to Eq. (3) when n = k = 1. Discussion. RPs@k, RDs@k and RRs@k focus on different robustness requirements in practice. High RPs@k does not necessarily mean low RDs@k or RRs@k, because the model may learn to utilize spurious correlation in the datasets to demonstrate better Pass@k or RP@k, which is not robust. We advocate to report all of them to provide a comprehensive estimation of model robustness.

heading: 4 Evaluation
text: Evaluation setup. In this work, we use execution-based code generation benchmarks HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) to demonstrate our ReCode robustness evaluation framework. We perform a comprehensive study of robustness evaluation on popular public models including CodeGen (Nijkamp et al., 2022), InCoder (Fried et al., 2022), and GPTJ (Wang and Komatsuzaki, 2021) to show the robustness comparisons across different model architectures and sizes. The perturbations and metrics implemented in ReCode are general and applicable to any code generation datasets and models.

heading: 4.1 Code Generation Robustness Evaluation
text: Tab. 3 and Tab. 4 show the general perturbation performances on all the models in terms of the four general perturbation categories including transformations on docstrings, function names, code syntax, and code format. The nominal baselines for docstrings and function name perturbations are the pass@k on nonperturbed datasets. For perturbations on code syntax and format, the nominal baseline is the pass@k on nonperturbed customized datasets with partial code (see Sect. 3.4). We use greedy sampling for all the models to eliminate randomness effect and enable fair comparisons as the default setting. We consider s = 5, i.e., we generate five different datasets with different random seeds for each type of perturbations and evaluate worst-case robustness performance according to the robustness evaluation metric defined in Sect. 3.6. To evaluate and compare model robustness in a unified fashion, we aggregate the worst performance across different perturbations under each category. Taking the docstring perturbation category as an example, we say the model is robust only when the model predicts correctly on all the s perturbed datasets for each transformation listed in Tab. 1. We present detailed numbers for each perturbation type in Appendix D, Tab. 11-18. In Appendix B, we showcase and analyze failure cases on CodeGen16B-mono under three top perturbations, causing significant performance drops.
(1) Diverse pretraining corpus helps with both generalization and worst-case robustness. Comparing all code generation models with the same size 6B, CodeGen models have much better nominal performance, and have better robustness on RP5@1, a very strict worst-case robustness metric. That is possibly because CodeGen models are pretrained over a more diverse corpus than InCoder and GPT-J and thus have more capacity to deal with unseen instances and perturbations. However, CodeGen models have worse performance on RD5@1 and RR5@1, two robustness metrics relative to nominal performance, indicating that CodeGen models cannot generalize in a robust way (e.g., may learn to use spurious features in data). 4
(2) Larger model size brings improvement in
4Although these models may have some subtle architecturewise differences in details, we follow the benchmarking and evaluation strategies in previous works to focus more on pretraining parts and model sizes (e.g., BigBench (Srivastava et al., 2022), HELM (Liang et al., 2022)). We leave further ablation study for future work.
worst-case robustness, but may risk overfitting. In general, we observe higher RP5@1 for larger models within the same model family (e.g., improved from 0.174 to 0.217 for CodeGen-mono 2B to 16B on average across all perturbations), indicating larger model helps improve worst-case robustness. Similarly, we observe that larger models usually have larger RR5@1 (e.g., increased from 27.90% to 35.91% for CodeGen-mono 2B to 16B on average), indicating that larger models may risk overfitting as the relative performance drops under perturbations are significant.
(3) Code generation models are most sensitive to syntax perturbation. Among all perturbation types and across MBPP and HumanEval, we observe that syntax perturbations often result in the most performance drops. That reveals a significant limitation of syntax understanding ability of the state-of-the-art code generation models.
(4) Datasets having more variances in code
style poses more challenges on model robustness. In Tab. 5, we can see that models show better robustness on HumanEval over MBPP on average. MBPP has more variances in code style (e.g., indent with 1 space), closer to natural code distribution hence more challenging for model robustness.

heading: 4.2 Ablation Study
text: Robustness with s perturbed datasets. As described in Sect. 3.6, our robustness metrics consider worst-case performance across s perturbed datasets for each perturbation. Larger s leads to stronger perturbations evaluated, larger performance drops, and more extensive coverage to practical failures. The performance drops will start converging when large enough s evaluated. We can clearly see such trends in Fig. 4 where we evaluate CodeGen-16Bmono RDs@1 and RRs@1 under greedy sampling with s = 1, ..., 10. Perturbation categories like docstring and syntax that involve larger searching space and more randomness tend to benefit more with larger s (see Appendix A for details). As a trade-off, evaluation cost linearly increase with s. Thus, we recommend s = 5 as a good balance between cost and evaluation strength. We summarize the ablation study in terms of larger sampling n in Appendix D.3 which can also benefit our proposed robustness estimation with additional sampling cost.
Stable RD@k and increasing RR@k under different k. Pass@k allows the model to have k trials and model performance is often reported with different k. With the sampling setting of n = 100, we plot the RD1@k and RR1@k in Fig. 5. Interestingly, we observe that RD@k stays stable across different k while RR@k increases with k. This is because larger k leads to higher nominal pass@k and RP@k but their relative ratio stays similar leading to stable RD. On the other hand, larger k involves more samples potentially changing results on perturbed datasets causing larger RR. Similar
trends on CodeGen-2B and 6B in Appendix D.2 further confirm the observations.

heading: 4.3 Perturbation Sample Quality
text: Human evaluation. To verify the naturalness of the perturbations in ReCode, we randomly sample and shuffle 100 and 50 perturbed and nonperturbed MBPP and HumanEval data points and create a shuffle mix of 300 samples. Each sample is shown to 5 human annotators who are familiar with Python and who are asked to rate naturalness out of 0: not natural, 0.5: possible to appear in practice but rare, and 1: natural. The scores for naturalness drop 14% on average for our perturbed data where drops mainly come from typos by Butterfingers, CharCaseChanges, SwapCharacter, etc.
In addition, we randomly sample 100 and 50 pairs perturbed and non-perturbed MBPP and HumanEval data points. Each pair is shown to 5 human annotators who are asked to rate semantics out of 0: totally changed, 0.5: slightly changed, and 1: exactly preserved. We summarize the main results in Tab. 6, and we present statistic details and setup in Appendix C.1. Notably, the majority vote (at least three out of five) is 1 for 90% of data points. We further provide automatic evaluation below to support the quality of our perturbed datasets, but human evaluation is in general more reliable.
Docstring/function names similarity. We measure the sentence cosine similarity between perturbed and non-perturbed docstrings and function names. We obtain the embeddings by sentence transformers using model all-mpnet-base-v25 (Song et al., 2020). Note that we split each function name into words to get
5Model embedding quality in https://www.sbert.net
sentence embeddings. On average, we have 0.93 and 0.81 for docstring and function name perturbations, showing that they well preserve the semantics. Scores for some function name perturbations are sensitive to typos due to the lack of sentence context (e.g., 0.21 for interperse and intErpErse). Appendix C.2 summarizes detailed numbers for each perturbation. Code syntax/format similarity. In Tab. 7, we also measure the code similarity using CodeBLEU scores (Lu et al., 2021) for perturbed and nonperturbed data involving code syntax/format transformations. Here we consider the CodeBLEU score with syntax and dataflow separately as the evaluation metrics. On average, we have score 0.96 and 0.97 for CodeBLEU syntax and dataflow, showing good quality of perturbed datasets. Note that a few perturbations should expect low CodeBLEU scores: doc2comments transforms docstrings into comments causing changes of syntax; Deadcode insertion and for-while switch involve new if-conditions, loops, and new variables causing changes of code syntax and dataflow. Please refer to Appendix C.3 for details.

heading: 5 Conclusion
text: In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We collect and customize over 30 natural transformations under categories of docstrings, function names, code syntax, and code format perturbations. These transformations are carefully selected and designed to be natural in practice and preserve the semantic meaning after perturbations. We further propose general worst-case robustness metrics to give a unified overview of the model robustness performance. We empirically demonstrate our ReCode benchmark on popular models including CodeGen, InCoder, and GPT-J using HumanEval and MBPP datasets and function completion tasks derived from them. With human evaluation, over 90% of our perturbed data are confirmed to preserve the original semantic meaning; sentence similarity and CodeBLEU scores additionally support the quality of perturbations in ReCode.

heading: Limitations
text: ReCode benchmark has several limitations: (1) It contains perturbed datasets based on HumanEval and MBPP which focuses on Python function completion use cases. Therefore, we only perform
evaluation on Python language and not be able to capture robustness in a wide variety of code completion use cases. However, our transformations are generalizable and could be easily extended to other languages and also other coderelated datasets (Athiwaratkun et al., 2023). We encourage researchers to apply and extend ReCode benchmark to additional languages and other coderelated tasks; (2) ReCode benchmark is designed for robustness evaluation and cannot mitigate the lack of robustness. Given that our benchmark can be used to generate comprehensive collection of perturbed data, we believe that it can be used for training data augmentation to enhance model robustness. We will consider corresponding robust training strategy design and evaluation in the future work.

heading: Ethics Statement
text: Our ReCode robustness benchmark aims to provide a comprehensive robustness evaluation framework for any code-generation models, which we believe is critical towards building robust and user-friendly language models for code. With the new robustness evaluation metrics, users can rely on ReCode and assess model predictions with more confidence. The model trainers, on the other hand, will be aware of the potential vulnerabilities that might cause mispredictions in practice and mitigate them before deployments. Therefore, we believe our ReCode benchmark is beneficial in terms of broader impact.

heading: A Transformation Details and Qualitative Examples
text: In this section, we give detailed descriptions and settings for each type of perturbations that are included in our ReCode benchmark with qualitative examples for illustrations.

heading: A.1 Natural Transformations on Docstrings
text: For natural transformations on docstrings, we aim to perturb the docstrings to their variances that preserve the semantics and also appear natural in practice. Specifically, we will first extract and perturb the docstrings with the following natural transformations in each prompt, and then attach their perturbed versions to the prompt. To preserve semantics for the code generation task prompts, we extract a blacklist of program keywords using treesitter as discussed in Sect. 3.2 that are excluded from perturbations. We extend most transformations from NL-Augmenter (Dhole et al., 2021), a standard library designed for data augmentation and robustness evaluation on text. We list some qualitative examples in Tab. 1.
BackTranslation. BackTranslation paraphrases the docstrings by translating them to another language (in this case, German) and then back to English. It is a common method for data augmentation in generating sentence variances with the same semantics (Li and Specia, 2019; Sugiyama and Yoshinaga, 2019). Overall, it can reliably generate high quality perturbed docstrings. We use the default implementation in NL-Augmenter (Dhole et al., 2021). BackTranslation contains no randomness in transformations.
ButterFingers. ButterFingers transformation randomly selects characters of the docstrings and perturbs each of them to a random subset of similar characters, it is from (Dhole et al., 2021) and is also used in (Mille et al., 2021). Since this transformation tends to introduce character-level typos, we set randomness for perturbing each character to be low as 0.05 for naturalness consideration.
ChangeCharCase. ChangeCharCase transformation randomly changes the selected characters to upper case in the docstrings. We use the default probability 0.35 where majority annotators vote 0.5 for naturalness in the setting of Sect. 4.3.
EnglishInflectionalVariation. This transformation randomly selects words in the docstring and change them to a random inflection variance. This can be from plural to singular (or vice versa) for nouns and tense changes for verbs. To maintain naturalness, the perturbation is constrained to be the same Part of Speech (POS) tag in the Penn Treebank (Marcus et al., 1993).
SwapCharacters. This transformation randomly selects pairs of adjacent characters in the docstring and swap them. This represents a common type of typos by humans. To ensure naturalness, we set the probability as 0.05 for making the swap.
SynonymInsertion. This transformation randomly select words in the docstrings and inserts their synonyms in WordNet (Miller, 1992). Punctuations and stopwords are excluded. We set the probability to be 0.35 considering low success rate after keywords filtering.
SynonymSubstitution. This transformation randomly selects words in the docstring and replaces each one with a synonym from WordNet (Miller, 1992). Similar to SynonymInsertion, we set the probability as 0.35 to balance naturalness and perturbation success rates.
TenseTransformationPast. This is a deterministic transformation that converts sentences in the docstring to past tense.
TenseTransformationFuture. This is a deterministic transformation that converts sentences in the docstring to future tense.
Whitespace. This transformation inserts or deletes a single white space at randomly selected locations in the docstring. This represents a common type of typos by humans. Folowing NL-Augmenter, we use probability 0.1 for adding whitespaces and 0.05 for removing whitespaces.

heading: A.2 Natural Transformations on Function Names
text: These transformations modify the name of the target function to generate. Any references to the function name in the prompt, e.g., in docstring, are also modified to maintain consistency. Qualitative examples can be found in Tab. 2.
CamelCase. A function name is often composed of multiple words. If the original function name concatenates the words in camel-case style, this
transformation changes it to snake-case, and vice versa. This transformation is deterministic.
ButterFingers. ButterFingers transformation randomly selects characters of the docstrings and perturbs each of them to a random subset of similar characters, it is from (Dhole et al., 2021) and is also used in (Mille et al., 2021). Since this transformation tends to introduce character-level typos, we set randomness for perturbing each character to be low as 0.05 for naturalness consideration.
SwapCharacters. This transformation randomly selects pairs of adjacent characters in the function name and swap each pair. This represents a common type of typos by humans. To control naturalness, we set the probability to be 0.05, same setting as the docstring perturbations.
ChangeCharCase. ChangeCharCase transformation randomly changes the selected characters to upper case in the docstrings. We use the default probability 0.35 where majority annotators vote 0.5 for naturalness in the setting of Sect. 4.3.
InflectionalVariation. This transformation randomly selects words in the function name and applies a random inflection on them. This can be from plural to singular (or vice versa) for nouns and tense change for verbs. To control naturalness, the perturbation is constrained to be the same Part of Speech (POS) tag in the Penn Treebank (Marcus et al., 1993).
SynonymSubstitution. This transformation randomly selects words in the docstring and replaces each one with a synonym from WordNet (Miller, 1992). Similar to SynonymInsertion, we set the probability as 0.35 to balance naturalness and perturbation success rates.

heading: A.3 Natural Transformations on Code Syntax
text: These transformations modify the code content in the prompt. We derived function completion tasks with half the code from the canonical solutions such that the following code transformations and robustness evaluation can be performed. To guarantee fair comparisons to the nominal baseline, we make sure that we have the same block of code before and after code perturbations. In the following part we show qualitative examples on the same MBPP sample baseline ( Fig. 6).
DeadCodeInserter. This transformation inserts a block of useless code at a random location. The added block can be a loop of zero iteration or an if condition that is always false. The code content inside the dummy loop or if condition is randomly selected from the adjacent code statements with limited tree-sitter node sizes.
For-While Switch. This transformation randomly selects a for-loop or while-loop in the prompt and transforms it to its equivalent counterpart.
OperandSwap. This transformation randomly selects a binary logical operation, swaps the two operands, and modifies the operator if necessary to maintain semantic equivalence.
VarRenamerCB. This transformation selects the most frequently referenced variable name in the partial code and replaces it throughout the prompt with a new name obtained by CodeBERT (Feng
et al., 2020). Specifically, we replace all occurrence of the variable name with a mask token, and then run CodeBERT inference to obtain candidate names at each location, where each candidate name comes with a probability score. We pick the candidate name with the highest aggregated score across locations. This transformation is inspired by (Jha and Reddy, 2022; Li et al., 2020).
VarRenamerNaive. This transformation selects the most frequently referenced variable name in the partial code and replaces it with "VAR_0". This is the original implementation in the NatGen package. This transformation is deterministic.
VarRenamerRN. This transformation selects the most frequently referenced variable name in the partial code and replaces it with a random string with half alphabetic and half numeric characters.

heading: A.4 Natural Transformations on Code Format
text: Tab-Indent. This transformation replaces any space indents with tabs or replaces tabs with 4 spaces for indent-sensitive languages like Python. This transformation is deterministic.
Line Split. This transformation splits the longest line in the partial code into two lines. This transformation is deterministic.
Doc2Comments. This transformation changes the style of the documentation in the prompt. For Python, it converts docstring (e.g., """ docstring """) to commented lines (e.g., # docstring) and vice versa. For Java, it converts comments in the
format of /* docstring */ to // docstring and vice versa. This transformation is deterministic.
NewlineRandom. This transformation inserts empty lines at randomly selected positions.
NewlineAfterCode. This transformation inserts an empty line at the end of the prompt. This transformation is deterministic.
NewlineAfterDoc. This transformation inserts an empty line between the docstring and the partial code. This transformation is deterministic.

heading: B Failure Case Study under Perturbations
text: In this section, we showcase and analyze some failure cases on CodeGen-16B-mono and perturbed HumanEval datasets under three top perturbations that will cause significant performance drops.
DeadCode insertion is one of the most effective
perturbations. It can commonly mislead the model predictions with the inserted dead code, especially when the completions are required right after the inserted dead code. Fig. 19 shows an failure example where CodeGen-mono-16B only predicts a newline after inserted meaningless for loop, which might be mislead by the inserted return statement.
(a) Correct completion without perturbation.
(b) Wrong completion perturbed by deadcode insertion.
Figure 19: HumanEval showcase 1 illustrating failure case under deadcode insertion.
Fig. 20 shows a failure example of CodeGen16B-mono on a prompt where an empty newline is inserted right before completion. Such simple perturbation causes wrong predictions for the following if-else conditions. It is especially effective when the required completion code is complicated.
ButterFingers perturbation on docstring causes large performance drops as well. Fig. 21 shows another falure example on CodeGen-16B-mono. The typos introduced in the perturbation might cause the model to misunderstand the targeted docstrings, leading to wrong model completions.
(a) Correct completion without perturbation.
(b) Wrong completion perturbed by NewlineAfterCode insertion.
Figure 20: HumanEval showcase 2 illustrating failure case under NewlineAfterCode insertion.

heading: C Perturbation Sample Quality

heading: C.1 Details for Human Evaluation
text: The annotators are all recruited from software engineers online who have good experience in Python via strict coding interview. To guarantee the reliability of the human evaluation results, we first conducted annotation trials with our annotators. We gave them clear definitions for each level of naturalness and semantic similarity.
We measure the inter-annotator agreement rate Fless Kappa in Tab. 8. The overall average Fleiss Kappa for the annotations is 0.52, 0.36 for semantic and naturalness measurements on perturbed samples. The confidence interval (95%) with bootstrap sampling (10K samples) is [0.515, 0.528] and [0.358, 0.364], indicating that our annotation reaches “moderate agreement” and thus our annotations are reliable (Gwet, 2014). The scores from annotators are not perfectly consistent especially for naturalness since people have different preferences for code.

heading: C.2 Sentence Transformers for Docstring/Function Names Similarity
text: In this subsection, we give experimental details for measuring the sentence similarity of perturbed and unperturbed data points using sentence transformers.
To measure the similarity scores for the docstring perturbations, we first extract docstrings from each pair of perturbed and unperturbed data points, and we use sentence transformer all-mpnet-base-v2 (Song et al., 2020) to predict
an embedding vector for each docstring. Then cosine similarity is calculated and reported for each pair of perturbed and unperturbed datapoints.
Same process cannot be directly applied to function name perturbations since function names are concatenations of words instead of common sentences, barely seen by the sentence transformer training data. In order get more accurate sentence embeddings for function names, we first split each name into words (e.g., has_close_elements to has close elements) and then calculate the corresponding cosine similarities.
In Table 9, we present the detailed results for each type of perturbations for sentence similarity. On average, we can have 0.93 and 0.92 similarity scores for docstring perturbations and 0.80 and 0.81 for function name perturbations on the HumanEval and MBPP datasets. The overall high similarity numbers provide support that our perturbations have good quality in naturalness and semantic preservation from the unperturbed inputs.
Some function name perturbations including ButterFinger, SynonymSubstitution, and CharCaseChange have relatively low sentence similarity. This is mainly because the function names only include keywords without complete sentence context and thus minor changes to each words could potentially cause large change in measured cosine similarity. For instance, character case changes on function name intersperse to intErspErse which lacks of context only has 0.21 similarity. On the other hand, the function names with more context has much higher scores, e.g., 1.0 similarity score for has_close_elements and has_ClosE_Elements.

heading: C.3 CodeBLEU Scores for Code Similarity
text: Here we present the experimental details for the CodeBLEU syntax and dataflow scores to quantitatively measure the quality of our code syntax and format transformations.
The measurement is straightforward. The unperturbed baseline is each data point from our customized partial code datasets derived from HumanEval and MBPP. The perturbed one is the same data point transformed by each type of our perturbations. The CodeBLEU syntax and dataflow scores are then directly measured using the CodeXGLUE (Lu et al., 2021) implementation.6
6https://github.com/microsoft/CodeXGLUE
In Table 10, we present the detailed CodeBLEU results for each type of perturbations. The average numbers are summarized in Table 7. Overall, 77% and 89% of our transformations have over 0.9 CodeBLEU syntax and dataflow scores, showing good quality in preserving semantics from the unperturbed code.
However, CodeBLEU syntax and dataflow are not perfect in quantitatively measuring naturalness and semantic preservation for the perturbations and thus some perturbations have expected relatively low scores: Doc2Comments transforms docstrings into comments causing changes of syntax; Deadcode insertion and for-while switch involve new if-conditions, loops, and new variables causing changes of code syntax and dataflow.

heading: D Additional Results

heading: D.1 Fine-grained Robustness Evaluation
text: We present the robustness evaluation for each type of perturbations from Table 11 to 18, . The evaluation setting is the same as Table 3 and 4 where we evaluate various sizes of CodeGen (Nijkamp et al., 2022), InCoder (Fried et al., 2022), and GPTJ (Wang and Komatsuzaki, 2021) with greedy sampling. For each type of perturbations, we randomly generate s = 5 different perturbed datasets derived from HumanEval and MBPP. For perturbations without randomness, only one single version of perturbed dataset is evaluated. The list of indeterministic perturbations can be found in Appendix A.

heading: D.2 Additional Results for Different k
text: As discussed in Sect. 4.2, we observe that Robust Drop stays stable across different k while Robust Relative increases linearly with k. We present additional results on CodeGen-2B-mono, CodeGen-6Bmono along with CodeGen-16B-mono in Fig. 22. We evaluate each model with large n (n = 100) using top-p sampling strategy with probability 0.95 and temperature 0.2.

heading: D.3 Additional Results for Large Sampling n
text: Larger sampling n is commonly used for preventing model generation variances and providing accurate estimations. The evaluation cost increases linearly to n. Here we show that larger n can also benefit our proposed three robustness metrics but not causing significant differences. In specific, we measure Robust Pass1@1, Robust Drop1@1, and Robust Relative1@1 on CodeGen-16B-mono and HumanEval dataset. The model is run with n = 100 using top-p sampling strategy with probability 0.95 and temperature 0.2. We present detailed results in Tab. 19.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: TenseTransformationFuture. This is a deterministic transformation that converts sentences in the docstring to future tense. Whitespace. This transformation inserts or deletes a single white space at randomly selected locations in the docstring. This represents a common type of typos by humans. Folowing NL-Augmenter, we use probability 0. 1 for adding whitespaces and 0. 05 for removing whitespaces