Code Execution with Pre-trained Language Models
Year: 2023

Abstract:
Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code. However, most pre-trained models for code intelligence ignore the execution trace and only rely on source code and syntactic structures. In this paper, we investigate how well pre-trained models can understand and perform code execution. We develop a mutation-based data augmentation technique to create a large-scale and realistic Python dataset and task for code execution, which challenges existing models such as Codex. We then present CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension. We evaluate CodeExecutor on code execution and show its promising performance and limitations. We also demonstrate its potential benefits for code intelligence tasks such as zero-shot code-tocode search and text-to-code generation. Our analysis provides insights into the learning and generalization abilities of pre-trained models for code execution.

heading: 1 Introduction
text: Pre-trained models have achieved remarkable results in natural language (NL) tasks (Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020), inspiring the development of pre-trained models for programming language (PL) tasks (Kanade et al., 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Wang et al., 2021b; Guo et al., 2021, 2022). These models leverage source code and code structures, such as abstract syntax tree (AST) (Wang et al., 2021a; Guo et al., 2022) and data flow (Guo et al., 2021), to learn code-related tasks. These structures, while useful, are not sufficient to represent the dynamic behavior of code during execution, which is reflected in the execution trace. Using Figure 1 as
∗Work done during internship at Microsoft. Shuai Lu and Nan Duan are corresponding authors.
an example, the execution trace shows how code behaves during execution, reflecting the control flow and the state changes of variables. On the other hand, as stated by Casalnuovo et al. (2020), source code contains two channels of information: natural & formal. The natural channel (Hindle et al., 2012), such as identifiers and comments, enables language models to be leveraged to understand code-related tasks. The formal channel is used by interpreters and compilers to specify execution and has precise semantics. The formal channel is unique to code and is what makes it executable. Execution trace falls into the second category since it reveals the formal channel of information that distinguishes code from natural language, as well as enabling code execution precisely (Casalnuovo et al., 2020; Chakraborty et al., 2022).
In this work, we aim to teach pre-trained models the real-world code execution process. We propose CodeExecutor, a Transformer-based model that learns to execute arbitrary programs and predict their execution traces. To support pre-training on large-scale data, we construct the Python CodeNetMut dataset by producing mutations based on submissions to competitive programming problems from CodeNet (Puri et al., 2021), along with single-line Python transformations and programs adapted from Python official tutorial. We design a pre-training task that predicts both the line order and the intermediate states of the execution trace, and apply curriculum learning to gradually increase the difficulty of the programs.
We evaluate CodeExecutor on code execution tasks and show that it outperforms existing models and demonstrates promising capabilities. We also conduct an in-depth analysis of the model’s performance and reveal its strengths and weaknesses. Furthermore, we show that CodeExecutor can improve downstream tasks like zero-shot codeto-code search and text-to-code generation, indicating the potential of leveraging execution trace to
4984
enhance code intelligence. Our models and datasets are publicly available1. In summary, the contributions of this paper are:
• We present the first attempt at building a largescale pre-training dataset for real-world code execution using a mutation-based data augmentation approach.
• We propose a novel pre-trained model named CodeExecutor that learns to predict the execution traces using a code execution pre-training task and curriculum learning.
• We conduct a comprehensive evaluation of CodeExecutor for code execution tasks, providing a detailed understanding of the model’s performance.
• CodeExecutor significantly improves code intelligence tasks like zero-shot code-to-code search and text-to-code generation.

heading: 2 Related Work

heading: 2.1 Learning to Execute
text: Previous works form the learning to execute task as a problem that reads a program and computes the program’s output. These works leverage architectures such as recurrent neural networks (Zaremba and Sutskever, 2014), graph neural networks (Bieber et al., 2020; Wang et al., 2020) and Transformers (Dehghani et al., 2019; Yan et al., 2020; Austin et al., 2021; Nye et al., 2021). Another related task algorithm induction is to read a short program, such as integer addition or polynomial evaluation, and computes the output. Algorithm induction task (Graves et al., 2014; Kurach et al., 2016; Kaiser and Sutskever, 2016; Graves
1https://github.com/microsoft/CodeBERT/tree/ master/CodeExecutor
et al., 2016; Reed and de Freitas, 2016; Dehghani et al., 2019; Velickovic et al., 2020a,b; Nye et al., 2021) targets a particular algorithm with direct algorithm-specific supervision compared with arbitrary programs in our code execution task.
Some emerging works also employ pre-trained models to tackle the two tasks. Lu et al. (2022) fine-tunes a small fraction of the weights in GPT2 (Radford et al., 2019) on non-language tasks, including simple algorithm induction tasks like Bit XOR. Austin et al. (2021) evaluates models pretrained on web documents and dialog data ranging in size from 2 million to 137 billion parameters and shows that largest models are generally unable to predict the output of a program, whether few-shot or fine-tuning. Nye et al. (2021) uses a "scratchpad" to store intermediate computation steps to perform multi-step computations, improving the ability of models in Austin et al. (2021).
Different from previous works that predict program’s output and mainly deal with specific algorithms, we predict the program’s whole execution trace and focus on imitating the real-world arbitrary program execution behavior. Besides, by using execution to capture code semantics, our work is beneficial for tasks related to code intelligence.

heading: 2.2 Mathematical Problem Solving
text: Mathematical problem solving is a related domain of code execution. Recent works show the ability of language models to solve math problems, which requires learning to execute a soft algorithm to arrive at a deterministic answer. Amini et al. (2019); Ling et al. (2017) map math problems to operation programs and focus on sequence-to-program generation. Saxton et al. (2019) introduce the DeepMind Mathematics dataset, which contains plugand-chug problems such as addition, list sorting, and function evaluation. Henighan et al. (2020)
shows that the majority of problems in the DeepMind Mathematics dataset can be straightforwardly solved with large Transformers. Hendrycks et al. (2021) introduces the MATH dataset, consisting of competition math problems with step-by-step solutions written in LATEX and natural languages. Cobbe et al. (2021) releases GSM8K, including grade school math questions and natural language solutions. Recently, Zhou et al. (2022) proposes algorithmic prompting to improve the performance of large language models on math problem solving, which starts from learning skills containing addition, subtraction, multiplication, and parity.
Code execution involves calculations such as addition, subtraction, multiplication, division, exponentiation, and modulus, which are similar to solving math problems. With the added complexity of managing variables, data structures, control flows, and other programming concepts, learning code execution requires a different set of skills and knowledge from learning mathematics, although some overlap exists.

heading: 3 Mutation-based Data Augmentation
text: The goal of code execution task is to learn to emulate the execution without running a program by an interpreter. We treat the task as a generation task: given a source code c, the execution trace t is required to be generated. Execution trace consists of two components: one is the order in which the computer executes statements, and the other is how the states of the variables change when jumping from one statement to another. Normally, the statements inside a program are not executed sequentially, especially in a real-world scenario where programs embody complex logic and rich semantics. More-
over, variables relate to various types of data structures with diverse characteristics and operations. Given the complexity and difficulty of this task, it is of great importance to build a large-scale dataset and explore the capabilities and boundaries of large language models for code execution.

heading: 3.1 Mutating Source Code
text: Constructing a large-scale Python dataset for realworld code execution is very challenging. Programs retrieved from software development platforms such as GitHub 2 are mostly not executable at scale, as they depend on specific external resources which are not easily available. Examples of external resources include program inputs, file contents, external modules, and third-party packages. For the same reason, it is not practical to collect programs from posts in coding question-answering websites like StackOverflow 3.
We build the Python code execution dataset based on submissions to competitive programming problems from CodeNet benchmark (Puri et al., 2021). We run each submission in a sandbox environment to get the execution trace and filter out programs that exceed time and trace limits or result in runtime errors.
To construct a large-scale dataset of executable programs, we propose a mutation-based data augmentation approach. For each submission, the approach modifies some parts of a program to generate diverse mutants, leading to different execution traces. Specifications of these modifications are called mutation operators. It is inspired by mutation testing (Hamlet, 1977; Jia and Harman,
2https://github.com/ 3https://stackoverflow.com/
2011) in software engineering, a popular technique that supports the design of high-quality test suites for programs. Following Derezińska and Hałas (2014) that applies mutation testing technique to Python programs, we first present a set of mutation operators as shown in Table 1. Most of them correspond to selected operators used in strongly typed general purpose languages and are adopted to the Python language. Operators designed for Python features are also included, such as Slice Index Removal (SIR) and Reverse Iteration Loop (RIL). Then we convert a program into an AST and extract its node type information to get a candidate list of all mutable literals, operators and statements. Finally, we generate mutants and eliminate those that are not executable. We use the CodeNet Mutants (CodeNetMut) to build the pre-training dataset. Greater detail of the dataset generation process can be found in Appendix A.

heading: 3.2 Dataset Construction
text: Given the difficulty of training the model on realworld complete programs, we build two simpler datasets along with CodeNetMut for pre-training.
The first is the Python SingleLine dataset collected by Fraser Greenlee 4, which consists of nearly nine million examples of single-line transformations. Each example contains several variables specified in initial values, a single line of Python code, and the new set of variables and values resulting from executing that line. We combine the first two as the input code, and use the last one as the target trace. We do not re-execute the dataset. When pre-training on SingleLine data, we only ask the model to predict the final states of the last code line without line-by-line illustration. Figure 2 (a)(b) show examples of these data. Since individual lines of code constitute real-world complex programs, the dataset serves as a foundation for learning about code execution.
The second is the Python Tutorial dataset. This dataset is created by crawling and filtering all the executable code examples that appear in the official Python tutorial 5. The official tutorial introduces the basic concepts and most noteworthy features of the Python language. To generate this dataset, we apply the Constant Replacement operator (first row in Table 1) to change numeric literals into diverse values. This approach results in 3.4 million pro-
4https://www.kaggle.com/frasergreenlee/ python-state-changes
5https://docs.python.org/3/tutorial
grams. Figure 2 (c) shows an example of a mutant. While the Tutorial dataset is not comprehensive and does not cover every single feature, it provides a good representation of Python’s flavor and style, which offers valuable supervision for modeling the execution of commonly used code blocks.
Therefore, the Python Code Execution datasets are a series of datasets following an easy-to-hard paradigm, including the SingleLine dataset, Tutorial dataset, and CodeNetMut dataset.

heading: 4 CodeExecutor
text: Our CodeExecutor utilizes a Transformer-based framework to learn code execution through pretraining. We will first describe the model architecture (§4.1), then the pre-training task (§4.2), and finally, the curriculum learning strategy (§4.3).

heading: 4.1 Model Architecture
text: The model is based on Transformer and adopts the same architecture as UniXcoder (Guo et al., 2022). UniXcoder is a unified cross-modal pre-trained model for programming language which has encoder-only, decoder-only and encoder-decoder modes. It utilizes mask attention matrices (Dong et al., 2019) with prefix adapters to control the behavior. We take the encoder-decoder manner by using a special token [E2D] as the prefix in front of the input. CodeExecutor consists of 12 Transformer layers. Each transformer layer is architecturally identical, containing a multi-headed self-attention pooling (Vaswani et al., 2017) followed by a feed forward network.

heading: 4.2 Pre-training Task
text: We propose a new pre-training task called code execution. Our motivation for the task is to improve the ability of our model to understand and execute code. Traditional pre-training tasks such as language modeling or denoising objective do not involve code execution, and thus, models trained on these tasks have limited ability to execute code. By pre-training our model on the task of code execution, we aim to improve its ability by learning useful patterns from bimodal data of code and trace. This will enable our model to generate more accurate traces and understand the behavior of the code, which is crucial for a wide range of code intelligence applications that require code understanding. With the knowledge of how the code works, the model can better understand the underlying logic of the code and use that understanding to better perform these tasks.
We continue pre-training UniXcoder on the task. At the pre-training stage, our model receives code as inputs and learns to generate traces. To facilitate a better understanding of code, special tokens [i] indicating line numbers and [INDENT ] [DETENT ] indicating indentation are inserted into the code. Each line in trace can be represented as [LINE], [i], [STATE], v1, : , s1, [DICTSEP ], ..., [DICTSEP ], vk, :, sk, [STATEEND], where k denotes the number of variables and the state of k-th variable vk is sk. The symbol [DICTSEP ] separates the pairs within the dictionary and [STATEEND] indicates the end of the states. This representation allows our model to learn the state of variables at each step of the execution, which is crucial for understanding the behavior of the code.

heading: 4.3 Curriculum Learning
text: To improve the generalization capacity, we follow the curriculum learning strategy during pre-training. Curriculum learning (Bengio et al., 2009) (CL) is a learning strategy that starts from easy instances and then gradually handles harder ones, which imitates the meaningful learning order in human curricula. In our pre-training process, we organize the learning of the Python code execution datasets according to a curriculum that starts with simple instances, i.e. SingleLine data. First, we employ all the 9 million SingleLine transformations to pre-train CodeExecutor until convergence. To achieve a balanced dataset, we then reserve 3 million instances in Sin-
gleLine that are most difficult for our model to generate and add Tutorial data into the pre-training corpus. We further add CodeNetMut data into the pretraining corpus and pre-train the model to converge on all the examples. To help distinguish difficulty level, we add a prefix p ∈ {[SINGLELINE], [TUTORIAL], [CODENETMUT ]} in front of the input, indicating the kind of data, e.g. [SINGLELINE] means receiving SingleLine data. More details about pre-training settings and model configurations can be found in Appendix B.

heading: 5 Experimental Setup

heading: 5.1 Dataset
text: We build our pre-training dataset as described in Section 3. Table 2 shows some basic statistics. The 19,541 examples in CodeNetMut test split are from 39 unseen programming problems in CodeNet and have not undergone the mutation process. Additionally, we held out 10k programs from each dataset as a validation split during pre-training. For Tutorial and CodeNetMut, the ground truth trace is the execution result of the whole program. For SingleLine, since the instances are simple programs consisting of variable declarations and one-line transformations, the model is only asked to predict the final states of variables, which is presented in the form of a one-line trace. We observe the average length of code and trace in CodeNetMut are about twice as long as those in Tutorial. Also, executing programs in CodeNetMut requires managing a larger number of variables in varying states.

heading: 5.2 Models
text: We evaluate several models on code execution task. Codex model code-cushman-001 is a specialized GPT model fine-tuned on GitHub code (Chen et al., 2021). We use few-shot learning
by giving Codex three code and execution trace pairs for the code execution task. CodeExecutorLimited (CEL) is a three-stage model pre-trained with the code execution objective. CEL can only access limited data in each stage, as opposed to CodeExecutor which can utilize all the datasets simultaneously (see Appendix C for a detailed comparison). It is initialized using the publicly available checkpoint of UniXcoder and continues to be trained with SingleLine data, resulting in the model CodeExecutorLimited-Stage1, which we call CELS1. In the second stage, we initialize it with CELS1 and employ Tutorial data to pre-train, so we get the model CEL-S2. By continuing pre-training CEL-S2, we use CodeNetMut to improve the capacity of executing real-world programs at the third stage. CEL-S3 is produced after these stages mentioned above. CodeExecutor without Curriculum Learning(CodeExecutor w/o CL) is a single-stage model trained on all three datasets together.

heading: 5.3 Evaluation Metrics
text: We test model capabilities of executing code on the test sets from three datasets. We measure functional correctness of the sampled trace from three perspectives. We report output accuracy and trace accuracy to evaluate the general aspect. Output accuracy checks if the model prints the same message as the code execution, calculated only for programs with standard output. Trace accuracy checks if the model produces the same trace as the code execution, regardless of the order of states in a line of the trace. To evaluate the correctness of each line and the states of identifiers in the trace, we also assess per-line score and identifier score. Line precision is determined by the ratio of correctly identified
lines among all the lines in the traces generated by the model. Line recall is the ratio of correctly identified lines predicted by the model among all the lines in the ground truth traces. Similarly, we also calculate scores for the identifiers in the trace.
To deepen our understanding of model behavior and error modes, we also conduct a qualitative analysis by examining samples.
We randomly sample 50 code-trace pairs from the test set and ask two programmers with at least 5 years of experience to evaluate whether CodeExecutor executes a program correctly in 7 aspects. The category Basic includes basic knowledge for a Python beginner like math operators, augmented assignment operators, comparison operators, variables. The category Lists, Tuples, etc. consists of typical Python data structures, such as lists, tuples, dictionaries, sets, and related manipulation functions. As shown in Table 4, we build the taxonomy, along with a handbook to guide classification. Each reviewer examines the generated trace line by line and counts the occurrence frequency of each category. They count all these categories if a trace line involves multiple categories. When an error occurs, they identify which kind of knowledge category the model mistakes. Finally, they work together to discuss the divergence of error attribution and come to an agreement.

heading: 6 Results and Analysis
text: In this section, we evaluate CodeExecutor on code execution task(§6.1), conduct an in-depth analysis to understand model behavior and error mode (§6.2), followed by two downstream tasks (§6.3).

heading: 6.1 Overall Results
text: We evaluate the performance of models on SingleLine, Tutorial and CodeNetMut datasets.
We show the result of SingleLine in Table 3 (top). CodeExecutor is able to execute around 94% of single-line transformations correctly, while Codex fails to do so in most cases. CodeExecutor also brings a 0.7% improvement over CEL-S1, indicating learning hard programs during pre-training helps better solve easier examples. Since each SingleLine program always produces a one-line trace without standard outputs, we do not report output accuracy, and the line precision/recall scores are equal to trace accuracy.
For the Tutorial experiments in Table 3 (medium), CodeExecutor significantly outperforms Codex on output accuracy (76.42% vs.13.07%). The lower score of CodeExecutor compared to CEL-S2 suggests a discrepancy between code examples in tutorials and CodeNet since the Tutorial dataset is composed of mutants from only a few programs in tutorial websites, limiting its diversity. CEL-S3 struggles to produce traces, indicating that it forgets most knowledge acquired in Tutorial data in the last training stage.
CodeNetMut results are much lower than those in SingleLine and Tutorial datasets, which shows that it is more challenging to generate traces in real-world scenarios. CodeExecutor produces the correct output for nearly half of the examples (48.06%), and about a third of the traces are the exact match for the ground truth (33.38%). By pretraining on the code execution task, CodeExecutor boosts the performance of output by 30.6% absolute points over Codex. Besides, CodeExecutor yields 4.3% output accuracy score and 3.9% trace accuracy score improvement than CEL-S3, which indicates the effectiveness of the training strategy described in 4.3. After removing curriculum learning, the output accuracy score drops from 48.06% to 45.93% and the trace accuracy score drops from 33.38% to 30.98%, which shows the contribution
of curriculum learning. These results demonstrate that the code execution task is challenging for pre-trained models on source code like Codex. However, our CodeExecutor model can achieve high performance to execute simple programs and are capable of predicting complex execution traces for real-world programs.

heading: 6.2 In-depth Study on Model Performance
text: We conduct a qualitative analysis of model performance by examining samples (Table 4), resulting in the following findings. More examples can be found in Appendix D.
The Model Typically Has a Basic Sense of Control Flows Conditional statements, loops, and function calls reveal the control flow of the program. Control flow reflects the order in which the program’s code executes. It is important for understanding a program and is often complex, as it controls the code through certain decisions and monitors which statements need to be executed and which should be skipped. From Table 4, we find that CodeExecutor has a rudimentary understanding of high-level multi-line control flows, especially expert at conditional statements and function calls. 57 out of 60 conditional statements and all 5 calls to user-defined functions are predicted
correctly. The accuracy of loops is 84%, while the incorrect loops undergo wrong iterative times. Take Figure 1 (a) as an example. CodeExecutor predicts exactly the same trace as the ground truth in (b). Our model recognizes that the for loop occurred on line 4 will execute several times. In the second iteration, “n” meets the condition of “n <= 0”, resulting in the “break” statement and terminating the loop. The model behaves well on the code block in the for loop, showing its capacity of understanding control flows.
The Model Struggles to Handle the Intricacies of Operations, Particularly in Relation to Data Structures Complex programs often involve multiple categories of programming knowledge. Figure 3 shows an example that uses lists and strings. It determines the maximum and minimum possible number of people among “n”, who subscribe to both Newspaper I and II, given that “a” people subscribe to I and “b” people subscribe to II. CodeExecutor incorrectly calculates “nmin” in line 5, expected 0 but got 2. This calculation involves retrieving values from a list, performing additions, subtractions, and using the "min" function. The compositionality of these operations makes it challenging for our model to fully comprehend the code and generate accurate states. Additionally, as presented by the relatively low accuracy on “Lists, Tuples, etc.” (77.27%) and “Strings” (52.63%) in Table 4, we observe that the model falls short of understanding data structures like lists and strings. The understanding of data structures requires the model to learn the behavior of objects after they are created, modified, added or deleted. These operations can be changeable and challenging for the model to grasp. This suggests that the model may struggle with complex programs that involve multiple operations and data structures.

heading: 6.3 Downstream Tasks
text: To verify the effectiveness of CodeExecutor in representing code semantics, we apply it to two code
intelligence tasks – the zero-shot code-to-codesearch task and text-to-code generation task.
Zero-shot Code-to-code Search The task is introduced by Guo et al. (2022). To avoid duplication between the associate dataset and our pre-training corpus, we construct a new dataset by collecting 9,987 Python functions from CodeNet (Puri et al., 2021). Each function solves one of the 48 problems. Given one function, we retrieve all the functions that solve the same problem.
We first use the mean vectors of last hidden states of a baseline model to calculate the similarity between two functions. To explore how code execution facilitates code-to-code-search, we execute each function by providing a test case. We then utilize the program outputs extracted from the execution trace generated by CodeExecutor, and sort the candidates according to the edit similarity compared with outputs of the query program.
From table 5, we find that CodeExecutor boosts over 32.8 points compared with GraphCodeBERT (Guo et al., 2021), and provides about 7.2 points improvement compared with UniXcoder, showing that code execution can significantly enhance the comprehension of code semantics.
Text-to-code Generation We use HumanEval benchmark (Chen et al., 2021) which includes 164 human-written programming problems.
We first leverage Codex (code-cushman-001) to generate 200 solutions for each problem. Then we use CodeExecutor to predict the outputs of each solution by feeding example test cases in problem descriptions. We rank the 200 solutions by the edit similarity between their outputs and expected outputs. Finally, we evaluate the correctness of the first 50 solutions for each problem. Note that different from other filtering strategies, our method doesn’t need a real-world code executor but only uses models to predict the execution results.
Table 6 demonstrates that with CodeExecutor as a solution filter, the performance of text-to-code generation is improved, indicating CodeExecutor
is beneficial to other code intelligence tasks.

heading: 7 Conclusion
text: We propose a mutation-based data augmentation method to create a large and realistic Python code execution dataset and task, which pose a significant challenge for current models such as Codex. We develop CodeExecutor, a Transformer model that leverages code execution as a pre-training objective and adopts a curriculum learning strategy. CodeExecutor not only outperforms existing models on code execution, but also demonstrates its generalizability to downstream tasks such as codeto-code search and text-to-code generation. Our work offers a novel and effective solution for code execution and other code intelligence tasks.
Limitations
Several limitations of CodeExecutor, such as its application to only Python, the lack of faithfulness in the results produced, and the maximum length limit for trace generation, point toward interesting directions for future work.
Programming Language One limitation of our current model is that it is currently only applied to Python, which limits its use and effectiveness in executing programs written in other programming languages. This highlights the need for future work to expand the model’s applicability to other languages.
Faithfulness The result may not be faithful enough when handling difficult examples, such as those with complex logic, long loops, or many branches. For example, we observe that in two complicated programs that both contain the assignment “alpha = list(’abcdefg’)”, our model correctly predicts the value of “alpha” in one case but incorrectly in the other. The lack of faithfulness needs to be studied for further research on code execution.
Generation Window Size We limit the length of generated trace to 1024 tokens. It can be a limitation for programs with long execution traces, particularly those with loops. Improving the ability of Transformers to handle longer sequences (Tay et al., 2021, 2022) would likely be beneficial for the code execution task.
Ethical Statement
The work is conducted in compliance with ethical principles. The datasets introduced in this paper only used publicly available data. The annotation in human evaluation was conducted by two authors of the paper, and thus there are no associated concerns, e.g. regarding compensation. Therefore, there are no potential risks associated with the research.

heading: A Dataset Detail
text: To obtain executable programs, we build the Python Code Execution dataset based on submissions to competitive programming problems from CodeNet (Puri et al., 2021). These human-written programs with real-world complexity are derived from online judge websites AIZU 6 and AtCoder 7. CodeNet contains 240k Python submissions, aiming to solve 8,00 distinct programming problems. Each submission is a single-file Python program that reads from stdin and writes to stdout. Each programming problem provides at least one sample input and at most four sample inputs. Since executing a program relies on an input, we replace the statements that read from input streams with assignment statements that assign input values to variables. We run each submission in a sandbox environment to get the execution trace for that program. Programs are restricted to one second of execution time and 1024 lines of execution trace, and will be filtered
6https://onlinejudge.u-aizu.ac.jp/ 7https://atcoder.jp/
out if they exceed the limits. We also remove the programs that result in runtime errors during parsing or execution, by catching Python exceptions raised in programs. This results in a dataset of 387k executable programs, each paired with a trace.
To construct a large-scale dataset of executable programs, we propose a mutation-based data augmentation approach. we first present a set of mutation operators as shown in Table 1. Most of them correspond to selected operators used in strongly typed general purpose languages and are adopted to the Python language. Operators designed for Python features are also included, such as Slice Index Removal (SIR) and Reverse Iteration Loop (RIL). Then we leverage the tree-sitter8 to convert a program into an abstract syntax tree and then extract its node type information to get a candidate list of all mutable literals, operators and statements. For each mutable candidate, we apply the related mutation operators with 50% probability. Specifically, we change a numeric literal x into a random number from a Gaussian distribution with mean x and standard deviation 100. We either extend a string with one or two random characters or shorten a string. We randomly pick one of the three looprelated operators or keep it as it is when handling each loop. All operators can be applied before a mutated program execution, and possible mutants with errors are to be detected and eliminated during execution. By mutating each program 20 times, we obtain 3.2M deduplicated programs, each paired with a trace.
We use the CodeNet Mutants (CodeNetMut) to build the pre-training dataset. To prevent data leakage, all submissions to the same problem become part of the same split. We use submissions of 710 problems with their mutants to build the pretraining dataset. Since mutation greatly enhances diversity, these programs embody rich semantics and complex operations. Other submissions (without mutations) are used to build the validation and test dataset. These human-authored programs ensure the quality of evaluation data.

heading: B Model Configurations
text: We build our model based on 12 layers of Transformer with 768 dimensional hidden states and 12 attention heads. We add 210 additional special tokens into the vocabulary to represent 200 line numbers, 3 pre-training dataset names, and trace
8https://tree-sitter.github.io/tree-sitter/
structure described in §4.2. During pre-training, we set the max length of input sequence and batch size to be 1024 and 256, respectively. We use the Adam optimizer to update model parameters with 4e-4 learning rate. We first employ the SingleLine dataset to pre-train the model with the code execution objective for 500k steps. We then reserve 3 million instances in SingleLine that are most difficult for our model to generate and add Tutorial data into the corpus, pre-training for 300k steps. We add CodeNetMut into the corpus and further pre-train for 300k steps. We pre-train the model on a cluster of 16 NVIDIA Tesla V100 with 32GB memory and the total training time is about a month. For inference, we set beam search as 10.

heading: C Three-stage Pre-training
text: In table 7, we list the datasets that CodeExecutorLimited (CEL) and CodeExecutor use for threestage pre-training, respectively.
The first stage of pre-training for CEL uses the SingleLine dataset, resulting in the model CEL-S1. In the second stage, CEL is initialized with CEL-S1 and pre-trained with the Tutorial dataset, resulting in the model CEL-S2. In the third stage, CEL is initialized with CEL-S2 and pre-trained with the CodeNetMut dataset, resulting in the model CELS3.
On the other hand, CodeExecutor is first pretrained with the SingleLine dataset, then the 3 million most challenging SingleLine data is selected for later training stages based on the model’s loss. In the second stage, CodeExecutor is pre-trained with the 3 million difficult SingleLine data, along with the Tutorial dataset. In the third stage, CodeExecutor is pre-trained with the 3 million difficult SingleLine data, the entire Tutorial dataset, and the CodeNetMut dataset.

heading: D Qualitative Examples
text: Additional examples are shown here. Figure 4 shows an example that covers all the categories of Python programming knowledge in Table 4. CodeExecutor generates the same trace as ground truth.
Figure 5 is an example of performing division calculations with decimals. CodeExecutor is able to produce the correct first fifteen digits and makes errors in the remaining two digits.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The "Limitations” section, which is after the conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: The "Ethical Statement” section, which is before the references.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and the first section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3, 4 and 5.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3, 4 and 5.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 3, 4 and 5.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3, 4 and 5.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The "Ethical Statement” section, which is before the references.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.
C 3 Did you run computational experiments? Section 5 and 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and appendix B.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and appendix A.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 5 and a handbook in the data package of supplemental material.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5, the "Ethical Statement” section and a handbook in the data package of supplemental material.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 5 and a handbook in the data package of supplemental material.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Limitations
Several limitations of CodeExecutor, such as its application to only Python, the lack of faithfulness in the results produced, and the maximum length limit for trace generation, point toward interesting directions for future work. Programming Language One limitation of our current model is that it is currently only applied to Python, which limits its use and effectiveness in executing programs written in other programming languages. This highlights the need for future work to expand the model’s applicability to other languages. Faithfulness The result may not be faithful enough when handling difficult examples, such as those with complex logic, long loops, or many branches. For example, we observe that in two complicated programs that both contain the assignment “alpha = list(’abcdefg’)”, our model correctly predicts the value of “alpha” in one case but incorrectly in the other. The lack of faithfulness needs to be studied for further research on code execution. Generation Window Size We limit the length of generated trace to 1024 tokens. It can be a limitation for programs with long execution traces, particularly those with loops. Improving the ability of Transformers to handle longer sequences (Tay et al. , 2021, 2022) would likely be beneficial for the code execution task