SlowGuess/NeurIPS_2023_OCR huggingface dataset:
John Yang Akshara Prabhakar Karthik Narasimhan Shunyu Yao
Department of Computer Science, Princeton University {jy1682, ap5697, karthikn, shunyuy}@princeton.edu
Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash [29], Spider [51], and MBPP [4] datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct [48] and Plan & Solve [40]. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. *
The art of computer programming is naturally an interactive process. When a human programmer writes code, she relies on several iterations of a 'write-execute-test' loop in order to iteratively refine solutions, plan changes, test sub-modules, and solve ambiguities by checking execution behavior. While this is reminiscent of other human endeavors like writing, code compilation and execution produce exact results that provide a deterministic form of feedback to make the refinement process more straightforward. Depending on the observed results, programmers perform various levels of debugging and rewriting, and continue the process until their code satisfies the requirements.
There has been increasing interest in recent years around the development of models that can automatically generate code given a specification in natural language [17, 43, 13, 27, 24]. Powered by large-scale pre-training over thousands of codebases [2, 21, 18], these models have shown solid performance on static benchmarks like HumanEval [9], APPS [19], MBPP [4], CodeXGLUE [30]. However, generating code in a static, sequence-to-sequence or auto-regressive fashion has several drawbacks: 1) simple errors (even typos) can propagate and there is no chance for recovery or
*Code and data available at https://intercode-benchmark.github.io/
Figure 1: Overview of InterCode. Setting up an interactive code environment with InterCode requires a Dockerfile, dataset, reward function definition, and a small amount of subclass implementation. The interactive loop between agent and environment closely mirrors real world software development processes. While InterCode task performance is generally quantified as a binary  $0/1$  completion score, InterCode allows for the design of more complex evaluation criteria that can incorporate execution output and the effects of interaction on the state space.
revision, 2) there is a disconnect between the code generation process and its downstream execution on the desired software and hardware environment, and 3) there is little room for human intervention or collaboration in the code generation process.
Recently, some works have proposed the use of execution feedback or interaction [44] to benefit code generation models [23, 20, 45, 19]. However, these papers consider their own individual setup and are difficult to compare with one other due to the use of different compilers, execution environments, feedback signals, and assumptions on the interactive process such as human participation to create task descriptions or provide natural language feedback. This makes it difficult to compare existing methods for code generation and to clearly understand the benefits of interactive generation.
To address these issues, we propose InterCode, the first standard coding benchmark designed natively with an interactive execution environment. Closely mimicking the human decision-making process, InterCode allows a coding agent to interactively receive feedback from compilers/interpreters that execute its code, and to submit further refinements. We design InterCode to be like a standard reinforcement learning (RL) environment that requires minimal human intervention and one in which generated code is treated as actions, which are executed to reveal observations. Our framework is (1) language and platform agnostic and can easily be used for new coding problems, (2) uses self-contained Docker environments to provide safe execution, and (3) compatible out-of-the-box with traditional seq2seq generation methods, while also enabling and empowering the development of new interactive techniques.
We demonstrate the power of the framework by implementing Bash, SQL, and Python tasks within InterCode, building on pre-existing static datasets [58, 29, 4]. We perform experiments across diverse models and prompting methods, including ReAct [48] and Plan & Solve [40]. Our findings concretely showcase the benefits of interaction towards solving coding tasks, discuss the distribution of distinct code understanding challenges across different task settings, and explore the ease with which new tasks and datasets can be defined using InterCode.
To summarize, our paper makes the following contributions:
- We develop InterCode, a new, universal framework for interactive code generation, which provides ease of use, extensibility, and safety.
- Using InterCode, we perform a comprehensive evaluation of state-of-the-art models and identify several avenues for improvements.
- We release our framework as a new benchmark along with useful empirical tools to customize any new static code datasets into interactive tasks.
Interactive environments for coding. Most coding benchmarks (e.g. SQL - Spider [51], KaggleDBQA [25]; Bash - NLC2CMD [1], NL2Bash [29]; Python - HumanEval [9], APPS [19], MBPP [4], CodeXGLUE [30], CodeNet [35]) frame the coding problem as a sequence transduction problem (from instruction to code), rather than an interactive decision making problem with an execution environment. Attempts have been made to simulate interaction by developing conversational, dialogue-style [53, 52], multi-step problem solving [33] datasets, which involve pre-annotated human-designed queries. The work closest to InterCode has been recent explorations of Python Jupyter Notebooks as a natural choice for interactive coding [20, 23, 50]. However, task data and settings often constrain allowed actions to a closed domain of code and libraries [23, 50], use evaluation procedures or metrics that may not generalize [20], require human-in-the-loop participation (i.e. create task contexts, write problems, evaluate execution per task instance) [23], or are Python-exclusive [20, 23, 50, 45]. InterCode provides a more general purpose foundation defining interactive coding tasks that enables easy construction of diverse task settings, can have any programming language(s) as the action space, and has automatic, execution-based evaluation.
Execution-based evaluation for coding. Evaluation for NL-to-code generation models has recently shifted away from surface form similarity metrics (BLEU [34, 2], ROUGE [28], Exact Match) towards execution oriented ratings (unit tests [4, 9, 20, 23, 19], output matching [15, 20, 58]). The rigidity of surface form analysis overlooks code syntax features, ignores execution effect, or over-penalizes alternative solutions [59]. On the contrary, execution-based assessment is a more thorough and comprehensive score of code functionality [19] and is a more natural fit for open-domain program usage that does not constrain code generation to a subset of the language space [45]. However, for newer benchmarks and datasets that put forth task definitions incorporating execution-based evaluation (APPS [19], ExeDS [20], ODEX [45]), the fundamental code generation task (Context + Code → Execution → Score) is still devoid of interaction. InterCode combines execution-based evaluation with flexible task construction, enabling more diverse problem-solving paradigms within a unified coding task formulation. InterCode's use of virtual containers as execution sandboxes protect against harmful actions and allow for advanced evaluation criteria beyond the aforementioned ones.
Methods for interactive or execution-based coding. The value of generative code models and interactive problem solving has motivated a recent proliferation of work to augment reasoning capabilities' of existing language models [48, 37, 40, 47, 56, 11] or propose new modeling techniques to tackle coding as a sequential decision making and reasoning tasks [6, 10, 16, 27, 8, 24], of which evaluation is unit test based. Approaches that leverage execution typically use re-ranking [57, 32, 49, 54] or majority vote [10, 27, 36] to decide on a final prediction. Additional work also explores incorporating human-in-the-loop [7, 22], compilers [41], and text [42, 55] feedback. A common thread among these contributions is that 1) the task setting can only provide the investigated form of feedback and 2) sought-after capabilities are exemplified by strong performance on favorably curated tasks and datasets, rendering comparisons across benchmarks tedious. InterCode has the potential to standardize the evaluation of these methods because 1) the interactive coding task is a conglomeration of many interesting interaction, reasoning, and decision-making challenges and 2) InterCode's task construction makes it possible to incorporate a wide variety of sources of feedback.
3 The InterCode Benchmark
The InterCode benchmark formalizes interactive coding with execution feedback as a partially observable Markov decision process (POMDP)  $(\mathcal{U},\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},\mathcal{R})$  with instruction space  $\mathcal{U}$ , state
Action Space	Environment	Dataset	Reward Function
Bash	Ubuntu Terminal	NL2Bash [29] (200)	Latest Std. Output + File System Δ
SQL	MySQL Database	Spider 1.0 [51] (1034)	Latest Std. Output
Python	Python Interpreter	MBPP [4] (117)	Submitted Function
Table 1: Rundown of the two environments with Bash and SQL as action spaces developed using the InterCode framework. The numbers in parentheses refer to the number of task instances adopted from each dataset. Each environment is defined in under 200 lines of code total. Specific discussion of the environment construction and reward function can be found in § A.2 and § A.3
space  $S$ , action space  $\mathcal{A}$ , observation space  $\mathcal{O}$ , transition function  $\mathcal{T}: S \times \mathcal{A} \to S$ , and reward function  $\mathcal{R}: S \times \mathcal{A} \to [0,1]$ . Given a coding instruction  $u \in \mathcal{U}$  in natural language, an agent issues code or a special submit keyword as an action  $a_t \in \mathcal{A}$ . An action is admissible [46] if it can be parsed and executed in the compiler/interpreter environment, and an admissible action incurs a change in the latent state space  $s_{t+1} \in S$ , and an execution feedback as observation  $o_{t+1} \in \mathcal{O}$ . The interaction loop repeats until the submit action is issued, wherein the task episode ends and a reward  $r = \mathcal{R}(s_T, \text{submit}) \in [0,1]$  is computed, with 1 representing task completion. We use the Success Rate (SR) metric, defined as the proportion of task episodes where  $r = 1$ . We also define the Error % metric, which is the percentage of non admissible actions across task episodes.
3.2 Construction pipeline
At a high level, InterCode decomposes the construction of an interactive coding task into three modular parts: (1) environment construction, (2) data collection, and (3) reward design. This workflow allows for the safe execution of transition functions, flexible reward design, and convenient adaptation of existing instructions to an interactive setting.
Docker-based environments. InterCode uses Docker [31] virtual containers as a general-purpose execution sandbox. Given a Dockerfile that defines a system and execution entrypoint, InterCode creates a corresponding, stateful virtual container that hosts the desired state space and transition function. We choose Docker as the basis of InterCode's environment construction for its safe execution in virtual containers, reproducibility of a Dockerfile across any Docker-equipped machine, and excellent coverage of application code, libraries, and dependencies offered by the Dockerfile DSL.
Data collection. InterCode requires that a dataset has at minimum two fields: query, a natural language instruction  $u \in \mathcal{U}$ , and gold, an answer or code block that is a procedure for generating the correct answer. We define these conditions to make it easy to adapt existing text-to-code datasets to an interactive setting while also leaving plenty of bandwidth for constructing new tasks and datasets.
Reward design. Across a single task episode, the action, observation, and state modification (if any) per interaction loop are implicitly logged by InterCode. InterCode's default reward function determines task completion via an exact match of the agent's execution output (observation and state modifications) against the gold command, where 1 is awarded only if all components match. Since Exact Match is usually too stringent of an evaluation criteria, InterCode exposes a reward function endpoint that has access to both the interaction history and the execution container, allowing for custom reward function definitions that can incorporate multiple signals.
Following the procedure discussed in Section 3.2, we create two separate InterCode based environments where Bash and SQL are the action spaces respectively. Table 1 summarizes them.
InterCode-Bash. We define a bash shell within an Ubuntu Operating System as the task setting. To evaluate an agent's ability to adapt generations to different situations, we architect four distinct file systems that can be swapped into the Bash environment by changing a single line in the Dockerfile.
We bootstrap the NL2Bash [29] dataset (which lacks specificity in queries and grounding to any underlying file system, preventing it from being used directly for interactive evaluations) to create an interactive coding task where an agent completes an instruction via bash actions. Transferring NL2Bash to the interactive task setting requires simple transformations to ground instructions and gold code blocks in the file system. First, we consider a subset of 1000 commands with each
having  $\geq 4$  utilities. We then filter out commands that are non-UNIX, non-Linux, or use utilities we currently do not support (eg. "ssh", "sudo", time, and GUI-dependent utilities). Finally, we enhance under-specified commands with specific file names/dictionary names/paths and update deprecated utilities/flags. The resulting 200 commands are grouped into 4 disjoint sets, 3 of which were grounded to custom-designed file systems, while one set is file-system agnostic. This categorization allows for a comprehensive evaluation of different command-grounding scenarios.
The InterCode-Bash dataset instructions typically make one or both of the following two types of requests. It either 1. Requests information that can be answered via execution output (i.e. "How many files...", "What is the size of...", "Where is <file> stored?") or 2. Requests a change to the location/ configuration/content of a file or folder (i.e. "Move dir1 folder...", "Set permissions of...", "Append a line to..."). Therefore, we define a custom reward function that evaluates an agent's performance against file system modifications and the latest execution output. Execution output is graded with a simple lexical similarity function. File system assessment is done in two parts. First, a comparison of the agent's and gold command's list of file system changes (list of [path, modification type ∈ [added, changed, deleted]]) entries) reveals any extraneous or missing changes. Second, md5sum hashes of each commonly edited file path are compared to determine if an added or changed file was altered correctly. A max score of 1 is achieved only if the correct file paths are changed, the changes are correct, and the latest execution output matches the gold command output exactly. Additional Bash statistics and design details are discussed in § A.2.
InterCode-SQL. We write a Dockerfile that defines a SQL interpreter within a MySQL database as the task setting. To create the databases and tables necessary for the task dataset, we write type resolution scripts and perform database conversions using thesqlite3mysql [38] Python library to adapt the Spider [51] database and table schema to a MySQL format. We then consolidate all setup code into a single, unified MySQL .sql dump that contains the complete set of schemas for all tables across 20 different databases. On container start-up, this file is invoked automatically, creating and populating databases with tables and tables with records.
The Spider [51] dataset is a large-scale cross-domain dataset originally meant for evaluating SQL query generations from natural language questions. We adapt the development set, which contains 1034 task instances, and remove all extraneous columns aside from the natural language questions and gold SQL command. The instruction and gold values do not require any additional pre-processing to be compatible with the MySQL task environment.
Finally, we employ Intersection over Union  $(IoU)$ , or more formally the Jaccard Index, to quantify the correctness of the latest execution output generated by the agent against the gold output, where both outputs are a list of records. A non-tabular execution output receives a reward of 0 by default. Among the items that lie in the intersection of the agent and gold execution outputs, we also apply a penalty if the records are in the incorrect order. To quantify how sorted the agent output is relative to the gold output, we lean on Kendall's  $\tau$  and adjust the output range to  $[0,1]$ . The  $IoU$  score is then directly scaled by this coefficient. All in all, only a correctly ordered list with the exact set of records found in the gold output receives a score of 1. Visualizations like Figure 1 for SQL along with a more extensive implementation discussion for this environment are in § A.3
InterCode-Python. In this setting, we define a Python interpreter running within an Ubuntu operating System as the task setting. The Dockerfile can be configured to run any Python version. The interpreter is not initialized with any dependencies, but PyPI packages can be installed and used by the agent.
We use the MBPP [4] dataset which presents the code completion task of synthesizing Python code from a method header and docstring. Evaluation of correctness is performed with an associated set of unit tests given by MBPP. The MBPP dataset is straightforward to adapt to the interactive setting, requiring no modifications to the query or evaluation components. Finally, we directly inherit MBPP's evaluation procedure of proportion of unit tests passed. With InterCode, it is easy to use existing datasets to evaluate how well models can use different programming languages as actions.
**Validations.** To verify the functionality of action execution in the task environment and the correctness of custom reward functions, we write testing scripts for both Bash and SQL that pass the gold command in as a dummy agent's action to ensure that the command is admissible and executes without error, and to verify that the reward received by the command is 1. To confirm that InterCode's dataset specification is enforced across multiple accepted file formats, we define a custom InterCode data loader class which is then rigorously unit tested.
Figure 2: Overview of Prompting Strategies adjusted for evaluation on InterCode. The "Try Again" termination constraint is conditioned on reward  $= 1$ , while ReAct [48] and Plan & Solve [40] are determined by the agent itself. This is because the purpose of the "Try Again" method is to explore how capable agents are at error correction from feedback, while the other two are more concerned with the overall success of general problem-solving strategies.
We perform preliminary experiments to gauge the proficiency and behavior of current large language models on interactive coding tasks with Bash and SQL. To observe and elicit relevant reasoning skills, we draw on several existing prompting strategies that have been put forth to augment language models' reasoning and problem-solving skills. We apply these prompting strategies to models across the following three families: OpenAI (text-davinci-003, gpt-3.5-turbo, gpt-4), PaLM-2 (text-bison-001, chat-bison-001) [3], and Open Source (Vicuna-13B [12], StarChat-16B [26]).
Figure 2 visualizes the four adjusted prompting strategies we evaluate on InterCode.
Single Turn is a zero-shot attempt. A model is given a simple description of the task setting and asked to generate code in a specific programming language that would address the query. The first generation in response to the user's question is then evaluated in the InterCode environment.
"Try Again" is an iterative feedback set up. In the initial message, the agent is informed of the task setting and its interactive nature; an agent has multiple turns to interact with the system, wherein each turn, upon generating an action, the execution output of the action is fed back as an observation. This continues until a reward of 1 (task completion) is achieved or the number of turns  $(n)$  is exhausted. The agent's position in this approach is meant to mirror human software development as closely as possible. The goal of this method is to probe language models' raw interactive coding abilities in addition to illustrating the benefits and different challenges that arise in interactive coding tasks.
ReAct and Plan & Solve. We write prompts and design workflows that follow the text and task configurations described in ReAct [48] and Plan & Solve [40] as faithfully as possible. For these two approaches, the termination of a task episode is conditioned upon the agent's own judgment, as our goal with these methods is to gauge the transferability to and efficacy of existing reasoning frameworks with respect to the interactive coding task. Full prompt templates are included in §B.7.
5.1 Base models comparison
Task performances. We first compare the success rate of models in the Single Turn and Try Again settings for both the InterCode-Bash and SQL datasets. From Table 2 and Table 3, we observe
InterCode-SQL
Model / Hardness	Single Turn	Try Again (n = 10)
Easy	Med	Hard	Extra	All	Easy	Med	Hard	Extra
text-davinci-003	20.6	4.9	1.7	0.0	7.4	32.4	14.6	5.2	4.2
gpt-3.5-turbo	22.6	8.3	5.7	3.6	10.5	72.5	44.3	43.7	21.1
gpt-4	19.8	7.2	4.6	3.0	9.1	87.5	76.7	66.7	52.4
text-bison-001	23.8	10.9	5.7	0.6	11.5	27.0	12.3	5.7	0.6
chat-bison-001	18.5	6.5	4.0	0.0	7.9	22.2	7.8	6.9	0.0
Vicuna-13B	8.1	1.3	0.6	0.0	2.6	18.9	3.4	1.7	0.0
StarChat-16B	21.8	7.4	2.9	0.0	8.9	22.3	8.5	2.9	1.2