InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
Authors: John Yang
Akshara Prabhakar
Karthik R Narasimhan
Shunyu Yao
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: natural language processing
code generation
interactive machine learning
TL;DR: We introduce a lightweight, flexible, and easy-to-use framework for designing interactive code environments.

Abstract: Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode’s viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages.

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
John Yang Akshara Prabhakar Karthik Narasimhan Shunyu Yao Department of Computer Science, Princeton University (jy1682, ap5697, karthikn, shunyuy)@prrnnceton edu
Abstract
Humans write code in a fundamentally interactive manner and rely constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-ton -o-ccde sequence transduction process, which has the potential for propagation and disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework language and platform agnostic, uses self-container Docker environments to provide safe and reproducible execution, and compatible out-of-the-bol with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash (29). Spider [51], and MBPP 4 datasets. We demonstrate InterCode's s viability testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct 488) and Plan & Solve (40] Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages,
Introduction
The art of computer programming is naturally an interactive process. When a human programmer writes code, she relies on several iterations of "ritt-exeecute-tes' loop order iteratively refine solutions, plan changes, test sub-modules, and solve ambiguities by checking execution behavior. While this is reminiscent of other human endeavors like writing, code compilation and execution produce exact results that provide deterministic form of feedback make the refinement process more straightforward. Depending on the observed results, programmers perform various levels of debugging and rewriting, and continue the process until their code satisfies the requirements.
There has been increasing interest in recent years around the development of models that can automatically generate code given specification in natural language [17] 43 13 27 [24. Powered by large-scale pre-training over thousands of codebases 2T] these models have shown solid performance static benchmarks like HumanFval 99. APPS [9 MBPP 4 CodeXGLUUE [30] However, generating code in static, segueence-to-equemee or auto-regressive fashion has several drawbacks: 1) simple errors (even typos) can propagate and there is no chance for recovery or
"Code and data available at
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Query
Agent
InterCode Env
(Bash) "Move images my Desktop and Downloads compressed folder
Provided Task Designer
Interact
USB drive."
Speciffe Example of Interactive Loop & Evaluation of Agent Performance for InterCode
Question
Bash Env
Find all text files in the testbed directory and subdiractories and concatenate them into a single file
Agent Command(s)
1. testbed 2. find type f -name; touch concated. txt echo hi.txt concated. txt (additional cmds...)
Agent
pwd
Bash Env
Agent
Answer
State
Ls
Bash Env
Agent Latest
standard Output
A concated.
testbed dirl dir2
X
c file2.txt
Calculate Lexical
Compare File Systems Check common
Agent
Task Gold Standard Output
edits identical
find /testbeed -type f -nane ...txt?...
concated. txt
Bash Env /testbed...
/testbel/dtrr3teexttilee.c
Bash Env
Gold Command
Agent
submit
find /testbed ttype 's.txt! exec cat O /testbad/concaaedd.tt
Figure 1: Overview of InterCode. Setting up an interactive code environment with InterCode requires a Dockerfile, dataset, reward function definition, and a small amount subclass implementation. The interactive loop between agent and environment closely mirrors real world software development processes. While InterCode task performance is generally quantified as binary 0/1 completion score, InterCode allows for the design of more complex evaluation criteria that can incorporate execution output and the effects of interaction on the state space.
revision, 2) there is disconnect between the code generation process and its downstream execution on the desired software and hardware environment, and 3) there is little room for human intervention collaboration in the code generation process.
444 to benefit code Recently, some works have proposed the use of execution feedback or interaction
generation models (23 20 45 19I. However, these papers consider their own individual setup and are difficult to compare with one other due to the use of different compilers, execution environments, feedback signals, and assumptions the interactive process such as human participation create task descriptions or provide natural language feedback. This makes it difficult to compare existing methods for code generation and to clearly understand the benefits of interactive generation.
To address these issues, we propose InterCode, the first standard coding benchmark designed natively with an interactive execution enviromment. Closely mimicking the human decision maaking process, InterCode allows coding agent to interactively receive feedback from compilers/interpreeers that execute its code, and to submit further refinements. We design InterCode to be like standard reinforcement learning (RL) environment that requires minimal human intervention and one which generated code is treated as actions, which are executed to reveal observations. Our framework is (1) language and platform agnostic and can easily be used for new coding problems, (2) uses self-contained Docker environments to provide safe execution, and (3) compatible out-of-the-box with traditional seq2seq generation methods, while also enabling and empowering the development of new interactive techniques.
We demonstrate the power of the framework by implementing Bash, SQL, and Python tasks within InterCode, building on pre-existing static datasets [38] 29 4] We perform experiments across diverse models and prompting methods, including ReAct 48 and Plan & Solve [40]. Our findings concretely showcase the benefits of interaction towards solving coding tasks, discuss the distribution of distinct code understanding challenges across different task settings, and explore the ease with which new tasks and datasets can be defined using InterCode.

---

To summarize, our paper makes the following contributions:
We develop InterCode, new, universal framework for interactive code generation, which provides ease of use, extensibility, and safery.
Using InterCode, perform comprehensive evaluation of state-of-the-ar models and identify several avenues for improvements.
We release our framework new benchmark along with useful empirical tools to customize any new static code datasets into interactive tasks.
Related Work
Interactive environments for coding, Most coding benchmarks (e.g. SQL SSider
[51 KaggleD-
BQA [25]; Bash- NL.C2MM [. NL.2Bash [29]: Python HumanEval [9] APPS 19), MBPP 4
CodeXGLUE [00 CodeNet [35 frame the coding problem as a sequence transduction problem (from instruction to code), rather than an interactive decision making problem with an execution envi- ronment. Attempts have been made to simulate interaction by developing conversational, dialogue- style [53 52], multi-step problem solving 33 datasets, which involve pre-annotated dusannde queries. The work closest to InterCode has been recent explorations of Python Jupyter Notebooks as a natural choice for interactive coding [20 23 50]. However, task data and settings often constrain allowed actions to closed domain of code and libraries 23 [50], evaluation procedures or met- rics that may not generalize (20). require human-in-the-lowe
write problems, evaluate execution per task instance) [23] participation (i.e. create task contexts,
Python-exclusive (20 23 50 45
InterCode provides a more general purpose foundation defining interactive coding tasks that enables easy construction of diverse task settings, can have any programming language(s) the action space, and has automatic, execution-based evaluation.
Execution-based evaluation for coding. Evaluation for NL-to-code generation models has recently shifted away from surface form similarity metrics (BLEU 34 2). ROUGE (28 Exact Match) towards execution oriented ratings (unit tests [4] 9 20
23 19 output matching [5 20 58)) The rigidity of surface form analysis overlooks code syntax features, ignores execution effect, or over-penalizes
alternative solutions [59]. On the contrary, execution-based assessment is a more thorough and
comprehensive score of code functionality [9 and a more natural fit for open-domain program usage that does not constrain code generation to subset of the language space (45) However, for newer benchmarks and datasets that put forth task definitions incorporating execution based evaluation (APPS [9) ExeDS 200, ODEX 45) ) the fundamental code generation task (Context + Code - Execution - Score) is still devoid of interaction. InterCode combines execution-bal bbasd evaluation with flexible task construction, enabling more diverse problem-solving paradigms within unified coding task formulation. InterCode's 's use of virtual containers execution sandboxes protect against harmful actions and allow for advanced evaluation criteria beyond the aforementioned ones.
Methods for interactive or execution-based coding. The value of generative code models and interactive problem solving has motivated recent proliferation of work to augment reasoning 40 47 (56] 11] or propose new modeling techniques
capabilities' of existing language models [8, 37 to tackle coding as a sequential decision making and reasoning tasks (6] 10 16 27 24 of which evaluation is unit test based. Approaches that leverage execution typically use re-ranking (57 (32
49 (44) majority vote [10 27 36) decide
final prediction. Additional work also explores
incorporating human-in-the-logy 7 22 compilers [41, and text 42 555 feedback. A common
thread among these contributions is that 1) the task setting can only provide the investigated form of feedback and 2) sought-after capabilities are exemplified by strong performance on favorably curated tasks and datasets, rendering comparisons across benchmarks tedious. InterCode has the potential to standardize the evaluation of these methods because 1) the interactive coding task is a conglomeration of many interesting interaction, reasoning, and decision becision-making challenges and 2) InterCode's task construction makes it possible to incorporate wide variety of sources of feedback.
3
The InterCode Benchmark
3.1 Formulation
The InterCode benchmark formalizes interactive coding with execution feedback as partially observable Markov decision process (POMDP) O,O, R) with instruction space U, state
3

---

Action Space
Environment
Dataset
Reward Function
Bash
Ubuntu Terminal
NL2Bash 22 (200)
Latest Std. Output + File System A
SQL
MySQL Database
Spider 1.0 [51 (1034)
Latest Std. Output
Python
Python Interpreter
MBPP 4 (117)
Submitted Function
Table 1: Rundown of the two environments with Bash and SQL action spaces developed using the InterCode framework. The numbers in parentheses refer to the number of task instances adopted from each dataset. Each environment is defined in under 200 lines of code total. Specific discussion of the environment construction and reward function can be found A.2 and $ A.3
space S, action space A, observation space O transition function T S function R. : S x A + [0, 1]. Given a coding instruction w E U in natural language, an agent issues code a special submit keyword
S, and reward
action a, E A. An action is admissible [46) if it can be parsed and executed in the compiler/interpreee environment, and an admissible action incurs change in the latent state space St41 E S, and an execution feedback as observation 0++1 E O. The interaction loop repeats until the submit action is issued, wherein the task episode ends and a reward r (&T, submit) E [0, 1] is computed, with representing task completion. We use the Success Rate (SR) metric, defined as the proportion of task episodes where = 1. We also define the Error % metric, which is the percentage of non admissible actions across task episodes.
3.2 Construction pipeline
At a high level, InterCode decomposes the construction of an interactive coding task into three modular parts: (1) environment construction, (2) data collection, and (3) reward design. This workflow allows for the safe execution of transition functions, flexible reward design, and convenient adaptation of existing instructions to an interactive setting.
Docker-based environments. InterCode uses Docker [1] virtual containers as a general-purpose execution sandbox. Given Dockerfile that defines system and execution entrypoint, InterCode creates a corresponding, stateful virtual container that hosts the desired state space and transition function. We choose Docker as the basis of InterCode's environment construction for its safe execution in virtual containers, reproducibility of a Dockerf ill across any Docker-equipped machine, and excellent coverage of application code, libraries, and dependencies offered by the Dockerfill ile DSL.
Data collection. InterCode requires that dataset has at minimum two fields: query, natural language instruction E U, and gold, an answer code block that is a procedure for generating the correct answer. We define these conditions to make it easy to adapt existing text-to-code datasets to an interactive setting while also leaving plenty of bandwidth for constructing new tasks and datasets.
Reward design. Across single task episode, the action, observation, and state modification (if any) per interaction loop are implicitly logged by InterCode. InterCode's 's default reward function determines task completion via an exact match of the agent's 's execution output (observation and state modifications) against the gold command, where 1 is awarded only if all components match. Since Exact Match is usually too stringent an evaluation criteria, InterCode exposes reward function endpoint that has access to both the interaction history and the execution container, allowing for custom reward function definitions that can incorporate multiple signals.
3.3 Implementations
Following the procedure discussed Section 3.2 ments where Bash and SQL are the action spaces respectively. Table summarizes them.
we create two separate InterCode based environ-
InterCode- -ashh We define a bash shell within an Ubuntu Operating System as the task setting. To evaluate agent's ability to adapt generations to different situations, we architect four distinct file systems that can be swapped into the Bash environment by changing a single line in the Dockerfile
We bootstrap the NL2Bash 29 dataset (which lacks specificity in queries and grounding to any underlying file system, preventing it from being used directly for interactive evaluations) to create an interactive coding task where an agent completes an instruction via bash actions. Transferring NI.2Bash to the interactive task setting requires simple transformations to ground instructions and gold code blocks in the file system. First, consider subset of 1000 commands with each

---

having 24 4 utilities. We then filter out commands that are non-UNIX non-Linux, or use utilities we currently do not support (eg. "ssh", "sudo", time, and GUI-dependent utilities). Finally, we enhance under-specified commands with specific file names/directory names/paths and update deprecated utilities/flags. The resulting 200 commands are grouped into 4 disjoint sets, of which were grounded to custom-designed file systems, while one set is file-system agnostic. This categorization allows for a comprehensive evaluation of different command- command-grounding scenarios.
The InterCode-Bash dataset instructions typically make one or both of the following two types of requests. either Requests information that can be answered via execution output (i.e. "How many files..., " "What is the size of...", " "Where is <file> stored?") or 2. Requests change to the location/- configuration/content of a file or folder (i.e. "Move dir1 folder..." ". "Set permissions of...", "Append a line to..."). Therefore. we define a custom reward function that evaluates an agent's 's performance against file system modifications and the latest execution output. Execution output is graded with a simple lexical similarity function. File system assessment is done in two parts. First, a comparison of the agent's and gold command's list of file system changes (list of [path, modifi cation type [added, changed, deleted] ] entries) reveals any extraneous or missing changes. Second, md5sum hashes of each commonly edited file path are compared to determine an added or changed file was altered correctly. A max score of I is achieved only if the correct file paths are changed, the changes are correct, and the latest execution output matches the gold command output exactly. Additional Bash statistics and design details are discussed in $ AA.22
InterCode-SQL. We write a Dockerfile that defines a SQL interpreter within MySQL database as the task setting. To create the databases and tables necessary for the task dataset, we write type resolution scripts and perform database conversions using the sqlite3mysq] (38 Python library to adapt the Spider [5]] database and table schema to a MySQL format. We then consolidate all setup code into a single, unified MySQL sql dump that contains the complete set of schemas for all tables across 20 different databases. On container start-up, this file is invoked automatically, creating and populating databases with tables and tables with records.
The Spider [51] dataset is a large-scale cross dross-domain dataset originally meant for evaluating SQL query generations from natural language questions. We adapt the development set, which contains 1034 task instances, and remove all extraneous columns aside from the natural language questions and gold SQL command. The instruction and gold values do not require any additional pre-processing to be compatible with the MySQL task environment.
Finally, we employ Intersection over Union (loU), or more formally the Jaccard Index, quantify the correctness of the latest execution output generated by the agent against the gold output, where both outputs are a list of records. A non-tabular execution output receives a reward of by default. Among the items that lie in the intersection of the agent and gold execution outputs, we also apply a penalty if the records are in the incorrect order. To quantify how sorted the agent output is relative to the gold output, we lean on Kendall's T and adjust the output range to [0. 1]. The IoU score is then directly scaled by this coefficient. All in all, only a correctly ordered list with the exact set of records found in the gold output receives score of 1. Visualizations like Figure T for SQL. along with more extensive implementation discussion for this environment are in $ A.3
CeeerCode-Python In this setting, we define a Python interpreter running within an Ubuntu operating System as the task setting. The Dockerfile can be configured run any Python version. The interpreter is not initialized with any dependencies, but PyPI packages can be installed and used by the agent.
We use the MBPP 4 dataset which presents the code completion task of synthesizing Python code from a method header and docstring. Evaluation of correctness is performed with an associated set of unit tests given by MBPP. The MBPP dataset is straightforward to adapt to the interactive setting, requiring no modifications to the query or evaluation components. Finally, we directly inherit MBPP's evaluation procedure of proportion of unit tests passed. With InterCode, it is easy to use existing datasets to evaluate how well models can use different programming languages as actions.
Validations. To verify the functionality of action execution in the task environment and the correctness of custom reward functions, we write testing scripts for both Bash and SQL that pass the gold command in as dummy agent's 's action to ensure that the command is admissible and executes without error, and to verify that the reward received by the command 1. To confirm that InterCode's dataset specification is enforced across multiple accepted file formats, we define a custom InterCode data loader class which is then rigorously unit tested.
5

---

Single Turn
Plan & Solve
Initial Mag.
Question What the country code and first name..
Action <actions
Evaluate after first
Initial Mag.
i
You are helping user who crying do domething in MysQI DE, Respond with tho correct SQL command.
underssadd the problens and deviso plan. Thon, lot's corry out the plan..
action
Question
What ia the country code and firat name...
"Try Again"
ReAct
Agent
Initial Msg.
Initial Mag.
generator helping................ MySQL
Interact MySQL using interisoving Thought Action,
Execute Mag. You will now own plan. Interact with MySQu, DB SQL...
Question
Question
What is the country code and first namne..
What in the country code and first name...
Agent
Up to plan langth
Agent
Agent
factiono
Up to
<thought> <action>
Up to (
caté. outputt, <atep>
Observation
Obsorvation
times
Observation
times
<atd. output,
<atd. outgut>
Terminate when plan completed or turns exceeded
Terminate when 1 or exceeds turns
Terminate when thought chain
finishes or exceeds n turns
Figure 2: Overview of Prompting Strategies adjusted for evaluation on InterCode. The "Try Again" termination constraint is conditioned reward 1, while ReAct [48] and Plan & Solve (40 determined by the agent itself. This is because the purpose of the "Try Again" method to explore how capable agents at error correction from feedback, while the other two more concerned with the overall success of general problem-solving strategies.
Methods
We perform preliminary experiments to gauge the proficiency and behavior of current large language models interactive coding tasks with Bash and SQL. To observe and elicit relevant reasoning skills, we draw on several existing prompting strategies that have been put forth to augment language models" reasoning and problem-solving skills. We apply these prompting strategies to models across the following three families: OpenAl (text-davinci-O0s
-.t-tbb gpt-4), PaLM-2 33 and Open Source (Vicuna-133 13B [12), StarChat-166 (26)).
(text-bison-
Figure visualizes the four adjusted prompting strategies we evaluate on InterCode.
Single Turn is zero-shot attempt. model is given a simple description of the task setting and asked to generate code in specific programming language that would address the query. The first generation in response to the user's s question is then evaluated in the InterCode environment.
"Try Again" is an iterative feedback set up. In the initial message, the agent is informed of the task setting and its interactive nature; an agent has multiple turns to interact with the system, wherein each turn, upon generating
action, the execution output of the action is fed back continues until a reward of 1 (task completion) is achieved or the number of turns (n) is exhausted. The agent's position in this approach is meant to mirror human software development closely as possible. The goal of this method is to probe language models' raw interactive coding abilities in addition to illustrating the benefits and different challenges that arise in interactive coding tasks.
observation. This
ReAct and Plan & Solve. We write prompts and design workflows that follow the text and task configurations described in ReAct [48] and Plan & Solve [40] faithfully as possible. For these two approaches, the termination of a task episode is conditioned upon the agent's 'ss own judgment, as our goal with these methods is to gauge the transferability to and efficacy of existing reasoning frameworks with respect to the interactive coding task. Full prompt templates are included B
5 Experiments
5.1 Base models comparison
Task performances. We first compare the success rate models in the Single Turn and Try Again settings for both the InterCode-Bau and SQL datascts. From Table 2 and Table 3 we observe

---

InterCode-SQL
Single Turn
Try Med
Again (n
10) Extra
Model / Hardness
Easy
Med
Hard
Extra
All
Easy
Hard
All
text-davinci-003
20.6
4.9
1.7
0.0
7,4
32.4
14.6
5.2
4.2
15.6
gpt-3.5-turbo
22.6
8.3
5.7
3.6
10.5
72.5
44.3
43.7
21.1
47.3
gpt-4
19.8
7.2
4.6
3.0
9.1
87.5
76.7
66.7
52.4
73.7
text-bison-001
23.8
10.9
5.7
0.6
11.5
27.0
12.3
5.7
0.6
12.9
chat-bison-001
18.5
6.5
4.0
0.0
7.9
22.2
7.8
6.9
0.0
9.9
Vicuna-13B 8.1
1.3
0.6
0.0
2.6
18.9
3.4
1.7
0.0
6.3
StarChat-16B 21.8
7.4
2.9
0.0
8.9
22.3
8.5
2.9
1.2
9.7
Table 2: Success Rate single vs, multi turn evaluation on InterCode-SQL (refer AA.3) Query
difficulty is adopted from Spider [51].
Best metrics are in bold.
InterCode-Bash
Single Turn
Try Again (n 10)
Model / File System
3
All
text-davinci-003
10.0
32.1
28.8
33.3
24.6
30.0
52.8
32,2
44.4
38.7
gpt-3.5-turbo
30.0
39.6
33.3
37.0
34.5
45.0
49.1
45.0
48.1
46.5
gpt-4
25.0
37.7
36.7
40.7
34.0
41.7
47.2
51.7
59.2
48.5
text-bison-001
15.0
22.6
11.7
22.2
17.0
23.3
28.3
16.7
22.2
22.5
chat-bison-001
12.1
22.5
16.7
22.2
17.7
13.8
24.5
18.3
22.2
19.2
Vicuna-13B
10.0
24.5
18.3
7.4
16.0
15.0
35.8
25.0
22.2
24.5
StarChat-168
15.5
22.6
13.3
22.2
17.7
17.2
30.2
21.7
29.6
23.7
Table 3: Success Rate across file systems for single vs. multi-turn evaluation on InterCode- -oerCode-Bash (refer A..2.. To evaluate models' ability interact with different task settings, we evaluate disjoint sets of Bash instructions across four different file systems. Best metrics are in bold.
that performance across different levels of task difficulty (SQL) and different file systems (Bash) is superior in the interactive setting for all models, with notable multi-fold increase for GPT-4 (9.1% + 73.7%) on the InterCode-SQL. task.
Analysis of interactions. Manual inspection of trajectory logs indicates that models actively exercise later turns for discovering relevant context, correcting errors via execution feedback observations, and solving problems via iteratively constructing and editing actions affirmed by Figure 3 In addition, models also demonstrate level of planning and modular problem solving; for instructions with gold commands that chain multiple commands together (i.c. with I, >, or in bash) or consist of multiple sub-problems (i.e. subqueries in SQL), models will use observations from solving smaller sub-problems in earlier turns to compose the higher-order action. Trajectories that exhibit these phenomena are in $ BB.4
Failure cases. With that said, both Figure exhibits a plateauing in Success Rate and and Error %. This suggests that the amount of context and feedback builds up. models less capable of discerning relevant past history toward future actions. In late-turn scenarios, task episode trajectories often reveal repetition of carlier actions, a failure to effectively use recent observations towards deciding an appropriate next action, or an inability to recognize that current problem-sol of thought inconclusive futile. This is particularly evident for hard and extra level InterCode- SQL task instructions that require context spanning across several tables and actions that incorporate multiple clauses. We note that even when the full schema of all tables and their descriptions are offered in addition to the original instructions, models still benefit greatly from using interaction to experiment with different JOIN and filtering operators across multiple turns, as demonstrated |B.2 A larger context window size, retrieval of useful memory, and more adaptive reasoning paradigms are just handful of potential solutions to overcoming such challenges.
chain
5.2 Prompting strategy comparison
Initiating language agents with prompting strategies that encourage different forms reasoning toward problem-solving improves performance on the interactive coding task to varying degrees. Table presents side-by-side comparisons of the success rate, number of turns, and error rate per strategy. Compared to Try Again, which lacks specific guidance leveraging multiple turns, more
7

---

chat-bison-001
45
70
qpt-3.3
60
starchat
Success Rate
chatistson-011
gpt-3.5
50
text-davinck003
Success
40
vicuna
30
text
30
Vicuna
20
20
10
Number of Interaction lurns
Number of Interaction Turns
(a) Success rate vs. turns for InterCode-Rash
(h) Success rate turns for InterCode-SQI.
Figure 3: Growth in Success Rate with increase in number of interaction turns across models configured with Try Again prompting strategy for InterCode-Bash and SQL tasks.
Try Again (n 10) Turns
ReAct (n = 10)
Plan & Solve
Error
Turns
Error
SR
Turns
Error
SQL
47.3
7.25
46.4
58.7
5,30
6.94
49.1
4.29
16.2
Bash
46.5
6.15
24.9
20.5
4.40
20.4
28.0
6.65
53.3
Table 4: Comparison of different prompting strategies across the entire InterCode-SQL and InterCode- Bash datasets using 3pt-3. 5-tubb as the base model. Turns refers to the average number of turns taken for single task episode. For Try Again and ReAct, the max number of turns 10. The highest Success Rate, fewest Turns, and lowest Error % are highlighted per dataset since they reflect more accuracy and efficient task solving. Best metrics are in bold.
explicit reasoning frameworks such as ReAct and Plan Solve policies generally achieve higher success rates LSQL:: 47.3% 58.7%) with fewer turns and higher rate of admissible commands.
Different tasks present difffrent learning challenges. An important skill to solving the InterCode- SQL task is the ability to discover context and construct actions conditionally based on information revealed in prior observations. Given that InterCode-SQQ task instructions are phrased most com- monly as questions, adapting to the task setting and new information discovered along the way puts more emphasis on error correction and context discovery. On the other hand, the more declarative and multi-step step nature of the InterCode- -ash task instructions is more aptly solved by planning and modular task completion. These distinctions manifest in the Plan & Solve strategy's 's performance gap between the InterCode-SQI. and InterCode-Bast tasks; while Plan & Solve encourages model to decompose problems into more manageable steps, the strategy is less favorable towards adjusting on the fly in response to execution feedback. Example trajectories supporting these claims are in $ B.4
More adaptive reasoning is favorable. Compared to "imperative" reasoning paradigms such Plan & Solve which prescribe a relatively rigid procedure, more flexible frameworks like ReAct, which do enforce any particular logical formula or roadmap, are more conducive eliciting broader set of reasoning capabilities. However, while ReAct's 's performance is generally superior to Plan & Solve, tasks solved by both strategies with g 3. 5-turbo make up 57% (407/ /708) and 27.6% (21/76) of the union of all successfully solved InterCode-SQL and InterCode- Bode-Bash tasks respectively. This discrepancy highlights trade-off between the guidance and structural constraints that are inherent to prompting strategies; schemes that draw out specific reasoning patterns often overlook other equally useful capabilities. InterCode's interactive coding task can serve as a strong litmus test toward more adaptable, variegated model reasoning.
5.3 New tasks & datasets opportunities
InterCode's formulation, modular design, flexible task construction, and use virtual containers enable task designers to manifest complex, code-driven tasks, where completion is much more

---

Task
Bash
Challange Descriptiore should utiise the add2-alpine image. we know the with ting named
offect Sectort
down-at-the--ottor bxt. The sleuthkit tools can be used for help.
uniia are byte sectore
Flug: pfcoCTP ( wtwseenne___eeeeeeeeee _tt.iiccstsssseeeieeeeeee r5565e7b)
o0s: fets iessessese sccesscece soccesoos orteary Table (41)
Agent
032:330 0005002045 0006252140 c00025099 Liist (B092)
/bin/bash "gunzip 0ds2-alpine, flag.e.e.a-------ee
Agent
2015 -r soci-alptne, flag. grop dout-at- cete--botttnnn.
dda2 aaaaaeliia.aaa...... flag-ing: ing: D0S/MBR beot sector; partition
Bash reot/dowr-at whw.swtttth---tttte...
I0n6x33, active, start-Ois (2,), end-CHs 81,4),
200096 sectors
Agent
Agent
icat no 2048 ddsz-alpine. flag. 18251
apt-get install steuthkit
Bash
Bash
Reeeing parkage Viste.. dependenty tree... littadii -onttttttt..-.ttttitiii
sidfe--))) ungraded, festatted, not uggraded.
Agent
subeit PicsCTF(foranstee návica, (S5c5a7b)
Agent
Bash
dds2-alpine. Plaiinn..aag..ee.
flag correct
Figure 4: GPT-4's interaction trajectory for a binary exploitation CTF task. This requires proficiency in Bash and Python, among additional knowledge and reasoning. Orange text and arrows highlight the feedback that the model attends to in generating the next action. In last step, agent submits flag.
attainable through interaction. We draw inspiration from Capture the Flag (CTF) [14], a competitive cybersecurity game that requires expertise in coding. cryptography (i.c. binary exploitation, forensics), reverse engineering, and recognizing security vulnerabilities to accomplish the primary objective of discovering encrypted "flags" concealed within code snippets or file systems. Compared to InterCode- Bash & -SQL, CTF is much more complicated, requiring an agent to exercise knowledge of multiple coding languages, modularize a higher-order objective into sub- aub-problems, construct multi-step plans towards solving each problem, and adjust strategy when a plan fails to yield any useful insights.
We establish InterCode
new dataset consisting of 100 CTF objectives from picoCTF [39] Following the interactive coding task formulation, each task instance in InterCode-CTF is given as a <instruction, assets, hidden flag> tuple. We first construct a Bourne Shell within an Ubuntu OS as the task environment. Here, InterCode's 's use of virtual containers is crucial, as necessary actions can be irreversibly damaging on real systems (i.c. rm -rf, sudo access). Per task instance, the associated assets (e.g., images, executables, code), necessary for task completion, are copied into the OS file system. Given this setting, a task worker must understand the given material and investigate the assets to develop potential solutions. Executing successful approach must be done across multiple steps with various conditionals, where the execution feedback of prior step could have significant effect on the next step. Figure 4 spotlights the diverse skills needed for CTE.
6 Discussion
Conclusion. We have developed InterCode, a novel lightweight framework that facilitates interaction between Language Models and the underlying environment, enabling them to mimic the human approach to language-to-con to state-of-the-ar models using different prompting styles. It effectively leverages the capabilities of LMs to break down complex tasks and recover from errors within a secure and isolated environment. The ability seamlessly convert existing datasets into the interactive format using InterCodeEnv API, and furthermore, the Bash and SQL environments, empowers task designers to construct new tasks to unlock the plethora of challenges that await in the space of interactive coding.
generation. Our framework has shown promising results when applied
Limitations and future directions. We point out several current limitations InterCode. At this time, the number of InterCode based environments is limited to Bash, SQL, and Python action spaces and datasets; within the near future, we plan to expand the number of offerings to cover a wider set of programming languages and datasets that should further deliver on InterCode's 's purported promises of efficient and expressive task construction. Second, the CTF dataset is limited to just four task instances due our manual curation procedure. We hope to release more formal work soon that provides a more thorough analysis of the reasoning and collaboration challenges of the CTF task along with more extensive dataset for evaluation purposes.

---

Acknowledgements
We thank Xiao Liu for the Vicuna/Alpaca APIs, Carlos Jimenez and Yuhan Liu for trying our code, and Princeton NLP Group for helpful discussion and feedback in general. We acknowledge support from the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
10

---

References
[1] M. Agarwal, T. Chakraborti, Q. Fu, D. Gros, X. V. Lin, J. Maene, K. Talamadupula, Z. Teng, and J. White. Neurips 2020 nlc2emd competition: Translating natural language to bash commands. In H. J. Escalante and K. Hofmann, editors, Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 302-324. PMLR, 06-12 Dec 2021. URL atttps:/pproceeding. mlr
hBtas//133/agaraal
html
[2] R. Agashe, S. Iyer, and L. Zettlemoyer. Juice: large scale distantly supervised dataset for open domain context- context-based code generation. ArXiv, abs/1910.02216, 2019.
[3] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern and et al. Palm technical report, 2023.
[4] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.
[5] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[6] R. Bunel, M. Hausknecht, J. Devlin, R. Singh, and P. Kohli. Leveraging grammar and reinforce- ment learning for neural program synthesis, 2018.
[7] A. Chen, J. Scheurer, T. Korbak, J. A. Campos, J. S. Chan, S. R. Bowman, K. Cho, and E. Perez. Improving code generation by training with natural language feedback, 2023.
[8] B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen. Codet: Code generation with generated tests. arXiv preprint ar Xiv:22200 10397, 2022.
[9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. Oliveira Pinto, Kaplan, Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Miskkin, B. Chan, Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Vors W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021.
[10] X. Chen, C. Liu, and D. X. Song. Exccution-guided neural program synthesis. International Conference on Learning Representations, 2018
[11] X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint ar arx1v:2304.00122 2023.
[12] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. Gonzalez, I. Stoica, and E. P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL htt.tst/ttsmssseeee ottp:loggggoooooeeee wttpbl/gg//2300333333333
1000-vicunaa vicuna/
[13] C. B. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers, 2020.
[14] C. Cowan, Arnold, S. Beattie, C. Wright, and J. Viega. Defcon capture the flag: defending vul- nerable code from intense attack. In Proceedings DARPA Information Survivability Conference and Exposition, volume 1, pages 120-129 vol.1, 2003. doi: 10.1 10./DDSSSEXXX.000 194878.
[15] L. Dong and M. Lapata. Language to logical form with neural attention, 2016.
[16] K. Ellis, M. Nye, Y. Pu, F. Sosa. J. Tenenbaum, and A. Solar-Lezama. Write, execute, assess: Program synthesis with a repl, 2019.
[17] Z. Feng, D. Guo, D. Tang, Duan, X. Feng, M. Gong, I. Shou, B. Qin, T. Liu, D. Jiang, and
M. Zhou. Codebert: A pre-trained model for programming and natural languages, 2020.
11

---

[18] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe. C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and Leahy. The pile: An 800gb dataset diverse text for language modeling, 2020.
[19] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
[20] J. Huang, C. Wang, J. Zhang, C. Yan, H. Cui, J. P. Inala, C. Clement, and N. Duan. Execution- based evaluation for data science code generation models. In Proceedings of the Fourth (Language Advances), pages 28-36, Abu Dhabi, United Arab Emirates (Hybrid), Dec. 2022. Association for Computational Linguistics. URL wttpe///acaaaaahooooooo
Workshop on Data Science with
org/ wwwpg//222.cco dash-1.5
[21] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search, 2020.
[22] S. K. Lahiri, A. Naik, G. Sakkas, P. Choudhury, C. von Veh, M. Musuvathi, J. P. Inala, C. Wang, and J. Gao. Interactive code generation via test-driven user-intent formalization, 2022.
[23] Y. Lai, C. Li, Y. Wang. T. Zhang, R. Zhong, L. Zettlemoyer. S. W. tau Yih, D. Fried, S. Wang. and T. Yu. Ds-1000: natural and reliable benchmark for data science code generation. ArXiv, abs/2211 1. 15011 2022.
[24] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328, 2022.
[25] C.-H. Lee, O. Polozov, and M. Richardson. KaggleDBQA: Realistic evaluation of text-to-SQL parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference (Volume 1: Long Papers). pages 2261-2273. Online, Aug. 2021. Association for Computational Linguistics. doi: 1111665//11/021.aclll acl-long.111 176
Natural Language Processing
176.
URL https: BttL:tttt////aaatthoolle
org/2021
[26] R. L.
L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, 'T Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, and et al. Starcoder: may the source be with you!, 2023.
[27] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, Gimeno, A. D. Lago. T. Hubert, P. Choy, C. de Masson d' Autume. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals. Competition-level code generation with AlphaCode. Science, URL htttttt////do...... org/ 111.//00121/////eieccceeee abq1158
dec 2022. doi: 10. 10.1122/science..bg) 158.
[28) C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL h org hrg/w04-1013
[29] X. V. Lin, C. Wang, L. Zettlemoyer, and M. D. Ernst. NL2Bash: A corpus and semantic parser for natural language interface the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https: //aclanthologe org/ 149-
[30] S. Lu, D. Guo, S. Ren, J. Huang. A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang. G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664. 2021.
[31] D. Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2, 2014.
12

---

[32] A. Ni, S. Iyer, D. Radev, V. Stoyanov, W. tau Yih, S. I. Wang, and X. V. Lin. Lever: Learning to verify language-to-code generation with execution, 2023.
[33] E. Nijkamp. B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. ICI.R, 2023.
[34] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.
[35] R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, I., Decker, V. Thost, L.. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021.
[36] F. Shi, D. Fried, M. Ghazvininejad, I., Zettlemoyer, and S. I. Wang. Natural language code translation with execution, 2022.
[37] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection, 2023.
[38] K. Tusar.
sq 3qlitemmssql, 2018.
URL https:://gtchub.com/teeeett
sqlite3-to-myse
[39] C. M. University. PIcoCTF, 2013. URL httppss:s//picoott org//
[40] L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023.
[41] X. Wang, Y. Wang, Y. Wan, F. Mi, Y. Li, P. Zhou, J. Liu, H. Wu, X. Jiang, and Q. Liu. Compilable neural code generation with compiler feedback, 2022.
[42] X. Wang, H. Peng, R. Jabbarvand, and H. Ji. Leti: Learning to generate from textual interactions, 2023.
[43] Y. Wang, W. Wang, S. Joty, and S. C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.
[44] Z. Wang, G. Zhang, K. Yang, N. Shi, W. Zhou, S. Hao, G. Xiong, Y. Li, M. Y. Sim, X. Chen, Q. Zhu, Z. Yang, A. Nik, Q. Liu, C. Lin, S. Wang, R. Liu, W. Chen, K. Xu, D. Liu, Y. Guo, and Fu. Interactive natural language processing, 2023.
[45] Z. Wang, S. Zhou, D. Fried, and G. Neubig. Execution-based evaluation for open-domain code generation, 2023.
[46] S. Yao, R. Rao, M. Hausknecht, and K. Narasimhan. Keep calm and explore: Language models for action generation in text-based games. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[47] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
[48] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models, 2023.
[49] P. Yin and G. Neubig. Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4553-4559, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/19-11 1447. URL https://aclanthologyy org/ PP9 1447
[50] P. Yin, W.D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation inter- active data science notebooks, 2022.
13

---

[51] T. Yu, R. Zhang, K. Yang. M. Yasunaga, D. Wang, Z. J. Ma, Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. Spider: A large-scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-SQI. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/11DD18-44222 URL httpe://aclantholl.gs otg/
[52] T. Yu, R. Zhang, H. Y. Er, S. Li, E. Xue. B. Pang. X. V. Lin, Y.C. Tan, T. Shi, Z. Li, Y. Jiang, M. Yasunaga, S. Shim, T. Chen, A. Fabbri, Z. Li. L. Chen, Y. Zhang, S. Dixit, V. Zhang, C. Xiong, R. Socher, W. Lasecki, and D. Radev. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases, 2019.
[53] T. Yu, R. Zhang, M. Yasunaga, Y. C. Tan, X. V. Lin, S. Li, H. Er, Li, B. Pang, T. Chen, E. Ji, S. Dixit, D. Proctor, S. Shim, J. Kraft, V. Zhang, C. Xiong. R. Socher, and D. Radev. SParC: Cross-domain semantic parsing in context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4511-4523, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1//199 1443. URL https 7 //aclanthology org. P19- 1443
[54] I.. Zeng. S. H. K. Parthasarathi, and D. Hakkani-Tur, N-best hypotheses reranking for text-to-sql systems, 2022.
[55] K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin. Self-edit: Fault-aware code editor for code generation, 2023.
[56] S. Zhang, Z. Chen, Shen, M. Ding, J. B. Tenenbaum, and C. Gan. Planning with large language models for code generation, 2023.
[57] T. Zhang, T. Yu, T. B. Hashimoto, M. Lewis, W. tau Yih, D. Fried, and S. I. Wang. Coder reviewer reranking for code generation, 2022.
[58] V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017.
[59] S. Zhou, U. Alon, S. Agarwal, and G. Neubig. Codebertscore: Evaluating code generation with pretrained models of code, 2023.
14

---

Appendix
In this appendix, provide additional details about the implementation and usage of the InterCode framework and the InterCodeEnv interface. We also provide visualizations and analyses of addi- tional experiments to demonstrate InterCode's utility and garner further insight into the extent of current models' performance on the interactive coding task. The full template for each prompting strategy is also included. Finally, we also discuss of the impacts, risks, and limitations of our hww.attterroode--bbebbaaaaeaeeee benchmark.................b..m.b github. io/ The code work. The webpage for InterCode wttpp/pppieeetooooo//ooiiiieieea https//////ttoooooo//////tttotet for InterCode lnttps://ge gighub. the InterCode webpage.
the link also included on
A Environment Details
A.1 InterCode Interface
The InterCode interface inherits OpenAl gym (5] environment API definition. Specifically, InterCodeEnv is written as an abstract class that primarily handles the main execution logic for processing code interactions, addition logging, data management, and sand-boxed execution, along with both environment-level and task-level customization.
InterCodeEnv exposes the following API. Creating an interactive coding environment requires defining subclass of InterCodeEnv. The methods denoted with an asterisk can be overridden for the purposes of customization.
init (self, data_path: str, image image_name: str, **kwargs)
Validates that the dataset specified by data_path is formatted correctly and can be used in interactive setting.
Uses the Docker image specified by image_nann name to create and connect with a Docker
container instance the image.
Initializes Logging Handler
Keyword arguments:
verbose (bool): If true, logging is enabled and environment interactions are shown to standard output
traj_dix (str)) If valid path is provided, task episode summaries are saved the given directory (generated by save_ (veeettraeetor))
preprocess (callable): If provided, this function is run before every task episode, It is way to provide task instance-spent specific customization of the execution environment.
reset (self, index: int None) -> Tuple [str, Dict]
Retrieves task record from data loader
Calls reset_containes
Reset task level logger, instance variables
step (self, action:
str) -> Tuple[str, int, bool, Dict]
Log (action, observation)
Invoke exec_action on action argument
If action=submit, invoke reward, save_trajectory
save ((ajecttory(se1f)
Saves task metadata, (action, obs.) sequence, and reward info to json in traj_dir
close (self)
Safely exit or stop any resources (i.e. docker container) used by the environment
execute_actions action
action:
str)
Defines how the action is executed within the context of the docker container. Requires impl. because the Dockerfile definition, particularly its entrypoint, affects how an action would be invoked within the container.
15

---

Interactive Loop
Intercode Env.
data_path
1. Provide docker image dataset
_datt_ img)
image_nane
env = BashEnv(...)
reset() 6 called by
Initialize new task episode
for of episodes: env.reset()
reset_container ()
policy ... done false
get_reward()
save (ove_trajector()
execute action() e
while done: policy(obs) + act env.step(act) + obs, done
3. Interact with env. until task done
called by
step(action)
close()
env.close()
4. Close, exit safely
Figure 5: Visualization demonstrating the intended invocations and usage of the InterCodeEnv inter- face, along with how the functions requiring implementation (get_reward() (), execute soccuut_action (), reset_container () are called by the methods of the main interactive loop.
Default impl. passes the action string directly into self container exec (action) call, which invokes the action the environment and returns execution output. A timeout is imposed execution duration.
(et__reaarddddee (self) Tuple [float, Dict]
Handles reward calculation of actions with respect to the gold command(s) for a task episode. Requires impl. because the concept and scoring for task completion varies across datasets and environments.
* reset_ cepeet_cottaieeeeee (self)
Handles resetting of execution container (i.e. resetting file system original state). Requires impl. because the approach to restoring setting to its initial state varies.
Figure conveys how each of these methods invoked and how they related to another. In summary, the technicalities for setting up an interactive coding task for specific system with one or more programming languages the action space involve:
Defining a Dockerfile
Providing dataset with the query and gold fields
(Optional) Defining a reward (get_reward) function to define task completion. (Optional) Creating an InterCodeEnv subelass that overrides the execute peccute_accioo action and get net_reward methods
A.2 Bash Environment
Environment Definition. The Dockerfile defining the Bash-based environment founded on the LTS version of the Ubuntu operating system. Several Linux dependencies that can potentially be used by an agent to address instructions in the InterCode- Basd Dataset are then installed via the Advanced Package Tool (apt) interface. Next, shell script is invoked within the Dockerfile fil to initialize one of the three file systems displayed in Figure The shell script consists of simple sequence mkdir, touch, and echo commands to deterministically create and populate the content of multiple files and folders. Finally, git is configured for the purposes determining file diffs per task episode (git status -s) and resetting environment to its original state (git reset -hard; git clean -fd;) before the beginning of a new task episode. The original code for the Dockerf ile along with the file system creation scripts be found on the project GitHub repository.
Dataset Details. The log-frequency distribution of the top-50 utilities is displayed in Figure 7
The NL2Bash [29] dataset is made available for use under the GPLv3 License
To assess the
16

---

generalizability of our approach, we designed three distinct file systems to accommodate the bash commands we collected. A key consideration during the construction of these file systems was to ensure that a significant portion of the executed commands would not result in operations that yield no changes. This deliberate design choice aimed to provide a more comprehensive assessment of our approach's is adaptability and effectiveness across various scenarios and command executions. The file systems encompass a wide range of file types, including text files (.Xx)), program files (.c, jjvva, .yy), compressed files (.zz,, shell scripts (.sh), PHP scripts (php)), JSON files (json), documents (.doc), spreadsheets (.esv), webpages (.html,, database schemas (.sql), hidden files, and files with special characters in their names, convoluted folder hierarchies. Their directory structures are illustrated in Figure 6 For simplicity, we consider the top-level folder created within the root directory (testbed, system, workspace) the root of each file system. This root folder contains files and sub-folders that necessitate access and manipulation, while changes are monitored throughout the entire container to accurately evaluate the models' actions. Notably, intentionally designed file system 1 to be more intricate and encompass relatively challenging hash tasks compared to the other two file systems. Thereby, the models' performance is relatively lower for file system 1.
Reward Function. Evaluation of an agent's 's trajectory across single task episode towards carrying out the given instruction is determined by modifications to the file system and the latest execution output. The instructions found in the InterCode- Bash dataset fall under one of two buckets: either 1. Requests information about the file system that can be answered via execution output generated from a correct sequence of Bash actions (i.e. "How many files...,, ", "What is the size of...". "Where is the .ung image stored?") or 2. Requests change to the location, configuration, or content of file or folder (i.e. "Move the dir1 folder from...", "Set the permissions to...", " "Append a line to....). Any relevant correct changes are therefore captured by considering both execution output and file system modifications during evaluation.
We define A and G as the outputs of the agent and gold commands respectively, where Aout and Gout refer to the execution output, and Afa and Gys refer to a list of entries reflecting file system modifications, where each entry is [file path, modification type [added, changed, deleted]]. . We then formally define the reward function as follows:
R = 0.34 * similarity Gout)
+0.33 (1 erf([A,, U G - AAs (Gsss))++++
(1)
+0.33 is_correct(A,,nGr
Asa
Where similarity refers to lexical similarity, which is determined by the cosine similarity score between TF-IDF vectors (calculated with TfidfVectoris from teikit-learry of the two execution outputs. The second component of the reward function reflects the number of file system modifications that were either not completed or not necessary; the error associated with the total number of misses is constrained to the range [o, using the Gauss error function (erf), where O corresponds to no file system modification mistakes. The third component checks what proportion of paths altered by both agent and gold were modified correctly. The is_ is_correct function returns the number of file paths that were changed correctly, determined by checking whether the md5sum hashes of each file path are identical for agent and gold. If Ass n Gfs 0. this reward is automatically 1. The scalar weights for each component are arbitrarily assigned.
A max score of 1 is achieved only if the correct file paths are changed, the changes are correct, and the latest execution output matches the gold command output exactly. Figure I visualizes the reward function. While an exact match comparison would have been a simpler choice to satisfy the Success Rate metric put forth in the main paper, we design this reward function to Demonstrate that InterCode can support complex reward functions that account for multiple forms of execution output, and 2. Provide practitioners who use the InterCode- Bash environment with scalar reward that reflects how "similar" the given output is to the expected, rather than a flat 0/1 reward value that may over ver-penalize and discount the efforts of more capable reasoning abilities. These reasons also motivate the SQL-based environment's 's reward function, discussed in the following section.
17

---

/teatbed
system
workspace
FooBar. html
_DS_Stor
.hidden. txt
Hello.java
MANIFEST
archive.tar tare gz
Hello1.java
a. out
dir1
JawClass.java
folder1
hiideen.. txt
dir1
a.out
a. txt
AnotherHello.
bata.csv
all.txt
info.php
doc1.doc
file.c
subdir1
doc2.doc
file. txt
jsonfile1.json
keep.tx: txt
hello.c
Pythonscript4.pp py
log1.1og
hello.tx
shellscript1.st sh
sh
long.txt
subsubdir1
old2. txt
new1.sh
pythonscripti py
recent txt
readonly. txt
shellscript4.sh sh
script1.sh
script1.sh
textfile4.txx
text2.txt
sum.c
subdir2
text3.
terminate. txt dir2
textfilei.ttxt
text4.txt
dir2
folder2
csvfile1.cav
shellscript2.eh sh
special
foo.txt
subdirl
text3. txt
hello.tz:
javafile1.java textfile2.txt
special_text1.. txt
mysql
special_text2. Deocial_txxt2.ccm txt
I sql1.sq!
subdir2
text1.
files. txt
pythonscript2.pp py shellscript5. sh subsubdir1
folder2.tar tar. gz
new.sh
folder3
old2. txt
backup_dbs
recent1. txt
textfile5.c txt
backup
results. txt
dir3
8ql1.sql
test
subdir1
text1_dbg. txt
2dir
pythonscript3.pp py
special
3dir
subsubdir1
text4.tx:
dirl
FooBar
temp
(c) File System
file. txt
empty. txt
file. txt
temp1
shellscript3, sh
temp
textfile3.txx
temp_1 1
tmp
text1. txt
tmp. txt
html1 html
subdir2
temp
csvfilel.cm
keep. txt
textfile6. txt
text1.txx txt
hello.c
text3 txt
hello.ph,
(b) File System
recent. txt
textfile7.tae txt
(a) File System
Figure 6: File System structures designed for InterCode-Bash.
18

---

cat
cut
aed
medium
echo
grop
head
awk
md5oum
446
usig
tail
hard
174
166
tree
248
extra
which
dirnamo
easy
df
dit
Figure 8: Distribution of gold command difficult for InterCode-SQL task data adapted from the Spider SQL dataset.
Figure 7: Top 30 most frequently occurring bash utilities of the 66 in InterCode-Ban Bash with their frequencies in log scale.
A.3 SQL Environment
The Dockerfile defining the SQL-based environment inherits from the MySQL image and adds a sqq file setup script to the /docker-eneeeeee teeexer-entryooitttinee irrpoint-iiiidb..... d directory within the Docker image; this is special directory made for container initialization. On container start-up, the added .sql file, which creates and populates databases with tables and tables with records, is automatically invoked. Since the InterCode-SQ!! dataset does not feature any queries that involve modifying the database in any manner (i.e. no INSERT, UPDATE, or DELETE commands), there no reset mechanism written into the Dockerfill definition that is invoked before each task episode; with that said, adding reset script or version control to the Dockerfill file is simple.
Environment Definition.
InterCode-SQL. Dataset. InterCode-SOL is adopted from the development of the Spider dataset [51] Spider 1.0 is a large-scale cross-domain dataset on generating SQL queries from natural language questions whose development contains 1034 pairs of <instruction, gold> task instances spanning 20 databases. The distribution of queries according to their hardness criterion is shown Figure As discussed in Section 3.3 filtering criterion narrows down the Spider dataset's information to only the necessary components. We do not add anything to the Spider dataset that was originally available. The Spider 1.0 dataset available for use under the CC BY-SA 4.0 license
MySQL Databases. We first resolve data types for primary. foreign key pairs across the provided table schemas in Spider for conflicting instances and generate the corresponding SQLite databases. Next, to align with our Docker-supportee
environment, we convert the SQLite databases to MySQL format using sqli selisqemyyql (38). Python library, and then generate unified MySQL dump having schemas for all the tables. To handle case-sensitive table name discrepancies between the queries and the underlying schema in the original Spider dataset, we activate the lower_case case_ table haable_naeen names setting in our evaluation environment. Additionally, for proper access controls, we create a test user and grant them all privileges for all the tables.
case_table
Reward Function. The completion evaluation mechanism compares the output of the gold SQL command with the latest execution output (i.c. latest observation) from the agent's interaction trajectory. The execution output of all gold SQL queries a list records. Each record a tuple of more values that may be different types. For any single execution output, the order of types for every record is identical. Given the agent command(s)) latest execution output and the gold command's execution output G, we formulate the reward function follows:
AnG
R ((endlltae(An((nG))) 1)/2 AUG
(2)
19

---

SQL Env
Agent Command
Question
SHOW TABLES DESC has_pet, SELECT StuID from has_pet 4. DESC
Find the first and age of students who dog but do not have cat as pet.
Student (additional cmds...)
Agent
SHOM TABLES
Answer
SQL Env [('has_pet'su ,), ('pets',) ,), ('Student' ,)]
Latest Std. Out: [RRss,, 29), (Ross, 29),
(Chandler, 28), (Rachel, 27), (Nonica, 25)]
Agent
Calculate Intersection Over Union
- 3/7
DESC has_pet
Gold Std. Output: [(Ross, 29), (Ross, 29), (Joey, 27), (Rachel, 27), (Phoebe, 26)]
SQL Env [('StuID', 'int', 'YES', 'HUL', None...]
SQL Env
Agent
SELECT fnane, age FROM student WHERE
Gold Command
SELECT TL. fname, Tg.age FROM Student JOIN Has_ AS T2 ON
SQL [( [('Tracy', 19)]
Agent
Evaluation of the results of agent interactions with the SQL Environment against the gold command associated with the task. A simple Intersection over Union formula that ac- counts for duplicates is used to quantify answer correctness. Task completion a reward I.
Figure 10:
subnit
Example of interactions between an agent and the InterCode SQL Environment
Figure 9:
We employ Intersection over Union (loU), more formally the Jaccard Index, to quantify the correctness of the latest execution output generated by the agent against the gold output. the latest execution output of the SQL query not in the form of list of records (i.e. a string error message), the reward is 0 by default. Among the items that lie in the intersection and gold execution outputs, we also apply penalty if the records achieving the correct order of fields in record is of non-trivias trivial importance to addressing many SQL queries correctly, we do not do any re-ordering or pre-processing of the list of records. Therefore, a record formatted as ("Ross", 29) is not awarded any credit against gold output that includes (29, "Ross"). To quantify how sorted the agent output is relative to the gold output, we lean on Kendall's v and adjust the output range to (0, 1). The IoU score is then directly scaled by this coefficient.
the agent in the incorrect order. Since
All all, only a correctly ordered list with the exact set of records found in the gold output would receive max function for an example set of outputs. Note that in the main paper, the Success Rate metric is used; the scalar 3/7 output shown in the figure is treated completed via the 0/1 Success Rate metric. As mentioned in the discussion of the Bash reward function, this reward function also aims be richer and fairer continuous evaluation metric of model's 's reasoning abilities compared to a binary 0/1 task completion score.
of which corresponds task completion. Figure TO visualizes the reward
0 when quantifying whether the task was
B
Experiment Details
B.1 Model Details
We do not perform any model training for configuring the methods running the experiments discussed this project. Our evaluations use inference call requests OpenAI, PaLM, and Hugging- Face API endpoints to run the baseline models on the InterCode tasks. For OpenAl models, we set temperature to top_p to 1, max_ tokens to 512, and n (number of completions) to 1. For PaLM models, we set temperature to 0, top_p to 1, and candidate _ccuttttcctttto (number of completions) to 1. For open source models, we set max
max_new, ____new____ tokens (maximum number of tokens generate) to 100 and temperature 0.01. Due constraints in the context window size, we limit the length of cach
20

---

observation to maximum of 1000 tokens across all inference calls. The code for configuring API calls can be found in the linked repository.
B.2 Additional Experiments & Analysis
SQL schema ablation. To confirm that the benefits of interaction exceed simple disparity in information between the Single Turn and Try Again settings, we add the full SQL database schema, providing holistic details of tables necessary to the given instruction, to the Question message of both prompts, then re-run the comparison for several. Table 5 indicates that while Single Turn performance improves drastically, non-trivial difference in favor of Try Again remains. Manual inspection of task episode trajectories shows that selective and fine- ine-grained context discovery (i.e. inspecting specific table records and file content that affect query construction) is still critical to solving tasks efficiently.
InterCode-SQL.
-Schema
Single Turn
Try Again (max 10 turn)
Model / Hardness
Easy
Med
Hard
Extra
All
Easy
Med
Hard
Extra
All
gpt-3.5-turbo
90.7
70.2
59.2
37.3
67.9
92.7
74.9
67.2
43.4
72.8
text-bison-001
89.5
68.2
44.2
19.3
61.4
90.7
70.4
50.0
21.1
63.9
chat-bison-001 79.0
52.0
32.1
15.1
49.2
82.2
56.0
42.5
24.1
54.9
Table 5: Success Rate across difficulty for single vs. multi-turn evaluation on the InterCode-SQL dataset, with the database schema relevant to each task episode's 's instruction, also provided in the Question message of the prompting strategy. Best metries are bold.
Trends admissible actions. Table shows that for the SQL task, models generate admissible actions with increasingly higher rates early on; in initial turns, models will tend to hallucinate query with fabricated table and column names at a high frequency. The drop error rate between the first and second turns can largely be attributed to the model's is decision to begin exploring context; 60.3% of second turn actions contain either the SHOW TABLES or DESC keywords. Prompting strategies (i.c. ReAct, Plan & Solve), explicit phrasing that encourages exploration, and demonstrations diminish a model's default tendency to hallucinate a query in the first turn. This trend is not found in Bash. This can likely be attributed to the nature of the instructions; unlike the SQL instructions which simply pose question and do not have any explicit references to SQL commands or clauses, Bash instructions will typically include keywords that correspond directly to useful Linux commands or give insight into the file system's 's internal structure. These signals reduce the need for context discovery. Therefore, successful task completion in Bash tends to lean towards 1) Figuring out which flags, options, and arguments to configure a command with and 2) How to string together commands or pass outputs from one command to the next correctly.
For both Bash and SQL, in later turns, the rate of admissible actions does not improve consistently. The actions in these later turns usually attempts to answer the original instruction. At these stages, a model will tend to make small, cursory adjustments to the prior action based on execution feedback, often resulting in both repetition of the same types of mistakes and hallucinations that introduce new issues. In these moments, compared to such minor perturbations, alternative reasoning capabilities such context discovery and modularized problem solving are often more efficient ways to get the relevant insights needed to better decide how to fix the prior turns' issues. As corroborated by Figure 3 models struggle to take advantage of additional context in longer task episodes or horizons. Making the most of multiple queries is an open challenge with exciting implications for solving more difficult coding tasks.
Turn
2
3
4
6
8
10
SQL
90.2
46.4
34.4
39.7
31.1
42,9
51.5
47.4
48.4
46.6
Bash
23.1
28.6
34.7
37.5
37.6
42.9
39.3
37.1
33.7
38.2
Table 6: Error % (Average ratio of non-admissible actions) per turn for the Try Again prompting scheme using a GPT 3.5 model on the Bash and SQL InterCode datasets.
21

---

Robustness results. We conducted an evaluation to assess the robustness of the reported accuracy metrics for the models. In order to maintain consistency in the evaluation, we focused on the performance across file systems 2, 3, and 4 (shown in 6). which were designed to have similar difficulty levels. File system I, intentionally made harder, was not included in this analysis. The standard errors for the Single Turn and Try Again modes are presented in Table 7 The Try Again mode leverages interaction to consistently outperform the Single Turn mode across all models.
Model
Single Turn
Try Again (n = 10)
text-davinci-003
31.40 + 1.35
43.13 5.98
gpt-3.5-turbo
36.63 + 1.83
47.40 1.23
gpt-4
38.37 1.20
52.70 3.50
text-bison-001
18.83 3.57
22.40 3.35
chat-bison-001
1.89
21.67 = 1.81
Vicuna-
16.73 5.00
27.67 = 4.15
StarChat- 16B
19.37 3.04
27.17
Table 7: (Robustness Results) Success Rate with standard errors for single vs. multi turn evaluation on InterCode-Bast (refer (A.)). Best metrics are in bold. Both modes display significant standard errors (as expected) but still Try Again outperforms Single Turn by huge margin.
B.3 Additional Prompting Strategy
To gauge the significance of designing prompting strategies that can successfully solve the interactive coding task, we attempt to devise a more performant approach by chaining together existing tech- niques, where each technique meant elicit different, relevant reasoning skill. To this end, we design a hybrid prompting strategy that combines Plan & Solve and Try Again, which we refer to as "Plan & Solve + Refine". This strategy is meant to complement a model's planning, modularized task completion, and context discovery abilities with error correction. Figure I1 visualizes this prompting strategy's 's workflow. The full prompting template included B.7
Plan & Solve + Refine
Initial Mag.
Agent
Agent Sectiono
[Ljei's underatiand the problem and devise
Faction
Up to (T
Up to (1)
plan. Then, let's carry out the plan..
Observation cotd. outpuut, sstepo
plan length
Observation <std. sutput, reward>
times
Question
What is the country code and first name...
Agent <plano
Terminate reward 1, else continue
Terminate reward: exceeds turns
Execute Msg.
Post Mag.
have finished executing the plan, but there
with..
Figure 11: Visualization of the Plan & Solve + Refine strategy. This prompting strategy is naive combination of Plan & Solve and Try Again; the model is first asked to formulate and exccute a plan. If it fails, then the model is asked to continue to work on generating the right solution until the reward is 1 or the number of allowed turns (n = 3) is exhausted.
We run small-scale experiments in the same fashion as discussed in Section 5.2 on the SQL task for 44 randomly selected task instances. These preliminary tests have an average Success Rate of 45.5, Number of Turns of 6.34 (across both planning and refinement stages), and an Error % of the original Plan & Solve performance presented in Section 5.2 this strategy does not yield a significant improvement. Upon inspection of the trajectories, it is evident that while models are capable of specific reasoning techniques, it not enough to elicit specific strategies in a pre-determined order. In addition, compared to human problem solvers, such reasoning frameworks do not intuitively allow for adjustment in strategy based on execution output. Therefore, mechanisms that allow model to freely self-determine what strategies at any turn in task episode are integral to improving Success Rate and non-trivial because simple adjustments in existing prompting strategies not enough to close this gap.
22.4. Compared
22

---

B.4 Task Trajectory Analyses
In this section, we discuss some of the behaviors and phenomena in different prompting schemes. We also include annotated trajectories that are representative of these observations.
Inst:
Count md5sum of all "eppy" files in /testbed folder with subfolders.
Inst: For each continent, list its id, name, and how many countries it has?
Gold:
find /testbed -type f -name "*.py " -exec md5sum O awk "print $13' sort I md5sum
Gold: SELECT F1.ContId , T1. Continent count(+) FROM CONTINENTS AS T1
JOIN COUNTRIES AS T2 ON T1.ContId
Out: 7F48339aabbe0ccbs33ccc4fffff
T2. Continent GROUP BY T1.ContII;
Act: find /testbed -name "*. "* -type -exec md5sum 0 + wc -1
Act: SELECT continent.ion continent na- me, COUNT (country. id) AS countries FROM continent LEFT JOIN country ON continent. country. wet.iyy.ootiiientteem GROUP BY continent. id, continent name Obs: Error: Table 'car 'car_I.comtinent" doesn't 't exist
Obs: 4
Act: find /testbed -name "t. -type -exec md5eum O I auk "END(print NR)
Obs: 4
Act: find /testbed -name "*, "*.py" type -exec md5aum 0 auk "ED(pprint)'
Act: SHOW TABLES
Obs: (fccarmaaeers'] .), (ecar_namess,,_ +.4 more...
cut -d' -f1
Act: DESC continents
Obs: 0dlcc200aoohhhs777752222222a
Obs: [('ContId', 'int', 'NO', 'PRI', None, "auto_ jitoo_nncremet"", ('Continent', 'text', 'YES'...
Act: find /testbed -name "* -type f -exec md5sum 0 I auk M[rrit $1)*
Act: SELECT ContId, Continent, (*UNT(+) AS NumCountries FROM countries GROUP Continent
sort | md5sum cut -d" -f1
Obs: 725ffffee95aab0000833666ef--f- (6 more turns)
...(5 more turns)
Table 8: Examples of Try Again interaction trajectories with GPT 3.5 based model from Bash (left) and SQL (right). Without any reasoning framework, models will tend to hallucinate an initial command, then make incremental adjustments (error correction) to the command based on the execution feedback. On occasion, a model will write commands to gather context about the task environment. For both trajectories, n= 10 turns was allotted and reward was 0 (max turns exhausted).
Default model behaviors. In both the Single Turn and Try Again setups, the templates describe the interactive coding task and do not provide the model with any reasoning framework or strategy, providing insight into a model's default approach when given the instruction and task environment. As evident in Table 8 across both the Bash and SQL tasks, model tends to directly generate an answer in the very first turn. For Try Again, the model will then perform error correction, then make the answer resolve the issue(s) reflected in prior execution output across the next several turns. With less frequency, for SQL, the model will also issue commands to get database and table descriptions. To the best of our knowledge, based on a close examination of the Try Again trajectories, current language models do not demonstrate any innate reasoning capacity for the interactive coding task beyond error correction and context discovery.
slight adjustments
Prompting strategy effects. In contrast with Try Again, the ReAct prompting strategy briefly introduces the interactive task environment and proposes reasoning framework for the model to abide by. Table
a side-by-side comparison of the Try Again and ReAct [48] strategies. The figure reflects the richer types of problem-solving that model exhibits when prompted to reason on its own thoughts about the execution output. This reflects that through better strategies, it may be possible make significant advancements in the interactive coding task with prompting strategies that attempt to elicit reasoning via an appropriate framework that also permits the model to be expressive and creative in devising its own solutions. This is particularly necessary for interactive code tasks, which pose multiple challenges that cannot be overcome by any isolated reasoning technique. As demonstrated in $ B.3] this direction is non-trivial, and InterCode is designed to facilitate the bench-marking of such approaches.
B.5
Capture the Flag Analysis
CTF challenges typically necessitate a trial-and-error methodology, where participants employ diverse techniques and exploit vectors to identify vulnerabilities to solve challenges. Processes such as exploring complex environments or executables, debugging, and dynamic exploitation, which
23

---

Inst: What is TV Channel TV series with Episode "A Love of a Lifetime"? List TV Channel's series name.
Gold: SELECT T series_naaaaee FROM TV_ IV_Channel AS T1 JOIN TV_series AS T2 ON T2. Channel T1.id WHERE T2. Episoie "A Love of Lifetime";
ReAct (r length = max turns 10)
Try Again (r= 0, length 10, max turms 10)
Thought: I should check what tables availahle in this MySQI. datahase that can help me answer this question
Act:
SELECT t1 channel, te.eeries, name
FROM table1 t1 JOIN table2 t2 ON
t1.series_id = t2. seriesies_ie WHERE
Act: SHOW TABLES
e2.episode_name 'A Love of Lifetime'
Obs: I( ('caarooon,,, J. ("tv_ [ww_channell,,, .), C"tv_ series' .]
...(1 turn for context discovery with DESC)
Obs: Error: Table 'tablel' doesn't exist
Thought: The tv_series table has a Channel column
Act: SHOW TABLES
that might be helpful. should check if there a
Obs: [('cartoon']] .) (tt_channel"") )) ("tv_series')) .)
row with the episode name "A Love of Lifetime". SELECT Channel FROM tv tv_series WHERE Episode 'A Love of Lifetime" Obs: [('700')]
Act: DESC tv_channel
Act:
Obs: [('id', 'varchar(255)', 'NO', 'PRI', None, "),
('series_name', "text,, 'YES', ". None...] + more...
Act: DESC tv_series
Thought: get a single value. 700, which is the TV Channel of the TV series with the episode "A Love of Lifetime". Check what series is this channel. Act: SELECT series_nane name FROM tv cv_chennel
Obs: I('id', "double". 'NO". 'PRI'. None, "), ('Episode', 'teet'. 'YES"." " ++7 more...
seriee,ssereee.... name FROM tv .channel WHERE id IN (SELECT Channel FROM tv tv_series WHERE Episode 'A Love of Lifetime')
Act: SELECT
WHERE channel_number
700 Unknown column
Obs: Error:
Thought: I should check what columns are avail- able in the tv clanneh table.
Obs: Error: Unknown column 'Channel' Act: SELECT channel __nmm,, serieeenaaaaaeee FROM tv_ tv_channel JOIN tv_ tv_series ON tv wy_channel.11 id tv_series. Channel WHERE Episode 'A Love of Lifetime' Obs: Eroo:: Unknown column 'channel_ne name' Act:
Act: DESC tv_ channel
Obs: [('dd', 'varchar(255)", 'NO', 'PRI' None, "),
('series_ name", 'text', 'YES', "... + more... Thought: The tv ">cchnnel table does not have a channel number column. Check if there is another way to get the series name for the TV Channel 700. Act: SELECT series_nan name FROM tv channel
SELECT Channel, series__errsssmme name FROM tv cv_chennel JOIN tv tv_series ON tv_ www.chhanee...m id tv_series, Channel WHERE Episode 'A Love of Lifetime' Continued hallucinations problem solving actions for last 4 turns...
WHERE id = '700'
Obs: [('Sky Radio' .)]
Thought: get a single value, Sky Radio, which the series name for the TV Channel 700.
Act: submit
Table 9: Comparison of Try Again & ReAct interaction trajectories with GPT 3.5 based model towards solving the same instruction. Without explicit strategy task demonstrations for Try Again, the model tends to hallucinate and doesn't 't build upon prior observations as well. With ReAct, there is greater evidence of context discovery, error correction, and problem solving.
involve sequential steps, require iterative interaction. Considering the inherently interactive nature of the task, it is crucial for an agent to employ an iterative approach and have access to an interactive platform to achieve success. In most instances, both humans and agents find it impracticable to solve a challenge in a single attempt.
Similar to the aforementioned exploitation task (Figure we now present a more intricate forensics task in Figure 12
is important to note that without the provided hint regarding the usefulness of the "sleuthkit" library, the agent fails to solve the task and engages in incorrect reasoning. However, upon receiving the prompt's 's hint, the agent adeptly utilizes this information to install the library and leverage its functionalities for its advantage. By analyzing given disk image file, the agent employs the "mmls" command to inspect the corresponding partition table. From the partition table, it deduces that a significant portion of the space remains unallocated, while a Linux partition initiates sector 2048. Subsequently, the agent attempts to access the contents of this sector using the "fls" command, searching for the hlow-at--he-bottom..n unable to locate the file, the agent speculates that recursive search might be necessary and adds the "-r" flag to command. Due to the immense output, it becomes arduous to track the file's 's location, prompting the agent to employ the "grep" command to search for the file within the output. By
file, which it anticipates will contain the flag. When
24

---

examining the grep output, the agent identifies the file's location (18291) and proceeds to inspect its contents. The flag, presented in a visual format, is accurately recognized and submitted by the agent.
human expert employs a very similar approach when provided with the hint. By furnishing an interactive framework, InterCode empowers agents emulate human-like behavior, enabling them to explore the environment, decompose tasks into subtasks, debug using traces and logs, and iteratively accumulate knowledge to successfully solve challenges.
Task
Challenge Description: You should utilize the add2--llpin..fig.iin image. know the file with the flag named The 'sluuthkit' tools can be used for help.
Flag: picocTT((eeransseettrr. F5565e7b)
Agent
Agent apt-get install sleuthkit
/bin/bash "gunzip dds2-alpine. flag.ing.-. ing.
Agent
Bash Env
>-
/bin/bash "file dds2-alpine. sappiinefllg.imee Flag.ing"
Reading package Hists....nluuiidiin dependency tree...
(oReading state insrmattoo...sstuuttii is already the newest version (4.28.2.ddaag -nn upgraded, installed,
Bash Env
upgraded.
dds2-alpine. flag.ingt DOS/NBR sectorj partition ID=0x83, active, start-cHS 32x),32,33), end-CHS 20,1), startsector 2048, 260096 sectors
Agent dds2-alpine. flag.ing
Bash Env
#3
Agent
>_
mats ads2-alpine.
Bash Env
#5
Agent
Partition
2048 -p dds2-alpine flag.
offset
grep coun-at-the-bottore
inits 512-byte
seez oseetsesss ceсо00e6ss c000000001 Primary Table (46)
Stot start End Description
Bash Env
ac6000en0s 05866422447 6490002648 unaltocated
r/r 182911
reat/dow-aat-the-bbtoo.
602: 000:000
0060092063 0886262142 6600260696
Linux
#6
Agent
#7
Agent subeit proecTF((eramssccterrvvves (
dds2-alpine. flag.ing
Bash Env
Bash Env
<ftag in visual form>
submitted flag
Figure 12: GPT-4's interaction trajectory for CTF forensics task. This requires proficiency in Bash and sleuthkit, among additional knowledge and reasoning. Highlighted in orange are intermediate logs/output observations that the agent intelligently captures and utilizes in the subsequent steps.
B.6 Human Performance Baseline
To explore the gap between human and agent performance on the interactive coding task, we the authors, all proficient in SQL, act as human task workers and perform the task on random sample of 15 InterCode-Se
task instances within the same task environment identical the agent's setting. max number of 10 turns is imposed, as was done with the Try Again prompting strategy. Similar to ReAct and Plan & Solve, the human task worker decides when to submit; in other words, the task does not terminate automatically when reward The trajectories for these 15 instances and the code for facilitating human interaction with the InterCode-SQI environment are available in the codebase.
The human task worker was able complete of 15 tasks (Success Rate = 0.87) with low Error %, most of the errors occurring not because of hallucinations of table columns and attributes, but rather because of SQL syntax errors that arose due to mistakes in relatively complex queries. What's noteworthy about the human task worker's 's trajectories is the presence of much more modularized mroblem-solvin that deviates heavily from an agent's s approach of generating query in a single go. Even with context discovery and error correction, an agent's action to produce answer for the
25

---

the other hand, human task worker will tend break up the query solution into multiple smaller sub-problems. This is particularly evident for instructions that must be answered with investigations across multiple tables with relations established by primary and foreign key columns. As an example, given an instruction "Find the average weight of the dog breed that is owned by the majority of pet owners", a human task worker might write commands that query the pet_ pet_owners table to determine field in the instruction will tend to be single, self-contained command that generates the answer in one go. On
what the most popular dog breed is, and then use the answer to this sub-problem WHERE clause of a second query that then determines the average weight using the pets table.
A more thorough and variegated study would be required to fully establish the performance gap between humans and agents. Nevertheless, from this small study, we are confident that humans generally exhibit more flexible and variegated reasoning capabilities compared to agents in the interactive coding task. Closing this gap is an exciting research direction, and beyond model-side improvements and scaling laws, incorporating human task reasoning and execution as guidance, feedback, or reward signals worthwhile consideration toward improving model performance.
B.7 Prompt Templates
As discussed in the paper, the main baseline evaluations for InterCode consist of presenting a language agent with an instruction and a prompting strategy that have been adapted for InterCode's 's interactive task setting. Each prompting strategy is defined as template with three components:
Initial Message: This is the first message presented to the agent. The initial message may describe the general task to accomplish, guidelines for interacting with the InterCode envi- ronment, the formats of the instruction and observation(s), and any additional information that pertains to the environment. In addition the environment and task specifications, the general prompting strategy and useful demonstrations may also be discussed. The initial message is presented once as the first message of task episode.
Instruction Message: This is the template for communicating the instructions that an agent is asked to solve for a particular task episode. The instruction message is presented once as the second message
task episode.
Observation Message: This template is for communicating the standard output and any additional information for a single interaction. This observation is what the agent will use generate the next action. The observation message may be presented multiple times depending on how many interactions the task episode lasts for.
Figures TO
present the corresponding prompt templates for the Try Again, ReAct, and Plan & Solve experiments, along with specific version for the toy Capture the Flag task.
11
12
26

---

Initial Message
## TASK DESCRIPTION
are (seelflanguaal code generator helping user answer a question using (self. language). The user will ask you a question, and your task is to interact system using {self language) commands to come up with the
with
answer.
## RULES
1. Do NOT ask queations
2. Your response should only be {self language} commands
## RESPONSE FORMAT
Your response should be a (self. Soelf.lange language) command. Format your (self langual))
command as follows:
">> (self. 1agguala Your (self. language) code here
Write (self language} commands to help you do things:
Learn more about sself.setting) you interacting with. example, you are interacting with a MySQL database, you can use the DESCRIBE command to learn more about the tables you have access to.
2. Execute {self language] commands based these tables generate the correct output.
DO NOT WRITE ANYTHING EXCEPT FOR CODE in your response.
## OUTPUT DESCRIPTION Given your (self language command input, the system will then give back output formatted as follows:
Output: <string> Reward: [o, 1]
The output the standard output from executing your (self .lagguage] command. The reward is decimal value between 0 and 1, which tells you how close your (self language) command is to the correct answer. The closer the reward is to 1, the closer your (self language) command is the correct answer
Instruction Message
Query:
"<Task Episode Instruction>"
Observation Message
Output: <Observation from Intercode Environment> Reward:
<Value [0, 1]>
Table 10: Initial, Query, and Retry Prompts for the Try Again Strategy Evaluation. {self language) is substituted with the programming language for interacting with the environment. {self. setting) substituted with a natural language description the environment
27

---

Initial Message
Interact with (self setting) system using (self language) queries question.
Use interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be tuo types: (1) execute, which executes (self. language) code the {self. setting) system (2) submit, which indicates that the previous observation is the Here are some examples.
Question: What are the names and grades for each high schooler? Thought 1: should write SQL command that selects the and grade fields from table about high achoolers.
Action execute [SELECT name, grade FROM scholeer
Observation 1: Error executing query: Table 'network_1 shghosco doesn't exist
Thought 2: The scholers table doesn't exist, need check what tables available to use.
Action 2: execute (SHOW TABLES]
Observation 2: [('friend', ,), ( ("highschooler'.) ,), (?likes' ,)]
Thought 3: There is a table called highschooler. I should check what information this table contains.
Action execute [DESC highschooler]
Observation 3: [('ID', 'int', 'NO', 'PRI', None, 'auto nuttoiincreent"/ ('name', 'text', 'YES', ", None, "), ('grade', 'int', 'YES', ", None, ")] Thought 4: The highschooler table contains the name and grade columns. I write SQL command that answers the question
Instruction Message
Query:
"<Task Episode Instruction>"
Observation Message
Output:
<Observation from Intercode Environment>
Table 11: Initial, Query, and Retry Prompts for ReAct Strategy Evaluation. Additional ReAct style demonstrations written for SQL and Bash can be found on the InterCode project GitHub.
Initial Message
You are an expert Capture the Flag (CTF) player.
Instruction Message
are in the '/ctf' folder of a standard Ubuntu machine. <Task Episode Instruction>. Your goal is to find this hidden flag by interacting with the bash terminal step-by-step. The flag is in the format 'picoCTF....
You can perform any
valid action in the terminal by starting your action with 'end <your_ cour__oomand>" For bash commands, prepend '/bin/bash 'python3 -c kyour_command>> the corresponding standard output as observation. You can also explore/inspect the file system or do anything that helps you find the flag. Once you feel you have found the flag, you choose to submit it by printing 'submit syour_flag>> Do NOT provide an explanation for your answer, only output the action you want.
for python commands, prepend and so on. For every action you take, you will receive
ppour_ccmmandi
Observation Message
Output:
<Observation from Intercode Environment>
Table 12: Initial, Query, and Retry Prompts for Capture the Flag Evaluation.
28

---

Plan Message
For the following question, let's first understand the problem and devise plan solve the problem. Then, let's carry out the plan solve the problem step by step.
Your plan should describe sequence of {self. language) queries you can write determine the
Here three examples of coming up with plan for
question.
Question:
What are the names and grades for each high schooler?
Plan:
Check tables are available for use.
2. Inspect each table to identify which has information about high schoolers.
3. Use the table to write query that selects the name and grade fields for each high schooler.
Execute Plan Message
You will execute your plan. Interact with (self setting) system using (self. language) queries answer question. turn, you will given the following information:
Observation: Standard output from executing previous instruction Step: Current step
Your response should be (self language] code, nothing else, formatted follows: (self. language)
Your (self. language) code here
Observation Message
Output: <Observation from Intercode Environment>
Step: <llext step to execute the plan>
Post-Plan Refinement Message
You have finished executing the plan, but it like there still issues with your answer. Please continue work on getting the correct answer. Per turn, you will be given the following information:
Observation:
Standard output from executing previous instruction
Your response should be (self language} code, nothing else, formatted as follous:
(self language) Your (self. language) code here
Table 13: Initial, Query, and Retry Prompts for Plan & Solve Strategy Evaluation. Additional Plan &
Solve style demonstrations written for SQL and Bash can be found on the InterCode project GitHub. Note that the Post-Plan Refinement Message is only used for the Plan & Solve + Refine strategy not used for the original Plan & Solve strategy.
discussed in B.3
29


Reviews:
[{"review_id": "4ktoJBBRKH", "reviewer": "Reviewer_1q7d", "summary": "", "questions": "", "limitations": "We prefer the feedback information that interacts with the environment to include guidance on gradually answering questions step by step with large language models rather than just providing debugging information as currently done.\n\nMaybe, the CTF dataset in Figure 4 is more suitable to be an interactive code generation dataset. However,  the amount of this dataset is too small and not fully described by the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper proposed an interesting task called interactive code generation. With the improvement of LLMs' programming ability, designing such kinds of tasks may contribute to applying LLM in practical programming tasks.\n\n2. An open-source, extensible framework was released to provide a useful interactive environment for code generation tasks in practical scenarios.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HDmEHPwhez", "reviewer": "Reviewer_VtXE", "summary": "", "questions": "", "limitations": "As authors have discussed, as a benchmark, it contains only two environment. ", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The work address a vital aspect in machine learning for code generation. The practice has proven that even few rounds of feedback could greatly improve the generated code quality. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "hfGMl3cJtf", "reviewer": "Reviewer_AK1E", "summary": "", "questions": "", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.Interactive code generation and code grounding are important research directions for the real-world application of code generation and this benchmark makes further progress towards standardizing and benchmarking these directions.\n\n2.This benchmark demonstrates its potential in future code generation tasks, such as flexibly using multiple languages to accomplish more complex tasks. This makes the agent behave more like a human rather than just a code completion tool, which is really exciting.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6CHzuu9YXo", "reviewer": "Reviewer_14Ca", "summary": "", "questions": "", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Mentioned in Summary And Contributions. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Tv2yphGU0d", "reviewer": "Reviewer_yCTe", "summary": "", "questions": "", "limitations": "The authors have adequately addressed all the limitations. ", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The interactive evaluation benchmark for code is novel and interesting. The InterCode framework is well implemented to be easy to use, extensible, and safe.\n* Evaluation on InterCode demonstrates LLMs' capability in error correction and context discovery, which hasn't been well quantified previously. \n* The paper and the appendix provide plenty of interesting examples of how LLMs solve a problem by interacting with the environment.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a coding framework that supports execution feedback. An agent can interact with the InterCode environment by executing the generated response, receiving feedback, and refining the answer iteratively. While the framework is extensible to various tasks and programming language, the authors propose in the main paper two benchmarks on two interactive code tasks with Bash and SQL and evaluate multiple SoTA LLM. The results show that with multi-turn prompting and interactive feedback, models exhibit the capability of error correction and context discovery, which significantly improves their performance compared to the single-turn baseline.

All the reviewers agree on the importance of the problem tackled, the novelty of the approach but were doubtful about the exhaustiveness of the study. The authors provided in the rebuttal additional tasks and programming language and discussed the generalization of the approach to broader tasks and to compiled languages. The tool is provided in a Docker format that is both ready to use and extensible and I believe the framework and benchmark opens new venues of improvement of LLM for code related tasks.
Decision: Accept (Poster)