SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Success Rate for single vs. multi turn evaluation on InterCode-SQL (refer §A.3). Query difficulty is adopted from Spider [51]. Best metrics are in bold.
InterCode-Bash
Model / File System	Single Turn	Try Again (n = 10)
1	2	3	4	All	1	2	3	4	All
text-davinci-003	10.0	32.1	28.8	33.3	24.6	30.0	52.8	32.2	44.4	38.7
gpt-3.5-turbo	30.0	39.6	33.3	37.0	34.5	45.0	49.1	45.0	48.1	46.5
gpt-4	25.0	37.7	36.7	40.7	34.0	41.7	47.2	51.7	59.2	48.5
text-bison-001	15.0	22.6	11.7	22.2	17.0	23.3	28.3	16.7	22.2	22.5
chat-bison-001	12.1	22.5	16.7	22.2	17.7	13.8	24.5	18.3	22.2	19.2
Vicuna-13B	10.0	24.5	18.3	7.4	16.0	15.0	35.8	25.0	22.2	24.5
StarChat-16B	15.5	22.6	13.3	22.2	17.7	17.2	30.2	21.7	29.6	23.7
Table 3: Success Rate across file systems for single vs. multi-turn evaluation on InterCode-Bash (refer §A.2). To evaluate models' ability to interact with different task settings, we evaluate disjoint sets of Bash instructions across four different file systems. Best metrics are in bold.
that performance across different levels of task difficulty (SQL) and different file systems (Bash) is superior in the interactive setting for all models, with a notable multi-fold increase for GPT-4  $(9.1\% \rightarrow 73.7\%)$  on the InterCode-SQL task.
Analysis of interactions. Manual inspection of trajectory logs indicates that models actively exercise later turns for discovering relevant context, correcting errors via execution feedback as observations, and solving problems via iteratively constructing and editing actions as affirmed by Figure 3. In addition, models also demonstrate a level of planning and modular problem solving; for instructions with gold commands that chain multiple commands together (i.e. with  $\mid , >$ , or ; in bash) or consist of multiple sub-problems (i.e. subqueries in SQL), models will use observations from solving smaller sub-problems in earlier turns to compose the higher-order action. Trajectories that exhibit these phenomena are in § B.4
Failure cases. With that said, both Figure 3 exhibits a plateauing in Success Rate and and Error  $\%$  . This suggests that as the amount of context and feedback builds up, models are less capable of discerning relevant past history toward future actions. In late-turn scenarios, task episode trajectories often reveal repetition of earlier actions, a failure to effectively use recent observations towards deciding an appropriate next action, or an inability to recognize that a current problem-solving chain of thought is inconclusive or futile. This is particularly evident for hard and extra level InterCodeSQL task instructions that require context spanning across several tables and actions that incorporate multiple clauses. We note that even when the full schema of all tables and their descriptions are offered in addition to the original instructions, models still benefit greatly from using interaction to experiment with different JOIN and filtering operators across multiple turns, as demonstrated in  $\S$  B.2. A larger context window size, retrieval of useful memory, and more adaptive reasoning paradigms are just a handful of potential solutions to overcoming such challenges.
5.2 Prompting strategy comparison
Initiating language agents with prompting strategies that encourage different forms of reasoning toward problem-solving improves performance on the interactive coding task to varying degrees. Table 4 presents side-by-side comparisons of the success rate, number of turns, and error rate per strategy. Compared to Try Again, which lacks specific guidance on leveraging multiple turns, more
(a) Success rate vs. turns for InterCode-Bash
(b) Success rate vs. turns for InterCode-SQL
Figure 3: Growth in Success Rate with increase in number of interaction turns across models configured with Try Again prompting strategy for InterCode-Bash and SQL tasks.
Try Again (n = 10)	ReAct (n = 10)	Plan & Solve
SR	Turns	Error %	SR	Turns	Error %	SR	Turns	Error %
SQL	47.3	7.25	46.4	58.7	5.30	6.94	49.1	4.29	16.2
Bash	46.5	6.15	24.9	20.5	4.40	20.4	28.0	6.65	53.3
Table 4: Comparison of different prompting strategies across the entire InterCode-SQL and InterCode-Bash datasets using gpt-3.5-turbo as the base model. Turns refers to the average number of turns taken for a single task episode. For Try Again and ReAct, the max number of turns  $n = 10$ . The highest Success Rate, fewest Turns, and lowest Error % are highlighted per dataset since they reflect more accuracy and efficient task solving. Best metrics are in bold.
explicit reasoning frameworks such as ReAct and Plan & Solve policies generally achieve higher success rates (SQL:  $47.3\% \rightarrow 58.7\%$ ) with fewer turns and a higher rate of admissible commands.
Different tasks present different learning challenges. An important skill to solving the InterCode-SQL task is the ability to discover context and construct actions conditionally based on information revealed in prior observations. Given that InterCode-SQL task instructions are phrased most commonly as questions, adapting to the task setting and new information discovered along the way puts more emphasis on error correction and context discovery. On the other hand, the more declarative and multi-step nature of the InterCode-Bash task instructions is more aptly solved by planning and modular task completion. These distinctions manifest in the Plan & Solve strategy's performance gap between the InterCode-SQL and InterCode-Bash tasks; while Plan & Solve encourages a model to decompose problems into more manageable steps, the strategy is less favorable towards adjusting on the fly in response to execution feedback. Example trajectories supporting these claims are in § B.4.
More adaptive reasoning is favorable. Compared to "imperative" reasoning paradigms such as Plan & Solve which prescribe a relatively rigid procedure, more flexible frameworks like ReAct, which do not enforce any particular logical formula or roadmap, are more conducive to eliciting a broader set of reasoning capabilities. However, while ReAct's performance is generally superior to Plan & Solve, tasks solved by both strategies with gpt-3.5-turbo make up  $57\%$  (407/708) and  $27.6\%$  (21/76) of the union of all successfully solved InterCode-SQL and InterCode-Bash tasks respectively. This discrepancy highlights a trade-off between the guidance and structural constraints that are inherent to prompting strategies; schemes that draw out specific reasoning patterns often overlook other equally useful capabilities. InterCode's interactive coding task can serve as a strong litmus test toward more adaptable, variegated model reasoning.
5.3 New tasks & datasets opportunities
InterCode's task formulation, modular design, flexible task construction, and use of virtual containers enable task designers to manifest new, complex, code-driven tasks, where completion is much more
Challenge Description: You should utilize the 'dds2-alpine.flag.img.gz' image. All we know is the file with the flag is named 'down-at-the-bottom.txt'. The 'slethkit' tools can be used for help. Flag: picoCTF{f0r3ns1c4t0r_n0v1c3_f5565e7b}
/bin/bash -c "gunzip dds2-alpine.flag.img.gz"
.Bash
dds2-alpine.flag.img: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x0,32,33), end-CHS (0x10,81,1), startsector 2048, 260096 sectors
apt-get install sleuthkit
Reading package lists...\NBuilding dependency tree...\NReading state information...\Nslsleuthkit is already the newest version (4.11.1+dfsg-1).\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Figure 4: GPT-4's interaction trajectory for a binary exploitation CTF task. This requires proficiency in Bash and Python, among additional knowledge and reasoning. Orange text and arrows highlight the feedback that the model attends to in generating the next action. In last step, agent submits flag.
mmls dds2-alpine.flag.img
-Bash
DOS Partition Table  
Offset Sector: 0  
Units are in 512-byte sectors  
Slot Start End Length Description  
000: Meta 000000000 000000000 000000001 Primary Table (#0)  
001: ———— 000000000 0000002047 0000002048 Unallocated  
002: 000:000 000002048 000262143 00026096 Linux (x8x3)
-Agent
fls -o 2048 -r -p dds2-alpine.flag.img | grep down-at-the-bottom.txt
r/r 18291: root/down-at-the-bottom.txt
icat -o 2048 dds2-alpine.flag.img 18291
<flag in visual form>
submit picoCTF{f0r3ns1c4t0r_n0v1c3_f5565e7b}
The submitted flag is correct.
attainable through interaction. We draw inspiration from Capture the Flag (CTF) [14], a competitive cybersecurity game that requires expertise in coding, cryptography (i.e. binary exploitation, forensics), reverse engineering, and recognizing security vulnerabilities to accomplish the primary objective of discovering encrypted "flags" concealed within code snippets or file systems. Compared to InterCode-Bash & -SQL, CTF is much more complicated, requiring an agent to exercise knowledge of multiple coding languages, modularize a higher-order objective into sub-problems, construct multi-step plans towards solving each problem, and adjust strategy when a plan fails to yield any useful insights.
We establish InterCode-CTF, a new dataset consisting of 100 CTF objectives from picoCTF [39]. Following the interactive coding task formulation, each task instance in InterCode-CTF is given as a <instruction, assets, hidden flag> tuple. We first construct a Bourne Shell within an Ubuntu OS as the task environment. Here, InterCode's use of virtual containers is crucial, as necessary actions can be irreversibly damaging on real systems (i.e. rm -rf, sudo access). Per task instance, the associated assets (e.g., images, executables, code), necessary for task completion, are copied into the OS file system. Given this setting, a task worker must understand the given material and investigate the assets to develop potential solutions. Executing a successful approach must be done across multiple steps with various conditionals, where the execution feedback of a prior step could have a significant effect on the next step. Figure 4 spotlights the diverse skills needed for CTF.
Conclusion. We have developed InterCode, a novel lightweight framework that facilitates interaction between Language Models and the underlying environment, enabling them to mimic the human approach to language-to-code generation. Our framework has shown promising results when applied to state-of-the-art models using different prompting styles. It effectively leverages the capabilities of LMs to break down complex tasks and recover from errors within a secure and isolated environment. The ability to seamlessly convert existing datasets into the interactive format using InterCodeEnv API, and furthermore, the Bash and SQL environments, empowers task designers to construct new tasks to unlock the plethora of challenges that await in the space of interactive coding.
Limitations and future directions. We point out several current limitations of InterCode. At this time, the number of InterCode based environments is limited to Bash, SQL, and Python action spaces and datasets; within the near future, we plan to expand the number of offerings to cover a wider set of programming languages and datasets that should further deliver on InterCode's purported promises of efficient and expressive task construction. Second, the CTF dataset is limited to just four task instances due to our manual curation procedure. We hope to release more formal work soon that provides a more thorough analysis of the reasoning and collaboration challenges of the CTF task along with a more extensive dataset for evaluation purposes.
We thank Xiao Liu for the Vicuna/Alpaca APIs, Carlos Jimenez and Yuhan Liu for trying our code, and Princeton NLP Group for helpful discussion and feedback in general. We acknowledge support from the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
[51] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1425. URL https://aclanthology.org/D18-1425.
[52] T. Yu, R. Zhang, H. Y. Er, S. Li, E. Xue, B. Pang, X. V. Lin, Y. C. Tan, T. Shi, Z. Li, Y. Jiang, M. Yasunaga, S. Shim, T. Chen, A. Fabbri, Z. Li, L. Chen, Y. Zhang, S. Dixit, V. Zhang, C. Xiong, R. Socher, W. S. Lasecki, and D. Radev. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases, 2019.
[53] T. Yu, R. Zhang, M. Yasunaga, Y. C. Tan, X. V. Lin, S. Li, H. Er, I. Li, B. Pang, T. Chen, E. Ji, S. Dixit, D. Proctor, S. Shim, J. Kraft, V. Zhang, C. Xiong, R. Socher, and D. Radev. SParC: Cross-domain semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4511-4523, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1443. URL https://aclanthology.org/P19-1443.
[54] L. Zeng, S. H. K. Parthasarathi, and D. Hakkani-Tur. N-best hypotheses reranking for text-to-sql systems, 2022.
[55] K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin. Self-edit: Fault-aware code editor for code generation, 2023.
[56] S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, and C. Gan. Planning with large language models for code generation, 2023.
[57] T. Zhang, T. Yu, T. B. Hashimoto, M. Lewis, W. tau Yih, D. Fried, and S. I. Wang. Coder reviewer reranking for code generation, 2022.
[58] V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017.
[59] S. Zhou, U. Alon, S. Agarwal, and G. Neubig. Codebertscore: Evaluating code generation with pretrained models of code, 2023.
In this appendix, we provide additional details about the implementation and usage of the InterCode framework and the InterCodeEnv interface. We also provide visualizations and analyses of additional experiments to demonstrate InterCode's utility and garner further insight into the extent of current models' performance on the interactive coding task. The full template for each prompting strategy is also included. Finally, we also discuss some of the impacts, risks, and limitations of our work. The webpage for InterCode is https://intercode-benchmark.github.io/. The code for InterCode is https://github.com/princeton-nlp/intercode; the link is also included on the InterCode webpage.
A Environment Details
A.1 InterCode Interface
The InterCode interface inherits the OpenAI gym [5] environment API definition. Specifically, InterCodeEnv is written as an abstract class that primarily handles the main execution logic for processing code interactions, in addition to logging, data management, and sand-boxed execution, along with both environment-level and task-level customization.
InterCodeEnv exposes the following API. Creating an interactive coding environment requires defining a subclass of InterCodeEnv. The methods denoted with an asterisk can be overridden for the purposes of customization.
__init__(self, data_path: str, image_name: str, **kwargs)
- Validates that the dataset specified by data_path is formatted correctly and can be used in an interactive setting.
- Uses the Docker image specified by image_name to create and connect with a Docker container instance of the image.
- Initialized Logging Handler
- Keyword arguments:
- verbose (bool): If true, logging is enabled and environment interactions are shown to standard output
- traj_dir (str): If a valid path is provided, task episode summaries are saved to the given directory (generated by save_trjectory)
- preprocess (callable): If provided, this function is run before every task episode. It is a way to provide task instance-specific customization of the execution environment.
reset(self, index: int = None) -> Tuple[str, Dict]
- Retrieves task record from data loader
- Calls resetcontainer
- Reset task level logger, instance variables
step(self, action: str) -> Tuple[str, int, bool, Dict]
- Log (action, observation)
- Invoke exec_action on action argument
- If action=submit, invoke get Reward, save_Trjectory
save_trjectory(self)
- Saves task metadata, (action, obs.) sequence, and reward info to .json in traj_dir
close(self)
- Safely exit or stop any resources (i.e. docker container) used by the environment
\*execute_action(self, action: str)
- Defines how the action is executed within the context of the docker container.
- Requires impl. because the Dockerfile definition, particularly its entrypoint, affects how an action would be invoked within the container.
Figure 5: Visualization demonstrating the intended invocations and usage of the InterCodeEnv interface, along with how the functions requiring implementation (get Reward(), execute_action(), resetcontainer() are called by the methods of the main interactive loop.
- Default impl. passes the action string directly into a selfcontainer.exec(action) call, which invokes the action in the environment and returns execution output. A timeout is imposed on execution duration.
\*get Reward(self) -> Tuple[float, Dict]
- Handles reward calculation of actions with respect to the gold command(s) for a task episode.
- Requires impl. because the concept and scoring for task completion varies across datasets and environments.
\*resetcontainer(self)
- Handles resetting of execution container (i.e. resetting file system to original state).
- Requires impl. because the approach to restoring a setting to its initial state varies.
Figure 5 conveys how each of these methods are invoked and how they related to one another. In summary, the technicalities for setting up an interactive coding task for a specific system with one or more programming languages as the action space involve:
- Defining a Dockerfile
- Providing a dataset with the query and gold fields
- (Optional) Defining a reward (get Reward) function to define task completion.
- (Optional) Creating an InterCodeEnv subclass that overrides the execute_action and get Reward methods
A.2 Bash Environment
Environment Definition. The Dockerfile defining the Bash-based environment is founded on the LTS version of the Ubuntu operating system. Several Linux dependencies that can potentially be used by an agent to address instructions in the InterCode-Bash Dataset are then installed via the Advanced Package Tool (apt) interface. Next, a shell script is invoked within the Dockerfile to initialize one of the three file systems displayed in Figure 6. The shell script consists of a simple sequence of mkdir, touch, and echo commands to deterministically create and populate the content of multiple files and folders. Finally, git is configured for the purposes of determining file diffs per task episode (git status -s) and resetting an environment to its original state (git reset -hard; git clean -fd;) before the beginning of a new task episode. The original code for the Dockerfile along with the file system creation scripts can be found on the project GitHub repository.
Dataset Details. The log-frequency distribution of the top-50 utilities is displayed in Figure 7. The NL2Bash [29] dataset is made available for use under the GPLv3 License. To assess the
generalizability of our approach, we designed three distinct file systems to accommodate the bash commands we collected. A key consideration during the construction of these file systems was to ensure that a significant portion of the executed commands would not result in operations that yield no changes. This deliberate design choice aimed to provide a more comprehensive assessment of our approach's adaptability and effectiveness across various scenarios and command executions. The file systems encompass a wide range of file types, including text files (.txt), program files (.c, .java, .py), compressed files (.gz), shell scripts (.sh), PHP scripts (.php), JSON files (.json), documents (.doc), spreadsheets (.csv), webpages (.html), database schemas (.sql), hidden files, and files with special characters in their names, convoluted folder hierarchies. Their directory structures are illustrated in Figure 6. For simplicity, we consider the top-level folder created within the root directory (testbed, system, workspace) as the root of each file system. This root folder contains files and sub-folders that necessitate access and manipulation, while changes are monitored throughout the entire container to accurately evaluate the models' actions. Notably, we intentionally designed file system 1 to be more intricate and encompass relatively challenging bash tasks compared to the other two file systems. Thereby, the models' performance is relatively lower for file system 1.
Reward Function. Evaluation of an agent's trajectory across a single task episode towards carrying out the given instruction is determined by modifications to the file system and the latest execution output. The instructions found in the InterCode-Bash dataset fall under one of two buckets: it either 1. Requests information about the file system that can be answered via execution output generated from a correct sequence of Bash actions (i.e. "How many files...","What is the size of...","Where is the .png image stored?") or 2. Requests a change to the location, configuration, or content of a file or folder (i.e. "Move the dir1 folder from...","Set the permissions to...","Append a line to..."). Any relevant correct changes are therefore captured by considering both execution output and file system modifications during evaluation.
We define  $A$  and  $G$  as the outputs of the agent and gold commands respectively, where  $A_{out}$  and  $G_{out}$  refer to the execution output, and  $A_{fs}$  and  $G_{fs}$  refer to a list of entries reflecting file system modifications, where each entry is [file path, modification type ∈ [added, changed, deleted]]. We then formally define the reward function as follows:
$$
\begin{array}{l} \mathcal {R} = 0. 3 4 * \text {s i m i l a r i t y} \left(A _ {\text {o u t}}, G _ {\text {o u t}}\right) \\ + 0. 3 3 * (1 - \operatorname {e r f} \left(\left| A _ {f s} \cup G _ {f s} - A _ {f s} \cap G _ {f s} \right|\right)) + \tag {1} \\ + 0. 3 3 * \frac {\text {i s} _ {-} \operatorname {c o r r e c t} \left(A _ {f s} \cap G _ {f s}\right)}{A _ {f s} \cap G _ {f s}} \\ \end{array}
$$
Where similarity refers to lexical similarity, which is determined by the cosine similarity score between TF-IDF vectors (calculated with TfidfVectorizer from scikit-learn) of the two execution outputs. The second component of the reward function reflects the number of file system modifications that were either not completed or not necessary; the error associated with the total number of misses is constrained to the range [0,1] using the Gauss error function (erf), where 0 corresponds to no file system modification mistakes. The third component checks what proportion of paths altered by both agent and gold were modified correctly. The is.correct function returns the number of file paths that were changed correctly, determined by checking whether the md5sum hashes of each file path are identical for agent and gold. If  $A_{fs} \cap G_{fs} = \emptyset$ , this reward is automatically 1. The scalar weights for each component are arbitrarily assigned.