SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A max score of 1 is achieved only if the correct file paths are changed, the changes are correct, and the latest execution output matches the gold command output exactly. Figure 1 visualizes the reward function. While an exact match comparison would have been a simpler choice to satisfy the Success Rate metric put forth in the main paper, we design this reward function to 1. Demonstrate that InterCode can support complex reward functions that account for multiple forms of execution output, and 2. Provide practitioners who use the InterCode-Bash environment with a scalar reward that reflects how "similar" the given output is to the expected, rather than a flat  $0/1$  reward value that may over-penalize and discount the efforts of more capable reasoning abilities. These reasons also motivate the SQL-based environment's reward function, discussed in the following section.
(a) File System 1
Figure 6: File System structures designed for InterCode-Bash.
(b) File System 2
Figure 7: Top 30 most frequently occurring bash utilities out of the 66 in InterCode-Bash with their frequencies in log scale.
Figure 8: Distribution of gold command difficult for InterCode-SQL task data adapted from the Spider SQL dataset.
A.3 SQL Environment
Environment Definition. The Dockerfile defining the SQL-based environment inherits from the MySQL image and adds a .sql file setup script to the /docker-entrypoint-initdb.d directory within the Docker image; this is a special directory made for container initialization. On container start-up, the added .sql file, which creates and populates databases with tables and tables with records, is automatically invoked. Since the InterCode-SQL dataset does not feature any queries that involve modifying the database in any manner (i.e. no INSERT, UPDATE, or DELETE commands), there is no reset mechanism written into the Dockerfile definition that is invoked before each task episode; with that said, adding a reset script or version control to the Dockerfile is simple.
InterCode-SQL Dataset. InterCode-SQL is adopted from the development set of the Spider dataset [51]. Spider 1.0 is a large-scale cross-domain dataset on generating SQL queries from natural language questions whose development set contains 1034 pairs of <instruction, gold> task instances spanning 20 databases. The distribution of queries according to their hardness criterion is shown in Figure 8. As discussed in Section 3.3, a filtering criterion narrows down the Spider dataset's information to only the necessary components. We do not add anything to the Spider dataset that was not originally available. The Spider 1.0 dataset is available for use under the CC BY-SA 4.0 license.
MySQL Databases. We first resolve data types for primary, foreign key pairs across the provided table schemas in Spider for conflicting instances and generate the corresponding SQLite databases. Next, to align with our Docker-supported environment, we convert the SQLite databases to MySQL format usingsqlite3mysql [38], a Python library, and then generate a unified MySQL dump having schemas for all the tables. To handle case-sensitive table name discrepancies between the queries and the underlying schema in the original Spider dataset, we activate the lower(case_table_names setting in our evaluation environment. Additionally, for proper access controls, we create a test user and grant them all privileges for all the tables.
Reward Function. The completion evaluation mechanism compares the output of the gold SQL command with the latest execution output (i.e. latest observation) from the agent's interaction trajectory. The execution output of all gold SQL queries is a list of records. Each record is a tuple of one or more values that may be different types. For any single execution output, the order of types for every record is identical. Given the agent command(s)' latest execution output  $A$  and the gold command's execution output  $G$ , we formulate the reward function as follows:
$$
\mathcal {R} = \frac {A \cap G}{A \cup G} * (k e n d a l l t a u ((A \cap (A \cap G)), (G \cap (A \cap G))) + 1) / 2 \tag {2}
$$
Figure 9: Example of interactions between an agent and the InterCode SQL Environment
Figure 10: Evaluation of the results of agent interactions with the SQL Environment against the gold command associated with the task. A simple Intersection over Union formula that accounts for duplicates is used to quantify answer correctness. Task completion is a reward of 1.
We employ Intersection over Union (IoU), or more formally the Jaccard Index, to quantify the correctness of the latest execution output generated by the agent against the gold output. If the latest execution output of the SQL query is not in the form of a list of records (i.e. a string error message), the reward is 0 by default. Among the items that lie in the intersection of the agent and gold execution outputs, we also apply a penalty if the records are in the incorrect order. Since achieving the correct order of fields in a record is of non-trivial importance to addressing many SQL queries correctly, we do not do any re-ordering or pre-processing of the list of records. Therefore, a record formatted as ("Ross", 29) is not awarded any credit against a gold output that includes (29, "Ross"). To quantify how sorted the agent output is relative to the gold output, we lean on Kendall's  $\tau$  and adjust the output range to [0, 1]. The IoU score is then directly scaled by this coefficient.
All in all, only a correctly ordered list with the exact set of records found in the gold output would receive a max score of 1, which corresponds to task completion. Figure 10 visualizes the reward function for an example set of outputs. Note that in the main paper, the Success Rate metric is used; the scalar  $3/7$  output shown in the figure is treated as a 0 when quantifying whether the task was completed via the 0/1 Success Rate metric. As mentioned in the discussion of the Bash reward function, this reward function also aims to be a richer and fairer continuous evaluation metric of a model's reasoning abilities compared to a binary 0/1 task completion score.
B.1 Model Details
We do not perform any model training for configuring the methods or running the experiments discussed in this project. Our evaluations use inference call requests to OpenAI, PaLM, and Hugging-Face API endpoints to run the baseline models on the InterCode tasks. For OpenAI models, we set temperature to 0, top_p to 1, max_tokens to 512, and n (number of completions) to 1. For PaLM models, we set temperature to 0, top_p to 1, and candidate_count (number of completions) to 1. For open source models, we set max_new_tokens (maximum number of tokens to generate) to 100 and temperature to 0.01. Due to constraints in the context window size, we limit the length of each
observation to a maximum of 1000 tokens across all inference calls. The code for configuring API calls can be found in the linked repository.
B.2 Additional Experiments & Analysis
SQL schema ablation. To confirm that the benefits of interaction exceed a simple disparity in information between the Single Turn and Try Again settings, we add the full SQL database schema, providing holistic details of tables necessary to the given instruction, to the Question message of both prompts, then re-run the comparison for several. Table 5 indicates that while Single Turn performance improves drastically, a non-trivial difference in favor of Try Again remains. Manual inspection of task episode trajectories shows that selective and fine-grained context discovery (i.e. inspecting specific table records and file content that affect query construction) is still critical to solving tasks efficiently.
InterCode-SQL + Schema
Model / Hardness	Single Turn	Try Again (max 10 turns)
Easy	Med	Hard	Extra	All	Easy	Med	Hard	Extra	All
gpt-3.5-turbo	90.7	70.2	59.2	37.3	67.9	92.7	74.9	67.2	43.4	72.8
text-bison-001	89.5	68.2	44.2	19.3	61.4	90.7	70.4	50.0	21.1	63.9
chat-bison-001	79.0	52.0	32.1	15.1	49.2	82.2	56.0	42.5	24.1	54.9
Trends of admissible actions. Table 6 shows that for the SQL task, models generate admissible actions with increasingly higher rates early on; in initial turns, models will tend to hallucinate a query with fabricated table and column names at a high frequency. The drop in error rate between the first and second turns can largely be attributed to the model's decision to begin exploring context;  $60.3\%$  of second turn actions contain either the SHOW TABLES or DESC keywords. Prompting strategies (i.e. ReAct, Plan & Solve), explicit phrasing that encourages exploration, and demonstrations diminish a model's default tendency to hallucinate a query in the first turn. This trend is not found in Bash. This can likely be attributed to the nature of the instructions; unlike the SQL instructions which simply pose a question and do not have any explicit references to SQL commands or clauses, Bash instructions will typically include keywords that correspond directly to useful Linux commands or give insight into the file system's internal structure. These signals reduce the need for context discovery. Therefore, successful task completion in Bash tends to lean towards 1) Figuring out which flags, options, and arguments to configure a command with and 2) How to string together commands or pass outputs from one command to the next correctly.
For both Bash and SQL, in later turns, the rate of admissible actions does not improve consistently. The actions in these later turns are usually attempts to answer the original instruction. At these stages, a model will tend to make small, cursory adjustments to the prior action based on execution feedback, often resulting in both a repetition of the same types of mistakes and hallucinations that introduce new issues. In these moments, compared to such minor perturbations, alternative reasoning capabilities such as context discovery and modularized problem solving are often more efficient ways to get the relevant insights needed to better decide how to fix the prior turns' issues. As corroborated by Figure 3, models struggle to take advantage of additional context in longer task episodes or horizons. Making the most of multiple queries is an open challenge with exciting implications for solving more difficult coding tasks.
Table 5: Success Rate across difficulty for single vs. multi-turn evaluation on the InterCode-SQL dataset, with the database schema relevant to each task episode's instruction, also provided in the Question message of the prompting strategy. Best metrics are in bold.
Turn	1	2	3	4	5	6	7	8	9	10
SQL	90.2	46.4	34.4	39.7	31.1	42.9	51.5	47.4	48.4	46.6
Bash	23.1	28.6	34.7	37.5	37.6	42.9	39.3	37.1	33.7	38.2
Table 6: Error % (Average ratio of non-admissible actions) per turn for the Try Again prompting scheme using a GPT 3.5 model on the Bash and SQL InterCode datasets.
Robustness results. We conducted an evaluation to assess the robustness of the reported accuracy metrics for the models. In order to maintain consistency in the evaluation, we focused on the performance across file systems 2, 3, and 4 (shown in Figure 6), which were designed to have similar difficulty levels. File system 1, intentionally made harder, was not included in this analysis. The standard errors for the Single Turn and Try Again modes are presented in Table 7. The Try Again mode leverages interaction to consistently outperform the Single Turn mode across all models.
Model	Single Turn	Try Again (n = 10)
text-davinci-003	31.40 ± 1.35	43.13 ± 5.98
gpt-3.5-turbo	36.63 ± 1.83	47.40 ± 1.23
gpt-4	38.37 ± 1.20	52.70 ± 3.50
text-bison-001	18.83 ± 3.57	22.40 ± 3.35
chat-bison-001	20.47 ± 1.89	21.67 ± 1.81
Vicuna-13B	16.73 ± 5.00	27.67 ± 4.15
StarChat-16B	19.37 ± 3.04	27.17 ± 2.74
Table 7: (Robustness Results) Success Rate with standard errors for single vs. multi turn evaluation on InterCode-Bash (refer §A.2). Best metrics are in bold. Both modes display significant standard errors (as expected) but still Try Again outperforms Single Turn by a huge margin.
B.3 Additional Prompting Strategy
To gauge the significance of designing prompting strategies that can successfully solve the interactive coding task, we attempt to devise a more performant approach by chaining together existing techniques, where each technique is meant to elicit a different, relevant reasoning skill. To this end, we design a hybrid prompting strategy that combines Plan & Solve and Try Again, which we refer to as "Plan & Solve + Refine". This strategy is meant to complement a model's planning, modularized task completion, and context discovery abilities with error correction. Figure 11 visualizes this prompting strategy's workflow. The full prompting template is included in § B.7.
Figure 11: Visualization of the Plan & Solve + Refine strategy. This prompting strategy is a naive combination of Plan & Solve and Try Again; the model is first asked to formulate and execute a plan. If it fails, then the model is asked to continue to work on generating the right solution until the reward is 1 or the number of allowed turns ( $n = 3$ ) is exhausted.
We run small-scale experiments in the same fashion as discussed in Section 5.2 on the SQL task for 44 randomly selected task instances. These preliminary tests have an average Success Rate of 45.5, a Number of Turns of 6.34 (across both planning and refinement stages), and an Error % of 22.4. Compared to the original Plan & Solve performance presented in Section 5.2, this strategy does not yield a significant improvement. Upon inspection of the trajectories, it is evident that while models are capable of specific reasoning techniques, it is not enough to elicit specific strategies in a pre-determined order. In addition, compared to human problem solvers, such reasoning frameworks do not intuitively allow for adjustment in strategy based on execution output. Therefore, mechanisms that allow a model to freely self-determine what strategies at any turn in task episode are integral to improving Success Rate and non-trivial because simple adjustments in existing prompting strategies are not enough to close this gap.
B.4 Task Trajectory Analyses
In this section, we discuss some of the behaviors and phenomena in different prompting schemes. We also include annotated trajectories that are representative of these observations.
Inst: Count md5sum of all ‘*.py’ files in /testbed folder with sub folders.
Gold: find /testbed -type f -name ‘*.py'
" -exec md5sum {} + | awk '{print $1}',
| sort | md5sum
Out: ff48e395ab4e0acb8336c4afe4504 -	Inst: For each continent, list its id, name, and how many countries it has?
Gold: SELECT T1.ContId , T1.Continent, count(*) FROM CONTINENTS AS T1
JOIN COUNTRIES AS T2 ON T1.ContId = T2.Continent GROUP BY T1.ContId;
Act: find /testbed -name ‘*.py’ -type f -exec md5sum {} + | wc -l
Obs: 4
Act: find /testbed -name ‘*.py’ -type f -exec md5sum {} + | awk 'END{print NR}',
Obs: 4
Act: find /testbed -name ‘*.py’ -type f -exec md5sum {} + | awk 'END{print}',
cut -d', '-f1
Obs: 0d1e201ac67b5a979c52f42eb8dd3702
Act: find /testbed -name ‘*.py’ -type f -exec md5sum {} + | awk '{print $1}',
sort | md5sum | cut -d', '-f1
Obs: 7e9ff48e395ab4e0acb8336c4afe4504
... (6 more turns)	Act: SELECT continent.id, continent.name, COUNT(country.id) AS num_countries FROM continent LEFT JOIN country ON continent.id = country.continent_id GROUP BY continent.id, continent.name
Obs: Error: Table 'car_1.continent' doesn't exist
Act: SHOW TABLES
Obs: [(‘car_makers’,), ('car_names’)] + 4 more...
Act: DESC continents
Obs: [(‘ContId’, 'int', 'NO', 'PRI', None, 'auto_increment'), ('Continent', 'text', 'YES')...
Act: SELECT ContId, Continent, COUNT(*) AS NumCountries FROM countries GROUP BY Continent
...(5 more turns)
Table 8: Examples of Try Again interaction trajectories with GPT 3.5 based model from Bash (left) and SQL (right). Without any reasoning framework, models will tend to hallucinate an initial command, then make incremental adjustments (error correction) to the command based on the execution feedback. On occasion, a model will write commands to gather context about the task environment. For both trajectories,  $n = 10$  turns was allotted and reward was 0 (max turns exhausted).
Default model behaviors. In both the Single Turn and Try Again setups, the templates describe the interactive coding task and do not provide the model with any reasoning framework or strategy, providing insight into a model's default approach when given the instruction and task environment. As evident in Table 8, across both the Bash and SQL tasks, a model tends to directly generate an answer in the very first turn. For Try Again, the model will then perform error correction, then make slight adjustments to the answer to resolve the issue(s) reflected in prior execution output across the next several turns. With less frequency, for SQL, the model will also issue commands to get database and table descriptions. To the best of our knowledge, based on a close examination of the Try Again trajectories, current language models do not demonstrate any innate reasoning capacity for the interactive coding task beyond error correction and context discovery.
Prompting strategy effects. In contrast with Try Again, the ReAct prompting strategy briefly introduces the interactive task environment and proposes a reasoning framework for the model to abide by. Table 9 presents a side-by-side comparison of the Try Again and ReAct [48] strategies. The figure reflects the richer types of problem-solving that a model exhibits when prompted to reason on its own thoughts about the execution output. This reflects that through better strategies, it may be possible to make significant advancements in the interactive coding task with prompting strategies that attempt to elicit reasoning via an appropriate framework that also permits the model to be expressive and creative in devising its own solutions. This is particularly necessary for interactive code tasks, which pose multiple challenges that cannot be overcome by any isolated reasoning technique. As demonstrated in § B.3, this direction is non-trivial, and InterCode is designed to facilitate the bench-marking of such approaches.
B.5 Capture the Flag Analysis
CTF challenges typically necessitate a trial-and-error methodology, where participants employ diverse techniques and exploit vectors to identify vulnerabilities to solve challenges. Processes such as exploring complex environments or executables, debugging, and dynamic exploitation, which
Inst: What is TV Channel of TV series with Episode "A Love of a Lifetime"? List TV Channel's series name. Gold: SELECT T1_series_name FROM TV_Channel AS T1 JOIN TV_series AS T2 ON T2.Channel = T1.id WHERE T2.Episode = "A Love of a Lifetime";
Try Again  $(\mathrm{r} = 0,$  length  $= 10$  max turns  $= 10$
Act: SELECT t1/channel, t2(series_name FROM table1 t1 JOIN table2 t2 ON t1_series_id = t2_series_id WHERE t2.episode_name = 'A Love of a Lifetime'
Obs: Error: Table 'table1' doesn't exist
Act:SHOW TABLES
Obs: [(‘cartoon’), ('tv_channel'), ('tv_series'),]
Act: DESC tv_channel
Obs: ['id', 'varchar(255)', 'NO', 'PRI', None,'],
('series_name', 'text', 'YES', "", None...] + 7 more...
Act: DESC tv_series
Obs: ['id', 'double', 'NO', 'PRI', None, ''], ('Episode', 'text', 'YES', " + 7 more...
Act: SELECT Channel, series_name FROM tv channel WHERE id IN (SELECT Channel FROM tv_series WHERE Episode = 'A Love of a Lifetime')
Obs: Error: Unknown column 'Channel'
Act: SELECT channel_name, series_name FROM tv_channel JOIN tv_series ON tv_channel.id = tv_series. Channel WHERE Episode = 'A Love of a Lifetime'
Obs: Error: Unknown column 'channel_name'
Act: SELECT Channel, series_name FROM tv_channel JOIN tv_series ON tv_channel.id = tv_series.Channel WHERE Episode = 'A Love of a Lifetime'
Continued hallucinations in problem solving actions for last 4 turns...
Thought: I should check what tables are available in this MySQL database that can help me answer this question
Act: SHOW TABLES
ReAct  $(\mathrm{r} = 1$  , length  $= 7$  max turns  $= 10$
Obs: ['cartoon', ('tv_channel'], ('tv_series'),] ... (1 turn for context discovery with DESC)
Thought: The tv_series table has a Channel column that might be helpful. I should check if there is a row with the episode name "A Love of a Lifetime".
Act: SELECT Channel FROM tv_series WHERE Episode = 'A Love of a Lifetime'  
Obs: ['700'].
Thought: I get a single value, 700, which is the TV Channel of the TV series with the episode "A Love of a Lifetime". Check what series is on this channel.
Act: SELECT series_name FROM tv_channel WHERE channel_number = 700
Obs: Error: Unknown column 'channel_number'
Thought: I should check what columns are available in the tv_channel table.
Obs: ['id', 'varchar(255)', 'NO', 'PRI', None, ('series_name', 'text', 'YES', "... + 7 more...
Thought: The tv_channel table does not have a channel_number column. Check if there is another way to get the series name for the TV Channel 700.
Act: SELECT series_name FROM tv_channel WHERE id = '700'
Obs: ['Sky Radio',]
Thought: I get a single value, Sky Radio, which is the series name for the TV Channel 700.
Act: submit
Table 9: Comparison of Try Again & ReAct interaction trajectories with GPT 3.5 based model towards solving the same instruction. Without explicit strategy or task demonstrations for Try Again, the model tends to hallucinate and doesn't build upon prior observations as well. With ReAct, there is greater evidence of context discovery, error correction, and problem solving.
involve sequential steps, require iterative interaction. Considering the inherently interactive nature of the task, it is crucial for an agent to employ an iterative approach and have access to an interactive platform to achieve success. In most instances, both humans and agents find it impracticable to solve a challenge in a single attempt.
Similar to the aforementioned exploitation task (Figure 4), we now present a more intricate forensics task in Figure 12.
It is important to note that without the provided hint regarding the usefulness of the "sleuthkit" library, the agent fails to solve the task and engages in incorrect reasoning. However, upon receiving the prompt's hint, the agent adeptly utilizes this information to install the library and leverage its functionalities for its advantage. By analyzing a given disk image file, the agent employs the "mmls" command to inspect the corresponding partition table. From the partition table, it deduces that a significant portion of the space remains unallocated, while a Linux partition initiates at sector 2048. Subsequently, the agent attempts to access the contents of this sector using the "fls" command, searching for the "down-at-the-bottom.txt" file, which it anticipates will contain the flag. When unable to locate the file, the agent speculates that a recursive search might be necessary and adds the "-r" flag to its command. Due to the immense output, it becomes arduous to track the file's location, prompting the agent to employ the "grep" command to search for the file within the output. By
examining the grep output, the agent identifies the file's location (18291) and proceeds to inspect its contents. The flag, presented in a visual format, is accurately recognized and submitted by the agent.
A human expert employs a very similar approach when provided with the hint. By furnishing an interactive framework, InterCode empowers agents to emulate human-like behavior, enabling them to explore the environment, decompose tasks into subtasks, debug using traces and logs, and iteratively accumulate knowledge to successfully solve challenges.
Figure 12: GPT-4's interaction trajectory for a CTF forensics task. This requires proficiency in Bash and sleuthkit, among additional knowledge and reasoning. Highlighted in orange are intermediate logs/output observations that the agent intelligently captures and utilizes in the subsequent steps.
B.6 Human Performance Baseline