SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To explore the gap between human and agent performance on the interactive coding task, we the authors, all proficient in SQL, act as human task workers and perform the task on a random sample of 15 InterCode-SQL task instances within the same task environment identical to the agent's setting. A max number of  $n = 10$  turns is imposed, as was done with the Try Again prompting strategy. Similar to ReAct and Plan & Solve, the human task worker decides when to submit; in other words, the task does not terminate automatically when reward  $= 1$ . The trajectories for these 15 instances and the code for facilitating human interaction with the InterCode-SQL environment are available in the codebase.
The human task worker was able to complete 13 of 15 tasks (Success Rate  $= 0.87$ ) with low Error  $\%$  , most of the errors occurring not because of hallucinations of table columns and attributes, but rather because of SQL syntax errors that arose due to mistakes in relatively complex queries. What's noteworthy about the human task worker's trajectories is the presence of much more modularized problem-solving that deviates heavily from an agent's approach of generating a query in a single go. Even with context discovery and error correction, an agent's action to produce an answer for the
instruction will tend to be a single, self-contained command that generates the answer in one go. On the other hand, a human task worker will tend to break up the query solution into multiple smaller sub-problems. This is particularly evident for instructions that must be answered with investigations across multiple tables with relations established by primary and foreign key columns. As an example, given an instruction "Find the average weight of the dog breed that is owned by the majority of pet owners", a human task worker might write commands that query the petOwners table to determine what the most popular dog breed is, and then use the answer to this sub-problem as a field in the WHERE clause of a second query that then determines the average weight using the pets table.
A more thorough and variegated study would be required to fully establish the performance gap between humans and agents. Nevertheless, from this small study, we are confident that humans generally exhibit more flexible and variegated reasoning capabilities compared to agents in the interactive coding task. Closing this gap is an exciting research direction, and beyond model-side improvements and scaling laws, incorporating human task reasoning and execution as guidance, feedback, or reward signals is a worthwhile consideration toward improving model performance.
B.7 Prompt Templates
As discussed in the paper, the main baseline evaluations for InterCode consist of presenting a language agent with an instruction and a prompting strategy that have been adapted for InterCode's interactive task setting. Each prompting strategy is defined as a template with three components:
- Initial Message: This is the first message presented to the agent. The initial message may describe the general task to accomplish, guidelines for interacting with the InterCode environment, the formats of the instruction and observation(s), and any additional information that pertains to the environment. In addition to the environment and task specifications, the general prompting strategy and useful demonstrations may also be discussed. The initial message is presented once as the first message of a task episode.
- Instruction Message: This is the template for communicating the instructions that an agent is asked to solve for a particular task episode. The instruction message is presented once as the second message of a task episode.
- Observation Message: This template is for communicating the standard output and any additional information for a single interaction. This observation is what the agent will use to generate the next action. The observation message may be presented multiple times depending on how many interactions the task episode lasts for.
Figures 10, 11, 12, and 13 present the corresponding prompt templates for the Try Again, ReAct, and Plan & Solve experiments, along with a specific version for the toy Capture the Flag task.
## TASK DESCRIPTION
You are a {self.language} code generator helping a user answer a question using {self.language}. The user will ask you a question, and your task is to interact with a {self.settingting} system using {self.language} commands to come up with the answer.
## RULES
1. Do NOT ask questions
2. Your response should only be {self.language} commands
## RESPONSE FORMAT
Your response should be a {self.language} command. Format your {self.language} command as follows:
\{\text{self的语言}\}
Your {self.language} code here
Write {self.language} commands to help you do two things:
1. Learn more about the {self.setting} you are interacting with. For example, if you are interacting with a MySQL database, you can use the DESCRIBE command to learn more about the tables you have access to.
2. Execute {self.language} commands based on these tables to generate the correct output.
DO NOT WRITE ANYTHING EXCEPT FOR CODE in your response.
## OUTPUT DESCRIPTION Given your {self.language} command input, the system will then give back output formatted as follows:
Output: <string>
Reward: [0, 1]
The output is the standard output from executing your {self.language} command. The reward is a decimal value between 0 and 1, which tells you how close your {self.language} command is to the correct answer. The closer the reward is to 1, the closer your {self.language} command is to the correct answer.
Instruction Message
Query: "<Task Episode Instruction>"
Observation Message
Output: <Observation from Intercode Environment>
Reward: <Value  $\in [0,1]$  >
Table 10: Initial, Query, and Retry Prompts for the Try Again Strategy Evaluation. {self .language} is substituted with the programming language for interacting with the environment. {self .setting} is substituted with a natural language description of the environment
Interact with a {self.setting} system using {self.language} queries to answer a question.
Use interleaving Thought, Action, Observation steps.
Thought can reason about the current situation, and Action can be two types:
(1) execute, which executes {self.language} code on the {self.setting} system
(2) submit, which indicates that the previous observation is the answer
Question: What are the names and grades for each high schooler?
Thought 1: I should write a SQL command that selects the name and grade fields from a table about high schoolers.
Action 1: execute[SELECT name, grade FROM high_schoolers]
Observation 1: Error executing query: Table 'network_1.high_schoolers' doesn't exist
Thought 2: The high_schoolers table doesn't exist, I need to check what tables are available to use.
Action 2: execute[SHOW TABLES]
Observation 2: [(‘friend’,), ('highschooler'), ('likes',)]
Thought 3: There is a table called highschooler. I should check what information this table contains.
Action 3: execute[DESC highschooler]
Observation 3: [(‘ID’, ‘int’, ‘NO’, ‘PRI’, None, ‘auto_increment’), ('name',
'text', 'YES', "", None, "", ('grade', 'int', 'YES', "", None, ])
Thought 4: The highschooler table contains the name and grade columns. I can now write a SQL command that answers the question
Table 11: Initial, Query, and Retry Prompts for ReAct Strategy Evaluation. Additional ReAct style demonstrations written for SQL and Bash can be found on the InterCode project GitHub.
You are an expert Capture the Flag (CTF) player.
You are in the '/ctf' folder of a standard Ubuntu machine. <Task Episode Instruction>. Your goal is to find this hidden flag by interacting with the bash terminal step-by-step. The flag is in the format 'picoCTF...'. You can perform any valid action in the terminal by starting your action with 'cmd <your_command>'. For bash commands, prepend '/bin/bash -c <your_command>', for python commands, prepend 'python3 -c <your_command>' and so on. For every action you take, you will receive the corresponding standard output as observation. You can also explore/inspect the file system or do anything that helps you find the flag. Once you feel you have found the flag, you can choose to submit it by printing 'submit <your_flag>'. Do NOT provide an explanation for your answer, only output the action you want.
Table 12: Initial, Query, and Retry Prompts for Capture the Flag Evaluation.
Plan Message
For the following user question, let's first understand the problem and devise a plan to solve the problem. Then, let's carry out the plan to solve the problem step by step.
Your plan should describe a sequence of {self.language} queries you can write to determine the answer. Here are three examples of coming up with a plan for a question.
1. Check what tables are available for use.
2. Inspect each table to identify which has information about high schoolers.
3. Use the table to write a query that selects the name and grade fields for each high schooler.
Execute Plan Message
You will now execute your own plan. Interact with a {self.setting} system using {self.language} queries to answer a question. Per turn, you will be given the following information:
Observation: Standard output from executing previous instruction
Step: Current step
Your response should be {self.language} code, nothing else, formatted as follows:  $\cdots\{\mathrm{self\_language}\}$
Step: <Next step to execute from the plan>
Post-Plan Refinement Message
You have finished executing the plan, but it seems like there are still issues with your answer. Please continue to work on getting the correct answer. Per turn, you will be given the following information:
Table 13: Initial, Query, and Retry Prompts for Plan & Solve Strategy Evaluation. Additional Plan & Solve style demonstrations written for SQL and Bash can be found on the InterCode project GitHub. Note that the Post-Plan Refinement Message is only used for the Plan & Solve + Refine strategy discussed in § B.3. It is not used for the original Plan & Solve strategy.