SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sample-terminal> Prompt
Here is an example of solution.   
# Task: Use the terminal below to delete a file ending with the extension .gif def solution(agent, start_from  $= 1$  ..   
# General plan: I should first type "ls" to list all files in the terminal, then identify the filename ending with ".gif" and type "rm [filename].gif" to delete the identified file. if start_from  $\Leftarrow$  1: print('[Step 1] type "ls" to list all files in the terminal') agent.type('ls') state_after_interaction  $=$  agent.press_key('enter') # the file list should be displayed on terminal. assert 'gif' in state_after_interaction, f'I cannot do [Step 1] correctly. The file list is not displayed on the terminal. Current state: {state_after_interaction}'   
if start_from  $\Leftarrow$  2: print('[Step 2] identify the filename ending with ".gif" and type "rm [filename].gif" to delete the identified file') # I should identify the filename ending with ".gif". I can ask assistant to do that. filename  $=$  ask(f'You are given some html code as follows: {state_after_interaction} What is the file ending with the extension .gif? You must directly output the full file name, including the extension.' agent.type(f'rem{filename}') state_after_interaction  $=$  agent.press_key('enter') assert 'not found' not in state_after_interaction, f'I cannot do [Step 2] correctly. The file ending with the extension .gif is not deleted. Current state: {state_after_interaction}'
The expert sample for the task tic-tac-toe:
<sample_tic-tac-toe> Prompt
Here is an example of solution.  
# Task: Playing as 'X', win a game of tic-tac-toe.  
def solution(agent, start_from=1):
The board layout and corresponding html xpath: top-left("//*[id='ttt-0']"), top-center("//*[id='ttt-1']"), top-right("//*[id='ttt-2']"), middle-left("//*[id='ttt-3']"), middle-center("//*[id='ttt-4']"), middle-right("//*[id='ttt-5']"), bottom-left("//*[id='ttt-6']"), bottom-center("//*[id='ttt-7']"), bottom-right("//*[id='ttt-8']"). Note that "mark-o" indicates the '0' placed on board, "mark-x" indicates the 'X' placed on board. # General plan: Currently, no grid is occupied. The plan is 1) put an 'X' in the middle-center("//*[id='ttt-4']"), 2) put an 'X' in the top-left("//*[id='ttt-0']"), 3) put an 'X' in the bottom-right("//*[id='ttt-8']"). place_to_put_X = ['4', '0', '8'] for idx, place_id in enumerate(place_to_put_X): print(f'[Step {idx}] put an X in {place_id}') # before interaction, I need to store the current state so that I can compare it with the state after interaction. state_before_interaction = agent.current_state state_after_interaction = agent.click_xpath(f"/*[id='ttt-{place_id}]") # if the current state does not change after interaction, that means I cannot put an 'X' in the desired location, and that location is already occupied and the plan will not work. assert state_before_interaction != state_after_interaction, f''I cannot do [Step {idx}] put an X in the "//*[id='ttt-{place_id}])", because it is occupied. I need to revise solution function according to the new plan. "" + ask(f''Playing as 'X', win a game of tic-tac-toe. The board layout and corresponding html xpath: top-left("//*[id='ttt-0']"), top-center("//*[id='ttt-1']"), top-right("//*[id='ttt-2']"), middle-left("//*[id='ttt-3']"), middle-center("//*[id='ttt-4']"), middle-right("//*[id='ttt-5']"), bottom-left("//*[id='ttt-6']"), botmum-center("//*[id='ttt-7']"), botmum-right("//*[id='ttt-8']").Note that "mark-o" indicates the '0' placed on board, "mark-x" indicates the 'X' placed on board. The game in progress is represented in html code: {agent.current_state} Report current board situation and generate a plan that the 'X' player should follow to continue this game. Use the format like "Currently, 'X' has been placed at <position>/<position> ("//*[id='ttt-x']") and '0' has been placed at <position>/<position> ("//*[id='ttt-x']"). Therefore, the plan is to: 1) put an 'X' in the <position>/<position> ("//*[id='ttt-x']") 2) put an 'X' in the ...""
Code Check. We use the same <code_check> prompt for MiniWoB++ tasks as ALFWorld.
Out-of-Plan Refinement. In the event of an assertion error, we use <refinement> to conduct the out-of-plan refinement. In this prompt, <basic_info> is replaced by the content of the <basic_info> prompt. The placeholder <solution_func> is replaced by the generated solution function, while <task> is replaced by the task description in natural language. Finally, <feedback> is replaced by the assertion error message returned by the solution function. Note that the <refinement> prompt used here differs slightly from the one used for ALFWorld. In this context, we include the previously generated solution function in the prompt, enabling the refiner to adjust the solution based on the error messages accordingly.
<refinement> Prompt
Task: <task>
You have generated code of solution() to solve the task: <solution_func>
However, you executed the solution() function and get an error message: [Error message] <feedback>
Let's think step by step. You must output the revised solution function based on the error message. You must only complete the revised solution function without any other words.
In the solution function, start with a commented "# General plan: ". For each step you plan to take, mark with '[Step xx]', and write an assertion to check if the step is successful.
Determining start_from. We use the same <start_from> prompt for MiniWoB++ tasks as ALFWorld.
E Hallucination with GPT-3.5 Model (gpt-3.5-turbo)
We present the trajectories of three tested methods (ReAct, Reflexion, and AdaPlanner) for two tasks. Here, each model utilizes gpt-3.5-turbo as the backend.
E.1 Case 1: examine the book with the desklamp.
ReAct's trajectory for this case is detailed below. Despite the presence of two expert demonstrations, ReAct encounters significant difficulties, as it extensively hallucinates inadmissible actions or scenarios. For example, search for a book titled "The Great Gatsby" and ask household members if they know where the book is are not feasible within the given environment.
ReAct Trajectory of Case 1 (Hallucinated actions are marked in pink.)
You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1. Your task is to: examine the book with the desklamp. > think: To solve the task, I need to find and take a book, then find and use a desklamp. First, I need to find a book. A book is more likely to appear in desk (1-2), shelf (1-6), drawer (1-6). I can check one by one, starting with desk 1. OK. > go to desk 1 On the desk 1, you see a alarmclock 1, a bowl 2, a cd 1, a cellphone 3, a desklamp 1, a mug 1, a pen 1, and a pencil 1. > take book from desk 1 Nothing happens. > think: I need to be more specific about which book I am looking for. Let me try looking for a specific title or author. OK. > search for a book titled "The Great Gatsby" Nothing happens. > think: Let me try looking for a book by a specific author. How about "To Kill a Mockingbird" by Harper Lee? OK. > search for a book by Harper Lee Nothing happens. > think: Maybe I should try looking in a different location. Let me check shelf 1. OK. > go to shelf 1 On the shelf 1, you see a bowl 1. > go to shelf 2 On the shelf 2, you see a creditcard 1, and a pencil 2. > go to shelf 3 On the shelf 3, you see a alarmclock 1, a bowl 2, and a desklamp 1. > take book from shelf 3 Nothing happens. > think: Let me try looking in a different location. How about drawer 1? OK. > go to drawer 1 The drawer 1 is closed. > open drawer 1 You open the drawer 1. The drawer 1 is open. In it, you see nothing. > go to drawer 2 On the drawer 2, you see a cellphone 2. > go to drawer 3
Nothing happens.  
> go to drawer 4  
The drawer 4 is closed.  
> open drawer 4  
You open the drawer 4. The drawer 4 is open. In it, you see nothing.  
> go to drawer 5  
On the drawer 5, you see nothing.  
> go to drawer 6  
The drawer 6 is closed.  
> open drawer 6  
You open the drawer 6. The drawer 6 is open. In it, you see nothing.  
> think: Let me try looking in a different location. How about desk 2? OK.  
> go to desk 2  
On the desk 2, you see a alarmclock 3, a bowl 3, a cd 3, a mug 2, and a pencil 3.  
> take book from desk 2  
Nothing happens.  
> think: Let me try looking in a different location. How about shelf 4? OK.  
> go to shelf 4  
On the shelf 4, you see a alarmclock 2.  
> think: Let me try looking in a different location. How about shelf 5? OK.  
> go to shelf 5  
On the shelf 5, you see nothing.  
> think: Let me try looking in a different location. How about shelf 6? OK.  
> go to shelf 6  
On the shelf 6, you see nothing.  
> think: It seems like the book I am looking for is not in any of the locations I have checked so far. Let me ask the household members if they know where it is. OK.  
> ask household members if they know where the book is Nothing happens.  
> think: Maybe I should try asking a specific household member. How about asking John? OK.  
> ask John if he knows where the book is Nothing happens.  
> think: It seems like I am unable to find the book. I may need to ask for more information or try a different approach to the task. OK.  
> ask the household members if they have any suggestions on where to look for book Nothing happens.  
> think: It looks like I may need to give up on this task for now and come back to it later with more information or a different approach. OK.  
> end task Nothing happens.  
> exit room Nothing happens.  
> end interaction Nothing happens.  
> leave the household Nothing happens.  
> end interaction and try again later Nothing happens.  
> end interaction and come back later with a new approach Nothing happens.  
> end interaction and move on to a different task or activity Nothing happens.  
> end interaction and leave the room/household/location Nothing happens.  
> end interaction and try again later with a different approach or more information Nothing happens.  
> end interaction and seek help from someone outside of the household
> end interaction and try to find book in a different room or location in the house Nothing happens.
The trajectory of Reflexion, after two rounds of iterative trials and errors, is detailed below. Despite multiple closed-loop corrections, Reflexion fails to formulate a feasible plan. Instead, it persists in generating unfeasible actions in its reflection part. For instance, it plans to ask the household members for help, or to look for a ladder or step stool outside of the room. These actions fall outside the scope of admissible actions in the ALFWorld environment.
Reflexion Trajectory of Case 1 (Hallucinated actions are marked in pink.)
Your memory for the task below:
Trial 0:
In this environment, my plan was to find a book then find and use a desklamp. However, I spent too much time checking every possible location without thoroughly examining each one. I should have focused on the most likely locations for a book (desk, shelf, drawer) and thoroughly searched each one before moving on to the next. Additionally, I should have asked the household members for help earlier in the process. In the next trial, I will prioritize searching the most likely locations and ask for help if I am unable to find the book after a thorough search.
Trial 1:
In the next trial, I will prioritize searching the most likely locations for a book (desk, shelf, drawer) and thoroughly examine each one before moving on to the next. If I am unable to find the book after a thorough search, I will ask the household members for help. If the book is out of reach, I will look for a ladder or step stool outside of the room that I can bring in to reach it.
Here is the task:
You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1.
Your task is to: examine the book with the desklamp.
> think: To solve the task, I need to find and take a book, then find and use a desklamp. First, I need to find a book. A book is more likely to appear in desk (1-2), shelf (1-6), drawer (1-6). I can check one by one, starting with desk 1.
> go to desk 1
On the desk 1, you see a alarmclock 1, a bowl 2, a cd 1, a cellphone 3, a desklamp 1, a mug 1, a pen 1, and a pencil 1.
> examine books on desk 1
> examine shelf 1
> go to shelf 2
On the shelf 2, you see a creditcard 1, and a pencil 2.
> go to shelf 3
On the shelf 3, you see a alarmclock 1, a bowl 2, and a desklamp 1.
> take book from shelf 3
> examine book on shelf 3
> ask household member for help
> think: I need to keep searching for the book. It could be in desk 2, shelf 4-6, or drawer 1-6. I will check drawer 1 next.
> go to drawer 1
The drawer 1 is closed.
>open drawer 1
You open the drawer 1. The drawer 1 is open. In it, you see nothing.
> go to drawer 2
On the drawer 2, you see a cellphone 2.
> take book from drawer 2
> examine book in drawer 2
Nothing happens. > think: The book is not in drawer 2. I will check drawer 3 next. OK. > go to drawer 3 Nothing happens. > examine books in drawer 3 Nothing happens. > think: The book is not in drawer 3. I will check drawer 4 next. OK. > go to drawer 4 The drawer 4 is closed. > open drawer 4 You open the drawer 4. The drawer 4 is open. In it, you see nothing. > think: The book is not in drawer 4. I will check drawer 5 next. OK. > go to drawer 5 On the drawer 5, you see nothing. > think: The book is not in drawer 5. I will check drawer 6 next. OK. > go to drawer 6 The drawer 6 is closed. > open drawer 6 You open the drawer 6. The drawer 6 is open. In it, you see nothing. > think: The book is not in drawer 6. I will check desk 2 next. OK. > go to desk 2 On the desk 2, you see a alarmclock 3, a bowl 3, a cd 3, a mug 2, and a pencil 3. > think: The book is not on desk 2. I will check shelf 4 next. OK. > go to shelf 4 On the shelf 4, you see a alarmclock 2. > think: The book is not on shelf 4. I will check shelf 5 next. OK. > go to shelf 5 On the shelf 5, you see nothing. > think: The book is not on shelf 5. I will check shelf 6 next. OK. > go to shelf 6 On the shelf 6, you see nothing. > think: The book is not on shelf 6. I have searched all the most likely locations for a book and have not found it. I will ask the household members for help. OK. > ask household member for help Nothing happens. > think: The household member did not respond. I will look for a ladder or step stool outside of the room that I can bring in to reach higher shelves or areas. OK. > go outside of the room Nothing happens. > look for a ladder or step stool Nothing happens. > think: I was unable to find a ladder or step stool. I will continue searching the room for the book and ask the household members for help again if necessary. OK. > go to desk 1 On the desk 1, you see a alarmclock 1, a bowl 2, a cd 1, a cellphone 3, a desktop lamp 1, a mug 1, a pen 1, and a pencil 1. > examine books on desk 1 Nothing happens. > think: I have searched the room thoroughly and have not found the book. I will ask the household members for help again. OK. > ask household member for help Nothing happens. > think: The household member is not responding. I will try to think of any other possible locations for the book and search those areas. OK. > think: The book could be in the laundry hamper or safe. I will check those areas next. OK.
> go to laundry hamper 1  
Nothing happens.  
> examine laundry hamper  
Nothing happens.  
> go to safe 1  
The safe 1 is closed.  
> try to open safe 1  
Nothing happens.
The trajectory of AdaPlanner is detailed below. For better readability, we only present the major sub-goals generated by the planner. Evidently, AdaPlanner successfully completes this task without the need for any revisions. This indicates that the code interface can effectively mitigate the hallucination in gpt-3.5-turbo.
AdaPlanner Trajectory of Case 1
You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1. Your task is to: examine the book with the desklamp. [Step 1] get a list of receptacles where a book is likely to appear. [Step 2] go to each receptacle in the list until seeing a book > go to shelf 6 On the shelf 6, you see nothing. > go to shelf 5 On the shelf 5, you see nothing. > go to shelf 4 On the shelf 4, you see a mug 2, and a pencil 1. > go to shelf 3 On the shelf 3, you see a creditcard 1, a desklamp 1, and a mug 1. > go to shelf 2 On the shelf 2, you see a pen 1. > go to shelf 1 On the shelf 1, you see a bowl 1. > go to desk 2 On the desk 2, you see a cellphone 2, a creditcard 2, a mug 3, and a pencil 2. > go to desk 1 On the desk 1, you see a alarmclock 1, a creditcard 1, a desklamp 1, a laptop 3, and a mug 1. > go to bed 1 On the bed 1, you see a book 1, a cellphone 1, a laptop 2, a laptop 1, a pillow 2, and a pillow 1. [Step 3] take the book from the receptacle > take book 1 from bed 1 You pick up the book 1 from the bed 1. [Step 4] get a list of receptacles where a desklamp is likely to appear. [Step 5] go to each receptacle in the list until seeing a desklamp > go to desk 1 On the desk 1, you see a alarmclock 1, a creditcard 1, a desklamp 1, a laptop 3, and a mug 1. [Step 6] turn on desklamp > use desklamp 1 You turn on the desklamp 1.
E.2 Case 2: put some watch on safe.
In the second trajectory, ReAct also hallucinates infeasible actions (e.g., look for a manual or instructions and ask someone in the household), ultimately leading to task failure.
ReAct Trajectory of Case 2 (Hallucinated actions are marked in pink.)
You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1.
> think: To solve the task, I need to find and take a watch, then put it on the safe. OK.  
> think: First I need to find a watch. A watch is more likely to appear in a drawer (1-5), a shelf (1-12), a cabinet (1-4), a dresser (1), a sidetable (1), or a sofa (1). I can check one by one, starting with drawer 1. OK.  
> go to drawer 1  
The drawer 1 is closed.  
> open drawer 1  
You open the drawer 1. The drawer 1 is open. In it, you see nothing.  
> go to drawer 2  
The drawer 2 is closed.  
> open drawer 2  
You open the drawer 2. The drawer 2 is open. In it, you see nothing.  
> go to drawer 3  
The drawer 3 is closed.  
> open drawer 3  
You open the drawer 3. The drawer 3 is open. In it, you see a creditcard 2.  
> go to drawer 4  
The drawer 4 is closed.  
> open drawer 4  
You open the drawer 4. The drawer 4 is open. In it, you see nothing.  
> go to drawer 5  
The drawer 5 is closed.  
> open drawer 5  
You open the drawer 5. The drawer 5 is open. In it, you see nothing.  
> go to shelf 1  
On the shelf 1, you see nothing.  
> go to shelf 2  
On the shelf 2, you see nothing.  
> go to shelf 3  
Nothing happens.  
> go to shelf 4  
On the shelf 4, you see nothing.  
> go to shelf 5  
On the shelf 5, you see nothing.  
> go to shelf 6  
On the shelf 6, you see nothing.  
> go to shelf 7  
On the shelf 7, you see a newspaper 3.  
> go to shelf 8  
On the shelf 8, you see nothing.  
> go to shelf 9  
On the shelf 9, you see nothing.  
> go to shelf 10  
On the shelf 10, you see nothing.  
> go to shelf 11  
On the shelf 11, you see nothing.  
> go to shelf 12  
On the shelf 12, you see nothing.  
> go to cabinet 1  
The cabinet 1 is closed.  
> open cabinet 1  
You open the cabinet 1. The cabinet 1 is open. In it, you see a vase 1.  
> go to cabinet 2  
Nothing happens.  
> go to cabinet 3  
The cabinet 3 is closed.  
> open cabinet 3  
You open the cabinet 3. The cabinet 3 is open. In it, you see a tissuebox 2.  
> go to cabinet 4  
Nothing happens.  
> go to dresser 1  
On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1.  
> take watch 1 from dresser 1  
You pick up the watch 1 from the dresser 1.  
> think: Now I have a watch (1). Next, I need to put it on the safe. OK.  
> go to safe 1
The safe 1 is closed.   
> open safe 1   
You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2.   
> put watch 1 on safe 1   
Nothing happens.   
> think: I need to be more specific about where to put the watch. Maybe on top of the cellphone 3? OK.   
> put watch 1 on cellphone 3 in safe 1   
Nothing happens.   
> think: Maybe I need to place the watch inside the safe instead of on top of something inside it. OK.   
> put watch 1 in safe 1   
Nothing happens.   
> think: Maybe I need to enter a code to properly open the safe and put the watch inside. OK.   
> look for a code or key to open the safe   
Nothing happens.   
> think: Maybe I need to ask someone in the household for the code or key to open the safe. OK.   
> ask someone in the household for the code or key to open the safe Nothing happens.   
> think: Maybe I need to search for a manual or instructions on how to open the safe. OK.   
> look for a manual or instructions on how to open the safe   
Nothing happens.   
> think: Maybe I need to try a combination of numbers to open the safe. OK.   
> try the combination 1234 on the safe   
Nothing happens.   
> think: Maybe I need to try a different combination of numbers to open the safe. OK.
Similarly, Reflexion experiences issues with hallucination during this task, such as seek additional resources or ask for help. This contributes to the eventual failure of the task.
Reflexion Trajectory of Case 2 (Hallucinated actions are marked in pink.)
Your memory for the task below:  
Trial 0:  
In this environment, my plan was to find a watch and put it on the safe. However, I did not consider that the safe may require a code or key to lock it. I should have looked for a code or lock mechanism on the safe or a key to the safe before attempting to put the watch inside. When I could not find a way to secure the watch inside the safe, I should have looked for another container or safe in the room. In the next trial, I will first look for a code or lock mechanism on the safe or a key to the safe. If I cannot find one, I will look for another container or safe in the room.  
Trial 1:  
To solve the task, I need to find a watch and a way to secure it inside the safe. First, I will look for a watch in drawers (1-5), cabinets (1-4), dressers (1), shelves (1-12), sidetables (1), and sofas (1). Once I find a watch, I will examine the safe for any clues or look for a key in other locations. If I cannot find a way to secure the watch inside the safe, I will look for another container or safe in the room. If I still cannot find a way to secure the watch, I will seek additional resources or ask for help.  
Here is the task:  
You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1.  
Your task is to: put some watch on safe.