SlowGuess/NeurIPS_2023_OCR huggingface dataset:
> think: To solve the task, I need to find a watch and a way to secure it inside the safe. First, I will look for a watch in drawers (1-5), cabinets (1-4), dressers (1), shelves (1-12), sidetables (1), and sofas (1). OK.  
> go to drawer 1  
The drawer 1 is closed.  
> open drawer 1  
You open the drawer 1. The drawer 1 is open. In it, you see nothing.  
> go to drawer 2  
The drawer 2 is closed.  
> open drawer 2  
You open the drawer 2. The drawer 2 is open. In it, you see nothing.  
> go to drawer 3  
The drawer 3 is closed.  
> open drawer 3  
You open the drawer 3. The drawer 3 is open. In it, you see a creditcard 2.  
> go to drawer 4  
The drawer 4 is closed.  
> open drawer 4  
You open the drawer 4. The drawer 4 is open. In it, you see nothing.  
> go to drawer 5  
The drawer 5 is closed.  
> open drawer 5  
You open the drawer 5. The drawer 5 is open. In it, you see nothing.  
> go to cabinet 1  
The cabinet 1 is closed.  
> open cabinet 1  
You open the cabinet 1. The cabinet 1 is open. In it, you see a vase 1.  
> go to cabinet 2  
Nothing happens.  
> think: The cabinet 2 may be locked or empty. I should try another location. OK.  
> go to cabinet 3  
The cabinet 3 is closed.  
> open cabinet 3  
You open the cabinet 3. The cabinet 3 is open. In it, you see a tissuebox 2.  
> go to cabinet 4  
Nothing happens.  
> think: The cabinet 4 may be locked or empty. I should try another location. OK.  
> go to dresser 1  
On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 2, and a watch 1.  
> think: Now I find a watch (1). Next, I need to examine the safe for any clues or look for a key in other locations. OK.  
> go to safe 1  
The safe 1 is closed.  
> think: Before attempting to put the watch inside the safe, I should look for a code or lock mechanism on the safe or a key to the safe. OK.  
> examine safe 1  
The safe 1 is closed.  
> look for key or code on safe 1  
Nothing happens.  
> think: I cannot find a key or code on the safe. I should look for another container or safe in the room. OK.  
> look for another container or safe in the room  
Nothing happens.  
> think: I still cannot find another container or safe in the room. I should seek additional resources or ask for help. OK.  
> seek additional resources or ask for help  
Nothing happens.  
> think: I may need to think of a different solution to the task. Perhaps I can place the watch near the safe instead of inside it. OK.  
> put watch 1 near safe 1  
Nothing happens.
> think: The task is incomplete. I need to put the watch on the safe. I should try to find a way to secure the watch on top of the safe or find a container to put the watch in that can be placed on top of the safe.
> look for a container to put the watch in that can be placed on top of the safe Nothing happens.
> think: I should try to find a way to secure the watch on top of the safe. OK.
> look for a way to secure the watch on top of the safe
> think: I may need to think of a different solution to the task. Perhaps I can find a way to attach the watch to the safe using tape or string. OK.
> look for tape or string
> think: I cannot find tape or string. I should seek additional resources or ask for help.
> seek additional resources or ask for help
> think: The task is incomplete. I need to put the watch on the safe. I should try to find a way to secure the watch on top of the safe or find a container to put the watch in that can be placed on top of the safe. Since I cannot find any resources or help, I will have to abandon the task for now and come back to it later with new ideas or resources.
> end task
> exit room
> think: I cannot exit the room until the task is complete or I have abandoned it.  
I will continue to look for a solution to the task. OK.
> continue searching for a solution to the task
> think: I have exhausted all possible options and resources in the room. I should take a break and come back to the task later with a fresh perspective or seek help from someone else.
> think: I cannot exit the room until the task is complete or I have abandoned it. I will take a break and come back to the task later with a fresh perspective or seek help from someone else.
Initially, AdaPlanner also experiences hallucination, presuming that the watch is located in the safe. This incorrect assumption triggers an assertion error. AdaPlanner then revises the plan to better adapt to the actual environment. Following this refinement, the task is successfully completed.
AdaPlanner Trajectory of Case 2 (Hallucinated actions are marked in pink.)
You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1.
Your task is to: put some watch on safe.
[Step 1] get a list of receptacles where the safe is likely to appear
[Step 2] go to each receptacle in the list until seeing a safe  
> go to safe 1  
The safe 1 is closed.  
> open safe 1  
You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2.  
[Step 3] identify the watch I just found and take it  
> take watch 1 from safe 1  
Nothing happens.  
[Step 4] go to the safe and put the watch in it  
> go to safe 1  
Nothing happens.  
> put watch 1 in/on safe 1  
Nothing happens.  
AssertionError: Error in [Step 4]: I cannot put the watch 1 in the safe 1. The last three interactions before error were:  
> take watch 1 from safe 1  
Nothing happens.  
> go to safe 1  
Nothing happens.  
> put watch 1 in/on safe 1  
Nothing happens.  
I am at safe 1 and holding None.  
Refinement is done with start_from = 1.  
[Step 1] get a list of receptacles where the watch is likely to appear  
[Step 2] go to each receptacle in the list until seeing a watch  
> go to safe 1  
Nothing happens.  
> go to cabinet 1  
The cabinet 1 is closed.  
> open cabinet 1  
You open the cabinet 1. The cabinet 1 is open. In it, you see a vase 1.  
> go to cabinet 2  
Nothing happens.  
> go to cabinet 3  
The cabinet 3 is closed.  
> open cabinet 3  
You open the cabinet 3. The cabinet 3 is open. In it, you see a tissuebox 2.  
> go to cabinet 4  
Nothing happens.  
> go to dresser 1  
On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1.  
[Step 3] identify the watch I just found and take it  
> take watch 1 from dresser 1  
You pick up the watch 1 from the dresser 1.  
[Step 4] go to the safe and put the watch in it  
> go to safe 1  
The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2.  
> put watch 1 in/on safe 1  
You put the watch 1 in/on the safe 1.
F Detailed Evaluation Results on MiniWoB++
Task Name	AdaPlanner	CC-Net [7]	WGE [11]	WebN-T5-3B [4]	RCI [8]
choose-list	100	99	16	26	100
click-button	100	100	100	100	100
click-button-sequence	100	100	100	100	100
click-checkboxes	100	98	100	96	100
click-checkboxes-large	100	71	84	22	94
click-checkboxes-soft	80	95	94	54	72
click-checkboxes-transfer	98	99	64	63	100
Continued on next page.
click-collapsible	100	100	100	0	100
click-collapsible-2	84	98	99	0	62
click-color	100	100	100	27	100
click-dialog	100	100	100	100	100
click-dialog-2	100	100	100	24	100
click-link	98	99	100	100	100
click-menu	78	94	n/a	37	100
click-option	100	99	100	87	100
click-scroll-list	100	60	n/a	0	100
click-shades	100	100	99	0	100
click-shape	75	95	64	53	98
click-tab	100	100	100	74	100
click-tab-2	85	98	98	18	74
click-tab-2-hard	78	98	n/a	12	76
click-test	100	100	100	100	100
click-test-2	100	100	100	100	100
click-widget	100	100	93	100	98
count-shape	50	85	76	41	40
email-inbox	98	100	99	38	98
email-inbox-forward-nl	100	100	n/a	60	100
email-inbox-forward-nl-turk	100	100	n/a	33	94
email-inbox-nl-turk	90	100	93	23	98
enter-date	100	100	96	0	96
enter-password	98	100	100	97	100
enter-text	98	100	100	89	100
enter-text-dynamic	96	100	100	98	100
enter-time	96	97	90	0	100
focus-text	100	100	100	100	100
focus-text-2	94	100	100	100	100
grid-coordinate	100	100	100	49	100
guess-number	88	100	0	0	20
identify-shape	96	100	100	88	76
login-user	100	100	100	82	100
login-user-popup	98	100	n/a	72	68
multi-layouts	84	100	100	83	72
multi-orderings	100	100	100	88	100
navigate-tree	82	99	99	91	86
search-engine	100	100	99	34	100
simple-algebra	82	75	n/a	n/a	100
social-media	82	90	100	21	98
social-media-all	100	75	1	0	100
social-media-some	90	85	42	2	90
terminal	98	0	n/a	n/a	100
tic-tac-toe	48	83	47	48	56
use-autocomplete	88	100	98	22	58
use-spinner	90	100	4	7	88
Table 8: Per-task success rate (\%) of AdaPlanner, CC-Net [7], WGE [11], WebN-T5-3B [4], and RCI [8]. "n/a" signifies that the corresponding success rate is not reported in the original paper of the method. The nine tasks with feedback are marked in gray.