SlowGuess/NeurIPS_2023_OCR huggingface dataset:
print("[Step 4] go to a cabinet and put the first cellphone found on it. ") # There are multiple countertops, and I only need to go to one of them. observation  $=$  agent.goto('cabinet 1') # check if the cabinet is closed. If so, open it. if 'closed' in observation: observation  $=$  agent.open_receptacle('cabinet 1') observation  $=$  agent.put(found_cellphone1,'cabinet 1') # expectation: I should be able to put the cellphone on the countertop. assert f'You put the {found_cellphone1} in/on the cabinet 1.' in observation, f'Error in [Step 4]: I cannot put the {found_cellphone1} on the cabinet 1.{agent.report()}'   
if start_from  $\leqslant$  5: print("[Step 5] go to each of the remaining receptacle in the list until seeing a second cellphone") for receptacle in recep_to_check: observation  $=$  agent.goto(receptacle) # check if the receptacle is closed. If so, open it. if 'closed' in observation: observation  $=$  agent.open_receptacle(receptacle) # check if a cellphone is in/on the receptacle. if 'cellphone' in observation: break # expectation: I should be able to find a receptacle where a cellphone is in/on it. assert 'cellphone' in observation, f'Error in [Step 5]: There is no second cellphone in/on{recep_to_check}. {agent.report()}'   
if start_from  $\leqslant$  6: print("[Step 6] identify the second cellphone I just found and take it") # I need to get the identifier of the cellphone. I can ask the assistant to do that. answer  $=$  ask(f'From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation}. The identifier of the cellphone? Only Output a single number without any other words. ') found_cellphone2  $=$  f'cellphone {answer}' observation  $=$  agent.take(found_cellphone2,receptacle) # expectation: I should be able to take the cellphone from the receptacle. assert agent.holding  $\equiv$  found_cellphone2,f'Error in [Step 6]: I cannot take {found_cellphone2} from the {receptacle}. {agent.report()}'   
if start_from  $\leqslant$  7: print("[Step 7] go to a cabinet and put the second cellphone found on it") observation  $=$  agent.goto('cabinet 1') observation  $=$  agent.put(found_cellphone2,'cabinet 1') # expectation: I should be able to put the cellphone2 on the countertop. assert f'You put the {found_cellphone2} in/on the cabinet 1.' in observation, f'Error in [Step 7]: I cannot put the {found_cellphone2} on the cabinet 1.{agent.report()}'
Code Check. After plan generation, we employ the following prompt to verify and rectify any syntax errors. The placeholder <solution_func> is replaced by the generated solution function. The <code_check> prompt prompts the model to return two questions. If the response to Question 1 is Yes, the answer to Question 2 is adopted as the corrected solution function. Otherwise, the solution function is kept unchanged.
<code_check> Prompt
You are given a Python code snippet to define a function called solution.
[Code]
<solution_func>
Question 1: Are there any syntax errors present in the code? Answer Yes/No.  
Question 2: Fix the syntax errors and output an error-free version of the code.  
Only Output the revised code after [Revised code] without any other words.
Out-of-Plan Refinement. In the event of an assertion error, we use <refinement> to conduct the out-of-plan refinement. In this prompt, <basic_info> is replaced by the content of the <basic_info> prompt. The placeholder <sample> is substituted with an expert trajectory, while <receptacle_list> is replaced by the list of interactive receptacles provided by the task environment. <task> is replaced by the task description in natural language. Finally, <error msg> is replaced by the assertion error message returned by the solution function. To adhere to the context length limit of the GPT-3/3.5 models, the previously generated solution function is not included in this prompt. Instead, we incorporate comprehensive information in the assertion error message, enabling the refiner to generate a revised plan based on these details.
<refinement> Prompt
<basic_info>
# Here is a example of successful solution for solving a similar task:
[Successful example]
receptacles = ['diningtable 1', 'hunter 2', 'hunter 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent = Agent(receptacles)
<sample>
# Here is the actual task.
# define environment and agent
receptacles = <receptacle_list>
agent = Agent(receptacles)
#
You have generated code of solution() to solve the task. However, you executed the solution() function and get an error message:
<error msg>
Let's think step by step. Referring to the successful case and the error message, you should complete the solution function with the correct code. def solution(agent, start_from=1):
Determining start_from. After formulating a revised plan, we utilize the following prompt to ascertain from which step the new solution function should commence. In this context, the <previous_solution> is replaced by the preceding solution function, while the <revised_solution> is replaced by the updated one. Subsequently, the argument start_from=1 is substituted with the step number that this prompt yields.
Previously, you generated some code defining a solution function as in [Previous solution]. The previous code is executed and outputs some error. Now you just revised the code as in [Revised solution]. Determine from which step these two versions differ. You should only output the step number without saying any other words.
<start_from> Prompt
[Previous solution] <previous_solution> [Revised solution] <revised_solution>
D.2 MiniWoB++
Basic Information. Similar to the ALFWorld tasks, the <basic_info> of MiniWoB++ defines the agent and admissible actions for AdaPlanner. Note that the actual definitions of action functions are not specified in the prompt. Instead, only a formalized definition with several examples is provided, such that the planner can acquire how to compose a plan based on these actions. As can be seen in the following part, this <basic_info> prompt is used in both <initial_planning> and <refinement> prompts.
Interact with the HTML webpage to finish the computer task. Here is some Python code defining a computer task environment:
<basic_info> Prompt
In the environment, you can ask questions to an assistant by ask(): from large_language_model import ask_11m as ask # for example: You want to solve a algebra problem  $x + 3 = 6$ . You can ask the assistant to solve it directly for you. answer = ask('Solve an algebra problem. You should directly output the value of the unknown. For example, solve 'y + 1 = 2' -> 1. Now solve 'x + 3 = 6' ->') # answer = '3'
Agent class represents the current state of the HTML webpage and the actions it can take.   
class Agent: def__init__(self,initial_state): self.current_state  $=$  initial_state #Here are the admissible actions the agent can take: #Action: type a string into the input box #this function returns the string of the HTML code after taking the action #e.g.,new_html_state  $\equiv$  agent.type("hello") deftype(self,characters:str）->str:
Action: press a key on the keyboard, the input can be one of the following:  
# enter, space, arrow_left, arrow_right, arrow_up, arrow_down, backspace  
# this function returns the string of the HTML code after taking the action  
# e.g., new_html_state = agent.press_key("enter")  
def press_key(self, key: str) -> str:
Action: click a <select> element in a list with an XPath  
# this function returns the string of the HTML code after taking the action  
# e.g., new_html_state = agent.click_option("/select [@id='cars']/option[1]")  
def click_option(self, xpath_of_option: str) -> str:
# Action: click an HTML element with its XPath
# this function returns the string of the HTML code after taking the action
# e.g., new_html_state = agent.click_xpath("/button [@id='button1']")
def click_xpath(self, xpath_of_element: str) -> str:
    ...
Action: move the mouse cursor on an HTML element with an XPath
# this function returns the string of the HTML code after taking the action
# e.g., new_html_state = agent.move Mouse_on("/button [@id='button1']")
def move Mouse_on(self, xpath_of_element: str) -> str:
    ...
Initial Planning. The <initial_planning> prompt is employed to generate the preliminary plan. In this context, <basic_info> is substituted by the content of the <basic_info> prompt. The <sample> is replaced with an expert trajectory (or discarded in the skill discovery stage). The <initial_state> is substituted by the initial HTML code of the task UI. Finally, <task> is substituted by the task description, expressed in natural language.
<initial_planning> Prompt
<basic_info>   
<sample>   
# Now complete the function solution() below to solve the task by composing the agent's methods to interact with the environment. # In the solution function, start with a commented "# General plan: ". For each step you plan to take, mark with '[Step xx]', and write an assertion to check if the step is successful.   
# Here is the actual task. # define environment and agent. The state is the list of HTML elements of the webpage. initial_state \''' <initial_state>   
\''' agent = Agent(initial_state)   
# Task: <task> # Here is the solution: def solution(agent, start_from=1):
Samples. In MiniWoB++, we select nine tasks where feedback is available. Here are the expert samples that we gathered for these tasks.
The expert sample for the task email-inbox:
<sample_email-inbox> Prompt
Here are three examples of solutions.   
# Task: Find the email by Brittani and reply to them with the text "Aliquet. Sollicitudin nam lectus"." def solution(agent, start_from  $= 1$  .   
# General plan: I should first click on the email by Brittani, then click on the reply button, then type the text "Aliquet. Sollicitudin nam lectus." and finally click on the send button. if start_from  $\leq 1$  .. print('[Step 1] click on the email by Brittani') agentclick_xpath("/div[@class='email- sender' and text()='Brittani']") state_after_interaction  $=$  agentclick_xpath("/span [@class='email-reply']") # the reply content should be displayed on page. assert 'reply-text' in state_after_interaction, 'I cannot do [Step 1] correctly. The reply button is not displayed on the page.' if start_from  $\leq 2$  .. print('[Step 2] type the text "Aliquet. Sollicitudin nam lectus．") agentclick_xpath("/textarea [@id='reply-text']") agent.type('Aliquet.Sollicitudin nam lectus.') state_after_interaction  $=$  agent.click_xpath("/*[ @id='send-reply']")   
# Task: Find the email by Blanca and forward that email to Agathe.   
def solution(agent, start_from  $= 1$  .   
# General plan: I should first click on the email by Blanca, then click on the forward button, then type "Agathe" and finally click on the send button.. if start_from  $\leq 1$  .. print('[Step 1] click on the email by Blanca') agentclick_xpath("/div [@class='email- sender' and text()='Blanca']") state_after_interaction  $=$  agentclick_xpath("/span [@class='email-forward']") # the forward content should be displayed on page. assert 'forward-sender' in state_after_interaction, 'I cannot do [Step 1] correctly. The forward button is not displayed on the page.'
if start_from <= 2: print(['Step 2] type "Agathe") agent.click_xpath("/input@class='forward-sender'")] agent.type('Agathe') state_after_interaction = agent.click_xpath("/*[@id='send-forward']") # Task: Find the email by Salli and click the trash icon to delete it. def solution(agent, start_from=1): # General plan: I should first click on the email by Salli, then click on the trash icon. if start_from <= 1: print(['Step 1] click on the email by Salli) agent.click_xpath("/div[#class='email-sender' and text()='Salli']") agent.click_xpath("/span[#class='trash']")
The expert sample for the task email-inbox-forward-n1:
<sample_email-inbox-forward-n1> Prompt
Here is an example of solution.
task: Send Alice the email from Beth / navigate to the message from Beth and send it to Alice.
General plan: I should first click on the email from Beth, then click on the "Forward" button, then type "Alice" in the "To" inputbox, finally click on the "Send" button.
if start_from <= 1:  
    print(['Step 1] click on the email from Beth')  
    agent.click_xpath('//*[@class="email-sender" and text()="Beth"]}  
    state_after_interaction =  
        agent.click_xpath('///span [@class="email-forward"]}  
    # the "To" inputbox should be displayed on page.  
    assert 'forward-sender' in state_after_interaction, f'I cannot do [Step 1] correctly. The "To" inputbox is not displayed on the page. Current state: {state_after_interaction}'
if start_from <= 2:  
    print(['Step 2] type "Alice" in the "To" inputbox')  
    agent.click_xpath('/input [@class="forward-sender"]}')  
    agent.type('Alice')  
    state_after_interaction = agent.click_xpath('///span [@id="send-forward"]}')  
    # the email should be sent successfully.  
    assert 'email-sender' in state_after_interaction, f'I cannot do [Step 2] correctly. The email is not sent successfully. Current state: {state_after_interaction}'
The expert sample for the task email-inbox-forward-nl-turk:
<sample_email-inbox-forward-nl-turk> Prompt
task: Send Alice the email from Beth / navigate to the message from Beth and send it to Alice / I want to forward the email from Beth over to Alice def solution(agent, start_from=1):
<sample_email-inbox-n1-turk> Prompt
if start_from <= 2:  
    print(['Step 2] type "Alice" in the "To" inputbox')  
    agent.click_xpath('/input [@class="forward-sender"]}')  
    agent.type('Alice')  
    state_after_interaction = agent.click_xpath('///span [@id="send-forward"]}')  
# the email should be sent successfully.  
assert 'email-sender' in state_after_interaction, f'I cannot do [Step 2] correctly. The email is not sent successfully. Current state: {state_after_interaction}'
The expert sample for the task email-inbox-n1-turk:
Here are three examples of solution.
Task: "Aliquet. Sollicitudin nam lectus." is my reply to Brittani's most recent email / Find the email by Brittani and reply to them with the text "Aliquet. Sollicitudin nam lectus."
General plan: I should first click on the email by Brittani, then click on the reply button, then type the text "Aliquet. Sollicitudin nam lectus." and finally click on the send button.
if start_from <= 1:  
    print(['Step 1] click on the email by Brittani')  
    agent.click_xpath("/div@[class='email-sender' and text()='Brittani']")  
    state_after_interaction =  
        agent.click_xpath("/ span [@class='email-reply']")  
# the reply content should be displayed on page.  
assert 'reply-text' in state_after_interaction, 'I cannot do [Step 1] correctly. The reply button is not displayed on the page.'
if start_from <= 2:  
    print(['Step 2] type the text "Aliquet. Sollicitudin nam lectus.")  
    agent.click_xpath("/textarea [@id='reply-text']")  
    agent.type('Aliquet. Sollicitudin nam lectus.')  
    state_after_interaction = agent.click_xpath("/*[ [@id='send-reply']")
Task: Find the last email by Blanca and send it to Agathe / Find the email by Blanca and forward that email to Agathe. def solution(agent, start_from=1):
General plan: I should first click on the email by Blanca, then click on the forward button, then type "Agathe" and finally click on the send button.
if start_from <= 1:  
    print(['Step 1] click on the email by Blanca')  
    agent.click_xpath("//div [@class='email-sender' and text()='Blanca']")  
    state_after_interaction =  
        agent.click_xpath("/ span [@class='email-forward']")  
# the forward content should be displayed on page.  
assert 'forward-sender' in state_after_interaction, 'I cannot do [Step 1] correctly. The forward button is not displayed on the page.'
if start_from <= 2:  
    print(['Step 2] type "Agathe")  
    agent.click_xpath("/input [@class='forward-sender']")  
    agent.type('Agathe')  
    state_after_interaction = agent.click_xpath("/*[@id='send-forward']")
Task: Delete this email from Salli / Please find Salli's email in the inbox and delete it.
# General plan: I should first click on the email by Salli, then click on the trash icon.
if start_from <= 1:  
    print(['Step 1] click on the email by Salli')
The expert sample for the task guess-number:
<sample Guess-number> Prompt
Here is an example of solution.   
# Task: Guess the number between 0-9 and press Submit. Use the feedback below to find the right number.   
# Here is the solution:   
def solution(agent, start_from  $= 1$  .   
# General plan: Given a list of possibleNumbers, I will try the number in the middle of the list and get feedback from the html state.   
# Now the given list of possibleNumbers is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. if start_from  $\leqslant 1$  print('[Step 1] maintain a list of the possible numbers left and try the middle one') # before making a guess, I should store the html state for future comparison. state_before_interaction  $=$  agent.current_state # I will choose the number 5, which is in the middle of possibleNumbers. guess_number  $= 5$  # click the input box, type the number I guessed and click submit. agent.click_xpath("/input [@id='tt']") agent.press_key("backspace") agent.type(str(guess_number)) state_after_interaction  $=$  agent.click_xpath('//* [@id="subbtn"]') # after input and submit the guessed number, the html_state should be changed and contain the feedback. Otherwise this step is not successful. assert state_after_interaction!  $=$  state_before_interaction,'I did [Step 1] but the html state did not change.'   
if start_from  $\leqslant 2$  print('[Step 2] get the feedback information from the new html state') # If the guess is successful, the keyword "higher" or "lower" should not be present in the observation. Otherwise I should use assert to jump out to pass the feedback. observation  $=$  ask(f'Answer a question based on the html code below: {state_after_interaction} Question: Which one is displayed? "The number is lower than" or "The number is higher than"? You must only output the displayed sentence without saying any other words.' assert "higher" not in observation, f'You tried the number {guess_number} in [Step 1], and the correct number is greater than this number. I need to revise solution function according to the new plan: Now the given list of possibleNumbers is [6, 7, 8, 9]. ' assert "lower" not in observation, f'You tried the number {guess_number} in [Step 1], and the correct number is smaller than this number. I need to revise solution function according to the new plan: Now the given list of possibleNumbers is [0, 1, 2, 3, 4].
The expert sample for the task login-user-popup:
<sample_login-user-popup> Prompt
Here is an example of solution.   
# Task: Enter the username "kanesha" and the password "oT" into the text fields and press login.   
def solution(agent, start_from  $= 1$  ..   
# General plan: I should first click on the username field, then type in the username, then click on the password field, then type in the password, then click on the login button. if start_from  $\Leftarrow$  1: print('[Step 1] Click on the username field') state_after_interaction  $=$  agent.click_xpath("/input@id  $\equiv$  'username'\$"")
agent.click_xpath("/div[@class='email-sender' and text()='Salli']")  
agent.click_xpath("/span [@class='trash']")
during interaction, some pickups may appear. If so, I need to jump out to handle the pickups. assert 'pickup' not in state_after_interaction, f"After [Step 1], some pickups appeared, you need to close the pickup at the beginning of [Step 1]. The current html is: {state_after_interaction} You need to add some code at the beginning of [Step 1] to cancel the pickup before any other actions.'
if start_from  $<= 2$
print('[Step 2] Type in the username')  
state_after_interaction = agent.type('kanesha')  
# during interaction, some popups may appear. If so, I need to jump out to handle the popups.  
assert '.]oup' not in state_after_interaction, f"After [Step 2], some popups appeared, you need to close the popup at the beginning of [Step 2]. The current html is: {state_after_interaction} You need to add some code at the beginning of [Step 2] to cancel the popup before any other actions.'
if start_from  $<= 3$
print('[Step 3] Click on the password field')  
state_after_interaction = agent.click_xpath("/input [@id=\'password\']") # during interaction, some popups may appear. If so, I need to jump out to handle the popups.  
assert '.]oup' not in state_after_interaction, f'After [Step 3], some popups appeared, you need to close the popup at the beginning of [Step 3]. The current html is: {state_after_interaction} You need to add some code at the beginning of [Step 3] to cancel the popup before any other actions.'
if start_from  $<  = 4$  ..
print('[Step 4] Type in the password')  
state_after_interaction = agent.type('oT')  
# during interaction, some popups may appear. If so, I need to jump out to handle the popups.  
assert '.]oup' not in state_after_interaction, f"After [Step 4], some popups appeared, you need to close the popup at the beginning of [Step 4]. The current html is: {state_after_interaction} You need to add some code at the beginning of [Step 4] to cancel the popup before any other actions.'
if start_from  $<= 5$
print('[Step 5] Click on the login button')  
state_after_interaction = agent.click_xpath("/button [@id='subbtn']")  
# during interaction, some popups may appear. If so, I need to jump out to handle the popups.  
assert '.]oup' not in state_after_interaction, f"After [Step 5], some popups appeared, you need to close the popup at the beginning of [Step 5]. The current html is: {state_after_interaction} You need to add some code at the beginning of [Step 5] to cancel the popup before any other actions.'
The expert sample for the task search-engine:
<sample_search-engine> Prompt
Task: Use the textbox to enter "Alice" and press "Search", then find and click the 7th search result.
General plan: I should first click on the inputbox, then type "Alice", then click on the "Search" button, finally look through pages and click on the 7th result.
if start_from  $<= 1$
print('[Step 1] click on the inputbox and type "Alice"')
agent.click_xpath('//*[@id="search-text"]') agent.type('Alice') state_after_interaction  $=$  agent.click_xpath('//*[@id="search"]') # the search content should be displayed on page. assert 'search-desc' in state_after_interaction,  $\prime \mathrm{I}$  cannot do [Step 1] correctly. The search content is not displayed on the page.'   
if start_from <= 2: print(['Step 2] calculate the page number of the 7th result and click on the page') # I should count the number of results on each page, iteratively turn to next page until seeing the 7th result. # I can use the following code to count the number of results on each page. num_results_displayed_per_page  $=$  state_after_interaction.count('search-desc') num_page  $= (7 - 1)$  // num_results_displayed_per_page state_after_interaction  $=$  agent.click_xpath(f'//*[@id="pageagination"]/li[{3+num_page}]*/a') # I should click on the 7th result. num_to.Click  $= 7$  - num_results_displayed_per_page  $\ast$  num_page state_after_interaction  $=$  agent.click_xpath(f'//*[@id="page-content"]/div[{num_to.Click}]*/a')
The expert sample for the task terminal: