SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here are the admissible actions the agent can take:  
# Go to a receptacle and update the agent's location.  
# For example, 'On the countertop 1, you see a candle 1, a cloth 2, and a soapbar 1.' = goto('countertop 1')  
# For example, 'On the sidetable 2, you see nothing.' = goto('sidetable 2')  
def goto(self, receptacle):  
    ...  
# Take an object from a receptacle if the agent is not holding anything.  
# For example, 'You pick up the soapbar 1 from the towelholder 1.' = take('soapbar 1', 'towelholder 1')  
def take(self, object, receptacle):  
    ...  
# Put an object in or on a receptacle if the agent is holding it.  
# For example, 'You put the soapbar 1 in/on the cabinet 1.' = put('soapbar 1', 'cabinet 1')  
def put(self, object, receptacle):  
    ...  
# Open a receptacle and observe its contents.  
# For example, 'You open the cabinet 1. The cabinet 1 is open. In it, you see a cloth 1.' = open_receptacle('cabinet 1')  
def open_receptacle(self, receptacle):  
    ...  
# Clean an object with a receptacle.  
# For example, 'You clean the soapbar 1 using the sinkbasin 1.' = clean('soapbar 1', 'sinkbasin 1')  
def clean(self, object, receptacle):  
    ...  
# Heat an object with a receptacle.  
# For example, 'You heat the tomato 1 using the microwave 1.' = heat('tomato 1', 'microwave 1')  
def heat(self, object, receptacle):  
    ...  
# Cool an object with a receptacle.  
# For example, 'You cool the pan 2 using the fridge 1.' = cool('pan 2', 'fridge 1')  
def cool(self, object, receptacle):  
    ...  
# Turn on an object.  
# For example, 'You turn on the desklamp 1.' = turn_on('desklamp 1')  
def turn_on(self, object):  
    ...  
# Report agent's current state, including its location, what it's holding, and last action and observation.  
# This function should only be used in assertion.  
def report(self):
Initial Planning. The <initial_planning> prompt is employed to generate the preliminary plan. In this context, <basic_info> is substituted by the content of the <basic_info> prompt. The <sample> is replaced with an expert trajectory, while <receptacle_list> is substituted by the list of interactive receptacles provided by the task environment. Finally, <task> is substituted by the task description, expressed in natural language.
<initial_planning> Prompt
<basic_info>
Now complete the function solution() below to solve the task by composing the agent's methods to interact with the environment.
For each step you plan to take, 1) mark with '[Step xx)', 2) give a reason why you think it is a good step to take 3) write an assertion to check if the step is successful.
Here is an example of a solution to the task:
<sample>
Here is the actual task.
define environment and agent  
receptacles = <receptacle_list>  
agent = Agent(receptacles)
# <task>
You should complete your solution function below: def solution(agent, start_from=1):
Samples. In ALFWorld, there are six types of tasks: Pick, Clean, Heat, Cool, Examine, and Pick two. For each type, we gather one expert sample of solutions that the planner can refer to. These six expert samples are presented as follows:
The expert sample for the task Pick:
<sample_pick> Prompt
define environment and agent
receptacles = ['diningtable 1', ' drawer 2', ' drawer 1', ' sinkbasin 1', ' toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent = Agent(receptacles)
Your task is to: put soapbar on countertop.
here is a solution:
def solution(agent, start_from=1):
General Plan: I need to get a list of receptacles where the soapbar is likely to appear, and then go to each receptacle in the list until seeing a soapbar. Then I can put get the identifier of the soapbar and take it.
Finally I can go to the countertop and put the soapbar.
if start_from <= 1:  
    print("Step 1] get a list of receptacles where the soapbar is likely to appear")  
    # I can ask the assistant to do that.  
    answer = ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a soapbar in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.')  
    recep_to_check = literal_eval的回答)  
    # expectation: the returned recep_to_check should not be empty.  
    assert recep_to_check, f'Error in [Step 1]: recep_to_check should not be empty. {agent.report()}'
if start_from <= 2:  
    print("[Step 2] go to each receptacle in the list until seeing a soapbar")  
    for receptacle in recep_to_check:  
        observation = agent.goto(receptacle)  
        # check if the receptacle is closed. If so, open it.  
        if 'closed' in observation:  
            observation = agent.open_receptacle(receptacle)  
        # check if a soapbar is in/on the receptacle.
if 'soapbar' in observation: break # expectation: I should be able to find a receptacle where a soapbar is in/on it. assert 'soapbar' in observation, f'Error in [Step 2]: There is no soapbar in/on {recep_to_check}. {agent.report()}' if start_from <= 3: print("[Step 3] identify the soapbar I juts found and take it") # I need to get the identifier of the soapbar. I can ask the assistant to do that. answer = ask(f'From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation} The identifier of the soap? Only Output a single number without any other words. ' found Soapbar = f'soapbar {answer}' observation = agent.take(foundSoapbar, receptacle) # expectation: I should be able to take the soapbar from the receptacle. assert agent.holding == foundSoapbar, f'Error in [Step 3]: I cannot take {foundSoapbar} from the {receptacle}. {agent.report()}' if start_from <= 4: print("[Step 4] go to a countertop and put the soapbar on it") # There are multiple countertops, and I only need to go to one of them. observation = agent.goto('countertop 1') # check if the countertop is closed. If so, open it. if 'closed' in observation: observation = agent.open_receptacle('countertop 1') observation = agent.put(foundSoapbar, 'countertop 1') # expectation: I should be able to put the soapbar on the countertop. assert f'You put the {foundSoapbar} in/on the countertop 1.' in observation, f'Error in [Step 4]: I cannot put the {foundSoapbar} on the countertop 1. {agent.report()}'
The expert sample for the task Clean:
define environment and agent  
receptacles = ['diningtable 1', ' drawer 2', ' drawer 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']  
agent = Agent(receptacles)  
# Your task is to: put a clean lettuce in diningtable / clean a lettuce and put it in diningtable.  
# here is a solution:  
def solution(agent, start_from=1):  
    # General plan: I need to get a list of receptacles to find the lettuce, take the lettuce to the sinkbasin, clean it and put it in a diningtable.  
    if start_from <= 1:  
        print("[Step 1] get a list of receptacles where the lettuce is likely to appear.")
    # I can ask the assistant to do that.  
        answer = ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a lettuce in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.")
        recep_to_check = literal_eval(answer)  
        # expectation: the returned recep_to_check should not be empty.  
        assert recep_to_check, f'Error in [Step 1]: recep_to_check should not be empty. {agent.report()}'  
    if start_from <= 2:  
        print("[Step 2] go to each receptacle in the list until seeing a lettuce")
for receptacle in recep_to_check: observation  $=$  agent.goto(receptacle) # check if the receptacle is closed. If so, open it. if 'closed' in observation: observation  $=$  agent.open_receptacle(receptacle) # check if a lettuce is in/on the receptacle. if 'lettuce' in observation: break # expectation: I should be able to find a receptacle where a lettuce is in/on it. assert 'lettuce' in observation, f'Error in [Step 2]: There is no lettuce in/on {recep_to_check}. {agent.report()}'   
if start_from  $\leq 3$  print("[Step 3] identify the lettuce I juts found and take it") # I need to get the identifier of the lettuce. I can ask the assistant to do that. answer  $=$  ask(f'From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation} The identifier of the lettuce? Only Output a single number without any other words. '） found_lettuce  $=$  f'lettuce {answer}' observation  $=$  agent.take(found_lettuce, receptacle) # expectation: I should be able to take the lettuce from the receptacle. assert agent.holding  $= =$  found_lettuce, f'Error in [Step 3]: I cannot take {found_lettuce} from the {receptacle}. {agent.report()}'   
if start_from  $\leq 4$  print("[Step 4] go to a sinkbasin to clean the lettuce. "） # I should go to the sinkbasin first if I want to clean the lettuce. observation  $=$  agent.goto('sinkbasin 1') # check if the sinkbasin is closed. If so, open it. if 'closed' in observation: observation  $=$  agent.open_receptacle('sinkbasin 1') observation  $=$  agent.clean(found_lettuce, 'sinkbasin 1') # expectation: I should be able to clean the lettuce. assert f'You clean the {found_lettuce} using the sinkbasin 1.' in observation, f'Error in [Step 4]: I cannot clean the {found_lettuce} using the sinkbasin 1. {agent.report()} I should have been at sinkbasin 1 and holding {found_lettuce}. '   
if start_from  $\leq 5$  print("[Step 5] go to a diningtable and put the lettuce on it. ") # There are multiple diningtables, and I only need to go to one of them. observation  $=$  agent.goto('diningtable 1') # check if the diningtable is closed. If so, open it. if 'closed' in observation: observation  $=$  agent.open_receptacle('diningtable 1') observation  $=$  agent.put(found_lettuce, 'diningtable 1') # expectation: I should be able to put the lettuce on the diningtable. assert f'You put the {found_lettuce} in/on the diningtable 1.' in observation, f'Error in [Step 5]: I cannot put the {found_lettuce} on the diningtable 1. {agent.report()}'
The expert sample for the task Heat:
<sample_heat> Prompt
define environment and agent  
receptacles = ['diningtable 1', ' drawer 2', ' drawer 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']  
agent = Agent(receptacles)
Your task is to: put a hot lettuce in diningtable / heat some lettuce and put it in diningtable.
# check if the diningtable is closed. If so, open it.  
if 'closed' in observation:  
    observation = agent.open_receptacle('diningtable 1')  
observation = agent.put(found_lettuce, 'diningtable 1')  
# expectation: I should be able to put the lettuce on the diningtable.  
assert f'You put the {found_lettuce} in/on the diningtable 1.' in observation, f'Error in [Step 5]: I cannot put the {found_lettuce} on the diningtable 1. {agent.report()}'
The expert sample for the task Cool1:
<sample_cool> Prompt
define environment and agent  
receptacles = ['diningtable 1', ' drawer 2', ' drawer 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']  
agent = Agent(receptacles)  
# Your task is to: put a cold lettuce in diningtable / cool some lettuce and put it in diningtable.  
# here is a solution:  
def solution(agent, start_from=1):  
    # General plan: I need to get a list of receptacles to find the lettuce, take the lettuce to the fridge, cool it and put it in a diningtable.  
if start_from <= 1:  
    print("[Step 1] get a list of receptacles where the lettuce is likely to appear."  
    # I can ask the assistant to do that.  
    answer = ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a lettuce in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.']  
    recep_to_check = literal_eval(answer)  
    # expectation: the returned recep_to_check should not be empty.  
    assert recep_to_check, f'Error in [Step 1]: recep_to_check should not be empty. {agent.report()}'  
if start_from <= 2:  
    print("[Step 2] go to each receptacle in the list until seeing a lettuce")  
    for receptacle in recep_to_check:  
        observation = agent.goto(receptacle)  
        # check if the receptacle is closed. If so, open it.  
        if 'closed' in observation:  
            observation = agent.open_receptacle(receptacle)  
        # check if a lettuce is in/on the receptacle.  
        if 'lettuce' in observation:  
            break  
        # expectation: I should be able to find a receptacle where a lettuce is in/on it.  
    assert 'lettuce' in observation, f'Error in [Step 2]: There is no lettuce in/on {recep_to_check}. {agent.report()}'  
if start_from <= 3:  
    print("[Step 3] identify the lettuce I juts found and take it")  
    # I need to get the identifier of the lettuce. I can ask the assistant to do that.  
    answer = ask(f'From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation} The identifier of the lettuce? Only Output a single number without any other words.')  
    found_lettuce = f'lettuce {answer}'  
    observation = agent.take(found_lettuce, receptacle)  
    # expectation: I should be able to take the lettuce from the receptacle.
assert agent.holding == found_lettuce, f'Error in [Step 3]: I cannot take {found_lettuce} from the {receptacle}. {agent.report()}'  
if start_from <= 4:  
    print("[Step 4] go to a fridge to cool the lettuce")  
    # I should go to a fridge to cool the lettuce.  
    observation = agent.goto('fridge 1')  
    # check if the fridge is closed. If so, open it.  
    if 'closed' in observation:  
        observation = agent.open_receptacle('fridge 1')  
        observation = agent.cool(found_lettuce, 'fridge 1')  
    # expectation: I should be able to cool the lettuce.  
    assert f'You cool the {found_lettuce} using the fridge 1.' in observation, f'Error in [Step 4]: I cannot cool the {found_lettuce} using the fridge 1. {agent.report()} I should have been at fridge 1 and holding {found_lettuce}.'  
if start_from <= 5:  
    print("[Step 5] go to a diningtable and put the lettuce on it")  
    # There are multiple diningtables, and I only need to go to one of them. observation = agent.goto('diningtable 1')  
    # check if the diningtable is closed. If so, open it.  
    if 'closed' in observation:  
        observation = agent.open_receptacle('diningtable 1')  
        observation = agent.put(found_lettuce, 'diningtable 1')  
    # expectation: I should be able to put the lettuce on the diningtable. assert f'You put the {found_lettuce} in/on the diningtable 1.' in observation, f'Error in [Step 5]: I cannot put the {found_lettuce} on the diningtable 1. {agent.report()}'
The expert sample for the task Examine:
define environment and agent receptacles  $=$  ['diningtable 1', 'hunter 2', 'hunter 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1'] agent  $=$  Agent(receptacles) #Your task is to: look at the bowl under the desklamp / examine the bowl with the desklamp #here is a solution: def solution(agent, start_from  $\coloneqq$  1): #General plan: I need to get a list of receptacles to find the bowl and take the bowl with me, then I get another list of receptacles to find the desklamp and turn it on. if start_from  $<= 1$  print("[Step 1] get a list of receptacles where a bowl is likely to appear.") #I can ask the assistant to do that. answer  $=$  ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a bowl in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.' recep_to_check  $=$  literal_eval(answer) #expectation: the returned recep_to_check should not be empty. assert recep_to_check, f'Error in [Step 1]: recep_to_check should not be empty.{agent.report()}' if start_from  $<= 2$  print("[Step 2] go to each receptacle in the list until seeing a pen") for receptacle in recep_to_check: observation  $=$  agent.goto(receptacle) # check if the receptacle is closed. If so, open it.
if 'closed' in observation:
observation = agent.open_receptacle(receptacle)
check if a bowl is in/on the receptacle.
if 'pen' in observation:
expectation: I should be able to find a receptacle where a bowl is in/on it.
assert 'pen' in observation, f'Error in [Step 2]: There is no bowl in/on {recep_to_check}. {agent.report()}'
if start_from  $<= 3$
print("[Step 3] take the bowl from the receptacle")
I need to get the identifier of the bowl so that I can take it. I can ask the assistant to do that.
answer = ask(f'From the observation, get the identifier of an object.
For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation} The identifier of the pen? Only Output a single number without any other words. '
foundpen  $\equiv$  f'pen{answer}
observation = agent.take(found_pen, receptacle)
expectation: I should be able to take the bowl from the receptacle.
assert agent.holding == found_pen, f'Error in [Step 3]: I cannot take {foundpen} from the {receptacle}. {agent.report()}'
if start_from  $<= 4$
print("[Step 4] get a list of receptacles where a desklamp is likely to appear.")
I can ask the assistant to do that.
answer = ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a desklamp in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.')
recep_to_check = literal_eval(answer)
expectation: the returned recep_to_check should not be empty.
assert recep_to_check, f'Error in [Step 4]: recep_to_check should not be empty. {agent.report()}'
if start_from  $<= 5$
print("[Step 5] go to each receptacle in the list until seeing a desklamp")
for receptacle in recep_to_check:
observation = agent.goto(receptacle)
check if the receptacle is closed. If so, open it.
check if a desklamp is in/on the receptacle.
if 'desklamp' in observation:
expectation: I should be able to find a receptacle where a desklamp is in/on it.
assert 'desklamp' in observation, f'Error in [Step 5]: There is no
```python
desklamp in/on {recep_to_check}. {agent.report()}'
if start_from  $<= 6$
print(" [Step 6] turn on desklamp")
There might be multiple deskslamps in the environment, and I need to get the identifier of the desklamp. I can ask the assistant to do that. answer = ask(f'From the observation, get the identifier of an object.
For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation} The identifier of the desklamp? Only Output a single number without any other words.'
found_desklamp = f'desklamp{answer}'
I can directly turn on the desklamp that I just found.
observation = agent.turn_on(found_desklamp)
expectation: the desklamp should be turned on now.  
assert 'turn on' in observation, f'Error in [Step 6]: I cannot turn on {foundDesklamp} in/on {receptacle}. {agent.report()}'
The expert sample for the task Pick two:
<sample_picktwo> Prompt
define environment and agent
receptacles = ['diningtable 1', 'driver 2', 'driver 1', 'sinkbasin 1', 'toilet 1', 'sidetable 2', 'sidetable 1', 'cabinet 1', 'countertop 1', 'microwave 1', 'fridge 1']
agent = Agent(receptacles)
# Your task is to: put two cellphone in cabinet / find two cellphone and put them in cabinet
# here is a solution:
def solution(agent, start_from=1):
    if start_from <= 1:
        print("[Step 1] get a list of receptacles where a cellphone is likely to appear.")
        # I can ask the assistant to do that.
        answer = ask(f'Given a list of receptacles, please sort them in descending order based on the likelihood of finding a cellphone in each of them. The list of receptacles is: {agent.receptacles}. You should directly return a Python list.")
        recep_to_check = literal_eval(answer)
        # remove the destination from the list
        recep_to_check.remove('cabinet 1')
        # expectation: the returned recep_to_check should not be empty.
        assert recep_to_check, f'Error in [Step 1]: recep_to_check should not be empty. {agent.report()}'
    if start_from <= 2:
        print("[Step 2] go to each receptacle in the list until seeing a cellphone")
        for receptacle in recep_to_check:
            observation = agent.goto(receptacle)
            # check if the receptacle is closed. If so, open it.
            if 'closed' in observation:
                observation = agent.open_receptacle(receptacle)
            # check if a cellphone is in/on the receptacle.
            if 'cellphone' in observation:
                break
            # expectation: I should be able to find a receptacle where a cellphone is in/on it.
            assert 'cellphone' in observation, f'Error in [Step 2]: There is no cellphone in/on {recep_to_check}. {agent.report()}'
    if start_from <= 3:
        print("[Step 3] identify the first cellphone found and take it")
        # I need to get the identifier of the cellphone. I can ask the assistant to do that.
        answer = ask(f'From the observation, get the identifier of an object. For example, On the cabinet 1, you see a cloth 2, and a toiletpaper 2. The identifier of cloth is 2. Now, {observation}. The identifier of the cellphone? Only Output a single number without any other words.')
        found_cellphone1 = f'cellphone {answer}'
        observation = agent.take(found_cellphone1, receptacle)
        # expectation: I should be able to take the cellphone from the receptacle.
        assert agent.holding == found_cellphone1, f'Error in [Step 3]: I cannot take {found_cellphone1} from the {receptacle}. {agent.report()}'
    if start_from <= 4: