O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models
Authors: Yuchen Xiao
Yanchao Sun
Mengda Xu
Udari Madhushani Sehwag
Jared Vann
Deepeka Garg
Sumitra Ganesh
Venue: COLM (2024)
Keywords: Large Language Model Agent, Sequential Decision-Making, Offline In-Context Learning
TL;DR: We propose an offline in-context learning framework that utilizes offline data at scale to enhance LLMs' sequential decision-making capability without model fine-tuning.

Abstract: Recent advancements in large language models (LLMs) have exhibited promising performance in solving sequential decision-making problems. By imitating few-shot examples provided in the prompts (i.e., in-context learning), an LLM agent can interact with an external environment and complete given tasks without additional training. However, such few-shot examples are often insufficient to generate high-quality solutions for complex and long-horizon tasks, while the limited context length cannot consume larger-scale demonstrations with long interaction horizons. To this end, we propose an offline learning framework that utilizes offline data at scale (e.g, logs of human interactions) to improve LLM-powered policies without fine-tuning. The proposed method O3D (Offline Data-driven Discovery and Distillation) automatically discovers reusable skills and distills generalizable knowledge across multiple tasks based on offline interaction data, advancing the capability of solving downstream tasks. Empirical results under two interactive decision-making benchmarks (ALFWorld and WebShop) verify that O3D can notably enhance the decision-making capabilities of LLMs through the offline discovery and distillation process, and consistently outperform baselines across various LLMs.

Published as a conference paper at COLM 2024
O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models
Yuchen Xiao", Yanchao Sun', Mengda Xu, Udari Madhushani Sehwag, Jared Vann Deepeka Garg, Sumitra Ganesh
J.P. Morgan Al Research (yuchen.xian) yanchao sun, mengda. xu @aagda..x]jpmccheeeeeeae @pmchase. com [udari. wwd.ushaaahinn.mm seehaaaniisswwa,,...s jared. vann, deepeka garg, sumitra. Jumishh.ganesh/ (ameshh@jjmmrgaaa jomorgan..co
Abstract
Recent advancements in large language models (LLMs) have exhibited promising performance in solving sequential decision-making problems. By imitating few-shot examples provided in the prompts (i.e., in-context learning), an LLM agent can interact with an external environment and complete given tasks without additional training. However, such few- shot examples are often insufficient to generate high-quality solutions for complex and long-horizon tasks, while the limited context length cannot consume larger-scale demonstrations with long interaction horizons. To this end, we propose an offline learning framework that utilizes offline data at scale (e.g, logs of human interactions) to improve LLM-powered policies without finetuning. The proposed method O3D (Offline Dala-driven Discomery and Distillation) automatically discovers reusable skills and distills generalizable knowledge across multiple tasks based on offline interaction data, adv advancing the capability of solving downstream tasks. Empirical results under two interactive decision-making benchmarks (ALFWorld and WebShop) verify that O3D can notably enhance the decision-making capabilities of LLMs through the offline discovery and distillation process, and consistently outperform baselines across various LLMs.
Introduction
Recent years have witnessed remarkable advancements in artificial intelligence (AI), partic- ularly in the development of Large Language Models (LLMs). One of the standout features of LLMs is their in-context learning ability, where the LLM can perform tasks with only few-shot examples provided in the prompts, making possible deploy LLMs various applications seamlessly.
Although most existing research focuses on one-step text generation such as question answering, many real-world scenarios desire autonomous agents that can interact with external environments and make sequential decisions to complete given tasks. Some recent works successfully showcase the application of LLMs in sequential decision-making (Yao)
et al. 2023b Shinn et al. 2023 Liu et al. 2023c Yang et al. 2023), by either directly providing few-shot examples of acting and reasoning in the prompt Yaoo et al., (023b), or learning via
trial and error during online interactions with the environment (Shinn et al., 2023).
However, existing methods still struggle to solve many complex domains with LLMs due to the intrinsic difficulties that arise from long-horizon interactive tasks. On the one hand, it is widely known that the task complexity increases exponentially with the interaction horizon (Sutton & Barto, 2018), such that a large amount of data or demonstrations can be desired for an agent fully understand the environment dynamics, especially for heteroge- neous and stochastic real-world tasks, where cross-task generalizability is important. On the other hand, the in-context learning ability is constrained by the limited context window
"qqual Contribution

---

Published as a conference paper at COLM 2024
Stage
interctionn
nadortes Please traat the
intrrction log inoo several sub- aoiis summarize
Skil-based
You middle
looking quiltly inound you, see a cabinet oacinot 1, lidge
Tradl (1njestortee,
Discovered Skills
tnettameer--ppeall subtosss.
apple
findjobject)
Segmentation
findoojjcc)
Inidgo.
(objec)
go oetiiel
Offline Dataset
Prompt
puttobject, receptable] cleanjobjsc()
takeloojest)
pepperatsker and
Act 2 taee apple from cobinert
putlobject,
Stage
interaction
haloriaa... Please sotruct the vardd rert-ppa---ee action with the cornect syntas
(mngeclarieg)
Discovered Primitives (receptacia) [ (id)
Enwisnment apeciis actions.
Interact with an anstroement
task.
DiscrverPromgl
(object) (receptacle) [d] Ecampler Lake from
considar the followinng
Improve Policy
(dipa)
Skill-based Datasets
generate following
TTM DistliPrompt You
Distiled Tips
(pnimitives)
actions:
When looking for object, make
Hore are same examples:
look all poss bic locations.
(traeectoress;
sucoossa Iistorios... Please lind yriennows in the iull trnd and jurmorrre gensrnll tip the
Uso commanc
Hore is your task:
(1tjed] inton jurbboof"
Follure (Prlectooies)
Policy P
Stacoeee Trale ((aperoee)) Summinzzo and Tip..
Stage
Hierarchical Polay Execulion
fintjnug)
skill of put
Initial Obs.
is: mug sidelable
Done: findinug)
Thase
skill of find 3T find
Environment (Task)
Primitive Action
skill take
go counteriso
Env. Obs.
skill
countertop you mug
Skll-condttooaa Policier
Figure 1: The proposed 03D framework. Stage 1: offline skill discovery and data segmenta- tion. Stage 2: offline policy im improvement with knowledge distillation. Stage 3: downstream interaction with hierarchical policy execution.
of an LLM. Even if many demonstrations exist, it is hard and inefficient to prompt LLMs with large number of examples. Although finetuning is a solution, it can be much more expensive and less accessible for normal users.
In response to these challenges, this paper asks and aims to answer the follow question:
Can we develop a data-driven learning framework for sequential decision- with LLMs,, which learns from large-scale offline data without the need of model training?
In this paper, define offline learning framework to enable LLMs to discover and distill useful knowledge from interaction trajectories on multiple tasks. For LLM-powered agents/ policies, we carefully design generic learning paradigm called Offline Data-driven Discovery and Distillation (03D), which can iterate over the offline dataset and keep improving the policy. Importantly, our method does not require high-quality expert offline dataset, as it can benefit from both positive examples and negative examples of environment interac- tions, making the framework easier and cheaper use in practice. As shown in Figure 1 03D is composed of 3 stages: the first stage aims to discover reusable skills by segmenting the offline interaction trajectories; the second stage then conducts skill-conditioned policy improvement by distilling knowledge from offline data; the third stage constructs the in- teractive policy by calling the learned skills given diverse tasks. All stages are based on querying LLMs and iterating policies. As a result, 03D can learn better policies from offline dataset at scale without any model finetuning. Experiments in two commonly used domains (ALlWorll and Web-
existing datasets, to improve or utilize LLM-powered
2

---

Published as a conference paper at COLM 2024
Shop) show that our LLM agent augmented by offline knowledge has much better few-shot performance than prior methods on various tasks.
Summary of contributions: (1) We establish an offline in-context learning framework for LLM sequential decision-making agents, such that the LLM can learn from offline experience without finetuning. This offline learning paradigm allows more effective usage of past interactions (including both good and bad behaviors) generated by humans or other agents, alleviating the cost of online learning.
(2) Different from prior work which prompts and solves different types of tasks independently, our algorithm leverages offline experience from multiple tasks. By letting LLMs automatically distill shared high-level knowledge, our algorithm achieves few-shot generalization to various types of tasks with a single set of prompts. (3) 03D is a versatile framework that can boost the performance of existing methods. Empirical results show that the knowledge distilled by 03D can be combined with existing in-context learning approaches to achieve superior performance as shown in Section 4 03D can also facilitate code generation in various tasks, as detailed in Appendix D
2 Related Work
In-context learning of LLMs for sequential decision-making. (1) Generating text. Yang et al. (023) and Liu et al. (003b) conduct extensive experiments to showcase the ability of Ls to make sequential decisions in range of challenging multi-step reasoning tasks. Yao et al. 2023b) propose ReAct, a method to combine multi-step reasoning and interactive decision-making, which achieves significant improvement in multiple domains, Shinn et al. (2023) develop agents that can verbally summarize from its past failures and incorporate the reflective text into subsequent trials of the same task, analogous to online reinforcement learning paradigm. Our method, in contrast, adopts an offline learning method to minimize the cost of online learning and can adapt to various new tasks. Two concurrent works, ExpeL et al. (2024) and AutoGuide Yang et al. (2023), also perform learning from offline experiences. Expel only extracts tips from offline data and provides them all in the prompt, while AutoGuide extracts tips conditioning on a summarized state and performs state-based similarity retrieval to select tips. Our approach differs from theirs along three dimensions: a) we discover skills from offline experiences and extract skill-oriented tips; b) we discover primitive actions for each skill to reduce the chance of action hallucination; c) our method can also improve oode-generation quality while theirs only consider text-based outputs. (2) Combined with planning. LLMs can also be combined with classical planning approaches, such Planning Domain Definition Language (PDDL) (Liu et al.. 2023a Silver et al., 2023).
Differently, our paper focuses on end-to-end LLM policies without additional planning algorithms or knowledge. can be empowered by LLM-written code (Liang et al. [2022] Wang et al. [2)2aab). Although the main focus of this paper is LLM agents interacting with environments by language, the proposed method naturally extends to code-based policies.
(3) Generating code. Recent works showing that embodied agents
Training or fine-tuning LLMs for sequential decision-making. Another line of related work includes training textual policies with imitation learning (Shridhar et al., 2021 2020 and fine-tuning language models to behave as policies (Wang et 2023a| Driess et al. 2023
Wang et al. 2023ch in sequential decision
Again, our work is different as it aims at achieving hi hugh-quality L.L.-based poccsion-making without fine-tuning.
Multi-step reasoning and task decomposition with LLMs. Multi-step reasoning using LLMs has been widely studied in language domains, including chain-of-thought style reasoning Yao et al. (2023a); Fu et al. (2022); ; Wei et al. (2022), step-by-step feedback based reasoning Lightman et al. (2023) Zheng et al. (2023), and self consistency and verification based reasoning Ling et al. (2023) y Wang et al. (2022). In contrast, we focus on sequential decision-making tasks in partially observable environments, where each step results in state transitions and only sparse rewards are available.
3

---

Published as a conference paper at COLM 2024
3 Methodology
3.1 Problem Formulation and Notations
LLM-powered policy. We first formally define an LLM-powered policy. A policy π for sequential decision-making a function that maps the interaction history to a distribution over actions. Let π[4| r) denote the probability of selection action a given interaction history T, which a sequence of past observations and actions, (01, 11, 42, ... ,t). Then, with pre-trained LLM which outputs text based on any text input, a policy can be written as Tilm (a|z) == LLM (a| pmt Prett)
(1)
The goal is to learn a policy that can maximize the total reward. During in-context learning, the pre-trained LLM weights 0peet are fixed, and our 03D learns policy by learning and optimizing the base prompt ppmt from an offline dataset containing multiple interaction sopeccories/historre
Skill-conditioned policy, A policy can be conditioned on specific skills or subgoals (e.g., find (ug) al. 2023), which compositional factors of the original task (e.g, heat some milk). Let z be a skill, then skill-conditional policy can be denoted as π², with xe" (a)T) the probability of selecting action a given history T when executing skill z.
3.2 Learning from Offline Data
In many real-world decision-making systems, there exists interaction logs from various users, including experts who can successfully perform the task, as well as non-experts who may fail and make mistakes. Although a final success ffilurr flag is given for each trajectory, there no intermediate rewards within the trajectory, making policy learning more difficult.
Intuitively, seeing the interaction logs from others performing task can be helpful for one to understand the environment and finish similar tasks. Since LLMs have strong abilities of interpretation and generalization, recent works such as ReAct (Yao et al. 2023b) have shown that LLMs can solve many interactive decision -aaiing problems when prompted with few expert demonstrations. This learning paradigm is analogous to behavior cloning (Torabi et al. 2018), where an agent learns to imitate how experts react to certain scenarios. However, behavior cloning could suffer from the distribution shift between expert demonstrations and the agent's own online interactions, especially when the expert dataset is small and not representative of all scenarios in the domain. Even if we increase the number of demonstrations, the performance is constrained by the limited context window and the high cost of collecting expert demonstrations and annotations (e.g., write reasoning examples). In contrast, we introduce an offline learning framework for LLM-powered policies which transfers knowledge across tasks and makes use of large amounts of sub-optimal interaction log to improve downstream decision-making.
3.3
03D: LLM -aadd Offline Policy Improvement
Our proposed offline policy learning framework consists of 3 stages, as depicted in Figure The first stage enables the LLM to discover and abstract reusable skills from offline datasets (potentially from diverse tasks). Then, the second stage aims to learn skill-based policy for each discovered skill, through an iterative process of distilling primitives and tips to achieve policy improvement. The final stage is to construct the main LLM-powered agent who can reason and call corresponding skills sequentially to solve given tasks. Below we explain each stage detail.
Stage 1: Offline Skill Discovery and Data Segmentation. Many real-world decision- making processes require a number of steps to complete a task, such as controlling a robot to pass several obstacles and navigate to the door, which results in two challenges for LLM- powered agents. First, the limited context length may not be enough to contain the few-shot demonstration and online interaction history. Second, the language model may lose track of its goal and not pay attention to the most important information. To mitigate this issue, we propose hierarchical policy learning framework that can iteratively extracts skills from
4

---

Published as a conference paper at COLM 2024
interaction logs with primitive-level executions. Here the skills are analogous to the options or temporally extended actions (Sutton et al. [199] in hierarchical reinforcement learning. It is well-known that discovering options is difficult in traditional RL, whereas we find that skill discovery with textual logs can be surprisingly well-achieved with the semantic understanding ability of state-of-the-art LLMs. Our skill discovery process iterates over the offline trajectories, using a DiscoverProme use is in Appendix E, We ask LLMs to divide the interaction histories into skill-oriented sub-trajectories, and abstract the skills in function forms.
as shown in Fig. | (Stage 1). The full prompt we
Stage 2: Offline Policy Improvement with Knowledge Distillation. The main idea of this stage is to distill generalizable knowledge from offline datasets. Such knowledge should be generalizable to tolerate the distribution shift between offline data and online interactions and facilitate downstream tasks. We aim to distill two types of knowledge from the segmented skill-based trajectories in an iterative process, which leads to improved skill-conditioned policies.
Distilling Primitive Actions. A common mistake of LLM-powered agents is hallucination, i.e., LLMs output actions that are not valid in the environment. To ensure effective usage of LLM in decision meciiinn-mmkking applications, it is important to specify an action space in the form of natural language or code. Prior works (Liang et al. 2022 Liu et al. 2023c) manually define the available primitive functions, which requires human labor and domain knowledge. Instead, we propose to distill primitive actions or functions from the offline interaction data with LLM, which is easy to scale up and automate the practical operation. Fig. [ describes how to distill the primitives with an example, and the full prompt is in Appendix E
Distilling Policy Improvement Tips with Trajectory Contrasting. Inspired by the policy gradient methods in RL, which increases the probability of selecting good actions and lowers the probability of selecting bad ones, we propose to distill knowledge that can enhance good (i.e., achieving high long-term reward) behaviors and avoid undesired ones in the task distribution. We propose to distill "policy improvement tips" about what actions are preferred under what circumstances. However, with the offline data that only provides sequences of interactions and final scores, it is non-trivial for an LLM to figure out the correct credit assignment and useful tips to guide policy improvement. To this end, we leverage Trajectory Contrasting to sample both successful (high-score) and failed (low-score) trajectories and let the LLM generate tips by comparing them. LLMs can identify the key to success and how to avoid failures. Fig. [ shows the iterative distillation process with DistillPrompt to update the policy. More details are in Section 3.4 and Appendix E
Stage 3: Downstream Interaction with Hierarchical Policy Execu- tion. So far, Stage discovers a set of skills, while Stage 2 produces and optimizes the corresponding skill-conditioned policies. The fi- nal stage is then to compose these skills and interact with the down- stream task. We prompt a base policy Tbase with few examples (come from LLM's own segmen- tation of offline trajectories) calling skills sequentially given a downstream task. Therefore, the policy is executed in a hierarchi- cal manner as Figure the high-level base policy receives the environment and calls the low-level skill-conditioned policies; the called skill-conditioned policy then interacts with the environment and returns terminate signal when its job is done; the high-level base policy receives the update from the low-level policy and deter- mines whether to execute the next skill. Fig. shows an example of how to construct the base policy by prompting and how a skill-conditionel policy is prompted when being called. Appendix D.3 p.rrvidds concrete example of the execution process.
Prompt for base policy
Prompt for the skill coniltiondddddddd policy "put"
Inteoecd with x hnowwhold complete a given You can call supp rocduurs tindlooectt cheantjhgecti, (esn))okpus))
Incoaac with houscholc complete givon saalo
[Donlicd Tics
Pheess only geoerste following and
cal: wwweleen
call pokoopayyoodin
You in the middle of room
coll Poloooa oft.. Lokkll
homercca inseactionn:
Your took & 3o dlean some applc and put kn
tack
go to shell
Cni the 1, you cup 2, cap
call chestippphe)
poppershakor
solshalior 1.
puzlepole adkraaii)
rision shelf
You put the mup livón the sheft
Done pul/mag, sbelli fi. have put a mug (31 on the shel required.
your
task tost)
Parrs your traks"
Your task ic: pubtistyood, (roccotabee)
Figure 2: Example prompts for the base policy and the text-based skill-conditioned policy for hierarchical policy execution in Stage 3.
5

---

Published as a conference paper at COLM 2024
3.4 Implementation Details of D3D
The main algorithm is presented in Algorithm 1 We provide all used prompts and additional implementation details in Appendix In Stage 1 (Line 1-5), we tra- verse the offline trajectories and ask LLM to summanzee potential sub-procedures as skills. Then, based on the discovered skill set, we ask an LLM to segment each trajectory into sub-trajectories, each of which corresponds to specific skill (Line 5). In Stage 2 (Line 6-13), the goal is to establish a set of skill-conditioned policies. For each skill, we sample mini-batch of its (sab-)trajcctories, then discover primitive actions and distill policy improvement tips, Stage 3 (Line 14-15) is the down- stream inference process, where the
Algorithm Offline Data-driven Discovery Distillation
Input: Pre-trained ILM,, offline dataset D, sizes N2, N2. max iteration steps Ti.T:
Output: LL.M-powered policy
Stage 1: skill disnovary Initialize sets of skills 2 =
for L..... ,T; do
Sample trajectories of Discover skills:
agent observes the task and se-
Segment trajectories in D based on skillset 2, obtain Di for each 24
quentially calls the learned skill-
Stage knowledge distilltton
conditioned policies to interact with
for 23.
Initialize primitive pritiiiizepimitivvee set pa and the knowledge set T
the environment, with few-shot ex-
Initialize 77 Initiate fpmm with a D and p
amples of how to call the skills.
for ",..., do
Sample N2 trajectories dft DD
Discover primitives:
The details of an extension of the
pi LLM((P%, Discverrrec)))))mmmm))
above Pethod,,3D-Code. for code-
Distill policy improvement tips:
T c- I.J TT di; (Prompt)
generation and empirical results are
Ask LLM incorporate T the prompt of policy xt'
in Appendix() where O3D-Code out-
// Stage 3: hiararchical policy acecution
Sample examples d ~ D and segment them based on skills
performs existing methods by learn-
Provide the examples demonsttations for 7im sklll-comddtionee
ing from offline trajectories.
policies ixe) given downstream tasks
4 Experiments
4.1 Experimental Setup
Problem Domains. We consider two sequential decision-making benchmarks, ALF- World (Shridhar et al. 2021) and WebShop (ao et al., 2022). ALFWorld mimics house- hold scenarios and allows an agent to interact with with the environment through a text-based interface We use the original test set, including 134 tasks in total across six distinct task types. Following Shinn et al. (2023), we make the problem more challenging by limiting the horizon of each episode to be 30 (original is 50) steps and terminating the episode if the agent takes the same action twice. WebShop requires an agent to explore an online shopping website, check relevant product candidates, and purchase the one that matches user's query. Our evaluation considers the first 500 out of 12, 087 instructions as test set (following the official implementation (Yao et al., 2022)).
Metrics. In ALFWorld, we assess method performance by measuring the success rate (SR) under each task type and a total success rate over 134 tasks. Besides the success rate, WebShop sets product matching score as an extra metric.
Models and Offline Data. To investigate the robustness of 03D across a range of LLMs, we consider two different GPT models in our experiments. These include GPT-4-0613 and GPT- 3.5-0613, each providing distinct épret defined in Equation (1) Our offline dataset comprises official human demonstrations in both domains as the success data, and set of failure data generated by running ReAct on the training task set, as introduced in the original ALFWorld and WebShop implementations (more details are referred to Appendix A.1)
Baselines and Experiment Organization. We evaluate and analyze the proposed 03D from multiple angles by comparison with several state-of-the-art baseline methods and their variants. We first compare the downstream performance of D3D with a well-known baseline ReAct (Yoo et al.) 2023b). The results demonstrate the capability of 03D to discover and distill knowledge from offline dataset and improve downstream task-solving. Then, we compare 03D with an online learning waaiingglggnnn algorithm Reflexion (Shinn et al. 2023) which interacts with each

---

Published as a conference paper at COLM 2024
downstream task for multiple trials and improves the policy by verbally reflecting on its failure history. We also evince the versatility of 03D by showing the advantage of combining 03D with existing methods (Reflexion+ D3D). Furthermore, we conduct comprehensive ablation study for each component of 03D including primitives, skills, policy improvement tips, and trajectory contrasting. We also show the comparison with ReAct-long, which inserts as many offline trajectories to the prompt context as possible, to further validate the efficiency of our distillation mechanism.
4.2 Main Results
Effectiveness of 03D in discov- ering skills and primitives, and distilling policy improvement tips. In our experiments, 03D can extract high-quality skills from raw successful trajectories in of- fline data, resulting in seven types of skills under ALFWorld domain, including: find(object), take(object), put(object, receptacle), cool(object), heat(object), use(object) and clean(object); and four types of skills under WebShop domain, in- cluding search item, select item, se- lect item's aftributes and purchase item. Each skill consists of a set of primitives for execution, accompa- nied by group of tips to facilitate skill completion. Fig shoows four skill examples from the ALFWorld domain (full results are available in Appendix E.4). Through learn- ing from offline data, 03D is able to capture correct primitives that can be composed to achieve each cor- responding skill. Importantly, we observe that tips distilled by 03D are functional to various degrees, such as suggesting general tips, encouraging exploration, realizing action preconditions and highlighting syntax (as shown in Fig. 3). To further assess the quality of LLM-distilled knowledge, we designed an oracle baseline, 03D-Human, where domain expert writes down tips and primitives in the prompt according to ReAct's failures. The comparative results in Appendix ?? demonstrate that 03D is promising to extract human-level knowledge from offline data such achieving the performance near 03D-Human, and even surpassing 03D-Human in certain cases.
Tips Categories:
Gemeral Tips
Exploration
Precondition
Syntax
Check possible locafions when looking for ilem Expand your search area when you can't find an item the expactad locallons.
skill of find
Available primitives:
(receptacle) [id]; (receptacie)
Ensure the object you trying from.
take actually location
akil of take
allention
description and correct object.
Available primitiees:
take (object [idj from [noceptack)] [id]
You can directly cool object with fridge by laking the corresponding action.
Avaid getting stuck in loop of actions that do not contribute to the completion the
skill of cool
Available primitivas:
go to [receptacte] [id]; [otject]
(receptacie)
Use the correct preposition when placing an objest on a surface. The correct action is to put the object "tnvonn the surtace, not just "on" the surface.
sidll of
put object devise (lkee microwave), remember trying to
anothar location.
primitives: go to [recapacce] [dd]; put [obyaet] (d] Inio (raceptadi]] [X]]
Figure 3: Examples of discovered skills and distilled
knowledge and tips in ALFWorld.
Table 1: Main results on ALFWorld and WebShop in comparison to ReAct.
(a) Results ALFWorld
(b) Results WebShop
Model
Method
Pick
Clean
Heat
Cool
Look
Pick2
All
Model
Method
SR
Score
GPT-4
ReAct 03D
67
74
72
GPT-4 (0613)
ReAct 03D
39
(0613)
92
100
96
95
100
53
91
41
58
GF11-3.5
ReAct
13
10
17
7
GP'1-3.5 (0613)
ReAct
27
60
(0613)
03D
71
41
61
03D consistently outperforms baselines across various LLMs under ALFWorld (Table. la and WebShop (Table. [bb. 03D achieves higher success rates than ReAct by 19% and 34% in ALFWorld, and 15% and 8% in WebShop, with using GPT-4-0613 and GPT-3.5-0613 respectively. Furthermore, as shown in Table la] the success rate achieved by 03D in each task category consistently exceeds that of ReAct with the two GPT models. This further confirms that the tips distilled by O3D from offline data are generalizable and beneficial
7

---

Published as a conference paper at COLM 2024
Table 2: Comparison with online-learning method Reflexion and combination between Reflexion and O3D.
(a) Results in ALFWorld
(b) Results in WebShop
Model
Method
Pick
Clean
Heat
Coul
Look
Pick2
All
Model
Method
SR
Score
GPT-3.5
Reflexion
26
18
29
GFT-3.5 (0613)
Reflexion
35
(0613)
030
71
35
4
67
44
24
41
03D
35
61
Reflexion+ 030
29
50
Reflexion
across diverse task types. For example, the tip "pay attention to the task description and take the correct object" helps the LLM agent avoid taking a similar object (a pot) rather than the requested one (a pan); and the tip "The correct action is put the object 'in/on' the surface, not just 'on' the surface" prevents the LLM agent from making syntactical errors with primitives, which are the two common mistakes made by ReAct.
We also observe that, the overall performance of GPT-3.5-0613 is significantly lower than GPT-4-0613 in the ALFWorld domain. Similar poor performance on ALFWorld is also reported by the AgentBench paper Liu et al. (2023b). This discrepancy may be attributed to the fact that GPT-3.5-0613 is fine- tine-tuned with data and objectives that are diverged from the ALFWorld domain. Appendix D.3 provide some interaction logs these tasks and analyze the potential causes of the failure. Despite the misalignment of GPT-3,5-0613 with the ALFWorld domain, our approach still achieves 34% improvement over ReAct.
Advantages of offline learning over online iterative reflection. We compare the perfor- mance of 03D with Reflexion (Shinn al., (202)) that involves 10 and 3 iterationn of online reflection in ALFWorld and WebShop respectively (see Table 2a and Table (b).. Reflexion allows an LLM agent reflect on its online interaction history via multiple iterations for performance improvement. However, such online interactions in a real-world environment can be expensive or risky. In contrast, 03D learns from offline data, which may include interaction logs from various users agents. Remarkably, 03D outperforms Reflexion in its first deployment trial without additional online reflection. More importantly, Reflexion learns task-specific textual tips, which only works for the current task it is solving. In contrast, 03D identifies reusable skills and generalizable textual tips that can facilitate a wide range of downstream tasks. Also, Reflexion relies on online failure experiences, however, 03D offers leverages both successful and failed experiences for improvements, which is more effective in generating proper tips to address mistakes.
03D can be combined with existing methods to enhance their performance. The compari- son between Reflexion and Reflexion+03D validates that, our offline learning framework can improve existing methods by prepending the discovered primitives and distilled tips from offline data into their prompts. The results in Table 2a and Table 2b show that 03D significantly boosts Reflexion's 's performance under both ALFWorld and Webshop domains.
Table 3: Comparison with ReAct-long which naively injects demonstrations into prompts.
(a) Results in ALFFWorld
(b) Results in WebShop
Model
Method
Pick
Clean
Heat
Cool
Look
Pick2
All
Model
Method
SR
Score
GPT-3.5 (0613)
ReAct
13
10
0
0
17
0
7
GPT-3.5 (0613)
ReAct
27
60
ReAct-long
71
17
ReAct-long
03D
71
35
67
44
24
03D
35
4.3
Ablations
03D is much more effective than naively injecting demonstrations into prompts. To demonstrate this, we add the same set of offline trajectories into the original ReAct's prompt many as we can, with results shown Table and Table 3b ReAct-long demonstrated improved performance over the original ReAct because of the richer context from the offline trajectories. Yet, 03D's outstanding performance in ALFWorld highlights its superior effectiveness in skill discovery and knowledge distillation from offline data. In Webshop, 03D reaches higher success rate but a lower score than ReAct-long, because it focuses on distilling tips to purchase the item exactly matching the instruction via success-failure trajectories contrasting, rather than
which is referred to as ReAct-loog,, As
8

---

Published as a conference paper at COLM 2024
O3D(TD-only)
O3D(TD-only
ALFWorld
0.5
Check all possible locations for the required iern, not just the most obvious ones.
0.4
Use the correct commands to interact with items
You can clean Items direclly with the sinkbasin without needing turn on.
0.2
Cellphanes can be found on sidetables and beds, After finding cellphane, place in dresser.
Kettles can be found on stavebumers. After finding the kettles, place them in the cabinet.
0.1
Spocns be found countertops. After finding the spoon. deen using the sink besin before placing the sidelable.
WebShop
0.4
Check the product's fealures to ensure meets al the requirementa in the instruction before making a purchese. if the first productt selected does not meet all the requirements, back the search results and select different product.
0.2
selacting an chock features to ensure meets task requirements before purchasing. the item does not meat the requiroments after chocking Its features, go back and solect another Item.
0.3
Figure 5: Comparison on averaged success rate over GPT models using contrastive and non-contrastive (NC) distillation methods, with tips in green and pink boxes respectively.
increasing the raw score. By having the full trajectories with scores, however, ReAct-long has a more fine-grained understanding of the relation between tasks and scores. A potential improvement for O3D is to contrast trajectories based on range of scores, rather than just success or failure. Besides, ReAct-long's long prompt would incur much higher API calling costs for large number of downstream tasks.
Primitives, skills and policy improvement tips independently advance base- line performance. 03D has three major processes: skill discovery (SD), primi- tives discovery (PD) and policy improvement tip distillation (TD). To investigate each component's contribution to performance improvement, we conducted an ab- lation study considering three variants of 03D, each with only one component. Fig. shows the three variants of 03D either outper- form the baseline or achieve the same performance as the baseline over two LLM models. In ALFWorld, 03D (PD-only) plays the dominant role in perfor- mance improvement with GPT-4-0613, because the major mistakes made by the baseline are outputting primitive actions with syntax errors or hallucinating unavailable actions. 03D (SD-only) boosts the perfor- mance the most with GPT-3.5-0613, because the tasks in ALFWorld are too complex for ReAct with GPT- 3.5-0613, and 03D (SD-only) solves the tasks in hier- archy by performing skill selections that greatly re- duces the complexity. In WebShop, the three compo- nents consistently benefit the baseline performance across the two GPT models, with their individual contributions also being model-dependent. Since the offline data was collected using GPT-3.5-0613, we observe that the highest overall improvement of the three components occurs in this model.
ReAct 03D(5D-only)
OJD(PD-only) O3D(TD-only)
ALFWorld
ALFWorld
1.0
0.8
0.8-
0.6
0.6
0.4
0.2
WebShop
WebShop
0.5
0.5
0.4
0.3
5
0.1
GPT 0613
GPT 3.5 0613
Figure 4: Comparison success rate (SR) with three variants of 03D against ReAct.
Trajectory contrasting produces domain-dependent
advantages over non-contrasting approach. 5 that contrasting successful and failed trials in offline data is relatively beneficial in certain domains, compared to the non- contrastive (NC) way using only successful data. In ALFWorld, failures often stem from disregarding domain-specific dynamics and rules, and the contrastive method effectively produces generic tips (green box in Fig. 5) to correct mistakes in failure cases. Conversely, the non-contrastive method merely summarizes the facts (pink box in Fig. 5) from successful trials, offering limited utility. In WebShop, however, there often no clear mistake, such as different searching keywords may result in the same products and multiple items fitting the task requirements. Therefore, the advantage of contrastive method is slight, and both methods output analogous tips as shown in Fig.

---

Published as a conference paper at COLM 2024
5 Conclusion
This paper introduces offline in-context learning framework, 03D, for sequential decision- making, where LLM agents can learn from previous experiences in a scalable offline manner to improve performance without model fine-tuning. 03D stands out by allowing LLMs to distill shared high-level knowledge from offline interaction logs, which is injected into single set of prompts to be reused in solving diverse downstream tasks. Empirically, 03D outperforms baseline methods in two challenging benchmark domains, and it is compatible with existing L.L.-based methods for sequential Decision-making Our work offers possibility for future LLM-based algorithm development in terms of efficiently leveraging offline data at scale for real-world sequential dlecision-making applications.
Disclaimer. This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co annd its affiliates ("JP Morgan"), and is not product of the Research Department of JP Morgan. JP Morgan makes representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or any person, if such solicitation under such jurisdiction or to such person would be unlawful.
References
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. ar preprint ar arXin:2303.03377 2023.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. ar arviv preprint ar arXiv:2210.00720 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arxiv preprint ar Xix:2209.0075.. 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ar arxiv preprint (aoo22305.20050 2023.
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. ar preprint ar drXio:200.00887, 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency. arxiv preprint ar xxxx:2304..1177. 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Iims as agents. arXiv preprint ar Xix:::2308.06888 2023b.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. ar preprint arXiv:2306.15722, 2023c.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ar preprint ar XrXi::2303..1333 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. httph:tt/////rtee wttp://wwww..a.///////1011/0131
10

---

Published as a conference paper at COLM 2024
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Inter- active Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL hhttps://arxiti http:://axxvvooo.aaaaae/ org/abs////eeeoooee....oo /
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arxiv preprint arXiv:2305.11011 2023.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint drXiv:2305.16633 2023.
Richard Sutton and Andrew G Barto. Reinforcement learning; An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112 (1-2):181-211, 1999.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, pp. 4950-4957. AAAI Press, 2018. ISBN 9780999241127.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar, Voyager: An open-ended embodied agent with large language models. Xru preprint arviv: Arxio-2305.16291, 2023a.
Huaxiaoyue Wang, Gonzalo Gonzalez-Puumariege Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain- of-thought. ar preprint arXivv2305.16744, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar Xrv preprint ar arXiv:2203.1117,, 2022.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, Xiv preprint ar xrXi:2332.001560 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Shain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. XSkill: Cross embodiment skill discovery. In 7th Annual Conference on Robot Learning, 2023. URL
https://openrevier
Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint ar arXivv2306.0222 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards
scalable real-world web interaction with grounded language agents. In ArXio, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Xrviv preprint ar Xix::22005. 10601, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Narasimhan Karthik, and Yuan Cao. React: Synergizing reasoning and acting in language models. Xiv preprint arXi::210000362, 2023b.
11

---

Published as a conference paper at COLM 2024
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. The 38th Annual AAAI Conference on Artificial Intelligence, 2024.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. ar Xie preprint ar Xit:23044.09777, 2023.
12

---

Published as a conference paper at COLM 2024
Appendix
A Details of Implementation and Extra Results
16
A.1 Offline Data Collection and Usage
16
B Comparison between LLM-Gemerated Knowledge and Human-Generated Knowl- edge
Additional Results
18
C.1 ALFWorld
18
C.2 WebShop
18
D Extension of 03D to Code-Generation
19
D.1 Methodology
19
D.2 Experiments
20
D.2.1 ALFWorld
20
D.2.2 WebShop
21
D.3 Detailed Interaction Logs: Examples and Comparisons
23
Prompts and Learned Knowledge
25
Text-based Policy
25
E.1.1
Skill Discovery Prompt
25
ALFWorld: Skill discovery prompt
25
WebShop: Skill discovery prompt
26
E.1.2
Primitive Discovery Prompt
28
ALFWorld: Skill-conditioned primitive discovery prompt
28
WebShop: Skill-conditioned primitive discovery prompt
29
E.1.3
Skill-conditoneed Policy Improvement Tips Distillation Prompt
30
ALFWorld: Policy improvement tips contrastive distillation prompt
30
ALFWorld: Policy improvemen tips non-contrastive distillation prompt
31
WebShop: Policy improvement tips contrastive distillation prompt
32
WebShop: Policy improvement tips non-contrastive distillation prompt
34
E.1.4
Prompts for Downstream Task Solving
35
ALFWorld: Base policy prompt
35
ALF-Woldd: subtask skill-conditioned policy prompt template
38
ALFWorld: subtask skill-conditioned policy prompt for heat(object)
38
WebShop: Base policy prompt
40
WebShop: skill-conditioned policy prompt template
42
WebShop: skill-conditonned policy prompt for select item attributes()
42
Code-based Policy
43
AL.FWorld: Skill discovery
43
13

---

Published as a conference paper at COLM 2024
WebShop: Skill discovery
45
Webshop: Skill discovery
47
E.3 Human-Generated Policy Improvement Tips and Primitives for Each Skill
48
ALFWorld
48
ALFWorld: Policy improvement tips and primitives for skill clean(object)
48
ALFWorld: Policy improvement tips and primitives for skill cool(object)
ALFWorld: Policy improvement tips and primitives for skill fond(object)
49
ALFWorld: Policy improvement tips and primitives for skill heat(object)
ALFWorld: Policy improvement tips and primitives for skill
put(object, receptacle)
50
ALFWorld: Policy improvement tips and primitives for skill take(object)
50
ALFWorld: Policy improvement tips and primitives for skill use(object)
50
E.3.2
WebShop
50
WebShop:
Policy improvement tips and primitives for skill search search_item/
(
50
WebShop:
Policy improvement tips and primitives for skill select item()
WebShop:
Policy improvement tips and primitives for skill se-
lect it..itm.ar attribbttr
WebShop:
Policy improvement tips and primitives for skill pur- chase_iteml
E.4
LLM-Distilled Policy Improvement Tips and Primitives for Each Skill
51
E.4.1
ALFWorld
51
ALFWorld:
Policy improvement tips and primitives for skill clean(object)
51
ALFWWrld:: Policy improvement tips and primitives for skill cool(object)
52
ALFWorld: Policy improvement tips and primitives for skill find(object)
52
ALFWorld: Policy improvement tips and primitives for skill heat(object)
53
ALFWorld: Policy improvement tips and primitives for skill put(object, receptacle)
53
ALFWorld: Policy improvement tips and primitives for skill take(object)
54
ALFWorld: Policy improvement tips and primitives for skill use(object)
54
E.4.2
WebShop
54
WebShop:
Policy improvement tips and primitives for skill pur-
chase-item()
55
WebShop: Policy improvement tips and primitives for skill search item
55
WebShop: Policy improvement tips and primitives for skill se- lect.item (ct.item.atrributes)) attributes().
55
WebShop: Policy improvement tips and primitives for skill select_item/
56
E.5
Ablation Study and Discussion on Policy Improvement Tips
56
E.5.1
Tip Distillation Only (TD-Only) Prompts
56
A.FFWorld: Contrastive tip distillation prompt
56
14

---

Published as a conference paper at COLM 2024
ALFWorld: Non-contrastive tip distillation prompi
57
WebShop: Contrastive tip distillation prompt
58
WebShop: Non-contrastive tip distillation prompt
60
h.5.2
Results ALFWorld
ALFWorld: Distilled policy improvement tips with contrastive prompt
61
ALFWorld: Distilled policy improvement tips with non-contrastive prompt
62
ALFWorld: Human distilled tips and primitives
63
E.5.3
Results in WebShop
64
WebShop: Distilled policy improvement tips with contrastive prompt
64
WebShop: Distilled policy improvement tips with non-contrastive prompt
WebShop: Human distilled tips and primitives
65
E,6
Generated Code
65
E.6.1
ALFWorld
65
ALFworld: Generated skill code
65
ALFworld: Skill Composition
67
E.6.2
WebShop
67
Webshop: Generated skill code
67
Webshop: Skill Composition (GPT-4)
68
Webshop: Skill Composition (GPT-3.5)
68
15

---

Published as a conference paper at COLM 2024
A
Details of Implementation and Extra Results
A.1
Offline Data Collection and Usage
The offline dataset used in our experiment consists both success and failure interaction trajectories. The success data in ALFWorld are gathered from a 'trai,json' file in the training dataset provided on their official website. We download human demonstrations from the official website of WebShop, and select out the demonstrations with 100% product matching score from the training dataset as the success data. We generate the failure data by running ReAct with GPT-335-0301 on the WebShop, which are both conducted on the training task set. Eventually, we have totally 80 pairs of success trajectories and failure trajectories on the same tasks in ALFWorld and 90 pairs of success trajectories and corresponding failure trajectories in WebShop.
ALFWorld and running ReAct with GPT-3.5-0613
In Table we detail the usage of the offline data for discovering primitives and skills and distilling policy improvement tips. We use GPT-4-0613 for all discovery and distillation processes. In our experiments, we note that the batch size may impact the quality of results. The tips distillation in 03D is skill-oriented, where we need contrastive data pairs for each skill. Among the first 30 offline data pairs, certain skills have zero failure data (e.g., cool) as the mistakes happened in other skills. We, therefore, generate more offline data (80) for the tips distillation part of 03D.
Table 4: Details of offline data usage in ALFworld
Method
Component
Data Type
Batch Size
of Trajs
Skill Discovery
success
6
03D
Primitives Discovery
success
12
Tips Distillation
success & failure
80 X 2
03D (SD-only)
Skill Discovery
success
6
03D (PD-only)
Primitives Discovery
success
12
03D (TD-only)
Tips Distillation
success & failure
30 x 2
03D (TD-only NC)
Tips Distillation
success
30
Table 5: Details of offline data usage for WebShop
Method
Component
Data Type
Batch Size
of Trajs
Skill Discovery
success
30
03D
Primitives Discovery
success
90
Tips Distillation
success & failure
30 x 2
03D (SD-only)
Skill Discovery
success
2
30
03D (PD-only)
Primitives Discovery
success
90
03D (TD-only)
Tips Distillation
success & failure
30 x 2
03D (TD-only NC)
Tips Distillation
success
30
16

---

Published as a conference paper at COLM 2024
B Comparison between LLM-Generated Knowledge and Human-Generated Knowledge
Table 6: Comparison with 03D-Human in ALFWorld and WebShop.
(a) Results ALFWorld
(b) Results in WebShop
Model
Method
Pick
Clean
Heat
Cool
Look
Pick2
All
Model
Method
SR
Score
GPT-4
DSD
92
100
96
100
91
GPT-4 (0613)
58
(0613)
OGD-Human
83
100
87
95
100
53
88
03D-Hunan Hunan
41
61
GPT-3.5
03D
71
35
4
67
44
24
GPT-3.5 (0613)
C3D
35
61
(0613)
03D-Huzan
71
83
03D-Human
31
61
Table 7: Tips of the "put(object. receptacle)" skill in ALFWorld
Method
Tips
LLM
1. Use the correct preposition when placing object surface. The correct action is to put the object "in/on" the surface, not just "on" the surface.
2. Place the object in the first available and suitable location instead of unnecessarily checking other locations.
3. If you put an object in device (like microwave), remember to take it out before trying to put it in another location.
Human
1. You can directly put an object in on an occupied receptacle.
2. Please strictly follow the syntax of primitives.
Table Knowledge the "select item's attributes" skill in WebShop
Method LLM
Tips
Primitives
1. Ensure to select all the necessary attributes as per the in- struction.
click(Attribute]]
click[Color]
2. Be aware that color and size options may not always be straightforward and may include additional information or codes.
click[Style]
click[Size]
click[Flavor]
3. Be careful with the case and spacing of the words when selecting item attributes.
Human
1. Remember to select all required attributes using
Attributt
click[Attribtee]] before click[Buy Now].
2. You should pay attention to the current observation to check
clickable buttons.
03D is promising to extract human-level knowledge from offline data such achiev- ing the performance near 03D-Human, and even surpassing 03D-Human in certain cases. 03D-Human designed an oracle, where a domain expert writes down tips and primitives in the prompt according to ReAct's failures. We noticed that human often gives more general tips/ prisiiiimiiiee, while LLMs can distill both general and granulated ones because of the contrastive way over success-failure pairs. Given the same sets of L.M--eenrater knowledge and guman-generatee which type of knowledge is better depends on the model's 's capability, such as 03D outper- forming 03D-Human when using GPT-4 in ALFWorld and GPT-3.5-0613 in WebShop, but being relatively worse than O3D-Human other models. Please see the concrete examples in Table. and Table. 8 where the -.M-gerrateed knowledge is more comprehensive than those generated by humans.
knowledge to three different models, as Table shown,
In Table. 7 the LLM-generated tips are more comprehensive such as the concrete in-line examples of obeying action syntax; the suggestion for avoiding redundant behaviors; and the
17

---

Published as a conference paper at COLM 2024
hint the action precondition. We think this is why 03D always outperformed -SD-Human on the "Pick" task actoss three GPT models.
In Table. [ the LLM-genreated tips offer detailed guidance aspects to pay attention during attribute selection, along with more granular primitives being discovered to assist the agent in focusing various types of attributes.
Overall, the comparison between 03D and O3D-Human verifies that LLMs can distill tips and discover primitives as good human-generated ones. Please see Appendix E.3 and Appendix E.4 for the full tips and primitives of human-generate and LLM-generated.
C Additional Results
In this section, we present additional experimental results in ALFWorld and WebShop domains.
C.1 ALFWorld
In our ablation study, we also further visualize the performance of the three variants of 03D under each type of task with three different models, as shown in Fig 6 From the perspective of each task type, the advantage of the three variants of 03D versus ReAct is bit model-dependent but overall, they outperform ReAct as shown in the last column of Fig.
ReAct
O3D(SD-only)
O3D(PD-only)
O3D(TD-only)
Put
Clean
Heat
Cool
Look
Pick
Total
1.0
0.8
0.4
GPT-4-0613
GPT-4-0613
GPT-4-0613
GPT-4-0613
GPT-4-6613
GPT-4-0613
GPT-4-0613
Clean
0.8
%
0.4
GPT-3.5-6613
GPT-3.5-0613
GPT-3.5-0613
GPT-3.5-0613
GPT-3.5-0613
GPT-3.5-0613
Figure 6: Comparison on success rate (SR) with three variants of 03D against ReAct.
C.2 WebShop
We show the performance of the three variants of 03D the product matching score against the baseline in Fig/7 With GP4-0613 and GPT-3.5-0613, the three variants of 03D all outperform ReAct, which validates the contribution of the offline data-driven discovery and distillation to policy improvement.
WebShop
WebShop
40
ReAct 03D:SD-only)
20
03D/PD-oniyl
O3D/TD-oniyl
GPT 0613
3.5 0613
Figure 7: Comparison the product matching score with three variants of 03D against ReAct.
18

---

Published as a conference paper at COLM 2024
D Extension of 03D to Code-Generation
D.1 Methodology
This paper mainly focuses on Text-based-Policy which takes in textual observations from the environment and outputs textual actions to interact with the environment. But the proposed approach also applicable to other scenarios of -eveennrnmmeent interaction. For example, it is well-known that LLMs can write code to solve tasks, where one asks LLMs to generate code and implement the policy function given proper prompts, i.e,
Code-based-Policy
code fal == Code(alT) <- (LM(0pmm Ppret).
(2)
In this case, the goal of learning to optimize the produced function code C) without tuning the pre-trained weights 0. The central idea of 03D naturally applies to Code-based- Policy: the LLM discovers reusable skills (callable functions) and distills policy improvement tips from offline dataset; the obtained knowledge helps LLM refine its generated code policy iteratively; during execution stage, the base policy sequentially determines which skills/functions to call, and then the corresponding skill function is executed.
Algorithm 2 Learning Code-based-Policcy with Offline Data-driven Discovery and Distilla- tion (03D)
Input: Pre-trained LLM, offline dataset D, batch sizes N1, N2, max iteration steps Tí, 72 Output: Code-based-Policy π
// Stage skill discovery
16 Initialize sets of skills Z = Ø
17 for t 1,..., Ti do
18
Sample N1 trajectories d ~ DD
19 Attempt to discover more skills Z + LLM d; DiscoverPromptt
20 Segment trajectories in D based on skillset 2, obtain D for each 2μ E Z
// Stage knowledge distillation
21
for z E 2 do
Initialize primitive set 7. and the knowledge set T Initialize 7 + Let LLM write function to reproduce d ~ D³ with primitive functions
22
pa
24
for 1or/=1,..., do
25
Sample N2 trajectories d D
Attempt discover more primitives p +- LLM DiscoverPromptt
26
Distill policy improvement tips T LLM(T* dPs DistillPromptt
27
Ask LLM to improve π² based on suggestions 7 hierarchical policy execution
// Stage 3:
Sample examples d ~ D and segment them based on skills
Provide the examples as demonstrations for Tilm to call skill-conditioned policies {nT hrezzez given downstream tasks
Algorithm illustrates the implementation of Code-based-Policy. The major algorithm flow is similar to the text-based one in Algorithm with several distinctions below: Policy Initilization with Primitives. Text-based-Policy directly provides the discovered primitives in the prompt of policy and advises the agent to follow the primitives, while Code-based-Policy first lets the LLM write primitive functions and then calls these primitive functions in the code of kkill-conditioned policies.
Policy Improvement. Since the distilled policy improvement tips are in natural language, we directly ask the LLM to merge the new suggestion into the prompt of Text-based-Poolle For didee-based-Poiccyy we let the LLM consider the policy improvement tips and re-write the policy code.
Policy Construction and Execution. In Stage 3, we prompt the base policy to call the learned Text- Noxt-based-Policee Policy Code-based-Policy Note that for Code-based-Polie Policy, it is possible
19

---

Published as a conference paper at COLM 2024
that the generated skill-conditione code has compilation errors, that it requires human checking or validation on a small set of tasks to verify that the code is executable.
Using LLMs to directly interact with environments (Text-based-Policyy and using LLMs to write code to interact with environments (Code-based-Pood Policcy are usually discussed separately. Our study also reveals the different pros and cons of these two approaches that is not well-discussed in literature.
Advantages of Code-based-Poligy code, which is more interpretable and reliable, and can fully avoid hallucination or syntax errors execution. Moreover, Code-based-Police
Code-based-Policy explicitly writes the acting policy in
usually more cost efficient, as the generated code can be reused in new tasks without calling LLMs. Therefore, Code-based- Policy can be more suitable for applications where reliability and efficiency are important. Text-based-Polin
is relatively easy to implement in practice with less human supervision. Also, in complicated environments such as WebShop where language understanding is important, Text Text-based-Police can achieve much better performance than Code-based-Policce as it retains the commonsense, expressivenes: and reasoning ability of pre-trained LLMs. Therefore, for language-orienter applications where reasoning and the ability of recovering from failure are crucial, Text-based -aoicd,, or a combination of the two approaches, can be better choice.
Adnantages of Fext-based-Polle
Reliability and robustness. stands the interaction logic and writes the acting policy in code, which is more interpretable and reliable, and can fully avoid hallucination or syntax errors in execution.
Advantages of Setle-baed-PPllic
Sede-based-Poliyy under-
Cost and efficiency. Code-based-Police is usually more cost efficient, the generated code can be reused in new tasks without calling LLMs.
Flexibility and expressiveness. In more complicated environments, Text-based-Policy may achieve much better performance than Code-based-Polit Policy, as it remains the commonsense and reasoning ability of pretrained LLMs, especially when language understanding is needed.
Implementation difficulty and anticipated human supervision. Text-based-Policy is relatively easy to implement in practice with less human supervision. For discussed above, supervision or certain level of online validation may be required to ensure the code is executable.
as
D.2 Experiments
In this section, we present additional experimental results of 03D-Code in ALFWorld and WebShop domains.
D.2.1 ALFWorld
We show the comparison of 03D-Code against two state-of-the-art code-as-policy baselines, Demo2Code Wang et al. (2023b) and AdaPlanner Sun et al. (2023) in Table. Note, we ran the official AdaPlanner code with GPT-3.5-06T3. It ended up with much worse performance than the results in the original paper. We think this is because their results were generated by GPT-3.5-0301.
The extension of 03D to code-generation, 03D-Code, demonstrates advancements comparing with two established baselines, Demo2Code Wang et al. (2023b) and AdaPlanner Sun al. (2023), in both domains. 03D-Code achieves an impressive leading performance against Demo2Code by 42% with GPT-4-0613, 32% with GPI-3.5-0613, 44% with GPT-3.5-0301, and outperforms AdaPlanner by 65% with GPT-3.5-0613, 9% with GPT-3.5-0301 The principal advantage of our approach lies in its unique method of generating code: adopts a bottom- up style, effectively constructing policies on top of robust skill functions. Through iterative skill refinement, the model cultivates robust skills by leveraging extensive and diverse data from demonstrations. Skill functions can then be efficiently utilized and reused to compose higher-level policies. In contrast, Demo2Code follows a top-down approach, requiring the generation of code for the same set of skills each time it receives new task. Due to the context length constraint inherent in LLMs, only limited number of demonstrations are used to guide skill code generation in Demo2Code, resulting in unstable
20

---

Published as a conference paper at COLM 2024
Table Results of Pode-based-Policy in ALFWorld
Model
Method
Pick
Clean
Heat
Cool
Look
Pick2
SR
Demo2Code
96
58
13
43
0
65
48
GPT-4-0613
03D-Code
100
84
87
90
89
88
90
Demo2Code
96
26
48
29
0
82
46
GPT-3.5-0613
AdaPlanner
13
13
03D-Code
100
71
91
86
89
18
78
Demo2Code
96
48
35
19
17
53
46
GPT-3.5-0301
AdaPlanner
78
94
70
94
63
78
81
03D-Code
84
91
89
and inconsistent skills. Although Adaplanner incorporates online reflection process for code correction, typically over iterations as its official setup, it still falls short in terms of success rate compared with 03D-Code. Notably, its performance is significantly worse than 03D-Code when utilizing GPT-3.5-0613. This comparison further confirms the superiority of D3D-Code's bottom-up way for code-generation and the iterative skill refinement over offline data.
We also conduct experiments to investigate whether the policy improvement tips distilled for text-based policy can improve the performance of code-based policy or not. As the results shown in Fig. in ALFWorld, the tips distilled with the contrastive approach and the non-contrastive approach both have a minor contribution to the improvement of code-based policy.
1.0
0.0
0.6
O3D-Code
03D Code/TD
03D (odoCoooe/Tmmm NC)
Figure 8: Results of 03D-Code with the tips distilled via contrastive and non-contrastive ways against the original 03D-Code in ALFWorld. We use the averaged success rate over the three GPT models the performance of each method.
D.2.2 WebShop
We show the comparison of 1SD-Code against Demo2Code Wang et al. (2023b) Table. 10 (Note that AdaPlanner does not provide a solution for WebShop, thus we did not compare with it.)
Table 10: Results of Code-based-Pollcy in WebShop
Model
Method
SR
Score
Demo2Code
1
5
GPT-4-0613
03D-Code
19
31
As the results shown in Table. 10 Webshop poses substantial challenge for code-based policies owing to its need for comprehensive natural language understanding within the environment feedback. We address this challenge by enabling LLMs to construct skills by employing a limited set of LLM-based functions that encapsulate the underlying LLM capabilities (see Appendix F.6.2 To ensure fair comparison, also offer the same set of
21

---

Published as a conference paper at COLM 2024
functions to Demo2Code. While our approach substantially enhances the performance of code-based policies in comparison to the baseline, it's important to note that skill generation remains considerably constrained by the intricate and diverse text-based environment feedback in Webshop-
Additionally, in WebShop, the results shown in Fig 9 the tips distilled for text-based policy is not helpful for the improvement of code-based policy.
02
0.2
Avg Score
0.1
0.2
B3D-Code(TD-omk) 030 Code(TD.only NC)
Figure 9: Results of 03D-Code with the tips distilled via contrastive and non-contrastive ways against the original 03D-Code in WebShop. We measure the averaged success rate and the averaged product matching score over GPT-4-0613 and GPT-3.--16K-0613 the performance of each method.
22

---

Published as a conference paper at COLM 2024
D.3 Detailed Interaction Logs: Examples and Comparisons
In Figure 10 we visualize the downstream interaction logs of both ReAct and our 03D on the same task from ALFWorld, to better illustrate why 03D has advanced performance. We can see that ReAct agent makes a syntax mistake when executing the action of putting potato, As a result, the invalid action does not work in the environment, but the agent does not realize the error and believes it has successfully finished the task. In contrast, 03D executes the policy in a hierarchical way, and each skill-conditioned policy works as expected. Also, the "put" agent can successfully avoid the syntax error as suggested by the distilled knowledge.
Inter Obe "You ar the ddde of roiry Looking cur aroundiyy you see calirat, coffeermactiee courterap
iniu Okt "We aocie the makrc of rooe, Lookl yo anly cound you you see oicemeet oicteee 5 clower driaee 2. ferwer 1 Merg gritggaon menoreee dkael a shad ereaa 1, Your bask cool sone petato and resrowens.
osd nal nat- mit cnd
serommn wnbelli sha whall
Aevosurn alandtuna
diciue stow humee and a beilke
put't
x Tark lodee bhe ker, Food to and polauss them souft trige, Ihe
Act tirec To rote the taak, need fnds and tak pornt, chan aod with Hegga, twen patt pérowarss
"OK"
( Act Baak My lask lo lisd pullts Apolau morr levyy appour ridge (1), mantaaiaa (I
Acto Fist lnwed fied a patrs A potrr more kole to appoor countatee 1-)0
staring
31 Oho
Obe "oxc"
ga totigg
Ols open Velge The kilge soen you see egg 1, latuuo and
pessoo (1) 'OXL"
Art go couuretton
Reramis tedjacaagg
Aet "ake polbio countetto top
Acts Nox taka pakato (TS, Mear, (nwed to go biige ouc
Obe 'Ok.
Art to Vegge
trage
Aut "hate potne hom countatop
Arth tqpar Vals
Like
Obx "Yu open télge Thiifiipe loopan in own age
Ad Stll codtpatttor
Act "ood polao with thdge I"
Obx the poado Lavng the todge
Aul Thes My tusk coct pulcie From par (T) Wrad, wee telge Ci
Anto taeek Now nooi pollts Noxt locod tn reformeeee" Obx 'Ox"
Adl 'go to ligge
Act- to mitromas
Ael taac patalo
ObX "Te marowoee is shooo?"
Reain
cooled
micooovt
Obc "our the nicoouue The ornwws spwn you apple
now have colled Medt
Obe "Notinghespprra"
yut polatc: Inrrsoooeel"
Osu Oil.
ancomsue
Olu Toleemmee Aloos"
Olis "Mou Sisoo riroooss 1."
Asl "opor mocecue
entrovaw Thareoovee
andaampt
"Mathing
Ad 'ou poass noe cossss
"Matling heppers"
Netue sase paaaya "Die patarass marrwaee hove (obak 11 Barstrewees seguiaal"
Example of downetream interaction with ReAct on "coof" task in ALFWorld
Example of downstream interaction with O3D on "coof" task in ALFWorld
Figure 10: Interaction logs of ReAct (left) and 03D (right) in the same task with GPT-4.
23

---

Published as a conference paper at COLM 2024
In experiments, we observe that the decision-making performance can be significantly different in different GPT versions. For example, GPT-3,5-0613 struggles to solve some ALFWorld tasks (ReAct with GPT-3.5-0613 fails on all Heat and Cool tasks as shown in Table 6) To better understand this phenomenon, we examine the downstream interaction log of the agent in these tasks. Figure 11 (left) shows the expert demonstration in task where the agent heats an apple and puts it in a fridge, which is the example provided by ReAct Yao et al. (2023b) in its base prompt. Given this example, however, we find that the ReAct agent with GPT-3.5-0613 model does not successfully solve the downsttrem task that is very similar to the expert example. When the agent finds that the microwave is closed, it does not follow the expert demonstration to heat the apple. Instead, it tries to open the microwave and retry going to the microwave. The agent finally gets stuck in a dead loop of going to the microwave and opening it. This is a common problem of GPT-335-0613 across many test tasks, showing the difficulty of letting LLM agent solve sequential decision-making problems.
St Inttal the You are the mábde o'a room, Losking zuichl sround you you sne cuberut 13, cabbett 12 oatanel 11. catine 10, scateee D. mo catered 7. excimst wradeet wi cecieat o cuterut uoadied 2. cab not
R. In Oba You are the midile of room Looking quidlty around you, you see cabinct a cubircl cubanet 4, oatannt caberet 2. culrnct 1 cofformechiee 1, countartu a counterta counteriop
drawer &, dlawer 2: a diowor a frigg 1. gollagooon 1,0
eofoormatto countetto a dranggate 1. drewer a lagg 1 gattogecan a retomwve shael shell 2t sheell a Moonin stombumer a stneburmer edwattuee attowhurmer and toader Your taok ta to: put hot apple fridge.
microwave 1 shefl 5, shelf 2. shelf airitasin 5, stoveburer a stowetumer a
stovetummr 2. atouetumer and loaster?
Your tank for put hot apple fridge.
tenn sohe the sodk reed and aale wr appte, then head it mecrowwre, tradly put lnagg..
Act trikk boseeea the leak, nese
an appie, then tear microwass, appeor trippe diningisble by
tidge
apple.
(1-15), gersegacan
shink
appeeriny
(1-5),
diseed.
Ast
i closed.
Act open top
Olee:
The trigee Init you and
Act open Hidge
Ube
som sowl 1, 2, letuue mup a plite 2, potato 2. potato 1, and tomano
vadge The teope
Act to diningtable
Obs On the divinaable 1. you see apple a, bread a fork 3. fork 2.
Adt go to countortop
tork 1, mig 2, peppershaver 2 pitte 2. pot a sonpbott 3, spasiil 1,
Obs: On the courterto 1, you see apple 2, apple 1, 3. potato 3. and spatulo
upoio 5, apcon it spouns 1. wimebolile' and ww
Act Jant: Now lound an apple GG) Moxt, reed to Jatwe
Act Brink how find on apple (T), Noxt I need to take Obe: OK
Oba: OK.
Act take apple hrom countevtop 1
Act trin appll from diinghhble
Obs You pick the appie hrom counterto
Obe: You plic up the replle
chringlable
Act dent Now have beken apple (2), Next, Ineadd to to the microwave (1) heat
Act trink: Now take an apples (1), Neod, need go to microoass (1)
and
Oba:
go to microwoee
Acc crowane
Otee: The microooe ln dosec.
Act teat appla I with microwaiee 1t
open, maa
Ots:: You hoat the apple 1 using the microware 1,
Act iisoowvve
Act Brink: Now heat an apple (1), Novt, need to put ivion Vidge 1,
Obe
Act diee micromeoe
Act go to ope
Oba dose missenwer
nidgo opon. you cup ogg
Ast ge microwiin
Act fidge
Obs: Nothing hoppers.
put tndge
Act open ricoowass
Obe You open the miooooee The ricrowawe 1i opan, In you now may
Act go ta recrowase
Oba: Nothing toppars.
Expert demonstration on "heat" task in ALFWorld
Interaction log of ReAct on "heat" task in ALFWorid
Figure 11: Expert demonstrations for a "heat" task in ALFWorld (left) and a failure trajectory of ReAct with GPT-3.5-0613 on a "heat" task given the expert demonstration in its prompt (right).
24

---

Published as a conference paper at COLM 2024
E Prompts and Learned Knowledge
E.1 Text-based Policy
E.1.1 Skill Discovery Prompt
ALFWorld: Skill discovery prompt
You will be given the interaction histories between agent and household environment. Break the full interaction log into several aub-tasks, and summarize these sub-taske in an abstract way. When there is already a list enviromment-specinn subtasks to start with, please keep them in your response improve them if you figure out any mistakes, and you can also append subtasks according the given trial. You should not add duplicate subtasks. Give your response after "New summarization:
Here is an example:
Task 1:
Success trial:
You are in the middle of room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop 1, diningtable 3, diningtable 2, diningtable 1, drawer 1, fridge garbagecan 1, microwave 1, sidetable
sinkbasin 1, stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1, a toaster 1.
Your task is to: find some apple and put it in sidetable.
go to fridge 1
The fridge is closed.
open fridge
You open the fridge 1. The fridge 1 is open. In it, you see a lettuce 2, mug 2, a potato 2, and tomato 1.
close fridge
go to garbagecan
Dn the garbagecan 1, you see a apple 3, and egg 3.
take apple 3 from garbagecan
You pick up the apple 3 from the garbagecan 1.
go to sidetable 1
the sidetable 1, you see a cup 1, a lettuce 1, peppershaker 3, a potato 1, and a saltshaker 1.
put apple 3 in/on sidetable 1
You put the apple 3 in/on the sidetable 1.
Here is no list subtasks to start with.
New summarization:
above interaction log, the agent completes including "go to", "open" and "close" as primitive operations. take (apple) including "take" as primitive operations, and put put (apple, sidetable) including "go to" and "put" primitive operations. We thus can segment the entire log based these three subtasks below.
subtasks: (apple)
You are in the middle of room. Looking quickly around you, you see cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop 1, diningtable 3, diningtable 2, diningtable 1, drawer 1, fridge 1, garbagecan 1, microwave 1, sidetable
sinkbasin 1, stoveburner 4, stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: find some apple and put it in sidetable.
find (apple)) go to fridge
The fridge is closed.
open fridge 1
25

---

Published as a conference paper at COLM 2024
You open the fridge The fridge is open. In it, you lettuce
mug 2, potato 2, and tomato close fridge
You close fridge 1.
go to garbagecan
the garbagecan you apple 3, and egg take (apple):
take apple 3 from garbagecan
You pick up the apple 3 from the garbagecan 1.
(apple, sidetable):
go sidetable
On the sidetable 1, you see a cup 1, a lettuce 1, a peppershaker 3, a potato 1, and a saltshaker 1.
put apple 3 in/on sidetable 1
You put the apple 3 in/on the sidetable 1.
As there is no a list of subtasks to start with, we list the Eaviroment-speciee specific subtasks below:
(object)
take (object)
(objectt, receptable)
WebShop: Skill discovery prompt
You will be given interaction histories between an agent and webshop environment. Break the full interaction log into several sub-tasks, and summarize these sub-tasks in abstrat way. when there is already list of environment-specifa subtasks to start with, please keep them in your response improve them you figure out any mistakes, and you can also append new subtasks according to the given new trial. You should not add duplicate subtasks. Give your response after "New summarization
Here is example:
Task 1:
Success trial:
WebShop
Instruction:
i'm looking for a light pink long handle back loofah shower brush, and price lower than 40.00 dollars [Search]
Action: (light pink long handle back loofah shower brush] Observation:
[Back to Search]
Page 1 (Total results: 50)
[Next >]
[B09N8YLMG]]
JulaJuyo Shower Back Scrubber, Long Handle Bath Body Brush Exfoliating Loofah Sponge on Stick for Men Women, Colorful Nylon Mesh Showering Bathing Luffa Cleaning Lufa Washer Brush (1 Pack-Gray)
\$7.99
[B09J8KCN4P]
JulaJuyo Back Scrubber for Shower, Long Handle Body Bath Brush Shovering Loofah Sponge on Stick for Men Women, Nylon Mesh Exfoliating Bathing Lufa Scrub Shower Cleaning Luffa Brush (1 Pack-White)
\$7.99
[B08Y6VZFT6]
Prozklves Back Scrubber for Shover, Long Handle Back Loofah Shower Brush, Soft Nylon Mesh Back Cleaner Washer, Bath Brush for Women Men, Exfoliating Body Scrubber Elderly (Light Pink)
\$7.0 .99
26

---

Published as a conference paper at COLM 2024
Action: click (B08Y6V2FT6)
Observation:
[Back to Search]
[< Prev]
scent [2 pack-beige] [2 pack-green] [2 pack-light pink] [2
pcs=green+beige][ [2 pcs=greentpink] [3 pack-beige] pack-light pink] pcs=colors] [beige] [green] [light pink]
Prozklves Back Scrubber for Shower, Long Handle Back Loofah Shower Brush, Soft Nylon Mesh Back Cleaner Washer, Bath Brush for Women Men, Exfoliating Body Scrubber for Elderly (Light Pink)
Price: \$7.99
Rating: N.
Description]
[Features]
[Reviews]
[Attributes]
[Buy Now]
Here is list subtasks start with.
New summarization.
In the above interaction log, the agent completes two subtasks: search_itens with the purpose of searching for the required item, including "search[UUERY] as primitive operation, as well as select_iteml with the purpose selecting the item that matches the requirement, including "click[ITEM, [ITEL_ " as a primitive operation. We thus can segment the entire log based these two subtasks below.
WebShop
Instruction:
i'm looking for light pink long handle back loofah shower brush, and price lower than 40.00 dollars [Search]
search_item
Action: search pink long handle back loofah shower brush] Observation: [Back to Search]
Page 1 (Total results: 50)
[Next >]
[B09N8YLMGJ]
JulaJuyo Shower Back Scrubber, Long Handle Bath Body Brush Exfoliating Loofah Sponge on Stick for Men Women, Colorful Nylon Meah Shovering Bathing Luffa Cleaning Lufa Washer Brush (1 Pack-Gray)
$$7.99 [B09J8KGN4P]
JulaJuyo Back Scrubber for Shower, Long Handle Body Bath Brush Showering
Loofah Sponge Stick for Men Women, Nylon Mesh Exfoliating Bathing Lufa Scrub Shower Cleaning Luffa Brush (1 Pack-White)
$$7.99
[B08Y6VZFT6]
Prozklves Back Scrubber for Shower, Long Handle Back Loofah Shower Brush, Soft Nylon Mesh Back Cleaner Washer, Bath Brush for Women Men, Exfoliating Body Scrubber for Elderly (Light Pink)
$7.99
select_item():
Action: click [B08Y6VZFT6]
Observation:
[Back Search]
Prev]
27

---

Published as a conference paper at COLM 2024
scent [2 pack-beigel [2 pack-green). [2 pack-light pink] [2 pcs-greentbeigel [2 pcs=green+pink] [3 pack-beige] pack-light pink] pca=colors] [beige] [green] [light pink]
Prozklves Back Scrubber for Shower, Long Handle Back Loofah Shower Brush, Soft Nylon Mesh Back Cleaner Washer, Bath Brush for Women Ken, Exfoliating Body Scrubber for Elderly (Light Pink)
Price: \$7.99
Rating: N.A.
[Description]
[Features] [Reviewa]
[Attributes]
[Buy Now]
As there is list of subtasks start with, list the -apirimint-specificc subtasks below:
search_item() select_item()
E.1.2 Primitive Discovery Prompt
ALFWorld: Skill-conditioned primitive discovery prompt
You will be given success histories in which you were placed an enviromment and given task to complete. Please extract the valid enviromment-specifle actions with the correct syntax from the trial, and list them under the corresponding subtask catergory. When there is already a list of enviromment-spe:ln to start with, please keep them in your response or improve them if any syntax also append new actions according the given new error detected, and you duplicate actions regardless of the examples. For trial. You should existing actions in the list, do not more examples. Give your response after "New summarization:
actions
Here is example:
Task 1:
Success trial:
Plan for Completing the Task:
1. (apple)
2. cake(apple)
Segmented Interaction History:
find(apple): find apple go to garbagecan
Dn the garbagecan 1, you see a apple 3, and egg 3.
take (apple): take the apple
take apple from garbagecan
You pick up the apple 3 from the garbagecan 1.
New summarization
find(ebject)
wwvironment-ssecifiic actions:
go to [receptacle] [id]
Example: go to garbagecan 1
take (object)
Environment-gpecific actions:
take [object] [id] from [receptacle] [id]
Example: take apple 3 from garbagecan
pubjebt receptacle)
28

---

Published as a conference paper at COLM 2024
Enviromentt-pecific actions:
cool (object)
nawironment-speciie actions:
use(object) Euvironment-specifa actions:
clean(object) Environment-specific actions:
heat(object)
Environment- actions:
WebShop: Skill-conditioned primitive discovery prompt
You will be given success histories in which you were placed an environment given a task to complete. Please extract the valid enviroment--pecific actions with the correct syntax from the trial, and 1ist them under the corresponding subtask catergory. When there is already list of enviromment-speciffy actions start with, please keep them in your response or improve them if any syntax error detected, and you can also append new actions according to the given new trial. You should not add duplicate actions regardless of the examplea. For existing actions in the list, do not add more examples. Give your response after "New aummarization;
Here is one example:
Webshop
Instruction:
would like a ounce bottle of bright citrus deodorant for sensitive skin, and price lover than 50.00 dollars [Search]
search_item(): search with query regarding instruction Action: search[3 ounce bright citrus deodorant sensitive skin]
Observation:
[Back to Search]
Page (Total resulta: 50)
[Next >]
[B078GWRC1J]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Qunce
\$10.99
[B078CTXVXY]
Ginger Fresh Deodorant by Earth Mama | Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[В08КBп]4XK]
Barrel and Dak Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar Patchouli Blend, Gentle Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack) $$15.95
Action: click (Back to Search]
Observation:
Webshop
Instruction:
would like 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
29

---

Failure trial: Published as a conference paper at COLM 2024
New summarization:
search_item()
Environment-specil actions:
bearch[Query]
Example search
bright citrus deodorant sensitive skin]
click [Back to Search]
Example click (Back to Search]
select_itemd Environment-specific actions:
select_iteed jallet_item__atrrbbueeee
Enviromment-specific actions:
purchase_item{ Euvironment actions:
E.1.3 Skill-conditioned Policy Improvement Tips Distillation Prompt
ALFWorld: Policy improvement tips contrastive distillation prompt
You will be given both failure and success histories in which you were placed in environment and given task complete. You should generate tips according your task description rather than the historical interactions. By comparing the two histories, please find the mistakes in the failure trial, and summarize tips to fix the mistakes. The tips can also be the rules in this particular environment. You will need these tips later to solve similar task, When there is already list of tips response or improve them, and you can also append new tips according to the given new trials. If the new mistake you detect can be solved by any existing tips, you do not need to generate new tips. You do not have to add new tips if there is no
with, please keep them in your
Give your response after "New summarization;
Here is one example:
Task 1:
You are the middle room. Looking quickly around you, you see cabinet
10, cabinet 9, cabinet 8, cabinet 7, a cabinet 6, cabinet 5, cabimet
4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop diningtable 2, diningtable 1, draver 3, drawer 2, a draver 1, fridge 1, garbagecan 1, a microwave 1, sinkbasin
2, countertop
stoveburner 4, stoveburner 3, stoveburner 2, a stoveburner 1, a toaster
1.
Your historical interactions:
go countertop
the countertop 1, you see a apple 2, butterknife 1, cup 1, a knife 2, knife 1, a pan 1, a potato 2, a spatula 2, a spatula 1, a spoon 2, spoon 1, a tomato 2, winebottle 3, and winebottle 1. go to countertop
countertop 2, you see butterknife take mug from countertop
cup 1, fork 2, and mug 1.
You pick up the mug 1 from the countertop 2.
Your task is to: heat the mug burner
go stoveburner 1, you see a pan 2. heat mug 1 with stoveburner 1 Nothing happens.
heat mug with stoveburner Nothing happens.
Success trial:
30

---

Published as a conference paper at COLM 2024
You the middle of room. Looking quickly around you, you cabinet 10, cabinet 9, cabinet 8, cabinet 7, cabinet 6, cabinet cabinet
4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop
2, a countertop 1, diningtable 2, diningtable 1, draver 3, draver
2, drawer 1, fridge 1, garbagecan 1, a microwave 1, sinkbasin 1, stoveburner 4, stoveburner 3, atoveburner 2, stoveburner 1, and toaster
1.
Your historical interactions:
go to countertop
you apple butterknife 1, cup 1, knife 2, knife 1, pan 1, a potato 2, apatula 2, a spatula 1, spoon 2, spoon 1, tomato 2, a winebottle 3, and a winebottle 1. go to countertop
On the countertop
On the countertop 2, you see butterkmife take mug from countertop You pick up the mug 1 from the countertop 2. Your task is to: heat the mug
cup 1, fork 2, and mug 1.
to microwave 1
You arrive at loc 22. The microwave is closed.
heat mug with microwave
You heat the mug using the microwave
New summarization. In task 1, my task is to heat the mug 1. I made mistake in heating the mug with stoveburner, and I also got stuck continually did this. shown the success trial, the correct way is to heat mug with a microwave. Thus, should try different actions when getting stuck in a loop; I should use a microwave to heat mug, and I can directly heat an object with microwave without opening the microwave and putting the object in it.
loop where I
Tips:
Try to execute different action when get stuck in a loop of taking identical actions.
Mug should heated by microwave rather than a stoveburner.
You can directly heat an object with a microwave by taking the corresponding action.
Note, you should directly include the above example tips in your response. Give your response after "New summarization:
ALFWorld: Policy improvement tips non-contrastive distillation prompt
You will be given success histories in which you placed in environment and given a task to complete. Please summarize tips according your task description rather than the historical interactions. The tips can be the rules in this particular environment. You will need these tips later to solve a similar task. When there is already list of tips to start with, please keep them your response improve them, and you also append tips according the given new trials. You not have to add new tips if there is no one. Give your response after "New summarization:
Here is example:
Task
Success trial:
You are in the middle of room. Looking quickly around you, you cabinet
cabinet cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, a coffeemachine 1, countertop
10, cabinet 9, cabinet 8, cabinet 7, cabinet
2, a countertop 1, diningtable 2, diningtable 1, draver 3, draver
2, a draver 1, fridge 1, garbagecan 1, a microwave 1, a sinkbasin 1, stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1, and toaster
1.
Your historical interactions:
31

---

have Published as a conference paper at COLM 2024
go countertop
countertop you see apple 2, butterknife 1, cup 1, a knife 2, knife 1, a pan 1, a potato 2, a spatula 2, a spatula 1, a spoon 2, spoon 1, a tomato 2, a winebottle 3, and winebottle 1. go to countertop
butterknife cup 1, fork 2, and mug take mug from countertop
the countertop
you
You pick up the mug 1 from the countertop 2. Your task is to: heat the go to microwave 1
You arrive at 22. The microwave is closed.
heat mug with microwave 1 You heat the mug 1 using the microwave 1.
New summarization: In task 1, my task is heat the mug shown the success trial, the correct way to heat mug with microwave. Thus, I should use a microwave to heat a mug, and can directly heat an object with microwave without opening the microwave and putting the object in it.
Tipa:
Mug should be heated by microwave.
You can directly heat an object with a microwave by taking the corresponding action.
Note, you should not directly include the above example tips in your response.
Give your response after "New summarization:
WebShop: Policy improvement tips contrastive distillation prompt
You will be given both failure and success histories in which you were placed in an environment and given a task to complete. You should generate tips according to your task description rather than the historical interactions. comparing the two histories, please find the mistakes in the failure trial, and summarize general tips fix the mistakes. The tips rules this particular environment. You will need these tips later to solve a similar task. When there is already a list of tips to start with, please keep them in your response according the given new trials. If the new mistake you detect can be solved by any existing tips, you do need generate new tipa. You add new tips if there is no one. Give your response after "New sumnarization:
also be about the
improve them, and you can also append new tips
Here is example:
Task1:
Failure Trial:
Webshop
Instruction:
bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
would like
Your historical interactions:
Action: search[3 ounce bright citrus deodorant sensitive skin]
Observation:
[Back to Search]
Page 1 (Total results: 50)
[Next >]
[B078GWRC1]]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin,
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[В078CТКVXY]
Ginger Fresh Deodorant by Earth Natural Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
32

---

Published as a conference paper at COLM 2024
\$10.99
[B08KBVJ4XN]
Barrel and Dak Aluminum-Free Deodorant, Deodorant for Men, Essential 0il-Based Scent, 24-Hour Odor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
\$15.95
Action: click [B078GWHCCC
Observation:
[Back to Search]
[< Prev]
[assorted scents] [bright citrua] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack of 1)] [3- [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
[escrippiio]]
[Features]
[Rvview]]
[Buy Now]
Your task is: solect_item,. aeleec_.tee_attrbbbtee
attributees ()
Action: click (BRIGHT CITRUS]
Observation: Invalid action!
Success Trial:
Webshop
Instruction:
would like ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
Your historical interactions:
Action: search(3 ounce bright citrus deodorant sensitive skin] Observation:
[Back to Search]
Page 1 (Total resulta: 50)
[Next >]
[B078GWRC1]]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[B078GTXVXY]
Ginger Fresh Deodorant by Earth Mama | Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Qunce
\$10.99
[B08KBVJ4XN]
Barrel and Oak Aluminum-Free Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Ddor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
\$15.95
Action: [B1cCWB07CC
Observation:
[Back to Search]
[< Prev]
[assorted scents] [bright citrus] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack of 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
[escription]
33

---

Published as a conference paper at COLM 2024
[Features]
[Reviess]
[Buy Now]
Your task is:
Action: click (bright citrus]
Observation: You have clicked bright citrus.
Action: click[3 ounce (pack of 1)] Observation: You have clicked 3 ounce (pack of 1).
New summarization: task 1, my task is select, item attributes made mistake on capitalizing the name scent option [bright citrusl, so that click [BRIGHT CITRUS] become invalid. As shown in the success trial, the correct action should be click (bright citrus]. Thus, I should strictly use the exact words of each option without making any changes.
Tips:
You should strictly use the exact words of each available option without making any changes.
Note, not directly include the above example tips in your response. Give your response after "New summarization:
WebShop: Policy improvement tips non-contrastive distillation prompt
You will be given success histories in which you were placed in an enviroument and given a task to complete. Please summarize tips according to your task description rather than the historical interactions. The tips can be the rules in this particular environment. You will need these tips later solve similar task. When there already list of tips to start with, please keep them in your response or improve them, and you can also append new tips according to the given new trials. You do not have to add new tips if there is no one. Give your response after "New summarization!
Here is one example:
Task1:
Success Trial:
Webshop
Instruction:
would like 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
Your historical interactions:
Action: search(3 ounce bright citrus deodorant sensitive skin]
Observation:
[Back to Search]
Page (Total resulta: 50)
[Next >]
[B078GWRC1J]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[B078GTXVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[В08КBVJ4XX]
Barrel and Dak Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
\$15.95
Action: [B078CWCCC
34

---

Published as a conference paper at COLM 2024
Observation:
[Back Search]
Prev]
scent [assorted scents] [bright citrus] [calming lavender] [ginger fresh] (simply non-scenta]
size [travel (4-pack)] [3 (pack of 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Barth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
Price: \$10.99
Rating: N.A.
[escription]
[Features]
[Reviess]
[Buy Now]
Your task is: beancc__iee_aattrbbiee
Action: click (bright citrus]
Observation: You have clicked bright citrus.
Action: click[3 ounce (pack of 1)]
Observation: You have clicked 3 ounce (pack of 1).
New summarization: In task 1, my task is select_ select_item item_ iite_aattibuteee.e As shown in the success trial, selected attributes according to the instruction.
Tips:
You should selected attributes according to instruction.
Note, do not directly include the tips in the above example in your response. Give your response after "New summarization:
E.1.4 Prompts for Downstream Task Solving
ALFWorld: Base policy prompt
Interact with a household to complete given task. You can call subprocedures (object)),), take (object), use (object), coben(o))ect,, heatebbjet), cool (object) and (ubjebj location). Here are some examples.
You are the middle room. Looking quickly around you, you see cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, countertop 1, a garbagecan 1, a handtowelholder 2, handtowelholder 1, sinkbasin 2, sinkbasin 1, a toilet 1, toiletpaperhanger 1, and towelholder 1. Your task to: put some spraybottle on toilet.
think: To solve the task, need find take sparybottle, then put it on toilet.
DK.
call: (apraybooy))))e) Subprocedure finished. call: take (spraybottle)) Subprocedure finished. call: (epraybottle, toilet)
Subprocedure finished.
You are in the middle of room. Looking quickly around you, you see cabinet
4, a cabinet 3, cabinet diningtable 3, diningtable 2, diningtable 1, drawer garbagecan 1, a microwave 1, a sidetable
cabinet 1, coffeemachine 1, countertop 1,
fridge sinkbasin 1, stoveburner 4, stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.
Your task is to: find some apple and put it in sidetable.
think: To solve the task, I need to find and take apple, then put it in sidetable.
DK.
call: (apple)
35

---

Published as a conference paper at COLM 2024
Subprocedure finished.
call: take (apple) Subprocedure finished.
call: put (apple, sidetable) Subprocedure finished.
You are the middle 13, cabinet 12, a cabinet 11, a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, cabinet 6, a cabinet 6, a cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine 1, countertop fridge 1, garbagecan 1, microwave 1, shelf 3, shelf 2, shelf 1, sinkbasin 1, stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.
room. Looking quickly around you, you see cabinet
diningtable
drawer
Your task is to: put clean lettuce in diningtable. think: To solve the task, need find and take lettuce, then clean it with sinkbasin, then put it in diningtable. OK.
call: (lettuc)
Subprocedure finished.
call: (lettuce)
Subprocedure finished.
call: clean (lettuce)
Subprocedure finished. call: put (lettuce, dimingtable) Subprocedure finished.
You are in the middle of room. Looking quickly around you, you see cabinet 10, cabinet 9, cabinet 8, cabinet 7, a cabinet 6, cabinet 5, cabinet 4, cabinet 3, cabinet
cabinet 1, coffeemachine 1, countertop countertop 2, a countertop 1, diningtable 1, a drawer 6, drawer 5, drawer 4, a drawer 3, a drawer 2, a draver 1, fridge 1, a garbagecan 1, a microwave 1, sinkbasin 1, stoveburner 4, stoveburner 3, a stoveburner 2, stoveburner and toaster 1.
Your task to: heat some egg and put in diningtable. think: To solve the task, need to find take an egg, then heat it with microwave, then put it in diningtable. OK.
call: find(egg)
Subprocedure finished.
call: take (egg)
Subprocedure finished. call: heat (egg)
Subprocedure finished.
call: put diningtable) Subprocedure finished.
You the middle of room. Looking quickly around you, you 13, cabinet 12, a cabinet 11, a cabinet 10, cabinet 9, cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, coffeenachine 1, a countertop 1, diningtable 1, draver 1, fridge 1, garbagecan 1, microwave 1, shelf 3, shelf 2, shelf 1,
cabinet
sinkbasin 1, stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1,
and a toaster 1.
Your task is to: put hot apple in fridge. think: To solve the task, I need to find and take apple, then heat it with microwave, then put it fridge. OK.
call: (apple
Subprocedure finished. call: take (apple) Subprocedure finished. call: heat(apple) Subprocedure finished.
36

---

Published as a conference paper at COLM 2024
call: (apple fridge) Subprocedure finished.
You are in the middle of room. Looking quickly around you, see cabinet 16, cabinet 15, cabinet 14, cabinet 13, cabinet 12, cabinet 11, cabinet 10, cabinet
cabinet 8, cabinet 7, cabinet 6, cabinet 5, a cabinet 4, a cabinet 3, cabinet 2, cabinet 1, coffeemachine 1, countertop 2, a countertop 1, a diningtable 1, a drawer 5, a drawer 4, drawer 3, drawer 2, drawer 1, fridge 1, a garbagecan 1, microwave 1, a safe 1, stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1, and toaster 1.
sinkbasin
Your task to: cool some pan and put in stoveburner. think: To solve the task, I need to find and take pan, then cool it with
fridge, then put it in stoveburner. OK.
call: faad
Subprocedure finished. call: take (pan)
Subprocedure finished.
call: (pan)
Subprocedure finished.
call: (pan, stoveburner)
Subprocedure finished.
room. Looking quickly around you, you see coffeetable 1, diningtable 1, a drawer 4, drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a sidetable 2, a sidetable 1, and a sofa 1. Your task is put two cellphone in sofa.
You are the middle
think: To solve the task, I need find and take the first cellphone, then put in sofa, then find and take the second cellphone, then put sofa. OK.
call: find(cellphone1)
Subprocedure finished. call: (celppho))))))1
Subprocedure finished.
call: (ctecllhhonel sofa)
Subprocedure finished.
call: (ellchel)))))e2)
Subprocedure finished.
call: take (cellphoe))))) Subprocedure finished. call: (cellphonee sofa)
Subprocedure finished.
You are in the middle of room. Looking quickly around you, you see bed 1, a desk 1, drawer 3, drawer 2, drawer shelf 4, ahelf 3, ghelf 2, shelf sidetable 2, and sidetable 1. boul under the desklamp. think: To solve the task, I need to find and take a boul, then find and use a desklamp.
garbagecan 1, safe 1, shelf 5,
Your task to:
DK.
call: find
Subprocedure finished. call: take (bowl)
Subprocedure finished. call: find (inddaallmp)) Subprocedure finished. call: use use (ddeklammp)) Subprocedure finished.
You the middle of room. Looking quickly around you, you bed 1, drawer 10, drawer 9, drawer 8, a drawer 7, a drawer 6, drawer 5, drawer 4, a drawer 3, a drawer 2, drawer 1, a dresser 1, a garbagecan 1, a shelf 9,
37

---

Published as a conference paper at COLM 2024
shelf 8, shelf 7, shelf shelf shelf 4, shelf 3, shelf and shelf
Your task is to: examine the pen with the desklamp.
think: To solve the task, I need to find and take pen, then find and use desklamp. First I need to find pen. pen is more likely to appear in drawer (1-10), shelf (1-9), bed (1), garbagecan (1). check by one, starting with drawer 1. OK.
call: (pen)
Subprocedure finished. call: (pen)
Subprocedure finished. call: find (dneklamm))) Subprocedure finished. call: use (desklamp) Subprocedure finished.
There are 7 skills in ALFWorld, which are find(object), take(object), use(object), clean(object), heat(object), cool(object) and put(object, location). Each of the skill induces skill- conditioned policy. Here first give general prompt template for these policies.
ALFWorld: skill-conditioned policy prompt template
Interact with a household to complete given task, Please consider the following tips to solve new tasks:
{policy improvement tipa} {primitives) <<(
Here examples.
<'< (examples}
In the above template, the policy improvement tips and primitives for each skill are provided in Appendix E.4, For the examples, we use the same examples provided by ReAct. Note that ReAct uses different examples to prompt different types of ALFWorld tasks, while reorganize the examples such that single set of prompts is used across all types of tasks. This is because we discovered skills that are shared across tasks, allowing more generalizable knowledge.
Below we provide the full prompt of skill-conditioned policy for skill "heat(object)". Prompts of other skills are available in our codebased, which will be released soon.
ALFWorld: skill-conditioned policy prompt for heat(object)
Interact with household to complete given task. Please consider the following rules to solve new tasks:
1. Try to execute a different action when you get stuck in a loop of taking identical actions.
Ensure you have the correct object in your possession before attempting perform actions on it.
2.
3. Check all possible locations if you can't find an object the expected places.
4. You can directly heat object with microwave by taking the corresponding
action.
5. an action does not work, try a different approach or command.
6. Focus on relevant locations to complete the task efficiently.
7. A cup is not the same as a mug. Make sure to find and use the correct object for the task.
38

---

Published as a conference paper at COLM 2024
8. You think about your plan the first step, and after that you only select following actions in your response (please strictly follow the syntax):
to [receptacle] [id]
# Example:
go to countertop
heat [object] [id] with [receptacle] [id]
Example:
heat egg with microwave 1
Here
examples.
You are in the middle of room. Looking quickly around you, you see cabinet 10, cabinet 9, cabinet 8, cabinet 7, a cabinet 6, cabinet 5, cabinet cabinet
4, cabinet 3, cabinet
coffeemachine 1, countertop countertop 2, a countertop 1, diningtable 1, drawer 6, drawer 5, drawer 4, a drawer 3, a drawer 2, a draser 1, fridge 1, a garbagecan 1, microwave 1, sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, stoveburner and toaster 1. Your historical interactions: open fridge
You open the fridge 1. The fridge 1 is open. In it, you see a lettuce 2, mug 2, and potato 3.
go to countertop
the countertop 1, you see a bread 1, fork 1, and saltshaker 1.
go to countertop On the countertop 2, you see nothing.
go to countertop
On the countertop plate 1, sink 1, and spatula 2. take egg 2 from countertop 3
you
boul 1, butterkmife 1, egg kettle
You pick up the egg 2 from the countertop 3. Your task is: heat (egg)
think: My task is heat an egg. Now take an egg (2). Next, need go heat it.
microwave (1) OK.
go to microwave 1 closed.
The microwave heat egg with microwave
You heat the egg 2 using the microwave 1.
Done heat(egg): I have heated the desired egg (2).
You are
room. Looking quickly around you, you see cabinet 13, cabinet 12, a cabinet 11, a cabinet 10, cabinet 9, a cabinet 8, a cabinet 7, cabinet 6, a cabinet 5, cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine 1, countertop
middle
diningtable 1, drawer 1,
fridge 1, garbagecan 1, microwave 1, shelf 3, shelf 2, shelf 1, sinkbasin 1, stoveburner 4, stoveburner 3, a stoveburner 2, a stoveburner 1,
and a toaster 1.
Your historical interactions: go to fridge The fridge is closed. open fridge You open the fridge 1. The fridge 1 is open. In it, you see a cup 1, and egg
1.
go diningtable 1
diningtable 1, you see apple 1, a bread 1, a mug 2, a peppershaker 3, a plate 2, a pot 1, soapbottle 3, a spatula 1, spoon 3, spoon 2, a spoon 1, a vinebottle 3, and a vinebottle 2.
fork 3, fork 2, a fork
take apple from diningtable 1 You pick up the apple 1 from the diningtable Your task is: heat (apple)
39

---

Published as a conference paper at COLM 2024
think: My task is heat apple. Now take apple (1). Next, need to go to microwave (1) heat it.
OK.
go to microwave 1
The microwave is closed.
heat apple with microwave You heat the apple using the microwave Done
I have heated the desired apple (1),
We note that in WebShop, whether to call a skill depends on the environment feedback. Therefore, the base policy needs to act based on environment observation at the end of each skill. For example, if the current item has available attribute to select, then the base policy will not call the select ilem altributes() skill. Otherwise will first select attributes then purchase the item. Below the prompt for the base policy.
WebShop: Base policy prompt
Interact with webshop solve task.
You can call subprocedures search_item() select_ item() stt.:tiiteeattributeettttte purchase_itent Here are some examples.
Webshop
Instruction:
would like a ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
Action: think[I need to firat search for available items, then select the item that matches the instruction. If there are additional options for certain attributes, I also need to select them before purchase. Otherwise I can directly purchase the item.] Observation:
Action: search_ (rrchhitem()
Observation:
Subprocedure Done have found list of items.]
Current Page:
[Back to Search]
Page 1 (Total results: 50)
[Next >]
[B078GWRC1]]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Qunce
\$10.99
[B078CTXVXY]
Ginger Fresh Deodorant by Earth Mama | Natural Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$$10.99
[B08KBVJ4XN]
Barrel and Dak Aluminum-Free Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Odor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
\$15.95
Action: select_iteml
Observation:
Subprocedure Done [I have selected the required item and it matches the instruction.]
Current Page:
[Back to Search]
[< Prev]
40

---

Published as a conference paper at COLM 2024
[assorted scents] [bright citrus] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack of 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
[Dscriptiio]] [Features]
[Reviews] [Buy Now]
Action: delect_item, (ttrittttmarbbuueee
Observation: Subprocedure Done have selected the additional attributes. ]]
Current Page: You have clicked 3 ounce (pack of 1).
Action: purchase_item()
WebShop
Instruction:
need living furniture, and price lower than 100.00 dollars
Action: think[I need to first search for available items, then select the item that matches the instruction. If there are additional options for certain attributes, I also need to select them before purchase. Otherwise I can directly purchase the item.] Observation:
Action: nearch_item/
Observation:
Subprocedure Done have found list of items.]
Current Page:
Observation:
[Back to Search]
Page (Total results: 50)
[Next >]
[B09G71YGSV]
Patterned Square Pouf in Black and Gray Boho Decor Bean Bag Bedroom Furniture Storage Ottoman Accent Chair Bean Bag Chair Papasan Chair Living Room Rugs Floor Pillow Bedroom Chair Foot Stool
\$100.0
[BOOUNYEOVE]
TOV Furniture The Gavin Collection Modern Tufted Linen Fabric Upholstered Living Room Sofa, Beige, 85. 25" 34.5" 27.75"
\$100.0
[B07194KS6C]
Hekman Furniture Round Coffee Table
\$100.0
Action: select_itemd
Observation:
Subprocedure Done [I have selected the required item and it matches the instruction.]
Current Page:
[Back to Search]
[< Prev] TOV Furniture The Gavin Collection Modern Tufted Linen Fabric Upholstered Living Room Sofa, Beige, 85. 25" 34.5" 75.75"
Price: \$100.0
Rating: N.A.
41

---

Published as a conference paper at COLM 2024
[Deccripiio]]
[Features]
[Reviews]
[Attributes]
[Buy Now]
Action:
purchase_itemi
Below is a template of the prompt for skill-conditioned policies. Then we provide example with skill select.temm.attributeei
WebShop: skill-conditioned policy prompt template
Interact with a webshop to solve task. Please consider the following tips to solve tasks:
'<< (policy improvement tips} (primitives)
Here are some examples.
(examples)
''(
WebShop: skill-conditioned policy prompt for select.item attemmatribuuts(
Interact with a webshop to solve a task. Please consider the following tips to solve new tasks:
1. You should strictly the exact words of each clickable option without
making any changes.
2. Ensure select the necessary attributes as per the instruction.
3. Pay attention to the product details to enaure they match the instruction.
4. Be aware that color and size options may not always be straightforward and may include additional information codes.
5. Be careful with the case and spacing of words when selecting item
attributes.
6. Make sure to select product that has all the required attributes as per the instruction.
7. Use quotation marks around specific phrases when searching for items get more accurate results.
8. Please only generate following actions in your response: click[Attributa
Example click [38 eu]
click[color]
Example click (browm]
[lickeStyle]
# Example click [powerlite 1785w]
click[Size]
Example click
click[Flavor:
# Example click (plantain chips]
cl click [Features]
Example click[Features
click (Digital Storage Capacity]
Example click[32 gb]
click[Offer Type]
Example (lcksseeen ad-aupported]
click (Dscciiptt))
Example click (Descripttinn)
42

---

Published as a conference paper at COLM 2024
Here examples.
Webshop
Instruction:
would like 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
Your historical interactions:
Action: search(3 ounce bright citrus deodorant sensitive skin] Observation:
[Back to Search]
Page 1 (Total results: 50)
[Next >]
[B078GWRC1]]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[В078CТКVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[B08KBVJ4XN]
Barrel and Oak Aluminum-Free Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Ddor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
$15.95
Action: click [B078CWRC11
Observation:
[Back to Search]
[< Prev]
scent [assorted scents] [bright citrus] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Qunce
Price: $10.99
Rating: N.A.
[escription]
[Features]
[Reviews] [Buy Now]
Your task is: select_attributes]
Action: Shink[For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options "bright citrus' and '3 (pack 1) a and good buy.]
Observation: OK.
Action:
click [bright citrus]
Observation: You clicked bright citrus.
Action:
click[3 ounce (pack of 1)] Observation: You have clicked 3 ounce (pack of 1).
Action: Done [I selected the additional attributes.]
E.2 Code-based Policy
ALFWorld: Skill discovery
You code generator.
43

---

Published as a conference paper at COLM 2024
You based the history of agenta interacting with the environment generate general code solution solve the task.
You have
the following primitive functions:
example: go to table => go('table 1*)
def glocatiion: str):
return "o to location
example: open draver open( 12)
def (pen(ob str): return "open object
example: The cabinet closed isclose ('The cabinet is closed') return True
def isclose (sbervaiobs:::ation:: str):
if "is closed" in observation:
return True
else: return False
example: input_str "cabinet 3, coffeemachine 1,cabinet 5,", object_nam " "coffeemachine" return "coffeemachine 1"
def extract_pattern (ppptterniinpt_stte (input_str, object_name) Using RegEx pattern to find the object_namd folloved by a space and a number
pattern = r'O \d+ (ormec(obeect_naan
match wee.sercc(ppateen. input_str)
If a match is found if match:
return match. group This returns the first match found else:
return "No match found."
def interact, with___tthhh__hhhehhhheeeeeeeeee wttp:/ittttttteeeeeeeeee (actiontonnt(ttti))
env env_lase env_last_obs,
info step step((action)) [action])
erint (esn__lastoos)
return env env_last_obs [0] info won '] [o]
def js_innobject,,n'' env_ltteevvllees check whether an object is in the envrionment observation if object in env env_last_obs: return True return False
example: use desklamp 1 => use u'deskkalmm 1') def use (deskkaam str):
return 'use desklamp
example: close drawer 1 => ('drwerd 1²)
def close (object: str):
return 'close object
example: put spraybottle in/on toilet p't'sspraboottle 1', 'toilet def put str, location: str): return 'put object in/on location
example: clean cloth with sinkbasin clean('cloth 'sinkbasin 1') def clean(bbjett_1: str, sinkbasin: str): return 'clean object_1 with sinkbasin
44

---

Published as a conference paper at COLM 2024
example: take keychain from sofa take ('keychknnniin 'sofa 1 def take (object: str, location: str):
return 'take object from location
example: heat tomato with microwave def heat (object: str, microwave: str): return heeat object with microwave
example: cool pot 1 with fridge def cool (object: str, fridge: str): return ccol object with fridge
Tips:
1. The basic interaction can be in the following format: action go(oocation) env eav_observation - interact_with envirrmmnnt (action)
You extract specific object by: specific.object
object_to_findt
will start showing you the interaction history of task find. You summarize tips based the interaction and generate the function based on the tips and primitives. The function should be in the format of
def You Your implementation here
the object is already with you.
WebShop: Skill discovery
You code generator.
You based on the history agent interacting with environment to generate a general code solution to solve the task.
You have access to the following primitive functions:
Example select def select return ('clich((item__unbl
Example buy() def buy (): return f'click Now]
Example:
click (brown)
click (Flavor)
click (rectangular)
click (Description)
click gb)
def click(feature)
return ('click[(featurr]
def crath__enirromenttcctiin
global env_id
caw.step(env_1d,action
printaaction
printeenv_be)
45

---

Published as a conference paper at COLM 2024
env_obs, reward
def search(instructiion):
#Usage: search item based the instruction
#Example #1:
#Instruction: im looking for a light pink long handle back loofah shower brush, and price lower than 40.00 dollars
#Return: search(light pink long handle back loofah shower brush]
def extract_.equirement (instruction):
#Usage: based on instruction, extract requirements
#Instruction: im looking for light pink long handle back loofah shower brush, and price lower than 40.00 dollars
#Return: list requirement ['easy apply', 'pine tar scented', 'price lower than 50.00 dollars']
def requirement cacttttmmtchhhac
#Usage: Check whether an item discription (not item attributes) satisfy certain requirement
#item_disccel discripisci = JulaJuyo Back Scrubber for Shower, Long Handle Body Bath Brush Showering Loofah Sponge Exfoliating Bathing Lufa Scrub Shower Cleaning Luffa Brush (1 Pack-White)
Stick for Men Women, Nylon Mesh
\$7
#requirement 'light pink' #return Ture/False
def parse__iemm__bberrvattte
oww.mssooerrroooo.ommm intervaaion_into_tie iiciiio__intt_ttt___edee itt.estditttttceeeeeeeeeee (eetetttt__seerratttiee #Usage: parse a string of item into dicticnary.
[B09GLCXXP6] ApexDesk MK Series Children's Height
Adjustable Chair with Study Desk w/Integrated Shelf Drawer (Pink Desk & MK Chair Bundle) V$419.99
[B0184JJR11]] ApexDesk Little Soleil 43" Children's Height Adjustable Study Dask w/ Integrated Shelf & Draver (Desk+Chair Bundle Blue) \$359.99 return ('B09GLCXXP6': 'ApeeDesk MK...', 'B0184JRW1I': 'ApexDesk Little..."
def
wtt.aatiiiieeeiiemmeesssiiiiiooi requirenrr)
#Usage: return list of most relevant attributes you can directly click
based dhen_description
#item_ deterpdescription: string
#requirements: return [attribute#1, attrrbbbttet..attrrbrreee.....
Tipa:
1. basic interaction can in the following format:
env observattion,ewaat
(action)
2. You should focus to generate the code for the interaction after "Your task is:select_item(] is:select, (ilect_item())??
i.e why the Action is executed (after "Your task
3. You need to based on the interaction history before "Your task is ra:select_item() to infer why the Action is executed
4. It is web shopping interface, you are browsing web pages.
5. You alloved code should be able generate action based interact with env with interact_vith antrrrtrtttthheeeeeommmettttttt environment (action)
numpy library: import numpy np
observation and
will start show you the interaction history of task find.
You summarize tipa based the interaction and generate the function based on the tips and primitives. The function should be in the format of
46

---

Published as a conference paper at COLM 2024
def items:
instruction):
dict {{'item number': discription)) instruction: str
# Hints:
1. You may maintain dictionary where key is the item number and the values containing number satified requirementa 2. Never select or click [<Prev] and [Next>]. Do not evenwrite [<Prev] and [Next>] in your code.
return the best match item number among all items. no need to match all requirements in the instruction
We encapsulate the LLM within the specified functions, exposing only the function signature and usage in the aforementioned prompt to the LLM.
Webshop: Skill discovery
def extract_ ierrrat_rrquiimmentttii (instruction
user_ content f"*"
Given instruction, you decompose and find the requirments: For example:
Example #1:
Instruction:
i'm looking for a light pink long handle back loofah shower brush, and price lower than 40.00 dollars
You return:
requirement_1 'light pink' requirement_2 'long handle back loofah'
requirement_3 = 'price lower than 40.00 dollars'
Example #2:
Instruction:
need easy apply pine tar scented mustache wax stick for men, and price lower than 50.00 dollars
You return:
{{'requirement__] : 'easy apply' 'requirement__ _2 'pine tar scented' "requirement33' 3²; ''pice lower than 50.00 dollars'}]
Now given new instruction:
(instruction)
You return:
res, prerr./ppnaa.aoooldeeer.rraaaarr return res
def eeguurreeent_macch... wattemmttcccccciitttt.. cieeccictt,rrequuurenntnt
segguirenent_emmedddeeee
get_embedding (requirement)
item_ stee_embedddig get_ embedding(item)
score seciine_similaitty erequirement__mbedddiee ittiiitemmmaeeee (tee_mmbeddinng)
print("simlees) acore", scoree if score>0.8. 8:
return True else: return False
def http::/wwtt....tttooe...eeeeeeee
47

---

Published as a conference paper at COLM 2024
user_content n** For example:
Example #1:
Given stem_description:
[citrus & spicel [essential [extra firm hold] [lime sage] [mountain fresh] [original blend] [pine tarl [unscented]
Mountaineer Brand Stache Stick All-Natural Convenient 1.5 Oz Mustache Wax Stick for Men (Extra Firm Hold)
Price: \$16.99
Rating: N.A.
[Dscriptiion] [Features]
[Reviews]
[Attributes]
[Buy Now]
And requirements:
requirement_1 'light pink'
requirement_2 'long handle back loofah' requirement_3 'price lower than 40 dollars'
You return: [[pine tar]]
Example #2:
Given stem_description:
[2 pack-beigel [2 pack-green] [2 pack-light pink] [2
pcs-green+beigel [2 pcs=greentpink] [3 pack-beige] pack-light pink] pca=colors] [beige] [green] [light pink]
Prozklves Back Scrubber for Shower, Long Handle Back Loofah Shower Brush, Soft Nylon Mesh Back Cleaner Washer, Bath Brush for Women Men, Exfoliating Body Scrubber for Elderly (Light Pink)
Price: \$7.99
Rating: A.A.
[escription]
[Features]
[Reviews]
[Attributes]
[Buy Now]
And requiremenss:
requirement_1 'light pink'
requirement_2 _ = 'long handle back loofah'
requirement_3 = 'price lower than 40.00 dollars'
You return:
[[light pink]]
E.3 Human-Generated Policy Improvement Tips and Primitives for Each Skill
E.3.1 ALFWorld
ALFWorld: Policy improvement tips and primitives for skill clean(object)
the sinkbasin; it does not work. 2. You directly clean object with sinkbasin by taking the
1. Do not try
corresponding action.
3. Avoid going to irrelevant places when you are not aure about the nert action;
it wastes time and does not help in accomplishing the task.
48

---

Published as a conference paper at COLM 2024
4. Focus on the task hand and avoid unnecessary actions.
Check the contents of location before moving another one.
6. Attempt to perform the task as soon as you have the necessary item and are
in the correct location.
7. You can think about your plan in the first step, and after that you can only select from the folloving actions in your response (please strictly follow the syntax):
go to [receptacle] [id]
# Example:
go to countertop
clean [object] [id] with [receptacle] [id]
Example:
clean apple 1 with sinkbasin 1
ALFWorld: Policy improvement tips and primitives for skill cool(object)
1. Please complete the task with as few stepa as possible.
2. You can think about your plan in the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
go to [receptacle] [id]
Example:
to countertop
cool [object] [id] with [receptacle] [id]
Example:
cool egg with fridge
[id] even when the 3. You can take action cool [object] [id] with [receptacle]
receptacle is closed.
ALFWorld: Policy improvement tips and primitives for skill find(object)
1. Try to execute a different action when you stuck in loop of taking identical actions.
2. Try to search all other locations if the object could not be found in initial likely locations.
3. Remember the location of each object you have seen in order to quickly find
it later.
4. Remember the number of objects you have seen in each location, you can directly go back the location to find another
5. Do not mismatch target obejct with similar object, for example, mug different with a cup.
6. Please complete the task with few steps possible.
7. You think about your plan in the first step, and after that you can only select from following actions in your response (please strictly follow the syntax):
to [receptacle] [id]
# Example:
go countertop open [receptacle] [id]
Example:
open drawer 2
ALFWorld: Policy improvement tips and primitives for skill heat(object)
1. You need to first go to a microwave to heat an object
2. You can heat any object with microwave.
3. Do not open the microwave, directly take the heat operation.
5. Only select from the following actions your reaponse (please strictly follow the syntax):
to [receptacle] [id] Example:
go to countertop heat [object]] [id] with [receptacle] [id]
49

---

Published as a conference paper at COLM 2024
Example: heat egg microwave
ALFWorld: Policy improvement tips and primitives for skill put(object, receptacle)
1. Try to execute a different action when you stuck loop taking identical actions.
2. You can directly put object in/on an occupied receptacle.
3. Please complete the task with few steps possible.
4. You can think about your plan the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
go to [receptacle] [id] Example:
go to countertop
open [receptacle] [id]
# Example:
open drawer 2
put [object] [id] in/on [receptacle] [id]
Example:
put lettuce 1 in/on countertop 1
ALFWorld: Policy improvement tips and primitives for skill take(object)
1. Try to execute different action when you stuck in loop of taking identical actions.
2. can think about your plan the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
take [object] [id] from [receptacle] [id]
Example:
take lettuce from countertop
ALFWorld: Policy improvement tips and primitives for skill uselobject)
1. Try to execute a different action when you stuck in a loop of taking identical actions.
2. Please complete the task with few steps possible.
3. You think about your plan the first step, and after that you only select from the following actions in your response (please strictly follow the syntax):
use [receptacle] [id]
Example:
desklamp
E.3.2
WebShop
WebShop: Policy improvement tips and primitives for skill search item()
Please pay attention to instruction description to find the product that exactly matches with the requirement in the instruction;
1.
2. You ahould pay attention to the current observation to check clickable buttons;
3. When the current selected product does not match the requirement in the instruction well, you should navigate to the results page to check other likely product candidates;
4. When there
product that matches the requirement in the instruction well, you should search again with new query;
5. You should click [Back to Search] before re-searching with a new query;
6. Please only generate following actions in your response:
search [Query]
Example search[3 ounce bright citrus deodorant sensitive skin]
50

---

Published as a conference paper at COLM 2024
click[Back to Search] Example click [Back Search]
WebShop: Policy improvement tips and primitives for skill select.item0
1. Please pay attention to instruction description to find the product that exactly matches with the requirement the instruction;
2. You can click [Dsccrppoti or click [Featurre to find the detailed description of product;
3. You should pay attention to the current observation to check clickable buttons;
4. When the current selected product does not match the requirement the instruction well, you should navigate to the results page to check other likely product candidates;
5. Please only generate following actions in your response:
click[Item ID]
Example click (B005B0004TM
click[< Prev]
# Example click[< Prev]
WebShop: Policy improvement tips and primitives for skill select.item item itattttttmmmatttttii attttiiiiiutes))
1, Please pay attention to instruction description to find the product that exactly matches with the requirement in the instruction;
2. You can click [Description] to find the detailed description of product;
3. Remember select all required attributes using click(Attributes] before click [Buy Now];
4. You should click (Descriptp) or click [Features] to double-check attributes when there is clickable attributes;
5. You should pay attention to the current observation check clickable buttons;
6. Please only generate following actions in your response: click[Attributed
Example 1: click[16.9 fl (pack of 1)]
Example click[blue]
click[< Prev]
# Example: click[< Prev]
cl click (Descriptio]]
Example: click [Description]
click
Example: click [Features]
WebShop: Policy improvement tips and primitives for skill purchase_itens item()
Please only generate following actions your response:
click Now]
# Example click [Buy Now]
E.4 LLM-Distilled Policy Improvement Tips and Primitives for Each Skill
E.4.1 ALFWorld
ALFWorld: Policy improvement tips and primitives for skill clean(object)
1. Do not try to turn on the sinkbasin; it does not work.
2. You can directly clean an object with a sinkbasin by taking the corresponding action.
3. Avoid going irrelevant places when you
wastes time and does not help accomplishing the task. about the action;
4. Focus on the task at hand and avoid unnecessary actions.
5. Check the contents of location before moving to another one.
51

---

Published as a conference paper at COLM 2024
you have the necessary item and 7. You can think about your plan in the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
6. Attempt to perform the task correct location.
go to [receptacle] [id]
Example:
go to countertop
clean [object] [id] with [receptacie] [id]
Example:
clean apple 1 with sinkbasin
ALFWorld: Policy improvement tips and primitives for skill cool(object)
1. Always remember to locate and pick up the object first before trying to perform an action on it.
2. An object can be cooled using fridge. action.
3. You directly cool object with fridge by taking the correspending
4. Avoid getting stuck in a loop of actions that do not contribute to the completion of the task. 5. Explore the environment thoroughly to locate the object needed complete the task.
6. If an object is not its usual place, check other possible locations.
7. If you can't find an object in one location, it might be in another. In this case, the lettuce was the countertop, not the fridge.
the given environment. 9. can think about your plan the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
8. Avoid attempting actions that not possible
go to [receptacle] [id] Example:
go to countertop
cool [object] [id] with [receptacie] [id]
# Example:
cool egg with fridge
ALFWorld: Policy improvement tips and primitives for skill find(object)
1. Thoroughly check each location before moving on to the next.
2. Do not pick up unnecessary items.
3. Do not waste time checking locations where the item is unlikely to be found.
4. Check all possible locations when looking for item.
5. Don't asaume item won't be certain location based where similar
itema were found.
6. If you can't find an item, try to revisit the locations you have already checked. The item might be hidden or covered by other items.
7. Do not exclude any locations when searching for item, if they unlikely.
8. Expand your search area when you can't find an item in the expected
locations.
9. Remember that items can be found in unexpected places.
10. Avoid trying to take item from location where it has not been found.
11. Revisit locations where similar items were found as they might contain the
item you are looking for.
12. Always check all possible locations, even if they seem unlikely to contain the item.
13. Remember check the dining table when looking for items like remote
control.
14. You can think about your plan in the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
go to [receptacle] [id]
52

---

Published as a conference paper at COLM 2024
Example:
go to countertop
open [receptacle] [id]
# Example:
open drawer 2
ALFWorld: Policy improvement tips and primitives for skill heat(object)
1. Try to execute a different action when you get stuck in loop of taking identical actions.
2. Ensure you have the correct object in your possession before attempting perform actions on it.
3. Check all possible locations if you can't find an object in the expected places.
4. You can directly heat object with microwave by taking the corresponding
action.
5. If an action does not work, try a different approach or command.
6. Focus on relevant locations to complete the task efficiently.
7. A cup is not the same as mug. Make sure to find and use the correct object for the task.
8. can think about your plan the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
go to [receptacle] [id] Example:
go to countertop
heat [object] [id] with [receptacie] [id]
# Example:
heat egg with microwave
ALFWorld: Policy improvement tips and primitives for skill put(object, receptacle)
1. When looking for an object, make sure to check all possible locations.
2. If object is not in the expected location, it may be someshere else in
the
loop unsuccessful actions. an action doeen't work, try a different approach.
3. Avoid getting stuck
4. Not all surfaces may be suitable for placing objects. If one doesn't work, try another.
5. Focus on the task at hand and avoid unnecessary actions, such as searching through irrelevant locations or performing unnecessary actions.
6. Use the correct preposition when placing an object on a surface. The correct action is put the object "in/on" the surface, not just "on" the surface. modify the object it is necessary for the task. 8. When placing an object on a surface, make sure the surface not already full. If it is, try a different surface.
7. Do not attempt
9. After picking up an object, perform the necessary action immediately instead
of moving around unnecessarily.
10. you already have
object needed for the task, not try to find more
the same object.
11. Place the object in the first available and suitable location instead of unnecessarily checking other locations.
12. Use the correct command to put object in cabinet. The correct command "put [object] in/on [surface]",
13. After cleaning an object, if the task is to put it in a draver, do it
immediately instead of checking other drawers.
14. If you put an object in device (like microwave), remember take it out before trying put in another location.
15. Do not attempt to put an object on top the fridge. The correct action to put the object "in/on" the fridge.
16. Do not attempt to put an object on top of other objects surface. The correct action is put the object directly "in/on" the surface.
53

---

Published as a conference paper at COLM 2024
17. You think about your plan in the first step, and after that you only select from the following actions your response (please strictly follow the syntax):
to [receptacle] [id]
# Example:
go to countertop
open [receptacle] [id]
Example:
open drawer 2
put [object] [id] in/on [receptacle] [id]
Example:
put lettuce 1 in/on countertop
ALFWorld: Policy improvement tips and primitives for skill takelobject)
1. Ensure object you want to take actually location you are trying to take it from.
2. Do not get stuck in loop of trying to take an object from location where it is not present.
3. Avoid unnecessary actions that do contribute the completion of the task.
4. Be aware of your current location before attempting to take an object.
5. Explore different locations if you cannot find the object in the initial locations you check.
6. If you can't find an object, try look for in other possible locations.
7. Don't waste time checking empty or irrelevant locations. Focus on the locations where the object is likely to be found.
8. Pay attention to the task description and take the correct object. Don't confuse similar objects. Make the task description.
take the correct object per
10. Check all possible locations before attempting to take an object.
11. Avoid unnecessary actions that are not related to the task, such as trying put object various shelves when the task is to take object. also be found sofas, 13. You can think about your plan in the first step, after that you can
12. Laptops
just tables
drawers.
only select from the following actions in your response (please strictly follow the syntax):
take [object] [id] from [receptacle] [id]
Example:
take lettuce from countertop
ALFWorld: Policy improvement tips and primitives for skill use(object)
1. Stay focused on the task and avoid unnecessary actions that are not related
to the task.
2. Some objects can be used directly without needing to be moved or interacted with other objects.
likely not necessary for task. Try 3. If action doesn't work,
different approach.
4. Avoid getting stuck in a loop of unproductive actions. If an action doean't produce result, it's unlikely to do so with repeated attempts.
5. Don't asaume that task requires complex actions interactions between objects. Sometimes,
solution straightforvard.
6. You can think about your plan in the first step, and after that you can only select from the following actions in your response (please strictly follow the syntax):
[receptacle] [id]
Example:
use desklamp
E.4.2 WebShop
54

---

Published as a conference paper at COLM 2024
WebShop: Policy improvement tips and primitives for skill purchase item(
1. You should strictly use the exact words of each clickable option without making any changes.
2. Always remember select the attributes of item before purchasing it.
correct color, size, style options as per task instruction before purchasing an item.
3. Ensure select
4. Do not rush to purchase an item without selecting its attributes.
5. Carefully read and understand the task instruction before making selection.
6. Always ensure that the options you want to select are available on the page before attempting to click on them.
7. Always select all the necessary attributes of an item as per the task instruction before purchasing it.
8. Always consider the price limit the instruction when selecting product.
9. Use simplified and more general terms when searching for items to get more accurate results.
10. Please only generate following actions in your response:
click[Buy Now]
Example click[Buy Now]
WebShop: Policy improvement tips and primitives for skill search_item()
1. Use specific search terms that include all the essential details of the item you are looking for.
2. Use the exact terminology when performing a search get the accurate results.
3. Always include all the necessary details in the search query to get the most accurate results.
4. Use quotation marks around specific search terms get the accurate search results.
5. Avoid being overly specific in your search terms as it may limit the search reaults.
6. Please only generate following actions in your response: search [Quer]]
Example search[3 ounce bright citrus deodorant sensitive skin]
click[Back to Search]
# Example click [Back to Search]
WebShop: Policy improvement tips and primitives for skill select item attributes()
1. You ahould strictly the exact words of each clickable option without making any changes.
2. Ensure to select all the necessary attributes as per the instruction.
3. Pay attention to the product details to ensure they match the instruction.
4. Be aware that color and size options may not always be straightforward and may include additional information codes.
5. Be careful with the case and spacing of the words when selecting item
attributes.
6. Make sure to select product that has all the required attributes per the instruction.
7. quotation marks around specific phrases when searching for items to get more accurate results.
8. Please only generate following actions in your response:
click [Attrbbutt]
Example [3ick[38 eu] click[Color]
# Example click [browm]
click[Style]
Example click (powerltte 1785w]
click[Size]
Example click[8.5)
[Flavo]
55

---

Published as a conference paper at COLM 2024
Example click(plantain chips]
click (Features]
Example click [Features]
cl click (Digital Storage Capacity]
# Example click[32 gb]
click[fffer Type]
Example click (lockscreen ad-supported]
click (Description
# Example click (Description]
WebShop: Policy improvement tips and primitives for skill select.itemt item
1. You should strictly use the exact words of each clickable option without making any changes.
2. When searching for items with specific characteristics,
quotation marks
around the specific characteristic narrow down the search results.
3. Review multiple items before making a selection to ensure the chosen item best fits the given criteria.
4. After finding the desired item, make sure to complete the task selecting
the item.
5. Avoid over-specifying the search terms to get more relevant results.
6. Always ensure to complete the task as per the instructions given.
7. Pay attention to the item details in the search reaults to ensure they match the given criteria.
given criteria. 9. Include all the criteria in the search query to narrow down the search
8. Ensure the selected item strictly meets
results and find the most suitable item more efficiently.
10. Confirm the specific characteristics of the item by checking the item details.
price limit selecting an item. 12. After clicking on an item, make sure to select it to complete the task.
11. Always consider
13. Please only generate following actions in your response:
click[Iten ID]
Example click [B0544400T]
click[< Prev]
Example click[< Prev]
E.5 Ablation Study and Discussion on Policy Improvement Tips
E.5.1 Tip Distillation Only (TD-Only) Prompts
ALFWorld:
Contrastive tip distillation prompt
histories in which you placed an environment and given a task to complete. By comparing the two histories, please find the mistakes in the failure trial, and summarize general tips to fix the mistakes. The tips can also be about the rules in this particular environment. You will need these tips later solve similar task. When there already list tips to start with, please keep them your response or improve them, and you can also append new tips according to the given new trials. You should not add duplicate tips. Give your response after "New summarization:
You will be given both failure and
Here is one example:
Task 1:
Failure trial:
You are the middle room. Looking quickly around you, you see cabinet
10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet
4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop diningtable 2, diningtable 1, draver 3, drawer 2, drawer 1, fridge garbagecan 1, microwave 1, sinkbasin
2, countertop
56

---

Published as a conference paper at COLM 2024
stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1, and toaster 1.
Your task is to: put a hot mug on a countertpp.
to countertop you see butterknife cup 1, fork 2, and mug 1.
Dn the countertop take mug from countertop
You pick up the mug 1 from the countertop 1.
go to stove burner
On the stoveburner 1, see pan 2. heat mug with stoveburner Nothing happens.
heat mug with stoveburner Nothing happens.
Success trial:
You are the middle room. Looking quickly around you, you see cabinet
10, cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, cabinet
4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop diningtable 2, diningtable 1, draver 3, drawer garbagecan 1, microwave 1, sinkbasin
2, countertop
2, drawer 1, fridge
stoveburner 4, stoveburner 3, stoveburner 2, a stoveburner 1, a toaster
1.
Your task is to: put hot mug countertop.
go to countertop countertop 1, you see a butterknife cup 1, fork 2, and mug 1, take mug from countertop
You pick up the mug 1 from the countertop 1.
go to microwave 1
You arrive at loc 22. The microwave is closed.
heat mug with microwave
You heat the mug 1 using the microwave 1. go to countertop
On the countertop put mug in/on countertop
you
butterknife cup 1, fork 2.
You put the mug in/on the countertop
New summarization: task 1, made mistake heating mug with stoveburner, and got stuck loop where I continually did this. As shown in the success trial, the correct way is to heat a mug with a microwave. Thus, should try different actions when getting stuck in loop; I should use a microwave heat mug, and directly heat object with microwave without opening microwave and putting the object in it.
Tips:
Try execute different action when you get stuck in loop of taking identical actions.
Mug should be heated by a microwave rather than a stoveburner.
You can directly heat an object with microwave by taking the corresponding action.
ALFWorld: Non-contrastive tip distillation prompt
You will be given success histories in which you placed in environment and given a task to complete. Please summarize general tips according the success histories. The tips can be the rules in this particular environment.
You will need these tips later to solve similar task. When there is already a list of tips to start with, please keep them in your response improve them, and you also append have to add new tips there is no one. You should not duplicate tips. Give your response after "New summarization:
tips according the given trials. You do not
57

---

Published as a conference paper at COLM 2024
Here is example:
Task 1:
Success trial:
You are in the middle of room. Looking quickly around you, you cabinet cabinet cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop
10, cabinet 9, cabinet 8, cabinet 7, cabinet
2, a countertop 1, diningtable 2, a diningtable 1, a drawer 3, a drawer
2, drawer 1, fridge 1, garbagecan 1, a microwave 1, sinkbasin 1, stoveburner 4, stoveburner 3, stoveburner 2, stoveburner 1, and toaster
1.
Your task to: put hot mug on a countertop.
to countertop
On the countertop 1, you see butterkmife cup 1, fork 2, and mug 1. take mug from countertop
You pick up the mug 1 from the countertop 1.
go to mi microwave 1
You arrive at loc 22. The microwave 1 is closed.
heat mug with microwave 1 You heat the mug using the microwave go to countertop
On the countertop 1, you see a butterkmife 1, cup 1, a fork 2.
put mug 1 in/on countertop
You put the mug in/on the countertop
New summarization: In task 1, as shown in the success trial, the correct way is to heat mug with a microwave. Thus, I should use a microwave to heat a mug, and I directly heat object with microwave without opening the microwave and putting the object in it.
Tips:
Mug should be heated by microwave rather than a stoveburner.
You directly heat an object with microwave by taking the corresponding action.
WebShop: Contrastive tip distillation prompt
You will be given both failure success histories in which you were placed in an environment and given a task to complete. By comparing the two histories, please find the mistakes in the failure trial, and summarize general tips to fix the mistakes. The tips environment. You will need these tips later solve similar task. When there already list tips to start with, please keep them your response or improve them, and you can also append new tips according to the given new trials. You should not add duplicate tips. You do not have to add new tips if there is
also be about the rules in this particular
Give your response after "New summarization;
Here is one example:
Task1:
Failure Trial:
Webshop
Instruction:
would like 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lover than 50.00 dollars [Search]
Action: search[3 ounce bright citrus deodorant sensitive skin]
Observation:
[Bakk to Search]
Page (Total results: 50)
[Next >]
[B078GWRC1J]
58

---

Published as a conference paper at COLM 2024
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy Breastfeeding, Contains Organic Calendula 3-Ounce ($10.99
[B078CТКVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
($10.99
[B08KBVJ4XN]
Barrel and Oak Aluminum-Free Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Ddor Protection, Cedar Patchouli Blend, Gentle Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack) \$15.95
Action: click
Observation:
[Back to Search]
[< Prev]
scent [assorted scents] [bright citrus] [calming lavender] [ginger fresh] (simply non-scents]
size [travel (4-pack)] [3 (pack of 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
[escription]
[Features]
[Reviens] [Buy Now]
Action: click[BRIGHT CITRUS]
Observation: Invalid action!
Action:
CITRUS] Observation: Invalid action!
Action:
Click[BRIGHT CITRUS] Observation: Invalid action!
Success Trial:
Webshop
Instruction:
would like ounce bottle of bright citrus deodorant sensitive skin, and price lower than 50.00 dollars [Search]
Action: search
bright citrus deodorant sensitive skin]
Observation:
[Back to Search]
Page 1 (Total results: 50)
[Next >]
[B078GWRC1J]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin,
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[B078CTКVXY]
Ginger Fresh Deodorant by Earth Natural Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$$10.99
[B08KBV]4XN]
Earrel and Dak Aluminum-Fred Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Ddor Protection, Cedar Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
59

---

Published as a conference paper at COLM 2024
$$15.95
Action:
click (B078WWRC11
Observation:
[Back to Search]
Prev]
scent [assorted scenta] [bright citrus] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack of 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mana Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
Description]
[Features]
[Reviews] [Buy Now]
Action: (bright citrus]
Observation: You have clicked bright citrus.
Action:
click[3 ounce (pack of 1)] Observation: You have clicked 3 ounce (pack of 1).
Action: click [Buy Now]
New summarization: In task 1, get stuck in loop where I continued taking invalid action [BRGH[ CITRUS]. Thus, I should try different actions when getting stuck scent option [bright citrus], so that (BRIGHT CITRUS] become invalid. As shown in the success trial, the correct action should be click (bright citrus]. Thus, should strictly the exact words of each option without making any changes.
loop; also made a mistake on capitalizing the name of
Tips:
different action when you get stuck in loop taking identical actions.
Try
You should strictly use the exact words of each clickable option without making any changes.
WebShop: Non-contrastive tip distillation prompt
You will be given success histories in which you were placed in an environment and given a task to complete. Please summarize general tips according to the success histories. The tips also be about the rules in this particular environment. You will need these tips later solve similar task. When there already list tips to start with, please keep them your response or improve them, and you can also append new tips according to the given new trials. You should not add duplicate tips. You do not have to add new tips if there is
Give your response after "New summarization;
Here is one example:
Task1:
Success Trial:
Webshop
Instruction:
would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lover than 50.00 dollars [Search]
Action:
search[3 ounce bright citrus deodorant sensitive skin]
60

---

Published as a conference paper at COLM 2024
Observation:
[Back Search]
Page (Total results: 50)
[Next >]
[B078GWRC1J]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[В078СТКVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
\$10.99
[B08KBVJ4XN]
Barrel and Oak Aluminum-Free Deodorant, Deodorant for Men, Essential Dil-Based Scent, 24-Hour Odor Protection, Cedar Patchouli Blend, Gentle Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack) ($15.95
Action: click (B078WWRCCC
Observation:
[Back to Search]
[< Prev]
scent [assorted scents] [bright citrus] [calming lavender] [ginger fresh] [simply non-scents]
size [travel set (4-pack)] [3 ounce (pack 1)] [3-ounce (2-pack)] Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce Price: \$10.99
Rating: N.A.
Description]
[Features]
[Revievs] [Buy Now]
Action: cbightt citrus]
Observation: You have clicked bright citrus.
Action:
click[3 ounce (pack of 1)] Observation: You have clicked ounce (pack of 1).
Action:
click [Buy Now]
New summarization: In task 1, as shown in the success trial, searched by several keywords, selected relevant item and clicked required attributes before placing the order.
Tips:
You should searched with keywords, select the relevant iten to check and click attributes according to the instruction.
E.5.2 Results in ALFWorld
ALFWorld: Distilled policy improvement tips with contrastive prompt
Please consider the following tips to solve new tasks:
1. Try to execute different action when you get stuck in loop of taking identical actions.
2. When stuck loop, try different action or look for the item different location.
3. Mug should be heated by microwave rather than stoveburner.
61

---

Published as a conference paper at COLM 2024
4. You directly heat object with microwave by taking the corresponding action.
5. Remember to heat items in the microwave when required.
6. Use the correct command to place items on the dining table.
7. Use the correct commands to interact with items.
8. Search all possible locations for the required item.
9. Check all possible locations for the required item, not just the most obvious ones.
10. Items can be found in unexpected places like the garbage can or microwave.
11. Remember to cool items in the fridge when required.
12. Remember to clean items when required before placing them their final
location.
13. You can clean items directly with the sinkbasin without needing to turn it
14. You don't need
them together. 15. Sometimes the simplest action is the correct one.
items
16. Perform tasks in the correct order.
ALFWorld: Distilled policy improvement tips with non-contrastive prompt
Please consider the following tips to solve new tasks:
1. Mug should heated by microwave rather than stoveburner. You corresponding action.
directly
object with microwave by taking
Bread can also be heated using a microwave.
2. After heating an object, it can be placed in the fridge for cooling. You can directly cool object with fridge by taking the corresponding action. After cooling cup the fridge,
be placed back in the microsave. 3. Cellphones can found on sidetables and beds. After finding cellphone,
place it in the dresser.
4. Toiletpaper can be found in drawers, not necessarily toiletpaperhangers.
5. Cups be found in microwaves, not necessarily countertops in cabinets.
6. Kettles can be found on stoveburners. After finding the kettles, place them in the cabinet.
7. Keychains can be found shelves. After finding the keychains, place them
in the sidetable.
8. Watches can be found in drawers and on dining tables. After finding the watches, place them on the desired shelf.
9. To clean a dish sponge, plate, or a knife, use the sink basin. After cleaning the dish sponge, the plate, the knife, place it the shelf, countertop, or drawer respectively.
10. To cool a boul or a mug, use the fridge. After cooling the boul or the mug, place it in the cabinet.
11. Vases can be found on shelves. After finding the vase, take it to the desk the desklamp to examine the 12. Eggs can be found in garbage cans. After finding the egg, heat it using the microwave before placing it back in the garbage can.
where the desklamp is located and
13. Apples can be found on countertops or in garbage cans. After finding the apple, cool it using the fridge before placing it in the microwave or heat it using the microwave before placing back in the garbage can.
14. Forks can found on countertops. After finding the fork, place it in the drawer.
15. Spoons can be found countertops. After finding the spoon, clean it using the sink basin before placing it the sidetable.
basin. After finding the lettuce, cool using the fridge before placing it in the garbage can.
16. Lettuce can found
17. Bowls can be found in cabinets. After finding the boul, cool it using the
fridge before placing it in another cabinet.
18. Potatoes
be found dining tables in microwaves. After finding the potato, heat it using the microwave before placing it the fridge or clean using the sink basin before placing it on the sidetable.
62

---

22. Credit cards can be found on sidetables. After finding the credit cards, place them on the dining table. desklamp to examine object without turning it Published as a conference paper at COLM 2024
19. Pillows found beds. After finding the pillow, take it the desk where the desklamp located and turn the desklamp to examine pillow. 20. CDs can be found on dining tables. After finding the CD, place it on the sidetable.
21. Butterknives can be found on countertops in drawers. After finding the butterkmife, clean it using the aink basin before placing or dining table.
the countertop
23. You directly
first.
24. Pots can be found in sink basins. After finding the pot, cool it using the fridge before placing it in the cabinet.
ALFWorld: Human distilled tips and primitives
Please consider the following tips to solve new tasks:
1. Try to execute different action when you stuck loop taking identical actions.
2. Try to search all other locations if the object could not be found in initial likely locations.
order quickly find 4. Remember the number of objects you have seen in each location, so you can directly go back the location to find another one.
3. Remember the location of each object you have later.
5. Carefully check the outcome of action before taking next action.
You directly put object in/on occupied receptacle.
7. You can only take actions to the object current scene.
or the objects the
8. Do not mismatch target obejct with similar object, for example, mug is different with cup.
9. Do not express sorry/apology in your response.
10. Please only generate following actions in your response and strictly follow the syntax:
go [receptacle] [id] Example:
to countertop
open [receptacle] [id]
Example:
open drawer 2
clean [object] [id] with [receptacle] [id]
# Example:
clean lettuce 1 with sinkbasin 1
take [object]] [id] from [receptacle] [ia]
Example:
take lettuce from countertop 1 close [receptacle] [id] Example:
close drawer
heat [object] [id] with [receptacle] [id]
# Example:
heat egg with microwave 1 put [object] [id] in/on [receptacle] [id] Example:
put lettuce 1 in/on countertop 1 use [object] [id]
Example:
desklamp
cool [object] [id] with [receptacle] [id]
Example:
cool egg with fridge
63

---

Published as a conference paper at COLM 2024
E.5.3 Results in WebShop
WebShop: Distilled policy improvement tips with contrastive prompt
Please consider the following tips to solve new tasks:
1. You should strictly the exact words of each clickable option without making any changes.
2. Always ensure to select all the necessary options before making purchase.
3. Pay attention to the specific requirements in the instruction, such as color and size, and make sure to select them correctly.
4. Use the correct search terms to find the desired product. Avoid using
special characters like slashes (/) in the search term.
5. Make sure the product selected meets all the requirements in the instruction.
6. Avoid making assumptions about the available options. Always select from the options provided.
7. Always select the color or any other specific attribute mentioned in the instruction before making a purchase.
8. Ensure to select the correct style or model of the product when multiple options available.
9. Check the product's features to ensure it meets the requirements the instruction before making a purchase.
10. If the search results are not satisfactory, adjust the search terms instead of repeating the same search.
11. Do include the price limit the search Instead, check the price each iten in the search results.
12. If the first product selected does not meet all the requirenents, go back to the search results and select different product.
13. Pay attention to the use of quotation marks in the search term to ensure
accurate results.
14. Only select the options that are specifically mentioned in the instruction. Avoid selecting unnecessary options.
WebShop: Distilled policy improvement tips with non-contrastive prompt
Please consider the following tips to solve new tasks:
1. You should search with specific keywords, including the color or specific feature if required, select the relevant item to check and click required attributes before placing the order.
2. Make sure to select the correct color and feature before purchasing the item.
3. Always check the price of the item to ensure it is within the specified
budget.
4. the first selected item does not meet the requirements, go back the search results and select another item. Repeat this process until a suitable item is found.
5. Ensure select the correct quantity
of the item before
purchasing.
6. Make sure to select the correct size before purchasing the item.
7. When searching for a apecific model of product, include the model name in the search keywords.
select the correct flavor variety when purchasing food items. 9. Check the material the item if is specified in the task.
8. Ensure
10. If the item does not meet the requirements after checking its features, go back and select another item.
11. When searching for wig, include the color and material in the search
keywords.
12. Check the features of the product to ensure it meets the requirements before purchasing.
13. Check the description of the product to ensure it meets the requirements before purchasing.
64

---

Published as a conference paper at COLM 2024
14. Always check the product's features to meets the specific requirements of task.
15. After selecting an item, check its features to ensure it meets the task requirements before purchasing.
WebShop: Human distilled tips and primitives
Please consider the following tips solve tasks:
1. Please pay attention to instruction description to find the product that exactly matches with the requirement in the instruction;
2. You click [Description] to find the detailed description of product; Remember select all required attributes using click(Attributes) before click[Buy Now];
4. You should click (Deacripppi or [lick[Feauure] to double-check attributes when there is clickable attributes;
5. You should pay attention to the current observation to check clickable buttons;
6. When the current selected product does not match the requirement in the instruction well, you should navigate to the reaults page to check other likely product candidates;
product that matches well, you should search again with a new query;
7. When there
requirement in the instruction
8. You should Blick[Back to Search] before re-searching with a new query;
9. Please only generate following actions in your response:
search [Query]
Example: search [noise cancelling cosycost usb microphone]
(lick(Attibbutes]
Example 1: click[16.9 fl (pack of 1)]
Example [lick[blue]
(Product Title]
Example: click (B09720117111
click [Back to Search]
Example: click [Back to Search]
click[< Prev]
Example: click[< Prev]
click (Descrippt)
# Example: click [Description]
[Fetuures
Example: click [Features]
click [Buy Now]
# Example: click (Buy Now]
E.6
Generated Code
E.6.1
ALFWorld
Examples of generated code for each type of tasks under ALFWorld.
ALFworld: Generated skill code
def clean, baean__bjectepppee (fjeciccppcciiccoojjjeet toairriiii_eeeettttt_eleeelll (tslecctttt_cean)) to sinkbasin
action g'issnnbaaiin 1')
env env_observation = interact_ enteeeeroettteeeevrmmeeeeeee (action)
Clean the object with the ainkbasin action poreciffc_objectttoooo env env.observation interact_ with enviromment (action)
"sinkbasin 1')
65

---

Published as a conference paper at COLM 2024
def coo._bbicccsppiiic....eet....... to fridge
action go 1')
env env_observation
Cool the object with the fridge
action cool (pheciifccc_cheeet.too ttoojjectttttttll______ cool, 'fridge 1')
env env_observation interact_ with environment ettentoronnttt (action)
return onv-observation
def oind_objecttobjees (ot. locations_to, to_seachh):
for location in
action golocaiion)
env_observation,
if is maloobecct_to_fing env awv_observation): specific_object.nig
extract_pattern object_to_find)
setrrific_ojecttmiee location
elif isclose Laccosee(ev_obbeevvaiion)
action open
env_observation, interact_withh.nvirooomee (action)
if is is_in(object_to tindddd eav_observation):
specific_.ohect_nnre extract.pattern(enn observvtii,,,,,
object_to_find
return opecific..bject_nanne location
return None, None
def heat_ heett.ojeccsspiiii....oc.......d
# Go to the microwave action goomiicrowave 1')
env_observation interact_with.r hawptteeevvrooonnnnnctttaaaa
# Heat the specific object with the microwave action heat (speciffiii jobeeicc_oojett___ to_heat, 'microwave 1²) env_observation interact_ withhwith__meeee anvironmentaaction
def put.cbbjett(ppccffcc..jeeee Go to the target location action go(specific__taget_.oooo env env_observation, won interact_ cectrratt_itt_heee www..hhevroommettteeeeeomommmo.e (action)
specific_targee_loattiin
If the target location is closed, open it isclose (ea_eeeetoobbeeaaiinn)))) action open (ppetpppccfficttttte__ooaaaa env env_observation, won interact_with envirommett (action)
Put the object the target location
action put env env_observation, won wtt.retttteeiiiimeeeeeeerrtt (action))
locatii)
return whether env return not return
def takke..jjec((@eeifc....eeee..
action take (sppccfic_oojeccttt to_ take, loc_tooc))io) env anv_obaervation interact wich__wttt___eeeee enwironmenit (action) return env env_observation
66

---

Published as a conference paper at COLM 2024
def jee_.bjjet(@peciicc.oeeeee : action iss(speciric_obbeet, to__se) env eav_observation won interact_ return won
(action)
ALFworld: Skill Composition
Based on the historical interaction and the new task description, the
sub-task sequence should be: find(b take (bowl)
clean (bowl)
put(bowl, cabinet 1)
Find the boul
srpeiffc_oojectttoo name, location find_obf_r caadd_object(boooe ["cabinet 6", "cabinet 5", "cabinet 4", "cabinet 3", "cabinet 2", "cabinet 1", "coffeemachine 1", "countertop 3", "countertop 2", "countertop 1", "drawer 3", "drawer 2", "drawer 1", "fridge 1", "garbagecan 1", "microwave 1", "shelf 3", "shelf "shelf 1",
"sinkbasin 1", "stoveburner 4", "stoveburner 3", "stoveburner 2", "stoveburner
1", "toaster 1"])
# Take the bowl
(twpcct(ppccifccccbbbeoooe Clean the bowl clean_objects (specccicccciccjejeeeee Put the boul in cabinet 1 won put_object (speciiicccciii..ojjetttt 'cabinet 1*)
location)
E.6.2 WebShop
In the WebShop, have observed that the GPT-4 model excels in skill composition, effectively generating responses based on instructions and context. In contrast, the GPT-3.55 model exhibits proficiency in task decomposition, breaking tasks into multiple sub-tasks; however, it struggles to seamlessly integrate these skill functions within the broader context. For instance, it cannot effectively pass the output of one function as input subsequent functions.
Webshop: Generated skill code
def select_item (itmms, instruction):
items: dict ('item number': discription) # instruction: str
Extract requirements from instruction requirements
(instructio)
# Initialize dictionary to store the number of satisfied requirements for each item
Snciiffied_rquireeeeen
fitem: for item items. keya
# Check each item against each requirement
for iten, description in (tems.item(()
for requirement in requirements:
requirenent_matche check (descriptoo,, requirement): satisfied_roquurrement [item]
Select the item with the most satisfied requirements
selected _itm max (et.atssfieereeiiieeeeeeeeee beersattiffialleeggiieeeettttt, return selected_ item
def select celeec_itemm_eee http:iitetttttiiieee attrriutesseeeeeeeee (item_ requirements):
67

---

Published as a conference paper at COLM 2024
deeerdeecriptoon: string
requirements: [requirement*#1, hrg.://eewa.aa.ooooooooooooeeeea
# Extract the most relevant attributes from the item description based on the requirements
celevant__atrrburee select_moat_ selert..est..rraraa.....tttttete requirements)
# Return the list of relevant attributes selevant_attributee
Webshop: Skill Composition (GPT-4)
Sub-task sequence based on historical interaction new task description:
1. search_itens item: search for a space-saving ottoman bench to match blue living room
2.
select_item: select the item that's 100r45r45cm and price than
410.00 dollars
select the attributes that match the requirements 4, purchase_item buy the selected item
3,
Start the task
Sub-task 1: search_ites.
instruction "i looking for a space-saving ottoman bench to match my blue living room. pick that one that's 100x45x45cm, and price lower than 410.00 dollars"
action search (instrucii))
env_obs, revard interact_wiee wttpracttttthhenvroooeeee environment (action)
Sub-task 2: select_iten
items parae_iteeee wtw.iiteee_eeeeeeee www.rrationnnaiiieoooeeeoooeeeee intoiiooonnni_oeeeeeeeeeeeeeee sitittttmddaae dict (env_obs)
selected_itee select ielectt_ttemiitmms instruction) action (seectesel env_obs, reward = iiberacttwithheeriirreee (action)
Sub-task bileet_jitem_ttrrbbee
requirements extract__rquiremeet (instruction)
selevant_atttibuttes relevant_ attributes = select_ite iet__item_aaae iet___iie_aarrbbtteeeeee attributes (env requirements)
for attribute in velevant_attribune
action click clich(attrbbute) env_obs, reward interact_vith. wtw.rreitteevvvroeeettcecommm (action)
Sub-task 4: purchase_iten action () env_obs, reward interott.wtthhenvirrmmeett (action)
Print the final reward print(f"Fina) reward: (reward}")
Webshop: Skill Composition (GPT-3.5)
Sub-task sequence:
1. search (haatrcutsttt
2. select_item()
3. select_item__et (
4. purchase_iteml
Code:
search ultta hd motion detection surveillance dome color black size 16pp wdr 2.8 mm, and price lower than 190.00 dollars')
select_itens
68

---

Published as a conference paper at COLM 2024
eeleet..ee..ttrrbueeeeeeeeeeeeee descriptii,, requirements)
purchase_itae __iemm) () select_itemm)
69


Reviews:
[{"review_id": "DQXrRpFrPH", "reviewer": "Reviewer_ubuS", "summary": "This paper proposed a prompting framework that can 1) automatically discover subtasks/skills along with their primitives and preconditions from trajectories, 2) apply discovered skills to new tasks in a hierarchical way. More specifically, for prediction, LM will first figure out the subtasks/skills and then retrieve relevant examples from the constructed database on how to use primitive actions to complete a task. Results based on GPT models on ALFWorld and WebShop demonstrate that it can improve upon ReAct. \n\nThough the general idea and presented results are interesting, the practical value of this work is unclear due to the limited experiment setting (see below).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the valuable feedback from Reviewer ubuS, and address the concerns below.\n\n> Q1: how often LMs fail to find generalisable actions or wrong actions.\n\nIn experiments, we find GPT4 is very reliable in discovery and distillation, with a few strategies we used:\n1. Provide few-shot examples (App. E.1.2), so the model can better understand the goal. \n2. Iteratively run discovery/distillation to keep refining summarized skills/actions/tips, so potential mistakes can be corrected in later iterations.\n3. Trajectory contrasting (compares success and failure trails), so the summarized knowledge is grounded on facts rather than hallucination.\n\n> Q2: what if DistillPrompt provides wrong preconditions/corrections of discovered actions\n\nTo test O3D’s robustness to errors, we run additional experiments by inserting misleading information into the discovered and distilled results.\n- Setting 1: Irrelevant Information. For every skill, we insert an action and a tip from a different skill, which is not useful to the current skill and may be distractors. \n- Setting 2: Misleading Information. For every skill, we insert a wrong action and an irrelevant tip that are not even valid in ALFWorld.\n\nResults: we run GPT-3.5-0613 on ALFWorld, where the scores are:\n- Original O3D: 41%\n- O3D + Irrelevant Info: 49%\n- O3D + Misleading Info: 38%\n\nWe can see that with the irrelevant or misleading info, the agent can still achieve high results (ReAct is 7%). Note that our distilled knowledge is summarized from real trajectories and provides additional context to the basic few-shot examples, so it can be more helpful than harmful. Moreover, Stage 1&2 of O3D are offline, so one can check, verify and modify the learned skills/actions/tips (on held-out validation sets) before deploying them to Stage 3 interactions.\n\n> Q3: It would be better to also provide results on one open-source model.\n\nThank you for the suggestion. We will release the codebase and add open-source models in the final version.\n\n> Q4: The performance of ReAct-Long’s retrieval-based variant.\n\nThis is an interesing point. \n1. ReAct uses manually selected relevant examples as prompts, which is already a retrieval-based baseline method.\n2. It is a great future direction to explore more fine-grained retrieval-based methods for long-horizon interactive tasks, but it is non-trivial and out of our current scope.\n3. O3D can be combined with retrieval methods, similar to how O3D+Reflexion improves Reflexion (Table 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The general idea of discovering skills based on trajectory data is interesting, and this paper reveals the potential of using offline data to discover better system prompt (i.e., with macro/template actions) for embodied tasks.", "reasons_to_reject": "The main components DiscoverPrompt and DistillPrompt all rely on LMs, and there seems no automatic verification to check how good are the distilled skills and so-called tips.  I'm concerned about 1) how often LMs fail to find generalisable actions (e.g., not too specific actions) or even wrong actions, 2) what if DistillPrompt provides wrong preconditions/corrections of discovered actions. Since these skills will serve as system prompt for new tasks, the errors in them will potentially propagate to all predictions.  Overall, I think how to ensure the quality of discovered skills is an important question, but there seems no experiments/discussion on it.", "questions_to_authors": "For reproducibility, it would be better to also provide results on one open-source models. \n\nFor the baseline ReAct-Long, I wonder the performance of its retrieval-based variant --retrieving relevant trajectories based on task descriptions. It's well-known that retrieval is crucial for few-shot in-context learning methods.", "extra_scores": {}}, {"review_id": "WCWawYUdzd", "reviewer": "Reviewer_KPqX", "summary": "The paper addresses the challenge of using Large Language Models (LLMs) for sequential decision-making tasks. Existing approaches often rely on high-quality few-shot demonstrations, but the limited context length of LLMs restricts their ability to process large-scale demonstrations with extended trajectories. To overcome this, the paper proposes a three-stage framework: First, it identifies reusable skills by segmenting offline datasets. Second, it enhances skill-conditioned policies through data distillation. The final stage involves constructing an interactive policy that utilizes the learned skills across various tasks. Experiments conducted on ALFWorld and WebShop demonstrate that this method surpasses traditional few-shot prompting baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the valuable feedback and positive comments from Reviewer KPqX. Below we address the reviewer’s concerns and questions.\n\n> Q1：difficult to determine which component contributes the most.\n\nSec. 4.3 shows an ablation study of each component. Although the impact of all components is task-dependent and model-dependent, dash-dot line in Fig. 4 shows that proposed components generally enhance performance across various tasks and models.\n\n> Q2：The extremely low performance of AdaPlanner is unusual. \n\nThank you for the suggestion. Unfortunately, due to our budgetary constraint and the time limitation, we are unable to include AdaPlanner+GPT-4 at this point. However, we conducted additional experiments using GPT-3.5-0301, \n\n|Method|Pick|Clean|Heat|Cool|Look|Pick2|ALL|\n|---|---|---|---|---|---|---|---|\n|AdaPlanner|78|94|70|94|63|78|81|\n|O3D-code|96|84|91|86|89|88|89|\n\nAdaPlanner achieves reasonable performance as the original paper showed, while O3D-code can still outperform it by 8%. \n\n> Q3: how much data O3D needs and how the quality of O3D affects performance.\n\n1. How much data: We discussed the data amount in App. A.1. We use a small batch size and iterate over multiple batches, and we can see the distilled tips converge only after a few iterations. So O3D does not need much data.\n2. How the quality of O3D affects performance: In experiments, we observe reliable action discovery and tip distillation with the proposed trajectory contrasting method and iterative improvements. Even if the quality of Stage 1&2 is not optimal, the following points are noteworthy:\n- The primitive actions and tips are prepended to the normal prompt with few-shot examples used in ReAct. Since the distilled knowledge is from real trajectories, such additional context would be more helpful than harmful.  \n- One can verify and modify the skills/actions/tips before deploying them to Stage 3, similar to verifying a trained ML model but the learned knowledge is in natural language and more interpretable. Compared to manual prompt design, O3D is more automatic and scalable. \n- We run additional experiments by inserting misleading information into the discovered and distilled results. The score of O3D (41%) does not drop with irrelevant information (49%), while slightly drops (38%) with misleading information, and is still much higher than baseline ReAct (7%) with GPT-3.5-0613. It shows the superior robustness of O3D. More details are in the response to Reviewer ubuS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper proposes an offline planning method, which differs from existing online methods.\n\n(2) The proposed method not only achieves commendable empirical performance but also shows compatibility with existing prompting-based approaches, adding value to current methodologies in LLM-based decision-making.\n\n(3) The paper is well-written.", "reasons_to_reject": "(1) The paper comprises multiple components and stages, making it difficult to determine which component contributes the most.\n\n(2) In Table 9 (Appendix D.2.1), the paper compares the proposed method with Demo2Code and AdaPlanner. The extremely low performance of AdaPlanner is unusual. I suggest adding results for AdaPlanner + GPT-4 to further validate the proposed method.\n\n(3) Since the method uses offline data, it is important to study how much data O3D needs and how the quality of O3D affects performance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OvFi6ADxmF", "reviewer": "Reviewer_JZjV", "summary": "This paper proposes an LLM system that can act in ALFWorld and WebShop interactive environments and improves on top of standard prompting baselines. The main idea stems for hierarchical modeling: by using prompted LLMs: \n- **(1)** they first learn/segment subtasks from human demonstrations, \n- **(2)** then automatically learn primitive actions (go to X, pick N X), \n- **(3)** by using human demonstrations (has both positives and negatives) learn tips for the learned subtasks, \n- **(4)** by using all of above, initial LLM decides what tasks to execute and another LLM executes each subtask.\n\nThe main technical contribution comes from automatic primitive skill learning and providing a way to use negative trajectories in prompting.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the valuable feedback and positive comments from Reviewer JZjV. Below we address the reviewer’s concerns and questions.\n\n> Q1. What is the intuitive reason for worse performance in Table 3(b) comparing to ReACT-long?\n\nTable 3(b) shows that on WebShop, O3D achieves a higher success rate but a lower average score than ReAct-long. This is because the current implementation of O3D distills tips by comparing successful trajectories (score=1, when the bought item exactly matches the instruction) and failed trajectories (score<1, when the purchase failed or when the bought item does not exactly match the instruction). As a result, the distilled tips would guide the agent to achieve an exact match rather than maximizing the raw score. \n\nIn contrast, ReAct-long utilizes the full trajectories with scores, it has a more fine-grained understanding of the relation between tasks and scores. One idea for improving the average score of O3D is to contrast the trajectories with higher scores and lower scores instead of simple successes and failures.\n\n> Q2. Not a reason to reject but it could be exciting to apply this method to more domains.\n\nThank you for the suggestion. We also believe that the method can be applied to a broader range of domains, and we will investigate it in our future work. We will be excited to see follow-up work in various applications as well.\n\n> Q3. Why not use \"interactive decision making\" instead of \"sequential decision making\"?\n\nIt’s an interesting point. Although the experiments are mainly conducted in interactive environments, the method can also be applied to non-interactive decision-making problems (using LLM), thus naming it as interactive decision-making would not be general enough. \n\n\n> Q4. Please add a small summary of the environment details to the main text.\n\nThank you for the suggestion. We will make it more clear in the final version.\n\n> Q5. \"This discrepancy may be attributed to the fact that GPT-3.5-0613 is fine-tuned with data and objectives that are diverged from the ALFWorld domain\", which seems ambiguous and generic. Aren't GPT-4 and GPT3.5 fine-tuned with same data?\n\nWe will clarify this in our final version. Here, we mean that GPT-3.5-0613 sometimes does not understand the objectives of ALFWorld tasks. For example, it sometimes wants to ask the house owner for help, which is not relevant to the ALFWorld domain. However, GPT-4 does a better job of understanding the restrictions of the environment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Learning what abstractions (what kind of hierarchy) to use in sequential decision making is a hard problem, but the authors automatically discovers the abstractions for primitives and subtasks by using LLMs.\n\n2) Their usage of both successfull and unsucessfull trajectories to genearate tips is an alternative to RL and clever in this context.\n\n3) They improve over various baselines (ReACT, Reflexion) for this task.\n\n4) Although the system complex, the paper does good job of ablating all the parts that I thought necessary in section 4.3\n\n5) The writing of text is fluent.", "reasons_to_reject": "1) What is the intuitive reason for worse performance in Table 3(b)\n comparing to ReACT-long? The authors could do more qualitative analysis here.\n\n2) This is not a reason to reject but, in Figure 3, the discovered skills seems simple and easy to infer from data. It could be exciting to apply this method to more domains, see what are the skills that can be learned.", "questions_to_authors": "1) Why not use \"interactive decision making\" instead of \"sequential decision making\" in the title and throughout the paper. I believe interactive part is more important than the sequential part?\n   \n2) Please add a small summary of the environment details to the main text (available actions, observations and states etc)\n\n3) I quote: \"This discrepancy may be attributed to the fact that GPT-3.5-0613 is fine-tuned with data and objectives that are diverged from the ALFWorld domain\". This sentence seems to me very ambigious and generic. Aren't GPT-4 and GPT3.5 fine-tuned with same data?", "extra_scores": {}}, {"review_id": "jPjh5G4KWU", "reviewer": "Reviewer_Fifg", "summary": "This paper proposes an offline learning framework called O3D (Offline Data-driven Discovery and Distillation) that enables large language models (LLMs) to improve their performance on sequential decision-making tasks without fine-tuning.\nThe framework has three stages: a) Skill discovery by segmenting offline trajectories into sub-tasks. b) Knowledge distillation by discovering primitives and extracting policy improvement tips from contrasting successful and failed trajectories. c) Hierarchical policy execution where a base policy sequentially calls the learned skill-conditioned policies. Through iterative skill refinement and knowledge distillation, O3D allows LLMs to benefit from extensive and diverse offline data, improving their few-shot performance on challenging benchmarks like ALFWorld and WebShop. The proposed approach outperforms state-of-the-art baselines and demonstrates the effectiveness of utilizing offline data at scale to enhance LLM capabilities in sequential decision-making without fine-tuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the valuable feedback and positive comments from Reviewer Fifg. Below we address reviewer's concerns and questions.\n\n> Q1. The effectiveness of each stage is not clear to me. \n\nOur Sec. 4.3 provides an ablation study to investigate the contribution of each component (see Fig. 4). Although the impact of each component is both task-dependent and model-dependent, the dash-dot line in Fig. 4 shows that our proposed components generally enhance performance across various tasks and models.\nWe will further clarify these in the final version.\n\n> Q2. How can the quality of the skill discovery be managed? LLMs can hallucinate.\n\nWe adopt several strategies to guarantee the quality of skill discovery: (1) provide offline trajectory data, ask LLM to first segment it into sub-trajectories, then label each sub-trajectory as a skill; (2) provide one demonstration in the prompt (see App. E.1.1) to guide LLM to discover the desired skills in a particular format; (3) iterate over multiple trajectories such that LLMs can refine the discovered skills, making sure the skills are general and non-duplicative.\n\nAs the discovered skills are grounded on factual trajectories, we observe stable and high-quality skill discovery results without hallucination. \n\n\n> Q3. What is the difference between O3D and other self-refine baselines like Reflexion?\n\nMajor differences between O3D and Reflexion are listed below:\n\n1. Reflexion lets the LLM reflect purely based on online interaction history, and it takes multiple interaction trials to summarize tips. However, in practice, online interaction can be expensive or risky. In contrast, O3D learns from offline data. O3D achieves improvement in the first trial of deployment without extra online interactions.\n2. Reflexion learns task-specific textual tips, only working for the current task. For example, the tip “in the next trial, I should put the apple on the countertop” cannot be directly transferred to a new task with a different goal, and such tips have to be re-generated for each downstream task. In contrast, O3D discovers reusable skills and generalizable textual tips. For example, the tip “I should strictly follow the task instruction and put the object in the desired location” can facilitate a wide range of downstream tasks.\n3. Reflexion relies on online failure experiences, however, our approach leverages both success and failure experiences, which is more effective in generating proper tips to fix mistakes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. O3D is a novel exploration aimed at enhancing the sequential decision-making performance of LLM agents through the utilization of large-scale offline data without the need for fine-tuning the model's weights.\n2. It shows the capability to effectively identify and distill generalizable skills and valuable insights that can be applied to unseen downstream tasks.\n3. It can also be utilized to augment these approaches with offline data. This possibility enables the achievement of high-quality policies prior to deployment while also reducing the cost of online interaction.\n4. This paper is well-written and easy to follow.", "reasons_to_reject": "1. The effectiveness of each stage is not clear to me. Further analysis of the design and experiments should be considered. \n2. Following the previous question, how can the quality of the skill discovery be managed? LLMs can hallucinate.\n3. What is the difference between O3D and other self-refine baselines like Reflexion?", "questions_to_authors": "See weakness", "extra_scores": {"ethics_concerns_details": "N/A"}}]

Author rebuttal:

Meta-review: This paper describes a method for discovering reusable skills from offline demonstrations, using an LLM to parse and segment demonstrations, then distilling each segmented-out skill into a dedicated policy. Reviewers agree that the proposed method is interesting and complementary to current work on deliberation methods for "LLM agents", and is convincingly evaluated. I think it will be a great fit for the conference.

One thought I had while reading the paper & reviews was that it might be nice to also include some discussion of pre-giant-LM methods that used language as a scaffold for skill learning, e.g. https://arxiv.org/abs/2110.07342, https://arxiv.org/abs/2110.01517, https://arxiv.org/abs/1903.01973.
Decision: Accept