O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 21:19:30

Abstract: Recent advancements in large language models (LLMs) have exhibited promising performance in solving sequential decision-making problems. By imitating few-shot examples provided in the prompts (i.e., in-context learning), an LLM agent can interact with an external environment and complete given tasks without additional training. However, such few-shot examples are often insufficient to generate high-quality solutions for complex and long-horizon tasks, while the limited context length cannot consume larger-scale demonstrations with long interaction horizons. To this end, we propose an offline learning framework that utilizes offline data at scale (e.g, logs of human interactions) to improve LLM-powered policies without fine-tuning. The proposed method O3D (Offline Data-driven Discovery and Distillation) automatically discovers reusable skills and distills generalizable knowledge across multiple tasks based on offline interaction data, advancing the capability of solving downstream tasks. Empirical results under two interactive decision-making benchmarks (ALFWorld and WebShop) verify that O3D can notably enhance the decision-making capabilities of LLMs through the offline discovery and distillation process, and consistently outperform baselines across various LLMs.

Main paper: 
 introduction: 
 Recent years have witnessed remarkable advancements in artificial intelligence (AI), particularly in the development of Large Language Models (LLMs). One of the standout features of LLMs is their in-context learning ability, where the LLM can perform tasks with only a few-shot examples provided in the prompts, making it possible to deploy LLMs to various applications seamlessly.Although most existing research focuses on one-step text generation such as question answering, many real-world scenarios desire autonomous agents that can interact with external environments and make sequential decisions to complete given tasks. There are some recent works that successfully showcase the application of LLMs in sequential decision-making (Yao et al., 2023b; Shinn et al., 2023; Liu et al., 2023c; Yang et al., 2023) , by either directly letting the language model interact with the environment, or using LLMs to write code which then executes in the environment. With a few examples of acting and reasoning (Yao et al., 2023b) , an LLM-based agent can interact with the environment and even learn by reflecting on historical errors (Shinn et al., 2023) .However, existing methods still struggle to solve many complex domains with LLMs due to the intrinsic difficulties that arise from long-horizon interactive tasks. On the one hand, it is widely known that the task complexity increases exponentially with the interaction horizon (Sutton & Barto, 2018) , such that a large amount of data or demonstrations can be desired for an agent to fully understand the environment dynamics, especially for heterogeneous real-world environments and tasks, where cross-task generalizability is important. On the other hand, the in-context learning ability is constrained by the limited context window of an LLM. Even if many demonstrations exist, it is hard to prompt LLMs with sufficient examples. Although finetuning is a solution, it can be much more expensive and less accessible for normal users. Your task is: put an apple on sidetable.On countertop 1, you see an apple 3.You pick up the apple 3. In response to these challenges, this paper asks and aims to answer the follow question:Can we develop a data-driven learning framework for sequential decision-making with LLMs, such that LLMs can learn from large-scale offline data without the need of model training?In this paper, we define an offline learning framework to enable LLMs to discover and distill useful knowledge from interaction trajectories on multiple tasks. We first formalize LLM-powered policies that unify the two parallel approaches in literature which generate text and code as policies, respectively. For these LLM-powered policies, we carefully design a generic learning paradigm called Offline Data-driven Discovery and Distillation (O3D), which can iterate over the offline dataset and keep improving the policy. Importantly, our method does not require a high-quality expert offline dataset, as it can benefit from both positive examples and negative examples of environment interactions, making the framework easier and cheaper to use in practice. As shown in Figure 1 , O3D is composed of 3 stages: the first stage aims to discover reusable skills by segmenting the offline interaction trajectories; the second stage then conducts skill-conditioned policy improvement by distilling knowledge from offline data; the third stage constructs the interactive policy by calling the learned skills given diverse tasks. All stages are based on querying LLMs and iterating over existing text-or code-based policies. As a result, O3D can learn better policies from offline dataset at scale without any model finetuning. Experiments in two commonly used domains (ALFWorld and WebShop) show that our LLM agent augmented by offline knowledge has much better few-shot performance than prior methods on a variety of downstream tasks.We establish the first offline in-context learning framework for LLM sequential decision-making agents, such that the LLM can learn from offline experience without any finetuning. This offline learning paradigm allows more effective usage of past interactions (including both good and bad behaviors) generated by human or other agents, alleviating the cost of online learning.(2) Our offline learning algorithm unifies two common approaches of LLM-based decision-making: textual action generation and code generation. For both approaches, our method achieves significant improvement over baseline methods on challenging domains.(3) Different from prior work which prompts and solves different types of tasks independently, our algorithm can leverage offline experience from multiple tasks. By letting LLMs automatically distill shared high-level knowledge, our algorithm achieves few-shot generalization to various types of tasks with a single set of prompts. 
related work: 
 LLMs for sequential decision-making with in-context learning.• Text-based methods. Yang et al. (2023) and Liu et al. (2023b) conduct extensive experiments to showcase the ability of LLMs to make sequential decisions in a range of challenging multi-step reasoning tasks. Yao et al. (2023b) propose ReAct, a method to combine multi-step reasoning and interactive decision-making, which achieves significant improvement in multiple domains. Shinn et al. (2023) develop agents that can verbally summarize from its past failures and incorporate the reflective text into subsequent trials of the same task, analogous to an online reinforcement learning paradigm. Our method, in contrast, adopts an offline learning method to minimize the cost of online learning and can adapt to various new tasks.• Code-based methods. It has been shown in multiple domains that embodied agents can be empowered by LLM-written code Liang et al. (2022) ; Wang et al. (2023a) . Sun et al. (2023) propose a method to refine the code-based plan during interaction rather than executing in a fixed loop. Liu et al. (2023c) propose to extract hierarchical summary from robot past experiences to adjust the plan. A concurrent work by Wang et al. (2023b) proposes a framework that generates robot task code from demonstrations via recursive task summarization. But our O3D differs from these methods as 1) O3D can iteratively improve the policy by utilizing offline data at scale, 2) O3D takes a bottom-up approach rather than a top-down recursive way to decompose the task, and 3) O3D works for both text-based policies and code-based policies.• Combining LLMs with existing planning approaches. It has also been shown that LLMs can be combined with classical planning approaches, such as Planning Domain Definition Language (PDDL) (Liu et al., 2023a; Silver et al., 2023) . Differently, our paper focuses on end-to-end LLM policies without additional planning algorithm or knowledge.Training or fine-tuning LLMs for sequential decision-making. Another line of related work includes training textual policies with imitation learning (Shridhar et al., 2021; 2020) and fine-tuning language models to behave as policies (Wang et al., 2023a; Driess et al., 2023; Wang et al., 2023c) in sequential decision-making. Again, our work is different as it aims at achieving high-quality LLM-based decision-making without fine-tuning.Multi-step reasoning and task decomposition with LLMs. Multi-step reasoning using LLMs has been widely studied in language domains, including chain-of-thought style reasoning Yao et al. Wang et al. (2022) . In contrast, we focus on sequential decision-making tasks in partially observable environments, where each step results in state transitions and only sparse rewards are available. 
problem formulation: 
 LLM-powered Policy. We first formally define an LLM-powered policy. A policy ⇡ for sequential decision-making is a function that maps the interaction history to a distribution over actions. Let ⇡(a|⌧ ) denote the probability of selection action a given interaction history ⌧ , which is a sequence of all past observations and actions, ho1 , a 1 , o 2 , a 2 , • • • , o t i.Then, with a pre-trained LLM, a policy can be realized in the following two ways.• Text-based-Policy. With a pre-trained LLM which outputs text based on any text input, a policy can be written as ⇡ text (a|⌧ ) := LLM (a|⌧ ; ✓ pmt , ✓ pret ).(1)• Code-based-Policy. It is well-known that LLMs can program, such that one can ask LLM to directly generate code to implement the policy function, i.e.,EQUATION)The goal is to learn a policy that can maximize the total reward. In both the above cases, the pretrained LLM weights ✓ pret are fixed, and our O3D learns a policy by learning and optimizing the base prompt ✓ pmt as well the written policy code Code(a|⌧ ) from offline data.Skill-conditioned Policy. A policy can be conditioned on specific skills or subgoals (e.g., find a mug), which are compositional factors of the original task (e.g., heat some milk). Let z be a skill, then a skill-conditional policy can be denoted as ⇡ z , with ⇡ z (a|⌧ ) the probability of selecting action a given history ⌧ when executing skill z.In many real-world decision-making systems, there exists interaction log from various users, including experts who can successfully perform the task, as well as non-experts who may fail and make mistakes.Our goal is to learn a good "policy" defined in Section 3.1 by utilizing the offline dataset to learn the base prompt, but having the model weights, ✓ pret , fixed.Intuitively, seeing the interaction logs from others performing a task can be helpful for one to understand the environment and finish similar tasks. Since LLMs have strong abilities of interpretation and generalization, recent works such as ReAct (Yao et al., 2023b) have shown that LLMs can solve many interactive decision-making problems when prompted with a few expert demonstrations. This learning paradigm is analogous to behavior cloning, where an agent learns to imitate how experts react to certain scenarios. However, traditional behavior cloning suffers from the distribution shift between expert demonstrations and the agent's own online interactions, especially when the expert dataset is small and not representative of all scenarios in the domain. Although LLMs are powerful at interpolating and generalizing with the pre-trained language understanding ability, their fixed context length only allows a limited number of expert demonstrations, making it hard to fully understand an external decision-making environment with specific dynamics and requirements. That is, even when there exist a rich and diverse offline dataset, in-context behavior cloning (Yao et al., 2023b ) is only able to utilize a small subset of the data and obtain sub-optimal policies. To overcome this issue, we introduce an offline learning framework for LLM-powered policies, including both Text-based-Policy and Code-based-Policy defined in Section 3.1.Our proposed offline policy learning framework consists of 3 stages, as depicted in Figure 1 . The first stage enables the LLM to discover and abstract reusable skills from offline datasets (potentially from diverse tasks). Then, the second stage aims to learn a skill-based policy for each discovered skill, through iterative discovery and primitives and iterative policy improvement with knowledge distillation. The final stage is to construct the main LLM-powered agent who can reason and call corresponding skills sequentially to solve given tasks. Below we explain each stage in detail.Stage 1: Offline Skill Discovery and Data Segmentation. Many real-world decision-making processes requires a number of steps to complete a task, such as controlling a robot to pass several obstacles and navigate to the door, which results in two challenges for LLM-powered agents. First, the limited context length may not be enough to contain the few-shot demonstration and online interaction history. Second, the language model may lose track of its goal and not pay attention to the most important information. To mitigate this issue, we propose a hierarchical policy learning framework that can iteratively extracts skills from interaction logs with primitive-level executions.Here the skills are analogous to the options or temporally extended actions (Sutton et al., 1999) in hierarchical reinforcement learning. It is well-known that discovering options is difficult in traditional RL, whereas we find that skill discovery with textual logs can be well-achieved with the semantic understanding ability of LLMs.Our skill discovery process iterates over the offline trajectories, using a DiscoverPrompt as shown in Figure 1 (Stage 1). The full prompt we use is in Appendix B. We ask LLMs to divide the interaction histories into skill-oriented sub-trajectories, and abstract the skills in function forms. For example, from all 6 types of ALFWorld (Shridhar et al., 2021) tasks, the LLM can reliably discover 7 skills: find(object), take(object), clean(object), heat(object), cool(object), use(object) and put(object, receptable), covering all the required sub-procedures in the domain and are reusable across tasks.Stage 2: Offline Policy Improvement with Knowledge Distillation. The main idea of this stage is to distill generalizable knowledge from offline datasets, such that the knowledge can improve the policy's performance in downstream tasks. Such knowledge should be generalizable to tolerate the distribution shift between offline data and online interactions. We propose to distill the following types of knowledge from the segmented skill-based trajectories in an iterative process, which leads to improved skill-conditioned policies.• Distilling Primitive Actions. A common mistake of LLM-powered agents is hallucination, i.e., the agent outputs actions that are not valid in the environment. To ensure effective usage of LLM in decision-making applications, it is important to specify the space of actions in the form of natural language or code. Many prior works (Liang et al., 2022; Liu et al., 2023c) manually define the available primitive functions, which requires human labor and domain knowledge. Instead, we propose to distill primitive actions or functions from the offline interaction data with LLM, which is easy to scale up and automate the practical operation. Figure 1 describes how to distill the primitives with an example, and the full prompt is in Appendix B. Here is your task: Your task is: put({object}, {receptable}) >Prompt for the skill-conditioned policy "put" The final stage is then to compose these policies and interact with the downstream task by calling the learned policies. We prompt a base policy ⇡ base with a few examples (come from LLM's own segmentation of offline trajectories) on calling the proper skills sequentially given a downstream task. Figure 2 shows an example of how to construct the base policy by prompting and how a skill-conditioned policy is prompted when being called.The main algorithm is presented in Algorithm 1, and the implementation-specific functions for Textbased-Policy and Code-based-Policy are defined in Algorithm 2 and Algorithm 3, respectively. We provide all used prompts and additional implementation details in Appendix B. The major difference in implementation between Text-based-Policy and Code-based-Policy is in the policy formulation and improvement processes:• Policy Initilization with Primitives. Text-based-Policy directly provides the discovered primitives in the prompt of policy and advises the agent to follow the primitives, while Code-based-Policy first lets the LLM write primitive functions and then calls these primitive functions in the code of skill-conditioned policies.• Policy Improvement. Since the distilled policy improvement tips are in natural language, we directly ask the LLM to merge the new suggestion into the prompt of Text-based-Policy. For Code-based-Policy, we let the LLM consider the policy improvement tips and re-write the policy code.• Policy Construction and Execution. In Stage 3, we prompt the base policy to call the learned Textbased-Policy or Code-based-Policy. Note that for Code-based-Policy, it is possible that the generated skill-conditioned code has compilation errors, so that it requires human checking or validation on a small set of tasks to verify that the code is executable. 2023), we make the problem even more challenging by limiting the horizon of each episode to be 30 (original is 50) steps and terminating the episode if the agent takes the same action twice. WebShop provides a real-world online shopping environment with 1.18M products, where an agent must explore the website, check relevant product candidates, and purchase the one that matches a user instruction (e.g., "I am looking for a queen sized bed that is black, and price lower than 140.00 dollars"). Our evaluation considers the first 500 out of 12, 087 instructions as test set (following the official implementation (Yao et al., 2022) ).Baselines and Metrics. We mainly compare the proposed O3D and O3D-Code with a popular text-based baseline method, ReAct (Yao et al., 2023b) and a state-of-the-art code-based baseline approach, Demo2code (Wang et al., 2023b) . Meanwhile, we have a variant of our method, named as O3D-Human, using the knowledge summarized by a human from the logs of ReAct in the test set of each domain, which we treat as an oracle baseline. This design is to investigate if the improvement tips distilled by LLMs can be as effective as the tips distilled by humans. In ALFWorld, we assess method performance by measuring the success rate (SR) under each task type as well as a total success rate over 134 tasks. Besides the success rate, in WebShop, there is a product matching score as an extra metric.Models and Offline Data. To investigate the robustness of O3D across various LLMs, we consider three GPT models (providing different ✓ pret defined in Equation (1) and Equation ( 2)) in our experiments, including GPT-4-0613, GPT-3.5-0613 and GPT-3.5-0301. The offline data include official human demonstrations in both domains as the success data, and a set of failure data generated by using ReAct on the training task set introduced in the original ALFWorld and WebShop implementations (more details are referred to Appendix A.1). 
results and analysis: 
 Effectiveness of discovering skills and primitives, and distilling policy improvement tips. In our experiments, O3D efficiently extracts high-quality skills from raw human demonstrations in the offline data, resulting in seven types of skills under ALFWorld domain, including: find(object), take(object), put(object, receptacle), cool(object), heat(object), use(object) and clean(object); and four types of skills under WebShop domain, including search item, select item, select item's attributes and purchase item. Each skill consists of a set of primitives to execute, along with a group of tips to assist with the skill completion. 3). By learning from offline data, O3D is able to capture correct primitives that can be composed to achieve each corresponding skill. Furthermore, we note that tips distilled by O3D is functional to various degrees, such as suggesting general tips, encouraging exploration, realizing action preconditions and highlighting syntax (as shown in Fig. 3 ). based policies, O3D respectively outperforms ReAct by 19%, 34% and 12% in ALFWorld, and 15%, 8% and 6% in WebShop in terms of success rate, with using GPT-4-0618, GPT-3.5-0613 and GPT-3.5-0301 repsectively. Furthermore, as shown in Table 1 , the success rate achieved by O3D under each task category is always greater than the one achieved by ReAct with GPT-4-0613 and GPT-3.5-0613. This further confirms that the tips distilled by O3D from offline data is generalizable and useful across diverse task types. For example, the tip of "pay attention to the task description and take the correct object" helps the LLM agent avoid taking a similar object (a pot) rather than the requested one (a pan), and the tip of "The correct action is to put the object 'in/on' the surface, not just 'on' the surface" prevents the LLM agent from using wrong syntax of primitives, which are the two common mistakes made by ReAct). Importantly, O3D can achieve competitive performance with O3D-Human in both domains, and even surpasses O3D-Human when using GPT-4 in ALFWorld and GPT-3.5-0613 in WebShop. This is strong evidence to validate that O3D's knowledge distillation is a promising approach to extract human-level tips from offline data to enhance the capability of an LLM to solve downstream tasks without finetuning or labor-intensive prompt engineering.In ALFWorld, O3D-Code surpasses Demo2Code, achieving a remarkable 32% higher performance with GPT-4-0613 and 14% with GPT-3.5-0613. Additionally, there's an 18% performance improvement on WebShop tasks using GPT-4-0613. However, GPT-3.5 struggles with these tasks due to the intricate nature of language processing and the complexity of the logic in WebShop. The principal advantage of our approach lies in its unique method of generating code: it adopts a bottom-up style, effectively constructing policies on top of robust skill functions. Through iterative skill refinement, the model cultivates robust skills by leveraging extensive and diverse data from demonstrations. Skill functions can then be efficiently utilized and reused to compose higher-level policies. In contrast, Demo2Code follows a top-down approach, requiring the generation of code for the same set of skills each time it receives a new task. Due to the context length constraint inherent in LLMs, only a limited number of demonstrations are used to guide skill code generation in Demo2Code, resulting in unstable and inconsistent skills.Webshop poses a substantial challenge for code-based policies owing to its need for comprehensive natural language understanding within the environment feedback. We address this challenge by enabling LLMs to construct skills by employing a limited set of LLM-based functions that encapsulate the underlying LLM capabilities (see Appendix B.5.2). To ensure a fair comparison, we also offer the same set of functions to Demo2Code. While our approach substantially enhances the performance of code-based policies in comparison to the baseline, it's important to note that skill generation remains considerably constrained by the intricate and diverse text-based environment feedback.Primitives, skills and policy improvement tips independently advance baseline performance in both ALFWorld and WebShop. O3D has three major processes: skill discovery (SD), primitives discovery (PD) and policy improvement tip distillation (TD). To investigate each component's contribution to performance in downstream task solving, we conducted an ablation study considering three variants of O3D, each with only one component. Fig. 4 shows that, in both domains, the three variants of O3D either outperform the baseline or achieve the same performance as the baseline across tested LLM models. In ALFWorld, O3D (PD-only) plays the dominant role in performance improvement with GPT-4-0613 and GPT-3.5-0301. This is because the major mistakes made by the baseline are in terms of outputing primitive actions with syntax errors or hallucinating unavailable actions. O3D (SD-only) boosts the performance the most with GPT-3.5-0613, becuase the tasks in ALFWorld are too complex for ReAct with GPT-3.5-0613, and O3D (SD-only) solves the tasks in hierarchy by performing skill selections that greatly reduces the complexity. In WebShop, the three components consistently benefit the baseline performance across the three GPT models, with their individual contributions also being model-dependent. Since the offline data was collected using GPT-3.5-0613, we observe that the highest overall improvement of the three components occurs in this model.-Check all possible locations for the required item, not just the most obvious ones.-Use the correct commands to interact with items.-You can clean items directly with the sinkbasin without needing to turn it on.-Cellphones can be found on sidetables and beds. After finding a cellphone, place it in dresser.-Kettles can be found on stoveburners. After finding the kettles, place them in the cabinet.-Spoons can be found on countertops. After finding the spoon, clean it using the sink basin before placing it on the sidetable.-Check the product's features to ensure it meets all the requirements in the instruction before making a purchase.-If the first product selected does not meet all the requirements, go back to the search results and select a different product.-After selecting an item, check its features to ensure it meets the task requirements before purchasing.-If the item does not meet the requirements after checking its features, go back and select another item. The advantage of using a contrastive method to distill improvement tips versus a non-contrastive method is domain-dependent. Fig. 5 shows that the proposed trajectory contrasting, which compares both successful and failed trials in offline data, is relatively helpful in certain domains, compared with the non-contrastive (NC) way based on only success data. In ALFWorld, failures in baseline method are often caused by violations of the domain-specific dynamics and rules. Therefore, the contrastive approach can generate general tips (green box in Fig. 5 ) to correct mistakes that occur in failure cases, while the non-contrastive approach only summarizes the facts (pink box in Fig. 5 ) from successful trials, which is less helpful. However, in WebShop, the two approaches achieve similar performance, as they output very analogous tips as shown in the corresponding boxes in Fig. 5 . 
conclusion: 
 This paper introduces an offline in-context learning framework, O3D, for LLM sequential decisionmaking agents, where agents can learn from previous experiences in a scalable offline manner to improve performance without the need of fine-tuning. O3D stands out by allowing LLMs to distill shared high-level knowledge from offline interaction logs, which is injected into a single set of prompts to be reused in solving diverse downstream tasks. Empirically, O3D leverages a unified algorithm that enhances both text-based policies and code-based policies with LLMs, outperforming baseline methods in two challenging benchmark domains. Our work offers a possibility for future LLM-based algorithm development in terms of efficiently leveraging offline data at scale for realworld sequential decision-making applications. 
 
 Appendix:  


Decision: This paper describes a method for discovering reusable skills from offline demonstrations, using an LLM to parse and segment demonstrations, then distilling each segmented-out skill into a dedicated policy. Reviewers agree that the proposed method is interesting and complementary to current work on deliberation methods for "LLM agents", and is convincingly evaluated. I think it will be a great fit for the conference.

One thought I had while reading the paper & reviews was that it might be nice to also include some discussion of pre-giant-LM methods that used language as a scaffold for skill learning, e.g. https://arxiv.org/abs/2110.07342, https://arxiv.org/abs/2110.01517, https://arxiv.org/abs/1903.01973.

Reviews: review: paper_summary: paper_summary: This paper proposed a prompting framework that can 1) automatically discover subtasks/skills along with their primitives and preconditions from trajectories, 2) apply discovered skills to new tasks in a hierarchical way. More specifically, for prediction, LM will first figure out the subtasks/skills and then retrieve relevant examples from the constructed database on how to use primitive actions to complete a task. Results based on GPT models on ALFWorld and WebShop demonstrate that it can improve upon ReAct. 

Though the general idea and presented results are interesting, the practical value of this work is unclear due to the limited experiment setting (see below).
strength_weakness: strength_weakness: The main components DiscoverPrompt and DistillPrompt all rely on LMs, and there seems no automatic verification to check how good are the distilled skills and so-called tips.  I'm concerned about 1) how often LMs fail to find generalisable actions (e.g., not too specific actions) or even wrong actions, 2) what if DistillPrompt provides wrong preconditions/corrections of discovered actions. Since these skills will serve as system prompt for new tasks, the errors in them will potentially propagate to all predictions.  Overall, I think how to ensure the quality of discovered skills is an important question, but there seems no experiments/discussion on it.
questions: questions: For reproducibility, it would be better to also provide results on one open-source models. 

For the baseline ReAct-Long, I wonder the performance of its retrieval-based variant --retrieving relevant trajectories based on task descriptions. It's well-known that retrieval is crucial for few-shot in-context learning methods.
paper_summary: paper_summary: The paper addresses the challenge of using Large Language Models (LLMs) for sequential decision-making tasks. Existing approaches often rely on high-quality few-shot demonstrations, but the limited context length of LLMs restricts their ability to process large-scale demonstrations with extended trajectories. To overcome this, the paper proposes a three-stage framework: First, it identifies reusable skills by segmenting offline datasets. Second, it enhances skill-conditioned policies through data distillation. The final stage involves constructing an interactive policy that utilizes the learned skills across various tasks. Experiments conducted on ALFWorld and WebShop demonstrate that this method surpasses traditional few-shot prompting baselines.
strength_weakness: strength_weakness: (1) The paper comprises multiple components and stages, making it difficult to determine which component contributes the most.

(2) In Table 9 (Appendix D.2.1), the paper compares the proposed method with Demo2Code and AdaPlanner. The extremely low performance of AdaPlanner is unusual. I suggest adding results for AdaPlanner + GPT-4 to further validate the proposed method.

(3) Since the method uses offline data, it is important to study how much data O3D needs and how the quality of O3D affects performance.
paper_summary: paper_summary: This paper proposes an LLM system that can act in ALFWorld and WebShop interactive environments and improves on top of standard prompting baselines. The main idea stems for hierarchical modeling: by using prompted LLMs: 
- **(1)** they first learn/segment subtasks from human demonstrations, 
- **(2)** then automatically learn primitive actions (go to X, pick N X), 
- **(3)** by using human demonstrations (has both positives and negatives) learn tips for the learned subtasks, 
- **(4)** by using all of above, initial LLM decides what tasks to execute and another LLM executes each subtask.

The main technical contribution comes from automatic primitive skill learning and providing a way to use negative trajectories in prompting.
strength_weakness: strength_weakness: 1) What is the intuitive reason for worse performance in Table 3(b)
 comparing to ReACT-long? The authors could do more qualitative analysis here.

2) This is not a reason to reject but, in Figure 3, the discovered skills seems simple and easy to infer from data. It could be exciting to apply this method to more domains, see what are the skills that can be learned.
questions: questions: 1) Why not use "interactive decision making" instead of "sequential decision making" in the title and throughout the paper. I believe interactive part is more important than the sequential part?
   
2) Please add a small summary of the environment details to the main text (available actions, observations and states etc)

3) I quote: "This discrepancy may be attributed to the fact that GPT-3.5-0613 is fine-tuned with data and objectives that are diverged from the ALFWorld domain". This sentence seems to me very ambigious and generic. Aren't GPT-4 and GPT3.5 fine-tuned with same data?
paper_summary: paper_summary: This paper proposes an offline learning framework called O3D (Offline Data-driven Discovery and Distillation) that enables large language models (LLMs) to improve their performance on sequential decision-making tasks without fine-tuning.
The framework has three stages: a) Skill discovery by segmenting offline trajectories into sub-tasks. b) Knowledge distillation by discovering primitives and extracting policy improvement tips from contrasting successful and failed trajectories. c) Hierarchical policy execution where a base policy sequentially calls the learned skill-conditioned policies. Through iterative skill refinement and knowledge distillation, O3D allows LLMs to benefit from extensive and diverse offline data, improving their few-shot performance on challenging benchmarks like ALFWorld and WebShop. The proposed approach outperforms state-of-the-art baselines and demonstrates the effectiveness of utilizing offline data at scale to enhance LLM capabilities in sequential decision-making without fine-tuning.
strength_weakness: strength_weakness: 1. The effectiveness of each stage is not clear to me. Further analysis of the design and experiments should be considered. 
2. Following the previous question, how can the quality of the skill discovery be managed? LLMs can hallucinate.
3. What is the difference between O3D and other self-refine baselines like Reflexion?
questions: questions: See weakness

Comments: title: Thanks for additional experiments
thank you
Discussion period
Thanks
comment: Thank you very much for taking time reading our response and giving positive feedback. We are happy to discuss further if there are any other questions. We will polish the final version based on your constructive suggestions and comments.
Thank you very much for taking time reading our response and raising the score. We will polish the final version based on your constructive suggestions. Please let us know if there are any other questions. We are happy to discuss further.
I would like to thank the authors for conducting additional experiments comparing the proposed method with AdaPlanner using GPT-3.5-0301. I'm pleased to see that it demonstrates decent performance. As a result, I will adjust my score accordingly.
Thank you for clarifying the metrics in Table 3(b), and your answers to my questions.

I will maintain my positive score as is.
Hi everyone! Thanks to reviewers ubuS and Fifg for your reply to author response. For the other reviewers, I'd appreciate it if you could look through the authors' comments soon and let us know if your concerns have been addressed / if there are any outstanding issues soon so there's time to discuss.

With appreciation,
Your AC
Thank you very much for reading our response and giving positive feedback. We really appreciate it. We will polish the final version of the paper based on your constructive suggestions. Please let us know if there are any other questions. We are happy to discuss further.
Thank you very much for reading our response and raising the score. We are keeping running the suggested experiments and will add them to the final version of the paper. 

We also finished another robustness test on WebShop. Similarly to the previous experiments on ALFWorld, we added irrelevant information and misleading information to the distilled knowledge. For example, we insert the action “rate [Item]” and tip “You can rate the items and leave feedback” which are not valid in the environment and can be misleading. We add these distractors to every skill-conditioned policy, so that it is a strong perturbation to the original method. In this way, we can simulate the case where the LLM hallucinated in the discovery and distillation process. The resulted success rates with GPT-3.5-0613 are as below.

- Original O3D: 35%
- O3D + Irrelevant Info: 31%
- O3D + Misleading Info: 34%

Compared to the baseline ReAct whose success rate is 27%, our O3D still achieves higher performance even when distractors exist. There suggests that **Stage 3 of O3D is robust to noise or mistakes from the first 2 stages, if any.**

In addition, we also would like to emphasize that **our algorithm design makes the first 2 stages grounded on facts in offline trajectories, so that the skills, actions and tips are relatively reliable.** Such design includes iterative discovery and distillation, trajectory contrasting, as well as segmentation-based skill summarization. For example, instead of asking the LLM to think what is the mistake and how to fix it (a common reflective method) which may lead to hallucination, we ask the LLM to compare a success sub-trajectory and a failure sub-trajectory and find their differences. In this way, it is much easier for the LLM to locate the key to success in the environment. 

We hope the above experiment results and explanations have addressed the reviewer’s concern on the quality control and robustness of the method. Please let us know if there is any remaining question or concern. We are very happy to discuss further. Thank you again for your time and suggestions.
Thanks the authors for the rebuttal. Most of my concerns have been resolved. I will maintain my positive score.
Thanks for the rebuttal.  I've increased the score given the new results and promised experiments.