Learning Grounded Action Abstractions from Language
Authors: Lionel Wong
Jiayuan Mao
Pratyusha Sharma
Zachary S Siegel
Jiahai Feng
Noa Korneev
Joshua B. Tenenbaum
Jacob Andreas
Venue: ICLR 2024 poster (2024)
Keywords: planning abstractions
hierarchical planning
library learning
learning from language
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We learn libraries of \textit{grounded action abstractions} from language that support efficient hierarchical planning in interactive environments.

Abstract: Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand. Decades of hierarchical planning techniques have used domain-specific temporal action abstractions to support efficient and accurate planning, almost always relying on human priors and domain knowledge to decompose hard tasks into smaller subproblems appropriate for a goal or set of goals. This paper describes Ada (Action Domain Acquisition), a framework for automatically constructing task-specific planning representations using task-general background knowledge from language models (LMs). Starting with a general-purpose hierarchical planner and a low-level goal-conditioned policy, Ada interactively learns a library of planner-compatible high-level action abstractions and low-level controllers adapted to a particular domain of planning tasks. On two language-guided interactive planning benchmarks (Mini Minecraft and ALFRED Household Tasks), Ada strongly outperforms other approaches that use LMs for sequential decision-making, offering more accurate plans and better generalization to complex tasks.

Published as a conference paper at ICLR 2024
LEARNING ADAPTIVE PLANNING REPRESENTATIONS WITH NATURAL LANGUAGE GUIDANCE
Lionel Wong i
Jiayuan Mao"
Pratyusha Sharma" !'
Zachary S. Siegel
Jiahai Fengà
Noa Korneev
Joshua B. Tenenbaum'
Jacob Andreas'
MITT Princeton University
*UC Berkeley *Microsoft
ABSTRACT
Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand. Decades of hierarchical planning techniques have used domain-specific temporal action abstractions to support efficient and accurate planning, almost always relying human priors and domain knowledge to decompose hard tasks into smaller subproblems appropriate for goal or set of goals. This paper describes Ada (Action Domain Acquisition), framework for automatically constructing task-specific planning representations using task-general background knowledge from language models (LMs). Starting with a general-purpose hierarchical planner and low-level coul-conditioned policy, Ada interactively learns library of planner-compatible high-level action abstractions and low-level controllers adapted to a particular domain of planning tasks. On two language-guide interactive planning benchmarks (Mini Minecraft and ALFRED Household Tasks), Ada strongly outperforms other approaches that use LMs for sequential decision- making, offering more accurate plans and better generalization to complex tasks.
INTRODUCTION
People make complex plans over long timescales, flexibly adapting what we know about the world in general to govern how act in specific situations. To make breakfast in the morning, we might convert a broad knowledge of cooking and kitchens into tens of fine-grained motor actions in order to find, crack, and fry specific egg: to achieve complex research objective, we might plan routine over days or weeks that begins with the low-level actions necessary to ride the subway to work. The problem of adapting general world knowledge to support flexible long-term planning is one of the unifying challenges of AI. While decades of research have developed representations and algorithms for solving restricted and shorter-term planning problems, generalized and long-horizon planning remains core, outstanding challenge for essentially all AI paradigms, including classical planning (Erol et al., 1994), reinforcement learning (Sutton et al., 1999), and modern generative AI (Wang et al., 2023a).
How do humans solve this computational challenge? A growing body of work in cognitive science suggests that people come up with hierarchical, problem-specipe representations of their actions and environment to suit their goals, tailoring how they represent, remember, and reason about the world to plan efficiently for particular set of tasks (e.g.. Ho et al., 2022). In AI, large body of work has studied hierarchical planning using domain-specific temporal abstractions progressively decomposing high-level goals into sequences abstract actions that eventually bottom in low-level control. An extensive body of work has explored how to plan using these hierarchical action spaces, including robotic task-and-motion planning (TAMP) systems (Garrett al., 2021) and hierarchical RL frameworks (Sutton et al., 1999).
However, identifying a set of abstract actions that are relevant and useful for achieving any given set of goals remains the central bottleneck in general. Intuitively, "useful" high-level actions must satisfy many different criteria: they should enable time-efficient high-level planning, correspond feasible low- level action sequences, and compose and generalize to new tasks. Despite efforts to learn high-level actions automatically both classical planning (Nejati et al., 2006) and RL formulations (Dietterich, 2000), most state-of-the-an robotics and planning systems rely on human expertise to hand-engineer new planning representations for each new domain (Ahn et al., 2022).
Asterisk indicates equal contribution. Correspondencee to 2y? @mi t.. edu. Code for this paper will
he released https:/githuh.com//// http://ggtttcccommoCCattttteeee Wwwaaaaheeennoooolllooooorraaae

---

Published as a conference paper at ICLR 2024
(a) To solve grounded planning taoks...
.e learn library of grounded actions with hierarchical planning framework
Propose symbolic abstractions
00 bi-evel planning
010 Veriteed grounded action lirary
Rice :iedd Ale hnooe we
ttactsos chil1 abbjet-
Scectdor cilll object-á
sprecandition (and
Flare
Crecentic resegetteee) (hoSds hol3
eotio 12
(aut siee castart)
Mini
Eoetteee cliiie objjecttt vx1
Call uee Sponk
Esectior beat doject-2
Pollcy
Pick
Hecly
Place
clactter elice- object-s ...)
grounded octions....
We solve complex planning tasks specified in language and grounded in interactive environments by jointly learning library of symbolic high-level action abstractions and modular low-level controllers associated with each abstraction. Our system leverages background information language as prior to propose useful action abstractions, then uses a hierarchical planning framework to verify and ground them.
Figure I:
In this paper, we introduce Action Domain Acquisition (Ada), a framework for using background knowledge from language (conveyed via language models) as initial source of task-relevant domain knowledge. Ada uses language models (LMs) in an interactive planning loop to assemble a library of composable. hierarchical actions tailored to given environment and task space. Each action consists of two components: (1) high-level abstraction represented as symbolic planning operutor (Fikes & Nilsson, 1971) that specifies preconditions and action effects as sets of predicates; and (2) a low-level controller that can achieve the action's 's effects by predicting sequence of low-level actions with neural network or local search procedure. We study planning in a multitask reinforcement learning framework, in which agents interact with their environments must solve collections of tasks of varying complexity. Through interaction, Ada incrementally builds a library of actions, ensuring at each step that learned high-level actions compose produce valid abstract plans and realizable low-level trajectories.
We evaluate Ada (Fig. 1) on two benchmarks, Mini Minecraft and ALFRED (Shridhar et al., 2020). We compare this approach against three baselines that leverage LMs for sequential decision-making
in other ways: to parse linguistic goals into formal specifications that are solved directly by planner (as in Liu et al. (2023)), to directly predict sequences of high-level subgoals (as in Ahn et al. (2022)), and to predict libraries of actions defined in general imperative code (as in Wang et al. (2023a)). In both domains, we show that Ada learns action abstractions that allow it to solve dramatically more tasks on each benchmark than these baselines, and that these abstractions compose to enable efficient and accurate planning in complex, unseen tasks.
2 PROBLEM FORMULATION
We assume access to environment (X, where X' is the (raw) state space, 2/ is the (low-level) action space (e.g., robot commands), and T is a deterministic transition function T : X x l + X. We also have set of features (or "predicates") P that define an abstract state space S: each abstract state 8 E S is composed of a set of objects and their features. For example, a simple scene that contains bread (bread(A) (A), table(B [B,, on(A, B)}. We assume the mapping from environmental states to abstract states X + S is given and fixed (though see Migimatsu & Bohg. 2022 for how it might be learned).
table could be encoded as abstract state with two objects A and B, and atoms
In addition to the environment, we have collection of tasks t. Each t is described by natural language instruction l, corresponding to goal predicate (which is not directly observed). In this paper, we assume that predicates may be defined in terms of abstract states, i.e., 9e : S - {T, F). Our goal is to build an agent that, given the initial state 2e E X' and the natural language instruction 4, can generate a sequence of low-level actions (u1, U2, , un) } E 4 such that ge (* )) is true (where vH is the terminal state of sequentially applying { on zo). The agent receives reward signal only upon achieving the goal specified by 9t-
Given a very large number of interactions, a sufficiently expressive reflex policy could, in principle, learn a policy that maps from low-level states to low-level actions conditioned on the language instruction [u x; f.). However, for very long horizons and large state spaces (e.g.. composed of many objects and compositional goals), such algorithms can be highly inefficient effectively infeasible. The key idea behind our approach is to use natural language descriptions & to bootstrap a high-level action space A over the abstract state space S to accelerate learning and planning.
2

---

Published as a conference paper at ICLR 2024
(a) Task Input
(b) Bi-Lovel Planning and Exccution
(c) State and Action Abstraction
Initial state:
Prodicates (P)
receptac letype(...)
Abstract Plan jed (pick egaen, ...) (enveta to eicrowase)
Tyjerty islot(....)
Abstract State-
High-Lovel Planner (e.g.. FD)
Operator (A4): (saction lesstdejact
Abstract Goal
spananebers (#) tr ho)
... (More actions)
:precondition (and
treceptacletype 2n Nicrowveeype)) (tolds to 20) ...)
seffect (and (1siorr 700) ...) operotons!
C Bring o hust <gg
LoweLLeell Plan
Hoya
Tove
Controllers: (or s4); Xxed
Figure 2: Representation for our (a) task input, (b) the bi-Ievel planning and execution pipeline for inference time, and (c) the abstract state and action representation.
for iteration = 1, 3,
Abstract State
Raw State
Execute and
Controller
Score Operators
{xg, {
Ai
A
Sym. Planner
{ag
{u+}
A
Candidate Ops.
High-Level Action
Low-Level Action
Symbolic Goal
Figure 3: The overall framework. Given task environment states and descriptions, at each iteration, we first propose candidate abstract actions (operators) Al, then uses bi-level planning and execution solve tasks. We add operators to the operator library based on the execution result.
Formally, our approach learns library of high-level actions (operators) A. As illustrated in Fig. 2b, each E A is a tuple of (name, args.pry pre, efff controller). name is the name of the action, args is list of variables, usually denoted by ?x, ?y, elc., pre is precondition formula based on the variables args and the features P. and eff is the effect, which also defined in terms of args and P. Finally, controller: X' + U is low-level policy associated with the action. The semantics of the preconditions and effects is: for any state I such that pre pre( (()), executing controller starting in x (for indefinite number of steps) will yield a state x' such that eff( (z²)) (Lifschitz, 1986). In this framework, A defines a partial, abstract world model of the underlying state space.
As shown in Fig. 2b, given the set of high-level actions and parse of the instruction l into first-order logic formula, we can leverage symbolic planners (e.g., Helmert, 2006) to first compute high-level plan {ai,.. ,ax) E A that achieves the goal & symbolically, and then refine the high-level plan into a low-level plan with the action controllers. This bi-level planning approach problems. Furthermore,
decomposes long-horizon planning problems into several can also leverage the compositionality of high-level actions A to generalize to longer plans.
3 ACTION ABSTRACTIONS FROM LANGUAGE
As illustrated Fig. 3, our framework, Action Domain Acquisition (Ada) learns action abstractions iteratively as it attempts solve tasks. Our algorithm is given dataset of tasks and their corre- sponding language descriptions, the feature set P, and optionally an initial set of high-level action operators Ao. At each iteration we first a large language model (L.LM) to propose set of novel high-level action definitions Af based on the features P and the language goals {&+] (Section 3. Next, we use LLM also translate each language instruction é, into a symbolic goal description Ft, and use bi-level planner to compute a low-level plan to accomplish & (Section 3.2). Then, based on the planning and execution results, we score each operator in A, and add ones the verified library if they have yielded successful execution results (Section 3.4). To accelerate low-level planning, we simultaneously learn local subgoal-conditione conditioned policies (i.e.. the controllers for each operator; Section 3.3). Algorithm summarizes the overall framework.
core goal of our approach is to adapt the initial action abstractions proposed from an LLM prior into a set of useful operators Ax that permit efficient and accurate planning on a dataset of tasks and ideally, that generalize to future tasks. While language provides a key initial prior, our formulation refines and verifies the operator library to adapt to given planning procedure and environment (similar to other action-learning formulations like Silver et al., 2021). ). Our formulation ensures not only that the learned operators respect the dynamics of the environment, but also fit their grain of abstraction according to the capacity of the controller, trading off between fast high-level planning and efficient low-level control conditioned each abstraction.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Action Abstraction Learning from Language
Input:
Dataset of tasks and their language descriptions (fx)
Input: Prodicate set P
Input: Optionally, an initial set abstract operators Ao. or AA
I: Initialize subgoal-conee conditioned policy ne.
2: for i fori=11 1,2. ,M do
3: A; 1:-1 U
(P:()))
b Section 3.1
4: for each unsolved task j: (x
(5) e(() do
5:
u. -- BiLevelPPann AA, 6 TT)
P Section 3.2
6:
resul((s) < Execute (Tu (5)
Execute the plan
7:
4- UpdateSSuggollPoiccyy result)
b Section 3.3
8:
A ScoreAndFilter(A,, resul))
Section 3,4
return AM
(al Stage 1: Propose Symbolic Task Decomposition
Prompt
LLM Generation
Bring a hot to the tuble.
Examples
up +gE)
3: Bake bread teing
table.
(eore steps)
(pick-up brewd) (place breed oven) (bakce bread)
(place pot stove)
Objects: eop,
(heat-obfect kitchen nicrowave egg) (core steps)
State: onlogs. tablo), coldfogal.
13 Saute some catbage.
opentmicrowardd
(wore Language to symbolic plans)
Extract Undefined
[b) Stage Propase Symbolic Operator Definitions
Operator
;; Examples
hwat-abject 2r pprecondition (and (receptac leTyp Microavetype)
(caction ...
precondition (receptaclolype tr Pan)
heat-cbject
seffect ...).
(saction
:preconditiin (and (recaptaclelype ir Oven)
(rcoopacIssttoccttinn
:effect ...)
(holds 20)) (1sHat 70)))
...... (cone aperator exarpies)
Figure 4: Our two-stage prompting method for generating candidate operator definitions. (a) Given a task instruction, we first prompt I.I.M to generate candidate symbolic task decomposition. (b) We then extract undefined operator names that appear the sequences and prompt I.IM generate symbolic definitions.
3.1 OPERATOR PROPOSAL: A, + A.-1 U PropsseOperaorDDeiimiiion// (4+)
At each iteration i, we use a pretrained LLM to extend the previous operator library A,-1 with a large set of candidate operator definitions proposed by the LLM based on the task language descriptions and environment features P. This yields extended candidate library Ai where each a E A² = (name, args, args,pre, eff) where name is a human- action name and args, pre, eff are a PDDL operator definition. We employ two-stage prompting strategy: symbolic task decomposition followed by symbolic operator definition.
Example. Fig. 4 shows concrete example. Given task instruction (Bring hot egg to the table) and the abstract state description, we first prompt the LLM to generate an abstract task decomposition, which may contain operator names that are undefined in the current operator library. Next, extract the names of those undefined operators and prompt LLMs to generate the actual symbolic operator descriptions, in this case, the new heat-object operator.
Symbolic task decomposition. For a given task & and initial state xp, we first translate the raw state 2y into symbolic description & (#a). To constrain the length of the state description, we only include unary features in the abstract state (i.e., only object categories and properties). Subsequently, we present a few-shot prompt to the LLM and query to generate proposed task decomposition conditioned on the language description &,- It generates sequence of named high-level actions and their arguments, which explicitly can include high-level actions that are not yet defined in the current action library. We then extract all the operator names proposed across tasks the candidate high-level operators. Note that while in principle we might use the LLM -rropood task decomposition itself as high-level plan, find empirically that this is less accurate and efficient than formal planner.
Symbolic operator definition. With the proposed operator names and their usage examples (i.e., the actions and their arguments in the proposed plans), we then few-shot prompt the LLM to generate candidate operator definitions in the PDDL format (argument types, and pre/postconditions defined based on features in P). We also post-process the generated operator definitions to remove feature
4

---

Published as a conference paper at ICLR 2024
names not present in P and correct syntactic errors. We describe implementation details for syntax correction strategy in the appendix.
3.2 GOAL PROPOSAL AND PLANNING: result < Execute(11)) (3) BBBLevelPlan(A, ep (T)
At each iteration i, we then attempt to BiLevelPlan for unsolved tasks in the dataset. This step attempts to find and execute a low-level action sequence {u,; U2..... ..... "
E 44 for each task using the proposed operators in A that satisfies the unknown goal predicate gr for each task. This provides the environment reward signal for action learning. Our BiLevelPlan has three steps.
Symbolic goal proposal: As defined in Sec. 2, each task is associated with queryable but unknown goal predicate 9r that can be represented as first-order logic formula fi over symbolic features in P. Our agent only has access to linguistic task description &,, so we use a few-shot prompted LLM to predict candidate goal formulas F/ conditioned on l and features P.
High-level planning: Given each candidate goal formula f E Fý, the initial abstract problem state So. and the current candidate operator library A', we search for high-level plan PA = {(11,0 01,...).... ..... (ax, ok,...) } as a sequence of high-level actions from A' concretized with object arguments such that executing the action sequence would satisfy J according to the operator definitions. This is a standard symbolic PDDL planning formulation; we use an off-the-shelf symbolic planner, FastDownward (Helmert, 2006) to find high-level plans.
Low-level planning and environment feedback: We then search for a low-level plan as a sequence of low-level actions {U1, u2.--. (H} E 4 conditioned on the high-level plan structure. Each concretized action tuple (a;, ..... E PA defines a local subgoal sgi, as the operator postcondition parameterized by the object arguments
For each (a,,01, .1.... E PA. therefore search for sequence of low-level actions W Wig... that satisfies the local subgoal 8g:. We search with a fixed budget per subgoal, and fail early if we are unable to satisfy the local subgoal sgc. If we successfully find a complete sequence of low-level actions satisfying all local subgoals sg, in PA. we execute all low-level actions and query the hidden goal predicate gt to determine environment reward. We implement a basic learning procedure to simultaneously learn subgoal- conditione controllers over time (described in Section 3.3), but our formulation is general and supports many hierarchical planning schemes (such as sampling-bass -bsed low-level planners (LaValle, 1998) or RL algorithms).
3.3 LOW-LEVEL LEARNING AND GUIDED SEARCH: 0 <- UpdateSubhocalPoliee(. result)
The sequence of subgoals sg; corresponding to high-level plans PA already restricts the local low- level planning horizon. However, we further learn congoal-conditooeed low-level policies T [u]a:; sg) from environment feedback during training to accelerate low-level planning. To exploit shared structure across subgoals, we learn a shared controller for all operators from x X and conjunctions of predicates in sg. To maximize learning during training, scheme (Andrychowicz et al., 2017), supervising on all conjunctions of predicates in the state as we roll out low-level search. While the shared controller could be learned we find that our learned operators sufficiently restrict the search to permit learning an even simpler count-based model from X, sg u E U. We provide additional details in the Appendix.
use hindsight goal relabeling
supervised neural policy,
3.4 SCORING LLM OPERATOR PROPOSALS: A; <- ScoreAndFilter(A: result)
Finally, we update the learned operator library A; retain candidate operators that were useful and successful in bi-le planning. Concretely, we estimate operator candidate af E A', accuracy across the bi-level plan executions as s/b where b counts the total times a' appeared in high-level plan and 8 counts successful execution of the corresponding low-level action sequence to achieve the subgoal associated with a/. We retain operators if b ть and s/b > Tr. where 7o, Ty are hyperparameters. Note that this scoring procedure learns whether operators are accurate and support low-level planning independently of whether the L.L.-predicted goals f matched the true unknown goal predicates gt-
EXPERIMENTS
Domains. We evaluate our approach two-language specified Hlanning-benchmaan Mini Minecraft and ALFRED (Shridhar et al., 2020). Mini Minecraft (Fig. 5, top) is a procedurally generaaeeee Minecraft-like benchmark (Chen et al., 2021; Luo et al., 2023) on 2D grid world that requires
5

---

Published as a conference paper at ICLR 2024
Mini Minecraft
(caettonn craft-bed tpanaretors 731 laventory Leventory
Investory
ta object (agent-at (okject-at 19) (on) type loristation)
abject sbgect sbject tiie]
plank
(Inventory 711 7et3
(obj-type 711 stedPTank) (onj-type knol) (m9j-type typothetical))
(imventory 2n0)
(Imventor--aptty
(1ot (Laventory empty PLA)) (inventary HL 28)
coal
(not [ob] type 20 tepothetical)) () (ob) Aype 76 sed)
(oot (inwentory 1o13) 22 mone effactt omarted
Cruft boo.
pickave
iron
shears
(a) Example task.
(b) An example operator proposed and verified by our algorithmm
Visualization of the crafting actions used.
ALFRED
(ractior cosldtject
(taction Slicediject
Sparaes
sparaesters
Put chilkad wine in the cotiinss.
Poodressptacee
11 lncation to object)
Atso-abjeet onject agent
object)
jprecanaition (and
preccnsation Cand
Worn a piste and place
treceptaciatype Procireceptacte
(objec:Type Ptosiebjest (niteType)
Fridgelype)
fariorati ha
Wast the dirty bowl before mutting bowl the casunier
(boldss 7 ho)
retireahle he)
(nesaptacakttoccreee reoottacaptccte 2333 sefficct
(bolds to Ttookoeject)
soffiect (and (1851teed ho))
70)3
Place a cold podato siice in the oven.
Example tasks.
[b] Example operators proposed and verifled by algorithrn.
Figure 5: Top: (a) The Mini Minecraft environment, showing an intermediate step towards crafting a bed. (b) Operator proposed by an LLM and verified by our algorithm through planning and execution. (c) Low-level actions involved in crafting the bed. Bottow: (a) The ALFRED household environment. (h) Example operators proposed by L.M and verified by our algorithm, which are composed to salve the cold potato slice task.
tools to mine resources and craft objects. The ability to create new objects that themselves permit new actions yields an enormous action space at each time step (>2000 actions) and very Long-horizon tasks (26 high-level steps for the most complex task, without path-planning.) ALFRED (Fig. 5, bottom) is a household planning benchmark of aunottan but formally verifiable tasks defined over a simulated Unity environment (Shridhar et al., 2020). The tasks include object rearrangements and those with object states such as heating and cleaning. Ground-trut high-level plans in the ALFRED benchmark compose 5-10 high-level operators, and low-level action trajectories have on average 50 low-level actions. There over 100 objects that the agent can interact with each interactive environment. See the Appendix for details.
complex, extended planning. The agent
Experimental setup. We evaluate in an iterative continual learning setting; except on the compo- sitional evaluations, we learn from n=2 iterations through all (randomly ordered) tasks and report final accuracy those tasks. All experiments and baselines use GPT-3.5. For each task, at each iteration, we sample n=4 initial goal proposals and == initial task decompositions, and n=3 operator definition proposals for each operator name. We report best-of accuracy, scoring task as solved if verification passes on at least one of the proposed goals. For Minecraft, we set the motion planning budget for each subgoal to <1000 nodes. For ALFRED, which requires slow Unity simulation, set it to 50 nodes. Additional temperature and sampling details are in the Appendix.
We evaluate on three Mini Minecraft benchmark variations to how our approach generalizes to complex, compositional goals. In the simplest Mining benchmark, all goals involve mining target item from an appropriate initial resource with appropriate tool (e.g.. Mining iron from iron_ore with an axe). In the harder Crafting benchmark, goals involve crafting a target artifact (e.g.. bed), which may require mining few target resources. The most challenging Compositional benchmark combines mining and crafting tasks, in environments that only begin with raw resources and two starting tools (axe and pickaxe). Agents may need to compose multiple skills to obtain other downstream resources (see Fig. 5 for example). To test action generalization. report evaluation on the Compositional using only actions learned previously the Mining and Crafting benchmarks.
We similarly evaluate ALFRED benchmark of Simple and Compositional tasks drawn from the original task distribution in Shridhar et al. (2020). This distribution contains simple tasks that location, picking up objects. applying a single
require picking up object and placing it in household skill to an object and moving them to a new location (e.g.. Put a clean apple on the dining table), and compositional tasks that require multiple skills (e.g., Place hot sliced potato the

---

Published as a conference paper at ICLR 2024
Mini Minecraft (n=3)
LLM Predicts?
Library?
Mining
Crafting
Compositional
Low-level Planning Only
Goal
31% (r=00.0%
9% (x=0.0%)
9% (a=0.0%)
Subgoal Prediction
Sub-goals
33% (o=1.6%)
36% (.6%)
6% (a=1.7%)
Code Policy Prediction
Sub-policies
15% (d=1.2%)
39% (0=3.2%%
10% (a=1.7%%
Ada (Ours)
Goal+Operators
100% (o=0.0%)
100% (d=7.5%)
100% (o=4.1%)
ALFRED (n=3 replications)
LLM Predicts?
Library?
Original (Simple Compositional Taskk)
Tasks)
Low-level Planning Only
Goal
21% (o=1.0%%
Suhgoal Prediction
Sub-goal
2% (r=0.4%)
Code Policy Prediction
Sub Sub-policies
2% (o=0.9%)
Ada (Ours)
Goal+Openssooss
79% (a=0.9%)
Table I: (Top) Results on Mini Minecraft. Our algorithm successfully recovers intermediate operators for mining and crafting, which enable generalization to more compositional tasks (which use up to 26 operators) without any additional learning. (Bottom) Results on ALFRED. Our algorithm recovers all required household operators, which generalize more complex compositional tasks. All results report mean performance and STD from n=3 random replications for all models.
counter). We use a random subset of n=223 tasks, selected from an initial 250 that we manually filter to remove completely misspecified goals (which omit any mention of the target object or skill).
Baselines. We compare our method to three baselines of language-guided planning.
Low-level Planning Only uses an LLM to predict only the symbolic goal specification conditioned on the high-level predicates and linguistic goal, then uses the low-level planner to search directly for actions that satisfy that goal. This baseline implements a model like LLM+P (Liu et al., 2023), which uses LLMs to translate linguistic goals into planning conpiiib-lom formal specifications, then attempt to plan directly towards these with no additional representation learning.
Subgoal Prediction uses an LLM to predict a sequence of high-level subgoals (as PDDL pre/postcon- ditions with object arguments), conditioned on the high-level predicates, and task goal and initial environment state. This baseline implements model like SayCan (Ahn et al., 2022), which uses LLMs to directly predict goal and a sequence of decomposed formal subgoal representations, then applies low-level planning over these formal subgoals.
Code Policy Prediction uses an LLM to predict the definitions of a library of imperative local code policies in Python (with cases and control flow) over imperative API that can query state and execute low-level actions.) Then, as FastDownward planning is no longer applicable, we also use the LLM to predict the function call sequences with arguments for each task. This baseline implements a model like Voyager (Wang et al., 2023a), which uses an LLM to predict a library of skills implemented as imperative code for solving individual tasks. Like Voyager, we verify the individual code skills during interactive planning, but do not use more global learning objective to attempt to learn a concise or non-redundant library.
4.1 RESULTS
What action libraries do learn? Fig. 5 shows example operators learned each domain (Appendix A.3 contains the full libraries of operators learned on both domains from a randomly sampled run of the n=3 replications). In Mini Minecraft, manually inspect the library and find that we learn operators that correctly specify the appropriate tools, resources, and outputs for all intermediate mining actions (on Mining) and crafting actions (on Crafting), allowing perfect direct generalization to the Compositional tasks without any additional training on these complex tasks. In ALFRED, we compare the learned libraries from all runs to the ground-truth operator library hand-engineered in Shridhar et al. (2020). The ground-truth operator set contains 8 distinct operators corresponding to different compositional skills (e.g.. Slicing, Heating. Cleaning, Cooling). Across all replications, model reliably recovers semantically identical (same predicate preconditions and postconditions) definitions for all of these ground-truth operators, except for single operator that defined disjunctively (the ground-truth Slice skill specifies either of two types of knives), which we occasionally learn as two distinct operators or only recover with one of these two types.
We also inspect the learning trajectory and find that, through the interactive learning loop, we successfully reject many initially proposed operator definitions sampled from the language model that turn out to be redundant (which would make high-level planning inefficient), inaccurate (including apriori reasonable proposals that do not fit the environment specifications, such as proposing to clean objects with just a towel, when our goal verifiers require washing them with water in sink), or
7

---

Published as a conference paper at ICLR 2024
underspecified (such those that omit key preconditions, yielding under-decomposed high-level task plans that make low-level planning difficult).
Do these actions support complex planning and generalization? Table 2 shows quantitative results from n=3 sampling from the language model and stochasticity in the underlying environment (ALFRED). On Minecraft, where goal specification is completely clear due to the synthetic language, we solve all tasks in each evaluation variation, including the challenging Compositional setting the action libraries learned from simpler mining/crafting tasks generalize completely to complex tasks that require crafting all intermediate resources and tools from scratch. On ALFRED, we vastly outperform all other baselines, demonstrating that the learned operators are much more effective for planning and compose generalizably to more complex tasks. We qualitatively find that failures on ALFRED occur for several reasons. One is goal misspecification, when the LLM does not successfully recover the formal goal predicate (often due to ambiguity in human language), though we find that on average, 92% of the time, the ground truth goal appears one of the top-4 goals translated by the LLM. We also find failures due to low-level policy inaccuracy, when the learned policies fail to account for low-level, often geometric details of the environment (e.g., the learned policies are not sufficiently precise to place a tall bottle on an appropriately tall shelf). More rarely, we see planning failures caused by slight operator overseecifccation (e.g., the Slice case discussed above, in which we do not recover the specific disjunction over possible knives that can be used to slice.) Both operator and goal specification errors could be addressed in principal by sampling more (and more diverse) proposals.
replications of all models, to account for random noise in
How does our approach compare to using the LLM to predict just goals, or predict task se- quences? As shown in Table 2, our approach vastly outperforms the Low-level Planning Only baseline on both domains, demonstrating the value of the action library for longer horizon planning. We also find a substantial improvement over the Subgoal Prediction baseline. While the LLM frequently predicts important high-level aspects of the task subgoal structure (as it does to propose op- erator definitions), frequently struggles to robustly sequence these subgoals and predict appropriate concrete object groundings that correctly obey the initial problem conditions or changing environment state. These errors accumulate over the planning horizon, reflected in decreasing accuracy on the compositional Minecraft tasks (on ALFRED, this baseline struggles to solve any more than the basic lock-and-place tasks, as the LLM struggles to predict subgoals that accurately track whether objects are in appliances or whether the agent's single gripper is full with an existing tool.)
How does our approach compare to using the LLM to learn and predict plans using imperative code libraries? Somewhat surprisingly, find that the Code Policy prediction baseline performs unevenly and often very poorly on our benchmarks. (We include additional results in A. showing that our model also dramatically outperforms this baseline using GPT-4 as the base LLM.) We find several key reasons for the poor performance of this baseline relative to our model, each which validate the key conceptual contributions of our approach. First, the baseline relies on the LLM as the planner as the skills are written as general Python functions, rather than any planner-specific representation, we do not use an optimized planner like FastDownwarr. As with Subgoal Prediction, we find that the LLM is not a consistent or accurate planner. While it retrieves generally relevant skills from the library for each task, it often struggles to sequence them accurately or predict appropriate arguments given the initial problem state. Second, we find that imperative code is less suited in general as a hicrarchical planning representation for these domains than the high-level PDDL and low-level local policy search representation we use in our model. This is because it uses control flow to account for environment details that would otherwise be handled by local search relative to high-level PDDL action. Finally, our model specifically frames the library learning objective around learning a compact library of skills that enables efficient planning, whereas our Voyager re-implemeenttionn (as in Wang al. (2023a)) simply grows library of skills which are individually executable and can be used to solve individual, shorter tasks. Empirically, as with the original model in Wang et al. (2023a), this baseline learns hundreds of distinct code definitions on these datasets, which makes it harder to accurately plan and generalize to more complex tasks. Taken together, these challenges support our overarching library learning objective for hierarchical planning.
RELATED WORK
Planning for language goals. A large body of recent work attempts to use LLMs to solve planning tasks specified in language. One approach is to directly predict action sequences (Huang et al., 2022; Valmeekam et al., 2022; Silver et al., 2022; Wang et al., 2023b), but this has yielded mixed
8

---

Published as a conference paper at ICLR 2024
results as LLMs can struggle to generalize produce correct plans as problems grow more complex. To combat this, one line of work has explored structured and iterative prompting regimes (e.g.. "chain-of-thouge and feedback) (Mu et al., 2023; Silver et al., 2023; Zhu et al., 2023). Increasingly. other neuro-symbolic work uses L.LMs to predict formal goal or action representations that can be verified or solved with symbolic planners (Song et al., 2023; Ahn et al., 2022; Xie al., 2023; Arora & Kambhampati, 2023). These approaches leverage the benefits of a known planning domain model. Our goal in this paper is to leverage language models to learn this domain model. Another line of research aims at using LLMs to generate formal planning domain models for specific problems (Liu et al., 2023) and subsequently uses classical planners to solve the task. However, they are not considering generating grounded or hierarchical actions in an environment and not learning library of operators that can be reused across different tasks. More broadly, we share the broad goal of building agents that can understand language and execute actions to achieve goals (Tellex et al., 2011; Misra et al., 2017; Nair et al., 2022). See also Luketina et al. (2019) and Tellex et al. (2020).
Learning planning domain and action representations from language. Another group of work has been focusing on learning latent action representations from language (Corona et al., 2021; Andreas et al., 2017; Jiang et al., 2019; Sharma et al., 2022; Luo et al., 2023). Our work differs from them in that we are learning a planning compnnibbl action abstraction from LLMs, instead of relying on human demonstrations and annotated step-by-step instructions. The more recent Wang et al. (2023a) adopts a similar overall problem specification, to learn libraries of actions as imperative code-based policies. Our results show that learning planning abstractions enables better integration with hierarchical planning, and, as result, better performance and generalization to more complex problems. Other recent work (Nottingham al., 2023) learns an environment model from interactive experience, represented as a task dependency graph; we seek to learn a richer state transition model (which represents the effects of actions) decomposed as operators that can be formally composed to verifiably satisfy arbitrarily complex new goals. Guan et al. (2024), published concurrently, seeks to learn PDDL representations; we show how these can be grounded hierarchically.
Language and code. In addition to Wang et al. (2023a), growing body of work in program synthesis, both by learning lifted program abstractions that compress longer cxisting or synthesized programs (Bowers et al., 2023; Ellis et al., 2023; Wong et al., 2021; Cao et al., 2023). These approaches (including Wang et al. (2023a)) generally learn libraries defined over imperative and functional programming languages, such as LISP and Python. Our work is closely inspired by these and seeks to learn representations suited specifically to solving long-range planning problems.
Hierarchical planning abstractions. The hierarchical planning knowledge that we learn from LLMs and interactions in the environments are related to hierarchical task networks (Erol et al., 1994; Nejati et al., 2006), hierarchical goal networks (Alford et al., 2016), 6), abstract PDDL domains (Konidaris et al., 2018; Bonet & Geffner, 2020; Chitnis et al., 2021; Asai & Muise, 2020; Mao et al., 2022: 2023), and domain control knowledge (de la Rosa & Mcllraith, 2011). Most of these approaches require manually specified hierarchical planning abstractions; others learn them from demonstrations or interactions. By contrast, we leverage human language to guide the learning of such abstractions.
6 DISCUSSION AND FUTURE WORK
Our evaluations suggest a powerful role for language within AI systems that form complex, long- horizon plans as rich source of background knowledge about the right action abstractions for everyday planning domains, which contains broad human priors about environments, task decomposi- tions, and potential future goals. A core goal of this paper was to demonstrate how integrate this knowledge into the search, grounding, and verification toolkits developed in hierarchical planning.
We leave open many possible extensions towards future work. Key limitations of our current framework point towards important directions for further integrating LMs and hierarchical planning to scale our approach: here, we build on an existing set of pre-defined symbolic predicates for initially representing the environment state; do not yet tackle fine- fine-grained, geometric motor planning; and use a general LLM (rather than one fine-tuned for extended planning). Future work might generally tackle these problems by further asking how else linguistic knowledge and increasingly powerful or multimodal LLMs could be integrated here: to propose useful named predicates over initial perceptual inputs (c.g., images) (Migimatsu & Bohg, 2022); to speed planning by bootstrapping hierarchical planning abstractions using the approach here, but then to progressively transfer planning to another model, including LLM,, later compose and use the learned representations.
9

---

Published as a conference paper at ICLR 2024
Acknowledgement. We thank anonymous reviewers for their valuable comments. We gratefully acknowledge support from ONR MURI grant N00014-16 1--20000--- from the Center for Brain. Minds, and Machines (CBMM, funded by NSF STC award
from NSF grant 2214177; from NSF grant CCF-221 7064 and IIS-2212310 from Air Force Office of Scientific Research (AFOSR) grant FA9550-22-11 22-1--249; from ONR MURI grant (00014-22-1-22 1-0034; from the MIT-IBM Watson AI Lab; from the MIT Quest for Intelligence; from Intel; and from the Boston Dynamics Artificial Intelligence Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
from ARO grant W91 INF-23-
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as I Can, Not as I Say: Grounding Language in Robotic Affordances. arXiv:2204.0166 2022. 2, 7,9
Ron Alford, Vikas Shivashankar, Mark Roberts, Jeremy Frank, and David W Aha. Hierarchical Planning: Relating Task and Goal Decomposition with Task Sharing. In ICCII, 2016. 9
Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforcement Learning with Policy Sketches. In ICML, 2017. 9
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. In NeurlPS, 2017. 5
Daman Arora and Subbarao Kambhampati. Learning and Leveraging Verifiers Improve Planning Capabilities of Pre-trained Language Models. arXiv:2305. 2023. 9
Masataro Asai and Christian Muise. Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (To Strips). IJCAI, 2020. 9
Blai Bonet and Hector Geffner. Learning First-Order Symbolie Representations for Planning from the Structure of the State Space. In ECAI, 2020.
Matthew Bowers, Theo X Olausson, Lionel Wong, Gabriel Grand, Joshua B Tenenbaum, Kevin Ellis, and Armando Solar-Lezama. Top-Down Synthesis for Library Learning. PACMPL, 7(POPL): 1182-1213, 2023. 9
David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, and Nadia Polikarpova. Babble: Learning Better Abstractions with E-graphs and Anti-unification. PACMPL, 7(POLL): 396-424, 2023. 9
Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask Your Humans: Using Human Instructions to Improve Generalization Reinforcement Learning. ICLR, 2021. 5, 13
Rohan Chitnis, Tom Silver, Joshua B Tenenbaum, Leslie Pack Kaelbling, and Tomás Mozzaoo-Péres GL.i: Efficient Exploration for Relational Model-Based Reinforcement Learning via Goal-Littral Babbling. In AAAI, 2021. 9
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modular Networks for Compositional Instruction Following. In NAACL-HILT, 2021. 9
Tomás de la Rosa and Sheila Mcllraith. Learning Domain Control Knowledge for TLPlan and Beyond. In ICAPS 201/ Workshop on Planning and Learning, 2011. 9
Thomas G Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function Decom-
position. JAIR, 13:227-303, 2000.
Kevin Ellis, Lionel Wong. Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. DreamCoder: Growing General- izable, Interpretable Knowledge with Wake-Sleep Bayesian Program Learning. Philosophical Transactions of the Royal Society, 381 381(12)))
2023. 9
10

---

Published as a conference paper at ICLR 2024
Kutluhan Erol, James Hendler, and Dana S Nau. HTN Planning: Complexity and Expressivity. In AAAI, 1994. 1,9
Richard E Fikes and Nils J Nilsson. STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artif. Intell., 2(3-4):189-22 10018292088 1971. 2
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Leslie Pack Kaelbling, and Tomás Lozano-Pérez Integrated Task and Motion Planning. Ann. Rev. Control Robot. Auton. Syst. 4:265-293, 2021.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, 36, 2024.
Malte Helmert. The Fast Downward Planning System. JAIR, 26:191-246, 2006. 3,
Mark K Ho, David Abel, Carlos Correa, Michael L Littman, Jonathan Cohen, and Thomas Griffiths. People Construct Simplified Mental Representations to Plan. Nature, 606(7912):129-136 2022.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML, 2022. 8
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an Abstraction for Hierarchical Deep Reinforcement Learning. NeurIPS, 2019.
George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning. JAIR, 61:215-289, 2018.
Steven LaValle. Rapidly-Explorinn Random Trees: A New Tool for Path Planning. Research Report 9811, 1998. 5
Vladimir Lifschitz. On the Semantics of STRIPS. In Workshop on Reasoning about Actions and Plans, 1986.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+ P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv:2304. 1477, 2023. 2,
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. In IJCAI, 2019.
Zhezheng Luo, Jiayuan Mao, Jiajun Wu, Tomás Lozano-Pérez, Joshua Tenenbaum, and Leslie Pack Kaelbling. Learning Rational Subgoals from Demonstrations and Instructions. In AAAI, 2023. 5, 9,
Jiayuan Mao, Tomas Lozano- Lozano-Perez, Joshua B. Tenenbaum, and Leslie Pack Kaelbing. PDSketch: Integrated Domain Programming, Learning, and Planning. In NeurlPS, 2022. 9
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum, and Leslie Pack Kaelbling. What planning problems can relational neural network solve? In NeurlPS, 2023. 9
Toki Migimatsu and Jeannette Bohg. Grounding Predicates through Actions, 2022. 2,
Dipendra Misra, John Langford, and Yoav Artzi. Mapping Instructions and Visual Observations to Actions with Reinforcement Learning. In EMNLP, 2017.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: LanguaeLe Pre- ree-training via Embodied Chain of Thought. 2rXiv:2305.1500 2023. 9
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, and Chelsea Finn. Learning Language- Conditioned Robot Behavior from Offline Data and Sowue- Annotation. In CoRL, 2022.
11

---

Published as a conference paper at ICLR 2024
Negin Nejati, Pat Langley, and Tolga Konik. Learning Hierarchical Task Networks by Observation. In ICML, 2006. 1,
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannanch Hajishirzi. Sameer Singh, and Roy Fox. Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling. In ICML, 2023. 9
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill Induction and Planning with Latent Language. ACL, 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020. 2, 5, 6, 7, 13
Tom Silver, Rohan Chitmis, Joshua Tenenbaum, Leslie Pack Kaclbling, and Tomás Lozano-Péres Learning Symbolic Operators for Task and Motion Planning. IROS, 2021.
Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano- Pérez, and Leslie Pack Kaelbling. PDDL. Planning with Pretrained Large Language Models. In NeurIPS Foundation Models for Decision Making Workshop, 2022.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized Planning in PDDL Domains with Pretrained Large Language Models. arXiv:2305. / 1014, 2023. 9
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. LLM-elanner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models. In ICCV, 2023. 9
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction Reinforcement Learning, Artif. Intell., 1 1211111: 1999. I
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In AAAI, 2011. 9
Stefanic Tellex, Nakul Gopalan, Hadas Kress- Gazit, and Cynthia Matuszek. Robots That Use Language. Annual Review of Control, Robotics, Autonomous Systems, 3525-55, 2020.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati, Large Language Models Still Can't i Plan (A Benchmark for LLMs on Planning and Reasoning about Change). In NeurIPS Foundation Models for Decision Making Workshop, 2022. 8
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models.
arXiv:2305.16291, 2023a. 7,2,7,8,
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-Wort World Multi-Task Agents, 2023b. 8
Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging Language to Learn Program Abstractions and Search Heuristics. In ICML, 2021. 9
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating Natural Language Planning Goals with Large-Language Models. arXiv:2302.05128, 2023.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the Minecraft: Generally Capable Agents for Open- World Environments via Large Language Models with Text- based Knowledge and Memory, 2023. 9
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
We will release a complete code repository containing full algorithm implementation, all baselines, and benchmark tasks. Here, we provide additional details on our implementational choices.
A.1 BENCHMARKS
Mini Minecraft (Fig. 5, top) is a procedurally gegerratd Minecraft-like benchmark (Chen et al., 2021; Luo et al., 2023) that requires complex, extended planning. The environment places agent 2D map containing various resources, tools, and crafting stations. The agent can use appropriate tools to mine new items from raw resources (e.g. use an axe to obtain wood from frees), collect resources into an inventory to craft new objects (e.g. combining sticks and iron ingots to craft a sword, which itself can be used obtain feathers from a chicken). The ability to create new objects that themselves permit new actions yields an enormous action space at each time step (>>2000 actions, considering different combinations of items use) and very long- Long-horizon tasks (26 steps for the most complex task, even without path-planning.) The provided environment predicates allow querying object types and inventory contents. Low-level actions allow the agent to move and apply tools to specific resources. To focus complex crafting, provide low-level move-to action to move directly to specified locations. Linguistic goal specifications are synthetically generated from a simple grammar over craftable objects and resources (e.g. Craft sword, Mine iron ore).
ALFRED (Fig. 5, bottom) is a household planning benchmark of human-annotated but formally verifiable tasks defined over simulated Unity environment (Shridhar et al., 2020). The interactive environment places an agent in varying 3D layouts, each containing appliances and dozens of household objects. The provided environment includes predicates for querying object types, object and agent locations, and classifiers over object states (eg. whether an object is hot or on). Low-level actions enable the agent to pick up and place objects, apply tools to other objects, and open, close, and turn on appliances. As specified in Shridhar et al. (2020), ground-truth high-level plans in the ALFRED benehmark compose 5-10 high-level operators, and low-level action trajectories have on average 50 low-level actions. There over 100 objects that the agent can interact with in each interactive environment.
As with Minecraft, we provide low-level method to move the agent directly to specified locations. While ALFRED is typically used to evaluate detailed instruction following, we focus on goal-only setting that only uses the goal specifications. The human-annotane underspecification, and errors with respect to the ground- ground-truth verifiable tasks (eg. people refer to tables without specifying if they mean the side table, dining table, or desk; light when there multiple distinct lamps; or a cabbage when they want lettuce).
goals introduce ambiguity,
A.2 ADDITIONAL METHODS IMPLEMENTATION DETAILS
A.2.1 LLM PROMPTING
We use 10C-33.-turrb-111 6k for all experiments and baselines. Here, we describe the contents of the I.IM few-shot prompts used our method in more detail. Symbolic Task Decomposition For all unsolved tasks, at each iteration, we sample a set of symbolic task descriptions as sequence of named high-level actions and their arguments. We construct a few-shot prompt consisting of the following components:
1. A brief natural language header (:s;; Given natural language goals, predict sequence of PDDL actions);
2. A sequence of example (l,, PA) tuples containing linguistic goals and example task de- compositions. To avoid biasing the language model in advance, we provide example task decompositions for similar, constructed tasks that do not use any of the skills that need to be learned in our two domains.
For example, on ALFRED, these example task decompositions are for example tasks (bake potato and put it in the fridge, place baked, grated apple top of the dining table, place a plate in a full sink., and pick up a laptop and then carry it over to the desk lamp, then restart the desk lamp.), and our example task decompositions suggest named operators
13

---

Published as a conference paper at ICLR 2024
BakeObject, GrateObject, FillObject, and RestartObject, none of which appear in the actual training set.
3. At iterations > O, we also provide a sequence of sampled (4, PA) tuples randomly sampled from any solved tasks and their discovered high-level plans. This means that few-shot prompting better represents the true task distribution over successive iterations.
In our experiments, we prompt with temperature=1.. and draw n=4 task decomposition samples per unsolved task.
Symbolic Operator Definition For all unsolved tasks, at each iteration, we sample proposed operator definitions consisting of pre,, eff conditioned on all undefined operator names that appear in the proposed task decompositions.
For each operator name, we construct few-shot prompt consisting of the following components:
1. A brief natural language header (You are a software engineer who will be writing planning operators in the PDDL planning language. These operators are based on the following PDDL domain definition.
2. The full set of environment predicates vocabulary of high-level environment predicates P, as well as valid named argument values (eg. object types).
3. A sequence of example name, prg,,pre, eff operator definitions demonstrating the PDDL definition format. As with task decomposition, of course, we do not provide any example operator definitions that we wish learn from our dataset.
4. At iterations > O, we include as many possible validated name, args, pre, eff operators defined in the current library (including new learned operators). If there are shared patterns between operators, this means that few-shot prompting also better represents the true operator structure over successive iterations.
In our experiments, we prompt with temperature=1.00 .0 and draw n=3 task decomposition samples per unsolved task. However, in our pilot experiments, we actually find that sampling directly from the token probabilities defined by this few-shot prompt does not produce sufficiently diverse definitions for each operator name. We instead directly prompt the LLM to produce up to N distinct operator definitions sequentially.
We find that GPT 3.5 frequently produces syntactically invalid operator proposals - proposed operators often include invent predicates and object types that are not defined in the environment vocabulary, do not obey the predicate typing rules, or do not have the correct number and types of arguments. While this might improve with finetuned or larger LLMs, we instead implement a simple post-processing heuristic to correct operators with syntactic errors, or reject operators altogether: as operator pre represented conjunctions of predicates, we remove any invalid predicates
and postconditions (predicates that are invented or that specify invalid arguments); we collect all arguments named across the predicates and use the ground truth typing produce the final args, and reject any operators that have valid postcondition predicates. This post-processing procedure frequently leaves operators underspecified (e.g., the resulting operators now missing necessary preconditions, which were partially generated but syntactically incorrect in the proposal); we allow our full operator learning algorithm to verify and reject these operators.
Symbolic Goal Proposal Finally, as described in 3.2, we also use an LLM to propose a set of candidate goal definitions as FOL formulas F defined over the environment predicates P for each task. Our prompting technique is very similar to that used in the rest of our algorithm. For each task, we we construct a few-shot prompt consisting of the following components:
1. A brief natural language header (You software engineer who will be writing goal definitions for robot in the PDDL plamning language.
2. The full set of environment predicates vocabulary of high-level environment predicates P. well as valid named argument values (eg. object types).
3. A sequence of example 4, fa language and FOL goal formulas. In our experiments, during training. unlike in the previous prompts (where including ground truth operators would solve the learning problem), do sample initial set of goal definitions from the training
14

---

Published as a conference paper at ICLR 2024
Mini Minecraft
LLM Predicts?
Library?
Mining
Crafting
Compositional
Code Policy Prediction
Sub-policies
12%
37% 100%
11%
Ours
Goel+Operasors
100%
100%
ALFRED (n=3 replications)
LLM Predicts?
Library?
Original (Simple + Compositional Tasks)
Code Policy Prediction Ours
Sub- Sub-policies
11%
Goal+Operators
70%
Table Results with GPT-4 as the LLM backbone: both Mini Minecraft (Top) ALFRED (Bottom). our algorithm recovers all required operators, which generalize to more complex compositional tasks. Switching to GPT-4 does not impact performance trends observed across the Code Policies (Voyager) baseline and our method.
distribution as our initial example supervision. We set supervision to a randomly sampled fraction (0.1) of the training distribution.
4. At iterations O, we also include 4, ft examples from successfully solved tasks.
In our experiments, we prompt with temperature=1.00 and draw n=4 task decomposition samples per unsolved task. As with the operator proposal, we also find that sampling directly from the token probabilities defined by this few-shot prompt does not produce sufficiently diverse definitions for each linguistic goal to correct for ambiguity in the human language (eg. to define the multiple concrete Table types that a person might mean when referring to table). We therefore again instead directly prompt the LLM to produce up to N distinct operator definitions sequentially.
We also post-process proposed goals using the same syntactic criterion remove invalid predicates in the FOL formula, and reject any empty goals.
A.2.2 POLICY LEARNING AND GUIDED LOW-LEVEL SEARCH
Concretely, we implement our policy-guided low-level action search as the following. We maintain dictionary D that maps subgoals (a conjunction of atoms) trajectories. When planning for a new subgoal if D contains the trajectory, we prioritize trying candidate low-level trajectories D. Otherwise, over all possible action trajectories. To populate D, during the BFS, we compute the difference in the environment state before and after the agent executes any sampled trajectory and the corresponding trajectory that caused the state change. Here the state difference can be viewed achieved by executing l. Rather than directly adding the (sg, I) as a key-value pair to D, we lift the trajectory and environment state change by replacing concrete objects in sg and by variables. Note that we update with each sampled trajectory in the BFS even if it doesn't It achieve the subgoal specified in the BFS search.
of candidate low-level action
fall back brute-force breadth-first search
subgoal sg
When the low-level search receives a subgoal sg, we again lift it by replacing objects with variables. and try match it with entries in D. If D contains multiple trajectories for given subgoal sg, track how often a given trajectory succeeds for a subgoal and prioritize trajectories with the most successes.
A.3 EXPERIMENTS
Learned Operator Libraries on Minecraft The following shows the full PDDL domain definition including the initial provided vocabulary of symbolic environment constants and predicates, initial pick and place operators and example operator, and all ensuing learned operators combined from the Mining and Crafting benchmarks.
(define (domain
2
(: requirements :strips) (:types tile object
6 inventory
object-type
(:constants
15

---

Published as a conference paper at ICLR 2024
Key object-type
11
WorkStation Object-type
12
Pickaxe - object-type
IronOreVein - -bjectt-ypp
IronOre object-type
Ironingot object-type
CoalOreVein - -bjeet-tyye
17
Coal object-type
GoldOreVein - objject-yee
GoldOre object-type
- obbect--ypp
21
22
Axe object-type
Tree object-type
25
Wood object-type
WoodPlank -oodPlank- -yy
26
27
Stick object-type
Sword object-type
Chicken object-type
Feather object-type
31
Arrow object-type
Shears object-type
Sheep object-type
34
Wool object-type
35
Bed object-type
36
Boat object-type
SugarcCanelant
SugarCane -
39
Paper object-type
40
Bowl object-type
41
PotatoPlant - tyee
Potato object-type
43
CookedPotalo object-type
44
BeetrootCrop -bbeec-ttye
45
Beetroot object-type
BeetrootSoup object-type
Hypothetical object-type 48
49
Trash object-type
(: predicates
(tile-up ?t1 tile ?t2 tile)
53
(tile-down ?t1 tile ?12 tile
54
(tile-left ?t1 tile ?t2 tile
(tile-right ?t1 tile ?12 tile)
57
(agent-at tile) (object-at ?x object ?t tile)
58
(inventory-holding ?i - inventory ?x object)
(inventory-empty ?i inventory)
62
(object-of-type object ?ot object-type)
6
(: action move-to
parameters (?t1 tile ?t2 tile)
67
precondition (and (agent-at ?t1))
68
efffect (and (agent-at ?12) (not (agent-at ))
(:action pick-up
71
:parameters (?i inventory object ?t tile
22
:precondition (and (agent-at ?t) (object-at ?x ?t) (inventory-empty ?i)
16

---

Published as a conference paper at ICLR 2024
effect (and (inventory-holddng ?i ?x) (not (object-at ?x ?t)) (not (
inventory-empty ?i)))
74
75
(cationn place-down
parameters (?i inventory ?x object ?t tile)
precondiiion (and (agent-at ?t) (inventory-holding ?i ?x))
effect (and (object-at ?x ?t) (not (inventory-holding ?i ?x))
78
inventory-emply ?i))
(: action mine-iron-ore
parameters (otoliinv inventory ?targetinv inventory ?x object ?
tool object ?target - object tile)
82
precondition (and
(agent-at ?t)
(object-at ?x ?t)
(object-of-type ?x IronOreVein)
86
(inventory-holding ?toolinv ?tool)
(object-of-type ?tool Pickaxe)
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
90
91
effect (and (not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
(not offject ?target Hypothetical)) (object-of-type ?target ironOre)
94
95
96
(: action mine-wood 2
99
parameters (?t tile ?x object ?toolinv inventory ?tool object
targetinv - inventory ?target - object)
100
precondition (and
102
(agent-at ?t)
103
(object-at ?t)
104
(object-of-type ?x Tree)
(inventory-holding ?toolinv ?tool)
(object-of-type ?tool Axe)
(inventory-emply ?targetinv)
107
(object-of-type ?target Hypothetical)
10s
effect (and
111
(not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
112
(not (object-of-type ?target Hypothetical))
113
114
(object-of-type ?target Wood)
116
117
(: action mine-wool1_0
118
parameters (?t tile ?x object ?toolinv inventory ?tool object
targetinv inventory ?target object)
120
precondition (and
121
(agent-at ?t)
122
(object-at ?x ?t)
(object-of-type ?x Sheep)
124
(inventory-holding ?toolinv ?tool)
125
(object-of-type ?tool Shears)
126
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
129
effect (and
130
(not (inventory-emply ?targetinv))
linventory-holdigg ?targetinv ?target)
(not offject ?target Hypothetical))
17

---

Published as a conference paper at ICLR 2024
(object-of-type ?target Wool)
134
135
136
(: action mine-potato.0 parameters (?t tile ?x object ?targetinv inventory ?target
object)
138
139
precondition (and
140
(agent-at ?t) (object-at ?x ?t)
(object-of-type ?x PotatoPlant)
145
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
144
effect (and
147
(not einventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
148
(not (object-of-type ?target Hypothetical))
149
(object-of-type Ptarget Potato)
152
153
(: action mine-sugar-cane parameters (?t tile ?x object ?toolinv inventory ?tool object targetinv inventory ?target object)
154
155
156
precondition (and
157
(agent-at ?t)
(object-at ?x ?t)
(object-of-type ?x SugarCanePlant)
100
(inventory-holding ?toolinv ?tool)
161
(object-of-type ?tool Axe)
162
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
164
165
effect (and
166
(not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
(object-of-type ?target SugarCane)
109
170
(: action mine-beetroot. parameters (?t tile ?x object ?toolinv inventory ?tool object
173
targetinv inventory ?target - object)
174
175
precondition (and
(agent-at ?t)
177
(object-at ?t)
178
(object-of-type ?x BeetrootCrop)
179
(inventory-holding ?toolinv ?tool)
180
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
182
183
effect (and
(not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
186
187
(object-of-type ?target Beetroot)
183
action mine-feather.1
191
parameters (?t tile ?x - object ?toolinv inventory ?tool object
targetinv inventory ?target - object)
192
precondition (and
18

---

Published as a conference paper at ICLR 2024
(agent-at ?t)
195
(object-at ?x ?t)
196
(object-of-type ?x Chicken)
197
(inventory-holding ?toolinv ?tool)
(object-of-type ?tool Sword)
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
200
201
effect (and
(not (inventory-empty ?targetinv))
(inventory-hoiding ?targetinv ?target)
205
(not (ofject-of-type ?target Hypothetical))
206
(object-of-type ?target Feather)
(: action mine-cobblestone parameters (?t tile ?x object ?toolinv
209
210
inventory ?tool object
targetinv inventory ?target - object)
precondition (and
213
(agent-at ?t)
214
(object-at ?x ?t)
215
(object-of-type ?x CobblestoneStash)
(inventory-holding ?toolinv ?tool)
217
(object-of-type ?tool Pickaxe)
218
(inventory-ampty ?targetinv)
219
(object-of-type ?target Hypothetical)
effect (and
222
(not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
223
224
(not (object-of-type ?target Hypothetical))
(object-of-type Ptarget Cobblestone)
226
237
238
(: action mine-gold-ore1 parameters (?t tile ?x object ?toolinv inventory ?tool object targetinv inventory ?target object)
230
231
precondition (and
(agent-at ?t)
(object-at ?x ?t)
(object-of-type ?x GoldOreVein)
235
(inventory-holding ?toolinv ?tool)
236
(object-of-type ?tool Pickaxe)
(inventory-empty ?targetinv)
(object-of-type Ptarget Hypothetical)
239
240
effect (and
(not (inventory-emply ?targetinv)) (inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical)) (object-of-type ?target GoldOre)
244
245
(: action mine-coal1_0 parameters (?t tile ?x object ?toolinv inventory ?tool object
248
targetinv inventory ?target - object)
249
precondition (and
(agent-at ?t)
252
(object-at ?t)
253
(object-of-type ?x CoalOreVein)
254
(inventory-holding ?toolinv ?tool)
(object-of-type ?tool Pickaxe)
19

---

Published as a conference paper at ICLR 2024
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
258
259
effect (and
(not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
262
263
(fbject-of-type ?target Coal)
264
(: action mine-beetroott parameters (?t - tile ?x object ?targetinv - inventory ?target object)
267
208
precondition (and (agent-at ?t) (object-at ?x ?t)
270
271
272
(object-of-type ?x BeetrootCrop)
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
275
276
effect (and
(not (inventory-emply ?targetinv))
(inventory-holding ?targetinv ?target) (not ofbject-of-type ?target Hypothetical)) (object-of-type ?target Beetroot)
279
280
281
action craft-wood-plank
284
parameters (?ingredientinvt inventory ?targetinv inventory ?
station object ?ingredient1 - object ?target object ?t tile
285
precondition (and
(agent-at ?t)
287
(object-at ?station ?t)
288
(object-of-type ?station WorkStation)
(inventory-holding ?ingredientinvl ?ingredient1)
289
(object-of-type Pingredient1 Wood)
iinventory-empty ?targetinv) (object-of-type ?target Hypothetical)
292
203
effect (and (not (inventory-empty ?targetinv)) (inventory-holding ?targetinv ?target) (not (object-of-type ?target Hypothetical))
296
297
208
(object-of-type ?target WoodPlank) (not (inventory-holding ?ingredientinvi ?ingredient1) ) inventory-empty ?ingredientinv1)
(not (object-of-type ?ingredientí Wood)
301
(fbject-of-type ?ingredient1 Hypothetical)
302
action craft-arrow
306
parameters (? (?ingredientinvl inventory ?ingredientinv2 inventory
targetinv inventory ?station object ?ingredient1 - object
ingredient2 object ?target object ?t tile)
precondition (and
(agent-at ?t)
309
(object-at ?station ?t)
310
(object-of-type ?station WorkStation)
(invantoyy-holding ?ingredientinv! ?ingredient1)
(object-of-type ?ingredient1 Stick)
(inventory-holding ?ingredientinv2 ?ingredient2)
313
314
(object-of-type ?ingredient2 Feather)
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
20

---

Published as a conference paper at ICLR 2024
318
effect (and
319
(not (inventory-empty ?targetinv))
320
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
(objectof--lyee Ptarget Arrow)
(not (inventory-holding ?ingredientinv1 ?ingredient1)
323
(inventory-emplyyynngeeee (ingreiiiii))g))iinnnnnnn))
324
325
(not (object-of-type ?ingredienti Stick))
(bjjettof-llype Pingreedent Hypothetical)
(not (inventory-holldin ?ingredientinv2 ?ingredient2))
(inennoryy-mmptyyyinneeee (ingredien)))))))))nninn))
308
329
(not (object-of-type ?ingredient2 Feather))
(object-of-type ?ingredient2 Hypothetical)
332
333
(: action craft-bbettoot--oupp..
334
parameters (?t - tile ?station object ?ingredientinvl inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2
object ?targetinv inventory ?target object)
335
336
precondition (and
(agent-at ?t)
(object-at ?station ?t)
339
(object-of-type ?station WorkStation)
340
(inventory-holding ?ingredientinvl ?ingredientt)
(object-of-type ?ingredientt Beetroot)
341
(inventory-holding ?ingredientinv2 ?ingredient2)
(object-of-type Pingredient2 Bowl)
344
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
345
346
effect (and
348
(not (inventory-empty ?targetinv))
349
(inventory-holding ?targetinv ?target)
350
(not (object-of-type ?target Hypothetical))
cobject-of-typeePtaree (not (inventory-holding ?ingredeeninn ?ingredient1)
BeetrootSoup)
eegeeooryeemmpyyyyiggereeeeennne (not (object-of-type ?ingredienti Beetroot))
353
354
Siiject-o--typeelinngeeeeeen Hypothetical) )
(not (inventory-holding ?ingredientinv2 ?ingredient2))
eeeeeooy-emmpyyyynnrreeeeettnt (not (object-of-type ?ingredient2 Bowl)) (object-of-type Pingredient2 Hypothetical)
357
358
359
362
(: action craft-paper
363
parameters (?t tile ?station - object ?ingredientinvt inventoryy ?
ingredient1 object ?targetinv inventory ?target object)
364
precondition (and
366
(agent-at ?t) (object-at ?station ?t)
307
(object-of-type ?station
?stiiiieessttiinn WorkStation)
(inventory-holding ?ingredientinvl ?ingredient1)
370
(object-of-type ?ingredient1 SugarCane)
371
(inventory-empty ?targetinv)
372
(object-of-type ?target Hypothetical)
effect (and
375
(not (inventory-empty ?targetinv))
376
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
(object-of-type ?target Paper)
21

---

Published as a conference paper at ICLR 2024
(not (inventory-holding ?ingredientinvi ?ingredient1) )
380
(inventory-emply ?ingredientinv1) (not (object-of-type ?ingredienti SugarCane))
381
382
(object-of-type ?ingredient1 Hypothetical)
385
acacioon craft-shears2
396
paraametess (?t tile ?station - object ?ingredientinv1 inventory ?
ingredient1 object ?targetinv inventory ?target object)
precondition (and
389
(agent-at ?t)
300
(object-at ?station ?t)
(object-of-type ?station WorkStation)
(inventory-holding ?ingredientinvl ?ingredientt)
(object-of-type Pingredient1 GoldIngot)
3V3
394
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
395
effect (and
308
(not (inventory-emppy ?targetinv))
(inventory-holding ?targetinv ?target)
309
(not (object-of-type ?target Hypothetical))
(object-of-type ?target Shears)
(not (inventory-holding ?ingredientinvt ?ingredient1) (inventory-empty ?ingredientinv1)
402
403
(not (object-of-type ?ingredient1 GoldIngot))
404
(object-of-type Pingredient1 Hypothetical)
407
408
action craft-bowl.1
409
parameters (?t tile ?station object ?ingredientinvt inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2
object ?targetinv inventory ?target object)
410
411
precondition (and
(agent-at ?t) (object-at ?station ?t)
414
(object-of-type ?station WorkStation)
(inventory-holding ?ingredientinvt ?ingredient1)
415
(object-of-type ?ingredient1 WoodPlank]
(inventory-holding ?ingredientinv2 ?ingredient2)
418
(object-of-type ?ingredient2 WoodPlank)
419
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
420
effect (and (not (inventory-empty ?targetinv)) (inventory-holding ?targetinv ?target)
423
424
(not (object-of-type ?target Hypothetical))
poppctt-o-ttypeeeParge Bowl)
(not (inventory-hollding ?ingredientinvt ?ingredient1)
unveentoyy-omoyyyynnneeeeennnn
428
(not (object-of-type ?ingredienti WoodPlank))
429
pibicttoo-typeeeiignreeeee Hypothetical)
(not (inventory-holding ?ingredientinv2 ?ingredient2))
peeeeooryemmmpyyyiigeeeeeennnnne (not (object-of-type ?ingredient2 WoodPlank))
432
433
434
(object-of-type ?ingredient2 Hypothetical)
437
(: action craft-boat :aarameeers (?t - tile ?station - object ?ingredientinv inventory
238
ingredient object ?targetinv inventory ?target object)
22

---

Published as a conference paper at ICLR 2024
precondition (and
(agent-at ?t)
442
(object-at ?station ?t)
44
(object-of-type ?station WorkStation)
(inventory-holding Pingredientiny ?ingredient)
(object-of-type Pingredient WoodPlank)
446
(inventory-ampty ?targetinv)
(object-of-type ?target Hypothetical)
447
effect (and (not (inventory-empty ?targetinv))
451
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
452
(object-of-lype ?target Boat)
?targe
(not (inventory-holding ?ingredientinv ?ingredient)) (inventory-emply Pingredieniyn
(not (object-of-type ?ingredient WoodPlank))
436
457
(object-of-type ?ingredient Hypothetical)
460
(: action cratt-cooked-potato..
461
parameters (?t - tile ?station - object Pingredientinvl inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2
object ?targetinv inventory ?target object)
462
463
precondition (and
464
(agent-at ?t) (object-at ?station ?t)
(object-of-type ?station WorkStation)
467
(inventory-holding ?ingredientinvi ?ingredient1) )
(object-of-type Pingredient1 Potato)
48
169
(inventory-holding ?ingredientinv2 ?ingredient2)
(object-of-type ?ingredient2 Coal)
471
(inventory-emply ?targetinv)
(object-of-type ?target Hypothetical)
472
473
effect (and (not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
476
(not (object-of-type ?target Hypothetical))
477
(object-of-type CookedPotato)
(not (inventory-holding ?ingredientivv ?ingredient1)
430
(not (object-of-type ?ingredienti Potato))
481
perect--f-typeeeiingedeeeeeee Hypothetical)
482
(not (inventory-holding ?ingredientinv2 ?ingredient2))
mnventryy-ommayyiingreeeennni (not (object-of-type ?ingredient2 Coal))
485
(object-of-type ?ingredient2 Hypothetical)
436
(: action craft-gold-ingot.
400
parameters (?t tile ?station object ?ingredientinvt inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2 object ?targetinv inventory ?target object)
192
precondition (and
193
(agent-at ?t)
(object-at ?station ?t)
(object-of-type ?station ? WorkStation)
(inventory-holding ?ingredientinvl ?ingredient1)
(object-oi-type ?ingredient GoldOre)
407
(inventory-holding ?ingredientinv2 ?ingredient2)
198
(object-of-type ?ingredient2 Coal)
(inventory-emply ?targetinv)
23

---

Published as a conference paper at ICLR 2024
(object-of-type ?target Hypothetical)
503
effect (and
504
(not (inventory-emply ?targetinv))
Linventory-holdingg ?tagggeinnnnnnrggggiiiii ?target)
(not (object-of-type ?target Hypothetical))
507
(object-of-yype Ptarget GoldIngot)
(not (inventory-holding ?ingredientinvi ?ingredient1) ))
508
(not (object-of-typp ?ingredienti GoldOre))
oea.oo--f--yypeeiinnneeeeeeee Hypothetical)
(not (inventory-holding ?ingredientinv2 ?ingredient2))
512
penettoyyemmpyyyiiigeeiiinnnn (not (object-of-type ?ingredient2 Coal))
513
(object-of-type Pingredient2 Hypothetical)
516
517
518
(: action craft-stick_0
parameters (?t tile ?station object Pingredientinv1 inventory ?
ingredient1 object ?targetinv inventory ?target object)
520
521
precondition (and
(agent-at ?t)
(object-at ?station ?t)
524
(object-of-type ?station WorkStation)
525
(inventory-holding ?ingredientinvl ?ingredient1)
536
(object-of-type ?ingredient1 WoodPlank)
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
$29
530
effect (and (not (inventory-empty ?targetinv))
(inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
533
534
(object-of-type Ptarge Stick)
(not (inventory-holding ?ingredientinvi ?ingredient1) )
535
(Inertooy-emmiiyy Pingrriiinn))n
(not (object-of-type ?ingredient1 WoodPlank))
(object-of-type ?ingredient1 Hypothetical)
538
539
(: action craft-sword_0
542
parameters (?t tile ?station object ?ingredientinvt inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2
object ?targetinv inventory ?target - object)
precondition (and
545
(agent-at ?t) (object-at ?station ?t)
546
(object-of-type ?station ?stattio WorkStation)
(inventory-holding ?ingredientinvl ?ingredientt)
(object-oi-type Pingredient Stick)
(inventory-holding ?ingredientinvv ( ?ingredient2)
550
(object-of-type ?ingredient2 Ironingot)
551
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
555
effect (and
556
(not (inventory-emply ?targetinv)) (inventory-holding ?targetinv ?target)
(not (object-of-type ?target Hypothetical))
<59
(object-of-type ?target Sword)
(not (inventory-holding ?ingredientinvi ?ingredient1) )
560
(inventory-emply
(not (object-of-type ?ingredienti Stick))
24

---

Published as a conference paper at ICLR 2024
(object-of-type ?ingredienti Hypothetical)
(not (Inventory-holding ?ingredientinv2 ?ingredient2))
565
(inventory-emply ?ingredientinv2) (not (object-of-type ?ingredient2 Ironingot))
566
(object-of-type ?ingredient2 Hypothetical)
500
570
(: action oraft-bed
parameters (?t tile ?station object ?ingredientinv1 inventory ?
ingredient1 object ?ingredientinv2 inventory ?ingredient2
object ?targetinv inventory ?target object)
572
573
precondition (and
(agent-at ?t)
(object-at ?station ?statioo ?t)
(object-of-type ?station WorkStation)
576
577
(inventory-holding ?ingredientinv1 ?ingredient1)
(object-of-type ?ingredient1 WoodPlank]
578
(inventory-holding ?ingredientinv2 ?ingredient2)
(object-of-type ?ingredient2 Wool)
(inventory-ampty ?targetinv)
581
(object-of-type ?target Hypothetical)
S82
effect (and
585
(not (inventory-emply ?targetinv))
(inventory-holding ?targetinv ?target)
596
(not (object-of-type ?target Hypothetical))
587
(object-of-type ?target Bed)
(not (inventory-hollding ?ingredientinvi ?ingredient1)
5X0
(not (object-of-type ?ingredienti WoodPlank))
591
Soopect-of-tyeee/iingeeeeeieel H
392
(not (inventory-holding ?ingredientinv2 ?ingredient2))
eerenooy-emmmyyy/nggeeeeeeennn (not (object-of-type ?ingredient2 Wool)] (object-of-type ?ingredient2 Hypothetical)
594
505
596
action craft-iron-ingot 2
599
parameters (?t - tile ?station - object ?ingredientinvt inventory ?
600
ingredient1 object ?ingredientinv2 inventory ?ingredient2 object ?targetinv inventory ?target object)
602
precondition (and
603
(agent-at ?t) (object-at ?station ?t)
(object-of-type ?station WorkStation)
(inventory-holding ?ingredientinvi ?ingredient1) )
606
(object-of-type ?ingredient1 IronOre)
607
(inventory-holding ?ingredientinv2 ?ingredient2)
(object-of-type ?ingredient2 Coal)
(inventory-empty ?targetinv)
(object-of-type ?target Hypothetical)
611
612
effect (and
(not (inventory-emppy ?targetinv)) (inventory-hoiding ?targetinv ?target)
615
616
(not (object-of-type ?target Hypothetical))
617
(object-of-type ?target Ironingot)
(not (inventory-holding ?ingredientinvt ?ingredient1)
(not (object-of-type ?ingredienti IronOre))
620
(object-of-type ?ingredient Hypothetical)
621
(not (inventory-holding ?ingredientinv2 ?ingredient2))
(inventory-emply Pingredientinv2)
25

---

Published as a conference paper at ICLR 2024
(not (object-of-type ?ingredient2 C
(object-of-type Pingredient2 Hypothetical)
626
627
Learned Operator Libraries ALFRED The following shows the full PDDL domain definition including the initial provided vocabulary symbolic environment constants and predicates, initial pick and place operators, and all ensuing learned operators.
(define (domain ) (: requireennts :adl
(:types agent location receptacle object rtype otype
5
(: constants
Candle otype
ShowerGlassType - otype
10
CDType otype
TomatoType otype
MirrorType otype
ScrubBrushType otype
MugType - otype
ToasterType otype
PaintingType otype
CellPhoneType otype
18
LadleType otype
19
BreadType otype
20
PotType otype
BookType otype
TennisFacketType otype
ButterKnifeType - otype
23
34
ShowerDoorType otype
25
KeyChainType otype
BaseballBatType otype
27
EggType otype
28
PenType otype
29
ForkType otype
VaseType otype
ClothType otype
32
Window otype
33
PencilType otype
34
StatueType otype
LightSwitchType otype
36
WatchType otype
37
SpatulaType -oype
38
PaperTowelRollTyppe otype
FloorLampType otype
KettleType - otype
41
SoapBottleType otype
42
BootsType otype
43
TowelType otype
PillowType otype
AlarmClockType otype
46
PotatoType otype
47
ChairType - otype
PlungerType otype
SprayBottle otype
HandTowelTyye otype
50
51
BathtubType otype
RemoteControlType otype
PopperShakerType otype
Plate otype
26

---

Published as a conference paper at ICLR 2024
BasketBallType otype
DeskLampType otype
57
FootstoolType otype
58
GlassbottleType otype
PaperTowelType otype
CreditCardType otype
61
PanType - otype
ToiletPaperType - otype
62
63
SaltShakerType otype
PosterType otype
ToiletPaperRollType otype
66
LettuceType- otype
67
WineBottleType - otype
KnifeType otype
LaundryHaameeLiddType otype
SpoonType - otype
70
71
TissueBoxType otype
BowlType - otype
otype
BoxType otype
SoapBarType otype
PlustPlantType otype
75
76
NewspaperType otype
CupType otype
DishSpongeTTyp otype
79
LaptopType otype
80
TelevisionType otype
81
StoveKnobType otype
CurtainsType otype
BlindsType otype
TeddyBearType otype
84
AppleType otype
86
WateringCanType otype SinkType otype
89
ArmChairType - rtype
90
BedType rtype BathtubBasinTypp rtype
DresserType rtype
Safe SafeType--ttype rttpp
93
DiningTableType - rtype
94
SofaType rtype
HandTowelHoiderType rtype
97
StoveBurnerType rtype CartType rtype
98
99
DeskType - rtype
CoffeeMacchieeTypee rtype
MicrowaveTyype rtype
102
ToiletType rtype
103
CounterTopType rtype
GarbageCanType rtype
105
CoffeeTableType rtype
CabinetType rtype
SinkBasinType rtype
107
108
OttomanType - rtype
ToiletPaperHangerTyyee rtype
TowelHolderTypee rtype
111
FridgeType rtype
112
DrawerType rtype
113
SideTableType rtype
ShelfType rtype LaundryHamperType rtype
116
117
118
:: Predicates defined on this domain. Note the types for each predicate.
27

---

Published as a conference paper at ICLR 2024
(: (:predicates
120
(atLocation ?a agent ?1 location)
121
(receptacleALLocationn ?r receptacle ?I location)
122
[objectAtLocation ?o - object ?I location)
123
iiReccetaccl ?o object ?r receptacle)
(receptacleType ?r receptacle ?t rtype)
125
(objectType ?0 object ?t
otype) -type)
126
(holds ?a agent
object)
127
(holdsAny ?a agent)
(holdsAny ReceptacleObject ?a agent)
130
(openable - receptacle)
131
(opened ?r receptacle)
(isClean ?o object)
(cleanable ?o object)
(isHot object)
134
135
(heatable ?o object)
136
(isCool ?o object)
(coolable ?o object)
(toggleable ?o object)
(isToggled ?o object)
139
140
(liceeblle ?o object)
(isSliced ?o object)
(:action PickupObjeccNotlneeeeeteeee
143
144
parameters (?a - agent precondition (and
location ?o object)
145
[atLocation ?a ?1)
lobjectAtLocation ?o ?1)
148
(not (holdsAny ?a))
149
(forall (?re receptacle)
150
(not (inReceptacle ?o ?re))
152
153
154
effect (and (not (objectAtLocation ?o ?I))
hhllss ?a ?o) (holdsAny ?a)
157
158
161
(: action PutObjectInReceptaclee
162
:parameters (?a - agent location ?ot otype ?o object ?r receptacle)
163
precondition (and
(atLocation ?a ?I)
165
(receptacleAlLocation ?r ?I)
166
(objectType ?o ?ot)
167
(holds ?a ?o)
(not (holdsAlyyRcceiaaclebbeee ?a))
170
effect (and
(inReceptacle ?o ?r)
171
(not (holds ?a ?0))
(not (holdsAny ?a)
174
(objectAtLocation ?o ?1)
175
176
a:atioo PickupobjectllReceetaelee :parameters (?a - agent location ?o object ?r receptacle
179
180
:rrecondition (and (atLocation ?a ?1)
28

---

Published as a conference paper at ICLR 2024
(objectAtLocation ?o ?1)
183
iinReceptacle ?o ?r)
184
(not (holdsAny ?a))
185
effect (and (not (objeciAtLocation ?o ?1))
188
(not (inReceptacle ?o ?r))
189
(holds ?a ?0)
190
(holdsAny ?a)
193
194
(:action RinseObject
:parameters (? (#toolreceptacle receptacle ?a agent ?1
location ?o object)
196
197
precondition (and
198
receptacleTTyp ?toolreceptacle SinkBasinType) attocation ?a ?1)
rrecppiaclettLocation ?toolreceptacle ?1) (objectAtLocation ?1)
201
202
(cleanable ?o)
efffect (and (isClean ?o)
205
206
207
acction TurnOnObject 2
210
:parameters (?a agent location ?o object)
211
212
precondition (and (atLocation ?a ?1)
214
(objectAtLocation ?o ?1)
215
(toggleable ?0)
216
effect (and (isToggled ?o)
219
220
a:action CoolObject.0
223
:parameters (? (?toolrecepptacle receptacle ?a agent ?1
location ?o object)
234
precondition (and
rreceptaceeType ?toolreccepaccee FridgeType) |atLocation ?a ?1) TreceptacleAtLocattonn ?toolreceptacle ?I) (holds ?a ?o)
227
228
229
effect (and (isCool ?0)
232
233
(: action SliceObject.1
236
:parameters (?toolobject object ?a agent ?1 location ?o object)
237
precondition (and
(objectType ?toolobject ButterKnifeType))
240
(atLocation ?a ?1)
241
(objectAtLocation ?o ?1)
(sliceable ?o)
(holds ?a ?toolobject)
29

---

Published as a conference paper at ICLR 2024
effect (and isSliced ?o)
246
247
(: action SliceObject.0
250
:parameters (otoolobject object ?a agent ?1 location ?o object)
251
precondition (and
(objectType ?toolobject KnifeType)
254
(atLocation ?a ?1)
(objectAtLocation ?o ?1)
255
256
(sliceable ?0)
(holss ?a ?toolobject)
258
259
effect (and (isSliced ?o)
260
263
(: action MiorowaveObject.0 :parameters (?toolreceppaccle receptacle ?a agent ?1 location ?o object)
264
206
precondition (and receptacleTTyp ?toolreceptacle MicrowaveType) [atLocation ?a ?1)
267
268
AecloticleAtLoocation ?toolreceptacle ?1)
(holds ?a ?o)
271
272
effect (and (isHot ?0)
273
276
277
30


Reviews:
[{"review_id": "GxQqV9RJvr", "reviewer": "Reviewer_uRvH", "summary": "The paper studies the challenge of long-horizon planning. To make this more tractable, the authors leverage hierarchical planning using temporal action abstractions, breaking down intricate tasks into manageable subproblems. The novel contribution is a system that harnesses language to derive symbolic action abstractions and associated learnable low-level policies. By querying large language models (LLMs), the system proposes symbolic action definitions, subsequently integrating these into a hierarchical planning framework for grounding and verification. This approach is framed within a multitask-reinforcement-learning objective, where an agent interacts with an environment to solve tasks described in natural language. The ultimate aim is to construct a library of grounded actions that are both planning-compatible and efficient.", "questions": "Questions:\n\n- The two-stage prompting strategy involves symbolic task decomposition followed by symbolic operator definition. How does the system ensure that the decomposition is optimal or near-optimal for complex tasks?\n\n- The author mentioned that one of the common failures on the ALFRED benchmark was due to goal misspecification, especially when faced with ambiguities in human language. Could you elaborate on how the system currently handles such ambiguities and if there are plans to improve this aspect?\n\n- The paper demonstrates that action libraries from simpler tasks in Mini Minecraft generalize to more complex tasks. Are there plans to test this generalization capability in more diverse environments or tasks outside of the current benchmarks?\n\n- How scalable is the proposed system? Specifically, if the number of tasks or the complexity of the environment increases significantly, how would the system's performance be affected?\n\nSuggestions:\n\n- Might consider introducing an interactive feedback loop where the system can ask clarifying questions when faced with ambiguous goals or tasks. This could help in refining the task understanding and improve planning accuracy.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The system leverages language to derive symbolic action abstractions, a unique approach to decomposing complex tasks, and subsequently verifies them within a hierarchical planning framework, ensuring the practical applicability of the abstractions,  which was tested on two benchmarks, Mini Minecraft and ALFRED, and outperformed other baseline methods that incorporate language models into planning.\n\nThe paper presents a commendable effort in bridging the capabilities of large language models with hierarchical planning, the innovative approach of using language to derive action abstractions is particularly noteworthy.", "weaknesses": "- Goal Misspecification: Failures on the ALFRED benchmark often occurred due to goal misspecification, where the LLM did not accurately recover the formal goal predicate, especially when faced with ambiguities in human language.\n\n- Policy Inaccuracy: The learned policies sometimes failed to account for low-level, often geometric details of the environment.\n\n- Operator Overspecification: Some learned operators were too specific, e.g., the learned SliceObject operator specified a particular type of knife, leading to planning failures if that knife type was unavailable.\n\n- Limitations in Hierarchical Planning: The paper acknowledges that it doesn't address some core problems in general hierarchical planning. For instance, it assumes access to symbolic predicates representing the environment state and doesn't tackle finer-grained motor planning. The paper also only considers one representative pre-trained LLM and not others like GPT-4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3sshSJaIIJ", "reviewer": "Reviewer_ukwA", "summary": "This paper proposed to leverage LLM to solve long-horizon planning problems by dynamically building a library of symbolic action abstractions and learning a low-level policy to execute the subgoals. They conducted experiments and relevant ablation studies on Mini Minecraft and ALFRED benchmarks to demonstrate the effectiveness of the proposed method.", "questions": "As stated in the Weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The method is technically feasible.\n+ The writing is easy to follow. \n+ Representing abstract actions with symbols and reasoning them with LLM is an interesting attempt.", "weaknesses": "+ **Lack of novelty.** This pipeline is reminiscent of the Voyager model, which also aims to tackle long-horizon tasks via the creation of a dynamic skill library. This limits the contributions of this paper. It is better to highlight the differences between this work and Voyager. \n\n+ **Benchmark is too simple.** It should be noted that the test environment used in the current work (Mini Minecraft) is significantly less complex than the original Minecraft version, resulting in reduced task difficulty. Especially the success rate in Mini Minecraft even reaches 100%. \n\n+ **Concerns about generalization ability.** The proposed method relies heavily on symbolic representations. I'm concerned that it may be difficult to generalize to complex real-world environments, which are often not easily symbolized. \n\n+ **Missing important citations.** [1, 2] are also important methods that leverage LLM as the planner to decompose the long-horizon task into subgoals. I suggest the authors to include some discussion and comparison of such methods. \n\n[1] Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, https://arxiv.org/abs/2302.01560\n\n[2] Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory, https://arxiv.org/abs/2305.17144", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbbERVwVEL", "reviewer": "Reviewer_xxKc", "summary": "In this paper, the authors propose to exploit the world knowledge of LLMs for learning action abstractions for hierarchical planning. These action abstractions can then be used to solve long-horizon planning problems, by decomposing a goal into subgoals and solving them using bi-level planning. More specifically, given a task and symbolic state, the authors use LLMs to propose symbolic (high-level) operators and their corresponding definitions (in PDDL) which are then used by a bi-level planner to generate a feasible low-level plan. The useful operators (planning-compatible and grounded) are retained in an operator library (i.e. reusable) and used for subsequent tasks, including those that require the composition of learned operators.", "questions": "1. In Sec 3.1 (Symbolic operator definition), is there a process through which you identify and/or discount semantically similar (redundant) operators from being added in the operator library since the LLM generation is not conditioned on it (i.e. the LLM is not aware of the existing operators in the library).\n\n2. In Sec 3.4 (Scoring LLM Operator Proposals), the operators are selected based on their executability in the low-level planning, but the overall goal is not accounted for. Wouldn't this lead to the selection of some operators that are just \"feasible\" but not \"useful\"?\n\nMinor Comment:\n* Sec 3.4: s/b > \\tau_r", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the paper is well-motivated, clearly written, and supported by strong empirical evidence.\n2. The proposed idea of exploiting knowledge of LLMs for action abstraction is intuitive and effective and would be of interest to the planning and decision-making community.", "weaknesses": "It would be nice to have some statistics on the length of the plan sequence (both in terms of high-level and low-level actions), and the number of learned operators, to get an understanding of the task complexity (especially for the compositionality experiments) in Minecraft and ALFRED domains. It is hard to get an idea from just the empirical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gar9SFqqgk", "reviewer": "Reviewer_ikxW", "summary": "The authors present a method to iteratively learn a set of action operators that can be used by a symbolic planner to generate high-level plans that are then refined into a series of low-level plans. The action operators are learned using a LLM and their selection is guided by a reward signal. To propose a set of action operators, the LLM decomposes language-based instructions into series of high-level actions. High-level actions that do not have a corresponding action operator are then passed to the LLM (along with a few-shot prompt) for the LLM to generate an operator definition consisting of the list of variables the action operates over, the preconditions that must be met to execute the action, and the effect the action has on the environment. The set of proposed action operators are then evaluated by planning with them to solve environment tasks, and are scored based on how often they are used and how often their use leads to task success. Only action operators with high scores are retained. The authors evaluate on Mini Minecraft and Alfred, and compare against several baselines that use LLMs to provide a low-level sequence of actions, specify subgoals, and to specify the plan as code. Across tasks and baselines, the proposed method performs best.", "questions": "- In section 3.2, the authors state that at each iteration operators are learned for only those tasks that were not solved in the previous iteration. How often were the found plans subpar? For example, taking unnecessary, but valid actions? In the experiment section, the tasks are listed as randomly ordered. How sensitive was the found action operator library to the task ordering?\n- In the results section the authors discuss Alfred failure cases as including \"operator over specification\". When over specification occurred, were multiple instances of the action with different objects seen? For example, a slice object with butter knife and one with steak knife. Were the over specifications arbitrary or driven by the training data? For example, a steak knife was chosen even through a butter knife would also work versus the sharpness level was needed to cut the object.\n- The authors suggest that encouraging more diverse proposals could address the failure mode. Was soliciting more diverse proposals attempted? Why would more diverse proposals address operator over specification? \n- Why Alfred instead of Habitat?\n- How accurate were the different parts of the LLM's output? How correct were the mappings from language description to goal specification? \n- Might the Mini Minecraft experiments, while good to test how composable the action abstractions are, simplify the action abstraction process by learning the action abstractions on the simpler tasks for which it is easier to identify action abstractions that are more primitive? Compared to learning the actions on the compositional tasks to see how well the method is able to identify useful and flexibly reusable action abstractions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and easy to follow. \n- The authors incorporate LLM to address the challenging problem of identifying abstract actions.\n- The baselines evaluate different ways to incorporate LLMs into planning and action selection tasks.\n- The environment on which the methods are evaluated assess actions of different complexity.", "weaknesses": "- It would be helpful in the results section, \"How does or approach compare to using the LLM to predict just goals, or predict task sequences?\", to call out the specific parts of the proposed algorithm that address the limitations observed in the baseline approaches.\n- It would have been beneficial to include experiments with multiple LLMs in order to understand the required LLM characteristics.\n- It is not clear from the reported experiments and results how much noise the system can handle.\n- There are no comparisons to systems that rely on hand-coded action abstractions or other methods for identifying/learning the action abstractions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ue6URa3Ohf", "reviewer": "Reviewer_geyi", "summary": "The paper proposes a method that learns a library of symbolic action abstractions (i.e., high-level actions) using LLMs.\nGiven a natural language instruction and a state (object classes, predicates, etc.), the proposed method uses LLMs to plan a sequence of high-level actions.\nIn the case of the undefined operator of any high-level action, it further uses LLMs to define the operator based on in-context examples.\nSuch obtained operators are iteratively refined.\nThe low-level actions to conduct each high-level action are acquired by BFS such that they satisfy the desired subgoal state.\nThe low-level action policies are updated based on rewards from the environment after task completion with the policies.\nThe proposed method outperforms baselines in its empirical validations based on ALFRED and Mini Mincraft.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally written well and easy to follow.\n- The paper tackles an important issue of action grounding present in LLMs used for action planning.\n- Using LLMs to acquire high-level actions and their unknown operators looks reasonable and intriguing.\n- The two-staged pipeline for the generation of candidate operator definition is well-motivated and sounds sensible.\n- The proposed method achieves strong performance over the baselines by noticeable margins.", "weaknesses": "- Some assumptions made are a bit practically unrealistic. For example, $\\Phi$ is assumed to be perfect and all environmental information is known, but they are usually not the case, especially for task planning for robotic agents (Shridhar et al., 2020; Krantz et al., 2020; Weihs et al., 2021).\n- Obtaining high-level action abstractions needs the corresponding low-level policies to generate low-level actions but it seems that it requires extensive interaction with environments (e.g., brute-force search to find a low-level action sequence that satisfies the subgoal condition). Can the proposed method be also applied to some offline scenarios without these interactive environments? And is there any efficient approach to this?\n- The detail of the baseline in Table 1 is unclear. For example, for \"Code Policy Prediction,\" the authors prompt the LLM to predict 1) imperative code policies in Python and 2) the function call sequences with arguments. What is the modification made for each? \n\n\\* Krantz et al. Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments. In ECCV, 2020.\n\n\\* Weihs et al. Visual Room Rearrangement. In CVPR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)