Learning Compositional Behaviors from Demonstration and Language
Authors: Weiyu Liu
Neil Nie
Ruohan Zhang
Jiayuan Mao
Jiajun Wu
Venue: CoRL 2024 (2024)
Keywords: Manipulation
Planning Abstractions
Learning from Language
TL;DR: We introduce a new framework for long-horizon robotic manipulation by integrating imitation learning and planning with models automatically constructed from language.

Abstract: We introduce Behavior from Language and Demonstration (BLADE), a framework for long-horizon robotic manipulation by integrating imitation learning and model-based planning. BLADE leverages language-annotated demonstrations, extracts abstract action knowledge from large language models (LLMs), and constructs a library of structured, high-level action representations. These representations include preconditions and effects grounded in visual perception for each high-level action, along with corresponding controllers implemented as neural network-based policies. BLADE can recover such structured representations automatically, without manually labeled states or symbolic definitions. BLADE shows significant capabilities in generalizing to novel situations, including novel initial states, external state perturbations, and novel goals. We validate the effectiveness of our approach both in simulation and on real robots with a diverse set of objects with articulated parts, partial observability, and geometric constraints.

Learning Compositional Behaviors from Demonstration and Language
Weiyu Liu",, Neil Nie",, Ruohan Zhang', Jiayuan Mao Jiajun Wull Sttanford University MIT
Abstract: We introduce Behavior from Language and Demonstration (BLADE), a framework for long-horizon robotic manipulation by integrating imitation learning and model-based planning. BLADE leverages language-annotatte demonstrations, extracts abstract action knowledge from large language models (LLMss), and con- structs a library of structured, high-level action representations. These represen- tations include preconditions and effects grounded in visual perception for each high-level action, along with corresponding controllers implemented as neural network-based policies. BLADE can recover such structured representations auto- matically, without manually labeled states or symbolic definitions. BLADE shows significant capabilities generalizing to novel situations, including novel initial states, external state perturbations, and novel goals. We validate the effectiveness of our approach both in simulation and real robot with diverse set of objects with articulated parts, partial observability, and geometric constraints.
Keywords: Manipulation, Planning Abstractions, Learning from Language
Introduction
Developing autonomous robots capable of completing long-horizen manipulation tasks a significant milestone. We want build robots that directly perceive the world, operate extended periods, generalize to various states and goals, and robust perturbations. A promising direction is to combine learned policies with model-based planners, allowing them to operate different time scales. In particular, imitation learning- based methods have proven highly successful in learning policies for various "behaviors," which usually operate over a short time span [e.g.. 1]. To solve more complex and longer-horizz horizon tasks, we can compose these behaviors by planning in abstract action spaces [2-4], latent spaces [5], or via large pre-trained models such as large language models [6].
However, of the key challenges of all high-level planning approaches is the automatic acquisition of an abstraction for the learned "behaviors" to support long-horizon planning. The goal of this behavior abstraction learning is to build representations that describe the preconditions and effects of behaviors, to enable chaining and search. These representations should depend on the environment, the set of possible goals, and the specifications of individual behaviors. Furthermore, these representations should be grounded on high-dimensional perception inputs and low-level robot control commands.
Our insight into tackling this challenge is to leverage knowledge from two sources: the low-level, mechanical understanding of robot-object contact, and the high-level, abstract understanding of object-object interactions described in language that can be extracted from language models as the knowledge source. Our framework, behavior from language and demonstration (BLADE), takes as input a small number of language-annokes based on which object is in contact with the robot. Then, uses a large language model (LLM), conditioned the contact sequences and the language annotations, propose abstract behavior descriptions with preconditions and effects that best explain the demonstration trajectories. During training, we extract the state abstraction terms from the preconditions and effects (e.g., turned-on,
demonstrations (Fig. la). segments each trajectory
denotes equal contribution. denotes equal advising. Project page and videos: https://blade-boo htw.bbllddd-bbg.iiihheh bot.github.io/.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

(A) Armotated Human Demonstrationa Task: Boll Water Stove
Picce in Bink
Move Foueet Head
Tarn On Fouces
Mave Faucet AwOy
Placo POt on Tobic
MOve kettle Stowe
Examples:
Geometry
(b) Generalizatioos
Perturbations
Recovery: move faucett place
Unseen Initial
State Parturbation
Candition
Kettll
Kettie Filled
Geometric constraint: stove blocked
Recovery: move pot to table
Geometric
Observabiiity
Constrolints
Stove not blocked
Kettle Filled & On Stove
Figure 1: BLADE, a robot manipulation framework combining imitation learning and model- planning. (a) BLADE takes language-amnotated annotated demonstrations as training data. (b) It generalizes to unseen initial conditions, state perturbations, and geometric constraints. (c) In the depicted scenarios, BLADE recovers from perturbations such moving the kettle of the sink, and resolves geometric constraints including blocked stove.
aligned-with), and learn their groundings on perception inputs. We also learn the control policies associated with each behavior (e.g.. turn on the faucer).
Our model offers several advantages. First, unlike prior work that relies on manually defined state abstractions additional state labels, our method automatically generates state abstraction labels annotated demonstrations and LLM-proposed behavior descriptions. BLADE recovers the visual grounding of these abstractions without any additional label. Second, BLADE generalizes to novel states and goals by composing learned behaviors using planner. Shown in Fig. lb, it can handle various novel initial conditions and external perturbations that lead to unseen states. Third, our method can handle novel geometric constraints (Fig. Ic) and partial observability from articulated bodies like drawers.
based on the language-annotated
Related Work
Composing skills for long-horizon manipulation. A large body of model- planning methods use manually-defined transition models [2, 7-12] or models learned from data [13-18] to generate long-horizon plans. However, learning dynamics models with accurate long-term predictions and strong generalization remains challenging. A related direction is to introduce hicrarchical structures into the policy models [19-25], where different methods can segment continuous demonstrations into thort-horizon skills [23, 26, 27]. Facing the challenges in modeling action dependencies, these methods limited to following sequentially specified subgoals. Some work addresses this issue by learning the dependencies between actions from data, but they require large-scale supervised datasets [28-31]. Our approach is related to methods that learn symbolic action representations [32- 36]; the difference is that BLADE uses a LLM to generates causal models of the environment and learns their groundings on sensory inputs.
Using LLMs for planning. Many researchers have explored using LLMs for planning. Methods for direct generation of action sequences [37, 38] struggle produce accurate plans [39, 40]. Researchers have also leveraged LLMs translators from natural language instructions to symbolic goals [41-44], as generalized solvers [45], memory modules [46], and as world models (47,48) To improve the planning accuracy, prior work has explored techniques including using programs [49, 50), learning affordance functions [6, 51], replanning [52], finetuning [53-55], embedding reasoning in a behavior tree [56], and VLM-based decision-makee
[57, 58]. BLADE shares similar spirit as methods using LLMs generate taaning--oopatiinn action representations [59-61]. However, they make assumptions on the availability of state abstractions, while BL.ADE grounds I.L.M-genrrate action definitions without additional labels. Also complementary methods that leverage these representations for skill learning [62, 63), our approach uses them for composing skills in novel ways.

---

(6) Annotated Human Demonstrations
(g) Beneralization
lractdas cpet-cabinetsdde" (precaudition (eat Likedeoneppes Mooo)) Egr
Contact Segnmettttnnnnnionnn
LLM Proposal
(clasearriper)) (opce-gr (rupper)))
Open Catinet Door
Place On Stove
Learning Algorithms
diceed]aeiinetddon aboodidrawer
Novel Goal States, initesbog, kectlel
Library
Classifier Library
Bi-Level Planning
"Grbb Kettie
Cabinet
Diffusion
Policy
7L
"heppeidsnwen"
"tchBlddccss"
Classifiar
Model
Falio
Fates
Robot Actions
(b) Structured Behavior Representations
Figure Overview BLADE. (a) BLADE receives language-annotann annotated human demonstrations. (b) segments demonstrations into contact primitives, and learns a structured behavior representation. (c) generalizes to novel conditions by leveraging bi-level planning and exccution to achieve goal states.
Problem Formulation
We consider the problem of learning lunguage-conditioned goal-reaching manipulation policy. Formally, the environment is modeled as tuple (X,4,T) where X' is the raw state space, 2/ is the low-level action space, and T X' x u + X' is the transition function (which may be stochastic and unknown). Furthermore, the rohot will receive observations o E O that may be partially observable views of the states. At test time, the robot also receives a natural language instruction 4, which corresponds to set of goal states. An oracle goal satisfaction function defines whether the language goal is reached, i.e., 9t. X (T, F). Given an initial state 20 X' and the instruction &,, the robot should generate sequence of low-level actions (u1, U2 .... Uz) E 4
In the language-annotated learning setting, the robot has a dataset of language-annotaan demonstra- tions D. Each demonstration is a sequence of robot actions {ui.... u paired with observations Each trajectory is segmented into M sub-trajectories, and natural language descriptions (f...., ex] are associated with the segments (e.g.. "place the kettle on the stove"). this paper, we assume that there is finite number of possible f-s-each corresponding
skill learn.
Directly learning single goal-conditioned policy that can generalize to novel states and goals is challenging. Therefore, we recover an abstract state and action representation of the environment and combine online planning in abstract states and offline policy learning for low-level control to solve the task. In BLADE, behaviors are represented as temporally extended actions with preconditions and effects characterized by state predicates. Formally, want to recover set of predicates P that define an abstract state space S. We focus on scenario where all predicates grounded on high-dimensionnl sensory inputs. Using P. a state can be described as set of grounded atoms such as (kettle A),,
binary. However, they
(A) (n, B)} for two-object scene. BLADE will learn function & O + S that maps observations to abstract states. In its current implementation, BLADE requires humans to additionally provide a list of predicate names natural language, which we have found to be helpful for L.LMs to generate action definitions. We provide additional ablations in the Appendix A.2. Based on S, we learn a library of behaviors (a.k.a., abstract actions). Each behavior E A is a tuple of (name, args, pre, efʃ, T) name is the name of the action. args is list of variables related to the action, often denoted by ?x, ?y. pre and eff are the precondition and effect formula defined in terms of the variables args and the predicates P. A low-level policy æ O - l is also associated with a. The semantics of the preconditions and effects is: for any state x such that pre (x:)) is satisfied, executing π at x. will lead to a state x' such that eff (Tx))) [64].
4 Behavior from Language and Demonstration
BLADE is a method for learning abstract state and action representations from language-annotatel demonstrations. works in three steps, as illustrated in Fig. 2. First, we generate a symbolic behavior definition conditioned on the language annotations and contact sequences in the demonstration using a large language model (LLM). Next, we learn the classifiers associated with all state predicates and the control policies, all from the demonstration without additional annotations. At test time, use bi-level planning and exccution strategy to generate robot actions.
3

---

(8) Demonsttrtions
Place Sink
Move Faucet Head
Turn On Faucet
Wait be Filled
(b) Temporal Segmentation with Contact Primitives
transit
close-gripper,
push
open-gripper
transit
close-gne geip-rripprr
push
open-grioper
Predicate Annotation
(d) Behavior Description Generation with LLMs
(saction meve_
=eve_fascet wwwcet wwwcet_head_ceeee Whedd_evee ovar__iik
(ix-plecwd-in
Lsaction turm, on. fascet
iprecondition (and 15s-tursef-off Tfaucet-Inob)
[Hn-nllgned
Haucet-hhaat
zel flett Lend (in-aligned li T(euee--bordd
uu-tumeddaway Trauces-hoad move
1ss-platef-in 7kettle 7sin81)
seffect Cand
[lauset-krab]
iss-turned--neg 7raucet-head)1)
Inot (ds-turned-off 7raucet-knebli1
tbody (then telose-gripper! (push Maucett-heatt) (opet-gripperl))
booyy
(then (close-gripper)
(peah (enddetkknde) (apen-griaper)))
Figure 3: Behavior Descriptions Learning. (a) A demonstration is provided along with corresponding language annotations. (b) The demonstration is segmented into a sequence contact primitives. (c) large language model interprets the annotation and contact sequence, generating a symbolic behavior definition. (d) The system automatically generates data to learn classificrs for state predicates.
4.1 Behavior Description Learning
Given a finite set of behaviors with language descriptions {L} and corresponding demonstration segments, we generate an abstract description for each by querying large language models. To facilitate LLM generation, we provide additional information on the list of objects with which the robot has contact. The generated operators are further refined with abstract verification.
Temporal segmentation. We first segment each demonstration (Fig. 3a) into sequence of contact- based primitives (Fig. 3b). In this paper consider seven primitives describing the interactions between the robot and other objects: open/close grippers without holding objects, move-to(z) which moves the gripper to an object, grasp(z, y) and place(x, y) which grasp and place object z from/onto another object move( which moves the currently holding object x and pash(z). We leverage proprioception, i.e., gripper open state, and object segmentation to automatically segment the con- tinuous trajectories into these basis segments. For example, pushing the faucet head away involves the sequence of (close-gripper, gush,, open- open-gripper). This segmentation will be used for LLMs to generate operator definitions and for constructing training data for control policies.
Behavior description generation with LLMs. Our behavior description language is based on PDDL [65]. We extend the PDDL definition to include body section which is a sequence of contact primitives. It will be generated by the LLM based on the demonstration data.
Our input to the LLM mainly contains: 1) general description of the environment, 2) the natural language descriptions associated with the behavior itself and other behaviors that have appeared preceding or following l in the dataset, 3) all possible sequence of contact primitive sequences associated with é across the dataset, and 4) additional instructions on the PDDL syntax, including a single PDDL definition example. We find the additional context useful. As shown in Fig. 3d, in addition to preconditions and effects of the operators, we also ask L.MSs to predict body of contact primitive sequence associated with the behavior, which we call body. We assume that each behavior has single corresponding contact primitive sequence, and use this step account for noises in the segmentation annotations. After LLM predicts the definition for all behavior, will re-segment the demonstrations associated with each behavior based on the LLM-predicted body section.
Behavior description refinement with abstract verification. In addition to checking for syntax errors, we also verify the generated behavior descriptions with abstract verification on the demon- stration trajectories. Given a segmented sequence of the trajectory where each segment associated with behavior, verify whether the preconditions of each behavior can be satisfied by the accumu- lated effects of the previous segments. This verification does not require learning the grounding of

---

predicates and can be done at the behavior level for incorrect preconditions and effects, and at the contact primitive level for missing or incorrect contact primitives (e.g., grasp cannot be immediately followed by other grasp). We resample behavior definitions that do not pass the verification.
4.2 Classifier and Policy Learning
Given the dataset of state-action segments associated with each behavior, we train the classifiers for different state predicates and the low-level controller for each behavior.
Automatic predicate annotation. We leverage all behavior descriptions automatically label an observation sequence {o1, 0H } based on its associated segmentation. In particular, at oo, label all state predicates as "unknown." Next, we unroll the sequence of behavior executed in ō. As illustrated in Fig. 3c, before applying behavior a at step oz, we label all predicates in pre, true and predicates in eff. false. When a finishes at step or, we label all predicates in eff.. In addition, we will propagate the labels for state predicates to later time steps until they are explicitly altered by another behavior a. In contrast to earlier methods, such as Migimatsu and Bohg [66] and Mao al. [67], which directly use the first and last state of state-action segments train predicate classifiers, our method greatly increases the diversity of training data. After this step, for each predicate p P, we obtain a dataset of paired observations and the predicate value of p at the corresponding time step.
Classifier learning, Based on the state predicate dataset generated from behavior definitions, we train a set of state classifiers fo(p) : O + (T, F), which are implemented as standard neural networks for classification. We include implementation details in Appendix A.6. In real-world environments with strong data-efficiency requirements, we additionally use an open vocabulary object detector [68] to detect relevant objects for the state predicate and crop the observation images. For example, only pixels associated with the object faucet will be the input to the turne-on (faucet) classifier.
Policy learning. For each behavior, we also train control policies my (a): : - U4, implemented as a diffusion policy [1]. simulation, we use a combination of frame-mounted and wrist-mounted RGB-D cameras the inputs the diffusion policy, while in the real world, the policy takes raw camera images imput. The high-level planner orchestrates which of these low-level policies to deploy based on the scene and states. Once trained on these diverse demonstrations of different skills, the resulting low-level policies can adapt to local changes, such as variations in object poses.
4.3 Bi-Level Planning and Execution
At test time, given novel state and novel goal, BLADE first uses L.MMs to translate the goal into first-order logic formula based on the state predicates. Next, it leverages the learned state abstractions to perform planning in a symbolic space to produce a sequence of behaviors. Then, we execute the low-level policy associated with the first behavior, and we re-run the planner after the low-level policy finishes-tie this enables us to handle various types of uncertainties and perturbations, including execution failure, partial observability, and human perturbation. In implementation, we use the fast-forward heuristic to generate plans [69]; however, method is planner-agnostic and other symbolic planners (e.g., Fast-Downward [70]) are compatible.
Visibility and geometric constraints are also modeled as preconditions, in addition to other object- state and relational conditions. For example, the behavior "opening the cabinet door" will have preconditions on the initial state of the door, visibility constraint that the door visible, and geometric constraint that nothing is blocking the door. When those preconditions the planner will automatically generate plans, such actions that move obstacles away, achieve them. Partial observability was handled by using the most-likely state assumption during planning and performing replanning. We include details in Appendix A.8.
not satisfied,
5
Experiments
5.1
Simulation Experimental Setup
We use the CALVIN benchmark [71] for simulation-based evaluations, which include teleoperated human-play data. We use the split D of the dataset, which consists of approximately 6 hours of interactions. Annotations of the play data are generated by script that detects goal conditions
5

---

Abstract Goals
Partial Observability
Geometric Constraints
Initial Condition
Goal State
Initial Condition
Goal State
Initial Condition
Geal State
Drawer closed blocks outside drewer blocks inside
Drawer
Blue black visible Blue block table
Purple cube
Slider the left
blocking
Language Coal: *Place All Blocks Inside Drawer"
Language Goal: "Find Block Slider*
Language Gool: "Move Sliding Door Left"
block(((() 7 in(x, drawer)
is-block(x), (s-blue(x), is.table(y), on(x,))
is-sliding -xliding-door(()... door(s)) left(s)
Figure 4: Generalization Tasks in CALVIN, Examples from the three generalization tasks in the CALVIN
simulation environment. Successfully completing these tasks require planning for and executing 3-7 actions.
Table 1: Generalization results in CALVIN. Mean success rates with STD from three seeds are reported.
BLADE outperforms latent planning, LLM, and VLM baselines in completing novel long-horizon tasks.
State
Latent
Generalization Task
Method
Classifier
Feasibility
Abstract Goal
Geometric Constraint
Partial Observability
HULC [72]
N/A
N/A
2.78 3.47
11.67 11.35
0.00 0.00
SayCan [6]
N/A
Short
23.89 1.92
1.67 = 2.89
1.67 2.89
VILA [57]
N/A
N/A
18.38 2.48
0.00 = 0.00
4.17 5.20
T2M-Shooting [51]
Learned
Long
57.78 + 12.29
0..0 + 0.00
13.33 + 1.44
Ours
Learned
N/A
68.33 + 10.14
26.67 7.64
75.83 + 3.82
T2M-Shooting [51]
GT
Long
61.67 = 5.00
0.00 = 0.00
0.83 1.44
Ours
GT
N/A
76.11 6.74
56.67 = 16.07
70.00 +: 5.00
on simulator states, and there are in total 34 types of behaviors. We use RGB-D images from the mounted camera for classifier learning and partial 3D point clouds recovered from the images for policy learning. The original benchmark focuses only on evaluating individual skills and instruction following. To evaluate the ability to compositionally combine previously learned policies to solve novel tasks, we design six new generalization tasks, with examples shown in Fig. 4. Each task has a language instruction, a sampler that generates random initial states, and goal satisfaction function for evaluation. For each task, we sample 20 initial states and evaluate all methods with three different random seeds. See Appendix B.1 for more details on the benchmark setup.
Baselines. We compare BLADE with two groups of baselines; hierarchical policies with planning in latent spaces and LLM/VLM SOMM/LLM-bseed based methods for robotic planning. For the former, we use HULC [72], a representative method in CALVIN, which learns a hicrarchical policy from language-annotee play data using hindsight labeling. For the latter, we use SayCan [6], Robot- VILA [57], and Text2Motion [51]. Note that Text2Motion assumes access to ground-truth symbolic states. Hence we compare TextMMotion with BLADE in two settings: one with the ground-truth states and the other with the state classifiers learned by BLADE. See Appendix B.2 for more details on these methods.
5.2 Results in Simulation
Table 1 presents the performance of different models in all three types of generalization tasks.
Structured behavior representations improve long-horizon planning, We first compare to the hierarchical policy HULC in Table. 1. BLADE with learned classifiers achieves a more than 65% improvement in the success rate for reaching abstract goals while using the same language-annotated play data. We attribute this the particular implementation of hindsight labeling in HULC being not sufficient to generate plans that chain multiple high-level actions: for example, the task of placing all blocks in the closed drawer requires chaining together minimum of 7 behaviors.
Structured transition models learned by BLADE facilitate long-horizon planning. Both SayCan and BEM--hooting uses learned action feasibility models for planning. Shown in Table. 1, learning accurate feasibility models directly from raw demonstration data remains a significant challenge. In our experiment, we find that first, when the LLM does not take into account state information (SayCan), using the short-horizon feasibility model is not sufficient produce sound plans. Second, since our model learns structured transition model, factorized into different state predicates, BLADE

---

is capable of producing more accurate longer- horizon plans than T2M-Shooting which learns long- horizon feasibility from data.
Structured scene representations facilitate making feasible plans. Compared to the Robot- VILA method, which directly predicts action sequences based the image state, BLADE first uses learned state classifiers to construct an abstract state representation. This contributes to 49% improvement on the Abstract Goal tasks in Table 1. We observe that the pre-trained VLM used in Robot-VILA often predicts actions that are not feasible in the current state, For example, Robot- consistently performs better in completing "placing all blocks in a closed drawer" than "placing all blocks in an open drawer" since it always predicts opening the drawer as the first step.
Explicit modeling of geometric constraints and object visibility improves performance in these scenarios. BLADE reason about these challenging situations without explicitly being trained in those settings. Table. 1 shows that our approach consistently outperforms baselines in these two settings. These generalization capabilities are built on the explicit modeling of geometric constraints and object visibility in behavior preconditions.
BLADE can automatically propose operators for the specific environment given demonstrations. Our experiment shows that the LLM can automatically propose high-quality behavior descriptions that resemble the dependency structures among operators. For example, the LLM discovers from the given contact primitive sequences and language-pairee demonstration that blocks can only be placed after the block is lifted and that a drawer needs to be opened before placing objects inside, etc. Some of these dependencies are unique to the CALVIN environment, therefore requiring the LLM to generate specifically for this domain. We provide more visualizations in the Appendix A.
BLADE's automatic predicate annotation enables better classifier learning. From Table 1, we observe that having accurate state classifier models critical for algo- rithms' performance (GT vs. Learned). Hence, we perform additional ablation stud- ies on classifier learning. Prior work such as Migimatsu and Bohg [66] also presented method for learning the preconditions and effects of actions from segmented trajectories and symbolic action descriptions. The key difference between BLADE and theirs is that they only use the first and last frame of each segment to supervise the learning of state classifiers. We compare the two classifier learning algorithms, given the same LLM-generated behavior definitions, by evaluating the classifier accuracy on held-out states. BLADE shows 20.7% improvement in FI (16.3% improvement for classifying object states and 38.6% improvement for classifying spatial relations) compared the baseline model. This also translates into significant improvements in the planning success rate, as shown in Table 2.
Table 2: Ablation on state classifier learning in CALVIN.
Method
Abstract
Geometric
Partial Obs.
[66]
33.89 + 5.85
9.17
5.20
3.33 = 2.89
BLADE
68.33 10.14
26.67
7.64
75.83 3.82
5.3 Real World Experiments
Environments. We use Franka Emika robot arm with a parallel jaw gripper. The setup includes five RealSense RGB-D cameras, with one being wrist-mounted the robot and the remaining positioned around the workspace. Fig. 5 shows the two domains: Make Tea and Boil Water. For each domain, we collect 85 language-annotate After segmenting the demonstrations using proprioception sensor data, an L.M used to generate behavior descriptions. These descriptions are subsequently used for policy and classifier learning.
demonstrations using teleoperation with 3D mouse.
Setup. We compare BLADE against the VLM-based baseline Robot- We omit SayCan and T2M-Shooting since they require additional training data. We first test the original action sequences seen in the demonstrations for each domain. We then test on tasks that require novel compositions of behaviors for four types of generalizations, i.e., unseen initial condition, state perturbation, geometric constraints, and partial observability. For each generalization type, we run six experiments and report the number of experiments that have been successfully completed. See Appendix D for details.
Results. In Fig. 5, we show that our model is able to successfully complete at least 4/6 tasks for all generalization types in the two different domains. In comparison, Robot-VILA struggles generate
7

---

Boll Domain
Domain
Results
Make Resulit
Mive Hiod
Open Caberal
RONTRIA
Stoow OniTaiee
On Bowe
35
35
16
Open Dravne
Place Teabogit Nottle
Seqlassweira Pwmuu Caner: DrD Seq LIVANT1
Sen Canor Parti
Figure Domains and Results in Real World. Make Tea features a toy kitchen designed to simulate hoiling water on a stove. The robot must assess the available space on the stove for the kettle. It also needs to manage the dependencies between actions, such as the faucet must be turned away before the kettle can be placed into the sink to avoid collisions. Boil Water involves tabletop task aimed at preparing a drawer, and stove. The robot must locate kettle, potentially hidden within the cabinet, and tcabag in the drawer. Additionally, must consider geometric constraints by removing obstacles that block the cabinet doors. In both environments, our model significantly outperforms the VLM- M-ddasdd planner Robot- VILA.
incorporating cabinet,
(a)
Initial Condition
Right
Place
BLADE
Visible
Cup Not Visibile
Cup Visible
Doal Achiaved
Robot-Vita
Plan: "open-drawer", "place-teabag"
(Policy cannot achieve the goal, teabag is an the stave)
Crpp Not Visibie
Initial Consition
Grab
(b)
Goometric
Unblock Doors
Door
Place
blocking doors
BLADE
& Right
Slocked
Goal Achisved
Robot-Vila
Plan: "grab-kettle", "grab-kettle", "grab-kettle".
(Policy unable to achieve the goal, resulting in loop)
Drawer Open
Inttial Condition
Grab Kettle
Figure Real World Planning and Execution. We show the execution traces from BLADE and Robot-VILA for two generalization tasks: (a) partial observability and (b) geometric constraints.
correct plans complete the tasks. In Fig. 6, visualize the generated plans and execution traces of both models. In example (a), we show that BLADE find the kettle initially hidden in the cabinet and then complete the rest of the task. In comparison, Robot-VILA directly predicts placing the teabag in the kettle when the kettle is not visible, resulting in a failure.
6 Conclusion and Discussion
BLADE a novel framework for long-horizen manipulation by integrating model-based planning and imitation learning. BLADE uses L.M generate behavior descriptions with preconditions and effects from language-ann annottted demonstrations and automatically generates state abstraction labels based on behavior descriptions for learning state classifiers. At performance time, BLADE generalizes to novel states and goals by composing learned behaviors with planner. Compared to latent-space and LLM/VLM-based planners, BLADE successfully completes significantly more long- long-horizon tasks with various types of generalizations.
Limitations. One limitation of BLADE is that the automatic segmentation of demonstrations is based on gripper states; more advanced contact detection techniques might be required for certain tasks such as caging grasps. We also assume the knowledge of given set of predicate names in natural language and focus on learning dependencies between actions using the given predicates. Automatically inventing task-specific predicates from demonstrations and language annotations, possibly with integration of vision- language models (VLMs) an important future direction. In our experiments, we also found that noisy state classification led to some planning failures. Therefore, developing robust to noises in state estimation necessary. Finally, achieving novel
planners that compositions of behaviors also requires policies with strong generalization to novel environmental states, which remain challenge for skills leamned from a limited amount of demonstration data.

---

Acknowledgments
This work is in part supported by Analog Devices, MIT Quest for Intelligence, MIT-IBM Watson AI Lab, ONR Science of AI, NSF grant 2214177, ONR N00014-23---225 1--3555,,, AFOSR YIP FA9550-23- 1-0127, AFOSR grant FA9550-22-1-088
ONR MURI - 22-1-2400 ARO grant W91 INF- 23-1-0034. We extend our gratitude to Jonathan Yedidia, Nicholas Moran, Zhutian Yang, Manling Li, Joy Hsu, Stephen Tian, Chen Wang, Wenlong Wang, Yunfan Jiang, Chengshu Li, Josiah Wong, Mengdi Xu, Sanjana Srivastava, Yunong Liu, Tianyuan Dai, Wensi Ai, Yihe Tang, the members of the Stanford Vision and Learning Lab, and the anonymous reviewers for insightful discussions.
References
[1] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023. 1, 5, 18, 19
[2] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling. PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning. In ICAPS, 2020. 1,
[3] D. Xu, A. Mandlekar, R. Martin-Martin, Zhu, S. Savarese, and L. Fei-Fei. Deep affordance foresight: Planning through what can be done in the future. In ICRA, 2021.
[4] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. RoboCraft: Learning to see, simulate, and shape clasto-plastic objects in 3d with graph networks. 1/RR, 43(4):533-549 2024.
[5] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. CoRI., 2020.
[6] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as I say: Grounding language in robotic affordances. In CoRL, 2023. 1, 6
[7] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined Task and Motion Planning through an Extensible Planner- Inneer--ndeeendent Interface Layer. In ICRA, 2014.
[8] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L.. E. Kavraki. Incremental task and motion planning: A constraint-basee approach. In RSS, 2016.
[9] M. Toussaint. Logic-Geometri Programming: An optimization-base approach combined task and motion planning. In IJCAI, 2015.
[10] A. Curtis, X. Fang, L. P. Kaelbling, Lozano-Pérez, and C. Garrett. Long-horizon manipu- lation of unknown objects via task and motion planning with estimated affordances. In ICRA, 2022.
[11] D. Driess, O. Oguz, J.-S. Ha, and M. Toussaint. Deep visual heuristics: Learning feasibility of mixed-integer programs for manipulation planning. In ICRA, 2020.
[12] Z. Yang, R. Garrett, T. Lozano-Pérer L. Kaelbling, and D. Fox. Sequence-based plan feasibility prediction for efficient task and motion planning. In RSS, 2023. 2
[13] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In ICRA, 2017.
[14] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. In ICLR, 2020.
[15] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. In CoRL, 2023.
[16] A. Simeonov, Y. Du, B. Kim, F. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. In CoRL, 2021.

---

[17] X. Lin, C. Qi, Y. Zhang, Z. Huang. K. Fragkiadaki, Y. Li, C. Gan, and D. Held. Planning with spatial and temporal abstraction from point clouds for deformable object manipulation. In CoRL, 2022.
[18) Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning, 1.Xiv:2310.10625. 2023. 2
[19] J. Luo. C. Xu, X. Geng, G. Feng, K. Fang, L. Tan, S. Schaal, and S. Levine. Multi-stage cable routing through hierarchical imitation learning. IEEE Transactions on Robotics, 2024. 2
[20] L. X. Shi, Z. Hu, Z. Zhao, A. Sharma, K. Pertsch, J. Luo, Levine, and C. Finn. Yell at your robot: Improving on-the-fly -fly from language corrections. arXiv:2403.129100 2024.
[21] S. Pirk, Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. In CoRL, 2020.
[22] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Zhu, and A. Anandkumar, Mimicplay: Long-horizon imitation learning by watching human play. CoRL, 2023.
[23] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. In RSS, 2021. 2
[24] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In ICRA, 2023. 21
[25] A. Mandlekar, C. Garrett, D. Xu, and D. Fox. Human- Human-in-thee in-the-loop task and motion planning for imitation learning. In CoRI, 2023. 2
[26] Z. Zhang, Y. Li, O. Bastani, A. Gupta, D. Jayaraman, Y. J. Ma, and L. Weihs. Universal Visual Decomposer: Long-horizon horizon manipulation made easy. In ICRA, 2024.
[27] Y. Zhu, P. Stone, and Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, (12):412641133 2022. 2
[28] Y. Zhu, Tremblay, Birchfield, and Y. Zhu. Hierarchical planning for long-horizon manipu- lation with geometric and symbolic scene graphs. In ICRA, 2020.
[29] D.-A. Huang, Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. CVPR, 2019.
[30] D.-.. Huang, D. Xu, Y. Zhu, A. Garg, S. Savarese, F-F. Li, and J. C. Niebles. Continuous relaxation symbolic planner for one-shot imitation learning. In IROS, 2019.
[31] Y. Huang, N. C. Taylor, A. Conkey, W. Liu, and T. Hermans. Latent space planning for multi-object manipulation with environment-aware relational classifiers. IEEE Transactions on Robotics, 2024.7
[32] G. Konidaris, I.. P. Kaelbling, and T. Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61: 215-289, 2018.
[33] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Pérez, L. Kaelbling, and J. B. Tenenbaum. Predicate invention for bilevel planning. In AAAI, 2023.
[34] A. Ahmetoglu, E. Oztop, and E. Ugur. Symbolic manipulation planning with discovered object and relational predicates. arXiv preprini wrXiv:2401.011223 2024.
[35] N. Shah, J. Nagpal, P. Verma, and S. Srivastava. From reals logic and back: Inventing symbolic vocabularies, actions and models for planning from data, arXiv preprint 11Xiv. 2024.
10

---

[36] M. Han, Y. Zhu, S.-C. Zhu, N. Wu, and Y. Zhu. Interpret: Interactive predicate learning from language feedback for generalizable task planning. In RSS, 2024.
[37] W. Huang, Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022. 2
[38] T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano- Pérez, and L. P. Kaelbling. Pddl planning with pretrained large language models. In NeurlPS 2022 foundation models for decision making workshop. 2022.
[39] K. Valmeekam, M. Marquez, Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. In NeurlPS, 2023. 2
[40] S. Kambhampati, K. Valmeekam, L. Guan, K. Stechly, M. Verma, S. Bhambri, L. Saldyt, and
A. Murthy. Lims can't plan, but can help planning in Ilm-modulo frameworks. arXiv:2402.01817, 2024.
[41] Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan. AutoTAMP: Autoregressive task and motion planning with lims translators and checkers. In ICRA, 2024. 2
[42] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. LLM+P: Empowering large language models with optimal planning proficiency. arXiv:2304 1477,
[43] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with large-language models. 2rXii:2302.05/28, 2023.
[44] A. Mavrogiannis, C. Mavrogiannis, and Y. Aloimonos. Cook2ltl: Translating cooking recipes to Itl formulae using large language models. In ICRA, 2024. 2
[45] Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L.. Kaelbling, and M. Katz. Generalized planning in PDDL domains with pretrained large language models. In AAAI, 2024. 2
[46] X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, B. Li, L. Lu, X. Wang, et al. Ghost in the Minecraft: Generally capable agents for open-world enviroments via large language models with text-hased knowledge and memory. 2rXiv:2305. 2023. 2
[47] K. Nottingham, P. Ammanabrolu, A. Suhr, Y. Choi, H. Hajishirzi, S. Singh, and R. Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In ICML, 2023.
[48] S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu. Reasoning with language model is planning with world model. EMNLP, 2023. 2
[49] Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In ICRA, 2023. 2
[50] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In ICRA, 2023.
[51] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 4745):1345 2023. 2,
[52] M. Skreta, Z. Zhou, J. L. Yuan, K. Darvish, A. Aspuru-Guzik, and A. Garg. Replan: Robotic replanning with perception and language models. arXiv:2401.04117,, 2024.
[53] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson,
Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. axXiv:2303.03378 2023. 2
11

---

[54] Z. Wu, Z. Wang, X. Xu, Lu, and H. Yan. Embodied task planning with large language models. arXiv22307.0/845 2023.
[55] Xiang, T. Tao, Y. Gu, T. Shu, Z. Wang, Z. Yang, and Z. Hu. Language models meet world models: Embodied experiences enhance language models. In NeurlPS, 2024. 2
[56] H. Wang, K. Kedia, J. Ren, R. Abdullah, A. Bhardwaj. A. Chao, K. Y. Chen, N. Chin, Dan,
X. Fan, et al. Mosaic: A modular system for assistive and interactive cooking. arXiv preprint
arXiv:2402.18727 2024.
[57] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao. Look before you leap: Unveiling the power of GPT-4v in robotic vision-language planning. arXiv:2311. 17842, 2023. 2,
[58] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi. ChatGPT empowered long robot control in various environments: A case application. IEEE Access, 2023. 2
[59] L. Wong, J. Mao, P. Sharma, S. Siegel, J. Feng, N. Korneev, J. B. Tenenbaum, and J. Andreas. Learning adaptive planning representations with natural language guidance. ICLR, 2024.
[60] L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In NeurlPS, 2023.
[61] Smirnov, F. Joublin, A. Ceravola, and M. Gienger. Generating consistent PDDL domains with large language models. arXiv:2404.07751, 2024. 2
[62] Z. Li, K. Yu, S. Cheng, and D. Xu. League+:: Empowering continual robot learning through guided skill acquisition with large language models. ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. 2
[63] M. Dalal, T. Chiruvolu, D. Chaplot, and R. Salakhutdinov. Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks. In ICLR, 2024.
[64] V. Lifschitz. On the semantics of STRIPS. In M. Georgeff, Lansky, and Amy, editors, Reasoning about Actions and Plans, pages 1-9. Morgan Kaufmann, San Mateo, CA, 1987.
[65] C. Aeronautiques, A. Howe, C. Knoblock, I. D. McDermott, A. Ram, M. Veloso, D. Weld, D. W. SRI, A. Barrett, and D. Christianson. PDDL: The Planning Domain Definition Language, 1998.
4
[66] T. Migimatsu and J. Bohg. Grounding predicates through actions. In ICRA, 2022. 5,7
[67] Mao, T. Lozano-Pérez, Tenenbaum, and L. Kaelbling. PDSketch: Integrated domain programming, learning, and planning. In NeurIPS, 2022.
[68] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding Dino: Marrying dino with grounded pre-training for open-set object detection. arXiv:2303.05423 2023. 5, 17
[69] J. Hoffmann and B. Nebel. The FF planning system: Fast plan generation through heuristic search. JAIR, 14:253-302, 2001.
[70] M. Helmert. The fast downward planning system. JAIR, 26:191-246 2006. 5
[71] O. Mees, L. Hermann, E. Rosete- Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. RA-L, 7:7327-7334, 2021.
[72] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. RA-L, 7:11205-11212 2022. 6, 21
12

---

[73] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin. T. Luo. Y. Li, S. Liu, et al. Recognize Anything: strong image tagging model. In CVPR, 2024. 14
[74] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-Actor; A multi-task transformer for robotic manipulation. In CoRL, 2023. 16
[75] T.-W. Ke, Gkanatsios, and K. Fragkiadaki. 3D Diffuser Actor: Policy diffusion with 3D scene representations. arXiv:2402.10885, 2024. 16
[76] Z. Zhang, Y. Li, O. Bastani, A. Gupta, D. Jayaraman, J. Ma, and L. Weihs, Universal Visual Decomposer: Long-horizon horizon manipulation made easy. ICRA, 2024. 16
[77] W. Wan, Y. Zhu, R. Shah, and Y. Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In ICRA, 2024. 16
[78] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J Mu, R. R. Martin, and S.-M. Hu. PCT: Point cloud transformer. Computational Visual Media, 7:187-199, 2021. 17
[79] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In ICRA, 2011. 19, 20
[80] C. R. Garretr, C. Paxton, T. Lozano-Pérez, L. P. Kaelbling, and D. Fox. Online replanning in belief space for partially observable task and motion problems. In ICRA, 2020. 20
13

---

Supplementary Material for Learning Compositional Behaviors from Demonstration and Language
This supplementary material provides additional details on the BLADE framework, the experiments, and qualitative examples. Section A provides detailed description of the method, including the behavior description generation, predicate generation, abstract verification, automatic predicate annotation, classifier implementation, and policy implementation. Section provides details on the simulation experiments, including the task design and bascline implementations. Section C provides qualitative examples of our method and baselines. Section D provides details of our setup of the real-robot experiment. Finally, Section E includes a full list of the prompts for the baselines used in the simulation experiments.
A
BLADE Details
A.1 Behavior Description Generation with LLMs
In Listing I, we show the behavior descriptions automatically generated by the LLM for the CALVIN domain. We also show the detailed prompt to the LLM for generating the behavior description. We break down the system prompt into four parts: definitions of primitive actions (Listings 2). definitions of predicates and environment context (Listings 3), an in-context example (Listings 4), and additional instructions (Listings 5). In Listings 6, we show one example of the specific user prompt that is used to generate the behavior description for place-in-dee innee-ee-drawer..
In our experiments, we find that the environment description is necessary for the LLMs to understand the context of the task. For the simulation experiment, we provide the environment description as list of objects and brief explanations, shown in Listings However, this description can be automatically generated using image tagging models such Recognize Anything [73] or general- purpose VLMs such GPT-4V.
A.2 Predicate Generation with LLMs
Our algorithms are agnostic to the source of predicates and can flexibly generate action descriptions based the given predicates. In our main experiment, assume that the predicates for each task domain are provided in natural language. Here, we show that given the task definition and the environment context, LLM can automatically generate the relevant predicates for the domain. Then, we generate behavior descriptions based on the automatically generated predicates.
To generate high-que high-quality predicates and behavior descriptions, we take the following steps. First, the LLM provided with the list of objects in the scene and the language-paired demonstration sequence and is required to generate relevant predicates for the domain. In Listing 7, we show an example of the input prompt to the LLM for the CALVIN domain. Second, the LLM is prompted with the generated predicates and is required to ground the predicates using the objects in the scene. This step helps the LLM eliminate errors in the first step, such as missing arguments. Finally, we instruct the LLM to remove semantically equivalent predicates and keep the most general ones. For example, is-on- (oo-etlovekettle) removed by the LLM in our experiment because is-on(kettle, stove) semantically equivalent. After the predicates are automatically generated and filtered, we proceed with the behavior description generation.
In Table AI, Table A2, and Table A3, we compare the generated predicates with the predicates defined by the domain expert for the CALVIN, Boil Water, and Make Tea domains. We observe that the LLM is able to generate 28 out of 30 predicates that match closely with the expert-designed predicates. These predicates provide abstract representations for object states (e.g., is-open, light-on), relations between objects (e.g., in-slider, in), and robot-centric states (e.g., holding). The LLM incorrectly generates the predicate next-to(?x,, ?y) characterize the effects of the push-left and push-right actions, possibly due to ambiguities in the definition of the actions. For the two real-worth domains,
14

---

we perform additional experiments to confirm that the generated predicates and behavior descriptions can be used to generate correct task plans on all nine real-world tasks presented in Section 5.3.
Table Al: Comparison of Manually Defined and Automatically Generated Predicates for CALVIN.
Manually Defined
Automatically Generated
rotated-left((3))
rotated-lef(((a))
roggted-righ
rotated- rig
lifted(?x)
holding(?:))
is-open(?x))
is-open(?a)
is-close({x)
ts-closed((a))
is so-tured--on on(?x))
light-on(?an
is oameed-off?">" slider-teft(Ras is-slider-righ((2n)) is-on(Ta, ?ÿ)
light-off( (2))
slider-left(?))
slider-righ((?a)
on-table(?x))
is is-in(?x, ?y)
in-slider(1x)) in-drawer((a:)
stacked(?sr Ty)
on(?x, ?V)
unstacked((1z ?n)
clear(?x)
pushed-lef((aa pushed-righ((2
next-to(?2,, ?y)
Table A2: Comparison of Manually Defined and Automatically Generated Predicates for Boil Water.
Manually Defined
Automatically Generated
Automatically Generated (Grounded)
is s--placedoon(21 Ix. "xx,y) ?y)
on-table(?x))
on-table(kettle), on-table(pot) in-sink(kettle), in-sink(pot)
is placed-inn in(?x, ?y)
in-sink(?z:)
is blocked((2))
on-stove(?=x)
on-stove(pot)
is -urntuuddaaa aaadddaww)?)))))
Soucet-over-sink(22 ?y)
Soucet-over-sink(faae sink)
is-at s-allgned( 2x, Ty)
faucet-over-sink(2a. ?y)
faucet-over-sin((( wiaaerrrnnfffucettt.cooeee---eee sink)
is tetttured--on pamned-on(?an on(?x)
faucet-on(?2))
(oucet-on(fauset)
is -u-tuuddoofff ?x)
faucer-off(Ix:)
faucer-offffacctt
is (x)le
filled(?x) (xx)
filled(kettle) filled(pot) holding(pot
holding(tx)
holding(?a)
Table A3: Comparison of Manually Defined and Automatically Generated Predicates Make Tea.
Manually Defined
Automatically Generated
Automatically Generated (Grounded)
is (s-placed-one on(?x, ?y)
is-on(?æ, ?y) is-open(?>)
is is-on(kettle, stove) is-open(left-door) is s--in-drawer(teabag) (--nn-ketlle(tcabag)
pacccaine-ddoor-peer is pplaced-inside(2-. ?n)
is-open(right-ddoo)
is-in-drawer(2a))
is-in-kertle(?aa)
(s-in-cubinee-le((((1)))
is-in- in--abinet-leff(keeeee is-in-cabiner sin--ubine--righikkkutt
is-in-cabinet-rigg(((a) is-open(?w)
is (t-drawer-open((2)
en-open(drawee is blocking left-door)
is B--left-cabinet-de bobiinet-door-blooed bloooor-booce((.. (x:)) wright-ccbbiet-done
is be-blookkig(?z.. ?y)
b-bbllooiin(?:; ?y)
blo.king (pot, right-door)
is closed((xx)
is closed( weeddddccftooooooooo door), is (s-closedtright-dooo))
is-closed drawer) is-moved-avayppot)
is-moved-away((aa)
A.3 Temporal Segmentation
Before the generation of behavior description, we segment each demonstration into a sequence of contact-based primitives. We consider seven primitives describing the interactions between the robot and other objects: openiclose grippers without holding objects, move-to(x) which moves the gripper to object, grasp(x, y) and place(æ, y) which grasp and place object from/onto another object y. move(x) which moves the currently holding object and push(x).
15

---

We use a set of heuristics to automatically segment the continuous trajectories using proprioception, i.e,, gripper open state, and object segmentation. Specifically, open and close are directly detected by checking whether the gripper width is at the maximum or minimum value. grasp(z, y) and place(æ, y) correspond to the other closing and opening gripper actions. move(x), push(r) and move-to(x) are matched to temporal segments between pairs of gripper actions. Their type can be inferred based on the preceding and following gripper actions. We make simplifying assumption that the robot moves freely in space only when the gripper is fully open and pushes objects only when the gripper is fully closed. These are given as instructions to the human demonstrators. In the simulator, the arguments of the primitives are obtained from the contact state, In the real world, they are inferred from the language annotations of the actions "place the kettle on the stove" corresponds to place(kettle, stove)) procedurally or by the LLMs.. The arguments can also be left unspecified; these arguments mainly provide additional contextual information about the target objects.
In Section 4. we discuss that we use LLMs to predict body of contact primitive sequence associated with each behavior description. This additional step helps account for noises in the segmentation anno- tations, which are prevalent in CALVIN's language-annotaan guage annotation tlittbllock-tbbee" correspond to the contact sequence {move-to, grasp, move, place). Based on the generated body, the behavior can correctly mapped to {grasp, move) and the demon- stration trajectories can then be re-segmented. This additional step crucial for learning accurate groundings of the states and actions.
demonstrations. For example, the lan-
Our approach to temporal segmentation are similar to keyframe-basee methods like PerAct [74] and 3D Diffuser Actor [75], which rely on end-effector states (e.g., grasp and release) and velocities to segment demonstrations. However, our method differs by distinguishing between prehensile and non prehenseee manipulation through detecting whether an object is grasped. This enables the development of behaviors such move-faucet-awayy where the robot pushes the faucet head without grasping it. In our preliminary studies, also experiment with other vision-based methods including UVD [76] and Lotus [77]. A main issue for incorporating these methods is that they provide less consistent segmentations for different occurrences of the same behavior. As we discussed in Section 6, more advanced contact detection techniques will be an important future direction for using contact primitives as meaningful interface between actions and language.
A.4 Abstract Verification
After the generation of the behavior descriptions, we verify the generated behavior descriptions by performing abstract verification on the demonstration trajectories. Given a segmented sequence of the trajectory where each segment is associated with a behavior, we verify whether the preconditions of each behavior can be satisfied by the accumulated effects of the previous behaviors. Pseudocode for this algorithm is shown in Algorithm 1.
A.5 Automatic Predicate Annotation
We leverage all behavior descriptions to automatically label an observation sequence {ol...., оh based on its associated segmentation. particular, at og, we label all state predicates "unknown." Next, we unroll the sequence of executed behaviors. As illustrated in Fig. 3c, before applying behavior step ot, label all predicates in pre, true and predicates in eff, false. When finishes at step ov, we label all predicates in eff.. In addition, we will propagate the labels for state predicates to later time steps until they are explicitly altered by another behavior Pseudocode for this algorithm is shown in Algorithm 2.
As result, we obtain both positive and negative examples to train the binary predicate classifiers. In particular, the negative examples of predicate come from two sources. First, the preconditions and effects of a behavior include negated predicates. Second, during automatic predicate annotation, we label all predicates in the effects false.
16

---

Based on the state predicate dataset generated from behavior definitions, we train a of state classifiers fo(p) O (T, F), which are implemented standard neural networks for classification. Algorithm Abstract Verification
Input: Dataset D. Behavior descriptions A
1: error croutter < counter for sequencing errors related to each behavior
2: counter <- counter for storing the occurrences of each behavior
3: for 1 K do
4: obtain a behavior sequence D. + .... d'y)
5: initialize dictionary for predicate state pred t 0
6: for & - to N do
7:
for each exp pr do
8:
(p. v) + EXTRACCTREEDCATEANBBBBO..CC
9:
if p not in pred then
10:
pred(p) + v
11:
else
12:
if pred + then increment error_counin =roo_ccounerrla]
13:
14:
for each exp in effa; do (p,v) t EXTRACTTREDDINTTEANDOOOL.CC predlp] +
15:
17:
increment counter[a]]
18:
for each in error counter do if error counter[a] larrlla/couune counter(a] > threshold then regenerate the behavior description for a
20:
A.6 Classifier Implementation
In the simulation experiment, the classifier model is based on pre-trained CLIP model (ViT-B/32) B2)... We use the image pre- precpeosesing pipeline from the CLIP model to process the input images. We use images from the static camera in the simulation. We perform one additional step of image processing mask out the robot arm, which we find in our preliminary experiment help avoid overfitting. We do patch tokens from the output of the vision transformer. We downsize the concatenated patch tokens with multilayer perceptron (MLP) and then concatenate with word embeddings of the predicate arguments (e.g., red-block, table). The final embedding is then passed through a predicate-specinee MI.P to output the logit for binary classification. The CLIP model is frozen, while all other learnable parameters are trained.
the global image embedding from the CLIP model, instead extract the
In the real-world experiment, find that, with more limited data than simulation, the pre-trained CLIP model often overfits to spurious relations in the training images (e.g., the state of the faucet is entangled with the location of the kettle). We also experiment with ResNet-50 model pre- trained on ImageNet and find similar behavior. To improve generalization, we choose to focus on relevant objects and regions. We achieve this by using segmented object point clouds. We use open vocabulary object detector Grounding-DDie Dino [68] to detect objects given object names. The predicted 2D bounding boxes are projected into 3D and used extract regions of the point cloud surrounding each object. The point-cloud-basse classifier is based on the shape classification model from the Point Cloud Transformer (PCT) [78]. We concatenate the segmented object point clouds and include one additional channel to indicate the identity of each point. The PCT is used to encode the combined point cloud and output the final logit. The PCT model is trained from scratch.
We also experiment with replacing trained classifiers general- purposs VLM gpt-4o. The VLM fails to robustly determine the boolean values of the predicates in different states. Our observation is that the VLM can more reliably detect the states of articulated objects (e.g., the states of drawers and cabinet doors) than more complex spatial relations (c.g., whether the faucet head is aligned with
17

---

Algorithm 2 Predicate Annotation
Input: Behavior sequence {a)...., ax Observation sequence (o,,... Or Descriptions A
1: propagated +- an empty list of propagated predicates
2: prev_eff's <- list for storing effects from previous step
3: pinds <- an empty list of predicates associated with time steps
4: pred_obs +- empty list for storing predicates paired with observations
5: for to N do
6:
// Precondition
7:
limed-preds + timed- preds U GEETTIEEDEEDDCCAEEESSSSS timed preds +- timed preds U GEETICEESREDDICATESS -efla.: // Propagated
8:
9:
10:
for each p in propagated do
11:
if not ALTERED(P, az) then UPDATETIME(P. t)
12:
13:
else
14:
rempagated.remoove] pendd.preds.aade
15:
// Previous effects
17:
for each p in eef-effs do ALTERED(p, a.) then
19:
propagated.addie add(p)
else
21:
timed preds.add(p)
22:
II Store effects for next step
23:
prev effs + PERTTMMEDPREDDCCCTSSS((......
timed-preds urp.dpdddddeeeeeeeeeeeeeeee (papattttppppppaee 25: timed upddate (prev (prev.effts efffs)
for each p in timed preds do
27: pred obs. upsste
MMFCCT/TINDDPEDCCCCKENTSSSSSSTTT (o,,,, (0
28: return pobd_obs
the kettle). Following our original experimental procedure, we combine the VLM classifier with our planner to the generalization cases for the Boil Water domain. Due to the inaccuracy of the state classification, the overall system achieves 0% success rate.
We imagine future VLMs will become more reliable in detecting the predicate states, and our method will beneficial to the paradigm of using VLMs as classifiers in the following ways: 1) our automatic predicate annotation method can generate examples for VLMs to recognize new concepts through visual in-context prompting or supervised fine-tuning: 2) our method provides data to train task- specific classifiers based on 3D representations, which complementary to general-purposs VLMs in recognizing geometric and spatial concepts; 3) method provides way to learn user-specific predicates (e.g., predicate that determines whether clean dishes are arranged according to user's s preferences) from demonstrations. In our preliminary experiment on the Boil Water domain, we find that using examples generated from our method as in-context visual examples improves the VLM's accuracy by around 5% on 220 withheld testing examples.
A.7 Policy Implementation
For each behavior, we train control policies To (a) : + U. implemented as diffusion policy [1]. We make three changes to the original implementation to facilitate chaining the learned behaviors. First, when training the model to predict the first raw action for each skill, we replace the history observations with observations sampled randomly from temporal window prior when the skill executed, avoid bias in the starting positions of the robot arm. Second, we perform biased sampling of the training sequences to ensure that the policy is trained on diverse set of starting positions. Third, the end of each training sequence, append a sequence of zeros actions the learned
18

---

policy can learned to predicate termination. These strategies are implemented for both the simulation and the real world.
In simulation, we construct the point cloud of the scene using the RGB-D image from the frame- mounted camera. We then obtain segmented object point clouds for the relevant objects of each behavior (e.g., table and block for pick-ble pick-block be-booloooobbbeee -tble) with groundtruth segmentation masks from the PyBullet simulator. The segmented point clouds of the objects are concatenated to form the input point cloud observation. The model uses the PCT to encode a sequence of point clouds as history observations and uses another time-series transformer encoder to reason over the history observations and predict the next actions. The time-series transformer is similar in design to the transformer- diffusion policy [1].
In the real world, we use RGB images from four stationary cameras mounted around the workspace and wrist-mounted camera as input to an image-based diffusion policy model. The input is processed using five separate ResNet-34 encoder heads. The policy directly predicts the gripper pose in the world frame. We found the wrist-mounted camera to be particularly helpful in the real-world setup.
A.8 Planner Implementation
Planning over geometric constraints. Geometric constraints, specifically the collision-fre con- straints for each action, are handled "in the now," right before an action is executed. This is because in order to classify the geometric constraints, we would need to know the exact pose of all objects in the environments. However, we do not explicitly learn models for predicting the exact location of objects after executing certain behaviors.
Our approach to handle this is to process them in the now. It follows the hierarchical planning strategy [79]. In particular, the precondition for actions is an ordered list. In our case, there are two levels: the second level contains the geometric constraint preconditions and the first level contains the rest of the semantic preconditions. During planning, only the first set of preconditions will be added to the subgoal list. After we have finished planning for the first-level preconditions, we consider the second-leve precondition for the first behavior in the resulting plan, by possibly moving other obstacles away.
As an example, let us consider the skill of opening the cabinet door. Its first-level precondition only considers the initial state of the cabinet door (i.e., should be initially closed). It also has a second-level precondition stating that nothing else should be blocking the door. In the beginning, the planner only considers the first-level preconditions. When this behavior is selected to be executed next, the planner checks for the second-level precondition. If this non-blocking precondition is not satisfied in the current state, we will recursively call the planner to achieve it (which will generate actions that move the blocking obstacles away). If this precondition has already been satisfied, we will proceed to execute the policy associated with this opening-cabinet cabinet-daeee door skill.
This strategy will work for scenarios where there is enough space for moving obstacles around and the robot does not need to make dedicated plans for arranging objects. In scenarios where space is tight and dedicated object placement planning is required, we can extend our framework to include the prediction of object poses after each skill execution.
Planning over partial observability. Partial observability is handled assuming the most likely state. In particular, the effect definitions for all behaviors are deterministic. denotes the most likely state that it will result in. For example, in the definition of behaviors for finding objects (e.g., the find-objec-in-leff-caee in-leff-cabbnet), we have a deterministic and "optimistic" effect statement that the object will be visible after executing this action.
At performance time, since we will replan after executing each behavior, if the object is not visible after we have opened the left cabinet, the planner will automatically plan for other actions to achieve this visibility subgoal.
This strategy works for simple partially observable Markov decision processes (POMDPs). A potential extension to it is to model belief state (e.g., representing distribution of possible object
19

---

poses) and execute belief updates on it. Planners can then use more advanced algorithms such as observation-based planning to generate plans. Such strategies have been studied task and motion planning literature [79, 80].
B Simulation Experiment Details
In the effort standardize our evaluation, adopt the standard CALVIN benchmark to evaluate all methods without any modifications to its language annotations, action space, skills, and environments.
B.1
Task Design
To evaluate generalization to long-horizon manipulation tasks, designed six tasks that fall into three categories: Abstract Goal, Geometric Constraint, and Partial Observability. Each task has language instruction, a sampler that generates random initial states, and goal satisfaction function for evaluation. We provide details for each task below.
Task-1
Task Category: Abstract Goal
Language Instruction: turn off all lights.
Logical Goal: (and (is-turnedooff led) (is-turned-off lightbulb))
Initial State: Both the led and the lightbulb are initially turned on.
Goal Satisfaction: The logical states of both the lightbulb and the led are off.
Variation: The initial states the led and the lightbulb are both on and the goal is to turn them off.
Task-2
Task Category: Abstract Goal
Language Instruction: move all blocks to the closed drawer:
Logical Goal: (and (is-in red-block drawer) (is-in blue-block drawer) (is-in pink-block drawer))
Initial State: The blocks visible and not in the drawer. The drawer is closed.
Goal Satisfaction: The blocks are in the drawer.
Task-3
Task Category: Abstract Goal
Language Instruction: move all blocks to the open drawer.
Logical Goal: (and (is-in red-block drawer) (is-in blue-block drawer) (is-in pink-block drawer))
Initial State: The blocks are visible and not the drawer. The drawer is open.
Goal Satisfaction: The blocks are in the drawer.
Task-4
Task Category:
Partial Observability
Language Instruction: place red block on the table.
Logical Goal: (is-on red-block table)
Initial State: The red block is in the drawer and the drawer is closed.
Goal Satisfaction: The red block is placed the table.
Variations: Find the blue block or the pink block.
Task-5
Task Category:
Partial Observability
Language Instruction: place a red block on the table.
Logical Goal: (is-on red-block table)
Initial State: The red block is behind the sliding door.
Goal Satisfaction: The red block is placed the table.
Variations: Find the blue block or the pink block.
Task-6
20

---

Task Category:
Geometric Constraint
Language Instruction: open the slider.
Logical Goal: (is-slider-left slider)
Initial State: The sliding door is on the right and there is a pink block on the path of the sliding
door the left.
Goal Satisfaction: The sliding door is within 5em of the left end.
Variations: Move the slider to the right.
B.2 Baseline Implementation
HULC. This baseline is a hierarchical policy learning method that learns from language-annotatee play data using hindsight labeling [72]. It's one of the best- pest-performing models on the D - D split of the CALVIN benchmark. We omit the comparison to the HULC++ method [24], the follow-up work of HULC that leverages affordance prediction and motion planning to improve the low-level skills, because evaluation is focused on the task planning ability of the learned hierarchical model.
SayCan. This baseline combines an LLM-baser planner that takes the language instruction and learned feasibility functions for skills to perform task planning. SayCan relies on well-trained affordance models to filter out inadmissible actions from the LLMs. Because we learn new skills from demonstrations, is infeasible to learn Q-functions for these skills through RL. We opt to use two other types of affordance functions*. The first affordance function based on the detected objects in the scene.. We provide the detected objects in the prompt. The second affordance function is an image-based neural network that predicts the likelihood of taking an action in a physical state, and the network is trained on our demonstration data. The backbone of the neural network is the same as our image-based predicate classifiers. The prompt of the model is listed in Listing 8.
Robot- VILA. This baseline performs task planning with a VLM. We adopt the prompts pro- vided in the original paper the CALVIN environment. The prompts are divided into the initial prompt that is used to generate the task plan given the initial observation (shown in Listing 9) and the follow-up prompt that is used for all subsequent steps (shown in Listing 10). We use 10L-4-ttrro-2024-0--
as the VLM. Because the model does not memorize the history. We store the history dialogue, including the text input and the image input, and concatenate the history dialogue with the current dialogue as the input to the VLM.
T2M-Shooting. This baseline (in particular, the shooting- based algorithm) is similar the SayCan algorithm except that: 1) it uses multi-step feasibility model in contrast to the single-step feasibility model used by SayCan; 2) the LLM additionally takes symbolic state description of object states and relationships. The original Tex12Motion method assumes access to ground-truth symbolic states. For comparison, in this paper, we compare Text2Motion with BLADE in two settings: one with the ground-truth states and the other with the state classifiers learned by BLADE. The prompt of the model is listed in Listing 11.
C Qualitative Examples
In this section, we include three qualitative examples from the CALVIN experiments to compare the generalization capabilities of BLADE with baselines. Specifically, Fig. A4 shows generalization to abstract goal, Fig. A5 shows generalization to partial observability, and Fig. A6 shows generalization to geometric constraint. In summary, BLADE is able to generate accurate long-hooe morizoo manipulation plans for novel situations while latent planning, I.M,, and VLM baselines fail.
D Real World Experiment Details
We validated our approach in two real-world domains. As shown in Fig. A1, employ 7-degree of freedom (DOF) Franka Emika robotic arm equipped with parallel jaw gripper. A total of Five Intel RealSense RGB-D cameras are used to provide observation for our policies and state classifiers. Four cameras are mounted on the frame and one additional camera is mounted on the robot's wrist.
"We closely follow the official open-source SayCan implementation, available at https://github.com/ google-rseerrchgooolee.ecc http://wwwwoggoeeeeeee//e//aa/aa
21

---

Franka Emika Robot
RealSense RGBD Camera
RealSense ROBD Camera
RealSense (Wrist-mounted)
RealSense
Figure A1: We 7-degree of freedom (DOF) Franka Emika robotic with parallel jaw gripper for real-world experiment. total Five Intel RealSense RGB-D cameras are to provide observation our policies and state classifiers. Four cameras are mounted on the frame and an additional one is mounted to the robot's wrist.
Our teleoperation system uses a 3DConnexion SpaceMouse for control. During the collection of demonstrations, we record the the pose of the end effector, the gripper width, and the RGB-D images from the five cameras. We collected 85 demonstrations for each the two real-world domains, which provide the training data for the diffusion policy models and the state classifiers.
D.1 Task Design
Similar to our simulation experiments, our evaluation protocol includes the design of six tasks aimed at assessing the model's generalization capabilities across new long-horizon tasks. These tasks are specifically crafted test the model's proficiency for four types of generalization: Unseen Initial Condition, State Perturbation, Partial Observability, and Geometric Constraint.
Task-1
Domain: Boil Water
Task Category: Unseen Initial Condition
Language Instruction: Fill the kettle with water and place it on the stove
Logical Goal: (and (is-filled kettle) (is placeedooon kettle stove) (is-turned-off faucet-knob))
Initial State: The kettle placed inside the sink, and the stove is not blocked. The faucet turned off with the faucet head turned away.
Task-2
Domain: Boil Water
Task Category: State Perturbation
Language Instruction: Fill the kettle with water and place it on the stove
Logical Goal: (and (is-filled kettle) (is-placed-on kettle stove) (is-turned-off) faucet-knob))
Initial State: The kettle is placed inside the sink and the stove is blocked.
Perturbation: The human user moves the kettle from the sink to the table after the robot turns the faucet head towards the sink. The robot needs to replan to move the kettle back to the sink.
Task-3
Domain: Boil Water
Task Category: Geometric Constraint
Language Instruction: Fill the kettle with water and place it the stove
Logical Goal: (and (is-filled kettle) (is-placed-on kettle stove) (is-turned-off faucet-knob))
22

---

Not Seen During Training
Training Demonstration States Intermediste States
1. L.Paae Sink
Nove Fauset head
tun On Foueet
Turn OIf raucet
Nove faucet Away
Placo Poton Tuble
7.Move Kottle Stove
Human Porturbation Porturbed States
Unseon inttaal Conditions Initial States
Deomctric Constraints
Inttaal States
Figure A2: Training and testing states for the Boil Water domain.
Not Soon During Training
Training Demonstration States Intermediate States
Hace Tesbag Kentle
Perfurbation
Constraints
Observabiiity
Figure A3: Training and testing states for the Make Tea domain.
Initial State: The kettle is placed inside the sink and the stove blocked, creating a geometric
constraint.
Task-4
Domain: Make Tea
Task Category: Unseen Initial Condition
Language Instruction: Place the kettle the stove and place the teabag inside the kettle.
Logical Goal: (and (is-placed-on kettle stove) (is-placed-insts inside teabag kettle))
Initial State: The kettle is placed inside cabinet. The cabinet doors are open. The drawer is closed.
Task-5
Domain: Make Tea
Task Category: State Perturbation
Language Instruction: Place the kettle on the stove and place the teabag inside the kettle.
Logical Goal: (and (is-placed-on kettle stove) dis-placed-insiie teabag kettle))
Initial State: The kettle is placed inside the cabinet and the cabinet door is open. The drawer is
initially closed.
Perturbation: Once the robot opens the drawer, human user closes the drawer.
23

---

Task-6
Domain: Make Tea
Task Category: Geometric Constraint
Language Instruction: Place the kettle on the stove and place the teabag inside the kettle.
Logical Goal: (and (is-placed-on kettle stove) (is-placed-inside teabag kettle))
Initial State: There is teapot blocking the cabinet doors. The kettle inside the cabinet. The
drawer is open with the teabag visible.
Task-7
Domain: Make Tea
Task Category: Partial Observability
Language Instruction: Place the kettle on the stove and place the teabag inside the kettle.
Logical Goal: (and (ispplaced-on kettle stove) (is-placed-inside teabag kettle))
Initial State: The kettle is placed inside a cabinet and is not visible.
D.2
Qualitative Examples of Novel States
In Fig. A2 and Fig. A3, visualize and confirm that more than half of the initial states and the perturbed states are not part of the demonstrations in our experiments; therefore, purely imitation- naming-bbssed methods will struggle to solve.
E Prompts for Baselines
In this section, we provide the prompts for the baselines used in the simulation experiments. We provide the prompts for SayCan in Listing Robot- VILA in Listing 9 and Listing 10, and T2M- Shooting in Listing 11.
24

---

Abstract Goal: "Place All Blocks Drawer"
Initial State
Open Drawer
Close Drawer
Planner loops
open drawer, close drawer
Initial State
Drawer
Place Block
Place Block Table
Robot-VILA
Planner fails to place any block into the drewer
Text2Mation Shooting
Planner fails pick up last block place in drawer
BLADE
Figure A4: BLADE and baseline performance on an Abstract Goal generalization task in the CALVIN environ- ment.
Partial Observability "Find Block In Slider"
Initial
Lights
HULK
fails generate relevant
Initial State
Robot-VILA
Planner fails generate plan
Initial State
Grab Block
Text2Motion Shooting
Planner consider the slider closed
Initial State
Open Drawer
Open Slider
Grab Block Slider
BLADE
Figure A5: BLADE and baseline performance on the Partial Observability generalization task the CALVIN environment.
25

---

Geometric Constraints: "Open Slider Door
Initial State
Move Slider
HULK
Planner fails consider the blocked
Robot-VILA
Planner fails to generate plan
Initial
Open Slider
Text2Motion- Shooting
Planner consider the door is blocked
Initial State
Move Block
Open Slider
BLADE
Figure A6: BLADE and baseline performance the Geometric Constraint generalization task in the CALVIN environment.
26

---

Listing 1: Behavior descriptions generated by the LLM for the CALVIN domain.
is bift_blockktable
(raction lift-block-table
:perametera 17block item stable item)
:precondition (and (is-block (block) (is-table ?table) (is-on Voleck ?table) (not (is-lifted (block)))
seffect (and (1iited Pblook) (not. (1s-on Pblock (table)))
bbody (then
(grasp ?block ?table)
(move 2alock)
4 Lift__blox__al.ol
fraction lift_
:perameters 17olock
2slider item)
:precondition (and (is-black bblack] ris-slider ialider) (is-in 7block ?alider)]
seffect (and (lifted 7block) (not (1s-1n 7block 7elider)): :body (then
(groap Pblock 7slider)
(move ?olock)
#1 lift_bloo Siift_block_dratte drawer
(raction lift block drawer
sparameters !7olock Item 7drawer item) sprecondition drawer))
block tis ?drawer) (is 2block 2dzawer) open
(lifted rblock) (not (is-in idrawer))
abody (then
(gzaep 7block ?drawer)
(move 2bleck)
4 place_in_slidor
(saction place-in-si in-siiesl
sparameters 17block item Pslider item)
:precondition (and (is-block (block) (is-slider ?slider) (is-liSted (block))
reffect (and (is in bblcck 2alider) (not (is lifted (block)))
:body
(place Phlock ?slider)
11 place_in__rawwrr draction place in-draver sparameters 1'block item 3drawer ittm) sprecondition (and (is-black Pblack] (is-drawer idrawer) (is-lifted Phlock) (is-open idrawer)
reffeet (and (is in lblckk ?drawer) (not (Le lifted (block)))
:body (then (place 7block ?drawer)
71 place_on_lablle
(raction place-on-table
speraneters !?block item 7table item)
sprecondition (ia-block Pblack) His-tbble ?table) (fa-1ifted ?block))
:effect (and (is-on bblcck Stable) (not (is-lifted (block!)) ibody (then
(place 7block ?table)
4 stack_block
(saction block
sparametera
item ?target (tem)
sprecondition (and (ia-block 7block) tis-bLock Starget) (is-lifted ?bloex))
reffect (and. (stacked 2block ?zerges) (not (is-lifted 2block))
:body (then
(place 7block Itarget)
1 unstack block
(saction unstzck_bleek
27

---

:paraneters W7olock1 item 7block2 item)
:precondition (and (is-block Pblack1) (is-block Phlock2) (stacked Pblock1 ?block2))
reffect (unatacked 7block1 ?block2) (ta-lirted 7blook1) (not (atacked 7blook1 Pblock2)))
:body (then
(graap Phlock1 7b1ock2)
(move 7olock1)
#1 cotale_block_tte
(saction rotate-block-righe
sparameters 17block item 7table item)
:preconditien (and (is block ?block) lis table Stable) (is on ?bleck ?table))
reffect (and
(rotated-right 7bloak) Inot trotated-le:t [block1)]
:body
(groap Pblock ?table)
(move 2alock)
(place ?block ?table)
4s cotate_bleck_it (saction rotate_blook_lert
sparameters 17block iten Stable iteml
:preconditien (and (is-block (block) (is-table 2table) (is-on Vbleck ?table))
reffect (and (ratsted-left ?block))
body (then
(gzsep ?block) (move 2bleck) (place Phlock)
4 mush_bloox_rige fraction push block_ihoc
sparaneters riblock item item)
sprecondition (and (is-black bblack] tis-table stable) (ia-on tblock ?table))
reffect (and (pushed-right (block) (not (pushed-left (block)))
bbody (then
(close)
(push 2olock) (open)
41 push_blosk___ blook_left atact lon push-block-lett sperameters 12block iten)
spreconditien (and (is-block (block)) seffect
(pushed-left tblock))
abody (then (clase)
(push ?a ook) (open)
nove_slider_lec_
(saction ion move move_alider_left sparameters 17slider item)
sprecondition (and (is slider 2slides) (is elider right felider))
:effect (is-slider-lest ?alider) (not (is-slider-rige) ?elider!))
:body (then
(graap 7alider)
Inove ?slidez] (place 7alider)
11 move_sl seo____ldd__rrooee (sact lon move-all nove-sllder-rigtt
sparameters (2slides item)
pprecondition (and (is-slider 2slidez) (not (is-slider-right (slider))) reffect
slider right ?slidez))
:body (then
(grasp Palider) (move 7elid (place ?slider)
28

---

#1 open_drawer
(saction epan-draver
sparametera 17drawer 1tem)
iprecondit fand (ie drawer 2drawe:) (is close ?dzawer)
reffect (and. (is-open 2dzawer) (act (is-close ?dcewer))) :body (then
(close)
(push ?drawer] (open)
47 alose_drawer (tact fon close-drawer
sparamelers 1?drawes item)
sprecondition (and (1s-drawer ?drawe") [is-open idrawer))
reffect (and (is close 7drawer! Inoc lis open ?drawer)))
:body (then
(close)
(push 7drawer] (open)
(taction tarn-an-lightbul.
paaaaeterr 17lightoutb
sprecondition (and (is lightbulb 2lightbulb) (is tuzned off (lightbulb)) :effect (is-turned-on 2lightbulb) lis-turned-off 2lightbulb))
:body (then
(close)
(push ?lightbulb) (opan)
turn_off__ lightbbl
(saction turn-off-Lighhbulb
sparametera t?lightbulh item)
eprecondition (and (18-11ghtbolb 71 llightbulb) (la-turned-on blightbulb))
turned (lightbulb) (not (is tuzned ?lightbulb)
bbody (then
(close) (push flightbulb) (open)
# turn_on_ied
(saction turn item) ittem)
sparameters 12led
sprecondition (is-led 2lad)
reffect (and (is-turned-on ?led) (not (la-turned-oft 71ed1))
body (then
(clase)
(push 21ed) (open)
1 turn_off_lee (sac ion turn-oft-led
sparameters !?led iten)
sprecondition
(is-led 21ed) (is-turaed-on 21ed)
:effect (and (is-turned-ofs ?led) (not (is-turnad-on (led1))
sbody (then
(close)
(push ?1ad) (open)
77 push_ ant_intt_ddaass fraction push-into into drawer
sparaneters !2block item 7drawer item) sprecondition reffect (and (is-in bblckk 2drawer) abody
block bblock] lis ?dzawer) (is open ?dzawer))
(clase)
(push 25lock)
(open)
29

---

Listing Example Prompt for CALVIN-Contact Primitives.
PePrint Act fonss** There
primitive actions that robot can perform, They are:
(grasp 2x Py) ix and 2y are two object variables. 2x is the objecz that the zobos will be grasping, ty the object
ix is currently on oz (plane 7x Py): 7x and ?y are two object variaales es. 7x is the object that the sobot is currently holding, 7y is the object that 7x will be placed on or in.
(move Px): Px Le the object that the robot is currently holding and will be moved by the robot.
(push (x) (move-to 2x): the robot arm will move without holding any object or pushing any object.
object
the
will pushing.
(open) : the robot gripper open fully. (close): the rabot gripper wi close without graaping object.
..Comiined Primitives:**
The primitive actions can be combined into a level routine. For example, (then (graap 7x
Pyl (move ?xt (place ?y1) means the robot will pick up from 2y, nove 2x, place
?z. The possible ccmbination of primitives are:
(thon (grasp ?y! imove 7x1)
B. (then (place 7y1)
C. (then (grasp 2x Py! Inove 2xt (place 2x 22))
(then (close) (push ?x) (open))
Listing 3: Example Prompt for CAIYIN-Eaviromment
Predicates for Bresonditions and kéfects:**
The 1sst of all possible predicates for defining the preconditions and effects of the high- level routine Listed below:
specifying the type the object:
(is-table 7x iteml: : ?x is table.
(is-slider ix iten): : 2x is a slider.
(ia-drawer iten):
drawer.
(la-lightbulb 7x iten): : 7x is a 1ightbu b.
led item):
(is-block 2x
iteml: 2x is bleck.
For spect fying the attributes of the object:
(is-red 7x item): 7x is red, This predicate applies a block. (la-blue item): 7x is blue. This predicate applies block,
(la-pink 2x - item): 7x 1s pink. This predicate applies to block.
for specifying the state of the object: (ratated-left
item): 7x rotated left. This predicate applies to a block.
(rotated-right 7x itemt: 7x is rotated right, This predicate applies
block.
(pushed left 2x itenl: ?x pushed left. This predicate applies to bleck. (pushed-right ght
puahec right. "his predicate applies (lifted 7x item): ix is lifced in the alc. This predicate applies to a block. predicate applies
item):
blook.
(is-opea item): 2x is cpea.
dzawer.
(is-close ix iteml: 7x is close. This predicate appliss to drawer.
(1s-turned-on
item): 7x i5 turnad on. This predicate applies 50 a Hightbu or lad, (la-turned-off 7x iteml: 7x is tuzned off, This predicate applies to Lightbulb led.
tis-slider-led 2x iten): the sliding door of the slider 2x is on the left.
(ta-slider-right
sliding
sllder
right.
specifying relationship between objects: (in-on 7x itam - item) 7x is on top of Py, This prodicaze applies when 7x is a black and 7y 18 table.
(is In 2x item
item): 2x is inside of Py, This predica.e applies when Px is block
and 2y is drawer oz slider.
(stacked
item ?y iLem): : stacked top Py, predicate applles
2y item): unstacked from ?y. This predicate applies when ix and 2y are blocks.
(unstacked Py are blocks.
.TTas Envirenment:**
in the environment where the demonstrations are being performed, there are the following objects:
A table. objests can be placed on the table.
drawer thet can opened. objects con be placed into the draver when is open.
A alider which is a cabinet with si dding door, The aliding door can be noved to the left or Lo the right. Objects can be placed into the slider no matter the posicion of the sliding docs.
A lightbulb that be cen turned on/off with a butten.
led that can turned on/off with button.
30

---

Three blocks that be rotated, pushed, lifted, and placed.
Listing 4: Example Prompt for CALVIN-In-Contew Example.
eemoosttaation Parsing:*.
Now, you will help to pärse several hunan demonstrattons of the robot performing taak and generate Lifted description of how Lo aecomplist; this task.
of performed primitives will be given, with actual object nomes. Threa demonstraticna for the taak of "place_ing in_siiderr ist
each demenstration,
coode "aam===ppimtive_eeepeeeeoe primiti
("name": "grasp", "argumenta"; ("ree_block", "Lable"]]
I"neme": "move", "arguments"; ["zed block"]]
""name": "plase", "arguments"; ("red_block", "slider"]] |"name": "move-to", "argumenta"; (*"])
</code>
<code name- "eaaereeeeealiieeeeeeeeeeee
primitives
I"name": "grasp", "arguments"; "rblu__boock" "table"])
I"name": *move", "arguments"; "iblue_block"
"naae"; "place", "argumenta"; "bbue_blook"" "elider*)]
{ I"name": "move-to", "arguments": [*"]]
</code>
<oode name= name="primis prop:m/ttoe____epeeeee primitives =
"name": "gzasp", "azguments"; ""pin_bbockk "table"])
"name": "move", "arguments"; ["pink
"naan": "plass", "arguments"; ("pink_block", "slider")]
["name": "move-to", "arguments": [*"]]
</code>
-reviious
A list of tasks that can be performed before the current task will also be provided as context possible previous tasks ara:
For task of "place_in_slider"" lift_block_table, lift_blockkdraver movess_ider_rihht
+*xammple Output:**
You should generate lifted description, treating all objects as variables. Foz example, the lifted description "place_in_stlder"" is:
<oode name= "mechanism">
(smochantan place-tn-allder
:paraneters !?block item ?slider item)
sprecondition (and (1s-black 7block] His-s' ider Felider) (ta-li Pblock)
bblcck ?alider) (la-lifted (blook))
:body (then
(place ?block ?slider)
//code>
Listing 5: Example Prompt for CAIVIN--Isttructions
wliink Step-by-Steg:*.
To generate the lifted description, you should think through the task in natural language in following steps. Be extremEly CAREFUT think through step 3a, 38, and 4a,
1. Parse the goal, For example "piace_in_ellder", the goal is place block into the slider
poseible effects achieved previous have been performed. Por "lift_blodd "lift_block_table"" a bleck is lifted from the table and the effect biift_blocc_drawer"
Think
previous actions that
that block lifted. the effect is that the block is lifted. For "move_al sove__sider_rihhte tha sliding door of the slider is moved the right and the effect that the sliding door is on the right. task. The demonstrations are noisy so that the demonstrated primitive sequences may include extra primitive actiens
drawer
3. Parse the demonstrations and cheose the combination of primisives for the
aot necessary for current extra primitive actions can be for the previcus tasks. combining with the understanding of the previous task 3a. Tn this csse, the previous tasks
beginning
The
tha correct combination primitives evant to the currenz task, We should think about how to sequence the previous Lasks with the current Lask. The primitive combination fer the current task should include primitive actions that have been performed. The above example for "placo_in_ale in_alider" is this case. we cen infer that "graap" in the demonstrated sequences is included
current task.
likely
previous
should
primitive combination.
31

---

for the task. therefore choose Ither (place ryl). The semantics is that the rabot place the lifted block in the alider. case, the 4. Think about the praconditicna. Also specify the types of all relevant objeets in the preconditions.
relevant
da, In this effects of the previous tasks. for
previous Laska are selevant to the curzent Lask. We should think about the the effects of previous tasks include the block is clready lifted, šo should specify that the bleck is lifted in the preconditions for the current task.
case, previous Think about the effects. For idde the effacts are that the block is in the slider and the block is not Lifced,
relevant
ourrent
5.
6, Write down the mechanism in the fornat of the example.
svdditional Instructions:vA
1. Make sure the generated lifted deacription atarta with scode name="nechanismi and ends with </code>,
2. Please do not invent any new predicates for the precondition and effect. You can only use
the predicates 1lated above.
3. Consides the physical constraints of the objects. For exemple, robot
go of predi catea for spect fying the type of the object to indicate its tyge (e.g., 18 1a-drawer, 1s-block, and eccl.
through a closed docr.
4. For
:parameters, you should
Listing 6: Example Prompt for CALVIN-Task Input.
AxCurzent Task:Ay in_aawe
+*xxample Saquences:**
coode namo="primi wwwpe/ppmmmmmeeeeeeeeee tie__soqueoe"
primitives =
I"name:: "gzasp", "arguments"; block", "table"]) ""name:: "move", "arguments"; "bblu_block"]]
{ "nane": "place", "azguments": ("blue block", "dzawe="]]
["name:: "move-to", "argumenta": [*"]]
</code>
<eede name- name="primee "priiiiiiiiveeeeeeeeee"""
primitives
["name": "grasp", "argumenta"; ["red ("red_blooke blook", "table"]]
I"name:: "move", "arguments"; [*red block"]ock"]]
I"name": "place", "arguments"s ("red_blook", "drawer"])
!"name:; move "argumenta"; [*"]]
</code>
<code pame="primitin "prinnttive_eeeeeece"""" primitives
["name": *grasp", "argumenta": "ppin_bloook" *table"]] {
("name:: "move", "arguments"; ("pink_block")
"naan": "place", "arguments"; ["pink_block', "drawer"]]
["name:: "move-to", "argumenta": (*"[)
</code>
PPPrvvouu Tasks:** puah_into_drawes Lirt_blook_table, iirt_block_elide
Listing 7: Example Prompt for Predicate Generation.
You are a halpful. zgent in helping rabot interpret hyman demanstrazions and discaver generalized high-level rout ine to accompLish a geven task. --Priiiivee Actionsze
There are seven primitive actions that the robot can perform. They are: (grasp 2x 7y]:
2y are object vacianles. grasping, 2y is the object that ix is currently on oz in.
objecs
will
(place 2x ty): ix 7y are object variables.
objecz
currently ha holding,
is the ohject that 7x will be placed or in.
(move 7x) : 7x la the ob jecect that the robot currently holding and will be moved by the zobot.
(push ix): fx is the object that the robot will be pushing. (move 2x):
without holding any object pushing any object. (open): the robot gripper will open fully. (close): rabot gripper closa without greeping any object.
AvTask Environment:**
In the envirenment where the demonstrations being performed, there the follewing
objects:
table. Objects placed
32

---

drawer that opened. ubjects be placed into the drawer when it is open. A alider which is a cabinet with sliding door. The aliding door can be noved to the left placed into the alider matter position the sliding
right. Objects docr.
Hightbul that can turned on/off with a outton.
à led that can be Lurned on/oft with button.
Three blocks that can be rotazed, pashed, lifted, and placed.
AvTaskva
help rosot write PEDL definitions following actions:
1. lift_red bered_bloo__tbbll
2. bift__ee__boosk_sl11
3. Lift_ree_bllsk_ddt
4. [ff_bblue_bocckttae
5. sift_blua_blook_ee
6. lift_blue_blee wwwp_lloooooooooo blooo_drawer
litt_piak_bloe block_abooeeebelee
8.
9. bitL_pink_bloox_dee
10. stack_bleck
11. unstsck_bleck
place_in_slides
13. place_in_drawe:
place_on_table
15. rotate_redLbioo___rrre
16. rotate_ree_biock__eee
17. beaate_blu__bloc_riig
18. Seaato_bloe_bloc____
rotate__ink__look_
20. salate_pink_blockccee
22. pash_red__lockk_ett
23. bol.kklkueeblokrighhhh
24. wush_blue_block__e
25. push_pink_bl book_bblokkkkhhhhhhh
push_pink_block_le.
27. move_slider_ieft
move_alider_rigit
29. open_drawer
30. close_draver
31. turnon__lihhbulb
32. turn_off_light
lare_on_led
34. tura_off_led
Refore writing the operators, define the predicates that should be used to write the preconditions and effects of the operaters. Group the predicates into unary predicates that dafine the of objects and hinary relationa that speci relations between objeats. For each predicate, 1iat actions that are zelevant.
Listing 8: Prompt for SayCan.
are a helpful agent helping zobot plan sequence actiens accemplish given task.
provide context then provi an example to perform task.
.TTask Envircnment:*.
robot's environment,
following objects:
A table. Ob'ects can be placed on the tabie.
that can opened. ubjects can placed A alider which is cabinet with sliding door. The aliding doar can he noved to the left or to the right. Objects can be placed into the alider no matter the postzion of the aliding door.
draver
open.
A lightbulb that be cen turned on/off with è butten. on/of: with Thres blocks that can be rotazed, pushed, lifted, and placed.
led that
wwActions:*w
There are the following sons that the robot can perform. They are: biff__eedblock_tttt lift the red bleck from the table. lift the red block from the slider. the
lift_red_blode
drawer.
-lift_blue block 1ift the blue block from the table. slider. bluee_lookk_ok boloekdrrawers lift the blue block from the drawer. lift_blun
-lift_blae_be
115t the pink block from the table.
hit__pink_blok_ki___ lift the pink block from the slider.
www.ppinkbbkkkkk.e lift the pink block from the draver.
block:
33

---

ploce_in_elider: place the block in the slider.
placa_in_ in__wer_ place the block in the drawer.
place_on_tables place the block
table.
rotate the rnd black to the right. socate_red__look_leerf rotate the red block to the left. rotate the blue block to the right. socste__Blue_Blockke rotate the blue bLock to the left.
rotate_red__loeke
toratt__iin__llock_iihh
the pink block to the right.
rocate_pink rocate__inn__look__ blt rotate the pink 5 ook to the left. right.
push__ee__look_righht puan the red block to
sesh_ree_blook_lefft push the red black to the left.
push_blue_Biookkright: push the blue block to the right.
jish_blue_blcsk_eeee push the blue block to the left. push_ push.pink.blckk.righht: push the pink bleck to the right.
tuah_pinn__bcck_llfft push the pink block the left. move_al move the alider to the left. move_elider_right: move open_drawer: open the drawer.
slider
right.
close_drawers close the drawer. the lightbulb.
surn_off_lighthen turn off the lightbulb. turn_or_lad: turn the led. turn_off_iee turn off the led. do_nothing: nothing.
ExExample Task:Av
Now, you will help parse the goal predicate and generate list of candidate actions the robot can patentially take to accomplish the task, You should rank the actions in torns of how likely they are to performed Goal predicate: (is Luzned off led) output:
"pyyhor
l'Eurn_off_ied', 'do_nothing')
In this example above, if the led is on, the robot should burn it off, If the led is already off, the robot should nothing,
when robot successfully completes an action, the will ask the next action take.
Considering the executed task: turm_off_led
Your output should be: "python
["d__nothiag']
since lad already off, the robot should nothing.
Adddittonns instructions:
1. Make sure the generated plan is Hstt of sctions. Flace the Hist between "Ppython and ends with 191, Think Step by Step.
Goal predicate: (Based on given taak]
Current symbolic state: (Based on the simulator state)
Executed actions: (Based on the previously executed actionsl
Listing 9: Initial Prompt for Robot-VILA.
You are highly skilled in robetic task planning, bresking down intricate and long-tezm taske into diatinot primitive
If the object is in sight, you need to directly nanigulate it. IS the object is not in sight, you to primitive skills find the object
first. If the target object in blacked by other abjests, naed to semove all the blocking objects before picking up the target object.
the tine, you need to ignore distracters that not zelaced to the Lask, And zemember your last step plan needs to be "done".
Consider the following skills a roboric arm can perform. the red block
the table. hift__rod_hocckktthll lift the red block from the alidar. hit_rredblook_llaee
-iift_red_blot
blockblocckcdreeeeeeeeeeeeeeeeee 1ift the red block from the drawer,
hitt_blee_bicc__tabeee 11st the blue block from the table.
lift blt._bbecck_sideerr.........eeee lift the blue block from the slider.
draver. pink block the table. slider. htt_ppinnkkbkkkoook draweerrddderrsoooo lift the pink block from the drawer. stack stack_block: block: stack the blocks.
lift_bl bt__buuebboo_ddreeeee 1ift
block from baank_block_tbblee 1ift the pink block from
slfttpinkkbbl
httt__inn_bbook_kle
lift
jisse_in__lider: place the block in the slider.
ialccaiin_dreeer place the block in the drawer.
place_on_tabl place the en table.
34

---

pestt_rre__booc_riiitt rotate the zed black the right. saaatt_red_bloo__eee rotate the red block to the left. block to the right. rotato_blue__lookkk blot rotate the blue block to the left. Doccat__Pnn__bloo_rrggg rotace the pink block to the right.
Bocate_blue_blookrrgghh rotate
tolaate_pin__Blooeleeet rotate the gink bLoek to the left. pusk__ee__loockriihtt push the red bleck to the right.
Fush_rod__boockllett: push the red block the left.
beah_blle__bccck_righ push the blue block to the right. push blabb_blue_llokklleff puan
block the left.
Beaah_pink_blcc_rtaa push the pink block to the right.
push_pink_Bloskklefft puan the pink block to the left.
move==lider_ieft:
the slider the Left.
move_alider_right move the slider to the right. cpen_drawar: open the droser. alose_drraer:: close the drawer, on bomm_off_iighhille turn off the lightbul..
Lura_on Lughioo__
lightbulb.
turn_on_led: Luzn on the led.
tura_off_ied; turn off the led.
done: the goal hes resched.
You are only allowed to use the provided skilla, You can first itemize the task-related jects help you plan. For the actions you choose, List them as List in the following tormat.
<eede> I'turn_of__edd; 'open_draver', "done"]
</code>
Listing 10: Follow-Up Prompt for Rohot-VILA.
This image displays scenaric after you have executed staps from the plan generated earlier. When interacting with people, sometimes the robetic azm needs to wait for the person' action. you searching elaewhare. continue to generate the plan given the updated ronment state.
target object
current image, you need continue
Listing 11: Prompt for Text2Motion.
#bbjecttvee
You are helpful agent in helping a robot plan sequence actions to accomplish given task.
I will firat provide context and then provide an example how to perfozn the taak.
**Task Enviroment::*
zobot's s environment, there aze following objects:
table. objects can be placed on the table.
-A that can opened, objects can placed
drawer when open.
slider which is cabinet with a sliding door, The sliding door can be moved to the left or to the right. Objects can be placed into the slider no mazter the position of the aliding door.
lightbulb that be can turned on/off with button.
that can turned en/off with button. Three blocks that can be rozated, pushed, lifted, and placad.
PPeeiccates for symbolic states..
The list of all possible predicates for defining the symbolic state are listed below: (rotated-left
rotated left. This predicate applies (cotated right 2x item) 2x is solated right. This predicate applies to block.
iten):
block.
(pushed-left ix iteml: 2x pushed left. this predicate applies to block.
(pushed-right 7x item): 7x is pushed right. This prodicate applies a black.
[lifted 7x item:: 7x lifted in the air. This predicate appl to block.
(is open item): 7x is open. This predicale applies to drawer.
(is-close 2x item) 2x is clese. This predicate applies to a drawer. Luzned
This predicate applies (is- his-turned-off 2x item): 2x is turned off. This predicate applies to lightbull or
iten): :
turned
lightbulb led.
(1s-slider=laft 7x item): the aliding door of the slider 7x is on the left.
(is-slider-right 7x item) ; the sliding door of the slider 7x on the right.
(is-on 2x iten ?y item):
of This predicate applies when ?x is block
and 2y is table.
item Py item) :
inside of Py. This predicate applies when
block
and ły is a draves or a slider.
item Py item) ix is stacked on top iy. This preéicate applies when ix and Py are blocks.
(stacked
(unstacked 2x item Py isent: 7x is unstacked from 2y. This predicate applles when and 2y blocks.
*Actiions:44
35

---

There the following actions that the robot perform. They lift_red bea_red_look_tableeee lift the red block from the toble. the
slider, -fft_redd_booo__llooo lift the red block from the drawer. seft_red_block__de
blook
[fft_blue_bbook blo__book_ttbles lift the blue block from the table.
btt_lue_lloo__iideee Lift the blue bLock from the slider.
lif: the blue block from the drawer.
iift_pint_blook_table lift the pink black from the table.
bt__pinn__bock_sllae lift the plnk à ook from the a ider. draver.
Lift the pink stack_blone stack the blooks.
[tt__pinx_blockddrec
place_in_slider: place the block in the alider.
plaee_in_drawer: place the block in the dzewer.
place_on_table: place the block on the table.
the red block to the right. rotaro_red_bioot_rigghe rotace the red aLock to the left. right. aetate_blue_blook_to rotate the blue block to the left. rotate volate__innk_blokk.iiia rotate the pink block to the right. sotate__pix__looc_lett retate the pink bleck to the left. push the red bleck to the right. the block the left.
rotate_blue
volate__luee_blockkiiphp
blue
push_red_blook__eee
btt_bblue_book_ripee push the blue block to the right. left. pueh_pink__look_riihht push the pink block to the right. pust_pink_blockkleltt pash the pink black to the left. move_slideriefft: the slider to the left.
pusk_bloe_block_let pash
to
move_alider_right meve the slider to the right. open_drawer: open
close_drawer: close the draver. lirn_on_lighhtuu::: turn
lightbulb.
curn_off_lighhhull turn eff the lightaulb.
turn_on_leds turn on the led.
curn_off_ledi
off the led.
Example Now, you will help to parse the goal predicate and generate a sequence of actions to accomplish task.
coal predicster (is-turned-off led]
Symbolic state: (2-tur is-turned-on (lightbuubb) not (i--tureed--f not (is- turnad-off(Lighhbulll Taak output:
"pyyhhon ['Earn_off_eed] '1
Exxmmpee Tesk:**
Goal predi cates (is-turnad-on lad)
Symoolic state: (o-turned-on((edl
ts-turneddonn(Lightbuul not (ie- not (is-
turned off(Lightbulb))
rask output: 'yttho
> Axxmmpl Ta
predicate: lis-in block drawez)
symbolic stete: net (is-in(tedd_blcck crawer)), not (is-in black, slider)), is-oní
red red_ block tabl e), not (ta-opan (drawer)), ss-close (drawerl, 1s-s) 1s-slider=lafttle ider), not (1a-
elider-right (sLidect), not not trif Taak output:
"tyytho
"ropen_dzawer/ "eift_ (11
in_ravei
AxBammple Task:A*
Goal predicate: lis block drawes)
Symonlic atate: not (1s-in(red_3 cok, drawer)), not (1a-in(red, block, alider)), not (1a-on block, lable)), (drawee,, not elider-right (slidesl), bidckk Task output:
slider left (elider), tis-
spython
>*Bxample
Goal predicete: (and lis-turned-en lightbulb) fis-slider-righ: slider))
Symbalic stete: is-slider-left (alider),, not (io-slldor-right (clidd is-turned-offi lightbllll, not (la-turned-on (1ihhbbubb)
Task outpui:
"yython
36

---

'move_sllder_=ihhe"
*dddittiona instractiong:**
Make sure the generated plan actiona, the between ">yyhhon and ends with **,,
2. Think Step-by-Step
37


Reviews:
[{"review_id": "DwciZNkmMD", "reviewer": "Reviewer_ZoiQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Learns symbolic predicates and predonditions/effects of skills to compose and replan.", "summary_of_recommendation": "Interesting paper, but seems incremental and stretching a lot results without clear step forward. Missing some major works in this direction.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "-Which planner is used in the work?\n\n-Why is SayCan performance so bad? Which grounding affordance policy is used?\n\n-Are there any negative examples from the predicate annotation? How are they generated?\n\n-How do you get reliable operators with preconditions and effects? It is stated that LLM automatically proposes them from the primitive sequences and language-paired demonstrations. However, it automatic predicate annotation it is assumed to have preconditions and effects to label the predicates.\n\n-How is segmentation done?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkqB9RjmPr", "reviewer": "Reviewer_cYEP", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed a framework for long-horizon robotic manipulation by combining imitation learning and model-based planning", "summary_of_recommendation": "This is a good work in pushing the field, however there are some design choices i would hope that the author could address, and i am open to changing my rating.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Refer to the reviews", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxNBiZ5xNU", "reviewer": "Reviewer_UF9m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes BLADE (Behavior from Language and Demonstration), an approach to constructing a skill library and set of high-level action representations that capture the precondition and effects of the skills using LLMs. The input is a set of demonstrations with language descriptions, and the output is a set of operators defining the precondition and effects along with learned grounded classifiers for the predicates, which can then be used by a bi-level planner to execute a novel task.", "summary_of_recommendation": "I am quite borderline on this paper, with a lean towards accepting this paper due to the clarity of the presentation, novelty and experimental results, but mostly conditioned on the authors properly addressing the questions raised, and mitigating my concerns over how dependent this method is on the quality of the input predicate names from the user.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- “To facilitate LLM generation, we provide additional information on the list of objects with which the robot has contact”. How was this additional information provided, by a human user? \n- So all the inferred effects from the LLM are based on the contact sequences and language-annotation with the skill? Would this make it difficult to learn effects that aren’t easily captured by the described contact-based primitives / doesn’t it put a lot of the burden on making sure the language annotation of the skill captures the intended effect? \n- “After LLM predicts the definition for all behavior, we will re-segment the demonstrations associated with each behavior based on the LLM-predicted body section”. Why is this done? If the LLM is fed the possible sequence of contact primitives associated with the language description from the dataset, why does the LLM need to predict the body (does it really differ from the input?) / why is this re-segmentation done? \n- On line 128, it is stated one of the inputs to the LLM is “a general description of the environment”. Where does this description come from? \n- On line 133, it references Fig 3c, but shouldn’t it be referencing 3d? \n- For classifier learning: “only pixels associated with the object faucet will be the input to the turned-on(faucet) classifier”. Does this imply the method assumes there is one instance of each category (i.e: there is a single faucet in the scene). If not, if there are multiple instances of the same category (i.e: multiple faucets), how does it associate the pixels with the correct instance?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper's strengths include:
BLADE integrates imitation learning and model-based planning, leveraging LLMs to generate symbolic action representations, which is seen as an interesting and relatively novel method.
The framework demonstrates robustness to disturbances, partial observability, and generalization to unseen initial states, making it promising for complex robotic manipulation tasks.
The paper is well-organized and clear, allowing readers to understand the core aspects of the proposed method easily. The experimental results, particularly those involving real-world robots, are impressive and well-documented.
The experiments show significant improvement over baseline approaches, demonstrating the effectiveness of BLADE in various scenarios, including handling geometric constraints and unseen conditions.
The paper's weaknesses include: 
The method relies heavily on symbolic models generated by LLMs, which raises concerns about the realism and practicality of these assumptions. The approach's effectiveness is contingent on high-quality input from LLMs, which may not always be feasible.
The requirement for users to provide a list of predicate names in natural language is seen as a significant limitation. This dependence on user input could make the system less autonomous and more challenging for non-experts to use effectively.
The paper does not compare its approach to other representative methods in Task and Motion Planning (TAMP), such as those by Silver et al. (2023), which could provide a more comprehensive evaluation of its contributions.
Certain technical aspects, such as the specifics of the planner used, segmentation, and how reliable operators with preconditions and effects are derived, are not thoroughly explained, leaving some doubts about the reproducibility and robustness of the method.
The choice of SayCan as a baseline is criticized, particularly given its poor performance in the experiments. Reviewers suggest that more relevant and recent approaches could provide a better benchmark for comparison.
While the framework shows potential, its applicability to a broader range of tasks is questioned, especially given its reliance on specific assumptions and symbolic models that may not generalize well to more varied or complex tasks.

Post rebuttal meta-review:
The paper introduces BLADE, a framework that integrates imitation learning and model-based planning for long-horizon robotic manipulation. It leverages language-annotated demonstrations and large language models (LLMs) to create structured action representations.

BLADE constructs a library of high-level action representations, including preconditions and effects grounded in visual perception. The framework is noted for its ability to generalize to novel situations, handle disturbances, and work with partial observability, which are validated both in simulations and on real robots. The approach is relatively novel in integrating LLMs for symbolic action representations in robotic manipulation tasks.

BLADE demonstrates robustness to disturbances and partial observability. The method generalizes well to unseen initial states and novel goals. The framework is well-documented with impressive experimental results, showing significant improvements over baseline methods. Weaknesses:

The reliance on symbolic models generated by LLMs raises concerns about realism and practicality. The method's effectiveness depends heavily on the quality of input from LLMs. The paper lacks a comprehensive comparison with other representative methods in Task and Motion Planning (TAMP), which affects the evaluation of its contributions. Certain technical aspects, like the specifics of the planner and the derivation of operators with preconditions and effects, are not thoroughly explained. Final Recommendation:

Borderline Accept: The paper has strong merits in its novel approach and experimental validation but has significant limitations, particularly in its reliance on LLMs and lack of thorough comparison with other methods. Two reviewers lean towards waek accept with minor reservations, while another expresses concerns about the limitations, suggesting a borderline reject depending on the perspective.
Decision: Accept