Grounding Language Plans in Demonstrations Through Counterfactual Perturbations
Authors: Yanwei Wang
Tsun-Hsuan Wang
Jiayuan Mao
Michael Hagenow
Julie Shah
Venue: ICLR 2024 spotlight (2024)
Keywords: Grounding LLM
Learning Mode Abstractions for Manipulation
Learning from Demonstration
Robotics
Task and Motion Planning
Primary area: applications to robotics, autonomy, planning

Abstract: Grounding the common-sense reasoning of Large Language Models in physical domains remains a pivotal yet unsolved problem for embodied AI. Whereas prior works have focused on leveraging LLMs directly for planning in symbolic spaces, this work uses LLMs to guide the search of task structures and constraints implicit in multi-step demonstrations. Specifically, we borrow from manipulation planning literature the concept of mode families, which group robot configurations by specific motion constraints, to serve as an abstraction layer between the high-level language representations of an LLM and the low-level physical trajectories of a robot. By replaying a few human demonstrations with synthetic perturbations, we generate coverage over the demonstrations' state space with additional successful executions as well as counterfactuals that fail the task. Our explanation-based learning framework trains an end-to-end differentiable neural network to predict successful trajectories from failures and as a by-product learns classifiers that ground low-level states and images in mode families without dense labeling. The learned grounding classifiers can further be used to translate language plans into reactive policies in the physical domain in an interpretable manner. We show our approach improves the interpretability and reactivity of imitation learning through 2D navigation and simulated and real robot manipulation tasks. Website: https://yanweiw.github.io/glide/

Published as a conference paper at ICLR 2024
GROUNDING LANGUAGE PLANS IN DEMONSTRATIONS THROUGH COUNTERFACTUAL PERTURBATIONS
Yanwei Wang Tsun-Hsuan Wang, Jiayuan Mao, Michael Hagenow, Julie Shah
MIT CSAIL
ABSTRACT
Grounding the common-senie sense reasoning of Large Language Models (LLMs) physical domains remains a pivotal yet unsolved problem for embodied Al. Whereas prior works have focused on leveraging LLMs directly for planning in symbolic spaces, this work uses LLMss to guide the search of task structures and constraints implicit in multi-step demonstrations. Specifically, we borrow from manipulation planning literature the concept of mode families, which group robot configurations by specific motion constraints, serve tween the high-level language representations of an LLM and the low-level physical traeectories of robot. By replaying few human demonstrations with synthetic perturbations, we generate coverage over the demonstrations' state space with additional successful executions well counterfactuals that fail the task. Our explanation-based learning framework trains an end-to-end differentiable neural network predict successful trajectories from failures and as by-product learns classifiers that ground low-level states and images in mode families without dense labeling. The learned grounding classifiers can further be used to translate language plans into reactive policies in the physical domain in an interpretable manner. We show our approach improves the interpretability and reactivity of imitation learning through 2D navigation and simulated and real robot manipulation tasks. Website: https://yanweiww github.io/gliche
abstraction layer be-
INTRODUCTION
Language models, in particular, pretrained large language models (LLMs) contain large amount of knowledge about physical interactions in an abstract space. However, grand open challenge lies in extracting such semantic knowledge and grounding tasks with embodied agents. Previous methods, given the symbolic and abstract nature of language, primarily focus on leveraging LLMs to propose abstract actions or policies in purely symbolic spaces top of manually defined high-level primitive abstractions (Liu et al., 2023; Ahn et al., 2022; Wang et al., 2023). Such approaches inherently require a set of predefined primitive skills and additional toolkits for estimating affordances or feasibility before executing a plan generated by LLM (Ahn et al., 2022; Lin et al., 2023).
physical domains to solve multi-step
To address this important limitation, in this paper, we consider the problem of grounding plans in abstract language spaces into robot demonstration trajectories, which lie in the low-level robot configuration spaces. Our key idea is that many verbs; such as reach, grasp, and transport; are all grounded on top of mode families that are lower-dimensions manifolds in the configuration space (as in manipulation mechanics, see Mason, 2001; Hauser Latombe, 2010). Therefore, LLMS can be prompted to describe the multi-step structure of demonstrations in terms of semantic mode abstractions: valid mode transitions describe pre-conditions for mode-based skills, and mode boundaries explicitly encode motion constraints in the physical space that are critical for task success.
Building upon this idea, we propose Grounding Language DEmonstrations (GLiDE, illustrated in Fig. 1), which casts the language grounding problem into two stages: learning to classify current modes from states, and learning mode-specific policies. The main challenge in mode classification is that learning decision boundary fundamentally requires both positive and negative labeled examples. To avoid having humans exhaustively provide dense mode annotation that covers the entire state space, we propose to systematically perturb demonstrations generate "counterfactual" trajectories and use a simple "overall" task success predictor as sparse supervision. Intuitively, perturbations to inconsequential parts of successful replay add unseen state coverage, while perturbations that cause
"Corresponding author; yanwei @mi edu

---

Published as a conference paper at ICLR 2024
Prompt LLM select features for demonstrations picking and placing the red
'ko': left
description) tight_ee....
K2' center ee, can xys, can bin
'k2' center_es venter_ee...>
can xyz.
task description
kB'
'4' table center...
bin....
and
k6' cereal bin...,
'K' bread bin....
k8' mou_iin...
state
(b) Plan mode sequence transporting the red can
Reach -> Grasp >>
Transport
(c) Collect demonstrations
discrete modes
continuous states
classifier
S
A
Sj
classifior
Φ
Figure 1: GLiDE framework Given common-sense LLM that understands (a) the appropriate state abstrac- tions for task and (b) how to solve the task via sequence of manipulation modes in semantic space and (c) unsegmented human demonstrations that embody the transitions through these modes, we learn grounding classifier that maps continuous physical states and observations to discrete semantic modes. Mode boundaries discovered by the classifier encode constraints implicit in the demonstrations that are critical for task success.
counterfactual failing outcomes reveal constraints in the demonstration. Next, based learning paradigm (DeJong & Mooney, 1986; Segre & DeJong, 1985) to recover the mode families that successful demonstrations implicitly transition through. With learned classifier that maps continuous physical states to discrete abstract modes, we can then learn mode-specifie policies and also LLMs plan for recovery from external perturbations other sources of partial failures. Our system improves both the interpretability and reactivity of robot learning of multi-step tasks.
explanation-
Our framework of grounding language plans recovering modes and learning mode-specifes policies brings two important advantages. First, compared to frameworks that generate robot behavior solely based text, we do not require pre-built policies and feasibility predictors for primitive actions. Experiments show that our learning paradigm can successfully identify each mode from the demonstration data without any human segmentation annotations, and from only a small number of expert-generater demonstrations. Second, connecting demonstrations with language suggests principled way to improve the interpretability and reactivity of motion imitation. While plenty of data collection systems (Zhao et al., 2023; Fang et al., 2023; Wu et al., 2023; Fu et al., 2024; Chi et al., 2024) allow humans to demonstrate complex multi-step tasks, these demonstrations are typically unsegmented without semantic annotations of individual steps. Neither do humans elaborate on the task constraints that successful trajectories implicitly satisfy. Consequently, the resulting imitation policies cannot detect whether current actions fail achieve pre-conditions (Garrett al., 2021) of subsequent actions or replan to recover from mistakes due to covariate shift (Ross et al., 2011). Our system enables the usage of LLMs for replanning and improves the overall system robustness.
2 METHOD
Our framework, GLiDE, takes in language description of the target task, and small of successful human demonstrations as input, and aims to produce a robust policy that can accomplish the task successfully even under perturbations. GLiDE first uses perturbation strategy to augment small set of human demonstrations with additional successful executions and failing counterfactuals. (Section 2.1). Next, prompts a large language model (LLM) decompose the very high-level instruction into a step-by-step abstract plan in language. At this step, the most important outcome feasibility matrix that encodes how we can transition between different modes in this task (Section 2.2). Given the augmented demonstration and perturbation dataset and the ILM-generated abstract plan, we ground each mode onto trajectories (Section 2.3) and generate motions for individual modes to be sequenced by language plan (Section 2.4).
2

---

Published as a conference paper at ICLR 2024
Example Perturbations
Extracting commonsense knowledge from LLMS in the semantic spece
Prompt
LLM Generation
Senoritt on atstract plan for the task
[('id": 1, "desc": "Reach the nut'),
Cloan the plate sink [C
("idf: 2, "deaef: "close gr gripser to grasp the rut"),
dese": "Reach plate"),
(1Id"; 3,
"dese": "Align the nut above the pec"), ("1d": 'desc": "Insert nut")]
greop slate"),
<1Adfs "deee": fnowe to the xink"),
(a)
Extract Modes and Transiiions
Mode number: 4 Transition 2-3
Instruction
Insert handle
tabie.
(d)
Prompt
(o)
LLM Generation
Sencrote relevant kioposints and Features,
(b)
cleae the plate ie sirk wat.teett.rooott..ooo
selactedd fretures
fectures
'nut, to www.oottoeeeeeeeee peoppss "nut_to_ robot8_eef..co _sut',
"plate
pasudd attructors [1"4d"; '2', "Larget':
preuzo attractors
plate to. roboto wut pes ....
("Id": '1' "torget": htw__to_oooot_e___eeee
Instruction
Insert nut with honole ovut:bli_festturss
table. robot, base.....
(c)
Figure 2: (a-c) Example perturbations causing replays (blue) to deviate from successful demonstrations (red). The task is to pick up the square nut and place it on the peg. End-effector perturhations at different locations (a) may or (h) may not cause grasp failures. (c) The gripper picks up the nut despite an initial end-effector perturbation but later drops it due to gripper perturbation. LLMs can be prompted (d) describe task solution
via discrete mode sequence or (e) to select relevant features and pseudo attractors for solving task.
2.1 DEMONSTRATION DATA AUGMENTATION WITH COUNTEREACTUN PERTURBATIONS
To learn grounding classifier that can partition the state space being considered into mode families, we need data coverage beyond the regions explored in few successful demonstrations. Additionally, to learn mode abstractions that can be used to predict task success-as opposed to clustering data based on statistical -ingaavety-eegative data that fail by crossing infeasible boundaries are necessary. Assuming an oracle that can label the execution outcome of a synthetically generated trajectory, we propose the following perturbations demonstration replays that might reveal task constraints:
End-effector perturbations Illustrated in Fig. 2c, given a successful demonstration shown in blue, we first sample two points the trajectory, namely X and Y. Next, we randomly sample third point Z in the state space. During the replay shown in red, we replace the XY segment with XZ and ZY. Depending on the location and magnitude of the perturbations, the robot may still succeed in the task (Fig. 2b) or fail (Fig. 2a). revealing that grasping the square nut is a pre-condition for the next step of peg insertion to be successful.
Gripper perturbations Illustrated in Fig. 2c, we randomly toggle the gripper state while otherwise adhering the original trajectory. Failure replays where the gripper drops the nut pre-maturely reveal the motion constraint of holding the nut during transportation.
Given the perturbed trajectories, we execute them using trajectory-folloiing controller in the environment and collect a binary task success signal for each trajectory. Essentially, this gives us a dataset of paired trajectories and their task success labels: (7' (r", where r' {s1 $2. ST . and suce' E {0, 1}. To learn the grounding classifier o(-) that can map 7' to its corresponding mode sequence <(r*) = {m,, m2, ... mr} (mode and mode families are used interchangeably this work), we ask LLMs what modes there are in demonstration, how they are connected, and what constitutes a state s, for given task.
2.2 SEMANTIC DESCRIPTION OF DEMONSTRATIONS AND TASK STRUCTURE FROM LLMS
Explaining continuous demonstrations with discrete mode sequence First, we assume given small set of demonstrations { .))]__ which can be variable at the motion level, satisfy the same sequential transition through K modes, defined words, if we reduce selfftransitions in the demonstrations where m4 = m4+1, mode sequence ΦOg*) for all demonstrations can be reduced to the same K-step transitions σ -> 02 - o- This the form of the language plan prompt LLMs to generate describe demonstrations. The plan informs the number of modes there are as well as the semantic grounding of each mode as seen in Fig. 2d.
E Σ {o and ((s:) = m, E Σ. In other
Representing states with task-informe the state representation s as a set of keypoint- based features or image observations that are relevant to mode classification. In particular, the keypoint-baseer features come from pre-defined exhaustive list of keypoints describing the scene as seen in Fig. la. Each keypoint definition contains (1) the keypoint name and (2) short description of its semantic meaning. Given task description, L.M
abstraction Second, we further prompt LLMs to define
3

---

Published as a conference paper at ICLR 2024
Feasibility tasks with 3, 4, modes
sequential
Differentiable calculation
Task prediction
Computing per-step feasibility
0
score
score 2
0
X
0 0
0
classifier d(
Feasibility of mode transitions
(b) abstraction groups states sharlng the same feasible transitions
$2
3
A few successful demonstrations
perturb
Success labels
mode
mode
mode A
(c) End-to-end explanation-base learning
Figure 3: (a) Example feasibility matrices. Specifically, I can describe the modal structure for pick- and-place task with solution maach-grasp->tranee definition mode transition implies every state in the second mode reachable from every state the first mode (states Y and Z are in the same mode but not X). We leverage this connection between the continuous states and the discrete modes to design (c) fully-differentte differeiffeee pipeline that calculates overall trajectory success based on the mode classification of individual states the trajectory.
where reach brach-ttrnspoor directly infeasible. (b) The
can be prompted to either select subset of keypoints tracking absolute locations or combine pairs of keypoints to track relative positions as shown Fig. 2e. For image observations, we either use the raw image as state representation or use pre-trained vision model (Kirillov et al., 2023) or vision-language model (Huang et al., 2023) to extract features from the image.
Encoding discrete modal structure in a feasibility matrix Lastly, while successful demonstrations ç can be reduced to a K-step language plan, not every perturbed trajectory r can be as it might minimal solution. Therefore, the reduced mode sequence may
not be successful or correspond contain back-and-forth steps such as σ + 02 - 01 - or simply invalid mode transitions. To describe the modal structure of a task in terms of the feasible transitions between modes, we generate a feasibility matrix F with K modes by first querying LLMs whether two semantic modes are directly connected. Then we compute the matrix entry Fsy from LLM responses as the negative shortest path between each pair of modes. In the case of sequential tasks with a linear temporal structure (true for most experiments considered in this work), zero entries Fi, encode valid transitions that incur zero costs. Negative entries F.j encode infeasible transitions, and the magnitudes denote the number of missing modes in between. particular, in Fig. 3a diagonal entries F;; are feasible self-transitions, and entries Fi,+ are demonstrated mode transitions towards the goal. Note for tasks with complex structures, the matrix may have more negative entries than the ones shown in Fig. 3a. The feasibility matrix is also interpretable and can be modified manually by humans.
2.3 END-TO-END EXPLANATION-BASE BASED LEARNING FOR MODE CLASSIFICATION
Given language plan, task-informed state representation, and feasibility matrix discrete structural information about a task, learning the grounding classifier is an inverse problem that tries recover the underlying modal structure from sparsely labeled continuous trajectories. To this end, we design a differentiable decision makinng pipeline to explain the task success of a trajectory on top of mode predictions. Having trajectory coverage with contrasting execution outcomes allows for recovering the precise grounding in terms of mode boundaries.
Mode classifier Our mode classifier is a neural network (with softmax output layers) that inputs state 8, and outputs a categorical distribution of the abstract mode at that state. Overloading the notation (--) to output both a predicted mode m, and mode belief, we have h = ((x)). The architecture of the classifier depends the state representation. The number of softmax categories K chosen based on the sequence length of the LLM-generated plan. If we had dense mode annotations ((s,, m4) T we could train the classifier directly with cross-entropy loss. However, only have supervision at the trajectory level via task success. Therefore, we need a differentiable forward model that predict task success from sequence of mode beliefs (b
Differentiable forward model to predict task success What makes a perturbed trajectory rollout unsuccessful (or still successful) in solving task? Following the approach by Wang et al. (2022a),
4

---

Published as a conference paper at ICLR 2024
consider successful trajectory according to F* and (2) eventually reaches the final mode σg seen in the demonstrations. Since the perturbations consider in this work do not affect the starting state $1 and final state sy, the success
that both (1) contains only feasible mode transitions
criteria for a trajectory r+ solely concerns intermediate transitions: ((({²) ) FK ((+++1 ) = 0, Vs," E
r+. Similarly, a failure trajectory r- is one that contains at least one invalid mode transition.
To operationalize this idea, let's consider the dataset of trajectories T containing both successful tra- jectories 7+ = [// and failure trajectories T- = {[>-)ー> }==1- First, we use a cross-entropy loss to enforce that the starting and ending continuous states for all trajectories must be in the initial and final mode being demonstrated: Linit En.TT Lce ( (0(s)), σ1) and Lannt = ET (c)( ox).
Second, we define the success and failure loss using ft,--1, which is shorthand for transition
feasibility score ((s;) 6($++1) between two states:
T-1 Σ fatt1
T-1 max(-1x Σ fls41)
Succe
1 M
Σ TET T-
Sjau
Σ TyET
(I)
Intuitively, minimizing Lance encourages the classifier to predict mode beliefs such that all transitions between consecutive states are feasible. Minimizing Laui encourages the classifier to predict mode beliefs such that there exists at least one invalid mode transition. The clipping in Ljaz at -1 makes the loss well-defined and treats all invalid mode transitions described by the negative entries in Fig. 3a equally". Fig. 3b gives another intuitive example, where states Y and Z constitute the same mode but not state X. A necessary condition to test if a state s is in mode o; is to check if can directly transition to at least one state in mode Ø++1 in the trajectory. Adding everything together, we have Lpai in Eq. 2, where Xg. Xj. and ); are hyperparameters for balancing loss terms:
Cuut Xuh + A + >e(Ciin Lffna)
(2)
Extension to underactuated systems. These conditions are sufficient for recovering modes from fully-actuated system. For underactuated systems (Tedrake, 2023) such as object manipulation where objects cannot directly move from one configuration to another via teleportation, it is not possible to generate a direct transition between any two modes such as the ones shown in 3b using synthetic perturbations. Hence, we need an additional regularization at the motion level to infer precise boundaries. Specifically,, states in the same mode should go through similar dynamics. In other words, one should be able to infer (8:-1 8x) from (s; 8t-1). Such mapping should be different for different modes. For example, the relative transformation between the end-effector pose and the object pose should remain the same when the robot is rigidly holding the object and change otherwise. Based on this observation, we instantiate a forward dynamics model (b-) that inputs the current state change and predicts how the state should change next for each mode. Coupled with mode belief, we can predict the next state change as ( (h st-1). Consequently, we can train mode classifiers for underactuated systems by introducing dynamics loss Lagn:
T-1
Lunder Liait + XaLayn where
Layy
Σ T
((b())) -8-1) - (8t+1 - -
(3)
Minimizing Cam groups states into modes based on similarity in dynamics. Since losses are differen- tiable with respect to @ and dú we use stochastic gradient descent to optimize learnale parameters.
2.4 MODE-BASED MOTION GENERATION
Having learned the explicit mode boundaries, we can leverage them in motion planning to ensure that the robot avoids invalid mode transitions (LaValle, 1998). Alternatively, we can use the classifier <(-) to segment demonstrations into mode-specific datasets, with which we can learn imitation policies (a(a)s) for each mode a and sequence them using a discrete plan (Wang et al., 2022a). To further improve the robustness of the learned policy for manipulation tasks, we use the mode for each mode. If the mode feature the absolute pose of the robot end-effector, we compute the mean end-effector poses at which mode transitions 0₂ - 0++1 occur as the pseudo-attractor; if it is relative pose, we transform that into an absolute pose of the end-effector at test time. We use this pscudo-attractor to construct potential field that guides the robot to move towards the next mode at inference time. Specifically, the final
feature identified by the LLM to construct a pssudo-attracte
"Empirically, setting all negative entries in the matrix to he -1 can get gradient descent optimization stuck.
5

---

Published as a conference paper at ICLR 2024
Mode 1
(a) Demonstrstions
(b) Our method
(c) No counterfactual
(d) Wrong feasibility
(e) Clustering
Figure 4: Grounding of 2D navigation task. (a) Given six demonstrations that start in mode I and end in mode visualized on top of the ground truth, (b) our method GLiDE recovers the underlying mode abstractions. (c) Without counterfactual data, GLiDE fails to learn precise boundaries. (d) Without a correct feasibility matrix (e.g. 4-mode instead of 5-mode), GLiDE results will miss modes. (e) Lastly, clustering the 2D state space to the nearest mode centers, discovered in the demonstrations by kmeans+++, produces an incorrect modal structure.
Method
3-Mode (+perturb)
4-Mode (+perturb)
5-Mode (+perturb)
Behavior Cloning (BC)
0.967
(0.908)
0.814
(0.614)
0.810
(0.596)
GLiDE +BC
0.963
(0.887)
0.892
(0.753)
0.893
(0.753)
GLiDE +Planning
0.996
(0.996)
0.987
(0.966)
0.991
(0.974)
Table 1: 2D navigation success rates for mode-agnostic imitation (BC) and mode-based (GLiDE) imitation or planning for environments consisting of 3, 4, and 5 modes. We test both perturbed and non con-perturbed settings 1000 times and report the average success rate. For neural network-based BC policies, we report the average performance across 10 different random seeds.
mode-based policy (a]ss) is weighted sum of the original TT. (als) and control command that moves the end-effector towards the pseudo-attractor for mode aı- We only apply the pseudo-attractor term when the distance between the current state and the pseudo-attractor is greater than threshold. Intuitively, when large perturbation leads to out-of-distribution states, the potential field will drive the system back to the demonstration distribution before the imitation policy иT& takes sole effect.
EXPERIMENT
We evaluate method on three sets of experiments: (1) a 2D navigation task, (2) simulated robot manipulation tasks in RoboSuite (Zhu et al., 2020), and (3) real-robot implementation of the 2D navigation and marble-scooping task. Since our robot experiments use end-effector pose control, we refer to task-space features states rather than configurations.
3.1 2D NAVIGATION
Setup The 2D navigation environment consists sequence connected randomly generated polygons, and the goal is to traverse from any state in the free space (mode 1) through the polygon sequence consecutively as demonstrated until reaching the final polygon. This environment serves 2D abstraction of the modal structure for multi-step manipulation tasks, where each polygon represents different mode with its boundaries showing the constraint of the mode. Illegal transitions include non-consccutive jumps between modes such direct transitions from free space any later modes other than mode 2. This system is fully-actuated with (x, y) coordinate as the state representation and (át, ú) the agent action. For all environments, we use fewer than 10 successful demonstrations for classifier learning and policy learning.
Results: Mode classification We visualize the learned grounding classifier in Fig. 4b and Ap- pendix A. Compared to baselines in Fig. 4(c-e), the mode boundaries recovered by GLiDE are the closest to the ground truth shown in Fig. 4a. particular, the poor grounding learned in Fig. 4(c-e) shows respectively the importance of learning with counterfactual data, correct task specification from LLMs, and a task prediction loss beyond clustering solely based on statistical similarities the website.
data. Quantitative results and more visualizations can be found
Task execution Next, we show the learned grounding classifier can be used to improve task success rates, especially in the face of external perturbations. We use behavior cloning (BC) as a mode-agnostic baseline to learn a single policy ([ass) from all successful trajectories. By contrast, method (GLiDE +BC) first segments the demonstrations and then learns mode-specific
Results:

---

Published as a conference paper at ICLR 2024
Method
Can
Lift
Square
Method
Can
Lift
Square
GLiDE
0.83
0.83
0.67
BC
0.93
0.99
0.38
GLiDE Dynamics Loss
0.67
0.75
0.46
BC (p)
0.20
0.18
0.03
GLiDE Prediction Loss
0.67
0.68
0.56
GLiDE +BC
0.85
0.99
0.25
GLiDE Feature Selection
0.55
0.70
0.57
GLiDE +BC (p)
0.40
0.39
0.15
Traj. Seg. Baseline
0.66
0.56
0.54
Table 3: The success rate of mooee-agnosie imitation (BC) drops more than that of mode- conditioned imitation (GLiDE +BC) after in- troducing perturbations (denoted by p).
Table 2:
Ablation study of the influence of different loss terms and baselines on the mode classification accuracy based on overlap (percentage) with the ground truth.
policies. Additionally, instead of mode-based imitation, could also do planning stay in the mode boundaries recovered by the classifier, since the system is fully-actuated with a intle--nt dynamics. Specifically, (GLiDE +Planning) uses RRT to compute waypoints to guide motion in non-convex mode and then uses potential fields in convex polygons to generate trajectories that stay in the mode until entering the next mode. Table I shows that our methods perform slightly better than BC across different environments. However, when external perturbations are introduced, BC suffers the biggest performance degradation as recovery at the motion level without attention mode boundaries may incur invalid transitions leading to task failures. The fact that (GLiDE +Planning) can almost maintain the same success rate despite perturbations validates the learned grounding.
Interpretability In the 2D environment, visualization of learned mode families can expose mode constraints and explain why some but not all perturbed demonstration replays fail the task execution.
3.2 ROBOSUITE
Setup We test GLiDE across three tasks from Robosuite: placing can in a bin (can), lifting a block (lift), and inserting a square nut into a peg (square). We use the default action and observation space of each environment unless an LLM suggests different features (e.g., relative distance to an object or keypoints). Since the manipulation tasks define underactuated systems, we use Eq. for training.
Results: Mode classification To evaluate the mode classification accuracy, we manually define the ground truth modes for each environment (details Appendix B). Table shows the percentage of overlap between mode predictions from different methods and the ground truth mode segmentation. The results show that including all of the loss terms in our method achieves the best boundary alignment with the ground truth visualized in Appendix B. Ablating the dynamics loss or the task prediction loss (Leuce and Laat) degrades the prediction accuracy as the classifier misses the precise location of important events such as dropping a grasped object. Comparing GLiDE training without feature selection shows the importance of using an L.MM to down-sample the feature space for efficient learning. A trajectory segmentation baseline using kmeans++ clustering on the features also underperforms GLiDE, highlighting the limitation of similarity-base
segmentation methods.
Results: Task execution To show the learned grounding can help recover from perturbations, we compare a mode-agnostic BC baseline, which is trained on unsegmented successful demonstrations. to a condiicondiioneed method (GLiDE +BC) described in Section 2.4, where each per-model BC policy is augmented with pseudo-attractor While our method is insufficient to recover from all potential failures, our goal to demonstrate how even basic control strategy that leverages the underlying mode families can benefit policy learning in robotics. Table 3 summarizes the methods performance without and with perturbations, which will randomly displace the end-effector or open the gripper. We see that for both methods, adding perturbations introduces some amount of performance drop. We find that the performance degradation for the BC baseline is much higher than with GLiDE +BC.
Interpretability In manipulation environments, it is challenging to directly visualize the mode state space. However, exposing the mode familics allows us
families given the high-dimensions to easily identify mode transition failures which can be used to generate post-hoc explanations of failures (e.g., videos on our website show invalid mode transitions associated with a task failure).
3.3 REAL ROBOT EXPERIMENTS: 2D NAVIGATION AND SCOOPING TASKS
2D navigation To illustrate GLiDE can also learn grounding classifiers directly from vision inputs, we implement the simulated 2D navigation task on real Franka robot, where the end-effector traces through sequence of colored polygons in plane. First, record 20 human demonstrations
7

---

Published as a conference paper at ICLR 2024
Modo
Mode
Mode
Mode
Mode
Counterfactual
(a) Demonstration
Inferred mode Ground truth mode
Grounding the entire dataset
Figure 5: Illustration the real robot 2D navigation task (a), where the end-effector traces through sequence of colored polygons. (b) shows perturbed trajectory, overlaid on ground truth mode boundaries, experiences an invalid transition from mode to mode 4. A vision-based classifier can predict from only pixels the inferred modes (first color bar) that match ground truth (second color bar) with high probability. (c) visualizes the mode prediction of individual image states seen the dataset. The location of the scattered dots indicates where images are recorded while the colors show the predictions, which are well-aligned with mode boundaries.
through kinesthetic teaching in the end-effector's state space that start in various parts of mode 1 and end in mode 5 as seen in 5a. Second, we add end-effector perturbations to the demonstration replays to generate coverage over the entire tabletop area and record the perturbed trajectories in image sequences as seen in 5b. For this planar task, we use a simple reset mechanism that brings the end-effector back to one of the demonstrations' starting states after each rollout to collect data continuously. Since we can check if the end-effect is within the convex hull of any colored regions whose vertices are known, log the mode sequence of each perturbed trajectory and automatically label if the trajectory a successful task execution by checking if all mode transitions are feasible. Consequently, we were able to collect 2000 labeled trajectories in 2 hours continuously without human supervision. To learn a vision- based classifier, we switch from using multi-layer perceptrons (MLP) for sate-space inputs convolutional neural networks (CNN) to encode image inputs. Figure 5c shows the learned classifier can group image observations into correct modes according to the ground truth mode boundaries.
Marble scooping task The second task requires spoon-holding robot to scoop marbles from a bowl and then transport least one marble to a second bowl across the table. typical robot implementation might require engineering marble detector to check if the spoon is holding marbles and plan actions accordingly (Wang et al., 2022a). Instead, we learn marble classifier on a wrist camera view to leverage LLM-based replanning shown in Fig. 6. To collect successful executions, we record the end-effector's pose and wrist camera view as a human demonstrates scooping from various starting states. Since it is non-trivial engineer reset mechanism for this task, ask humans to demonstrate various failures through kinesthetic teaching as well. To improve learning efficiency, preprocess the raw wrist image extract mask corresponding marble objects, where an empty spoon returns an empty mask. Specifically, we prompt an L.M for relevant object types to track, with which we employ the Segment Anything Model (SAM) (Kirillov et al., 2023) to generate segmentation masks. The classifier is then trained on state representation consisting of end-effector poses and the marble mask. In Fig. 6, we show how our mode classifier can be used to develop a reactive robot program.
RELATED WORK
Learning abstractions from demonstrations. A large body of work focuses on learning action abstractions from language and interaction. This includes the grounding of natural language (Corona et al., 2021; Andreas et al., 2017; Andreas & Klein, 2015; Jiang et al., 2019; Sharma et al., 2022; Luo et al., 2023), programs (Sun al., 2020), and linear temporal logic (LTL) formulas (Bradley et al., 2021; Toro Icarte et al., 2018; Tellex et al., 2011). In contrast to learning policies for individual action terms, this paper focuses on learning mode families in robot manipulation domains. These learned mode families enable to construct robust policies under perturbation (Wang al., 2022a). Furthermore, our framework is capable of recovering the underlying modes from a small number of unsegmented demonstrations, Through the use of synthetic noise (Delaney al., 2021; Wang et al.,
8

---

Published as a conference paper at ICLR 2024
Imitating Continuous Demonstration Start State
Demonstration
End
Spoon
Marble Fallure
(8) Mode Mode-Agnostoe Agnostic
Replanning
(b) Mode Conditioned
Re-Scoop Marbles
Marble Transfer Success
Classifier
Reaching Mode
Transporting Mode
Reaching
LLM
Next
Step: Transport
Next
Figure 6: Illustration of the rohot scooping task. Here the perturbations are human-initiated (e.g.. moving the spoon to drop fail to grasp marbles). A mode-agnostic BC that imitates continuous motion demonstrations cannot replan the task level when all marbles dropped during the transporting mode (a). In contrast, our mode-conditioned policy enabled by grounding classifier can leverage LLMs to replan given external perturbations (b).
2022b), we relieve humans from the burden of providing negative demonstrations that fail task by automatically generating positive and negative variations of task executions.
Grounding language in robot behavior. With the rise of LLMs that can decompose high-level commands into sequences of actions, there has been much recent interest in the ability ground these commands in embodied agents. Given that data from environment interactions (including human demonstrations) do not explicitly identify constraints and success criteria of task, previous work has investigated how to infer affordances directly from observations (Ahn et al., 2022). Compared to prior work (e.g., Lynch et al. (2023)), our method does require dense labels to learn grounding operator. We are also not directly using large language models for planning (Huang et al., 2022; Li al., 2022; Huang al., 2023). Rather, abstractions in demonstrations, and as a result, we can also acquire a grounding operator for high- level language commands.
using L.M to guide the discovery of mode
contrast the discovery of language-conddtioned skills (Lynch Sermanet, 2020; Garg et al., 2022), which can consist of multiple modes, our mode decomposition occurs at lower level and can explain why certain trajectories fail task execution.
Counterfactuals. Counterfactuals describe hypothetical situations of alternative outcomes compared to the original data (Byrne, 2019). other words, they are fake (non-human generated) data with different result (e.g., failing task instead of succeeding) (Karimi et al., 2020). In this paper, we define counterfactual perturbations as non-human -ennratte synthetic probes that test which parts of the time-series trajectory data (Delaney et al., 2021) demonstrated by humans have implicit constraints, the violation of which will change the outcome of the successful human demonstrations.
5 CONCLUSION
In conclusion, this work introduces framework, Grounding Language in DEmonstrations (GLiDE), to effectively ground the knowledge within large language models into physical domains, via mode families. Given small number of human demonstrations and task descriptions, we show how GLiDE successfully recovers mode families and their transitions required in the task and enables the learning of robust robot control policies.
L Limitations and future work. While GLiDE does not need large number of human demonstrations, requires a large number of trial-and-errors and an environment with a reset capability in order to collect task success labels of trajectory. This data inefficiency, however, can be addressed through active learning where the current belief of mode segmentation can be used to probe demonstrations only in regions with high uncertainty. Additionally, prompting the LLM to find suitable state representation for learning the classifier also requires skill. In future work, we would like to learn the state representation in conjunction with the mode classifiers in an end-to-end fashion.

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, al. Do as I Can, Not as Say: Grounding Language in Robotic Affordances. (xXiv:2004.0691) 2022. 1,9
Jacob Andreas and Dan Klein. Alignment-Based Compositional Semantics for Instruction Following. In EMNLP, 2015.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforcement Learning with Policy Sketches. In ICMI,, 2017. 8
Christopher Bradley, Adam Pacheck, Gregory J. Stein, Sebastian Castro, Hadas Kress-Gazit and Nicholas Roy. Learning and planning for temporally extended tasks in unknown environments, 2021. 8
Ruth MJ Byrne Counterfactuals in explainable artificial intelligence (xai): Evidence from human reasoning. In IJCAI, pp.
2019. 9
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.
2024. 2
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modular Networks for Compositional Instruction Following. NAACL-HLT, 2021.
Gerald DeJong and Raymond Mooney. Explanation-bart learning: An alternative view. Machine learning, 1: 1:145-176, 1986. 2
Eoin Delaney, Derek Greene, and Mark T Keane. Instance- based counterfactual explanations for time series classification. In International Conference on Case-Based Reasoning, pp. 32-47. Springer, 2021. 8,
Hongjie Fang, Hao-Shu Fang, Yiming Wang, Jieji Ren, Jingjing Chen, Ruo Zhang, Weiming Wang, and Cewu Lu. Low-cost exoskeletons for learning whole-arm manipulation in the wild. arXiv preprint arXiv:2309. 14975, 2023. 2
Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.021 2024. 2
Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learning interpretable skill abstractions from language. Advances in Neural Information Processing Systems, 35:21711-21724, 2022. 9
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Leslie Pack Kaelbling, and Tomás Lozzno-Pérez. Integrated Task and Motion Planning. Ann. Rev. Control Robot. Auton. Syst., 4:265-293, 2021. 2
Kris Hauser and Jean-Claude Latombe. Multi-modal motion planning in non-expansive spaces. The International Journal of Robotics Research, 29(7)8897-915, 2010. 1
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML, 2022. 9
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv:2307.05973.
2023. 4, 9
Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an Abstraction for Hicrarchical Deep Reinforcement Learning. In NeurIPS, 2019. 8
Amir- Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. A survey of algorithmic recourse: definitions, formulations, solutions, and prospects. arXiv preprint
2020. 9
10

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015 14264 2023. 4,
Steven LaValle. Rapidly-Exploring Random Trees: A New Tool for Path Planning. Research Report 9811, 1998. 5
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 55:31199-31211 2022.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-135,, 2023,
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+ P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv:2304. 115304.11447, 2023.
Zhezheng Luo, Jiayuan Mao, Jiajun Wu, Tomás Lozano-Pérez, Joshua B Tenenbaum, and Leslie Pack Kaelbling. Learning Rational Subgoals from Demonstrations and Instructions. In AAAI, 2023. 8
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.077 07648, 2020. 9
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence, Interactive language: Talking to robots real time, IEEE Robotics and Automation Letters, 2023. 9
Matthew T Mason. Mechanics of robotic manipulation. MIT press, 2001.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011. 2
Alberto Segre and Gerald DeJong. Explanation espllnnatoonbbasdd manipulator learning: Acquisition of planning ability through observation. In Proceedings. 1985 IEEE International Conference on Robotics and Automation, volume pp. 555-560. IEEE, 1985. 2
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas, Skill Induction and Planning with Latent Language. In ACL,, 2022. 8
Shao-Hua Sun, Te-Lin Wu, and Joseph J Lim. Program guided agent. In ICLR, 2020.
Russ Tedrake. Underactuated Robotics. 2023. URL https://underactuutel.call mit.edu. 5
Stefanic Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In AAAI, 2011. 8
Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A. Mcllraith. Teaching multiple tasks to an rl agent using Itl. In AAMAS, 2018. 8
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. vrXiv:2305.1629; 2023.
Yanwei Wang, Nadia Figueroa, Shen Li, Ankit Shah, and Julie Shah. Temporal logic imitation: Learning plan-satisficing motion policies from demonstrations. arXiv preprint arXiv: 04632, 2022a. 4, 5,8
Yanwei Wang. Ching Yun Ko, and Pulkit Agrawal. Visual pre-training for navigation: What can we learn from noise? arXiv preprint arXiv:2207.00052 2022b. 8

---

Published as a conference paper at ICLR 2024
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general. low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.13037, 2023. 2
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705,, 2023. 2
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 2020. 6
12

---

Published as a conference paper at ICLR 2024
A MORE MODE CLASSIFICATION RESULTS FOR 2D NAVIGATION TASKS
Figure visualizes data augmentation of generating additional successful trajectories (middle) and failing counterfactuals (right) from a few successful demonstrations (left). Figure visualizes learned grounding for additional randomly generated 2D navigation environments with 3-, 4- and 5-mode task structure.
Successful demonstrations task
Perturbed trajectories that
Perturbed trajectories that failing
Figure 7: Additional 2D grounding examples. Column (a) shows the ground truth mode segmentation and successful demonstrations. Column (b) shows the learned grounding and its percentage overlap with the ground truth. Columns (c-e) visualize grounding learned without counterfactual data, correct feasibility matrix. and task prediction loss respectively.
(a) Demonstrations and ground truth modes
(b) Grounding learned by GLIDE
(c) GLIDE without counterfactual data
(d) GLIDE with incorrect number of mades
(e) Baseline: similarity- based clustering
Figure 8: Additional 2D grounding examples. Column shows the ground truth mode segmentation and successful demonstrations. Column (b) shows learned grounding its percentage overlap with the ground truth. Columns (c-e) visualize the grounding learned without counterfactual data, correct feasibility matrix, and task prediction loss respectively.
13

---

Published as a conference paper at ICLR 2024
B HEURISTIC RULES FOR MODE FAMILY GROUNDTRUTH IN ROBOSUITE
We use the following heuristic rules to define the ground truth mode families to evaluate the grounding learned by GLiDE:
Can (3 modes): the ground truth modes reaching for the (until the end effector makes contact with the can), transporting the can to the bin, and finally hovering about the target bin. Lift (3 modes): the ground truth modes are reaching for the cube (until the end effector makes contact with the cube), lifting the cube off the table, and finally moving to a certain height above the table.
Square (4 modes): the ground truth modes are reaching for the nut (until the end effector makes contact with the nut), transporting the nut the peg, aligning the above the peg, and finally lowering the nut into the assembled position.
We assess the predicted and ground truth mode (based on the heuristics) for each sample in the robosuite demonstrations to compute accuracy (i.e., the percentage of samples where the predicted and ground -trthh modes are the same). Fig. 9 shows the visualization of the mode segmentation from GLiDE, compared to the ground truth on the can placing task. Our model faithfully identifies all the modes and yields high consistency with the modes defined by human-crafted rules.
"Can" Task
truth
ours
"Lifting" Task
ground truth
Ours
"Square Peg" Task
ground
ours
Figure 9: Comparison the robosuite can task between method's 's segmented modes and the ground truth modes. Generally, our system is able to accurately recover mode boundaries.
C PROMPT ENGINEERING FOR ROBOSUITE TASKS
C.I PROMPT EXAMPLE
14

---

Published as a conference paper at ICLR 2024
You are expest generating rabos action plana and features.
language desur iptiritioo of Lack, suth as Rolean the plate in aink, you should firat generale an abstract plan for the task, Fat than in splanse/stann, The plan is a 1ist of atepa. You should zgnoze object Finding local ization actions. For exerp e,
<plan>
staps I(riar: 1, "daaefs "Beach
platef),
Mfidf 2, ''desff; rclose gripger to greep the plate!),
(fidfs 3, !desefs "Meve the sink'
(ridds ddasefs reamn faucet"]]
</pl
Next, should generate feasibility matrix betweer all steps, Pot then in <teaeibi httpe//eeiitttt/teeeeeeellll <feesibility
feasibillly
Cl 233 True.
aateer reaching the plate, we can directly close the gripper the sink
33: vel
# *after reaching the plate, can't directly
plate, can't diseatly efter closing the gripper, se ca directly ly nove to the sink
seaching
fancel
(2, 3)2 True,
(2, 452 Palse,
after closing the gripper, we pen't direct ly turn en the fauces can't directly
True,
moving the
raucet
6/ d/Eeasibiityy
The will sl give you list of available features, auch rebot object Following:
example is the
<availabie_testurr>>> restures>
avaiable wble features ("plate_pod "platt "plase so roboto eef pos', 'plate rebet eef quat",
siak_pos', "sick_euat', "sink_to_robott rtt___ott_______opoo_ peet_pas"" "sink_to_robet co__rooot__eet patott__ett_uur.. "scuset_pas', 'raucet_quas " "raucet_to_rsbott
"taucet soboto pettot__ett_tee eet_quat', poboss__eet_pon "resotd_eet sea__
gr gripger_stett statte|
</available_ cewailable_ttatrreee tosturess
will the tolloxing conventions relacive
XXX
object, XXX to Wo__ttorbbtteeo
object
end-effector.
You should select a subset of testures, Bach as: <sclecoddtoattre>>> reasures 'plate_to_ clatt_tt__robee wtt_too_ttttto___e_eeoe pepp____ee_pd",, "plate_to pilate_to_rbbott to_.ttoooooooeeeeooooooooooooo ottoooootttttttoooeeeeoe qustt, 'eink tank__torobott rotootttt_etttt setppsss
sink roboto set, quat', "faucet to robot eet pos', to rebotO eef eet__qat' 'gripper_atate") </polectte_restuure>>
For each step, you also need to generate "goal" posa, which is either an absolute pose of the robez or relative between the robot ard another oadest. In thin case,
<goal>
0cals
I'id': "1', 'teature'; "plate_ to robott eet_ ly
('id": '2", "fezture": "piate_to_robort rot__ooooo_______oooo _ee___os"
('id': '3", 'fozture': "eink_to_r cat_t__o_oo__ossseeeee______ssse coc___o__peee
('id": '4", "testure"; "aaueet_to cereet____rrott_________ rewott__ett_tttteeeeeoeeeeeoeeee cet_____e__ps""
4/goa1>
C.2 LLM RESPONSE FOR CAN TASK
Tasks: the tray tray trays fixed. Available foatures:
<availabbe booollleoeerccture>>>>>
available, Teaalle___attessss ("ean_pos", reangouat", car_ioo_roott_aaarppooo rcan_Lo_robott__o que Frobot f poa ", "robott_oat_quat"" 'oripper_atate") </available_toctarc
<plan>
steps [rridrs 1, 'deasf's
bray
(fidr: "dase": relose grippez to grasp the can"),
(rid": 3, 'dass": "Meve to tray 2
(ridfss 4, 'desef' "Ogwn
relmase the can
</plan>
<Eeas (billlt Ceastbility
23:
reaching
tray
directly
gripper
3) Falow,
aflwe swaching can Leay we can't diswelly neve to Leay
(2y 4) False,
afcer zeaching the can in tray 1, we can't disectly open the grisper
(2, 35 True,
after elowing the gripper, we own di dirwett
tray 2 direelly open the ippee after noving to tray 2, carts directly open the grippez
clasing geipgee,
4) False,
</ S/aaibblllyy>
Featurres
Coa-ueus
'oan Gan_Loo_ccooi__ee_eeam "sobotO_wwi__pr post, "robot eaef
gripper state!]
<poal>
goals
('idr: "I' "teature": "can_to_renes robott_ eet_pos"!
Prid's *2", "eeturee; "oonuto_ rebnt eee. pos'
15

---

Published as a conference paper at ICLR 2024
1rid": 13", feature": sobbacC_esc__ sac_p
111d": 14" "Iontare": "sobat0__eF_poo""
</goa1>
C.3 LLM RESPONSE FOR LIFTING TASK
Tass: litt plock tron the table. Available teatures: 'availabc_ffaatrre svasiabls_foaturee l'block_pos', 'Block_quat', Fioci_to_rooott_ett__00 saoos_tt_robos_eee____oe "robett_ect_poer" "reaoto_cet_quurt" "gesppot_atate") </available beatilabeeeetttrees....
stepe
[irief: 'desets
I, (fidf: 2, "dase" rclose gripper to geeap the block"
("id': 3, "dese"s "litt the block' 1]
</plan>
stoaribility> foasibility
21: after noving block, dirsetly close
331 Felse, ascer moving block, we nirectly 1ift block
(2, 2)1 True, after closing the qrippet, we can dirsetly lift the bleck
</teasiallityy
ceclocted_fcatrreee
feacures ["s_esk_to__dbooo_oodooo tobok__o_rrbott_eeeee.eo "eobotD_eef_poo', "sobot__ww__guet"
oripper_stata") </poletto__reatucoo>>
<coa1>
"1, "faatrre":
('id': '2" "fosture": "bloci_to Rooos__o__roooB_cc poo 1y
1'd''s 13', "Inture"" "subol (_poa
</goal>
C.4 LLM RESPONSE FOR SQUARE PEG TASK
Task: insert
hardic
Available foatures:
restrrrs resiibbs_ttaturees l'aut_pos', 'nat_quat', torrottt_oo_______
nut robot0 eet quat', "hendle post, "handle_quat', "handle to rebotd eet__os',,
rot_ttoooooo_____eee___ "peg_pos", "pez_quat', [ee_tt_rrbot___e__pooo "pe__to_ratot__aa__eurr, reboto_cet_pes" "robott_oet_quaa" 'grippor_atate" s/availabl_ffattrre>>
<plar> (fid': "dase": "close gripper grasp mat"),
[[fidr: 1, 'dase"s
aut'l,
(rid's 3, 'dece": "Nove to the peg' by
(ridf: 4, 'dase": "Insort tha nuz into the pog'
</plan>
<foaribility> roasibility
2)
3) False,
(Ly 471 Faise,
(2, 3) True,
4) True,
</feasibility>
<selectad_Foaturr
#casures ['"et__terrbbott se_rebott__cc__ppa "aut pout_to__rbae cout_to_rebec__oe__co to Q_ew qqaal', "grigper_atatee] //soluctod_feaatrrep>>
"poa_to_robotb__effuo
<goala>
goals
P'id": "1", feature": "put_to_rolae
1,
P'id": "enture" "put_to S p_ 1,
1'id": "3", "onturef: touebto_ o_stoo_rtttttt__________ p_s___po' 1,
1'id"s 14r
Testurel: "peg_to_roaus robut ew pppwl
</goalos
16

---

Published as a conference paper at ICLR 2024
D ADDITIONAL INFORMATION FOR MARBLE SCOOPING TASK
We prompt LLM to generate a subset of features relevant to predicting task success: X, and Y locations the robot end-effector the robot base frame as well as the wrist camera mask. Due to lack of contact sensors, omit the scooping mode, and prompt the LLM to generate plan: Reaching + Transporting - Dropping (assuming scooping is always successful when transitioning from the reaching mode to the transporting mode. The corresponding feasibility matrix is F In Fig. 10 (top), we plot demonstrations in and Y and use the color the scattered plot to indicate ground truth modes (reaching is red, transporting is green, and dropping is blue). Examples of logged spoon masks along these trajectories are shown at the top. At the bottom, we visualize the learned classifier, which has correctly learned three modes (indicated by three distinct colors) by partitioning the space according to X and Y locations and the masks. Note that the location of the learned blue mode matches the dropping bowl location. Figure shows the classifier successfully learns the threshold function that turns the continuous mask values to the discrete information that differentiates reaching mode from transporting mode given the same X, locations. Note that for the majority of the wrist camera images we collected, the learned classifier infers the correct ground truth mode.
axis robot base frame
Target bowl location
Scooping location
Dropping location
-0.6
-0.4
-0.2
0.0
0.2
axis of robot base frame
Dropping location
Clessifier (
Dropping mode
Drapping mode
Classifier
df
Any other location
other
location
Classifier
Transporting r modee
Learned mode partition Reaching mode
Classifier
dl-)
Learned mode partition in robot base frame when marbles detected on
#1
robot base frame when marbils delected on
Figure 10: Visualizing the state representation of the scooping task demonstrations and the learned mode partitions.
Ground truth mode
Figure 11: Learned classification over wrist camera images compared to ground truth. The red color indicates "No Marble" and green color indicates "Has Marble".
17


Reviews:
[{"review_id": "cH1dUxTUu8", "reviewer": "Reviewer_Rhgd", "summary": "The central idea is to leverage mode families (defines a specific type of motion constraint among a set of objects and can be chained together to represent complex manipulation behaviors) from manipulation research, in combination with LLMs for robust and grounded manipulation.  The authors framework Manipulation Modes from Language Plans (MMLP)  learns manipulation mode families from language plans and counterfactual perturbations. Given a small number of human demonstrations for the task and a small language description of the task, MMLP automatically reconstructs the sequence of mode switches required to achieve the task, and learns classifiers and control policies for each individual mode families. MMLP has four stages: prompting LLMs with a short task description to create a multi-step physical plan; generating a vast amount of counterfactual perturbed trajectories based on a small set of successful demonstrations, and subsequently, learning a classifier for each mode family outlined in the plan; and using the learned classifiers to segment all trajectories and derive mode-specific policies. Evaluation is shown using a simple 2D continuous-space domain and Robosuite (Zhu et al., 2020) and a simulated robot manipulation environment.", "questions": "- The loss terms were a bit difficult to parse at the first go — I’d love a visualization if possible to better understand the loss terms.\n- While it was clear that the LLM generated sub-plans were used to identify number of modes etc., it wasnt clear to me how exactly were additional things obtained from prompting LLM in the first stage of MMLP used as state in the later stages of MMLP. Could the authors explain this more clearly perhaps by adding an example of what “s” looks like for the classifiers?\n- Unclear how the method would work without a simulator to generate success labels for counterfactuals. Is the availability of sim an assumption for MMLP?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- In the sea of prompt-based planning approaches for embodied AI, this work felt creative and novel. MMLP seems to combine best of many worlds e.g. capabilities of LLMs for generating abstract/high-level plans, mode families form manipulation research to ground these plans into motion constraints, and idea of counterfactuals to efficiently learn these mode families and corresponding policies.\n- The use of mode families enables MMLP to be interpretable and robustly generalizable.", "weaknesses": "- Well-thought out analysis but weak evaluation: The evaluation seemed a bit on the weaker side. \n    - Even though I liked the systematic comparison with BC-based baselines and ablations on loss analysis, the two eval domains are both really simple and in simulation. The comparison with BC really helps in validating the use of mode families, however, the evaluation doesn't seem to provide me an understanding of how MMLP would compare with other SOTA manipulation planning approaches such text2motion, vima etc. To that end, I think addition of a more broader set of baselines would strengthen the paper. \n    - Real-world eval or eval with more complex tasks is also highly encouraged in the same vain. Currently I don’t have any intuition or understanding about how well would MMLP work in real world. I’d love to understand how MMLP would deal with partial observability as well. Would the addition of a “search mode” for finding the right object in clutter for instance, make MMLP brittle/ineffective? \n- Opensourcing plans? The authors do not talk about releasing their code, which is important for reproducibility. I encourage the authors to consider and comment on this in their rebuttal.\n- The authors highlight how prompting the LLM to find a suitable state representation for learning the classifier requires skill. It seems like this is the case for generation of perturbations as well. It is not clear to me if same style of perturbations would work for more complex manipulation setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mngFnfdezx", "reviewer": "Reviewer_Mzap", "summary": "The paper presents a method to learn mode-conditioned policy for sequential manipulation tasks. The method proceeds in four stages: 1) prompting LLMs to generate a plan that contains multiple modes as well as keypoints and features for detecting the mode, 2) gather human demonstrations, augment them with noise, and execute in the environment to obtain success/failure labels, and 3) learning a mode classifier, and 4) learning a mode-conditioned policy. The empirical results are shown on a 2D polygon domain, where an agent needs to sequentially traverse through all the modes and reach a configuration in the final mode, and on three tasks in Robosuite.", "questions": "- How are the keypoints and features being grounded in the demonstrations?\n- How is the feasibility matrix generated from LLMs and is it being kept fixed for each task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper text is overall clear\n- The presented idea seems to be novel\n- The finding may be of general interest for the ICLR community", "weaknesses": "- It seems that the paper is conflated with two distinct problems: 1) generating modes and key detection features using LLMs and grounding them for sequential manipulation tasks, and 2) learning a robust mode-conditioned policy from a few demonstrations. The paper sufficiently demonstrates (2) but more evidence needs to be shown for (1), and currently the two contributions seem quite disconnected. For example, although in the method section the paper discusses how LLMs may generate reasonable mode families and key detection features for those modes, in experiment section the experiments use manually-defined modes instead of LLM-generated modes and use manually-labeled features instead of relying on automatic mechanism for grounding. Therefore, it is questionable whether the paper should be made relevant to LLMs at all.\n- The clarity of the figures need to be greatly improved. (Figure 1) It’s unclear what the task even is and what the different colored lines represent, and this has to be inferred by the reader. (Figure 3) What does the coloring mean in the center figure? How can we interpret the feasibility matrix? (Figure 4) Again, the readers may need to guess what the task is. (Figure 5) In what order of the color should the agent follows? What does the difference in coloring between each sub-figure mean here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Urs8L3BGcO", "reviewer": "Reviewer_i3As", "summary": "This paper introduces a framework that autonomously discovers mode families within robot manipulation trajectories with the help of an LLM. Once these mode families are identified, they serve two key purposes: 1) facilitating learning mode-conditioned policies, and 2) enabling the creation of pseudo-attractors that enhance the robustness of the policies against out-of-distribution states. \n\nThe LLM plays a vital role in the framework by: 1) decomposing tasks into a sequence of modes, and 2) generating relevant state features for each mode as inputs to the mode classifiers. To train these mode classifiers, the authors propose to augment expert-demonstrated trajectories with negative ones generated by counterfactual perturbations. Additionally, they design several loss terms aimed at aligning the trajectories with the mode sequences generated by the LLM. Specifically, the loss terms encourage: 1) consistent state transitions within the same mode family, and 2) matched mode transitions between the success trajectories and those generated by the LLM.\n\nIn a 2D toy example and a robot manipulation domain, quantitative evaluations verify the effectiveness of the proposed method in 1) accurately identifying modes, and 2) learning mode-conditioned policies that are robust against perturbations.", "questions": "1. When prompting an LLM to generate keypoints, how to make the correspondence between the generated textual description such as “nut-center” and the continuous positions? Do you need to add the list of available keypoints into the prompt?\n2. Sometimes the demonstrated trajectories do not perfectly follow the LLM-generated mode transitions. Can the learning framework handle these cases?\n3. How much does the LLM-based feature selection contribute to learning the mode classifiers and motion policies? Do you by any chance evaluate the framework using the full state for mode learning?\n4. I wonder in the RoboSuite experiment, how do the proposed method (MMLP-Conditional BC) work in the absence of the pseudo-attractor? It will be great to add these results as well as the authors have a better understanding on how different modules contribute to the robustness against perturbations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is a good practice to utilize an LLM to facilitate the learning of motion-level robot - by learning task structures and identifying important state features. The paper is well-written: the problem is sufficiently motivated and the framework is clearly-explained. In the experiments, the authors properly conduct comparisons and ablation studies to demonstrate the effectness of the proposed framework.", "weaknesses": "My primary concern of the paper pertains to details of the method, specifically the transition feasibility matrix and transition loss. \n\nBased on the definition of transition feasibility matrix $T_{i,j}=max(i-j+1, 0)$, we get $T_{i,j} = 0$ for $i < j$ and $T_{i,j} > 0$ otherwise. But this doesn’t match the illustration in the right subfigure in Figure 3, where there is “-1” in the matrix. More importantly, the usage of the transition feasibility matrix in the transition loss (in Section 3.2) might be problematic. My guess is that $T_{k,l}$ helps penalize mode transitions following the opposite direction for successful trajectories; however, the successful trajectories seem not to contribute to the loss term at all since $\\tau(succ)=0$. I would appreciate more clarifications from the authors on this.\n\nI also think “this work introduces a groundbreaking framework” in the conclusion section is overclaimed. I agree that this is an interesting work, and could be a solid one if the authors properly addess my concern above. However, I would still regard the contribution as incremental, as it only offers an alternative solution to many of the unsupervised trajectory segmentation approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)