Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
Authors: brian ichter
Pierre Sermanet
Corey Lynch
Venue: CoRL2021 Poster (2021)
Keywords: RRT
Task and Motion Planning
Model-based Planning
Tree-search

Abstract: Long-horizon planning in realistic environments requires the ability to reason over sequential tasks in high-dimensional state spaces with complex dynamics. Classical motion planning algorithms, such as rapidly-exploring random trees, are capable of efficiently exploring large state spaces and computing long-horizon, sequential plans. However, these algorithms are generally challenged with complex, stochastic, and high-dimensional state spaces as well as in the presence of small, topologically complex goal regions, which naturally emerge in tasks that interact with the environment. Machine learning offers a promising solution for its ability to learn general policies that can handle complex interactions and high-dimensional observations. However, these policies are generally limited in horizon length. Our approach, Broadly-Exploring, Local-policy Trees (BELT), merges these two approaches to leverage the strengths of both through a task-conditioned, model-based tree search. BELT uses an RRT-inspired tree search to efficiently explore the state space. Locally, the exploration is guided by a task-conditioned, learned policy capable of performing general short-horizon tasks. This task space can be quite general and abstract; its only requirements are to be sampleable and to well-cover the space of useful tasks. This search is aided by a task-conditioned model that temporally extends dynamics propagation to allow long-horizon search and sequential reasoning over tasks. BELT is demonstrated experimentally to be able to plan long-horizon, sequential trajectories with a goal conditioned policy and generate plans that are robust.

Broadly-Exploring Local-Policy Trees for Long-Horizon Task Planning
Brian Ichter Robotics Google i Sehter@googlee com
Pierre Sermanet Robotics Google se commantt@googe com
Corey Lynch Robotics at Google
corey wecllyycchhhhheeeoco com
Abstract: Long-horizon planning in realistic environments requires the ability to reason over sequential tasks in high-dimensional state spaces with complex dy- namics. Classical motion planning algorithms, such as rapidly-exploring random trees, are capable of efficiently exploring large state spaces and computing long- horizon, sequential plans. However, these algorithms are generally challenged with complex, stochastic, and sigh-dimensional state spaces as well as in the presence of small, topologically complex goal regions, which naturally emerge tasks that interact with the environment. Machine learning offers a promising solution for its ability to learn general policies that can handle complex interac- tions and digensiimeni observations. However, these policies are generally limited in horizon length. Our approach, Broadly-Exploring, Local-policy Trees (BELT), merges these two approaches to leverage the strengths of both through task-conditioned, model-based tree search. BELT uses RRT-inspired tree search to efficiently explore the state space. Locally, the exploration is guided by task-conditioned, learned policy capable of performing general short-horizon tasks. This task space can be quite general and abstract; its only requirements are to be sampleable and to well-cover the space of useful tasks. This search is aided by task-conditioned model that temporally extends dynamics propagation to allow long-horizon search and sequential reasoning over tasks. BELT is demon- strated experimentally to be able to plan long-horizon, sequential trajectories with goal conditioned policy and generate plans that are robust.
Keywords: RRT, Task and Motion Planning, Model-based Planning, Tree-search
1
Introduction
For a robot to plan complex long-horizon tasks in realistic environments, it must be capable of reasoning over sequences of subtasks. It must broadly search the space of possible trajectories while considering both what the subtasks are and the order in which they are accomplished. It must handle the complex and high dimensional state space and dynamics of the real world. Consider, for example, robot cleaning kitchen. A planner must find trajectories put utensil away, to wash a pot, to throw away trash, and more. Each of these tasks requires the ability to understand and execute lower-level tasks involved, as well as their order; to put utensil away, the robot must first open drawer, then grasp the utensil, then place it in the drawer, and finally close the drawer.
Classical planning algorithms are capable of efficiently and broadly searching state spaces and plan- ning over long-horizon [1, 2). However, they have difficulty in the presence of complex, stochastic dynamics, high-dimensional systems, so-called narrow passages, and small, complex goal regions (which tasks naturally create). Machine learning has emerged as the state of the art for problems with nigh-dimensional observations (e.g., pixels) and challenging dynamics (e.g., environment in- teraction). Furthermore, it allows learning policies for general and abstract tasks [3, 4, 5, 6, 7]. However, many such approaches are limited to short-horizons [8, 9].
We propose a method which leverages the best of both approaches, a tree search which explores the state space in an RRT-like manner, but where edges maintain a consistent task executed by learned, cundccondtto policy (Fig. 1). In this work, we define task in a general sense, with a requirement only that the task space is sampleable and well-covers the space of useful actions; for example, this includes coal-condi policies, explicit tasks, learned skills, and latent task embeddings. The local, cask-conditioned policy is thus capable of executing complex short-horizon tasks. The skeleton of the RRT search allows rapid exploration of the state space. This exploration is aided by a task-conditioned model, capable of rolling long-horizon trajectories, which can then
Sth Conference Robot Learning (CoRL 2021), London, UK.

---

Pot
MoN DOOREAS
Mous to poodiin
Nove pom3oon
Move to p4c
Mone P41
chiir
Move
(a) Initialize
Sample task rollout
Iterate
Return salution
Figure 1: BELT plans mor-horizon trajectories via a task-condittoned tree and task model.
be planned over and reasoned about in model-based manner to the limit of the known, predictable environment. Broadly-Explord
Local-policy Trees (BELT) is thus able create long-horizon plans with sequential, complex tasks.
1.1 Related Work.
Planning Algorithms are capable of efficiently, and globally, exploring state spaces to plan long- horizon trajectories which avoid collision [1, 2, 10]. Within robotics, as the robot must interact with the world, planning algorithms often consider the task and motion planning setting (contrasting with Al planning). Approaches in this setting generally plan in a hierarchical manner by defining explicit task-primitives, symbolic languages, or transition logic constraints. These methods then plan over them, for example, via trees or optimization- These methods perform well in long-horizon planning, but are limited by their task representations, the dimensionality of the search space, and their ability to execute high-dimensional, complex tasks robustly. BELT approaches such problems through minimal constraints and structure on the task space, instead relying on learned local policies guide the search towards task relevant regions.
methods [11. 12, 13, 14, 15, 16, 17].
Model-Free Learning offers promising direction to execute such tasks, though theses methods have difficulty reasoning over long horizons. To enable long-horizon performance, previous works often enforce hierarchy embed policies within classical structure. Hierarchical reinforcement learning [4, 5, 18] works to temporally extend planning at lower levels, reducing the horizon the high-level policy must reason over [19]. Recent work on coal-conditioned reinforcement learning learns to efficiently achieve arbitrary goal states [20. 3]. This casts the problem for the high-level policy as finding a sequence of goal states [8, 9, 21, 22]. This also allows the embedding of policies in classical motion planning structures [23, 24, 25, 26, 27, 28, 29, 30]. Search on the replay buffer (SoRB) [27] for example builds a searchable graph of the replay buffer given learned distance metric for goal- conditioned policy. In contrast to SoRB and other previous work, BELT uses tree search equipped with condiinddt model to enable searching for plans outside of previous experience and is capable solving non-goal-cen www.--onniiiionee... tasks with its model-based success check.
Model-bbaee Planning allows horizon, sequential planning by moving reasoning to the model level. In this way planning can be performed by rolling out models and evaluating such rollouts, [31, 32]. To extend the performance of models to longer-horizons, models can be trained for multi- step prediction [33] and planned over [34]. Others use models within classical algorithms, such sampling-bast based motion planning [35] or control [36, 37]. In this work, leverage task- conditioned model to temporally extend the model prediction over edges with consistent tasks, re- sulting in more robust long-range performance. We additionally use the model to broadly search the state space via RRT-inspired search.
1.2 Statement of Contributions.
The contribution of this paper an algorithm Broadly-Exploring, Local-policy Trees (BELT) that is capable of planning long-horizon, sequential tasks in complex environments. BELT leverages (1 Sections and 2.2) the state space exploration abilities of planning through RRT-inspired search, (2 Section 2.1) the local guidance of a Icarned condiconditioned policy, capable of execut- ing complex tasks in high-dimensional spaces, and (3 - Section 2.3) the long-horizon propagation and reasoning of learned, temporally extended task-conditioned model. BELT is demonstrated with a learned goal-conditioned policy in a realistic environment to plan effectively, significantly outperforming basclines. Furthermore, these plans are directly executable by the policy.

---

Learning from play (LfP)
Broadly-Exploring Lpolicy Trees (BELT)
Long-horizon plan execution
goil conditione
Sction
current
goal
goal concitioned polkcy
ourrent
gool
Tidy up the room"
unstructured Ielesperated play
BELT with Play Latent Motor Plans [38] learns a general, goal-conditioned policy as well as a model from teleoperation data, Given a long-horizon task, BELT translates this into an RRT-inspired, model- based tree search through the space, where each edge represents single lask sampled from demonstration data. Trajectories are verified successful via trajectory-wise success check on the model. Finally, the full long-horizon plan is executed online by the policy.
Figure 2:
Bropdly-Exploring, Local-policy Trees
We begin with general outline of BELT and then discuss concrete implementations of algorithmic subroutines in Sections 2.1 2.3. Briefly, BELT searches for long-horizon, sequential plans via a model-based, task-conditioned tree search. This search is built on a backhone of an RRT with a local, learned task-conditioned policy. This leverages the efficient giobal-exploration of RRTs, the local directed actions of a learned task-conditioned policy, and the sequential reasoning of a model.
Problem Statement. In this work we refer to a task and task space in a general sense and we consider planning in environments which are predictable and known, with a fully observed state space X. The only requirements of the task space Z is that is sampleable and well-covers the space of useful actions. For example, this can be explicitly given as a set of tasks, implicitly learned via a latent task space, or structured as the state space X' with goal-conditioned policy. To allow for general task definitions, a success criteria Success is used that is considered a "black box". Given a trajectory (Tinit...., )T Success outputs if the trajectory satisfies all necessary conditions. This formula- tion allows for general problem definition that includes goal-conditioned planning. but also allows for transient tasks that cannot be expressed by a goal state. The cost used in this work, referred to interchangeably with distance, is the time of trajectory, though the approach is general to other costs. The environment considered in this work is continuous, known, and static, with the exception of motion as a result of interaction with the robot. The complexity of the environment arises gener- ally from high dimensional states (e.g., many objects) and challenging interactions (e.g.. grasping) and the sequentiality arises from tasks which are bottlenecked by previous task completion.
BELT. The search is outlined in Algorithm I and BELT is shown with progressively more concrete- ness in Figs. 1 - 3. Fig. 1 shows hypothetical form of BELT with an explicit task space, Fig. 2 shows BELT with a goal-conditioned policy that is learned from play [38], and Fig. 3 shows BELT as applied to the experimental environment used herein. A new problem is defined by an initial state, Tinits and success criteria Success. The search initializes a tree T with Vini its root (Line 1) and iteratively adds task-conditioned edges. For each iteration i, a state Texpand E T within the tree is selected, along with task 2; E Z (Lines 3-4). BELT requires a sampleable task space Z and task-conditioned policy, <a, [ 2), which outputs an action a, at time / given the current state Ie and task 2, The goal of the task space is to well cover the state space and guide the search towards useful actions; the task space and policy used herein are discussed in detail in Section 1. The goal of the selection of the tree node Ieeppndd is guide the search towards unexplored regions and to exploit promising paths; this is discussed in detail in Section 2.2. Given the task 2; and state Texpand. the policy is then propagated for T timesteps via a model p(xx+1 a,) to create tra- jectory {Zexpand, .I,, .... TT-1, æT), saving the final state Tnew = ZT (Lines 6-7). The goal of the model is to predict trajectories given task (in the absence of known dynamics); the model used herein is discussed in more detail in Section 2.3. In this way, each edge may be thought of having a temporally extended consistent intent, such as attempting a single task or reaching a goal. This edge then added to the tree and the trajectory from Fint to Texpand to Zaow checked for success with Success (Line 8). This search continues for N iterations, at which time the lowest-cost path marked as successful is returned if it exists (Line 9).
3

---

Initialize (LT)
next state, x
edge (LB)
task- action- sim. model (L6-L7)
rollout (1.37 Fig. check success, and edd edge (L8)
ten
latent plan
sample latent plan
button pushing
drawer
graspin
all other
sweepin
play
plan distribution proposal
state space
X
expand
sample goal from demo data D (L3)
sample 'sooand from tree
return solution (L9)
(a) An individual rollout of Alg. with Play LMP. Sample a goal state (b) Following Alg. itcratively samples and latent "plan" (Line and takes input a goal statc, latent "plan", and the expansion state. This is trajectory is selected the lowest solution, verified hy then input to the model and rolled our (Lines 6-7.) The new trajectory is the binury success check Success (Line 9). After this oflline (iness
and tasks.
expand from (Line 4). Policy and rolls them out to add odges to tree (Line 3-8). The solution
checked
planning. the sequence of subkasks then executed online by the policy. Larger figure in Appendix Section 5.9.
Figure 3: Broadly- Exploly-x,pooring, Local-policy Trees with Play-LMP policy (Section 2.1).
Algorithm Broadly-Exploring, Local-policy Trees
Input: initial state Tink, sample count N, and success check Success I: Initialize T with state Tinit-
for in Sample task =; from space Z.
Section
4: Seleet a state Texyund from T to expand from.
b Section 2.2
5: for in 1 T do
6: Run policy d: ~ π (atl2a, 2.).
b Section 2.1
7: Rollout model Me+1 ~ [
b Section 2.3
Update T with state Jnew = æT, trajectory (Jrespnd: ..., Jnew task 2,, and Success ((Finit Tnew return minimum time successful trajectory failure if none
2.1
Learning a Task Space and Task-Conditionel Policy From Play
In this work we implement BELT with general-purpose goal-reaching policy at the low level, called Learning from Play (LiP) [38]. To learn from "play" we assume access to an unsegmented teleoperated play dataset D of semantically meaningful behaviors provided by users, without a set of predefined tasks in mind. For example, user might open and close doors, rearrange objects, or simply tidy the scene.
To learn control, this long temporal state-action log D {(,,, at) a o is relabeled [3, 20], treating each visited state in the dataset as a "reached goal state", with the preceding states and actions treated
as optimal behavior for reaching that goal. Relabeling yields dataset of Dplay (TT xg ) ) where each example consists of a goal state Tg and a demonstration F = (( ao)....} solving for the goal. These can be fed to a simple maximum likelihood goal conditioned imitation objective,
Dplay
i=0
LLP E(,, apayy
D log Te (ae)=,,x,)) to learn policy г (ae|2,, xgy), parameterized by
The motivation behind this collection to allow play to fully cover the state space using prior human knowledge, then to distill short- hootzooriizon behaviors into single goal-directed policy.
We could in principle use any goal-directed imitation network architecture to implement the policy To (4,, For direct comparison to prior work, use Play Supervised Latent Motor Plans (Play-LMP) [38]. Play-LMP addresses the inherent multimodality in free-form imitation datasets. Concretely it is sequence-to-sequene conditional variational autoencoder (seq2seq CVAB), au-

---

toencoding contextual demonstrations through latent "plan" space. The decoder is a policy trained to reconstruct input actions, conditioned on state 2t, goal Tg. and inferred plan for how get from Ij to Eg. At test time, Play-LMI takes a goal as imput, and infers and follows plan a internally. A t-SNE of this plan space is shown in Fig. 3a. See [38] for details. With Play-LMP as our policy, the task space is the union of the goal 2g and the latent plan 2.
2.2 Biasing the Search
Given a goal-conditioned policy, as Play-LMP is, RRT [1] is used with a few adaptions to broadly search the state space, chose achievable edge tasks, and bias towards low-cost paths (Fig. 4a).
To broadly search the state space, key component of RRT's ability to efficiently explore is what is known as the Voronoi bias. For an RRT, this bias states that at each iteration, the selection of the tree node for expansion is proportional to the volume of its Voronoi region, meaning nodes in larger un- explored regions are more likely be selected and thus RRT's rapidly explore [1]. To select node in RRT, a sample is drawn from the state space and the nearest node within the tree is selected as the expansion node. For BELT, we seek to leverage a Voronoi bias via a temporal distance A/ between states Ft, T E X' as the L2 distance is not good distance for this problem. For example, consider the block in Fig. 3b. it is inside the drawer or on top of the desk, the 1.2 distance is quite small, but to connect these two states requires opening the drawer, grasping the block, placing the block, and closing the drawer- - a significant distance. the temporal distance is not easily quantifiable, we learn to predict it with a temporal distance classifier, Tidel (Zi, I ) with K exponential categories cor- responding to temporal distance intervals, de {[0. 1), 2), [2, 4), [4, 8), (8, 16), -.... [128, 256)) based on [39]. The temporal distance classifier is implemented as a neural network and trained over policy rollouts to predict the distribution over class labels via a cross-entropy classification loss, Lute = - Ei=1 M/ ((i) with و Twe (1, 2zy) ) where y and ú are the true and predicted label distributions respectively. Briefly, we note that we also considered a temporal distance regressor, but found it considerably less accurate.
The selection of an expansion state thus proceeds by first selecting goal state from demonstration dataset from rollouts of the policy Tsample x-- The temporal distance is then predicted between Example and each node in the tree. All nodes below a cutoff temporal distance demott are then added to a set of possible nodes to expand from. Within this set the node with the lowest-cost to come is selected for expansion, Fexpand- This biases the tree search towards nodes that are capable of reaching the goal state and towards nodes with efficient trajectories. It further allows a Voronoi bias by tem- poral distance. The value of deunotf represents a trade-off between exploiting more promising paths (larger values of deunoot) and exploring new ones (smaller values of deunont); this trade-off is studied experimentally in the Appendix Section 5.3. This procedure modifies lines 3 and 4 of Algorithm 1.
We note that a common and effective approach to biasing search in sampling-based -baee motion planning goal bias, i.c., biasing samples towards states within the goal region. As success often requires full sequences of trajectories rather than individual states we do not generally have access to goal states in this problem setting and thus we do not use goal biasing of this form.
2.3 Learning a Task-Conditione Model
Given a policy, the tree search plans via a model. In this work we consider three options for models: a simulator, an action-conditioned model, and a task-conditioned model, each shown in Fig. 4b and 4c. Access to simulator or the exact dynamics is assumed for much of the motion planning community when planning (unless explicitly planning under uncertainty). This acts as an upper bound herein and tests the ability of BELT to explore the state space and plan sequential tasks given a perfect model. It also shows bounds on the robustness of our policy. The action-conditioned model learns to predict the next state Paction (x++1 It, at) given the current state x, and action a₁ For a given trajectory, this is recursively applied for T timesteps. The task-conditiones model learns a prediction of where the robot will be after T timesteps, i.c., (1 [x;,,] 2). This accounts for the closed- loop nature of the policy for a fixed task and temporally extends the prediction to reduce recursive errors. For the Play LMP policy, the task conditioning corresponds to goal state and a latent plan. Each model is represented as a neural network and trained from a dataset of demonstration policy rollouts to minimize the L2 reconstruction loss. For given rollout for task 2, states { (xo, I1, .... 2T ). -TD1 TT || M441l
and actions (0o, "1,,.. }. the action-model seeks to minimize S = -1= with de+1 ~ Paction PactionFs,,00 at), while the task-model seeks to minimize Lyw = fr Frll with ft Ptask (J0,
5

---

initial
Task model
True
Init
sample
TDC
Action model
3
L2
(a) Blasing the state space tree scarch
Task- ssdolssmdeels
Simulator
Task-
Model
Action- Model
(c) Task-modela action model, and simulator our experiment envirenment (see video link). Larger ligure available in Appendix Section 5.9.
TTumpl has been sampled. The blue region shows how the temporal distance between states may differ from the L2 distance, necessitating learning temporal distance classifier. The choice between state and demonstrates the bias towards lower cost paths: though state closer to Jiample, node has a much lower cost to come, and thus state is selected. (4b-4c) shows the two types of models
Figure 4: (4a) shows the bias used by BELT choose the state Tespind
in this work, an action- and a task-conditioned model. The action- conditioned model often exhibits compounding errors as it is recursively applied along the trajectory, while the task-cone lusk-conditioned model avoids this by temporally extending the prediction and conditioning on the fixed task for the edge. This compounding error can be seen in the second task (block lifting) where the action-model becomes unstable and the end effector flails (see video link).
3 Experimental Results
The experiments herein are based on a simulated "playground environment" [38] (Discussed in more detail in the Appendix and shown in Appendix Fig 7) built in Mujoco [40]. The robot is position- controlled 8-DOF robotic arm and gripper. The robot receives as observations at each timestep of the cartesian position and orientation of all objects including its end effector. The agent performs high-free freefrregency, closed-loop control, sending continuous position, rotation, and gripper commands to its arm at 30 Hz. The environment is known and static (unless the direct result of interaction with the robot) and contains desk with a sliding door and drawer that can be opened and closed as well movable rectangular block and 3 buttons that can be pushed turn on different colored lights (for a total of 19 continuous dimensions, detailed in full in Appendix 5.1).
Long-horizon evaluation. We evaluate our model and its baselines on large set of challenging long- horizon manipulation tasks in [41], which require several subtasks to be executed in order. Here, multi-stage tasks are constructed by considering all valid N-stage transitions between a set of 18 core short- horizon manipulation tasks. See [38, 41] for a complete discussion of the 18 tasks, which span different families, e.g. pushing buttons, opening and closing doors/drawers, grasping and lifting block, etc. We evaluate on "Chain-4", which results in 925 unique task chains (Appendix Fig. 7).
Note that prior work [41] assumes a human is in the loop providing each next subtask to the agent in the sequence. In our setup, agent must provide itself subgoals achieve the multi-stage behavior. Other prior long-horizon evaluations, such as [9], describe the long-horizon task as a final goal state to reach. While this valid for many useful manipulation tasks, many are impossible to infer from the final goal state alone. For example, in this environment, when button is longer pressed down, the corresponding light turns off. In this way, the final state for task "1) open drawer. 2) push green button, 3) close drawer" may appear to an agent as "no change to the environment". We consider more general task specification scheme-an indicator function, defined over the full agent trajectory, which returns true if the full desired sequence was executed correctly and false otherwise. We refer to this here as the "success detector", which our high-level agent plans against.

---

3.1 Algorithmic Parameters and Baselines
Baselines. We compare against two baselines that use the same underlying Play-supervised Latent Motor Plan (Play-LMP) [38] policy. We compare to a model-based planner (cross entropy method (CEM) [42]) and a Play-LMP with given goal state. CEM was implemented in the task space to select four sequential tasks. The tasks were then rolled out with the simulator and task-model. The distribution was then refit to successful trajectories. We also compare to a Play-LMM policy that is given single goal-state that satisfies all the subtasks (this single goal state existed for 36% of the chains). This baseline demonstrates that simple approach of using long-horizon goals with policy trained on shor-hhorizons performs poorly and that BELT is able to plan long- long-horizons with local policies. Finally, we also compare to an "oracle" LMP policy, Chain-LMM, which is given the four chain tasks in a row, demonstrating an upper bound of what the BELT plan may achieve given the performance of the underlying policy.
Parameters. The low-level policy is trained with the same play logs collected in [38]; ~7h of teleoperated play relabeled into play, containing 10M short-horizon demonstrations, each 1-2 seconds (see video for examples of this data). The TDC and models were trained on demonstration data for ~ 100k rollouts. Tule was trained on this dataset to an accuracy of 82% and deutoi was set to 64 based on analysis in the next section. The edges were rolled out with randomly drawn timestep of T E {32, 64, 96), as inspired by [10]. BELT used 2500 goal samples drawn from demonstration data of policy rollouts, rather than the play data generated by human operators. We note that we also tried samples from the play dataset, but found many of the goals unreachable by the policy, and overall worse performance. See the Appendix for detail on parameters and tradeoffs.
Metrics. We evaluated the performance of BELT across three metrics intended to capture both its ability to plan and how robust those plans are. "Solution Found" the rate which the planner returns that it has found a solution that satisfics the success criteria, capturing an algorithm's 's ability to explore and plan. "Success Rate" is the rate at which the solution is successfully executed if it found (in this work we rollout successful solutions nine times), capturing how robust the plan is. This robustness may be with respect to a stochastic policy, errors in the learned model, or a stochastic environment. "Feasibility" is the rate at which any of the replays for a given solution was able to solve the problem, capturing whether the plan was feasible (partially isolating the impact of an inconsistent policy). These are discussed in more detail in Appendix Section 5.7.
3.2 Results
We begin by comparing the overall performance of BELT, different models, and the baselines in Table on the chain tasks and Fig. 5 shows several planned trajectories. We study each algorithm's 's performance on the axis of finding solutions (based on the algorithm output) and on robustness. Chain-LMP shows the upper bound of robustness of the policy repeat four successive tasks. CEM with the simulator and LMP with a single-goal state fail to solve most chains. CEM with the model solves more tasks, but the solutions are not robust as the search seems to exploit inaccuracies in the model. BELT with the action model too exploits model inaccuracies the model quickly becomes unstable as it is recursively rolled out over every timestep. Solutions are found for all chains, but the solutions are not robust or feasible for most, possibly due to impossible states input to the success detectors. The task model performs much better due to its temporally extended prediction, though we note multistep predictions hold promise to improve the action-conditiones results [33].
BELT with the task model and the simulator perform well, solving 66% and 82% of the problems respectively, and replay at similar rates and only slightly worse than the oracle. This demonstrates BELT's s ability to both search the space of solutions and find feasible plans. The task model solves 16% fewer problems than directly using the simulator.
Algorithm
Model
Solution Found
Success Rate
Feasible
Chain-IMP (Oracle)
32%
87%
LMP
6%
21%
CEM
Simulator
3%
40%
58%
CEM
Task-Model
11%
7%
15%
BELT
Action-Model
100%
3%
8%
BELT
Task-Model
66%
27 %
56%
BELT
Simulator
82%
26%
68%
Table 1: Success rate and robustness for BELT, CEM, and LMP. Plan robustness is measured by the percent of solutions executed successfully (success rate) and if there was at least one successful in nine attempts (feasible). Note that LMP is only evaluated on chains that can be represented by single goal state (36% of the chains). All approaches evaluated on only this set (which are comparatively easier) is shown in the Appendix Table 3.
7

---

Solution Found
Feasible
BELT
BELT Task-Modee
BELT Simulator
BELT
Chain Contains Task
Simulator
Ratio
Task-Model
Ratio
overall
82%
66%
0.81
68%
56%
0.82
button, push any
73%
54%
0.74
63%
32%
0.50
no buttons
99%
89%
0.90
76%
86 %
1.12
drawer open/close
82%
70%
0.85
69%
64%
0.93
grasp
92%
77%
0.84
73%
70%
0.95
knock
91% %
71%
0.78
58%
55%
0.95
rotate left
93%
0 %
0.00
38%
rotate right
89%
78%
0.88
77%
74%
0.97
shelf, in/out slider, open/close sweeping
77%
56%
0.72
57%
54%
0.95
81%
63%
0.78
67%
58%
0.87
76%
64%
0.84
65%
76%
1.17
Table 2:
Task performance breakdown. The task-model particularly has difficulty planning "rotate left" tasks an angle discontinuity, and the policy trouble executing the task for the same reason. With button presses both approaches plan poorly (BELT with simulator solves 99% without and 73% with) the task- model's plans execute poorly, because the button tasks are transient and thus often not appear goal states.
Figure 5: Plans from BELT, demonstrating its ability to plan long- horg-horizon, sequential trajectories (vidco link).
In Table 2, we analyze which tasks proved most difficult for BELT, and particularly where the task-conditioned model and the simulator performance most diverged. For both the simulator and tosk-model, the button presses proved This is a because the button presses are transient and do not leave a permanent state change, which is particularly challenging for gondicond low-level policy well cover. The task-model has difficulty with chains with the "rotate left" task on the block, while the simulator plans quite well over them but fail to execute many. This is a result of the discontinuity of angles, i.e., if the block starts at 0 radians and is rotated left, the angle jumps 2n radians, which is particularly difficult for the model. This indicates a need for an angular representation such as quaternions or axis-angles,
part of the most difficult chains to plan and execute.
Conclusion and Future Work
Conclusions. This work presented BELT, an algorithm capable of planning long-horizon tasks. BELT leverages the rapid state space exploration of RRT, the robust and general local performance of learned task-conditioned policies, and the long-horizon reasoning of a task-conditioned model. It demonstrated realistic, complex, manipulation environment with a goal-conditioned policy to be capable of planning consistently and robustly.
Real-world Experiments. This method builds upon a scalable approach to learning goal- goal-conditioned low-level policies ([38)), which were demonstrated yield high success mark using onboard sensors only. While these results were obtained simulation, the scalability of the local policy learning makes real-world deployment promising. Our work also relies task success detectors, real-world experiments could make use of learned success classifiers such as [43]. Hence we estimate that there is a reasonable and scalable path to real-world experiments and plan to explore this in future work.
the 18-task bench-
Future Work. In the future wish to show how BELT be extended well beyond the dataset and to other task representations, state representations (c.g., images), and models. We further wish to study how it can be used in replanning, particularly how the tree may be reused, as discussed in Appendix Section 5.8. We also wish study how BELT can be used with constraints on obstacles, states, or model feasibility. Finally, we wish to study methods for generating not just any feasible plan, but robust and efficient plans.
8

---

Acknowledgments
The authors wish to thank Karol Hausman, Vincent Vanhoucke, Alexander Toshev, and Aleksandra Faust for helpful discussions.
References
[1] S. M. Lalale and J. Kuffner Randomized kinodynamic planning. IJRR, 2001.
[2] S. Karaman and Frazzoli. Sampling-bases algorithms for optimal motion planning. IJRR,
2011.
[3] M. Andrychowicz, F, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin,
O. P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in neural information processing systems, 2017.
[4] P.-.. Bacon, J. Harb, and D. Precup. The option-critic architecture. In AAAI, 2017.
[5] R. Fox, S. Krishman, Stoica, and K. Goldberg, Multi-level discovery of deep options. arXiv preprint arXiv: 170300824, 2017.
[6] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 2018.
[7] D. Xu, S. Nair, Zhu, J. Gao, A. Garg, L. Fei-Fei, and Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In ICRA, 2018
[8] O. Nachum, S. S. Gu, H. Lee, and Levine. Data-efficient hierarchical reinforcement learning.
In Advances in Neural Information Processing Systems, 2018.
[9] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving
long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.1199: 2019.
[10] Y. Li, Z. Littlefield, and K. E. Bekris. Sparse methods for efficient asymptotically optimal kinodynamic planning. In Algorithmic founitaiooo of robotics XI. 2015.
[11] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical planning in the now. In Workshops at AAAI, 2010.
[12] L. P. Kaelbling and T. Lozeno-Pérez. Integrated task and motion planning in belief space. IJRR, 2013.
[13] K. Hauser and J.-C. Latombe. Integrating task and prm motion planning: Dealing with many infeasible motion planning queries. In ICAPS09 Workshop on Bridging the Gap between Task and Motion Planning, 2009.
[14] D. S. Nau, T.-C. Au, O. Ilghami. U. Kuter, J. W. Murdock, D. Wu, and F. Yaman. Shop2: An htn planning system. Journal of artificial intelligence research, 2003.
[15] J. Wolfe, B. Marthi, and S. Russell. Combined task and motion planning for mobile manipu- lation. In ICAPS, 2010.
[16] M. Toussaint. Logic-geometre programming: An optimization approach combined task and motion planning. In IJCAI, 2015.
[17] Srivastava, C. Li, M. Lingelbach, R. Martin-Martin, Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. arXiv preprint arXiv:2108.03332. 2021.
[18) A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv: /703.0 161, 2017.
[19] O. Nachum, H. Tang, X. Lu, S. Gu, H. Lee, and S. Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint a9X9v: 10618, 2019.
[20] L. P. Kaelbling, Learning to achieve goals. In IJCAI, 1993.

---

[21] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of tasks via visual subgoal generation. arXiv preprint urXiv:199.008229 2019.
[22] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, 2018.
[23] A. Faust, K. Oslund, O. Ramirez, A. Francis, I.. Tapia, M. Fiser, and Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling- -baeed planning. In ICRA, 2018.
[24] H.-T. Chiang, J. Hsu, M. Fiser, L. Tapia, and A. Faust. Rl-rrt: Kinodynamic motion planning via learning reachability estimators from policies. IEEE RAL, 2019.
[25] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-parametric topological memory for naviga- tion. arXiv preprint arXiv:1803. 00653, 2018.
[26] C. Paxton, V. Raman, G. D. Hager, and M. Kobilarov. Combining neural networks and tree search for task and motion planning in challenging environments. In IROS, 2017.
[27] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search the replay buffer: Bridging planning and reinforcement learning. In Advances Neural Information Processing Systems, 2019.
[28] A. Zhang, S. Sukhbaatar, A. Lerer, A. Szlam, and R. Fergus. Composable planning with attributes. In ICML, 2018.
[29] B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion plan- ning. In ICRA, 2018.
[30] A. H. Qureshi, A. Simeonov, M. J. Bency, and M. C. Yip. Motion planning networks. In ICRA, 2019.
[31] L. Kaiser, M. Babacizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. C.Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for atari. arXiv preprint 190X.v
2019.
[32] J. Schritwiisser, I. Antonoglou, Tr. Hubert, K. Simonyan, I.. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv: 1.08265, 2019.
[33] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.
[34] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0160) 2019.
[35] B. Ichter and M. Pavone. Robot motion planning learned latent spaces. IEEE RAL, 2019.
[36] M. Watter, Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, 2015.
[37] N. Levine. Y. Chow, R. Shu, A. Li. M. Ghavamzadeh, and H. Bui. Prediction, consistency, curvature: Representation learning for locally-linear control. arXiv preprint 1rXiv::909.0155:00 2019.
[38] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In CoRL, 2020.
[39] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, 2018.
[40] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In IROS, 2012.
[41] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020. URL https: /ttp///xivv..////////e/////ee.lt org/ abw/abas/oooo 22005.00... 07648.
10

---

[42] R. Y. Rubinstein and D. P. Kroese. The cross-entrope method: unified approach to com- binatorial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media, 2013.
[43] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering, 2019.


Reviews:
[{"review_id": "bkZUMqfEQH", "reviewer": "Reviewer_FQLy", "summary": "This paper proposes a framework that combines imitation learning and planning approaches to solve long-horizon control tasks. Specifically, a goal-conditioned model is trained via imitation learning to achieve short-horizon behaviors, and an RRT planner uses this goal-conditioned model to find a sequence of temporally abstract waypoints to solve long-horizon tasks. Experiments on a simulated playground environment demonstrate that the proposed approach can solve long-horizon tasks more effectively than a pure goal-conditioned baseline and a variant using an alternative planning method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall goal of this paper — solving long-horizon tasks — is motivated well in the introduction, and the related work section sets the context for relevant work well. The first three figures — showing a progressively more concrete overview of the method — also complement the paper well. These components all help to understand the high-level ideas of the paper. That said, there are a number of areas for improvement regarding the clarity and technical quality of the methods and experiments section:\n1. The definition of the goal-conditioned model is a bit unclear. The paper mentions that the underlying goal-conditioned policy is based on Play-LMP, which consists of both a goal state $g$ and a task vector $z$, yet in this paper the policy is only conditioned on $z$.\n2. The metrics in the main experimental results (table 1) should be defined more clearly. It is possible to infer the distinction between “Solution Found” and “Success Rate” but these details should not be up to interpretation.\n3. How long does it take the planner to find a solution? This is an important consideration if we seek reactive agents that can operate in real-time.\n4. What is the dimensionality of the state space? This can go in the appendix.\n5. Is the underlying Play-LMP policy trained on short-horizon windows of (state, goal) pairs or arbitrary horizons? If it is trained for short-horizon windows only then it makes sense why the LMP baseline performs poorly — in that case the LMP comparison would not be fair because the baseline was not trained to solve long-horizon tasks.\n6. The LMP baseline is only evaluated on tasks that can be expressed as a single goal state, in contrast with the other baselines. It may be that single-goal state tasks entail dealing with a unique set of challenges, making this an unfair comparison.\n7. Clarify the distinction between the play dataset and the demonstration dataset. Why is a separate demonstration dataset needed?\n8. In the cases when the method does not find a solution, what are the underlying reasons behind these cases? Expand on the analysis.\n9. It is entirely possible that the dynamics model predicts infeasible states, eg. when two blocks penetrate with each other. The success function may erroneously predict success for infeasible states, unless it can filter such states to start with. How does the success function deal with such states?\n10. The paper claims that the proposed method can deal with high-dimensional states. Have you considered tasks operating over images? Are there specific modifications you need to make to your framework to deal with images?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is well-motivated and the high-level idea of the paper is easy to understand. There are however a number of low-level details that are unclear in the current iteration of the paper. Refer to the “Issues” section for details. These issues are hopefully easy to address, **at which point I will raise my recommendation.**\n\n---\nUpdate: raised recommendation to weak accept, **but with some hesitation.** Please refer to my followup post for details.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See the list in the “Strengths and Weaknesses” section.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "Mb4zjiO3o7I", "reviewer": "Reviewer_qgUz", "summary": "The authors propose a planning method for using imitation-learned policies to form plans over multiple sequential tasks. They use imitation learning to train a goal-based policy, and train a temporal distance classifier to predict the time to reach the sampled point. The proposed planning algorithm is similar to performing RRT in a space of task manipulations. They sample points from the dataset of demonstrations and attempt to reach those points, expanding the lowest-cost node on the tree that they predict can reach the point in under a certain time. They find that when using a simulator or task-conditioned model, they are frequently able to complete difficult strings of tasks that are difficult for existing methods to solve. They also find that the task-conditioned model is essential for preventing the planner from exploiting inaccuracies in the model and maintaining accuracy over long time horizons. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This method moves towards making it feasible to plan over sequences of tasks, which is a challenging problem. In particular, the complexity of the tasks that they attempt is impressive. I find it interesting as well that this is done without learning different networks for different tasks, instead learning by imitation without well-delineated 'tasks'. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The experimental results for this method are strong, and planning method combined with the goal-conditioned policies make sense as a general problem-solving tool for complicated complicated tasks. The actual planning algorithm bears very strong similarity to RL-RRT, so the interest here is primarily on the technical details needed to achieve these kinds of results. Luckily, the method delivers these technical details and results. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "It's not clear what the Chain-LMP method is or what it's doing. It's also not clear in what sense this method is an oracle, considering its low success rate. Another detail that I had difficulty making sense of was the difference between the success rate and the 'solution found' rate when using the simulator. It was my understanding that the experiments are performed on a simulator, so if the method finds a sequence of actions that solves the task with the simulator, why can't that sequence of actions just be directly applied to deliver a solution? "}}, {"review_id": "Iea-X0QEe5E", "reviewer": "Reviewer_yRzc", "summary": "The paper presents a task and motion planning algorithm that combines the classical motion planning algorithm, RRT, and reinforcement learning to solve long-horizon problems. The authors have shown its superior performance over baselines in simulation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors tackled an interesting problem that lies in the intersection of classical planning and end-to-end policy learning, which is gaining its popularity recently. I find the particular architecture of the proposed method novel. \n- Many figures are included to help better understand the paper.\n- The results seem promising.\n\nWeaknesses:\n- The motivation of the work should be improved in comparison with task and motion planning algorithms in literature. Task and motion planning algorithms can also solve the problems in known, quasi-static environments efficiently so it is not clear why one should use the proposed complex method that involves with RL and user inputs. More emphasis on complex tasks or giving a concrete example that cannot be solved without RL would be helpful.\n- Some parts are not technically sound. Detailed comments are included in Issues below.\n- The paper is not well readable. It includes heavy terminologies, which might be improved by unifying similar ones, and figures require more information, such as notations, explanation on different lines and colors used, and what space it represents. The placement of figures, algorithm, and texts needs to be revised so that texts match with the right figure and algorithm. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper proposes an interesting method in robot learning and planning, I find the paper hard to read and understand. Also, there are technically vague parts in the method that need to be clarified. Overall, the presentation needs an improvement. Especially, it is confusing how RRT interacts with RL setting, which is the most important part in the paper. Thus, I think the paper is on the borderline in its current form.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- As mentioned in Weaknesses, the motivation should be enhanced in comparison with literature on task and motion planners.\n- I think the paper addresses planning only, not interleaved planning and execution. However, the right figure in Fig. 2 and images of intermediate steps in Fig. 3(b) might make readers think that the method proposes an interleaved planning and execution approach. Please clarify this.\n- The motivation of employing RRT is not clear. RRT is used to find a trajectory that does not collide with obstacles in the high-dimensional configuration space. Are all x points in configuration space and does the Fig. 4(a) represent the configuration space? In its current form, it looks like RRT is used to find a trajectory in obstacle-free state space, which can also be done with simpler planners.\n- Fig. 3(a) needs a huge improvement since it represent the main method. It would be helpful to indicate which part is connected to RRT and include used notations. Also, since there are the task space and the state space, please indicate which space it is in all figures.\n- The state is not an image but the state of the robot and object as explained in the experiments. From figures, readers might think that the bird-eye view images are used as a state. It would be good to clarify what state is considered.\n- Along with the above comment, the problem description is not clearly stated before explaining the method. Please clearly state what is known and unknown to the robot, and what are inputs (e.g. what kind of data) to the proposed method.\n- In the below texts in page 3, it would be helpful to explain the overall method using the line number of Algorithm 1 and Figure 3(a). Currently, texts, figure and algorithm are disconnected.\n- The authors used the narrow passage as a fundamental issue in classical planning but it is only a fundamental problem of sampling-based motion planners. There are other algorithms in classical planning, such as trajectory optimization, that does not have the narrow passage problem so this is incorrect.\n- The theoretical results do not seem to add much value but interrupts the flow of the reading. I would suggest to take it out.\n- The considered task in experiments involves with pushing buttons. Is the state that the button being pushed or not represented by a state x? This is a discrete variable so how can RRT handle discrete variables with other continuous variables in search?\n- I understand there is a page limit so that including all experimental results is hard. Although the validation on entire task is critical, readers might also be interested in seeing how each component performs. For example, is x_sample obtained by RRT for a given x_goal semantically meaningful?; is sampled z from task space also semantically meaningful with respect to neighboring points in task space?"}}, {"review_id": "-KQLt8G8h0v", "reviewer": "Reviewer_Da4d", "summary": "The paper aims to solve long horizon state-based robotic tasks by leveraging RRT-style tree search with goal conditioned policy and task-dynamics model. The goal conditioned policy is learned from play-data using variational learning. It is able to solve subtasks with a shorter task horizon within a longer-horizon task. Simultaneously, the paper learns a task dynamics model which takes in latent task parameters to predict the final state after completion of the task. Finally, the paper uses a task dynamics model with RRT-style tree search to find the sequence of subtasks needed to complete the task and then solves each of the subtasks with the goal conditioned policy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe proposed technique is able to nicely solve hard long horizon tasks. For context, it is very hard to solve these tasks with existing deep RL (model free or model based) techniques with end to end training. The paper does a good job of leveraging past data to learn a goal conditioned policy as well as a task dynamics model and then using them to plan for longer horizon tasks. \n\nWeaknesses:\nThe writing could have been a bit more clear with respect to the metrics reported. I don’t completely understand the difference between solution found and feasibility. By solution found, do we mean the agent is able to find some solution using the dynamics model (but that solution could be totally wrong if the dynamics model is bad)? If yes, when the simulator is used as the dynamics model, shouldn't the solution found be very similar to percentage feasibility? Also, if CEM finds a solution in 3% of the cases, how is the success rate 40%? It would be really nice to explain this clearly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the use of goal conditioned policy, task dynamics model and RRT style tree search to solve hard long horizon tasks. That's why I am giving it a weak accept. I will be happy to increase the score if authors are able to more clearly explain the metrics used in reporting the results (as pointed above) ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Mentioned above. See the weakness section.\n\nPOST-REBUTTAL UPDATE: The authors have addressed my concerns. I am keeping my score to a weak accept. It's more of an accept for me (but the only other option available is strong accept)"}}]

Author rebuttal:

Meta-review: ### Final Meta-Review

The authors have carefully responded to reviewers' questions and, after the revision and discussion process, reviewers agree that the updated manuscript has gained in clarity. The initial concerns have been mostly resolved and all reviewers now recommend paper acceptance. I echo their recommendation—nonetheless, the authors should note the reservations expressed by reviewer FQLy and address them to the extent possible in the final version of the paper.


### Original Meta-Review

The paper presents a hierarchical planning scheme combining task-level tree search with learned motion-level control policies. The reviewers agree the the paper is generally well motivated and introduces an interesting new approach. On the other hand, the reviewers express shared concerns regarding technical clarity of terminology and definitions as well as the evaluation metrics used; raised issues regarding the comparison to the state of the art and choice of baselines also need to be addressed.

Three of the four reviewers point out the lack of an explicit upfront discussion of the three metrics used to evaluate the performance of the method (solution found, success rate, and feasibility). As far as I can tell, "feasibility" is only defined in the caption of Table 1, but its meaning (somewhat surprising) has significant implications on gauging the method's effectiveness. I ask that the authors provide clear and formal definitions of these metrics so that the reviewers can fully assess the results.

One reviewer stresses that state-of-the-art task-motion planners are able to solve problems in this class successfully, and the proposed method needs to be more strongly motivated with respect to these existing approaches. The authors must respond to these comments, also explaining why their choice of experimental baselines did not include any methods in this family.

Two of the reviewers also raise questions regarding the state space and its representation (both continuous and discrete) in the proposed algorithm. The authors should respond to these questions, clarifying what representation is being used in each case (search tree, learned policy, paper figures, etc.). Similarly, please address the questions regarding goal and task representation.
Decision: Accept (Poster)