Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
Authors: Bohan Wu
Suraj Nair
Li Fei-Fei
Chelsea Finn
Venue: CoRL2021 Poster (2021)
Keywords: model-based reinforcement learning
long-horizon planning

Abstract: In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks. Reinforcement learning (RL) is a promising approach for acquiring short-horizon skills autonomously. However, the focus of RL algorithms has largely been on the success of those individual skills, more so than learning and grounding a large repertoire of skills that can be sequenced to complete extended multi-stage tasks. The latter demands robustness and persistence, as errors in skills can compound over time, and may require the robot to have a number of primitive skills in its repertoire, rather than just one. To this end, we introduce EMBR, a model-based RL method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. EMBR learns and plans using a learned model, critic, and success classifier, where the success classifier serves both as a reward function for RL and as a grounding mechanism to continuously detect if the robot should retry a skill when unsuccessful or under perturbations. Further, the learned model is task-agnostic and trained using data from all skills, enabling the robot to efficiently learn a number of distinct primitives. These visuomotor primitive skills and their associated pre- and post-conditions can then be directly combined with off-the-shelf symbolic planners to complete long-horizon tasks. On a Franka Emika robot arm, we find that EMBR enables the robot to complete three long-horizon visuomotor tasks at 85% success rate, such as organizing an office desk, a file cabinet, and drawers, which require sequencing up to 12 skills, involve 14 unique learned primitives, and demand generalization to novel objects.

Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
Bohan Wu, Suraj Nair, Li Fei-Fei", Chelsea Finnt Stanford University, Stanford, CA (bohanwu, surajn, feifeili, cbfinn) stanford. edu
Abstract: In this paper, study the problem of learning repertoire of low- level skills from raw images that can be sequenced to complete long-horizon vi- suomotor tasks. Reinforcement learning (RL) is promising approach for ac- quiring short-horizon skills autonomously. However, the focus of RL algorithms has largely been the success of those individual skills, more than learning and grounding a large repertoire of skills that can be sequenced to complete ex- tended multi-stage tasks. The latter demands robustness and persistence, errors in skills can compound over time, and may require the robot to have a number primitive skills in its repertoire, rather than just EMBR, a model-based RI. method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. EMBR learns and plans using a learned model, critic, and success classifier, where the success classifier serves both as a reward function for RL and grounding mechanism to continuously detect if the robot should retry skill when unsuccessful or under perturbations. Further, the learned model aank-agnottic and trained using data from all skills, enabling the robot to efficiently learn number of distinct primitives. These vi- suomotor primitive skills and their associated pre- and post-conditions can then be directly combined with off-the-shelf symbolic planners to complete hong-horizon tasks. On a Franka Emika robot arm, we find that EMBR enables the robot complete three long-horizon visuomotor tasks at 85% success rate, such as orga- nizing a desk, a cabinet, and drawers, which require sequencing up to 12 skills, involve 14 unique learned primitives, and demand generalization to novel objects.
To this end, we introduce
Keywords: model-based reinforcement learning, long-horizon tasks
Introduction
We want robots to robustly complete a variety of long- long-horizon visuomotor manipulation tasks. Such tasks can be completed by composing sequence of low-level primitive skills, otherwise known as "manipulation primitives" [1] or "options" " [2]. For example, the long-horizon task of "organizing office desk" often means performing long sequence of shorter tasks, such as picking objects up, opening or closing drawers, placing objects into drawers, and putting pens into pen-holders. How- ever, completing such long-horizon task may require the robot to learn many low-level visuomotor skills, compose them sequentially, and generalize across novel objects. During task execution, the robot also needs to detect its own failures (e.g. accidentally dropping an object) and correct them (e.g. picking the object up again) before continuing to the remaining parts of the task. We hope to endow robots with such generalization and robustness.
Approaches such as symbolic planning [3], hierarchical RL [4, 5, 6, 7, 8], and hierarchical plan- ning [9] can accomplish compositional task generalization by reasoning in abstract symbolic space. While these approaches have shown promising results on tasks such as simulated tool use [10] and non-vision-based manipulation [9], they rely the ability acquire repertoire of robust, low-level motor skills. These low-level skills can in principle be acquired through reinforcement learning, but number of challenges remain. First, the acquisition of robust and persistent low-level visuomotor manipulation skills is still challenging, and the errors in completing any sub-sequence of
1 Equul advising and contribution.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

250+ Timesteps
Long-Horizon Task: Put object 1 in cabinet, object 2 in small drawer, and object in big drawer
Figure 1: Example-Dris Model-Based RL (EMBR), EMBR enables a Franka Emika robot to complete long-horizon manipulation tasks by sequencing up to 12 unique skills at 85% success with novel objects from raw image observations. Here, the robot organizes desk, which involves a sequence of 10 unique skills.
a long-horizon task can compound over time and lead to overall task failure. Second, acquiring the requisite low-level visuomotor skills requires data-cfficient learning and reward or task specification across a broad set of skills. Finally, sequencing the learned primitive skills to perform a long-horizon task requires persistent grounding of the robot's visual observation within the broader abstract plan.
To address these challenges, we introduce example-driven model- RL (EMBR), an algorithm that learns dynamics model and a set of Q-functions and success classifiers to acquire repertoire of visuomotor skills that are both robust and can be directly grounded into symbols for symbolic planning of long-horizon visuomotor task. More specifically, the success classifiers in EMBR enable both robustness and skill grounding, as they allow the robot to determine the pre- and post- conditions of skill. For example, reward function and post-condition for the skill of opening a drawer can be obtained from learning a binary image classifier of "whether the drawer is open" given the robot's pixel observation. These success classifiers require modest number of example images, making it easy to specify a variety of skills. Finally, by learning a task-agnostic dynamics model, EMBR allows data be shared across all tasks, leading to greater data-efficiency.
The main contribution of this work a framework for learning grounded visuomotor skills that can be directly sequenced by off-the-shelf symbolic planners to complete long-horizon tasks. Exper- iments on a Franka Emika Panda robot arm find that EMBR enables the robot to complete three long-horizon tasks from raw pixels with 85% success, such as organizing a desk, cabinet, and two drawers, which require up to 14 learned unique primitives, sequencing up to 12 skills, and generalization to novel objects. Moreover, our ablation studies confirm that all components of our method including the dynamics model, success classifiers, Q-functions, and stages are essential to performance for tackling long-horizon tasks.
2 Related Work
Reinforcement learning (RL) has a rich history of being used for robotic control problems on tasks
from locomotion [11, 12, 13] to object manipulation [14, 15, 16]. In this work, we focus on learning vision-based robotic manipulation skills, a problem studied by many prior works [17, 18, 19, 20]. Model-free deep RL has been effective on short-horizon skills like object grasping [21, 20, 22], pushing and throwing [19, 23, 24], and multi-task learning [23, 25] from images. Alternatively, model-based approaches [26, 27], which explicitly learn the environment forward dynamics from images have also been employed for multi-task vision- bassd robotic manipulation tasks, either with planning algorithms [28, 29, 30, 31, 32, 33, 34, 35] or for optimizing parametric policy [36]. Unlike these prior works, our algorithm uses a model in conjunction with task-specific success classifiers and Q-functions to learn a wide range of skills suitable for sequencing into long-horizon tasks. In Section 5, we find that both components our approach are essential for robust performance.
Even with powerful reinforcement learning methods, task specification on real robots remains chal- lenging, as engineering rewards on physical systems can be costly and time-consuming [37]. Mo- tivated by this, many works have studied the reward specification, including inverse reinforcement learning [38] with robot demonstrations [39, 40, 41, 42, 43], learning rewards from user preferences [44, 45, 46, 47], and learning rewards from videos of humans [48, 49]. One common approach in visual RL is learn to reach a goal image using a coarse measure of reward like negative & pixel distance [30, 50, 51] or temporal distance [52, 53, 34]. While these techniques have had some success on real robots, such rewards often provide sparse and difficult to optimize reward signal. Alternatively, number of recent works learn classifier from modest number of examples goal states [54, 55, 24, 56, 57, 25] and tries to learn agents which maximize the classifier score. this work, we extend this approach to multiple stages of human- provided goals, which we find enables

---

more effective learning of hard exploration skills, and find that such reward specification can be used to learn 14 unique visuomotor skills on real robot without any demonstrations.
Like many prior works, the goal of this work is to complete challenging long-horizon visuomotor tasks. One class of prior work is hierarchical RI. which aims to learn temporall--extendee primitive skills and a high-level policy over them [4. 5, 6, 7, 8, 58J. Such approaches include jointly learning the primitive skills with the high-level policy through goal generation or an "options" framework I6, 8, 59], learning primitives through intrinsic motivation or other auxiliary objectives [60, 61, 62), from demonstration behavior [63, 64, 65, 66, 67], or from low-level, potentially goal conditioned, reward functions [7. 58, 68, 69]. EMBR is not hierarchical RL algorithm, as it only learns low- level skills provided to the high-level planner and does not learn model over options. Nevertheless, our work is similar to the last group, except that we learn visuomotor primitives on a real robot from rewards derived from human examples. Other works have also explored learning hierarchical policies on top of hand-designed skills [70, 71, 72]. These methods are complementary to EMBR, as skills learned via EMBR can be incorporated into most hierarchical policy learning algorithm.
Alternatively, a number of recent works have studied long horizon visual planning [52, 53, 73, 74, 75, 50, 76, 77, 78], often by combining a structured search approach with learned components like generative models [73, 50, 76] or distance functions [53, 34]. While these approaches have demon- strated impressive results on problems such as visual navigation [52, 53], unlike prior work we learn manipulation tasks on a real robots that consists of up to 12 unique skills and 250+ timesteps.
Lastly, the task and motion planning (TAMP) literature [79, 80) has extensively studied long-horizon robotic tasks. Unlike these works, we do not assume pre-defined state representations or accurate state estimators, making it possible to handle novel objects in cluttered scenes. Like our work, a number of works have also explored combining learned models and skills with hand-designed or symbolic planning [10, 9, 81] and learning ground skills for planning [82]. Unlike these techniques, key insight in this work is that we can jointly learn visuomotor skills and the associated grounding of their pre- and post-conditions, allowing us leverage off-the-shelf symbolic planners to execute long horizon tasks in a closed-loop fashion directly from pixels on a real robot.
3 Preliminaries
Tasks and skills. We consider the problem of learning a repertoire of skills that can be sequenced to complete a long-horizon task M. We model the robot's environment as a controlled Markov process = Po: A. 'T ,, H), with an image observation space s E S, an initial state distribution po. an action space a E A, a dynamics model T S x A S - IR, a discount factor 7 E [0. 1), and a finite horizon Ht. We notice that many long-horizon tasks are composed of sequence of lover-level skills. For example, to organize an office desk, the robot needs to acquire a number as grasping object, opening drawer, and placing an object in drawer. Let K denote the total number of unique skills the robot has acquired, and k E [1, K] denote the 4tth skill in the robot's skill repertoire. Depending
skills such
the environment state, the order in which the skills should be executed to successfully complete a Long-horizon task varies. For example, to organize a desk, the robot may not need to use the skill of closing drawer if the drawer is already closed. Here, each skill is an MDP M = (E, R*), where the robot's environment E is shared across all tasks and skills, and R² S x A A R is the reward function for the tt skill (e.g. the skill of opening a drawer).
Problem Definition. Formally, EMBR takes as input N human-provided example images y $1...., Sy for each skill we intend robot to learn. Our goal (i.e. EMBR's 's output) is to learn the symbolic grounding of the pre- and post-conditions Spres gpest : S + {0, 1}. and policy x² : S > I[(A) for each skill k, where II(.) defines probability distribution over a set. Each policy has a Q-function, Q+ S x A- + IR for taking action a from and following n² onward.
Variational Autoencoders (VAEs). VAEs compress high-dimensional observations s such as im- ages into an embedding 2. VAES can be optimized by maximizing the evidence lower bound (ELBO): maxp. Lvae (8), where (v) = Eyysle) [log (s =)) - DKL (u(= (b) 1| p(2)). where p denotes the generative model, and q denotes the variational distribution.
(u(((((
Symbolic planning.
symbolic planner performs task planning in the symbolic domain. Con- cretely, given a set of candidate actions, each action's 's associated pre- and post-conditions, the current condition and goal condition g. a symbolic planner outputs the sequence of actions that allows the robot to reach from h to g. The pre-condition of an action is a set of predicates that defines
3

---

Image Reconstruction
Future Image
Human
Patik
Examples
Current Image
daec
fenc
fenc
fR
fo
l2
Robot Observation (+)
Q-learning
fenc
fy
RC-L
Ce+l
Trained on across skills
Trained on séill-specifcc data
Figure 2: Example-Driven Model-based RL (EMBR). During training. EMBR learns VAF fixe and dynamics model fi in the VAE latent space (left). For each skill, EMBR uses human provided example images to train a success-classfffr--ass time actions are selected by model predictive control over the latent dynamics model using the Q-value as the planning cost. The reward function further successful state, such as inserting the marker as shown the figure.
reward function fr, and associated Q function fo (right). At test
success classifier that encourages the robot to reach
the condition that needs be satisfied in order for the action to be executable. The post-condition of a skill is a set of predicates that defines the effect of executing an action. For example, the pre- condition and post-condition of "Open left drawer" action is "Drawer is closed" and "Drawer open" respectively. In our context, the candidate actions are the primitive skills learned by EMBR.
4 Example-Driven Model-Based RL (EMBR)
To learn repertoire of skills that can be directly sequenced by off-the-shelf symbolic planner to complete long-horizon visuomotor tasks, we propose Example-driven Model-Basee RI., or EMBR. Below first describe how EMBR learns cach visuomotor skill without any demonstration skills or trajectories during training, and then elaborate on how these learned skills can be directly sequenced to complete a long-ho long-horizon task at test time.
4.1 Learning a Repertoire of Visuomotor Skills
During training, EMBR learns repertoire of low-level persistent visuomotor skills, such as opening a drawer or inserting a marker, using a set of example success images provided by a person for each task. The model learned by EMBR, denoted f. is composed of four components: global variational autoencoder (VAE) fvve= global latent dynamics model fT. stage-specific binary reward functions f and stage-specific Q-functions f5- The VAE and latent dynamics model jointly learn a www-dimessional representation of the image observations to address the challenge of efficiently learning many primitive skills; both are trained data collected across all skills. The skill- -selifiee binary classifiers allow us to learn a reward function for each skill from human-provin provided images. We the latent dynamics model to perform model predictive control with the skill-specific Q-function as the terminal planning objective. The skill-specific Q-functions and reward functions are trained on skill-specific data. Next, we elaborate on each of these components.
Learning a low-dimensional latent space with a VAE and latent dynamics model. Performing a Long-horizon task using large number of visuomotor skills demands the algorithm to be sample- efficient and fast at execution time. To address both of these demands, we learn a single VAE fvae {fenee face } to construct latent space Z. This VAE consists of an encoder fene S II(Z) and a decoder face : Z + II(S). We train fvae using data collected across all skills via Eq. ??, where p(s 2) = (Se 21), p(z) ~ (0, and q(2 s) ( | st). By learning from all skills, the VAE can improve data efficiency. In practice, we use the greedy hierarchical VAE [51] as frae with the current image observation as a context frame. Based on this latent space, EMBR learns
latent dynamics model JT jointly with fvne with data collected across all skills via 12 loss:
Lra (st41-1)))) + (F 2441) 2
min ESEEAL, freIT
(1)
where 2, ~ fene(: st), L is the rollout length of fr and D is the dataset collected by the robot.
Obtaining rewards by learning image classifiers. Obtaining a reward function in RL typically requires some forms of human supervision such as reward engineering or demonstrations. EMBR, binary image classifiers serve not only as reward functions for learning primitive visuomotor skills,

---

but also as success detectors as well as grounding of raw pixels for long-horizon symbolic planning. Concretely, we learn a reward function f for each skill _ by training binary image classifiers to distinguish human-provided example images from robot-collected non-example images using the binary cross-entropy objective [55, 56]. The reward learning objective for skill l: is:
max Eg Dt wtw.:/ttttttttttttttett (oo log (Fi(et)) + WD.tt [log (1 (1- (((())))
(2)
Here, D+ is the dataset of images labeled as positive by person (typically 100-200 images), st are the images sampled from D* and s are sampled from all images in D, the dataset collected by the robot, and are considered negative by default. In our experiments, these positive examples can be collected in 15-30 minutes per visuomotor skill, and 345 minutes in total for all 14 skills. No demonstration trajectories are collected.
At test time, the learned image classifiers are re- re-purposed as post-condition grounding of each skill for long-horizon task execution. Some of the pre-conditions the task planner requires can be de- rived from existing post-conditions. For example, for a drawer-opening skill, the pre-condition of "drawer is not open" can be derived from the post-condition of "drawer is open" by negating this post-condition. For the remaining pre-conditions that cannot be derived from any existing post- conditions, we train additional image classifiers using Eq. 2.
Learning Q-functions for model-based control. While one can directly use the latent dynamics model for model-based planning, the difficulty of learning accurate dynamics model for the full horizon of a primitive skill increases with the length of the skill itself. To alleviate this challenge and further accelerate skill learning, learn Q-function f in the latent space. Concretely, perform Q-learning via the following objective for skill k:
min Bs, 88
S (z),
THEE (zet1) 7 (ze+1 max ((41)) 1111
(3)
where (=) = (z) > 0.5). ~ Svae(- st). Ze+1 ~ S (- 1241), and the Q-value target is computed by maximizing over randomly and uniformly sampled actions 1:m1
Algorithm EMBR at Training Time:
Learn Repertoire of Visuomotor Skills
1: Input: Example images per skill; Output: K primitive visuomotor skills
2: 'D < to empty datasct, f + random weights, J + gradient update steps, L + rollout horizon
3: for Skill & [1, K] do
4: while Not reaching target success rate for skill k as determined by success classifier f do
5: Collect trajectory d = (*1.,, an.rr } using actions selected by Eq. 4 and append to dataset D
6: for 1: J do
Sample minibatch trajectories of timesteps L H: D {sti o | ~ D Update fve. fr jointly via Eq. 1. f% via Eq. 2, and f8 via Eq. all with D
8:
Algorithm 2 EMBR at Test Time: Long- Horizon Task Execution via Skill Composition
1: Input: trained model f. symbolic goal condition
2: Compute current symbolic state h using fx; + new image observation
3: while h + g do
4: Compute the first skill perform le E [] using symbolic planner given goal g and current state h
5: while f (ffo < 0.5 de
6: Execute robot action via Eq. 4 given
7: s f current image observation
8:
Compute current symbolic condition h from fR
At both training and test time, we perform model-based planning in the latent space by first rolling out fT for L timesteps into the future and then evaluating the predicted future latent space observa- tion timestep i + L using the learned Q-function, process that is illustrated in Fig. 2:
te (fr (ze, auc+L-1) ( ...t)))))++)))))))))))))))))). where 2 frao ( (s)
0
arg max
(4)
EA
5

---

(a) Robot Environment
(b) Train Objects
(c) Validation Objects
(d) Test Objects
Figure 3: Real-Robot Experimental Sctup. In Fig. (a), Franka Emika Panda robot is equipped with five
RGB cameras (red circles) capturing 64 RGB images. Quantitative justification for having five cameras Appendix F. The robot interacts with desk that has two drawers and cabinet, as identified by three pink handles in Fig. (a). Fig. (b), (c) and (d) visualize train, validation, and test objects used in all experiments.
Here, the outer and inner maximization operations are taken over mg and my samples respectively.
Integrating these four components, EMBR is trained simultaneously across all components from scratch (i.e. without pre-training) for learning a repertoire of visuomotor primitive skills (Alg. I). Concretely, the robot begins with empty dataset D and can learn each skill in almost any order (Appendix C.2). During skill learning, the robot collects a new trajectory using actions selected from Eq. 4 and updates all model components using D. This process repeats until the current skill reaches a desirable success rate as computed by the success classifiers, before the robot moves on to learn the next skill. As the robot acquires more skills, the dataset 'D accumulates and improves the generalization of the visual dynamics model, which in-turn accelerates future skill learning. Wall- clock training time per skill and hyperparameters are detailed Appendix G and H respectively.
4.2 Composing Learned Visuomotor Primitive Skills for Long-Horizen Task Execution
At test time, EMBR provides the learned primitive skills and the symbolic grounding of each skill's post-conditions (in the form of success classifiers) to high-level task planner (standard PDDL STRIPS symbolic planner [83] in our case) to perform long-horizon task. As elaborated in Alg. 2, the human first specifies long-horizon task using symbolic goal condition g. Next, EMBR computes the current symbolic condition of the environment from raw pixel observations by passing the current image observations each of the success classifiers. Using both the symbolic goal condition and the current symbolic condition, the symbolic planner computes the sequence of skills to perform and executes the first skill of this sequence using actions selected from Eq. 4. This procedure repeats until the current condition matches the goal condition, after which task execution terminates. Note that the arm position of the robot is reset after executing each skill. The predicate definitions, the pre- and post-conditions and the full details of the symbolic planner used in Section 5 are in Appendix C.3.
Experiments
Our experiments aim to answer four key questions: 1) how does EMBR compare to prior RL al- gorithms on both performing primitive skills and when sequencing those skills to complete long- horizon tasks? 2) how necessary are stage-specific reward functions (as elaborated Appendix ??) compared to a single skill-specific reward function? 3) how important is replanning using the classifiers for the robustness of the algorithm? and 4) how important is each of the five cam- era observations to the performance of the primitive skills? To answer these questions, we con- duct experiments across 14 primitive skills and three long-horizon tasks, using the experimental setup in Fig. 3, in which a Franka Emika Panda robot has access to five cameras, each capturing 64x64 RGB images. One of the five cameras is mounted on the wrist. To test object gener- alization, all objects used in all evaluations are novel and unseen during training, except for the desk, drawers, cabinet, and the marker holder. Fig. 3 visualizes the train, validation, and test objects used in all experiments. See Appendix C for experimental details. Project website at https://sites: google com/view/embr- site.
Comparisons. We compare EMBR to three prior methods: (1) EMBR w/o fr, which corresponds to EMBR without the dynamics model, i.e. using only the Q-function and the VAE for action selection; (2) EMBR w/o fvee- fr (Qt-Opt) [20], which is our re-implemented version of Qt-Opt [20] with

---

Table I: Successful trials (out of 20) and success rates of visuomotor primitive skills, Here, EMBR outperforms the three prior methods by 10-20%, 15-25%, and 50-65% respectively on high-precision tasks.
Prior Method
Prior Methed 2:
Prior
Ablation 1:
EMBR (Ours)
&
Primitiiee Skill
Precision
EMBR
EMBR w/o fax.
Method 3:
EMBR w/ Skill-
w/n fr
(Qr-Opt [20]
BEE [56]
specific Rewards
1 Insert marker into marker holder
High
20 (100%)
16 (80%)
15 (75%)
(40%)
20 (100%)
2
Grasp object from left drawer (single)
High
18 (90%)
15 (75%)
14 (70%)
(40%)
15 (75%)
Grasp object from left drawer (clutter)
High
18 (90%)
14 (70%)
15 (75%)
(35%)
14 (70%)
3
Grasp object from right drawer (single)
High
18 (90%)
16 (80%)
14 (70%)
5 (25%)
16 (80%)
Grasp object from right drawer (clutter)
High
(90%)
15 (75%)
15 (75%)
(25%)
16 (80%)
4
Grasp object from desk (single)
High
18 (90%)
16 (80%)
15 (75%)
(255%)
12 (60%)
Grasp object from desk (clutter)
High
(90%)
16 (80%)
15 (75%)
(305)
12 (60%)
5
Open right drawer
Medium
20 (100%)
18 (90%)
16 (80%)
14 (70%)
17 (85%))
6
Open left drawer
Medium
20 (100%)
19 (95%)
16 (80%)
(40%)
16 (80%)
7
Open cabinct
Medium
20 (100%)
18 (905))
17 (85%)
(60%)
18 (90%)
8
Place object on desk
Low
20 (100%)
20 (100%)
20 (100%)
20 (100%)
20 (100%)
Place object cahinet
(100%)
(100%)
(100%)
20 (100%)
(100%)
10
Place object into left drawer
Low
20 (100%)
20 (100%)
20 (100%)
20 (100%)
20 (100%)
11
Place objecc into right drawer
Low
20 (100%)
(100%)
20 (100%)
20 (100%)
20 (100%)
12
Close right drawer
Low
20 (100%)
20 (100%)
20 (100%)
20 (100%)
20 (100%)
13
Close left drawer
Low
20 (100%)
20 (100%)
20 (100%)
20 (100%)
20 (100%)
14
Close cabinet door
(100%)
(100%)
20 (100%)
20 (100%)
20 (100%)
Table 2: Successful trials (out of 20) and success rates of long-horizon tasks. K" and refer to the minimum number of skills that need to be sequenced and the minimum number of unique skills required. respectively. Here, EMBR outperforms the three prior methods by 15-25%, 20-30%, and 85% respectively.
Prior Method
Prior Method 2:
Prior
Ablation 2:
EMBR (Ours)
Long horizon Task
K
I: EMBR w/o fT
EMBR w/o fue. fT (Qt-Opt) [20]
Method 3:
EMBR
BEE [56] w/o Replanning
Organize Desk and Cabinet
8
17 (85%)
14 (70%)
11 (55%)
(0%)
11 (55%)
2
Organize Markers
9
5
17 (85%)
13 (65%)
11 (55%)
(0%)
10 (50%)
3
Rearrange Objects
12
12
17 (85%)
15 (75%)
13 (65%)
(0%) 11 (55%)
VICE-like success classifier [55] that does not learn latent space and uses only the Q-function for action selection; and (3) BEE [56], which uses the success classifier instead of the Q-function for model predictive control. All comparisons are representative of prior methods for -baedd robotic RL. For all experiments, the data used to train each method is the same dataset collected by EMBR using Alg. 1, which contains 300-400 trajectories collected in 16 hours per skill on average. This provides a more direct comparison and in effect makes the prior methods stronger since the challenge of online
explorre exploration is addressed by EMBR during data collection.
5.1 Short-horizon Primitive Skill Performance
To answer the first question, we compare all methods across 14 visuomotor skills. In Table 1, we ob- serve that "EMBR (Ours)" can reliably complete all 14 skills with at least 90% success. Compared to "EMBR w/o f-" and "EMBR w/o free: fo (Qt-Opt)", "EMBR (Ours)" improves success rates by 10-20% and 15-25% across high-precision skills, 5-10% and 15-20% for medium- prrecsi-reeiiiion skills and 0% for low- wwwprrcission skills. We hypothesize that EMBR's 's improvement over "EMBR w/o fr" is due to its ability to alleviate overestimation of Q-values in the presence of limited training trajec- tories (300-400 per skill), and the addition success rate improvement of EMBR over "EMBR w/o ST= is due to less overfitting of the success classifiers and Q-functions in the VAE latent space. Finally, EMBR outperforms BEE [56] by 50-65% on high-precision tasks because BEE [56] cannot capture action consequences beyond its planning horizon and the model is not always accurate for the full horizon of skill.
5.2 Long-horizon Task Performance
Next, we compare EMBR to the three prior methods when sequencing the learned visuomotor skills with symbolic planning to complete three challenging long-horizon tasks (Fig. 4):
1. Organize Desk and Cabinet (8 Skills) - putting three novel objects cluttered on the desk into the cabinet.
Organize Markers (9 Skills) picking up three markers and inserting them into a holder."
3. Rearrange Objects (12 Skills) picking up three objects and placing them to the desk, cabinet or drawers.
210 this task, we also equip all methods with a scripted skill uses the edge the desk as a supporting point to re-orient marker.
7

---

Organize Desk and Cabinet
Organize Markers
Rearrange Objects
Figure 4: Long-horizon Task Experiments. EMBR enables a Franka Emika robot to complete long-horizon manipulation tasks with novel objects from raw image observations. Here. the robot performs three long- horizon tasks with 85% success, which involve sequence of up 12 unique primitive visuomotor skills.
Detailed task descriptions are in Appendix C.7. In Table 2, we find that "EMBR (Ours)" completes these long- hongzoorizon visuomotor tasks with 85% success, including the most challenging tasks that require up to 12 unique skills. Compared to "EMBR w/o JT" and "EMBR w/o f fr (Qt-Opt)", success rates respectively because EMBR
EMBR leads to 10-20% and 20-30% improvement achieves higher success rate on individual primitive skills that require higher precision, such as inserting marker and grasping object from desk or drawers. Finally, BEE [56], is unable to perform any long-horizon task due to low performance skills with medium high precision. This reflects the importance of learning robust, high-success-rate primitive skills using EMBR.
5.3 Ablations
Next, we conduct six ablations to examine the importance of various components of EMBR. Ab- lation 3, which compares EMBR with vs. without any one camera observation, is in Appendix F. Comparisons to the original BEE and training without stages are in Appendix and E.
Stage-specific vs. skill-specific reward function. While EMBR accelerates skill acquisition by learning reward function for each stage of skill, one can in principle learn just one reward function for each skill. In this ablation, we answer Question 2 by comparing EMBR to an ablated EMBR that learns each skill with a single skill-specific reward function. This does not affect data collection and exploration, which stage-specific rewards can also help. In Table 1, we find that using stage-specific reward functions outperforms using single skill-specific reward function by 10-30% for higher precision skills such as grasping an object in various locations, indicating the importance of using stage-specific reward functions in long-horizon or higher precision skills.
Ablation 1:
Task performance with vs. without classifier based replanning, While EMBR improves the robustness of long-horizon task execution by using success classifiers during planning. one can in principle perform model-based planning without these classifiers. In this ablation, we answer Question by comparing EMBR an ablation that executes each skill of task sequentially without using the success classifiers to determine when the skill is successful. In Table 2, we find that long-horizon task execution with classifiers improves performance by 30-35% success rate, due to the robot's ss ability to detect success in closed-loop manner until successful. These ablations and Table 2 suggest that all components of EMBR are critical completing long-horize tasks.
Ablation 2:
Conclusion and Future Work
In this work, we propose EMBR: an example-driven model-based reinforcement learning algorithm for learning visuomotor skills that can be directly sequenced to perform long-horizon tasks from raw pixels. Across three long-horizon visuomotor tasks which require up to 14 unique primitives and sequeneing up to 12 skills, we find that EMBR consistently achieves 85% success rates, out- performs prior methods, and generalizes to novel objects. Nonetheless, EMBR also has a number of important limitations, including the limited scope of tasks, the amount of supervision required to complete these complex long-horizon tasks, and the limited degree of generalization especially across different scenes. More specifically, the visuomotor skill definitions object-specific, such as "open left drawer", which will scale poorly in the presence of many objects. We discuss limitations in more detail in Appendix A, each of which presents interesting directions for future work.
EMBR are currently
8

---

Acknowledgments
The authors would like thank members of the IRIS and RAIL labs for providing valuable feed- back. Suraj Nair is funded in part by an NSF GREP. This work was also supported part by ONR grants N00014-20-1-2675 and N00014-21-1-2685.
References
[1] T. Kröger, Finkemeyer, and F. M. Wahl. Manipulation primitives-a universal interface between sensor-based motion control and robot programming. In Robotic systems for handling and assembly, pages 293-313. Springer, 2010.
[2] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
[3] Smolensky, Connectionist ai, symbolic ai, and the brain. Artificial Intelligncee Review, (2):95-109) 1987.
[4] R. Sutton, D. Precup, and S. Singh. Between mdps and semi mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 112:811-211 1999.
[5] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning, Dis- crete Event Dynamic Systems, 311-2))41-77. Jan. 2003. ISSN 0924-6703.
[6] Bacon, J. Harb, and D. Precup. The option-critt architecture. CoRR, abs/ 600.055400, 2016.
[7] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. CoRR, abs/1805.088 008.8055.099 2018.
[8] A. Levy, R. Platt, and K. Saenko. Hierarchical reinforcement learning with hindsight. In International Conference Learning Representations, 2019.
[9] G. Konidaris, L.. Kaelbling, and Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. J. Artif. Intell. Res., 61:215-289, 2018.
[10] M. Toussaint, K. R. Allen, K. Smith, and J. Tenenbaum. Differentiable physies and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
[11] H. Benbrahim and J. A. Franklin. Biped dynamic walking using reinforcement learn- ing. Robotics and Autonomous Systems, 2233)228-330, 1997. ISSN 0921-8890. doi:
URI. htttps://wwe sci enceerieer.... Robot Learning: The New Wave.
[12] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, 3:2619-2623 Vol.3, 2004.
[13] R. Tedrake, T. Zhang, and H. Seung. Stochastic policy gradient reinforcement learning simple 3d biped. 2004 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), 2-2849-2854 vol.3, 2004.
[14] V. Gullapalli. Skillful control under uncertainty via direct reinforcement learning. Robotics and Autonomous Systems, 15(4):237-246, 1995. ISSN 0921 1-8900. doi.https:/doi.orr//...r 1011/ 0921-88900955000006-2 URL https: //ww. sci encedirect. wtt.e//eeettttc//c/eee//eattttti com//ccience/articcle// pi1/00211889995000068 Reinforcement Learning and Robotics.
[15] J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682-697, 2008. ISSN 0893-6080. doi:httss//doi.ort www.t/iiiogg/111// ceunett.000.022.003 URL https: http///wwws..e $0893608008000701. Robotics and Neuroscience.
org/10..
[16] M. Deisenroth, C. Rasmussen, and D. Fox. Learning control low-cost manipulator using data-efficient reinforcement learning. Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2011. doi: 1. VIL.088.

---

[17] S. Lange, M. Riedmiller, and A. Voigtländer. Autonomous reinforcement learning on raw visual imput data in real world application. In The 2012 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2012. 10 109/UCCNN.2022.62528222
[18] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor poli- cies. Journal of Machine Learning Research, 17(39):1-40, 2016. URL http://jmmr.og papers/v17 10tp1///////21111 522. html.
[19] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2351-2358, 2017.
[20] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[21] L. Pinto and A. Gupta. Supersizing self supervuuevi::::: Learning grasp from 50k tries and 700 robot hours. 2016 IEEE International Conference on Robolics and Automation (ICRA), pages 3406-3413, 2016.
[22] A. Zeng, S. Song, Welker, J. Lee, A. Rodriguez, and Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. 2018.
[23] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31:9191-9200, 2018.
[24] A. Singh, L. Yang, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgauu, Germany, June 2019.
[25] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. ArXiv, abs/2104.08212, 2021.
[26] D. Hafner, T. Lillicrap, Fischer, Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Pro- ceedings of Machine Learning Research, pages 2555-2566 PMLR, 09-15 Jun 2019. URL http: httt:://proceediggs. . mlr. www..mll....ss.ee.. www.ppprssss///ee httpe//haaaeerr9r html.
[27] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representattions, 2019.
[28] C. Finn, Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pages 64-72, 2016.
[29] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[30] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.
[31] Ebert, S. Dasari, A. X. Lee, S. Levine, and C. Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. In Conference on Robot Learning (CoRL), 2018.
[32] Y-C. Lin, M. Bauzá, and P. Isola. Experience-emheddded visual foresight. hts/1911.0507// 2019.
ArXiv,
[33] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. ArXiv, abx/2000..00093, 2020.
10

---

[34] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine. hodel-based vi- sual planning with self-super functional distances. International Conference on Learning Representations (ICLR), 2021.
[35] M. Bhardwaj. A. Handa, D. Fox, and B. Boots. Information theoretic model predictive q- learning. In Learning for Dynamics and Control, pages
PMLR, 2020.
[36] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. arXiv preprint arXiv:2012.11547, 2020.
[37] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, Kumar, and Levine The ingredients of real-world robotic reinforcement learning. ArXiv, abs/2004. 12570,, 2020.
[38] Abbeel and A. Y. Ng. In Proceedings of the Twenty-First International Conference on Machine Learning, ICML '04, page 1, 2004.
[39] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 729-736, 2006.
[40] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proc. AAAI, pages 1433-1438, 2008.
[41] M. Wulfmeier, P. Ondruska, and I. Posner. Maximum entropy deep inverse reinforcement learning, 2016.
[42] C. Finn, Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[43] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference Learning Representations, 2018.
[44] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), 7 2017. doi: 10. 255007/SSS.2017XXII
[45] D. S. Brown and S. Nickum. Deep bayesian reward learning from preferences, 2019.
[46] M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions by integrat- ing human demonstrations and preferences. In Proceedings of Robotics: Science and Systems (RSS), 6 2019. doi:100.1560//ss..011.vvv
[47] D. S. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe imitation learning via fast bayesian reward inference from preferences, 2020.
[48] L. Shao, T. Migimatsu, Q Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. In Proceedings of Roboties: Science and Systems (RSS), 2020.
[49] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos, 2021.
[50] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. ICLR, 2020.
[51] B. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei*, and C. Finn*. Greedy hierarchical variational autoencoders for large-scale video prediction. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[52] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-parametric topological memory for naviga- tion, 2018.
[53] B. Eysenbach, R. Salakhutdinov, and Levine. Search on the replay buffer: Bridging planning and reinforcement learning, 2019.

---

[54] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning, In CoRL, 2018.
[55] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events:
A general framework for data-driven reward definition. In Advances in Neural Information Processing Systems, 2018.
[56] A. S. Chen, H. Nam, S. Nair, and C. Finn. Batch exploration with examples for scalable robotic reinforcement learning. arXiv preprint arXiv:2010. /1917, 2020.
[57] B. Eysenbach, S. Levine, and R. Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification, 2021.
[58] Y. Jiang, S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NewrIPS, 2019.
[59] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining, In International Conference on Learning Representations, 2019.
[60] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.00077, 2018.
[61] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills, 2020.
[62] O. Biza, Wang. R. W. Platt, J.-W. van de Meent, and I. L. S. Wong. Action priors for large action spaces in roboties. ArXiv, abs/2101.04178, 2021.
[63] S. Krishnan, Fox, I. Stoica, and Goldberg. DDCO: discovery of deep continuous options forrobot learning from demonstrations. CoRR, abs/1710.0544) 2017.
[64] R. Fox, R. Shin, S. Krishnan, K. Goldherg, D. Song, and I. Stoica. Parametrized hierarchical procedures for neural programming. In International Conference on Learning Representations, 2018.
[65] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. ArXiv, abs/18 1810. 1043, 2018
[66] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In CoRL, 2019.
[67] A. Mandlekar, D. Xu, R. Martin-Martin, Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations, ArXiv, abs/2003.06088, 2020.
[68] X. Chen, Y. Gao, A. Ghadirzadeh, M. Bjorkman, G. Castellano, and P. Jensfelt. Skew-explore: Learn faster in continuous spaces with sparse rewards, 2020. URL /tttps://openreviie (tte:forrmmmddddteeexxLLLLL.L
[69] R. Hafner, T. Hertweck, P. Kloppner, M. Bloesch, M. Neunert, M. Wulfmeier, S. Tunyasuvu- nakool, N. Heess, and M. A. Riedmiller. Towards general and autonomous learning of core skills: A case study in locomotion. ArXiv, abs/2008. 12228, 2020.
[70] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795-3802. IEEE, 2018.
[71] D.-A. Huang. S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei. S. Savarese, and J. C. Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019.
[72] M. Janner, S. Levine, W. Freeman, Tenenhaum, C. Finn, and J. Wu. Reasoning about physi- cal interactions with object-oriented prediction and planning. ArXiv, Abs/1812.11997 2019.
12

---

[73] T. Kurutach, A. Tamar, G. Yang, S. Russell, and P. Abbeel. Learning plannable representa- tions with causal infogan. CoRR, abs/1807. 807.09341, 2018.
[74] A. Wang, T. Kurutach, K. Liu, Abbeel, and A. Tamar. Learning robotic manipulation through visual planning and acting. CoRR, abs/ 1ess1995,0441,, 2019.
[75] D. Jayaraman, F. Ebert, A. Efros, and S. Levine. Time-agnostic prediction: Predicting pre- dictable video frames. International Conference on Learning Representations, 2019.
[76] K. Liu, T. Kurutach, C. Tung, Abheel, and A. Tamar. Hallucinative topological memory for zero-shot visual planning, 2020.
[77] B. Ichter, Sermanet, and C. Lynch. Broadly-exploring, local-policy trees for long-horizel task planning, 2020.
[78] K. Pertsch, O. Rybkin, F. Ebert, C. Finn, D. Jayaraman, and S. Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors, 2020.
[79] L. P. Kaelbling and T. Lozano-Penez. Hierarchical task and motion planning in the now. In IEEE Conference Roboties and Automation (ICRA), 2011. Finalist, Best Manipulation Paper Award.
[80] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planncr-independent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014.
[81] OpenAl, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, Ribas, Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's 's cube with a robot hand. ArXiv, abs/1910.07113, 2019.
[82] L. Kaelbling and T. Lozano-Perez. Learning composable models of parameterized skills. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 886-893, 2017.
[83] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins. Phdl-the planning domain definition language-version 1.2. Yale Center for Com- putational Vision and Control, Tech. Rep. CVC TR-98-003/DCS TR-1 165, 1998.
13


Reviews:
[{"review_id": "oZmcE1ebFs", "reviewer": "Reviewer_Au3T", "summary": "The paper presents a skill learning algorithm to solve long horizon tasks, where human teachers provide skill annotation using trajectories and images of the terminal states. These demonstration skills are then used to learn skill termination conditions and skill policies. The skill termination conditions are used to plan with a symbolic planner, where when a skill is chosen to solve a long horizon task, its policy is used to get to the skill termination state. The skill learning framework EMBER uses all skill demonstrations to learn skill policies at the same time, while also learning an underlying latent representation of the skill under use. The paper then presents ablation results and comparisons to some previous methods to show the EMBER's superior performance. \n\n ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper uses latent representations to learn a large number of skills within the same network. \nWeaknesses:\nHuman and expert annotation of skills. Annotations of skills with termination symbols, and overall model over options are hard to specify for for TAMP problems, and even harder to learn from demonstrations. In my opinion the paper solves the easier problem to learn multiple policies using latent variables, but does show how the skills themselves would be specified. Further, the symbolic planning is performed over the sequence of sub-tasks provided by humans and it is not required to be over the skill termination classifiers learned. Moreover, the comparisons are not fair as there are significant changes in the BEE and Qt-Opt comparisons as listed in the appendix. A fair comparison for skill discovery would be skill learning work such as Bagaria et al.\n\nBagaria et al: Option Discovery using Deep Skill Chaining, Akhil Bagaria, George Konidaris   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper needs to clarify its claims as to whether it is learning a skill model or using human annotations. This is a hard problem, especially to solve using deep methods. A modification of the claims of the paper would be useful.   ", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "The paper in its current state seems to overclaim its capacity to learn a model over options. This model's associated skills are defined by the expert as clarified in the appendix. The paper demonstrates a method to then learn the skill policies and terminations which is an easier problem than learning a model.  \nMoreover a clear section talking about inputs and outputs of the algorithms along with a clear problem definition would help the paper's readability.\n\nAfter author response:\nI have taken the author's response in mind and their improved clarity of presentation of their approach. I move to accept the paper, and have changed my review accordingly. I also think that the comparison to skill chaining is unnecessary and not an apples to apples comparison. "}}, {"review_id": "hLs_cV7om6T", "reviewer": "Reviewer_vnfa", "summary": "This paper proposes a reinforcement learning architecture that can be supplemented by a classical symbolic planner to solve so-called long-horizon tasks. These tasks require sequencing different 'skills' to realize a given goal (a set of logical states). These skills are grounded action-object-location tuples (e.g., 'open left-drawer', 'grasp marker from left-drawer') and comprise either one or two stages. The proposed learning approach relies on stage-specific components (Q-functions, rewards, binary classifier). The results illustrate the success of the algorithm and how it compares against several other methods. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is its attempt at solving multi-step tasks via a combination of reinforcement learning and classical planning formulation, compared to the focus on learning low-level manipulation skills within RL research domain.\nBoth the training and testing happen on the physical robot, which also enhances the value proposition of the paper. \n\n\nWeaknesses:\\\nThe critical weakness of this work is its non-scalable skill definition. The distinction between a skill and a stage seems somewhat unnecessary. Skills as they are defined in this work sound more like a command, and/or impose a hierarchy of skills (stages as they are called in this work). Wording choice might be unimportant, but this choice affects the analysis and comparison to different methods. For example, a comparison to a hierarchical RL framework, or a model-based task and motion planner might be more relevant when terminology and what they imply are better/properly framed. Because as the authors also highlighted classical RL methods are not designed for multi-step tasks.\n\nSecond, the learned system is strongly biased on the specific environment, i.e., the robot actually does not learn how to open a drawer, but rather it learns how to open that specific (left/right) drawer. A reliable and effective visuomotor skill learning system should not depend on the left/rightness of a drawer at all.\n\nThird significant weakness is the reliance on a large set of perception systems (here five cameras). This hinders the applicability of this framework in many realistic settings. Furthermore, there isn't any analysis and/or discussion on this design choice, except for the inclusion of the wrist camera. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work is a good initial step towards getting closer to what model-based task and motion planners can already reliably accomplish, which is to solve multi-step reasoning and manipulation tasks. It effectively uses visual perception to ground logical predicates, which are then used for a binary classifier and reward function. However, the skill definition is not scalable, and the learned policies are too dependent on the actual setup which in turn impedes the generalization of the skills to different environments. The visual perception setup, comprising five cameras, is also demanding, hindering the applicability, transfer, and replicability of the framework.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "In addition to my suggestions above (Strengths & Weaknesses), here I provide some further improvement points.\n\n- Skill vs. stage distinction seems rather redundant/arbitrary. Why, for example, 'place object on desk' is a different skill than 'place object in cabinet' or 'open left drawer' vs. 'open right drawer'? Isn't 'place' (or 'open', 'grasp', 'close') a skill by itself? Otherwise, we need to iterate over all possible actions and objects and create a new 'skill', which results in a combinatorial explosion of number of skills that robots will not ever be able to learn.  \n\n- The camera setup is unrealistic. The need for 5 cameras just for ultimately pick-and-place tasks have not been discussed or justified (except the importance of the wrist camera, which is appreciated).\n\n- The paper requires going back and forth between the main text and the supplementary material a lot to better understand and evaluate the experimental and implementation details, along with arguments and discussions that are of critical importance for the work. \n\n- The compared algorithms are not exact replicas (different architectures, input spaces, etc.) and above else they are targeted for different scenarios (i.e., short horizon tasks), I'm not sure then whether these are fair comparisons.\n\n- Human collected images do not seem to be always accurate or representative as shown in Fig.5 in the supp. material, or is it due to the low resolution that it's harder to understand?\n\n- Robot policy execution speed is very slow, why is that so?"}}, {"review_id": "UeyXboZJmgn", "reviewer": "Reviewer_REyR", "summary": "The paper presents a method to sequence learned visuomotor skills by learning a representation space with a VAE, a global dynamics model (shared across skills and trained jointly with VAE) and skill-specific reward classifiers and Q-functions. Model-predictive control is used with the Q-functions to execute skills at runtime. The reward classifiers are used as success detectors for when an executed skill is achieved. The Q-functions are learned to account for model inaccuracy and compounding errors over the planning horizon. Long-horizon planning (sequencing skills) is achieved using the skill success classifiers to ground to predicates that can be utilized in a symbolic planner to achieve a particular goal predicate. The predicates are manually defined (in appendix). Experiments are performed on a real robot with long-horizon tasks of cleaning and organizing environments with containers, drawers, cabinets, and various objects.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n    + The method is technically sound and intuitive.\n    + The paper is well-written and organized, I especially liked the appendix organization and thoroughness \n    + The experiments are thorough and very nicely organized. The precision column Table 1 is fantastic, it is an important attribute to consider in the tasks and makes the reported statistics more interpretable across the different baseline methods. \n    + Several relevant baselines were compared to, and there were different experiments for comparing performance of the individual skills.\n    + The real-robot results are impressive.\nWeaknesses:\n    - The connection to symbolic planning is tenuous (see Issues below for further explanation).\n    - A number of relevant citations are missing (again see Issues)\n    - The method itself is not drastically novel, but this is mitigated by the impressive experimental real-robot results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method itself is not drastically novel, I find the high-level idea to be a combination of ideas from citations [1-3] provided in the Issues section below. However, demonstrating the efficacy of a visuomotor MBRL algorithm on a real robot system is a major contribution, as it is all too common for experiments to be performed on simple simulated environments. Demonstrating success on long-horizon tasks is even better. A number of relevant baselines were compared and the experiments were insightful, particularly the correlation of skill precision to method and baseline performance. The paper was extremely well-written and organized, and extensive details are provided in the appendix. I think this method provides a good basis for further extensions and research.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "EMBER doesn’t make sense as an acronym for Example-driven Model-based RL. It's a real stretch, just to get an acronym that is a nice word. I advise against this, but these acronyms are rampant in deep RL so it'll fit right in.\n\nThe description of the symbolic planning in the main paper is lacking in details necessary to understand this aspect of the approach. I would suggest adding a more concrete description of the planner utilized. Symbolic planning is described in Section 4.2 and Algorithm 2, but it is described only at a high-level that a symbolic planner is used. There are details in the tables in the appendix, but it is left unclear what is the symbolic planning algorithm used in the paper. The grounding to a symbolic language suitable for planning is contrived as each skill will have one success classifier, but that classifier will signify potentially a complex combination of predicates (e.g. skill 1 in Table 7 will have one success classifier that binds to a combination of three predicates, symbolic action 1 in Table 6). It will in general be non-trivial for a human to exhaustively consider the logical predicates that are to be satisfied for a particular skill pre- and post-condition. The long-horizon performance depends critically on the ability of the engineers to define these correctly. To the authors' credit, this limitation is mentioned in Appendix H. I only want to highlight the limitation since the long-horizon capabilities of the approach is one of the main novelties, and the method will be susceptible to failure on more complex environments and tasks.\n\nFive cameras is a relatively high number of cameras compared to what is utilized on most research robot platforms. The ablation of the wrist camera is nice, I would also be curious to know how crucial it is to have so many cameras, and if some camera placements are more useful than others. Further ablations on the number and placement of cameras could be informative in extensions to this work. I wouldn’t expect more experiments for the revision, but it would be nice to discuss this in the text.\n\nSome citations that are relevant to include: \n    - [1] does model-predictive control using a learned latent dynamics predictor and learned reward predictor (which can be effectively a classifier as a sparse reward). \n    - [2] extends [1] to predict Q-values instead of just rewards. However, these works do not try to do long-horizon sequencing and they have not been demonstrated to work on real robot systems. \n    - [3] is an info-theoretic MPC formulation that learns Q-functions from real data to mitigate the influence of the planning horizon choice and model-error. I think these papers are sufficiently relevant to the present paper to be cited. \n\n[1] Hafner, Danijar, et al. \"Learning latent dynamics for planning from pixels.\" International Conference on Machine Learning. PMLR, 2019.\n[2] Hafner, Danijar, et al. \"Dream to Control: Learning Behaviors by Latent Imagination.\" International Conference on Learning Representations. 2019.\n[3] Bhardwaj, Mohak, et al. \"Information theoretic model predictive q-learning.\" Learning for Dynamics and Control. PMLR, 2020.\n"}}]

Author rebuttal:

Meta-review: The paper proposes an RL architecture that uses a symbolic planner for tasks that involve many steps.

Quoting the summary of reviewer REyR:
The authors present a sound approach to doing visuomotor skill learning and show very impressive results on a real robot platform. 
The experiments are thorough, detailed, and insightful. 

The submission contained major weaknesses, in particular, the claims of the paper were incorrectly stated and that skills are supervised by human experts. The authors have addressed these concerns and put a lot of effort into the revision process. 
The remaining weakness of scalability is discussed in the appendix. I ask the authors to move some of this discussion to the main text.

I recommend accepting this paper.
 

Decision: Accept (Poster)