Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
Authors: Suraj Nair
Chelsea Finn
Venue:  (2020)
Keywords: video prediction
reinforcement learning
planning

Abstract: Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which generates subgoal images conditioned on a goal image, and uses them for planning. The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identifies semantically meaningful states as subgoals. Across three out of four simulated vision-based manipulation tasks, we find that our method achieves more than 20% absolute performance improvement over planning without subgoals and model-free RL approaches. Further, our experiments illustrate that our approach extends to real, cluttered visual scenes.

Published as a conference paper at ICLR 2020
HIERARCHICAL FORESIGHT:
LEARNING OF LONG-HORIZON TASKS VIA VISUAL SUBGOAL GENERATION
Suraj Nair Chelsea Finn 1.2 Stanford University. "Google Brain
ABSTRACT
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self- supervision, reaching novel goals in cluttered scenes with unseen objects. How- ever, due the compounding uncertainty in long horizon video prediction and poor scalability of sampling-basee based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hi- erarchical visual foresight (HVF), which generates subgoal images conditioned goal image, and uses them for planning. The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identifies semantically meaningful states as sub- goals. Across four simulated baseed manipulation tasks, we find that our method achieves more than 20% absolute performance improvement over plan- ning without subgoals and model-free RL approaches. Further, our experiments illustrate that our approach extends to real, cluttered visual scenes.
INTRODUCTION
Developing robotic systems that can complete long horizon visual control tasks, while generalizing to novel scenes and objectives, remains an unsolved and challenging problem. Generalization to unseen objects and scenes requires robots to be trained across diverse environments, meaning that detailed supervision during data collection in not practical to provide. Furthermore, reasoning over long-horizon tasks introduces two additional major challenges. First, the robot must handle large amounts of uncertainty the horizon increases. And second, the robot must identify how to reach distant goals when only provided with the final goal state, sparse indication of the task, as opposed to shaped cost that implicitly encodes how get there. In this work, we aim develop method that can start to address these challenges, leveraging self-supervised models learned using only unlabeled data, to solve novel temporally-extendee tasks.
Model-based reinforcement learning has shown promise in generalizing to novel objects and tasks, as learned dynamics models have been shown to generalize new objects (Finn & Levine 2016 Ebert et al. 2018b), and can be used in conjunction with planning to reach goals unseen during training. However, planning reach temporally distant goals is difficult. As the planning horizon increases model error compounds, and the cost function often provides only a noisy or sparse signal of the objective. Both of these challenges are exacerbated when planning in visual space.
In this work, the key insight that we leverage is that while model eтror and sparse cost signals can make long horizon planning difficult, we can mitigate these issues by learning to break down long- horizon tasks into short horizon segments. Consider, for example, the task of opening drawer and putting a book in it, given supervision only in the form of the final image of the open drawer containing the book. The goal image provides nearly no useful cost signal until the last stage of the task, and model predictions are likely to become inaceurate beyond the first stage of the task. However, if we can generate sequence of good subgoals, such as (1) the robot arm grasping the
Work completed at Google Brain Videos and code available at:

---

Published as a conference paper at ICLR 2020
drawer handle, (2) the open drawer, and (3) the robot arm reaching for the book, planning from the initial state to (1), from (1) to (2), from (2) to (3), and from (3) to the final goal, the problem becomes substantially easier. The subgoals break the problem into short horizon subsegments each with some useful cost signal coming from the next subgoal image.
Our main contribution is a self-superriser hierarchical planning framework, hierarchical visual fore- sight (HVF), which combines generative models of images and model predictive control to decom- pose a long-horizon visual task into a sequence of subgoals. In particular, we propose optimizing over subgoals such that the resulting task subsegments have low expected planning cost. However, in the case of visual planning, optimizing over subgoals corresponds to optimizing within the space of natural images. To address this challenge, we train a generative latent variable model over im- ages from the robot's environment and optimize over subgoals in the latent space of this model. This allows us to optimize over the manifold of images with only a small number of optimization variables. When combined with visual model predictive control, we observe that this subgoal op- timization naturally identifies semantically meaningful states in long horizon tasks as subgoals, and that when using these subgoals during planning, we achieve significantly higher success rates on long horizon, multi-stage visual tasks. Furthermore, since our method outputs subgoals condi- tioned on goal image, we can use the same model and approach to plan to solve many different long-horizon tasks, even with previously unseen objects. We first demonstrate our approach in sim- ulation on a continuous control navigation task with tight bottlenecks, and then evaluate on a set of four different multi-stage object manipulation tasks in simulated desk environment, which require interacting with up to 3 different objects. In the challenging desk environment, we find that our method yields at least 20% absolute performance improvement over prior approaches, including model-fie free reinforcement learning and a state of the art subgoal identification method. Finally, we show that our approach generates realistic subgoals on real robot manipulation data.
2 RELATED WORK
Developing robots that can execute complex behaviours from only pixel inputs has been a well
studied problem, for example with visual servoing (Mohta et al.) [014] Espiau et al.) 1992 Wilson
iet al.
1996. Yoshimi & Allen 1994, Jagersand et al. 1997] Lampe & Riedmiller] 2013 Sadeghi
al.
2018 Sadeghi 2019). Recently, reinforcement learning has shown promise in completing
complex tasks from pixels (Ghadirzadeh et al. 2017 Levine et al. 2015 Kalashnikov et al. 2018
Lange et al.] 2012: OpenAl et al.J | [Schenck Fox 2016; Matas et al.] James et al. Singh et al.; 2019). including in goal-conditioned settings (Kaelbling) 1993| Schaul et al.,
2017
2015
Andrychowicz et al. 2017, (aadehi et al. 2018, Saddghhi 2019: Nair et al. 2018a). While
model free RI. approaches have illustrated the ability to generalize to new objects (Kalashnikov
cct al.
2018) and learn tasks such as grasping and pushing through self-supervision (Pinto & Guptal Zeng et al. 2018), pure model-free approaches generally lack the ability to explicitly reason plans, making them ill-suited for the problem of learning long-horizon tasks with limited supervision.
2015
over cemporaly-extendde
Video prediction and planning have also shown promise in enabling robots to complete a diverse set of visuomotor tasks while generalizing to novel objects (Finn & Levine 2016: Kalchbrenner let al. 2016 Boots et al.] 2014 Byravan & Fox 2016). Since then, a number of video predic- frameworks have been developed specifically for robotics (Babaeizadeh et al. 2017 Lee al.)
tion
2018
Ebert et al. (017). which combined with planning have been used to complete diverse behav-
1ors Nair et al. 2018b: Ebert et al. 2018b Paxton et al. 2018 Xie al.] (2019). However, these
approaches still struggle with long horizon tasks, which we specifically focus on.
One approach handle long horizon tasks is add compositional structure to policies, either from demonstrations (Krishman et al. 2017, Fox et al.) 2018). with manually-specifie primitives (Xu 2017, |Huang et al., 2) learned temporal abstractions (Neitz et al., 2018). or through model free reinforcement learning (Sutton et al. 1999) Barto& & Mahadevan 2003 Bacon et al.)
et al.
2016
Nachum et al. 2018 Levy et 2019). These works have studied such hierarchy in grid
worlds (Bacon et al. 2016) and simulated control tasks (Nachum et al.) 2018 Eysenbach et al.)
2018
Levy et al., 2019) with known reward functions. In contrast, we study how to incorporate compositional structure in learned model- based planning with video prediction models. Our ap- proach is entirely self supervieee without motion primitives, demonstrations, or shaped rewards, and scales to vision-based manipulation tasks.
2

---

Published as a conference paper at ICLR 2020
Classical planning methods have been successful in solving long- nor-hhoizzon tasks Vale 2006
Choset et al.) 2005), ) but make restrictive assumptions about the state space and reachability between states, limiting their applicability to complex visual manipulation tasks. Similarly, completing long horizon tasks has also been explored with symbolic models (Toussaint et al., 2019) and Task and
Motion Planning (TAMP) (Kaelbling & Lozano-Peres 2011] Srivastava et al. 2014). However, un- like these approaches our method requires no additional knowledge about the objects in the scene nor any predefined symbolic states. Recently, there have been several works that have explored planning in learned latent spaces (Kurutach et al., 2018 Ichter & Pavone 2018 Watter et al. 2015 Srinivas al., 2018). This has enabled planning in higher dimensional spaces, however these methods still struggle with long-horizon tasks. Furthermore, our hierarchical planning framework is agnostic to state space, and could directly operate one the above latent spaces.
A number of recent works have explored reaching novel goals using only self-supervision (Finn & Levine| 2016 Eysenbach et al., 2019; Kurutach et al. 2018) Wang et al. 2019 [Jayaraman al,
2019 Nair et al. 2018a). In particular, time-agnostic prediction (TAP) (Jayaraman et al. 2019) aims
to identify bottlenecks in long-horizon visual tasks, while other prior works (Nair et al. [00 Sa Finn
& Levine 2016) reach novel goals using model-free model- RL. We compare all three of
these methods in Section 5 and find that HVF significantly outperforms all of them.
PRELIMINARIES
We formalize our problem setting goal-conditioned Markov decision process (MDP) defined by the tuple (S, A. G. C, where s E S is the state space, which in our case corresponds to images, E A is the action space, p(s st at) governs the environment dynamics, 9 c represents the set of goals which is a subset of possible states, C(st, $g) represents the cost of being in state 8, E S given that the goal is &g E 9, and À the discount factor. practice, acquiring cost functions that accurately reflect the distance between two images is challenging problem (Yu et al. 2019). We make no assumptions about having a shaped cost, assuming the simple yet sparse distance metric of lg distance in pixel space in all of our experiments. Approaches that aim to recover more shaped visual cost functions are complementary to the contributions of this work.
In visual foresight, or equivalently, visual MPC (Finn & Levine 2016 2018b), the robot collects a data set of random interactions [(11, a1), (32, 02 (ST, ar) ] from a pre-determined policy. This dataset is used to learn model of dynamics fo($4+1, St-2,..., 8tth Qt, a.+1:-.., a++A-1) through maximum likelihood supervised learning. Note the states are camera images, and thus fo is an action-conditioned video prediction model. Once the model is trained, the robot is given some objective and plans a sequence of actions that op- timize the objective via the cross entropy method (CEM) (Rubinstein Kroese 2004). In this work, we will assume the objective is specified in the form of an image of the goal Sg. while CEM aims to find a sequence of actions that minimize the cost C between the predicted future frames from fo and the goal image. While standard visual foresight struggles with onng-hoizoon tasks due to uncertainty in So as the prediction horizon h increases and sparse for CEM to optimize, in the next section we describe how proposed approach uses subgoal generation to mitigate these issues.
Ebert et al.
4 HIERARCHICAL VISUAL FORESIGHT
Overview: We propose hierarchical visual foresight (HVF), a planning framework built on top of visual foresight (Finn& & Levine 2016 Ebert al., 2018b) for long horizon visuomotor tasks. We observe that when planning to reach a long horizon goal given only a goal image, standard planning frameworks struggle with (1) sparse noisy cost signals and (2) compounding model error. Critically, we observe that if given the ability to sample possible states, there is potential to decompose long horizon planning task into shorter horizon tasks. While this idea has been explored in classical planning (Betts, 2010), state generation is significant challenge when dealing with high dimensional image observations.
One of our key insights is that we can train deep generative model, trained exclusively on self- supervised data, as a means to sample possible states. Once we can sample states, we also need to evaluate how easy it is to get from one sampled state to another, to determine if state makes for a good subgoal. We can do so through planning: running visual MPC to get from one state to another and measuring the predicted cost of the planned action sequence. Thus by leveraging the low-dimensional space of a generative model and the cost acquired by visual MPC, we can optimize over sequence of subgoals that lead to the goal image. In particular, we can explicitly search in
3

---

Published as a conference paper at ICLR 2020
goal state 5g
Hierarchical Visual Foresight
generated subgoels $2
52
generative model
9ole)
prediction
video
generated subgoels $1
model
s1
with &
current state
Optimize subgoals with CEM to minimize planning cost
50
Visual planning with
Figure 1: Hierarchical visual foresight: Our method takes as input the current image, goal image, an action conditioned video prediction model for and generative model go(=). Then, it samples sets of possible states from g0(2) as sub-goals. It then plans between each sub-goal, and iteratively optimizes the sub-goals to min- imize the worst case planning cost between any segment. The final set of sub-goals that minimize planning cost selected, and finally the agent completes the task by performing visual model predictive control with the sub-goals in sequence. this example the task is to push a block off the table, and close the door. Given only the final goal image, HIVF produces sub-goals for (1) pushing the block and reaching to the handle and (2) closing the door.
latent image space for subgoals, such that no segment is too long-horizon, mitigating the issues around sparse costs and compounding model error.
In the following sections, we will describe more formally how HVF works, how we learn the gener- ative model, how we learn the dynamics model, how goal-condee condiconnd planning with visual MPC is executed, and lastly how subgoals are optimized.
Hierarchical visual foresight: Formally, we assume the goal conditioned MDP setting in Section 3 where the agent has current state 8o, goal state &g. cost function C, and dataset of environment interaction {(11, 41, $2, 02, .... ST, a.r) This data can come from any exploration policy; in prac- tice, we find that interaction from a uniformly random policy in the continuous action space of the agent works well. From this data, we train both dynamics model So using maximum likelihood supervised learning, as well as generative model
9p.
Now given so and sy. the objective is to find K subgoals $1, 52, ... SK that enable casier completion of the task. Our hope is that the subgoals will identify steps in the task such that, for each sub- segment, the planning problem is easier and the horizon is short. While one way to do this might be to find subgoals that minimize the total planning cost, we observe that this does not necessarily encourage splitting the task into shorter segments. Consider planning point on that line as subgoal would equally minimize the total cost. Therefore, we instead optimize for subgoals that minimize the worst expected planning cost across any segment. This corresponds to the following objective:
straight line: using any
min max{Crlon (50, $1), Cytan (31, 82), ...y Cptan (SK, {s),
(1)
where Cptan( (8,, 83) is the cost achieved by the planner when planning from 8; to 8j, which we compute by planning a sequence of actions to reach sy from 8; using fo and measuring the predicted cost achieved by that action sequence Once the subgoals are found, then the agent simply plans using visual MPC (Finn& & Levine 2016 |Ebert et al., 2018b) from each Sk-1 to Sy until cost threshold is reached or for a fixed, maximum number of timesteps, then from s to 8241. until planning to the goal state, where ot the actual state reached when running MPC to get to 8y. For full summary of the algorithm, see Algorithm 1. Next, we describe individual components in detail.
Generative model: To optimize subgoals, directly optimizing in image space is nearly impossible due to the high dimensionality of images and the thin manifold on which they lie. Thus, we learn
We compare max/mean cost in Section 5.4
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Hierarchical Visual Foresight
82. 8g)
1: Receive current state 8, and goal state $g
2: Initialize = (0, I), M = 200, M = 40, number of subgoals K
3: while (a > le - 3) or (iterations <, 5) do
4: for 1,,,..., M do
5: sg sK+1 = $1
6: x" ~ q
Sample latent subgoal lists
7:
for h = 1,2, .... K do
8:
8." (0 Th [k), 88')
# Map latent to image subgoal
9:
Agi Apian. plan MPC fo, 1,
Optimal actions and planning cost
10:
end for
11:
Ai Cpan,KK11
(PC( K 8 xk+1)
12:
Cost = -maxx (Cplan,0.... ... Cpann, K+1} end for
# Max planning cost across segments
13:
14: Zuors Soot(K: [Cost" Cost" J, V [z* 1,,",..?' 1)
Rank by Cost
15:
Refit 9 to o (Zoort [1], Zoon [M' 1 with low cost
16:
end while
17:
for A: = 1,2...... K do
18: Set final subgoal 8% = Gol (ootted
19: Execute Ag,- MPC(fo, 8k-1, 81)
20:
end for
21: Execute Ax. MPC(Jo, (K,,
generative model 8 ~ (z() such that samples represent possible futures in the scene, and optimize in the low-dimensional latent space In settings where aspects of the scene or objects change across episodes, we only want to sample plausible future states, rather than sampling states corresponding all scenes. To handle this setting, we condition the generative model on the current image, which contains information about the current scene and the objects within view. Hence, in our experiments, we use either variational autoencoder (VAE) or conditional variational autoencoder (CVAE), with a diagonal Gaussian prior over 2, i.e. 2 ~ (O. I). In the case of the CVAE, the decoder also takes as input an encoding of the conditioned image, i.e. 8 ~ go(z, sn). In practice, we use the inital state so as the conditioned image. We train the generative model on randomly collected interaction data. The architecture and training details can be found in Appendix A.3
Dynamics model: The forward dynamics model fo can be any forward dynamics model that estimates p(5t+1, St42, ..., St+h St, ag at+1s..., dt+h-1): In our case states are images, so we action conditioned video prediction model, stochastic variational video prediction (SV2P) [Babaeizadeh et al., 2017) for fe. We train the dynamics model on randomly collected interaction data. Architecture and training parameters are identical to those used in (Babaeizadeh et al. 2017) details can be found in Appendix A.4.
MPC & planning with subgoals: When optimizing subgoals, need some way measure the
ease of planning to and from the subgoals. We explicitly compute this expected planning cost between two states Sk, Sk+1 by running model predictive control A, C MPC fe, Sk. Sk+1 as done in previous work (Finn& & Levine, 2016 Nair et al. 20188 Ebert et al. 2018b). This uses the model fo to compute the optimal trajectory between 8y and 8k+1, and returns the optimal action A and associated cost C. Note this does not step any actions in the real environment, but simply produces an estimated planning cost. Details on this procedure can be found in Appendices A. and A.2 Once the subgoals have been computed, the same procedure is run MPC(Je, 8k 88) until 8
is reached (measured by cost threshold) or for a fixed, maximum number of timesteps, then from MPC(fe, s $k+1), .... MPC sk.,s, sg) (Alg. I Lines 17:21), where & represents the state actually reached after trying to plan to sk- In this case, the best action at each step is actually applied in the environment until the task is completed, once in the beginning calling HVF(fe, go(2), Su, $y), the plan with those subgoals. In principle HVF can be called every step, but for computational efficiency we only call HVF once.
the horizon runs out. Note only compute subgoals
Subgoal optimization: Since we need to search in the space of subgoals to optimize Equation we perform subgoal optimization using the cross entropy method (CHM) (Rubinstein & Kroese 2004) in the latent space of our generative model (z(2). Note this subgoal optimization CEM is distinct
Pscuddocde for MPC can be found in Appendix A.1
5

---

Published as a conference paper at ICLR 2020
Sub Goal
Sub Goals
Current State
Sub Goal
Goal State
Currant State
Sub Goal
Sub Goal
State
Medium
Hard
Figure 3: Qualitative Results Maze Navigation. Here we show some of the generated subgoals from HVF for the Maze Navigation task. On the left we observe samples for K = 1 and on the right we observe samples for K = 2. We see in the "Medium" difficulty and K = 1. the generated subgoal is very close to the intuitive bottleneck in the task. Similarly, in the "Hard" case with K = 2 we observe that the discovered subgoals are close the gaps the walls, which represent the bottlenecks in the task.
from the CEM used for computing the planning cost, which we use as subroutine within this outer level of CEM. At a high-level, the subgoal optimization samples list of subgoals, evaluates their effectiveness towards reaching the goal, and then iteratively refines the subgoals through resampling and reevaluation. We begin by drawing M samples from diagonal Gaussian distribution = (O I) where the dimensionality of q is K L, where K is the number of subgoals and L is the size of the latent dimension of the generative model go (=) (Alg. 1 Line 6). Thus, one sample from y gives K latents each of size T., each of which decoded into a subgoal image (Alg. 1 Line 8). Then, described in Equation [ the cost for one sample (one set of K subgoals) is computed as the maximum planning cost across the subsegments (Alg. Lines 9:12). The M samples are ranked by this cost, and then g is refit to the latents of the best M samples (Alg. Lines 14:15). In all of our experiments we use M 200, M = 40 and L 8.
EXPERIMENTS
In our experiments, we aim to evaluate (1) if, by using HVE, robots can perform challenging goal-conditioned hong-horizon tasks from raw pixel observations more effectively than prior self- supervised approaches, (2) if HVF is capable of generating realistic and semantically significant subgoal images, and (3) if HVF can scale to real images of cluttered scenes. do so, we test on three domains: simulated visual maze navigation, simulated desk manipulation, and real robot ma- nipulation diverse objects. The simulation environments use the MuJoCo physics engine (Todorov) et al., 2012). We compare against three prior methods: (a) visual foresight (Finn & Levine [016]; Ebert et al. 2018b), which uses no subgoals, (b) RIG (Nair et al., 2018a) which trains a model-free policy reach generated goals using latent space distance as the cost, and (c) visual foresight with subgoals generated by time-agnosee prediction (TAP) (Jayaraman et al.) 2019), a mate-of-the-an method for self-supervised generation of visual subgoals, which generates subgoals by predicting the most likely frame between the current and goal state.
D 5.1 MAZE NAVIGATION
100
First, we study HVF in 2D maze with clear bottleneck states, as it allows us to study how HVF compares to oracle subgoals. In the the maze navigation environment, the objective is for the agent (the green block) to move to goal position, which is always in the right- most section. To do so, the agent must navi- gate through narrow gaps in walls, where the position of the gaps, goal, and initial state of the agent are randomized within each episode. The agent's action space 2D Cartesian move- ment, and the state space is top down images. We consider three levels of difficulty, "Easy" where the agent spawns in the rightmost sec- tion, "Medium" where the agent spawns in the middle, and "Hard" where the agent spawns in the leftmost section. Details in Appendix B.T
50
50-
VF (No Subgoals)
HVF, Subgoal (Ours)
HVF, Subgoals (Ours)
Ground Truth Bottleneck
Medium
Hard
Figure 2:
Quantitative Results for Maze Naviga-
tion. Success for navigation tasks using sub- goals. HVF subgoals, and ground truth hand specified subgoals. We observe that HVF significantly improves performance on the "Medium" and "Hard" difficulty compared to not using subgoals. Computed over 100 randomized trials.
Videos/cnde available httpt:: / googl wttp:/e..cccsttaaaafeeaaea htwpgoge...//sstaooo.eeeee/////e

---

Published as a conference paper at ICLR 2020
Sub Goal
2 Sub Goals
Current State
Sub Goal
Goal State
Current State
Sub Goal
Sub Goal
State
Door
2 Block
Door Block
Door 2 Block
Figure 5: Qualitative Results for Desk Manipulation. Example generated subgoals from HVF for the desk manipulation tasks with one two subgoals. We observe interesting behavior: in the Door Closing + Block Pushing task with one subgoal, the subgoal to slide the door then push the block, while in the Door Closing 2 Block Pushing the first subgoal is push the blocks, then grasp the door handle, then slide the door.
Results: In Figure 2 we observe that using HVF subgoals consistently improves success rate over visual foresight (Finn& & Levine 2016) without subgoals, indicating that it is able to find subgoals that make the long-horizon task more manageable. Additionally, we compare to the "Ground Truth Bottleneck" that
manually-designen subgoals, where the subgoal is exactly at the gaps in the walls (or the midpoint between states in the "casy" case). We find that while using the oracle subgoals tends yield the highest performance, the oracle subgoals do not perform perfectly, sug- gesting that a non-trivial amount of performance gains are to be had from improving the consistency of the video prediction model and cost function for short-horizon problems, as opposed to the sub- goals. We also show that HVF outperforms time agnostic prediction (TAP) (Jayaraman et al. 2019) Appendix
We next qualitatively examine the subgoals discovered by HVF in Figure and find empirically that they seem correspond to semantically meaningful states. In this task there are clear bottlenecks specifically reaching the gaps in the walls and find that HVF outputs close to these states as subgoals. For example, when the agent starts in the leftmost section, and has to pass through two narrow gaps in walls, we observe the first subgoal goal corresponds to the agent around the bottleneck for the first gap and the second subgoal near the second gap.
5.2
SIMULATED DESK MANIPULATION
VF (No Subgoals)
HVE, 1 Subgoal (Ours)
We now study the performance improvement and subgoal quality of HVF in a challeng- ing simulated robotic manipulation domain. Specifically, simulated Franka Emika Panda robot arm is mounted in front of a desk (as used (Lynch et al. 2019)). The desk consists of 3 blocks, sliding door, three buttons, and drawer. We explore four tasks this space: door closing, block pushing, door closing block pushing, and door closing 2 block push- ing. Example start and goal images for each task are visualized in Figure 5) and task de- tails in Appendix B.2 The arm is controlled with 3D Cartesian velocity control of the end- effector position. Across the 4 different tasks in this environment, we use single dynamics model fo and generative model (2). Experi- mental details are in the Appendix B.2
Subgoal TAP Subgnel TAP
HVF, Subgoals (Ours)
Success Raai
Door 2
Door Block
Door: 2 Blocck
Figure 4: Quantitative Results for Desk Manipu- lation: Using HVF drastically improves performance. Across all tasks, HVF with two subgoals leads a more 20% absolute performance improvement
over visual foresight (Finn & Levine (2016). TAP (Ja-) yaraman et al. 2019), and RIG (Nair et al] (018a)). Computed over 100 trials with random initial scenes.
Results: As seen 4 we find that using HVF subgoals dramatically improves performance, providing at least 20% absolute improvement in success rate across the board. In the task with the longest horizon, closing the door and sliding two blocks off the table, we find that using no subgoals or subgoal has approximately 15% performance, but subgoals leads over 42% success rate. We compare to subgoals generated by time agnostic prediction (TAP) (Jayaraman et al. 2019) and find that while it does generate plausible subgoals, they are very close to the start or goal, leading to

---

Published as a conference paper at ICLR 2020
benefit in planning. We also compare against RIG (Nair et al. 2018a) where train model free policy in the latent space of the VAE to reach "imagined" goals, then try and reach the unseen goals.
However, due to the complexity of the environ- ment, we find that RIG struggles to reach even the sampled goals during training, and thus fails on the unseen goals. Qualitatively, in Figure 5 observe that HVF outputs meaningful sub- goals on the desk manipulation tasks, often pro- duces subgoals corresponding grasping the handle, sliding the door, or reaching to a block.
Sub Goal
Current State
Sub Goal
Goal State
5.3 REAL ROBOT MANIPULATION
Lastly, we aim to study whether HVF can ex- tend to real images and cluttered scenes. To do
2 Sub Goals
s0, we explore the qualitative performance of
Current State
Sub Goal
Sub Goal
Goal State
method on the BAIR robot pushing dataset
(Ebert et al. 2018a). We train fe and (2( so) on the training set, and sample current and goal states from the beginning and end of test trajec- tories. We then qualitatively evaluate the sub- goals outputted by HVF. Further implementa- tion details are in the Appendix B.3
Results:
Our results are illustrated in Fig. 6 serve that even in cluttered scenes, HVF pro- duces meaningful subgoals, such grasping ob- jects which need to be moved. For more exam- ples, Figure C.2 We observe that in both the BAIR dataset and the desk manipulation ex- periments, the most common failure cases corresponded to the subgoal prediction and model ignor- ing the objects and focusing exclusively on the arm. Cost functions which can more effectively capture objects and their poses would be a step towards addresing this.
We ob-
Figure 6: BAIR Dataset Qualitative Results. The subgoals generated by HVF on the BAIR dataset, which we find correspond to meaningful states between the start and goal. For example, when moving objects we subgoals corresponding to reaching/grasping.
5.4 ABLATIONS
# subgoals 0
5
10
In our ablations, we explore three primary ques- tions: (1) What is the optimal number of sub- goals?, (2) Is there difference between HVF using the max and mean cost across subseg- ments?, and (3) Is HVF as valuable when the samples used for visual MPC is significantly increased? We evaluate these questions in the maze navigation task on the "Hard" difficulty. All results report success rates over 100 trials using random imitialization/gaal.. Additional ablations on planning horizion and latent space cost can be found in Appendix C.I
success
33%
47%
54%
39%
2%
0%
Table 1: Number of Subgoals. With a fixed sampling
budget. as the number of subgoals increases beyond 2, performance drops as the subgoal search is challenging.
in- crease the number of subgoals in Table Interestingly, we observe that as scale the number of subgoals beyond 2, the performance starts to drop, and with 5 or more subgoals the method fails. We conclude that this is due to the increasing complexity of the sub- goal search problem: the sampling budget allocated for subgoal op- timization is likely insufficient to find a large sequence of subgoals.
Number of Subgoals: We explore how HVF performs
# subgoals
1
mean cost
0.45
0.53
cost
0.47
0.54
Table 2: Max vs Mean. Suc- cess rates minimizing mean vs. max cost across subsegments. Max is marginally better.
Max vs Mean: In our HVF formulation, we define the subgoal optimization objective as finding the subgoals that minimize the max cost across subsegments. Table 2 com- pares to using the mean cost. We find that using the max cost is marginally better.
subgoals
0
2
200 samples
0.33
0.47
0.54
1000 samples
0.35
0.54
0.55
Table 3: Sample Quantity. Success using
visual MPC with 200 1000 samples each CEM iteration. Using more samples hetter, hut HVF still outperforms standard visual foresight by wide margin.
Sample Quantity: Table 3 we examine how the num- ber of action samples affects the relative improvement of HVF. Using more samples should also mitigate the chal-
lenges of sparse costs, might suspect that HVF would
be less valuable in these settings. On
8

---

Published as a conference paper at ICLR 2020
the contrary, we find that HVF still significantly outperforms subgoals, and the improvement between using and subgoals is even more significant.
CONCLUSION AND LIMITATIONS
We presented an self-supervised approach for hicrarchical planning with vision-based tasks, hier- archical visual foresight (HVF), which decomposes visual goal into a sequence of subgoals. By explicitly optimizing for subgoals that minimize planning cost, HVF is able to discover semanti- cally meaningful goals in visual space, and when using these goal for planning, perform variety of challenging, long-horizon rision-based tasks. Further, HVF learns these tasks in an entirely self- supervised manner without rewards or demonstrations.
While HVF significantly extends the capabilities of prior methods, a number of limitations remain. First, HVF assumes access to an exploration policy which can cover enough of the state space to learn a good model. While the random policy used works for our environments, more complex environments may require better exploration techniques, such as intrinsic motivation methods.
Another limitation of the current framework is its computational requirements, as the nested op- timization procedure requires many iterations of MPC with expensive video prediction models. Specifically, assume that the horizon of the task is T, one iteration of visual MPC has com- putational cost C. and HVF is using subgoals, and searching in a space of subgoal se- quences. Then normal visual foresight or TAP would have cost T x C, while HVF would have cost (T' x C* + (K X N X C). However note the computation (K X N X C*) can be heavily parallelized because it is done completely offline (without environment interaction). We also expect that this can be mitigated by reformulating the subgoal prediction problem as policy inference, and training a subgoal generation policy in the loop of training, interesting direction for future work.
Further, while the development of accurate visual cost functions and predictive models were not the main aim of this work, the performance with oracle subgoals suggests this to be a primary bottle- neck in furthering the performance of HVE. Specifically, learning dynamics models and cost func- tions which more effectively capture the state of all the objects in the scene would significantly improve the performance of HVF.
Lastly, development of better generative models that can capture the scene and generate possible futures remains an open and challenging problem. Work in generative models that enable better generation of cluttered scenes with novel objects could improve the applicability of HVF in real- world settings.
ACKNOWLEDGMENT
We would like to thank Sudeep Dasari, Vikash Kumar, Sherry Moore, Michael Ahn, and Tuna Tokosz for help with infrastructure, well as Sergey Levine, Abhishek Gupta. Dumitru Erhan, Brian Ichter, and Vincent Vanhouke for valuable discussions. This work was supported in part by an NSF GRFP award. Chelsea Finn is CIFAR Fellow in the Learning in Machines & Brains program.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR, 0a//170700195. 2017.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. CoRR, abs/1710. 11252, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup.
The option-critic architecture. CoRR,
abss/1609.05 050..014110000 2016.
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13(1-2):41-77, January
January 2003. ISSN 0924-6703.
J. Betts. Practical Methods for Optimal Control and Estimation Using Nonlinear Programming. Society for Industrial and Applied Mathematics, second edition, 2010.
B. Boots, A. Byravan, and D. Fox. Learning predictive models of a depth camera amp; manipulator from raw execution traces. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4021-4028 May 2014. doi: 10.1109//CR..001 4..64444444
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural
networks. CoRR, abs/1606.001 603788278. 2016.
9

---

Published as a conference paper at ICLR 2020
Howie Choset, Kevin M. Lynch, Seth Hutchinson, George Kantor, Wolfram Burgard, Lydia Kavraki, and Sebastian Thrun. Principles of Robot Motion: Theory, Algorithms, and Implementations. MIT Press. 2005.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRR, abs/1710.002668, 2017.
Frederik Ebert, Sudeep Dasari, Alex X. Lee, Sergey Levine, and Chelsea Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. CoRR, abs/ b/bs/1810.00 180.0183,,,,43, 2018a.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex X. Lee, and Sergey Levine. Visual foresight: Model-hased deep reinforcement learning for vision-hased robotic control. CoRR,
abs/1812.00566 2018b.
B. Espiau, F. Chaumette, and P. Rives. A new approach to visual servoing in robotics. IEEE Transactions on Robotics and Automation, 833):313-326, June 1992. ISSN 1042-296X.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/ bbss/802.00677, 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov. and Sergey Levine. Search on the replay buffer: Bridg-
ing planning and reinforcement learning. CoRR, abs/ abs//9906,0553, 2019.
Chelsea Finn and Sergey Levine Deep visual foresight for planning robot motion. CoRR,
1ess0660.., 2016.
Roy Fox, Richard Shin, Sanjay Krishnan, Ken Goldberg, Dawn Song, and Ion Stoica. Parametrized
hierarchical procedures for neural programming. In International Conference on Learning Rep- resentations, 2018 8.
Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Marten Björkman. Deep predictive policy train- ing using reinforcement learning. CoRR, abs/1703.001 000727, 2017.
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and
Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demon- stration. CoRR, abs/1 807.03480 2018.
Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. CoRR,
abs/1807.10036. 2018
M. Jagersand, O. Fuentes, and R. Nelson. Experimental evaluation of uncalibrated visual servoing for precision manipulation. In Proceedings of International Conference on Robotics and Automa- tion, volume 4, pp. 2874-28725 2880 vol.4, April 1997.
Stephen James, Andrew J. Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation real world for multi-stage task. CoRR, 0tns/1707.02266 2017.
Dinesh Jayaraman, Frederik Ebert, Alexei Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. In International Conference on Learning Representations, 2019.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1098-1098, 1993.
Leslie Pack Kaclbling and Tomas Lozano- Perez. Hicrarchical task and motion planning in the now. In IEEE Conference Robotics and Automation (ICRA), 2011. Finalist, Best Manipulation Paper Award.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arxiv:Preprim, 2018. Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. CoRR, abs/1610.00522 2016.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. DDCO: discovery of deep continuous options forrobot learning from demonstrations. CoRR, abs/1771000542. 2017.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J. Russell, and Pieter Abbeel. Learning plannable
representationns with causal infogan. CoRR, abs/1801 .0341,, 2018.
Thomas Lampe and Martin Riedmiller. Acquiring visual servoing reaching and grasping skills using neural reinforcement learning. In International Joint Conference on Neural Networks (IJCNN 2013), 2013.
S. Lange, M. Riedmiller, and A. Voigtländer. Autonomous reinforcement learning on raw visual input data in real world application. The 2012 International Joint Conference on Neural Networks (LJCNN), June 2012.
Steven M. La Valle. Planning Algorithms. Cambridge University Press, 2006. doi: 10.1017/ CBO97805 1546877.
10

---

Published as a conference paper at ICLR 2020
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. CoRR, abs/1804.01523,, 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. CoRR, Mos//54440072, 2015.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight.
In International Conference on Learning Representations, 2019.
Corey Lynch. Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. CoRR, abs/1903.019933 2019.
Jan Matas, Stephen James, and Andrew J. Davison. Sim-to-real reinforcement learning for de- formable object manipulation. CoRR,
2018.
K. Mohta, V. Kumar, and K. Daniilidis. Vision-based control of quadrotor for perching on lines.
In 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014. Ofir Nachum, Shixiang Gu. Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. CoRR, abs/ 005/1805.08220 00096, 2018.
Ashvin Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR, abs/1807. (bs/1827.04772, 2018a.
Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, and Vikash Kumar. Time rever- sal as self-supervision. CoRR, abs/1810.01122 2018b.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In Advances in Neural Informa- fion Processing Systems 31. 2018
OpenAI, Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Józefowiez, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in- hand manipulation. CoRR, abs/1808. 00177,007777700 2018.
Chris Paxton, Yotam Barnoy, Kapil D. Katyal, Raman Arora, and Gregory D. Hager. Visual robot task planning. CoRR, abs/1 804.0006,, 2018.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. CORR, abs/1500.06825, 2015.
R Rubinstein and D Kroese. The Cross-Entropyy Method: A Unified Approach to Combinatorial Optimization, Monte-Card Carto Simulation and Machine Learning. 01 2004.
Fereshteh Sadeghi. Divis: Domain invariant visual servoing for collision-free goal reaching. CoRR,
abs/1902.0522 00999000.577..2. 2019.
Fereshtch Sadeghi, Alexander Toshev, Eric Jang, and Sergey Levine. Sim2real viewpoint invariant
visual servoing by recurrent control. In The IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, 2015.
Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. abs/1610.02610, 2016.
CORR,
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. CoRR, abs/ 111111944774444 2019. Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. CoRR, abs/ $04.00645, 2018.
Srivastava, F. Fang, I.. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible plannee-indepeedent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1 112:181-211 1999. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In IROS. pp. 5026-5033. IEEE, 2012. ISBN 178-1-4673-17333 173755.
Marc Toussaint, Kelsey R Allen, Kevin A Smith, and Josh B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning extended abstract, 2019. Sister Conference Best Paper Track Extended abstract of the R:SS'18 18 paper.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/18033 003.0000377 2018.
11

---

Published as a conference paper at ICLR 2020
Angelina Wang, Thanard Kurutach, Kara Liu, Pieter Abbeel, and Aviv Tamar. Learning robotic manipulation through visual planning and acting. CoRR, abs/1905.04411 2019.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Em- bed to control: A locally linear latent dynamics model for control from raw images. CoRR,
50.//506.07366, 2015.
W. J. Wilson, C. C. Williams Hulls, and G. S. Bell. Relative end-effector control using cartesian position based visual servoing. IEEE Transactions on Robotics and Automation, Oct 1996.
Annie Xie. Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical un- Using novel objects as tools with visual foresight. CoRR, abs/1904 01111994405533 2019.
derstanding:
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. CoRR, abs/1710.01813,
2017.
B. H. Yoshimi and P. K. Allen. Active, uncalibrated visual servoing. In Proceedings of the 1994 IEEE International Conference Robotics and Automation, pp. 156-161 vol.1, May 1994. Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control through distributional planning networks. CoRR, abs/1902.05542, 2019.
Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas A. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep 2018.
inforcement learning. CoRR,
12

---

Published as a conference paper at ICLR 2020
A METHOD DETAILS
A.1
VISUAL MPC
In HVF, evaluating the cost of potential subgoals, as well as actually taking actions given computed subgoals uses visual MPC. One step of visual MPC is described in Algorithm 1. Given the current state and goal state, MPC samples action trajectories of length H, then feeds them through the model fe. Then the cost of the output images are computed relative to the goal image, which is then used to sort the actions and refit the action distribution. After 5 iterations or convergence the best action is returned.
Algorithm MPC(fe, St,
Receive current state and goal state sy from environment Initialize N(u, = N(0, 1), cost function 85),
while (o² le - 3) or (iterations < 5) do
d1...., ap (p) ²2)
St. = fo (1, A1....
1, D= (8,+ 8y)
C(sL+H.D:&g)
Osorted = -0rt( [0) ap
Refit o(, to asorted
D
end while
Return (sorted [1], asorted []]
where = 200, D
A.2 PLANNING WITH SUBGOALS
In the main text describe how given, 8o and 8g. HVF produces subgoal images $1 82, 8K- Given these subgoals. planning with them is executed as follows. For an episode starting at so of maximum length T, the agent plans using visual MPC from So to $1 until some cost threshold C(st, 81) < for some maximum number of steps T Once this criteria is reached, the agent plans from its current state to the next subgoal 82, again until the cost threshold C(st, 82) < a or some maximum number of steps T*, and so until the agent is planning form 8, to the true goal 8g* at which point it simply plans until the environment returns that the task has been a success or the total horizon I" runs out.
A.3 GENERATIVE MODEL
The generative model we use is either a Variational Autoencoder ~ go(2) or a Conditional Varia- tional Autoencoder [2(2, so).
In the VAE, training is done by sampling images from the dataset, and each image is encoded using an image encoder into the mean and standard deviation of normal distribution of dimension 8. Then a sample from the distribution are decoded back to the original input image. This is trained with maximum likelihood loss well KI. penalty on the normal distribution restricting it to be a unit gaussian N (0, 1).
In the conditional VAE, training is done by sampling pairs of images from the dataset, specifically images from random episodes as well as the corresponding first image of that episode so- Then both Sn and s, are encoded using the same image encoder, which are then flattened and concatenated. This is then encoded into the mean and standard deviation of a normal distribution of dimension 8. sample from the resulting distribution is then fed through fully connected layers and reshaped into the output shape of the encoder, concatenated with the spatial embedding of the conditioned image from the encoder and then decoded. This is done to enable easier conditioning on the scene in the decoding. The resulting image is trained with maximum likelihood loss as well as a KL penalty on the normal distribution restricting to be a unit gaussian A A(00, 1).
In the Simulated Maze Navigation experiments, we use a Conditional VAE which encodes the 64x64 RGB image with 4 convolutional layers ([16, 32, 64, 128] filters, kernel size 3x3, stride 2), as well as 3 fully connect layers of size 256 for the generated image and 3 fully connected layers of size [512, 512, 256] for the conditioned image, and decoder with the same architecture as the encoder inverted. In the Simulated Desk Manipulation experiments, we use VAE which encodes the 64x64
13

---

Published as a conference paper at ICLR 2020
RGB image with 7 convolutional layers ((8, 16, 32, 32, 32, 64, 128] filters, kernel size 3, stride alternating between and 2). as well as a single fully connected layer of size 512 before mapping the latent distribution, which is decoded with the inverted encoder architecture. In the Real Robot Manipulation experiments we use a Conditional VAE which encodes the 48x64 RGB image with convolutional layers (18, 16, 32, 64, 64, 128, 128, 256, 512] filters, with kernel size 3, and stride sizes [I.III.I,III222222]] as well as 3 fully connect layers of size 256 for the generated image and fully connected layers of size [512, 512, 256] for the conditioned image, and decoder which the inverted architecture of the encoder. The three experiments use Adam optimizer with learning rate le-4, 1e-3, le-4 respectively.
A.4 DYNAMICS MODEL.
The dynamics model is an action conditioned video prediction model, stochastic variational video prediction (SV2P) (Babaeizadeh et al. 2017). See architecture below:
~Nij of)
1ar10e64
16x Tox54
(
6406663
Su5
Seb
Sxd
Ee5
Sid
Se5
5xi
Sxd
MASKS
32
sexdbica
actioms
dtatte
Toee bemed Images
GAeDAd
&.
Figure 7: Architecture of stochastic variational video prediction model (SV2P) (Babacizadeh et al. 201 used for dynamics model fo (taken from (Babacizadeh al. 2017) with permission). The architecture has two main suh-networks, one convolutional network which approximates the distrihution of latent values give all the frames, and recurrent convolutional network which predicts the pixels the next frame, given the previous frame, sampled latent, and action (if available). The code is open sourced in Tensor2Tensor (Vaswani et al. 2018) library.
For all experimental settings the dynamics models are trained for approximately 300K iterations.
B EXPERIMENT DETAILS
B. MAZE NAVIGATION
Data Collection:
For the maze navigation environment data is collected through random policy interaction. That is, for each action we sample uniformly in the delta æ, action space of the green block. We collect 10000 episodes, each with random initialization of walls, block, and goal. Fach episode contains 100 transitions. The images are of size 64x64x3.
Cost Function: In this experiment, the cost function C(s,, 8.) is simply the squared lg pixel dis- tance between the images, that is ,
Planning Parameters:
When planning in the maze navigation environment, MPC samples action trajectories of length H 5. Additionally, the cost of a sequence of 5 actions is the cost of the last of the subsequent frames, where the cost of each frame is the temporal cost C' described above. Lastly, we use 50 and T
in these experiments.
B.2 SIMULATED DESK MANIPULATION
Task Details: Door: The first task is reaching to and closing the sliding door. From the initialization position of the arm, needs reach around the door handle into the correct position, then slide the
14

---

Published as a conference paper at ICLR 2020
door shut. The position of the door and distractor blocks on the table are randomized each episode. The agent is given 50 timesteps to complete the task. 2 Block: The second task requires the agent to push two different blocks off the table, one located on the left end of the table and one located in the middle. It requires pushing first one block, then re positioning to push the other block off the table. The agent is given 50 timesteps complete the task. Door Block: The agent must both slide the door closed from a random position, as well as push a block off of the table. This requires both grasping and sliding the door from the previous task, as well as positioning the end effector behind the block to slide it off the table. The agent is given 100 timesteps to complete the task. Door + 2 Block: A combination of the 2 Block and Door + Block task. The agent must close the door and slide both blocks off the table within 100 timesteps
For the simulated desk manipulation environment data is again collected through random policy interaction. That is, for each action we sample uniformly in the delta x, v, action space of the robot end effector. We collect 10000 episodes, each with random initialization of the desk door/drawer and blocks. Each episode contains 100 transitions. The images are of size 64x64x3.
Data Collection:
Cost Function: In this experiment, the cost function C(ss, 8y) is simply the squared lg pixel dis- tance between the images, that is ||s: - 8
When planning the desk manipulation environment MPC samples action trajectories of length H = 15, which actually consists of 5 actions, each repeated 3 times. Addi- tionally, the cost of a sequence of 15 actions the l pixel cost of the last frame only. Note this is distinct from the maze environment. Lastly, we use T = 50 or 100 and T 20 in these experiments, depending on the task.
Planning Parameters:
B.3 REAL ROBOT MANIPULATION
Data: We use the BAIR robot manipulation dataset from (Ebert al., 2018a), which consists of roughly 15K trajectories split into a train/test split. We train Jo and Io on the train set and display qualitative results on the test set.
Cost Function: Like the desk manipulation set up, the cost function C(s;, 8y) is the squared & pixel distance between the images, that is ||s; - s,11
Planning Parameters: When planning in the desk manipulation environment MPC samples ac- tion trajectories of length H = 15, which actually consists of actions, each repeated 3 times. Additionally, the cost of a sequence of 15 actions is the fz pixel cost C' of the last frame only.
C ADDITIONAL RESULTS
C.I ADDITIONAL ABLATIONS
C.1.1 PLANNING HORIZON
# SG
0
2
In the maze enviromment, on the hard difficulty, we study how increasing the planning horizon of visual MPC impacts the benefit of using HVF (See Table Interestingly, we find that for longer planning horizons, performance does not necessarily improve (as a longer planning horizon constitutes harder search problem). This supports the idea that simply using longer planning horizon is not necessarily the solution to doing long- horiiorizon tasks. Further, we find that even with longer planning horizons, HVF outperforms standard visual foresight, but that performance with subgoals is worse than with subgoal.
5 Step Planning
0.33
0.47
0.54
10 Step Planning
0.46
0.55
0.37
15 Step Planning
0.31
0.39
0.24
Table 4: Planning Horizon. Compares success rates using visual MPC with planning horizon of a0 15. Using subgoals always performs best, but for larger planning horizons 2 subgoals can hurt performance.
Easy
Medium
Hard
VAE Latent &g Cost Pixel Eo Cost
0.62
0.49
0.23
0.93
0.68
0.33
Table 5: Latent vs Pixel Ea cost. Compares success rates using visual MPC with either the squared la pixel distance or squared & distance in the latent space of the VAE, The latent space cost is notably worse due to it providing weak signal at close distances.
C.1.2 VAE LATENT SPACE COST
To address the sparsity of pixel cost, we explore using distance in the latent space of the VAE as
15

---

Published as a conference paper at ICLR 2020
cost signal. However, find that this cost actually performs worse across all difficulties. In Table S we show the success rate of standard visual foresight for either pixel & cost or VAE latent space 22 cost. We find that the reason for the poor performance of the latent space cost is that provides close to zero cost anytime the green blocks are reasonably close together, leading to CEM often getting stuck close the goal but not actually the goal.
C.2
REAL ROBOT MANIPULATION EXAMPLES
1 Sub Goal
Sub Goals
Current State
Sub Goal
Goal State
Current State
Sub Goal
Sub Goal
Goal State
Figure 8: BAIR Dataset Additional Results.
C.3
TIME AGNOSTIC PREDICTION JAYARAMAN (AODOCCTAONJYAAAAAAA ET AL. 2019 IN THE MAZE NAVIGATION TASK
We observe that while TAP gets similar performance to HVF on the "Easy" and "Medium" cases, it has significantly lower performance the longest horizon "Hard" setting.
Additionally we see that Recursive TAP with subgoals has lower performance across all diffi- culties as the subgoals it outputs are very close current/goal state (See Figure 9..
Difficulty
Easy (1 SG)
Medium
Hard
TAP (1 SG)
0.89
0.82
0.35
TAP (2 SG)
0.91
0.68
0.28
HVF (1 SG)
0.96
0.80
0.47
HVF SG)
0.90
0.79
0.54
Table 6: TAP Maze Environment. Compares cess rates of TAP vs HVF on the maze environment. Computed over 100 randomized trials.
16

---

Published as a conference paper at ICLR 2020
Sub Goal
Sub Goals
Current State
Sub Goal
Goal State
Current State
Sub Goal
Sub Goal
Goal
Easy
Medium
Hard
Figure 9: Qualitative examples of TAP. Notice it is prone to predicting very close to the current or goal state for subgoals.
17


Reviews:
[{"review_id": "rylPuR2RKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, hierarchical visual foresight (HVF) that learns to break down the long horizon tasks into short horizon segments. It first generates the subgoals conditioned on the main goal. These subgoals are optimized to have meaningful states and used for planning. The experiments on Maze navigation, simulated desk manipulation, and real robot manipulation show significant performance gain over the planning method without subgoals and model-free RL. \n\nThe paper tackles a novel and challenging problem for the long-horizon tasks. Also, the paper is well written and easy to understand.\n\nQuestions/Concerns:\n\n- How important is the output quality of the video prediction model? \n\n- Finding subgoals that split into the optimal subtasks seems not easy. Are the learned subgoals actually reasonable for the final task? I can see that Fig. 6 has some examples of subgoals but is hard to recognize what are the subtasks. \n- Also, is there a chance that subgoals are randomly selected? Could you verify that subgoals are consistent across different initialization/runs? \n- Could you provide us any failure cases if there's any and explain us why this happens? \n\n- The number of subgoals needs to be fixed. I assume this number depends on how complex of the task is. It is surprising that only 2 subgoals were enough for the BAIR Robot push dataset (Tab. 1). The authors commented that \"the sampling budget allocated for subgoal optimization is likely insufficient to find a large sequence of subgoals.\". Will it actually be solved by increasing the sampling? Or is it possible that the dataset is too simple or simply finding optimal subgoals is hard.  Could authors comment more about this issue?\n- Also, how can you find the right number of subgoals? \n\n- The computational requirement is one of the weaknesses. Please provide the comparison of how much is the computational cost of the proposed method compared to other ones such as VF (without subgaols), TAP, and RIG. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Bygq9J7zKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a hierarchical extension to existing work in vision-based model predictive control. Here, a hierarchical model is optimised to find sub-goals that minimise the planning cost (bottleneck states), so as to allow for improved planning to goal states expressed in higher dimensional state spaces. As expected, results show that this hierarchy improves tasks execution success rates. \n\nThe paper is clearly written, although many sub-components of the architecture are described in other articles. While I appreciate that the work is incremental, and that it relies on a number of previously modules, this brevity and the architectural complexity means that the work will be challenging to reproduce or adapt to different tasks. \n\nNevertheless, I believe the idea is interesting, and the paper does achieve its stated goals of allowing for entirely self-supervised learning without motion primitives, demonstrations and rewards, so makes a useful contribution here.  Compositional planning through sub-goals is a good idea, and this paper adds a useful mechanism to identify suitable sub-goals in a latent space and plan through these.\n\nHowever, I don't believe that this approach is in any way practically feasible at present, and the paper makes a number of hyperbolic claims that should be toned down to give a more balanced perspective (I appreciate that the authors attempted to do so in their limitations section, but this needs to be done throughout the paper).\n\nSpecifically:\n\n1.) Please avoid the use of relative improvement percentages (200% performance improvements in the abstract - the 20% absolute performance is strong enough to stand alone).\n\n2.) The paper oversimplifies the importance of exploration and data collection in the proposed approach. The paper states that a uniform random policy in the continuous action space of the agent works well, and offsets the data collection to \"any exploration policy\", but this is a key requirement if the proposed approach is to be useful. At present the method seems to be relying on typewriting monkeys writing Shakespeare as a precursor to control. An exploration policy using identified sub-goal of interest may be a particularly valuable extension in future work, but for now the paper glosses over this aspect.\n\n3.) The use of the term \"long-horizon tasks\" and \"manipulation\". In line with point 2, my personal feeling is that any task or state that can conceivably be explored and accessed by uniformly sampling from a continuous action space does not qualify as a long-horizon task. From the results and videos, it seems that all of the tasks could be solved by following a single trajectory in 3D space, ie. swing left to knock objects off the table, then right to close the door. Terming this \"long-horizon\" is a bit of a stretch, as is the notion that flopping about a table bashing into objects counts as \"manipulation\". Motion planning and trajectory following are not long-term horizon manipulation tasks, solving towers of Hanoi is. Moreover, I appreciate that experiments were conducted on real robot datasets, but this seems to be more of an exercise in latent space anthropomorphism than practical evidence of a feasible control policy.\n\n4.) Limitations regarding latent space expressiveness.  As mentioned in the limitations section of the paper, the proposed approach is heavily reliant on a latent space that fully captures the scene and can roll-out future states sensibly. This is an extremely challenging problem, and one which appears to affect the proposed approach, for example, in the accompanying video (3:.40 - 3:59), the desired goal state contains two objects standing on the table and a closed door, but the policy only closes the door, while objects are knocked over. This seems to indicate that the latent space and planning is unable to learn good object embeddings and spatial representations, and that the type of task that can be solved is along the lines of move forward and to the right when the black blob is on the left of the image, or move down to the left, when there are some coloured blobs on the table. \n\n5.) Limitations around goal state representations . Following on from 4, the identification of important aspects in a given image or latent space represents a major challenge to the proposed approach. Given an image of a desired state, how can the proposed approach identify which elements in the scene are required for success, and which are simply distractors? At present, I see no mechanism by which this could ever be learned in a self-play setting or through an image-based goal state. Eg. what if the image indicated that I want the two objects to be in a specific position, and I never cared about the state of the door?\n\nDespite these limitations, the granularity of the solution is fine for a proof of concept work like this, and is itself a commendable achievement. Unfortunately, the paper's choice of language, relying on terms like \"long term horizon\" and \"manipulation\" for simple reaching and pushing tasks exaggerates the state of robot learning, and is potentially misleading to those less familiar with the field.\n\nDespite my gripes, the paper definitely meets the ICLR threshold of \"accept if you'd share it with a colleague\", and I believe it is is a useful approach to sub-goal identification and a nice piece of work, so I recommend acceptance.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method that uses subgoals for planning when using video prediction. The reviewers thought that the paper was clearly written and interesting. The reviewer questions and concerns were mostly addressed during the discussion phase, and the reviewers are in agreement that the paper should be accepted.
Decision: Accept (Poster)