Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation

1 INTRODUCTION . Developing robotic systems that can complete long horizon visual control tasks , while generalizing to novel scenes and objectives , remains an unsolved and challenging problem . Generalization to unseen objects and scenes requires robots to be trained across diverse environments , meaning that detailed supervision during data collection in not practical to provide . Furthermore , reasoning over long-horizon tasks introduces two additional major challenges . First , the robot must handle large amounts of uncertainty as the horizon increases . And second , the robot must identify how to reach distant goals when only provided with the final goal state , a sparse indication of the task , as opposed to a shaped cost that implicitly encodes how to get there . In this work , we aim to develop a method that can start to address these challenges , leveraging self-supervised models learned using only unlabeled data , to solve novel temporally-extended tasks . Model-based reinforcement learning has shown promise in generalizing to novel objects and tasks , as learned dynamics models have been shown to generalize to new objects ( Finn & Levine , 2016 ; Ebert et al. , 2018b ) , and can be used in conjunction with planning to reach goals unseen during training . However , planning to reach temporally distant goals is difficult . As the planning horizon increases model error compounds , and the cost function often provides only a noisy or sparse signal of the objective . Both of these challenges are exacerbated when planning in visual space . In this work , the key insight that we leverage is that while model error and sparse cost signals can make long horizon planning difficult , we can mitigate these issues by learning to break down longhorizon tasks into short horizon segments . Consider , for example , the task of opening a drawer and putting a book in it , given supervision only in the form of the final image of the open drawer containing the book . The goal image provides nearly no useful cost signal until the last stage of the task , and model predictions are likely to become inaccurate beyond the first stage of the task . However , if we can generate a sequence of good subgoals , such as ( 1 ) the robot arm grasping the †Work completed at Google Brain Videos and code are available at : https : //sites.google.com/stanford.edu/hvf drawer handle , ( 2 ) the open drawer , and ( 3 ) the robot arm reaching for the book , planning from the initial state to ( 1 ) , from ( 1 ) to ( 2 ) , from ( 2 ) to ( 3 ) , and from ( 3 ) to the final goal , the problem becomes substantially easier . The subgoals break the problem into short horizon subsegments each with some useful cost signal coming from the next subgoal image . Our main contribution is a self-supervised hierarchical planning framework , hierarchical visual foresight ( HVF ) , which combines generative models of images and model predictive control to decompose a long-horizon visual task into a sequence of subgoals . In particular , we propose optimizing over subgoals such that the resulting task subsegments have low expected planning cost . However , in the case of visual planning , optimizing over subgoals corresponds to optimizing within the space of natural images . To address this challenge , we train a generative latent variable model over images from the robot ’ s environment and optimize over subgoals in the latent space of this model . This allows us to optimize over the manifold of images with only a small number of optimization variables . When combined with visual model predictive control , we observe that this subgoal optimization naturally identifies semantically meaningful states in a long horizon tasks as subgoals , and that when using these subgoals during planning , we achieve significantly higher success rates on long horizon , multi-stage visual tasks . Furthermore , since our method outputs subgoals conditioned on a goal image , we can use the same model and approach to plan to solve many different long-horizon tasks , even with previously unseen objects . We first demonstrate our approach in simulation on a continuous control navigation task with tight bottlenecks , and then evaluate on a set of four different multi-stage object manipulation tasks in a simulated desk environment , which require interacting with up to 3 different objects . In the challenging desk environment , we find that our method yields at least a 20 % absolute performance improvement over prior approaches , including model-free reinforcement learning and a state of the art subgoal identification method . Finally , we show that our approach generates realistic subgoals on real robot manipulation data . 2 RELATED WORK . Developing robots that can execute complex behaviours from only pixel inputs has been a well studied problem , for example with visual servoing ( Mohta et al. , 2014 ; Espiau et al. , 1992 ; Wilson et al. , 1996 ; Yoshimi & Allen , 1994 ; Jagersand et al. , 1997 ; Lampe & Riedmiller , 2013 ; Sadeghi et al. , 2018 ; Sadeghi , 2019 ) . Recently , reinforcement learning has shown promise in completing complex tasks from pixels ( Ghadirzadeh et al. , 2017 ; Levine et al. , 2015 ; Kalashnikov et al. , 2018 ; Lange et al. , 2012 ; OpenAI et al. , 2018 ; Schenck & Fox , 2016 ; Matas et al. , 2018 ; James et al. , 2017 ; Singh et al. , 2019 ) , including in goal-conditioned settings ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Sadeghi et al. , 2018 ; Sadeghi , 2019 ; Nair et al. , 2018a ) . While model-free RL approaches have illustrated the ability to generalize to new objects ( Kalashnikov et al. , 2018 ) and learn tasks such as grasping and pushing through self-supervision ( Pinto & Gupta , 2015 ; Zeng et al. , 2018 ) , pure model-free approaches generally lack the ability to explicitly reason over temporally-extended plans , making them ill-suited for the problem of learning long-horizon tasks with limited supervision . Video prediction and planning have also shown promise in enabling robots to complete a diverse set of visuomotor tasks while generalizing to novel objects ( Finn & Levine , 2016 ; Kalchbrenner et al. , 2016 ; Boots et al. , 2014 ; Byravan & Fox , 2016 ) . Since then , a number of video prediction frameworks have been developed specifically for robotics ( Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ; Ebert et al. , 2017 ) , which combined with planning have been used to complete diverse behaviors ( Nair et al. , 2018b ; Ebert et al. , 2018b ; Paxton et al. , 2018 ; Xie et al. , 2019 ) . However , these approaches still struggle with long horizon tasks , which we specifically focus on . One approach to handle long horizon tasks is to add compositional structure to policies , either from demonstrations ( Krishnan et al. , 2017 ; Fox et al. , 2018 ) , with manually-specified primitives ( Xu et al. , 2017 ; Huang et al. , 2018 ) , learned temporal abstractions ( Neitz et al. , 2018 ) , or through model-free reinforcement learning ( Sutton et al. , 1999 ; Barto & Mahadevan , 2003 ; Bacon et al. , 2016 ; Nachum et al. , 2018 ; Levy et al. , 2019 ) . These works have studied such hierarchy in grid worlds ( Bacon et al. , 2016 ) and simulated control tasks ( Nachum et al. , 2018 ; Eysenbach et al. , 2018 ; Levy et al. , 2019 ) with known reward functions . In contrast , we study how to incorporate compositional structure in learned model-based planning with video prediction models . Our approach is entirely self-supervised , without motion primitives , demonstrations , or shaped rewards , and scales to vision-based manipulation tasks . Classical planning methods have been successful in solving long-horizon tasks ( LaValle , 2006 ; Choset et al. , 2005 ) , but make restrictive assumptions about the state space and reachability between states , limiting their applicability to complex visual manipulation tasks . Similarly , completing long horizon tasks has also been explored with symbolic models ( Toussaint et al. , 2019 ) and Task and Motion Planning ( TAMP ) ( Kaelbling & Lozano-Perez , 2011 ; Srivastava et al. , 2014 ) . However , unlike these approaches our method requires no additional knowledge about the objects in the scene nor any predefined symbolic states . Recently , there have been several works that have explored planning in learned latent spaces ( Kurutach et al. , 2018 ; Ichter & Pavone , 2018 ; Watter et al. , 2015 ; Srinivas et al. , 2018 ) . This has enabled planning in higher dimensional spaces , however these methods still struggle with long-horizon tasks . Furthermore , our hierarchical planning framework is agnostic to state space , and could directly operate in one of the above latent spaces . A number of recent works have explored reaching novel goals using only self-supervision ( Finn & Levine , 2016 ; Eysenbach et al. , 2019 ; Kurutach et al. , 2018 ; Wang et al. , 2019 ; Jayaraman et al. , 2019 ; Nair et al. , 2018a ) . In particular , time-agnostic prediction ( TAP ) ( Jayaraman et al. , 2019 ) aims to identify bottlenecks in long-horizon visual tasks , while other prior works ( Nair et al. , 2018a ; Finn & Levine , 2016 ) reach novel goals using model-free or model-based RL . We compare to all three of these methods in Section 5 and find that HVF significantly outperforms all of them . 3 PRELIMINARIES . We formalize our problem setting as a goal-conditioned Markov decision process ( MDP ) defined by the tuple ( S , A , p , G , C , λ ) where s ∈ S is the state space , which in our case corresponds to images , a ∈ A is the action space , p ( st+1|st , at ) governs the environment dynamics , G ⊂ S represents the set of goals which is a subset of possible states , C ( st , sg ) represents the cost of being in state st ∈ S given that the goal is sg ∈ G , and λ is the discount factor . In practice , acquiring cost functions that accurately reflect the distance between two images is a challenging problem ( Yu et al. , 2019 ) . We make no assumptions about having a shaped cost , assuming the simple yet sparse distance metric of ` 2 distance in pixel space in all of our experiments . Approaches that aim to recover more shaped visual cost functions are complementary to the contributions of this work . In visual foresight , or equivalently , visual MPC ( Finn & Levine , 2016 ; Ebert et al. , 2018b ) , the robot collects a data set of random interactions [ ( s1 , a1 ) , ( s2 , a2 ) , ... , ( sT , aT ) ] from a pre-determined policy . This dataset is used to learn a model of dynamics fθ ( st+1 , st+2 , ... , st+h|st , at , at+1 , ... , at+h−1 ) through maximum likelihood supervised learning . Note the states are camera images , and thus fθ is an action-conditioned video prediction model . Once the model is trained , the robot is given some objective and plans a sequence of actions that optimize the objective via the cross entropy method ( CEM ) ( Rubinstein & Kroese , 2004 ) . In this work , we will assume the objective is specified in the form of an image of the goal sg , while CEM aims to find a sequence of actions that minimize the cost C between the predicted future frames from fθ and the goal image . While standard visual foresight struggles with long-horizon tasks due to uncertainty in fθ as the prediction horizon h increases and sparse C for CEM to optimize , in the next section we describe how our proposed approach uses subgoal generation to mitigate these issues .

Summary: This paper proposes a method, hierarchical visual foresight (HVF) that learns to break down the long horizon tasks into short horizon segments. It first generates the subgoals conditioned on the main goal. These subgoals are optimized to have meaningful states and used for planning. The experiments on Maze navigation, simulated desk manipulation, and real robot manipulation show significant performance gain over the planning method without subgoals and model-free RL.