Learning what you can do before doing anything

1 INTRODUCTION . Agents behaving in real-world environments rely on perception to judge what actions they can take and what effect these actions will have . Purely perceptual learning may play an important role in how these action representations are acquired and used . In this work , we focus on the problem of learning an agent ’ s action space from unlabeled visual observations . To see the usefulness of this strategy , consider an infant that is first learning to walk . From around 10 months of age , infants rapidly progress from crawling , to irregular gaits with frequent falling , and finally to reliable locomotion ( Adolph et al . ( 2012 ) ) . But before they first attempt to walk , infants have extensive sensory exposure to adults walking . Unsupervised learning from sensory experience of this type appears to play a critical role in how humans acquire representations of actions before they can reliably reproduce the corresponding behaviour ( Ullman et al . ( 2012 ) ) . Infants need to relate the set of motor primitives they can generate to the action spaces exploited by adults ( Dominici et al . ( 2011 ) ) , and a representation acquired by observation may allow an infant to more efficiently learn to produce natural , goal-directed walking behavior . Reinforcement learning ( RL ) provides an alternative to the ( passive ) unsupervised learning approach as it implicitly discovers an agent ’ s action space and the consequences of its actions . Recent breakthroughs in model-free and model-based RL suggest that end-to-end training can be used to learn mappings between sensory input and actions ( Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2016 ) ; Levine et al . ( 2016 ) ; Finn & Levine ( 2017 ) ; Schulman et al . ( 2015 ) ) . However , these methods require active observations and the sensorimotor mappings learned in this way can not be easily generalized to new agents with different control interfaces . Methods for sensorimotor learning from purely visual ∗Equal contribution . Ordering determined by a coin flip . 1Project website : https : //daniilidis-group.github.io/learned_action_spaces data may facilitate learning where action information is not available , such as when using video data collected from the Internet . Such methods may also be useful for imitation learning , where ground truth actions are often hard or impossible to collect other than by visual observation ( Finn et al . ( 2017 ) ; Pathak et al . ( 2018 ) ) . More generally , learning from passive observations may make it easier to reuse action representations between systems with different effectors and goals . The representations learned by unsupervised methods are invariant to these choices because the model does not have access to motor commands or goals during training . In this work , we evaluate the proposal that learning what you can do before doing anything can lead to action space representations that make subsequent learning more efficient . To this end , we develop a model that learns to represent an agent ’ s action space given only unlabeled videos of the agent . The resulting representation enables direct planning in the latent space . Given a small number of action-labeled sequences we can execute the plan by learning a simple mapping from latent action representations to the agent ’ s controls . This representation may be analogous to those in the parietal and premotor areas of cortex , which include populations of neurons that represent the structure of actions produced both by the self and by others ( Rizzolatti et al . ( 1996 ) ; Romo et al . ( 2004 ) ) and that are critical for reliably producing flexible , voluntary motor control ( see Kandel et al . ( 2012 ) , Chapter 38 ) . In the brain , representations of this kind could plausibly be learned using specialized loss functions ( Marblestone et al . ( 2016 ) ) whose effect is to induce the prior needed to determine the structure of actions in observation data . In contrast to most approaches to unsupervised learning of dynamics , which focus on learning the statistical structure of the environment , we focus on disentangling action information from the instantaneous state of the environment ( Fig . 1 ) . We base our work on recent stochastic video prediction methods ( Babaeizadeh et al . ( 2018 ) ; Denton & Fergus ( 2018 ) ; Lee et al . ( 2018 ) ) and impose two properties on the latent representation . First , we train the representation to be minimal , i.e . containing minimal information about the current world state . This forces the representation to focus on dynamic properties of the sensory input . A similar objective has been used in previous work to constrain the capacity of video prediction models ( Denton & Fergus ( 2018 ) ) . Second , we train the representation to be composable by introducing a novel loss term that enforces that the cumulative effect of a sequence of actions can be computed from the individual actions ’ representations ( Fig . 1 , left ) . Composability encourages disentangling : as a composed representation does not have access to the static content of the intermediate frames , a representation is composable only if the individual action representations are disentangled from the static content . Taken together , these two properties lead to a representation of sensory dynamics that captures the structure of the agent ’ s actions . We make the following three contributions . First , we introduce a method for unsupervised learning of an agent ’ s action space by training the latent representation of a stochastic video prediction model for the desiderata of minimality and composability . Second , we show that our method learns a representation of actions that is independent of scene content and visual characteristics on ( i ) a simulated robot with one degree of freedom and ( ii ) the BAIR robot pushing dataset ( Ebert et al . ( 2017 ) ) . Finally , we demonstrate that the learned representation can be used for action-conditioned video prediction and planning in the learned action space , while requiring orders of magnitude fewer action-labeled videos than extant supervised methods . 2 RELATED WORK . Learning structured and minimal representations . Several groups have recently shown how an adaptation of the variational autoencoder ( VAE , Kingma & Welling ( 2014 ) ; Rezende et al . ( 2014 ) ) can be used to learn representations that are minimal in the information-theoretic sense . Alemi et al . ( 2017 ) showed that the Information Bottleneck ( IB ) objective function ( Tishby et al . ( 1999 ) ; Shwartz-Ziv & Tishby ( 2017 ) ) can be optimized with a variational approximation that takes the form of the VAE objective with an additional weighting hyperparameter . In parallel , Higgins et al . ( 2017 ) showed that a similar formulation can be used to produce disentangled representations . The connection between disentaglement and minimality of representations was further clarified by Burgess et al . ( 2018 ) . In this work , we apply the IB principle to temporal models to enforce minimality of the representation . Several groups have proposed methods to learn disentangled representations of static content and pose from video ( Denton & Birodkar ( 2017 ) ; Tulyakov et al . ( 2018 ) ) . Jaegle et al . ( 2018 ) learn a motion representation by enforcing that the motion acts on video frames as a group-theoretic action . In contrast , we seek a representation that disentangles the motion from the static pose . Thomas et al . ( 2017 ) attempt to learn a disentangled representation of controllable factors of variation . While the goals of their work are similar to ours , their model relies on active learning and requires an embodied agent with access to the environment . In contrast , our model learns factors of variation purely from passive temporal visual observations , and thus can be applied even if access to the environment is costly or impossible . Unsupervised learning with video data . Several recent works have exploited temporal information for representation learning . Srivastava et al . ( 2015 ) used the Long Short-Term Memory ( LSTM , Hochreiter & Schmidhuber ( 1997 ) ) recurrent neural network architecture to predict future frames and showed that the learned representation was useful for action recognition . Vondrick et al . ( 2016 ) showed that architectures using convolutional neural networks ( CNNs ) can be used to predict actions and objects several seconds into the future . Recently , work such as Finn et al . ( 2016 ) ; Villegas et al . ( 2017 ) ; Denton & Birodkar ( 2017 ) has proposed various modifications to the convolutional LSTM architecture ( Xingjian et al . ( 2015 ) ) for the task of video prediction and shown that the resulting representations are useful for a variety of tasks . Others have explored applications of video prediction models to RL and control ( Weber et al . ( 2017 ) ; Ha & Schmidhuber ( 2018 ) ; Wayne et al . ( 2018 ) ) . Chiappa et al . ( 2017 ) and Oh et al . ( 2015 ) propose models that predict the consequences of actions taken by an agent given its control output . Similar models have been used to control a robotic arm ( Agrawal et al . ( 2016 ) ; Finn & Levine ( 2017 ) ; Ebert et al . ( 2017 ) ) . The focus of this work is on learning action-conditioned predictive models . In contrast , our focus is on the unsupervised discovery of the space of possible actions from video data . Our model is inspired by methods for stochastic video prediction that , given a sequence of past frames , capture the multimodal distribution of future images ( Goroshin et al . ( 2015 ) ; Henaff et al . ( 2017 ) ) . We use the recently proposed recurrent latent variable models based on the variational autoencoder ( Babaeizadeh et al . ( 2018 ) ; Denton & Fergus ( 2018 ) ; Lee et al . ( 2018 ) ) . We develop these methods and propose a novel approach to unsupervised representation learning designed to capture an agent ’ s action space . Sensorimotor abstractions for behavior . There is a long history of work developing sensorimotor representations for applications in reinforcement learning and robotics . Previous work in this domain has primarily focused on introducing hand-crafted abstractions and hierarchies to make sensorimotor mappings more generalizable . Methods for aggregating low-level controls into higher- level representations on which planning and RL can be performed are well-studied in the robotics literature : notable examples include the options framework for hierarchical RL ( Sutton et al . ( 1999 ) ; Bacon et al . ( 2017 ) ) , dynamic motion primitives for manipulation ( Schaal et al . ( 2005 ) ; Schaal ( 2006 ) ; Niekum et al . ( 2015 ) ) , and recent work that abstracts a learned policy away from low-level control and perception to ease simulation-to-real transfer ( Clavera & Abbeel ( 2017 ) ; Müller et al . ( 2018 ) ) . Other work has learned to separate robot-instance specific controls from task-related skills through modular policies , but this work does not enforce any structure onto the intermediate representation and requires extensive interaction with the environment ( Devin et al . ( 2017 ) ) .

Summary: The paper proposes a Variational IB based approach to learn action representations directly from video of actions being taken. The basic goal of the work is to disentangle the dynamic parts of the scene in the video from the static parts and only capture those dynamic parts in the representation. Further, a key property of these learned representations is that they contain compositional structure of actions so as to their cumulative effects. The outcome of such a method is better efficiency of the subsequent learning methods while requiring lesser amount of action label videos.