Learning what you can do before doing anything

1 INTRODUCTION . Agents behaving in real-world environments rely on perception to judge what actions they can take and what effect these actions will have . Purely perceptual learning may play an important role in how these action representations are acquired and used . In this work , we focus on the problem of learning an agent ’ s action space from unlabeled visual observations . To see the usefulness of this strategy , consider an infant that is first learning to walk . From around 10 months of age , infants rapidly progress from crawling , to irregular gaits with frequent falling , and finally to reliable locomotion ( Adolph et al . ( 2012 ) ) . But before they first attempt to walk , infants have extensive sensory exposure to adults walking . Unsupervised learning from sensory experience of this type appears to play a critical role in how humans acquire representations of actions before they can reliably reproduce the corresponding behaviour ( Ullman et al . ( 2012 ) ) . Infants need to relate the set of motor primitives they can generate to the action spaces exploited by adults ( Dominici et al . ( 2011 ) ) , and a representation acquired by observation may allow an infant to more efficiently learn to produce natural , goal-directed walking behavior . Reinforcement learning ( RL ) provides an alternative to the ( passive ) unsupervised learning approach as it implicitly discovers an agent ’ s action space and the consequences of its actions . Recent breakthroughs in model-free and model-based RL suggest that end-to-end training can be used to learn mappings between sensory input and actions ( Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2016 ) ; Levine et al . ( 2016 ) ; Finn & Levine ( 2017 ) ; Schulman et al . ( 2015 ) ) . However , these methods require active observations and the sensorimotor mappings learned in this way can not be easily generalized to new agents with different control interfaces . Methods for sensorimotor learning from purely visual ∗Equal contribution . Ordering determined by a coin flip . 1Project website : https : //daniilidis-group.github.io/learned_action_spaces data may facilitate learning where action information is not available , such as when using video data collected from the Internet . Such methods may also be useful for imitation learning , where ground truth actions are often hard or impossible to collect other than by visual observation ( Finn et al . ( 2017 ) ; Pathak et al . ( 2018 ) ) . More generally , learning from passive observations may make it easier to reuse action representations between systems with different effectors and goals . The representations learned by unsupervised methods are invariant to these choices because the model does not have access to motor commands or goals during training . In this work , we evaluate the proposal that learning what you can do before doing anything can lead to action space representations that make subsequent learning more efficient . To this end , we develop a model that learns to represent an agent ’ s action space given only unlabeled videos of the agent . The resulting representation enables direct planning in the latent space . Given a small number of action-labeled sequences we can execute the plan by learning a simple mapping from latent action representations to the agent ’ s controls . This representation may be analogous to those in the parietal and premotor areas of cortex , which include populations of neurons that represent the structure of actions produced both by the self and by others ( Rizzolatti et al . ( 1996 ) ; Romo et al . ( 2004 ) ) and that are critical for reliably producing flexible , voluntary motor control ( see Kandel et al . ( 2012 ) , Chapter 38 ) . In the brain , representations of this kind could plausibly be learned using specialized loss functions ( Marblestone et al . ( 2016 ) ) whose effect is to induce the prior needed to determine the structure of actions in observation data . In contrast to most approaches to unsupervised learning of dynamics , which focus on learning the statistical structure of the environment , we focus on disentangling action information from the instantaneous state of the environment ( Fig . 1 ) . We base our work on recent stochastic video prediction methods ( Babaeizadeh et al . ( 2018 ) ; Denton & Fergus ( 2018 ) ; Lee et al . ( 2018 ) ) and impose two properties on the latent representation . First , we train the representation to be minimal , i.e . containing minimal information about the current world state . This forces the representation to focus on dynamic properties of the sensory input . A similar objective has been used in previous work to constrain the capacity of video prediction models ( Denton & Fergus ( 2018 ) ) . Second , we train the representation to be composable by introducing a novel loss term that enforces that the cumulative effect of a sequence of actions can be computed from the individual actions ’ representations ( Fig . 1 , left ) . Composability encourages disentangling : as a composed representation does not have access to the static content of the intermediate frames , a representation is composable only if the individual action representations are disentangled from the static content . Taken together , these two properties lead to a representation of sensory dynamics that captures the structure of the agent ’ s actions . We make the following three contributions . First , we introduce a method for unsupervised learning of an agent ’ s action space by training the latent representation of a stochastic video prediction model for the desiderata of minimality and composability . Second , we show that our method learns a representation of actions that is independent of scene content and visual characteristics on ( i ) a simulated robot with one degree of freedom and ( ii ) the BAIR robot pushing dataset ( Ebert et al . ( 2017 ) ) . Finally , we demonstrate that the learned representation can be used for action-conditioned video prediction and planning in the learned action space , while requiring orders of magnitude fewer action-labeled videos than extant supervised methods . 2 RELATED WORK . Learning structured and minimal representations . Several groups have recently shown how an adaptation of the variational autoencoder ( VAE , Kingma & Welling ( 2014 ) ; Rezende et al . ( 2014 ) ) can be used to learn representations that are minimal in the information-theoretic sense . Alemi et al . ( 2017 ) showed that the Information Bottleneck ( IB ) objective function ( Tishby et al . ( 1999 ) ; Shwartz-Ziv & Tishby ( 2017 ) ) can be optimized with a variational approximation that takes the form of the VAE objective with an additional weighting hyperparameter . In parallel , Higgins et al . ( 2017 ) showed that a similar formulation can be used to produce disentangled representations . The connection between disentaglement and minimality of representations was further clarified by Burgess et al . ( 2018 ) . In this work , we apply the IB principle to temporal models to enforce minimality of the representation . Several groups have proposed methods to learn disentangled representations of static content and pose from video ( Denton & Birodkar ( 2017 ) ; Tulyakov et al . ( 2018 ) ) . Jaegle et al . ( 2018 ) learn a motion representation by enforcing that the motion acts on video frames as a group-theoretic action . In contrast , we seek a representation that disentangles the motion from the static pose . Thomas et al . ( 2017 ) attempt to learn a disentangled representation of controllable factors of variation . While the goals of their work are similar to ours , their model relies on active learning and requires an embodied agent with access to the environment . In contrast , our model learns factors of variation purely from passive temporal visual observations , and thus can be applied even if access to the environment is costly or impossible . Unsupervised learning with video data . Several recent works have exploited temporal information for representation learning . Srivastava et al . ( 2015 ) used the Long Short-Term Memory ( LSTM , Hochreiter & Schmidhuber ( 1997 ) ) recurrent neural network architecture to predict future frames and showed that the learned representation was useful for action recognition . Vondrick et al . ( 2016 ) showed that architectures using convolutional neural networks ( CNNs ) can be used to predict actions and objects several seconds into the future . Recently , work such as Finn et al . ( 2016 ) ; Villegas et al . ( 2017 ) ; Denton & Birodkar ( 2017 ) has proposed various modifications to the convolutional LSTM architecture ( Xingjian et al . ( 2015 ) ) for the task of video prediction and shown that the resulting representations are useful for a variety of tasks . Others have explored applications of video prediction models to RL and control ( Weber et al . ( 2017 ) ; Ha & Schmidhuber ( 2018 ) ; Wayne et al . ( 2018 ) ) . Chiappa et al . ( 2017 ) and Oh et al . ( 2015 ) propose models that predict the consequences of actions taken by an agent given its control output . Similar models have been used to control a robotic arm ( Agrawal et al . ( 2016 ) ; Finn & Levine ( 2017 ) ; Ebert et al . ( 2017 ) ) . The focus of this work is on learning action-conditioned predictive models . In contrast , our focus is on the unsupervised discovery of the space of possible actions from video data . Our model is inspired by methods for stochastic video prediction that , given a sequence of past frames , capture the multimodal distribution of future images ( Goroshin et al . ( 2015 ) ; Henaff et al . ( 2017 ) ) . We use the recently proposed recurrent latent variable models based on the variational autoencoder ( Babaeizadeh et al . ( 2018 ) ; Denton & Fergus ( 2018 ) ; Lee et al . ( 2018 ) ) . We develop these methods and propose a novel approach to unsupervised representation learning designed to capture an agent ’ s action space . Sensorimotor abstractions for behavior . There is a long history of work developing sensorimotor representations for applications in reinforcement learning and robotics . Previous work in this domain has primarily focused on introducing hand-crafted abstractions and hierarchies to make sensorimotor mappings more generalizable . Methods for aggregating low-level controls into higher- level representations on which planning and RL can be performed are well-studied in the robotics literature : notable examples include the options framework for hierarchical RL ( Sutton et al . ( 1999 ) ; Bacon et al . ( 2017 ) ) , dynamic motion primitives for manipulation ( Schaal et al . ( 2005 ) ; Schaal ( 2006 ) ; Niekum et al . ( 2015 ) ) , and recent work that abstracts a learned policy away from low-level control and perception to ease simulation-to-real transfer ( Clavera & Abbeel ( 2017 ) ; Müller et al . ( 2018 ) ) . Other work has learned to separate robot-instance specific controls from task-related skills through modular policies , but this work does not enforce any structure onto the intermediate representation and requires extensive interaction with the environment ( Devin et al . ( 2017 ) ) .

Summary: The authors propose a way to learn models that predict what will happen next in scenarios where action-labels are not available in abundance. The agents extend previous work by proposing a compositional latent-variable model. Results are shown on BAIR (robot pushing objects) and simulated reacher datasets. The results indicate that it is possible to learn a bijective mapping between the latent variables inferred from a pair of images and the action executed between the observations of the two images.