Large-Scale Study of Curiosity-Driven Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has emerged as a popular method for training agents to perform complex tasks . In RL , the agent ’ s policy is trained by maximizing a reward function that is designed to align with the task . The rewards are extrinsic to the agent and specific to the environment they are defined for . Most of the success in RL has been achieved when this reward function is dense and well-shaped , e.g. , a running “ score ” in a video game ( Mnih et al. , 2015 ) . However , designing a wellshaped reward function is a notoriously challenging engineering problem . An alternative to “ shaping ” an extrinsic reward is to supplement it with dense intrinsic rewards ( Oudeyer & Kaplan , 2009 ) , that is , rewards that are generated by the agent itself . Examples of intrinsic reward include “ curiosity ” ( Mohamed & Rezende , 2015 ; Schmidhuber , 1991b ; Singh et al. , 2005 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) which uses prediction error as reward signal , and “ visitation counts ” ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Poupart et al. , 2006 ; Lopes et al. , 2012 ) which discourages the agent from revisiting the same states . The idea is that these intrinsic rewards will bridge the gaps between sparse extrinsic rewards by guiding the agent to efficiently explore the environment to find the next extrinsic reward . But what about scenarios with no extrinsic reward at all ? This is not as strange as it sounds . Developmental psychologists talk about intrinsic motivation ( i.e. , curiosity ) as the primary driver in the early stages of development ( Smith & Gasser , 2005 ; Ryan , 2000 ) : babies appear to employ goal-less exploration to learn skills that will be useful later on in life . There are plenty of other examples , from playing Minecraft to visiting your local zoo , where no extrinsic rewards are required . Indeed , there is evidence that pre-training an agent on a given environment using only intrinsic rewards allows it to learn much faster when fine-tuned to a novel task in a novel environment ( Pathak et al. , 2017 ; 2018 ) . Yet , so far , there has been no systematic study of learning with only intrinsic rewards . In this paper , we perform a large-scale empirical study of agents driven purely by intrinsic rewards across a range of diverse simulated environments . In particular , we choose the dynamics-based curiosity model of intrinsic reward presented in Pathak et al . ( 2017 ) because it is scalable and trivially parallelizable , making it ideal for large-scale experimentation . The central idea is to represent intrinsic reward as the error in predicting the consequence of the agent ’ s action given its current state , i.e. , the prediction error of learned forward-dynamics of the agent . We thoroughly investigate the dynamics-based curiosity across 54 environments : video games , physics engine simulations , and virtual 3D navigation tasks , shown in Figure 1 . To develop a better understanding of curiosity-driven learning , we further study the crucial factors that determine its performance . In particular , predicting the future state in the high dimensional raw observation space ( e.g. , images ) is a challenging problem and , as shown by recent works ( Pathak et al. , 2017 ; Stadie et al. , 2015 ) , learning dynamics in an auxiliary feature space leads to improved results . However , how one chooses such an embedding space is a critical , yet open research problem . To ensure stable online training of dynamics , we argue that the desired embedding space should : 1 ) be compact in terms of dimensionality , 2 ) preserve sufficient information about the observation , and 3 ) be a stationary function of the observations . Through systematic ablation , we examine the role of different ways to encode agent ’ s observation such that an agent can perform well , driven purely by its own curiosity . Here “ performing well ” means acting purposefully and skillfully in the environment . This can be assessed quantitatively , in some cases , by measuring extrinsic rewards or environment-specific measures of exploration , or qualitatively , by observing videos of the agent interacting . We show that encoding observations via a random network turn out to be a simple , yet surprisingly effective technique for modeling curiosity across many popular RL benchmarks . This might suggest that many popular RL video game testbeds are not as visually sophisticated as commonly thought . Interestingly , we discover that although random features are sufficient for good performance in environments that were used for training , the learned features appear to generalize better ( e.g. , to novel game levels in Super Mario Bros. ) . The main contributions of this paper are : ( a ) Large-scale study of curiosity-driven exploration across a variety of environments including : the set of Atari games ( Bellemare et al. , 2013 ) , Super Mario Bros. , virtual 3D navigation in Unity ( Juliani et al. , 2018 ) , multi-player Pong , and Roboschool ( Schulman et al. , 2017 ) environments . ( b ) Extensive investigation of different feature spaces for learning the dynamics-based curiosity : random features , pixels , inverse-dynamics ( Pathak et al. , 2017 ) and variational auto-encoders ( Kingma & Welling , 2013 ) and evaluate generalization to unseen environments . ( c ) Analysis of some limitations of a direct prediction-error based curiosity formulation . We observe that if the agent itself is the source of stochasticity in the environment , it can reward itself without making any actual progress . We empirically demonstrate this limitation in a 3D navigation task where the agent controls different parts of the environment . 2 DYNAMICS-BASED CURIOSITY-DRIVEN LEARNING . Consider an agent that sees an observation xt , takes an action at and transitions to the next state with observation xt+1 . We want to incentivize this agent with a reward rt relating to how informative the transition was . To provide this reward , we use an exploration bonus involving the following elements : ( a ) a network to embed observations into representations φ ( x ) , ( b ) a forward dynamics network to predict the representation of the next state conditioned on the previous observation and action p ( φ ( xt+1 ) |xt , at ) . Given a transition tuple { xt , xt+1 , at } , the exploration reward is then defined as rt = − log p ( φ ( xt+1 ) |xt , at ) , also called the surprisal ( Achiam & Sastry , 2017 ) . An agent trained to maximize this reward will favor transitions with high prediction error , which will be higher in areas where the agent has spent less time , or in areas with complex dynamics . Such a dynamics-based curiosity has been shown to perform quite in some cases ( Pathak et al. , 2017 ) , especially when the dynamics are learned in an embedding space rather than raw observations . In this paper , we explore dynamics-based curiosity further . We use mean-squared error corresponding to a fixed-variance Gaussian density as surprisal , i.e. , ‖f ( xt , at ) − φ ( xt+1 ) ‖22 where f is the learned dynamics model . However , any other density model could be used . 2.1 FEATURE SPACES FOR FORWARD DYNAMICS . Consider the representation φ in the curiosity formulation above . If φ ( x ) = x , the forward dynamics model makes predictions in the observation space . A good choice of feature space can make the prediction task more tractable and filter out irrelevant aspects of the observation space . But what makes a good feature space for dynamics driven curiosity ? We propose the qualities that a good feature space must have : • Compactness : The features should be easy to model by being low ( er ) -dimensional and filtering out irrelevant parts of the observation space . • Sufficiency : The features should contain all the important information . Otherwise , the agent may fail to be rewarded for exploring some relevant aspect of the environment . • Stability : Non-stationary rewards make it difficult for reinforcement agents to learn . Exploration bonuses by necessity introduce non-stationarity since what is new and novel becomes old and boring with time . In a dynamics-based curiosity formulation , there are two sources of non-stationarity : the forward dynamics model is evolving over time as it is trained and the features are changing as they learn . The former is intrinsic to the method , and the latter should be minimized where possible In this work , we systematically investigate the efficacy of a number of feature-learning methods , summarized briefly as follows : Pixels The simplest case is where φ ( x ) = x and we fit our forward dynamics model in the observation space . Pixels are sufficient , since no information has been thrown away , and stable since there is no feature learning component . However , learning from pixels is tricky because the observation space may be high-dimensional and complex . Random Features ( RF ) The next simplest case is where we take our embedding network , a convolutional network , and fix it after random initialization . Because the network is fixed , the features are stable . The features can be made compact in dimensionality , but they are not constrained to be . However , random features may fail to be sufficient . Variational Autoencoders ( VAE ) VAEs were introduced in ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) to fit latent variable generative models p ( x , z ) for observed data x and latent variable z with prior p ( z ) using variational inference . The method calls for an inference network q ( z|x ) that approximates the posterior p ( z|x ) . This is a feedforward network that takes an observation as input and outputs a mean and variance vector describing a Gaussian distribution with diagonal covariance . We can then use the mapping to the mean as our embedding network φ . These features will be a low-dimensional approximately sufficient summary of the observation , but they may still contain some irrelevant details such as noise , and the features will change over time as the VAE trains . Inverse Dynamics Features ( IDF ) Given a transition ( st , st+1 , at ) the inverse dynamics task is to predict the action at given the previous and next states st and st+1 . Features are learned using a common neural network φ to first embed st and st+1 . The intuition is that the features learned should correspond to aspects of the environment that are under the agent ’ s immediate control . This feature learning method is easy to implement and in principle should be invariant to certain kinds of noise ( see ( Pathak et al. , 2017 ) for a discussion ) . A potential downside could be that the features learned may not be sufficient , that is they do not represent important aspects of the environment that the agent can not immediately affect . A summary of these characteristics is provided in Table 1 . Note that the learned features are not stable because their distribution changes as learning progresses . One way to achieve stability could be to pre-train VAE or IDF networks . However , unless one has access to the internal state of the game , it is not possible to get a representative data of the game scenes to train the features . One way is to act randomly to collect data , but then it will be biased to where the agent started , and won ’ t generalize further . Since all the features involve some trade-off of desirable properties , it becomes an empirical question as to how effective each of them is across environments .

Summary: The authors consider the setting of a RL agent that exclusively receives intrinsic reward during training that is intended to model curiosity; technically, ‘curiosity’ is quantified by the ability of the agent to predict its own forward dynamics [Pathak, et al., ICML17]. This study primarily centers around an initially somewhat surprising result that non-trivial policies can be learned for many ’simpler’ video games (e.g., Atari, Super Mario, Pong) using just curiosity as reward. While this is primarily an empirical study, one aspect considered was the observation representation (raw pixels, random features, VAE, and inverse dynamics features [Pathak, et al., ICML17]). In examining reward curves (generally extrinsic during testing), ‘curiosity-based’ reward generally works with the representation effectiveness varying across different testbeds. They also conduct more in-depth experiments on specific testbeds to study the dynamics (e.g., Super Mario, Juggling, Ant Robot, Multi-agent Pong) — perhaps most interestingly showing representation-based transfer of different embeddings across levels in Super Mario. Finally, they consider the Unity maze testbed, combining intrinsic rewards with the end-state goal reward to generate a more dense reward space.