Unsupervised Exploration with Deep Model-Based Reinforcement Learning

1 INTRODUCTION Many mammals , including humans , are incapable after birth of providing for themselves for extended periods of time ( years in the case of humans ) . This behavior is in stark contrast with other species that are born with complex motor skills already available to them . One hypothesis is that this forced developmental period provides a safe environment for the infant to perform unsupervised ( i.e. , task-agnostic ) exploration in order to acquire more flexible skills and enhanced adaptation capabilities that will prove useful in the long term . This hypothesis is at least partially supported by the observation that reduced exploration in the early periods ( 7-9 months ) of infancy is correlated with long-term poorer cognitive and language outcomes ( Ruff et al. , 1984 ; Muentener et al. , 2018 ) . From a machine learning perspective on reinforcement learning , this hypothesis raises intriguing questions . The predominant formalization of RL is concerned with solving a single task . Is learning to solve a single task “ in the void ” without any context and prior knowledge truly the right benchmark to develop intelligent and efficient algorithms ? Can human-like learning capabilities be attained through iterative progress in such simplified settings ? Or can we formulate a problem statement that more closely matches the one outlined above , where an unsupervised critical period is used to acquire knowledge about the world , and that knowledge is then used for goal-directed task acquisition later ? Guided by these overwhelming questions , in this paper we do a timid step towards better understanding the importance of unsupervised exploration and reinforcement learning . To do so , we consider a setting composed of a first unsupervised exploration phase where the agent can interact with the environment – but without having any specific task defined – and a second phase which require the agent to quickly adapt to solve multiple tasks not previously known . We then address a specific piece of the critical questions : if we are to use an unsupervised exploration phase to learn about the world , what sort of knowledge about the world should we retain that will be helpful for then achieving specific goals ? The particular hypothesis that we explore in this regard is that model-based reinforcement learning offers a promising solution , if it is augmented with an appropriate unsupervised exploration strategy . In contrast to model-free reinforcement learning , which learns task-specific policies or value functions , model-based reinforcement learns to predict the future – specifically , to predict the next state from the previous state and action . This predictive model can then be used to optimize for near-optimal behavior by planning or offline policy optimization . Model-based reinforcement learning offers an appealing approach to unsupervised exploration : explore the environment to acquire a good model , in effect learning how the world works , and then simply use this model to achieve new user-specified tasks . In contrast , model-free unsupervised reinforcement learning methods require considerable additional machinery to extract useful knowledge from unsupervised exploration . While this basic principle is simple and well known , to our knowledge no prior work has proposed an unsupervised approach to exploration in model-based reinforcement learning with expressive deep neural network models . This formulation raises several challenges . First , model-based RL with deep neural networks is inherently challenging because high-capacity models have a tendency to overfit in the model-based RL setting , leading to poor performance . The optimization over actions or policies exacerbates this issue , since a model that overfits to data will be easy for a strong optimizer to “ exploit ” and discover erroneously optimistic outcomes . Second , unsupervised reinforcement learning requires an effective exploration procedure to achieve good coverage of the state , and while exploration for model-free RL has been studied extensively , as we discuss in the following section , exploration for model-based RL has been studied substantially less . In this work , we use model uncertainty to address both of these issues . By formulating an uncertainty-aware deep neural network dynamics model , we show that we can both obtain a readily usable exploration criterion for the unsupervised training phase and produce a model-based RL procedure that can remain robust to “ exploitation ” when presented with goals at test-time . Our primary contributions are the formulation of the unsupervised deep model-based RL learning procedure , and the introduction of a practical algorithm for tackling this problem . Our experimental results show that our method can provide excellent performance in terms of achieving new goals , even without any additional training beyond the unsupervised exploration phase , and can substantially improve performance through additional finetuning . We compare to alternative exploration heuristics from the model-free RL literature , as well prior baseline methods that do not perform explicit exploration . 2 RELATED WORK . Exploration generally aims to acquire information about an environment to help an agent better optimize its reward function . A large number of exploration techniques have been proposed in the reinforcement learning literature suited for continuous state-action environments , from simple inefficient methods to complex methodical exploration strategies . Simple but computationally-cheap methods used to increase training data diversity include occasionally selecting random actions ( - greedy ) for discrete actions , additional Gaussian noise for continuous actions , and policy parameters perturbations ( Plappert et al. , 2017 ; Fortunato et al. , 2017 ) . A disadvantage of -greedy exploration is even actions that are known to have low Q-values are executed with the same probability as other actions . Exploration can instead be biased towards actions of higher expected value using Boltzman exploration or maximum entropy methods ( Haarnoja et al. , 2018 ) . However the robot may still repeatedly ‘ explore ’ actions whose values it is already certain . Information value theory defines the value of exploration as the additional rewards we expect to obtain given the expected information of taking an action ( Howard , 1966 ) . In reinforcement learning , the value of exploration ( information ) can be formulated with Bayesian method by first placing a prior over the space of MDPs and treating all subsequent observations as likelihoods . This Bayesian reinforcement leaning transforms the MDP learning problem into a POMDP planning problem : a POMDP whose state is a hybrid of dynamics function parameters ( that are partially observed , or uncertainty ) and the robot state ( Duff & Barto , 2002 ) . Whilst treating RL as a POMDP defines exploratory value in MDP environments , computing values is almost always intractable . Instead approximate myopic and ‘ optimistic ’ exploration strategies can form effective explorers in model-free or model-based RL ( Dearden et al. , 1998 ; 1999 ; Ghavamzadeh et al. , 2015 ) . Such frameworks reason probabilistically about plausible value functions or dynamics functions respectively . Directing exploration towards such known-unknowns yields exponentially faster learning in discrete spaces ( Thrun , 1992 ) . For continuous state-action spaces , an effective method periodically samples value function or dynamics function , acting greedily w.r.t . the sample for a period of time , achieving temporarily-extended ( or deep ) exploration Thompson ( 1933 ) ; Osband et al . ( 2017 ) ; Touati et al . ( 2018 ) . An alternate definition of data-efficiency for exploration is minimizing the number of episodes that the agent fails to perform within a specified bound of the optimal performance . Algorithms which address this alternate criteria are PAC-MDP ( probably approximately correct Markov decisions process ) methods which aim to discover ‘ near-optimal ’ policies with high-probability within time polynomial to the number of states and actions . Examples include R-max Brafman & Tennenholtz ( 2003 ) , E3 Kearns & Singh ( 2002 ) , and delayed Q-learning Strehl et al . ( 2006 ) . PAC-MDP methods provide powerful probabilistic-guarantees on the data-complexity required before asymptotic convergence to an optimal policy . However , such strong guarantees are only possible by systematic over-exploration of the complete state-space Delage & Mannor ( 2007 ) ; Kolter & Ng ( 2009 ) . In the context of model-based RL , uncertainty has been previously used both to encourage exploration in states of high-uncertainty ( Depeweg et al. , 2017 ) , and to perform risk-sensitive control by avoiding states with high uncertainty ( Buechler et al. , 2018 ) . Whilst the aforementioned Bayesian methods are typically effective in typical RL problems where the reward function or reward samples are accessible , this paper considers the problem an initial exploration training phase that is unsupervised w.r.t . the reward . Hence our problem setting is more akin to using state-uncertainty-based exploration methods ( Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Tang et al. , 2017 ) , model error ( Pathak et al. , 2017 ) , and novelty-based exploration ( Fu et al. , 2017 ) . However , in contrast to all of these prior methods , which are concerned either with exploring to maximize a reward on a single specific task , or else exploring with no task reward at all and without any downstream tasks ( Pathak et al. , 2017 ) , our method is concerned with transfer : using unsupervised exploration to acquire knowledge about the world that can help to perform downstream tasks . We therefore employ model-based reinforcement learning , in contrast to all of these prior methods that are based on model-free RL . We show that , in the model-based RL setting , transfer is straightforward , and uncertainty-aware models directly provide a readily usable exploration criterion . To purposefully explore a state space , model-based RL can be used to intentionally plan towards uncertainty regions . Probabilistic model-based RL learns particularly fast , being robust to overfitting to small datasets during the early stages of learning ( Deisenroth & Rasmussen , 2011 ; Chua et al. , 2018 ) . In addition , probabilistic models are well suited to exploration problems since an agent can methodically plan actions sequences of maximal trajectory uncertainty . Such behavior yields temporarily-extended ( or deep ) exploration as opposed to myopic exploration which only considers what act be learned within one future transition . 3 UNSUPERVISED EXPLORATION WITH DEEP MODEL-BASED RL . We consider the class of learning tasks , shown in Figure 1 , that have two phases : 1 . A longer unsupervised exploration phase , without any reward feedback . 2 . A supervised phase , where the agent is provided with a goal , and must achieve that goal either immediately or after a small amount of fine-tuning . In order to transfer knowledge from phase 1 to phase 2 , we propose to use model-based reinforcement , which learns to predict the outcomes of actions in the environment . The dynamics model learned in model-based RL is , insofar as it is globally accurate , agnostic to the particular task , making it ideal for transferring to phase 2 . However , model-based RL with high-capacity function approximators , such as deep neural network models , is prone to overfitting , which can lead to poor results in practice . Furthermore , the unsupervised phase requires an objective to ensure the learner can visit a wide variety of states in the environment . To address both of these challenges , we extend the PETS algorithm ( Chua et al. , 2018 ) , which estimates model uncertainty using an ensemble of dynamics models . While PETS has been previously demonstrated to achieve good performance on purely goal-directed tasks , prior work used simple random exploration , and did not study the unsupervised reinforcement learning setting . Since a probabilistic models predict distributions over future states , we extend PETS to explore by selecting action sequences that lead to a wide distribution over possible states . Through recursive prediction of state distributions , this method can consider the fact that multiple deterministic transitions may be required to reach novel states with uncertain dynamics . Such temporally extended exploration ( also called deep , or non-myopic exploration ) is desirable to learn complex sequential decisions making problems like walking or running . During phase 2 , we propose that our algorithm act in a greedy fashion : simply maximizing the expected rewards without further consideration of exploration . If the exploration phase was successful , this will result in good performance . If performance is insufficient , several additional episodes can provide enough data to finetune the model , as we illustrate in our experiments .

Summary: The authors address the problem of how to use unsupervised exploration in a first phase of reinforcement learning to gather knowledge that can be transferred to new tasks to improve performance in a second task when specific reward functions are available. The authors proposed a model-based approach which uses deep neural networks as a model for the environment. The model is PETS (probabilistic ensembles with trajectory sampling), an ensemble of neural networks whose outputs parametrize predictive distributions for the next state as a function of the current state and the action applied. To collect data during the unsupervised exploration phase, they use a metric of model uncertainty computed as follows: the average over all the particles assigned to each bootstrap is computed and the variance over these computed means is the