Exploration by random network distillation

1 INTRODUCTION . Reinforcement learning ( RL ) methods work by maximizing the expected return of a policy . This works well when the environment has dense rewards that are easy to find by taking random sequences of actions , but tends to fail when the rewards are sparse and hard to find . In reality it is often impractical to engineer dense reward functions for every task one wants an RL agent to solve . In these situations methods that explore the environment in a directed way are necessary . Recent developments in RL seem to suggest that solving the most challenging tasks ( Silver et al. , 2016 ; Zoph & Le , 2016 ; Horgan et al. , 2018 ; Espeholt et al. , 2018 ; OpenAI , 2018 ; OpenAI et al. , 2018 ) requires processing large numbers of samples obtained from running many copies of the environment in parallel . In light of this it is desirable to have exploration methods that scale well with large amounts of experience . However many of the recently introduced exploration methods based on counts , pseudo-counts , information gain or prediction gain are difficult to scale up to large numbers of parallel environments . This paper introduces an exploration bonus that is particularly simple to implement , works well with high-dimensional observations , can be used with any policy optimization algorithm , and is efficient to compute as it requires only a single forward pass of a neural network on a batch of experience . Our exploration bonus is based on the observation that neural networks tend to have significantly lower prediction errors on examples similar to those on which they have been trained . This motivates the use of prediction errors of networks trained on the agent ’ s past experience to quantify the novelty of new experience . As pointed out by many authors , agents that maximize such prediction errors tend to get attracted to transitions where the answer to the prediction problem is a stochastic function of the inputs . For example if the prediction problem is that of predicting the next observation given the current observation and agent ’ s action ( forward dynamics ) , an agent trying to maximize this prediction error will tend to seek out stochastic transitions , like those involving randomly changing static noise on a TV , or outcomes of random events such as coin tosses . This observation motivated the use of methods that quantify the relative improvement of the prediction , rather than its absolute error . Unfortunately , as previously mentioned , such methods are hard to implement efficiently . We propose an alternative solution to this undesirable stochasticity by defining an exploration bonus using a prediction problem where the answer is a deterministic function of its inputs . Namely we predict the output of a fixed randomly initialized neural network on the current observation . Atari games have been a standard benchmark for deep reinforcement learning algorithms since the pioneering work by Mnih et al . ( 2013 ) . Bellemare et al . ( 2016 ) identified among these games the hard exploration games with sparse rewards : Freeway , Gravitar , Montezuma ’ s Revenge , Pitfall ! , Private Eye , Solaris , and Venture . RL algorithms tend to struggle on these games , often not finding even a single positive reward . In particular , Montezuma ’ s Revenge is considered to be a difficult problem for RL agents , requiring a combination of mastery of multiple in-game skills to avoid deadly obstacles , and finding rewards that are hundreds of steps apart from each other even under optimal play . Significant progress has been achieved by methods with access to either expert demonstrations ( Pohlen et al. , 2018 ; Aytar et al. , 2018 ; Garmulewicz et al. , 2018 ) , special access to the underlying emulator state ( Tang et al. , 2017 ; Stanton & Clune , 2018 ) , or both ( Salimans & Chen , 2018 ) . However without such aids , progress on the exploration problem in Montezuma ’ s Revenge has been slow , with the best methods finding about half the rooms ( Bellemare et al. , 2016 ) . For these reasons we provide extensive ablations of our method on this environment . We find that even when disregarding the extrinsic reward altogether , an agent maximizing the RND exploration bonus consistently finds more than half of the rooms in Montezuma ’ s Revenge . To combine the exploration bonus with the extrinsic rewards we introduce a modification of Proximal Policy Optimization ( PPO , Schulman et al . ( 2017 ) ) that uses two value heads for the two reward streams . This allows the use of different discount rates for the different rewards , and combining episodic and non-episodic returns . With this additional flexibility , our best agent often finds 22 out of the 24 rooms on the first level in Montezuma ’ s Revenge , and occasionally ( though not frequently ) passes the first level . The same method gets state of the art performance on Venture and Gravitar . 2 METHOD . 2.1 EXPLORATION BONUSES . Exploration bonuses are a class of methods that encourage an agent to explore even when the environment ’ s reward et is sparse . They do so by replacing et with a new reward rt = et + it , where it is the exploration bonus associated with the transition at time t. To encourage the agent to visit novel states , it is desirable for it to be higher in novel states than in frequently visited ones . Count-based exploration methods provide an example of such bonuses . In a tabular setting with a finite number of states one can define it to be a decreasing function of the visitation count nt ( s ) of the state s. In particular it = 1/nt ( s ) and it = 1/ √ nt ( s ) have been used in prior work ( Bellemare et al. , 2016 ; Ostrovski et al. , 2018 ) . In non-tabular cases it is not straightforward to produce counts , as most states will be visited at most once . One possible generalization of counts to non-tabular settings is pseudo-counts ( Bellemare et al. , 2016 ) which uses changes in state density estimates as an exploration bonus . In this way the counts derived from the density model can be positive even for states that have not been visited in the past , provided they are similar to previously visited states . An alternative is to define it as the prediction error for a problem related to the agent ’ s transitions . Generic examples of such problems include forward dynamics and inverse dynamics ( Schmidhuber , 1991b ; Stadie et al. , 2015 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2018 ; Haber et al. , 2018 ) . Non-generic prediction problems can also be used if specialized information about the environment is available , like predicting physical properties of objects the agent interacts with ( Denil et al. , 2016 ) . Such prediction errors tend to decrease as the agent collects more experience similar to the current one . For this reason even trivial prediction problems like predicting a constant zero function can work as exploration bonuses ( Fox et al. , 2018 ) . 2.2 RANDOM NETWORK DISTILLATION . This paper introduces a different approach where the prediction problem is randomly generated . This involves two neural networks : a fixed and randomly initialized target network which sets the prediction problem , and a predictor network trained on data collected by the agent . The target network takes an observation to an embedding f : O → Rk and the predictor neural network f̂ : O → Rk is trained by gradient descent to minimize the expected MSE ‖f̂ ( x ; θ ) − f ( x ) ‖2 with respect to its parameters θf̂ . This process distills a randomly initialized neural network into a trained one . The prediction error it = ‖f̂ ( x ) − f ( x ) ‖2 is expected to be higher for novel states dissimilar to the ones the predictor has been trained on . This allows to use it as an exploration bonus . To build intuition we consider a toy model of this process on MNIST . We train a predictor neural network to mimic a randomly initialized target network on training data consisting of a mixture of images with the label 0 and of a target class , varying the proportion of the classes , but not the total number of training examples . We then test the predictor network on the unseen test examples of the target class and report the MSE . In this model the zeros are playing the role of states that have been seen many times before , and the target class is playing the role of states that have been visited infrequently . The results are shown in Figure 2 . The figure shows that test error decreases as a function of the number of training examples in the target class , suggesting that this method can be used to detect novelty . Figure 1 shows that the intrinsic reward is high in novel states in an episode of Montezuma ’ s Revenge . One objection to this method is that a sufficiently powerful optimization algorithm might find a predictor that mimics the target random network perfectly on any input ( for example the target network itself would be such a predictor ) . However the above experiment on MNIST shows that standard gradient-based methods don ’ t overgeneralize in this undesirable way . 2.2.1 SOURCES OF PREDICTION ERRORS . In general , prediction errors can be attributed to a number of factors : 1 . Amount of training data . Prediction error is high where few similar examples were seen by the predictor ( epistemic uncertainty ) . 2 . Stochasticity . Prediction error is high because the target function is stochastic ( aleatoric uncertainty ) . Stochastic transitions are a source of such error for forward dynamics prediction . 3 . Model misspecification . Prediction error is high because necessary information is missing , or the model class is too limited to fit the complexity of the target function . 4 . Learning dynamics . Prediction error is high because the optimization process fails to find a predictor in the model class that best approximates the target function . Factor 1 is what allows one to use prediction error as an exploration bonus . In practice the prediction error is caused by a combination of all of these factors , not all of them desirable . For instance if the prediction problem is forward dynamics , then factor 2 results in the ‘ noisy-TV ’ problem . This is the thought experiment where an agent that is rewarded for errors in the prediction of its forward dynamics model gets attracted to stochastic transitions in the environment . A TV randomly switching between channels would be such an attractor , as would a coin flip . To avoid the undesirable factors 2 and 3 , methods such as those by Schmidhuber ( 1991a ) ; Oudeyer et al . ( 2007 ) ; Lopes et al . ( 2012 ) ; Achiam & Sastry ( 2017 ) instead use a measurement of how much the prediction model improves upon seeing a new datapoint . However these approaches tend to be computationally expensive and hence difficult to scale . RND obviates factors 2 and 3 since the target network can be chosen to be deterministic and inside the model-class of the predictor network .

Summary: The algorithm proposed in this paper consists in driving exploration in RL through an intrinsic reward, computed as the prediction error of a neural network whose target is the output of a randomly initialized network (with the state reached by the agent as input). The intuition is that rarely seen states will have a large prediction error, thus encouraging the agent to visit them (until they have been seen often enough that the error goes down). Among potential benefits of this method, compared to previously proposed intrinsic curiosity techniques for RL, are its simplicity and its robustness to environment stochasticity. Extensive experiments on the Atari game Montezuma’s Revenge investigate several variants of this idea (combined with PPO), with the best results significantly outperforming the current state-of-the-art. Other results on five other hard exploration Atari games show competitive performance as well.