Prioritized Level Replay

1 INTRODUCTION . Environments generated using procedural content generation ( PCG ) have garnered increasing interest in RL research , leading to a surge of PCG environments such as MiniGrid ( Chevalier-Boisvert et al. , 2018 ) , the Obstacle Tower Challenge ( Juliani et al. , 2019 ) , the Procgen Benchmark ( Cobbe et al. , 2019 ) , and the NetHack Learning Environment ( Küttler et al. , 2020 ) . Unlike singleton environments , like those in the Arcade Learning Environment benchmark ( Bellemare et al. , 2013 ) , which are exploitable by memorization and deterministic reset strategies ( Ecoffet et al. , 2019 ; 2020 ) , PCG environments create novel environment instances or levels algorithmically . Every such level exhibits a unique configuration of underlying factors of variation , such as layout , positions of game entities , asset appearances , or even different rules governing environment transitions , making them a promising target for evaluating systematic generalization in RL ( Risi & Togelius , 2020 ) . Each level can be associated with a level identifier ( e.g . an index , a random number generator seed , etc . ) used by the PCG algorithm to generate a specific level . This allows for a clean notion of train-test split and testing on held-out levels , in line with the best practices from supervised learning . Due to the variation among algorithmically-generated levels , a random selection of PCG levels can , in principle , correspond to levels of varying difficulty as well as reveal different—perhaps rare—environment dynamics . This diversity among levels implies that different levels hold differing learning potential for an agent at any point in training , a fact exploited by methods that both learn to generate levels as well as to solve them ( Wang et al. , 2019 ; 2020 ) . Here , we focus on the less intrusive setting where we do not have control over level generation , but can instead replay ( that is , revisit ) any previously visited level during training to generate entirely new experiences from it . We introduce Prioritized Level Replay , illustrated in Figure 1 , a method for sampling training levels that exploits this difference among levels . Our method induces a level-sampling distribution that prioritizes levels based on the learning potential of replaying each level anew . Throughout agent training , our method updates a heuristic score appraising the agent ’ s learning potential on a level based on temporal-difference ( TD ) errors collected along the last trajectory sampled from that level . Rather than sampling from the default , typically static ( e.g . uniform ) , training level distribution , our method samples from a distribution derived from a normalization procedure over these level scores . Prioritized Level Replay does not make any assumption about how the policy is updated , and is therefore compatible with any RL method . Furthermore , our method does not rely on some external or general method for quantifying difficulty of a level , but instead derives a level score directly from the policy . The only requirements—satisfied in a wide number of settings where a simulator or game is used to collect experience—are as follows : ( i ) some notion of “ level ” exists , ( ii ) such levels can be sampled from the environment in an identifiable way , and ( iii ) given a level identifier , it is possible to set the environment to that level to be able to collect new experiences from it . While previous works in off-policy RL devised effective methods to directly reuse past experiences for training ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) , Prioritized Level Replay uses past experiences to inform the collection of future experiences by assessing how much replaying each level anew will benefit learning . Our method can thus be seen as a forward-view variation of prioritized experience replay , and an online counterpart to this off-policy method for policy-gradient algorithms . This paper makes the following core contributions : ( i ) we introduce a computationally efficient algorithm for adaptively prioritizing levels throughout training using a heuristic-based assessment of the learning potential of replaying each level , ( ii ) we empirically demonstrate our method leads to significant gains on 11 of 16 Procgen Benchmark environments and two challenging MiniGrid environments , ( iii ) we demonstrate our method combines with a previous leading method to set a new state-of-the-art on Procgen Benchmark , and ( iv ) we provide evidence that our method induces an implicit curriculum over training levels in sparse reward settings . 2 BACKGROUND . In this paper , we refer to a PCG environment as any computational process that , given a level identifier ( e.g . an index or a seed ) , generates a level , defined as an environment instance exhibiting a unique configuration of its underlying factors of variation , such as layout , positions of game entities , asset appearances , or even rules that govern the environment transitions ( Risi & Togelius , 2020 ) . For example , MiniGrid ’ s MultiRoom environment instantiates mazes with varying numbers of rooms based on the seed ( Chevalier-Boisvert et al. , 2018 ) . We refer to sampling a new trajectory generated from the agent ’ s latest policy acting on a given level l as replaying that level l. The level diversity of PCG environments makes them useful testbeds for studying the robustness and generalization ability of RL agents , measured by agent performance on unseen test levels . The standard test evaluation protocol for PCG environments consists of training the agent on a finite number of training levels , Λtrain and evaluating performance on unseen test levels Λtest , drawn from the set of all levels . A common variation of this protocol sets Λtrain to the set of all levels , though the agent will still effectively only sees a finite set of levels during training . Training levels are sampled from an arbitrary distribution Ptrain ( l|Λtrain ) . We call this training process direct level sampling . In practice , for the case of a finite training set , typically Ptrain ( l|Λtrain ) = Uniform ( l ; Λtrain ) . See Appendix D for the pseudocode outlining this procedure . In generating levels of varying difficulty , PCG environments naturally lend themselves to the study of curriculum-based training protocols . For example , Justesen et al . ( 2018 ) and Wang et al . ( 2019 ; 2020 ) show that generating progressively harder levels improves generalization of agents across difficulty settings , demonstrating that the set of levels most useful for improving an agent ’ s performance varies throughout the course of training . When levels can not directly be generated , many previous works have exploited handcrafted curricula to learn policies on harder levels ( ChevalierBoisvert et al. , 2018 ; Zhong et al. , 2020 ) . Instead , we are interested in answering how an effective curriculum over an existing set Λtrain can be automatically discovered . 3 PRIORITIZED LEVEL REPLAY . In this section , we present Prioritized Level Replay , a level-sampling algorithm that exploits differences in the learning potential for the current agent in replaying previously visited levels . It is a drop-in replacement for the experience-collection process used in a wide range of RL algorithms . To exemplify this , we show how it is straight-forward to incorporate Prioritized Level Replay into a generic policy-gradient training loop , as described in Algorithm 1 where , for clarity , we focus on training on batches of complete trajectories ( see Appendix D for detailed pseudocode on how our procedure works with T -step rollouts ) . Our method , illustrated in Figure 1 and fully specified in Algorithm 2 , induces a dynamic and nonparametric sampling distribution Preplay ( l|Λseen ) over previously visited training levels Λseen that prioritizes visited levels with higher learning potential based on properties of the agent ’ s past trajectories . We refer to Preplay ( l|Λseen ) as the replay distribution . Throughout an agent ’ s training , our method updates this replay distribution adaptively according to a heuristic score , assigning greater weight to visited levels with higher future learning potential . Through variable-size lists S and C of equal length to Λseen , Prioritized Level Replay tracks level scores Si ∈ S for each visited training level li based on the latest episode trajectory on li , as well as the episode count Ci ∈ C at which each level li ∈ Λseen was last sampled . Our method updates Preplay after each terminated episode by computing a mixture of two distributions , PS , based on the level scores , and PC , based on how long ago each level was last sampled : Preplay = ( 1− ρ ) · PS + ρ · PC , ( 1 ) where the staleness coefficient ρ ∈ [ 0 , 1 ] is a hyperparameter . We discuss how we compute level scores ( Si ) parameterizing the scoring ( PS ) distribution , and the staleness ( PC ) distribution , in Sections 3.1 and 3.2 , respectively . Prioritized Level Replay chooses the next level at the start of every training episode by first sampling a replay-decision from a Bernoulli ( or similar ) distribution PD ( d ) to determine whether to replay a level sampled from the replay distribution Preplay ( l|Λseen ) or to experience a new , unseen level from Λtrain , according to some distribution Pnew ( l|Λtrain\Λseen ) . In practice , for the case of a finite number of training levels , we implement Pnew as a uniform distribution over the remaining unseen levels . For the case of a countably infinite number of training levels , we simulate Pnew by sampling levels from Ptrain until encountering an unseen level . In our experiments based on a finite number of training levels , we opt to perform a “ warm start ” by setting PD ( d = 0 ) = 1 ( i.e . sampling a new level ) as long as |Λtrain \ Λseen| > 0 ( i.e . not every level has been seen ) , ensuring the replay distribution takes into account scores from at least one trajectory from each level . The following sections describe the main steps of how Prioritized Level Replay updates the replay distribution Preplay ( l|Λseen ) , namely level scoring and staleness-aware prioritization . 3.1 SCORING LEVELS FOR LEARNING POTENTIAL . After collecting each full trajectory τ over a completed episode on level li using policy π , our method assigns li a score Si = score ( τ , π ) measuring the learning potential of replaying li in the future . We employ a function of the TD-error at timestep t , δt = rt+γV ( st+1 ) −V ( st ) , as a proxy Algorithm 1 Generic policy-gradient training loop with Prioritized Level Replay Input : Training levels Λtrain of environment E , policy πθ , policy update function U ( B , θ ) → θ′ , and batch size Nb . Initialize level scores S and level timestamps C , global episode counter c← 0 Initialize the ordered set of visited levels Λseen = ∅ Initialize experience buffer B = ∅ while training do B ← ∅ while collecting experiences do B ← B ∪ collect experiences ( Λtrain , Λseen , πθ , S , C , c ) . Using Algorithm 2 end while θ ← U ( B , θ ) . Update policy using collected experiences end while Algorithm 2 Experience collection with Prioritized Level Replay Input : Training levels Λtrain , visited levels Λseen , policy π , the training level distribution Ptrain , global level scores S , global level timestamps C , and global episode counter c. Output : A sampled trajectory τ c← c+ 1 Sample replay decision d ∼ PD ( d ) if d = 0 and |Λtrain \ Λseen| > 0 then . Sample an unseen level , if any Define new index i← |S|+ 1 Sample li ∼ Pnew ( l|Λtrain , Λseen ) Add li to Λseen , add initial value 0 to S and C else . Sample a level for replay Sample li ∼ ( 1− ρ ) · PS ( l|Λseen , S ) + ρ · PC ( l|Λseen , C , c ) end if Sample τ ∼ Pπ ( τ |l ) Update score Si ← score ( τ , π ) and timestamp Ci ← c . Update level replay lists for this learning potential . Its expectation over next states is equivalent to the advantage estimate , and therefore higher-magnitude TD-errors imply greater discrepancy between expected and actual returns , making δt a useful measure of the learning potential in revisiting a particular state transition . To prioritize the learning potential of future experiences resulting from replaying a level , we use a scoring function based on the average magnitude of the Generalized Advantage Estimate ( GAE ; Schulman et al. , 2015 ) over each of T time steps in the latest trajectory τ from that level : Si = score ( τ , π ) = 1 T T∑ t=0 ∣∣∣∣∣ T∑ k=t ( γλ ) t−kδt ∣∣∣∣∣ . ( 2 ) Importantly , while the GAE at time t is most commonly expressed as the discounted sum of all 1-step TD-errors starting at t as in Equation 2 , it is equivalent to an exponentially-discounted sum of all k-step TD-errors from t , with discount factor λ . By considering all k-step TD-errors , the GAE mitigates the bias introduced by the bootstrap term in 1-step TD-errors . The discount factor λ then controls the trade-off between bias and variance . Our scoring function considers the absolute value of the GAE , as we assume the degree of learning potential grows with the magnitude of the TD-error irrespective of its sign . Another useful interpretation of Equation 2 comes from observing that the GAE at t is equivalent to the L1 value loss |V̂t − Vt| when applied in the context of a policy-gradient algorithm that uses GAE for its own advantage estimates—and therefore value targets V̂t—as done in state-of-the-art implementations of PPO ( Schulman et al. , 2017 ) used in our experiments . For this reason , we will refer to this instantiation of our algorithm as value-based level replay . While we provide principled motivations for our specific choice of scoring function , we emphasize that in general , the scoring function can be any approximation of learning potential based on pertrajectory values . In Section 4 , we will compare our choice of the unsigned GAE , or equivalently , the L1 value loss , to alternative TD-error-based and uncertainty-based scoring approaches listed in Table 1 . Given level scores , the normalized outputs of a prioritization function h evaluated over these scores and tuned using a temperature parameter β defines the score-prioritized distribution PS ( Λtrain ) over the training levels , under which PS ( li|Λseen , S ) = h ( Si ) 1/β∑ j h ( Sj ) 1/β . ( 3 ) The function h defines how differences in level scores translate into differences in prioritization . The temperature parameter β allows us to tune how much h ( S ) ultimately determines the resulting distribution . We make the design choice of using rank prioritization , for which h ( Si ) = 1/rank ( Si ) , where rank ( Si ) is the rank of level score Si among all scores sorted in descending order . We also experimented with proportional prioritization ( h ( Si ) = Si ) , which performs comparably to rank prioritization , as well as greedy prioritization ( the level with the highest score receives probability 1 ) , which tends to perform worse .

Summary: This paper concerns about the use of experience replay in a way that past experience is sampled based on (implicit) levels so as for the agent to better adapt to the current task at hand. The authors defined a replay distribution (where experience is sampled) based on two scores relevant to learning potential and staleness. Due to its formulation, the change of replay distribution can be used as an outer-layer of a learning algorithm without any modification of the underlying learning mode. The authors conducted experiments over a set of benchmark data sets relevant to level-ness and found statistically significant improvements over more than half of the tasks.