Replay-Guided Adversarial Environment Design

1 Introduction . While deep reinforcement learning ( RL ) approaches have led to many successful applications in challenging domains like Atari [ 21 ] , Go [ 35 ] , Chess [ 36 ] , Dota [ 4 ] , and StarCraft [ 40 ] in recent years , deep RL agents still prove to be brittle , often failing to transfer to environments only slightly different from those encountered during training [ 44 , 9 ] . To ensure learning of robust and well-generalizing policies , agents must train on sufficiently diverse and informative variations of environments ( e.g . see Section 3.1 of [ 8 ] ) . However , it is not always feasible to specify an appropriate training distribution or a generator thereof . Agents may therefore benefit greatly from methods that automatically adapt the distribution over environment variations throughout training [ 10 , 17 ] . Throughout this paper we will call a particular environment instance or configuration ( e.g . an arrangement of blocks , race tracks , or generally any of the environment ’ s constituent entities ) a level . Two recent works [ 10 , 17 ] have sought to empirically demonstrate this need for a more targeted agentadaptive mechanism for selecting levels on which to train RL agents , so to ensure efficient learning and generalization to unseen levels—as well as to provide methods implementing such mechanisms . The first method , Protagonist Antagonist Induced Regret Environment Design ( PAIRED ) [ 10 ] , ∗Equal contribution . Correspondence to msj @ fb.com and michael_dennis @ berkeley.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . introduces a self-supervised RL paradigm called Unsupervised Environment Design ( UED ) . Here , an environment generator ( a teacher ) is co-evolved with a student policy that trains on levels actively proposed by the teacher , leading to a form of adaptive curriculum learning . The aim of this coevolution is for the teacher to gradually learn to generate environments that exemplify properties of those that might be encountered at deployment time , and for the student to simultaneously learn a good policy that enables zero-shot transfer to such environments . PAIRED ’ s specific adversarial approach to environment design ensures a useful robustness characterization of the final student policy in the form of a minimax regret guarantee [ 31 ] —assuming that its underlying teacher-student multi-agent system arrives at a Nash equilibrium [ NE , 24 ] . In contrast , the second method , Prioritized Level Replay ( PLR ) [ 17 ] , embodies an alternative form of dynamic curriculum learning that does not assume control of level generation , but instead , the ability to selectively replay existing levels . PLR tracks levels previously proposed by a black-box environment generator , and for each , estimates the agent ’ s learning potential in that level , in terms of how useful it would be to gather new experience from that level again in the future . The PLR algorithm exploits these scores to adapt a schedule for revisiting or replaying levels to maximize learning potential . PLR has been shown to produce scalable and robust results , improving both sample complexity of agent training and the generalization of the learned policy in diverse environments . However , unlike PAIRED , PLR is motivated with heuristic arguments and lacks a useful theoretical characterization of its learning behavior . In this paper , we argue that PLR is , in and of itself , an effective form of UED : Through curating even randomly generated levels , PLR can generate novel and complex levels for learning robust policies . This insight leads to a natural class of UED methods which we call Dual Curriculum Design ( DCD ) . In DCD , a student policy is challenged by a team of two co-evolving teachers . One teacher actively generates new , challenging levels , while the other passively curates existing levels for replaying , by prioritizing those estimated to be most suitably challenging for the student . We show that PAIRED and PLR are distinct members of the DCD class of algorithms and prove in Section 3 that all DCD algorithms enjoy similar minimax regret guarantees to that of PAIRED . We make use of this result to provide the first theoretical characterization of PLR , which immediately suggests a simple yet highly counterintuitive adjustment to PLR : By only training on trajectories in replay levels , PLR becomes provably robust at NE . We call this resulting variant PLR⊥ ( Section 4 ) . From this perspective , PLR effectively performs level design in a diametrically opposite manner to PAIRED—through prioritized selection rather than active generation . A second corollary to the provable robustness of DCD algorithms shows that PLR⊥ can be extended to make use of the PAIRED teacher as a level generator while preserving the robustness guarantee of PAIRED , resulting in a method we call Replay-Enhanced PAIRED ( REPAIRED ) ( Section 5 ) . We hypothesize that in this arrangement , PLR⊥ plays a complementary role to PAIRED in robustifying student policies . Our experiments in Section 6 investigate the learning dynamics of PLR⊥ , REPAIRED , and their replay-free counterparts on a challenging maze domain and a novel continuous control UED setting based on the popular CarRacing environment [ 5 ] . In both of these highly distinct settings , our methods provide significant improvements over PLR and PAIRED , producing agents that can perform out-of-distribution ( OOD ) generalization to a variety of human designed mazes and Formula 1 tracks . In summary , we present the following contributions : ( i ) We establish a common framework , Dual Curriculum Design , that encompasses PLR and PAIRED . This allows us to develop new theory , which provides the first robustness guarantees for PLR at NE as well as for REPAIRED , which augments PAIRED with a PLR-based replay mechanism . ( ii ) Crucially , our theory suggests a highly counterintuitive improvement to PLR : the convergence to NE should be assisted by training on less data when using PLR—namely by only taking gradient updates from data that originates from the PLR buffer , using the samples from the environment distribution only for computing the prioritization of levels in the buffer . ( iii ) Our experiments in a maze domain and a novel car racing domain show that our methods significantly outperform their replay-free counterparts in zero-shot generalization . We open source our methods at https : //github.com/facebookresearch/dcd . 2 Background . 2.1 Unsupervised Environment Design . Unsupervised Environment Design ( UED ) , as introduced by [ 10 ] , is the problem of automatically designing a distribution of environments that adapts to the learning agent . UED is defined in terms of an Underspecified POMDP ( UPOMDP ) , given byM = 〈A , O , Θ , SM , TM , IM , RM , γ〉 , where A is a set of actions , O is a set of observations , S is a set of states , T : S × A × Θ → ∆ ( S ) is a transition function , I : S → O is an observation ( or inspection ) function , R : S → R is a reward function , and γ is a discount factor . This definition is identical to a POMDP with the addition of Θ to represent the free-parameters of the environment . These parameters can be distinct at every time step and incorporated into the transition function TM : S ×A×Θ→∆ ( S ) . For example , Θ could represent the possible positions of obstacles in a maze . We will refer to the environment resulting from a fixed θ ∈ Θ asMθ , or with a slight abuse of notation , simply θ when clear from context . We define the value of π inMθ to be V θ ( π ) = E [ ∑T i=0 rtγ t ] where rt are the rewards attained by π in Mθ . Aligning with terminology from [ 17 ] , we refer to a fully-specified environment as a level . 2.2 Protagonist Antagonist Induced Regret Environment Design . Protagonist Antagonist Induced Regret Environment Design [ PAIRED , 10 ] presents a UED approach consisting of simultaneously training agents in a three player game : the protagonist πA and the antagonist πB are trained in environments generated by the teacher θ̃ . The objective of this game is defined by U ( πA , πB , θ̃ ) = Eθ∼θ̃ [ REGRET θ ( πA , πB ) ] , where regret is defined by REGRETθ ( πA , πB ) = V θ ( πB ) − V θ ( πA ) . The protagonist and antagonist are both trained to maximize their discounted environment returns while the teacher is trained to maximize U . Note that by maximizing regret , the teacher is disincentivized from generating unsolvable levels , which will have a maximum regret of 0 . As shorthand , we will sometimes refer to the protagonist and antagonist jointly as the student agents . The counterclockwise loop beginning at the student agents in Figure 2 summarizes this approach , with the students being both the protagonist and antagonist . As both student agents grow more adept at solving different levels , the teacher continues to adapt its level designs to exploit the weaknesses of the protagonist in relation to the antagonist . As this dynamic unfolds , PAIRED produces an emergent curriculum of progressively more complex levels along the boundary of the protagonist ’ s capabilities . PAIRED is a creative method in the sense that the teacher may potentially generate an endless sequence of novel levels . However , as the teacher only adapts through gradient updates , it is inherently slow to adapt to changes in the student policies . 2.3 Prioritized Level Replay . Prioritized Level Replay [ PLR , 17 ] is an active-learning strategy shown to improve a policy ’ s sample efficiency and generalization to unseen levels when training and evaluating on levels from a common UPOMDP , typically implemented as a seeded simulator . PLR maintains a level buffer Λ of the top K visited levels with highest learning potential as estimated by the time-averaged L1 value loss of the learning agent over the last episode on each level . At the start of each training episode , with some predefined replay probability p , PLR uses a bandit to sample the level from Λ to maximize the estimated learning potential ; otherwise , with probability 1− p , PLR samples a new level from the simulator . In contrast to the generative but slow-adapting PAIRED , PLR does not create new levels , but instead , acts as a fast-adapting curation mechanism for selecting the next training level among previously encountered levels . Also unlike PAIRED , PLR does not provide a robustness guarantee . By extending the theoretical foundation of PAIRED to PLR , we will show how PLR can be modified to provide a robustness guarantee at NE , as well as how PAIRED can exploit PLR ’ s complementary curation to quickly switch among generated levels to maximize the student ’ s regret . 3 The Robustness of Dual Curriculum Design . The previous approaches of PAIRED and PLR reveal a natural duality : Approaches that gradually learn to generate levels like PAIRED , and methods which can not generate levels , but instead , quickly curate existing ones , like PLR . This duality suggests combining slow level generators with fast level curators . We call this novel class of UED algorithms Dual Curriculum Design ( DCD ) . For instance , PLR can be seen as curator with a prioritized sampling mechanism with a random generator , while PAIRED , as a regret-maximizing generator without a curator . DCD can further consider Domain Randomization ( DR ) as a degenerate case of a random level generator without a curator . To theoretically analyze this space of methods , we model DCD as a three player game among a student agent and two teachers called the dual curriculum game . However , to formalize this game , we must first formalize the single-teacher setting : Suppose the UPOMDP is clear from context . Then , given a utility function for a single teacher , Ut ( π , θ ) , we can naturally define the base game between the student s and teacher t as G = 〈S = Ss × St , U = Us × Ut〉 , where Ss = Π is the strategy set of the student , St = Θ is the strategy set of the teacher , and Us ( π , θ ) = V θ ( π ) is the utility function of the student . In Sections 4 and 5 , we will study settings corresponding to different choices of utility functions for the teacher agents , namely the maximum-regret objective URt ( π , θ ) and the uniform objective UUt ( π , θ ) . These two objectives are defined as follows ( for any constant C ) : URt ( π , θ ) = argmax π∗∈Π { V θ ( π∗ ) − V θ ( π ) } ( 1 ) UUt ( π , θ ) = C ( 2 ) In the dual curriculum game G , the first teacher plays the game with probability p , and the second , with probability ( 1− p ) —or more formally , G = 〈S = Ss × St × St , U = Us × U 1 t × U 2 t 〉 , where the utility functions for the student and two teachers respectively , Us , U 1 t , U 2 t , are defined as follows : U 1 t ( π , θ 1 , θ2 ) = pU1t ( π , θ 1 ) ( 3 ) U 2 t ( π , θ 1 , θ2 ) = ( 1− p ) U2t ( π , θ2 ) ( 4 ) Us ( π , θ 1 , θ2 ) = pUs ( π , θ 1 ) + ( 1− p ) Us ( π , θ2 ) ( 5 ) Our main theorem is that NE in the dual curriculum game are approximate NE of both the base game for either of the original teachers and the base game with a teacher maximizing the joint-reward of pU1t + ( 1− p ) U2t , where the quality of the approximations depends on the mixing probability p. Theorem 1 . Let B be the maximum difference between U1t and U2t , and let ( π , θ1 , θ2 ) be a NE for G. Then ( π , pθ1 + ( 1− p ) θ2 ) is an approximate NE for the base game with either teacher or for a teacher optimizing their joint objective . More precisely , it is a 2Bp ( 1− p ) -approximate NE when Ut = pU 1 t + ( 1− p ) U2t , a 2B ( 1− p ) -approximate NE when Ut = U1t , and a 2Bp-approximate NE when Ut = U2t . The intuition behind this theorem is that , since the two teachers do not affect each other ’ s behavior , their best response to a fixed πs is to choose a strategy θ that maximizes U1t and U 2 t respectively . Moreover , the two teachers ’ strategies can be viewed as a single combined strategy for the base game with the joint-objective , or with each teacher ’ s own objective . In fact , the teachers provide an approximate best-response to each case of the base game simply by playing their individual best responses . Thus , when we reach a NE of the dual curriculum game , the teachers arrive at approximate best responses for both the base game with the joint objective and with their own objectives , meaning they are also in an approximate NE of the base game with either teacher . The full details of this proof are outlined in Appendix A .

Summary: In this paper, the authors propose a common framework,  Dual Curriculum Design, that augments PAIRED with a PLR-based replay mechanism, named as REPAIRED. The theory also suggests convergence to Nash Equilibrium should be assisted by training on fewer data when using PLR—namely by only taking policy-gradient updates from data that originates from the PLR buffer, and only using samples from the environment distribution to populate the buffer. The authors further conducted experiments on a maze environment and on car-racing tracks. Results showed that the proposed approaches outperformed the PLR and other baselines in general.