Replay-Guided Adversarial Environment Design

1 Introduction . While deep reinforcement learning ( RL ) approaches have led to many successful applications in challenging domains like Atari [ 21 ] , Go [ 35 ] , Chess [ 36 ] , Dota [ 4 ] , and StarCraft [ 40 ] in recent years , deep RL agents still prove to be brittle , often failing to transfer to environments only slightly different from those encountered during training [ 44 , 9 ] . To ensure learning of robust and well-generalizing policies , agents must train on sufficiently diverse and informative variations of environments ( e.g . see Section 3.1 of [ 8 ] ) . However , it is not always feasible to specify an appropriate training distribution or a generator thereof . Agents may therefore benefit greatly from methods that automatically adapt the distribution over environment variations throughout training [ 10 , 17 ] . Throughout this paper we will call a particular environment instance or configuration ( e.g . an arrangement of blocks , race tracks , or generally any of the environment ’ s constituent entities ) a level . Two recent works [ 10 , 17 ] have sought to empirically demonstrate this need for a more targeted agentadaptive mechanism for selecting levels on which to train RL agents , so to ensure efficient learning and generalization to unseen levels—as well as to provide methods implementing such mechanisms . The first method , Protagonist Antagonist Induced Regret Environment Design ( PAIRED ) [ 10 ] , ∗Equal contribution . Correspondence to msj @ fb.com and michael_dennis @ berkeley.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . introduces a self-supervised RL paradigm called Unsupervised Environment Design ( UED ) . Here , an environment generator ( a teacher ) is co-evolved with a student policy that trains on levels actively proposed by the teacher , leading to a form of adaptive curriculum learning . The aim of this coevolution is for the teacher to gradually learn to generate environments that exemplify properties of those that might be encountered at deployment time , and for the student to simultaneously learn a good policy that enables zero-shot transfer to such environments . PAIRED ’ s specific adversarial approach to environment design ensures a useful robustness characterization of the final student policy in the form of a minimax regret guarantee [ 31 ] —assuming that its underlying teacher-student multi-agent system arrives at a Nash equilibrium [ NE , 24 ] . In contrast , the second method , Prioritized Level Replay ( PLR ) [ 17 ] , embodies an alternative form of dynamic curriculum learning that does not assume control of level generation , but instead , the ability to selectively replay existing levels . PLR tracks levels previously proposed by a black-box environment generator , and for each , estimates the agent ’ s learning potential in that level , in terms of how useful it would be to gather new experience from that level again in the future . The PLR algorithm exploits these scores to adapt a schedule for revisiting or replaying levels to maximize learning potential . PLR has been shown to produce scalable and robust results , improving both sample complexity of agent training and the generalization of the learned policy in diverse environments . However , unlike PAIRED , PLR is motivated with heuristic arguments and lacks a useful theoretical characterization of its learning behavior . In this paper , we argue that PLR is , in and of itself , an effective form of UED : Through curating even randomly generated levels , PLR can generate novel and complex levels for learning robust policies . This insight leads to a natural class of UED methods which we call Dual Curriculum Design ( DCD ) . In DCD , a student policy is challenged by a team of two co-evolving teachers . One teacher actively generates new , challenging levels , while the other passively curates existing levels for replaying , by prioritizing those estimated to be most suitably challenging for the student . We show that PAIRED and PLR are distinct members of the DCD class of algorithms and prove in Section 3 that all DCD algorithms enjoy similar minimax regret guarantees to that of PAIRED . We make use of this result to provide the first theoretical characterization of PLR , which immediately suggests a simple yet highly counterintuitive adjustment to PLR : By only training on trajectories in replay levels , PLR becomes provably robust at NE . We call this resulting variant PLR⊥ ( Section 4 ) . From this perspective , PLR effectively performs level design in a diametrically opposite manner to PAIRED—through prioritized selection rather than active generation . A second corollary to the provable robustness of DCD algorithms shows that PLR⊥ can be extended to make use of the PAIRED teacher as a level generator while preserving the robustness guarantee of PAIRED , resulting in a method we call Replay-Enhanced PAIRED ( REPAIRED ) ( Section 5 ) . We hypothesize that in this arrangement , PLR⊥ plays a complementary role to PAIRED in robustifying student policies . Our experiments in Section 6 investigate the learning dynamics of PLR⊥ , REPAIRED , and their replay-free counterparts on a challenging maze domain and a novel continuous control UED setting based on the popular CarRacing environment [ 5 ] . In both of these highly distinct settings , our methods provide significant improvements over PLR and PAIRED , producing agents that can perform out-of-distribution ( OOD ) generalization to a variety of human designed mazes and Formula 1 tracks . In summary , we present the following contributions : ( i ) We establish a common framework , Dual Curriculum Design , that encompasses PLR and PAIRED . This allows us to develop new theory , which provides the first robustness guarantees for PLR at NE as well as for REPAIRED , which augments PAIRED with a PLR-based replay mechanism . ( ii ) Crucially , our theory suggests a highly counterintuitive improvement to PLR : the convergence to NE should be assisted by training on less data when using PLR—namely by only taking gradient updates from data that originates from the PLR buffer , using the samples from the environment distribution only for computing the prioritization of levels in the buffer . ( iii ) Our experiments in a maze domain and a novel car racing domain show that our methods significantly outperform their replay-free counterparts in zero-shot generalization . We open source our methods at https : //github.com/facebookresearch/dcd . 2 Background . 2.1 Unsupervised Environment Design . Unsupervised Environment Design ( UED ) , as introduced by [ 10 ] , is the problem of automatically designing a distribution of environments that adapts to the learning agent . UED is defined in terms of an Underspecified POMDP ( UPOMDP ) , given byM = 〈A , O , Θ , SM , TM , IM , RM , γ〉 , where A is a set of actions , O is a set of observations , S is a set of states , T : S × A × Θ → ∆ ( S ) is a transition function , I : S → O is an observation ( or inspection ) function , R : S → R is a reward function , and γ is a discount factor . This definition is identical to a POMDP with the addition of Θ to represent the free-parameters of the environment . These parameters can be distinct at every time step and incorporated into the transition function TM : S ×A×Θ→∆ ( S ) . For example , Θ could represent the possible positions of obstacles in a maze . We will refer to the environment resulting from a fixed θ ∈ Θ asMθ , or with a slight abuse of notation , simply θ when clear from context . We define the value of π inMθ to be V θ ( π ) = E [ ∑T i=0 rtγ t ] where rt are the rewards attained by π in Mθ . Aligning with terminology from [ 17 ] , we refer to a fully-specified environment as a level . 2.2 Protagonist Antagonist Induced Regret Environment Design . Protagonist Antagonist Induced Regret Environment Design [ PAIRED , 10 ] presents a UED approach consisting of simultaneously training agents in a three player game : the protagonist πA and the antagonist πB are trained in environments generated by the teacher θ̃ . The objective of this game is defined by U ( πA , πB , θ̃ ) = Eθ∼θ̃ [ REGRET θ ( πA , πB ) ] , where regret is defined by REGRETθ ( πA , πB ) = V θ ( πB ) − V θ ( πA ) . The protagonist and antagonist are both trained to maximize their discounted environment returns while the teacher is trained to maximize U . Note that by maximizing regret , the teacher is disincentivized from generating unsolvable levels , which will have a maximum regret of 0 . As shorthand , we will sometimes refer to the protagonist and antagonist jointly as the student agents . The counterclockwise loop beginning at the student agents in Figure 2 summarizes this approach , with the students being both the protagonist and antagonist . As both student agents grow more adept at solving different levels , the teacher continues to adapt its level designs to exploit the weaknesses of the protagonist in relation to the antagonist . As this dynamic unfolds , PAIRED produces an emergent curriculum of progressively more complex levels along the boundary of the protagonist ’ s capabilities . PAIRED is a creative method in the sense that the teacher may potentially generate an endless sequence of novel levels . However , as the teacher only adapts through gradient updates , it is inherently slow to adapt to changes in the student policies . 2.3 Prioritized Level Replay . Prioritized Level Replay [ PLR , 17 ] is an active-learning strategy shown to improve a policy ’ s sample efficiency and generalization to unseen levels when training and evaluating on levels from a common UPOMDP , typically implemented as a seeded simulator . PLR maintains a level buffer Λ of the top K visited levels with highest learning potential as estimated by the time-averaged L1 value loss of the learning agent over the last episode on each level . At the start of each training episode , with some predefined replay probability p , PLR uses a bandit to sample the level from Λ to maximize the estimated learning potential ; otherwise , with probability 1− p , PLR samples a new level from the simulator . In contrast to the generative but slow-adapting PAIRED , PLR does not create new levels , but instead , acts as a fast-adapting curation mechanism for selecting the next training level among previously encountered levels . Also unlike PAIRED , PLR does not provide a robustness guarantee . By extending the theoretical foundation of PAIRED to PLR , we will show how PLR can be modified to provide a robustness guarantee at NE , as well as how PAIRED can exploit PLR ’ s complementary curation to quickly switch among generated levels to maximize the student ’ s regret . 3 The Robustness of Dual Curriculum Design . The previous approaches of PAIRED and PLR reveal a natural duality : Approaches that gradually learn to generate levels like PAIRED , and methods which can not generate levels , but instead , quickly curate existing ones , like PLR . This duality suggests combining slow level generators with fast level curators . We call this novel class of UED algorithms Dual Curriculum Design ( DCD ) . For instance , PLR can be seen as curator with a prioritized sampling mechanism with a random generator , while PAIRED , as a regret-maximizing generator without a curator . DCD can further consider Domain Randomization ( DR ) as a degenerate case of a random level generator without a curator . To theoretically analyze this space of methods , we model DCD as a three player game among a student agent and two teachers called the dual curriculum game . However , to formalize this game , we must first formalize the single-teacher setting : Suppose the UPOMDP is clear from context . Then , given a utility function for a single teacher , Ut ( π , θ ) , we can naturally define the base game between the student s and teacher t as G = 〈S = Ss × St , U = Us × Ut〉 , where Ss = Π is the strategy set of the student , St = Θ is the strategy set of the teacher , and Us ( π , θ ) = V θ ( π ) is the utility function of the student . In Sections 4 and 5 , we will study settings corresponding to different choices of utility functions for the teacher agents , namely the maximum-regret objective URt ( π , θ ) and the uniform objective UUt ( π , θ ) . These two objectives are defined as follows ( for any constant C ) : URt ( π , θ ) = argmax π∗∈Π { V θ ( π∗ ) − V θ ( π ) } ( 1 ) UUt ( π , θ ) = C ( 2 ) In the dual curriculum game G , the first teacher plays the game with probability p , and the second , with probability ( 1− p ) —or more formally , G = 〈S = Ss × St × St , U = Us × U 1 t × U 2 t 〉 , where the utility functions for the student and two teachers respectively , Us , U 1 t , U 2 t , are defined as follows : U 1 t ( π , θ 1 , θ2 ) = pU1t ( π , θ 1 ) ( 3 ) U 2 t ( π , θ 1 , θ2 ) = ( 1− p ) U2t ( π , θ2 ) ( 4 ) Us ( π , θ 1 , θ2 ) = pUs ( π , θ 1 ) + ( 1− p ) Us ( π , θ2 ) ( 5 ) Our main theorem is that NE in the dual curriculum game are approximate NE of both the base game for either of the original teachers and the base game with a teacher maximizing the joint-reward of pU1t + ( 1− p ) U2t , where the quality of the approximations depends on the mixing probability p. Theorem 1 . Let B be the maximum difference between U1t and U2t , and let ( π , θ1 , θ2 ) be a NE for G. Then ( π , pθ1 + ( 1− p ) θ2 ) is an approximate NE for the base game with either teacher or for a teacher optimizing their joint objective . More precisely , it is a 2Bp ( 1− p ) -approximate NE when Ut = pU 1 t + ( 1− p ) U2t , a 2B ( 1− p ) -approximate NE when Ut = U1t , and a 2Bp-approximate NE when Ut = U2t . The intuition behind this theorem is that , since the two teachers do not affect each other ’ s behavior , their best response to a fixed πs is to choose a strategy θ that maximizes U1t and U 2 t respectively . Moreover , the two teachers ’ strategies can be viewed as a single combined strategy for the base game with the joint-objective , or with each teacher ’ s own objective . In fact , the teachers provide an approximate best-response to each case of the base game simply by playing their individual best responses . Thus , when we reach a NE of the dual curriculum game , the teachers arrive at approximate best responses for both the base game with the joint objective and with their own objectives , meaning they are also in an approximate NE of the base game with either teacher . The full details of this proof are outlined in Appendix A .

Summary: This paper considers a recent proposed problem, unsupervised environment design, that provides an environment selection scheme to enable the policy being trained under a curriculum learning setting. The proposed method combines two new approaches in this direction, PAIRED and PLR, where the former one learns to generate an environment while the latter one does not include a generator but instead gives a sampling scheme to choose previously encountered environments. The combined method is explained as a two-teacher dual curriculum game and explained theoretically. Experiments demonstrate the effectiveness of the combined method compared to PAIRED and PLR, respectively.