That Escalated Quickly: Compounding Complexity by Editing Levels at the Frontier of Agent Capabilities

1 INTRODUCTION . Reinforcement Learning ( RL , Sutton & Barto ( 1998 ) ) considers the problem of an agent learning from experience in an environment to maximize total ( discounted ) of reward . The past decade has seen a surge of interest in RL , with high profile successes in games ( Vinyals et al. , 2019 ; Berner et al. , 2019 ; Silver et al. , 2016 ; Mnih et al. , 2013 ; Hu & Foerster , 2020 ) and robotics ( OpenAI et al. , 2019 ; Andrychowicz et al. , 2020 ) . As such , there is tremendous excitement that RL may be a path towards generally capable agents ( Silver et al. , 2021 ) . Despite these successes , deploying RL agents in the real world remains a challenge ( Dulac-Arnold et al. , 2019 ) . Notably , strong training performance in simulation may not result in policies that are robust to the many sources of variation in the real world . Addressing this challenge on the agent side has become an active area of research ( Zhang et al. , 2021a ; Agarwal et al. , 2021a ; Raileanu & Fergus , 2021 ) , but in this paper we instead focus on the impact of the training environment itself , which often has a significant impact on agent ’ s ability to generalize ( Co-Reyes et al. , 2020 ) . For example in locomotion tasks , Reda et al . ( 2020 ) found that the initial state distribution , survival bonus , reward structure and control frequency had a significant impact on the performance of an agent . Indeed , curricula over environments can also influence the generalization performance of the agent ( Jiang et al. , 2021b ) . Throughout this paper we consider distributions of environments , referring to each individual sample as a level . Given a parameterized environment , the simplest approach one can consider is Domain Randomization ( DR , Jakobi , 1997 ; Tobin et al. , 2017 ; Sadeghi & Levine , 2017 ; Risi & Togelius , 2020 ; Peng et al. , 2017 ) , whereby an agent trains on individual levels uniformly sampled from an underlying environment distribution . It has been shown that training an agent with a DR-type approach can produce agents capable of complex real-world skills ( OpenAI et al. , 2019 ) . However , the performance of DR is only as good as the sampling distribution available—thus it can be ineffective when the probability of sampling useful levels is too low . Recently , Unsupervised Environment Design ( UED , Dennis et al. , 2020 ) has emerged as formalism for methods to design effective curricula . Given a parameterized environment , UED methods frame learning as a game between a teacher which generates a curriculum of levels , and a student seeking to maximize some notion of return . UED is a generalization of several other approaches . Indeed , DR can be considered as a UED algorithm whereby the teacher generates environments uniformly at random from the environment distribution . Other approaches to UED consider learning a teacher agent ( or generator ) , with a variety of adversarial objectives proposed ( Dennis et al. , 2020 ; Gur et al. , 2021 ) . However , training a teacher is a challenging optimization problem , suffering from both nonstationarity and sparse reward , as the teacher ’ s feedback only comes after evaluation by a changing student policy . Recent work showed it can be more effective to simply curate levels produced by DR ( Jiang et al. , 2021b ; a ; Matiisen et al. , 2020 ) , producing a curriculum of increasingly complex randomly generated levels . Despite their promise , these methods can only be as effective as the best of the random levels they sample , which can be a limitation in high dimensional design spaces . Finally , another series of promising works seek to evolve populations of environments ( Wang et al. , 2019 ; 2020 ; Dharna et al. , 2020 ) , but these methods heavily rely on handcrafted heuristics and also use up to 20x more compute since they also train a population of agents . In this paper , we seek a general method which harnesses the benefits of all three of these approaches . We posit the following : Rather than generate levels from scratch , it may be more effective to edit previously curated levels . Our primary contribution is to propose a new method which we call Adversarially Compounding Complexity by Editing Levels , or ACCEL . ACCEL is an evolutionary process , with levels constantly changing to remain at the frontier of the student agent ’ s capabilities ( see : Figure 2 ) . As such , levels generated by ACCEL begin simple but quickly become more complex . This benefits both the beginning of training ( Berthouze & Lungarella , 2004 ) , as the student begins learning much faster , while it also facilitates the construction of complex structures ( see Figure 1 ) . We believe ACCEL provides the best of both worlds : an evolutionary approach that can generate increasingly complex environments , combined with a regret-based curator which provides theoretical robustness guarantees in equilibrium . We evaluate ACCEL on a series of challenging procedurally generated grid world environments , where ACCEL demonstrates the ability to rapidly increase complexity while maintaining performance . Finally , we show ACCEL makes it possible to train agents capable of transfer to mazes an order of magnitude larger than training levels , achieving over double the success rate of the next best baseline . 2 BACKGROUND . 2.1 FROM MDPS TO UNDERSPECIFIED POMDPS . A Markov Decision Process ( MDP ) is defined as a tuple 〈S , A , T , R , γ〉 where S andA stand for the sets of states and actions respectively and T : S ×A→∆ ( S ) is a transition function representing the probability that the system/agent transitions from a state st ∈ S to st+1 ∈ S given action at ∈ A . Each transition also induces an associated reward rt generated by a reward functionR : S → R , and γ is a discount factor . When provided with an MDP , the goal of Reinforcement Learning ( RL , Sutton & Barto , 1998 ) is to learn a policy π that maximizes expected discounted reward , i.e . E [ ∑T i=0 rtγ t ] . Despite the generality of the MDP framework , it is often an unrealistic model for real world environments . First , it assumes full observability of the state , which is often impossible in practice . This is addressed in partially observable MDPs , or POMDPs , which include an observation function I : S → O which maps the true state ( which is unknown to the agent ) to a ( potentially noisy ) set of observations O. Secondly , the traditional MDP framework assumes a single reward and transition function , which are fixed throughout learning . Instead , in the real world , agents may experience variations not seen during training , which makes it crucial that policies are capable of robust transfer . To address both of these issues , we use the recently introduced Underspecified POMDP , or UPOMDP , given byM = 〈A , O , Θ , SM , TM , IM , RM , γ〉 . This definition is identical to a POMDP with the addition of Θ to represent the free parameters of the environment , similar to the context in a Contextual MDP ( Modi et al. , 2017 ) . These parameters can be distinct at every time step and incorporated into the transition function TM : S ×A×Θ→∆ ( S ) . Following Jiang et al . ( 2021a ) we define a levelMθ as an environment resulting from a fixed θ . We define the value of π inMθ to be V θ ( π ) = E [ ∑T i=0 rtγ t ] where rt are the rewards achieved by π inMθ . UPOMDPs benefit from their generality , since Θ can represent possible dynamics ( for example in sim2real ( Peng et al. , 2017 ; OpenAI et al. , 2019 ; Andrychowicz et al. , 2020 ) ) , changes in observations , different reward functions or differing game maps in procedurally generated environments . 2.2 METHODS FOR UNSUPERVISED ENVIRONMENT DESIGN . The goal of Unsupervised Environment Design ( UED , Dennis et al. , 2020 ) is to generate a series of levels that form a curriculum for a student agent , such that the student agent is capable of transfer , by maximizing some utility function Ut ( π , θ ) . In the case of DR , the utility function is simply : UUt ( π , θ ) = C ( 1 ) for any constant C. When learning a teacher , recent approaches proposed to use objectives seeking to maximize regret , defined as the difference between the expected return of the current policy and the optimal policy , ie : URt ( π , θ ) = argmax π∗∈Π { REGRETθ ( π , π∗ ) } = argmax π∗∈Π { V θ ( π∗ ) − V θ ( π ) } ( 2 ) Unlike other objectives , which may promote unsolvable environments , regret-based objectives have been shown to promote the simplest possible environments that the agent can not currently solve ( Dennis et al. , 2020 ) in a range of settings . However , since we do not have access to π∗ , a key challenge in UED algorithms utilizing objectives inspired by Equation 2 is to approximate the regret . Recently , the Prioritized Level Replay ( PLR , Jiang et al. , 2021b ; a ) algorithm introduced an additional teacher agent in the form of a curator , forming a “ dual curriculum game ” . The curator maintains a buffer of previously experienced levels and selects levels to be replayed by the student policy using objectives approximating regret . One of the objectives used by PLR is Positive Value Loss , given by : 1 T T∑ t=0 max ( T∑ k=t ( γλ ) k−tδk , 0 ) ( 3 ) where λ and γ are the Generalized Advantage Estimation ( GAE , Schulman et al . ( 2016 ) ) and MDP discount factors respectively , and δt , the TD-error at timestep t. Since Positive Value Loss approximates regret , if the student trains solely on curated levels ( i.e . does not take gradient steps on levels from the generator ) , then PLR achieves robustness guarantees in equilibrium . More formally , if St = Π is the strategy set of the student and St = Θ is the strategy set of the teacher ( in this case the curator ) , then ( by Corollary 1 of Jiang et al . ( 2021a ) ) , in equilibrium the resulting student policy π converges to a minimax regret policy , ie : π = argmin πA∈Π { max θ , πB∈Θ , Π { REGRETθ ( πA , πB ) } } ( 4 ) Empirically PLR has also been shown produce policies with strong generalization capabilities1 , yet it ’ s main weakness is that it still relies on randomly sampling useful levels . Next , we introduce our new approach which seeks to leverage the curator to produce batches of high regret levels . 1To see the impact of PLR on a simple example , we include a visualization in Figure 19 in the Appendix .

Summary: This paper introduces Adversarially Compounding Complexity by Editing Levels (ACCEL). ACCEL is an Unsupervised Environment Design (UED) algorithm, a method of generating a curriculum of environments so as to train agents that generalize well to either a training distribution of environments or off-distribution environments. ACCEL bares similarity to a recent addition in the UED literature, Robust PLR, but importantly uses an editor to modify previously seen environments. Edited levels are used if they satisfy a criterion based on the PLR score, and hence this can be thought of as a sort of evolutionary algorithm.  After introducing ACCEL, the paper presents a series of experiments in the Lava grid and Minigrid environments, benchmarking ACCEL against a suite of UED and simpler baselines. It demonstrates emergent complexity generated in algorithms' curricula (with metrics such as number of lava tiles/blocks, shortest path length) as well as generalization to held-out levels.