COLA: Consistent Learning with Opponent-Learning Awareness

Optimization problems with multiple , interdependent losses , such as Generative Adversarial Networks ( GANs ) or multi-agent RL , are commonly formalized as differentiable games . Learning with Opponent-Learning Awareness ( LOLA ) introduced opponent shaping to this setting . More specifically , LOLA introduced an augmented learning rule that accounts for the agent ’ s influence on the anticipated learning step of the other agents . However , the original LOLA formulation is inconsistent because LOLA models other agents as naive learners rather than LOLA agents . In previous work , this inconsistency was suggested as a root cause of LOLA ’ s failure to preserve stable fixed points ( SFPs ) . We show that , contrary to claims in previous work , Competitive Gradient Descent ( CGD ) does not solve the consistency problem and does not recover high-order LOLA ( HOLA ) as a series expansion . Working towards a remedy , we formalize consistency and show that HOLA is consistent whenever it converges ; however , it may fail to converge altogether . We propose a new method called Consistent LOLA ( COLA ) which learns update functions that are consistent under mutual opponent shaping . We prove that even such consistent update functions do not preserve SFPs , contradicting the hypothesis that this shortcoming is due to inconsistency . Finally , we empirically compare the performance and consistency of aforementioned algorithms on a range of general-sum learning games . 1 INTRODUCTION . Multi-objective problems can be found in many domains , such as GANs ( Goodfellow et al. , 2014 ) or single- and multi-agent reinforcement learning ( RL ) in the form of imaginative agents ( Racanière et al. , 2017 ) , hierarchical RL ( Barto & Mahadevan , 2002 ) , and intrinsic curiosity ( Schmidhuber , 1991 ) . A popular framework to understand systems with multiple , interdependent losses is differentiable games ( Balduzzi et al. , 2018 ) . For example , in the case of GANs , the differentiable game framework models the generator and the discriminator as competing agents , each trying to optimize their respective loss . The action space of the game consists of choosing the respective network parameters ( Balduzzi et al. , 2018 ) . An effective paradigm to improve learning in differentiable games is opponent shaping , where the players use their ability to shape each other ’ s learning steps . LOLA ( Foerster et al. , 2018 ) was the first work to make explicit use of opponent shaping in the differentiable game setting . LOLA is also one of the only general learning methods designed for differentiable games that obtains mutual cooperation with the Tit-for-Tat strategy in the Iterated Prisoner ’ s Dilemma ( IPD ) . The Tit-for-Tat strategy starts out cooperating and retaliates once whenever the opponent does not cooperate . It achieves mutual cooperation and has proven to be successful at IPD tournaments ( Axelrod , 1984 ; Harper et al. , 2017 ) . In contrast , naive gradient descent and other more sophisticated methods typically converge to the mutual defection policy under random initialization ( Letcher et al. , 2019b ) . While LOLA discovers these interesting equilibria , the original LOLA formulation is inconsistent because LOLA agents assume that their opponent is a naive learner . This assumption is clearly violated if two LOLA agents learn together in a game . It has been suggested that this inconsistency is the root cause for LOLA ’ s shortcomings , such as not converging to SFPs in some simple quadratic games ( Letcher 2018 , p. 2 , 26 ; see also Letcher et al . 2019b ) . Contributions . How can LOLA ’ s inconsistency be resolved ? To answer this question , we first revisit the concept of higher-order LOLA ( HOLA ) ( Foerster et al. , 2018 ) in Section 4.1 . For example , second-order LOLA assumes that the opponent is a first-order LOLA agent ( which in turn assumes the opponent is a naive learner ) and so on . Assuming that HOLA converges with increasing order , we define infinite-order LOLA ( iLOLA ) as the limit of HOLA whenever it exists . Intuitively , it should follow that two iLOLA agents have a consistent view of each other , meaning they make an accurate assumption about the learning behavior of the opponent under mutual opponent shaping . We introduce a formal definition of consistency and prove that iLOLA is indeed self-consistent under mutual opponent shaping . Previous work has claimed that a series expansion of Competitive Gradient Descent ( CGD ) ( Schäfer & Anandkumar , 2020 ) recovers high-order LOLA . This would imply that CGD corresponds to iLOLA and thus solves the consistency problem . In Section 4.2 , we prove that this is false : CGD does not in general correspond to iLOLA , and , unlike iLOLA , does not resolve the problem of consistency . In particular , we show that , contrary to previous claims , the series expansion of CGD does not correspond to higher-order LOLA . There are a number of problems with addressing consistency using a limiting update ( iLOLA ) : the process may not converge , and requires computation of arbitrarily high derivatives . In Section 4.3 , we propose Consistent LOLA ( COLA ) as a more general and efficient alternative . Instead of repeatedly applying the LOLA learning rule ( iLOLA ) , COLA learns a pair of consistent update functions by explicitly minimizing a consistency loss . By reframing the problem as such , the method only requires up to second-order derivatives , and instead of having a handcrafted update function as for LOLA or CGD , we use the representation power of neural networks to learn the update step . In Section 4.4 , we prove initial results about COLA . First , we show that COLA ’ s solutions are not necessarily unique . Second , despite being consistent , COLA does not recover SFPs , contradicting the prior belief that this shortcoming is caused by inconsistency . Third , we provide an example in which COLA converges more robustly , i.e. , under a wider range of learning rates , than LOLA . Finally , in Sections 5 and 6 , we report our experimental setup and results , investigating COLA and HOLA and comparing it to LOLA and CGD in a range of games . We show that , despite its nonuniqueness , COLA tends to find similar solutions in different runs empirically . Moreover , we show that COLA finds the iLOLA solution when HOLA converges but finds different solutions when HOLA diverges . These solutions have lower consistency loss and converge under a broader range of learning rates than LOLA and HOLA . Our experiments also show that , while COLA does not find Tit-for-Tat on the IPD ( unlike LOLA ) , it does learn policies with near-optimal total payoff . 2 RELATED WORK . General-sum learning algorithms and their consequences have been investigated from different perspectives in the reinforcement learning , game theory , and GAN literature , see e.g . ( Schmidhuber , 1991 ; Barto & Mahadevan , 2002 ; Racanière et al. , 2017 ; Goodfellow et al. , 2014 ) to name a few . Next , we will highlight a few of the approaches to the mutual opponent shaping problem . Opponent modeling maintains an explicit belief of the opponent , which allows to reason over their strategies and compute optimal responses . Opponent modeling can be divided into different subcategories : There are classification methods , classifying the opponents into pre-defined types ( Weber & Mateas , 2009 ; Synnaeve & Bessiere , 2011 ) , or policy reconstruction methods , where we explicitly predict the actions of the opponent ( Mealing & Shapiro , 2017 ) . Most closely related to opponent shaping is recursive reasoning , where methods model nested beliefs of the opponents ( He et al. , 2016 ; Albrecht & Stone , 2019 ; Wen et al. , 2019 ) . In comparison , COLA assumes that we have access to the ground-truth model of the opponent , e.g. , the opponent ’ s payoff function , parameters , and gradients , which puts COLA into the framework of differentiable games ( Balduzzi et al. , 2018 ) . Various methods have been proposed , investigating the local convergence properties to different solution concepts ( Mescheder et al. , 2018 ; Mazumdar et al. , 2019 ; Letcher et al. , 2019b ; Azizian et al. , 2020 ; Schäfer & Anandkumar , 2020 ; Schäfer et al. , 2020 ; Hutter , 2020 ) . Most of the work in differentiable games has not focused on the issue of opponent shaping and consistency . Mescheder et al . ( 2018 ) and Mazumdar et al . ( 2019 ) focus solely on zerosum games without shaping . Letcher et al . ( 2019b ) improve on LOLA , but do not investigate the consistency issue . CGD ( Schäfer & Anandkumar , 2020 ) addresses the consistency issue of LOLA for zero-sum games but not for general-sum games . The exact difference between CGD and LOLA is addressed in the Section 4.2 . 3 BACKGROUND . 3.1 DIFFERENTIABLE GAMES . The framework of differentiable games has become increasingly popular to model the problem of multi-agent learning . Whereas in the framework of stochastic games we are typically limited to parameters such as action-state probabilities , differentiable game generalizes to any parameters as long as the loss function is differentiable with respect to them ( Balduzzi et al. , 2018 ) . We restrict our attention on two-player games , as is standard in the current differentable games literature . Definition 1 ( Differentiable games ) . In a two-player differentiable game , players i = 1 , 2 control parameters θi ∈ Rdi to minimize twice continuously differentiable losses Li : Rd1+d2 → R. We adopt the convention to write −i to denote the respective other player . A fundamental challenge of the multi-loss setting is finding a meaningful solution concept . Whereas in the single loss setting the typical solution concept is local minima , in multi-loss settings there are different sensible solution concepts . Most prominently , there are Nash Equilibria ( Osborne & Rubinstein , 1994 ) . However , Nash Equilibria include unstable saddle points that can not be reasonably found via gradient-based learning algorithms ( Letcher et al. , 2019b ) . A more appropriate concept are stable fixed points ( SFPs ) , which could be considered a differentiable game analogon to local minima in single loss optimization . We will omit a formal definition here for brevity and point the interested reader to previous work on the topic ( Letcher et al. , 2019a ) . 3.2 LOLA AND SOS . Consider a differentiable game with two players . A LOLA agent θ1 uses its access to the opponent ’ s parameters θ2 to differentiate through the learning step of the opponent . In other words , agent 1 reformulates their loss to L1 ( θ1 , θ2 + ∆θ2 ) , where ∆θ2 represents the assumed learning step of the opponent . In first-order LOLA we assume the opponent to be a naive learner : ∆θ2 = −α∇2L2 , which is what makes LOLA inconsistent if the opponent was any other type of learner . Note that ∇2 denotes the gradient with respect to θ2 . Also note that α represents the look-ahead rate , which is the assumed learning rate of the opponent . In the original paper the loss was approximated using a Taylor expansion L1 + ( ∇2L1 ) > ∆θ2 . For agent 1 , their first-order ( Taylor ) LOLA update is then defined as ∆θ1 : = −α ( ∇1L1 +∇12L1∆θ2 + ( ∇1∆θ2 ) > ∇2L1 ) . Alternatively , in exact LOLA , the derivative is taken directly with respect to L1 ( θ1 , θ2 + ∆θ2 ) . LOLA has had some empirical success , being one of the first general learning methods to discover Tit-for-Tat like solutions in social dilemmas . However , later work showed that LOLA does not preserve SFPs θ̄ since the rightmost term can be nonzero at θ̄ . In fact , LOLA agents show “ arrogant ” behavior : they assume they can shape the learning of their naive opponents without having to adapt to the shaping of the opponent . Prior work hypothesized that this arrogant behavior is due to LOLA ’ s inconsistent formulation ( Letcher 2018 , p. 2 , 26 ; see also Letcher et al . 2019b ) . To improve upon LOLA , Letcher et al . ( 2019b ) have suggested the Stable Opponent Shaping ( SOS ) algorithm . SOS applies a correction to the LOLA update , leading to theoretically guaranteed convergence to SFPs . However , despite its desirable convergence properties , SOS still does not solve the conceptual issue of inconsistent assumptions about the opponent .

Summary: The authors tackle the consistency problem of the original LOLA formulation. The paper investigates HOLA convergence, demonstrates that CGD does not correspond to high-order LOLA in general, and proposes COLA to directly address the consistency problem. The proposed method COLA seems more robust to different look-ahead values where HOLA diverges. The authors also find that COLA is still sometime susceptible to the ‘arrogant’ LOLA behavior, opening questions for future work.