COLA: Consistent Learning with Opponent-Learning Awareness

Optimization problems with multiple , interdependent losses , such as Generative Adversarial Networks ( GANs ) or multi-agent RL , are commonly formalized as differentiable games . Learning with Opponent-Learning Awareness ( LOLA ) introduced opponent shaping to this setting . More specifically , LOLA introduced an augmented learning rule that accounts for the agent ’ s influence on the anticipated learning step of the other agents . However , the original LOLA formulation is inconsistent because LOLA models other agents as naive learners rather than LOLA agents . In previous work , this inconsistency was suggested as a root cause of LOLA ’ s failure to preserve stable fixed points ( SFPs ) . We show that , contrary to claims in previous work , Competitive Gradient Descent ( CGD ) does not solve the consistency problem and does not recover high-order LOLA ( HOLA ) as a series expansion . Working towards a remedy , we formalize consistency and show that HOLA is consistent whenever it converges ; however , it may fail to converge altogether . We propose a new method called Consistent LOLA ( COLA ) which learns update functions that are consistent under mutual opponent shaping . We prove that even such consistent update functions do not preserve SFPs , contradicting the hypothesis that this shortcoming is due to inconsistency . Finally , we empirically compare the performance and consistency of aforementioned algorithms on a range of general-sum learning games . 1 INTRODUCTION . Multi-objective problems can be found in many domains , such as GANs ( Goodfellow et al. , 2014 ) or single- and multi-agent reinforcement learning ( RL ) in the form of imaginative agents ( Racanière et al. , 2017 ) , hierarchical RL ( Barto & Mahadevan , 2002 ) , and intrinsic curiosity ( Schmidhuber , 1991 ) . A popular framework to understand systems with multiple , interdependent losses is differentiable games ( Balduzzi et al. , 2018 ) . For example , in the case of GANs , the differentiable game framework models the generator and the discriminator as competing agents , each trying to optimize their respective loss . The action space of the game consists of choosing the respective network parameters ( Balduzzi et al. , 2018 ) . An effective paradigm to improve learning in differentiable games is opponent shaping , where the players use their ability to shape each other ’ s learning steps . LOLA ( Foerster et al. , 2018 ) was the first work to make explicit use of opponent shaping in the differentiable game setting . LOLA is also one of the only general learning methods designed for differentiable games that obtains mutual cooperation with the Tit-for-Tat strategy in the Iterated Prisoner ’ s Dilemma ( IPD ) . The Tit-for-Tat strategy starts out cooperating and retaliates once whenever the opponent does not cooperate . It achieves mutual cooperation and has proven to be successful at IPD tournaments ( Axelrod , 1984 ; Harper et al. , 2017 ) . In contrast , naive gradient descent and other more sophisticated methods typically converge to the mutual defection policy under random initialization ( Letcher et al. , 2019b ) . While LOLA discovers these interesting equilibria , the original LOLA formulation is inconsistent because LOLA agents assume that their opponent is a naive learner . This assumption is clearly violated if two LOLA agents learn together in a game . It has been suggested that this inconsistency is the root cause for LOLA ’ s shortcomings , such as not converging to SFPs in some simple quadratic games ( Letcher 2018 , p. 2 , 26 ; see also Letcher et al . 2019b ) . Contributions . How can LOLA ’ s inconsistency be resolved ? To answer this question , we first revisit the concept of higher-order LOLA ( HOLA ) ( Foerster et al. , 2018 ) in Section 4.1 . For example , second-order LOLA assumes that the opponent is a first-order LOLA agent ( which in turn assumes the opponent is a naive learner ) and so on . Assuming that HOLA converges with increasing order , we define infinite-order LOLA ( iLOLA ) as the limit of HOLA whenever it exists . Intuitively , it should follow that two iLOLA agents have a consistent view of each other , meaning they make an accurate assumption about the learning behavior of the opponent under mutual opponent shaping . We introduce a formal definition of consistency and prove that iLOLA is indeed self-consistent under mutual opponent shaping . Previous work has claimed that a series expansion of Competitive Gradient Descent ( CGD ) ( Schäfer & Anandkumar , 2020 ) recovers high-order LOLA . This would imply that CGD corresponds to iLOLA and thus solves the consistency problem . In Section 4.2 , we prove that this is false : CGD does not in general correspond to iLOLA , and , unlike iLOLA , does not resolve the problem of consistency . In particular , we show that , contrary to previous claims , the series expansion of CGD does not correspond to higher-order LOLA . There are a number of problems with addressing consistency using a limiting update ( iLOLA ) : the process may not converge , and requires computation of arbitrarily high derivatives . In Section 4.3 , we propose Consistent LOLA ( COLA ) as a more general and efficient alternative . Instead of repeatedly applying the LOLA learning rule ( iLOLA ) , COLA learns a pair of consistent update functions by explicitly minimizing a consistency loss . By reframing the problem as such , the method only requires up to second-order derivatives , and instead of having a handcrafted update function as for LOLA or CGD , we use the representation power of neural networks to learn the update step . In Section 4.4 , we prove initial results about COLA . First , we show that COLA ’ s solutions are not necessarily unique . Second , despite being consistent , COLA does not recover SFPs , contradicting the prior belief that this shortcoming is caused by inconsistency . Third , we provide an example in which COLA converges more robustly , i.e. , under a wider range of learning rates , than LOLA . Finally , in Sections 5 and 6 , we report our experimental setup and results , investigating COLA and HOLA and comparing it to LOLA and CGD in a range of games . We show that , despite its nonuniqueness , COLA tends to find similar solutions in different runs empirically . Moreover , we show that COLA finds the iLOLA solution when HOLA converges but finds different solutions when HOLA diverges . These solutions have lower consistency loss and converge under a broader range of learning rates than LOLA and HOLA . Our experiments also show that , while COLA does not find Tit-for-Tat on the IPD ( unlike LOLA ) , it does learn policies with near-optimal total payoff . 2 RELATED WORK . General-sum learning algorithms and their consequences have been investigated from different perspectives in the reinforcement learning , game theory , and GAN literature , see e.g . ( Schmidhuber , 1991 ; Barto & Mahadevan , 2002 ; Racanière et al. , 2017 ; Goodfellow et al. , 2014 ) to name a few . Next , we will highlight a few of the approaches to the mutual opponent shaping problem . Opponent modeling maintains an explicit belief of the opponent , which allows to reason over their strategies and compute optimal responses . Opponent modeling can be divided into different subcategories : There are classification methods , classifying the opponents into pre-defined types ( Weber & Mateas , 2009 ; Synnaeve & Bessiere , 2011 ) , or policy reconstruction methods , where we explicitly predict the actions of the opponent ( Mealing & Shapiro , 2017 ) . Most closely related to opponent shaping is recursive reasoning , where methods model nested beliefs of the opponents ( He et al. , 2016 ; Albrecht & Stone , 2019 ; Wen et al. , 2019 ) . In comparison , COLA assumes that we have access to the ground-truth model of the opponent , e.g. , the opponent ’ s payoff function , parameters , and gradients , which puts COLA into the framework of differentiable games ( Balduzzi et al. , 2018 ) . Various methods have been proposed , investigating the local convergence properties to different solution concepts ( Mescheder et al. , 2018 ; Mazumdar et al. , 2019 ; Letcher et al. , 2019b ; Azizian et al. , 2020 ; Schäfer & Anandkumar , 2020 ; Schäfer et al. , 2020 ; Hutter , 2020 ) . Most of the work in differentiable games has not focused on the issue of opponent shaping and consistency . Mescheder et al . ( 2018 ) and Mazumdar et al . ( 2019 ) focus solely on zerosum games without shaping . Letcher et al . ( 2019b ) improve on LOLA , but do not investigate the consistency issue . CGD ( Schäfer & Anandkumar , 2020 ) addresses the consistency issue of LOLA for zero-sum games but not for general-sum games . The exact difference between CGD and LOLA is addressed in the Section 4.2 . 3 BACKGROUND . 3.1 DIFFERENTIABLE GAMES . The framework of differentiable games has become increasingly popular to model the problem of multi-agent learning . Whereas in the framework of stochastic games we are typically limited to parameters such as action-state probabilities , differentiable game generalizes to any parameters as long as the loss function is differentiable with respect to them ( Balduzzi et al. , 2018 ) . We restrict our attention on two-player games , as is standard in the current differentable games literature . Definition 1 ( Differentiable games ) . In a two-player differentiable game , players i = 1 , 2 control parameters θi ∈ Rdi to minimize twice continuously differentiable losses Li : Rd1+d2 → R. We adopt the convention to write −i to denote the respective other player . A fundamental challenge of the multi-loss setting is finding a meaningful solution concept . Whereas in the single loss setting the typical solution concept is local minima , in multi-loss settings there are different sensible solution concepts . Most prominently , there are Nash Equilibria ( Osborne & Rubinstein , 1994 ) . However , Nash Equilibria include unstable saddle points that can not be reasonably found via gradient-based learning algorithms ( Letcher et al. , 2019b ) . A more appropriate concept are stable fixed points ( SFPs ) , which could be considered a differentiable game analogon to local minima in single loss optimization . We will omit a formal definition here for brevity and point the interested reader to previous work on the topic ( Letcher et al. , 2019a ) . 3.2 LOLA AND SOS . Consider a differentiable game with two players . A LOLA agent θ1 uses its access to the opponent ’ s parameters θ2 to differentiate through the learning step of the opponent . In other words , agent 1 reformulates their loss to L1 ( θ1 , θ2 + ∆θ2 ) , where ∆θ2 represents the assumed learning step of the opponent . In first-order LOLA we assume the opponent to be a naive learner : ∆θ2 = −α∇2L2 , which is what makes LOLA inconsistent if the opponent was any other type of learner . Note that ∇2 denotes the gradient with respect to θ2 . Also note that α represents the look-ahead rate , which is the assumed learning rate of the opponent . In the original paper the loss was approximated using a Taylor expansion L1 + ( ∇2L1 ) > ∆θ2 . For agent 1 , their first-order ( Taylor ) LOLA update is then defined as ∆θ1 : = −α ( ∇1L1 +∇12L1∆θ2 + ( ∇1∆θ2 ) > ∇2L1 ) . Alternatively , in exact LOLA , the derivative is taken directly with respect to L1 ( θ1 , θ2 + ∆θ2 ) . LOLA has had some empirical success , being one of the first general learning methods to discover Tit-for-Tat like solutions in social dilemmas . However , later work showed that LOLA does not preserve SFPs θ̄ since the rightmost term can be nonzero at θ̄ . In fact , LOLA agents show “ arrogant ” behavior : they assume they can shape the learning of their naive opponents without having to adapt to the shaping of the opponent . Prior work hypothesized that this arrogant behavior is due to LOLA ’ s inconsistent formulation ( Letcher 2018 , p. 2 , 26 ; see also Letcher et al . 2019b ) . To improve upon LOLA , Letcher et al . ( 2019b ) have suggested the Stable Opponent Shaping ( SOS ) algorithm . SOS applies a correction to the LOLA update , leading to theoretically guaranteed convergence to SFPs . However , despite its desirable convergence properties , SOS still does not solve the conceptual issue of inconsistent assumptions about the opponent .

Summary: This paper investigates the inconsistency problem in LOLA: each LOLA agent assumes the other agent as a naive learner, resulting in LOLA agents not converging to SFPs in some games. This paper aims to address this problem by the infinite-order LOLA, which can have a consistent view of each other and empirically observes that HOLA may not resolve LOLA's convergence issues. Instead of HOLA, this paper proposes COLA that employs neural networks to explicitly minimize the consistency loss. COLA empirically shows that COLA finds the consistent solution when HOLA converges and finds more stable solutions when HOLA diverges.