Online Learning in Periodic Zero-Sum Games

1 Introduction . The study of learning dynamics in zero-sum games is arguably as old of a field as game theory itself , dating back to the seminal work of Brown and Robinson [ 7 , 27 ] , which followed shortly after the foundational minmax theorem of von Neumann [ 33 ] . The dynamics of online no-regret learning algorithms [ 10 , 29 ] are of particular interest in zero-sum games as they are designed with an adversarial environment in mind . Moreover , well known results imply that such dynamics converge in a time-average sense to a minmax equilibrium in zero-sum games [ 10 , 15 ] . Despite the classical nature of the study of online no-regret learning dynamics in zero-sum games , the actual transient behavior of such dynamics was historically not as understood . However , in the past several years this topic has gained attention with a number of works studying such dynamics in zerosum games ( and variants thereof ) with a particular focus on continuous-time analysis [ 24 , 25 , 20 , 6 , 32 , 23 , 22 ] . The unifying emergent picture is that the dynamics are “ approximately cyclic '' in a formal sense known as Poincaré recurrence . Moreover , these results have acted as fundamental building blocks for understanding the limiting behavior of their discrete-time variants [ 2 , 12 , 21 , 11 , 3 ] . ∗Joint first authors †Joint last authors 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite the plethora of emerging results regarding online learning dynamics in zero-sum games , an important and well motivated aspect of this problem has only begun to receive attention . How do online learning dynamics behave if the zero-sum game evolves over time ? Clearly , the answer to this question depends critically on how the game is allowed to evolve . Problem Setting and Model . We study periodic zero-sum games with a time-invariant equilibrium , which is a class of games we formally define in Section 2 . In a periodic zero-sum game , the payoffs that dictate the game are both T -periodic and zero-sum at all times . We consider both periodic zero-sum bilinear games on infinite , unconstrained strategy spaces and periodic zero-sum matrix games ( along with network generalizations thereof ) on finite strategy spaces . The goal of this work is to evaluate the robustness of the archetypal online learning behaviors in zero-sum games , Poincaré recurrence and time-average equilibrium convergence , to this natural model of game evolution . Connections to Repeated Game Models . The time-evolving game model we study can be seen as a generalization of usual repeated game formulations . A time-invariant game is a trivial version of a periodic game , in which case we recover the repeated static game setting . For a general periodic zero-sum game with period T , each stage game now is chosen according to a fixed length T sequence of games , capturing interactions between the players with time-dependent payoffs . Periodic zero-sum games can also fit into the frameworks of multi-agent contextual games [ 28 ] and dynamic games ( see , e.g. , [ 5 ] ) . In a multi-agent contextual game [ 28 ] , the environment selects a context from a set before each round of play and this choice defines the game that is played . Periodic zero-sum games can be seen as a multi-agent contextual game where the environment draws contexts from the available set in a T -periodic fashion with each context defining a zero-sum game with a common equilibrium . In the class of dynamic games , there is a game state on which the payoffs may depend that evolves with dynamics . Periodic zero-sum games can be interpreted as a dynamic game where the state transitions do not depend on the strategies of the players , the state is T -periodic , and the payoffs are completely defined by the state . We remark that the focus of existing work on contextual games and dynamic games is distinct from the questions investigated in this paper . The periodic zero-sum game model allows us to capture competitive settings where exogenous environmental variations manifest in an effectively periodic/epochal fashion . This naturally occurs in market competitions where time-of-day effects , week-to-week trends , and seasonality can dictate the game between players . To illustrate this point , consider a competition between service providers that wish to maximize their users , while the total market size evolves seasonally over time . This evolution affects the utility functions , even if the fundamentals of the market , and consequently the equilibrium , remain invariant . Contributions and Approach . In this paper , for the classes of periodic zero-sum bilinear games and periodic zero-sum polymatrix games with time-invariant equilibrium , we investigate the day-today and time-average behaviors of continuous-time gradient descent-ascent ( GDA ) and follow-theregularized-leader ( FTRL ) learning dynamics , respectively . This study highlights the careful attention that must be given to the dynamical systems in periodic zero-sum games which preclude standard proof techniques for Poincaré recurrence , while also revealing that intuition from existing results on static zero-sum games can be totally invalidated even by simple restricted examples in periodic zero-sum games . Contribution 1 : Poincaré Recurrence . A key technical challenge in this work is that the dynamical systems which emerge from learning dynamics in periodic zero-sum games correspond to nonautonomous ordinary differential equations , whereas learning dynamics in static zero-sum games correspond to autonomous ordinary differential equations . Consequently , the usual proof methods from static zero-sum games for showing Poincaré recurrence are insufficient on their own in periodic zero-sum games . We overcome this challenge by delicately piecing together properties of periodic systems to construct a discrete-time autonomous system that we are able to show is Poincaré recurrent . This approach allows to prove both the GDA and FTRL learning dynamics are Poincaré recurrent in the respective classes of periodic zero-sum games ( Theorems 1 & 2 ) . Finally , we show both periodicity and a time-invariant equilibrium are necessary for such results in evolving games ( Proposition 1 ) . Contribution 2 : Time-Average Strategy Equilibration Fails . Given that Poincaré recurrence provably generalizes from static zero-sum games to periodic zero-sum games , it may be expected that the time-average strategies in periodic zero-sum games converge to the time-invariant equilibrium as in static zero-sum games . Surprisingly , we show that counterexamples can be constructed to this intuition even in the simplest of periodic zero-sum games . In particular , we prove the negative result that the time-average GDA and FTRL strategies do not necessarily converge to the time-invariant equilibrium in the respective classes of zero-sum games ( Propositions 2 & 3 ) . Contribution 3 : Time-Average Equilibrium Utility Convergence . Despite the negative result on the time-average strategy convergence , in the special case of periodic zero-sum bimatrix games we are able to show a complimentary positive result on the time-average utility convergence . Specifically , we show that the time-average utilities of the FTRL learning dynamics converge to the average of the equilibrium utility values of all the zero-sum games included in a single period of our time-evolving games . ( Theorem 3 ) . Organization . In Section 2 , we formalize the classes of games that we study . We present characteristics of dynamical systems as they pertain to this work in Section 3 . Section 4 and 5 contain our results analyzing GDA and FTRL learning dynamics in continuous and finite strategy periodic zero-sum bilinear and polymatrix games , respectively . We present numerical experiments in Section 6 and finish with a discussion in Section 7 . Proofs of are theoretical results are deferred to the appendix . 2 Game-Theoretic Preliminaries . 2.1 Continuous Strategy Periodic Zero-Sum Games . For continuous strategy periodic zero-sum games , we study periodic zero-sum bilinear games . We begin by formalizing zero-sum bilinear games and then define the periodic variant . Zero-Sum Bilinear Games . Given a matrix A ∈ Rn1×n2 , a zero-sum bilinear game on continuous strategy spaces can be defined by the max-min problem maxx1∈Rn1 minx2∈Rn2 x > 1 Ax2 . Formally , the game is defined by the pair of payoff matrices { A , −A > } and the action space of agents 1 and 2 are given by Rn1 and Rn2 , respectively . Player 1 seeks to maximize the utility function u1 ( x1 , x2 ) = x > 1 Ax2 while player 2 optimizes the utility u2 ( x1 , x2 ) = −x > 2 A > x1 . The game is zero-sum since for any x1 ∈ Rn1 and x2 ∈ Rn2 , the sum of utility over each player is zero . For zero-sum bilinear games , a Nash equilibrium corresponds to a joint strategy ( x∗1 , x ∗ 2 ) such that for each player i and j 6= i , ui ( x∗i , x∗j ) ≥ ui ( xi , x∗j ) , ∀xi ∈ Rni . Note that ( x∗1 , x∗2 ) = ( 0,0 ) is always a Nash equilibrium of a zero-sum bilinear game . Periodic Zero-Sum Bilinear Games . We study the continuous-time GDA learning dynamics in a class of games we refer to as periodic zero-sum bilinear games . The key distinction from a typical static zero-sum bilinear game is that the payoff matrix is no longer fixed in this class of games . Instead , the payoff matrix may change at each time instant as long as game remains zero-sum and the continuous-time sequence of payoffs is periodic . The next definition formalizes this class of games . Definition 1 ( Periodic Zero-Sum Bilinear Game ) . A periodic zero-sum bilinear game is an infinite sequence of zero-sum bilinear games { A ( t ) , −A ( t ) > } ∞t=0 in which the player set and strategy spaces are fixed and the payoff matrix is such that A ( t ) = A ( t+ T ) for a finite period T and all t ≥ 0 . Note that in such a game , ( 0 , 0 ) is always a time-invariant Nash equilibrium . Furthermore , we assume that the dependence of the payoff entries on time is smooth everywhere except for a finite set of points . 2.2 Finite Strategy Periodic Zero-Sum Games . For finite strategy periodic zero-sum games , we analyze periodic zero-sum polymatrix games . In what follows we define a zero-sum polymatrix game , which is a network generalization of a bimatrix game , and then detail the periodic variant considered in this paper . Zero-Sum Polymatrix Games . An N -player polymatrix game is defined by an undirected graph G = ( V , E ) where V is the player set and E is the edge set where a bimatrix game is played between the endpoints of each edge [ 8 ] . Each player i ∈ V has a set of actions Ai = { 1 , . . . , ni } that can be selected at random from a distribution xi called a mixed strategy . The mixed strategy set of player i ∈ V is the simplex in Rni denoted by Xi = ∆ni−1 = { xi ∈ Rni≥0 : ∑ α∈Ai xiα = 1 } where xiα denotes the probability of action α ∈ Ai . The joint strategy space is denoted by by X = Πi∈V Xi . The bimatrix game on edge ( i , j ) is described using a pair of matrices Aij ∈ Rni×nj and Aji ∈ Rnj×ni . The utility or payoff of agent i ∈ V under the strategy profile x ∈ X is given by ui ( x ) = ∑ j : ( i , j ) ∈E x > i A ijxj and corresponds to the sum of payoffs from the bimatrix games the player participates in . We further denote by uiα ( x ) = ∑ j : ( i , j ) ∈E ( A ijxj ) α the utility of player i ∈ V under the strategy profile x = ( α , x−i ) ∈ X for α ∈ Ai . The game is called zero-sum if ∑ i∈V ui ( x ) = 0 for all x ∈ X . Each bimatrix edge game is not necessarily zero-sum in a zero-sum polymatrix game . A Nash equilibrium in a polymatrix game is a mixed strategy profile x∗ ∈ X such that for each player i ∈ V , ui ( x∗i , x∗−i ) ≥ ui ( xi , x∗−i ) , ∀xi ∈ Xi . A Nash equilibrium is said to be an interior if supp ( x∗i ) = Ai ∀i ∈ V where supp ( x∗i ) = { α ∈ Ai : xiα > 0 } is the support of x∗i ∈ Xi . Periodic Zero-Sum Polymatrix Games . We analyze the continuous-time FTRL learning dynamics in a class of games we call periodic zero-sum polymatrix games . This class of games is such that the payoffs defined by the edge games evolve periodically . We consider that this periodic evolution is such that there is a common interior Nash equilibrium that arises in each zero-sum polymatrix game that arrives . The following definition formalizes the games we study on finite strategy spaces . Definition 2 ( Periodic Zero-Sum Polymatrix Game ) . A periodic zero-sum polymatrix game is an infinite sequence of zero-sum polymatrix games { G ( t ) = ( V ( t ) , E ( t ) ) } ∞t=0 in which the set of players , strategy spaces , and edges are fixed and each bimatrix game on an edge ( i , j ) is such that Aij ( t ) = Aij ( t+ T ) and Aji ( t ) = Aji ( t+ T ) for some finite period T and all t ≥ 0 . We assume there is a common interior Nash equilibrium x∗ ∈ X of the polymatrix game G ( t ) for all t ≥ 0 . Furthermore , we assume that the dependence of the payoff entries on time is smooth everywhere except for a finite set of points .

Summary: In this paper, the authors aim to analyze the asymptotic behavior of online learning algorithms in periodic rather than static zero-sum games. In terms of positive results, they show that the time-evolving *periodic* versions of both zero-sum bilinear games (under Gradient Descent Ascent) and zero-sum polymatrix games with time-invariant equilibria (under FTRL) satisfy the notion of Poincare recurrence -- a formal guarantee of cyclic behavior. However, the authors provide negative results too: in the described cases, the time-average of the GDA/FTRL plays does not converge to a time-invariant Nash equilibrium (in the latter case there is utility convergence).