Online Learning in Periodic Zero-Sum Games

1 Introduction . The study of learning dynamics in zero-sum games is arguably as old of a field as game theory itself , dating back to the seminal work of Brown and Robinson [ 7 , 27 ] , which followed shortly after the foundational minmax theorem of von Neumann [ 33 ] . The dynamics of online no-regret learning algorithms [ 10 , 29 ] are of particular interest in zero-sum games as they are designed with an adversarial environment in mind . Moreover , well known results imply that such dynamics converge in a time-average sense to a minmax equilibrium in zero-sum games [ 10 , 15 ] . Despite the classical nature of the study of online no-regret learning dynamics in zero-sum games , the actual transient behavior of such dynamics was historically not as understood . However , in the past several years this topic has gained attention with a number of works studying such dynamics in zerosum games ( and variants thereof ) with a particular focus on continuous-time analysis [ 24 , 25 , 20 , 6 , 32 , 23 , 22 ] . The unifying emergent picture is that the dynamics are “ approximately cyclic '' in a formal sense known as Poincaré recurrence . Moreover , these results have acted as fundamental building blocks for understanding the limiting behavior of their discrete-time variants [ 2 , 12 , 21 , 11 , 3 ] . ∗Joint first authors †Joint last authors 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite the plethora of emerging results regarding online learning dynamics in zero-sum games , an important and well motivated aspect of this problem has only begun to receive attention . How do online learning dynamics behave if the zero-sum game evolves over time ? Clearly , the answer to this question depends critically on how the game is allowed to evolve . Problem Setting and Model . We study periodic zero-sum games with a time-invariant equilibrium , which is a class of games we formally define in Section 2 . In a periodic zero-sum game , the payoffs that dictate the game are both T -periodic and zero-sum at all times . We consider both periodic zero-sum bilinear games on infinite , unconstrained strategy spaces and periodic zero-sum matrix games ( along with network generalizations thereof ) on finite strategy spaces . The goal of this work is to evaluate the robustness of the archetypal online learning behaviors in zero-sum games , Poincaré recurrence and time-average equilibrium convergence , to this natural model of game evolution . Connections to Repeated Game Models . The time-evolving game model we study can be seen as a generalization of usual repeated game formulations . A time-invariant game is a trivial version of a periodic game , in which case we recover the repeated static game setting . For a general periodic zero-sum game with period T , each stage game now is chosen according to a fixed length T sequence of games , capturing interactions between the players with time-dependent payoffs . Periodic zero-sum games can also fit into the frameworks of multi-agent contextual games [ 28 ] and dynamic games ( see , e.g. , [ 5 ] ) . In a multi-agent contextual game [ 28 ] , the environment selects a context from a set before each round of play and this choice defines the game that is played . Periodic zero-sum games can be seen as a multi-agent contextual game where the environment draws contexts from the available set in a T -periodic fashion with each context defining a zero-sum game with a common equilibrium . In the class of dynamic games , there is a game state on which the payoffs may depend that evolves with dynamics . Periodic zero-sum games can be interpreted as a dynamic game where the state transitions do not depend on the strategies of the players , the state is T -periodic , and the payoffs are completely defined by the state . We remark that the focus of existing work on contextual games and dynamic games is distinct from the questions investigated in this paper . The periodic zero-sum game model allows us to capture competitive settings where exogenous environmental variations manifest in an effectively periodic/epochal fashion . This naturally occurs in market competitions where time-of-day effects , week-to-week trends , and seasonality can dictate the game between players . To illustrate this point , consider a competition between service providers that wish to maximize their users , while the total market size evolves seasonally over time . This evolution affects the utility functions , even if the fundamentals of the market , and consequently the equilibrium , remain invariant . Contributions and Approach . In this paper , for the classes of periodic zero-sum bilinear games and periodic zero-sum polymatrix games with time-invariant equilibrium , we investigate the day-today and time-average behaviors of continuous-time gradient descent-ascent ( GDA ) and follow-theregularized-leader ( FTRL ) learning dynamics , respectively . This study highlights the careful attention that must be given to the dynamical systems in periodic zero-sum games which preclude standard proof techniques for Poincaré recurrence , while also revealing that intuition from existing results on static zero-sum games can be totally invalidated even by simple restricted examples in periodic zero-sum games . Contribution 1 : Poincaré Recurrence . A key technical challenge in this work is that the dynamical systems which emerge from learning dynamics in periodic zero-sum games correspond to nonautonomous ordinary differential equations , whereas learning dynamics in static zero-sum games correspond to autonomous ordinary differential equations . Consequently , the usual proof methods from static zero-sum games for showing Poincaré recurrence are insufficient on their own in periodic zero-sum games . We overcome this challenge by delicately piecing together properties of periodic systems to construct a discrete-time autonomous system that we are able to show is Poincaré recurrent . This approach allows to prove both the GDA and FTRL learning dynamics are Poincaré recurrent in the respective classes of periodic zero-sum games ( Theorems 1 & 2 ) . Finally , we show both periodicity and a time-invariant equilibrium are necessary for such results in evolving games ( Proposition 1 ) . Contribution 2 : Time-Average Strategy Equilibration Fails . Given that Poincaré recurrence provably generalizes from static zero-sum games to periodic zero-sum games , it may be expected that the time-average strategies in periodic zero-sum games converge to the time-invariant equilibrium as in static zero-sum games . Surprisingly , we show that counterexamples can be constructed to this intuition even in the simplest of periodic zero-sum games . In particular , we prove the negative result that the time-average GDA and FTRL strategies do not necessarily converge to the time-invariant equilibrium in the respective classes of zero-sum games ( Propositions 2 & 3 ) . Contribution 3 : Time-Average Equilibrium Utility Convergence . Despite the negative result on the time-average strategy convergence , in the special case of periodic zero-sum bimatrix games we are able to show a complimentary positive result on the time-average utility convergence . Specifically , we show that the time-average utilities of the FTRL learning dynamics converge to the average of the equilibrium utility values of all the zero-sum games included in a single period of our time-evolving games . ( Theorem 3 ) . Organization . In Section 2 , we formalize the classes of games that we study . We present characteristics of dynamical systems as they pertain to this work in Section 3 . Section 4 and 5 contain our results analyzing GDA and FTRL learning dynamics in continuous and finite strategy periodic zero-sum bilinear and polymatrix games , respectively . We present numerical experiments in Section 6 and finish with a discussion in Section 7 . Proofs of are theoretical results are deferred to the appendix . 2 Game-Theoretic Preliminaries . 2.1 Continuous Strategy Periodic Zero-Sum Games . For continuous strategy periodic zero-sum games , we study periodic zero-sum bilinear games . We begin by formalizing zero-sum bilinear games and then define the periodic variant . Zero-Sum Bilinear Games . Given a matrix A ∈ Rn1×n2 , a zero-sum bilinear game on continuous strategy spaces can be defined by the max-min problem maxx1∈Rn1 minx2∈Rn2 x > 1 Ax2 . Formally , the game is defined by the pair of payoff matrices { A , −A > } and the action space of agents 1 and 2 are given by Rn1 and Rn2 , respectively . Player 1 seeks to maximize the utility function u1 ( x1 , x2 ) = x > 1 Ax2 while player 2 optimizes the utility u2 ( x1 , x2 ) = −x > 2 A > x1 . The game is zero-sum since for any x1 ∈ Rn1 and x2 ∈ Rn2 , the sum of utility over each player is zero . For zero-sum bilinear games , a Nash equilibrium corresponds to a joint strategy ( x∗1 , x ∗ 2 ) such that for each player i and j 6= i , ui ( x∗i , x∗j ) ≥ ui ( xi , x∗j ) , ∀xi ∈ Rni . Note that ( x∗1 , x∗2 ) = ( 0,0 ) is always a Nash equilibrium of a zero-sum bilinear game . Periodic Zero-Sum Bilinear Games . We study the continuous-time GDA learning dynamics in a class of games we refer to as periodic zero-sum bilinear games . The key distinction from a typical static zero-sum bilinear game is that the payoff matrix is no longer fixed in this class of games . Instead , the payoff matrix may change at each time instant as long as game remains zero-sum and the continuous-time sequence of payoffs is periodic . The next definition formalizes this class of games . Definition 1 ( Periodic Zero-Sum Bilinear Game ) . A periodic zero-sum bilinear game is an infinite sequence of zero-sum bilinear games { A ( t ) , −A ( t ) > } ∞t=0 in which the player set and strategy spaces are fixed and the payoff matrix is such that A ( t ) = A ( t+ T ) for a finite period T and all t ≥ 0 . Note that in such a game , ( 0 , 0 ) is always a time-invariant Nash equilibrium . Furthermore , we assume that the dependence of the payoff entries on time is smooth everywhere except for a finite set of points . 2.2 Finite Strategy Periodic Zero-Sum Games . For finite strategy periodic zero-sum games , we analyze periodic zero-sum polymatrix games . In what follows we define a zero-sum polymatrix game , which is a network generalization of a bimatrix game , and then detail the periodic variant considered in this paper . Zero-Sum Polymatrix Games . An N -player polymatrix game is defined by an undirected graph G = ( V , E ) where V is the player set and E is the edge set where a bimatrix game is played between the endpoints of each edge [ 8 ] . Each player i ∈ V has a set of actions Ai = { 1 , . . . , ni } that can be selected at random from a distribution xi called a mixed strategy . The mixed strategy set of player i ∈ V is the simplex in Rni denoted by Xi = ∆ni−1 = { xi ∈ Rni≥0 : ∑ α∈Ai xiα = 1 } where xiα denotes the probability of action α ∈ Ai . The joint strategy space is denoted by by X = Πi∈V Xi . The bimatrix game on edge ( i , j ) is described using a pair of matrices Aij ∈ Rni×nj and Aji ∈ Rnj×ni . The utility or payoff of agent i ∈ V under the strategy profile x ∈ X is given by ui ( x ) = ∑ j : ( i , j ) ∈E x > i A ijxj and corresponds to the sum of payoffs from the bimatrix games the player participates in . We further denote by uiα ( x ) = ∑ j : ( i , j ) ∈E ( A ijxj ) α the utility of player i ∈ V under the strategy profile x = ( α , x−i ) ∈ X for α ∈ Ai . The game is called zero-sum if ∑ i∈V ui ( x ) = 0 for all x ∈ X . Each bimatrix edge game is not necessarily zero-sum in a zero-sum polymatrix game . A Nash equilibrium in a polymatrix game is a mixed strategy profile x∗ ∈ X such that for each player i ∈ V , ui ( x∗i , x∗−i ) ≥ ui ( xi , x∗−i ) , ∀xi ∈ Xi . A Nash equilibrium is said to be an interior if supp ( x∗i ) = Ai ∀i ∈ V where supp ( x∗i ) = { α ∈ Ai : xiα > 0 } is the support of x∗i ∈ Xi . Periodic Zero-Sum Polymatrix Games . We analyze the continuous-time FTRL learning dynamics in a class of games we call periodic zero-sum polymatrix games . This class of games is such that the payoffs defined by the edge games evolve periodically . We consider that this periodic evolution is such that there is a common interior Nash equilibrium that arises in each zero-sum polymatrix game that arrives . The following definition formalizes the games we study on finite strategy spaces . Definition 2 ( Periodic Zero-Sum Polymatrix Game ) . A periodic zero-sum polymatrix game is an infinite sequence of zero-sum polymatrix games { G ( t ) = ( V ( t ) , E ( t ) ) } ∞t=0 in which the set of players , strategy spaces , and edges are fixed and each bimatrix game on an edge ( i , j ) is such that Aij ( t ) = Aij ( t+ T ) and Aji ( t ) = Aji ( t+ T ) for some finite period T and all t ≥ 0 . We assume there is a common interior Nash equilibrium x∗ ∈ X of the polymatrix game G ( t ) for all t ≥ 0 . Furthermore , we assume that the dependence of the payoff entries on time is smooth everywhere except for a finite set of points .

Summary: The paper aims to analyze online learning behaviors of the gradient descent-ascent(GDA) and the follow the regularized leader(FTRL) algorithms for periodic zero-sum games. Seminal works [23, 19, 7] show that online no-regret learning dynamics is Poincar\'e recurrent in repeated static zero-sum games, but the techniques used in the seminal works does not work well in periodic one. To resolve this, the paper exploits an important theorem about volume preservation for periodic systems by [1], and shows that GDA dynamics in periodic zero-sum bilinear games and FTRL in periodic strategy polymatrix zero-sum games are Poincar\'e recurrent. The paper also shows a negative result that time-average strategy does not converge to time-invariant equilibrium, as a counter example. The empirical evaluation support the theoretical results. (Reference number follows the paper.)