Title: Higher-Order Uncoupled Dynamics Do Not Lead to Nash Equilibrium - Except When They Do
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning in games

Nash equilibrium

Uncoupled Dynamics
Abstract: The framework of multi-agent learning explores the dynamics of how an agent's strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well known solution concepts such as Nash Equilibrium (NE). In "higher order'' learning, agent dynamics include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are "payoff based'' and "uncoupled'' in that each agent's dynamics depend on its own evolving payoff and has no explicit dependence on the utilities of other agents. We first show that for any specific game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. Finally, we show that convergence to the mixed-strategy equilibrium in coordination games, comes at the expense of the dynamics being inherently internally unstable.

Higher-Order Uncoupled Dynamics Do Not Lead to Nash Equilibrium - Except When They Do

Abstract

The framework of multi-agent learning explores the dynamics of how an agent's strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well-known solution concepts such as Nash Equilibrium (NE). In 'higher-order' learning, agent dynamics include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are 'payoff-based' and 'uncoupled' in that each agent's dynamics depend on its own evolving payoff and has no explicit dependence on the utilities of other agents. We first show that for any polymatrix game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. Finally, we show that convergence to the mixed-strategy equilibrium in coordination games can come at the expense of the dynamics being inherently internally unstable.

1 Introduction

The field of learning in games explores how game-theoretic solution concepts emerge as the outcome of dynamic processes where agents adapt their strategies in response to the evolving strategies of other agents (1; 2; 3; 4). There is a multitude of specific cases of learning dynamics/game combinations that result in a range of outcomes, including convergence, limit cycles, chaotic behavior, and stochastic stability (5; 6; 7; 8; 9; 10; 11; 12; 13). The emphasis in the literature is on simple adaptive procedures, called "natural" dynamics in (14), that can result in various solution concepts (e.g., Nash equilibrium, correlated equilibrium, and coarse correlated equilibrium). (A separate concern is the complexity associated with such computations (e.g. (15; 16)).)

One 'natural' restriction for learning is that the dynamics of one agent should not depend explicitly on the utility functions of other agents. This restriction was referred to as 'uncoupled' dynamics in (17), where the authors constructed a specific anti-coordination matrix game for which no uncoupled learning dynamics could converge to the unique mixed-strategy NE. The dynamics considered in that setting were of fixed order, i.e., the order of the learning dynamics was restricted to match the dimension of the strategy space. More recent work showed that specific instances of fixed order uncoupled learning dynamics can never lead to mixed-strategy NE (18). Furthermore, there exist games for which any fixed order learning dynamics are bound to have an initial condition starting from which the dynamics do not converge to NE (19).

The restriction on the order of the learning dynamics in learning mixed-strategy NE turns out to be essential. In particular, by introducing additional auxiliary states, (20) showed that higher-order learning could overcome the obstacle of convergence to NE in the same anti-coordination game considered in (17) while remaining uncoupled.

Higher-order learning in games can be seen as a parallel to higher-order optimization algorithms, such as momentum-based or optimistic gradient algorithms (e.g., (21; 22)). Such algorithms utilize history to update the underlying search parameter. In this way, there is a path dependency on the trajectory of information. An early utilization of higher-order learning is in (23), in which a player's strategy update uses two stages of history of an opponent's strategies in a zero-sum setting to eliminate oscillations. Similar ideas were used in (24). Reference (25) modified gradient-based algorithms through the introduction of a cumulative (integral) term. In (20), higher-order dynamics were used to create a myopic forecast of the action of other agents. Authors in (26) introduce a version of higher-order replicator dynamics and show that, unlike fixed order replicator dynamics, weakly dominated strategies become extinct. Reference (27) utilizes the system theoretic notion of passivity to analyze a family of higher-order dynamics.

In this paper, we further explore the implications of learning dynamics that are uncoupled. We address 'payoff based' dynamics, in which the learning dynamics depend on the evolution of a payoff vector that is viewed as an externality. When players are engaged in a game, then the payoff stream of one agent depends on the actions of other agents. However, the learning dynamics themselves do not change based on the source of the payoff streams.

First, we show that for any polymatrix game with a mixed-strategy NE, there exist payoff-based dynamics that converge locally to that NE. This result is established by making a connection between convergence to NE and the existence of decentralized stabilizing control (28; 29). A consequence of the payoff-based structure is that the dynamics also converge to the NE of nearby perturbations of the original game. The form of higher-order learning used for this stability result is higher-order gradient play, which generalizes gradient ascent. Next, we show that for any such dynamics, there exists a game with a unique mixed-strategy NE that is unstable under given dynamics. The tool utilized is a classical analysis method in feedback control systems known as root-locus (e.g., (30), (31)), which characterizes the locations of the eigenvalues of a matrix as a function of a scalar parameter. A combination of the above results suggests the lack of universality on the side of both learning dynamics and games. While any mixed-strategy NE can be stabilized by suitable higher-order gradient dynamics, any such dynamics can be destabilized by a suitable anti-coordination game. Finally, we examine the implications of higher-order dynamics being able to converge to the mixed-strategy NE of a 2 × 2 coordination game, which has two pure NE and one mixed-strategy NE. We show that such higher-order gradient play dynamics must have an inherent internal instability, which makes them unsuitable, if not irrational, as a model of learning.

2 Payoff-based learning dynamics

2.1 Finite polymatrix games

Weconsider finite (normal form) games over mixed-strategies. There are n players. The strategy space of player i ∈ { 1 , 2 , ..., n } is the probability simplex, ∆( k i ) , where k i is a positive integer and ∆( · ) is defined as ∆( κ ) = { s ∈ R κ ∣ ∣ ∣ s j ≥ 0 , j = 1 , ..., κ, & ∑ κ j =1 s j = 1 } . The utility function of player i is a function u i : ∆( k 1 ) × ... × ∆( k n ) → R . We sometimes will write u i ( x 1 , ..., x n ) = u i ( x i , x -i ) , for x i ∈ ∆( k i ) and x -i ∈ X -i , where X -i = ∆( k 1 ) × ... × ∆( k i -1 ) × ∆( k i +1 ) × ... × ∆( k n ) .

For convenience, we will restrict our discussion to pairwise interactions, also called polymatrix games. That is, the utility function of player i is defined as

̸

̸

for matrices, M ij , j = 1 , . . . , n , j = i . We can write the utility function of player i as the inner product u i ( x 1 , ..., x n ) = x T i P i ( x -i ) where P i ( x -i ) = ∑ n j =1 j = i M ij x j ∈ R k i . Accordingly, each element of P i ( x -i ) can be viewed as a payoff that is associated with a component of player i 's strategy vector, x i .

̸

A Nash equilibrium (NE) is a tuple ( x ∗ 1 , ..., x ∗ n ) ∈ X such that for all i = 1 , ..., n ,

A completely mixed-strategy NE is such that each x ∗ i is in the interior of the simplex.

2.2 Fixed order learning

Our model of learning is a dynamical system that relates trajectories of a payoff vector, p i ( t ) , to trajectories of the strategy, x i ( t ) . In particular, learning dynamics for player i are specified by a function f i : ∆( k i ) × R k i → R k i according to

where x i ( t ) ∈ ∆( k i ) and p i : R + → R k i . We assume implicitly that f i and p i are such that there exists a unique solution whenever x i (0) ∈ ∆( k i ) . We further assume that the dynamics satisfy the invariance property that

Note that we define learning dynamics without specifying the source of the payoff vector, p i ( t ) , hence the terminology 'payoff-based'. Only once a player is coupled with other players in a game through their own (possibly heterogeneous) learning dynamics is when we make the connection p i ( t ) = P i ( x -i ( t )) .

This formulation is illustrated in Figure 1, where the LD i denote payoff-based learning dynamics that are interconnected through the game matrices, M ij . Note that such learning dynamics are uncoupled by construction since each player can only access its own payoff vector. There is no dependence on the payoff stream of other players. Indeed, there is no dependence on the parameters of one's own utility function.

2.3 Higher-order learning

The learning dynamics described in the previous section have a fixed order associated with the dimension of the strategy space. Higher-order learning dynamics allow for the introduction of auxiliary states as follows. For any fixed order learning dynamics, f i , we can define a higher-order version as

As before, x i ( t ) ∈ ∆( k i ) and p i : R + → R k i . The new variable z i ∈ R ℓ i represents ℓ i dimensional auxiliary states that evolve according to the p i -dependent dynamics, g i . These enter into the original fixed order dynamics through ϕ i . Accordingly, we can view p i ( t ) + ϕ i ( p i ( t ) , z i ( t )) as a modified payoff stream that captures path dependencies in p i , and the original learning dynamics react to this modified payoff stream.

As with the fixed order counterparts, there is no specification of game parameters in higher-order learning dynamics. In order to enforce that the auxiliary states have no effect on the equilibria of games, we make the following assumption.

Assumption 2.1. If p ∗ i and z ∗ i are an equilibrium of the higher-order dynamics, i.e.,

This assumption assures that the auxiliary states represent purely transient phenomena that disappear at equilibrium.

Example 1 ( Anticipatory higher-order dynamics ) . A special case of higher-order dynamics is

where the the higher-order modification is the linear system (see Appendix A.1)

and γ, λ ∈ R + . Intuition behind the connection to anticipation can be seen by viewing λ ( p i -z i ) as an approximation of ˙ p i , and so p i ( t ) + γλ ( p i ( t ) -z i ( t )) ≈ p i ( t + γ ) . Similar higher-order dynamics were used in reference (20) for smooth fictitious play to overcome the lack of convergence of uncoupled dynamics to NE in the anti-coordination game analyzed by (17). See also reference (32) for an analysis with replicator dynamics.

Anticipatory higher-order dynamics can also be linked to optimistic optimization algorithms (e.g., (22)). An Euler discretization of step size, h , results in

Setting h = 1 /λ and γ = 1 /λ results in

where the superscripts ' + ' and ' -' indicate the next and previous discrete time steps, respectively. There are also optimistic variants of discrete-time no-regret learning algorithms (33; 34; 35) that guarantee faster convergence rates to coarse correlated equilibria compared to the standard versions.

3 Gradient play

The main results of the paper will examine the behavior of gradient play and higher-order gradient play, which are the focus of this section.

3.1 Fixed order gradient play

In gradient play dynamics, a player adjusts its strategy in the direction of the payoff stream, i.e.,

where Π ∆ [ x ] : R n → ∆( n ) is the projection of x into the simplex, i.e., Π ∆ ( x ) = arg min s ∈ ∆( n ) ∥ x -s ∥ .

̸

The terminology 'gradient play' stems from the gradient of an agent's utility function in (1) with respect to its own strategy, x i , namely ∇ x i u i ( x i , x -i ) = P i ( x -i ) = ∑ n j =1 j = i M ij x j . As was done in the description of payoff-based learning, we replace P i ( x -i ) with the payoff stream p i without regard to the game matrices M ij .

Our primary concern will be studying these dynamics near a completely mixed-strategy NE. To this end, let x ∗ = ( x ∗ 1 , . . . , x ∗ n ) be an isolated completely mixed-strategy NE. The strategy vector x i then

with a modified payoff stream of

where where

Given the zero trace of M , standard gradient play is always unstable at a completely mixed-strategy NE (see Appendix A.3 for stability conditions). Also, for this equilibrium to be isolated, M must be non-singular. Otherwise, M has a non-trivial null space leading to an equilibrium subspace.

3.2 Higher-order gradient play

We will be interested in a specific form of higher-order gradient play that uses the following linear structure of higher-order dynamics:

for some matrices E i , F i , G i and H i . Here, the auxiliary states are z i = ( v i , ξ i ) , which enter into the dynamics through ϕ i ( p i , ξ i , v i ) = N i ( G i ξ i + H i ( N T i p i -v i )) , where N i is defined as in (6).

The motivation behind this structure, illustrated in Figure 2, assures the enforcement of Assumption 2.1. The payoff stream is first preprocessed by a specific linear system to produce y i and then by a general linear system K i to produce u i . The matrices ( E i , F i , G i , H i ) create the dynamical system

that maps y i = N T i p i -v i to u i = G i ξ i + H i y i via ˙ ξ i = E i ξ i + F i y i . The preprocessing system has the property (see 'washout filters' in Appendix A.2) that if p i ( t ) converges to a constant, then y i ( t ) converges to zero. Accordingly, when E i is non-singular, the preprocessing system guarantees Nash stationarity, i.e., if

then x ∗ = ( x ∗ 1 , . . . , x ∗ n ) is a NE. To conclude, the linear system K i is the central entity that performs the payoff modification, and the preprocessing system ensures compliance with Assumption 2.1.

evolves on the simplex, which is a subset of dimension k i -1 . Hence, the local behavior of the dynamics around x ∗ i is characterized by evolution on a lower-dimensional subset. Thus, we can write

Therefore, w i ∈ R ( k i -1) represents deviations from x ∗ i and satisfies w i ( t ) = N T i ( x i ( t ) -x ∗ i ) . When all players utilize fixed order gradient play, the collective dynamics near a completely mixed-strategy NE take the form

3.3 Local stability analysis

As before, we can analyze the behavior near a completely mixed-strategy NE x ∗ through the variable w i defined as in (5). Using the fact that N T i ∑ j = i M ij x ∗ j = 0 , we can write the local dynamics of a player as

̸

̸

The collective dynamics near a mixed-strategy NE can be written as

where E , F , G , and H are block diagonal matrcies with appropriate dimensions and M is defined in (8). Local stability of a completely mixed NE is determined by whether the above collective dynamics are stable, i.e., the dynamics matrix in (10) is a stability matrix.

4 Uncoupled dynamics that lead to mixed-strategy NE

4.1 Decentralized control formulation

The stability of a mixed-strategy equilibrium is tied to the existence of K 1 , K 2 , ..., K n so that the linear system in (10) is stable. When the K i have yet to be determined, we can rewrite (10) as

where u =    u 1 . . . u n    , y =    y 1 . . . y n    , and the y i and u i are to be related through K i .

4.2 Decentralized stabilization

Let with

̸

̸

We first establish that P can be stabilized by verifying the conditions for stabilizability and detectability (see Appendix A.3). The assumption that M is non-singular stems from our interest in isolated NE.

Proposition 4.1. For M non-singular, the pair ( A , B ) is stabilizable, and the pair ( A , C ) is detectable.

While Proposition 4.1 establishes that P can be stabilized, that property alone is inadequate for our purposes. In particular, for the learning dynamics to be uncoupled, we seek to establish decentralized stabilization (see Appendix A.4) according to the partition

̸

where

Here, M i · denotes the i th block row of M , i.e.,

and

where I has a dimension (suppressed in the notation) of k i -1 .

Theorem 4.1. For any isolated (i.e., M is non-singular) completely mixed-strategy NE, there exist uncoupled higher-order gradient play dynamics such that (10) is stable.

The proof of Theorem 4.1 relies on the conditions of Theorem A.1 and is presented in Appendix B.1.

Theorem 4.1 should be viewed as a statement regarding whether uncoupled learning in itself is a barrier to learning dynamics leading to NE. The theorem makes no claim that the higher-order learning dynamics are interpretable (e.g., as in anticipatory learning). Nor does the theorem offer guidance on how agents may construct the matrices of higher-order learning that lead to convergence. In the next section, we will see that, while the structure is universal, any specific set of parameters is not universal in that one can construct a game for which they do not lead to NE. Despite the lack of universality, there is an inherent robustness that is a consequence of stability. The following follows from standard arguments on linear systems.

Proposition 4.2. Let the K i and M ij , i = 1 , ..., n and j = 1 , ..., n , be such that (10) is stable. Then there exists a δ > 0 such that (10) is stable with the M ij replaced by any ˜ M ij as long as ∥ ∥ .

In words, this proposition guarantees that learning dynamics that lead to NE for a specific game continue to do so for nearby games.

4.3 Stabilization through a single higher-order player

The previous section's analysis allowed all players to utilize higher-order learning. In some cases, it may not be necessary for all players to utilize higher-order learning. In this section, we present sufficient conditions under which a single player using higher-order gradient play with the remainder utilizing fixed order gradient play can still lead to NE.

Assumption 4.1.

A. Let ( w,λ ) be a left eigenvalue pair of M , i.e.,

with Re [ λ ] ≥ 0 and w T = ( w T 1 ... w T n ) partitioned consistently with (8). Then w i = 0

B. Let ( v, λ ) be a right eigenvalue pair of M , i.e.,

with Re [ λ ] ≥ 0 and v = ( v T 1 ... v T n ) T partitioned consistently with (8). Then v i = 0 for all i .

Recall the definitions of A , B i , and C i from (14).

Proposition 4.3. Let M be non-singular and satisfy Assumption 4.1. Then for any i , the pair ( A , B i ) is stabilizable and the pair ( A , C i ) is detectable.

As a consequence of Proposition 4.3, it is possible for a completely mixed-strategy NE to be stabilized where a single player utilizes higher-order gradient play with the remaining players utilizing fixed order gradient play.

̸

̸

5 Non-convergence to NE in higher-order gradient play

We now show that linear higher-order gradient play dynamics need not lead to NE. Given any such dynamics, we construct a game with a unique NE that is unstable under given dynamics.

5.1 The Jordan anti-coordination game

The Jordan anti-coordination game, introduced in (36), was used in (17) to prove that fixed order uncoupled learning dynamics do not lead to NE. The game consists of three players with

and a unique mixed-strategy NE at x ∗ 1 = x ∗ 2 = x ∗ 3 = ( 1 2 1 2 ) T . We will let Γ( µ ) denote the Jordan anti-coordination game but with the utility function of player 1 modified to u 1 ( x 1 , x 2 ) = x T 1 ( µM 12 ) x 2 , where µ ∈ R + . Since scaling payoffs does not change the nature of the game, Γ( µ ) has the same unique NE as Γ(1) .

5.2 Destabilization using rescaled anti-coordination

Suppose all three players use variants of linear higher-order gradient play in Γ( µ ) . As before, we denote the higher-order dynamics of player i as the linear dynamical system (9). To study the local behavior of the dynamics around the unique mixed-strategy NE of Γ( µ ) , we define w 1 ( t ) , w 2 ( t ) , and w 3 ( t ) as in (5). Then, we analyze the local stability of the mixed-strategy NE through (10).

The following lemma will be essential in proving our main result.

Lemma 5.1. Let A ∈ R n × n , B ∈ R n × 1 and C ∈ R 1 × n . If CB = CAB = 0 , and CA m B = 0 for some m ≥ 2 . Then for sufficiently large µ > 0 , A -µBC is not a stability matrix.

̸

The proof of Lemma 5.1 is presented in Appendix B.2 and it uses root-locus arguments (see references (31; 30)).

Proposition 5.1. If linear higher-order gradient play dynamics are locally exponentially stable at the unique NE of Γ(1) , then there exists µ > 0 such that the unique NE of Γ( µ ) is unstable under such dynamics.

The proof of Proposition 5.1 is presented in Appendix B.3. The main idea of the proof is to write the local dynamics matrix in the form A -µBC , and then use Lemma 5.1. We also provide a similar proof for sufficiently small µ in Appendix C.

The results might be puzzling because, for all µ > 0 , all games Γ( µ ) are strategically equivalent. Convergence guarantees for learning dynamics are usually established amongst classes of games. Thus, it is generally expected that dynamics will behave similarly for all games in a particular class. In this case, we design linear learning dynamics that are affected by simple rescaling of the payoff matrices.

6 Strong stabilization of mixed-strategy NE

Results from Section 4 imply that the mixed-strategy NE in a two-player 2 × 2 (identical-interest) coordination game can be stabilized. Here, we argue why dynamics that stabilize this mixed-strategy equilibrium are not reasonable. Specifically, we show that such dynamics must be inherently unstable as an open system, i.e., as dynamics that respond to an exogenous payoff stream, and this instability is problematic with respect to such payoffs.

First, we inspect which type of mixed-strategy NE requires unstable learning dynamics for stabilization. For this purpose, consider the system in (12) for n = k 1 = k 2 = 2 :

x,（1)

x(2)

0.5

0

5

10

15

20

25

35

40

45

50

t

x(1)

x(2)

0.5

0

5

10

15

20

25

30

35

40

45

50

t

x(1)

x(2)

0.5

0

5

10

15

20

25

30

35

40

45

50

t

̸

Around an isolated mixed-strategy NE, the matrix A above should be non-singular. Accordingly, it must be that m 12 = 0 and m 21 = 0 . The ability to stabilize a system via another stable system is referred to as strong stabilization (see Appendix A.3). The next proposition gives a sufficient condition under which an isolated mixed-strategy NE is not strongly stabilizable.

̸

Proposition 6.1. If m 12 m 21 > 0 , then (15) is not strongly stabilizable.

The proof of Proposition 6.1 uses the 'parity interlacing principle' (see reference (37)) and is presented in Appendix B.4.

The nature of the game can be inferred from the scalars m 12 and m 21 . For example in zero-sum games we have M 12 = -M T 21 , which gives

In coordination games, we have M 12 = M 21 , which gives

Therefore, the mixed-strategy NE in a coordination game is not strongly stabilizable.

Now let us examine the implications of inherently unstable learning dynamics. A reasonable expectation of learning dynamics is that in the case of a constant payoff vector, i.e., p i ( t ) ≡ p ∗ , we expect

where β ( p ∗ ) is a best response to p ∗ , i.e.,

For any higher-order gradient play dynamics, if E i is a stability matrix, then whenever p i ( t ) ≡ p ∗ for some constant vector p ∗ , one can show ξ i ( t ) → 0 , which implies that x i ( t ) is generated by standard gradient play dynamics in the limit. However, if p i ( t ) ≡ p ∗ and the dynamics are inherently unstable, the term N i G i ξ i ( t ) need not vanish. Indeed, one can construct p ∗ such that x i ( t ) does not converge to the best response of p ∗ (see the example in Section 7.2). The inability of learning dynamics to converge to the best response of a constant payoff vector does not reflect 'natural' behavior.

7 Numerical experiments

7.1 Jordan anti-coordination game: Stabilization through a single player

The payoff matrices of the Jordan anti-coordination game, introduced in Section 5.1, satisfy Assumption 4.1. Therefore, we can stabilize its mixed-strategy NE, allowing only one player to use higher-order learning while others continue to use standard gradient play. For this purpose, let ξ 1 ∈ R and choose H 1 = γλ , G 1 = -γλ , F 1 = λ and E 1 = -λ , where λ = 50 and γ = 5 . Such dynamics resemble anticipatory gradient play but on the filtered low-dimensional payoff. Figure 3 illustrates convergence of the players' strategies to NE.

(a) Stabilization of the mixed- strategy NE.

(b) Inherently unstable dynamics

do not converge to best response

of

p

∗

=

(

0

1

)

T

.

x,（1)

X(2)

0.5

D

5

10

15

20

25

t

30

35

40

45

50

x（1)

x(2)

0.5

D

0

5

10

15

20

25

30

35

40

45

50

t

6°0

x,（1)

x,(2)

8'0

0.7

0.6

0.5

0.4

'0

0.2

0.1

0

0

1

2

3

5

6

7

8

9

10

7.2 Stabilization of mixed-strategy NE in coordination games

Consider the (identical interest) coordination game:

The game has two pure strategy NE and one completely mixed-strategy NE at x ∗ = ( 1 2 1 2 ) T . Consider the following set of parameters for higher-order gradient play: E 1 = λ , F 1 = -2 λ , G 1 = γλ , H 1 = -γλ , E 2 = -λ 2 , F 2 = λ 2 , G 2 = -γ 2 λ 2 , and H 2 = γ 2 λ 2 . The numerical values are λ = 0 . 5 , γ = 20 , λ 2 = 50 , and γ 2 = 1 . Figure 4a illustrates convergence to the mixed-strategy NE of this coordination game. Suppose we break the feedback loop and use p ∗ = (0 1) T as the input to the first player dynamics. The response to such input is illustrated in Figure 4b. We see that E 1 > 0 is a scalar, and so ξ 1 grows without bound. The strategy x 1 , which is projected to the simplex, converges to (1 0) T , which is not a best response to the input p ∗ .

8 Concluding remarks

To recap, we studied the role of higher-order gradient play with linear higher-order dynamics. We showed that for any game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead to that NE, both for the original game and for nearby games. On the other hand, we showed that for any higher-order gradient play dynamics, the dynamics do not lead to NE for a suitably rescaled anti-coordination game. We also provided an argument against dynamics that lead to the mixed-strategy NE in a coordination game, showing they are not reasonable.

Regarding the higher-order gradient play dynamics that lead to NE, the interpretation of the results herein should not be that these dynamics are either a descriptive model of learning or a prescriptive recommendation for computation. Rather, the results are a contribution towards delineating what is possible or impossible in multi-agent learning. In that sense, they may be seen as a complement to the contributions in (17). Namely, dynamics being uncoupled is not a barrier to converging to mixed-strategy NE when allowing higher-order learning.

More generally, the present results open new questions related to the discussion in (14) on what constitutes 'natural' learning dynamics. In the case of anticipatory higher-order learning, there is a clear interpretation of the effect of higher-order terms. However, it is unclear how to interpret general higher-order dynamics. Furthermore, the results herein regarding inherent instability of higher-order dynamics that converge to the mixed-strategy NE of a coordination game suggests that higher-order learning can be 'unnatural'. Possible restrictions on dynamics, in addition to being uncoupled, could include having no asymptotic regret; maintaining qualitative behavior in the face of strategically equivalent games (cf., Section 5.2); or having an interpretable relationship between payoff streams and strategic evolutions such as 'passivity', which generalizes and extends the notion of contractive games to contractive learning dynamics (e.g., (38; 39; 40)).

In terms of limitations, the current paper only addresses the payoff vector setup and does not address the setup where players only have access to instantaneous scalar payoffs. Furthermore, results in Sections 5 and 6 are limited to certain setups and require further generalization.

References

D. Fudenberg and D. K. Levine, The Theory of Learning in Games . Economic Learning and Social Evolution Series, MIT Press, 1998.

D. Fudenberg and D. K. Levine, 'Learning and equilibrium,' Annual Review of Economics , vol. 1, no. 1, pp. 385-420, 2009.

S. Hart, 'Adaptive heuristics,' Econometrica , vol. 73, no. 5, pp. 1401-1430, 2005.

H. Young, Strategic Learning and its Limits . Arne Ryde memorial lectures, Oxford University Press, 2004.

S. Hart and A. Mas-Colell, 'A simple adaptive procedure leading to correlated equilibrium,' Econometrica , vol. 68, no. 5, pp. 1127-1150, 2000.

U. Berger, 'Fictitious play in 2 × n games,' Journal of Economic Theory , vol. 120, no. 2, pp. 139-154, 2005.

D. Monderer and L. S. Shapley, 'Fictitious play property for games with identical interests,' Journal of Economic Theory , vol. 68, no. 1, pp. 258-265, 1996.

J. S. Shamma and G. Arslan, 'Unified convergence proofs of continuous-time fictitious play,' IEEE Transactions on Automatic Control , vol. 49, no. 7, pp. 1137-1141, 2004.

L. S. Shapley, 'Some topics in two-person games,' in Advances in Game Theory (L. Shapley, M. Dresher, and A. Tucker, eds.), pp. 1-29, Princeton, NJ: Princeton University Press, 1964.

D. P. Foster and H. Young, 'On the nonconvergence of fictitious play in coordination games,' Games and Economic Behavior , vol. 25, no. 1, pp. 79-96, 1998.

D. P. Foster and R. V. Vohra, 'Calibrated learning and correlated equilibrium,' Games and Economic Behavior , vol. 21, no. 1, pp. 40-55, 1997.

G. Piliouras and J. S. Shamma, 'Optimization despite chaos: Convex relaxations to complex limit sets via Poincaré recurrence,' in Proceedings of the 2014 Annual ACM-SIAM Symposium on Discrete Algorithms , pp. 861-873, 2014.

H. P. Young, Individual Strategy and Social Structure: An Evolutionary Theory of Institutions . Princeton University Press, 1998.

S. Hart and A. Mas-Colell, Simple Adaptive Strategies . WORLD SCIENTIFIC, 2013.

C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou, 'The complexity of computing a Nash equilibrium,' SIAM Journal on Computing , vol. 39, no. 1, pp. 195-259, 2009.

Y. Babichenko and A. Rubinstein, 'Communication complexity of approximate Nash equilibria,' Games and Economic Behavior , vol. 134, pp. 376-398, 2022.

S. Hart and A. Mas-Colell, 'Uncoupled dynamics do not lead to Nash equilibrium,' American Economic Review , vol. 93, pp. 1830-1836, December 2003.

E.-V. Vlatakis-Gkaragkounis, L. Flokas, T. Lianeas, P. Mertikopoulos, and G. Piliouras, 'Noregret learning and mixed Nash equilibria: They do not mix,' in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 1380-1391, Curran Associates, Inc., 2020.

J. Milionis, C. Papadimitriou, G. Piliouras, and K. Spendlove, 'An impossibility theorem in game dynamics,' Proceedings of the National Academy of Sciences , vol. 120, no. 41, 2023.

J. S. Shamma and G. Arslan, 'Dynamic fictitious play, dynamic gradient play, and distributed convergence to Nash equilibria,' IEEE Transactions on Automatic Control , vol. 50, pp. 312-327, March 2005.

M. Muehlebach and M. I. Jordan, 'Optimization with momentum: Dynamical, control-theoretic, and symplectic perspectives,' J. Mach. Learn. Res. , vol. 22, no. 1, 2021.

C. Daskalakis and I. Panageas, 'The limit points of (optimistic) gradient descent in min-max optimization,' in Advances in Neural Information Processing Systems (NeurIPS) , 2018.

T. Ba¸ sar, 'Relaxation techniques and asynchronous algorithms for on-line computation of noncooperative equilibria,' Journal of Economic Dynamics and Control , vol. 11, no. 4, pp. 531-549, 1987.

J. Conlisk, 'Adaptation in games: Two solutions to the Crawford puzzle,' Journal of Economic Behavior & Organization , vol. 22, no. 1, pp. 25-50, 1993.

S. Flam and J. Morgan, 'Newtonian mechanics and Nash play,' International Game Theory Review , vol. 06, 07 2003.

R. Laraki and P. Mertikopoulos, 'Higher order game dynamics,' Journal of Economic Theory , vol. 148, pp. 2666-2695, 06 2013.

B. Gao and L. Pavel, 'On passivity, reinforcement learning and higher order learning in multiagent finite games,' IEEE Transactions on Automatic Control , vol. 66, no. 1, pp. 121-136, 2021.

S.-H. Wang and E. Davison, 'On the stabilization of decentralized control systems,' IEEE Transactions on Automatic Control , vol. 18, no. 5, pp. 473-478, 1973.

E. Davison and T. Chang, 'Decentralized stabilization and pole assignment for general proper systems,' IEEE Transactions on Automatic Control , vol. 35, no. 6, pp. 652-664, 1990.

A. M. Krall, 'An extension and proof of the root-locus method,' Journal of the Society for Industrial and Applied Mathematics , vol. 9, no. 4, pp. 644-653, 1961.

A. M. Krall, 'The root locus method: A survey,' SIAM Review , vol. 12, no. 1, pp. 64-72, 1970.

G. Arslan and J. S. Shamma, 'Anticipatory learning in general evolutionary games,' in Proceedings of the 45th IEEE Conference on Decision and Control , pp. 6289-6294, 2006.

V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire, 'Fast convergence of regularized learning in games,' in Advances in Neural Information Processing Systems (C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, eds.), vol. 28, Curran Associates, Inc., 2015.

X. Chen and B. Peng, 'Hedging in games: Faster convergence of external and swap regrets,' in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 18990-18999, Curran Associates, Inc., 2020.

C. Daskalakis, M. Fishelson, and N. Golowich, 'Near-optimal no-regret learning in general games,' in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 27604-27616, Curran Associates, Inc., 2021.

J. Jordan, 'Three problems in learning mixed-strategy Nash equilibria,' Games and Economic Behavior , vol. 5, no. 3, pp. 368-386, 1993.

D. Youla, J. Bongiorno, and C. Lu, 'Single-loop feedback-stabilization of linear multivariable dynamical plants,' Automatica , vol. 10, no. 2, pp. 159-173, 1974.

M. J. Fox and J. S. Shamma, 'Population games, stable games, and passivity,' Games , vol. 4, no. 4, pp. 561-583, 2013.

M. Arcak and N. C. Martins, 'Dissipativity tools for convergence to Nash equilibria in population games,' IEEE Transactions on Control of Network Systems , vol. 8, no. 1, pp. 39-50, 2021.

L. Pavel, 'Dissipativity theory in game theory: On the role of dissipativity and passivity in Nash equilibrium seeking,' IEEE Control Systems Magazine , vol. 42, no. 3, pp. 150-164, 2022.

J. P. Hespanha, Linear Systems Theory: Second Edition . Princeton University Press, 2018.

W. J. Rugh, Linear System Theory . Prentice Hall, 1996.

M. Hassouneh, H.-C. Lee, and E. Abed, 'Washout filters in feedback control: benefits, limitations and extensions,' in Proceedings of the 2004 American Control Conference , pp. 3950-3955, 2004.

Appendices

In Appendix A, we review some relevant topics from linear systems theory. In Appendix B, we provide proofs of theorems, lemmas, and propositions that were presented in the paper. In Appendix C, we provide an alternative destabilization proof for sufficiently small gain µ .

A Background on linear systems

Here, we review some standard background material for linear dynamics systems. There are several references (e.g., (41; 42)) with more detailed exposition.

A.1 Notation

The partitioned matrix whose solution is

then

represents a general linear dynamical system

The variables x ∈ R n , u ∈ R m , and y ∈ R p here are used temporarily as generic placeholders (and will play a different role in the ensuing discussion). The notation P ∼ ( A B C D ) assigns the label P to the dynamics (16).

A.2 Washout filters

In the special case of where

and x, u, y ∈ R n , it is straightforward to show that if

Linear systems with this property are known as 'washout filters'. For an extended discussion, see reference (43).

A.3 Stability and stabilization

A matrix, M , is a stability matrix if, for every eigenvalue, λ , of M , Re [ λ ] < 0 .

The linear system, P ∼ ( A B C D ) is stable if A is a stability matrix.

The linear system K ∼ ( E F G H ) stabilizes P if the combined linear dynamics

or are stable, i.e., M is a stability matrix.

The following conditions are necessary and sufficient for the existence of such a K . For all complex λ with Re [ λ ] ≥ 0 :

The pair ( A,B ) is stabilizable : ( A -λI B ) = n has full row rank.

The pair ( A,C ) is detectable : ( C A -λI ) = n has full column rank.

The linear system K strongly stabilizes P if (i) K stabilizes P and (ii) E is a stability matrix. Necessary and sufficient conditions for the existence of a strongly stabilizing K are presented in reference (37).

A.4 Decentralized stabilization

Let P ∼ ( A B C 0 ) , with A having dimensions n × n , have the structure

for some integer k . Suppose there exist linear systems

such that the combined linear dynamics

are stable. Then the ( K 1 , ..., K k ) achieve decentralized stabilization of P . The previous conditions of ( A,B ) stabilizable and ( A,C ) detectable are necessary, but not sufficient, conditions for decentralized stabilization.

The following theorem from reference (29) provides necessary and sufficient conditions for decentralized stabilization. First, for any partition Q ∪ R = { 1 , 2 , ..., k } define B | Q as the matrix formed by extracting the block columns of B = ( B 1 ... B k ) with indices in Q , i.e.,

with { q 1 , ..., q | Q | } = Q . Likewise, define C | R as the matrix formed from the block rows of

with { r 1 , ..., r | R | } = R .

Theorem A.1 ((29), Theorem 3) . There exist ( K 1 , ..., K k ) that achieve decentralized stabilization of P if and only if

for all complex λ with Re [ λ ] ≥ 0 and all partitions, Q ∪ R = { 1 , 2 , ..., k } .

The above rank condition must hold for all partitions Q ∪ R = { 1 , ..., k } . If R = ∅ , one recovers the rank condition for (centralized) stabilizability. Likewise, Q = ∅ results in the rank condition for detectability.

B Proofs

B.1 Proof of Theorem 4.1

We will examine the conditions of Theorem A.1 on the system (13)-(14). We need to inspect the rank of

for all partitions Q ∪ R = { 1 , 2 , ..., n } . Note that the partitions of either Q = ∅ or R = ∅ are already covered by Proposition 4.1.

First, note that

Let M be an ℓ × ℓ matrix. Then

We need the rank of the above matrix to be 2 ℓ for all λ with Re [ λ ] ≥ 0 . Because of the presence of AλI , loss of rank below 2 ℓ is only possible at eigenvalues of A . Since we are only concerned with Re [ λ ] ≥ 0 , we focus on eigenvalues of M (which excludes λ = 0 by hypothesis, since M is non-singular).

Without affecting the rank, we can multiply the bottom block row by -( λ +1) and add the middle block rows corresponding R to the rescaled bottom block row to get

Switching the top and bottom block rows results in

We can now exploit the block triangular structure. The bottom block row provides a row rank of

The middle block row provides a row rank of ℓ . Finally, the top block row provides a row rank of

The last assertion is because M is non-singular, by hypothesis, and therefore it has linearly independent rows. Since Q ∪ R = { 1 , ...n } , we have the desired row rank of 2 ℓ .

B.2 Proof of Lemma 5.1

Define

Since H ( s ) is a rational function, we can write it as

for polynomials p and q that have no common roots. The assumption that CA m B = 0 for some m assures that H ( s ) is not identically equal to zero.

Suppose that for some µ and s ′ that is not an eigenvalue of A ,

Then s ′ is an eigenvalue of A -µBC , since

Note that the roots of q ( s ) are a subset of the roots of det [ sI -A ] . For sufficiently large | s | , we can rewrite H ( s ) as

By assumption, CB = 0 and CAB = 0 , which implies that the first two terms of the series equal zero. Accordingly,

The main implication here is that the degree of q ( s ) is at least 3 more than the degree of p ( s ) . Root-locus arguments in references (30) and (31) (asymptote rule) imply that

has roots with positive real parts for large µ .

̸

B.3 Proof of Proposition 5.1

The local dynamics matrix in the game Γ( µ ) can be written in the form A -µBC with

where we reordered the variables according to ( w 1 , w 3 , w 2 , ξ 1 , v 1 , ξ 3 , v 3 , ξ 2 , v 2 ) for convenience. Following the same arguments in proving Lemma 5.1, if CA m B = 0 for all m , then eigenvalues of A -BC are the same as the eigenvalues of A . Writing A in block matrix form yields

where A 11 is 3 × 3 . Notice that A 11 is strictly lower triangular, and A 21 is strictly block lower triangular. Now examine

̸

One can show that A 21 ( sI -A 11 ) -1 A 12 is strictly block lower triangular. Therefore, we have det [ sI -A ] = det [ sI -A 11 ] det [ sI -A 22 ] . Thus, A has eigenvalues at 0 with multiplicity 3 or more because of A 11 . By exponential stability, there exists m ≥ 2 such that CA m B = 0 . Since CB = 0 , and CAB = 0 , we can now apply Lemma 5.1.

B.4 Proof of Proposition 6.1

Reference (37) presents a necessary and sufficient condition for strong stabilizability. First, we compute

There is blocking zero (i.e., T ( s ) = 0 ) at s = 0 and | s | → ∞ . According to reference (37), a necessary and sufficient condition for strong stabilizability is that there should be an even number of eigenvalues in between such pairs of real zeros. This property is known as the 'parity interlacing principle'. The eigenvalues of A are -1 , -1 , ± √ m 12 m 21 , and so there is a single (and hence, an odd number) of real eigenvalue in between two real zeros of T ( s ) .

C Destabilization with a sufficiently small µ

The proof in Section 5 can be modified to consider non-convergence over bounded games, e.g.,

Consider first the following lemma.

Lemma C.1. Let A ∈ R n × n , B ∈ R n × 1 and C ∈ R 1 × n . Assume that A has eigenvalues at 0 with multiplicity 3 or more. Then for sufficiently small µ > 0 , A -µBC is not a stability matrix.

Proof. As in the proof of Lemma 5.1, we have

Recall that the roots of q ( s ) are a subset of the roots of det [ sI -A ] . If q ( s ) does not have at least 3 roots at zero, then A -µBC is not a stability matrix. Otherwise, root-locus arguments in references (30), and (31) (angle of departure rule) imply there exist roots of q ( s ) + µp ( s ) with positive real parts for small µ . ■

Using the structure of the local dynamics and the fact that A in (17a) has eigenvalues at 0 with multiplicity 3 or more, one can directly use Lemma C.1 to show the existence of a sufficiently small offending µ .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies higher order dynamics, i.e., dynamics that can rely on more auxiliary states than those limited by the dimensionality of the action spaces, in network games with pairwise interactions between players. Importantly, these dynamics only depend on the sequence of payoff signals that each player receives and are, thus, called uncoupled (an important property in multi-agent settings). The paper shows that linear versions of such dynamics can locally lead to any isolated mixed Nash equilibrium. However, for each such dynamic, there exists a \"simple\" anticoordination game for which this dynamic will not work, i.e., will not stabilize the unique isolated mixed NE. Furthermore, as the paper explains, a linear dynamic that converges to a mixed NE may not be meaningful after all.\n\n**Post-rebuttal**: After reading the other reviews and the authors' responses, I conclude that my concerns stem from my limited understanding of the techniques that are used in the paper. In response, I increase my score from 3 to 5 (and the contribution subscore from 2 to 3) to reflect my evaluation of the results, but I decrease my confidence from 3 to 2 to reflect that I didn't not understand (parts of) the techniques used in the paper.\n\nSTRENGTHS:\n- The paper provides existence results for uncoupled dynamics that (locally) converge to mixed Nash equilibria. To do so, the paper creates higher-order dynamics that overcome known limitations of lower-order dynamics, i.e., of dynamics whose dimension is constrained by the dimension of the action spaces. \n- The main takeaways of the paper are clearly presented.\n\nWEAKNESSES:\n- The paper is hihgly not self-contained. There is a strong reliance to prior literature and to the appendix. \n- As a result of the above and of the complicated notation, the paper was very hard for me to read. Although, I couldn't follow some parts and I couldn't verify the derivation of some results, the results seems plausible and the main takeaways are still clear (as mentioned above). \n- I found the motivation of main dynamics in line 140 (paragraph 3.2) inadequate - but this may be related to the fact, that the exposition was not good enough for me to be able to follow.\n- While the results inform the discussion on paper [16], they are merely existential and quite general. So, their practical and theoretical scope may be rather limited.\n- The paper could have done a better job in building upon more recent literature regarding convergence of dynamics to NE in games.\n\nQUESTIONS:\n- Can the authors address the weaknesses mentioned above? \n- Line 22: not the most updated list of papers. Here is an indicative reference to help the authors locate more recent papers in the area in my opinion: https://papers.nips.cc/paper_files/paper/2020/file/0ed9422357395a0d4879191c66f4faa2-Paper.pdf\n- Line 77: P_i(x_-i) and P(x_-i): I don't understand the subscript i in P_i. Also, the next sentence in lines 77-78 is even more confusing.\n- Line 80: is \"a\" tuple (and other such minor typos - the paper needs a proofreading)\n- Equations (4), (5) and (7): I had a hard time to follow the derivation of these equations. This is one instance of my comment above that the paper is highly not self-contained.\n- Line 160: wasn't y_i defined above as \\dot v_i? Is that the same?\n- Line 171: stabilizability and detectability seem to be important but are only defined in the Appendix. I think that the paper needs to be rewritten in a way that it is self-contained and easier to follow.\n- Line 241: locally exponentially stable - the same here. This notion has not been defined before.\n- Line 288-289: I missed this argument. Is it that the dynamics fail to monotonically improve with respect to input payoffs (line 282)? Again, I couldn't understand the argument without relying on the Appendix.\n- Lines 297/304: anticipatory higher-oder learning/passivity, contractive games - another series of terms that are used without having being defined before. At least the conclusions should be accessible by a wider audience, but this is not the case.\n\nLIMITATIONS:\nYes, adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-agent learning dynamics and the central question is if there is an iterative learning process in a multiplayer game that leads to a Nash equilibrium. There has been substantial prior work on this question for many specific games and learning strategies—generally it is not the case that known dynamics lead to a Nash equilibrium and this is perhaps frustrating as it would be convenient to be able to find Nash equilibria this way. The authors approach this problem in a way that is novel, as far as I know, by specifying a class of interesting/acceptable learning dynamics and using tools from feedback control systems to prove properties of that class. For interesting/acceptable, they require that dynamics are \"payoff based,\" which essentially limits the ability of each agent to see \"into\" the action space of other agents, and, I believe for tractability of the analysis, they limit the dynamics to a kind of generalized higher-order gradient play. They specifically show that\n1. If a game has a strictly mixed Nash equilibrium, there exist payoff-based dynamics that converge locally to that NE. (As a consequence, they show that these dynamics also converge to NE of \"nearby\" games.)\n2. However, there is no \"overall good\" dynamics—for any such dynamics, there exists a game with unique mixed NE such that these dynamics are unstable at that NE.\n\nSTRENGTHS:\nThe results are likely to be novel and provide important context to those who study multi-agent learning dynamics. As is typical, there are some asterisks (the particular learning dynamics are general, but I think the broader questions could motivate looking at an even more general class). The negative result about games with a unique NE seems particularly strong.\n\nThe authors have done substantial work to make the results interpretable, at least on the surface, to a broader audience that is not familiar with the control-theoretic tools they are using. I would challenge the authors to go even further with this (see weaknesses below).\n\nWEAKNESSES:\nFor someone who is not familiar with the specific tools the authors use, the paper is quite hard to follow, even if the game-theoretic aspects are clear and familiar. Appendix D has some worked examples with plots in them. I would suggest thinking about whether it is possible to move some of this content to the main paper. The figures in the paper itself are not that helpful and could be improved, potentially with more detail. The paper lacks good figures currently.\n\nAnything the authors can do to further broaden the context of their results would be helpful.\n\nQUESTIONS:\nI would be interested in the authors' thoughts on whether they view the class of higher-order dynamics they study as restrictive or not. It would be helpful to know, of the papers they cite, in which cases their dynamics class subsumes that of the methods proposed by that paper.\n\nLIMITATIONS:\nYes, this is a theoretical work and the assumptions of the theorems are clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies higher order payoff based learning, and in particular higher order gradient play in this setting. The authors show that for games with isolated completely mixed NE, there are higher-order gradient play dynamics that converge to that NE. Moreover, that same NE is converged to in ‘nearby’ games as well. However, the authors also show that there exist anti-coordination games where higher-order gradient play dynamics fail to converge to NE. Finally the authors also argue that dynamics that do lead to NE in a coordination game must be inherently unstable.\n\nSTRENGTHS:\nI find the paper to be well structured and readable. The results in the context of uncoupled learning dynamics in games are also interesting and meaningfully extend existing ideas into higher order dynamics.\n\nWEAKNESSES:\nA weak point in the paper for me is that the results are based upon the assumption that the mixed NE is a practically useful solution concept in learning in games. However, recent works have shown that the NE can often not only be a poor metric for players’ performance, but also it is unnatural for players using decentralized dynamics to converge to an NE in general games. Thus, for a paper that focuses on higher order learning it would have been much more compelling to focus on a more complete picture of higher order gradient dynamics. What characterizes stable equilibria/fixed points for these dynamics in this setting? In cases where NE are not stable, what do the dynamics look like? In my opinion, a broader view of the dynamical system properties would make the results more interesting and useful.\n\nQUESTIONS:\nFor the higher order dynamics, is there intuition about the bandit setting where players only observe (potentially random) realizations of their payoffs? This seems more reasonable for the cases where payoff vectors are large/there are a large number of players and complexity is a concern.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of the dynamics studied.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper shows that for any finite game with an isolated completely mixed Nash Equilibrium, there exist a payoff based higher-order
gradient play dynamics that lead (locally) to that Nash equilibrium, both for this game and all payoff nearby games. Conversely, they show that for
any higher-order gradient play dynamics, there is a game with a unique isolated completely mixed Nash equilibrium for which that dynamics do not locally converge to that Nash Equilibrium. 

 Hart and A. Mas-Colell proved using an anti-coordination game that no first-order uncoupled dynamics leads to the unique interior Nash equilibrium of that game.  Shamma and Arslan (2005) proved that there are higher order dynamics which leads to that equilibrium. This paper shows that this extends to all games.

STRENGTHS:
Tools used are, up to my knowledge, not standard in evolutionary game theory: decentralized stabilizing control and root-locus which characterizes the locations of the eigenvalues of a matrix as a function of a scalar parameter!

WEAKNESSES:
The biggest weakness for me is that there is no micro-foundation of the class higher order dynamics nor a comparison with the previously studied higher order dynamics.

QUESTIONS:
- Are your dynamics more general than all the previously studied one (such as [18, 19, 20, 23] etc)?
 - Do you have any micro-foundation of your higher order dynamics?

LIMITATIONS:
The converse result (for any higher-order dynamics, there is a game with a unique isolated completely mixed Nash equilibrium) is proved only for gradient play dynamics and not all higher order dynamics !
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows the lack of universality on the side of both games and learning dynamics (even for higher-order ones)! Particularly, for any game with a mixed-strategy Nash equilibrium (NE), there exists uncoupled payoff-based (possibly high-order) dynamic converging locally to the NE. However, any such dynamics can also be destabilized by a suitable anti-coordination game. Notably, the paper uses classical analysis methods in feedback control systems. Highlighted similarities between higher-order learning in games and higher-order optimization algorithms, such as momentum-based or optimistic gradient algorithms, are also interesting.\n\nSTRENGTHS:\nThe widely studied fictitious play dynamics are known to converge equilibrium in many interesting games but not all of them, e.g., see Shapley's counterexample. Therefore, researchers were looking for a learning dynamic that can converge to equilibrium in every game to justify equilibrium analysis. However, Hart and Mas-Colell, Ref. (16), proved the negative result that there does not exist (first-order) uncoupled learning dynamics that can converge to equilibrium in anti-coordination games, and therefore, there cannot be universally convergent (uncoupled) learning dynamics.\n\nLater, Shamma and Arslan, Ref. (17), showed that higher-order learning dynamics can converge to equilibrium in anti-coordination games. This paper provides a more general result saying that for any game (with a mixed-strategy Nash equilibrium), there exists a (possibly high-order) payoff-based learning dynamic that can converge locally to that equilibrium. Seeing this, researchers may start looking for universally convergent higher-order learning dynamic that can converge to equilibrium in every game. However, the paper proves the negative result that given any such higher-order dynamics, there always exists a certain anti-coordination game in which the dynamics do not converge to equilibrium. Therefore, we can view this paper as a generalization of Hart and Mas-Collel's seminal negative result to higher-order learning dynamics.\n\nNote that Foster and Young designed uncoupled stochastic rules, known as regret testing, that can converge probabilistically to Nash equilibrium in every two-player strategic-form game. However, the convergence is in the relatively weak sense by saying that players will be at equilibrium most of the time though they may move away from it.\n\nNote also that complexity results related to Nash equilibrium computation are not relevant since the paper focuses on asymptotic convergence for finite games. A mixed-strategy Nash equilibrium always exists in strategic-form games, with finitely many player and action.\n\nBecause of these reasons, I believe that this result is worth being taught in (advanced) game theory courses.\n\nI acknowledge that I have read the rebuttal.\n\nWEAKNESSES:\n- Figures might include captions with more detailed descriptions.\n\nQUESTIONS:\n- What is the main obstacle to address instantaneous scalar payoffs rather than payoff vector setup?\n\nLIMITATIONS:\nThe limitations are highlighted explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}