Title: Exponential Lower Bounds for Fictitious Play in Potential Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: fictitious play

convergence rate

potential games
Abstract: Fictitious Play (FP) is a simple and natural dynamic for repeated play with many applications in game theory and multi-agent reinforcement learning. It was introduced by Brown and its convergence properties for two-player zero-sum games was established later by Robinson. Potential games [Monderer and Shapley 1996] is another class of games which exhibit the FP property [Monderer and Shapley 1996], i.e., FP dynamics converges to a Nash equilibrium if all agents follows it. Nevertheless, except for two-player zero-sum games and for specific instances of payoff matrices [Abernethy et. al. 2021] or for adversarial tie-breaking rules [Daskalakis and Pan, 2014], the \textit{convergence rate} of FP is unknown. In this work, we focus on the rate of convergence of FP when applied to potential games and more specifically identical payoff games. We prove that FP can take exponential time (in the number of strategies) to reach a Nash equilibrium, even if the game is restricted to \textit{two agents}. To prove this, we recursively construct a two-player coordination game with a unique Nash equilibrium. Moreover, every approximate Nash equilibrium in the constructed game must be close to the pure Nash equilibrium in $\ell_1$-distance.

Exponential Lower Bounds for Fictitious Play in Potential Games

Abstract

Fictitious Play (FP) is a simple and natural dynamic for repeated play with many applications in game theory and multi-agent reinforcement learning. It was introduced by Brown [4, 5] and its convergence properties for two-player zero-sum games was established later by Robinson [22]. Potential games [19] is another class of games which exhibit the FP property [18], i.e., FP dynamics converges to a Nash equilibrium if all agents follows it. Nevertheless, except for two-player zero-sum games and for specific instances of payoff matrices [1] or for adversarial tie-breaking rules [9], the convergence rate of FP is unknown. In this work, we focus on the rate of convergence of FP when applied to potential games and more specifically identical payoff games. We prove that FP can take exponential time (in the number of strategies) to reach a Nash equilibrium, even if the game is restricted to two agents and for arbitrary tie-breaking rules. To prove this, we recursively construct a two-player coordination game with a unique Nash equilibrium. Moreover, every approximate Nash equilibrium in the constructed game must be close to the pure Nash equilibrium in ℓ 1 -distance.

1 Introduction

In 1949 Brown [4, 5] proposed an uncoupled dynamics called fictitious play so as to capture the behavior of selfish agents engaged in a repeatedly-played game. Fictitious play assumes at round t = 1 , each agent selects an arbitrary action. At each round t ≥ 2 , each player plays a best response pure action to the opponents' empirical strategy ; empirical strategy is defined to be the average of the past chosen strategies.

Due to its simplicity and natural behavioral assumptions, fictitious play is one of the most seminal and well-studied game dynamics [7]. Despite the fact that fictitious play does not converge to a Nash equilibrium (NE) in general normal-form games, there are several important classes of games at which the empirical strategies always converge to a NE. In her seminal work, Robinson [22] showed that in the case of two-player zero-sum games, the empirical strategy profiles converge to a min-max equilibrium of the game; Robinson's proof use a smart inductive argument on the number of strategies of the game. Later, Monderer and Shapley [18] established that in the case of N -player potential games, the empirical strategies also converge to a NE. Summing up, the following theorem is true:

Nikolas Patris ∗

Informal Theorem [22],[18] In two-player zero-sum and N -player potential games, the empirical strategy profiles of fictitious play converge to a NE for any initialization and tie-breaking rule * .

The above convergence results are asymptotic in the sense that they do not provide guarantees on the number of rounds needed by fictitious play to reach an approximate NE. Karlin [13] conjectured that in the case of two-player zero-sum games, fictitious play requires O (1 /ϵ 2 ) rounds to reach an ϵ -approximate NE. Daskalakis et. al. [9] disproved the strong version of Karlin's conjecture by providing an adversarial tie-breaking rule for which fictitious play requires exponential number of rounds (with respect to the number of strategies) in order to converge to an ϵ -approximate NE. On the other hand, for the case of potential games and despite the foundational work of Monderer et al. [18], no lower bound results have been established. Therefore, the central focus of our work is to address the following question."

Q: Does fictitious play in potential games admit convergence to an approximate NE with rates that depend polynomially on the number of actions and the desired accuracy?

Our contributions Our work provides a negative answer on the above question. Specifically, we present a two-player potential game for which fictitious play requires super-exponential time with respect to the number of actions to reach an approximate NE.

Theorem 1.1 (Main result, formally stated Theorem 3.1) . There exists a two-player potential game (more specifically both agents have identical payoffs) in which both agents admit n actions and for which fictitious play requires Ω ( 4 n (( n 2 -2 ) ! ) 4 + 1 n √ ϵ ) rounds in order to reach an ϵ -approximate NE. Moreover, the result holds for any tie-breaking rule and uniformly random initialization.

Remark 1.2. Daskalakis et al. [9] provide an exponential lower bound on the convergence of fictitious play assuming an adversarial tie-breaking rule, meaning that ties are broken in favor of slowing down the convergence rate. To this point, it is not known whether fictitious play with a consistent tie-breaking rule (e.g. lexicographic) converges in polynomial time or not, except for special cases, e.g., diagonal payoff matrices [1]. We would like to note that our lower bound construction holds for any tie-breaking rule. The latter indicates an interesting discrepancy between two-player zero-sum and two-player potential games.

Related Work The work of Daskalakis et al. [9] provides a lower bound on the convergence rate of fictitious play for the case of two-player zero-sum games using an adversarial tie-breaking rule and is the most closely related to our. Another important work is [1], in which they demonstrate that when the tie-breaking rule is predetermined (e.g., using lexicographical tie-breaking), the convergence rate of fictitious play becomes polynomial in the number of strategies, specifically with a rate of O ( 1 ϵ 2 ) , as Karlin's conjecture states. It is important to note that this result holds true only for diagonal matrices.

Other works include the examination of fictitious play convergence for near-potential games [6], and the investigation of the necessary conditions that games must meet for fictitious play to converge to a NE [8]. We note that neither of these works provide convergence rates for fictitious play. On the other hand, fast convergence rates of continuous time fictitious play for regular , as introduced by Harsanyi [10], potential games are established [25]. Additional works on continuous-time fictitious play can be found in [21], and references there in.

Fictitious play has found various application in Multi-agent Reinforcement Learning as well (see [20, 23, 2, 3] and [26] for a survey and references therein), extensive form games [11], and control theory [15, 14]. It possesses not only an intuitive game theoretic interpretation but also other attractive properties, such as its simplicity with no need for a step-size parameter. These traits have contributed to its status as a pivotal topic of study, leading to numerous works aiming to establish convergence in broader settings [16, 24, 17, 12]. *

Technical overview The technical overview of this paper provides a high-level roadmap of the key contributions. In Section 3, we outline the steps towards proving Theorem 3.1 by recursively constructing a payoff matrix of carefully crafted structural properties. In that matrix, starting from the

lower-left element, the sequence of successive increments form a spiraling trajectory that converge towards the element of maximum value. We demonstrate that fictitious play has to follow the same trajectory, passing through all non-zero elements, to reach the unique pure Nash equilibrium, as illustrated in Figure 2. A crucial component of the proof is provided by an induction argument that emulates the movement of fictitious play and provides a super-exponential lower bound on the number of rounds needed.

2 Preliminaries

2.1 Notation and Definitions

Notation Let R be the set of real numbers, and [ n ] = { 1 , 2 , . . . , n } be the set of actions. We define ∆ n as the probability simplex, which is the set of n -dimensional probability vectors, i.e., ∆ n := { x ∈ R n : x i ≥ 0 , ∑ n i =1 x i = 1 } . We use e i to denote the i -th elementary vector, and to refer to a coordinate of a vector, we use either x i or [ x ] i . The superscripts are used to indicate the time at which a vector is referring to.

Normal-form Games In a two-player normal-form game we are given a pair of payoff matrices A ∈ R n × m and B ∈ R n × m where n and m are the respective pure strategies of the row and the column player. If the row player selects strategy i ∈ [ n ] , and the column player selects strategy j ∈ [ m ] , then their respective payoffs are A ij for the row player and B ij for the column player.

The agents can use randomization. A mixed strategy for the row player is a probability distribution x ∈ ∆ n over the n rows, and a mixed strategy for the column player is a probability distribution y ∈ ∆ m over the m columns. After the row player selects mixed strategy x ∈ ∆ n , and the column player selects mixed strategy y ∈ ∆ m , their expected payoffs are x ⊤ Ay and x ⊤ By , respectively.

Potential Games A two-player potential game is a class of games that admit a unique function Φ , referred to as a potential function, which captures the incentive of all players to modify their strategies. In other words, if a player deviates from their strategy, then the difference in payoffs is determined by a potential function Φ evaluated at those two strategy profiles. We can express this formally as follows:

Definition 2.1 (Two-player Potential Game) . For any given pair of strategies ( x, y ) and a pair of unilateral deviations x ′ by the row player and y ′ by the column player, the difference in their utility is equivalent to the difference in the potential function.

Remark 2.2. We note Φ is a function that characterizes the equilibria of the game as the strategy profiles that maximize the potential function.

In this work, we focus on a specific type of potential games called identical payoff games , where both players receive the same payoff.

Definition 2.3 (Identical Payoff Games) . A two-player normal-form game ( A,B ) is called identical payoff if and only if A = B .

In this scenario, it is apparent that the potential function is given by Φ( x, y ) = x ⊤ Ay . Finally we provide the definition of an approximate NE.

Definition 2.4 ( ϵ -Nash Equilibrium) . A strategy profile ( x ⋆ , y ⋆ ) ∈ ∆ n × ∆ m is called an ϵ -approximate NE if and only if

In words, an approximate Nash equilibrium is a strategy profile in which no player can improve their payoff significantly by unilaterally changing their strategy, but the strategy profile may not necessarily satisfy the precise definition of a Nash equilibrium.

Remark 2.5. We highlight two special cases of the ϵ -NE. Firstly, when ϵ is equal to zero, it is referred to as an exact Nash equilibrium . Secondly, when the support of strategies is of size 1, it is called a pure Nash equilibrium . It is worth noting that potential games always admit a pure NE.

2.2 Fictitious Play

Fictitious play is a natural uncoupled game dynamics at which each agent chooses a best response to their opponent's empirical mixed strategy. Since there might be several best response actions at a given round, fictitious play might contain different sequences of play; see Definition 2.6.

Definition 2.6 (Fictitious Play) . An infinite sequence of pure strategy profiles ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . is called a fictitious play sequence if and only if at each round t ≥ 2 ,

The empirical strategy profile of row and column player at time T is defined as ˆ x ( T ) = ( 1 T ∑ T s =1 e i ( s ) ) and ˆ y ( T ) = ( 1 T ∑ T s =1 e j ( s ) ) where e i ( t ) , e j ( t ) are the elementary basis vectors.

Definition 2.7 (Cumulative utility vector) . For an infinite sequence of pure strategy profiles ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . , the cumulative utility vectors of the row and column player at round t ≥ 1 are defined as,

Remark 2.8. Fictitious play assumes that each agent selects at each round t ∈ [ T ] a strategy with maximum cumulative utility . The latter decision-making algorithm is also known as Follow the Leader . We remark that the latter alternative interpretation provides a direct generalization of fictitious play in N -player games.

In their seminal work, Monderer et al.[18] established that in case of identical payoff games the empirical strategies of any fictitious play sequence converges asymptotically to a NE.

Theorem 2.9 ([18]) . Let a fictitious play sequence ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . for an identical payoff game described with matrix A . Then, there exists a round T ⋆ ≥ 1 such that for any t ≥ T ⋆ , the empirical strategy profile (ˆ x ( t ) , ˆ y ( t ) ) converges to a NE with a rate of 1 /t .

On the positive side, Theorem 2.9 establishes that any fictitious play sequence converges to a Nash equilibrium in the case of potential games * On the negative side, Theorem 2.9 does not provide any convergence rates, since the round T ⋆ depends on the specific fictitious play sequence and its dependence on the number of strategies is rather unclear.

3 Main Result

In this section, we outline the steps towards proving Theorem 3.1, as it is stated below. Firstly, we introduce a carefully constructed payoff matrix A of size n × n and analyze its structural properties in Section 3.1. Next, in Section 3.2, we investigate the behavior of fictitious play when the game is an two-player identical payoff game with this matrix A . We also present a set of key statements that are necessary for proving the main theorem. Finally, in Section 3.3, we provide a proof for the fundamental Lemma 3.8.

Theorem 3.1. Let an identical payoff game defined with the matrix A of size n × n and consider any fictitious play sequence ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . with ( i (1) , j (1) ) = ( n, 1) . In case the empirical strategy profile (ˆ x ( T ) , ˆ y ( T ) ) is an ϵ -approximate Nash equilibrium then it holds

Moreover, the lower bound on T is independent of the tie-breaking rule. Finally, if the initialization is chosen uniformly at random, then the expected number of rounds to reach an ϵ -approximate Nash equilibrium is Ω ( 4 n (( n/ 2 -2)!) 4 +1 /n √ ϵ n 2 ) .

3.1 Construction and Analysis of the Payoff Matrix A

We begin by introducing our recursive construction for the payoff matrix, which we use to establish the formal statement of Theorem 3.1.

Definition 3.2. For any z > 0 and n even, consider the following n × n matrix K n ( z ) .

For n ≥ 4 ,

K n n 1 ( z ) = z +1 and K n nj ( z ) = 0 for j / ∈ { 1 } . Row n

K n n 1 ( z ) = z +1 , K n 11 ( z ) = z +2 and K n 1 j ( z ) = 0 for any j / ∈ { 1 , n } . Column 1

K n 11 ( z ) = z +2 , K n 1 n ( z ) = z +3 and K n 1 j ( z ) = 0 for j / ∈ { 1 , n } . Row 1

K n 1 n ( z ) = z +3 , K n n -1 n ( z ) = z +4 and K n nj ( z ) = 0 for j / ∈ { 1 , n -1 } . Column n

For all i, j ∈ { 2 , . . . , n -1 } × { 2 , . . . , n -1 } , K n ij ( z ) := K n -2 i -1 j -1 ( z +4) .

In Figures 1a and 1b, we provide a schematic representation of Definition 3.2 and an illustrative example for n = 6 and z = 0 . For the sake of simplicity, we have intentionally omitted the remaining zeros in the outer rows, and columns of the matrix.

The construction of the payoff matrix exhibits an interesting circular pattern, which begins at the lower left corner and extends along the outer layer of the matrix. More specifically, the first increment occurs in the same column as the starting point, i.e., at position (1 , 1) on the first row. The pattern then proceeds to the next greater element on the same row but a different column, i.e., at position (1 , n ) , and the last increment before entering the inner sub-matrix is located on the same column but on the ( n -1) -th row.

The increments have been carefully selected to ensure that there are alternating changes in row and column when starting from the lower-left corner and following the successive increments, until reaching the sub-matrix in the center. Once inside the sub-matrix, a similar pattern continues. As we will explore later on, the structure of the payoff dictates the behavior of fictitious play. We denote the payoff matrix under consideration as A , which is defined as A := K n (0) . The subsequent statements establish the key properties of A .

Observation 3.3 (Structural Properties of matrix A ) . Let the matrix A = K n (0) , then for i ∈ { 0 , . . . , n 2 -1 } the following hold:

The only elements with non-zero values in column i +1 are located at positions i +1 and n -i , and have values 4 i +2 and 4 i +1 , respectively.

The only non-zero elements of row i +1 are located at positions i +1 and n -i , and have values 4 i +2 and 4 i +3 , respectively.

The only non-zero elements of column n -i are located at positions i +1 and n -i -1 and have values 4 i +3 and 4 i +4 , respectively.

The only non-zero elements of row n -i are located at positions i +1 and n -i +1 , and have values 4 i +1 and 4 i , respectively.

Observation 3.4. The maximum value of A is 2 n -2 and is located at the entry ( n 2 , n 2 +1 ) .

Proposition 3.5. For any non-zero element in the matrix A , there is at most one non-zero element that is greater and at most one non-zero element that is smaller in the same column or row.

Proof. By Observation 3.3, each row and column of matrix A contains at most two non-zero elements that are necessarily different to each other. Thus, if ( i, j ) is a non-zero element of A , there can be at most two additional non-zero elements in row i and column j combined. Moreover, at most one of these elements can be greater and at most one can be smaller.

One of the central components of the main theorem is presented below. Lemma 3.6 establishes that in an identical payoff game with matrix A , any approximate Nash equilibrium must distribute the majority of its probability mass to the maximum element in A , which is located at the entry ( n 2 , n 2 +1) . The proof of this theorem is based solely on the structural properties presented in Observation 3.3, and is provided in the Appendix.

Lemma 3.6 (Unique ϵ 2 -NE) . Let ϵ ∈ O ( 1 n 3 ) and consider an ϵ 2 -approximate Nash Equilibrium ( x ∗ , y ∗ ) . Then the following hold,

Lemma 3.6 not only establishes that the only approximate Nash equilibrium of the identical-payoff game with matrix A corresponds to the strategies ( n 2 , n 2 +1) , but it also implies that this is the only exact Nash equilibrium (i.e., ϵ = 0 ). This observation follows from Observation 3.4, which states that the entry ( n 2 , n 2 +1) corresponds to a maximum value of A and hence it is a Nash equilibrium as it dominates both the row and the column that it belongs to. We can formally state this observation as follows.

Corollary 3.7 (Unique pure NE) . In an identical-payoff game with payoff matrix A , there exists a unique pure Nash equilibrium at ( n 2 , n 2 +1 ) .

3.2 Lower Bound for Fictitious Play in a Game with Matrix A

In this subsection, we present the proof of Theorem 3.1. To achieve this, we first prove that fictitious play requires super-exponential time before placing a positive amount of mass in entry ( n 2 , n 2 +1) . This result is established by our main technical contribution of the subsection, which is Lemma 3.8.

Lemma 3.8. Let an identical-payoff game with payoff matrix A and a fictitious play sequence ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . with ( i (1) , j (1) ) = ( n, 1) . Then, for all ℓ = { 0 , . . . , n 2 -1 } there exists a round T ℓ ≥ 1 such that:

the agents play the strategies ( n -ℓ, ℓ +1) for the first time,

all rows r ∈ [ ℓ +1 , n -ℓ -1] admit 0 cumulative utility, R ( T ℓ ) r = 0 ,

all columns c ∈ [ ℓ +2 , n -ℓ ] admit 0 cumulative utility, C ( T ℓ ) c = 0 .

Moreover for ℓ ≥ 2 , the cumulative utility of row n -ℓ at round T ℓ is greater than

Using Lemma 3.8 we are able to establish that for a very long period of time the row player has never played row n 2 and that the column player has never played column n 2 +1 .

Lemma 3.9 (Exponential Lower Bound) . Let an identical-payoff game with matrix A and a fictitious play sequence ( i (1) , j (1) ) , . . . , ( i ( t ) , j ( t ) ) , . . . with ( i (1) , j (1) ) = ( n, 1) . In case ( i ( T ) , j ( T ) ) = ( n 2 , n 2 +1) then T ≥ Ω(4 n (( n/ 2 -2)!) 4 ) .













j

i

′

i













j

i

′

i













j

i

′

i













j

i

′

i













j

i

′

i

Proof. Based on Lemma 3.8, we can guarantee the existence of a round T ⋆ := T n/ 2 -1 when the players choose the strategy profile ( n 2 +1 , n 2 ) for the first time. In addition, at round T ⋆ , it holds that R ( T ⋆ ) n/ 2 -1 > 0 and C ( T ⋆ ) n/ 2+1 = R ( T ⋆ ) n/ 2 = 0 . The latter condition ensures that the strategy profile ( n 2 , n 2 +1) has not been played up to time T ⋆ .

As indicated by Observation 3.3, row n 2 has non-zero entries at columns n 2 and n 2 +1 . Therefore, if the cumulative utilities R ( T ⋆ ) n/ 2 at time T ⋆ is zero, this implies that neither of these columns has been chosen up to that point. By the same reasoning, column n 2 +1 has a non-zero entry only at row n 2 , indicating that this row has not been chosen as well.

In order to continue, we require an estimate of the duration during which the strategy profile ( n 2 +1 , n 2 ) will be played. Observation 3.3 guarantees that the utility vector of the row player is the following.

̸

We can combine this information with the fact that R ( T ⋆ ) n/ 2+1 = 0 , R ( T ⋆ ) n/ 2 = 0 , and that the respective entries in the payoff vector (3) of the row player differ by exactly one. This allows us to conclude that it will take at least R ( T ⋆ ) n 2 +1 iterations for the cumulative utilities to become equal, i.e R n 2 +1 = R n 2 . Therefore, the lower bound on the number of iterations holds regardless of the tie-breaking rule.

Now, if at any time T the agents play the strategy profile ( i ( T ) , j ( T ) ) = ( n 2 , n 2 +1) , we can conclude that T ≥ R ( T ⋆ ) n 2 +1 . Using Equation (2) of Lemma 3.8 we obtain:

Now that we have established the necessary technical result in Lemma 3.9, we are ready to present the proof of Theorem 3.1.

Proof of Theorem 3.1. Let T ⋆ ≥ 1 denote the first time at which ( i ( t ) , j ( t ) ) = ( n 2 , n 2 +1) . For any t ≤ T ⋆ -1 , it holds that ˆ x ( t ) n 2 = ˆ y ( t ) n 2 +1 = 0 . Thus, Lemma 3.6 implies that (ˆ x ( t ) , ˆ y ( t ) ) is not an approximate NE. On the other hand, at each round t ≥ T ⋆ we are ensured that ˆ x ( t ) n 2 , ˆ y ( t ) n 2 +1 converges to 1 with rate 1 /t . Applying Lemma 3.6 for ϵ : -√ ϵ we get that if (ˆ x ( t ) , ˆ y ( t ) ) is an ϵ -NE then t ≥ T ⋆ + 1 n √ ϵ ≥ Ω ( 4 n (( n/ 2 -2)!) 4 ) + 1 n √ ϵ . It is evident that, even with a uniformly random initialization, the probability of selecting ( n, 1) as the starting point for fictitious play is 1 /n 2 and so the claim of the theorem follows.

3.3 Strategy Switches and Proof of Lemma 3.8

This subsection is dedicated to presenting the proof of Lemma 3.8. However, before delving into the proof, we first provide additional statements that serve to shed light on the sequence of strategy profiles generated by fictitious play.

According to Proposition 3.5, if the strategy ( i, j ) is chosen by fictitious play, then either in row i or in column j , there is at most one element greater than A ij . Therefore, in a subsequent round, fictitious play will necessarily choose this element as its strategy. Let i ′ be the row where the element of greater value is located. Intuitively, we can imagine that as fictitious play continues to play the same strategy ( i, j ) , the payoff vector Ae j is repeatedly added to the cumulative vector of the row player. Since this vector has a greater value at coordinate i ′ than at coordinate i , a strategy switch will eventually occur. We state this observation in Proposition 3.10 and defer the proof to the Appendix.

̸

Proposition 3.10 (Strategy Switch) . Let ( i ( t ) , j ( t ) ) be a strategy selected by fictitious play at round t , and ( i ( t ) , j ( t ) ) = ( n 2 , n 2 ) . Then, in a subsequent round, fictitious play will choose the strategy of greater value that is either on row i ( t ) or column j ( t ) .

As previously hinted, the structure of the payoff matrix dictates the sequence of strategy profiles chosen by fictitious play. The sequence of strategy switches alternates between rows and columns. Continuing with the example from the previous paragraph, let us assume that the row player made the most recent strategy switch from ( i, j ) to ( i ′ , j ) . This implies that the element A i ′ j is greater than A ij , as otherwise a strategy switch would not have taken place, as established in Proposition 3.10. Moreover, by Proposition 3.5, we know that there must be an element of greater value in either row i ′ or column j . Since A i ′ j is greater than A ij , any element of greater value must be located in row i ′ . The same reasoning applies for the case where the column player was the last to switch strategies. We can summarize this observation by stating that if one player is the last to switch, then the other player must switch next. We formally state this in Corollary 3.11 and defer the proof to the Appendix.

Corollary 3.11 (Successive Strategy Switches) . Let t be a round in which a player changes their strategy. Then exactly one of the following statements is true:

̸

If the row player changes their strategy at round t , i.e. i ( t ) = i ( t -1) , then the column player can only make the next strategy switch.

̸

If the column player changes their strategy at round t , i.e. j ( t ) = j ( t -1) , then the row player can only make the next strategy switch.

Applying the same concept, we can observe that starting from the lower-left corner, fictitious play follows a spiral trajectory. The resulting spiral is illustrated in Figure 2. We now proceed with the main result of this section, Lemma 3.8.

Proof. Since ( i (1) , j (1) ) = ( n, 1) all the above claims trivially for T 0 = 1 . We assume that the claim holds for i and will now establish it inductively for i +1 .

By the induction hypothesis, agents play strategies ( n -i, i +1) at round T 0 i := T i . Furthermore, row n -i admits cumulative utility of R ( T 0 i ) n -i while row i +1 admits cumulative utility of R ( T 0 i ) i +1 = 0 . According to Observation 3.3, the payoff vectors of the row and column agent are highlighted in Figure 3a. By combining these facts, we can establish the following.

Proposition 3.12 (Abridged; Full Version in Proposition B.3) . There exists a round T 1 i > T 0 i at which the strategy profile is ( i +1 , i +1) for the first time, column i +1 admits cumulative utility C ( T 1 i ) ≥ (4 i +1) · ( R ( T 0 i ) +1) , and C ( T 1 i ) = 0 .

By Proposition 3.12, at round T 1 i , the agents play strategies ( i +1 , i +1) . Furthermore, the cumulative utility of column n -i equals to C ( T 1 i ) n -i = 0 . According to Observation 3.3, the payoff vectors of the row and column agent are highlighted Figure 3b. Combining these facts, we get the following:

Proposition 3.13 (Abridged; Full Version in Proposition B.4) . There exists a round T 2 i > T 1 i at which the strategy profile is ( i + 1 , n -i ) for the first time, row i + 1 admits cumulative utility R ( T 2 i ) i +1 ≥ (4 i +2) · C ( T 1 i ) i +1 , and R ( T 2 i ) n -i -1 = 0 .

By Proposition 3.13, at round T 2 i , the agents play strategies ( i +1 , n -i ) . Furthermore, the cumulative utility of row n -i -1 equals to R ( T 2 i ) n -i -1 = 0 . According to Observation 3.3, the payoff vectors of the row and column agent are highlighted Figure 3c. By combining these facts, we get the following:

Proposition 3.14 (Abridged; Full Version in Proposition B.5) . There exists a round T 3 i > T 2 i at which the strategy profile is ( n -( i +1) , n -i ) for the first time, R ( T 3 i ) i +2 = 0 .

By Proposition 3.14, at round T 3 i , the agents play strategies ( n -i -1 , n -i ) . the cumulative of column i +2 equals to R ( T 3 i ) i +2 = 0 . According to Observation 3.3, the payoff vectors of the row and column agent are highlighted Figure 3d. By combining these facts, we can establish the following.

Proposition 3.15 (Abridged; Full Version in Proposition B.6) . There exists a round T 4 i > T 3 i at which the strategy profile is ( n -( i +1) , ( i +1)+1) for the first time, row n -i -1 admits cumulative utility R ( T 4 i ) n -( i +1) ≥ (4 i +4) · C ( T 3 i ) n -i , all rows k ∈ [( i +1) + 1 , n -( i +1) -1] admit R ( T 4 i ) k = 0 and all columns k ∈ [( i +1) + 2 , n -( i +1)] admit C ( T 4 i ) k = 0 .

Proposition 3.15 establishes that there exits a round T i +1 := T 4 i at which the strategy profile ( n -( i +1) , ( i +1) + 1) is played for the first time. Furthermore, Proposition 3.15 confirms that all rows k ∈ { ( i +1)+1 , n -( i +1) -1 } admit R ( T i +1 ) k = 0 and all columns k ∈ { ( i +1)+2 , n -( i +1) } admit C ( T i +1 ) k = 0 . We still need to verify the the recursive relation Equation (2). By combining Proposition 3.12, 3.13, 3.14 and 3.15, we can deduce

4 Experiments

In this section, we aim to experimentally validate our findings on a 4 × 4 payoff matrix. Our analysis focuses on three key aspects: the round in which a new strategy switch occurs, the Nash gap throughout the game, and the empirical strategy employed by the x player. We present the plot from the row player's perspective, which is identical to that of the column player.

In Figure 4a, we provide the time steps of all strategy switches. As it is expected from the analysis, fictitious play visits all strategies, specifically in increasing order of their utility, to reach the pure Nash equilibrium. Moreover, in Figure 4b we observe a recurring pattern in the Nash gap diagram, where the gap increases after the selection of a new strategy with a higher utility and decreases until the next strategy switch. However, this pattern stops after the pure Nash equilibrium is reached, which is the unique approximate Nash equilibrium in accordance with Lemma 3.8.

(a) Table of transitions

Nash gap

1.0

0.8

0.6

0.4

0.2

0.0

100

101

102

103

104

105

iterations

(b) Nash Gap

1.0

0.8

0.6

1

2

m

0.4

4

0.2

0.0

100

101

102

103

104

105

iterations

(c) Empirical strategy of x player

5 Conclusion

In summary, this paper has provided a thorough examination of the convergence rate of fictitious play within a specific subset of potential games. Our research has yielded a recursive rule for constructing payoff matrices, demonstrating that fictitious play, regardless of the tie-breaking rule employed, may require super exponential time to reach a Nash equilibrium even in two-player identical payoff games. This contribution to the literature differs from previous studies and sheds new light on the limitations of fictitious play in the particular class of potential games.

Limitations and Broader Impacts: Our work is theoretical in nature, and we do not identify any limitations or negative ethical or societal implications.

Acknowledgments

Ioannis Panageas would like to acknowledge a start-up grant and SF50818. This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021 _ 205011 , by Hasler Foundation Program: Hasler Responsible AI (project number 21043) and Innovation project supported by Innosuisse (contract agreement 100.960 IP-ICT).

References

J. D. Abernethy, K. A. Lai, and A. Wibisono. Fast convergence of fictitious play for diagonal payoff matrices. In D. Marx, editor, Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms, SODA 2021, Virtual Conference, January 10 - 13, 2021 , pages 13871404. SIAM, 2021. doi: 10.1137/1.9781611976465.84. URL https://doi.org/10.1137/1. 9781611976465.84 .

L. Baudin and R. Laraki. Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 1664-1690. PMLR, 2022. URL https://proceedings.mlr.press/v162/ baudin22a.html .

L. Baudin and R. Laraki. Smooth fictitious play in stochastic games with perturbed payoffs and unknown transitions. Advances in Neural Information Processing Systems , 35:20243-20256, 2022.

G. W. Brown. Some notes on computation of games solutions. Technical report, RAND CORP SANTA MONICA CA, 1949.

G. W. Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13(1): 374, 1951.

O. Candogan, A. E. Ozdaglar, and P. A. Parrilo. Dynamics in near-potential games. Games Econ. Behav. , 82:66-90, 2013. doi: 10.1016/j.geb.2013.07.001. URL https://doi.org/10. 1016/j.geb.2013.07.001 .

N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge University Press, 2006. ISBN 978-0-521-84108-5. doi: 10.1017/CBO9780511546921. URL https: //doi.org/10.1017/CBO9780511546921 .

Y. Chen, X. Deng, C. Li, D. Mguni, J. Wang, X. Yan, and Y. Yang. On the convergence of fictitious play: A decomposition approach. In L. D. Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 179-185. ijcai.org, 2022. doi: 10.24963/ijcai.2022/26. URL https://doi.org/10.24963/ijcai.2022/26 .

C. Daskalakis and Q. Pan. A counter-example to karlin's strong conjecture for fictitious play. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014, Philadelphia, PA, USA, October 18-21, 2014 , pages 11-20. IEEE Computer Society, 2014. doi: 10.1109/FOCS.2014.10. URL https://doi.org/10.1109/FOCS.2014.10 .

J. C. Harsanyi. Oddness of the number of equilibrium points: a new proof. International Journal of Game Theory , 2:235-250, 1973.

J. Heinrich, M. Lanctot, and D. Silver. Fictitious self-play in extensive-form games. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 805-813. JMLR.org, 2015. URL http://proceedings.mlr. press/v37/heinrich15.html .

J. Hofbauer and W. H. Sandholm. On the global convergence of stochastic fictitious play. Econometrica , 70(6):2265-2294, 2002.

S. Karlin. Mathematical Methods and Theory in Games, Programming & Economics . AddisonWesley Professional, 1959.

W.-C. Ma, D.-A. Huang, N. Lee, and K. M. Kitani. Forecasting interactive dynamics of pedestrians with fictitious play. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 774-782, 2017.

J. R. Marden, G. Arslan, and J. S. Shamma. Joint strategy fictitious play with inertia for potential games. IEEE Transactions on Automatic Control , 54(2):208-220, 2009.

K. Miyasawa. On the convergence of the learning process in a 2 x 2 non-zero-sum two-person game . Princeton University Princeton, 1961.

D. Monderer and A. Sela. A2 × 2game without the fictitious play property. Games and Economic Behavior , 14(1):144-148, 1996.

D. Monderer and L. S. Shapley. Fictitious play property for games with identical interests. Journal of economic theory , 68(1):258-265, 1996.

D. Monderer and L. S. Shapley. Potential games. Games and economic behavior , 14(1): 124-143, 1996.

P. Muller, S. Omidshafiei, M. Rowland, K. Tuyls, J. Pérolat, S. Liu, D. Hennes, L. Marris, M. Lanctot, E. Hughes, Z. Wang, G. Lever, N. Heess, T. Graepel, and R. Munos. A generalized training approach for multiagent learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=Bkl5kxrKDr .

G. Ostrovski and S. van Strien. Payoff performance of fictitious play. CoRR , abs/1308.4049, 2013. URL http://arxiv.org/abs/1308.4049 .

J. Robinson. An iterative method of solving a game. Annals of Mathematics , 54(2):296-301, 1951. ISSN 0003486X. URL http://www.jstor.org/stable/1969530 .

M. O. Sayin, F. Parise, and A. E. Ozdaglar. Fictitious play in zero-sum stochastic games. SIAM J. Control. Optim. , 60(4):2095-2114, 2022. doi: 10.1137/21m1426675. URL https: //doi.org/10.1137/21m1426675 .

L. Shapley. Some topics in two-person games. Advances in game theory , 52:1-29, 1964.

B. Swenson and S. Kar. On the exponential rate of convergence of fictitious play in potential games. In 55th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2017, Monticello, IL, USA, October 3-6, 2017 , pages 275-279. IEEE, 2017. doi: 10.1109/ALLERTON.2017.8262748. URL https://doi.org/10.1109/ALLERTON.2017. 8262748 .

Y. Yang and J. Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. CoRR , abs/2011.00583, 2020. URL https://arxiv.org/abs/2011.00583 .

Appendix

A Proof of Lemma 3.6

In this section, we present a comprehensive proof of Lemma 3.6. This lemma establishes a fundamental property of our recursive construction, as defined in Definition 3.2: the only ϵ 2 -approximate Nash equilibrium is located at the maximum value element. Consequently, the fictitious play dynamics fail to reach even an approximate Nash equilibrium (within the specified degree of approximation) unless the strategy ( n 2 , n 2 +1 ) has been played for a sufficient duration. Our theoretical findings are further supported by our experiments. As shown in Figure 4b, the Nash gap does not vanish before the final strategy switch occurs. To provide a precise definition of the Nash gap, we present it below.

Definition A.1 (Nash Gap for Identical Payoff) . The Nash gap at round t for a two-player identical payoff games is

To establish Lemma 3.6, we employ a similar approach as in the other proofs presented in this work. Specifically, we heavily rely on the structure of our payoff matrix and utilize an induction technique to demonstrate that the majority of the probability mass must be concentrated in the maximum element. The induction argument starts from the outermost elements of the matrix, namely row n -1 , column 1 , and row 1 , column n -1 . By successive induction steps, we establish that until the maximum element is reached, none of the elements in those rows or columns can possess a significant probability mass.

Lemma A.2 (Proof of Lemma 3.6) . Let ϵ ∈ (0 , 1 / 56 n 3 ] and ( x ⋆ , y ⋆ ) an ϵ 2 -NE. Then for each i ∈ [0 , n/ 2 -2] ,

x ⋆ i +1 ≤ ϵ and x ⋆ n -i ≤ ϵ .

y ⋆ i +1 ≤ ϵ and y ⋆ n -i ≤ ϵ .

Proof. We will prove the claim by induction. First, assume that for all j ∈ [0 , i -1] , we have:

x ⋆ j +1 ≤ ϵ and x ⋆ n -j ≤ ϵ .

y ⋆ j +1 ≤ ϵ and y ⋆ n -j ≤ ϵ .

Next, we proceed to establish the inequalities x ⋆ n -i , y ⋆ i +1 , x ⋆ i +1 , y ⋆ n -i ≤ ϵ . We demonstrate these inequalities in the exact order as presented, as their proof relies on the underlying structure of the matrix. It is important to note that there are interdependencies between these inequalities, which we will address accordingly.

Let assume that x ⋆ n -i > ϵ and we will reach a contradiction. From Observation 3.3, we notice that the utility of row n -i equals

At the same time the utility of row i +1 equals

As a result, by taking the difference on the utilities of row i +1 and n -i we get,

where the last inequality follows by the fact that y ⋆ n -i +1 ≤ ϵ (Inductive Hypothesis). As a result, we conclude that

In case y ⋆ i +1 + y ⋆ n -i ≥ (4 i +1) ϵ then [ Ay ⋆ ] i +1 -[ Ay ⋆ ] n -i ≥ ϵ . Hence if the row player puts x ⋆ n -i probability mass to row i +1 by transferring the probability mass from row n -i to row i +1 then it increases its payoff by x ⋆ n -i ([ Ay ⋆ ] i +1 -[ Ay ⋆ ] n -i ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus, we conclude in the following two statements,

where the last inequality is obtained by combining the first inequality with Equation (4). Now consider the sum of the utilities of rows k ∈ [ i +2 , n -i -1] . By the construction of the payoff matrix A we can easily establish the following claim.

Proposition A.3. The sum of utilities of rows k ∈ [ i +2 , n -i -1] satisfies the inequality,

By Proposition A.3 we are ensured that

where the second inequality follows by the fact that y ⋆ i +1 + y ⋆ n -i ≤ (4 i +1) ϵ and the fact that y ⋆ j ≤ ϵ for all k ∈ [1 , i ] ∪ [ n -i +1 , n ] (Inductive Hypothesis).

Due to the fact that

we are ensured that there exists a row k ⋆ ∈ [ i +2 , n -i -1] with utility [ Ay ⋆ ] k ⋆ ≥ 1 -(5 n +1) ϵ n . Now consider the difference between the utility of row k ⋆ and the row n -i .

where the last inequality holds for ϵ ≤ 1 / 56 n 3 . Hence if the row player puts x ⋆ n -i probability mass to row k ⋆ then it increases its payoff by x ⋆ n -i ([ Ay ⋆ ] k ⋆ -[ Ay ⋆ ] n -i ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we have reached to a final contradiction that x ⋆ n -i > ϵ .

Case 2: y ⋆ i +1 ≤ ϵ

Similar to the previous case, we assume that y ⋆ i +1 > ϵ and proceed to derive a contradiction. From Observation 3.3, we notice that the utility of column i +1 is given by:

At the same time the utility of column n -i equals

As a result, by taking the difference on the utilities of columns n -i and i +1 we get,

where the last inequality follows by the fact x ⋆ n -i ≤ ϵ (Inductive Step Case 1). As a result, we conclude that

In case x ⋆ i +1 + x ⋆ n -i -1 ≥ (4 i +2) ϵ then [( x ⋆ ) ⊤ A ] n -i -[( x ⋆ ) ⊤ A ] i +1 ≥ ϵ . Hence if the column player puts y ⋆ i +1 probability mass to column n -i then it increases its payoff by y ⋆ i +1 ([( x ⋆ ) ⊤ A ] n -i -[( x ⋆ ) ⊤ A ] i +1 ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we conclude in the following two statements:

Now consider the sum of the utilities of columns k ∈ [ i +2 , n -i -1] . By the construction of the payoff matrix A we can easily establish the following claim.

Proposition A.4. The sum of utilities of columns k ∈ [ i +2 , n -i -1] satisfies the inequality,

By Proposition A.4 we are ensured that

where the second to last inequality follows by the facts: x ⋆ i +1 + x ⋆ n -i -1 ≤ (4 i + 2) ϵ and so x ⋆ i +1 ≤ (4 i +2) ϵ , and the Inductive Step Case 1, x ⋆ n -i ≤ ϵ . Moreover, the last inequality holds by the Inductive Hypothesis: x ⋆ k ≤ ϵ for all k ∈ [1 , i ] ∪ [ n -i +1 , n ] .

Due to the fact that

we are ensured that there exists a column k ⋆ ∈ [ i +2 , n -i -1] with utility [( x ⋆ ) ⊤ A ] k ⋆ ≥ 1 -(5 n +3) ϵ n . Now consider the difference between the utility of column k ⋆ and the column i +1 .

where the last inequality follows by the fact that ϵ ≤ 1 / 56 n 3 . Hence if the column player puts y ⋆ i +1 probability mass to column k ⋆ then it increases its payoff by y ⋆ i +1 ([( x ⋆ ) ⊤ A ] k ⋆ -[( x ⋆ ) ⊤ A ] i +1 ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we have reached to a final contradiction that y ⋆ i +1 > ϵ .

Let assume that x ⋆ i +1 > ϵ and we will reach a contradiction. From Observation 3.3, we notice that the utility of row i +1 equals,

Next, we examine the row n -( i +1) in the inner submatrix. If this row is not well-defined, it implies that the inductive step j = i has reached the 2 × 2 submatrix. In this case, the proof of Lemma 3.6 has already been completed. Otherwise, the utility of row n -( i +1) equals

As a result, by taking the difference on the utilities of row i +1 and n -( i +1) we get,

where the last inequality follows by the fact that y ⋆ i +1 ≤ ϵ (Inductive Step Case 2). As a result, we conclude that

In case y ⋆ n -i ≥ (4 i +3) ϵ then [ Ay ⋆ ] n -( i +1) -[ Ay ⋆ ] i +1 ≥ ϵ . Hence, if the row player puts x ⋆ i +1 probability mass to row n -( i +1) then it increases its payoff by x ⋆ i +1 ([ Ay ⋆ ] n -( i +1) -[ Ay ⋆ ] i +1 ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus, we conclude the following two statements:

Now consider the sum of the utilities of rows k ∈ [ i +2 , n -i -1] . From Proposition A.3

Thus, we are ensured that there exists a row k ⋆ ∈ [ i +2 , n -i -1] with utility [ Ay ⋆ ] k ⋆ ≥ 1 -(5 n +3) ϵ n . Now consider the difference between the utility of row k ⋆ and the row n -i .

where the last inequality follows by the fact that ϵ ≤ 1 / 56 n 3 . Hence if the row player puts x ⋆ i +1 probability mass to row k ⋆ then it increases its payoff by x ⋆ i +1 ([ Ay ⋆ ] k ⋆ -[ Ay ⋆ ] i +1 ) > ϵ 2 . The

latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we have reached to a final contradiction that x ⋆ i +1 > ϵ .

Case 4: y ⋆ n -i ≤ ϵ

Let assume that y ⋆ n -i > ϵ and we will reach a contradiction. From Observation 3.3, we notice that the utility of column n -i equals,

Now, we consider the column ( i +1) + 1 in the inner submatrix. In case this column is not welldefined, it means that the inductive step j = i has reached the 2 × 2 submatrix and so the proof of the Lemma 3.6 has already been completed. Otherwise, the utility of column ( i +1) + 1 equals

As a result, by taking the difference on the utilities of columns i +1 and n -( i +1) we get,

where the last inequality follows by the fact that x ⋆ i +1 ≤ ϵ (Inductive Step Case 3). As a result, we conclude that

In case x ⋆ n -i ≥ (4 i +4) ϵ then [( x ⋆ ) ⊤ A ] ( i +1)+1 -[( x ⋆ ) ⊤ A ] n -i ≥ ϵ . Hence if the column player puts y ⋆ n -i probability mass to column ( i +1) + 1 then it increases its payoff by y ⋆ n -i ([( x ⋆ ) ⊤ A ] ( i +1)+1 -[( x ⋆ ) ⊤ A ] n -i ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we conclude in the following two statements:

Now consider the sum of the utilities of columns k ∈ [ i +2 , n -i -1] . From Proposition A.4

Thus, we are ensured that there exists a column k ⋆ ∈ [ i +2 , n -i -1] with utility [( x ⋆ ) ⊤ A ] k ⋆ ≥ 1 -(5 n +4) ϵ n . Now consider the difference between the utility of column k ⋆ and the column n -i .

where the last inequality follows by the fact that ϵ ≤ 1 / 56 n 3 . Hence if the column player puts y ⋆ n -i probability mass to column k ⋆ then it increases its payoff by y ⋆ n -i ( [( x ⋆ ) ⊤ A ] k ⋆ -[( x ⋆ ) ⊤ A ] n -i ) > ϵ 2 . The latter contradicts with the assumption that ( x ⋆ , y ⋆ ) is an ϵ 2 -NE. Thus we have reached to a final contradiction that y ⋆ n -i > ϵ .

A.1 Proof of Theorem A.3

Proposition A.5. The sum of utilities of rows k ∈ [ i +2 , n -i -1] admits,

Proof. By Observation 3.3 we derive the following equation.

The claim can be immediately derived from the inequality given above, n -i -1 ∑ k = i +2 [ Ay ⋆ ] k ≥ n -i -1 ∑ k = i +2 y ⋆ k .

A.2 Proof of Theorem A.4

Proposition A.6. The sum of utilities of columns k ∈ [ i +2 , n -i -1] admits,

Proof. By Observation 3.3 we derive the following equation.

The claim can be immediately derived from the inequality given above, .

B Proof of Lemma 3.8

B.1 Omitted Proofs of Section 3.3

̸

Proposition B.1 (Proof of Proposition 3.10) . Let ( i ( t ) , j ( t ) ) be a strategy selected by fictitious play at round t , and ( i ( t ) , j ( t ) ) = ( n 2 , n 2 ) . Then, in a subsequent round, fictitious play will choose the strategy of greater value that is either on row i ( t ) or column j ( t ) .

Proof. To establish the claim, we employ the concept of a cumulative utility vector, as defined in Definition 2.7. According to Proposition 3.5, the row i ( t ) and column j ( t ) combined have three distinct non-zero elements. Without loss of generality, let's assume that the greater element is in column j ( t ) , but in a different row, denoted as i ′ .

Firstly, we observe that any subsequent strategy will involve only those three elements. This is because in the cumulative utility vector, which determines the strategy to be played in each round, only the coordinates corresponding to those elements are updated as long as the strategy ( i ( t ) , j ( t ) ) is being played.

Moreover, we can demonstrate that among these elements, the one with the greater value will be played next, and this transition is deterministic. This means that the row player will choose the strategy associated with the greater element. We note that this decision is implicitly affected by the strategy of column player.

Let's first exclude the case where the next strategy switch involves the column player. We initially assumed that the greatest element is in column j ( t ) but on a different row. Consequently, the only non-zero element in a different column than j ( t ) must have a smaller utility compared to the element ( i ( t ) , j ( t ) ) . Therefore, there is no incentive to switch to a strategy with lower utility. This confirms that the column player will not opt for a different strategy, ensuring that the next strategy switch, if it occurs, will necessarily involve the row player.

Regarding the row player, we aim to prove that there will be a round where they will change to a different strategy, and consequently, to a different row. To analyze this, let's examine how the cumulative utility vector of the row player changes from round to round.

Therefore, as long as the column player continues to use the same strategy, the row Ae j ( t ) will repeatedly be added to the cumulative vector, reinforcing the coordinate of row i ′ . However, this cannot happen indefinitely, as the cumulative utility vector is bounded. After a certain number of rounds, the row player will eventually choose the strategy associated with row i ′ . This proves that claim for the case of the row player.

Similarly, if the greater element is located in the same row but on a different column, a similar argument can be made to prove that the column player will switch strategies next.

Corollary B.2 (Proof of Corollary 3.11) . Let t be a round in which a player changes their strategy. Then exactly one of the following statements is true:

̸

If the row player changes their strategy at round t , i.e. i ( t ) = i ( t -1) , then the column player can only make the next strategy switch.

̸

If the column player changes their strategy at round t , i.e. j ( t ) = j ( t -1) , then the row player can only make the next strategy switch.

̸

Proof. This corollary is a simple application of Proposition 3.10. We will only prove the first claim. Let t be the round when the row player changes their strategy, i.e., i ( t ) = i ( t -1) . According to Proposition 3.5, there are three non-zero elements combined in i ( t ) , j ( t ) . Since the row player changes their strategy, it follows from Proposition 3.10 that the other element in column j ( t ) but not in row i ( t ) must necessarily have a smaller value than ( i ( t ) , j ( t ) ) . Therefore, the element with the greater value must necessarily be in a different column. Hence, by applying Proposition 3.10, we conclude that the column player can only make the next strategy switch. This proves the claim.

B.2 Auxiliary Propositions for Lemma 3.6

In this subsection, we provide the full version of the proposition used to establish the proof Lemma 3.6.

Proposition B.3. There exists a round T 1 i > T 0 i at which

(i) the strategy profile is ( i +1 , i +1) for the first time,

(ii) for all rounds t ∈ [ T 0 i , T 1 i -1] , the strategy profile is ( n -i, i +1) ,

(iii) column i +1 admits cumulative utility C ( T 1 i ) i +1 ≥ (4 i +1) · ( R ( T 0 i ) i +1 +1) ,

(iv) all rows k ∈ [( i +1) + 1 , n -i -1] admit R ( T 1 i ) k = 0 and all columns k ∈ [ i +2 , n -i ] admit C ( T 1 i ) k = 0 .

Proof. The proposition is composed of multiple parts, each of which is proven separately. To begin with, we must establish that the new strategy profile chosen by fictitious play will be ( i +1 , i +1) .

According to the inductive hypothesis in Theorem 3.1, we know that the strategy ( n -i, i +1) was played at round T 0 i for the first time, while any strategy involving a row ∈ [ i +1 , n -( i +1)] has not been played until that point. Additionally, the inductive hypothesis also states that the strategy played before time T 0 i was ( n -i, n -i +1) . Therefore, according to Corollary 3.11, it is guaranteed that the next strategy switch will be initiated by the row player.

In other words, as long as the column player continues to play their current strategy, strategy ( n -i, i +1) will be played in every subsequent round, which establishes Item (ii). This implies that the row player's cumulative utility will increase by Ae i +1 . As per Observation 3.3, column i +1 only has non-zero elements in positions i +1 and n -i .

Since strategy i +1 has a higher value, a strategy switch in a later time step is certain. Consequently, there will be a round T 1 i in which the strategy ( i +1 , i +1) will be played for the first time, establishing Item (i).

We must determine the point at which the strategy switch to ( i +1 , i +1) will take place. According to Equation (5), the value of strategy i +1 is exactly one greater than the value of strategy n -i . From the inductive hypothesis, we know that row i +1 has a cumulative utility of zero, whereas row n -i has a cumulative utility of R ( T 0 i ) i +1 . Therefore, it will take precisely R ( T 0 i ) i +1 steps for those strategies to have equal cumulative utility values. Consequently, the strategy switch will either occur in that round or the immediate next, as the cumulative utility of row i +1 will have surpassed that of row n -i .

In order to proceed with Item (iii), we compute the updated cumulative utility of column ( i +1) . As shown in Equation (5), column i +1 has a value of (4 i +1) at position n -i . Thus, if row n -i has been played for a minimum of R ( T 0 i ) i +1 rounds, then the cumulative utility of column i +1 is greater than (4 i +1) · R ( T 0 i ) i +1 . Given that row n -i has also been played previously (inductive hypothesis), it is reasonable to conclude that:

Lastly, according to Observation 3.4, there exists a non-zero element in row i +1 at position i +1 . Due to the fact that column i +1 has already been played (according to Item (ii)), we can conclude that the cumulative utility of row i +1 must be non-zero. Combining this with inductive hypothesis, it concludes the proof of Item (iv).

B.2.1 Proof of Theorem 3.13

Proposition B.4. There exists a round T 2 i > T 1 i at which

(i) the strategy profile is ( i +1 , n -i ) for the first time,

(ii) for all rounds t ∈ [ T 1 i , T 2 i -1] , the strategy profile is ( i +1 , i +1) ,

(iii) row i +1 admits cumulative utility R ( T 2 i ) i +1 ≥ (4 i +2) · C ( T 1 i ) i +1 ,

(iv) all rows k ∈ [( i +1)+1 , n -i -1] admit R ( T 2 i ) k = 0 and all columns k ∈ [ i +2 , n -( i +1)] admit C ( T 2 i ) k = 0 .

Proof. We repeat the same reasoning as in the proof of Proposition 3.12. To begin with, we must establish that the new strategy profile chosen by fictitious play will be ( i +1 , n -i ) .

We can infer from both Items (i) and (ii) in Proposition 3.12 that the most recent strategy switch was made by the row player. Therefore, based on Corollary 3.11, we are certain that the next strategy switch will be initiated by the column player.

In other words, as long as the row player continues to play their current strategy, strategy ( i +1 , i +1) will be played in every subsequent round, which establishes Item (ii). This implies that the column player's cumulative utility will increase by e ⊤ i +1 A . As per Observation 3.3, row i + 1 only has non-zero elements at positions i +1 and n -i .

Since strategy n -i has a higher value, a strategy switch in a later time step is certain. Consequently, there will be a round T 2 i in which the strategy ( i +1 , n -i ) will be played for the first time, establishing Item (i).

We must determine the point at which the strategy switch to ( i +1 , n -i ) will take place. According to Equation (6), the value of strategy n -i is exactly one greater than the value of strategy i +1 . From the Proposition 3.12, we know that column n -i has a cumulative utility of zero, whereas column i +1 has a cumulative utility of C ( T 1 i ) i +1 . Therefore, it will take precisely C ( T 1 i ) i +1 steps for those strategies to have equal cumulative utility values. Consequently, the strategy switch will either occur in that round or the immediate next, as the cumulative utility of column n -i will have surpassed that of column i +1 .

In order to proceed with Item (iii), we compute the updated cumulative utility of row ( i +1) . As shown in Equation (6), row i +1 has a value of (4 i +2) at position i +1 . Thus, if column i +1 has been played for a minimum of C ( T 1 i ) i +1 rounds, then the cumulative utility of row i +1 satisfies R ( T 2 i ) i +1 ≥ (4 i +2) · C ( T 1 i ) i +1 .

Lastly, according to Observation 3.4, there exists a non-zero element in column n -i at position i +1 . Due to the fact that row i +1 has already been played (according to Item (ii)), we can conclude that the cumulative utility of column n -i must be non-zero. Combining this with Proposition 3.12, it concludes the proof of Item (iv).

B.2.2 Proof of Theorem 3.14

Proposition B.5. There exists a round T 3 i > T 2 i at which

(i) the strategy profile is ( n -( i +1) , n -i ) for the first time,

(ii) for all rounds t ∈ [ T 2 i , T 3 i -1] , the strategy profile is ( i +1 , n -i ) ,

(iii) column n -i admits cumulative utility C ( T 3 i ) n -i ≥ (4 i +3) · R ( T 2 i ) i +1 ,

(iv) all rows k ∈ [( i + 1) + 1 , n -( i + 1) -1] admit R ( T 3 i ) k = 0 and all columns k ∈ [ i +2 , n -( i +1)] admit C ( T 3 i ) k = 0 .

Proof. We repeat the same reasoning as in the proof of Proposition 3.12. To begin with, we must establish that the new strategy profile chosen by fictitious play will be ( n -i -1 , n -i ) .

We can infer from both Items (i) and (ii) in Proposition 3.13 that the most recent strategy switch was made by the column player. Therefore, based on Corollary 3.11, we are certain that the next strategy switch will be initiated by the row player.

In other words, as long as the column player continues to play their current strategy, strategy ( i +1 , n -i ) will be played in every subsequent round, which establishes Item (ii). This implies that the row player's cumulative utility will increase by Ae n -i . As per Observation 3.3, column n -i only has non-zero elements at positions i +1 and n -( i +1) .

Since strategy n -( i + 1) has a higher value, a strategy switch in a later time step is certain. Consequently, there will be a round T 3 i in which the strategy ( n -i -1 , n -i ) will be played for the first time, establishing Item (i).

We must determine the point at which the strategy switch to ( n -i -1 , n -i ) will take place. According to Equation (7), the value of strategy n -( i +1) is exactly one greater than the value of strategy i +1 . From the Proposition 3.13, we know that row n -( i +1) has a cumulative utility of zero, whereas row i +1 has a cumulative utility of R ( T 2 i ) i +1 . Therefore, it will take precisely R ( T 2 i ) i +1 steps for those strategies to have equal cumulative utility values. Consequently, the strategy switch will either occur in that round or the immediate next, as the cumulative utility of row n -( i +1) will have surpassed that of row i +1 .

In order to proceed with Item (iii), we compute the updated cumulative utility of column n -i . As shown in Equation (7), column n -i has a value of (4 i +3) at position i +1 . Thus, if row i +1 has been played for a minimum of R ( T 2 i ) i +1 rounds, then the cumulative utility of column n -i satisfies C ( T 3 i ) ≥ (4 i +3) · R ( T 2 i ) .

Lastly, according to Observation 3.4, there exists a non-zero element in row n -( i +1) at position n -i . Combining this with the Proposition 3.13 and the fact that column n -i has already been played (according to Item (ii)), we can conclude that the cumulative utility of row n -( i +1) must be non-zero. This concludes the proof of Item (iv).

B.2.3 Proof of Theorem 3.15

Proposition B.6. There exists a round T 4 i > T 3 i at which

(i) the strategy profile is ( n -( i +1) , ( i +1) + 1) for the first time,

(ii) for all rounds t ∈ [ T 3 i , T 4 i -1] , the strategy profile is ( n -( i +1) , n -i ) ,

(iii) row n -i -1 admits cumulative utility R ( T 4 i ) n -( i +1) ≥ (4 i +4) · C ( T 3 i ) n -i ,

(iv) all rows k ∈ [( i + 1) + 1 , n -( i + 1) -1] admit R ( T 4 i ) k = 0 and all columns k ∈ [( i +1) + 2 , n -( i +1)] admit C ( T 4 i ) k = 0 .

Proof. We repeat the same reasoning as in the proof of Proposition 3.12. To begin with, we must establish that the new strategy profile chosen by fictitious play will be ( n -( i +1) , ( i +1) + 1) .

We can infer from both Items (i) and (ii) in Proposition 3.14 that the most recent strategy switch was made by the row player. Therefore, based on Corollary 3.11, we are certain that the next strategy switch will be initiated by the column player.

In other words, as long as the row player continues to play their current strategy, strategy ( n -( i + 1) , n -i ) will be played in every subsequent round, which establishes Item (ii). This implies that the column player's cumulative utility will increase by e ⊤ n -( i +1) A . As per Observation 3.3, row n -( i +1) only has non-zero elements at positions ( i +1) + 1 and n -( i +1) .

Since strategy ( i + 1) + 1 has a higher value, a strategy switch in a later time step is certain. Consequently, there will be a round T 4 i in which the strategy ( n -( i +1) , ( i +1) + 1) will be played for the first time, establishing Item (i).

We must determine the point at which the strategy switch ( n -( i +1) , ( i +1) + 1) will take place. According to Equation (8), the value of strategy ( i +1) + 1 is exactly one greater than the value of strategy n -i . From the Proposition 3.14, we know that column ( i +1) + 1 has a cumulative utility of zero, whereas column n -i has a cumulative utility of C ( T 3 i ) n -i . Therefore, it will take precisely

C ( T 3 i ) n -i steps for those strategies to have equal cumulative utility values. Consequently, the strategy switch will either occur in that round or the immediate next, as the cumulative utility of column ( n -( i +1) , ( i +1) + 1) will have surpassed that of column n -i .

In order to proceed with Item (iii), we compute the updated cumulative utility of row n -( i +1) . As shown in Equation (8), row n -( i +1) has a value of (4 i +4) at position n -i . Thus, if column n -i has been played for a minimum of C ( T 3 i ) n -i rounds, then the cumulative utility of row n -( i +1) satisfies R ( T 4 i ) n -( i +1) ≥ (4 i +4) · C ( T 3 i ) n -i .

Lastly, according to Observation 3.4, there exists a non-zero element in column ( i +1)+1 at position n -( i +1) . Combining this with the Proposition 3.14 and the fact that row n -( i +1) has already been played (according to Item (ii)), we can conclude that the cumulative utility of column ( i +1)+1 must be non-zero. This concludes the proof of Item (iv).

Peer reviews:
comments: SUMMARY:
This paper studies fictitious play in potential games and in particular, two-player identical payoff games. It is shown that fictitious play, under arbitrary tie-breaking, needs super-exponential time with respect to the number of actions to find an approximate Nash equilibrium. The lower bound is proved through recursive construction of a hard game in which fictitious play spirals slowly and takes super exponential time to reach the unique pure Nash equilibrium.

STRENGTHS:
- This paper makes significant progress on a fundamental problem, namely the convergent behavior of fictitious play in potential games. While Monderer et al. proved an asymptotic $O(1/t)$ rate, no finite-time guarantee is known before to my knowledge; this paper shows that any finite-time guarantee for classes of games that include identical-payoff games has to be super-exponential in the number of actions. It suggests that fictitious play in potential games, while asymptotically convergent, would have a burn-in period that is super-exponentially long.

- The paper is remarkably well written. The results are presented clearly and the construction is clean and easy to follow.

WEAKNESSES:
- The asymptotic part of the lower bound translates to $\Omega(1/(n^2t^2))$, which I believe is not tight in light of the $O(1/t)$ upper bound.

QUESTIONS:
A couple of minor questions:

- Line 221: In Eq (2), should the last term be $R^{T_{l-1}}_{n-l+1}$ instead?

- Line 247: To evoke Lemma 3.6, wouldn't one need $n\sqrt{\epsilon}<1$? I can't seem to find where this is assumed.

- Line 248: "On the other hand, at each round $t\ge T^*$, ..., with rate $1/t$". Where is result statement proven?

LIMITATIONS:
This work is of theoretical nature and there is no foreseeable negative ethical, societal implication.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the convergence rate of Fictitious Play (FP) in potential games. The paper proves that FP for potential games can take exponential time to reach a Nash equilibrium. The research has yielded a recursive rule for constructing payoff matrices, demonstrating that fictitious play, regardless of the tie-breaking rule employed, may require exponential time to reach a Nash equilibrium even in two-player identical payoff games. The same theorem holds for general N-player potential games.\n\nSTRENGTHS:\nThey have provided a concrete example to illustrate the exponential lower bound with figures, which greatly enhances the clarity and understanding of the paper. Furthermore, I am satisfied with the overall readability of the paper. The writing style is approachable, making it easier for readers to engage with the content. One aspect that I particularly enjoyed is the authors' explanation of intuition to construct counterexamples of learning dynamics. This approach helps readers grasp the underlying concepts more effectively, enhancing the overall learning experience. The proof idea is simple and intuitive.\nThe paper is well-written, provides clear explanations through intuitive examples, and offers an enjoyable reading experience.\n\nWEAKNESSES:\nI have some concerns regarding this paper that I would like the authors to address.\n\n1. In reference [1], it is mentioned that if certain regularity conditions are met, FP dynamics achieve an exponential rate. Can the authors provide more details on these regularity conditions and their implications? Like why this game is not in the regularity conditions. [1] are saying that almost every potential game are regular game. \n\n2. For FP dynamics, this paper proves that finding NE using fictitious play potential games has exponentially lower bounds. Does this exponential lower bound also hold for stochastic fictitious play? For example, if people have some exploration in each turn,  is it possible to break the exponential lower bound? \n\n3. I am curious about the upper-bound of the convergence rate in the case where the conditions of regularity are not satisfied. I guess that the example that this paper gave does not satisfy the regularity condition. Is there a specific upper-bound that can be established?\n\nQUESTIONS:\nI want to clarify the issues of the weakness section. If this is well-addressed, I am down to re-evaluate this paper.\n\nLIMITATIONS:\nThe authors adequately addressed the limitation and potential negative societal impact on their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper constructs a common-payoff two-player game for which fictitious play must take a number of iterations exponential in the number of actions to converge to an $\epsilon$-equilibrium.  The paper closes with some empirical demonstrations of the behavior of fictitious play on the game.

STRENGTHS:
The construction of $A$ is very clever, and the description of why this guarantees an exponential lower bound does an excellent job of delivering the important aspects without getting bogged down in the arithmetic details.

Fictitious play is an extremely well-studied and widely-used algorithm.  The structural simplicity of the constructed example gives an excellent intuition about a class of scenarios in which we should expect FP to perform poorly.

WEAKNESSES:
My only concern about this work is that it might have limited impact; a link to the structural properties of games that FP is commonly applied to might have helped.

QUESTIONS:
1. What is $z$ for in the construction of the game?  Once $K_n(z)$ has been defined, we only ever think about $A=K_n(0)$ for the rest of the paper.


#### Minor issues (did not affect my evaluation)
- on p.3, $x^\top A(y') - x^\top Ay = \Phi(x,y') - \Phi(x,y)$ should be $x^\top B(y') - x^\top By = \Phi(x,y') - \Phi(x,y)$
- on p.7, "Applying Lemma 3.6 for $\epsilon := \sqrt\epsilon$...": This is needlessly confusing

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

comments: SUMMARY:
This paper studies the convergence rate of Fictitious Play (FP) in potential games. While prior work shows that FP asymptotically converges to a NE in the case of $N$-player potential games, the current paper proves that FP can take exponential time to do so. Specifically, the paper recursively constructs a two-player coordination game with identical interests and a unique pure NE. The paper shows that FP requires super-exponential time before placing positive probability in the unique NE, even with an arbitrary tie-breaking rule. The paper also contains simulations to validate the shown learning dynamics.

STRENGTHS:
1. The paper provides a complete answer to an important problem (convergence rate of FP in potential games). 
2. The paper is very well-written and easy to follow. Related works are discussed extensively. Technical results are delivered in a clear way. 
3. The paper also provides simulations to validate findings and provide intuition.

WEAKNESSES:
1. It might be better to include more background information on FP and potential games, e.g., some concrete examples of potential games in practice.

QUESTIONS:
Line 58 - 60: I believe [1] is only for diagonal games?

Line 100 - 101: The latter A should be B.

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08