Title: Team-Fictitious Play for Reaching Team-Nash Equilibrium in Multi-team Games
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-agent reinforcement learning

fictitious play

multi-team games
Abstract: Multi-team games, prevalent in robotics and resource management, involve team members striving for a joint best response against other teams. Team-Nash equilibrium (TNE) predicts the outcomes of such coordinated interactions. However, can teams of self-interested agents reach TNE? We introduce Team-Fictitious Play (Team-FP), a new variant of fictitious play where agents respond to the last actions of team members and the beliefs formed about other teams with some inertia in action updates. This design is essential in team coordination beyond the classical fictitious play dynamics. We focus on zero-sum potential team games (ZSPTGs) where teams can interact pairwise while the team members do not necessarily have identical payoffs. We show that Team-FP reaches near TNE in ZSPTGs with a quantifiable error bound. We extend Team-FP dynamics to multi-team Markov games for model-based and model-free cases. The convergence analysis tackles the challenge of non-stationarity induced by evolving opponent strategies based on the optimal coupling lemma and stochastic differential inclusion approximation methods. Our work strengthens the foundation for using TNE to predict the behavior of decentralized teams and offers a practical rule for team learning in multi-team environments. We provide extensive simulations of Team-FP dynamics and compare its performance with other widely studied dynamics such as smooth fictitious play and multiplicative weights update. We further explore how different parameters impact the speed of convergence.

Team-Fictitious Play for Reaching Team-Nash Equilibrium in Multi-team Games

Abstract

Multi-team games, prevalent in robotics and resource management, involve team members striving for a joint best response against other teams. Team-Nash equilibrium (TNE) predicts the outcomes of such coordinated interactions. However, can teams of self-interested agents reach TNE? We introduce Team-Fictitious Play (Team-FP), a new variant of fictitious play where agents respond to the last actions of team members and the beliefs formed about other teams with some inertia in action updates. This design is essential in team coordination beyond the classical fictitious play dynamics. We focus on zero-sum potential team games (ZSPTGs) where teams can interact pairwise while the team members do not necessarily have identical payoffs. We show that Team-FP reaches near TNE in ZSPTGs with a quantifiable error bound. We extend Team-FP dynamics to multi-team Markov games for model-based and model-free cases. The convergence analysis tackles the challenge of non-stationarity induced by evolving opponent strategies based on the optimal coupling lemma and stochastic differential inclusion approximation methods. Our work strengthens the foundation for using TNE to predict the behavior of decentralized teams and offers a practical rule for team learning in multi-team environments. We provide extensive simulations of Team-FP dynamics and compare its performance with other widely studied dynamics such as smooth fictitious play and multiplicative weights update. We further explore how different parameters impact the speed of convergence.

1 Introduction

Multi-team games are increasingly common, e.g., in robotics and resource management [Silva and Chaimowicz, 2017, Vinyals et al., 2019, Jaderberg et al., 2019]. Unlike non-cooperative multi-agent settings, team members strive for a joint best response against other teams as if the entire team is a single decision-maker. Team-Nash equilibrium (TNE), where team members coordinate in the best team response against other teams, can capture this to predict the outcome of coordinated team interactions [Farina et al., 2018, Zhang et al., 2021]. Game-theoretical equilibrium is often justified by its emergence from non-equilibrium adaptation of self-interested learners (e.g., see [Fudenberg and Levine, 2009]). However, the question of whether the teams of self-interested agents can reach TNE in multi-team games remains largely unexplored. This paper investigates this very question.

For example, TNE generally can arise if the team members can learn to correlate their actions in the best team response independent of the opponent. However, the widely studied fictitious play (FP) dynamics do not necessarily reach the best team outcome even when there are no opponents, e.g., in

potential games. We present a slight adjustment of FP, called Team-FP , provably reaching TNE in multi-team competition even with networked interconnections, where agents' payoffs depend only on the neighbors' actions. Similar to the FP, here, agents respond greedily to the beliefs formed about the opponent teams' joint play based on past observations. Different from the FP, Team-FP incorporates the key features: ( i ) response to the last actions of team members, and ( ii ) inertia in action updates. These features, inspired by log-linear learning dynamics [Blume, 1993], play a crucial role in driving team coordination towards TNE. Notably, Team-FP reduces to the smoothed FP [Fudenberg and Kreps, 1993] (or log-linear learning) when each team has a single agent (or there is a single team).

Multi-team competition spans diverse domains. For example, robotics, resource management, online gaming, and financial markets [Kitano et al., 1997, Cardenas et al., 2009, Silva and Chaimowicz, 2017, Vinyals et al., 2019, Jaderberg et al., 2019] involve multi-team competition. To model such interactions, we consider multiple teams with possibly different number of team members. These team members have networked interconnections, as depicted in Figure 1. We focus on multi-team zero-sum potential team games where teams have pairwise interactions (ZSPTGs). For any opponent team strategy, team members effectively play an underlying potential game, as in distributed optimization applications, e.g., see [Arslan et al., 2007, Xu et al., 2012, Zheng et al., 2014]. Notably, ZSPTGs reduce to zero-sum polymatrix games [Cai et al., 2016] if each team has a single agent, and to potential games [Monderer and Shapley, 1996a] if there is a single team. Additionally, widely studied two-team zero-sum games, e.g., see [Farina et al., 2018, Zhang et al., 2021, Carminati et al., 2022, Kalogiannis et al., 2022], are a special case of ZSPTGs.

We show that the Team-FP dynamics reach near TNE in ZSPTGs. This means that the empirical average of team actions converge to the near best response each team can take against the average actions of other teams. We quantify the approximation error, showing it decreases with the level of exploration in the agents' responses. Similar to the FP dynamics, Team-FP is also rational : teams can learn (near) optimal strategies if opponent teams play stationary strategies. These results strengthen the applicability of TNE for predicting team behavior in multi-team competition and provide a practical rule for teams of self-interested agents to learn coordination in multi-team settings.

A key challenge in our analysis is handling the non-stationary nature of learning, as opponent teams' strategies change over time. We address this by leveraging the optimal coupling lemma (e.g. see [Levin and Peres, 2017, Chapter 4]) and stochastic differential inclusion approximation methods (e.g., see [Benaïm et al., 2005, Perkins and Leslie, 2013]) to the repeated play of games. Motivated from the recent interest in multi-agent reinforcement learning, we can extend Team-FP dynamics to finite horizon multi-team Markov games for both model-based and model-free cases. We discuss this extension and analyze its convergence numerically in Appendix C.

Related works. FP and its variants offer convergence guarantees in important classes of games [Fudenberg and Levine, 2009], yet not in every class of games [Hart and Mas-Colell, 2003]. For example, they reach equilibrium in potential games [Monderer and Shapley, 1996a], but not necessarily the most efficient one for the team. Log-linear learning can achieve efficient equilibrium for the team [Marden and Shamma, 2012, Tatarenko, 2017]. However, it is not clear whether such dynamics can track efficient equilibrium in dynamic environments (induced by the evolving strategies of opponent teams). Notably, Tatarenko [2018] and Donmez et al. [2024] addressed, resp., efficient learning under non-stationarity induced by the decaying exploration in agents' responses for the repeated play of potential games and non-stationarity induced by evolving stage games in Markov team problems

(also known as identical-interest Markov games). These approaches are orthogonal to our analysis to extend our results to the exact TNE convergence in repeated multi-team games or to learning in infinite horizon Markov games.

FP and its variants can reach equilibrium in two-agent zero-sum games [Hofbauer and Sandholm, 2002] yet not necessarily in multi-agent zero-sum games with more than two agents. We can transform any general-sum game to a multi-agent zero-sum game by introducing a non-effective auxiliary agent (with a single action). There have been several attempts to address zero-sum games beyond two-agent cases [Bergman and Fokin, 1998, Cai and Daskalakis, 2011, Cai et al., 2016]. For example, Ewerhart and Valkanova [2020] addressed the convergence of continuous and discrete-time FP in zero-sum network games, where each agent plays multiple two-agent games with separate actions, and the overall utilities sum to zero. Notably, Cai et al. [2016] introduced zero-sum polymatrix games where agents have network separable pairwise interactions with applications in security. Recently, FP has been shown to reach Nash equilibrium in zero-sum polymatrix games [Park et al., 2023]. Following the same trend, we focus on learning in ZSPTGs extending two-team zero-sum games to multi-team games with pairwise team interactions. However, we highlight that two-team zero-sum or ZSPTGs are not necessarily multi-agent zero-sum polymatrix games (as the agent payoffs do not necessarily sum to zero) and the classical FP dynamics do not necessarily converge TNE or Nash equilibrium in such multi-team games.

Recent studies on two-team zero-sum games and adversarial team games (e.g., see [Celli and Gatti, 2018, Farina et al., 2018, Zhang et al., 2022, Carminati et al., 2022]) have primarily focused on the efficient computation of team-minimax equilibrium. In particular, Celli and Gatti [2018] examines the efficiency of different communication types, highlighting the promising results of ex ante communication, referring to pre-play communication among team members. Consequently, the studies of Farina et al. [2018], Zhang et al. [2022], Carminati et al. [2022] often model teams as a single agent with imperfect recall, incorporating ex ante communication within the team. Notably, Farina et al. [2018] introduced the Fictitious Team Play (FTP) algorithm for extensive-form two-team zero-sum games with imperfect information, where team members communicate ex ante. In this approach, Farina et al. [2018] used fictitious play (FP) on a simplified version of the original game, embedded within the game tree. This method essentially applied FTP to the original adversarial team game. To find the best response, they used mixed-integer linear programming. Team-FP differs from such approaches by letting agents learn to team up while following their self-interest based on simple behavioral rules, as in the log-linear learning and FP dynamics.

The rest of the paper is organized as follows. We describe ZSPTGs in §2. We present the (independent) Team-FP dynamics in §3. We provide analytical and numerical results, resp., in §4 and §5. We conclude the paper with some remarks in §6. Appendices include the proofs of the technical results, some further numerical experiments and the extension to multi-team Markov games.

Notation: Given a finite set A , we let ∆( A ) denote the probability simplex over A . We let f ( µ ) = E a ∼ µ [ f ( a )] for any probability distribution µ ∈ ∆( A ) and any functional f : A → R . Furthermore, we define the smoothed best response to any functional q : A → R by

for some temperature parameter τ > 0 , where H ( µ ) := E a ∼ µ [ -log( µ ( a ))] for all µ ∈ ∆( A ) is the entropy regularization.

2 Game Formulation

Consider a multi-team game , characterized by the tuple 〈T , ( A i , u i ) i ∈I 〉 , where T and I denote, resp., the teams' and agents' index sets, and A i and u i : A → R (with A := ∏ j ∈I A j ) denote, resp., the agent i 's finite action set and payoff function. Agents take their actions to maximize their payoff functions.

Definition 2.1 (Zero-sum Potential Team Game) . Let I m denote the index set of agents in team m and A m := ∏ i ∈I m A i denote the set of joint actions for team m . We say that a multi-team game is zero-sum potential team game (ZSPTG) if for each team m ∈ T , there exists a potential function

φ m : A → R such that

glyph[negationslash]

for all (ˆ a i , a ) ∈ A i × A and i ∈ I m , where a -i := { a j } j ∈I m \{ i } are the actions of other team members, a -m := { a glyph[lscript] } glyph[lscript] = m are the action profiles of other teams, where a glyph[lscript] ∈ A glyph[lscript] is team glyph[lscript] 's action profile. The potential functions sum to zero, i.e., we have

Furthermore, the actions have network separable interactions across teams such that we can separate the potential functions and correspondingly payoff functions as glyph[negationslash]

glyph[negationslash]

for some φ mglyph[lscript] : A m × A glyph[lscript] → R and u iglyph[lscript] : A m × A glyph[lscript] → R .

The following example generalizes the potential game formulation for distributed optimization (e.g., see [Arslan et al., 2007, Xu et al., 2012, Zheng et al., 2014]) to two-team zero-sum potential games. Example 2.2 . Consider two teams of agents interacting over a network. We can represent their interactions via a graph G = ( V, E ) , where the set of vertices V refers to the agents and the set of (undirected) edges refers to their interactions. Let agent i from team m receive a local payoff r i : A i × ∏ j :( i,j ) ∈ E A j → R depending on the actions of the neighboring agents only. Agent i adds the neighboring team members' local payoffs whereas subtracts the other neighbors' local payoffs in his/her total payoff. In other words, his/her total payoff is given by

This yields that the team m has the potential function

and therefore, these potential functions sum to zero. However, the potential function is not generally the sum of the payoffs in potential games.

Remark 2.3 (General-sum ZSPTGs) . In ZSPTGs, the underlying game can be a general-sum game although the team-potentials sum to zero, as described in (3). For example, consider two competing teams whose team members have identical payoffs corresponding to their team potentials. If the teams have different number of members, then the agents' payoffs do not sum to zero or a constant while the team potentials do so.

In the following, we introduce TNE, generalizing team-minimax equilibrium for two-team zero-sum games, e.g., see [Von Stengel and Koller, 1997], to multi-team games. Particularly, at TNE, no team has an incentive to change their team strategy.

Definition 2.4 (Team-Nash Gap) . Given the strategy profile of teams { π m ∈ ∆( A m ) } m ∈T , we define the team-Nash gap for team m as

glyph[negationslash]

and TNG( π ) := ∑ m ∈T TNG m ( π ) , where π -m := { π glyph[lscript] } glyph[lscript] = m . Correspondingly, we say that the strategy profile of teams { π m } m ∈T is glyph[epsilon1] -TNE if TNG( π ) ≤ glyph[epsilon1] .

3 Team-FP Dynamics

We first present the Team-FP dynamics combining the log-linear learning and fictitious play for learning in multi-team games played repeatedly, and then extend Team-FP to multi-team MGs in Appendix C.

Algorithm 1 (Independent) Team-FP

glyph[negationslash]

glyph[negationslash]

Let a i k ∈ A i denote the agent i 's action at the k th repetition in the repeated play of the underlying ZSPTG. Correspondingly, a m k = ( a i k ) i ∈ T m denote the team m 's action profile. Observing the joint actions of team m , agents j / ∈ I m can form a belief about the team m 's joint strategy. Let π m k ∈ ∆( A m ) denote the belief they formed. Consider actions as pure strategy where the associated action gets played with probability 1 . Then, agents j / ∈ I m can update their beliefs about team m 's strategy according to

such that the belief π m k +1 also corresponds to the (weighted) empirical average of the past action profiles { a m 0 , . . . , a m k } .

glyph[negationslash]

Agent i ∈ I m either takes the previous action a i k -1 (i.e., a i k = a i k -1 ), or takes the action a i k ∼ br τ ( u i ( · , a -i k -1 , π -m k )) according to the smoothed best response (as described in (1)) to the previous actions of team members a -i k -1 := { a j k -1 } j ∈I m \{ i } and the beliefs π -m k := { π glyph[lscript] k } glyph[lscript] = m formed about other teams. It is instructive to note that the definition of potential function φ m ( · ) , as described in (2), yields that

We introduce Team-FP and independent Team-FP dynamics depending on how agents update their actions. In the former, a single agent can get chosen randomly, as in the classical log-linear learning. In the latter, each agent can update his/her action with probability δ ∈ (0 , 1) independent of others, as in the independent log-linear learning. The latter addresses the coordination burden in the update of actions within teams. Algorithm 1 provides descriptions of these dynamics for the typical agent i from team m .

Remark 3.1 (Scalability) . Agents can have networked interactions such that their payoff functions depend only on the actions of certain agents such as one/two-hop neighbors. For such cases, agents can form beliefs about these neighbors' strategies only, as if these neighbors play according to some stationary strategy. For example, assume that the payoff of agent i / ∈ I m (outside team m ) depends only on the actions of teamm members from some neighborhood N i , i.e., { j : j ∈ N i ∩ I m } . Agent i can form a belief about these agents' strategies according to

where a im k = { a j k } j ∈N i ∩I m . Then, the linearity of the update rule yields that the local empirical average π im k corresponds to the marginalization of π m k such that

Therefore, local observations (within neighborhoods) would still be sufficient to follow Algorithm 1. We demonstrate the scalability of Team-FP by a large-scale experiment in Appendix D, Figure 7.

We focus on the homogeneous cases where agents i / ∈ I m have a common belief about team m 's strategy. Homogeneous beliefs are possible if the agents have common initial beliefs and step sizes.

We moved the extension of (Independent) Team-FP to multi-team Markov games and its numerical analysis to Appendix C.

4 Convergence Results

Team-FP and Independent Team-FP dynamics reduce, resp., to the classical log-linear learning and independent log-linear learning dynamics if there is only one team. These log-linear learning

dynamics are known to reach team-optimal solution in potential games. For example, consider an n -agent potential game 〈 ( A i , u i ) i ∈ [ n ] 〉 with the potential function φ ( · ) . In both dynamics, the action profiles played form irreducible and aperiodic Markov chains. Let ̂ µ and ̂ µ denote the unique stationary distributions, resp., for the classical and independent versions. For the former, we have ̂ µ = br τ ( φ ( · )) [Marden and Shamma, 2012, Section 3] and (1) yields that

On the other hand, the smaller δ > 0 implies closer stationary distributions in the classical and independent versions. Particularly, we have ̂

for some function Λ( · ) decaying to zero as δ → 0 + for any glyph[epsilon1] φ > 0 , and 0 < glyph[epsilon1] φ ≤ br τ ( φ ( · , a -i )) for any a -i and i is a lower bound on local actions get played in the smoothed best response [Donmez et al., 2024, Lemma 5.6].

Team-FP dynamics have similar convergence guarantees for multi-team games under the following assumption on the step sizes used.

Assumption 4.1. The step size α k ∈ [0 , 1] satisfies the stochastic approximation conditions: α k → 0 as k →∞ , ∑ ∞ k =0 α k = ∞ , and ∑ ∞ k =0 α 2 k < ∞ . Additionally, we have lim k →∞ α k /α k +1 = 1 , and α k -α k +1 ≥ α k α k +1 .

The last condition in Assumption 4.1 ensures that recent observations have comparable weight in the beliefs formed. The classical choice α k = 1 / ( k +1) , leading to the empirical averages of the actions played, satisfies Assumption 4.1.

The following theorem shows the convergence of Algorithm 1 to near TNE almost surely with the approximation levels (similar to (12) and (13)) inherited from the (independent) log-linear learning dynamics.

Theorem 4.2. Given a ZSPTG characterized by 〈T , ( A i , u i ) i ∈I 〉 , let every agent follow either Team-FP or Independent Team-FP, as described in Algorithm 1. If Assumption 4.1 holds, then the team-Nash gap for π k := ( π m k ) m ∈T satisfies

almost surely, where φ := max ( m,l,a ) | φ ml ( a ) | .

The key challenge in our convergence analysis is the non-stationarity arising from opponent teams adapting their strategies while the team members learn to coordinate against them. We address this by constructing a reference scenario where the team members' beliefs about opponent strategies are frozen over finite-length epochs, allowing them to hypothetically "team up" under these fixed beliefs. By comparing the dynamics in the actual scenario and the reference scenario, and exploiting the averaging nature of belief formation, we can bound the approximation error between the two. The main proof concept, along with the Team-FP algorithm for a two-team scenario, is illustrated in Figure 2. This approach is similar to the one used in [Donmez et al., 2024] to handle non-stationarity in Markov team problems. However, unlike [Donmez et al., 2024], we cannot directly show the error bound decay due to the lack of a contraction property in our dynamics.

To overcome this limitation, we view Team-FP as smoothed fictitious play dynamics in zero-sum polymatrix games with an additive bounded error term. The additive error captures the fact that team members may not perfectly achieve team coordination. We then relax the problem by considering any approximation error within the formulated bounds, rather than the actual error. To address this relaxation, we leverage stochastic differential inclusion approximation methods [Benaïm et al., 2005, Perkins and Leslie, 2013]. Finally, by constructing a suitable Lyapunov function addressing arbitrary bounded errors in continuous-time smoothed best response dynamics, we establish the convergence of the discrete-time Team-FP updates.

The following corollary to the main result shows the rationality of the (independent) Team-FP dynamics.

Update

Transton

Kernel

Transton

Kernel

Update

Update

Update

Update

Update

Team 1

Team 2

...

...

3T

2T

T

0

4T

Epsode

Corollary 4.3. Given a ZSPTG characterized by 〈T , ( A i , u i ) i ∈I 〉 , let agents from team m follow either Team-FP or Independent Team-FP while other teams play according to some stationary strategy π -m . If Assumption 4.1 holds, then empirical average of the action profiles played by team m satisfy

Theorem 4.2 can be generalized to the case where the rewards are random with bounded support, rather straightforwardly. Therefore, the proof of Corollary 4.3 follows from Theorem 4.2 if we view the underlying game as there is a single team receiving random rewards with bounded support.

5 Illustrative Examples

In this section, we present various simulation results demonstrating the coordination speed of TeamFP and compare it to pure FP, no-regret algorithms, and a stationary opponent. We also observe the effect of parameters on the convergence speed. In addition, we examine the long-run behavior of team-FP in games beyond ZSPTG games, where we intuitively expect it to converge. All simulations are averaged over 10 independent trials to reduce the randomness. In all figures, the mean is plotted with a thick colorful line, while one standard deviation below and above the mean is shown with a shaded area of the same color. Also, for all simulations, temperature parameter τ is chosen to be 0.1 unless another option is mentioned. We conduct simulations for ZSPTG with two different setups: one with three teams, each consisting of three agents, and another with two teams, each consisting of four agents. Unless explicitly stated otherwise, the default setting consists of two teams, with four agents in each team. The step size is chosen to be α k = 1 / ( k +1) for all simulations.

All the simulations are executed on a computer equipped with an Intel Xeon W7-3455 CPU and 128 GB RAM. Run-time for 10 independent trials over 10 6 iterations vary between 1-5 hours depending on the experiment.

Achieving Implicit Coordination in Team-FP In this section, we compare the performance of Team-FP to the explicit coordination of team members. We also compare Team-FP to algorithms such as Multiplicative Weights Update (MWU) and Smoothed FP (SFP), and show that these algorithms fail to achieve team coordination. We also show that Team-FP achieves near-optimum policy against a stationary opponent as stated in Corollary 4.3.

10 1

10 2

10 3

10 4

10 5

10 6

Number of Iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Team Nash Gap

Team-FP and Coordination

No coordination

Half coordination

Full coordination

(a) Varying group sizes of 1, 2, and 4.

10 1

10 2

10 3

10 4

10 5

10 6

Number of Iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Team Nash Gap

Team-FP vs. MWU and SFP

Independent Team-FP

Team-FP

MWU

SFP

(b) Algorithm 1 (Independent) compared to SFP, and MWU

10

10

10

10

10

10 6

Number of Iterations

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Team Nash Gap

Effect of Competition

Stationary Opponent

Competing Opponent

(c) Competitive vs Stationary

Environment

Competing Opponent

Stationary Opponent

Groups: 1

Groups: 2

Groups: 4

Team-FP

Independent Team-FP

MWU

Smoothed FP

Impact of Team Size in Team Coordination: In Team-FP self-interested team members act together without explicit communication in a coordinated way to reach TNE. We can measure how this independent cooperation compares to the explicit coordination of team members. For that, we propose an example game with two teams and four agents in each team. For the first scenario, four agents act separately and use Team-FP. In the second scenario, the agents form groups of two and act in a coordinated way as if they are a single agent, using Team-FP. In the final scenario, all agents in a team behave as if they were a single agent, equivalent to the standard fictitious play (FP) dynamics in a two-person zero-sum game. This case mimics the ex-ante communication scheme from [Farina et al., 2018]. The scenarios are described by group sizes. Group sizes are one, two, and four respectively for these scenarios. The simulation results are presented in Figure 3a. As expected, the explicit coordination of all members in a team converges the fastest, followed by the explicit coordination of groups of two, and finally, Team-FP converges slowly as the agents do not coordinate explicitly.

Team-FP Compared to MWU and SFP: The strong side of Team-FP is, even though the agents do not communicate, the average of joint actions of teams can reach TNE, unlike other algorithms. We compare the equilibrium behavior of team-FP with a well-known no-regret learning algorithm Multiplicative Weights Update (MWU), in which the average strategies converge to NE in zero-sum polymatrix games [Cai et al., 2016]. We also compare Team-FP with the usual SFP, where each agent holds beliefs about other agents and uses the smoothed best response against them. In Figure 3b, we see that both Team-FP and Independent Team-FP dynamics converge to near TNE, while the other algorithms fail to do so.

Rationality Against Stationary Opponent : In this part, we use 3-team setting where each team has 3 agents. We let a team using Team-FP compete against two other stationary teams and compared it with the performance of the same team in the same game when the opponents are also using Team-FP competitively. In Figure 3c, we observe that both algorithms converge, while the convergence is much faster against stationary opponents.

Team-FP in Application In this part, we provide an example to demonstrate that Team-FP has applications in various contexts.

Security Game Example : We model an airport security scenario as a two-team game between defender and attacker teams. In our example, a security chief on the defender team faces three independent intruders on the attacker team, as shown in Figure 4a. The chief can defend a gate at a cost, while intruders decide whether to attack a gate or remain idle. Intruders gain or lose payoffs based on whether they attack undefended or defended gates, and the chief's payoffs mirror these outcomes. We conducted multiple trials and presented the evolution of the average Team-Nash Gap with standard deviations on the right side of Figure 4b. From a higher level, this example shows that team-minimax equilibrium can predict the outcome of games with different uncoordinated attackers. It also justifies the algorithms developed to compute team-minimax equilibrium efficiently.

106

(a)

(b)

AirportDefence Simulation

1.0

80

Team Nash Gap

0.6

0.4

0.2

0.0

101

102

103

104

105

Number of Iterations

(a) Simulation network for a 3 vs 3 vs 3 game.

10 1

10 2

10 3

10 4

10 5

10 6

Number of Iterations

0.0

0.5

1.0

1.5

2.0

2.5

Team Nash Gap

Team-FP Parameter

τ

= 0

.

1

τ

=0

.

2

τ

=0

.

15

(b) Varying

τ

in Team-FP

10 1

10 2

10 3

10 4

10 5

10 6

Number of Iterations

0.0

0.5

1.0

1.5

2.0

2.5

Team Nash Gap

τ

=0

.

1

,

δ

=0

.

2

τ

= 0

.

1

τ

=0

.

1

,

δ

=0

.

5

(c) Varying

δ

in Independent

Team-FP

Team-FP Parameter 6

T = 0.1

t = 0.1, 6 = 0.2

= 0.1, 5 = 0.5

1

9

2

5

3

8

T = 0.1

T = 0.15

T = 0.2

Effect of parameters in Team-FP In this part, we examine how Team-FP performs for different τ and δ values. Given an example ZSPTG of three teams where each team has three agents in Figure 6a, we examine the evolution of TNG in the Team-FP dynamics for different values of τ ∈ { 0 . 1 , 0 . 15 , 0 . 2 } . We also compare the evolution of NG in the Team-FP and Independent TeamFP dynamics for τ = 0 . 1 , δ = 0 . 2 , δ = 0 . 5 . All simulation results (see Figure 5) show convergence, and we observe lower final values of NG ( π ) for smaller τ as we predicted (see. Figure 5b). The Independent Team-FP requires more iterations when δ = 0 . 2 for its convergence, while it is much faster when δ = 0 . 5 (see. Figure 5c). This is expected as updates are much more frequent as δ increases. However, increasing δ too much may harm the coordination behavior of the team.

Beyond ZSPTG In this part, we try several other games for Team-FP without proof of convergence. We expect Team-FP to converge in 2xN and potential games other than zero-sum games as FP converges in these settings. For the first case, we try a game where one team has a single agent with only two actions with random rewards, while the other team has three agents with an underlying potential function. In this case, Team-FP converges (see. Figure 6b). In another setting, we try two teams of four agents. However, the potential functions for both teams are identical rather than summing to zero, resulting in a potential function that encompasses the individual potential functions. The team-FP algorithm converges to TNE in this case as well (see. Figure 6c). However, the equilibrium is not unique in this case. Finally, we create an extension of the team-FP for Markov games (see. Appendix C) in an RL framework and try simulations on this setting. In this case, there

10 1

10 2

10 3

10 4

10 5

Number of Iterations

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Team Nash Gap

Model Free

(a) Markov Games

10 1

10 2

10 3

10 4

10 5

10 6

Number of Iterations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Team Nash Gap

General Sum

10

10

10

10

10

10 6

Number of Iterations

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Team Nash Gap

Team-FP in Potential Games

(c) Potential of Potentials Game

Model Free

2 × N game

Potential Game

are two teams each having two agents, competing against each other in a finite horizon Markov game with a horizon length of ten. The number of states is two, and the state transition matrices are generated randomly. We observe that team-Nash Gap for MG's defined in (65), converges to near-zero.

6 Conclusion

In this paper, we introduced Team-FP, a novel fictitious play variant designed for multi-team games. We showed that Team-FP provably achieves near-TNE in ZSPTGs, with a quantifiable error bound. Our convergence analysis addressed the non-stationarity challenge arising from evolving opponent team strategies, by leveraging the optimal coupling lemma and stochastic differential inclusion approximation methods. We also extended Team-FP to multi-team Markov games, encompassing both model-based and model-free scenarios, with applications in multi-team reinforcement learning. Furthermore, we conducted detailed numerical analysis of the Team-FP dynamics, focusing on the trade-off in learning to team up and competition in comparison to the classical FP and no-regret learning dynamics. We further examined the convergence of Team-FP dynamics to TNE in multi-team games beyond ZSPTGs. These results strengthen the theoretical foundations for applying TNE to predict decentralized team behavior and provide a framework for team learning in multi-team settings.

Limitations and Broader Impacts Our work quantified the almost sure convergence of the TeamFP dynamics asymptotically yet did not provide guarantees for the convergence rate. We conducted detailed numerical examples and presented the evolution of the gap in TNE to exemplify this rate qualitatively. The challenge for the non-asymptotic analysis is inherited from the discrete-time FP dynamics for which there are only rough rate analysis [Karlin, 1959] and negative examples for some edge cases [Brandt et al., 2010, Daskalakis and Pan, 2014].

Our work introduces no new ethical concerns in multi-team systems and shares the assumption of stationary opponents with learning dynamics like Q-learning and fictitious play. This assumption does not disadvantage our approach. We argue that treating uncoordinated attackers as a single decision-maker is necessary, as they can learn to bypass security measures. Our paper provides a theoretical basis for this, ensuring more reliable AI-based solutions.

Future Research Directions This work paves the way to further explore the behavior of decentralized teams in multi-team interactions when the team members follow different types of dynamics for teaming up within teams (other than log-linear learning) and adapting to other teams' play (other than FP). Numerical examples we conducted for multi-team games beyond ZSPTGs are also promising to show the provable convergence of Team-FP dynamics in multi-team (Markov) games reducing to games with the fictitious play property (e.g., see [Monderer and Shapley, 1996b]) if the teams coordinate in acting as a single player.

References

Arslan, M. F. Demirkol, and Y. Song. Equilibrium efficiency improvement in MIMO interference systems: A decentralized stream control approach. IEEE Transactions on Wireless Communications , 6(8):2984-2993, 2007.

Benaïm, J. Hofbauer, and S. Sorin. Stochastic approximations and differential inclusions. SIAM J. Control Optim , 44(1):328-348, 2005.

M. Bergman and I. N. Fokin. On separable non-cooperative zero-sum games. Optimization , 44(1): 69-84, 1998.

E. Blume. The statistical mechanics of strategic interaction. Games Econom. Behav. , 5(3):387-424, 1993.

Brandt, F. Fischer, and P. Harrenstein. On the rate of convergence of fictitious play. In Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010. Proceedings 3 , pages 102-113. Springer, 2010.

Cai and C. Daskalakis. On minmax theorems for multiplayer games. In Proceedings of the 22nd Annual ACM-SIAM Symposium on Discrete Algorithms , pages 217-234. SIAM, 2011.

Cai, O. Candogan, C. Daskalakis, and C. Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Math. Oper. Res , 41(2):648-655, 2016.

Cardenas, S. Amin, B. Sinopoli, A. Giani, A. Perrig, S. Sastry, et al. Challenges for securing cyber physical systems. In Workshop on Future Directions in Cyber-physical Systems Security , volume 5, page 7, 2009.

Carminati, F. Cacciamani, M. Ciccone, and N. Gatti. A marriage between adversarial team games and 2-player games: Enabling abstractions, no-regret learning, and subgame solving. In Int. Conf. Machine Learn. (ICML) , pages 2638-2657. PMLR, 2022.

Celli and N. Gatti. Computational results for extensive-form adversarial team games. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Daskalakis and Q. Pan. A counter-example to Karlin's strong conjecture for fictitious play. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 11-20. IEEE, 2014.

S. Donmez, O. Unlu, and M. O. Sayin. Logit-Q dynamics for efficient learning in stochastic teams. arXiv preprint arXiv:2302.09806 , 2024.

Ewerhart and K. Valkanova. Fictitious play in networks. Games Econom. Behav. , 123:182-206, 2020.

Farina, A. Celli, N. Gatti, and T. Sandholm. Ex ante coordination and collusion in zero-sum multi-player extensive-form games. Advances in Neural Inform. Process. (NeurIPS) , 31, 2018.

Fudenberg and D. M. Kreps. Learning mixed equilibria. Games Econom. Behav. , 5(3):320-367, 1993.

Fudenberg and D. K. Levine. Learning and equilibrium. Annual Rev. Econ. , 1(1):385-420, 2009.

Hart and A. Mas-Colell. Uncoupled dynamics do not lead to Nash equilibrium. The American Econ. Rev. , 83(5):1830-1836, 2003.

Hofbauer and W. H. Sandholm. On the global convergence of stochastic fictitious play. Econometrica , 70(6):2265-2294, 2002.

Hu and M. P. Wellman. Nash Q-learning for general-sum stochastic games. J. Machine Learn. Res. , pages 1039-1069, 2003.

Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Kalogiannis, I. Panageas, and E-V. Vlatakis-Gkaragkounis. Towards convergence to nash equilibria in two-team zero-sum games. In Internat. Conf. Learning Representations (ICLR) , 2022.

Karlin. Mathematical methods and theory in games. Programming, and Economics , 1,2, 1959.

Kitano, M. Asada, Y. Kuniyoshi, I. Noda, E. Osawa, and H. Matsubara. Robocup: A challenge problem for AI. AI magazine , 18(1):73-73, 1997.

A. Levin and Y. Peres. Markov Chains and Mixing Times . American Mathematical Society, 2nd edition, 2017.

L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Int. Conf. Machine Learn. (ICML) , 1994.

R. Marden and J. S. Shamma. Revisiting log-linear learning: Asynchrony, completeness and payoff-based implementation. Games Econom. Behav. , 75(2):788-808, 2012.

Monderer and L. S. Shapley. Potential games. Games Econom. Behav. , 14(1):124-143, 1996a.

Monderer and L. S. Shapley. Fictitious play property for games with identical interests. Journal of economic theory , 68(1):258-265, 1996b.

Park, K. Zhang, and A. Ozdaglar. Multi-player zero-sum Markov games with networked separable interactions. In Advances in Neural Inform. Process. (NeurIPS) , pages 37354-37369, 2023.

Perkins and D. S. Leslie. Asynchronous stochastic approximation with differential inclusions. Stochastic Systems , 2(2):409-446, 2013.

S. Shapley. Stochastic games. Proc. Natl. Acad. Sci. USA , 39(10):1095-1100, 1953.

N. Silva and L. Chaimowicz. MOBA: A new arena for game AI. arXiv preprint arXiv:1705.10443 , 2017.

Tatarenko. Game-theoretic Learning and Distributed Optimization in Memoryless Multi-agent Systems . Springer, 2017.

Tatarenko. Independent log-linear learning in potential games with continuous actions. IEEE Trans. on Control of Network Systems , 5(3):913-923, 2018.

Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Von Stengel and D. Koller. Team-maxmin equilibria. Games Econom. Behav. , 21(1-2):309-321, 1997.

Xu, J. Wang, Q. Wu, A. Anpalagan, and Y. D. Yao. Opportunistic spectrum access in cognitive radio networks: Global optimization using local interaction games. IEEE Journal of Selected Topics in Signal Processing , 6(2):180-194, 2012.

Zhang, L. Carminati, F. Cacciamani, G. Farina, P. Olivieri, N. Gatti, and T. Sandholm. Subgame solving in adversarial team games. Advances in Neural Inform. Process. (NeurIPS) , 35:2668626697, 2022.

Zhang, Z. Yang, H. Liu, T. Zhang, and T. Ba¸ sar. Finite-sample analysis of decentralized batch multiagent reinforcement learning with networked agents. IEEE Trans. on Automatic Control , 66 (12):5925-5940, 2021.

Zheng, Y. Cai, Y . Liu, Y . Xu, B. Duan, and X. Shen. Optimal power allocation and user scheduling in multicell networks: Base station cooperation using a game-theoretic approach. IEEE Transactions on Wireless Communications , 13(12):6928-6942, 2014.

A Proof of Theorem 4.2

We can separate the proof into two main steps: ( i ) showing that team members can learn to team up approximately by constructing a reference scenario where beliefs got frozen, and ( ii ) addressing the bounded approximation error by leveraging the stochastic differential inclusion approximations.

A.1 Step ( i ) - Reference Scenario and Error Analysis

We divide the horizon into T -length epochs. Then, we can write the belief update (8) accumulated from k = nT to ( n +1) T as

for any n = 0 , 1 , . . . denoting the epoch index. Let π m ( n ) := π m nT denote the belief about team m in epoch n . Furthermore, for notational simplicity, we define

Then, we can simplify (16) as

Due to 4.1, the step size β ( n ) decays to zero monotonically at a certain rate such that [Donmez et al., 2024, Lemma 5.4]

Let F ( n ) be a filtration generated by the σ -algebra σ ( A 0 , . . . , A nT -1 ) , where A t denotes the joint actions taken by each team at stage t , i.e., A t = ( a 1 t , . . . , a |T | t ) . It is instructive to note that π m ( n ) is F ( n ) -measurable. We define the joint action distributions of team m at time k in epoch n by

Then, we can write (18) as in the form of stochastic approximation

where ω m ( n +1) is a Martingale difference sequence defined by

Now, let's consider a reference scenario for the analysis in which the beliefs (denoted by ̂ π t m ) are only updated at the ends of T -length epochs. In other words, we have ̂ π t m = π m ( n ) for all nT ≤ t ≤ ( n +1) T -1 and m ∈ T . Due to the fixed beliefs about the opponent plays, Team-FP dynamics reduce to the log-linear learning in the reference scenario. Let ̂ a m ( n ) ,k denote the joint action of team m in the reference scenario. By the nature of the log-linear learning, { ̂ a m ( n ) ,k } ∞ k = nT form a homogeneous Markov chain (MC) even though the actual action profiles { a m k } ∞ k = nT do not necessarily do so. Denote the stationary distribution of the MC in the reference scenario by ̂ µ m ( n ) ,glyph[star] and ̂ µ m ( n ) ,glyph[star] for Team-FP and Independent Team-FP, respectively. The former is given by

̂

µ m ( n ) ,glyph[star] = br τ ( φ m ( · , π -m ( n ) )) due to the log-linear learning update [Blume, 1993, Marden and Shamma, 2012]. Then, we can write the belief update (21) for team m as

̂

where we decompose the error e m ( n ) := ̂ e m ( n ) + e m ( n ) as

The update (23) corresponds to the SFP dynamics of teams acting as a single decision-maker with the additive error e m ( n ) .

The following lemma, based on designing the optimal coupling between the actual and reference scenarios as in [Donmez et al., 2024, Lemma 5.5], plays an important role in bounding ‖ ̂ e m ( n ) ‖ from above.

Lemma A.1. For some constants c, d, ρ ≥ 0 , we have

By (24), Lemma A.1 yields that we can bound the error ̂ e m ( n ) from above by

Due to the no-recency-bias condition in 4.1, we have β k +1 /β k ≤ 1 by the definition (17). Then, we have monotonically decaying β k , which yields

By the assumption 4.1, we have

By (27), (28), (29), and the decaying property of α k , we obtain

where

Note that C ( ε, T ) can become arbitrarily close to Λ( δ, glyph[epsilon1] ) for sufficiently large T and small ε , which are chosen arbitrarily just for the analysis.

̂

On the other hand, e m ( n ) is non-zero only for Independent Team-FP and corresponds to the difference between the stationary distributions for the classical and independent log-linear learning. We can bound ̂ e m ( n ) ≤ Λ( δ, glyph[epsilon1] ) for some glyph[epsilon1] > 0 based on (13). Hence, given any ε > 0 , there exists N ε such that

A.2 Step ( ii ) - Convergence Analysis with Relaxed Errors

We focus on the convergence analysis of (23) based on the bound (31). To this end, we define the set-valued mapping

for all π ∈ Π := ∏ m ∈T ∆( A m ) . Then, the update (23) and (31) yield that for sufficiently large n , the empirical averages { π ( n ) } n ≥ 0 satisfy

where the Martingale difference sequence { ω ( n +1) } is as described in (22). We have set inclusion in (34) different from the equality in (23) since we relax the update rule by considering any error satisfying the bound (31), rather than the actual error.

The following proposition shows that F ( · ) is a peculiar set-valued mapping. Particularly, given the sets X,Y from some underlying Euclidean space, we say that a set-valued function F ( · ) mapping each point x ∈ X to a set F ( x ) ⊂ Y is a Marchaud map , e.g., see [Perkins and Leslie, 2013, Definition 2.1] and [Benaïm et al., 2005, Hypothesis 1.1], if it satisfies the following conditions:

(i) F ( · ) is upper semi-continuous, or equivalently, Graph( F ) = { ( x, y ) : y ∈ F ( x ) } is a closed subset of X × Y .

(ii) For all x ∈ X , the set F ( x ) is a non-empty, compact, and convex subset of Y .

(iii) There exists a c > 0 such that sup y ∈ F ( x ) ‖ y ‖ ≤ c (1 + ‖ x ‖ ) for all x ∈ X .

Proposition A.2. The set-valued function F ( · ) is a Marchaud map.

Next, we approximate the relaxation (34) by the differential inclusion

Particularly, due to Proposition A.2 and (19), a linear interpolation of the iterative process { π ( n ) } n ≥ 0 is a perturbed solution to the differential inclusion (35) [Benaïm et al., 2005, Proposition 1.4] and its limit set is internally chain transitive [Benaïm et al., 2005, Theorem 3.6]. Furthermore, we can characterize internally chain transitive sets (and therefore, the limit set of linear interpolations) through a Lyapunov function. Particularly, let Φ t ( π ) be the set of solutions to (35) with initial point π . We say that a continuous function V : Π → R is a Lyapunov function of (35) for a subset Λ ⊂ Π provided that

V (˜ π ) < V ( π ) for all π ∈ Π \ Λ , ˜ π ∈ Φ t ( π ) , t > 0 ,

V (˜ π ) ≤ V ( π ) for all π ∈ Λ , ˜ π ∈ Φ t ( π ) , t ≥ 0 .

Given such a Lyapunov function for Λ , every chain internally chain transitive set L is contained in Λ if the set { V ( π ) : π ∈ Λ } has empty interior [Benaïm et al., 2005, Proposition 3.27]. Therfore, we propose the candidate Lyapunov function

where the auxiliary terms are defined by

for some arbitrary λ > 1 to characterize the long-run behavior of (34) based on (35).

Proposition A.3. The continuous function V ( · ) is a Lyapunov function of (35) for the set Λ = { π : V ( π ) = 0 } .

Proposition A.3 yields that there exists some sequence { ζ k ∈ R } k ≥ 0 such that | ζ k |→ 0 as k →∞ almost surely and

By (3), we can write L ( π k ) as

Since max µ { φ m ( µ, π -m ) } -φ m ( π ) ≥ 0 for all π , the inequalities (38) and (39) yield that

Recall that we can select λ > 1 , ε > 0 and T ∈ N arbitrarily. The definitions (32) and (37b) yield that Ξ( λ, ε, T ) can be arbitrarily close to τ log | A | for Team-FP and τ log | A | + |T | 2 φ · Λ( δ, glyph[epsilon1] ) for Independent Team-FP. Furthermore, ζ k decays to zero. Therefore, we can obtain (14) based on (40) and Definition 2.4. This completes the proof.

B Proofs of Technical Results

The followings are the proofs of Lemma A.1, and Propositions A.2 and A.3 used in Appendix A.

B.1 Proof of Lemma A.1

We define two discrete-time processes from the beginning of epoch n . One of them is the joint actions of teams in the reference scenario, defined as { ̂ ω k } k ≥ nT := { ( ̂ a m k ) m ∈T |F ( n ) } k ≥ nT , and the other one is the joint actions of teams in the actual scenario, with { ω k } k ≥ nT := { ( a m k ) m ∈T |F ( n ) } k ≥ nT .

The transition probabilities between states depend only on the beliefs on the other teams. Therefore, for the reference scenario, during an epoch, this process is a homogeneous MC. Let, P ( n ) ,k be the transition probabilities between the states, i.e., joint action profiles of all teams, of the original discrete-time process at step k, and let ̂ P ( n ) be the transition probabilities between the states for the reference scenario. Let ̂ µ ( n ) ,k be the distribution of ̂ ω k , and let µ ( n ) ,k be the distribution of ω k . In this case, the expected joint actions of a team at step k within an epoch in real and fictional scenarios can be expressed in terms of the distributions as:

Step 1. In classical log-linear learning, at each step, only one agent can change their action and due to the soft-max nature of this action change, any action has positive probability which is bounded from below. This bound only depends on the minimum and maximum values of any agents' reward, which are defined by the game and bounded by definition, and the temperature parameter. If we divide that bound by the number of agents in the team ( |I m | ), we can obtain a lower bound on the probability of changing to any joint action which can be reached within a single step. Let's call that bound ξ . Then, for any state ω or ̂ ω , there is a ξ > 0 such that

glyph[negationslash]

Then, we can find a path with positive probability for both of these scenarios, where their state does not match until they reach κ = max m |I m | . In other words, ω nT + κ = ̂ ω nT + κ and ω k = ̂ ω k for all k = nT,. . . , nT + κ -1 with

Hence, (44) satisfies the first condition for [Donmez et al., 2024, Lemma 2].

Step 2. For this part, we introduce a notation a jm ∈ A jm , where jm represents the j th agent from team m . For example, if all teams have identical agent numbers and agent indexes are ordered for teams, jm = ( m -1) | T m | + j . Also, let π ( n ) ,k := ( π m ( n ) ,k ) m ∈T , and ̂ π ( n ) ,k := ( ̂ π m ( n ) ,k ) m ∈T . Now, consider the total variation distance between two transition probabilities. Transition probabilities between state ω = { ( a im , a -im ) } m ∈T and ˜ ω = { ( ˜ a im , a -im ) } m ∈T , where im can be any random agent from team m with a slight abuse of notation, can be written as a function of the belief as

where P m ω → ˜ ω is defined to be

glyph[negationslash]

and π -m ( n ) ,k := ( π glyph[lscript] ( n ) ,k ) glyph[lscript] = m . Remember that due to the separable structure of the network in ZSPTG we have glyph[negationslash]

glyph[negationslash]

where Φ mglyph[lscript] is the matrix form of the potential function whose rows are the joint actions of team m and columns are the joint actions of team glyph[lscript] .

Now, for all ω k +1 , which is reachable in one step from the state ω k , i.e., at most a single agent changes action in each team, the relation between state transition probabilities and P m ω → ˜ ω ( · ) can be expressed as

Note that π ( n ) ,k is a function of history of actions and the initial π ( n ) . Therefore, it can be computed given the past states ( ω k , . . . , ω nT ) .

We know that P m ω → ˜ ω ( π ( n ) ,k ) is bounded as it is a probability. Now, using the Lipschitz property of the soft-max function, and since φ m ( · , π -m ( n ) ,k ) is a linear function of π ( n ) ,k , we can say that P m ω k → ( · ) ( π ) is a Lipschitz continuous function with respect to the input π . Also, using the fact that the product of bounded Lipschitz continuous functions is also Lipschitz continuous, and (48), we can say that there exists an L < ∞ such that the total variation distance between two transition probabilities is bounded as

The distance between the belief of the original scenario and the reference scenario can also be bounded thanks to the small step size. If we bound ‖ a glyph[lscript] k -π glyph[lscript] k ‖ 1 < ‖ a glyph[lscript] k ‖ 1 + ‖ π glyph[lscript] k ‖ 1 = 2 . Then using triangle inequality

Let's consider late epochs where α nT < 1 2 T L|T | , and set 2 T L|T | α nT = 1 -λ nT with 0 < λ nT ≤ 1 . Then,

Hence, the second condition of [Donmez et al., 2024, Lemma 2] is met, and we can invoke the corresponding Lemma such that given the distributions of the original and reference scenarios, following inequality holds for all k ≥ nT ,

where ε = ξ 2 κ . If we define constants, c := 2 (1 -ε ) 1 k -1 , ρ := (1 -ε ) 1 κ , d := 4 1 + ε ε , and assume that the reference scenario initial distribution is the stationary distribution of the MC, ̂ µ ( n ) ,glyph[star] , we can rewrite the inequality (52) as follows

for all k ≥ nT . Then, by (41), and triangle inequality, we can conclude

for all k ≥ nT .

B.2 Proof of Proposition A.2

The set Π is compact set by definition as it is a Cartesian product of probability simplexes. Let's consider a convergent sequence ( π n , µ n -π n + e n ) n =1 , 2 ,... in the set { ( π n , y ): y ∈ F ( π ) } , and let ( π glyph[star] , µ glyph[star] -π glyph[star] + e glyph[star] ) be the point that the sequence converges to. Given π n , any µ n is a fixed and unique value, and it is an element of the compact set that is generated by mapping probability simplex with the continuous soft-max function. Then, for any π glyph[star] ∈ Π , µ glyph[star] -π glyph[star] is a fixed value within another compact set. Furthermore, the error term must remain within the compact set e glyph[star] ∈ e . As a result, ( π glyph[star] , µ glyph[star] -π glyph[star] + e glyph[star] ) is also within the set { ( π n , y ): y ∈ F ( π ) } , and F : Π → A ∑ m ∈T ‖ A m ‖ is a closed-set valued map. Therefore, the condition (i) is satisfied. Given a π ∈ Π , µ glyph[star] ∈ Π is a fixed value corresponding to the smoothed best responses to π -m { m ∈T } . Hence, µ glyph[star] -π is a fixed value for a given π . Note that each e m ∈ e is a non-empty, bounded, closed and convex subset of R ∑ m ∈T ‖ A m ‖ . Therefore, for any given π ∈ Π , F ( π ) = µ glyph[star] -π + e is a non-empty, compact, convex subset of R ∑ m ∈T ‖ A m ‖ . As a result, (ii) is also satisfied. The function F is bounded such that

Hence, it satisfies the condition (iii). Since all three conditions are satisfied, F is a Marchaud Map.

B.3 Proof of Proposition A.3

The smoothness of the entropy regularization in (1) yields that we can invoke the envelope theorem to compute the time derivative of L ( π ) as

glyph[negationslash]

where µ m glyph[star] := br τ ( φ m ( · , π -m )) and ( a ) follows from (4). By (35) and (33), we have ˙ π glyph[lscript] = µ glyph[lscript] glyph[star] -π glyph[lscript] + e glyph[lscript] . Therefore, we can write (57) as glyph[negationslash]

For the first two terms on the right-hand side, we have glyph[negationslash]

where ( a ) is due to (3) and (4), and ( b ) follows from the definition (37a) as H ( µ m glyph[star] ) ≤ log | A m | . On the other hand, we have glyph[negationslash]

glyph[negationslash]

due to the bound on the errors. By the bounds (60) and (61), we can bound the time derivative of L ( π ) as

where the constant Ξ( λ, ε, T ) > 0 is as described in (37b). Since we have V ( π ) = min(0 , L ( π ) -Ξ( λ, ε, T )) , the strict inequality in (62), yields that V ( · ) is a Lyapunov function.

C Extension to Multi-team Markov Games

Markov games (MGs), introduced by Shapley [1953], generalizes Markov decision processes (MDPs) to non-cooperative multi-agent environments. We can characterize a multi-team MG by the tuple 〈 H, T , S, ( A i , r i ) i ∈I , p, p o 〉 , where H is the horizon length, T and I again denote the index sets for the teams and agents, S denotes the finite set of states, and A i and r i : S × A → R denote, resp., the agent i 's finite action set and immediate reward function, depending on current state and joint actions. 1 The state of the underlying game can change according to the transition kernel p ( · | s, a ) , depending on the current state and joint actions, and the initial state is determined by the probability distribution p o ∈ ∆( S ) .

Let each team m randomize their actions contingent on the current state s ∈ S and stage h ∈ [ H ] := { 1 , . . . , H } via a stationary strategy π m : S × [ H ] → ∆( A m ) . Note that team members do not necessarily randomize their actions independently. Given the strategy profile of teams π = ( π m ) m ∈T , agent i 's utility function is defined by

where the pair ( s h , a h ) denotes the state and action profile at stage h , the expectation is taken with respect to the randomness on these pairs ( s h , a h ) induced by the strategy profile π and the underlying transition kernel.

For each state s , let the reward functions { r i ( s, · ) } i ∈I induce a ZSPTG where team m ∈ T has the potential function φ m ( s, · ) : A → R satisfying (2), (3), and (4). Correspondingly, team m 's utility function is given by

Let Π m := { π m | π m : S × [ H ] → ∆( A m ) } denote the set of stationary strategy profiles for team m . Then, the following is the counterpart of Definition 2.4 for multi-team MGes.

Definition C.1 (Team-Nash Gap for MGs) . Given the strategy profile of teams { π m ∈ Π m } m ∈T , we define the team-Nash gap for team m as

glyph[negationslash]

and TNG( π ) := ∑ m ∈T TNG m ( π ) , where π -m := { π glyph[lscript] } glyph[lscript] = m . Correspondingly, we say that the strategy profile of teams { π m } m ∈T is glyph[epsilon1] -TNE if TNG( π ) ≤ glyph[epsilon1] .

Next, we describe the stage-game framework, going back to the introduction of Markov games [Shapley, 1953], and also used in multi-agent reinforcement learning algorithms such as Minimax-Q [Littman, 1994] and Nash-Q [Hu and Wellman, 2003]. Particularly, at each stage, the agents' joint actions determine the immediate rewards they receive and the next state, and therefore, the future rewards to be received. Given the strategy profile of teams π = ( π m ) m ∈T , let v i : S × [ H ] → R denote agent i 's value function such that v i ( s ) is the game value for the stage game associated with state s . Similarly, let Q i : S × [ H ] × A → R be the Q-function such that Q i ( s, · ) corresponds to agent i 's payoff function for the stage game associated with state s . The definition of the utility (63) yields that

The Q-function and value function implicitly depend on π .

We can extend Team-FP dynamics to MGs played repeatedly. Based on the stage game framework, agents play some stage game associated with each state s ∈ S and stage h ∈ [ H ] pair repeatedly whenever the underlying MG visits state s at stage h . Correspondingly, agents form beliefs about the opponent teams as if the opponent teams play according to some stationary strategies across these repetitions.

glyph[negationslash]

Let π glyph[lscript] k : S × [ H ] → ∆( A glyph[lscript] ) denote the beliefs formed by agents i / ∈ I glyph[lscript] about team glyph[lscript] , and Q i k : S × [ H ] × A → R denote agent i 's Q-function estimate at the k th repetition. If the underlying MGvisits state s at stage h at the k th repetition, then agent i follows Team-FP dynamics as if the payoff function is the Q-function estimate Q i k ( s, h, · ) . Agents also recall the previous actions of their teams specific to each stage game. We denote agent i 's previous action for the pair of state s and stage h until and including the k th repetition by a i k ( s, h ) ∈ A i , with a slight abuse of notation. Then, at stage k , agent i ∈ I m either takes the previous action for that stage game (i.e., a i k ( s, h ) = a i k -1 ( s, h ) ), or takes the action a i k ( s, h ) ∼ br τ ( Q i k ( s, h, · , a -i k -1 ( s, h ) , π -m k ( s, h ))) according to the smoothed best response to the previous actions of the team members a -i k -1 ( · ) := { a j k -1 ( · ) } j ∈I m \{ i } and the beliefs π -m k := { π glyph[lscript] k } glyph[lscript] = m formed about other teams.

Let s h,k and a h,k denote, resp., the state and action profile at stage h at the k th repetition. Then, given some reference step size { α c } c ≥ 0 , agents j / ∈ I m update their beliefs about team m 's strategy according to

where a m k ( · ) := { a j k ( · ) } j ∈I m and c k ( s, h ) is the number of times state s get visited at stage h until the k th repetition. Correspondingly, by (66), each agent i ∈ I m updates their Q-function estimates according to

where λ k ( s, h, a ) ∈ [0 , 1] is also some step size. If the agent i knows the model of the underlying MG, then we have

for all a ∈ A and

If the agent does not know the model, then we have

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

where r i h,k denotes the reward received at stage h at the k th repetition and we approximate the expected continuation payoff by looking one stage ahead, as in the classical Q-learning algorithm, and c k ( s, h, a ) is the number of times the pair ( s, a ) gets realized at stage h until the k th repetition. Algorithms 2 and 3 provide descriptions of the extensions, resp., for the model-based and model-free cases from the perspective of the typical agent i ∈ I m from the typical team m ∈ T .

Remark C.2 . Algorithm 2 reduces to Algorithm 1 if | S | = 1 and H = 1 .

D Large-scale Numerical Examples

In this section, we give a large-scale experiment that show the scalability of Team-FP in a networked game where only 2-hop neighbors of agents affect their payoff function. We simulated a threeteam game with nine agents per team, resulting in a large joint action space of size 2 2 7 . After ten independent trials, we plotted the evolution of the Team-Nash Gap in Figure 7. Despite the problem's scale, the empirical averages of team actions converge to the Team-Nash equilibrium at a similar rate, even with sparse network interconnections, as shown in the top right of Figure 7.

Algorithm 3 Model-free (Independent) Team-FP for MGs

glyph[negationslash]

glyph[negationslash]

13: update the beliefs and the Q-functions

14: end for

Team-FP

4.0

Team-FPLarge Scale

3.5

3.0

Team Nash Gap

2.5

2.0

1.5

1.0

0.5

0.0

102

103

104

105

106

Number of Iterations

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction is clear about the claims of the paper.

Guidelines:

glyph[negationslash]

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed at the end before the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For all the theorems, we provide assumptions and proofs along with proper references.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Although there are some randomness in the generation, and we did not save the seeds, we share the code, and it should work as in the simulations for any other randomly generated games. Therefore, anyone can reproduce the results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We share code for experiments. However, the code may be too complex to understand as most of the documentation for the code is missing.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify experimental settings of each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include mean and standard deviation of independent trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We give specifications of the computer and rough run-time of the algorithms. However, we do not provide exact run-times or required time for preliminary experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicMGuidelines ?

Answer: [Yes]

Justification: We discuss how our work does not raise any ethical concerns in the conclusion section.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no social impact of the work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not contain risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use existing assets

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There are no human subjects or potential risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There are no human subjects or potential risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces a new variant of fictitious play where agents respond to the last actions of team members. The authors need to improve their academic writing skills largely. The expression between paragraphs and sentences lacks logic and consistency. Moreover, the paper does not list the challenges and gaps in current research and why this research can solve the recent issue. Furthermore, from my point of view, I did not see particular novelty or practical value in this research.

STRENGTHS:
This paper introduces a new variant of fictitious play where agents respond to the last actions of team members.

WEAKNESSES:
The authors need to improve their academic writing skills largely. The expression between paragraphs and sentences lacks logic and consistency. Moreover, the paper does not list the challenges and gaps in current research and why this research can solve the recent issue. Furthermore, from my point of view, I did not see particular novelty or practical value in this research.

QUESTIONS:
I suggest the author reorganize the paper and improve the academic writing skills.

LIMITATIONS:
I suggest the author reorganize the paper and improve the academic writing skills.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the complex problem of multi-team games by introducing a new variant of fictitious play called Team-FP. The method aims to enable teams of self-interested agents to reach Team-Nash Equilibrium (TNE) in multi-team games, with a particular focus on zero-sum potential team games (ZSPTGs). The authors present a rigorous convergence analysis and extend the Team-FP dynamics to multi-team Markov games. Extensive simulations are provided, comparing Team-FP with other algorithms to demonstrate its effectiveness and practical applicability.\n\nSTRENGTHS:\nThe paper introduces Team-FP, a novel variant of fictitious play specifically designed for multi-team games. The approach incorporates inertia in action updates and agents’ responses to the last actions of team members, which enhances team coordination. This creative extension of classical fictitious play is valuable in advancing the understanding of team dynamics in multi-agent settings.\n\nThe paper is well-structured and organized, making it easy to follow the progression from the introduction of the problem to the presentation of the proposed methods.\n\nWEAKNESSES:\nDespite the novelty of the Team-FP approach, the modifications to classical fictitious play are not significantly groundbreaking. The methods, while creative, represent incremental improvements rather than major innovations in the field.\n\nI am not familiar with the convergence proof; thus, I cannot and do not have the time to verify the proof. The authors put a lot of effort into the theoretical part. The experiment setting and results are simple. For a theoretical paper, it is better to prove the convergence through experimental results.\n\nThus, I think the paper is below the acceptance bar of NeurIPS.\n\nQUESTIONS:\n1. How can we get the Eq.4?\n\n2. Can you provide experimental evidence to support your convergence proof?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Team-Fictitious Play (Team-FP) dynamics as a novel approach for teams of self-interested agents to converge to Team-Nash equilibrium in multi-team games. The study focuses on games where multiple teams interact strategically, aiming to maximize their collective utilities. For this purpose, this paper (1) introduces Team-FP as a method for teams to converge to Team-Nash equilibrium in multi-team games; (2) extends the convergence analysis of Team-FP dynamics to multi-team Markov games; (3) demonstrates the effectiveness of Team-FP dynamics through theoretical analysis and empirical evaluations in various multi-team game scenarios.\n\nSTRENGTHS:\n1. The introduction of Team-FP represents a novel approach to address the challenge of teams reaching equilibrium in multi-team games.\n2. The detailed numerical analysis and simulations conducted to evaluate the behavior of Team-FP dynamics in various multi-team game settings reflect the thoroughness and quality of the method.\n\nWEAKNESSES:\n1. The notations are confusing and it's hard to keep up. For example, 'agent index' and 'team index' both use lowercase letters. \n2. Limited discussion on computational complexity. The paper could provide more insights into the computational complexity of implementing Team-FP dynamics in large-scale multi-team games. Discussing the scalability of the approach, potential bottlenecks, and computational efficiency considerations would be beneficial for understanding the practical feasibility of deploying Team-FP in complex settings.\n\nQUESTIONS:\nThe authors provide some experiments, but it would be helpful to see more extensive experiments, including a larger-scale multi-team game to demonstrate the effectiveness of the algorithm in a more complex setting.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors introduce a novel variant of virtual play, referred to as **Team-Fictitious Play (Team-FP)**, aimed at assisting self-interested agents within teams to reach **Team Nash Equilibrium (TNE)** in multi-team games. The paper focuses on zero-sum potential team games (ZSPTGs), where teams interact pairwise, but the payoffs to team members are not necessarily identical. The main contributions include the introduction of inertia in action updates and responses to the last actions of team members, which are crucial for team coordination. The authors provide theoretical convergence guarantees and validate the efficacy of the approach through extensive simulations.\n\nSTRENGTHS:\n1. The introduction of Team-FP fills a gap in the multi-team game theory literature, particularly in the context of zero-sum potential team games.\n2. The paper offers rigorous theoretical analysis and practical insights, including convergence proofs and error bounds.\n3. Extensive simulations compare Team-FP with other algorithms, demonstrating its effectiveness and exploring the impact of various parameters on convergence speed.\n4. The grammar and expression are accurate and professional.\n\nWEAKNESSES:\n- Adding more background or appendix sections on Nash equilibrium and related solution algorithms would greatly help the reader's understanding.\n- The theoretical analysis section is quite technical; including more intuitive explanations and illustrations could be beneficial. For instance, adding a diagram in Section 3 to visually depict the workflow of Team-FP might help.\n- A more detailed analysis of the parameters used in Team-FP and their sensitivity to performance could provide deeper insights.\n- While the paper focuses on zero-sum potential team games, discussing how Team-FP could be adapted or extended to other types of multi-team games would be valuable.\n- The numerical experiments section could benefit from more qualitative analysis.\n- The discussion on practical applications is not sufficiently thorough; adding more experiments and analysis in specific application scenarios would be beneficial.\n- Section 5 could include experimental comparisons with other multi-team learning algorithms.\n\nQUESTIONS:\nAll my questions are written inside the weakness part.\n\nLIMITATIONS:\nThe paper explicitly discusses its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}