SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 4 (Challenges for analyzing general cases). One might ask why we had to focus on a star-shaped structure. First, for general networked structures, even in the matrix-game case, it is known that the NE values of a zero-sum NG may not be unique [24]. Hence, suppose one performs Nash-value iteration, i.e., solving for the NE of the stage game and conducting backward induction, this value iteration process does not converge in general as the number of backward steps increases, since the solution at each stage is not even unique, and there may not exist a unique fixed point. This is in stark contrast to the max and max min operators in the value iteration updates for single-player and two-player zero-sum cases, respectively. By exploiting a star-shaped structure, we managed to reformulate a minimax optimization problem when solving each stage game, which makes the corresponding value iteration operator contracting, and thus iterating it infinitely converges to the unique fixed point. Second, suppose there exists some other network structure (other than star-shaped ones) that also leads to a contracting value iteration operator, then for a fixed constant  $\gamma$ , the fixed point (which corresponds to the Markov stationary CCE/NE of the zero-sum NMG) becomes unique and can be computed efficiently, which contradicts our hardness result in Theorem 1. Indeed, it was the exclusion of a star-shaped structure in Theorem 1 that inspired us to consider this structure in proving the convergence of FP dynamics. That being said, we note that having a contracting value iteration operator is only a sufficient condition for the FP dynamics to converge. It would be interesting to explore other structures that enjoy the FP property for reasons beyond this contraction property. We leave this as an immediate future work.
Next, we present another positive result in light of the hardness in Theorem 1, regarding the computation of non-stationary equilibria in multi-player zero-sum NMGs.
6 Non-Stationary NE Computation
We now focus on computing an (approximate) Markov non-stationary equilibrium in zero-sum NMGs. In particular, we show that when relaxing the stationarity requirement, not only CCE, but NE, can be computed efficiently. Before introducing our algorithm, we first recall the folklore result that approximating Markov non-stationary NE in infinite-horizon discounted settings can be achieved by finding approximate Markov NE in finite-horizon settings, with a large enough horizon length (c.f. Proposition 10). Hence, we will focus on the finite-horizon setting from now on.
Before delving into the details of our algorithm, we introduce the notation  $Q_{h,i}(s)$  and  $Q_{h}(s)$  for  $h \in [H], i \in \mathcal{N}, s \in \mathcal{S}$  as follows:
$$
\boldsymbol {Q} _ {h, i} (s) := \left(Q _ {h, i, 1} (s), \dots , Q _ {h, i, i - 1} (s), \mathbf {0}, Q _ {h, i, i + 1} (s) \dots , Q _ {h, i, n} (s)\right) \in \mathbb {R} ^ {\left| \mathcal {A} _ {i} \right| \times \sum_ {i \in \mathcal {N}} \left| \mathcal {A} _ {i} \right|}
$$
$$
\boldsymbol {Q} _ {h} (s) := \left(\left(\boldsymbol {Q} _ {h, 1} (s)\right) ^ {\intercal}, \left(\boldsymbol {Q} _ {h, 2} (s)\right) ^ {\intercal}, \dots , \left(\boldsymbol {Q} _ {h, n} (s)\right) ^ {\intercal}\right) ^ {\intercal} \in \mathbb {R} ^ {\sum_ {i \in \mathcal {N}} | \mathcal {A} _ {i} | \times \sum_ {i \in \mathcal {N}} | \mathcal {A} _ {i} |}.
$$
Here,  $Q_{h,i,j}$  represents an estimate of the equilibrium value function with canonical decomposition (Proposition 2). Hereafter, we similarly define the notation of  $Q_{h,i}^{\pi}$  and  $Q_{h}^{\pi}$ . Our algorithm is based on value iteration, and iterates three main steps from  $h = H$  to 1 as follows: (1)  $Q$ -value computation: compute  $Q_{h,i,j}$ , which estimates the equilibrium  $Q$ -value with a canonical decomposition form; in particular, when  $\mathcal{N}_C \neq \emptyset$ ,  $Q_{h,i,j}$  is updated for all  $s \in S$ ,  $(i,j) \in \mathcal{E}_Q$ ,  $a_i \in \mathcal{A}_i$ , and  $a_j \in \mathcal{A}_j$ :
$$
Q _ {h, i, j} (s, a _ {i}, a _ {j}) = r _ {h, i, j} (s, a _ {i}, a _ {j}) + \sum_ {s ^ {\prime} \in \mathcal {S}} \bigg (\frac {1}{| \mathcal {E} _ {Q , i} |} \mathbf {1} (i \in \mathcal {N} _ {C}) \mathbb {F} _ {h, i} (s ^ {\prime} \mid s, a _ {i}) + \mathbf {1} (j \in \mathcal {N} _ {C}) \mathbb {F} _ {h, j} (s ^ {\prime} \mid s, a _ {j}) \bigg) V _ {h + 1, i} (s ^ {\prime}),
$$
(2) Policy update: update  $\pi_h(s)$  with an NE-ORACLE: finding (approximate)-NE of some zero-sum NG  $(\mathcal{G},\mathcal{A},(Q_{h,i,j}(s))_{(i,j)\in \mathcal{E}_Q})$  for all  $s\in S$ , and (3) Value function update: compute  $V_{h,i}$ , which estimates the equilibrium value function as follows for all  $s\in S$ ,  $i\in \mathcal{N}$ :  $V_{h,i}(s) = \pi_{h,i}^{\top}(s)Q_{h,i}(s)\pi_h(s)$ . The overall procedure is summarized in Algorithm 6.
NE-ORACLE and iteration complexity. The NE-ORACLE in Algorithm 6 can be instantiated by several different algorithms that can find an NE in a zero-sum NG. Depending on the algorithms, the convergence guarantees can be either in terms of average-iterate, best-iterate, or last-iterate. Note that for algorithms with average-iterate convergence, one may additionally need to marginalize the output joint policy, i.e., the approximate CCE, and combine them as a product policy that is an approximate NE (Proposition 6). For those with best-/last-iterate convergence, by contrast, the best-/last-iterate is already in product form, and one can directly output it as an approximate NE. Moreover, last-iterate convergence is known to be a more favorable metric than the average-iterate one in learning in games [45, 46, 47, 48, 49], which is able to characterize the day-to-day behavior of the iterates and implies the stability of the update rule. Hence, one may prefer to have last-iterate convergence for solving zero-sum N(M)Gs. To this end, two algorithmic ideas may be useful: adding regularization to the payoff matrix [39, 42, 50, 51, 52], and/or using the idea of optimism [47, 36, 53]. Recent results [54, 51] have instantiated the ideas of optimism-only and optimism + regularization, respectively, for best-/last-iterate convergence in zero-sum polymatrix games. We additionally established results for the idea of regularization-only in obtaining last-iterate convergence in these games. Specifically, we propose to study the vanilla Multiplicative Weight Update (MWU) algorithm [55] in the regularized zero-sum NG, as tabulated in Algorithm 9. We have also introduced a variant with diminishing regularization, and summarize the update rule in Algorithm 10.
Given the results above, aggregating  $\epsilon$ -approximate NE for the zero-sum NGs  $(\mathcal{G}, \mathcal{A}, (Q_{h,i,j}(s))_{(i,j) \in \mathcal{E}_Q})$  for all  $h \in [H], i \in \mathcal{N}, s \in S$  provides an  $H\epsilon$ -approximate NE for the corresponding zero-sum NMG. We have the following formal result.
Proposition 4. Suppose that for all  $h \in [H], i \in \mathcal{N}, s \in \mathcal{S}$ , NE-ORACLE  $(\mathcal{G}, \mathcal{A}, (Q_{h,i,j}(s))_{(i,j) \in \mathcal{E}_Q})$  provides an  $\epsilon_{h,s}$ -approximate NE for the zero-sum NG  $(\mathcal{G}, \mathcal{A}, (Q_{h,i,j}(s))_{(i,j) \in \mathcal{E}_Q})$  in Algorithm 6. Then, the output policy  $\pi$  in Algorithm 6 is an  $(\sum_{h \in [H]} \max_{s \in S} \epsilon_{h,s})$ -approximate NE for the corresponding zero-sum NMG  $(\mathcal{G} = (\mathcal{N}, \mathcal{E}_Q), \mathcal{S}, \mathcal{A}, H, (\mathbb{P}_h)_{h \in [H]}, (r_{h,i,j}(s))_{(i,j) \in \mathcal{E}_Q, s \in S})$ .
The proof of Proposition 4 is deferred to Section F. In light of Proposition 4 and Table 1, we obtain Table 2, which summarizes the iteration complexities required to find an  $\epsilon$ -NE for zero-sum NMGs, with different NE-ORACLE subroutines. Note that the iteration complexities are all polynomial in  $H, n, |S|$ , and inherit the order of dependencies on  $\epsilon$  from Table 1 for the matrix-game case. In particular, Algorithm 6 with the OMWU in [51] yields the fast rate of  $\widetilde{\mathcal{O}}(1/\epsilon)$  for the last iterate.
The authors would like to thank the anonymous reviewers of NeurIPS for their helpful feedback. C.P. acknowledges support from the Xianhong Wu Fellowship, the Korea Foundation for Advanced Studies, and the Siebel Scholarship. K.Z. acknowledges support from the Northrop Grumman – Maryland Seed Grant Program. A.O. acknowledges support from the MIT-Air Force AI Innovation Accelerator Grant and the MIT-DSTA grant 031017-00016.
[1] John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton University Press, 1947.
[2] John F Nash Jr. Equilibrium points in n-person games. Proceedings of the National Academy of sciences, 36(1):48-49, 1950.
[3] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge University press, 2006.
[4] Drew Fudenberg and David K Levine. The theory of learning in games, volume 2. MIT Press, 1998.
[5] George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):374, 1951.
[6] Julia Robinson. An iterative method of solving a game. Annals of mathematics, pages 296-301, 1951.
[7] Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and Economic Behavior, 5:320-367, 1993.
[8] Dov Monderer and Lloyd S Shapley. Fictitious play property for games with identical interests. Journal of economic theory, 68(1):258-265, 1996.
[9] Koichi Miyasawa. On the convergence of the learning process in a  $2 \times 2$  non-zero-sum two-person game. Princeton University Press, 1961.
[10] Ulrich Berger. Fictitious play in 2 x n games. Journal of Economic Theory, 120(2):139-154, 2005.
[11] Aner Sela. Fictitious play in “one-against-all” multi-player games. *Economic Theory*, 14(3):635–651, 1999.
[12] Christian Ewerhart and Kremena Valkanova. Fictitious play in networks. Games and Economic Behavior, 123:182-206, 2020.
[13] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.
[14] David S Leslie, Steven Perkins, and Zibo Xu. Best-response dynamics in zero-sum stochastic games. Journal of Economic Theory, 189:105095, 2020.
[15] Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized Q-learning in zero-sum Markov games. NeurIPS, 2021.
[16] Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero-sum stochastic games. SIAM Journal on Control and Optimization, 60(4):2095-2114, 2022.
[17] Lucas Baudin and Rida Laraki. Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. ICML, 2022.
[18] Zaiwei Chen, Kaiqing Zhang, Eric Mazumdar, Asuman Ozdaglar, and Adam Wierman. A finite-sample analysis of payoff-based independent learning in zero-sum stochastic games. NeurIPS, 2023.
[19] Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in markov games with single controller. EC, 2022.
[20] Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing, 39(1):195-259, 2009.
[21] Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player nash equilibria. Journal of the ACM (JACM), 56(3):1-57, 2009.
[22] Christos H Papadimitriou and Tim Roughgarden. Computing correlated equilibria in multiplayer games. Journal of the ACM (JACM), 55(3):1-29, 2008.
[23] Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. SODA, 2011.
[24] Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Mathematics of Operations Research, 41(2):648-655, 2016.
[25] Zhigang Cao, Haoyu Gao, Xinglong Qu, Mingmin Yang, and Xiaoguang Yang. Fashion, cooperation, and social interactions. PLoS One, 8(1):e49441, 2013.
[26] Zhigang Cao and Xiaoguang Yang. The fashion game: Network extension of matching pennies. Theoretical Computer Science, 540:169-181, 2014.
[27] Boyu Zhang, Zhigang Cao, Cheng-Zhong Qin, and Xiaoguang Yang. Fashion and homophily. Operations Research, 66(6):1486-1497, 2018.
[28] L M Bergman and I N Fokin. On separable non-cooperative zero-sum games. Optimization, 44(1):69-84, 1998.
[29] Yujia Jin, Vidya Muthukumar, and Aaron Sidford. The complexity of infinite-horizon general-sum stochastic games. *ITCS*, 2022.
[30] Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games.  $COLT$ , 2023.
[31] L M Bergman and I N Fokin. Methods of determining equilibrium situations in zero-sum polymatrix games. Optimizatsia, 40(57):70-82, 1987.
[32] Christos H Papadimitriou. On the complexity of the parity argument and other inefficient proofs of existence. Journal of Computer and System Sciences, 48(3):498-532, 1994.
[33] János Flesch, Frank Thuijsman, and Okko Jan Vrieze. Stochastic games with additive transitions. European Journal of Operational Research, 179(2):483-497, 2007.
[34] Jerzy Filar and Koos Vrieze. Competitive Markov decision processes. Springer Science & Business Media, 2012.
[35] Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. NeurIPS, 2020.
[36] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games.  $COLT$ , 2021.
[37] Yulai Zhao, Yuandong Tian, Jason D Lee, and Simon S Du. Provably efficient policy gradient methods for two-player zero-sum markov games. AISTATS, 2022.
[38] Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum markov game. ICLR, 2021.
[39] Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. NeurIPS, 2021.
[61] Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. JMLR, 2003.
[62] Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. ICML, 2020.
[63] Aaron Sidford, Mengdi Wang, Lin Yang, and Yinyu Ye. Solving discounted stochastic two-player games with near-optimal time and sample complexity. AISTATS, 2020.
[64] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. COLT, 2020.
[65] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. NeurIPS, 2020.
[66] Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. ICML, 2021.
[67] Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. NeurIPS, 2020.
[68] Gen Li, Yuejie Chi, Yuting Wei, and Yuxin Chen. Minimax-optimal multi-agent rl in Markov games with a generative model. NeurIPS, 2022.
[69] Jayakumar Subramanian, Amit Sinha, and Aditya Mahajan. Robustness and sample complexity of model-based MARL for general-sum Markov games. Dynamic Games and Applications, pages 1-33, 2023.
[70] Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum markov games with a large number of players sample-efficiently? ICLR, 2021.
[71] Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent RL. ICLR 2022 workshop “Gamification and Multiagent Solutions”, 2022.
[72] Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. ICML, 2022.
[73] Weichao Mao and Tamer Başar. Provably efficient reinforcement learning in decentralized general-sum Markov games. Dynamic Games and Applications, pages 1-22, 2022.
[74] Qiwen Cui, Kaiqing Zhang, and Simon S Du. Breaking the curse of multiagents in a large state space: RL in Markov games with independent linear function approximation.  $COLT$ , 2023.
[75] Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent RL with function approximation.  $COLT$ , 2023.
[76] Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. ICML, 2023.
[77] Aviad Rubinstein. Settling the complexity of computing approximate two-player Nash equilibria. ACM SIGcom Exchanges, 15(2):45-49, 2017.
[78] Constantinos Daskalakis. Non-concave games: A challenge for game theory's next 100 years. 2022.
[79] Matthew O Jackson and Yves Zenou. Games on networks. In Handbook of game theory with economic applications, volume 4, pages 95-163. Elsevier, 2015.
[80] Michael Kearns, Michael L Littman, and Satinder Singh. Graphical models for game theory. UAI, 2001.
[81] Sham Kakade, Michael Kearns, John Langford, and Luis Ortiz. Correlated equilibria in graphical games. EC, 2003.
[103] Jens Grossklags, Nicolas Christin, and John Chuang. Secure or insure? a game-theoretic analysis of information security games. WWW, 2008.
[104] Robert Wilson. Game-theoretic analysis of trading processes. Technical report, Stanford University, 1985.
[105] Robert Wilson. Game theoretic analysis of trading. In Advances in Economic Theory: Fifth World Congress, number 12, page 33. CUP Archive, 1989.
[106] Dayong Zhang, Lei Lei, Qiang Ji, and Ali M Kutan. Economic policy uncertainty in the US and china and their impact on the global markets. *Economic Modelling*, 79:47–56, 2019.
[107] Michael Beckley. The power of nations: Measuring what matters. International Security, 43(2):7-44, 2018.
[108] Charles Freedman, Michael Kumhof, Douglas Laxton, Dirk Muir, and Susanna Mursula. Global effects of fiscal stimulus during the crisis. Journal of monetary economics, 57(5):506-526, 2010.
[109] Klaus Armingeon. The politics of fiscal responses to the crisis of 2008–2009. Governance, 25(4):543–565, 2012.
[110] Fábio Augusto Reis Gomes, Sergio Naruhiko Sakurai, and Gian Paulo Soave. Government spending multipliers in good times and bad times: The case of emerging markets. *Macroeconomic Dynamics*, 26(3):726-768, 2022.
[111] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior, 10(1):6-38, 1995.
[112] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for extensive form games. Experimental economics, 1:9-41, 1998.
[113] Panayotis Mertikopoulos and William H Sandholm. Learning in games via reinforcement and regularization. Mathematics of Operations Research, 41(4):1297-1324, 2016.
[114] Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. NeurIPS, 2021.
[115] Ioannis Anagnostides, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, and Tuomas Sandholm. Uncoupled learning dynamics with  $o(\backslash \log t)$  swap regret in multiplayer games. NeurIPS, 2022.
[116] Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. NeurIPS, 2022.
[117] David Reeb and Michael M Wolf. Tight bound on relative entropy by entropy difference. IEEE Transactions on Information Theory, 61(3):1458-1473, 2015.
[118] Haipeng Luo. Introduction to online optimization/learning (fall 2022), lecture note 1.
[119] Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Uncoupled and convergent learning in two-player zero-sum markov games. arXiv preprint arXiv:2303.02738, 2023.