SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fengzhuo Zhang<sup>1</sup> Vincent Y. F. Tan<sup>1</sup> Zhaoran Wang<sup>2</sup> Zhuoran Yang<sup>3</sup>  
<sup>1</sup>National University of Singapore <sup>2</sup>Northwestern University <sup>3</sup>Yale University fzzhang@u.nus.edu, vtan@nus.edu.sg, zhaoranwang@gmail.com, zhuoranyang.work@gmail.com
This paper studies two fundamental problems in regularized Graphon Mean-Field Games (GMFGs). First, we establish the existence of a Nash Equilibrium (NE) of any  $\lambda$ -regularized GMFG (for  $\lambda \geq 0$ ). This result relies on weaker conditions than those in previous works for analyzing both unregularized GMFGs ( $\lambda = 0$ ) and  $\lambda$ -regularized MFGs, which are special cases of GMFGs. Second, we propose provably efficient algorithms to learn the NE in weakly monotone GMFGs, motivated by Lasry and Lions [2007]. Previous literature either only analyzed continuous-time algorithms or required extra conditions to analyze discrete-time algorithms. In contrast, we design a discrete-time algorithm and derive its convergence rate solely under weakly monotone conditions. Furthermore, we develop and analyze the action-value function estimation procedure during the online learning process, which is absent from algorithms for monotone GMFGs. This serves as a sub-module in our optimization algorithm. The efficiency of the designed algorithm is corroborated by empirical evaluations.
In Multi-Agent Reinforcement Learning (MARL), the sizes of state and action spaces grow exponentially in the number of agents, which is known as the "curse of many agents" [Sonu et al., 2017, Wang et al., 2020] and restrict its applicability to large-scale scenarios. The Mean-Field Game (MFG) has thus been proposed to mitigate this problem by exploiting the homogeneity assumption [Huang et al., 2006, Lasry and Lions, 2007], and it has achieved tremendous successes in many real-world applications [Cousin et al., 2011, Achdou et al., 2020]. However, the homogeneity assumption is an impediment when modeling scenarios in which the agents are heterogeneous. GMFGs, as extensions of MFGs, have thus been proposed to model the behaviors of heterogeneous agents and ameliorate the "curse of many agents" at the same time [Parise and Ozdaglar, 2019, Carmona et al., 2022].
Despite the empirical successes of the Graphon Mean-Field Game (GMFG) [Aurell et al., 2022a], its theoretical understanding is lacking. First, sufficient conditions for Nash Equilibrium (NE) existence in regularized GMFG have not been established. Most works only address the existence of the NE in unregularized GMFGs. However, regularization is employed in practical implementations for improved exploration and robustness [Geist et al., 2019]. Moreover, previous works prove the existence of NE in regularized MFGs, a special case of GMFGs, only under the contraction condition, which is overly restrictive for real-world applications. Second, the analysis of discrete-time algorithms for monotone GMFGs is lacking. Most existing works design provably efficient discrete-time algorithms only under contraction conditions, as shown in Table 1. Complementarily, previous works on monotone GMFGs either only derive the convergence rate for continuous-time algorithms, which ignores the discretization error, or require extra conditions, (e.g., potential games) to analyze discrete-time algorithms.
Table 1: Comparison of GMFGs and MFGs learning algorithms
Condition	No population manipulation	Online playing	Heterogeneity	Discrete-time algorithm	Convergence rate
Anahtarci et al. [2022]	Contraction	No	No	No	Yes	Yes
Xie et al. [2021a]	Contraction	No	Yes	No	Yes	Yes
Zaman et al. [2022]	Contraction	No	Yes	No	Yes	Yes
Yardim et al. [2022]	Contraction	Yes	Yes	No	Yes	Yes
Perrin et al. [2020]	Monotone	No	No	No	No	Yes
Geist et al. [2021]	Potential &Monotone	No	No	No	Yes	Yes
Perolat et al. [2021]	Monotone	Yes	Yes	Yes	No	No
Fabian et al. [2022]	Monotone	Yes	Yes	Yes	No	No
Our work	Monotone	Yes	Yes	Yes	Yes	Yes
In this paper, we first consider GMFGs in full generality, i.e., without any contraction or monotone conditions. The goal is to establish the existence of NE in the regularized GMFG in this general setting. Then we focus on monotone GMFGs motivated by Lasry and Lions [2007]. We aim to learn the unique NE from the online interactions of all agents with and without the action-value function oracle. When the oracle is absent, the action-value functions should be estimated from the data of sampled agents generated in the online game.
In the analysis, we have to overcome difficulties that arise from both the existence of the NE problem and the learning of the NE. First, the proof of the existence of NE in regularized GMFG involves establishing some topological spaces and operators related to NE on which fixed point theorems are applicable. However, the direct construction of the space and the operators for GMFGs with uncountably infinite agents is challenging. Second, the design and analysis of the discrete-time NE learning algorithm require subtle exploitation of the monotone condition. Unlike continuous-time algorithms with infinitesimal step sizes, the design of appropriate step sizes is additionally required for the discrete-time algorithm to guarantee that iterates evolve appropriately. This guarantee originates from the delicate utilization of the monotone condition in the optimization procedures.
To address the difficulty of the existence problem, we construct a regularized MFG from the regularized GMFG and show that the NE of the constructed game can be converted into the NE of the original game, thus mitigating the difficulty of having an uncountable number of agents. To handle the difficulty in the NE learning problem, we design the Monotone GMFG Policy Mirror Descent (MonoGMFG-PMD) algorithm, which iteratively implements policy mirror descent for each player. We show that this procedure results in a decrease of the KL divergence between the iterate and the NE, and this decrease is related to the gap appearing in the weakly monotone condition. When the action-value function oracle is absent, we also design and analyze action-value functions estimation algorithms to serve as a submodule of the optimization procedures.
Main Contributions: We first establish the existence of the NE in the  $\lambda$ -regularized GMFG with  $\lambda \geq 0$  assuming Lipschitzness of graphons and continuity of transition kernels and reward functions. Our result relaxes the assumption of the Lipschitzness of transition kernels and rewards required in previous works on unregularized GMFGs and the contraction condition in the literature on regularized MFG [Cui and Koeppl, 2021a]. Then we design and analyze the MonoGMFG-PMD algorithm. Using an action-value function oracle, the convergence rate for MonoGMFG-PMD is proved to be  $\tilde{O}(T^{-1/2})$  after  $T$  iterations. Without the oracle, the convergence rate includes an additional  $\tilde{O}(K^{-1/2} + N^{-1})$  term that arises from sampling  $N$  agents and collecting data from  $K$  episodes, reflecting the generalization error and the approximation error of the estimation algorithm. As shown in Table 1, our algorithm can be implemented from the online interaction of agents and does not require the distribution flow manipulation. Detailed explanations of the properties stated in the columns of Table 1 are provided in Appendix A. Our result for MonoGMFG-PMD provides the first convergence rate for discrete-time algorithms in monotone GMFGs.
MFGs were proposed by Huang et al. [2006] and Lasry and Lions [2007] to model the interactions among a set of homogeneous agents. In recent years, learning the NE of the MFGs formulated by
discrete-time Markov Decision Process (MDP)s has attracted a lot of interest. There is a large body of works that design and analyze algorithms for the MFGs under contraction conditions [Anahtarci et al., 2019, 2022, Cui and Koeppl, 2021a, Xie et al., 2021a, Zaman et al., 2022, Yardim et al., 2022]. Typically, these works design reinforcement learning algorithms to approximate the contraction operators in MFGs, and the NE is learned by iteratively applying this operator. In contrast, another line of works focuses on the MFGs under monotone conditions. Motivated by Lasry and Lions [2007], the transition kernels in these works are independent of the mean fields. Perrin et al. [2020] proposed and analyzed the continuous-time fictitious play algorithm, which dynamically weighs the past mean fields and the best response policies to derive new mean fields and policies. With the additional potential assumption, Geist et al. [2021] derived the convergence rate for the discrete-time fictitious play algorithm. Perolat et al. [2021] then proposed the continuous-time policy mirror descent algorithm but only proved the asymptotic convergence, i.e., the consistency. In summary, there is no convergence rate result for any discrete-time algorithm for MFGs under the monotone condition. In addition, the relationship between the contraction conditions and the monotone conditions is not clear from existing works, but they complement each other.
To capture the potential heterogeneity among agents, GMFGs have been proposed by Parise and Ozdaglar [2019] in static settings as an extension of MFGs. The heterogeneous interactions among agents are represented by graphons. Aurell et al. [2022b], Caines and Huang [2019] then extended the GMFGs to the continuous-time setting, where the existence and the uniqueness of NE were established. Vasal et al. [2020] formulated discrete-time GMFGs and provided way to compute the NE with master equations. With the precise underlying graphons values, Cui and Koeppl [2021b] proposed algorithms to learn the NE of GMFGs with the contraction condition by modifying MFGs learning algorithms. Fabian et al. [2022] considered the monotone GMFG and proposed the continuous-time policy mirror descent algorithm to learn the NE. However, only consistency was provided in the latter two works.
Notation Let  $[N] := \{1, \dots, N\}$ . Given a measurable space  $(\Omega, \mathcal{F})$ , we denote the collection of all the measures and the probability measures on  $(\Omega, \mathcal{F})$  as  $\mathcal{M}(\Omega)$  and  $\Delta(\Omega)$ , respectively. For a metric space  $(\mathcal{X}, \| \cdot \|)$ , we use  $C(\mathcal{X})$  and  $C_b(\mathcal{X})$  to denote the set of all continuous functions and the set of all bounded continuous functions on  $\mathcal{X}$ , respectively. For a measurable space  $(\mathcal{X}, \mathcal{F})$  and two distributions  $P, Q \in \Delta(\mathcal{X})$ , the total variation distance between them is defined as  $\mathrm{TV}(P, Q) := \sup_{A \in \mathcal{F}} |P(A) - Q(A)|$ . A sequence of measures  $\{\mu_n\}$  on  $\mathcal{X}$  is said to converge weakly to a measure  $\mu$  if  $\int_{\mathcal{X}} g(x) \mu_n(\mathrm{d}x) \to \int_{\mathcal{X}} g(x) \mu(\mathrm{d}x)$  for all  $g \in C_b(\mathcal{X})$ .
3.1 Graphon Mean-Field Games
We consider a GMFG that is defined through a tuple  $(\mathcal{I},\mathcal{S},\mathcal{A},H,P,r,W,\mu_1)$ . The horizon (or length) of the game is denoted as  $H$ . In GMFG, we have infinite players, each corresponding to a point  $\alpha \in \mathcal{I} = [0,1]$ . The state and action space of them are the same, denoted as  $\mathcal{S} \subseteq \mathbb{R}^{d_{\mathrm{s}}}$  and  $\mathcal{A} \subseteq \mathbb{R}^{d_{\mathrm{a}}}$  respectively. The interaction among players is captured by graphons. Graphons are symmetric functions that map  $[0,1]^2$  to  $[0,1]$ . Symmetry here refers to that  $W(\alpha ,\beta) = W(\beta ,\alpha)$  for all  $\alpha ,\beta \in [0,1]$ . We denote the set of graphons as  $\mathcal{W} = \{W:[0,1]^2\to [0,1]\mid W$  is symmetric.}. The set of graphons of the game is  $W = \{W_h\}_{h = 1}^H$  with  $W_{h}\in \mathcal{W}$ . The state transition and reward of each player are influenced by the collective behavior of all the other players through an aggregate  $z\in \mathcal{M}(\mathcal{S})$ . At time  $h\in [H]$ , we denote the state distribution of player  $\beta \in \mathcal{I}$  as  $\mu_h^\beta \in \Delta (\mathcal{S})$ . The aggregate for player  $\alpha \in \mathcal{I}$  with the underlying graphon  $W_{h}\in \mathcal{W}$  is then defined as
$$
z _ {h} ^ {\alpha} = \int_ {0} ^ {1} W _ {h} (\alpha , \beta) \mu_ {h} ^ {\beta} \mathrm {d} \beta . \tag {1}
$$
The transition kernels  $P = \{P_h\}_{h=1}^H$  of the game are functions  $P_h : \mathcal{S} \times \mathcal{A} \times \mathcal{M}(\mathcal{S}) \to \mathcal{S}$  for all  $h \in [H]$ . At time  $h$ , if player  $\alpha$  takes action  $a_h^\alpha \in \mathcal{A}$  at state  $s_h^\alpha \in \mathcal{S}$ , her state will transition according to  $s_{h+1}^\alpha \sim P_h(\cdot | s_h^\alpha, a_h^\alpha, z_h^\alpha)$ . The reward functions are denoted as  $r_h : \mathcal{S} \times \mathcal{A} \times \mathcal{M}(\mathcal{S}) \to [0,1]$  for all  $h \in [H]$ . We note that the players in GMFG are heterogeneous. This means that different players will, in general, receive different aggregates from other players. The distribution  $\mu_1 \in \Delta(\mathcal{S})$  is the initial state distribution for all the players. A policy for an player  $\alpha$  is  $\pi^\alpha = \{\pi_h^\alpha\}_{h=1}^H \in \Pi^H$ , where  $\pi_h^\alpha : \mathcal{S} \to \Delta(\mathcal{A})$  takes action based only on the current state, and  $\Pi$  is the set of all these policies.
A policy for all the players  $\pi^{\mathcal{I}}\in \tilde{\Pi} = \Pi^{H\times \mathcal{I}}$  is the collection of the policies of each player, i.e,  $\pi^{\mathcal{I}} = \{\pi^{\alpha}\}_{\alpha \in \mathcal{I}}$ . In the following, we denote the state distributions of all the players at time  $h$  and the state distributions of all the players at any time (distribution flow) respectively as  $\mu_h^\mathcal{I} = \{\mu_h^\alpha \}_{\alpha \in \mathcal{I}}$  and  $\mu^{\mathcal{I}} = \{\mu_h^{\mathcal{I}}\}_{h = 1}^H\in \tilde{\Delta} = \Delta (\mathcal{S})^{H\times \mathcal{I}}$ . Eqn. (1) shows that the aggregate  $z_{h}^{\alpha}$  is a function of  $\mu_h^\mathcal{I}$  and  $W_{h}$ , so to make this dependence explicit, we also write it as  $z_{h}^{\alpha}(\mu_{h}^{\mathcal{I}},W_{h})$ .
We consider the entropy-regularized GMFG. It has been shown that the regularization results in policy gradient algorithms converging faster [Shani et al., 2020, Cen et al., 2022]. In this game, the rewards of each player are the sum of the original rewards and the negative entropy of the policy multiplied by a constant. In a  $\lambda$ -regularized GMFG ( $\lambda \geq 0$ ), the value function and the action-value function of player  $\alpha$  with policy  $\pi^{\alpha}$  on the MDP induced by the distribution flow  $\mu^{\mathcal{I}}$  are defined as
$$
V _ {h} ^ {\lambda , \alpha} (s, \pi^ {\alpha}, \mu^ {\mathcal {I}}) = \mathbb {E} ^ {\pi^ {\alpha}} \left[ \sum_ {t = h} ^ {H} r _ {t} \left(s _ {t} ^ {\alpha}, a _ {t} ^ {\alpha}, z _ {t} ^ {\alpha} \left(\mu_ {t} ^ {\mathcal {I}}, W _ {t}\right)\right) - \lambda \log \pi_ {t} ^ {\alpha} \left(a _ {t} ^ {\alpha} \mid s _ {t} ^ {\alpha}\right) \Bigg | s _ {h} ^ {\alpha} = s \right], \tag {2}
$$
$$
Q _ {h} ^ {\lambda , \alpha} (s, a, \pi^ {\alpha}, \mu^ {\mathcal {I}}) = r _ {h} \left(s, a, z _ {h} ^ {\alpha} \left(\mu_ {h} ^ {\mathcal {I}}, W _ {h}\right)\right) + \mathbb {E} ^ {\pi^ {\alpha}} \left[ V _ {h + 1} ^ {\lambda , \alpha} \left(s _ {h + 1} ^ {\alpha}, \pi^ {\alpha}, \mu^ {\mathcal {I}}\right) \mid s _ {h} ^ {\alpha} = s, a _ {h} ^ {\alpha} = a \right] \tag {3}
$$
for all  $h\in [H]$ , where the expectation  $\mathbb{E}^{\pi^{\alpha}}$  is taken with respect to the stochastic process induced by implementing policy  $\pi^{\alpha}$  on the MDP induced by  $\mu^{\mathcal{I}}$ . The cumulative reward function of player  $\alpha$  is defined as  $J^{\lambda ,\alpha}(\pi^{\alpha},\mu^{\mathcal{I}}) = \mathbb{E}_{\mu_1}[V_1^{\lambda ,\alpha}(s,\pi^{\alpha},\mu^{\mathcal{I}})]$ . Then the notion of an NE is defined as follows.
Definition 1. An NE of the  $\lambda$ -regularized GMFG is a pair  $(\pi^{*,\mathcal{I}}, \mu^{*,\mathcal{I}}) \in \tilde{\Pi} \times \tilde{\Delta}$  that satisfies: (i) (player rationality)  $J^{\lambda, \alpha}(\pi^{*,\alpha}, \mu^{*,\mathcal{I}}) = \max_{\tilde{\pi}^{\alpha} \in \Pi^{H}} J^{\lambda, \alpha}(\tilde{\pi}^{\alpha}, \mu^{*,\mathcal{I}})$  for all  $\alpha \in \mathcal{I}$  up to a zero-measure set on  $\mathcal{I}$  with respect to the Lebesgue measure. (ii) (Distribution consistency) The distribution flow  $\mu^{*,\mathcal{I}}$  is equal to the distribution flow induced by implementing the policy  $\pi^{*,\mathcal{I}}$ .
Similar to the NE for the finite-player games, the NE of the  $\lambda$ -regularized GMFG requires that the policy of each player is optimal. However, in GMFGs, the optimality is with respect to  $\mu^{*,\mathcal{I}}$ .
3.2 Mean-Field Games
As an important subclass of GMFG, MFG corresponds to the GMFG with constant graphons, i.e,  $W(\alpha, \beta) = p$  for all  $\alpha, \beta \in \mathcal{I}$ . MFGs involve infinite homogeneous players. All the players employ the same policy and thus share the same distribution flow. The aggregate in Eqn. 1 degenerates to  $z_h^\alpha = \int_0^1 p \cdot \mu_h^\beta \mathrm{d}\beta = p \cdot \mu_h$  for all  $\alpha \in \mathcal{I}$ . Here  $\mu_h$  is the state distribution of a representative player. Thus, an MFG is denoted by a tuple  $(\bar{S}, \bar{A}, \bar{H}, \bar{P}, \bar{r}, \mu_1)$ . The state space, the action space, and the horizon are respectively denoted as  $\bar{S}$ ,  $\bar{A}$ , and  $\bar{H}$ . Here, the transition kernels  $\bar{P} = \{\bar{P}_h\}_{h=1}^H$  are functions  $\bar{P}_h : S \times A \times \Delta(S) \to S$ , and reward functions  $\bar{r}_h : S \times A \times \Delta(S) \to [0,1]$  for all  $h \in [H]$ . In the MFG, all the players adopt the same policy  $\pi = \{\pi_h\}_{h=1}^H$  where  $\pi_h \in \Pi$ . The value function and the action-value function in the  $\lambda$ -regularized MFG with the underlying distribution flow  $\mu = \{\mu_h\}_{h=1}^H \in \Delta(S)^H$  can be similarly defined as Eqn. (2) and (3) respectively as follows
$$
\bar {V} _ {h} ^ {\lambda} (s, \pi , \mu) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = h} ^ {H} \bar {r} _ {t} \left(s _ {t}, a _ {t}, \mu_ {t}\right) - \lambda \log \pi_ {t} \left(a _ {t} \mid s _ {t}\right) \Bigg | s _ {h} = s \right],
$$
$$
\bar {Q} _ {h} ^ {\lambda} (s, a, \pi , \mu) = \bar {r} _ {h} (s, a, \mu_ {h}) + \mathbb {E} ^ {\pi} \big [ \bar {V} _ {h + 1} ^ {\lambda} (s _ {h + 1}, \pi , \mu) | s _ {h} = s, a _ {h} = a \big ]
$$
for all  $h \in [H]$ . The cumulative reward is  $\bar{J}^{\lambda}(\pi, \mu) = \mathbb{E}_{\mu_1}[\bar{V}_1^{\lambda}(s, \pi, \mu)]$ . The notion of NE can be similarly derived as follows.
Definition 2. An NE of the  $\lambda$ -regularized MFG is a pair  $(\pi^{*}, \mu^{*}) \in \Pi^{H} \times \Delta(\mathcal{S})^{H}$  that satisfies: (i) (player rationality)  $J^{\lambda}(\pi^{*}, \mu^{*}) = \max_{\tilde{\pi} \in \Pi^{H}} J^{\lambda}(\tilde{\pi}, \mu^{*})$ . (ii) (Distribution consistency) The distribution flow  $\mu^{*}$  is equal to the distribution flow induced by the policy  $\pi^{*}$ .
Remark 1. Compared with Definition 1, the definition of NE in MFG only involves the policy and the distribution flow of a single representative player, since the agents are homogeneous in MFGs.
4 Existence of the NEs in Regularized GMFGs and MFGs
We now state some assumptions to demonstrate the existence of a NE for  $\lambda$ -regularized GMFGs.
Assumption 1. The state space  $S$  is compact, and the action space  $\mathcal{A}$  is finite.
This assumption imposes rather mild constraints on  $S$  and  $\mathcal{A}$ . In real-world applications, the states are usually physical quantities and thus reside in a compact set. For the action space, many deep reinforcement learning algorithms discretize the potential continuous action sets into finite sets [Lowe et al., 2017, Mordatch and Abbeel, 2018].
Assumption 2. The graphons  $W_{h}$  for  $h\in [H]$  are continuous functions.
The stronger version of this assumption (Lipschitz continuity) is widely adopted in GMFG works [Cui and Koeppl, 2021b, Fabian et al., 2022]. It helps us to build the continuity of the transition kernels and rewards with respect to players.
Assumption 3. For all  $h \in [H]$ , the reward function  $r_h(s, a, z)$  is continuous on  $S \times \mathcal{A} \times \mathcal{M}(S)$ , that is if  $(s_n, a_n, z_n) \to (s, a, z)$  as  $n \to \infty$ , then  $r_h(s_n, a_n, z_n) \to r_h(s, a, z)$ . The transition kernel  $P_h(\cdot | s, a, z)$  is weakly continuous in  $S \times \mathcal{A} \times \mathcal{M}(S)$ , that is if  $(s_n, a_n, z_n) \to (s, a, z)$  as  $n \to \infty$ ,  $P_h(\cdot | s_n, a_n, z_n) \to P_h(\cdot | s, a, z)$  weakly.
This assumption states the continuity of the models, i.e., the transition kernels and the rewards, as functions of the state, action, and aggregate. We note that the Lipschitz continuity assumption of the model in the previous works implies that our assumption is satisfied [Cui and Koeppl, 2021b, Fabian et al., 2022]. Next, we state the existence result of regularized GMFG.
Theorem 1. Under Assumptions 1, 2 and 3, for all  $\lambda \geq 0$ , the  $\lambda$ -regularized GMFG  $(\mathcal{I},\mathcal{S},\mathcal{A},H,P,r,W,\mu_1)$  admits an NE  $(\pi^{\mathcal{I}},\mu^{\mathcal{I}}) \in \tilde{\Pi} \times \tilde{\Delta}$ .
This theorem strengthens previous existence results in Cui and Koeppl [2021b] and Fabian et al. [2022] in two aspects. First, our assumptions are weaker. These two existing works require a finite state space and the Lipschitz continuity of the models. In contrast, we only need a compact state space and the model continuity. Second, their results only hold for the unregularized case  $(\lambda = 0)$ , whereas ours holds for any  $\lambda \geq 0$ . In the proof of Theorem 1, we construct a MFG from the given GMFG and show that an NE of the constructed MFG can be converted to an NE of the GMFG. Then we prove the existence of NE in the constructed regularized MFG. Our existence result for the regularized MFG is also a significant addition to the MFG literature.
Remark 2. Although we show that an NE of the constructed MFG can be converted to an NE of GMFG, this proof does not imply that GMFG forms a subclass of or is equivalent to MFG. This is because we have only demonstrated the relationship between the NEs of these two games, but the exact realizations of the GMFG and the conceptually constructed MFG may differ. It means that the sample paths of these two games may not be the same, which include the realizations of the states, actions, and rewards of all the players.
We next state the assumption needed for the MFG.
Assumption 4. The MFG  $(\bar{S},\bar{\mathcal{A}},\bar{H},\bar{P},\bar{r},\mu_1)$  satisfies that: (i) The state space  $\bar{S}$  is compact, and the action space  $\bar{\mathcal{A}}$  is finite. (ii) The reward functions  $\bar{r}_h(s,a,\mu)$  for  $h\in [H]$  are continuous on  $\mathcal{S}\times \mathcal{A}\times \Delta (\mathcal{S})$ . The transition kernels are weakly continuous on  $\mathcal{S}\times \mathcal{A}\times \Delta (\mathcal{S})$ ; that is if  $(s_n,a_n,\mu_n)\to (s,a,\mu)$  as  $n\to \infty$ ,  $\bar{P}_h(\cdot |s_n,a_n,\mu_n)\to \bar{P}_h(\cdot |s,a,\mu)$  weakly.
Then the existence of the NE is stated as follows.
Theorem 2. Under Assumption 4, the  $\lambda$ -regularized MFG  $(\bar{S},\bar{A},\bar{H},\bar{P},\bar{r},\mu_1)$  admits an  $NE(\pi ,\mu)\in \Pi^H\times \Delta (\mathcal{S})^H$  for any  $\lambda \geq 0$ .
Our result in Theorem 2 imposes weaker conditions than previous works [Cui and Koeppl, 2021a, Anahtarci et al., 2022] to guarantee the existence of an NE. These existing works prove the existence of NE by assuming a contractive property and the finiteness of the state space. They also require a strong Lipschitz assumption [Anahtarci et al., 2022], where the Lipschitz constants of the models should be small enough. In contrast, we only require the continuity assumption in Theorem 2. This is due to our analysis of the operator for the regularized MFG and the use of Kakutani fixed point theorem Guide [2006].
5 Learning NE of Monotone GMFGs