SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we focus on GMFGs with transition kernels that are independent of the aggregate  $z$ , i.e.,  $P_{h}: S \times \mathcal{A} \to S$  for  $h \in [H]$ . This model is motivated by the seminal work Lasry and Lions [2007], where the state evolution in continuous-time MFG is characterized by the Fokker-Plank equation. However, the form of the Fokker-Plank equation results in the state transition of each player being independent of other players. This model is also widely accepted in the discrete-time GMFG and MFG literature [Fabian et al., 2022, Perrin et al., 2020, Perolat et al., 2021].
5.1 Monotone GMFG
We first generalize the notion of monotonicity from multi-population MFGs in Perolat et al. [2021] to GMFGs.
Definition 3 (Weakly Monotone Condition). A GMFG is said to be weakly monotone if for any  $\rho^{\mathcal{I}},\tilde{\rho}^{\mathcal{I}}\in \Delta (\mathcal{S}\times \mathcal{A})^{\mathcal{I}}$  and their marginalizations on the states  $\mu^{\mathcal{I}},\tilde{\mu}^{\mathcal{I}}\in \Delta (\mathcal{S})^{\mathcal{I}}$ , we have
$$
\int_ {\mathcal {I}} \sum_ {a \in \mathcal {A}} \int_ {\mathcal {S}} \left(\rho^ {\alpha} (s, a) - \tilde {\rho} ^ {\alpha} (s, a)\right) \Big (r _ {h} \big (s, a, z _ {h} ^ {\alpha} (\mu^ {\mathcal {I}}, W _ {h}) \big) - r _ {h} \big (s, a, z _ {h} ^ {\alpha} (\tilde {\mu} ^ {\mathcal {I}}, W _ {h}) \big) \Big) \mathrm {d} s \mathrm {d} \alpha \leq 0
$$
for all  $h \in [H]$ , where  $W_h$  is the underlying graphon. It is strictly weakly monotone if the inequality is strict when  $\rho^{\mathcal{I}} \neq \tilde{\rho}^{\mathcal{I}}$ .
In two MDPs induced by the distribution flows  $\mu^{\mathcal{I}}$  of  $\pi^{\mathcal{I}}$  and  $\tilde{\mu}^{\mathcal{I}}$  of  $\tilde{\pi}^{\mathcal{I}}$ , the weakly monotone condition states that we can achieve higher rewards at stage  $h \in [H]$  by swapping the policies. This condition has two important implications. The first is the uniqueness of the NE.
Proposition 1. Under Assumptions 1, 2, and 3, a strictly weakly monotone  $\lambda$ -regularized GMFG has a unique NE for any  $\lambda \geq 0$  up to a zero-measure set on  $\mathcal{I}$  with respect to the Lebesgue measure.
In the following, we denote this unique NE as  $(\pi^{*,\mathcal{I}},\mu^{*,\mathcal{I}})$ , and we aim to learn this NE. The second implication concerns the relationship between the cumulative rewards of two policies.
Proposition 2. If a  $\lambda$ -regularized GMFG satisfies the weakly monotone condition, then for any two policies  $\pi^{\mathcal{I}}$ ,  $\tilde{\pi}^{\mathcal{I}} \in \tilde{\Pi}$  and their induced distribution flows  $\mu^{\mathcal{I}}$ ,  $\tilde{\mu}^{\mathcal{I}} \in \tilde{\Delta}$ , we have
$$
\int_ {0} ^ {1} J ^ {\lambda , \alpha} (\pi^ {\alpha}, \mu^ {\mathcal {I}}) + J ^ {\lambda , \alpha} (\tilde {\pi} ^ {\alpha}, \tilde {\mu} ^ {\mathcal {I}}) - J ^ {\lambda , \alpha} (\tilde {\pi} ^ {\alpha}, \mu^ {\mathcal {I}}) - J ^ {\lambda , \alpha} (\pi^ {\alpha}, \tilde {\mu} ^ {\mathcal {I}}) d \alpha \leq 0
$$
If the  $\lambda$ -regularized GMFG satisfies the strictly weakly monotone condition, then the inequality is strict when  $\pi^{\mathcal{T}} \neq \tilde{\pi}^{\mathcal{T}}$ .
Proposition 2 shows that if we have two policies, we can improve the cumulative rewards on the MDP induced by these policies by swapping the policies. This implies an important property of the NE  $(\pi^{*,\mathcal{I}},\mu^{*,\mathcal{I}})$ . Since  $\pi^{*,\mathcal{I}}$  is optimal on the MDP induced by  $\mu^{*,\mathcal{I}}$ , we have  $\int_0^1 J^{\lambda ,\alpha}(\pi^{*,\alpha},\mu^{*,\mathcal{I}})\mathrm{d}\alpha \geq \int_0^1 J^{\lambda ,\alpha}(\pi^\alpha ,\mu^{*,\mathcal{I}})\mathrm{d}\alpha$  for any  $\pi^{\mathcal{I}}\in \tilde{\Pi}$ . Then Proposition 2 shows that  $\int_0^1 J^{\lambda ,\alpha}(\pi^{*,\alpha},\mu^{\mathcal{I}})\mathrm{d}\alpha \geq \int_0^1 J^{\lambda ,\alpha}(\pi^\alpha ,\mu^{\mathcal{I}})\mathrm{d}\alpha$  for any policy  $\pi^{\mathcal{I}}$  and the distribution flow  $\mu^{\mathcal{I}}$  it induces. This means that the NE policy gains cumulative rewards not less than any policy  $\pi^{\mathcal{I}}$  on the MDP induced by  $\pi^{\mathcal{I}}$ . This motivates the design of our NE learning algorithm.
5.2 Policy Mirror Descent Algorithm for Monotone GMFG
In this section, we introduce the algorithm to learn the NE, which is called MonoGMFG-PMD and whose pseudo-code is outlined in Algorithm 1. It consists of three main steps. The first step estimates the action-value function (Line 3). We need to evaluate the action-value function of a policy on the MDP induced by itself. This estimate can be obtained for each player independently by playing the  $\pi_t^{\mathcal{I}}$  several times. We assume access to a sub-module for this and quantify the estimation error in our analysis. The second step is the policy mirror descent (Line 4). Given  $\lambda \eta_t < 1$ , This step can be equivalently formulated as
$$
\hat {\pi} _ {t + 1, h} ^ {\alpha} (\cdot | s) = \underset {p \in \Delta (\mathcal {A})} {\operatorname {a r g m a x}} \frac {\eta_ {t}}{1 - \lambda \eta_ {t}} \left[ \left\langle \hat {Q} _ {h} ^ {\lambda , \alpha} (s, \cdot , \pi_ {t} ^ {\alpha}, \mu_ {t} ^ {\mathcal {I}}), p \right\rangle - \lambda R (p) \right] - D _ {\mathrm {k l}} \big (p \| \pi_ {t, h} ^ {\alpha} (\cdot | s) \big) \quad \forall s \in \mathcal {S},
$$
Algorithm 1 MonoGMFG-PMD
Procedure:
1: Initialize  $\pi_{1,h}^{\alpha}(\cdot |s) = \mathrm{Unif}(\mathcal{A})$  for all  $s\in S,h\in [H]$  and  $\alpha \in \mathcal{I}$
2: for  $t = 1,2,\dots ,T$  do
3: Compute the action-value function  $\hat{Q}_h^{\lambda ,\alpha}(s,a,\pi_t^\alpha ,\mu_t^{\mathcal{I}})$  for all  $\alpha \in \mathcal{I}$  and  $h\in [H]$ , where  $\mu_t^{\mathcal{I}}$  is the distribution flow induced by  $\pi_t^{\mathcal{I}}$ .
4:  $\hat{\pi}_{t+1,h}^{\alpha}(\cdot|s) \propto (\pi_{t,h}^{\alpha}(\cdot|s))^{1-\lambda\eta_t}\exp(\eta_t\hat{Q}_h^{\lambda,\alpha}(s,a,\pi_t^\alpha,\mu_t^\mathcal{T}))$  for all  $\alpha \in \mathcal{I}$  and  $h \in [H]$ .
5:  $\pi_{t + 1,h}^{\alpha}(\cdot |s) = (1 - \beta_{t})\hat{\pi}_{t + 1,h}^{\alpha}(\cdot |s) + \beta_{t}\mathrm{Unif}(\mathcal{A})$
6: end for
7: Output  $\bar{\pi}^{\mathcal{I}} = \mathrm{Unif}\bigl (\pi_{[1:T]}^{\mathcal{I}}\bigr)$
where  $R(p) = \langle p, \log p \rangle$  is the negative entropy function. This step aims to improve the performance of the policy  $\pi_t^{\mathcal{I}}$  on its own induced MDP. Intuitively, since the policy  $\pi_{t}^{*,\mathcal{I}}$  in NE performs better than  $\pi_t^{\mathcal{I}}$  on the MDP induced by  $\mu_t^{\mathcal{I}}$  as shown in Section 5.1, the improved policy  $\pi_{t + 1}^{\mathcal{I}}$  should be closer to  $\pi^{*,\mathcal{I}}$  than  $\pi_t^{\mathcal{I}}$ . The third step mixes the current policy with the uniform policy (Line 5) to encourage exploration.
MonoGMFG-PMD is different from previous NE learning algorithms for monotone GMFG in Perolat et al. [2021], Fabian et al. [2022] in three different ways. First, MonoGMFG-PMD is designed to learn the NE of the  $\lambda$ -regularized GMFG with  $\lambda > 0$ , whereas other algorithms learn the NE of the unregularized GMFGs. As a result, our policy improvement (Line 4) discounts the previous policy as  $(\pi_{t,h}^{\alpha})^{1 - \lambda \eta_t}$ , but other algorithms retain  $\pi_{t,h}^{\alpha}$ . Second, our algorithm is discrete-time and thus amenable for practical implementation. However, other provably efficient algorithms evolve in continuous time. Finally, MonoGMFG-PMD encourages exploration in Line 5, which is important for the theoretical analysis. Such a step is missing in other algorithms.
5.3 Theoretical Analysis for MonoGMFG-PMD with Estimation Oracle
This section provides theoretical analysis for the MonoGMFG-PMD algorithm given an action-value function oracle in Line 3, i.e.,  $\hat{Q}_h^{\lambda,\alpha} = Q_h^{\lambda,\alpha}$ . We denote the unique NE of the  $\lambda$ -regularized GMFG as  $(\pi^{*,\mathcal{I}}, \mu^{*,\mathcal{I}})$ . For any policy  $\pi^{\mathcal{I}}$ , we measure the distance to the policy  $\pi^{*,\mathcal{I}}$  of NE using
$$
D (\pi^ {\mathcal {I}}) = \int_ {0} ^ {1} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mu_ {h} ^ {*, \alpha}} \Big [ D _ {\mathrm {k l}} \big (\pi_ {h} ^ {*, \alpha} (\cdot | s _ {h} ^ {\alpha}) \| \pi_ {h} ^ {\alpha} (\cdot | s _ {h} ^ {\alpha}) \big) \Big ] \mathrm {d} \alpha .
$$
This metric measures the weighted KL divergence between policy  $\pi^{\mathcal{I}}$  and the NE policy, and the weights are the NE distribution flow  $\mu^{*,\mathcal{I}}$ .
Theorem 3. Assume that the GMFG is strictly weakly monotone and we have an action-value function oracle. Let  $\eta_t = \eta = O(T^{-1/2})$  and  $\beta_t = \beta = O(T^{-1})$  in MonoGMFG-PMD. For any  $\lambda > 0$  we have
$$
D \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \pi_ {t} ^ {\mathcal {I}}\right) = O \left(\frac {\lambda \log^ {2} T}{T ^ {1 / 2}}\right).
$$
Theorem 3 provides the first convergence rate result for a discrete-time algorithm on strictly weakly monotone GMFGs under mild assumptions. In contrast, Perolat et al. [2021], Fabian et al. [2022] only consider the continuous-time policy mirror descent, which is difficult for the practical implementation, and only provide the asymptotic consistency results. Geist et al. [2021] derive exploitability results for a fictitious play algorithm but require the potential structure and the Lipschitzness of the reward function. Our proof for Theorem 3 mainly exploits properties of NE discussed in Section 5.1. Concretely, we use the fact that the policy mirror descent procedure reduces the distance between the policy iterate and the NE as  $D(\pi_{t + 1}^{\mathcal{I}}) - D(\pi_t^{\mathcal{I}})\approx \int_0^1 J^{\lambda ,\alpha}(\pi_t^\alpha ,\mu_t^{\mathcal{I}}) - J^{\lambda ,\alpha}(\pi^{*,\alpha},\mu_t^{\mathcal{I}})\mathrm{d}\alpha$ . Thus, the policy iterate becomes closer to the NE. However, the discretization error and the exploration influence (Line 5) also appear, requiring additional care to show that  $D(\pi_t^{\mathcal{I}})$ , in general, decreases.
Algorithm 2 Estimation of Action-value Function
1: Sample  $N$  players  $\{i / N\}_{i = 1}^{N}\subseteq [0,1]$
2: The  $i^{\mathrm{th}}$  player implements  $\pi_t^{\mathrm{b},i}$  for  $i\in [N]$ , and the other players implement  $\pi_t^\mathcal{T}$ .
3: Collect data  $\{(s_{\tau ,h}^i,a_{\tau ,h}^i,r_{\tau ,h}^i)\}_{i,\tau ,h = 1}^{N,K,H}$  of sampled players from  $K$  independent episodes.
4: Initialize  $\hat{V}_{H + 1}^{\lambda ,i}(s,a) = 0$  for all  $s\in S,a\in \mathcal{A}$  and  $i\in [N]$
5: for time  $h = H, \dots, 1$  do
6: for Player  $i = 1,\dots ,N$  (in parallel) do
7:  $\hat{Q}_h^{\lambda ,i} = \operatorname *{argmin}_{f\in \mathcal{F}_h}\sum_{\tau = 1}^{K}\left(f(s_{\tau ,h}^i,a_{\tau ,h}^i) - r_{\tau ,h}^i -\hat{V}_{h + 1}^{\lambda ,i}(s_{\tau ,h + 1}^i)\right)^2.$
8:  $\hat{V}_h^{\lambda ,i}(s) = \langle \hat{Q}_h^{\lambda ,i}(s,\cdot),\pi_{t,h}^{i / N}(\cdot ,s)\rangle -\lambda R\bigl (\pi_{t,h}^{i / N}(\cdot ,s)\bigr).$
9: end for
10: end for
11: Output  $\{\hat{Q}_h^{\lambda ,i}\}_{i,h = 1}^{N,H}$
5.4 Theoretical Analysis for MonoGMFG-PMD with General Function Classes
In this section, we remove the requirement that one is given oracle access to an action-value function and we estimate it in Line 3 of MonoGMFG-PMD using general function classes. We consider the action-value function class  $\mathcal{F} = \mathcal{F}_1\times \dots \times \mathcal{F}_H$ , where  $\mathcal{F}_h\subseteq \{f:\mathcal{S}\times \mathcal{A}\to [0,(H - h + 1)(1 + \lambda \log |\mathcal{A}|)]\}$  is the class of action-value functions at time  $h\in [H]$ . Then we estimate the action-value functions using Algorithm 2.
This algorithm mainly involves two steps. The first is involves data collection (Line 3). Here we assign policies to players and collect data from their interactions. We let the sampled  $N$  players implement behavior policies  $\{\pi_t^{b,i}\}_{i=1}^N$ , which can be different from  $\{\pi_t^{i/N}\}_{i=1}^N$ . This will not change the aggregate  $z_h^\alpha(\mu_h^{\mathcal{I}}, W_h)$  for any  $\alpha \in \mathcal{I}$ , since only a zero-measure set of players change their policies. The second is the action-value function estimation (Lines 7 and 8). The action-value function is selected based on the previous value function, and the value function is updated from the derived estimate. This can be implemented in parallel for all players. We highlight that the estimation analysis cannot leverage results from general non-parametric regression [Wainwright, 2019], since the response variable  $\hat{V}_{h+1}^{\lambda,i}$  is not independent of  $s_{\tau,h+1}^i$  in our setting.
Assumption 5 (Realizability). For any policy  $\pi^{\mathcal{I}} \in \tilde{\Pi}$  and the induced distribution flow  $\mu^{\mathcal{I}} \in \tilde{\Delta}$ , we have  $Q_h^{\lambda, \alpha}(\cdot, \cdot, \pi^\alpha, \mu^{\mathcal{I}}) \in \mathcal{F}_h$  for  $h \in [H]$ .
This assumption ensures that we can find the nominal action-value function in the function class. For a policy  $\pi \in \Pi^H$  and a function  $f: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ , we define the operator  $(\mathcal{T}_h^\pi f)(s,a) = \mathbb{E}_{s' \sim P_h(\cdot|s,a)}[\langle f(s',\cdot), \pi_{h+1}(\cdot|s') \rangle - \lambda R(\pi_{h+1}(\cdot|s'))]$ . For a policy  $\pi^{\mathcal{I}}$  and the induced distribution flow  $\mu^{\mathcal{I}}$ , we have  $Q_h^{\lambda,\alpha}(s,a,\pi^\alpha,\mu^{\mathcal{I}}) = r_h(s,a,z_h^\alpha) + (\mathcal{T}_h^{\pi^\alpha}Q_{h+1}^{\lambda,\alpha})(s,a)$ .
Assumption 6 (Completeness). For any policy  $\pi^{\mathcal{I}} \in \tilde{\Pi}$  and the induced distribution flow  $\mu^{\mathcal{I}} \in \tilde{\Delta}$ , we have that for all  $f \in \mathcal{F}_{h+1}$ ,  $r_h(\cdot, \cdot, z_h^\alpha(\mu_h^{\mathcal{I}}, W_h)) + (\mathcal{T}_h^{\pi^\alpha} f) \in \mathcal{F}_h$  for all  $\alpha \in \mathcal{I}$ ,  $h \in [H-1]$ .
This completeness assumption ensures that the estimates from  $\mathcal{F}$  also satisfy the relationship between nominal action-value functions through  $\mathcal{T}_h^{\pi^\alpha}$ . These realizability and completeness assumptions are widely adopted in the off-policy evaluation and offline reinforcement learning literature [Uehara et al., 2022, Xie et al., 2021b].
Assumption 7. The reward functions  $\{r_h\}_{h=1}^H$  are Lipschitz in  $z$ , i.e.,  $|r_h(s, a, z) - r_h(s, a, z')| \leq L_r \|z - z'\|_1$  for all  $s \in S$ ,  $a \in \mathcal{A}$ ,  $h \in [H]$ . The graphons  $W_h$  for  $h \in [H]$  are Lipschitz continuous functions, i.e., there exists a constant  $L = L_W > 0$  (depending only on  $W = \{W_h\}_{h=1}^H$ ) such that  $|W_h(\alpha, \beta) - W_h(\alpha', \beta')| \leq L_W(|\alpha - \alpha'| + |\beta - \beta'|)$  for all  $\alpha, \alpha', \beta, \beta' \in [0,1]$ , and  $h \in [H]$ .
The Lipschitzness assumption is common in the GMFG works [Parise and Ozdaglar, 2019, Carmona et al., 2022, Cui and Koeppl, 2021b]. It helps us to approximate the action-value function of a player by that of sampled players. We denote the state distributions of player  $i$  induced by policy  $\pi_t^{\mathrm{b},i}$  as  $\mu_t^{\mathrm{b},i}$ . Then we require the behavior policies  $\{\pi_t^{\mathrm{b},i}\}_{i=1}^N$  to satisfy the following requirements.
(a) Beach Bar problem with SBM graphons
(b) Beach Bar problem with exp-graphons.
Figure 1: Simulation results for Beach Bar problem with SBM and exp-graphons.
Assumption 8. For any  $t \in [T]$ , the behavior policies explore sufficiently. More precisely, for any policy  $\pi \in \Pi^H$  and induced distributions  $\mu \in \Delta(\mathcal{S})^H$ , we have  $\sup_{s \in \mathcal{S}, a \in \mathcal{A}} \pi_h(a|s) / \pi_{t,h}^{\mathrm{b},i}(a|s) \leq C_1$  and  $\sup_{s \in \mathcal{S}} \mathrm{d}\mu_h(s) / \mathrm{d}\mu_{t,h}^{\mathrm{b},i}(s) \leq C_2$  for  $h \in [H]$  and  $i \in [N]$  where  $C_1, C_2 > 0$  are constants.
This assumption guarantees that the behavior policies explore the actions that may be adopted by  $\pi_t^{\mathcal{I}}$  and  $\pi^{*,\mathcal{I}}$ . Such an assumption is widely adopted in offline reinforcement learning and off-policy evaluation works [Uehara et al., 2022, Xie et al., 2021b].
Theorem 4. Assume that the GMFG is weakly monotone and that Assumptions 5, 6, 7, and 8 hold. Let  $\eta_t = \eta = O(T^{-1/2})$  and  $\beta_t = \beta = O(T^{-1})$  in Algorithm 1 (MonoGMFG-PMD). Then with probability at least  $1 - \delta$ , Algorithms 1 and 2 yield
$$
D \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \pi_ {t} ^ {\mathcal {T}}\right) = O \left(\frac {\lambda \log^ {2} T}{\sqrt {T}} + C _ {1} C _ {2} \frac {H ^ {3 / 2} B _ {H} ^ {2}}{\lambda \sqrt {K}} \log \frac {T N H \cdot \mathcal {N} _ {\infty} (5 B _ {H} / K , \mathcal {F} _ {[ H ]})}{\delta} + \frac {H \log T}{N}\right),
$$
where  $B_H = H(1 + \lambda \log |\mathcal{A}|)$ , and  $\mathcal{N}_{\infty}(5B_H / K,\mathcal{F}_{[H]}) = \max_{h\in [H]}\mathcal{N}_{\infty}(5B_H / K,\mathcal{F}_h)$  is the  $\ell_{\infty}$ -covering number of the function class.
The error in Theorem 4 consists of both the optimization and estimation errors. The optimization error corresponds to the first term, which also appears in Theorem 3. The estimation error consists of the generalization error and the approximation error, in the second and third terms respectively. When the function class  $\mathcal{F}$  is finite, this term scales as  $O(K^{-1/2})$ , which originates from the fact that we estimate the action-value function from the empirical error instead of its population counterpart. The approximation error scales as  $O(N^{-1})$ . This term originates from the fact that the action-value function of player  $\alpha$  is approximated by that of the sampled player near  $\alpha$ . To learn a policy that is at most  $\varepsilon > 0$  far from the NE, we can set  $T = \tilde{O}(\varepsilon^{-2})$ ,  $K = \tilde{O}(\varepsilon^{-2})$ , and  $N = \tilde{O}(\varepsilon^{-1})$ , which in total results in  $TK = \tilde{O}(\varepsilon^{-4})$  episodes of online plays.
In this section, we conduct experiments to corroborate our theoretical findings. We run different algorithms on the Beach Bar problem Perrin et al. [2020], Fabian et al. [2022]. The underlying graphons are set to Stochastic Block Model (SBM) and exp-graphons. The details of experiments are deferred to Appendix B. Since the NEs of the games are not available, we adopt the exploitability to measure the proximity between a policy and the NE. For a policy  $\pi^{\mathcal{I}}$  and its induced distribution flow  $\mu^{\mathcal{I}}$ , the exploitability for the  $\lambda$ -regularized GMFG is defined as
$$
\operatorname {E x p l o i t} (\pi^ {\mathcal {I}}) = \int_ {0} ^ {1} \max  _ {\tilde {\pi} \in \Pi^ {H}} J ^ {\lambda , \alpha} (\tilde {\pi}, \mu^ {\mathcal {I}}) - J ^ {\lambda , \alpha} (\pi^ {\alpha}, \mu^ {\mathcal {I}}) \mathrm {d} \alpha .
$$
First, the experimental results demonstrate the necessity of modelling the heterogeneity of agents. Figure 1 demonstrates the performance degradation of approximating GMFG by MFG. Here we let the agents play in the GMFG with constant graphons  $W_{h}(\alpha ,\beta) = p$  for  $p\in \{0,0.5,1\}$ . The agents have oracle access to the action-value function. We observe that this approximation results in gross errors for learning the NEs of GMFGs with non-constant graphons.
Second, the experiments show that the algorithms designed for unregularized GMFG cannot learn the NE of regularized GMFG. We implement the discrete-time version of the algorithm in Fabian et al.
[2022]; results marked "Unreg PMD" show that the exploitability first decreases and then increases. In line with the discussion in Section 5.2, this originates from keeping too much gradient knowledge in previous iterates  $\pi_t^{\mathcal{T}}$ . The gradient of the policy is largely correct in the several initial iterations, but a large amount of past knowledge results in it deviating in later iterations, since the past knowledge accumulates. In contrast, our algorithm discounts the past knowledge as  $(\pi_t^{\mathcal{T}})^{1 - \lambda \eta_t}$ .
Finally, the results indicate the influence of action-value function estimation. In the experiments, we run our algorithm when  $N = 5$ ,  $K = 300$ ,  $N = 10$ ,  $K = 100$ , and  $N = 10$ ,  $K = 300$ . Figure 1 shows that the algorithm with  $N = 10$ ,  $K = 300$  can achieve a smaller error than the algorithms both with  $N = 10$ ,  $K = 100$  and  $N = 5$ ,  $K = 300$ . This is in agreement with Theorem 4.
In this paper, we focused on two fundamental problems of  $\lambda$ -regularized GMFG. Firstly, we established the existence of NE. This result greatly weakened the conditions in the previous works. Secondly, the provably efficient NE learning algorithms were proposed and analyzed in the weakly monotone GMFG motivated by Lasry and Lions [2007]. The convergence rate of MonoGMFG-PMD features the first performance guarantee of discrete-time algorithm without extra conditions in monotone GMFGs. We leave the lower bound of this problem to the future works.
Fengzhuo Zhang and Vincent Tan acknowledge funding by the Singapore Data Science Consortium (SDSC) Dissertation Research Fellowship, the Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 under grant number A-8000423-00-00, and AcRF Tier 1 under grant numbers A-8000980-00-00 and A-8000189-01-00. Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their support.
Y. Achdou, P. Cardaliaguet, F. Delarue, A. Porretta, F. Santambrogio, Y. Achdou, and M. Laurière. Mean field games and applications: Numerical aspects. *Mean Field Games: Cetraro, Italy* 2019, pages 249–307, 2020.
B. Anahtarci, C. D. Kariksiz, and N. Saldi. Fitted q-learning in mean-field games. arXiv preprint arXiv:1912.13309, 2019.
B. Anahtarci, C. D. Kariksiz, and N. Saldi. Q-learning in regularized mean-field games. Dynamic Games and Applications, pages 1-29, 2022.
A. Aurell, R. Carmona, G. Dayanikl, and M. Lauriere. Finite state graphon games with applications to epidemics. Dynamic Games and Applications, 12(1):49-81, 2022a.
A. Aurell, R. Carmona, and M. Lauriere. Stochastic graphon games: II. the linear-quadratic case. Applied Mathematics & Optimization, 85(3):1-33, 2022b.
D. Bertsekas and S. E. Shreve. Stochastic optimal control: the discrete-time case, volume 5. Athena Scientific, 1996.
Q. Cai, Z. Yang, C. Jin, and Z. Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
P. E. Caines and M. Huang. Graphon mean field games and the gmfg equations:  $\varepsilon$ -nash equilibria. In 2019 IEEE 58th conference on decision and control (CDC), pages 286-292. IEEE, 2019.
R. Carmona, D. B. Cooney, C. V. Graves, and M. Lauriere. Stochastic graphon games: I. the static case. Mathematics of Operations Research, 47(1):750-778, 2022.
S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4):2563-2578, 2022.