SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to establish convergence to (approximate) NE, we need to show that our optimistic OMD policy updates achieve "no regret" with respect to the value estimate sequence at each state, i.e., to upper bound (3). If we were to use the celebrated  $\alpha_{t} = \frac{H + 1}{H + t}$  learning rate [32] to update the value function estimates, we will inevitably need to show a no-weighted-regret guarantee for optimistic OMD, because such a time-varying learning rate assigns non-uniform weights to each history step. However, incorporating OMD with a dynamic learning rate is known to be challenging and can easily lead to linear regret [50]. While a stabilization technique [19] has been introduced to tackle this challenge, we take a different route by resorting to an alternative value update method, namely stage-based value updates [79]. Specifically, we divide the total  $T$  iterations into multiple stages and only update our value estimates at the end of a stage (Line 9). We let the lengths of the stages grow exponentially at a rate of  $(1 + 1 / H)$  (Line 8) [79, 45]. The exponential growth ensures that the total  $T$  iterations can be covered by a small number of stages, while the  $(1 + 1 / H)$  growth rate guarantees that the value estimation error does not blow up during the  $H$  steps of recursion (Lemma 9). Compared with the incremental  $\alpha_{t} = \frac{H + 1}{H + t}$  update rule that modifies the value estimates at every step, stage-based updates are more stationary and allow us to assign uniform weights to each history step. This leads to a simpler no(-average)-regret problem [47] that can be easily addressed by (optimistic) OMD.
We introduce a few notations before presenting the convergence analysis of Algorithm 1. Let  $\tau(t)$  denote the index of the stage that iteration  $t$  belongs to. We denote by  $\bar{\tau}$  the total number of stages, i.e.,  $\bar{\tau} := \tau(T)$ . For any  $(\tau, h, s) \in [\bar{\tau}] \times [H] \times S$ , define the per-state regrets for the max-player as
$$
\operatorname {r e g} _ {h, 1} ^ {\tau} (s) := \max  _ {\mu_ {h} ^ {\tau , \dagger} (\cdot | s) \in \Delta (\mathcal {A})} \frac {1}{L _ {\tau}} \sum_ {j = t _ {\tau} ^ {\text {s t a r t}}} ^ {t _ {\tau} ^ {\text {e n d}}} \left\langle \mu_ {h} ^ {\tau , \dagger} - \mu_ {h} ^ {j}, Q _ {h} ^ {\tau} \nu_ {h} ^ {j} \right\rangle (s). \tag {3}
$$
The per-state regret  $\mathrm{reg}_{h,2}^{\tau}(s)$  for the min-player can be defined symmetrically (see (14) in Appendix B). We define the maximal regret (over the states and the two players) as  $\mathrm{reg}_h^\tau \coloneqq \max_{s\in S}\max_{i = 1,2}\{\mathrm{reg}_{h,i}^\tau (s)\}$ . An upper bound for the per-state regrets is provided in Lemma 8 of Appendix B, which is useful in the analysis of Algorithm 1. We use the standard notion of
$$
\mathrm {N E - g a p} (\mu , \nu) := V _ {1} ^ {\dagger , \nu} (s _ {1}) - V _ {1} ^ {\mu , \dagger} (s _ {1})
$$
to measure the optimality of a policy pair  $(\mu, \nu)$ . The initialization-dependent convergence rate of Algorithm 1 is as follows.
Theorem 1. If Algorithm 1 is run on a two-player zero-sum Markov game for  $T$  iterations with a learning rate  $\eta \leq 1 / (8H^2)$ , the output policy pair  $(\bar{\mu},\bar{\nu})$  satisfies:
$$
\mathrm {N E - g a p} (\bar {\mu}, \bar {\nu}) \leq \frac {1 9 2 H ^ {3}}{T} \sum_ {h = 1} ^ {H} \sum_ {\tau = 1} ^ {\bar {\tau}} \max _ {s} \left(D _ {R} \left(\mu_ {h} ^ {\tau , \dagger} (\cdot | s), \tilde {\mu} _ {h} ^ {\tau} (\cdot | s)\right) + D _ {R} \left(\nu_ {h} ^ {\tau , \dagger} (\cdot | s), \tilde {\nu} _ {h} ^ {\tau} (\cdot | s)\right)\right).
$$
In addition, if the players' policies are initialized to be uniform policies, i.e.,  $\tilde{\mu}_h^\tau (\cdot |s) = \mathbf{1} / A$  and  $\tilde{v}_h^\tau (\cdot |s) = \mathbf{1} / B,\forall s\in S,\tau \in [\bar{\tau}],h\in [H],$  we further have
$$
\mathrm {N E} - \operatorname {g a p} (\bar {\mu}, \bar {\nu}) \leq \frac {7 6 8 H ^ {5} \log T \log (A B)}{T}. \tag {4}
$$
Compared to existing results [77, 72], Theorem 1 directly associates the convergence rate with the quality of the initial policy  $(\tilde{\mu},\tilde{\nu})$ . Even when a good policy initialization is unknown and the algorithm is initialized with uniformly random policies, our convergence rate in (4) still matches the best-known result in the literature [72] except for an extra factor of  $O(\log T)$ . When suppressing the logarithmic terms, Theorem 1 immediately implies that for any  $\varepsilon >0$ , Algorithm 1 takes no more than  $T = \widetilde{O} (H^5 /\varepsilon)$  steps to learn an  $\varepsilon$ -approximate NE in an individual zero-sum Markov game.
3.2 Sharper Convergence with Meta-Learning
Having settled the initialization-dependent convergence in a zero-sum game, we proceed to show how meta-learning can learn a set of related games collectively and more rapidly. We consider an online setting with a sequence of  $K$  games  $\mathcal{G} = (\mathbb{G}^1,\dots ,\mathbb{G}^K)$ . For the max-player, let  $\tilde{\mu}^{k}$  and  $\bar{\mu}^{k}$ , respectively, denote the initial policy and output policy of Algorithm 1 on game  $\mathbb{G}^k$ . By putting together  $\mu_h^{\tau ,\dagger}(\cdot |s)$  over all  $(\tau ,h,s)\in [\bar{\tau} ]\times [H]\times S$ , we let  $\mu^{k,\dagger}:[\bar{\tau} ]\times [H]\times S\to \Delta (A)$  denote the best fixed policies in hindsight on  $\mathbb{G}^k$ . Define  $\tilde{\nu}^{k}$ ,  $\bar{\nu}^{k}$  and  $\nu^{k,\dagger}$  analogously for the min-player. Let  $\mu^{\star} = \frac{1}{K}\sum_{k = 1}^{K}\mu^{k,\dagger}$  and  $\nu^{\star} = \frac{1}{K}\sum_{k = 1}^{K}\nu^{k,\dagger}$  be the empirical averages of the best response policies. To ensure that the knowledge gained from previous games is useful for learning future tasks, we need to impose some similarity assumptions on the games  $\mathcal{G}$ . We consider the following similarity metric:
$$
\Delta_ {\mu , \nu} := \sum_ {k = 1} ^ {K} \left(\operatorname {K L} \left(\mu^ {k, \dagger} \| \mu^ {\star}\right) + \operatorname {K L} \left(\nu^ {k, \dagger} \| \nu^ {\star}\right)\right).
$$
Intuitively, since  $\{\nu^{k,t}\}_{t\in [T]}$  converges to an equilibrium policy for  $\mathbb{G}^k$  when  $T$  is large, the best fixed responses  $\mu^{k,\dagger}$  can be considered as an approximation of the max-player's NE policy on  $\mathbb{G}^k$ . In this sense,  $\Delta_{\mu ,\nu}$  essentially measures the distances between the NE policies of different games. It considers a set of games  $\vec{G}$  to be "similar" if their NE policies lie in a close neighborhood of each other. We remark that there might be multiple NE policies (with the same value) in a zero-sum game, and  $\Delta_{\mu ,\nu}$  only takes into account the NE policy pairs that Algorithm 1 actually delivers.
Our meta-learning procedure proceeds as follows: Within each game  $\mathbb{G}^k$ , we run Algorithm 1 as our base algorithm (1) to find a NE of  $\mathbb{G}^k$ . In a new game  $\mathbb{G}^{k + 1}$ , the initial policy of Algorithm 1 is given
by the following meta-updates in the outer loop (2), which essentially averages the best response policies of the previous tasks under  $\alpha$ -greedy parameterization:
$$
\tilde {\mu} ^ {k + 1} = \frac {1}{k} \sum_ {k ^ {\prime} = 1} ^ {k} \left[ \mu^ {k ^ {\prime}, \dagger} \right] _ {\alpha}, \quad \text {a n d} \quad \tilde {\nu} ^ {k + 1} = \frac {1}{k} \sum_ {k ^ {\prime} = 1} ^ {k} \left[ \nu^ {k ^ {\prime}, \dagger} \right] _ {\alpha}. \tag {5}
$$
In particular, for any vector  $\mathbf{x} \in \mathbb{R}^d$ , we define its  $\alpha$ -greedy parameterization  $[\mathbf{x}]_\alpha := (1 - \alpha)\mathbf{x} + \frac{\alpha}{d}\mathbf{1}$  to be a weighted average with a uniform vector  $\mathbf{1} / d \in \mathbb{R}^d$  of a proper dimension, where  $\alpha \in (0, 1/2)$ . Since  $\mu^{k,\dagger}$  denotes a set of vectors, we apply the operator  $[\cdot]_\alpha$  element-wise to each of the vectors. The reason for using  $\alpha$ -greedy is mainly technical: KL  $(\cdot|\cdot)$  is not Lipschitz continuous near the boundary of the probability simplex, and  $\alpha$ -greedy parameterization helps to stay  $\alpha$ -distance away from the boundary. We are now ready to present our sharper convergence rates for meta-learning.
Theorem 2. In a sequence of  $K$  two-player zero-sum Markov games, if Algorithm 1 is run for  $T$  iterations as the base algorithm and (5) with  $\alpha = 1 / \sqrt{K}$  as the meta-updates, we have
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathrm {N E - g a p} \left(\bar {\mu} ^ {k}, \bar {\nu} ^ {k}\right) \leq \frac {1 9 2 H ^ {5}}{T} \left(\frac {\Delta_ {\mu , \nu}}{K H ^ {2}} + \frac {1 0 (A + B) \log K}{\sqrt {K} H ^ {2}} + \frac {1 6 \log T \log (A B K)}{\sqrt {K}}\right). \tag {6}
$$
Consequently, for any  $\varepsilon > 0$ ,  $T = \tilde{O}\left(\frac{H^3}{\varepsilon} \left(\frac{\Delta_{\mu,\nu}}{K} + \frac{A + B + H^2}{\sqrt{K}}\right)\right)$  steps on average suffice to find an  $\varepsilon$ -approximate Nash equilibrium in each game.
When the number of games  $K$  is large, the last two terms on the RHS of (6) become negligible. Hence, compared to the best-known results  $\widetilde{O}(H^5 / T)$  of learning each game individually, Theorem 2 implies a significantly sharper convergence rate when the games are similar, i.e., when  $\Delta_{\mu, \nu} \ll KH^2$ .
4 Meta-Learning for Markov Potential Games
In this section, we study meta-learning for NE in Markov potential games. We show that a straightforward refinement to the analysis of an existing algorithm [15] provides initialization-dependent bounds. Building on it, in Section 4.1, we first investigate the sharper convergence of meta-learning in a sequence of similar MPGs. Further, since there exists an optimization objective universally agreed on by all the players in an MPG (i.e., the potential function), we can formulate the meta-learning problem in the same way as MAML [20]. In Section 4.2, by choosing a proper base algorithm, we establish the non-asymptotic convergence of MAML in the highly non-stationary multi-agent scenario, without even imposing any smoothness assumptions as in existing works [17, 18, 30].
4.1 Sharper Rates in Similar Games
To be consistent with existing results in the literature, in this section, we consider an infinite-horizon  $\gamma$ -discounted reward setting for MPGs [43, 39, 76, 15]. A detailed description of the setup is provided in Appendix C for completeness. Equivalent results for the finite-horizon episodic setting (as we defined in Section 2) can be derived in a straightforward way. We choose an existing state-of-the-art algorithm, namely independent projected Q-descent [15], as our base algorithm (1). Specifically, in an MPG  $\mathbb{G}^k$ , each agent independently runs policy gradient ascents to update its own policy for  $T$  iterations:
$$
\pi_ {i} ^ {k, t} (\cdot | s) \leftarrow \operatorname {P r o j} _ {\Delta \left(\mathcal {A} _ {i}\right)} \left(\pi_ {i} ^ {k, t - 1} (\cdot | s) + \alpha \bar {Q} _ {i} ^ {\pi^ {k, t - 1}} (s, \cdot)\right), \forall t \in [ T ], \tag {7}
$$
where  $\bar{Q}_i^\pi$  is the "averaged" Q-function formally defined in Appendix C. Let  $\Phi (\cdot ;\mathbb{G}^k)$  denote the potential function of  $\mathbb{G}^k$ . Through a simple refinement of the analysis in [15], we can establish the following initialization-dependence bound for our base algorithm (7).
Proposition 1. (Theorem 1 in [15]) Suppose that all players in a Markov potential game  $\mathbb{G}^k$  run independent projected  $Q$ -descent (7) for  $T$  iterations with  $\alpha \leq \frac{(1 - \gamma)^4}{8\kappa^3NA_{\mathrm{max}}}$ . Then, we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathrm {N E - g a p} (\pi^ {k, t}) \leq \sqrt {\frac {\kappa (\mathbb {G} ^ {k}) (\Phi (\pi^ {k , T} ; \mathbb {G} ^ {k}) - \Phi (\pi^ {k , 0} , \mathbb {G} ^ {k}))}{\alpha T (1 - \gamma) ^ {2}}},
$$
where  $\kappa (\mathbb{G}^k)$  is the standard distribution mismatch coefficient for  $\mathbb{G}^k$  formally defined in Appendix C.
Proposition 1 immediately implies that if we learn each MPG individually, it takes  $T = O\left(\frac{N A_{\max} \kappa^4 \Phi_{\max}}{(1 - \gamma)^6 \varepsilon^2}\right)$  steps to find an  $\varepsilon$ -approximate NE. To show the effectiveness of meta-learning, we consider the following similarity metric for a sequence of  $K$  games, which measures the maximal point-wise deviations of the potential functions:
$$
\Delta_ {\Phi} := \sum_ {k = 1} ^ {K - 1} \max  _ {\pi} \left(\Phi (\pi ; \mathbb {G} ^ {k}) - \Phi (\pi ; \mathbb {G} ^ {k + 1})\right). \tag {8}
$$
As for the meta-updates, we simply instantiate (2) as  $\pi_i^{k,0} \gets \pi_i^{k-1,T}$ , which lets each agent play the converged policy in the previous game. The intuition is that after running  $T$  steps on  $\mathbb{G}^{k-1}$ , the agents will converge to an approximate NE policy of  $\mathbb{G}^{k-1}$ . Since (8) requires the potential functions to be close, the converged policy  $\pi^{k-1,T}$  should serve as a good starting point to search for NE in  $\mathbb{G}^k$ . We formally characterize such an intuition in the following theorem, which shows the sharper convergence of meta-learning in a large set of similar MPGs (i.e., when  $K$  is large and  $\Delta_\Phi$  is small):
Theorem 3. In a sequence of  $K$  Markov potential games, if (7) is run for  $T$  iterations as the base algorithm and  $\pi_i^{k,0} \gets \pi_i^{k-1,T}$  as the meta-updates, then, for any  $\varepsilon > 0$ ,  $T = O\left(\frac{NA_{\max}\kappa^4(\Phi_{\max} + \Delta_\Phi)}{K(1 - \gamma)^6\varepsilon^2}\right)$  steps on average suffice to find an  $\varepsilon$ -approximate Nash equilibrium in each game.
4.2 Convergence to MAML Objective
In this subsection, we study meta-learning for MPGs under exactly the same formulation as in the seminal work of MAML [20]. Let  $\mathcal{G} = \{\mathbb{G}^j\}$  be a set of different MPGs, where the games are now drawn from a fixed distribution  $p$  that we can sample from. We consider parametric policy classes where agent  $i$ 's policy is parameterized by  $\theta_{i} = \{\theta_{i}(a_{i}|s)\in \mathbb{R}\}_{s\in S,a_{i}\in \mathcal{A}_{i}}$ . We focus on softmax parameterization where
$$
\pi_ {\theta_ {i}} (a _ {i} | s) = \frac {\exp (\theta_ {i} (a _ {i} | s))}{\sum_ {a _ {i} ^ {\prime} \in \mathcal {A} _ {i}} \exp (\theta_ {i} (a _ {i} ^ {\prime} | s))}.
$$
Let  $\zeta(\cdot; \mathbb{G})$  denote the operator of performing one step of policy gradient update on game  $\mathbb{G}$ , i.e.,  $\zeta(\theta; \mathbb{G}) := \theta + \alpha \nabla \Phi(\theta; \mathbb{G})$ , where  $\alpha > 0$  is the learning rate. The  $T$ -step MAML objective [20, 18, 30] can be formulated as
$$
\max  _ {\theta \in \Theta} F _ {T} (\theta) := \mathbb {E} _ {\mathbb {G} \sim p (\mathcal {G})} \left[ \Phi \left(\zeta \left(\dots \left(\zeta (\theta ; \mathbb {G})\right) \dots\right); \mathbb {G}\right) \right], \tag {9}
$$
where  $\theta = (\theta_{1},\dots,\theta_{N})\in \Theta$ , and the operator  $\zeta (\cdot ;\mathbb{G})$  is applied  $T$  times. Intuitively, MAML tries to find a good parameter initialization from which running  $T$  steps of gradient ascents on any new task  $\mathbb{G}$  leads to well-performing policy parameters.
Similar to Section 2, the MAML procedure consists of two nested stages. For the inner stage (1), we let each agent independently run  $T$  steps of policy gradient ascents to update its policy parameter  $\theta_{i}^{(t)}$  on each encountered MPG. It is known (Theorem 5 of [78]) that  $T = O(1 / \varepsilon^2)$  steps will find an  $\varepsilon$ -approximate NE for each individual MPG. For the outer stage (2), MAML directly performs gradient ascents with respect to the meta-objective (9). The gradient of  $F_{T}$  can be written in closed-form as
$$
\nabla F _ {T} (\theta) = \mathbb {E} _ {\mathbb {G} \sim p (\mathcal {G})} \left[ \left(\prod_ {t = 0} ^ {T - 1} \left(I + \alpha \nabla^ {2} \Phi \left(\theta^ {(t)}; \mathbb {G}\right)\right)\right) \nabla \Phi \left(\theta^ {(T)}; \mathbb {G}\right) \right]. \tag {10}
$$
A detailed discussion of MAML and its instantiation in our problem are provided in Appendix D. Most importantly, Appendix D shows that both the policy gradient  $\nabla \Phi (\theta)$  and the policy Hessian  $\nabla^2\Phi (\theta)$  can be written in closed-form, which allows us to construct unbiased estimators of (10) from samples. Despite the fact that the learning agents update their policies independently in an intertwined multi-agent system, our next result shows that the MAML updates converge to a stationary point of the meta-objective (9) in a non-asymptotic manner. A key step of the proof is to prove (rather than assume, as in existing works [17, 30]) that the meta-objective is Lipschitz smooth in the policy parameter  $\theta$ . The smoothness constant can also be written in a closed form (Lemma 14).
Theorem 4. Suppose that the agents run independent policy gradient ascents with softmax parameterization on each encountered  $MPG$  as the inner stage, and perform gradient ascents w.r.t the MAML objective as the outer stage. For any  $\varepsilon >0$ ,  $K = \frac{4NL_F}{(1 - \gamma)\varepsilon^2}$  iterations of MAML updates can find a policy  $\theta^{\star}$  such that  $\| \nabla F_T(\theta^\star)\| \leq \varepsilon$ , where  $L_{F}$  is given in Lemma 14 of Appendix D.
5 Meta-Learning for General-Sum Markov Games
In this section, we consider learning coarse correlated equilibria in general-sum Markov games with no assumption on reward structures. Similar to Section 3, we start by developing an initialization-dependent algorithm, followed by investigating the sharper convergence of meta-learning.
Our base algorithm for learning CCE also uses optimistic OMD with stage-based value updates. Detailed descriptions are deferred to Algorithm 2 in Appendix E due to space limitations. Algorithm 2 follows a similar structure as Algorithm 1, but the output policy  $\bar{\pi}$  of Algorithm 2 is no longer a state-wise average policy and is instead a correlated policy [3, 44, 33]. For any correlated policy  $\pi$ , we use the notion
$$
\mathrm {C C E - g a p} (\pi) := \max _ {i \in \mathcal {N}} V _ {1, i} ^ {\dagger , \pi_ {- i}} (s _ {1}) - V _ {1, i} ^ {\pi} (s _ {1})
$$
to measure its distance to a CCE. Let  $\bar{\tau}$  denote the total number of stages of Algorithm 2. Similar to zero-sum games (3), for any  $(\tau, h, s) \in [\bar{\tau}] \times [H] \times S$ , we define the per-state regret for each player  $i \in \mathcal{N}$  as
$$
\mathrm {r e g} _ {h, i} ^ {\tau} (s) := \max _ {\pi_ {h, i} ^ {\tau , \dagger} (\cdot | s) \in \Delta (\mathcal {A} _ {i})} \frac {1}{L _ {\tau}} \sum_ {j = t _ {\tau} ^ {\mathrm {s t a r t}}} ^ {t _ {\tau} ^ {\mathrm {e n d}}} \left\langle \pi_ {h, i} ^ {\tau , \dagger} - \pi_ {h, i} ^ {j}, Q _ {h, i} ^ {\tau} \pi_ {h, - i} ^ {j} \right\rangle (s),
$$
where  $Q_{h,i}^{\tau}$  is player  $i$ 's Q-function estimate at stage  $\tau$ . We define the maximal regret (over all states and players) as  $\mathrm{reg}_h^\tau \coloneqq \max_{s\in S}\max_{i\in \mathcal{N}}\{\mathrm{reg}_{h,i}^\tau (s)\}$ . The initialization-dependent convergence rate of Algorithm 2 is established in the following theorem.
Theorem 5. If Algorithm 2 is run on a general-sum Markov game for  $T$  iterations with a learning rate  $\eta >0$ , the output policy  $\bar{\pi}$  satisfies:
$$
\mathrm {C C E - g a p} (\bar {\pi}) \leq \frac {3}{\eta T} \sum_ {\tau = 1} ^ {\bar {\tau}} \sum_ {h = 1} ^ {H} \max _ {i \in \mathcal {N}, s \in \mathcal {S}} D _ {R} (\pi_ {h, i} ^ {\tau , \dagger} (\cdot | s), \tilde {\pi} _ {h, i} ^ {\tau} (\cdot | s)) + 3 6 N ^ {2} \eta^ {2} H ^ {4}.
$$
In addition, if the players' policies are initialized to be uniform policies  $\tilde{\pi}_{h,i}^{\tau}(\cdot |s) = 1 / A_{i},\forall i\in \mathcal{N}$  and  $\eta$  is chosen as  $\eta = H^{-2 / 3}T^{-1 / 3}(N - 1)^{-2 / 3}$ , then we have
$$
\mathrm {C C E} - \operatorname {g a p} (\bar {\pi}) \leq \frac {1 2 N ^ {\frac {2}{3}} H ^ {\frac {8}{3}} \log T \log A _ {\max }}{T ^ {\frac {2}{3}}}. \tag {11}
$$
Compared to existing results, Theorem 5 directly associates the convergence rate with the quality of the initial policy  $\tilde{\pi}$ . With uniform initialization, the convergence rate in (11) has a slightly worse dependence on  $T$  than the best known result  $\widetilde{O} (\sqrt{N} H^{11 / 4} / T^{3 / 4})$  [77]. Such deterioration is due to the potential lack of a smoothness condition for optimistic OMD that directly connects the stability of policies to the stability of utility functions (Lemma 18), unlike in optimistic FTRL. Although we believe that our rate in (11) can almost certainly be improved via a refined stability analysis, we leave the tightening of it to our future work as it would be a departure from the main focus of this work.
Let  $\tilde{\pi}^k$  and  $\bar{\pi}^k$ , respectively, denote the initial policy and output policy of Algorithm 2 on game  $\mathbb{G}^k$ . For player  $i \in \mathcal{N}$ , by putting together  $\pi_{h,i}^{\tau,\dagger}(\cdot|s)$  over all  $(\tau,h,s)$ , we use  $\pi_i^{k,\dagger} : [\bar{\tau}] \times [H] \times S \to \Delta(\mathcal{A}_i)$  to denote the best fixed policies in hindsight on  $\mathbb{G}^k$ . We consider a game similarity metric defined as
$$
\Delta_ {\pi} := \sum_ {k = 1} ^ {K} \sum_ {i = 1} ^ {N} \mathrm {K L} (\pi_ {i} ^ {k, \dagger} \| \pi_ {i} ^ {\star}), \mathrm {w h e r e} \pi_ {i} ^ {\star} = \frac {1}{K} \sum_ {k = 1} ^ {K} \pi_ {i} ^ {k, \dagger}.
$$
The following theorem presents the convergence rate of meta-learning, which again is sharper than learning each game individually when the games are similar, i.e., when  $\Delta_{\pi}$  is sufficiently small.
Theorem 6. In a sequence of  $K$  general-sum Markov games, if Algorithm 2 is run for  $T$  iterations as the base algorithm and the meta-updates  $\tilde{\pi}_i^k = \frac{1}{k - 1}\sum_{k' = 1}^{k - 1}[\pi_i^{k',\dagger}]_\alpha, \forall i \in \mathcal{N}$  are used with  $\alpha = 1 / \sqrt{K}$  for policy initializations, then, for any  $\varepsilon > 0$ ,  $T = \tilde{O}\left(\frac{HN}{\varepsilon^{3/2}}\left(\frac{\Delta_\pi^{3/2}}{K^{5/4}} + \frac{A_{\max}^{3/2} + H^3}{K^{1/2}}\right)\right)$  steps on average suffice to find an  $\varepsilon$ -approximate CCE in each game.
6 Simulations
We numerically evaluate our meta-learning algorithms from Sections 3 and 4 on a sequence of  $K$  games. In this section, we evaluate on a sequence of  $K = 10$  zero-sum Markov games and Markov potential games with two states, two players, and two candidate actions for each player. In Appendix F, we further demonstrate the scalability of our methods by providing numerical results on larger-scale tasks, including a simplified version of the Poker endgame considered in [27] and a 1D linear-quadratic tracking problem [37] with 4 cooperative players.
(a) Zero-sum Markov game
Figure 1: NE-gap of policies output by individual learning and meta-learning in (a) zero-sum Markov games, and (b) Markov potential games. Shaded areas denote the standard deviations. (c) visualizes the NE policies of the  $K$  games in the normalized space  $[0,1] \times [0,1]$  to illustrate their closeness.
(b) Markov potential game
(c) NE visualization
We generate the  $K = 10$  games by first specifying a "base game" and then adding random perturbations to its reward function to get  $K$  slightly different games. Each of the  $K$  games is run for  $T = 1000$  iterations. To better visualize the similarity level of these games, in Figure 1(c), we plot the NE policies of the perturbed zero-sum matrix games at each of the two states for the  $K = 10$  games. We remark that due to the existence of state transitions, the NE policies with respect to the stage Q-functions can be more diversified than Figure 1(c). Detailed descriptions of the simulation setup are deferred to Appendix F.