SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi_ {\iota}, \nu_ {\iota}}} \left[ \sum_ {o _ {h}, a _ {h}, b _ {h}} \| \left(\boldsymbol {M} _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h})\right) \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \sigma^ {\iota} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) \right] ^ {2} \\ \leq 2 \sum_ {\iota = 1} ^ {t - 1} \underbrace {\left[ \sum_ {\tau_ {h}} \| M _ {h} ^ {t} \left(o _ {h} , a _ {h} , b _ {h}\right) \bar {\boldsymbol {q}} ^ {t} \left(\tau_ {h - 1}\right) - M _ {h} \left(o _ {h} , a _ {h} , b _ {h}\right) \bar {\boldsymbol {q}} \left(\tau_ {h - 1}\right) \| _ {1} \mathbf {P} \left(\tau_ {h - 1}\right) \sigma^ {\iota} \left(\tau_ {h}\right) \right] ^ {2}} _ {\text {T e r m (i)}} \\ + 2 \sum_ {\iota = 1} ^ {t - 1} \underbrace {\left[ \sum_ {\tau_ {h}} \| M _ {h} ^ {t} \left(o _ {h} , a _ {h} , b _ {h}\right) \left(\bar {\boldsymbol {q}} ^ {t} \left(\tau_ {h - 1}\right) - \bar {\boldsymbol {q}} \left(\tau_ {h - 1}\right)\right) \| _ {1} \mathbf {P} \left(\tau_ {h - 1}\right) \sigma^ {\iota} \left(\tau_ {h}\right) \right] ^ {2}} _ {\text {T e r m (i i)}}. \\ \end{array}
$$
For Term(i), we equivalently rewrite this term as
$$
\begin{array}{l} \left[ \sum_ {\tau_ {h}} \| M _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) \bar {\boldsymbol {q}} ^ {t} \left(\tau_ {h - 1}\right) - M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) \bar {\boldsymbol {q}} \left(\tau_ {h - 1}\right) \| _ {1} \mathbf {P} \left(\tau_ {h - 1}\right) \sigma^ {\iota} \left(\tau_ {h}\right) \right] ^ {2} \\ = \Big[ \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {\iota}, \nu^ {\iota}}} {\mathbb {E}} \sum_ {o _ {h}} \sum_ {a _ {h}, b _ {h}} \| M _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) \bar {\boldsymbol {q}} ^ {t} (\tau_ {h - 1}) - M _ {h} (o _ {h}, a _ {h}, b _ {h}) \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \\ \left. \cdot \sigma^ {\iota} \left(o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}\right) \right] ^ {2}. \\ \end{array}
$$
We first consider the case when  $h < H$ . According to the definitions of  $\bar{\mathbf{q}}_h^t$  and  $\bar{\mathbf{q}}_h$ , we obtain
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i}) = \left[ \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {\iota}, \nu^ {\iota}}} {\mathbb {E}} \sum_ {o _ {h}} \sum_ {a _ {h}, b _ {h}} \| M _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) \bar {\boldsymbol {q}} ^ {t} \left(\tau_ {h - 1}\right) - M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) \bar {\boldsymbol {q}} \left(\tau_ {h - 1}\right) \| _ {1} \right. \\ \left. \cdot \sigma^ {\iota} \big (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1} \big) \right] ^ {2} \\ = \left[ \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {\iota}, \nu^ {\iota}}} {\mathbb {E}} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {i = 1} ^ {O} \left| \Pr_ {f ^ {t}} \left(o _ {h}, o _ {i} \mid \tau_ {1: h - 1}, \mathbf {d o} \left(a _ {h}, b _ {h}\right)\right) \right. \right. \\ \left. - \Pr_ {f ^ {*}} \left(o _ {h}, o _ {i} \mid \tau_ {h - 1}, \mathbf {d o} \left(a _ {h}, b _ {h}\right)\right) \mid \sigma^ {\iota} \left(o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}\right) \right] ^ {2} \\ = \left[ \sum_ {\tau_ {h - 1}, o _ {h}} \left| \Pr_ {f ^ {t}} ^ {\pi^ {\iota}, \nu^ {\iota}} \left(\tau_ {h - 1}, o _ {h}\right) - \Pr_ {f ^ {*}} ^ {\pi^ {\iota}, \nu^ {\iota}} \left(\tau_ {h - 1}, o _ {h}\right) \right| \right] ^ {2} \\ \leq \left\| \mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot) - \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot) \right\| _ {1} ^ {2} \leq 8 D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot), \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot)\right), \\ \end{array}
$$
where  $\operatorname{Pr}_{f^t}^{\pi^\iota, \nu^\iota}(\tau_{h-1}, o_h)$  denotes the probability of  $(\tau_{h-1}, o_h)$  if the actions are taken following  $(\pi^\iota, \nu^\iota)$  and the observations follow the omission and transition process in the model  $f^t$ . The first inequality is by the fact that the  $L_1$  difference of two marginal distributions is no more than the  $L_1$  difference of two uniformed distributions, and the second inequality is due to  $\|P - Q\|_1^2 \leq 8D_{\mathrm{He}}^2(P, Q)$ . Similarly, when  $h = H$ , Term(i) can be bounded as
$$
\begin{array}{l} \operatorname {Term}(\mathrm{i}) = \Bigl[\mathop{\mathbb{E}}_{\tau_{h - 1}\sim \mathbf{P}_{f^{*},h - 1}^{\pi^{\iota},\nu^{\iota}}}\sum_{o_{h},a_{h},b_{h}}\left|\operatorname{Pr}_{f^{t}}\big(o_{H}\mid \tau_{H - 1},\mathbf{d o}(a_{H},b_{H})\big) - \operatorname{Pr}_{f^{*}}\big(o_{H}\mid \tau_{H - 1},\mathbf{d o}(a_{H},b_{H})\big)\right| \\ \left. \cdot \sigma^ {\iota} \big (o _ {H}, a _ {H}, b _ {H}; \tau_ {H - 1} \big) \right] ^ {2} \\ = \left[ \sum_ {\tau_ {H - 1}, o _ {H}} \left| \Pr_ {f t} ^ {\pi^ {\iota}, \nu^ {\iota}} \left(\tau_ {H - 1}, o _ {H}\right) - \Pr_ {f ^ {*}} ^ {\pi^ {\iota}, \nu^ {\iota}} \left(\tau_ {H - 1}, o _ {H}\right) \right| \right] ^ {2} \\ \leq \| \mathbf {P} _ {f ^ {t}, H} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot) - \mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot) \| _ {1} ^ {2} \leq 8 D _ {\mathrm {H e}} ^ {2} \big (\mathbf {P} _ {f ^ {t}, H} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot), \mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {\iota}, \nu^ {\iota}} (\cdot) \big). \\ \end{array}
$$
For Term(ii), Lemma 4 is used to give an upper bound. Specifically, we obtain that
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i}) = \left[ \sum_ {\tau_ {h}} \| M _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) \left(\bar {\boldsymbol {q}} ^ {t} \left(\tau_ {h - 1}\right) - \bar {\boldsymbol {q}} \left(\tau_ {h - 1}\right)\right) \| _ {1} \mathbf {P} \left(\tau_ {h - 1}\right) \sigma^ {\iota} \left(\tau_ {h}\right) \right] ^ {2} \\ \leq \frac {S}{\alpha^ {2}} \left[ \sum_ {\tau_ {h - 1}} \| \bar {\boldsymbol {q}} ^ {t} (\tau_ {h - 1}) - \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \mathbf {P} (\tau_ {h - 1}) \sigma^ {\iota} (\tau_ {h - 1}) \right] ^ {2} \\ \leq \frac {S}{\alpha^ {2}} \| \mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} - \mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}} \| _ {1} ^ {2} \leq \frac {8 S}{\alpha^ {2}} D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}). \\ \end{array}
$$
Combining the above results completes the proof of this lemma.
Lemma 7. For any  $(o,a,b)\in \mathcal{O}\times \mathcal{A}\times \mathcal{B},$  we have
$$
\underset {\tau_ {h - 1} \sim \mathbb {P} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \| x _ {\tau_ {h - 1}, \sigma^ {t}, o, a, b} \| _ {2} ^ {2} \leq S, \quad \sum_ {j = 1} ^ {O} \| w _ {h, t, j, o, a, b} \| _ {2} \leq \frac {2 A B d \sqrt {S}}{\alpha}.
$$
Proof. We prove the former statement first. By noting that the  $\mathbb{O}_{h - 1}^{\dagger}\bar{\boldsymbol{q}} (\tau_{h - 1}) = [\operatorname *{Pr}(s_i\mid \tau_{h - 1})]_{i = 1}^{S}$  and  $\sigma^t (o_h,a_h,b_h;\tau_{h - 1})\leq 1$  , we obtain
$$
\begin{array}{l} \left\| x _ {\tau_ {h - 1}, \sigma^ {t}, o _ {h}, a _ {h}, b _ {h}} \right\| _ {2} ^ {2} \\ = \| \mathbb {O} _ {h - 1} ^ {\dagger} \bar {\boldsymbol {q}} (\tau_ {h - 1}) \sigma^ {t} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) \| _ {2} ^ {2} \\ \leq \sum_ {i = 1} ^ {S} \Pr \left(s _ {i} \mid \tau_ {h - 1}\right) ^ {2} \leq S. \\ \end{array}
$$
Therefore the first statement in Lemma 7 is proven. To prove the second statement, we write
$$
\begin{array}{l} \sum_ {j = 1} ^ {O} \left\| w _ {h, t, j, o, a, b} \right\| _ {2} \\ \leq \sum_ {o, a, b} \sum_ {j = 1} ^ {O} \| w _ {h, t, j, o, a, b} \| _ {2} \\ \leq \sum_ {o, a, b} \sum_ {j = 1} ^ {O} \| w _ {h, t, j, o, a, b} \| _ {1} \\ = \sum_ {o, a, b} \sum_ {i = 1} ^ {O} \| \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \mathbb {O} _ {h - 1} \widetilde {\boldsymbol {e}} _ {i} \| _ {1}, \\ \end{array}
$$
where the second inequality is due to  $\| \cdot \|_2 \leq \| \cdot \|_1$ , and  $\widetilde{e}_i$  stands for the  $i$ -th standard basis vector in  $\mathbb{R}^O$ . Then we apply Lemma 4 to obtain
$$
\begin{array}{l} \sum_ {o, a, b} \sum_ {i = 1} ^ {O} \| \left(M _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \mathbb {O} _ {h - 1} \widetilde {e} _ {i} \| _ {1} \\ \leq \frac {2 A B \sqrt {S}}{\alpha} \sum_ {i = 1} ^ {O} \| \mathbb {O} _ {h - 1} \widetilde {\boldsymbol {e}} _ {i} \| _ {1} \leq \frac {2 A B O \sqrt {S}}{\alpha} \| \mathbb {O} _ {h - 1} \| _ {1} = \frac {2 A B O \sqrt {S}}{\alpha}, \\ \end{array}
$$
where the second inequality is by  $\| Ax \|_1 \leq \| A \|_1 \| x \|_1$ . The proof is completed.
B.5 Decodable POMG
In this section, we propose a new class of POMGs, dubbed decodable POMG, by generalizing decodable POMDPs [23, 22] from the single-agent setting to the multi-agent setting.
Definition 9 (Decodable POMG). We say a POMG is a decodable POMG if an unknown decoder function  $\phi_h$  exists, which recovers the state at step  $h$  from the observation at step  $h$ . We have for any  $1 \leq h \leq H$  that
$$
\phi_ {h} (o _ {h}) = s _ {h}.
$$
Given the decoder, we can define the transition from observation to observation as follows,
$$
\begin{array}{l} \mathbb {P} _ {h} \left(o _ {h + 1} \mid o _ {h}, a _ {h}, b _ {h}\right) \\ = \sum_ {s _ {h + 1} \in \mathcal {S}} \mathbb {O} (o _ {h + 1} \mid s _ {h + 1}) \mathbb {P} (s _ {h + 1} \mid s _ {h} = \phi_ {h} (o _ {h}), a _ {h}, b _ {h}). \\ \end{array}
$$
Our next proposition will show that the class of decodable POMGs is subsumed by the class of MGs with low self-play and adversarial GEC.
Proposition 7 (Decodable POMG  $\subset$  Low Self-Play/Adversarial GEC). For a decodable POMG as defined in Definition 9, with the loss  $\ell(f, \xi_h)$  defined in Definition 3 for POMGs, we have
Decodable  $POMG \subset POMG$  with low self-play and adversarial GEC
$$
d _ {\mathrm {G E C}} = O \left(H ^ {3} | \mathcal {O} | ^ {3} | \mathcal {A} | ^ {2} | \mathcal {B} | ^ {2} \varsigma\right),
$$
where  $\varsigma = 2\log \left(1 + 4|\mathcal{A}|^2 |\mathcal{B}|^2 |\mathcal{O}|^2 |\mathcal{S}|\right)$ .
Proof. The proof of this proposition is largely the same as the proof of Proposition 6, except that the coefficient before  $\| x\| _1$  in Lemma 3 and 4 is changed, from  $\frac{\sqrt{S}}{\alpha}$  to 1, as is proved in Lemma 8 and Lemma 9. Following the proof of Proposition 6 and using Lemmas 8 and 9 yields our result.
B.5.1 Lemmas for Poof of Proposition 7
We present the lemmas for Proposition 7. These lemmas are analogous to Lemma 3 and Lemma 4. For convenience in analysis, we define some notations as follows.
Similar to the case of  $\alpha$ -revealing POMG in Appendix B.4, we define  $q_0, \tau_{h:h'}$ ,  $q(\tau_h)$  and  $\bar{q}(\tau_h)$ . We now define a different observable operator as follows,
$$
\boldsymbol {M} _ {h} (o, a, b) := \mathbb {P} _ {h, a, b} \mathrm {d i a g} (\boldsymbol {e} _ {o}),
$$
where  $e_{o} \in \mathbb{R}^{O}$  is the basis vector with only the  $o$ -th entry being 1. With these definitions, one can show that
We define  $M_{h^{\prime}:h}(o_{h:h^{\prime}},a_{h:h^{\prime}},b_{h:h^{\prime}}) \coloneqq M_{h^{\prime}}(o_{h^{\prime}},a_{h^{\prime}},b_{h^{\prime}})M_{h^{\prime}}(o_{h^{\prime} - 1},a_{h^{\prime} - 1},b_{h^{\prime} - 1})\ldots M_{h}(o_{h},a_{h},b_{h})$  and rewrite  $M_{H:h + 1}(o_{h + 1:H},a_{h + 1:H},b_{h + 1:H})$  as  $\pmb{m}(o_{h + 1:H},a_{h + 1:H},b_{h + 1:H})$  to emphasize that  $\pmb{m}(o_{h + 1:H},a_{h + 1:H},b_{h + 1:H})$  is a vector since  $M_H(o_H,a_H,b_H)$  is a vector.
Lemma 8. For any  $x \in \mathbb{R}^O$ , any policy pair  $(\pi, \nu)$ , and any  $\tau_{h-1}$ , we have
$$
\sum_ {\tau_ {h: H}} | \boldsymbol {m} (\tau_ {h: H}) x | \sigma (\tau_ {h: H}; \tau_ {h - 1}) \leq \| x \| _ {1}.
$$
Proof. We can first bound the LHS as
$$
\begin{array}{l} \sum_ {\tau_ {h: H}} | \boldsymbol {m} \left(\tau_ {h: H}\right) x | \sigma \left(\tau_ {h: H}; \tau_ {h - 1}\right) \\ = \sum_ {\tau_ {h: H}} \left| \sum_ {i = 1} ^ {O} \tilde {e} _ {i} ^ {\top} x \Pr \left(\tau_ {h + 1: H} \mid o _ {i}\right) \right| \sigma \left(\tau_ {h: H}; \tau_ {h - 1}\right) \\ \leq \sum_ {\tau_ {h: H}} \sum_ {i = 1} ^ {O} | \tilde {e} _ {i} ^ {\top} x | \Pr (\tau_ {h + 1: H} \mid o _ {i}) \sigma (\tau_ {h: H}; \tau_ {h - 1}), \\ \end{array}
$$
where  $\widetilde{e}_i$  is the  $i$ -th basis vector of the space  $\mathbb{R}^O$ . Since  $\sum_{\tau_{h + 1:H}}\operatorname*{Pr}(\tau_{h + 1:H}\mid o_i)\sigma (\tau_{h:H};\tau_{h - 1})\leq 1$ , we further obtain
$$
\begin{array}{l} \sum_ {\tau_ {h: H}} \sum_ {i = 1} ^ {O} | \tilde {e} _ {i} ^ {\top} x | \Pr (\tau_ {h + 1: H} \mid o _ {i}) \sigma (\tau_ {h: H}; \tau_ {h - 1}) \\ \leq \sum_ {i = 1} ^ {O} | \tilde {\boldsymbol {e}} _ {i} ^ {\top} x | = \| x \| _ {1}. \\ \end{array}
$$
Lemma 9. For any  $x\in \mathbb{R}^O$  , policy pair  $(\pi ,\nu)$  and  $\tau_{h - 1}$  , we have
$$
\sum_ {o \in \mathcal {O}, a \in \mathcal {A}, b \in \mathcal {B}} \| M _ {h} (o, a, b) x \| _ {1} \sigma (o, a, b; \tau_ {h - 1}) \leq \| x \| _ {1}.
$$
Proof. We can first show that
$$
\begin{array}{l} \sum_ {o \in \mathcal {O}, a \in \mathcal {A}, b \in \mathcal {B}} \| M _ {h} (o, a, b) x \| _ {1} \sigma (o, a, b; \tau_ {h - 1}) \\ = \sum_ {o \in \mathcal {O}, a \in \mathcal {A}, b \in \mathcal {B}} \sum_ {i = 1} ^ {O} | \tilde {\boldsymbol {e}} _ {i} ^ {\top} \boldsymbol {M} _ {h} (o, a, b) x | \sigma (o, a, b; \tau_ {h - 1}) \\ \leq \sum_ {o \in \mathcal {O}, a \in \mathcal {A}, b \in \mathcal {B}} \sum_ {i = 1} ^ {O} | \widetilde {\boldsymbol {e}} _ {i} ^ {\top} x | \Pr (o _ {i} \mid o, a, b) \sigma (o, a, b; \tau_ {h - 1}). \\ \end{array}
$$
Since  $\sum_{o\in \mathcal{O},a\in \mathcal{A},b\in \mathcal{B}}\operatorname *{Pr}(o_i\mid o,a,b)\sigma (o,a,b;\tau_{h - 1})\leq 1$  , we further obtain
$$
\begin{array}{l} \sum_ {o \in \mathcal {O}, a \in \mathcal {A}, b \in \mathcal {B}} \sum_ {i = 1} ^ {O} | \widetilde {\boldsymbol {e}} _ {i} ^ {\top} x | \mathbb {P} (o _ {i} \mid o, a, b) \sigma (o, a, b; \tau_ {h - 1}) \\ \leq \sum_ {i = 1} ^ {O} | \boldsymbol {e} _ {i} ^ {\top} x | = \| x \| _ {1}. \\ \end{array}
$$
C Computation of  $\omega (\beta ,p^0)$
In this section, we discuss the upper bound of the quantity  $\omega (\beta ,p^0)$  for both FOMGs and POMGs. For FOMGs, the analysis can be adopted from Lemma 2 in [2]. For POMGs, we give the detailed analysis to show the bound of  $\omega (\beta ,p^0)$ , which is the first proof for the partially observable setting.
C.1 Fully Observable Markov Game
We can adopt the result from Lemma 2 in [2] to give a bound for  $\omega (\beta ,p^0)$  in the context of fully observable Markov games.
Proposition 8 (Lemma 2 of [2]). When  $\mathcal{F}$  is finite,  $p^0$  is a uniform distribution over  $\mathcal{F}$ , then  $\omega (\beta ,p^{0})\leq \log |\mathcal{F}|$ . When  $\mathcal{F}$  is infinite, suppose a transition kernel  $\mathbb{P}_0$  satisfies  $\mathbb{P}_{f^*}\ll \mathbb{P}_0$  and  $\left\| \frac{\mathrm{d}\mathbb{P}_{f^*}}{\mathrm{d}\mathbb{P}_0}\right\|_{\infty}\leq B$ , then for  $\epsilon \leq 2 / 3$  and  $B\geq \log (6B^{2} / \epsilon)$ , there exists a prior  $p^0$  on  $\mathcal{F}$  such that
$$
\omega (\beta , p ^ {0}) \leq \beta \epsilon + \log \left(\mathcal {N} \left(\frac {\epsilon}{6 \log (B / \nu)}\right)\right),
$$
where  $\nu = \epsilon /(6\log (6B^{2} / \epsilon))$  and  $\mathcal{N}(\epsilon)$  stands for the  $\epsilon$ -covering number w.r.t. the distance
$$
d(f,f^{\prime}):= \sup_{s,a,b,h}\bigl|D^{2}_{\mathrm{He}}\bigl(\mathbb{P}_{f,h}(\cdot |s,a,b),\mathbb{P}_{f^{*},h}(\cdot |s,a,b)\bigr) - D^{2}_{\mathrm{He}}\bigl(\mathbb{P}_{f^{\prime},h}(\cdot |s,a,b),\mathbb{P}_{f^{*},h}(\cdot |s,a,b)\bigr)\bigr|.
$$
To apply Proposition 8 in FOMGs, we can further show that  $|D_{\mathrm{He}}^2 (P,R) - D_{\mathrm{He}}^2 (Q,R)| \leq \sqrt{2} D_{\mathrm{He}}^2 (P,R)| \leq \frac{\sqrt{2}}{2}\| P - Q\| _1$  for distributions  $P,Q,R$ , so that the covering number under the distance  $d$  can be bounded by the covering number w.r.t. the  $\ell_1$  distance denoted as  $\mathcal{N}_1(\epsilon)$ , i.e.,
$$
\mathcal {N} (\epsilon) \leq \mathcal {N} _ {1} (\sqrt {2} \epsilon),
$$
where  $\mathcal{N}_1(\epsilon)$  is defined w.r.t. the distance
$$
d _ {1} (f, f ^ {\prime}) = \sup  _ {s, a, b, h} \| \mathbb {P} _ {f, h} (\cdot | s, a, b) - \mathbb {P} _ {f ^ {\prime}, h} (\cdot | s, a, b) \| _ {1}.
$$
The covering number under the  $\ell_1$  distance is more common and is readily applicable to many problems.
Taking linear mixture MGs for an instance (defined in Definition 5), we calculate  $\omega(4HT, p^0)$ , which is the term quantifying the coverage of the initial sampling distribution in our theorems. We first obtain that
$$
\begin{array}{l} \sup  _ {s, a, b, h} \| P _ {f, h} (\cdot | s, a, b) - P _ {f ^ {\prime}, h} (\cdot | s, a, b) \| _ {1} \\ = \sup  _ {s, a, b, h} \int_ {s ^ {\prime} \in \mathcal {S}} \phi \left(s ^ {\prime}, s, a, b\right) ^ {\top} \left(\boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {\prime}, h}\right) d s ^ {\prime} \\ \leq \sup  _ {s, a, b, h} \int_ {s ^ {\prime} \in \mathcal {S}} \| \phi (s ^ {\prime}, s, a, b) \| _ {2} d s ^ {\prime} \| \boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {\prime}, h} \| _ {2} \\ \leq \sup  _ {h} \left\| \boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {\prime}, h} \right\| _ {2}, \tag {16} \\ \end{array}
$$
where the first inequality is by Cauchy-Schwarz, and the second inequality is due to  $\int_{s' \in S} \| \phi(s', s, a, b) \|_2 \mathrm{d}s' \leq 1$ . Since the model space is  $\{\pmb{\theta} : \sup_h \| \pmb{\theta}_h \|_2 \leq B\}$ , under the  $\ell_2$  distance measure of
$$
d _ {2} (f, f ^ {\prime}) = \sup  _ {h} \left\| \boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {\prime}, h} \right\| _ {2},
$$
$$
\mathcal {N} (\epsilon) \leq \mathcal {N} _ {1} (\sqrt {2} \epsilon) \leq \left(1 + \frac {2 \sqrt {2} B}{\varepsilon}\right) ^ {H d}. \tag {17}
$$
according to (16) and the covering number for an Euclidean ball. Combining this result (17) with Proposition 8 together, we can show that there exists a prior distribution  $p^0$  with a sufficiently large  $T$  such that
$$
\omega (4 H T, p ^ {0}) \lesssim H d \log \left(1 + B T \log \left(T \log (T)\right)\right).
$$
C.2 Partially Observable Markov Game
In this subsection, we prove the bound of  $\omega (\beta ,p^0)$  for the partially observable setting, inspired by the proof of Proposition 8 (Lemma 2 in [2]).
Proposition 9. When  $\mathcal{F}$  is finite,  $p^0$  is a uniform distribution over  $\mathcal{F}$ , then  $\omega (\beta ,p^{0})\leq \log |\mathcal{F}|$ . When  $\mathcal{F}$  is infinite, suppose a distribution  $\mathbf{P}_0$  satisfies that for any policy pair  $(\pi ,\nu)$ ,  $\mathbf{P}_{f^{*},H}^{\pi ,\nu}\ll \mathbf{P}_0^{\pi ,\nu}$  and  $\| \mathrm{d}\mathbf{P}_{f^{*},H}^{\pi ,\nu} / \mathrm{d}\mathbf{P}_{0}^{\pi ,\nu}\|_{\infty}\leq B$ , where  $B\geq 1$ . Then for  $\epsilon \leq 2 / 3$  and  $B\geq \log (6B^{2} / \epsilon)$ , there exists a prior  $p^0$  on  $\mathcal{F}$  such that
where  $\nu = \epsilon /(6\log (6B^2 /\epsilon))$  and  $\mathcal{N}(\cdot)$  is the covering number w.r.t. the distance
$$
d (f, f ^ {\prime}) = \sup _ {\pi , \nu} \left| D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f, H} ^ {\pi , \nu}, \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}) - D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu}, \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}) \right|.
$$
The assumption in the theorem above covers the case when  $\mathcal{S}$  is finite, where we choose  $P_0$  to be uniform on  $\mathcal{S}$  regardless of the policy. We also note that
$$
\| \mathrm {d} \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} / \mathrm {d} \mathbf {P} _ {0} ^ {\pi , \nu} \| _ {\infty} = \sup _ {\tau_ {H}} \left| \frac {\mathrm {d} \mathbf {P} _ {f ^ {*} , H} ^ {\pi , \nu} (\tau_ {H})}{\mathrm {d} \mathbf {P} _ {0} ^ {\pi , \nu} (\tau_ {H})} \right| = \sup _ {\tau_ {H}} \left| \frac {\mathrm {d} \mathbf {P} _ {f ^ {*} , H} (\tau_ {H}) \sigma (\tau_ {H})}{\mathrm {d} \mathbf {P} _ {0} (\tau_ {H}) \sigma (\tau_ {H})} \right| = \sup _ {\tau_ {H}} \left| \frac {\mathrm {d} \mathbf {P} _ {f ^ {*} , H} (\tau_ {H})}{\mathrm {d} \mathbf {P} _ {0} (\tau_ {H})} \right|,
$$
which does not depend on the joint policy  $\sigma = (\pi, \nu)$ .
To apply the above proposition in POMGs, by the relation between different distances:  $|D_{\mathrm{He}}^2 (P,R) - D_{\mathrm{He}}^2 (Q,R)| \leq \sqrt{2} D_{\mathrm{He}}^2 (P,R)| \leq \frac{\sqrt{2}}{2}\| P - Q\| _1$  for distributions  $P,Q,R$ , we can show the covering number under the distance  $d$  can be bounded by the covering number w.r.t. the  $\ell_1$  distance denoted as  $\mathcal{N}_1(\epsilon)$ , i.e.,
$$
d _ {1} (f, f ^ {\prime}) := \sup  _ {\pi , \nu} \left\| \mathbf {P} _ {f, H} ^ {\pi , \nu} - \mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu} \right\| _ {1}.
$$
Such a covering number under  $\ell_1$  distance is analyzed in the work [78], generalizing whose results give that POMGs with different structures admit a log-covering number  $\log \mathcal{N}_1(\epsilon) = \mathrm{ploy}(|\mathcal{O}|,|\mathcal{A}|,|\mathcal{B}|,|S|,H,\log (1 / \epsilon))$ . We refer readers to [78] for detailed calculation of log-covering numbers under  $d_{1}(f,f^{\prime})$ . Therefore, we can eventually show that  $\omega (4HT,p^{0}) = \mathrm{ploy}(|\mathcal{O}|,|\mathcal{A}|,|\mathcal{B}|,|S|,H,\log (HT))$ . Next, we show the detailed proof for Proposition 9.
Proof. When  $\mathcal{F}$  is finite and  $p^0$  is the uniform distribution on  $\mathcal{F}$ , the proof is straightforward as we have  $\omega (\beta ,p^{0})\leq \beta \varepsilon +\log |\mathcal{F}|$  for any  $\varepsilon \geq 0$  and a uniform distribution  $p^0$ . Setting  $\varepsilon$  to approach  $0^{+}$  completes the proof.
When  $\mathcal{F}$  is infinite, we start by setting up a  $\gamma$ -covering  $\mathcal{C}(\gamma) \subset \mathcal{F}$  w.r.t. the distance  $d(f, f') = \sup_{\pi, \nu} \left| D_{\mathrm{He}}^{2}(\mathbf{P}_{f,H}^{\pi,\nu}, \mathbf{P}_{f^{*},H}^{\pi,\nu}) - D_{\mathrm{He}}^{2}(\mathbf{P}_{f',H}^{\pi,\nu}, \mathbf{P}_{f^{*},H}^{\pi,\nu}) \right|$ , where  $\gamma > 0$  is a variable to be specified. Since  $f^{*}$  is covered, an  $\widetilde{f} \in \mathcal{C}(\gamma)$  satisfies  $d(\widetilde{f}, f^{*}) = \sup_{\pi, \nu} D_{\mathrm{He}}^{2}(\mathbf{P}_{\widetilde{f},H}^{\pi,\nu}, \mathbf{P}_{f^{*},H}^{\pi,\nu}) \leq \gamma$ . We further define  $\mathcal{C}_{\nu}(\gamma) := \{\nu \mathbf{P}_0 + (1 - \nu) \mathbf{P}_f | f \in \mathcal{C}(\gamma)\}$  and also  $\mathbf{P}_{f'} := \nu \mathbf{P}_0 + (1 - \nu) \mathbf{P}_{\widetilde{f}}$ . We note that  $\sup_{\pi, \nu} \| \frac{\mathrm{d}\mathbf{P}_{f^{*},H}^{\pi,\nu}}{\mathrm{d}\mathbf{P}_{f',H}^{\pi,\nu}} \|_{\infty} \leq \frac{B}{\nu}$ . Then, we obtain
$$
\begin{array}{l} D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu}, \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}) \\ = 1 - \int \sqrt {\mathrm {d} (\nu \mathbf {P} _ {0} ^ {\pi , \nu} + (1 - \nu) \mathbf {P} _ {\tilde {f} , H} ^ {\pi , \nu}) \mathrm {d} \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}} \\ \leq 1 - \left(\nu \int \sqrt {\mathrm {d} \mathbf {P} _ {0} ^ {\pi , \nu} \mathrm {d} \mathbf {P} _ {f ^ {*} , H} ^ {\pi , \nu}} + (1 - \nu) \int \sqrt {\mathrm {d} \mathbf {P} _ {\tilde {f} , H} ^ {\pi , \nu} \mathrm {d} \mathbf {P} _ {f ^ {*} , H} ^ {\pi , \nu}}\right) \\ = D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {\tilde {f}, H} ^ {\pi , \nu}, \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}\right) + \nu \int \sqrt {\mathrm {d} \mathbf {P} _ {\tilde {f} , H} ^ {\pi , \nu} \mathrm {d} \mathbf {P} _ {f ^ {*} , H} ^ {\pi , \nu}} - \nu \int \sqrt {\mathrm {d} \mathbf {P} _ {0} ^ {\pi , \nu} \mathrm {d} \mathbf {P} _ {f ^ {*} , H} ^ {\pi , \nu}} \\ \leq \gamma + \nu , \\ \end{array}
$$