SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \leq \sqrt {1 6 H ^ {3} d \log \left(1 + \frac {H T}{\epsilon}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota} , h)} \ell (f ^ {t} , \xi_ {h} ^ {\iota})} \\ + \epsilon H T + 2 \sqrt {2 H ^ {4} T d \log \left(1 + \frac {H T}{\epsilon}\right)} \\ \end{array}
$$
by  $x\wedge y\leq \sqrt{xy}$  and setting  $\epsilon = \lambda Hd$  where  $\lambda >0$  and  $\ell (f,\xi_h)\coloneqq D_{\mathrm{He}}^2\bigl (\mathbb{P}_{f,h}(\cdot |\xi_h),\mathbb{P}_{f^*,h}(\cdot |\xi_h)\bigr)$  with  $\xi_{h} = (s_{h},a_{h},b_{h})$ . Thus, we obtain that when  $T$  is sufficiently large, we have
Note that the above results hold for arbitrary policy pair  $(\nu^t,\pi^t)$ . This implies that linear MGs with the dimension  $d$  is subsumed by the class of MGs with self-play and adversarial GEC  $d_{\mathrm{GEC}} = 16H^{3}d\log (1 + \frac{HT}{\epsilon})$ . This completes the proof.
B.2 Linear Mixture MG
In this subsection, we discuss the linear mixture MG setting, where the transition kernel is a linear combination of some basis transition kernels and thus admits a linear structure. Specifically, we have the following definition of the linear mixture MG.
Definition 5 (Linear Mixture Markov Game [16]). There exists a  $d$ -dimensional feature map  $\phi : S \times \mathcal{A} \times \mathcal{B} \times S \mapsto \mathbb{R}^d$  and an unknown coefficient  $\theta_h \in \mathbb{R}^d$ , such that the transition of the Markov game can be represented by
$$
\mathbb {P} _ {h} \left(s _ {h + 1} \mid s _ {h}, a _ {h}, b _ {h}\right) = \boldsymbol {\theta} _ {h} ^ {\top} \phi \left(s _ {h + 1} \mid s _ {h}, a _ {h}, b _ {h}\right),
$$
where  $\| \pmb{\theta}_h\| _2\leq B$  and  $\| \int_{\mathcal{S}}\phi (s,a,b,s')V(s')\mathrm{d}s'\| _2\leq H$  for any bounded function  $V:\mathcal{S}\mapsto [0,H]$
In the context of linear mixture MGs, both self-play and adversarial GEC can be calculated.
Proposition 4 (Linear Mixture MG  $\subset$  Low Self-play/Adversarial GEC). For linear mixture MGs with  $\phi(s, a, b, s') \in \mathbb{R}^d$  defined as in Definition 4, and for  $\epsilon > 0$ , when  $T$  is sufficiently large, choosing  $\ell(f, \xi_h) = D_{\mathrm{He}}^2\left(\mathbb{P}_{f,h}(\cdot \mid \xi_h), \mathbb{P}_{f^*,h}(\cdot \mid \xi_h)\right)$  as in (3) with  $\xi_h = (s_h, a_h, b_h)$ , we have
linear mixture  $MG \subset MG$  with low self-play and adversarial GEC
$$
d _ {\mathrm {G E C}} = 1 6 H ^ {3} d \log \left(1 + \frac {T B ^ {2}}{d \epsilon H}\right).
$$
Proof. The proof of GEC bound in a linear mixture Markov game is similar to the one in the linear Markov game case. We first define the Bellman residual in expectation as
$$
\mathcal {E} _ {f, h} ^ {\pi , \nu} := \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \left[ Q _ {f, h} ^ {\pi , \nu} (s _ {h}, a _ {h}, b _ {h}) - \mathbb {E} _ {s _ {h + 1} \sim \mathbb {P} _ {f ^ {*}   , h} (\cdot | s _ {h}, a _ {h}, b _ {h})} \left(r _ {h} (s _ {h}, a _ {h}, b _ {h}) + V _ {f, h + 1} ^ {\pi , \nu} (s _ {h + 1})\right) \right].
$$
One can derive the following property of  $\mathcal{E}_{f,h}^{\pi,\nu}$  by precisely the same derivation as in (4), which is
$$
V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} = \sum_ {h = 1} ^ {H} \mathcal {E} _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}}.
$$
We next show that in a linear mixture Markov game, the Bellman residual  $\mathcal{E}_{f,h}^{\pi,\nu}$  can be linearly represented, whose form is
$$
\begin{array}{l} \mathcal {E} _ {f, h} ^ {\pi , \nu} = \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} [ \mathbb {P} _ {f, h} (s | s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f ^ {*}, h} (s | s _ {h}, a _ {h}, b _ {h}) ] V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s \\ = \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} \left[ \phi \left(s _ {h}, a _ {h}, b _ {h}, s\right) ^ {\top} \left(\boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {*}, h}\right) \right] V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s \\ = \langle \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} \phi (s _ {h}, a _ {h}, b _ {h}, s) V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s, \boldsymbol {\theta} _ {f, h} - \boldsymbol {\theta} _ {f ^ {*}, h} \rangle \\ = \left\langle \phi \left(s _ {h}, a _ {h}, b _ {h}, f, \pi , \nu\right), \boldsymbol {q} _ {h} (f) \right\rangle , \\ \end{array}
$$
where we define  $\phi(s_h, a_h, b_h, f, \pi, \nu) \coloneqq \mathbb{E}_{\pi, \nu, \mathbb{P}_{f^*}} \int_S \phi(s_h, a_h, b_h, s') V_{f,h+1}^{\pi, \nu}(s') \mathrm{d}s'$  and  $\pmb{q}_h(f) \coloneqq \pmb{\theta}_{f,h} - \pmb{\theta}_{f^*,h}$ . Note that  $\| \phi(s_h, a_h, b_h, f, \pi, \nu) \| \leq H$  and  $\| \pmb{q}_h(f) \| \leq 2B$  by Definition 5.
We next define  $\phi_h^t \coloneqq \phi(s_h^t, a_h^t, b_h^t, f^t, \pi^t, \nu^t)$ ,  $\pmb{q}_h^t \coloneqq \pmb{q}_h(f^t)$ ,  $\Phi_h^t \coloneqq \lambda I_d + \frac{1}{H^2} \sum_{\iota=1}^{t} \phi_h^\iota(\phi_h^\iota)^\top$  and  $\varphi_h^t \coloneqq \| \phi_h^t \|_{(\Phi_h^{t-1})^{-1}}$ . Since we have
$$
\begin{array}{l} | \langle \boldsymbol {\phi} _ {h} ^ {\iota}, \boldsymbol {q} _ {h} ^ {\iota} \rangle | ^ {2} = \left| \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} \left(\mathbb {P} _ {f ^ {\iota}, h} (s \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}) - \mathbb {P} _ {f ^ {*}, h} (s \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota})\right) V _ {f ^ {\iota}, h + 1} ^ {\pi^ {\iota}, \nu^ {\iota}} (s) d s \right| ^ {2} \\ \leq H ^ {2} \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}} \| \mathbb {P} _ {f ^ {t}, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}) - \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}) \| _ {1} ^ {2} \\ \leq 8 H ^ {2} \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t}, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right), \mathbb {P} _ {f ^ {*}, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right)\right), \tag {12} \\ \end{array}
$$
where the first inequality is due to  $V_{f^t,h + 1}^{\pi^t,\nu^t}(s)\leq H$  and the second inequality is due to  $\| P - Q\| _2^2\leq$ $8D_{\mathrm{He}}^{2}(P,Q)$  . By utilizing  $\| \pmb {q}_h(f)\| \leq 2B$  and (12), we further obtain an upper bound of  $\| \pmb {q}_h^t\|_{\Phi_h^{t - 1}}$  as follows,
$$
\begin{array}{l} \| \boldsymbol {q} _ {h} ^ {t} \| _ {\Phi_ {h} ^ {t - 1}} \leq \sqrt {\sum_ {\iota = 1} ^ {t - 1} [ \langle \boldsymbol {\phi} _ {h} ^ {t} , \boldsymbol {q} _ {h} ^ {t} \rangle ] ^ {2} + \lambda \| \boldsymbol {q} _ {h} ^ {t} \| _ {2} ^ {2}} \\ \leq \sqrt {\sum_ {\iota = 1} ^ {t - 1} 8 H ^ {2} \mathbb {E} _ {\pi^ {\iota , \nu^ {\iota} , \mathbb {P} _ {f ^ {*}}}} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t} , h} (\cdot \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}) , \mathbb {P} _ {f ^ {*} , h} (\cdot \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})\right) + 4 \lambda B ^ {2}}. \\ \end{array}
$$
The rest of the proof is the same as the proof in the linear Markov game, from (5) to (11), except with a scaling factor. We provide the final result here as follows,
$$
\left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right|
$$
$$
\begin{array}{l} \leq \sqrt {1 6 H ^ {3} d \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t} , h} (\cdot | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}) , \mathbb {P} _ {f ^ {*} , h} (\cdot | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})\right)} \\ + \lambda B ^ {2} T + \min  \left\{2 H ^ {2} d \log \left(1 + \frac {T}{d \lambda}\right), H ^ {2} T \right\} + \min  \left\{\frac {3 H ^ {2} d}{\log 2} \log \left(1 + \frac {1}{\lambda \log 2}\right), H ^ {2} T \right\} \\ \end{array}
$$
$$
\begin{array}{l} \leq \sqrt {1 6 H ^ {3} d \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {\iota} , h} (\cdot | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}) , \mathbb {P} _ {f ^ {*} , h} (\cdot | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})\right)} \\ + \lambda B ^ {2} T + 2 \left[ 2 H ^ {2} d \log \left(1 + \frac {T}{d \lambda}\right) \wedge H ^ {2} T \right], \\ \end{array}
$$
Note that here  $(\pi^t,\nu^t)$  can be any arbitrary policy pair, including the ones required as in Definitions 1 and 3. Moreover, we will see that the above result satisfies the definitions of both self-play and adversarial GEC when using the inequality  $x\wedge y\leq \sqrt{xy}$  and choosing  $\lambda = \epsilon H / B^{2}$  and  $\ell (f,\xi_h) = D_{\mathrm{He}}^2\bigl (\mathbb{P}_{f,h}(\cdot |\xi_h),\mathbb{P}_{f^*,h}(\cdot |\xi_h)\bigr)$ . Therefore, we have
$$
d _ {\mathrm {G E C}} = 1 6 H ^ {3} d \log \left(1 + \frac {T B ^ {2}}{d \epsilon H}\right)
$$
for  $T$  sufficiently large, which shows that linear mixture MGs with the dimension  $d$  is subsumed by MGs with self-play and adversarial GEC  $d_{\mathrm{GEC}} = 16H^3 d\log (1 + \frac{T B^2}{d\epsilon H})$ . This completes the proof.
B.3 Low-Witness-Rank MG
Witness rank is a complexity measure that absorbs a wide range of settings in single-agent RL[60]. The work [28] further studies the witness rank under the MG setting with self-play. In this subsection, we show that MGs with low self-play GEC subsumes the class of MGs with a low witness rank in the
self-play setting. Since the witness rank under the adversarial setting was not investigated in prior works, we here only analyze the self-play setting.
Before presenting our main proposition, we first show the definitions of witnessed model misfit and witness rank in MGs with self-play [28].
Definition 6 (Witnessed model misfit in Markov game). Suppose that there exists a discriminator class  $\mathcal{U}$  with  $u\in \mathcal{U}:S\times \mathcal{A}\times \mathcal{B}\times S\mapsto [0,H]$ . The witnessed model misfit in Markov games  $\mathcal{W}(f,g,\rho ,h)$  for the models  $f,g,\rho \in \mathcal{F}$  at step  $h$  is characterized by
$$
\begin{array}{l} \mathcal {W} (f, g, \rho , h) = \sup  _ {u \in \mathcal {U}} \left| \mathbb {E} _ {(s _ {h}, a _ {h}, b _ {h}) \sim \mathbb {P} _ {f ^ {*}}, \tilde {\pi}, \check {\nu}} \left[ \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {\rho , h} (\cdot | s _ {h}, a _ {h}, b _ {h})} u (s _ {h}, a _ {h}, b _ {h}, s ^ {\prime}) \right. \right. \\ \left. \left. - \mathbb {E} _ {s _ {h + 1} \sim \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) u (s _ {h}, a _ {h}, b _ {h}, s _ {h + 1}) \right] \right|, \\ \end{array}
$$
where  $\check{\pi} := \operatorname{argmax}_{\pi} \min_{\nu} V_f^{\pi, \nu}$  and  $\check{\nu} := \operatorname{argmin}_{\nu} V_g^{\check{\pi}, \nu}$  with  $f, g \in \mathcal{F}$ .
The definition of  $(\check{\pi},\check{\nu})$  matches the requirement for the policy pair in the self-play GEC, i.e., condition (1) in Definition 1. We note that condition (2) in our definition of self-play GEC is designed for Algorithm 3, which is thus a symmetric condition to (1). We define the Bellman residual as
Then, we have the following definition of witness rank for MGs with self-play.
Definition 7 (Witness Rank for Self-Play). There exist two functions  $\mathbf{W}_h: \mathcal{F} \times \mathcal{F} \mapsto \mathbb{R}^d$  and  $\mathbf{X}_h: \mathcal{F} \mapsto \mathbb{R}^d$  and constant  $\kappa > 0$  such that
$$
\kappa_ {\mathrm {w i t}} | \mathcal {E} _ {\rho , h} ^ {\breve {\pi}, \breve {\nu}} | \leq \langle \boldsymbol {W} _ {h} (f, g), \boldsymbol {X} _ {h} (\rho) \rangle , \quad \mathcal {W} (f, g, \rho , h) \geq \langle \boldsymbol {W} _ {h} (f, g), \boldsymbol {X} _ {h} (\rho) \rangle ,
$$
where  $\check{\pi} := \operatorname{argmax}_{\pi} \min_{\nu} V_f^{\pi, \nu}$  and  $\check{\nu} := \operatorname{argmin}_{\nu} V_g^{\check{\pi}, \nu}$  with  $f, g \in \mathcal{F}$ ,  $\| \pmb{W}_h(\cdot) \|_2 \leq HB$  for a constant  $B$ , and  $\| \pmb{X}_h(\cdot) \|_2 \leq 1$  for any step  $h$ . Then, the witness rank of MGs with self-play is defined as the dimension  $d$  of the range spaces for functions  $\pmb{W}$  and  $\pmb{X}$ .
We show that in the self-play setting,  $d_{\mathrm{GEC}}$  is bounded for MGs with a low witness rank. It is worth noting that the proof of GEC bound for MGs with a low witness rank generalizes the proof for linear MGs in Appendix B.1.
Proposition 5 (Low Self-Play Witness Rank  $\subset$  Low Self-play GEC). For a Markov game with witness rank  $d$  as defined in Definition 7, when  $T$  sufficiently large, choosing  $\ell(f, \xi_h) = D_{\mathrm{He}}^2\left(\mathbb{P}_{f,h}(\cdot \mid \xi_h), \mathbb{P}_{f^*,h}(\cdot \mid \xi_h)\right)$  as in (3) with  $\xi_h = (s_h, a_h, b_h)$ , we have
MG with low self-play witness rank  $\subset$  MG with low self-play GEC
$$
d _ {\mathrm {G E C}} = \frac {4 H ^ {3} d}{\kappa_ {\mathrm {w i t}} ^ {2}} \log \left(1 + \frac {T H B ^ {2}}{4 d \epsilon \kappa_ {\mathrm {w i t}} ^ {2}}\right).
$$
Proof. We first give a bound of the model misfit in the Hellinger distance. By  $0 \leq u \leq H$ , we have
$$
\begin{array}{l} \left. \left[ \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {\rho , h} (\cdot | s _ {h}, a _ {h}, b _ {h})} u (s _ {h}, a _ {h}, b _ {h}, s ^ {\prime}) - \mathbb {E} _ {s _ {h + 1} \sim \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h})} u (s _ {h}, a _ {h}, b _ {h}, s _ {h + 1}) \right] ^ {2} \right. \\ \leq H ^ {2} \left\| \mathbb {P} _ {\rho , h} (\cdot | s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) \right\| _ {1} ^ {2} \\ \leq 2 H ^ {2} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {\rho , h} (\cdot | s _ {h}, a _ {h}, b _ {h}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h})\right), \\ \end{array}
$$
where  $\rho$  can be  $f$  or  $g$  as in the definition of self-play GEC. By the definition of  $\mathcal{W}(f,g,\rho,h)$  and Jensen's inequality, we further get
$$
\mathcal {W} (f, g, \rho , h) ^ {2} \leq 2 H ^ {2} \mathbb {E} _ {\left(s _ {h}, a _ {h}, b _ {h}\right) \sim \mathbb {P} _ {f ^ {*}, \check {\pi}, \check {\nu}} D _ {\mathrm {H e}} ^ {2}} \left(\mathbb {P} _ {\rho , h} (\cdot | s _ {h}, a _ {h}, b _ {h}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h})\right), \tag {13}
$$
where we use  $\sup_v\mathbb{E}_xf(x,v)\leq \mathbb{E}_x\sup_vf(x,v)$  and the definition of  $(\check{\pi},\check{\nu})$  . Note that following our definition of self-play GEC, if we choose  $f = f^{t}$  and  $g = g^t$  in the model misfit, then correspondingly we have  $\check{\pi} = \pi^t$  and  $\check{\nu} = \nu^{t}$  as in the condition (1) of Definition 1.
By the definition of  $\mathcal{E}_{\rho,h}^{\pi,\nu}$ , we can decompose the value difference as
$$
\left| \sum_ {t = 1} ^ {T} \left(V _ {\rho^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| = \left| \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {E} _ {\rho^ {t}, h} ^ {\pi^ {t}; \nu^ {t}} \right|.
$$
Hereafter we define  $\pmb{W}_h^t \coloneqq \pmb{W}_h(f^t, g^t)$ ,  $\pmb{X}_h^t \coloneqq \pmb{X}_h(\rho^t)$ ,  $\Sigma_h^t \coloneqq \lambda \mathrm{I} + \sum_{\iota=1}^{t} \pmb{X}_h^\iota(\pmb{X}_h^\iota)^\top$ , and  $\varphi_h^t \coloneqq \| \pmb{X}_h^t\|_{(\Sigma_h^t)^{-1}}^2$ . Using the definition of  $\pmb{W}_h$  and  $\pmb{X}_h$  and  $\mathcal{E}_{\rho^t,h}^{\pi^t,\nu^t} \leq H$ , we obtain
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {E} _ {\rho^ {t}, h} ^ {\pi^ {t}, \nu^ {t}} \right| \leq \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} H \min  \left\{\frac {1}{H \kappa_ {\mathrm {w i t}}} \langle \boldsymbol {W} _ {h} (f ^ {t}, g ^ {t}), \boldsymbol {X} _ {h} (\rho^ {t}) \rangle , 1 \right\} \\ = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} H \min  \left\{\frac {1}{H \kappa_ {\mathrm {w i t}}} \langle \boldsymbol {W} _ {h} (f ^ {t}, g ^ {t}), \boldsymbol {X} _ {h} (\rho^ {t}) \rangle , 1 \right\} \left(\mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} \leq 1 \}} + \mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} > 1 \}}\right) \\ \leq \underbrace {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {1}{\kappa_ {\mathrm {w i t}}} \| \boldsymbol {W} _ {h} ^ {t} \| _ {\Sigma_ {h} ^ {t - 1}} \min  \left\{\| \boldsymbol {X} _ {h} ^ {t} \| _ {(\Sigma_ {h} ^ {t - 1}) ^ {- 1}}, 1 \right\} \mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} \leq 1 \}}} _ {\text {T e r m (i)}} \\ + \underbrace {H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} > 1 \}} ,} _ {\text {T e r m (i i)}} \\ \end{array}
$$
where  $\mathbf{1}_{\{\cdot \}}$  denotes the indicator function. For Term(ii), we can apply Lemma 16 to obtain
$$
H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} > 1 \}} \leq \min \left\{\frac {3 H ^ {2} d}{\log 2} (1 + \frac {1}{\lambda \log 2}), H ^ {2} T \right\}.
$$
It remains to bound Term(i). We first calculate the sum of  $\| \pmb{W}_h^t\|_{\Sigma_h^t}^2$ , which is
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \| \boldsymbol {W} _ {h} ^ {t - 1} \| _ {\Sigma_ {h} ^ {t}} ^ {2} \\ = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\lambda \| \boldsymbol {W} _ {h} ^ {t} \| _ {2} ^ {2} + \sum_ {\iota = 1} ^ {t - 1} \langle \boldsymbol {W} _ {h} ^ {t}, \boldsymbol {X} _ {h} ^ {\iota} \rangle^ {2}\right) \\ \leq H ^ {3} T B ^ {2} \lambda + \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left\langle \boldsymbol {W} _ {h} ^ {t}, \boldsymbol {X} _ {h} ^ {\iota} \right\rangle^ {2} \\ \leq H ^ {3} T B ^ {2} \lambda + 2 H ^ {2} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \big (\mathbb {P} _ {\rho^ {\iota}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) \big), \\ \end{array}
$$
where the first inequality uses  $\| W_h^t\| \leq HB$ , and the second inequality uses the definition of witness rank and (13). We also let  $\xi_h^t \coloneqq (s_h^t, a_h^t, b_h^t)$ . We next use Lemma 15 to obtain
$$
\sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \min  \{\varphi_ {h} ^ {t}, 1 \} \leq \min  \left\{2 H d \log \left(1 + \frac {T}{d \lambda}\right), H T \right\}.
$$
Finally, by using the Cauchy-Schwarz inequality, we obtain the bound for Term(i) as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {1}{\kappa_ {\mathrm {w i t}}} \| \boldsymbol {W} _ {h} ^ {t} \| _ {\Sigma_ {h} ^ {t - 1}} \min  \{\| \boldsymbol {X} _ {h} ^ {t} \| _ {(\Sigma_ {h} ^ {t - 1}) ^ {- 1}}, 1 \} \mathbf {1} _ {\{\varphi_ {h} ^ {t - 1} \leq 1 \}} \\ \leq \sqrt {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\frac {1}{\kappa_ {\mathrm {w i t}}} \| \boldsymbol {W} _ {h} ^ {t} \| _ {\Sigma_ {h} ^ {t - 1}}\right) ^ {2}} \sqrt {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \min  \{\varphi_ {h} ^ {t} , 1 \}} \\ \leq \frac {1}{\kappa_ {\mathrm {w i t}}} \sqrt {\lambda H ^ {3} T B ^ {2} + 2 H ^ {2} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {\rho^ {\iota}, h} (\cdot | \xi_ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h} ^ {\iota})\right)} \\ \cdot \sqrt {\min  \left\{2 H d \log \left(1 + \frac {T}{d \lambda}\right) , H T \right\}} \\ \leq \left[ \sqrt {\frac {\lambda H ^ {3} T B ^ {2}}{\kappa_ {\mathrm {w i t}} ^ {2}}} + \sqrt {\frac {2 H ^ {2}}{\kappa_ {\mathrm {w i t}} ^ {2}} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \big (\mathbb {P} _ {\rho^ {\iota}, h} (\cdot | \xi_ {h} ^ {\iota}) , \mathbb {P} _ {f ^ {*} , h} (\cdot | \xi_ {h} ^ {\iota}) \big)} \right] \\ \cdot \sqrt {\min  \left\{2 H d \log \left(1 + \frac {T}{d \lambda}\right) , H T \right\}}, \\ \end{array}
$$
where the first inequality is by Cauchy-Schwarz, and the last inequality is by  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ . We further invoke  $\sqrt{ab} \leq \frac{a}{4} + b$  to obtain that
$$
\begin{array}{l} \mathrm {T e r m} (\mathrm {i}) \\ \leq \frac {\lambda H ^ {2} T B ^ {2}}{4 \kappa_ {\mathrm {w i t}} ^ {2}} + \min  \left\{2 H ^ {2} d \log \left(1 + \frac {T}{d \lambda}\right), H ^ {2} T \right\} \\ + \sqrt {\frac {4 H ^ {3} d}{\kappa_ {\mathrm {w i t}} ^ {2}} \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \big (\mathbb {P} _ {\rho^ {\iota}, h} (\cdot | \xi_ {h} ^ {\iota}) , \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h} ^ {\iota}) \big) .} \\ \end{array}
$$
Finally, we combine the bounds for term(i) and term(ii) to obtain that for  $T$  sufficiently large,
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {\rho^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \leq \frac {\lambda H ^ {2} T B ^ {2}}{4 \kappa_ {\mathrm {w i t}} ^ {2}} + 2 \sqrt {2 H ^ {4} d T \log \left(1 + \frac {T}{d \lambda}\right)} \\ + \sqrt {\frac {4 H ^ {3} d}{\kappa_ {\mathrm {w i t}} ^ {2}} \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \big (\mathbb {P} _ {\rho^ {\iota}, h} (\cdot | \xi_ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h} ^ {\iota}) \big)}, \\ \end{array}
$$
where we also use the inequality  $x \wedge y \leq \sqrt{xy}$ . Then, we can see that the above results satisfy Definition 1 with choosing  $\lambda = \frac{4\epsilon\kappa_{\mathrm{wt}}^2}{HB^2}$ . Thus, we conclude that under the self-play setting, we have
B.4 Weakly Revealing POMG
In this section, we consider a weakly revealing POMG. Recall that  $\{\mathbb{O}_h\}_{h=1}^H$  stands for the emission kernels, which can be viewed in the form of matrices  $\mathbb{O}_h \in \mathbb{R}^{|\mathcal{O}| \times |\mathcal{S}|}$ , with the  $(o,s)$ -th element being  $\mathbb{O}_h(o|s)$  for step  $h$ . Similarly, the transition matrix  $\mathbb{P}_{h,a,b} \in \mathbb{R}^{|\mathcal{S}| \times |\mathcal{S}|}$  is such that the  $(s',s)$ -th element is  $\mathbb{P}_h(s'|s,a,b)$ .
Here we consider undercomplete POMGs where there are more observations than hidden states,  $|\mathcal{O}| \geq |\mathcal{S}|$ . Formally, it can be defined as an  $\alpha$ -revealing POMG as follows:
Definition 8 ( $\alpha$ -Revealing POMG [41]). A POMG is  $\alpha$ -revealing if it satisfies that there exists  $\alpha > 0$  such that  $\min_h \sigma_S(\mathbb{O}_h) \geq \alpha$ , where  $\sigma_S(\cdot)$  denotes the  $S$ -th singular value of a matrix with  $S = |\mathcal{S}|$ .