SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i i}) = \frac {1}{\gamma_ {2}} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota})\right) \\ + \sum_ {t = 1} ^ {T} \Delta V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, *} + \frac {\gamma_ {2} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T. \\ \end{array}
$$
We further bound the RHS of the above equality. By the definitions of  $\ell(f, \xi_h)$  for FOMGs and POMGs and also the definitions of  $L_h^t(f)$  as in (1) and (2), according to Lemmas 11 and 12, we can obtain
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {\underline {{f}} ^ {t} \sim q ^ {t}} \left[ \gamma_ {2} \Delta V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, *} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {\underline {{f}} ^ {t} \sim q ^ {t}} \left[ \gamma_ {2} \Delta V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\underline {{f}} ^ {t}) + \log \frac {q ^ {t} (\underline {{f}} ^ {t})}{q ^ {0} (\underline {{f}} ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}, \overline {{f}} ^ {t} \sim p ^ {t}} \inf _ {q} \mathbb {E} _ {f \sim q} \left[ \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right], \\ \end{array}
$$
where the expectation for  $\overline{f}^t\sim p^t$  exists due to that  $\pi^t$  is computed based on  $\overline{f}^t$ , and also according to Lemma 10 and (25), the last equality can be achieved by that the updating rule for the distribution  $q^{t}$  in Algorithm 1, i.e., equivalently
$$
\begin{array}{l} q ^ {t} (f) \propto \exp \left(- \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log q ^ {0} (f)\right) \\ = q ^ {0} (f) \exp \left(- \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f)\right), \\ \end{array}
$$
can minimize  $\mathbb{E}_{f\sim q}\left[\gamma_2\Delta V_f^{\pi^t,*} - \sum_{h = 1}^H\sum_{\iota = 1}^{t - 1}\Delta L_h^\iota (f) + \log \frac{q(f)}{q^0(f)}\right]$ .
By Definition 2, further letting  $\widetilde{q} \coloneqq q^0(f) \cdot \mathbf{1}(f \in \mathcal{F}(\varepsilon)) / q^0(\mathcal{F}(\varepsilon))$  with  $\varepsilon \leq 1$ , we have
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}, \overline {{f}} ^ {t} \sim p ^ {t}} \inf  _ {q} \mathbb {E} _ {f \sim q} \left[ \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {f \sim \tilde {q}} \left[ \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right] \\ \leq \left(3 H ^ {2} \gamma_ {2} + \eta H T\right) \varepsilon - \log q ^ {0} (\mathcal {F} (\varepsilon)) \\ \leq \omega (4 H T, q ^ {0}), \\ \end{array}
$$
where the second inequality is by Lemma 13 and Lemma 14 as well as  $\varepsilon \leq 1$ , and the last inequality is by our setting that  $\eta = \frac{1}{2}$  and  $H\gamma_{2} \leq T$ . Therefore, combining the above results, we have
$$
\begin{array}{l} \mathbb {E} [ \text {T e r m (i i)} ] \leq \frac {\omega (H T , q ^ {0}) T}{\gamma_ {2}} + \frac {\gamma_ {2} d _ {\mathrm {G E C}} H}{4} + 2 H \left(d _ {\mathrm {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq 4 \sqrt {d _ {\mathrm {G E C}} H T \cdot \omega \left(4 H T , q ^ {0}\right)}, \tag {27} \\ \end{array}
$$
where we set  $HT > d_{\mathrm{GEC}}$ ,  $\epsilon = 1 / \sqrt{HT}$  for self-play GEC in Definition 1, and  $\gamma_2 = 2\sqrt{\omega(4HT,q^0)T / d_{\mathrm{GEC}}}$ .
Combining Results. Finally, combining the results in (26) and (27), we eventually obtain
$$
\mathbb {E} \left[ \operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) \right] = \mathbb {E} \left[ \operatorname {T e r m} (\mathrm {i}) \right] + \mathbb {E} \left[ \operatorname {T e r m} (\mathrm {i i}) \right] \leq 6 \sqrt {d _ {\mathrm {G E C}} H T \left(\omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right)\right)}.
$$
E.2 Proof of Proposition 2
Proof. Due to the symmetry of Algorithms 1 and 3, we can similarly bound  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$  in the way of bounding  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$ . Therefore, in this subsection, we only present the main steps for the proof. Specifically, by Algorithm 3, we have a decomposition as
$$
\operatorname{Reg}_{2}^{\mathrm{sp}}(T) = \underbrace{\sum_{t = 1}^{T}[V_{f^{*}}^{*,\nu^{t}} - V_{\overline{f}^{*}}^{\pi^{t},\nu^{t}}]}_{\mathrm{Term}(\mathrm{iv})} + \underbrace{\sum_{t = 1}^{T}[V_{\overline{f}^{*}}^{\pi^{t},\nu^{t}} - V_{f^{*}}^{*}]}_{\mathrm{Term}(\mathrm{iii})}.
$$
Bound  $\mathbb{E}[\text{Term(iii)}]$ . To bound  $\mathbb{E}[\text{Term(iii)}]$ , we give the following decomposition
$$
\begin{array}{l} \text {T e r m} (i i i) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} - V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} + V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} + V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {*} \right] \\ \leq \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} - V _ {\overline {{f}} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} + V _ {\overline {{f}} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {*} ] \\ = \sum_ {t = 1} ^ {T} \Delta V _ {\bar {f} ^ {t}} ^ {*} + \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} - V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu^ {t}} \right], \\ \end{array}
$$
where according to Algorithm 3, we have  $(\overline{\pi}^t,\nu^t) = \mathrm{argmax}_{\pi}\mathrm{argmin}_{\nu}V_{\overline{f}}^{\pi ,\nu}$ , which thus leads to
$$
V _ {\bar {f} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \nu^ {t}} = \max  _ {\pi} V _ {\bar {f} ^ {t}} ^ {\pi , \nu^ {t}} \geq V _ {\bar {f} ^ {t}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}}.
$$
According to the condition (2) in Definition 1 for self-play GEC and the updating rules in Algorithm 3, setting the exploration policy pair as  $\sigma^t = (\underline{\pi}^t,\nu^t)$ , we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {\bar {f} ^ {t}, 1} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}} - V _ {\bar {f} ^ {*}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}}\right) \right| \\ \leq \frac {1}{\gamma_ {1}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\underline {{\pi}} ^ {\iota}, \nu^ {\iota}, h)} \ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {\iota})\right) + \frac {\gamma_ {1} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T. \\ \end{array}
$$
We need to further bound the RHS of the above equality. For FOMGs and POMGs, by their definitions of  $\ell(f, \xi_h)$  as well as  $L_h^t(f)$ , according to Lemmas 11 and 12, we obtain
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\overline {{f}} ^ {t}) + \log \frac {p ^ {t} (\overline {{f}} ^ {t})}{p ^ {0} (\overline {{f}} ^ {t})} \right], \\ \end{array}
$$
where  $\Delta L_h^t (f)\coloneqq L_{h}^t (f) - L_h^t (f^*)$  is defined in (23). According to Lemma 10, we obtain that the distribution  $p^t$  defined in Algorithm 3, equivalently
$$
p ^ {t} (f) \propto p ^ {0} (f) \exp \left(- \gamma_ {1} \Delta V _ {f} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {t} (f)\right)
$$
minimizes the last term in the above inequality. This result implies that
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\overline {{f}} ^ {t}) + \log \frac {p ^ {t} (\overline {{f}} ^ {t})}{p ^ {0} (\overline {{f}} ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \inf _ {p} \mathbb {E} _ {f \sim p} \left[ \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f \sim \widetilde {p}} \left[ \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right] \\ \leq \left(3 H ^ {2} \gamma_ {1} + \eta H T\right) \varepsilon - \log p ^ {0} (\mathcal {F} (\varepsilon)) \\ \leq \omega (4 H T, p ^ {0}), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \text {T e r m (i i i)} ] \leq \frac {\omega (H T , p ^ {0}) T}{\gamma_ {1}} + \frac {\gamma_ {1} d _ {\text {G E C}} H}{4} + 2 H (d _ {\text {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T \\ \leq 4 \sqrt {d _ {\mathrm {G E C}} H T \omega \left(4 H T , p ^ {0}\right)}, \tag {28} \\ \end{array}
$$
where we set  $HT > d_{\mathrm{GEC}}$ ,  $\epsilon = 1 / \sqrt{HT}$  for self-play GEC in Definition 1, and  $\gamma_1 = 2\sqrt{\omega(4HT,p^0)T / d_{\mathrm{GEC}}}$ .
Bound  $\mathbb{E}[\text{Term}(\mathbf{iv})]$ . Next, we consider to bound  $\mathbb{E}[\text{Term}(\mathbf{iv})]$  as follows,
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i v}) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*}, \nu^ {t} - \underline {{V _ {f ^ {t}} ^ {\pi}}} ^ {t, \nu^ {t}} + \underline {{V _ {f ^ {t}} ^ {\pi}}} ^ {t, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right] \\ = - \sum_ {t = 1} ^ {T} \Delta V _ {\underline {{f}} ^ {t}} ^ {*}, ^ {\nu^ {t}} + \sum_ {t = 1} ^ {T} [ V _ {\underline {{f}} ^ {t}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}} - V _ {\underline {{f}} ^ {*}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}} ], \\ \end{array}
$$
where the second equality is by the fact that  $\underline{\pi}^t = \operatorname{argmax}_{\pi} V_{\underline{f}^t}^{\pi, \nu^t}$  according to the updating rule.
By Definition 1 and the updating rules in Algorithm 3, we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {\underline {{f}} ^ {t}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\underline {{\pi}} ^ {t}, \nu^ {t}}\right) \right| \\ \leq \frac {1}{\gamma_ {2}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \Big (\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\underline {{\pi}} ^ {\iota}, \nu^ {\iota}, h)} \ell (\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) \Big) + \frac {\gamma_ {2} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T. \\ \end{array}
$$
By the definitions of  $\ell(f, \xi_h)$  for FOMGs and POMGs and also the definitions of  $L_h^t(f)$  as in (1) and (2), we have
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {\underline {{f}} ^ {t} \sim q ^ {t}} \left[ - \gamma_ {2} \Delta V _ {\underline {{f}} ^ {t}} ^ {*}, \nu^ {t} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\underline {{\pi}} ^ {\iota}, \nu^ {\iota}, h)} \ell (\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {\underline {{f}} ^ {t} \sim q ^ {t}} \left[ - \gamma_ {2} \Delta V _ {\underline {{f}} ^ {t}} ^ {*}, \nu^ {t} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\underline {{f}} ^ {t}) + \log \frac {q ^ {t} (\underline {{f}} ^ {t})}{q ^ {0} (\underline {{f}} ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}, \overline {{J}} ^ {t} \sim p ^ {t}} \inf  _ {q} \mathbb {E} _ {f \sim q} \left[ - \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right], \\ \end{array}
$$
where the expectation for  $\overline{f}^t\sim p^t$  exists due to that  $\nu^{t}$  is computed based on  $\overline{f}^t$ , and also according to Lemma 10, the last equality can be achieved by that the updating rule for the distribution  $q^{t}$  in Algorithm 3, i.e., equivalently
$$
q ^ {t} (f) \propto q ^ {0} (f) \exp \left(\gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f)\right),
$$
can minimize  $\mathbb{E}_{f\sim q}\left[-\gamma_2\Delta V_f^{*\pi^t,\nu^t} - \sum_{h = 1}^H\sum_{\iota = 1}^{t - 1}\Delta L_h^\iota (f) + \log \frac{q(f)}{q^0(f)}\right]$ . By Definition 2, further letting  $\widetilde{q} \coloneqq q^0 (f)\cdot \mathbf{1}(f\in \mathcal{F}(\varepsilon)) / q^0 (\mathcal{F}(\varepsilon))$  with  $\varepsilon \leq 1$ , we have
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \inf  _ {q} \mathbb {E} _ {f \sim q} \left[ \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}, \bar {f} ^ {t} \sim p ^ {t}} \mathbb {E} _ {f \sim \tilde {q}} \left[ \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {q (f)}{q ^ {0} (f)} \right] \\ \leq \left(3 H ^ {2} \gamma_ {2} + \eta H T\right) \varepsilon - \log q ^ {0} (\mathcal {F} (\varepsilon)) \\ \leq \omega (4 H T, q ^ {0}), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \text {T e r m (i v)} ] \leq \frac {\omega (H T , q ^ {0}) T}{\gamma_ {2}} + \frac {\gamma_ {2} d _ {\mathrm {G E C}} H}{4} + 2 H \left(d _ {\mathrm {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq 4 \sqrt {d _ {\mathrm {G E C}} H T \cdot \omega \left(4 H T , q ^ {0}\right)}, \tag {29} \\ \end{array}
$$
Combining Results. Finally, combining the results in (28) and (29), we eventually obtain
$$
\mathbb {E} \left[ \operatorname {R e g} _ {2} ^ {\mathrm {s p}} (T) \right] = \mathbb {E} \left[ \text {T e r m} (\text {i i i}) \right] + \mathbb {E} \left[ \text {T e r m} (\text {i v}) \right] \leq 6 \sqrt {d _ {\mathrm {G E C}} H T \left(\omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right)\right)}.
$$
E.3 Proof of Theorem 1
Proof. The proof of Theorem 1 is immediate by the relation that  $\mathrm{Reg}^{\mathrm{sp}}(T) = \mathrm{Reg}_1^{\mathrm{sp}}(T) + \mathrm{Reg}_2^{\mathrm{sp}}(T)$ . Thus, according to the above proofs of Proposition 1 and Proposition 2, under the conditions in both propositions, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {R e g} ^ {\mathrm {s p}} (T) \right] = \mathbb {E} \left[ \operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) \right] + \mathbb {E} \left[ \operatorname {R e g} _ {2} ^ {\mathrm {s p}} (T) \right] \\ \leq 1 2 \sqrt {d _ {\mathrm {G E C}} H T \cdot \left(\omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right)\right)}. \\ \end{array}
$$
F Proofs for Algorithm 2
In this section, we provide the detailed proof for Theorem 2 and unify the proofs for both the FOMG and POMG settings together.
We recall that the value difference and the likelihood function difference are already defined in (22) and (23), which are
$$
\Delta V _ {f} ^ {*} := V _ {f} ^ {*} - V _ {f ^ {*}} ^ {*},
$$
$$
\Delta L _ {h} ^ {t} (f) = L _ {h} ^ {t} (f) - L _ {h} ^ {t} \left(f ^ {*}\right).
$$
Then, similar to (24), the updating rules of posterior distribution in Algorithm 2 have the following equivalent form as
$$
p ^ {t} (f) \propto p ^ {0} (f) \exp \left[ \gamma V _ {f} ^ {*} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} L _ {h} ^ {\tau} (f) \right]
$$
$$
\Longleftrightarrow p ^ {t} (f) \propto p ^ {0} (f) \exp \left[ \gamma \Delta V _ {f} ^ {*} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} \Delta L _ {h} ^ {\tau} (f) \right], \tag {30}
$$
since adding or subtracting terms irrelevant to  $f$ , i.e.,  $V_{f^*}^*$  and  $L_h^t(f^*)$ , within the power of all exponential terms will not change the posterior distribution of  $f$ .
F.1 Proof of Theorem 2
Proof. To bound the expected value of the adversarial regret  $\mathrm{Reg}^{\mathrm{adv}}(T)$ , i.e.,  $\mathbb{E}[\mathrm{Reg}^{\mathrm{adv}}(T)]$ , we first decompose the regret  $\mathrm{Reg}_{w}^{\mathrm{adv}}$  as follows
$$
\begin{array}{l} \mathrm {R e g} ^ {\mathrm {a d v}} (T) = \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {*} - V _ {f ^ {t}} ^ {*} ] + \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} ] \\ = - \sum_ {t = 1} ^ {T} \Delta V _ {f ^ {t}} ^ {*} + \sum_ {t = 1} ^ {T} [ V _ {f ^ {t}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} ]. \\ \end{array}
$$
Furthermore, for the term  $\sum_{t=1}^{T}[V_{f^t}^* - V_{f^*}^{\pi^t, \nu^t}]$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {t}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right] = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {t}} ^ {\pi^ {t}, \bar {\nu} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right] \\ \leq \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right], \\ \end{array}
$$
where the above result is by  $(\pi^t,\overline{\nu}^t) = \mathrm{argmax}_{\pi}\mathrm{argmin}_{\nu}V_{f^t}^{\pi ,\nu}$ , i.e.,  $(\pi^{t},\overline{\nu}^{t})$  is the NE of  $V_{f^t}^{\pi ,\nu}$ , according to Algorithm 2, and also due the relation that
$$
V _ {f ^ {t}} ^ {\pi^ {t}, \overline {{\nu}} ^ {t}} = \min  _ {\nu} V _ {f ^ {t}} ^ {\pi^ {t}, \nu} \leq V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}}.
$$
$$
\mathrm {R e g} ^ {\mathrm {a d v}} (T) = - \sum_ {t = 1} ^ {T} \Delta V _ {f ^ {t}} ^ {*} + \sum_ {t = 1} ^ {T} [ V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} ].
$$
According to Definition 3 and the updating rules in Algorithm 2, setting  $\sigma_{\mathrm{exp}}^t = (\pi^t,\nu^t)$ , and  $\ell (f^{t},\xi_{h}^{\iota}) = D_{\mathrm{He}}^{2}(\mathbb{P}_{f^{t},h}(\cdot |s_{h}^{\iota},a_{h}^{\iota},b_{h}^{\iota}),\mathbb{P}_{f^{*},h}(\cdot |s_{h}^{\iota},a_{h}^{\iota},b_{h}^{\iota}))$  for Definition 1, we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \leq \left[ d _ {\text {G E C}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\left(\sigma_ {\exp} ^ {\iota}, h\right)} \ell \left(f ^ {t}, \xi_ {h} ^ {\iota}\right)\right) \right] ^ {\frac {1}{2}} + 2 H \left(d _ {\text {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq \frac {1}{\gamma} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \ell (f ^ {t}, \xi_ {h} ^ {\iota})\right) + \frac {\gamma d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T, \\ \end{array}
$$
where the second inequality is due to  $\sqrt{xy} \leq \frac{1}{\gamma} x^2 + \frac{\gamma}{4} y^2$ . Combining the above results, we have
$$
\begin{array}{l} \operatorname {R e g} ^ {\mathrm {a d v}} (T) \leq \frac {1}{\gamma} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota})} \ell \left(f ^ {t}, \xi_ {h} ^ {\iota}\right)\right) - \sum_ {t = 1} ^ {T} \Delta V _ {f ^ {t}} ^ {*} \\ + \frac {\gamma d _ {\mathrm {G E C}}}{4} + 2 H \left(d _ {\mathrm {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T. \\ \end{array}
$$
For FOMGs and POMGs, we have different definitions of  $\ell(f, \xi_h)$ . According to (3), for FOMGs, we define
$$
\ell \left(f ^ {t}, \xi_ {h} ^ {\iota}\right) = D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t}, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right), \mathbb {P} _ {f ^ {*}, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right)\right),
$$
and for POMGs, the definition of  $\ell$  ensures
$$
\mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \ell (f ^ {t}, \xi_ {h} ^ {\iota}) = D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}).
$$
For different MG settings, we also define  $L_h^t (f)$  as in (1) and (2) which are
$$
L _ {h} ^ {t} (f) = \eta \log \mathbb {P} _ {f, h} \left(s _ {h + 1} ^ {t} \mid s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}\right), \quad L _ {h} ^ {t} (f) = \eta \log \mathbf {P} _ {f, h} \left(\tau_ {h} ^ {t}\right).
$$
Then, combining Lemmas 11 and 12, we know that the following result holds for both FOMGs and POMGs,
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {Z} ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \ell (f ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left[ L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*}) \right] + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right], \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \gamma \Delta V _ {f ^ {t}} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota})} \ell (f ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \gamma \Delta V _ {f ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f ^ {t}) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \gamma \Delta V _ {f ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f ^ {t}) - \log p ^ {0} (f ^ {t}) + \log p ^ {t} (f ^ {t}) \right], \\ \end{array}
$$
where  $\Delta L_h^t (f) = L_h^t (f) - L_h^t (f^*)$  as in (23). Therefore, according to Lemma 10 and (30), we know that the distribution
$$
\begin{array}{l} p ^ {t} (f) \propto \exp \left(\gamma \Delta V _ {f} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log p ^ {0} (f)\right) \\ = p ^ {0} (f) \exp \left(\gamma \Delta V _ {f} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f)\right) \\ \end{array}
$$
minimizes the last term in the above inequality, which is the posterior distribution updating rule for  $p^t$  adopted in Algorithm 2. This result implies that
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \gamma \Delta V _ {f ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f ^ {t}) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ - \gamma \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f ^ {t}) + \log \frac {p (f)}{p ^ {0} (f)} \right]. \\ \end{array}
$$
For the adversarial setting, we define  $\mathcal{F}(\varepsilon)$  as in Definition 2 similar to the definition of  $\mathcal{F}(\varepsilon)$  in the self-play setting. Further letting  $\widetilde{p} \coloneqq p^0(f) \cdot \mathbf{1}(f \in \mathcal{F}(\varepsilon)) / p^0(\mathcal{F}(\varepsilon))$  with  $\varepsilon \leq 1$  and  $\omega$  be associated with  $\mathcal{F}(\varepsilon)$ , we have