SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} \right| \\ \leq H \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \sum_ {h = 1} ^ {H} \left\| \mathbb {P} _ {f ^ {*}, h} \left(\cdot | s _ {h}, a _ {h}, b _ {h}\right) - \mathbb {P} _ {f, h} \left(\cdot | s _ {h}, a _ {h}, b _ {h}\right) \right\| _ {1} \\ \leq H ^ {2} \sup  _ {h, s, a, b} \| \mathbb {P} _ {f ^ {*}, h} (\cdot | s, a, b) - \mathbb {P} _ {f, h} (\cdot | s, a, b) \| _ {1} \\ \leq 3 H ^ {2} \sup _ {h, s, a, b} \mathrm {K L} ^ {\frac {1}{2}} (\mathbb {P} _ {f ^ {*}, h} (\cdot | s, a, b) \| \mathbb {P} _ {f, h} (\cdot | s, a, b)) \leq 3 H ^ {2} \varepsilon , \\ \end{array}
$$
where the third inequality is by Pinsker's inequality and the last inequality is by the assumption of this lemma. On the other hand, we can show that the above result also holds for POMGs. Then, for this setting, we have
$$
\begin{array}{l} \left| V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} \right| = \int_ {(\mathcal {O} \times \mathcal {A} \times \mathcal {B}) ^ {H}} \left(\mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} (\tau_ {H}) - \mathbf {P} _ {f, H} ^ {\pi , \nu} (\tau_ {H})\right) \left(\sum_ {h = 1} ^ {H} r _ {h} (o _ {h}, a _ {h}, b _ {h})\right) d \tau_ {H} \\ = H \int_ {(\mathcal {O} \times \mathcal {A} \times \mathcal {B}) ^ {H}} \left| \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} \left(\tau_ {H}\right) - \mathbf {P} _ {f, H} ^ {\pi , \nu} \left(\tau_ {H}\right) \right| d \tau_ {H} \\ = H \| \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} (\tau_ {H}) - \mathbf {P} _ {f, H} ^ {\pi , \nu} (\tau_ {H}) \| _ {1} \\ = 3 H \sup  _ {\pi , \nu} \mathrm {K L} ^ {\frac {1}{2}} \left(\mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} \| \mathbf {P} _ {f, H} ^ {\pi , \nu}\right) \leq 3 H \varepsilon . \\ \end{array}
$$
To unify our results, we enlarge the above bound by a factor of  $H$  and eventually obtain that
$$
\left| V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} \right| \leq 3 H ^ {2} \varepsilon ,
$$
for both FOMGs and POMGs. Moreover, by the properties of the operators min, max, and max min, we have
$$
V _ {f ^ {*}} ^ {*} - V _ {f} ^ {*} = \max _ {\pi} \min _ {\nu} V _ {f ^ {*}} ^ {\pi , \nu} - \max _ {\pi} \min _ {\nu} V _ {f} ^ {\pi , \nu} \leq \sup _ {\pi , \nu} | V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} | \leq 3 H ^ {2} \varepsilon .
$$
$$
\sup  _ {\pi} (V _ {f ^ {*}} ^ {\pi , *} - V _ {f} ^ {\pi , *}) = \sup  _ {\pi} (\min  _ {\nu} V _ {f ^ {*}} ^ {\pi , \nu} - \min  _ {\nu} V _ {f} ^ {\pi , \nu}) \leq \sup  _ {\pi , \nu} | V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} | \leq 3 H ^ {2} \varepsilon .
$$
$$
\sup  _ {\nu} \left(V _ {f ^ {*}} ^ {* , \nu} - V _ {f} ^ {* , \nu}\right) = \sup  _ {\nu} \left(\sum_ {\pi} V _ {f ^ {*}} ^ {\pi , \nu} - \sum_ {\pi} V _ {f} ^ {\pi , \nu}\right) \leq \sup  _ {\pi , \nu} \left| V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} \right| \leq 3 H ^ {2} \varepsilon .
$$
This concludes the proof of this lemma.
D.5 Proof of Lemma 14
Proof. We first prove the upper bound under the FOMG setting. By (1), we know that
$$
\begin{array}{l} L _ {h} ^ {t} (f ^ {*}) - L _ {h} ^ {t} (f) = \eta \log \mathbb {P} _ {f ^ {*}, h} (s _ {h + 1} ^ {t} | s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}) - \eta \log \mathbb {P} _ {f, h} (s _ {h + 1} ^ {t} | s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}) \\ = \eta \log \frac {\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {t} | s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t})}{\mathbb {P} _ {f , h} (s _ {h + 1} ^ {t} | s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t})}, \\ \end{array}
$$
$$
\begin{array}{l} | \mathbb {E} (L _ {h} ^ {t} (f ^ {*}) - L _ {h} ^ {t} (f)) | = \eta \left| \mathbb {E} \log \frac {\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {t} \mid s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t})}{\mathbb {P} _ {f , h} (s _ {h + 1} ^ {t} \mid s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t})} \right| \\ = \eta \left| \mathbb {E} _ {\left(s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}\right)} \mathbb {E} _ {s _ {h + 1} ^ {t} \sim \mathbb {P} _ {f ^ {*}, h} \left(\cdot | s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}\right)} \log \frac {\mathbb {P} _ {f ^ {*} , h} \left(s _ {h + 1} ^ {t} \mid s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t}\right)}{\mathbb {P} _ {f , h} \left(s _ {h + 1} ^ {t} \mid s _ {h} ^ {t} , a _ {h} ^ {t} , b _ {h} ^ {t}\right)} \right| \\ \leq \eta \sup  _ {s, a, b} \mathrm {K L} (\mathbb {P} _ {f ^ {*}, h} (\cdot | s, a, b) \| \mathbb {P} _ {f, h} (\cdot | s, a, b)) \\ \leq \eta \varepsilon^ {2}, \\ \end{array}
$$
where we use the definition of KL divergence in the first inequality and the last inequality is by the condition that  $\sup_{h,s,a,b}\mathrm{KL}^{\frac{1}{2}}(\mathbb{P}_{f^{*},h}(\cdot |s,a,b)\| \mathbb{P}_{f,h}(\cdot |s,a,b))\leq \varepsilon$
We next prove the upper bound under the POMG setting. According to (2), we have
$$
L _ {h} ^ {t} (f ^ {*}) - L _ {h} ^ {t} (f) = \eta \log {\bf P} _ {f ^ {*}, h} (\tau_ {h} ^ {t}) - \eta \log {\bf P} _ {f, h} (\tau_ {h} ^ {t}) = \eta \log \frac {{\bf P} _ {f ^ {*} , h} (\tau_ {h} ^ {t})}{\bf P} _ {f, h} (\tau_ {h} ^ {t}),
$$
$$
\begin{array}{l} | \mathbb {E} (L _ {h} ^ {t} (f ^ {*}) - L _ {h} ^ {t} (f)) | = \eta \left| \mathbb {E} _ {\tau_ {h} ^ {t} \sim \mathbf {P} _ {f ^ {*} , h} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \log \frac {\mathbf {P} _ {f ^ {*} , h} (\tau_ {h} ^ {t})}{\mathbf {P} _ {f , h} (\tau_ {h} ^ {t})} \right| \\ = \eta \left| \mathbb {E} _ {\tau_ {h} ^ {t} \sim \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \log \frac {\mathbf {P} _ {f ^ {*} , h} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h} ^ {t})}{\mathbf {P} _ {f , h} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h} ^ {t})} \right| \\ = \eta \mathrm {K L} (\mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, h} ^ {\pi^ {t}, \nu^ {t}}), \\ \end{array}
$$
where we use the definition of KL divergence and the relation between  $\mathbf{P}_{f,h}$  and  $\mathbf{P}_{f,h}^{\pi,\nu}$ . Now we consider to lower bound  $\mathrm{KL}(\mathbf{P}_{f^{*},H}^{\pi^{t},\nu^{t}}\| \mathbf{P}_{f,H}^{\pi^{t},\nu^{t}})$ . We have
$$
\begin{array}{l} \mathrm {K L} (\mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, H} ^ {\pi^ {t}, \nu^ {t}}) = \mathbb {E} _ {\tau_ {H} ^ {t} \sim \mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \log \frac {\mathbf {P} _ {f ^ {*} , H} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {H} ^ {t})}{\mathbf {P} _ {f , H} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {H} ^ {t})} \\ = \mathbb {E} _ {\tau_ {H} ^ {t} \sim \mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \left[ \log \frac {\mathbf {P} _ {f ^ {*} , h} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h} ^ {t})}{\mathbf {P} _ {f , h} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h} ^ {t})} + \log \frac {\operatorname * {P r} _ {f ^ {*}} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h + 1 : H} ^ {t} | \tau_ {h} ^ {t})}{\operatorname * {P r} _ {f} ^ {\pi^ {t} , \nu^ {t}} (\tau_ {h + 1 : H} ^ {t} | \tau_ {h} ^ {t})} \right] \\ = \mathrm {K L} \left(\mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, h} ^ {\pi^ {t}, \nu^ {t}}\right) + \mathbb {E} _ {\tau_ {h} ^ {t} \sim \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \mathbb {E} _ {\tau_ {h + 1: H} \sim \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} (\cdot | \tau_ {h} ^ {t})} \log \frac {\operatorname * {P r} _ {f ^ {*}} ^ {\pi^ {t} , \nu^ {t}} \left(\tau_ {h + 1 : H} ^ {t} \mid \tau_ {h} ^ {t}\right)}{\operatorname * {P r} _ {f} ^ {\pi^ {t} , \nu^ {t}} \left(\tau_ {h + 1 : H} ^ {t} \mid \tau_ {h} ^ {t}\right)} \\ = \mathrm {K L} (\mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, h} ^ {\pi^ {t}, \nu^ {t}}) + \mathbb {E} _ {\tau_ {h} ^ {t} \sim \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} (\cdot)} \mathrm {K L} (\mathrm {P r} _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} (\cdot | \tau_ {h} ^ {t}) \| \mathrm {P r} _ {f} ^ {\pi^ {t}, \nu^ {t}} (\cdot | \tau_ {h} ^ {t})) \\ \geq \mathrm {K L} (\mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, h} ^ {\pi^ {t}, \nu^ {t}}), \\ \end{array}
$$
where  $\mathrm{Pr}_f^{\pi^t,\nu^t}$  denote the probability under the model  $f$  and the policy pair  $\pi^t,\nu^t$ , the third equality is by the definition of KL divergence, and the inequality is by the non-negativity of KL divergence. Therefore, combining the above results and the condition  $\sup_{\pi ,\nu}\mathrm{KL}^{\frac{1}{2}}(\mathbf{P}_{f^{*},H}^{\pi ,\nu}\| \mathbf{P}_{f,H}^{\pi ,\nu})\leq \varepsilon$ , we obtain
$$
| \mathbb {E} (L _ {h} ^ {t} (f ^ {*}) - L _ {h} ^ {t} (f)) | \leq \eta \mathrm {K L} (\mathbf {P} _ {f ^ {*}, H} ^ {\pi^ {t}, \nu^ {t}} \| \mathbf {P} _ {f, H} ^ {\pi^ {t}, \nu^ {t}}) \leq \eta \varepsilon^ {2}.
$$
E Proofs for Algorithms 1 and 3
In this section, we provide the detailed proof for Theorem 1. In particular, our proof is compatible with both the FOMG and the POMG settings. Thus, we unify both setups in a single theorem and present their proof together in this section.
To characterize the value difference under models  $f$  and  $f^{*}$  we define the following terms, which are
$$
\Delta V _ {f} ^ {\pi , *} (s) := V _ {f} ^ {\pi , *} (s) - V _ {f ^ {*}} ^ {\pi , *} (s), \tag {21}
$$
$$
\Delta V _ {f} ^ {*} (s) := V _ {f} ^ {*} (s) - V _ {f ^ {*}} ^ {*} (s). \tag {22}
$$
In addition, we define the difference of likelihood functions at step  $h$  of time  $t$  as
$$
\Delta L _ {h} ^ {t} (f) = L _ {h} ^ {t} (f) - L _ {h} ^ {t} \left(f ^ {*}\right). \tag {23}
$$
Then, the updating rules of the posterior distribution in Algorithm 1 have the following equivalent forms
$$
p ^ {t} (f) \propto p ^ {0} (f) \exp \left[ \gamma_ {1} V _ {f} ^ {*} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} L _ {h} ^ {\tau} (f) \right]
$$
$$
\Longleftrightarrow p ^ {t} (f) \propto p ^ {0} (f) \exp \left[ \gamma_ {1} \Delta V _ {f} ^ {*} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} \Delta L _ {h} ^ {\tau} (f) \right], \tag {24}
$$
$$
q ^ {t} (f) \propto q ^ {0} (f) \exp \left[ - \gamma_ {2} V _ {f} ^ {\pi^ {t}, *} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} L _ {h} ^ {\tau} (f) \right]
$$
$$
\Longleftrightarrow q ^ {t} (f) \propto q ^ {0} (f) \exp \left[ - \gamma_ {2} \Delta V _ {f} ^ {\pi^ {t}, *} + \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} \Delta L _ {h} ^ {\tau} (f) \right]. \tag {25}
$$
since here adding or subtracting terms irrelevant to  $f$ , i.e.,  $V_{f^*}^{\pi, *}, V_{f^*}^*$ , and  $L_h^t(f^*)$ , within the power of all exponential terms will not change the posterior distribution of  $f$ .
To learning a sublinear upper bound for the expected value of the total regret, i.e.,  $\mathbb{E}[\mathrm{Reg}^{\mathrm{sp}}(T)] = \mathbb{E}\sum_{t=1}^{T}[V_{f^*}^{*,\nu^t} - V_{f^*}^{\pi^t,*}]$  for the self-play setting, we need to execute both Algorithm 1 and 3, which are two symmetric algorithms. We can decompose  $\mathrm{Reg}^{\mathrm{sp}}(T)$  as
$$
\operatorname {R e g} ^ {\mathrm {s p}} (T) = \operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) + \operatorname {R e g} _ {2} ^ {\mathrm {s p}} (T),
$$
$$
\operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) := \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, *} \right], \qquad \operatorname {R e g} _ {2} ^ {\mathrm {s p}} (T) := \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {* , \nu^ {t}} - V _ {f ^ {*}} ^ {*} \right].
$$
In fact, executing Algorithm 1 leads to a low regret upper bound for  $\mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T)]$  while running Algorithm 3 incurs a low regret upper bound for  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$ . Since the two algorithms are symmetric, the derivation of their respective regret bounds is thus similar. In the following subsections, we present the proofs of Proposition 1, Proposition 2, and Theorem 1 sequentially.
Proof. We start our proof by first decomposing the regret as follows
$$
\operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) = \underbrace {\sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t} , \underline {{\nu}} ^ {t}} ]} _ {\text {T e r m (i)}} + \underbrace {\sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {\pi^ {t} , \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t} , *} ]} _ {\text {T e r m (i i)}}.
$$
Our goal is to give the upper bound for the expected value of the total regret, which is  $\mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T)]$ . Thus, we need to derive the upper bounds of  $\mathbb{E}[\mathrm{Term}(\mathrm{i})]$  and  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$  separately. Intuitively, according to our analysis below,  $\mathbb{E}[\mathrm{Term}(\mathrm{i})]$  can be viewed as the regret incurred by the updating rules for the main player in Line 3 and Line 4 of Algorithm 1, and  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$  is associated with the exploiter's updating rules in Line 5 and Line 6 of Algorithm 1,
Bound  $\mathbb{E}[\mathrm{Term}(\mathbf{i})]$ . To bound  $\mathbb{E}[\mathrm{Term}(\mathbf{i})]$ , we give the following decomposition
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i}) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*} - V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \bar {\nu} ^ {t}} + V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \bar {\nu} ^ {t}} - V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} + V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} \right] \\ \leq \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {*} - V _ {\overline {{f}} ^ {t}} ^ {\pi^ {t}, \overline {{\nu}} ^ {t}} + V _ {\overline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} ] \\ = - \sum_ {t = 1} ^ {T} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} + \sum_ {t = 1} ^ {T} [ V _ {\overline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} ], \\ \end{array}
$$
where according to Algorithm 1, we have  $(\pi^t,\overline{\nu}^t) = \mathrm{argmax}_{\pi}\mathrm{argmin}_{\nu}V_{\overline{f}^t}^{\pi ,\nu}$ , i.e.,  $(\pi^{t},\overline{\nu}^{t})$  is the NE of  $V_{\overline{f}}^{\pi ,\nu}$ . Thus, the second equality is by (22), and the inequality is due to
$$
V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \bar {\nu} ^ {t}} = \min  _ {\nu} V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \nu} \leq V _ {\bar {f} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}}.
$$
According to the condition (1) in Definition 1 for self-play GEC and the updating rules in Algorithm 1, setting the exploration policy pair as  $\sigma^t = (\pi^t,\underline{\nu}^t)$ , and  $\rho^t = \overline{f}^t$  for Definition 1, we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {\overline {{f}} ^ {t}, 1} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}}\right) \right| \\ \leq \left[ d _ {\text {G E C}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\sigma^ {\iota}, h)} \ell \left(\bar {f} ^ {t}, \xi_ {h} ^ {\iota}\right)\right) \right] ^ {1 / 2} + 2 H \left(d _ {\text {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq \frac {1}{\gamma_ {1}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\bar {f} ^ {t}, \xi_ {h} ^ {\iota})\right) + \frac {\gamma_ {1} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T, \\ \end{array}
$$
where the second inequality is due to  $\sqrt{xy} \leq \frac{1}{\gamma_1} x^2 + \frac{\gamma_1}{4} y^2$ . Combining the above results, we have
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {i}) = \frac {1}{\gamma_ {1}} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell \left(\bar {f} ^ {t}, \xi_ {h} ^ {\iota}\right)\right) - \sum_ {t = 1} ^ {T} \Delta V _ {\bar {f} ^ {t}} ^ {*} \\ + \frac {\gamma_ {1} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T. \\ \end{array}
$$
We need to further bound the RHS of the above equality. Note that for FOMGs and POMGs, we have different definitions of  $\ell(f, \xi_h)$ . Specifically, according to our definition of  $\ell(f, \xi_h^t)$ , for FOMGs, we define
$$
\ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) = D _ {\mathrm {H e}} ^ {2} (\mathbb {P} _ {\overline {{f}} ^ {t}, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota})),
$$
and for POMGs, we have the relation
$$
\mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) = D _ {\mathrm {H e}} ^ {2} (\mathbb {P} _ {\overline {{f}} ^ {t}, h} ^ {\pi^ {\iota}, \underline {{\nu}} ^ {\iota}}, \mathbb {P} _ {f ^ {*}, \overline {{h}}} ^ {\pi^ {\iota}, \underline {{\nu}} ^ {\iota}}).
$$
On the other hand, for the two MG settings, we also define  $L_h^t(f)$  as in (1) and (2). According to Lemmas 11 and 12, unifying their results, we can obtain
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {t}, \nu^ {t}, h)} \ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {t}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left[ L _ {h} ^ {t} \left(\overline {{f}} ^ {t}\right) - L _ {h} ^ {t} \left(f ^ {*}\right) \right] + \log \frac {p ^ {t} \left(\overline {{f}} ^ {t}\right)}{p ^ {0} \left(\overline {{f}} ^ {t}\right)} \right], \\ \end{array}
$$
which further gives
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ - \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\overline {{f}} ^ {t}, \xi_ {h} ^ {\iota}) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ - \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {t} (\overline {{f}} ^ {t}) + \log \frac {p ^ {t} (\overline {{f}} ^ {t})}{p ^ {0} (\overline {{f}} ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ - \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\overline {{f}} ^ {t}) - \log p ^ {0} (\overline {{f}} ^ {t}) + \log p ^ {t} (\overline {{f}} ^ {t}) \right]. \\ \end{array}
$$
where  $\Delta L_h^t (f)\coloneqq L_h^t (f) - L_h^t (f^*)$  is defined in (23). Note that in the above analysis, we slightly abuse the notation of  $\pi^t$  and  $\underline{\nu}^{t}$ , which denote the Markovian policies for FOMG and history-dependent policies for POMGs. Therefore, according to Lemma 10, we obtain that the distribution
$$
\begin{array}{l} p ^ {t} (f) \propto \exp \left(\gamma_ {1} \Delta V _ {f} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log p ^ {0} (f)\right) \\ = p ^ {0} (f) \exp \left(\gamma_ {1} \Delta V _ {f} ^ {*} + \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f)\right) \\ \end{array}
$$
minimizes the last term in the above inequality. As discussed in (24), this distribution is an equivalent form of the posterior distribution updating rule for  $p^t$  adopted in Line 3 of Algorithm 1. This result implies that
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {\overline {{f}} ^ {t} \sim p ^ {t}} \left[ - \gamma_ {1} \Delta V _ {\overline {{f}} ^ {t}} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (\overline {{f}} ^ {t}) + \log \frac {p ^ {t} (\overline {{f}} ^ {t})}{p ^ {0} (\overline {{f}} ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \inf _ {p} \mathbb {E} _ {f \sim p} \left[ - \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right]. \\ \end{array}
$$
By Definition 2, further letting  $\widetilde{p} \coloneqq p^0(f) \cdot \mathbf{1}(f \in \mathcal{F}(\varepsilon)) / p^0(\mathcal{F}(\varepsilon))$ , we have
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ - \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f \sim \widetilde {p}} \left[ - \gamma_ {1} \Delta V _ {f} ^ {*} - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \Delta L _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right] \\ \leq \left(3 H ^ {2} \gamma_ {1} + \eta H T\right) \varepsilon - \log p ^ {0} (\mathcal {F} (\varepsilon)) \\ \leq \omega (4 H T, p ^ {0}), \\ \end{array}
$$
where the second inequality is by Lemma 13 and Lemma 14 as well as  $\varepsilon \leq 1$ , and the last inequality is by our setting that  $\eta = \frac{1}{2}$  and  $H\gamma_{1} \leq T$ . Therefore, by combining the above results, we obtain
$$
\begin{array}{l} \mathbb {E} [ \text {T e r m (i)} ] \leq \frac {\omega (H T , p ^ {0}) T}{\gamma_ {1}} + \frac {\gamma_ {1} d _ {\mathrm {G E C}} H}{4} + 2 H \left(d _ {\mathrm {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq 4 \sqrt {d _ {\mathrm {G E C}} H T \omega \left(4 H T , p ^ {0}\right)}, \tag {26} \\ \end{array}
$$
where we set  $HT > d_{\mathrm{GEC}}$ ,  $\epsilon = 1 / \sqrt{HT}$  for self-play GEC in Definition 1, and  $\gamma_1 = 2\sqrt{\omega(4HT,p^0)T / d_{\mathrm{GEC}}}$ . Since we set  $\epsilon = 1 / \sqrt{HT}$ , the self-play GEC  $d_{\mathrm{GEC}}$  is defined associated with  $\epsilon = 1 / \sqrt{HT}$  by Definition 1. However, our analysis in Appendix B shows that this setting only introduces  $\log T$  factors in  $d_{\mathrm{GEC}}$  in the worst case.
Bound  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$ . Next, we consider to bound  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$ . We start by decomposing Term(ii) as follows,
$$
\begin{array}{l} \mathrm {T e r m} (\mathrm {i i}) = \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} + V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, *} ] \\ = \sum_ {t = 1} ^ {T} [ V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} ] + \sum_ {t = 1} ^ {T} \Delta V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, *}, \\ \end{array}
$$
where the second equality is by (21) and the fact that  $\underline{\nu}^t = \mathrm{argmin}_{\nu} V_{\underline{f}^t}^{\pi^t, \nu}$ .
In addition, according to Definition 1 and the updating rules in Algorithm 1, setting  $\sigma^t = (\pi^t,\underline{\nu}^t)$ ,  $\rho^t = \underline{f}^t$ , we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {\underline {{f}} ^ {t}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}}\right) \right| \\ \leq \left[ d _ {\mathrm {G E C}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\sigma^ {\iota}, h)} \ell \left(\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota}\right)\right) \right] ^ {1 / 2} + 2 H \left(d _ {\mathrm {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T \\ \leq \frac {1}{\gamma_ {2}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \underline {{\nu}} ^ {\iota}, h)} \ell (\underline {{f}} ^ {t}, \xi_ {h} ^ {\iota})\right) + \frac {\gamma_ {2} d _ {\mathrm {G E C}}}{4} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T, \\ \end{array}
$$
where the second inequality is due to  $\sqrt{xy} \leq \frac{1}{\gamma_2} x^2 + \frac{\gamma_2}{4} y^2$ . Thus, we obtain that