SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the first inequality uses Jensen's inequality and the second inequality is by  $\sup_{\pi, \nu} D_{\mathrm{He}}^{2}(\mathbf{P}_{\tilde{f}, H}^{\pi, \nu}, \mathbf{P}_{f^{*}, H}^{\pi, \nu}) \leq \gamma$  and  $0 \leq \int \sqrt{\mathrm{d}P \mathrm{d}Q} \leq 1$ . To connect to the definition of  $\mathcal{F}(\varepsilon)$ , we further invoke Theorem 9 from [55] and obtain
$$
\mathrm {K L} \left(\mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} | | \mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu}\right) \leq \zeta (B / \nu) D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu}, \mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu}\right)
$$
for any policy pair  $(\pi, \nu)$ , where  $\zeta(b) \leq \max \left\{1, \frac{b \log b}{(1 - \sqrt{b})^2} \right\}$  for  $b > 1$ . Plugging the above inequalities together, we obtain
$$
\mathrm {K L} \left(\mathbf {P} _ {f ^ {*}, H} ^ {\pi , \nu} \mid \left| \mathbf {P} _ {f ^ {\prime}, H} ^ {\pi , \nu} \right.\right) \leq \zeta (B / \nu) (\gamma + \nu). \tag {18}
$$
It remains to find a proper choice of  $\gamma$  and  $\nu$  to obtain  $\zeta(B / \nu)(\gamma + \nu) \leq \varepsilon$ . We choose  $\nu = \varepsilon / (6\log(6B^2 / \varepsilon))$  and  $\gamma = \varepsilon / (6\log(B / \nu))$ . Given the condition  $B \geq \log(6B^2 / \varepsilon)$ , we have
$$
\nu = \frac {\varepsilon}{6 \log \left(\frac {6 B ^ {2}}{\varepsilon}\right)} \leq \frac {\varepsilon}{6 B},
$$
$$
\nu = \frac {\varepsilon}{6 \log \left(\frac {6 B ^ {2}}{\varepsilon}\right)} \leq \frac {\varepsilon}{6 \log \left(\frac {B}{\nu}\right)} = \gamma . \tag {19}
$$
Given  $\varepsilon \leq 2 / 3$  and  $B\geq 1$  , we obtain
$$
\nu \leq \frac {\varepsilon}{6 B} \leq \frac {1}{9 B} \leq \frac {B}{9},
$$
namely  $B / \nu \geq 9$ . We note that when  $b > 9$ , it holds that
$$
\zeta (b) \leq \frac {b \log b}{(1 - \sqrt {b}) ^ {2}} \leq \frac {b \log b}{b - 2 \sqrt {b}} \leq 3 \log b.
$$
Thus we have  $\zeta(B / \nu) \leq 3 \log(B / \nu)$ . Finally, we obtain
$$
\zeta (B / \nu) (\gamma + \nu) \leq 2 \gamma \zeta (B / \nu) = \frac {\varepsilon}{3 \log \left(\frac {B}{\nu}\right)} \zeta (B / \nu) \leq \varepsilon ,
$$
where the first inequality uses (19), the first equation uses the choice of  $\gamma$ , and the second inequality uses  $\zeta(B / \nu) \leq 3 \log(B / \nu)$ . Choosing  $p^0$  to be a uniform distribution on  $\mathcal{C}_{\nu}(\gamma)$  completes the proof.
D Technical Lemmas for Main Theorems
In this section, we first provide several important supporting lemmas used in the proofs of Theorems 1 and 2. We then present detailed proofs for these lemmas.
D.1 Lemmas
Lemma 10. Let  $v$  be any probability distribution over  $f \in \mathcal{F}$  where  $\mathcal{F}$  is an arbitrary set. Then,  $\mathbb{E}_{f \sim v(\cdot)}[G(f) + \log v(f)]$  is minimized at  $v(f) \propto \exp(-G(f))$ .
Proof. This lemma is a corollary of Gibbs variational principle. For the detailed proof of this lemma, we refer the readers to the proof of Lemma 4.10 in [64]. This completes the proof.  $\square$
The above lemma states that  $v(f)$  in the above-described form solves the minimization problem  $\min_{v \in \Delta(\mathcal{F})} \mathbb{E}_{f \sim v}[G(f) + \log v(f)]$ , which helps to understand the design of the posterior sampling steps in our proposed algorithms. This lemma is also used in the proofs of the following two lemmas.
The following two lemmas provide the upper bounds for the expectation of the Hellinger distance by the likelihood functions defined in (1) and (2) for FOMGs and POMGs respectively.
Lemma 11. Under the FOMG setting, for any  $t \geq 1$ , let  $Z^t$  be the system randomness history up to the  $t$ -th episode,  $p^t(\cdot|Z^{t-1})$  be any posterior distribution over the function class  $\mathcal{F}$  with  $p^0(\cdot)$  denoting an initial distribution, and  $(\pi^t, \nu^t)$  be any Markovian policy pair for Player 1 and Player 2 depending on  $f^t \sim p^t$ . Suppose that  $(s_h^t, a_h^t, b_h^t, s_{h+1}^t)$  is a data point sampled independently by executing the policy pair  $(\pi^t, \nu^t)$  to the  $h$ -th step of the  $t$ -th episode. If we define  $L_h^t(f) \coloneqq \eta \log \mathbb{P}_{f,h}(s_{h+1}^t | s_h^t, a_h^t, b_h^t)$  with  $\eta = 1/2$  as in (1), we have the following relation
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \left[ D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t}, h} \left(\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right), \mathbb {P} _ {f ^ {*}, h} \left(\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right)\right) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right], \\ \end{array}
$$
where  $\mathbb{E}_{(\pi^t,\nu^t,h)}$  denotes taking an expectation over the data  $(s_h^t,a_h^t,b_h^t)$  sampled following the policy pair  $(\pi^{t},\nu^{t})$  and the true model  $\mathbb{P}_{f^*}$  up to the  $h$ -th step at any  $t$ .
Proof. Please see Appendix D.2 for a detailed proof.
Lemma 12. Under the POMG setting, for any  $t \geq 1$ , let  $Z^t$  be the system randomness history up to the  $t$ -th episode,  $p^t (\cdot |Z^{t - 1})$  be any posterior distribution over the function class  $\mathcal{F}$  with  $p^0 (\cdot)$  denoting an initial distribution, and  $(\pi^t,\nu^t)$  be any general history-dependent policy pair for Player 1 and Player 2 depending on  $f^t \sim p^t$ . Suppose that  $\tau_h^t = (o_1^t,a_1^t,b_1^t\dots,o_h^t,a_h^t,b_h^t)$  is a data point sampled independently by executing the policy pair  $(\pi^t,\nu^t)$  to the  $h$ -th step of the  $t$ -th episode. If we define  $L_h^t (f)\coloneqq \eta \log \mathbb{P}_f(\tau_h^t)$  with  $\eta = 1 / 2$  as in (2), we have the following relation
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} [ D _ {\mathrm {H e}} ^ {2} (\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}}, \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}}) ] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right], \\ \end{array}
$$
where  $\mathbb{P}_{f,h}^{\pi,\nu}$  denotes the distribution for  $\tau_h = (o_1, a_1, b_1, \ldots, o_h, a_h, b_h)$  under the model  $\theta_f$  and the policy pair  $(\pi, \nu)$  up to the  $h$ -th step.
Proof. Please see Appendix D.3 for a detailed proof.
The next lemma shows that when the model is sufficiently close to  $f^{*}$  with the distances employed in Definition 2, the following value differences are small enough under both FOMG and POMG settings.
Lemma 13. If the model  $f$  satisfies  $\sup_{h,s,a,b} \mathrm{KL}^{\frac{1}{2}}(\mathbb{P}_{f^{*},h}(\cdot |s,a,b)\| \mathbb{P}_{f,h}(\cdot |s,a,b)) \leq \varepsilon$  for FOMGs and  $\sup_{\pi,\nu} \mathrm{KL}^{\frac{1}{2}}(\mathbf{P}_{f^{*},H}^{\pi,\nu}\| \mathbf{P}_{f,H}^{\pi,\nu}) \leq \varepsilon$  for POMGs, we have that their corresponding value function satisfies
$$
V _ {f ^ {*}} ^ {*} - V _ {f} ^ {*} \leq 3 H \varepsilon , \quad \sup  _ {\pi} (V _ {f ^ {*}} ^ {\pi , *} - V _ {f} ^ {\pi , *}) \leq 3 H \varepsilon , \quad \sup  _ {\nu} (V _ {f ^ {*}} ^ {*, \nu} - V _ {f} ^ {*, \nu}) \leq 3 H \varepsilon .
$$
Proof. Please see Appendix D.4 for detailed proof.
Finally, we show that when the model is sufficiently close to  $f^{*}$ , we will obtain that the following likelihood function difference is small under both FOMG and POMG settings.
Lemma 14. If the model  $f$  satisfies  $\sup_{h,s,a,b} \mathrm{KL}^{\frac{1}{2}}(\mathbb{P}_{f^{*},h}(\cdot |s,a,b)\| \mathbb{P}_{f,h}(\cdot |s,a,b)) \leq \varepsilon$  for FOMGs and  $\sup_{\pi,\nu} \mathrm{KL}^{\frac{1}{2}}(\mathbf{P}_{f^{*},H}^{\pi,\nu}\| \mathbf{P}_{f,H}^{\pi,\nu}) \leq \varepsilon$  for POMGs, we have that their corresponding likelihood function defined in (1) and (2) satisfies
$$
\left| \mathbb {E} \left(L _ {h} ^ {t} (f) - L _ {h} ^ {t} \left(f ^ {*}\right)\right) \right| \leq \eta \varepsilon^ {2},
$$
where the expectation is taken with respect to the randomness in  $L_{h}^{t}$ .
Proof. Please see Appendix D.5 for detailed proof.
D.2 Proof of Lemma 11
Proof. The proof of Lemma 11 can be viewed as a multi-agent generalization of the proof for Lemma E.5 in [83]. We start our proof by first considering the following equality
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \eta \log \frac {\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}{\mathbb {P} _ {f ^ {t} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})} + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right]. \\ \end{array}
$$
Next, we lower bound RHS of the above equality. We define
$$
\overline {{L}} _ {h} ^ {\iota} (f) := \eta \log \frac {\mathbb {P} _ {f , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}{\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}, \qquad \widetilde {L} _ {h} ^ {\iota} (f) := \overline {{L}} _ {h} ^ {\iota} (f) - \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}, h}) [ \exp (\overline {{L}} _ {h} ^ {\iota} (f)) ]},
$$
where  $\mathbb{E}_{(\pi^{\iota},\nu^{\iota},\mathbb{P}_{f^{*}},h)}$  denotes taking an expectation over the data  $(s_h^\iota ,a_h^\iota ,b_h^\iota ,s_{h + 1}^\iota)$  sampled following the policy pair  $(\pi^{\iota},\nu^{\iota})$  and the true model  $\mathbb{P}_{f^*}$  to the  $h$ -th step with  $s_{h + 1}^{\iota}\sim \mathbb{P}_{f^{*},h}(\cdot |s_{h}^{\iota},a_{h}^{\iota},b_{h}^{\iota})$  at round  $\iota$ . Then, we will show that  $\mathbb{E}_{Z^{t - 1}}[\exp (\sum_{h = 1}^{H}\sum_{\iota = 1}^{t - 1}\widetilde{L}_h^\iota (f))] = 1$  by induction, following from [80]. Suppose that for any  $k$ , we have at  $k - 1$  that  $\mathbb{E}_{Z^{k - 1}}[\exp (\sum_{h = 1}^{H}\sum_{\iota = 1}^{k - 1}\widetilde{L}_h^\iota (f))] = 1$ . Then, at  $k$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {Z} ^ {k}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {k} \widetilde {L} _ {h} ^ {\iota} (f)\right) \right] = \mathbb {E} _ {\mathcal {Z} ^ {k - 1}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {k - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \exp \left(\sum_ {h = 1} ^ {H} \widetilde {L} _ {h} ^ {k} (f)\right) \right] \\ = \mathbb {E} _ {Z ^ {k - 1}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {k - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \mathbb {E} _ {f ^ {k} \sim p ^ {k}} \prod_ {h = 1} ^ {H} \mathbb {E} _ {(\pi^ {k}, \nu^ {k}, \mathbb {P} _ {f ^ {*}}, h)} \exp \left(\widetilde {L} _ {h} ^ {k} (f)\right) \right] \\ = \mathbb {E} _ {Z ^ {k - 1}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {k - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \right] = 1, \\ \end{array}
$$
where the second equality uses the fact that the data is sampled independently, the third equality is due to  $\mathbb{E}_{(\pi^k,\nu^k,\mathbb{P}_{f^*},h)}\exp (\widetilde{L}_h^k (f)) = \mathbb{E}_{(\pi^k,\nu^k,\mathbb{P}_{f^*},h)}\exp (\overline{L}_h^k (f)) / \mathbb{E}_{(\pi^k,\nu^k,\mathbb{P}_{f^*},h)}[\exp (\overline{L}_h^k (f))] = 1$  by the definition of  $\widetilde{L}_h^k (f)$ , and the last equality is by  $\mathbb{E}_{Z^{k - 1}}[\exp (\sum_{h = 1}^{H}\sum_{\iota = 1}^{k - 1}\widetilde{L}_h^\iota (f))] = 1$  in the above assumption. Moreover, for  $k = 1$ , we have a trivial result that  $\mathbb{E}_{Z^1}[\exp (\sum_{h = 1}^H\widetilde{L}_h^1 (f))] =$
$\mathbb{E}_{f^1 \sim p^1} \prod_{h=1}^{H} \mathbb{E}_{(\pi^1, \nu^1, \mathbb{P}_{f^*}, h)} \exp(\widetilde{L}_h^1(f)) = 1$ . Consequently, we conclude that for any  $k$ , the above equality holds. Then, when  $k = t - 1$ , we have
$$
\mathbb {E} _ {Z ^ {t - 1}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \right] = 1. \tag {20}
$$
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f ^ {t}) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ \geq \mathbb {E} _ {Z ^ {t - 1}} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f) + \log \frac {p (f)}{p ^ {0} (f)} \right] \\ = - \mathbb {E} _ {Z ^ {t - 1}} \log \mathbb {E} _ {f \sim p ^ {0}} \exp \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f) \right] \\ \geq - \log \mathbb {E} _ {f \sim p ^ {0}} \mathbb {E} _ {Z ^ {\iota - 1}} \exp \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f) \right] = 0, \\ \end{array}
$$
where the last inequality is by Jensen's inequality and the last equality is due to (20). For the first equality, we use the fact that the following distribution is the minimizer
$$
\begin{array}{l} p (f) \propto \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f) + \log p ^ {0} (f)\right) = p ^ {0} (f) \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \\ \iff p (f) = \frac {p ^ {0} (f) \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right)}{\int_ {\mathcal {F}} p ^ {0} (f) \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \mathrm {d} f} = \frac {p ^ {0} (f) \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right)}{\mathbb {E} _ {f \sim p ^ {0}} \left[ \exp \left(\sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f)\right) \right]} \\ \end{array}
$$
according to Lemma 10, such that plugging in the above distribution leads to the first equality. Thus, according to the definitions of  $\widetilde{L}_h^t$  and  $\overline{L}_h^t$ , we have
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \eta \log \frac {\mathbb {P} _ {f , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}{\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})} + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ \geq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}, h)} \exp \left(\eta \log \frac {\mathbb {P} _ {f , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}{\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}\right) \right]. \\ \end{array}
$$
Moreover, to further lower bound the RHS of the above inequality, by the inequality that  $\log x\leq x - 1$  and the setting  $\eta = \frac{1}{2}$ , we have
$$
\begin{array}{l} - \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}, h)} \exp \left(\log \frac {\sqrt {\mathbb {P} _ {f , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}}{\sqrt {\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}}\right) \\ = - \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}, h)} \frac {\sqrt {\mathbb {P} _ {f , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}}{\sqrt {\mathbb {P} _ {f ^ {*} , h} (s _ {h + 1} ^ {\iota} | s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota})}} \\ \geq 1 - \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, \mathbb {P} _ {f ^ {*}}, h)} \frac {\sqrt {\mathbb {P} _ {f , h} \left(s _ {h + 1} ^ {\iota} \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}\right)}}{\sqrt {\mathbb {P} _ {f ^ {*} , h} \left(s _ {h + 1} ^ {\iota} \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}\right)}} \\ = 1 - \mathbb {E} _ {\left(\pi^ {\prime}, \nu^ {\prime}, h\right)} \int_ {\mathcal {S}} \sqrt {\mathbb {P} _ {f , h} \left(s \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}\right) \mathbb {P} _ {f ^ {*} , h} \left(s \mid s _ {h} ^ {\iota} , a _ {h} ^ {\iota} , b _ {h} ^ {\iota}\right)} \mathrm {d} s \\ = \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} [ D _ {\mathrm {H e}} ^ {2} (\mathbb {P} _ {f, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota})) ], \\ \end{array}
$$
where the last equality is by  $D_{\mathrm{He}}^2 (P,Q) = \frac{1}{2}\int (\sqrt{\mathrm{d}P(x)} -\sqrt{\mathrm{d}Q(x)})^2 = 1 - \int \sqrt{\mathrm{d}P(x)\mathrm{d}Q(x)}$  Here  $\mathbb{E}_{(\pi^{\iota},\nu^{\iota},h)}$  denotes taking expectation over  $(s_h^\iota ,a_h^\iota ,b_h^\iota)$  sampled following the policy pair  $(\pi^{\iota},\nu^{\iota})$  and the true model  $\mathbb{P}_{f^{*}}$  to the  $h$  -th step but without the next state  $s_{h + 1}^{\iota}$  generated by  $\mathbb{P}_{f^{*},h}(\cdot |s_h^\iota ,a_h^\iota ,b_h^\iota)$  at round  $\iota$  . In the sequel, combining the above results, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {Z} ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ \geq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \left[ D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right), \mathbb {P} _ {f ^ {*}, h} \left(\cdot \mid s _ {h} ^ {\iota}, a _ {h} ^ {\iota}, b _ {h} ^ {\iota}\right)\right) \right] \right]. \\ \end{array}
$$
D.3 Proof of Lemma 12
Proof. The proof of Lemma 12 is similar to the proof of Lemma 11. We will give a brief description of the main steps for our proof of Lemma 12. We start our proof by considering the following equality
$$
\begin{array}{l} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \\ = \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \eta \log \frac {\mathbf {P} _ {f ^ {*} , h} \left(\tau_ {h} ^ {\iota}\right)}{\mathbf {P} _ {f ^ {t} , h} \left(\tau_ {h} ^ {\iota}\right)} + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right]. \\ \end{array}
$$
$$
\overline {{L}} _ {h} ^ {\iota} (f) := \eta \log \frac {\mathbf {P} _ {f , h} (\tau_ {h} ^ {\iota})}{\mathbf {P} _ {f ^ {*} , h} (\tau_ {h} ^ {\iota})}, \qquad \widetilde {L} _ {h} ^ {\iota} (f) := \overline {{L}} _ {h} ^ {\iota} (f) - \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} [ \exp (\overline {{L}} _ {h} ^ {\iota} (f)) ],
$$
where  $\mathbb{E}_{(\pi^{\iota},\nu^{\iota},h)}$  denotes taking an expectation over the data  $\tau_h^\iota$  sampled following the policy pair  $(\pi^{\iota},\nu^{\iota})$  and the true model  $\theta_{f^{*}}$  to the  $h$ -th step at round  $\iota$ . Then, we can show that
$$
\mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \widetilde {L} _ {h} ^ {\iota} (f ^ {t}) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right] \geq 0,
$$
following a similar derivation as (20) in the proof of Lemma 11. With setting  $\eta = \frac{1}{2}$ , this result further leads to
$$
\begin{array}{l} - \log \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \exp \left(\log \frac {\sqrt {\mathbf {P} _ {f , h} (\tau_ {h} ^ {\iota})}}{\sqrt {\mathbf {P} _ {f ^ {*} , h} (\tau_ {h} ^ {\iota})}}\right) \\ \geq 1 - \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \frac {\sqrt {\mathbf {P} _ {f , h} (\tau_ {h} ^ {\iota})}}{\sqrt {\mathbf {P} _ {f ^ {*} , h} (\tau_ {h} ^ {\iota})}} = 1 - \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} \frac {\sqrt {\mathbf {P} _ {f , h} ^ {\pi^ {\iota} , \nu^ {\iota}} (\tau_ {h} ^ {\iota})}}{\sqrt {\mathbf {P} _ {f ^ {*} , h} ^ {\pi^ {\iota} , \nu^ {\iota}} (\tau_ {h} ^ {\iota})}} \\ = 1 - \int_ {(\mathcal {O} \times \mathcal {A}) ^ {h}} \sqrt {\mathbf {P} _ {f , h} ^ {\pi^ {\iota} , \nu^ {\iota}} (\tau_ {h}) \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota} , \nu^ {\iota}} (\tau_ {h})} d \tau_ {h} \\ = D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}\right). \\ \end{array}
$$
In the sequel, combining the above results, we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbb {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}\right) \right] \\ \leq \mathbb {E} _ {Z ^ {t - 1}} \mathbb {E} _ {f ^ {t} \sim p ^ {t}} \left[ - \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \left(L _ {h} ^ {\iota} (f ^ {t}) - L _ {h} ^ {\iota} (f ^ {*})\right) + \log \frac {p ^ {t} (f ^ {t})}{p ^ {0} (f ^ {t})} \right]. \\ \end{array}
$$
D.4 Proof of Lemma 13
Proof. We first prove the upper bound of  $|V_{f^*}^{\pi, \nu} - V_f^{\pi, \nu}|$  for any  $(\pi, \nu)$  and  $f$  under FOMG and POMG settings separately.
For FOMGs, we let  $V_{f^*}^{\pi,\nu} = V_{f^*}^{\pi,\nu}$  and  $V_{f}^{\pi,\nu} = V_{f}^{\pi,\nu}$ . Then, according to the Bellman equation that  $Q_{f,h}^{\pi,\nu}(s,a,b) = r_{h}(s,a,b) + \langle \mathbb{P}_{f,h}(\cdot|s,a,b), V_{f,h+1}^{\pi,\nu}(\cdot) \rangle$  and also  $V_{f,h}^{\pi,\nu}(s,a,b) = \mathbb{E}_{a \sim \pi_{h}(\cdot|s), b \sim \nu_{h}(\cdot|s)}[Q_{f,h}^{\pi,\nu}(s,a,b)]$ , we have
$$
\begin{array}{l} \left| V _ {f ^ {*}} ^ {\pi , \nu} - V _ {f} ^ {\pi , \nu} \right| \\ \leq \mathbb {E} _ {\pi , \nu} \left| \langle \mathbb {P} _ {f ^ {*}} (\cdot | s _ {1}, a _ {1}, b _ {1}), V _ {f ^ {*}, 2} ^ {\pi , \nu} (\cdot) \rangle - \langle \mathbb {P} _ {f, 1} (\cdot | s _ {1}, a _ {1}, b _ {1}), V _ {f, 2} ^ {\pi , \nu} (\cdot) \rangle \right| \\ \leq H \mathbb {E} _ {\pi , \nu} \| \mathbb {P} _ {f ^ {*}} (\cdot | s _ {1}, a _ {1}, b _ {1}) - \mathbb {P} _ {f, 1} (\cdot | s _ {1}, a _ {1}, b _ {1}) \| _ {1} + \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \left| V _ {f ^ {*}, 2} ^ {\pi , \nu} (s _ {2}) - V _ {f, 2} ^ {\pi , \nu} (s _ {2}) \right| \\ \end{array}
$$
(Recursively applying the above derivation)
$$
\leq H \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \sum_ {h = 1} ^ {H} | | \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) | | _ {1},
$$