SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The  $\alpha$ -revealing condition measures how hard it is to recover states from observations. Before presenting the GEC bound for an  $\alpha$ -revealing POMG, we first define some notations. We define  $\tau_{h:h'} := (o_i, a_i, b_i)_{i=h}^{h'}$  to be a set of observation-action tuples from step  $h$  to  $h'$ . For simplicity, we denote  $\tau_{1:h}$  as  $\tau_h$ . We define  $\pmb{q}_0 := \mathbb{O}_1 \pmb{\mu}_1 \in \mathbb{R}^{|O|}$ , where  $\pmb{\mu}_1 \in \mathbb{R}^{|S|}$  is a vector formed by the distribution of the initial state. We define  $\pmb{q}(\tau_h) := [\operatorname*{Pr}(o_{1:h}, o_{h+1} | \mathbf{d}\mathbf{o}(a_{1:h}, b_{1:h}))]_{o \in O} \in \mathbb{R}^{|O|}$  when  $\tau_h = (o_i, a_i, b_i)_{i=1}^h$ . Here  $\mathbf{d}\mathbf{o}(a_{1:h}, b_{1:h})$  means executing actions  $a_{h'}$  and  $b_{h'}$  at step  $h'$ . In other words,  $\operatorname*{Pr}(o_{1:h}, o_{h+1} | \mathbf{d}\mathbf{o}(a_{1:h}, b_{1:h}))$  describes the probability of receiving observations  $o_{1:h}$  given the actions fixed as  $(a_{1:h-1}, b_{1:h-1})$ . We define  $\bar{\pmb{q}}(\tau_h) := \pmb{q}(\tau_h) / \mathbf{P}(\tau_h)$ , which is the concatenation of the probabilities under condition  $\tau_h$ . We have  $\bar{\pmb{q}}(\tau_h) = [\operatorname*{Pr}(o | \tau_{1:h})]_{o \in O}$ , where  $\operatorname*{Pr}(o | \tau_{1:h})$  means the probability of receiving  $o_{h+1} = o$  under the condition that the previous trajectory is  $\tau_h$ .
Based on the above definitions, we then derive the GEC bound for an  $\alpha$ -revealing POMG in the following proposition.
Proposition 6 ( $\alpha$ -Revealing POMG  $\subset$  Low Self-Play/Adversarial GEC). For an  $\alpha$ -revealing POMG defined in Definition 8, with the loss  $\ell(f, \xi_h)$  defined in (3) for POMGs, we have
$\alpha$  -revealing  $POMG\subset MG$  with low self-play and adversarial GEC
$$
d _ {\mathrm {G E C}} = O \left(\frac {H ^ {3} | \mathcal {O} | ^ {3} | \mathcal {A} | ^ {2} | \mathcal {B} | ^ {2} | \mathcal {S} | ^ {2} \varsigma}{\alpha^ {4}}\right),
$$
where  $\varsigma = 2\log \left(1 + \frac{4|\mathcal{A}|^2|\mathcal{B}|^2|\mathcal{O}|^2|\mathcal{S}|^2}{\alpha^2}\right)$ .
For our proof of this proposition, we prove several supporting lemmas presented in Appendix B.4.1.
Proof. For ease of notation, we let  $S = |\mathcal{S}|$ ,  $A = |\mathcal{A}|$ ,  $B = |\mathcal{B}|$ , and  $O = |\mathcal{O}|$  in our proof. We define the following observed operator representation:
$$
\boldsymbol {M} _ {h} (o, a, b) := \mathbb {O} _ {h + 1} \mathbb {P} _ {h, a, b} \operatorname {d i a g} \left(\mathbb {O} _ {h} (o | \cdot)\right) \mathbb {O} _ {h} ^ {\dagger} \quad \in \mathbb {R} ^ {O \times O}
$$
for  $1 \leq h \leq H$ . One can verify that the previously defined  $M_h(o, a, b)$  has the following property,
$$
\boldsymbol {q} (\tau_ {h}) = \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h}) \boldsymbol {M} _ {h - 1} (o _ {h - 1}, a _ {h - 1}, b _ {h - 1}) \dots \boldsymbol {M} _ {1} (o _ {1}, a _ {1}, b _ {1}) \boldsymbol {q} _ {0}.
$$
Specifically,  $\mathbf{P}(\tau_H) = M_H(o_H,a_H,b_H)\ldots M_1(o_1,a_1,b_1)\pmb{q}_0$ . For simplicity, we define
$$
\boldsymbol {M} _ {h ^ {\prime}: h} \left(o _ {h: h ^ {\prime}}, a _ {h: h ^ {\prime}}, b _ {h: h ^ {\prime}}\right) := \boldsymbol {M} _ {h ^ {\prime}} \left(o _ {h ^ {\prime}}, a _ {h ^ {\prime}}, b _ {h ^ {\prime}}\right) \boldsymbol {M} _ {h ^ {\prime} - 1} \left(o _ {h ^ {\prime} - 1}, a _ {h ^ {\prime} - 1}, b _ {h ^ {\prime} - 1}\right) \dots \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right).
$$
We also rewrite  $M_{H:h+1}(o_{h+1:H}, a_{h+1:H}, b_{h+1:H})$  as  $m(o_{h+1:H}, a_{h+1:H}, b_{h+1:H})$  to emphasize that  $M_{H:h+1}(o_{h+1:H}, a_{h+1:H}, b_{h+1:H})$  is a vector since  $M_H(o_H, a_H, b_H)$  is a vector.
We start by using Lemma 5 to decompose the regret as follows
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \leq \sum_ {t = 1} ^ {T} \frac {H \sqrt {S}}{\alpha} \Big [ \sum_ {h = 1} ^ {H} \sum_ {\tau_ {h}} \| (M _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - M _ {h} (o _ {h}, a _ {h}, b _ {h})) \boldsymbol {q} (\tau_ {h - 1}) \| _ {1} \sigma^ {t} (\tau_ {h}) + \| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \| _ {1} \Big ]. \\ \end{array}
$$
Combine this with the fact that  $V_{f^t}^{\pi^t,\nu^t} - V_{f^*}^{\pi^t,\nu^t}\leq H$  ,we have
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \leq \sum_ {t = 1} ^ {T} \frac {H \sqrt {S}}{\alpha} \left[ \sum_ {h = 1} ^ {H} \min  \left\{\frac {\alpha}{\sqrt {S}}, \sum_ {\tau_ {h}} \| \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) \| _ {1} \sigma^ {t} \left(\tau_ {h}\right) \right\} \right. \\ \left. + \left\| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \right\| _ {1} \right]. \\ \end{array}
$$
$$
\pi (\tau_ {h}) = \prod_ {h ^ {\prime} = 1} ^ {h} \pi_ {h ^ {\prime}} (a _ {h ^ {\prime}} | \tau_ {h ^ {\prime} - 1}, o _ {h ^ {\prime}}), \quad \nu (\tau_ {h}) = \prod_ {h ^ {\prime} = 1} ^ {h} \nu_ {h ^ {\prime}} (b _ {h ^ {\prime}} | \tau_ {h ^ {\prime} - 1}, o _ {h ^ {\prime}}), \quad \sigma^ {t} (\tau_ {h}) = \pi^ {t} (\tau_ {h}) \nu^ {t} (\tau_ {h}),
$$
and  $M_h^t$  and  $q_0^t$  correspond to the model  $f^t$  sampled in time step  $t$ .
We next focus on  $\sum_{\tau_h}\| (M_h^t (o_h,a_h,b_h) - M_h(o_h,a_h,b_h))q(\tau_{h - 1})\| _1\sigma^t (\tau_h)$ . By the definition of  $\bar{\pmb{q}}$  and  $\pmb{q}$ , we obtain
$$
\begin{array}{l} \sum_ {\tau_ {h}} \left\| \left(\boldsymbol {M} _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h})\right) \boldsymbol {q} (\tau_ {h - 1}) \right\| _ {1} \sigma (\tau_ {h}) \\ = \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} | \bar {\pmb {e}} _ {j} ^ {\top} (\pmb {M} _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - \pmb {M} _ {h} (o _ {h}, a _ {h}, b _ {h})) \bar {\pmb {q}} (\tau_ {h - 1}) | \sigma^ {t} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) \\ = \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} \left| \tilde {\boldsymbol {e}} _ {j} ^ {\top} (\boldsymbol {M} _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h})) \mathbb {O} _ {h - 1} \mathbb {O} _ {h - 1} ^ {\dagger} \right. \\ \left. \cdot \bar {\pmb {q}} (\tau_ {h - 1}) \sigma^ {t} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) \right|, \\ \end{array}
$$
where  $\widetilde{e}_j$  denotes the  $j$ -th standard basis vector in  $\mathbb{R}^O$ , and  $\sigma^t(\tau_{h:h'}; \tau_{h-1}) := \pi^t(\tau_{h:h'}; \tau_{h-1}) \cdot \nu^t(\tau_{h:h'}; \tau_{h-1})$ , where  $\pi^t(\tau_{h:h'}; \tau_{h-1}) = \prod_{h''=h}^{h'} \pi_{h''}^t(a_{h''} | \tau_{h''-1}, o_{h''})$ ,  $\nu^t(\tau_{h:h'}; \tau_{h-1}) = \prod_{h''=h}^{h'} \nu_{h''}^t(b_{h''} | \tau_{h''-1}, o_{h''})$ . For simplicity, we define  $w_{h,t,j,o,a,b} := (\widetilde{e}_j^\top(M_h^t(o,a,b) - M_h(o,a,b)) \mathbb{O}_{h-1})^\top$  and  $x_{\tau_{h-1},\sigma,o,a,b} := \mathbb{O}_{h-1}^\dagger \bar{\boldsymbol{q}}(\tau_{h-1})\sigma(o,a,b; \tau_{h-1})$ . Then, we have
$$
\begin{array}{l} \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} | \widetilde {\boldsymbol {e}} _ {j} ^ {\top} (\boldsymbol {M} _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h})) \mathbb {O} _ {h - 1} \mathbb {O} _ {h - 1} ^ {\dagger} \bar {\boldsymbol {q}} (\tau_ {h - 1}) \sigma^ {t} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) | \\ = \sum_ {o, a, b} \mathbb {E} _ {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {t}, \nu^ {t}}} \sum_ {j = 1} ^ {O} | w _ {h, t, j, o, a, b} ^ {\top} x _ {\tau_ {h - 1}, \sigma^ {t}, o, a, b} |. \\ \end{array}
$$
Next, we will apply the  $\ell_2$  eluder technique [13, 83] in Lemma 17 to derive the upper bound. We first analyze the upper bounds of  $\sum_{j=1}^{O}\|w_{h,t,j,o,a,b}\|_2$  and  $\mathbb{E}_{\tau_{h-1}}\sim \mathbf{P}_{f^*,h-1}^{\pi^t,\nu^t}\|x_{\tau_{h-1},\sigma^t,o,a,b}\|_2^2$  for the usage of the  $\ell_2$  eluder technique. According to Lemma 7, for any  $(o,a,b) \in \mathcal{O} \times \mathcal{A} \times \mathcal{B}$ , we have
$$
\underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \| x _ {\tau_ {h - 1}, \sigma^ {t}, o, a, b} \| _ {2} ^ {2} \leq S, \quad \sum_ {j = 1} ^ {O} \| w _ {h, t, j, o, a, b} \| _ {2} \leq \frac {2 A B d \sqrt {S}}{\alpha}.
$$
According to Lemma 17, setting  $R = \frac{\alpha}{\sqrt{S}}$  and  $R_{x}^{2}R_{w}^{2} \leq \frac{4A^{2}B^{2}O^{2}S^{2}}{\alpha^{2}}$  in the new  $\ell_{2}$  eluder technique, we can obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \min  \left\{\frac {\alpha}{\sqrt {S}}, \underset {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*} , h - 1} ^ {\pi^ {t}, \nu^ {t}}} {\mathbb {E}} \sum_ {j = 1} ^ {O} \sum_ {o, a, b} | w _ {h, t, j, o, a, b} ^ {\top} x _ {\tau_ {h - 1}, \sigma^ {t}, o, a, b} | \right\} \\ \leq \sum_ {o _ {h}, a _ {h}, b _ {h}} \left\{O \varsigma \left[ \frac {\alpha^ {2}}{S} T + \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi_ {\iota}, \nu_ {\iota}}} \left(\| \left(M _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - M _ {h} (o _ {h}, a _ {h}, b _ {h})\right) \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \right. \right. \right. \\ \left. \cdot \sigma^ {\iota} \left(o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}\right)\right) ^ {2} \bigg ] \Bigg \} ^ {\frac {1}{2}}, \\ \end{array}
$$
where  $\varsigma = 2\log \left(1 + \frac{4A^2B^2O^2S^2}{\alpha^2}\right)$ . By Jensen's inequality, the last term above is further relaxed as
$$
\begin{array}{l} \left\{O ^ {2} A B \varsigma \left[ \frac {O A B \alpha^ {2}}{S} T + \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {\iota}, \nu^ {\iota}}} \left(\sum_ {o _ {h}, a _ {h}, b _ {h}} \| \left(M _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - M _ {h} (o _ {h}, a _ {h}, b _ {h})\right) \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \right. \right. \right. \\ \left. \cdot \sigma^ {\iota} \left(o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}\right)\right) ^ {2} \bigg ] \Bigg \} ^ {\frac {1}{2}}. \\ \end{array}
$$
Then, invoking Lemma 6, we obtain
$$
\begin{array}{l} \sum_{t = 1}^{T}\min \Bigl\{\frac{\alpha}{\sqrt{S}},\underset {\tau_{h - 1}\sim \mathbf{P}_{f^{*},h - 1}^{\pi^{t},\nu^{t}}}{\mathbb{E}}\sum_{j = 1}^{O}\sum_{o,a,b}|w_{h,t,j,o,a,b}^{\top}x_{\tau_{h - 1},\sigma^{t},o,a,b}|\Bigr \} \\ \lesssim \left[ O ^ {2} A B \varsigma \left(\frac {O A B \alpha^ {2}}{S} T + \frac {S}{\alpha^ {2}} \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}}, \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {t}, \nu^ {t}}\right)\right) \right] ^ {\frac {1}{2}}, \tag {14} \\ \end{array}
$$
We next focus on  $\| \pmb{q}_0^t -\pmb {q}_0\| _1$  . Using that  $\| \pmb {q}_0^t -\pmb {q}_0\| _1\leq 1$  and  $2(t - 1)\geq t$  when  $t\geq 2$  , we obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \right\| _ {1} \leq 1 + \sum_ {t = 2} ^ {T} \left[ \frac {2 (t - 1)}{t} \left\| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \right\| _ {1} ^ {2} \right] ^ {\frac {1}{2}} \\ \lesssim 1 + \left[ \sum_ {t = 2} ^ {T} \frac {2}{t} \right] ^ {\frac {1}{2}} \left[ \sum_ {t = 2} ^ {T} (t - 1) D _ {\mathrm {H e}} ^ {2} \left(\boldsymbol {q} _ {0} ^ {t}, \boldsymbol {q} _ {0}\right) \right] ^ {\frac {1}{2}} \\ \lesssim 1 + \left[ \log T \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} D _ {\mathrm {H e}} ^ {2} \left(\boldsymbol {q} _ {0} ^ {t}, \boldsymbol {q} _ {0}\right) \right] ^ {\frac {1}{2}}. \tag {15} \\ \end{array}
$$
Thus, by invoking (14) and (15), we get the regret bound by
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \lesssim H \left(O A B H \sqrt {O _ {\zeta} T} + \sqrt {\frac {O ^ {2} A B _ {\zeta} S ^ {2} H}{\alpha^ {4}} \sum_ {h = 0} ^ {H} \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {t} , h} ^ {\pi^ {\iota} , \nu^ {\iota}}, \mathbf {P} _ {f ^ {*} , h} ^ {\pi^ {\iota} , \nu^ {\iota}}\right)}\right) \\ \lesssim \left[ \frac {O ^ {2} A B _ {\varsigma} S ^ {2} H ^ {3}}{\alpha^ {4}} \sum_ {h = 0} ^ {H} \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}\right) \right] ^ {\frac {1}{2}} + \left(O ^ {3} A ^ {2} B ^ {2} H ^ {3} \varsigma \cdot H T\right) ^ {\frac {1}{2}}. \\ \end{array}
$$
Since the above derivation is for any policy pair  $(\pi^t,\nu^t)$ , we can show that  $\alpha$ -revealing POMG is subsumed by the classes of both self-play and adversarial GEC class with a bounded  $d_{\mathrm{GEC}}$ . We finally prove
$$
d _ {\mathrm {G E C}} = O \left(\frac {O ^ {3} A ^ {2} B ^ {2} H ^ {3} S ^ {2} \varsigma}{\alpha^ {4}}\right),
$$
where  $\varsigma = 2\log \left(1 + \frac{4A^2B^2O^2S^2}{\alpha^2}\right)$ . This completes the proof.
B.4.1 Lemmas for Proof of Proposition 6
Here, we present and prove all the lemmas used for the proof of Proposition 6. All the lemmas presented here follow the notations of Proposition 6.
Lemma 3. In an  $\alpha$ -revealing POMG, for any policy pair  $(\pi, \nu)$ , any vector  $x \in \mathbb{R}^O$ , and any  $\tau_{h-1}$  we have
$$
\sum_ {\tau_ {h: H}} | \boldsymbol {m} (\tau_ {h: H}) x | \sigma \left(\tau_ {h: H}; \tau_ {h - 1}\right) \leq \frac {\sqrt {S}}{\alpha} \| x \| _ {1}.
$$
Proof. We first obtain
$$
\begin{array}{l} \sum_ {\tau_ {h: H}} | \boldsymbol {m} (\tau_ {h: H}) x | \sigma (\tau_ {h: H}; \tau_ {h - 1}) \\ = \sum_ {\tau_ {h: H}} | \boldsymbol {m} \left(\tau_ {h: H}\right) \mathbb {O} _ {h} \mathbb {O} _ {h} ^ {\dagger} x | \sigma \left(\tau_ {h: H}; \tau_ {h - 1}\right) \\ \leq \sum_ {i = 1} ^ {S} \sum_ {\tau_ {h: H}} | \boldsymbol {m} (\tau_ {h: H}) \mathbb {O} _ {h} \boldsymbol {e} _ {i} | \cdot | \boldsymbol {e} _ {i} ^ {\top} \mathbb {O} _ {h} ^ {\dagger} x | \sigma (\tau_ {h: H}; \tau_ {h - 1}), \\ \end{array}
$$
where  $\pmb{e}_i\in \mathbb{R}^S$  denotes the standard basis vector whose  $i$ -th element is 1 and 0 for others. Note that  $\sum_{\tau_{h:H}}|\pmb {m}(\tau_{h:H})\mathbb{O}_h\pmb {e}_i|\sigma (\tau_{h:H};\tau_{h - 1}) = \sum_{\tau_{h:H}}\operatorname *{Pr}^{\sigma}(\tau_{h:H}\mid \tau_{1:h - 1},s_i) = 1$ . Thus we obtain
$$
\begin{array}{l} \sum_ {i = 1} ^ {S} \sum_ {\tau_ {h: H}} | \boldsymbol {m} (\tau_ {h: H}) \mathbb {O} _ {h} \boldsymbol {e} _ {i} | \cdot | \boldsymbol {e} _ {i} ^ {\top} \mathbb {O} _ {h} ^ {\dagger} x | \sigma (\tau_ {h: H}; \tau_ {h - 1}) \\ = \| \mathbb {O} _ {h} ^ {\dagger} x \| _ {1} \leq \| \mathbb {O} _ {h} ^ {\dagger} \| _ {1} \cdot \| x \| _ {1} \leq \sqrt {S} \| \mathbb {O} _ {h} ^ {\dagger} \| _ {2} \cdot \| x \| _ {1} \leq \frac {\sqrt {S}}{\alpha} \| x \| _ {1}, \\ \end{array}
$$
Lemma 4. In an  $\alpha$ -revealing POMG, for any policy pair  $(\pi, \nu)$  and any  $x \in \mathbb{R}^O$ , we have
$$
\sum_ {\left(o _ {h}, a _ {h}, b _ {h}\right) \in \mathcal {O} \times \mathcal {A} \times \mathcal {B}} \| M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) x \| _ {1} \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \leq \frac {\sqrt {S}}{\alpha} \| x \| _ {1}.
$$
$$
\begin{array}{l} \sum_ {\left(o _ {h}, a _ {h}, b _ {h}\right) \in \mathcal {O} \times \mathcal {A} \times \mathcal {B}} \| M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) x \| _ {1} \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ = \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} | \tilde {e} _ {j} ^ {\top} M _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) x | \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ \leq \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} \sum_ {i = 1} ^ {S} | \widetilde {\boldsymbol {e}} _ {j} ^ {\top} \boldsymbol {M} _ {h} (o _ {h}, a _ {h}, b _ {h}) \mathbb {O} _ {h} \boldsymbol {e} _ {i} | \cdot | \boldsymbol {e} _ {i} ^ {\top} \mathbb {O} _ {h} ^ {\dagger} x | \sigma (o _ {h}, a _ {h}, b _ {h}), \\ \end{array}
$$
where  $\widetilde{\pmb{e}}_j$  is the  $j$ -th standard basis vector in  $\mathbb{R}^O$  and  $\pmb{e}_i$  is the  $i$ -th standard basis vector in  $\mathbb{R}^S$ . Note that we have
$$
\begin{array}{l} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} | \tilde {\boldsymbol {e}} _ {j} ^ {\top} \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) \mathbb {O} _ {h} \boldsymbol {e} _ {i} | \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ = \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} \Pr (o _ {j}, o _ {h}, a _ {h}, b _ {h} \mid s _ {i}) \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ = \sum_ {o _ {h}, a _ {h}, b _ {h}} \Pr \left(o _ {h}, a _ {h}, b _ {h} \mid s _ {i}\right) \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ \leq \sum_ {o _ {h}, a _ {h}, b _ {h}} \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) = 1. \\ \end{array}
$$
Combining the above result, we obtain
$$
\begin{array}{l} \sum_ {o _ {h}, a _ {h}, b _ {h}} \sum_ {j = 1} ^ {O} \sum_ {i = 1} ^ {S} | \widetilde {\boldsymbol {e}} _ {j} ^ {\top} \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right) \mathbb {O} _ {h} \boldsymbol {e} _ {i} | \cdot | \boldsymbol {e} _ {i} ^ {\top} \mathbb {O} _ {h} ^ {\dagger} x | \sigma \left(o _ {h}, a _ {h}, b _ {h}\right) \\ \leq \sum_ {i = 1} ^ {S} | e _ {i} ^ {\top} \mathbb {O} _ {h} ^ {\dagger} x | \\ = \| \mathbb {O} _ {h} ^ {\dagger} x \| _ {1} \leq \| \mathbb {O} _ {h} ^ {\dagger} \| _ {1} \cdot \| x \| _ {1} \leq \sqrt {S} \| \mathbb {O} _ {h} ^ {\dagger} \| _ {2} \cdot \| x \| _ {1} \leq \frac {\sqrt {S}}{\alpha} \| x \| _ {1}, \\ \end{array}
$$
Lemma 5. The value difference in a POMG can be decomposed as
$$
V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \leq \frac {H \sqrt {S}}{\alpha} \sum_ {h = 1} ^ {H} \left[ \sum_ {\tau_ {h}} \| (\boldsymbol {M} _ {h} ^ {t} - \boldsymbol {M} _ {h}) \boldsymbol {q} (\tau_ {h - 1}) \| _ {1} \sigma^ {t} (\tau_ {h}) + \| q _ {0} ^ {t} - q _ {0} \| _ {1} \right].
$$
Proof. Denoting  $r(\tau_H)$  as the sum of rewards from  $h = 1$  to  $H$  on  $\tau_H$ , we obtain
$$
\begin{array}{l} V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \\ = \sum_ {\tau_ {H}} \left(\mathbf {P} _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} (\tau_ {H}) - \mathbf {P} _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} (\tau_ {H})\right) r (\tau_ {H}) \\ \leq H \sum_ {\tau_ {H}} \left| \mathbf {P} _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} (\tau_ {H}) - \mathbf {P} _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} (\tau_ {H}) \right| \\ = H \sum_ {\tau_ {H}} \left| \boldsymbol {M} _ {H: 1} ^ {t} \left(o _ {1: H}, a _ {1: H}, b _ {1: H}\right) \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {M} _ {H: 1} \left(o _ {1: H}, a _ {1: H}, b _ {1: H}\right) \boldsymbol {q} _ {0} \right| \sigma^ {t} (\tau_ {H}). \\ \end{array}
$$
Using the triangle inequality, we further obtain
$$
\begin{array}{l} H \sum_ {\tau_ {H}} | \boldsymbol {M} _ {H: 1} ^ {t} \left(o _ {1: H}, a _ {1: H}, b _ {1: H}\right) \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {M} _ {H: 1} \left(o _ {1: H}, a _ {1: H}, b _ {1: H}\right) \boldsymbol {q} _ {0} | \sigma^ {t} (\tau_ {H}) \\ \leq H \sum_ {\tau_ {H}} \left[ \sum_ {h = 1} ^ {H} | \boldsymbol {m} ^ {t} \left(\tau_ {h + 1: H}\right) \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) | \sigma^ {t} \left(\tau_ {H}\right) \right. \\ \left. + \left| \boldsymbol {m} ^ {t} \left(\tau_ {H}\right) \left(\boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0}\right) \right| \sigma^ {t} \left(\tau_ {H}\right) \right], \\ \end{array}
$$
where  $\pmb{m}(\tau_{h+1:H}) = \pmb{M}_{H:h+1}(\tau_{h+1:H})$  and  $\pmb{q}(\tau_{h-1}) = \pmb{M}_{h-1:1}(\tau_{h-1})\pmb{q}_0$ . Note that Lemma 3 shows that
$$
\begin{array}{l} \sum_ {\tau_ {h + 1: H}} \left| \boldsymbol {m} ^ {t} \left(\tau_ {h + 1: H}\right) \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) \right| \sigma^ {t} \left(\tau_ {h + 1: H}; \tau_ {1: h}\right) \\ \leq \frac {\sqrt {S}}{\alpha} \| \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) \| _ {1} \\ \end{array}
$$
$$
\sum_ {\tau_ {H}} | \boldsymbol {m} ^ {t} (\tau_ {H}) (\boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0}) | \sigma^ {t} (\tau_ {H}) \leq \frac {\sqrt {S}}{\alpha} \| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \| _ {1}.
$$
$$
\begin{array}{l} H \sum_ {\tau_ {H}} \left[ \sum_ {h = 1} ^ {H} | \boldsymbol {m} ^ {t} \left(\tau_ {h + 1: H}\right) \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) | \sigma^ {t} \left(\tau_ {H}\right) \right. \\ \left. + \left| \boldsymbol {m} ^ {t} \left(\tau_ {H}\right) \left(\boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0}\right) \right| \sigma^ {t} \left(\tau_ {H}\right) \right] \\ = H \left[ \sum_ {h = 1} ^ {H} \sum_ {\tau_ {H}} | \boldsymbol {m} ^ {t} \left(\tau_ {h + 1: H}\right) \left(\boldsymbol {M} _ {h} ^ {t} \left(o _ {h}, a _ {h}, b _ {h}\right) - \boldsymbol {M} _ {h} \left(o _ {h}, a _ {h}, b _ {h}\right)\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) | \sigma^ {t} \left(\tau_ {H}\right) \right. \\ \left. + \sum_ {\tau_ {H}} \left| \boldsymbol {m} ^ {t} \left(\tau_ {H}\right) \left(\boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0}\right) \right| \sigma^ {t} \left(\tau_ {H}\right) \right] \\ \leq \frac {H \sqrt {S}}{\alpha} \sum_ {h = 1} ^ {H} \left[ \sum_ {\tau_ {h}} \| \left(\boldsymbol {M} _ {h} ^ {t} - \boldsymbol {M} _ {h}\right) \boldsymbol {q} \left(\tau_ {h - 1}\right) \| _ {1} \sigma \left(\tau_ {h}\right) + \| \boldsymbol {q} _ {0} ^ {t} - \boldsymbol {q} _ {0} \| _ {1} \right], \\ \end{array}
$$
The next lemma gives the bound of training error by Hellinger distances.
Lemma 6. For any model  $f^t$  and policy  $\sigma^t$ , we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} _ {\tau_ {h - 1} \sim \mathbf {P} _ {f ^ {*}, h - 1} ^ {\pi^ {\iota}, \nu^ {\iota}}} \mathbb {E} _ {\substack {\pi^ {\iota}, \nu^ {\iota} \\ f ^ {*}, h - 1}} \left[ \sum_ {o _ {h}, a _ {h}, b _ {h}} \| \left(M _ {h} ^ {t} (o _ {h}, a _ {h}, b _ {h}) - M _ {h} (o _ {h}, a _ {h}, b _ {h})\right) \bar {\boldsymbol {q}} (\tau_ {h - 1}) \| _ {1} \sigma^ {\iota} (o _ {h}, a _ {h}, b _ {h}; \tau_ {h - 1}) \right] ^ {2} \\ \lesssim \frac {S}{\alpha^ {2}} \sum_ {\iota = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} D _ {\mathrm {H e}} ^ {2} \left(\mathbf {P} _ {f ^ {t}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}, \mathbf {P} _ {f ^ {*}, h} ^ {\pi^ {\iota}, \nu^ {\iota}}\right). \\ \end{array}
$$
where  $\lesssim$  omits absolute constants.
Proof. By using  $(a + b)^2 \leq 2a^2 + 2b^2$  and the triangle inequality, we decompose the LHS into two parts to bound them separately. We have