SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathrm{Term}(\mathrm{i}) \leq \sum_{t=1}^{T}\left[-\Delta V_{\underline{f}^t}^* + V_{\overline{f}^t}^{\pi^t, \underline{\nu}^t} - V_{f^*}^{\pi^t, \underline{\nu}^t}\right]$ ,  $\mathrm{Term}(\mathrm{ii}) = \sum_{t=1}^{T}\left[V_{f^*}^{\pi^t, \underline{\nu}^t} - V_{\underline{f}^t}^{\pi^t, \underline{\nu}^t} + \Delta V_{\underline{f}^t}^{\pi^t, *}\right]$ ,
where  $\Delta V_{\overline{f}^t}^* := V_{\overline{f}^t}^{\pi^t, \overline{\nu}^t} - V_{f^*}^*$  and  $\Delta V_{\underline{f}^t}^{\pi^t, *} = V_{\underline{f}^t}^{\pi^t, \underline{\nu}^t} - V_{f^*}^{\pi^t, *}$  are associated with the optimism terms in posterior distributions. The inequality above for Term(i) is due to Line 4 such that  $V_{\overline{f}^t}^{\pi^t, \overline{\nu}^t} = \min_\nu V_{\overline{f}^t}^{\pi^t, \nu} \leq V_{\overline{f}^t}^{\pi^t, \underline{\nu}^t}$ . By Definition 1 for self-play GEC, we obtain that  $\sum_{t=1}^{T} \left( V_{\overline{f}^t, 1}^{\pi^t, \underline{\nu}^t} - V_{f^*}^{\pi^t, \underline{\nu}^t} \right)$  and  $\sum_{t=1}^{T} \left( V_{f^*}^{\pi^t, \underline{\nu}^t} - V_{\underline{f}^t, 1}^{\pi^t, \underline{\nu}^t} \right)$  can be bounded by
$$
\left[ d _ {\mathrm {G E C}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\sigma_ {\exp} ^ {\iota}, h)} \ell (\rho^ {t}, \xi_ {h} ^ {\iota})\right) \right] ^ {1 / 2} + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T,
$$
where  $\rho^t$  is chosen as  $\overline{f}^t$  or  $f^t$  respectively. By Lemma 11 and Lemma 12, we prove that for both FOMGs and POMGs, the accumulation of the losses  $\ell(\overline{f}^t, \xi_h^t)$  in (3) connects to the likelihood function  $L_h^t$  defined in (1) and (2). Thus, we obtain  $\mathbb{E}[\mathrm{Term}(\mathrm{i})] \leq \sum_{t=1}^{T} \mathbb{E}_{Z^{t-1}} \mathbb{E}_{\overline{f}^t \sim p^t} \{-\gamma_1 \Delta V_{\overline{f}^t}^* - \sum_{h=1}^{H} \sum_{\iota=1}^{t-1} [L_h^t (\overline{f}^t) - L_h^t(f^*)] + \log \frac{p^t(\overline{f}^t)}{p^0(\overline{f}^t)}\} + 2H(d_{\mathrm{GEC}} HT)^{\frac{1}{2}} + \epsilon HT$  and  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$  has a similar bound based on  $q^t$ , where  $Z^{t-1}$  is the randomness history. By Lemma 10, the posterior distributions  $p^t$  and  $q^t$  following Lines 3 and 5 of Algorithm 1 can minimize the above upper bounds for  $\mathbb{E}[\mathrm{Term}(\mathrm{i})]$  and  $\mathbb{E}[\mathrm{Term}(\mathrm{ii})]$ . Therefore, we can relax  $p^t$  and  $q^t$  to be distributions defined around the true model  $f^*$  to enlarge above bounds. When  $T$  is sufficiently large and  $\eta = 1/2$ , we have
$$
\mathbb {E} \left[ \text {T e r m (i)} \right] \leq \omega \left(H T, p ^ {0}\right) T / \gamma_ {1} + \gamma_ {1} d _ {\text {G E C}} H / 4 + 2 H \left(d _ {\text {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T,
$$
$$
\mathbb {E} [ \text {T e r m (i i)} ] \leq \omega \left(H T, q ^ {0}\right) T / \gamma_ {2} + \gamma_ {2} d _ {\text {G E C}} H / 4 + 2 H \left(d _ {\text {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T.
$$
Choosing proper values for  $\epsilon, \gamma_1$ , and  $\gamma_2$ , we obtain the bound for  $\mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T)]$  in Theorem 1 via  $\mathrm{Reg}_1^{\mathrm{sp}}(T) = \mathrm{Term}(\mathrm{i}) + \mathrm{Term}(\mathrm{ii})$ . In addition, we can prove the bound of  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$  in a symmetric manner. Finally, combining  $\mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T)]$  and  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$  gives the result in Theorem 1.
Proof Sketch of Theorem 2. Under the adversarial setting, the policy of the opponent  $\nu^t$  is not generated by the algorithm, which could be arbitrarily time-varying. We decompose  $\mathrm{Reg}^{\mathrm{adv}}(T) = \sum_{t=1}^{T} \Delta V_{f^t}^* + \sum_{t=1}^{T} [V_{f^t}^* - V_{f^*}^{\pi^t, \nu^t}]$  where  $\Delta V_{f^t}^* := V_{f^*}^* - V_{f^t}^*$  relates to optimism. Since  $(\pi^t, \overline{\nu}^t)$  is NE of  $V_{f^t}^{\pi, \nu}$  as in Line 3 of Algorithm 2, we have  $V_{f^t}^* = \min_\nu V_{f^t}^{\pi^t, \nu} \leq V_{f^t}^{\pi^t, \nu^t}$ , which leads to
$$
\operatorname {R e g} ^ {\operatorname {a d v}} (T) \leq \sum_ {t = 1} ^ {T} \Delta V _ {f ^ {t}} ^ {*} + \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right].
$$
We can bound  $\sum_{t=1}^{T}[V_{f^t}^{\pi^t,\nu^t} - V_{f^*}^{\pi^t,\nu^t}]$  via adversarial GEC in Definition 3 by
$$
\left[ d _ {\text {G E C}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\left(\sigma_ {\exp} ^ {\iota}, h\right)} \ell \left(f ^ {t}, \xi_ {h} ^ {\iota}\right)\right) \right] ^ {\frac {1}{2}} + 2 H \left(d _ {\text {G E C}} H T\right) ^ {\frac {1}{2}} + \epsilon H T.
$$
Connecting the loss  $\ell(\overline{f}^t, \xi_h^t)$  to the likelihood function  $L_h^t$  defined in (1) and (2) via Lemmas 11 and 12, we obtain  $\mathbb{E}[\mathrm{Reg}^{\mathrm{adv}}(T)] \leq \sum_{t=1}^{T} \mathbb{E}_{Z^{t-1}} \mathbb{E}_{f^t \sim p^t} \left\{ \gamma \sum_{t=1}^{T} \Delta V_{f^t}^* - \sum_{h=1}^{H} \sum_{\iota=1}^{t-1} [L_h^t(f^t) - L_h^t(f^*)] + \log \frac{p^t(f^t)}{p^0(f^t)} \right\} + 2H(d_{\mathrm{GEC}} HT)^{\frac{1}{2}} + \epsilon HT$ . Lemma 10 shows  $p^t$  in Line 3 of Algorithm 2 can minimize this bound. Thus, relaxing  $p^t$  to be distribution defined around the true model  $f^*$ , with sufficiently large  $T$  and  $\eta = 1/2$ , we have
$$
\mathbb {E} [ \mathrm {R e g} ^ {\mathrm {a d v}} (T) ] \leq \omega (4 H T, p ^ {0}) T / \gamma + \gamma d _ {\mathrm {G E C}} H / 4 + 2 H (d _ {\mathrm {G E C}} H T) ^ {\frac {1}{2}} + \epsilon H T.
$$
Choosing proper values for  $\epsilon$  and  $\gamma$ , we eventually obtain the bound for  $\mathbb{E}[\mathrm{Reg}^{\mathrm{adv}}(T)]$  in Theorem 2.
Discussion of Limitations. Our work has studied several but a limited number of tractable MG classes in both FOMGs and POMGs. It is interesting to further define new MG classes by generalizing their single-agent counterparts and explore the relation between these MG classes and low self-play/adversarial GEC classes. It is also intriguing to generalize our method to general-sum settings.
The authors would like to thank the anonymous reviewers for their valuable comments. The authors would like to thank Wei Xiong for the helpful discussions. Shuang Qiu and Tong Zhang acknowledge the funding supported by the GRF 16310222.
[75] Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael I Jordan. On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622, 2020.
[76] Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954–1964. PMLR, 2020.
[77] Wenhao Zhan, Jason D Lee, and Zhuoran Yang. Decentralized optimistic hyperpolicy mirror descent: Provably no-regret learning in markov games. arXiv preprint arXiv:2206.01588, 2022.
[78] Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. Pac reinforcement learning for predictive state representations. arXiv preprint arXiv:2207.05738, 2022.
[79] Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pages 26447-26466. PMLR, 2022.
[80] Tong Zhang. From  $\varepsilon$ -entropy to kl-entropy: Analysis of minimum information complexity density estimation. 2006.
[81] Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
[82] Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. arXiv preprint arXiv:2202.07511, 2022.
[83] Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.01962, 2022.
[84] Han Zhong, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Can reinforcement learning find stackelberg-nash equilibria in general-sum markov games with myopically rational followers? Journal of Machine Learning Research, 24(35):1-52, 2023.
[85] Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
A Omitted Algorithm in the Main Text 17
B Computation of Self-Play and Adversarial GEC 17
B.1 Linear MG 17
B.2 Linear Mixture MG 20
B.3 Low-Witness-Rank MG 21
B.4 Weakly Revealing POMG 24
B.5 Decodable POMG 32
C Computation of  $\omega (\beta ,p^0)$  34
C.1 Fully Observable Markov Game 34
C.2 Partially Observable Markov Game 35
D Technical Lemmas for Main Theorems 37
D.1 Lemmas 37
D.2 Proof of Lemma 11 38
D.3 Proof of Lemma 12 40
D.4 Proof of Lemma 13 41
D.5 Proof of Lemma 14 42
E Proofs for Algorithms 1 and 3 43
E.1 Proof of Proposition 1 43
E.2 Proof of Proposition 2 47
E.3 Proof of Theorem 1 49
F Proofs for Algorithm 2 50
F.1 Proof of Theorem 2 50
G Other Supporting Lemmas 52
A Omitted Algorithm in the Main Text
Algorithm 3 Model-Based Posterior Sampling for Self-Play (Min-Player)
1: Input: Model class  $\mathcal{F}$ , prior distributions  $p^0$  and  $q^0$ ,  $\gamma_1$ , and  $\gamma_2$ .  
2: for  $t = 1, \dots, T$  do  
3: Draw a model  $\overline{f}^t \sim p^t(\cdot)$  with defining  $p^t(f) \propto p^0(f) \exp[-\gamma_1 V_f^* + \sum_{\iota=1}^{t-1} \sum_{h=1}^{H} L_h^\tau(f)]$   
4: Compute  $\nu^t$  by letting  $(\overline{\pi}^t, \nu^t)$  be NE of  $V_{\overline{f}^t}^{\pi,\nu}$   
5: Draw a model  $\underline{f}^t \sim q^t(\cdot)$  with defining  $q^t(f) \propto q^0(f) \exp[\gamma_2 V_f^{*,\nu^t} + \sum_{\iota=1}^{t-1} \sum_{h=1}^{H} L_h^\tau(f)]$   
6: Compute  $\underline{\pi}^t$  by letting  $\underline{\pi}^t$  be the best response of  $\nu^t$  w.r.t.  $V_{\underline{f}^t}^{\pi,\nu}$   
7: Collect data  $\mathcal{D}^t$  by executing the joint exploration policy  $\sigma^t$   
8: Define the likelihood functions  $\{L_h^t(f)\}_{h=1}^H$  using the collected data  $\mathcal{D}^t$   
9: end for  
10: Return:  $(\nu^1, \dots, \nu^T)$ .
B Computation of Self-Play and Adversarial GEC
In this section, we present the computation of the self-play and adversarial GEC for different FOMG and POMG classes, including linear MGs, linear mixture MGs, low-witness-rank MGs, weakly revealing POMGs, and a novel POMG class, dubbed decodable POMGs, newly proposed by this work.
Note that the GEC  $d_{\mathrm{GEC}}$  essentially depends on  $\epsilon$  as shown in Definition 1 and Definition 3 such that it should be expressed as  $d_{\mathrm{GEC}}^{\epsilon}$ . For convenience, we omit such a dependence in the notation. In our main theorems, we set  $\epsilon = 1 / \sqrt{HT}$ , and according to our derivation in this section,  $d_{\mathrm{GEC}}$  has an  $O(\log (1 / \epsilon)) = O(\cdot \log (HT))$  factor which only scales logarithmically in  $T$ .
B.1 Linear MG
In this subsection, we show that the linear MG is in the class of MGs with low self-play and adversarial GEC and further calculate  $d_{\mathrm{GEC}}$  for the linear MG.
Definition 4 (Linear Markov Game [71]). The linear  $MG$  is a FOMG admitting the following linear structures on the reward function and transition kernel,
$$
r _ {h} (s, a, b) = \mathbf {w} _ {h} ^ {\top} \phi (s, a, b), \quad \mathbb {P} _ {h} \left(s ^ {\prime} \mid s, a, b\right) = \boldsymbol {\theta} _ {h} \left(s ^ {\prime}\right) ^ {\top} \phi (s, a, b),
$$
where there exist a known feature map  $\phi : \mathcal{S} \times \mathcal{A} \times \mathcal{B} \mapsto \mathbb{R}^d$  and unknown coefficients  $\mathbf{w}_h \in \mathbb{R}^d$  and  $\pmb{\theta}_h(s') \in \mathbb{R}^d$  that should be learned in algorithms. We assume that the feature map and the coefficients satisfy  $\| \phi(s, a, b) \|_2 \leq 1$ ,  $\int_{\mathcal{S}} \| \pmb{\theta}_h(s) \|_2 \mathrm{d}s \leq \sqrt{d}$ , and  $\| \mathbf{w}_h \|_2 \leq \sqrt{d}$ .
Then,  $d_{\mathrm{GEC}}$  for the linear MG can be calculated in the following proposition. We can show that both low self-play and adversarial GEC classes subsume the class of linear MGs.
Proposition 3 (Linear MG  $\subset$  Low Self-Play/Adversarial GEC). For linear MGs with  $\phi(s, a, b) \in \mathbb{R}^d$  as defined in Definition 4, and for  $\epsilon > 0$ , when  $T$  is sufficiently large, choosing  $\ell(f, \xi_h) = D_{\mathrm{He}}^2\big(\mathbb{P}_{f,h}(\cdot | \xi_h), \mathbb{P}_{f^*,h}(\cdot | \xi_h)\big)$  as in (3) with  $\xi_h = (s_h, a_h, b_h)$ , we have
linear  $MG\subset MG$  with low self-play and adversarial GEC
with  $d_{\mathrm{GEC}}$  satisfying
$$
d _ {\mathrm {G E C}} = 1 6 H ^ {3} d \log \left(1 + \frac {H T}{\epsilon}\right).
$$
Proof. Since the value function is defined as  $V_{f,h}^{\pi,\nu}(s_h) \coloneqq \mathbb{E}_{\pi,\nu,\mathbb{P}_f}[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'},b_{h'})|s_h]$  at step  $h$  with denoting  $V_f^{\pi,\nu} = V_{f,1}^{\pi,\nu}(s_1)$  and we also have a relation  $V_f^{\pi,\nu} =$
$\mathbb{E}_{\pi, \nu, \mathbb{P}_{f'}}[\sum_{h=1}^{H}(V_{f,h}^{\pi, \nu}(s_h) - V_{f,h+1}^{\pi, \nu}(s_{h+1}))]$  for any  $f'$  using the telescoping summation as well as  $V_{f,H+1}^{\pi, \nu}(\cdot) = 0$ , we thus can decompose the value difference as follows,
$$
\begin{array}{l} V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \\ = \mathbb {E} _ {\pi^ {t}, \nu^ {t}, \mathbb {P} _ {f ^ {*}}} \left[ \sum_ {h = 1} ^ {H} \left(V _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}} \left(s _ {h}\right) - V _ {f ^ {t}, h + 1} ^ {\pi^ {t}, \nu^ {t}} \left(s _ {h + 1}\right)\right) \right] - \mathbb {E} _ {\pi^ {t}, \nu^ {t}, \mathbb {P} _ {f ^ {*}}} \left[ \sum_ {h = 1} ^ {H} r _ {h} \left(s _ {h}, a _ {h}, b _ {h}\right) \right] \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {t}, \nu^ {t}, \mathbb {P} _ {f ^ {*}}} \left[ V _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}} (s _ {h}) - \left(r _ {h} (s _ {h}, a _ {h}, b _ {h}) + V _ {f ^ {t}, h + 1} ^ {\pi^ {t}, \nu^ {t}} (s _ {h + 1})\right) \right] \\ = \sum_ {h = 1} ^ {H} \mathcal {E} _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}}, \tag {4} \\ \end{array}
$$
where we define the following Bellman residual in expectation as
$$
\mathcal {E} _ {f, h} ^ {\pi , \nu} := \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \left[ Q _ {f, h} ^ {\pi , \nu} (s _ {h}, a _ {h}, b _ {h}) - \mathbb {E} _ {s _ {h + 1} \sim \mathbb {P} _ {f ^ {*} , h} (\cdot | s _ {h}, a _ {h}, b _ {h})} \left(r _ {h} (s _ {h}, a _ {h}, b _ {h}) + V _ {f, h + 1} ^ {\pi , \nu} (s _ {h + 1})\right) \right].
$$
The Bellman residual for the linear MG can be linearly represented, whose form is
$$
\begin{array}{l} \mathcal {E} _ {f, h} ^ {\pi , \nu} = \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} \left[ \mathbb {P} _ {f, h} (s | s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f ^ {*}, h} (s | s _ {h}, a _ {h}, b _ {h}) \right] V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s \\ = \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \int_ {\mathcal {S}} [ \phi (s _ {h}, a _ {h}, b _ {h}) ^ {\top} (\boldsymbol {\theta} _ {f, h} (s) - \boldsymbol {\theta} _ {f ^ {*}, h} (s)) ] V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s \\ = \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \phi (s _ {h}, a _ {h}, b _ {h}) ^ {\top} \int_ {\mathcal {S}} [ (\boldsymbol {\theta} _ {f, h} (s) - \boldsymbol {\theta} _ {f ^ {*}, h} (s)) ] V _ {f, h + 1} ^ {\pi , \nu} (s) \mathrm {d} s \\ = \left\langle \mathbb {E} _ {\pi , \nu , \mathbb {P} _ {f ^ {*}}} \phi (s _ {h}, a _ {h}, b _ {h}), \boldsymbol {q} _ {h} (f, \pi , \nu) \right\rangle , \\ \end{array}
$$
where we let
$$
\boldsymbol {q} _ {h} (f, \pi , \nu) := \int_ {\mathcal {S}} [ (\boldsymbol {\theta} _ {f, h} (s) - \boldsymbol {\theta} _ {f ^ {*}, h} (s)) ] V _ {f, h + 1} ^ {\pi , \nu} (s) d s,
$$
such that we have  $\| q_h(f,\pi ,\nu)\| _2\leq 2\sqrt{d} H$  according to the definition of the linear MG. Therefore, the value difference term can also be rewritten as
$$
V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} = \sum_ {h = 1} ^ {H} \mathcal {E} _ {f ^ {t}, h} ^ {\pi^ {t}, \nu^ {t}} = \sum_ {h = 1} ^ {H} [ \langle \mathbb {E} _ {\pi^ {t}, \nu^ {t}, \mathbb {P} _ {f ^ {*}}} \phi (s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}), q _ {h} (f ^ {t}, \pi^ {t}, \nu^ {t}) \rangle ],
$$
where  $(s_h^t, a_h^t, b_h^t)$  can be viewed as sampled data following  $\pi^t, \nu^t$ , and  $\mathbb{P}_{f^*}$ .
For simplicity, we denote  $\phi_h^t \coloneqq \mathbb{E}_{\pi^t,\nu^t,\mathbb{P}_{f^*}}\phi(s_h^t,a_h^t,b_h^t)$ ,  $\pmb{q}_h^t \coloneqq \pmb{q}_h(f^t,\pi^t,\nu^t)$ , and  $\xi_h^t \coloneqq (s_h^t,a_h^t,b_h^t)$ , and define  $\Phi_h^t \coloneqq \lambda I_d + \sum_{\iota = 1}^{t}\phi_h^\iota (\phi_h^\iota)^\top$  with  $\lambda >0$  and  $\varphi_h^t \coloneqq \| \phi_h^t\|_{(\Phi_h^{t - 1})^{-1}}$ . Note that we have
$$
\left\| \boldsymbol {q} _ {h} ^ {t} \right\| _ {\Phi_ {h} ^ {t - 1}} = \sqrt {\sum_ {\iota = 1} ^ {t - 1} \left\langle \phi_ {h} ^ {\iota} , \boldsymbol {q} _ {h} ^ {t} \right\rangle^ {2} + \lambda \left\| \boldsymbol {q} _ {h} ^ {t} \right\| _ {2} ^ {2}}, \tag {5}
$$
where the term  $\langle \phi_h^t, \mathbf{q}_h^t \rangle^2$  can be bounded as
$$
\begin{array}{l} [ \langle \boldsymbol {\phi} _ {h} ^ {\iota}, \boldsymbol {q} _ {h} ^ {t} \rangle ] ^ {2} = \left[ \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, f ^ {*}} \int_ {\mathcal {S}} \left(\mathbb {P} _ {f ^ {t}, h} (s \mid s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f ^ {*}, h} (s \mid s _ {h}, a _ {h}, b _ {h})\right) V _ {f ^ {t}, h + 1} ^ {\pi^ {t}, \nu^ {t}} (s) \mathrm {d} s \right] ^ {2} \\ \leq H ^ {2} \mathbb {E} _ {\pi^ {\iota}, \nu^ {\iota}, f ^ {*}} \| \mathbb {P} _ {f, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) - \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}) \| _ {1} ^ {2} \\ \leq 8 H ^ {2} \mathbb {E} _ {\pi^ {t}, \nu^ {t}, f ^ {*}} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t}, h} (\cdot | s _ {h}, a _ {h}, b _ {h}), \mathbb {P} _ {f ^ {*}, h} (\cdot | s _ {h}, a _ {h}, b _ {h})\right), \tag {6} \\ \end{array}
$$
where the first inequality uses  $V_{f,h+1}^{\pi,\nu}(s) \leq H$  and the second inequality uses  $\| P - Q \|_1^2 \leq 8D_{\mathrm{He}}^2(P, Q)$ . Moreover, using  $\boldsymbol{a}^\top \boldsymbol{b} \leq \| \boldsymbol{a} \|_A \| \boldsymbol{b} \|_{A^{-1}}$ , we also have
$$
\begin{array}{l} \langle \phi_ {h} ^ {t}, \boldsymbol {q} _ {h} ^ {t} \rangle = \langle \phi_ {h} ^ {t}, \boldsymbol {q} _ {h} ^ {t} \rangle (\mathbf {1} _ {\{\varphi_ {h} ^ {t} <   1 \}} + \mathbf {1} _ {\{\varphi_ {h} ^ {t} \geq 1 \}}) \\ \leq H \min  \left\{\frac {1}{H} | \langle \phi_ {h} ^ {t}, \boldsymbol {q} _ {h} ^ {t} \rangle |, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}} + H \cdot \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}} \\ \leq \left\| \boldsymbol {q} _ {h} ^ {t} \right\| _ {\Phi_ {h} ^ {t - 1}} \min  \left\{\varphi_ {h} ^ {t}, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}} + H \cdot \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}}, \tag {7} \\ \end{array}
$$
where  $\mathbf{1}_{\{\cdot \}}$  denotes an indicator function.
Plugging (5) into (7) and taking a summation from  $t = 1$  to  $T$ , we obtain
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \| \boldsymbol {q} _ {h} ^ {t} \| _ {\Phi_ {h} ^ {t - 1}} \min  \left\{\varphi_ {h} ^ {t}, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}} + H \cdot \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}} \\ \leq \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \sqrt {\sum_ {\iota = 1} ^ {t - 1} [ \langle \phi_ {h} ^ {\iota} , \boldsymbol {q} _ {h} ^ {t} \rangle ] ^ {2} + 4 \lambda d H ^ {2}} \cdot \min  \{\varphi_ {h} ^ {t}, 1 \} \mathbf {1} _ {\{\varphi_ {h} ^ {t} <   1 \}} + H \cdot \mathbf {1} _ {\{\varphi_ {h} ^ {t} \geq 1 \}} \\ \leq \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sqrt {\sum_ {\iota = 1} ^ {t - 1} [ \langle \phi_ {h} ^ {\iota} , \boldsymbol {q} _ {h} ^ {t} \rangle ] ^ {2}} + \sqrt {4 \lambda d H ^ {2}}\right) \cdot \min  \left\{\varphi_ {h} ^ {t}, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}} + H \cdot \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}} \\ \leq \sqrt {\sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} [ \langle \phi_ {h} ^ {\iota} , \boldsymbol {q} _ {h} ^ {t} \rangle ] ^ {2}} \sqrt {\sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \min  \left\{\left(\varphi_ {h} ^ {t}\right) ^ {2}, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}}} \\ + \sqrt {4 \lambda d H ^ {3} T \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \min  \left\{\left(\varphi_ {h} ^ {t}\right) ^ {2}, 1 \right\} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} <   1 \right\}}} + H \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}}, \tag {8} \\ \end{array}
$$
where the second inequality uses  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  and the last inequality is by Cauchy-Schwarz. According to the elliptical potential lemma (Lemma 15), we have the following bound,
$$
\sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \min  \left\{\left(\varphi_ {h} ^ {t}\right) ^ {2}, 1 \right\} \leq \sum_ {h = 1} ^ {H} 2 \log \left(\frac {\det  \Phi_ {h} ^ {T}}{\det  \Phi_ {h} ^ {0}}\right) \leq 2 H d \log \left(1 + \frac {T}{d \lambda}\right), \tag {9}
$$
where the first inequality uses Lemma 15, and the last inequality uses  $\log \det \Phi_h^t\leq d\log \frac{\mathrm{tr}(\Phi_h^T)}{d}\leq d\log \frac{\lambda d + T}{d}$ . We also note that  $\phi_h^t$  cannot exceed 1 too many times, as detailed in Lemma 16. By Lemma 16 and the fact that  $\mathbf{1}_{\{\varphi_h^t\geq 1\}}\leq 1$ , we obtain
$$
\sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \mathbf {1} _ {\left\{\varphi_ {h} ^ {t} \geq 1 \right\}} \leq \min  \left\{\frac {3 H d}{\log 2} \log \left(1 + \frac {1}{\lambda \log 2}\right), H T \right\}. \tag {10}
$$
Combining (6), (7), (8), (9), and (10), we obtain
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \\ \leq \sqrt {2 H d \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \sum_ {\iota = 1} ^ {t - 1} [ \langle \phi_ {h} ^ {\iota} , \omega_ {h} ^ {t} \rangle ] ^ {2}} \\ + \sqrt {4 \lambda d H ^ {3} T \min \left\{2 H d \log \left(1 + \frac {T}{d \lambda}\right) , H T \right\}} + \min  \left\{\frac {3 H ^ {2} d}{\log 2} \log \left(1 + \frac {1}{\lambda \log 2}\right), H ^ {2} T \right\} \\ \leq H \sqrt {1 6 H d \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {\left(\pi^ {\iota}, \nu^ {\iota}, h\right)} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t} , h} (\cdot | \xi_ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h} ^ {\iota})\right)} \\ + \lambda d H ^ {2} T + \min  \left\{2 H ^ {2} d \log \left(1 + \frac {T}{d \lambda}\right), H ^ {2} T \right\} + \min  \left\{\frac {3 H ^ {2} d}{\log 2} \log \left(1 + \frac {1}{\lambda \log 2}\right), H ^ {2} T \right\} \\ \leq \sqrt {1 6 H ^ {3} d \log \left(1 + \frac {T}{d \lambda}\right) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {\iota = 1} ^ {t - 1} \mathbb {E} _ {(\pi^ {\iota}, \nu^ {\iota}, h)} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f ^ {t} , h} (\cdot | \xi_ {h} ^ {\iota}), \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h} ^ {\iota})\right)} \\ + \lambda d H ^ {2} T + 2 \left[ 2 H ^ {2} d \log \left(1 + \frac {T}{d \lambda}\right) \wedge H ^ {2} T \right], \tag {11} \\ \end{array}
$$