SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Because  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  and for any  $k \in [K]$ ,  $h \in [H]$ ,  $\sqrt{\max_{(s,a) \in S \times A} \left\| P_h^{\star,k+1}(\cdot|s,a) - P_h^{\star,k}(\cdot|s,a) \right\|_{TV}} =$
$\max_{(s,a)\in \mathcal{S}\times \mathcal{A}}\sqrt{\left\|\left|P_h^{\star,k + 1}(\cdot|s,a) - P_h^{\star,k}(\cdot|s,a)\right|\right\|_{TV}}$  , then for any  $\mathcal{H},\mathcal{I}$  , we can convert  $\ell_{\infty}$  variation budgets to square-root  $\ell_{\infty}$  variation budgets.
$$
\sqrt {\Delta_ {\mathcal {H} , \mathcal {I}} ^ {P}} \leq \Delta_ {\mathcal {H}, \mathcal {I}} ^ {\sqrt {P}}. \tag {36}
$$
Recall that  $W_{h,\phi}^{k} = \sum_{i=1\vee(k-W)}^{k-1}\mathbb{E}_{(s_{h},a_{h})\sim(P^{\star,i},\pi^{i})}\left[\phi_{h}(s_{h},a_{h})\phi_{h}(s_{h},a_{h})^{\top}\right] + \lambda_{k,W}I_{d}$ . We derive the following bound:
$$
\underset {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \pi)} {\mathbb {E}} \left[ \hat {b} _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \Bigg | s _ {h - 1}, a _ {h - 1} \right]
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \underset {a _ {h - 1} \sim \pi} {\mathbb {E}} \left[ \left\| \phi_ {h - 1} ^ {\star , k} \left(s _ {h - 1}, a _ {h - 1}\right) \right\| _ {\left(W _ {h - 1, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \right. \\ \times \sqrt{\sum_{i = 1\vee(k - W)}^{k - 1}\mathbb{E}_{\substack{s_{h - 1},a_{h - 1}\sim(P^{\star,i},\pi^{i})\\ s_{h}\sim P^{(\star,i)}_{h - 1}(\cdot |s_{h - 1},a_{h - 1})\\ a_{h}\sim\mathcal{U}(A)}}\left[\hat{b}_{h}^{k}(s_{h},a_{h})^{2}\right] + A\Delta_{h - 1,[k - W,k - 1]}^{P} + \lambda_{k,W}d} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} \underset {a _ {h - 1} \sim \pi} {\mathbb {E}} \left[ \left\| \phi_ {h - 1} ^ {\star , k} (s _ {h - 1}, a _ {h - 1}) \right\| _ {\left(W _ {h - 1, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \times \sqrt {9 d A \alpha_ {k , W} ^ {2} + \lambda_ {k , W} d} \right] + \sqrt {\frac {A}{d} \Delta_ {h - 1 , [ k - W , k - 1 ]} ^ {P}} \\ \stackrel {(i i i)} {\leq} \mathbb {E} _ {a _ {h - 1} \sim \pi} \left[ \left\| \phi_ {h - 1} ^ {\star , k} (s _ {h - 1}, a _ {h - 1}) \right\| _ {\left(W _ {h - 1, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \times \sqrt {9 d A \alpha_ {k , W} ^ {2} + \lambda_ {k , W} d} \right] + \sqrt {\frac {A}{d}} \Delta_ {h - 1, [ k - W, k - 1 ]} ^ {\sqrt {P}} \\ \end{array}
$$
where  $(i)$  follows from Lemma A.12,  $(ii)$  follows from Equation (35), and  $(iii)$  follows Equation (36).
Before next lemma, we first introduce a notion related to matrix norm. For any matrix  $A$ ,  $\| A \|_2$  denotes the matrix norms induced by vector  $\ell_2$ -norm. Note that  $\| A \|_2$  is also known as the spectral norm of matrix  $A$  and is equal to the largest singular value of matrix  $A$ .
Lemma A.18. With probability at least  $1 - \delta$ , the summation of the truncated value functions  $\hat{V}_{\hat{P}^k,\hat{b}^k}^{\pi_k}$  under exploration policies  $\{\tilde{\pi}_k\}_{k\in [K]}$  is bounded by:
$$
\sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\tilde {\pi} ^ {k}} \leq O \left(\sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] \right.
$$
$$
\left. + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right).
$$
Proof. For any  $k \in [K]$ , any policy  $\pi$ , we have
$$
\begin{array}{l} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} - V _ {P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi} \leq \underset {\pi} {\mathbb {E}} \left[ \hat {P} _ {1} ^ {k} \hat {V} _ {2, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {1}, a _ {1}) - P _ {1} ^ {\star , k} V _ {2, P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi} (s _ {1}, a _ {1}) \right] \\ = \mathbb {E} _ {\pi} \left[ \left(\hat {P} _ {1} ^ {k} - P _ {1} ^ {\star , k}\right) \hat {V} _ {2, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {1}, a _ {1}) + P _ {1} ^ {\star , k} \left(\hat {V} _ {2, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} - V _ {2, P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi}\right) (s _ {1}, a _ {1}) \right] \\ \leq \underset {\pi} {\mathbb {E}} \left[ f _ {1} ^ {k} \left(s _ {1}, a _ {1}\right) + P _ {1} ^ {\star , k} \left(\hat {V} _ {2, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} - V _ {2, P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi}\right) \right] \\ \leq \underset {\pi} {\mathbb {E}} \left[ f _ {1} ^ {k} (s _ {1}, a _ {1}) \right] + \underset {P ^ {\star , k}, \pi} {\mathbb {E}} \left[ \hat {V} _ {2, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} - V _ {2, P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi} \right] \\ \leq \underset {P ^ {\star , k}, \pi} {\mathbb {E}} \left[ \sum_ {h = 1} ^ {H} f ^ {k} \left(s _ {h}, a _ {h}\right) \right] = V _ {P ^ {\star , k}, f ^ {k}} ^ {\pi}, \tag {37} \\ \end{array}
$$
$$
\sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} \leq \sum_ {k = 1} ^ {K} V _ {P ^ {\star , k}, f ^ {k}} ^ {\pi} + \sum_ {k = 1} ^ {K} V _ {P ^ {\star , k}, \hat {b} ^ {k}} ^ {\pi}.
$$
Step 1: We first bound  $\sum_{k=1}^{K} V_{P^{\star,k},f^k}^{\tilde{\pi}^k}$  via an auxiliary anchor representation.
Recall that  $U_{h,\phi^{\star},k}^{k,W} = \sum_{i=1\vee(k-W)}^{k-1}\mathbb{E}_{\substack{s_h\sim(P^{\star,i},\tilde{\pi}^i)\\ a_h\sim\mathcal{U}(\mathcal{A})}}\left[\phi_h^{\star,k}(s_h,a_h)\phi_h^{\star,k}(s_h,a_h)^\top\right] + \lambda_{k,W}I_d$  and we define  $\tilde{U}_{h,\phi^{\star},k}^{k,W,t} = \sum_{i=tW+1}^{k-1}\mathbb{E}_{\substack{s_h\sim(P^{\star,i},\tilde{\pi}^i)\\ a_h\sim\mathcal{U}(\mathcal{A})}}\left[\phi_h^{\star,k}(s_h,a_h)\phi_h^{\star,k}(s_h,a_h)^\top\right] + \lambda_{k,W}I_d$ . We first note that for any  $h$ , the following equation holds.
$$
\begin{array}{l} \sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \pi^ {k}\right)} \left[ \alpha_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \right] \\ = \sqrt {\sum_ {k \in [ K ]} \alpha_ {k , W} ^ {2} \sum_ {k \in [ K ]} \mathbb {E} _ {(s _ {h} , a _ {h}) \sim (P ^ {\star , k} , \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h} , a _ {h}) \right\| _ {(U _ {h , \phi^ {\star}, k} ^ {k, W}) ^ {- 1}} \right] ^ {2}} \\ = \sqrt {\sum_ {k \in [ K ]} \alpha_ {k , W} ^ {2} \sum_ {t = 0} ^ {[ K / W ]} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(U _ {h , \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} \right] ^ {2}} \tag {38} \\ \end{array}
$$
The  $\phi^{\star ,k}$  and  $U$  in Equation (38) both change with the round index  $k$ . To deal with such an issue, we divide the entire round into  $\lfloor \frac{K}{W} \rfloor + 1$  blocks with an equal length of  $W$ . For each block  $t \in \{0, \dots, \lfloor \frac{K}{W} \rfloor\}$ , we select an auxiliary anchor representation  $\phi^{\star,tW+1}$  and decompose Equation (38) as follows. We first derive the following equation:
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ - \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. + \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]. \\ = \underbrace {\sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(U _ {h , \phi^ {\star , k}} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(U _ {h , \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]} _ {(I)} \\ + \underbrace {\sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(U _ {h , \phi^ {\star} , t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]} _ {(I I)}. \tag {39} \\ \end{array}
$$
For term (II), we have
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) \right\| _ {(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \right\| _ {(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1}} ^ {2} \right] \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star}, k, \tilde {\pi} ^ {k})} \left[ \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) ^ {\top} (U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1} \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) - \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) ^ {\top} (U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \right. \\ \left. + \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right] \\ \stackrel {(i)} {\leq} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) \right\| _ {2} \left\| (U _ {h, \phi^ {\star , t W + 1}} ^ {k, W}) ^ {- 1} \right\| _ {2} \left\| \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) - \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \right\| _ {2} \right. \\ + \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \left\| \left(U _ {h, \phi^ {\star , t W + 1}} ^ {k, W}\right) ^ {- 1} \right\| _ {2} \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \Bigg ] \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \frac {2}{\lambda_ {W}} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \right] \\ \leq \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1, t (W + 1) - 1 ]} ^ {\phi}, \tag {40} \\ \end{array}
$$
where  $(i)$  follows from the property of the matrix norms induced by vector  $\ell_2$ -norm.
For term  $(I)$ , we have
$$
\begin{array}{l} \leq \frac {2}{\lambda_ {W} ^ {2}} \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , i}, \tilde {\pi} ^ {i}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \right] \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h, [ t W + 1, k - 1 ]} ^ {\phi}, \tag {41} \\ \end{array}
$$
where  $(i)$  follows from the property of the matrix norms induced by vector  $\ell_2$ -norm and  $(ii)$  follows from that  $\left\| \phi_h^{\star ,k}(s_h,a_h)\right\| _2\leq 1$
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \operatorname {t r} \left(\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \left(U _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}\right) \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \operatorname {t r} \left(\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \left(\tilde {U} _ {h, \phi^ {\star}, t W + 1} ^ {k, W, t}\right) ^ {- 1}\right) \\ \leq \sum_{k = tW + 1}^{(t + 1)W}A\underset { \begin{array}{c}s_{h}\sim (P^{\star ,k},\bar{\pi}^{k})\\ a_{h}\sim \mathcal{U}(\mathcal{A}) \end{array} }{\mathbb{E}}\operatorname {tr}\left[\phi_{h}^{\star ,tW + 1}(s_{h},a_{h})\phi_{h}^{\star ,tW + 1}(s_{h},a_{h})^{\top}\right](\tilde{U}_{h,\phi^{\star ,tW + 1}}^{k,W,t})^{-1} \\ \leq 2 A d \log \left(1 + \frac {W}{d \lambda_ {0}}\right), \tag {42} \\ \end{array}
$$
where the last equation follows from Lemma D.2.
Then combining Equations (39) to (42), we have
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ \leq 2 A d \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1, t (W + 1) - 1 ]} ^ {\phi} + \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h, [ t W + 1, k - 1 ]} ^ {\phi}. \tag {43} \\ \end{array}
$$
Substituting Equation (43) into Equation (38), we have
$$
\begin{array}{l} \sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \alpha_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k}\right) ^ {- 1}} \right] \\ \leq \sqrt {\sum_ {k \in [ K ]} \alpha_ {k , W} ^ {2} \sum_ {t = 0} ^ {[ K / W ]} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h} , a _ {h}) \sim (P ^ {\star , k} , \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h} , a _ {h}) \right\| _ {(U _ {h, \phi^ {\star}, k} ^ {k, W}) ^ {- 1}} \right] ^ {2}} \\ \leq \sqrt {\sum_ {k \in [ K ]} \alpha_ {k , W} ^ {2} \sum_ {t = 0} ^ {\lfloor K / W \rfloor} \left[ 2 A d \log (1 + \frac {W}{d \lambda_ {0}}) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1 , t (W + 1) - 1 ]} ^ {\phi} + \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \lor k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h , [ t W + 1 , k - 1 ]} ^ {\phi} \right]} \\ \leq \sqrt {K \left(2 W A \zeta_ {k , W} + \lambda_ {k , W} d\right) \left[ \frac {2 K A d}{W} \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ K ]} ^ {\phi} + \frac {2 W ^ {2}}{\lambda_ {W} ^ {2}} \Delta_ {\{h \}, [ K ]} ^ {\phi} \right]} \tag {44} \\ \end{array}
$$
where the second equation follows from Equation (43).
Then we derive the following bound:
$$
\sum_ {k = 1} ^ {K} V _ {P ^ {\star , k}, f ^ {k}} ^ {\tilde {\pi} ^ {k}}
$$
$$
\begin{array}{l} = \sum_ {k \in [ K ]} \sum_ {h \in [ H ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ f _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right] \\ \overset {(i)} {\leq} \sum_ {k \in [ K ]} \left\{\sum_ {h = 2} ^ {H} \left[ \mathbb {E} _ {(s _ {h - 1}, a _ {h - 1}) \sim (P ^ {\star}, k, \bar {\pi} ^ {k})} \left[ \alpha_ {k, W} \left| \left| \phi_ {h - 1} ^ {\star , k} (s _ {h - 1}, a _ {h - 1}) \right| \right| _ {(U _ {h - 1, \phi^ {\star}, k} ^ {k}) ^ {- 1}} \right] + \sqrt {\frac {1}{2 d} W A C _ {B} \Delta_ {[ h - 1 , h ], [ k - W , k - 1 ]} ^ {P}} \right] \right. \\ \left. + \sqrt {A \left(\zeta_ {k , W} + \frac {1}{2} C _ {B} \Delta_ {1 , [ k - W , k - 1 ]} ^ {P}\right)} \right\} \\ \leq \sum_ {h = 1} ^ {H - 1} \sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \alpha_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star , k}} ^ {k}\right) ^ {- 1}} \right] \\ + \sum_ {k \in [ K ]} \sum_ {h = 1} ^ {H} \sqrt {W A C _ {B} \Delta_ {[ h - 1 , h ] , [ k - W , k - 1 ]} ^ {P}} + \sum_ {k \in [ K ]} \sqrt {A \zeta_ {k , W}} \\ \stackrel {(i i)} {\leq} \sum_ {h = 1} ^ {H - 1} \sqrt {K \left(2 W A \zeta_ {k , W} + \lambda_ {k , W} d\right) \left[ \frac {2 A K d}{W} \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ K ]} ^ {\phi} + \frac {2 W ^ {2}}{\lambda_ {W} ^ {2}} \Delta_ {\{h \}, [ K ]} ^ {\phi} \right]} \\ + \sum_ {k \in [ K ]} \sum_ {h = 1} ^ {H} \sqrt {W A C _ {B} \Delta_ {[ h - 1 , h ] , [ k - W , k - 1 ]} ^ {P}} + \sum_ {k \in [ K ]} \sqrt {A \zeta_ {k , W}} \\ \leq \sqrt {K (2 W A \zeta_ {k , W} + \lambda_ {k , W} d)} \left[ H \sqrt {\frac {2 A K d}{W} \log \left(1 + \frac {W}{d \lambda_ {0}}\right)} + \sqrt {\frac {2 H W}{\lambda_ {W}} \Delta_ {[ H ] , [ K ]} ^ {\phi}} + \sqrt {\frac {2 H W ^ {2}}{\lambda_ {W} ^ {2}} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] \\ + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} + \sum_ {k \in [ K ]} \sqrt {A \zeta_ {k , W}} \\ \leq O \left(\sqrt {K (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {2 A K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right), \tag {45} \\ \end{array}
$$
where  $(i)$  follows from Lemma A.14, and  $(ii)$  follows from Equation (44).
Step 2: We next bound  $\sum_{k=1}^{K} V_{P^{\star,k},\hat{b}^k}^{\tilde{\pi}^k}$  via an auxiliary anchor representation.
Similarly to the proof Step 1, we further bound  $\sum_{k\in [K]}\mathbb{E}_{(s_h,a_h)\sim (P^{\star ,k},\tilde{\pi}^k)}\left[\beta_{k,W}\left\| \phi_h^{\star ,k}(s_h,a_h)\right\|_{(W_{h,\phi^{\star},k})^{-1}}\right]$ .
We define  $W_{h,\phi^{\star},k}^{k,W} = \sum_{i = 1\vee (k - W)}^{k - 1}\mathbb{E}_{(s_h,a_h)\sim (P^{\star ,i},\tilde{\pi}^i)}\left[\phi_h^{\star ,k}(s_h,a_h)\phi_h^{\star ,k}(s_h,a_h)^\top \right] + \lambda_{k,W}I_d$  and  $\tilde{W}_{h,\phi^{\star},k}^{k,W,t} = \sum_{i = tW + 1}^{k - 1}\mathbb{E}_{(s_h,a_h)\sim (P^{\star ,i},\tilde{\pi}^i)}\left[\phi_h^{\star ,k}(s_h,a_h)\phi_h^{\star ,k}(s_h,a_h)^\top \right] + \lambda_{k,W}I_d.$  We first note that for any  $h$  we have
$$
\begin{array}{l} \sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \beta_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \right] \\ = \sqrt {\sum_ {k \in [ K ]} \beta_ {k , W} ^ {2} \sum_ {k \in [ K ]} \mathbb {E} _ {(s _ {h} , a _ {h}) \sim (P ^ {\star , k} , \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h} , a _ {h}) \right\| _ {(W _ {h, \phi^ {\star}, k} ^ {k}) ^ {- 1}} \right] ^ {2}} \\ = \sqrt {\sum_ {k \in [ K ]} \beta_ {k , W} ^ {2} \sum_ {t = 0} ^ {\lfloor K / W \rfloor} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(W _ {h , \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} \right] ^ {2}} \tag {46} \\ \end{array}
$$
The  $\phi^{\star ,k}$  and  $W$  in Equation (46) both change with the round index  $k$ . To deal with this issue, we decompose it as follows. We first derive the following equation:
$$
\sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k}\right) ^ {- 1}} ^ {2} \right]
$$
$$
\begin{array}{l} - \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k}\right) ^ {- 1}} ^ {2} - \| \phi_ {h} ^ {\star} \left(s _ {h}, a _ {h}\right) \| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right. \\ \left. + \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]. \\ = \underbrace {\sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(W _ {h , \phi^ {\star} , k} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(W _ {h , \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]} _ {(I I I)} \\ + \underbrace {\sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h} , a _ {h}\right) \sim \left(P ^ {\star , k} , \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(W _ {h , \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h} , a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right]} _ {(I V)}. \tag {47} \\ \end{array}
$$
For term  $(IV)$ , we have