SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star}, k, \bar {\pi} ^ {k})} \left[ \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) ^ {\top} (W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1} \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) - \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) ^ {\top} (W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \right. \\ \left. + \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right] \\ \stackrel {(i)} {\leq} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \left\| \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \right\| _ {2} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \right. \\ + \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \left\| \left(W _ {h, \phi^ {\star , t W + 1}} ^ {k, W}\right) ^ {- 1} \right\| _ {2} \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \frac {2}{\lambda_ {W}} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \right] \\ \leq \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1, t (W + 1) - 1 ]} ^ {\phi}, \tag {48} \\ \end{array}
$$
For term (III), we derive the following bound:
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} - \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(\left(W _ {h, \phi^ {\star , k}} ^ {k, W}\right) ^ {- 1} - \left(W _ {h, \phi^ {\star , t W + 1}} ^ {k, W}\right) ^ {- 1}\right) \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \left(W _ {h, \phi^ {\star , k}} ^ {k, W}\right) ^ {- 1} \right. \\ \left. \times \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W} - W _ {h, \phi^ {\star}, k} ^ {k, W}\right) \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \left\| \left(W _ {h, \phi^ {\star , k}} ^ {k, W}\right) ^ {- 1} \right\| _ {2} \left\| \left(W _ {h, \phi^ {\star , t W + 1}} ^ {k, W} - W _ {h, \phi^ {\star , k}} ^ {k, W}\right) \right\| _ {2} \right. \\ \times \left\| \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1} \right\| _ {2} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \Bigg ] \\ \stackrel {(i i)} {\leq} \frac {1}{\lambda_ {W} ^ {2}} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \tilde {\pi} ^ {k})} \left[ \left\| \sum_ {i = 1 \vee k - W} ^ {k - 1} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , i}, \tilde {\pi} ^ {i})} \right. \left[ \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) ^ {\top} \right. \right. \\ \left. \left. - \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \right\rVert_ {2} \Bigg ] \\ \leq \frac {1}{\lambda_ {W} ^ {2}} \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star}, i, \bar {\pi} ^ {i})} \left[ \left| \left| \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \phi_ {h} ^ {\star , t W + 1} (s _ {h}, a _ {h}) \right. ^ {\top} - \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) \phi_ {h} ^ {\star , k} (s _ {h}, a _ {h}) \right. ^ {\top} \right\rvert   \left| _ {2} \right] \\ \leq \frac {1}{\lambda_ {W} ^ {2}} \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star}, i, \bar {\pi} ^ {i}\right)} \left[ \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \right\| _ {2} \right. \\ \left. + \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} - \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) ^ {\top} \right\| _ {2} \right] \\ \leq \frac {2}{\lambda_ {W} ^ {2}} \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , i}, \tilde {\pi} ^ {i}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) - \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {2} \right] \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h, [ t W + 1, k - 1 ]} ^ {\phi}. \tag {49} \\ \end{array}
$$
Furthermore, we derive the following bound:
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ = \sum_ {k = t W + 1} ^ {(t + 1) W} \operatorname {t r} \left(\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \left(W _ {h, \phi^ {\star}, t W + 1} ^ {k, W}\right) ^ {- 1}\right) \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \operatorname {t r} \left(\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \left(\tilde {W} _ {h, \phi^ {\star}, t W + 1} ^ {k, W, t}\right) ^ {- 1}\right) \\ \leq \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \operatorname {t r} \left[ \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) \phi_ {h} ^ {\star , t W + 1} \left(s _ {h}, a _ {h}\right) ^ {\top} \right] \left(\tilde {W} _ {h, \phi^ {\star}, t W + 1} ^ {k, W, t}\right) ^ {- 1} \\ \leq 2 d \log \left(1 + \frac {W}{d \lambda_ {0}}\right), \tag {50} \\ \end{array}
$$
Then combining Equations (39) to (41) and (50), we have
$$
\begin{array}{l} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k} ^ {k, W}\right) ^ {- 1}} ^ {2} \right] \\ \leq 2 d \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1, t (W + 1) - 1 ]} ^ {\phi} + \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h, [ t W + 1, k - 1 ]} ^ {\phi}. \tag {51} \\ \end{array}
$$
Substituting Equation (51) into Equation (46), we have
$$
\sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \bar {\pi} ^ {k}\right)} \left[ \beta_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(U _ {h, \phi^ {\star}, k}\right) ^ {- 1}} \right]
$$
$$
\begin{array}{l} \leq \sqrt {\sum_ {k \in [ K ]} \beta_ {k , W} ^ {2} \sum_ {t = 0} ^ {[ K / W ]} \sum_ {k = t W + 1} ^ {(t + 1) W} \mathbb {E} _ {(s _ {h} , a _ {h}) \sim (P ^ {\star , k} , \tilde {\pi} ^ {k})} \left[ \left\| \phi_ {h} ^ {\star , k} (s _ {h} , a _ {h}) \right\| _ {(U _ {h, \phi^ {\star}, k} ^ {k, W}) ^ {- 1}} \right] ^ {2}} \\ \leq \sqrt {\sum_ {k \in [ K ]} \beta_ {k , W} ^ {2} \sum_ {t = 0} ^ {\lfloor K / W \rfloor} \left[ 2 d \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ t W + 1 , t (W + 1) - 1 ]} ^ {\phi} + \sum_ {k = t W + 1} ^ {(t + 1) W} \sum_ {i = 1 \vee k - W} ^ {k - 1} \frac {2}{\lambda_ {W} ^ {2}} \Delta_ {h , [ t W + 1 , k - 1 ]} ^ {\phi} \right]} \\ \leq \sqrt {K (9 d A (2 W A \zeta_ {k , W} + \lambda_ {k , W} d) + \lambda_ {k , W} d) \left[ \frac {2 K d}{W} \log \left(1 + \frac {W}{d \lambda_ {0}}\right) + \frac {2 W}{\lambda_ {W}} \Delta_ {\{h \}, [ K ]} ^ {\phi} + \frac {2 W ^ {2}}{\lambda_ {W} ^ {2}} \Delta_ {\{h \}, [ K ]} ^ {\phi} \right]}. \tag {52} \\ \end{array}
$$
where the second equation follows from Equation (51).
Then, we derive the following bound:
$$
\begin{array}{l} \sum_ {k \in [ K ]} V _ {P ^ {\star , k}, \hat {b} ^ {k}} ^ {\tilde {\pi} ^ {k}} = \sum_ {k \in [ K ]} \sum_ {h \in [ H ]} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim (P ^ {\star , k}, \tilde {\pi} ^ {k})} \left[ \hat {b} _ {h} ^ {k} (s _ {h}, a _ {h}) \right] \\ \stackrel {(i)} {\leq} \sum_ {k \in [ K ]} \left\{\sum_ {h = 2} ^ {H} \left\{\mathbb {E} _ {\left(s _ {h - 1}, a _ {h - 1}\right) \sim \left(P ^ {* , k}, \tilde {\pi} ^ {k}\right)} \left[ \beta_ {k, W} \left\| \phi_ {h - 1} ^ {\star , k} \left(s _ {h - 1}, a _ {h - 1}\right) \right\| _ {\left(W _ {h - 1}, \phi^ {*}, k\right) ^ {- 1}} \right] \right. \right. \\ \left. \left. + \sqrt {\frac {A}{d} \Delta_ {h - 1 , [ k - W , k - 1 ]} ^ {P}} \right\} + \sqrt {\frac {9 A d \alpha_ {k , W} ^ {2}}{w}} \right\} \\ \leq \sum_ {h = 1} ^ {H - 1} \sum_ {k \in [ K ]} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \left(P ^ {\star , k}, \tilde {\pi} ^ {k}\right)} \left[ \beta_ {k, W} \left\| \phi_ {h} ^ {\star , k} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(W _ {h, \phi^ {\star}, k} ^ {k}\right) ^ {- 1}} \right] \\ + W \sqrt {\frac {A}{d}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} + \sum_ {k \in [ K ]} \sqrt {\frac {9 A d \alpha_ {k , W} ^ {2}}{W}} \\ \stackrel {(i i)} {\leq} \sqrt {K (9 d A (2 W A \zeta_ {k , W} + \lambda_ {k , W} d) + \lambda_ {k , W} d)} \left[ H \sqrt {\frac {2 K d}{W} \log \left(1 + \frac {W}{d \lambda_ {0}}\right)} + \sqrt {\frac {2 H W}{\lambda_ {W}} \Delta_ {[ H ], [ K ]} ^ {\phi}} \right. \\ \left. + \sqrt {\frac {2 H W ^ {2}}{\lambda_ {W} ^ {2}} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] + W \sqrt {\frac {A}{d}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \\ \leq O \left(\sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] \right. \\ \left. + W \sqrt {A} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right), \tag {53} \\ \end{array}
$$
where  $(i)$  follows from Lemma A.17, and  $(ii)$  follows from Equation (52).
Finally, combining Equations (37), (45) and (53), we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \hat {V} _ {\hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\tilde {\pi} ^ {k}} \leq O \Big (\sqrt {K (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \Big [ H \sqrt {\frac {2 A K d}{W} \log (W)} + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \Big ] + \sqrt {W ^ {3} A C _ {B}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} \Big) \\ + O \left(\sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W}} \log (W) + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] + W \sqrt {A} \Delta_ {[ H ], [ K ]} ^ {\overline {{P}}} \right. \\ \leq O \left(\sqrt {K d A (A \log (| \Phi | | \Psi | K H / \delta) + d ^ {2})} \left[ H \sqrt {\frac {K d}{W}} \log (W) + \sqrt {H W ^ {2} \Delta_ {[ H ] , [ K ]} ^ {\phi}} \right] + \sqrt {W ^ {3} A \Delta_ {[ H ] , [ K ]} ^ {\sqrt {P}}} \right). \\ \end{array}
$$
The following visitation probability difference lemma is similar to lemma 5 in Fei et al. (2020), but we remove their Assumption 1.
Lemma A.19. For any transition kernels  $\{P_h\}_{h=1}^H, h \in [H], j \in [h-1], s_h \in S$  and policies  $\{\pi_i\}_{i=1}^H$  and  $\pi_j'$ , we have
$$
\left| P _ {1} ^ {\pi_ {1}} \dots P _ {j} ^ {\pi_ {j}} \dots P _ {h - 1} ^ {\pi_ {h - 1}} (s _ {h}) - P _ {1} ^ {\pi_ {1}} \dots P _ {j} ^ {\pi_ {j} ^ {\prime}} \dots P _ {h - 1} ^ {\pi_ {h - 1}} (s _ {h}) \right| \leq \max  _ {s \in \mathcal {S}} \left\| \pi_ {j} (\cdot | s) - \pi_ {j} ^ {\prime} (\cdot | s) \right\| _ {T V}
$$
Proof. To prove this lemma, the only difference from lemma is that we need to show  $\max_{s_j}\sum_{s_{j + 1}}|P_j^{\pi_j}(s_{j + 1}|s_j) - P_j^{\pi_j'}(s_{j + 1}|s_j)|\leq 2\max_{s\in \mathcal{S}}\left\| \pi_j(\cdot |s) - \pi_j'(\cdot |s)\right\|_{TV}$  holds without assumption. We show this as follows:
$$
\begin{array}{l} \max _ {s _ {j}} \sum_ {s _ {j + 1}} \left| P ^ {\pi_ {j}} \left(s _ {j + 1} \mid s _ {j}\right) - P ^ {\pi_ {j} ^ {\prime}} \left(s _ {j + 1} \mid s _ {j}\right) \right| \\ = \max  _ {s _ {j}} \sum_ {s _ {j + 1}} \left| \sum_ {a} P (s _ {j + 1} | s _ {j}, a) \pi_ {j} (a | s _ {j}) - \sum_ {a} P (s _ {j + 1} | s _ {j}, a) \pi_ {j} ^ {\prime} (a | s _ {j}) \right| \\ \leq \max  _ {s _ {j}} \sum_ {s _ {j + 1}} \sum_ {a} P (s _ {j + 1} | s _ {j}, a) | \pi_ {j} (a | s _ {j}) - \pi_ {j} ^ {\prime} (a | s _ {j}) | \\ = \max  _ {s _ {j}} \sum_ {a} \sum_ {s _ {j + 1}} P (s _ {j + 1} | s _ {j}, a) | \pi_ {j} (a | s _ {j}) - \pi_ {j} ^ {\prime} (a | s _ {j}) | \\ = \max _ {s _ {j}} \sum_ {a} | \pi_ {j} (a | s _ {j}) - \pi_ {j} ^ {\prime} (a | s _ {j}) | = 2 \max _ {s \in \mathcal {S}} \| \pi_ {j} (\cdot | s) - \pi_ {j} ^ {\prime} (\cdot | s) \| _ {T V}. \\ \end{array}
$$
B Further Discussion and Proof of Corollary 4.5
In this section, we first provide a detailed version and further discussion of Corollary 4.5 in Appendix B.1, then present the proof in Appendix B.2, and finally present an interesting special case in Appendix B.3.
B.1 Further Discussion of Corollary 4.5
We present a detailed version of Corollary 4.5 as follows. Let  $\Pi_{[1,K]}(N) = \min \{K, \max \{1,N\}\}$  for any  $K, N \in \mathbb{N}$ .
Corollary B.1 (Detailed version of Corollary 4.5). Under the same conditions of Theorem 4.4, if the variation budgets are known, then for different variation budget regimes, we can select the hyper-parameters correspondingly to attain the optimality for both  $(I)$  w.r.t.  $W$  and  $(II)$  w.r.t.  $\tau$  in Equation (3). For  $(I)$ , with  $W = \Pi_{[1,K]}(\lfloor H^{\frac{1}{3}}d^{\frac{1}{3}}K^{\frac{1}{3}}(\Delta^{\sqrt{P}} + \Delta^{\phi})^{-\frac{1}{3}}\rfloor)$ , part  $(I)$  is upper-bounded by
$$
\left\{ \begin{array}{l l} \sqrt {\frac {H ^ {4} d ^ {2} A}{K} (A + d ^ {2})}, & \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) \leq \frac {H d}{K ^ {2}}, \\ H ^ {2} d ^ {\frac {5}{6}} A ^ {\frac {1}{2}} (A + d ^ {2}) ^ {\frac {1}{2}} (H K) ^ {- \frac {1}{6}} \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) ^ {\frac {1}{6}}, & \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) > \frac {H d}{K ^ {2}}, \end{array} \right. \tag {54}
$$
For (II) in Equation (3), with  $\tau = \Pi_{[1,K]}\big(\lfloor K^{\frac{2}{3}}(\Delta^{P} + \Delta^{\pi})^{-\frac{2}{3}}\rfloor \big)$ , part (II) is upper bounded by
$$
\left\{ \begin{array}{l l} \frac {2 H}{\sqrt {K}}, & \left(\Delta^ {P} + \Delta^ {\pi}\right) \leq \frac {1}{\sqrt {K}}, \\ 2 H ^ {\frac {4}{3}} (H K) ^ {- \frac {1}{3}} \left(\Delta^ {P} + \Delta^ {\pi}\right) ^ {\frac {1}{3}}, & \frac {1}{\sqrt {K}} <   \left(\Delta^ {P} + \Delta^ {\pi}\right) \leq K, \\ H + \frac {H \left(\Delta^ {P} + \Delta^ {\pi}\right)}{K}, & K <   \left(\Delta^ {P} + \Delta^ {\pi}\right) \end{array} \right. \tag {55}
$$
For any  $\epsilon \geq 0$ , if nonstationarity is not significantly large, i.e., there exists a constant  $\gamma < 1$  such that  $(\Delta^{P} + \Delta^{\pi}) \leq (2HK)^{\gamma}$  and  $(\Delta^{\sqrt{P}} + \Delta^{\phi}) \leq (2HK)^{\gamma}$ , then PORTAL can achieve  $\epsilon$ -average suboptimal with polynomial trajectories.
As a direct consequence of Theorem 4.4, Corollary 4.5 indicates that if variation budgets are known, then the agent can choose the best hyper-parameters directly based on the variation budgets. The  $\mathrm{Gap}_{\mathrm{Ave}}$  can be different depending on which regime the variation budgets fall into, as can be seen in Equations (54) and (55).
At the high level, we further explain how the window size  $W$  depends on the variations of environment as follows. If the nonstationarity is moderate and not significantly large, Corollary 4.5 indicates that for any  $\epsilon$ , Algorithm 1 achieves  $\epsilon$ -average suboptimal with polynomial trajectories (see the specific form in Equation (59) in Appendix B.2). If the environment is near stationary and the variation is relatively small, i.e.,  $(\Delta^{\sqrt{P}} + \Delta^{\phi}) \leq Hd / K^{2}$ ,  $(\Delta^{P} + \Delta^{\pi}) \leq 1 / \sqrt{K}$ , then the best window size  $W$
and the policy restart period  $\tau$  are both  $K$ . This indicates that the agent does not need to take any forgetting rules to handle the variation. Then the  $\mathrm{Gap}_{\mathrm{Ave}}$  reduces to  $\tilde{O}\left(\sqrt{H^4d^2A(A + d^2) / K}\right)$ , which matches the result under a stationary environment.
Furthermore, it is interesting to consider a special mildly changing environment, in which the representation  $\phi^{\star}$  stays identical and only the state-embedding function  $\mu^{\star,k}$  changes over time. The average dynamic suboptimality gap in Equation (3) reduces to
$$
\tilde {O} \Big (\underbrace {\sqrt {\frac {H ^ {4} d ^ {2} A (A + d ^ {2})}{W}} + \sqrt {\frac {H ^ {2} W ^ {3} A}{K ^ {2}}} \Delta^ {\sqrt {P}}} _ {(I)} + \underbrace {\frac {H}{\sqrt {\tau}} + \frac {H \tau (\Delta^ {P} + \Delta^ {\pi})}{K}} _ {(I I)} \Big).
$$
The part (II) is the same as (II) in Equation (3) and by choosing the best window size of  $\overline{W} = H^{\frac{1}{2}}d^{\frac{1}{2}}(A + d^{2})^{\frac{1}{4}}K^{\frac{1}{2}}(\Delta^{\sqrt{P}})^{-\frac{1}{2}}$ , part (I) becomes
$$
\tilde {O} \left(H ^ {2} d ^ {\frac {3}{4}} A ^ {\frac {1}{2}} (A + d ^ {2}) ^ {\frac {3}{8}} (H K) ^ {- \frac {1}{4}} \left(\Delta^ {\sqrt {P}}\right) ^ {\frac {1}{4}}\right). \tag {56}
$$
Compared with the second regime in Equation (54), Equation (56) is much smaller, benefited from identical representation function  $\phi^{\star}$ . In this way, samples in previous rounds can help to estimate the representation space so that  $\overline{W}$  can be larger than  $W$  in terms of the order of  $K$ , which yields efficiency gain compared with changing  $\phi^{\star}$ .
On the other hand, if the nonstationarity is significantly large, for example, scales linearly with  $K$ , then for each round, the previous samples cannot help to estimate current best policy. Thus, the best  $W$  and  $\tau$  are both 1, and the average dynamic suboptimality gap reduces to  $\tilde{O}\left(\sqrt{H^4d^2A(A + d^2)}\right)$ .
This indicates that for a fixed small accuracy  $\epsilon \geq 0$ , no matter how large the round  $K$  is, Algorithm 1 can never achieve  $\epsilon$ -average suboptimality.
B.2 Proof of Corollary B.1 (i.e., Detailed Version of Corollary 4.5)
If variation budgets are known, for different variation budgets regimes, we can tune the hyperparameters correspondingly to reach the optimality for both the term  $(I)$  that contains  $W$  and the term  $(II)$  that contains  $\tau$ .
For the first term  $(I)$  in Equation (14), there are two regimes:
- Small variation:  $\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right) \leq \frac{Hd}{K^{2}}$ ,
- The best window size  $W$  is  $K$ , which means that the variation is pretty mild and the environment is near stationary. In this case, by choosing window size  $W = K$ , the agent takes no forgetting rules to handle the variation. Then the first term  $(I)$  reduces to  $\sqrt{\frac{H^4d^2A}{K}(A + d^2)}$ , which matches the result under a stationary environment. $^3$
- Then for any  $\epsilon \geq 0$ , with  $HK$  no more than  $\tilde{O}\left(\frac{H^5d^2A(A + d^2)}{\epsilon^2}\right)$ , term  $(I) \leq \epsilon$ .
Large variation:  $\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right) > \frac{Hd}{K^{2}}$
- By choosing the window size  $W = H^{\frac{1}{3}}d^{\frac{1}{3}}K^{\frac{1}{3}}\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right)^{-\frac{1}{3}}$ , the term (I) reduces to  $H^{2}d_{6}^{\frac{5}{6}}A^{\frac{1}{2}}\left(A + d^{2}\right)^{\frac{1}{2}}(HK)^{-\frac{1}{6}}\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right)^{\frac{1}{6}}$ .
- Since  $\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right) \leq 2HK$ , there exists  $\gamma \leq 1$  s.t.  $\left(\Delta^{\sqrt{P}} + \Delta^{\phi}\right) \leq (2HK)^{\gamma}$ . Then  $(I) \leq 2H^{2}d_{6}^{\frac{5}{6}}A^{\frac{1}{2}}(A + d^{2})^{\frac{1}{2}}(HK)^{-\frac{1 - \gamma}{6}}$ . Then for any  $\epsilon \geq 0$ , if  $\gamma \neq 1$ , with  $HK$  no more than  $\tilde{O}\left(\frac{d^{\frac{5}{1 - \gamma}}H^{\frac{12}{1 - \gamma}}A^{\frac{3}{1 - \gamma}}(A + d^{2})^{\frac{3}{1 - \gamma}}}{\epsilon^{\frac{6}{1 - \gamma}}}\right)$ , term  $(I) \leq \epsilon$ .
2We convert the sample complexity bound under infinite horizon MDPs in Uehara et al. (2022) to the average dynamic suboptimality gap under episodic MDPs.
<sup>3</sup>We convert the regret bound under infinite horizon MDPs in Uehara et al. (2022) to the average dynamic suboptimality gap under episodic MDP.
For the second term (II) in Equation (14), there are three regimes elaborated as follows:
- Small variation:  $(\Delta^P + \Delta^\pi) \leq \frac{1}{\sqrt{K}}$ ,
- The best policy restart period  $\tau$  is  $K$ , which means that the variation is pretty mild and the agent does not need to handle the variation. Then term  $(II) \leq \frac{2H}{\sqrt{K}}$ .
- Then for any  $\epsilon \geq 0$ , with  $HK$  no more than  $\tilde{O}\left(\frac{H^2}{\epsilon^2}\right)$ , term  $(II) \leq \epsilon$ .
- Moderate variation:  $\frac{1}{\sqrt{K}} < (\Delta^P + \Delta^\pi) \leq K$
- The best policy restart period  $\tau = K^{\frac{2}{3}}(\Delta^{P} + \Delta^{\pi})^{-\frac{2}{3}}$ , and the term (II) reduces to  $2HK^{-\frac{1}{3}}(\Delta^{P} + \Delta^{\pi})^{\frac{1}{3}}$ .