SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Since  $\left(\Delta^{P} + \Delta^{\pi}\right) \leq 2HK$ , there exists  $\gamma \leq 1$  s.t.  $\left(\Delta^{P} + \Delta^{\pi}\right) \leq (2HK)^{\gamma}$ . Then the term  $(II) \leq 4H^{\frac{4}{3}}(HK)^{-\frac{1 - \gamma}{3}}$ . Then for any  $\epsilon \geq 0$ , if  $\gamma \neq 1$ , with  $HK$  no more than  $\tilde{O}\left(\frac{H^{\frac{4}{1 - \gamma}}}{\epsilon^{\frac{3}{1 - \gamma}}}\right)$ , term  $(II) \leq \epsilon$ .
Large variation:  $K < (\Delta^{P} + \Delta^{\pi})$
- The variation budgets scale linearly with  $K$ , which indicates that the nonstationarity of the environment is significantly large and lasts for the entire rounds. Hence in each round, the previous sample can not help to estimate the current best policy. So the best policy restart period  $\tau = 1$ , and the second term  $(II)$  reduces to  $H + \frac{H(\Delta^P + \Delta^\pi)}{K} = O(H)$ , which implies that Algorithm 1 can never achieve small average dynamic suboptimality gap for any large  $K$ .
In conclusion, the first term is upper bounded by
$$
(I) \leq \left\{ \begin{array}{l l} \sqrt {\frac {H ^ {4} d ^ {2} A}{K} (A + d ^ {2})}, & \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) \leq \frac {H d}{K ^ {2}}, \\ H ^ {2} d ^ {\frac {5}{6}} A ^ {\frac {1}{2}} (A + d ^ {2}) ^ {\frac {1}{2}} (H K) ^ {- \frac {1}{6}} \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) ^ {\frac {1}{6}}, & \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) > \frac {H d}{K ^ {2}}, \end{array} \right. \tag {57}
$$
and the second term is upper bounded by
$$
(I I) \leq \left\{ \begin{array}{l l} \frac {2 H}{\sqrt {K}}, & \left(\Delta^ {P} + \Delta^ {\pi}\right) \leq \frac {1}{\sqrt {K}}, \\ 2 H ^ {\frac {4}{3}} (H K) ^ {- \frac {1}{3}} \left(\Delta^ {P} + \Delta^ {\pi}\right) ^ {\frac {1}{3}}, & \frac {1}{\sqrt {K}} <   \left(\Delta^ {P} + \Delta^ {\pi}\right) \leq K, \\ H + \frac {H \left(\Delta^ {P} + \Delta^ {\pi}\right)}{K}, & K <   \left(\Delta^ {P} + \Delta^ {\pi}\right) \end{array} \right. \tag {58}
$$
In addition, if the variation budgets are not significantly large, i.e. scale linearly with  $K$ , for any  $\epsilon \geq 0$ , Algorithm 1 can achieve  $\epsilon$ -average dynamic suboptimality gap with at most polynomial samples. Specifically, if there exists a constant  $\gamma < 1$  such that the variation budgets satisfying  $(\Delta^{P} + \Delta^{\pi}) \leq (2HK)^{\gamma}$  and  $(\Delta^{\sqrt{P}} + \Delta^{\phi}) \leq (2HK)^{\gamma}$ , then to achieve  $\epsilon$ -average dynamic suboptimality gap, i.e.,
$\mathrm{Gap}_{\mathrm{Ave}}(K) \leq \epsilon$ , Algorithm 1 only needs to collect trajectories no more than
$$
\left\{ \begin{array}{l} \tilde {O} \left(\frac {H ^ {5} d ^ {2} A (A + d ^ {2})}{\epsilon^ {2}}\right), \\ \text {i f} \quad \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) \leq \frac {H d}{K ^ {2}}, (\Delta^ {P} + \Delta^ {\pi}) \leq \frac {1}{\sqrt {K}}; \\ \tilde {O} \left(\frac {d ^ {\frac {5}{1 - \gamma}} H ^ {\frac {1 2}{1 - \gamma}} A ^ {\frac {3}{1 - \gamma}} (A + d ^ {2}) ^ {\frac {3}{1 - \gamma}}}{\epsilon^ {\frac {6}{1 - \gamma}}} + \frac {H ^ {2}}{\epsilon^ {2}}\right), \\ \text {i f} \quad \frac {H d}{K ^ {2}} <   \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) \leq (H K) ^ {\gamma}, (\Delta^ {P} + \Delta^ {\pi}) \leq \frac {1}{\sqrt {K}}; \\ \tilde {O} \left(\frac {H ^ {5} d ^ {2} A (A + d ^ {2})}{\epsilon^ {2}} + \frac {H ^ {\frac {4}{1 - \gamma}}}{\epsilon^ {\frac {3}{1 - \gamma}}}\right), \\ \text {i f} \quad \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi}\right) \leq \frac {H d}{K ^ {2}}, \frac {1}{\sqrt {K}} <   (\Delta^ {P} + \Delta^ {\pi}) \leq (H K) ^ {\gamma}; \\ \tilde {O} \left(\frac {d ^ {\frac {5}{1 - \gamma}} H ^ {\frac {1 2}{1 - \gamma}} A ^ {\frac {3}{1 - \gamma}} (A + d ^ {2}) ^ {\frac {3}{1 - \gamma}}}{\epsilon^ {\frac {6}{1 - \gamma}}}\right), \\ \text {i f} \quad \frac {H d}{K ^ {2}} <   (\Delta^ {\sqrt {P}} + \Delta^ {\phi}) \leq (H K) ^ {\gamma}, \frac {1}{\sqrt {K}} <   (\Delta^ {P} + \Delta^ {\pi}) \leq (H K) ^ {\gamma}. \end{array} \right. \tag {59}
$$
B.3 A Special Case
In this subsection, we provide a characterization of a special case, where the representation  $\phi^{\star}$  stays identical and only the state-embedding function  $\mu^{\star ,k}$  changes over time. In such a scenario, the variation budget  $\Delta_{[H],[K]}^{\phi} = 0$  and the average dynamic suboptimality gap bound in Equation (14) reduces to
$$
\begin{array}{l} \operatorname {G a p} _ {\mathrm {A v e}} (K) \\ \leq \tilde {O} \left(\sqrt {\frac {H ^ {4} d ^ {2} A}{W} (A + d ^ {2})} + \sqrt {\frac {H ^ {2} W ^ {3} A}{K ^ {2}}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}} + \frac {2 H}{\sqrt {\tau}} + \frac {3 H \tau}{K} \left(\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}\right)\right) \\ \leq \tilde {O} \left(H ^ {\frac {7}{4}} d ^ {\frac {3}{4}} A ^ {\frac {1}{2}} \left(A + d ^ {2}\right) ^ {\frac {3}{8}} K ^ {- \frac {1}{4}} \left(\Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right) ^ {\frac {1}{4}} + H K ^ {- \frac {1}{3}} (\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}) ^ {\frac {1}{3}}\right), \\ \end{array}
$$
where the last equation follows from the choice of the window side  $W = \tilde{O}\left(H^{\frac{1}{2}}d^{\frac{1}{2}}(A + d^{2})^{\frac{1}{4}}K^{\frac{1}{2}}\left(\Delta_{[H],[K]}^{\sqrt{P}}\right)^{-\frac{1}{2}}\right)$  and the policy restart period  $\tau = \tilde{O}\left(K^{\frac{2}{3}}(\Delta_{[H],[K]}^{P} + \Delta_{[H],[K]}^{\pi})^{-\frac{2}{3}}\right)$  with known variation budgets.
C Proof of Theorem 5.1 and Detailed Comparison with Wei & Luo (2021)
C.1 Proof of Theorem 5.1
Proof of Theorem 5.1. Before our formal proof, we first explain several notations on different choices of  $W$  and  $\tau$  here.
-  $(W^{\star},\tau^{\star})$  : We denote  $W^{\star} = d^{\frac{1}{3}}H^{\frac{1}{3}}K^{\frac{1}{3}}(\Delta^{\phi} + \Delta^{\sqrt{P}} + 1)^{-\frac{1}{3}}$ , and  $\tau^{\star} = \tilde{O}\left(K^{\frac{2}{3}}(\Delta^{P} + \Delta^{\pi} + 1)^{-\frac{2}{3}}\right)$ .
-  $(\overline{W}, \overline{\tau})$ : Because  $W^{\star} = d^{\frac{1}{3}}H^{\frac{1}{3}}K^{\frac{1}{3}}(\Delta^{\phi} + \Delta^{\sqrt{P}} + 1)^{-\frac{1}{3}} \leq d^{\frac{1}{3}}H^{\frac{1}{3}}K^{\frac{1}{3}} \leq J_{W}$  and  $\tau^{\star} = \tilde{O}\left(K^{\frac{2}{3}}(\Delta^{P} + \Delta^{\pi} + 1)^{-\frac{2}{3}}\right) \leq K^{\frac{2}{3}} \leq J_{\tau}$ . As a result, there exists a  $\overline{W} \in \mathcal{J}_W$  such that  $\overline{W} \leq W^{\star} \leq 2\overline{W}$  and a  $\overline{\tau} \in \mathcal{J}_{\tau}$  such that  $\overline{\tau} \leq \tau^{\star} \leq 2\overline{\tau}$ .
-  $(W^{\dagger},\tau^{\dagger})$ :  $(W^{\dagger},\tau^{\dagger})$  denotes the set of best choices of the window size  $W$  and the policy restart period  $\tau$  in feasible set that maximize  $\sum_{i=1}^{\lceil K/M\rceil}R_i(W,\tau)$ .
Then we can decompose the average dynamic suboptimality gap as
$$
\begin{array}{l} \operatorname {G a p} _ {\mathrm {A v e}} (K) = \frac {1}{K} \sum_ {k \in [ K ]} \left[ V _ {P ^ {\star}, k} ^ {\pi^ {\star}} - V _ {P ^ {\star}, k} ^ {\pi^ {k}} \right] \\ = \frac {1}{K} \underbrace {\sum_ {k = 1} ^ {K} V _ {P ^ {\star} , k} ^ {\pi^ {\star}} - \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} \left[ R _ {i} (\overline {{W}} , \overline {{\tau}}) \right]} _ {(I)} + \frac {1}{K} \underbrace {\sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} [ R _ {i} (\overline {{W}} , \overline {{\tau}}) ] - \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} [ R _ {i} (W _ {i} , \tau_ {i}) ]} _ {(I I)}, \\ \end{array}
$$
where the last inequality follows because if  $\{\pi^k\}_{k=1}^K$  is the output of Algorithm 3 with the chosen window size  $\{W_i\}_{i=1}^{[T/M]}$ ,  $\mathbb{E}[R_i(W_i, \tau_i)] = \mathbb{E}\left[\sum_{k=(i-1)M+1}^{\min\{iM, K\}} V_1^k\right] = \sum_{k=(i-1)M+1}^{\min\{iM, K\}} V_{P^\star, k}^{\pi^k}$  holds.
We next bound Terms  $(I)$  and  $(II)$  separately.
Term (I): We derive the following bound:
$$
\begin{array}{l} \frac {1}{K} \left\{\sum_ {k = 1} ^ {K} V _ {1} ^ {\pi^ {*}, k, k} - \sum_ {i = 1} ^ {\lceil K / M \rceil} R _ {i} (\overline {{W}}, \bar {\tau}) \right\} \\ \stackrel {(i)} {\leq} \tilde {O} \left(\sqrt {\frac {H ^ {4} d ^ {2} A}{\overline {{W}}}} (A + d ^ {2}) + \sqrt {\frac {H ^ {3} d A}{K}} (A + d ^ {2}) \overline {{W}} ^ {2} \Delta_ {[ H ], [ K ]} ^ {\phi} + \sqrt {\frac {H ^ {2} \overline {{W}} ^ {3} A}{K ^ {2}}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right) \\ + \tilde {O} \left(\frac {2 H}{\sqrt {\bar {\tau}}} + \frac {3 H \bar {\tau}}{K} \left(\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}\right)\right) \\ \stackrel {(i i)} {\leq} \tilde {O} \left(\sqrt {\frac {H ^ {4} d ^ {2} A}{W ^ {\star}} (A + d ^ {2})} + \sqrt {\frac {H ^ {3} d A}{K} (A + d ^ {2}) W ^ {\star 2} \Delta_ {[ H ], [ K ]} ^ {\phi}} + \sqrt {\frac {H ^ {2} W ^ {\star 3} A}{K ^ {2}}} \Delta_ {[ H ], [ K ]} ^ {\sqrt {P}}\right) \\ + \tilde {O} \left(\frac {2 H}{\sqrt {\tau^ {\star}}} + \frac {3 H \tau^ {\star}}{K} \left(\Delta_ {[ H ], [ K ]} ^ {P} + \Delta_ {[ H ], [ K ]} ^ {\pi}\right)\right) \\ \stackrel {(i i i)} {\leq} \tilde {O} \left(H ^ {\frac {1 1}{6}} d _ {6} ^ {\frac {5}{6}} A ^ {\frac {1}{2}} \left(A + d ^ {2}\right) ^ {\frac {1}{2}} K ^ {- \frac {1}{6}} \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi} + 1\right) ^ {\frac {1}{6}}\right) + \tilde {O} \left(2 H K ^ {- \frac {1}{3}} (\Delta^ {P} + \Delta^ {\pi} + 1) ^ {\frac {1}{3}}\right), \\ \end{array}
$$
where  $(i)$  follows from Equation (14),  $(ii)$  follows from the definition of  $\overline{W}$  and  $(iii)$  follows from the definition of  $W^{\star}$  at the beginning of the proof.
Term (II): We derive the following bound:
$$
\begin{array}{l} \frac {1}{K} \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} \left[ R _ {i} (\bar {W}, \bar {\tau}) \right] - \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} \left[ R _ {i} \left(W _ {i}, \tau_ {i}\right) \right] \\ \stackrel {(i)} {\leq} \frac {1}{K} \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} \left[ R _ {i} \left(W ^ {\dagger}, \tau^ {\dagger}\right) \right] - \sum_ {i = 1} ^ {\lceil K / M \rceil} \mathbb {E} \left[ R _ {i} \left(W _ {i}, \tau_ {i}\right) \right] \\ \stackrel {(i i)} {\leq} \tilde {O} (M \sqrt {J \lceil K / M \rceil} / K) \\ = \tilde {O} (\sqrt {J K M}) = \tilde {O} \left(H ^ {\frac {1}{6}} d ^ {\frac {1}{6}} K ^ {- \frac {1}{6}}\right), \\ \end{array}
$$
where  $(i)$  follows from the definition of  $W^{\dagger}$  and  $(ii)$  follows from Theorem 3.3 in Bubeck & Cesa-Bianchi (2012) with the adaptation that reward  $R_{i} \leq M$  and the number of iteration is  $\lceil K / M\rceil$ . Then combining the bounds on terms  $(I)$  and  $(II)$ , we have
$$
\mathrm {G a p} _ {\mathrm {A v e}} (K) \leq \tilde {O} \left(H ^ {\frac {1 1}{6}} d ^ {\frac {5}{6}} A ^ {\frac {1}{2}} \left(A + d ^ {2}\right) ^ {\frac {1}{2}} K ^ {- \frac {1}{6}} \left(\Delta^ {\sqrt {P}} + \Delta^ {\phi} + 1\right) ^ {\frac {1}{6}}\right) + \tilde {O} \left(2 H K ^ {- \frac {1}{3}} \left(\Delta^ {P} + \Delta^ {\pi} + 1\right) ^ {\frac {1}{3}}\right).
$$
C.2 Detailed Comparison with Wei & Luo (2021)
Based on the theoretical results Theorem 4.4 and taking Algorithm 1 PORTAL as a base algorithm, we can also use the black-box techniques called MASTER in Wei & Luo (2021) to handle the unknown variation budgets. But such an approach of MASTER+PORTAL turns out to have a larger average dynamic suboptimality gap than Algorithm 3. Denote  $\Delta = \Delta^{\phi} + \Delta^{\sqrt{P}} + \Delta^{\pi}$ .
To explain, first choose  $\tau = k$  and  $W = K$  in Algorithm 1. Then Algorithm 1 reduces to a base algorithm with the suboptimality gap of  $\tilde{O} (\sqrt{H^4d^2A(A + d^2) / K} +\sqrt{H^3dAK(A + d^2)}\Delta)$ , which satisfies Assumption 1 in Wei & Luo (2021) with  $\rho (t) = \sqrt{H^4d^2A(A + d^2) / t}$  and  $\Delta_{[1,t]} = \sqrt{H^3dAt(A + d^2)}\Delta$ . Then by Theorem 2 in Wei & Luo (2021), the dynamic regret using MASTER+PORTAL can be upper-bounded by  $\tilde{O} (H^{\frac{11}{6}}d^{\frac{5}{6}}A^{\frac{1}{2}}(A + d^{2})^{\frac{1}{2}}K^{\frac{2}{3}}\Delta^{\frac{1}{3}}) = \tilde{O} (K^{\frac{5}{6}}\Delta^{\frac{1}{3}})$ . Here, we find the order dependency on  $d,H,A$  is the same as Theorem 5.1 and hence mainly focus on the order dependency on  $K$  and  $\Delta$  in the following comparison. Such a bound on dynamic regret can be converted to the average dynamic suboptimality gap as  $\tilde{O} (K^{-\frac{1}{6}}\Delta^{\frac{1}{3}})$ . Then it can be observed that for not too small variation budgets, i.e.,  $\Delta \geq \tilde{O}(1)$ , the order dependency on  $\Delta$  is higher than that of Algorithm 3.
Specifically, if we consider the special case when the representation stays identical and denote  $\tilde{\Delta} = \Delta^{\sqrt{P}} + \Delta^{\pi}$ , then the average dynamic suboptimality gap of MASTER+PORTAL is still  $\tilde{O}(K^{-\frac{1}{6}}\tilde{\Delta}^{\frac{1}{3}})$ . With small modifications on the parameters, by following the analysis similar to that of Theorem 5.1 and Appendix B.3, we can show that the average dynamic suboptimality gap of Algorithm 3 satisfies  $\tilde{O}(K^{-\frac{1}{4}}\tilde{\Delta}^{\frac{1}{4}})$ , which is smaller than MASTER+PORTAL.
In this section, we provide two lemmas that are commonly used for the analysis of MDP problems.
The following lemma (Dann et al., 2017) is useful to measure the difference between two value functions under two MDPs and reward functions.
Lemma D.1. (Simulation Lemma). Suppose  $P_{1}$  and  $P_{2}$  are two MDPs and  $r_{1}, r_{2}$  are the corresponding reward functions. Given a policy  $\pi$ , we have,
$$
\begin{array}{l} V _ {h, P _ {1}, r _ {1}} ^ {\pi} \left(s _ {h}\right) - V _ {h, P _ {2}, r _ {2}} ^ {\pi} \left(s _ {h}\right) \\ = \sum_{h^{\prime} = h}^{H}\underset { \begin{array}{s}s_{h^{\prime}}\sim (P_{2},\pi)\\ a_{h^{\prime}}\sim \pi \end{array} }{\mathbb{E}}\left[r_{1}(s_{h^{\prime}},a_{h^{\prime}}) - r_{2}(s_{h^{\prime}},a_{h^{\prime}}) + (P_{1,h^{\prime}} - P_{2,h^{\prime}})V_{h^{\prime} + 1,P_{1},r}^{\pi}(s_{h^{\prime}},a_{h^{\prime}})|s_{h}\right] \\ = \sum_{h^{\prime} = h}^{H}\mathbb{E}_{\substack{s_{h^{\prime}}\sim (P_{1},\pi)\\ a_{h^{\prime}}\sim \pi}}\left[r_{1}(s_{h^{\prime}},a_{h^{\prime}}) - r_{2}(s_{h^{\prime}},a_{h^{\prime}}) + (P_{1,h^{\prime}} - P_{2,h^{\prime}})V_{h^{\prime} + 1,P_{2},r}^{\pi}(s_{h^{\prime}},a_{h^{\prime}})|s_{h}\right]. \\ \end{array}
$$
The following lemma is a standard inequality in the regret analysis for linear MDPs in reinforcement learning (see Lemma G.2 in Agarwal et al. (2020b) and Lemma 10 in Uehara et al. (2022)).
Lemma D.2 (Elliptical Potential Lemma). Consider a sequence of  $d \times d$  positive semidefinite matrices  $X_{1},\ldots ,X_{N}$  with  $\operatorname {tr}(X_n)\leq 1$  for all  $n\in [N]$ . Define  $M_0 = \lambda_0I$  and  $M_{n} = M_{n - 1} + X_{n}$ . Then the following bound holds:
$$
\sum_ {n = 1} ^ {N} \operatorname {t r} \left(X _ {n} M _ {n - 1} ^ {- 1}\right) \leq 2 \log \det  (M _ {N}) - 2 \log \det  (M _ {0}) \leq 2 d \log \left(1 + \frac {N}{d \lambda_ {0}}\right).
$$