SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {D} - \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right), \tag {4}
$$
where  $\pi_1^c, \ldots, \pi_K^c$  is any sequence of compared policies. We define  $\pi_0^c = \pi_1^c$  to simplify the notation.
The non-stationarity measure is defined as  $P_{T} = \sum_{k=1}^{K} \sum_{h=1}^{H} \left\| \pi_{k,h}^{c} - \pi_{k-1,h}^{c} \right\|_{1,\infty}$ .
4 Optimal Dynamic Regret with Known  $P_{T}$
We present our proposed algorithm in Algorithm 1. Similar to previous studies, the algorithm consists of (i) policy improvement phase, and (ii) policy evaluation phase. We introduce the details below. In Sections 4.1, we first consider the case when the transition is known to highlight the challenges even under the ideal setting. Then, we extend the results to the unknown transition setting in Section 4.2.
4.1 Policy Improvement Phase
In the policy improvement phase, the algorithm updates  $\pi_{k}$  based on  $\pi_{k - 1}$  using the proximal policy optimization (PPO) method [50]. Specifically, at episode  $k$ , we define the following linear function:
$$
L _ {k - 1} (\pi) = V _ {k, 1} ^ {\pi_ {k - 1}} (s _ {k, 1}) + \mathbb {E} _ {\pi_ {k - 1}} \left[ \sum_ {h = 1} ^ {H} \langle Q _ {k - 1, h} ^ {\pi_ {k - 1}}, \pi_ {h} (\cdot | s _ {h}) - \pi_ {k - 1, h} (\cdot | s _ {h}) \rangle \Big | s _ {1} = s _ {k, 1} \right],
$$
which is the first-order Taylor approximation of  $V_{k - 1,1}^{\pi}(s_{k,1})$  around  $\pi_{k - 1}$ . Then, we update  $\pi_k$  by
$$
\pi_ {k} = \underset {\pi \in \Delta (\mathcal {A} \mid \mathcal {S}, H)} {\arg \max } L _ {k - 1} (\pi) - \frac {1}{\eta} \mathbb {E} _ {\pi_ {k - 1}} \left[ \sum_ {h = 1} ^ {H} D _ {\mathrm {K L}} \left(\pi_ {h} (\cdot \mid s _ {h}) \| \pi_ {k - 1, h} (\cdot \mid s _ {h})\right) \right], \tag {5}
$$
where  $\eta > 0$  is the stepsize and the KL-divergence encourages  $\pi_k$  to be close to  $\pi_{k-1}$  so that  $L_{k-1}(\pi)$  is a good approximation of  $V_{k-1,1}^{\pi}(s_{k,1})$ . The update rule in (5) takes the following closed form,
$$
\pi_ {k, h} (\cdot \mid s) \propto \pi_ {k - 1, h} (\cdot \mid s) \cdot \exp \left(\eta \cdot Q _ {k - 1, h} ^ {\pi_ {k - 1}} (s, \cdot)\right), \forall h \in [ H ], s \in \mathcal {S}. \tag {6}
$$
We show the update rule in (6) ensures the following guarantee and the proof is in Appendix C.1.
Lemma 1. The update rule in (6) ensures the following dynamic regret guarantee:
$$
\text {D - R e g r e t} (K) \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \widetilde {D} _ {\mathrm {K L}} \left(\pi_ {k} ^ {c} \| \pi_ {k}\right) - \widetilde {D} _ {\mathrm {K L}} \left(\pi_ {k} ^ {c} \| \pi_ {k + 1}\right) \right]. \tag {7}
$$
Note that the expectation in the last term in (7) is taken over  $\pi_k^c$  which may change over episode  $k$ . For static regret, i.e.,  $\pi_1^c = \ldots = \pi_K^c = \pi^*$ , we can control this term by a standard telescoping argument, which is not viable for dynamic regret analysis. Fei et al. [35] propose a restart strategy to handle this term. Specifically, they restart the algorithm every certain number of steps and decompose the above expectation into  $\mathbb{E}_{\pi_k^c}[\cdot] = \mathbb{E}_{\pi_{k_0}^c}[\cdot] + \mathbb{E}_{\pi_k^c - \pi_{k_0}^c}[\cdot]$  where  $k_0 < k$  is the episode in which restart takes place most recently before episode  $k$ . For the first expectation  $\mathbb{E}_{\pi_{k_0}^c}[\cdot]$ , they apply a customized telescoping argument to each period as the expectation is taken over the fixed policy. The second expectation  $\mathbb{E}_{\pi_k^c - \pi_{k_0}^c}[\cdot]$  involves the difference  $\pi_k^c - \pi_{k_0}^c$  and can be bounded by  $P_T$ . However, as we will show in Theorem 4, their regret bound is suboptimal in terms of  $K$  and  $P_T$ .
We introduce our approach below. Let us first consider taking expectations over any fixed policy  $\pi$ . Denote by  $\delta$  the minimal probability over any action at any state for policies  $\pi_1, \ldots, \pi_K$ , i.e.,  $\delta = \min_{k \in [K]} \pi_k(a \mid s), \forall a \in \mathcal{A}, s \in S$ , the last term in (7) can be upper bounded by  $\sum_{k=1}^{K} \mathbb{E}_{\pi}[\widetilde{D}_{\mathrm{KL}}(\pi_k^c \| \pi_k) - \widetilde{D}_{\mathrm{KL}}(\pi_k^c \| \pi_{k+1})] \leq H \log A + P_T \log \frac{1}{\delta}$ , showing that we need to control the minimal value of  $\delta$  to obtain a favorable dynamic regret bound. To this end, we slightly modify the update rule in (6) and add a uniform distribution  $\pi^u(\cdot \mid s) = \frac{1}{A} \mathbf{1}, \forall s \in S$ . That is, the policy  $\pi^u$  chooses each action with equal probability at any state. Thus, the update rule in (6) is modified as:
$$
\pi_ {k, h} ^ {\prime} (\cdot \mid s) \propto \pi_ {k - 1, h} (\cdot \mid s) \exp \left(\eta \cdot Q _ {k - 1, h} ^ {\pi_ {k - 1}} (s, \cdot)\right), \pi_ {k, h} (\cdot \mid s) = (1 - \gamma) \pi_ {k, h} ^ {\prime} (\cdot \mid s) + \gamma \pi^ {u} (\cdot \mid s) \tag {8}
$$
for any  $s \in S, h \in [H]$ , where  $\gamma \geq 0$  is the exploration parameter. This update is called the fixed-share mechanism in online learning literature [37]. While the fixed-share mechanism is standard to obtain dynamic regret in modern online learning [38], several important new challenges arise in online MDPs due to taking expectations over the policy sequence of changing policies  $\pi_1^c, \ldots, \pi_K^c$ . In particular, we prove that performing (8) ensures the following dynamic regret.
Lemma 2. Set  $\pi_1$  as uniform distribution on  $\mathcal{A}$  for any state  $s\in S$ . The update rule in (8) ensures
$$
\begin{array}{l} \mathrm {D} - \operatorname {R e g r e t} (K) \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(P _ {T} \log \frac {A}{\gamma} + K H \log \frac {1}{1 - \gamma}\right) \\ + \frac {1}{\eta} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \right] \tag {9} \\ \end{array}
$$
The proof can be found in Appendix C.2. In the dynamic regret analysis in online learning, the last term in (9) is usually canceled out through telescoping since we do not need to take expectations [38].
However, this is not the case in online MDPs. Since the expectation is taken over the policy sequence of changing policies  $\pi_1^c,\ldots ,\pi_K^c$  , this term cannot be canceled out, which requires a more refined analysis. To address this issue, we decompose one step of the expectation in (9) as follows.
$$
\left(\mathbb {E} _ {\pi_ {k - 1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k - 1, h} ^ {c} \log \frac {1}{\pi_ {k , h} ^ {\prime}} \right] - \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k, h} ^ {c} \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime}} \right]\right) + \mathbb {E} _ {\pi_ {k} ^ {c} - \pi_ {k - 1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k - 1, h} ^ {c} \log \frac {1}{\pi_ {k , h} ^ {\prime}} \right].
$$
With this decomposition, the first term can be canceled out through telescoping, yet it remains to control the second term — the weighted difference between the state-action occupancy measures of policy  $\pi_k^c$  and  $\pi_{k - 1}^c$  with weight  $-\pi_{k - 1,h}^{c}(a|s_{h})\log \pi_{k,h}^{\prime}(a|s_{h})$  for state-action  $(s_h,a)$ . To control it, we need to (i) ensure the weight is upper bounded by some universal constant, and (ii) bound the unweighted difference between the state-action occupancy measures, which are new challenges that arose in online MDPs compared with standard online learning.
For the first challenge, note that the weight  $-\pi_{k-1,h}^{c}(a|s_{h})\log\pi_{k,h}^{\prime}(a|s_{h})$  can be large or even infinite since  $\pi_{k,h}^{\prime}$  is the policy before uniform exploration and  $\pi_{k,h}^{\prime}(a|s_{h})$  can be arbitrarily small. Fortunately,  $\pi_{k,h}^{\prime}$  is obtained by one-step descent from  $\pi_{k-1,h}$ , which is the policy after uniform exploration and can be lower bounded. We provide the following multiplicative stability lemma for the one-step update, which shows  $\pi_{k,h}^{\prime}$  is not far from  $\pi_{k-1,h}$  and thus is also lower bounded.
Lemma 3 (Multiplicative Stability). For any distributions  $p \in \Delta(\mathcal{A})$  with  $p(a) > 0$ , for all  $a \in A$  and any function  $Q: S \times \mathcal{A} \to [0, H]$ , it holds for  $p' \in \Delta(\mathcal{A})$  with  $p'(a) \propto p(a) \cdot \exp(\eta \cdot Q(s, a))$  and  $\eta \leq 1 / H$  that  $p'(a) \in [p(a) / 4, 4p(a)]$ , for all  $a \in \mathcal{A}$ .
For the second challenge, we show the unweighted difference between the state-action occupancy measures can be bounded by the path length of policies. In particular, we have the following lemma.
Lemma 4. For any policy sequence  $\pi_1^c,\ldots ,\pi_K^c$  , it holds that
$$
\sum_ {k = 1} ^ {K} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} - \mathbb {E} _ {\pi_ {k - 1} ^ {c}}\right) \left[ \sum_ {h = 1} ^ {H} \mathbb {1} (s _ {h}) | s _ {1} = s _ {k, 1} \right] \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {h} \| \pi_ {k, i} ^ {c} - \pi_ {k - 1, i} ^ {c} \| _ {1, \infty} = H P _ {T}.
$$
Remark 1. We note that a similar argument is also used in Fei et al. [35, Appendix B.2.2]. However, they prove this lemma by imposing an additional smooth visitation measures assumption [35, Assumption 1], which is not required in our analysis.
The proofs for Lemmas 3 and 4 can be found in Appendices C.3 and C.4 respectively. Combining Lemmas 2, 3 and 4, we can prove the guarantee for update rule (8). The proof is in Appendix C.5.
Theorem 1. Set  $\pi_1$  as uniform distribution on  $\mathcal{A}$  for any state  $s\in S$ . The update rule in (8) ensures
$$
\mathrm {D - R e g r e t} (K) \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(H \log A + (1 + H) \log \frac {4 A}{\gamma} P _ {T} + K H \log \frac {1}{1 - \gamma}\right).
$$
Remark 2. Considering the static regret where  $\pi_1^c = \ldots = \pi_K^c = \pi^*$ , we can recover the  $\mathcal{O}(\sqrt{H^4K\log A})$  static regret in Cai et al. [32] under the stationary scenario by setting  $\gamma = 0$ , that is, without uniform exploration. However, when  $\gamma = 0$ , the dynamic regret is not bounded as there lacks an upper bound for  $-\log \gamma$ , showing the necessity of the fixed-share mechanism.
4.2 Policy Evaluation Phase
Sections 4.1 focus on the simplified scenario where the transition is known. In this subsection, we further consider the unknown transition setting such that it is necessary to evaluate the policy  $\pi_{k}$  based on the  $(k - 1)$  historical trajectories. To see how the model estimation error enters the dynamic regret, we decompose the dynamic regret in the following lemma.
Lemma 5 (Fei et al. [35, Lemma 1]). Define the model prediction error as  $\iota_{k,h} = r_{k,h} + \mathbb{P}_h V_{k,h+1} - Q_{k,h}$ , the dynamic regret D-Regret  $(K) = \sum_{k=1}^{K} V_{k,1}^{\pi_k^c}(s_{k,1}) - \sum_{k=1}^{K} V_{k,1}^{\pi_k}(s_{k,1})$  can be written as
$$
\sum_ {k, h} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle \right] + \mathcal {M} _ {K, H} + \sum_ {k, h} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} [ \iota_ {k, h} (s _ {h}, a _ {h}) ] - \iota_ {k, h} (s _ {k, h}, a _ {k, h})\right),
$$
where  $\mathcal{M}_{K,H} = \sum_{k=1}^{K}\sum_{h=1}^{H}M_{k,h}$  is a martingale that satisfies  $M_{k,h} \leq 4H$ ,  $\forall k \in [k], h \in [H]$ .
<sup>1</sup>With a slight abuse of notations, we omit  $(\cdot \mid s_h)$  for simplicity.
Remark 3. Lemma 5 is independent of the structure of MDPs. The first term in Lemma 5 is the dynamic regret over the estimated action-value function  $Q_{k,h}$ , which can be upper bounded by Theorem 5. The second term is a martingale, which can be bounded by Azuma-Hoeffding inequality. The third term is the model estimation error, which is the main focus of this section. Note the model prediction error  $\iota_{k,h}(s_h,a_h)$  can be large for the state-action pairs that are less visited or even unseen. The general approach is incorporating the bonus function into the estimated  $Q$ -function such that  $\iota_{k,h}(s_h,a_h) \leq 0$  for all  $s \in S, a \in \mathcal{A}$  (i.e.,  $\mathbb{E}_{\pi_k^c}[\iota_{k,h}(s_h,a_h)] \leq 0$ ) and we only need to control  $-\iota_{k,h}(s_{k,h},a_{k,h})$ , which is the model estimation error at the visited state-action pair  $(s_{k,h},a_{k,h})$ .
When applied to linear mixture MDPs, the key idea is learning the unknown parameter  $\theta_h^*$  of the linear mixture MDP and using the learned parameter  $\theta_{k,h}$  to build an optimistic estimator  $Q_{k,h}(\cdot ,\cdot)$  such that the model prediction error is non-positive, which is more or less standard. From the definition of linear mixture MDP, for the learned value function  $V_{k,h}(\cdot)$ , we have  $[\mathbb{P}_hV_{k,h + 1}](s,a) = \langle \sum_{s'}\phi (s'|s,a)V_{k,h + 1}(s'),\theta_h^*\rangle = \langle \phi_{V_{k,h + 1}}(s,a),\theta_h^*\rangle$ . Inspired by recent advances in policy evaluation for linear mixture MDPs [31], we adopt the weighted ridge regression to estimate the parameter  $\theta_h^*$ , that is, we construct the estimator  $\widehat{\theta}_{k,h}$  by solving the following weighted ridge regression problem:
$$
\widehat {\theta} _ {k, h} = \underset {\theta \in \mathbb {R} ^ {d}} {\arg \min } \sum_ {j = 1} ^ {k - 1} \left[ \left\langle \phi_ {V _ {j, h + 1}} (s _ {j, h}, a _ {j, h}), \theta \right\rangle - V _ {j, h + 1} (s _ {j, h + 1}) \right] ^ {2} / \bar {\sigma} _ {j, h} ^ {2} + \lambda \| \theta \| _ {2} ^ {2}.
$$
Here,  $\bar{\sigma}_{j,h}^{2}$  is the upper confidence bound of the variance  $[\mathbb{V}_hV_{j,h + 1}](s_{j,h},a_{j,h})$ , and we set it as  $\bar{\sigma}_{k,h} = \sqrt{\max\{H^2 / d,[\bar{\mathbb{V}}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h}) + E_{k,h}\}}$ , where  $[\bar{\mathbb{V}}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h})$  is a scalar-valued empirical estimate for the variance of the value function  $V_{k,h + 1}$  under the transition probability  $\mathbb{P}_h(\cdot \mid s_k,a_k)$ , and  $E_{k,h}$  is the bonus term to guarantee that the true variance  $[\mathbb{V}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h})$  is upper bounded by  $[\bar{\mathbb{V}}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h}) + E_{k,h}$  with high probability. Then, the confidence set  $\widehat{C}_{k,h}$  is constructed as follows:
$$
\widehat {\mathcal {C}} _ {k, h} = \left\{\theta \mid \| \widehat {\Sigma} _ {k, h} ^ {1 / 2} \left(\theta - \widehat {\theta} _ {k, h}\right) \| _ {2} \leq \widehat {\beta} _ {k} \right\}. \tag {10}
$$
where  $\widehat{\Sigma}_{k,h}$  is a covariance matrix based on the observed data, and  $\widehat{\beta}_k$  is a radius of the confidence set. Given  $\widehat{C}_{k,h}$ , we estimate the  $Q$ -function following the principle of "optimism in the face of uncertainty" [51] and set it as  $Q_{k,h}(\cdot ,\cdot) = [r_{k,h}(\cdot ,\cdot) + \max_{\theta \in \widehat{C}_{k,h}}\langle \theta ,\phi_{V_{k,h + 1}}(\cdot ,\cdot)\rangle ]_{[0,H - h + 1]}$ .
It remains to estimate the variance  $[\mathbb{V}_hV_{k,h + 1}](s_{k,h},a_{k,h})$ . By the definition of linear mixture MDPs, we have  $[\mathbb{V}_hV_{k,h + 1}](s_{k,h},a_{k,h}) = \langle \phi_{V_{k,h + 1}^2}(s_{k,h},a_{k,h}),\theta_h^*\rangle -[\langle \phi_{V_{k,h + 1}}(s_{k,h},a_{k,h}),\theta_h^*\rangle ]^2$ . Therefore, we estimate  $[\bar{\mathbb{V}}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h})$  by the expression below
$$
\left[ \left\langle \phi_ {V _ {k, h + 1} ^ {2}} \left(s _ {k, h}, a _ {k, h}\right), \widetilde {\theta} _ {k, h} \right\rangle \right] _ {[ 0, H ^ {2} ]} - \left[ \left\langle \phi_ {V _ {k, h + 1}} \left(s _ {k, h}, a _ {k, h}\right), \widehat {\theta} _ {k, h} \right\rangle \right] _ {[ 0, H ]} ^ {2}, \tag {11}
$$
where  $\widetilde{\theta}_{k,h} = \arg \min_{\theta \in \mathbb{R}^d}\sum_{j = 1}^{k - 1}[\langle \phi_{V_{j,h + 1}^2}(s_{j,h},a_{j,h}),\theta \rangle -V_{j,h + 1}^2 (s_{j,h + 1})]^2 +\lambda \| \theta \| _2^2$  . The details are summarized in Lines 10-20 of Algorithm 1 and we provide the following guarantee.
Theorem 2. Set the parameters as in Lemma 8, with probability at least  $1 - \delta$ , we have
$$
\sum_ {k = 1} ^ {K} \left(V _ {k, 1} \left(s _ {1} ^ {k}\right) - V _ {k, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right)\right) = \mathcal {M} _ {K, H} - \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K}\right).
$$
The proof is given in Appendix C.6. Theorem 2 shows the model estimation error can be bounded. Combining Theorems 1, 2 and Lemma 5, we present the dynamic regret bound in the next section.
4.3 Regret Guarantee: Upper and Lower Bounds
In this section, we provide the regret bound for our algorithm and present a lower bound of the dynamic regret for any algorithm for adversarial linear mixture MDPs with the unknown transition.
Theorem 3. Set  $\eta = \min \{\sqrt{(P_T + \log A) / K}, 1\} / H, \gamma = 1 / (KH)$  and  $\widehat{\beta}_k$  as in Lemma 8, then with probability at least  $1 - \delta$ , it holds
$$
\operatorname {D} - \operatorname {R e g r e t} (K) \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K} + \sqrt {H ^ {4} \left(K + P _ {T}\right) \left(1 + P _ {T}\right)}\right), \tag {12}
$$
where  $P_{T} = \sum_{k = 1}^{K}\sum_{h = 1}^{H}\| \pi_{k,h}^{c} - \pi_{k - 1,h}^{c}\|_{1,\infty}$  is the path length of the compared policies.
Remark 4 (recovering static regret). Since static regret is a special case with  $\pi_k^c = \pi^*,\forall k$  , our result can recover the optimal  $\widetilde{\mathcal{O}} (\sqrt{d^2H^3K})$  static regret when  $H\leq d$  , same as the result in He et al. [33].
Remark 5 (improving linear mixture case). Our result improves upon the previously best-known  $\tilde{\mathcal{O}}(dH^{7/4}K^{3/4} + H^2K^{2/3}P_T^{1/3})$  dynamic regret for adversarial linear mixture MDPs of Zhong et al. [36, Theorem 4.6] in terms of the dependence on  $H$ ,  $K$  and  $P_T$ .
Remark 6 (improving tabular case). For the adversarial tabular MDPs, our result implies an  $\widetilde{\mathcal{O}} (\sqrt{H^4S^2AK} +H^2\sqrt{(K + P_T)(1 + P_T)})$  dynamic regret. This improves upon the best-known  $\widetilde{\mathcal{O}} (\sqrt{H^4S^2AK} +H^2 K^{2 / 3}P_T^{1 / 3})$  result of Fei et al. [35, Theorem 1]. The details are in Appendix B.
We finally establish the lower bound of this problem. The proof can be found in Appendix C.8.
Theorem 4. Suppose  $B \geq 2, d \geq 4, H \geq 3, K \geq (d - 1)^{2}H / 2$ , for any algorithm and any constant  $\Gamma \in [0,2KH]$ , there exists an episodic  $B$ -bounded adversarial linear mixture MDP and compared policies  $\pi_1^c, \ldots, \pi_K^c$  such that  $P_T \leq \Gamma$ , and D-Regret  $(K) \geq \Omega(\sqrt{d^2H^3K} + \sqrt{HK(H + \Gamma)})$ .
When  $H \leq d$ , the upper bound is  $\widetilde{\mathcal{O}}(\sqrt{d^2H^3K} + \sqrt{H^4(K + P_T)(1 + P_T)})$ . Combining it with Theorem 4, we discuss the optimality of our result. We consider the following three regimes.
- Small  $P_T$ : when  $0 \leq P_T \leq d^2 / H$ , the upper bound (12) can be simplified as  $\widetilde{\mathcal{O}}(\sqrt{d^2 H^3 K})$ , and the lower bound is  $\widetilde{\mathcal{O}}(\sqrt{d^2 H^3 K})$ , hence our result is optimal in terms of  $d, H$  and  $K$ .
- Moderate  $P_T$ : when  $d^2 / H \leq P_T \leq K$ , the upper bound (12) can be simplified as  $\widetilde{\mathcal{O}}(\sqrt{d^2 H^3 K} + \sqrt{H^4 K (1 + P_T)})$ , and it is minimax optimal in  $d, K$  and  $P_T$  but looses a factor of  $H^{\frac{3}{2}}$ .
- Large  $P_T$ : when  $P_T \geq K$ , any algorithm suffers at most  $\mathcal{O}(HK)$  dynamic regret, while the lower bound is  $\Omega(K\sqrt{H})$ . So our result is minimax optimal in  $K$  but looses a factor of  $\sqrt{H}$ .
5 Towards Optimal Dynamic Regret with Unknown  $P_{T}$
This section further considers the case when the non-stationarity measure  $P_T$  is unknown. By Theorem 1, we need to tune the step size  $\eta$  optimally to balance the number of episodes  $K$  and  $P_T$  to achieve a favorable dynamic regret. To address the difficulty of not knowing  $P_T$  ahead of time, we develop an online ensemble method to handle this uncertainty, in which a two-layer meta-base structure is maintained. While the methodology can be standard in recent non-stationary online learning [52, 39, 40, 41], new challenges arise in online MDPs. We introduce the details below.
By the performance difference lemma in Cai et al. [32, Lemma 3.2] (as restate in Lemma 13), we can rewrite the dynamic regret as
$$
\sum_ {k = 1} ^ {K} \left[ V _ {k, 1} ^ {\pi_ {k} ^ {c}} (s _ {1} ^ {k}) - V _ {k, 1} ^ {\pi_ {k}} (s _ {1} ^ {k}) \right] = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {h = 1} ^ {H} \left\langle Q _ {k, h} ^ {\pi_ {k}} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \right\rangle \right],
$$
where the expectation is taken over the randomness of the state trajectory sampled according to  $\pi_k^c$ . The dynamic regret of online MDPs can be written as the weighted average of some "multi-armed bandits" problems over all states, where the weight for each state is the unknown and changing probability of being visited by  $\pi_1^c,\ldots ,\pi_K^c$ . As the optimal step size depends on the unknown nonstationarity measure  $P_{T}$  as shown in Section 4, a natural idea is to the two-layer structure to learn the optimal step size as in recent online convex optimization literature [52, 39, 40, 41].
The general idea is constructing a step size pool  $\mathcal{H} = \{\eta_1, \dots, \eta_N\}$  to discretize the value range of the optimal step size; and then maintaining multiple base-learners  $\mathcal{B}_1, \dots, \mathcal{B}_N$ , each of which works with a specific step size  $\eta_i$ . Finally, a meta-algorithm is used to track the best base-learner and yield the final policy. Then, the dynamic regret can be decomposed as follows (omit  $(\cdot \mid s_h)$  for simplicity):
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {h = 1} ^ {H} \left\langle Q _ {k, h} ^ {\pi_ {k}}, \pi_ {k, h} ^ {c} - \pi_ {k, h} ^ {i} \right\rangle \right] + \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {h = 1} ^ {H} \left\langle Q _ {k, h} ^ {\pi_ {k}}, \pi_ {k, h} ^ {i} - \pi_ {k, h} \right\rangle \right].
$$
Since the above decomposition holds for any index  $i \in [N]$ , we can always choose the base-learner with optimal step size to analyze and the first term is easy to control. The challenge is to control the
second term, which is the regret of the meta-algorithm. Different from the standard "Prediction with Expert Advice" problem, it involves an additional expectation over the randomness of states sampled according to  $\pi_k^c$ . This poses a grand challenge compared to conventional online convex optimization where the expectation is not required. Although we can bound this term by  $P_T$  again, optimal tuning of the meta-algorithm is hindered as  $P_T$  is unknown. Consequently, we opt to upper bound it by the worst-case dynamic regret [53], that is, benchmarking the performance with the best choice of each round, which in turn introduces the dependence on the switching number of the best base-learner.
We introduce our approach as follows. We maintain multiple base-learners, each of which works with a specific step size  $\eta_{i}$ . All base-learners update their policies according to the same action-value function  $Q_{k - 1,h}(s_h,\cdot)$  of the combined policy  $\pi_{k - 1}$ , that is, the base-learner  $\mathcal{B}_i$  updates policy by
$$
\pi_ {k, h} ^ {i, \prime} (\cdot | s) \propto \pi_ {k - 1, h} ^ {i} (\cdot | s) \exp \left(\eta_ {i} Q _ {k - 1, h} (s, \cdot)\right), \pi_ {k, h} ^ {i} (\cdot | s) = (1 - \gamma) \pi_ {k, h} ^ {i, \prime} (\cdot | s) + \gamma \pi^ {u} (\cdot | s), \tag {13}
$$