SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Long-Fei Li, Peng Zhao, Zhi-Hua Zhou
{lilf, zhaop,zhouzh}@lamda.nju.edu.cn
We study reinforcement learning in episodic inhomogeneous MDPs with adversarial full-information rewards and the unknown transition kernel. We consider the linear mixture MDPs whose transition kernel is a linear mixture model and choose the dynamic regret as the performance measure. Denote by  $d$  the dimension of the feature mapping,  $H$  the length of each episode,  $K$  the number of episodes,  $P_T$  the non-stationary measure, we propose a novel algorithm that enjoys an  $\widetilde{\mathcal{O}}\big(\sqrt{d^2H^3K} +\sqrt{H^4(K + P_T)(1 + P_T)}\big)$  dynamic regret under the condition that  $P_T$  is known, which improves previously best-known dynamic regret for adversarial linear mixture MDP and adversarial tabular MDPs. We also establish an  $\Omega \big(\sqrt{d^2H^3K} +\sqrt{HK(H + P_T)}\big)$  lower bound, indicating our algorithm is optimal in  $K$  and  $P_T$ . Furthermore, when the non-stationary measure  $P_T$  is unknown, we design an online ensemble algorithm with a meta-base structure, which is proved to achieve an  $\widetilde{\mathcal{O}}\big(\sqrt{d^2H^3K} +\sqrt{H^4(K + P_T)(1 + P_T) + H^2S_T^2}\big)$  dynamic regret and here  $S_{T}$  is the expected switching number of the best base-learner. The result can be optimal under certain regimes.
Reinforcement Learning (RL) aims to learn a policy that maximizes the cumulative reward through interacting with the environment, which has achieved tremendous successes in various fields, including games [1, 2], robotic control [3, 4], and dialogue generation [5, 6]. In reinforcement learning, the Markov Decision Process (MDP) [7] is the most widely used model to describe the environment.
Traditional MDPs assume that the reward functions are stochastic and the number of actions and states is small. However, in many real-world applications, the reward functions may be adversarially changing and the state and action spaces are large or even infinite. Previous work studies these two problems separately. To deal with the adversarial reward functions, Even-Dar et al. [8] first consider learning adversarial MDPs with known transition and full-information reward feedback.
They propose the MDP-E algorithm that enjoys  $\tilde{\mathcal{O}} (\sqrt{\tau^3T})$  regret, where  $\tau$  is the mixing time and  $T$  is the number of steps. There is a line of subsequent work studying adversarial MDPs [9, 10, 11, 12, 13, 14, 15, 16], which studies various settings depending on whether the transition kernel is known, and whether the feedback is full-information or bandit. To overcome the large state and action space issue, a widely used method is linear function approximation, which reparameterizes the value function as a linear function over some feature mapping that maps the state and action to a low-dimensional space. Amongst these work, linear MDPs [17, 18, 19, 20, 21, 22, 23] and linear mixture MDPs [24, 25, 26, 27, 28, 29, 30, 31] are two of the most popular MDP models with linear function approximation. In particular, He et al. [23] and Zhou et al. [31] attain the minimax optimal  $\tilde{\mathcal{O}} (\sqrt{d^2H^3K})$  regret for linear MDPs and linear mixture MDPs with stochastic rewards respectively.
Recent studies try to combine two lines of work to establish the theoretical foundation of adversarial MDPs with large state and action space. In particular, Cai et al. [32] study adversarial linear mixture
MDPs and propose the OPPO algorithm that enjoys an  $\widetilde{\mathcal{O}} (\sqrt{d^2H^4K})$  regret. He et al. [33] improve the result to  $\widetilde{\mathcal{O}} (\sqrt{d^2H^3K})$  and show it is (nearly) optimal by presenting a matching lower bound. However, these studies choose static regret as the performance measure, defined as the performance difference between the learner's policy and that of the best-fixed policy in hindsight, namely,
$$
\operatorname {R e g r e t} (K) = \max  _ {\pi \in \Pi} \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right), \tag {1}
$$
where  $V$  is the value function and  $\Pi$  is the set of all stochastic policies. One caveat in (1) is that the best fixed policy may behave poorly in non-stationary environments. To this end, we introduce dynamic regret, which benchmarks the learner's performance with changing policies, defined as
$$
\operatorname {D} - \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right), \tag {2}
$$
where  $\pi_1^c, \ldots, \pi_K^c \in \Pi$  is compared policies. Define  $P_T = \sum_{k=2}^{K} d(\pi_k^c, \pi_{k-1}^c)$  with a certain distance measure  $d(\cdot, \cdot)$  as the non-stationary measure. A favorable dynamic regret should scale with  $P_T$ .
Dynamic regret is a more appropriate measure in non-stationary environments, but it is more challenging to optimize such that few studies focus on it in the literature. Zhao et al. [34] investigate the dynamic regret of adversarial tabular MDPs with the known transition kernel and present an algorithm with optimal dynamic regret. Importantly, their algorithm does not require the non-stationarity measure  $P_T$  as the algorithmic input ahead of time. For the unknown transition setting, Fei et al. [35] study adversarial tabular MDPs and propose an algorithm with dynamic regret guarantee. Zhong et al. [36] further extend the algorithm of Fei et al. [35] to accommodate non-stationary transition kernels with linear function approximation. Both algorithms of Fei et al. [35] and Zhong et al. [36] require the quantity of  $P_T$  as the input. Moreover, their dynamic regret bounds are suboptimal in terms of  $K$  and  $P_T$  as demonstrated by the lower bound established by our work (see Theorem 4).
This work investigates the dynamic regret of adversarial linear mixture MDPs, with a focus on the full-information feedback and the unknown transition. We first propose POWERS-FixShare algorithm when  $P_T$  is known, an algorithm combining optimistic policy optimization with a Bernstein bonus and fixed-share mechanism. We show it enjoys an  $\tilde{\mathcal{O}}(\sqrt{d^2H^3K} + \sqrt{H^4(K + P_T)(1 + P_T)})$  dynamic regret, where  $d$  is the dimension of the feature mapping,  $H$  is the length of each episode,  $K$  is the number of episodes,  $P_T$  is the non-stationary measure. We also establish a dynamic regret lower bound of  $\Omega(\sqrt{d^2H^3K} + \sqrt{HK(H + P_T)})$ . We stress four remarks regarding our results:
(1) Our result can recover the  $\tilde{\mathcal{O}} (\sqrt{d^2H^3K})$  minimax optimal static regret in He et al. [33].
(2) Our result improves upon the previously best-known  $\widetilde{\mathcal{O}} (dH^{7 / 4}K^{3 / 4} + H^2 K^{2 / 3}P_T^{1 / 3})$  dynamic regret for the fixed transition of Zhong et al. [36, Theorem 4.6] in terms of  $H$ ,  $K$  and  $P_T$ .
(3) Our result can imply an  $\widetilde{\mathcal{O}} (\sqrt{H^4S^2AK} +H^2\sqrt{(K + P_T)(1 + P_T)})$  dynamic regret for adversarial tabular MDPs, strictly improving upon the previously best-known  $\widetilde{\mathcal{O}} (\sqrt{H^4S^2AK} + H^2 K^{2 / 3}P_T^{1 / 3})$  dynamic regret of Fei et al. [35, Theorem 1], in terms of both  $K$  and  $P_T$ .
(4) As the lower bound suggests, our result is the first optimal regarding the dependence on  $d$ ,  $K$  and  $P_T$  and can be optimal in terms of  $H$  under certain regimes ( $H \leq d$  and  $P_T \leq d^2 / H$ ).
Furthermore, we study the case when  $P_T$  is unknown and design a novel algorithm equipped with the dynamic regret guarantee by the meta-base two-layer structure. Our algorithm enjoys an  $\widetilde{\mathcal{O}}(\sqrt{d^2H^3K} + \sqrt{H^4(K + P_T)(1 + P_T) + H^2S_T^2})$  dynamic regret, where  $S_T$  is the expected switching number of the best base-learner. Though  $S_T$  is a data-dependent quantity, it also reflects the degree of environmental non-stationarity to some extent. Moreover, under specific regimes, the magnitude of  $S_T$  may be relatively negligible, resulting in our results still being optimal. Indeed, given that  $S_T$  is a data-dependent quantity, its inclusion in the regret bound is not ideal. Deriving bounds that rely exclusively on problem-dependent quantities, like  $P_T$ , remains an open challenge. We discuss the technical difficulty of removing  $S_T$  in Section 5 and take this issue for future work.
Finally, we also highlight the main technical challenges and our solutions as follows.
- We first show that the dynamic regret depends on the inverse of the minimal probability over the action space of our policies, which can be arbitrarily small. To this end, we propose a novel
algorithm with the fixed-share mechanism [37]. While this mechanism is proved to enjoy favorable dynamic regret in online convex optimization [38], it suffers an additional term that can be regarded as the weighted sum of the difference between the occupancy measure of the compared policies in online MDPs. To overcome the difficulty, we exploit the multiplicative stability to bound this term, eliminating the need for a restart strategy to handle the environmental non-stationarity as in previous studies [35, 36] and allows us to attain the dynamic regret optimal in terms of  $K$  and  $P_T$ .
- We show the dynamic regret of online MDPs can be written as the weighted average of "multiarmed bandits" problems over all states, where the weight for each state is the unknown and changing probability of being visited by  $\pi_1^c, \ldots, \pi_K^c$ . For the unknown  $P_T$  case, we first show the standard two-layer structure used in non-stationary online learning studies [39, 40, 41] fails to achieve a favorable dynamic regret guarantee, which characterizes the unique difficulty of online MDPs. Then, we present an initial attempt to address this issue by a specifically designed two-layer structure. We prove our algorithm enjoys nice dynamic regret guarantees under certain regimes.
Notations. We denote by  $\Delta(\mathcal{A})$  the set of probability distributions on a set  $\mathcal{A}$  and denote the KL-divergence by  $D_{\mathrm{KL}}(p \| p') = \sum_{a \in \mathcal{A}} p(a) \log \frac{p(a)}{p'(a)}$  for any  $p, p' \in \Delta(\mathcal{A})$ . We define  $\Delta(\mathcal{A} | \mathcal{S}, H) = \{\{\pi_h(\cdot | \cdot)\}_{h=1}^H | \pi_h(\cdot | x) \in \Delta(\mathcal{A}), \forall s \in \mathcal{S}, h \in [H]\}$  for any set  $\mathcal{S}$  and  $H \in \mathbb{Z}_+$ . Further, for any  $\pi, \pi', \pi'' \in \Delta(\mathcal{A} | \mathcal{S}, H)$ , we define  $\mathbb{E}_{\pi}[\widetilde{D}_{\mathrm{KL}}(\pi' \| \pi'')] = \mathbb{E}_{\pi}[\sum_{h=1}^{H} D_{\mathrm{KL}}(\pi_h' (\cdot | s_h) \| \pi_h'' (\cdot | s_h))]$ . For any policy pair  $\pi_h, \pi_h'$ , we define  $\| \pi_h - \pi_h' \|_{1,\infty} = \max_{s \in \mathcal{S}} \| \pi_h(\cdot | s) - \pi_h' (\cdot | s) \|_1$ . For any  $a, b, x \in \mathbb{R}$  with  $a \leq b$ , let  $[x]_{[a,b]}$  denote min $\{\max(x, a), b\}$ .  $\widetilde{\mathcal{O}}(\cdot)$  omits the logarithmic factors.
RL with adversarial rewards. There are many studies on learning adversarial MDPs where the reward functions are adversarially chosen, yielding fruitful results that can be categorized into three lines [8, 9, 10, 11, 12, 13, 14, 15, 16]. In particular, the first line of work considers the infinite-horizon MDPs with uniform mixing assumption. In the known transition and full-information setting, the seminal work of Even-Dar et al. [8] proposes MDP-E algorithm that achieves the  $\widetilde{\mathcal{O}} (\sqrt{\tau^3T})$  regret, where  $\tau$  is the mixing time and  $T$  is the number of steps. Another concurrent work of Yu et al. [9] achieves  $\widetilde{\mathcal{O}} (T^{2 / 3})$  in the same setting. In the known transition and bandit-feedback setting, Neu et al. [10] propose MDP-EXP3 algorithm that attains  $\widetilde{\mathcal{O}} (T^{2 / 3})$  regret. The second line of work considers the episodic loop-free MDPs. Neu et al. [11] first study this problem under the known transition setting and propose algorithms that achieve  $\widetilde{\mathcal{O}} (\sqrt{T})$  and  $\widetilde{\mathcal{O}} (\sqrt{T / \alpha})$  for full-information and bandit-feedback respectively, where  $\alpha$  is the lower bound of the probability of all states under any policy. Zimin and Neu [12] propose O-REPS algorithm that enjoys  $\widetilde{\mathcal{O}} (\sqrt{T})$  regret in both full-information and bandit-feedback setting without any additional assumption. Rosenberg and Mansour [13] and Jin et al. [14] further consider the harder unknown transition and bandit-feedback setting. The last line of work studies the episodic Stochastic Shortest Path (SSP) setting [15, 16]. In this paper, we focus on episodic MDPs with the unknown transition and full-information setting.
RL with linear function approximation. To design RL algorithms in large state and action space scenarios, recent works focus on solving MDPs with linear function approximation. In general, these works can be divided into three lines based on the specific assumption of the underlying MDP. The first line of work considers the low Bellman-rank assumption [42, 43, 44, 45], which assumes the Bellman error matrix has a low-rank factorization. The second line of work is based on the linear MDP assumption [17, 18, 19, 20, 21, 22, 23], where both the transition kernel and reward functions can be parameterized as linear functions of given state-action feature mappings  $\phi : S \times \mathcal{A} \to \mathbb{R}^d$ . The last line of work studies the linear mixture MDP [24, 25, 26, 28, 29, 30, 31], where the transition kernel can be parameterized as a linear function of a feature mapping  $\phi : S \times \mathcal{A} \times S \to \mathbb{R}^d$  but without the linear reward functions assumption. Amongst these works, He et al. [23] and Zhou et al. [31] attain the minimax optimal  $\widetilde{\mathcal{O}}(\sqrt{d^2 H^3 K})$  regret for both episodic linear MDPs and linear mixture MDPs respectively. However, all the above studies consider the stochastic reward setting. In this paper, we study the episodic linear mixture MDP setting but with adversarial reward functions.
Non-stationary RL. Another related line of research is online non-stationary MDPs. In contrast to adversarial MDPs where the reward functions are generated in an adversarial manner, online non-stationary MDPs consider the setting where the reward functions are generated stochastically
according to some distributions that may vary over time. Jaksch et al. [46] study the piecewise stationary setting where the transitions and rewards are allowed to change certain times and propose UCRL2 with restart technique to deal with the non-stationarity. Later, Ortner et al. [47] consider the generalized setting where the changes are allowed to take place every step. However, the above studies need prior knowledge about the magnitude of non-stationary. To address this issue, Cheung et al. [48] propose the Bandit-over-RL algorithm to remove this requirement. A recent breakthrough by Wei and Luo [49] introduces a black-box method that can convert any algorithm satisfying specific conditions and enjoying optimal static regret in stationary environments into another with optimal dynamic regret in non-stationary environments, without requiring prior knowledge about the degree of non-stationarity. However, this approach does not apply to the adversarial setting. Specifically, their reduction requires the base algorithm to satisfy a certain property enjoyed by typical UCB-type algorithms. When a new instance of the base algorithm surpasses the optimistic estimator, it can be inferred that the environment has changed, prompting a restart of the algorithm to disregard prior information. However, this approach of constructing an optimistic estimator by a UCB-type algorithm can only be applied to a stochastic setting. In the adversarial setting, where no model assumptions are made and comparators can be arbitrary, this approach encounters significant difficulties.
Dynamic Regret. Dynamic regret of RL with adversarial rewards is only recently studied in the literature [34, 35, 36]. Zhao et al. [34] investigate the dynamic regret of adversarial tabular MDPs with the known transition kernel and present an algorithm with optimal dynamic regret. Importantly, their algorithm does not require the non-stationarity measure  $P_T$  as the algorithmic input ahead of time. For the unknown transition setting, Fei et al. [35] study adversarial tabular MDPs and propose an algorithm with dynamic regret guarantees. Zhong et al. [36] further extend the algorithm of Fei et al. [35] to accommodate non-stationary transition kernels with linear function approximation. Both algorithms [35, 36] require the quantity of  $P_T$  as the input. Moreover, their dynamic regret bounds are suboptimal in  $K$  and  $P_T$  as shown by the lower bound established in our work. In this work, we first design an optimal algorithm in terms of  $K$  and  $P_T$  when  $P_T$  is known. Further, we develop the first algorithm to handle the unknown  $P_T$  issue in adversarial MDPs with unknown transition.
We focus on episodic inhomogeneous MDPs with full-information reward functions and the unknown transition kernel. Denote by  $M = \{\mathcal{S},\mathcal{A},H,\{r_{k,h}\}_{k\in [K],h\in [H]},\{\mathbb{P}_h\}_{h\in [H]}\}$  an episodic inhomogeneous MDP, where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $K$  is the number of episodes,  $H$  is the horizon,  $r_{k,h}:S\times \mathcal{A}\to [0,1]$  is the reward function,  $\mathbb{P}_h(\cdot |\cdot ,\cdot):S\times \mathcal{A}\times S\rightarrow [0,1]$  is the transition kernel. We assume the rewards are deterministic without loss of generality and extending our results to stochastic rewards is straightforward. Let  $S = |\mathcal{S}|$  and  $A = |\mathcal{A}|$ .
The learner interacts with the MDP through  $K$  episodes without knowledge of transition kernel  $\{\mathbb{P}_h\}_{h \in [H]}$ . In each episode  $k$ , the environment chooses the reward function  $\{r_{k,h}\}_{h \in [H]}$  and decides the initial state  $s_{k,1}$ , where the reward function may be chosen in an adversarial manner and depend on the history of the past  $(k - 1)$  episodes. Simultaneously, the learner decides a policy  $\pi_k = \{\pi_{k,h}\}_{h \in [H]}$  where each  $\pi_{k,h}: S \to \Delta(\mathcal{A})$  is a function that maps a state  $s$  to a distribution over action space  $\mathcal{A}$ . In the  $h$  stage in episode  $k$ , the learner observes current state  $s_{k,h}$ , chooses an action  $a_{k,h} \sim \pi_{k,h}(\cdot | s_{k,h})$ , and transits to the next state  $s_{k,h + 1} \sim \mathbb{P}_h(\cdot | s_{k,h}, a_{k,h})$ . Then the learner obtains the reward  $r_{k,h}(s_{k,h}, a_{k,h})$  and observes the reward function  $r_{k,h}$  as we consider the full-information setting. At stage  $H + 1$ , the learner observes the final state  $s_{k,H + 1}$  but does not take any action, and the episode  $k$  terminates. Denote by  $T = KH$  the total steps throughout  $K$  episodes.
Linear Mixture MDPs. In this work, we focus on a special class of MDPs called linear mixture MDPs, a setting initiated by Ayoub et al. [24] and further studied in the subsequent work [32, 31, 33]. In this setup, the transition kernel can be parameterized as a linear function of a feature mapping  $\phi : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}^d$ . The formal definition of linear mixture MDPs is as follows.
Definition 1 (Linear Mixture MDPs). An MDP  $M = \{\mathcal{S},\mathcal{A},H,\{r_{k,h}\}_{k\in [K],h\in [H]},\{\mathbb{P}_h\}_{h\in [H]}\}$  is called an inhomogeneous, episode  $B$ -bounded linear mixture MDP, if there exist a known feature mapping  $\phi(s' | s,a): \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}^d$  and an unknown vector  $\theta_h^* \in \mathbb{R}^d$  with  $\| \theta_h^* \|_2 \leq B$ ,  $\forall h \in [H]$  such that (i)  $\mathbb{P}_h(s' | s,a) = \phi(s' | s,a)^\top \theta_h^*$  for all  $(s,a,s') \in \mathcal{S} \times \mathcal{A} \times \mathcal{S}$  and  $h \in [H]$ , (ii)  $\| \phi_V(s,a) \|_2 \triangleq \| \sum_{s' \in \mathcal{S}} \phi(s' | s,a)V(s') \|_2 \leq 1$  for any  $(s,a) \in \mathcal{S} \times \mathcal{A}$  and function  $V: \mathcal{S} \to [0,1]$ .
Algorithm 1 POWERS-FixShare
Input: step size  $\eta$ , exploration parameter  $\gamma$  and regularization parameter  $\lambda$ .
1: Initialize  $\{\pi_{0,h}(\cdot |s)\}_{h = 1}^{H}, s\in \mathcal{S}$  as uniform distribution, and set  $\{Q_{0,h}(\cdot ,\cdot)\}_{h = 1}^{H}$  as zero function.
2: for  $k = 1, 2, \dots, K$  do
3: Receive the initial state  $s_{k,1}$ .
4: for  $h = 1,2,\dots ,H$  do
5: For all  $h \in [H]$ ,  $s \in S$ , updates policy by
$$
\pi_ {k, h} ^ {\prime} (\cdot \mid s) \propto \pi_ {k - 1, h} (\cdot \mid s) \exp \left(\eta Q _ {k - 1, h} (s, \cdot)\right), \pi_ {k, h} (\cdot \mid s) = (1 - \gamma) \pi_ {k, h} ^ {\prime} (\cdot \mid s) + \gamma \pi^ {u} (\cdot \mid s).
$$
6: Take the action following  $a_{k,h} \sim \pi_{k,h}(\cdot \mid s_{k,h})$  and transit to the next state  $s_{k,h+1}$ .
7: Obtain reward  $r_{k,h}(s_{k,h},a_{k,h})$  and observe the reward function  $r_{k,h}(\cdot ,\cdot)$
8: end for
9: Initialize  $V_{k,H + 1}(\cdot)$  as a zero function.
0: for  $h = H, H - 1, \dots, 1$  do
1: Set  $Q_{k,h}(\cdot ,\cdot)\gets \left[r_{k,h}(\cdot ,\cdot) + \langle \widehat{\theta}_{k,h},\phi_{V_{k,h + 1}}(\cdot ,\cdot)\rangle +\widehat{\beta}_k\big|\big|\widehat{\Sigma}_{k,h}^{-1 / 2}\phi_{V_{k,h + 1}}(\cdot ,\cdot)\big|\big|_2\right]_{[0,H - h + 1]}$
2: Set  $V_{k,h}(\cdot) \gets \mathbb{E}_{a \sim \pi_{k,h}(\cdot \mid \cdot)}[Q_{k,h}(\cdot, \cdot)]$ .
3: Set the estimated variance  $\left[\mathbb{V}_{k,h} V_{k,h+1}\right] (s_{k,h}, a_{k,h})$  as in (11), bonus  $E_{k,h}$  as in (21).
4:  $\bar{\sigma}_{k,h} \gets \sqrt{\max\left\{H^2 / d, \left[\bar{\mathbb{V}}_{k,h} V_{k,h + 1}\right] (s_{k,h}, a_{k,h}) + E_{k,h}\right\}}.$
5:  $\widehat{\Sigma}_{k+1,h} \gets \widehat{\Sigma}_{k,h} + \bar{\sigma}_{k,h}^{-2}\phi_{V_{k,h+1}}(s_{k,h},a_{k,h})\phi_{V_{k,h+1}}(s_{k,h},a_{k,h})^{\top}.$
6:  $\widehat{b}_{k + 1,h}\gets \widehat{b}_{k,h} + \bar{\sigma}_{k,h}^{-2}\phi_{V_{k,h + 1}}(s_{k,h},a_{k,h})V_{k,h + 1}(s_{h + 1}^{k}).$
7:  $\widetilde{\Sigma}_{k+1,h} \gets \widetilde{\Sigma}_{k,h} + \phi_{V_{k,h+1}^2}(s_{k,h}, a_{k,h}) \phi_{V_{k,h+1}^2}(s_{k,h}, a_{k,h})^\top.$
8:  $\widetilde{\mathrm{b}}_{k + 1,h}\gets \widetilde{b}_{k,h} + \phi_{V_{k,h + 1}^2}\left(s_{k,h},a_{k,h}\right)V_{k,h + 1}^2\left(s_{h + 1}^k\right).$
9:  $\widehat{\theta}_{k+1,h} \gets \widehat{\Sigma}_{k+1,h}^{-1}\widehat{b}_{k+1,h}, \widetilde{\theta}_{k+1,h} \gets \widetilde{\Sigma}_{k+1,h}^{-1}\widetilde{b}_{k+1,h}.$
20: end for
21: end for
Dynamic Regret. For any policy  $\pi = \{\pi_h\}_{h\in [H]}$  and any  $(k,h,s,a)\in [K]\times [H]\times S\times A,$  we define the action-value function  $Q_{k,h}^{\pi}$  and value function  $V_{k,h}^{\pi}$  as
$$
Q _ {k, h} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {k, h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, a _ {h} = a \right], V _ {k, h} ^ {\pi} (s) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {k, h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s \right].
$$
The Bellman equation is given by  $Q_{k,h}^{\pi} = r_{k,h} + \mathbb{P}_{h}V_{k,h + 1}^{\pi}$ , and  $V_{k,h}^{\pi}(s) = \mathbb{E}_{a\sim \pi_h(\cdot |s)}[Q_{k,h}^{\pi}(s,a)]$  with  $V_{k,H + 1}^{\pi} = 0$ . For simplicity, for any function  $V:\mathcal{S}\to \mathbb{R}$ , we define the operator
$$
\left[ \mathbb {P} _ {h} V \right] (s, a) = \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {h} (\cdot | s, a)} V \left(s ^ {\prime}\right), \quad \left[ \mathbb {V} _ {h} V \right] (s, a) = \left[ \mathbb {P} _ {h} V ^ {2} \right] (s, a) - \left(\left[ \mathbb {P} _ {h} V \right] (s, a)\right) ^ {2}. \tag {3}
$$
As stated in Section 1, dynamic regret is a more appropriate measure compared with static regret for the adversarial environments, which is defined in (2) and we rewrite it below for clarity: