Title: Dynamic Regret of Adversarial Linear Mixture MDPs
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamic regret

adversarial MDPs

linear mixture MDPs

policy optimization
Abstract: We study reinforcement learning in episodic inhomogeneous MDPs with adversarial full-information rewards and the unknown transition kernel. We consider the linear mixture MDPs whose transition kernel is a linear mixture model and choose the \emph{dynamic regret} as the performance measure. Denote by $d$ the dimension of the feature mapping, $H$ the horizon, $K$ the number of episodes, $P_T$ the non-stationary measure, we propose a novel algorithm that enjoys an $\widetilde{\mathcal{O}}\big(\sqrt{d^2 H^3K} + \sqrt{H^4(K+P_T)(1+P_T)}\big)$ dynamic regret under the condition that $P_T$ is known, which improves previously best-known dynamic regret for adversarial linear mixture MDP and adversarial tabular MDPs. We also establish an $\Omega\big(\sqrt{d^2 H^3 K} + \sqrt{H K (H+P_T)}\big)$ lower bound, indicating our algorithm is \emph{optimal} in $K$ and $P_T$. Furthermore, when the non-stationary measure $P_T$ is unknown, we design an online ensemble algorithm with a meta-base structure, which is proved to achieve an $\widetilde{\mathcal{O}}\big(\sqrt{d^2 H^3K} + \sqrt{H^4(K+P_T)(1+P_T) + H^2 S_T^2}\big)$ dynamic regret and here $S_T$ is the expected switching number of the best base-learner. The result can be optimal under certain regimes.

Dynamic Regret of Adversarial Linear Mixture MDPs

Abstract

We study reinforcement learning in episodic inhomogeneous MDPs with adversarial full-information rewards and the unknown transition kernel. We consider the linear mixture MDPs whose transition kernel is a linear mixture model and choose the dynamic regret as the performance measure. Denote by d the dimension of the feature mapping, H the length of each episode, K the number of episodes, P T the non-stationary measure, we propose a novel algorithm that enjoys an ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T ) ) dynamic regret under the condition that P T is known, which improves previously best-known dynamic regret for adversarial linear mixture MDP and adversarial tabular MDPs. We also establish an Ω ( √ d 2 H 3 K + √ HK ( H + P T ) ) lower bound, indicating our algorithm is optimal in K and P T . Furthermore, when the non-stationary measure P T is unknown, we design an online ensemble algorithm with a meta-base structure, which is proved to achieve an ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T ) + H 2 S 2 T ) dynamic regret and here S T is the expected switching number of the best base-learner. The result can be optimal under certain regimes.

1 Introduction

Reinforcement Learning (RL) aims to learn a policy that maximizes the cumulative reward through interacting with the environment, which has achieved tremendous successes in various fields, including games [1, 2], robotic control [3, 4], and dialogue generation [5, 6]. In reinforcement learning, the Markov Decision Process (MDP) [7] is the most widely used model to describe the environment.

Traditional MDPs assume that the reward functions are stochastic and the number of actions and states is small. However, in many real-world applications, the reward functions may be adversarially changing and the state and action spaces are large or even infinite. Previous work studies these two problems separately. To deal with the adversarial reward functions, Even-Dar et al. [8] first consider learning adversarial MDPs with known transition and full-information reward feedback. They propose the MDP-E algorithm that enjoys ˜ O ( √ τ 3 T ) regret, where τ is the mixing time and T is the number of steps. There is a line of subsequent work studying adversarial MDPs [9, 10, 11, 12, 13, 14, 15, 16], which studies various settings depending on whether the transition kernel is known, and whether the feedback is full-information or bandit. To overcome the large state and action space issue, a widely used method is linear function approximation , which reparameterizes the value function as a linear function over some feature mapping that maps the state and action to a low-dimensional space. Amongst these work, linear MDPs [17, 18, 19, 20, 21, 22, 23] and linear mixture MDPs [24, 25, 26, 27, 28, 29, 30, 31] are two of the most popular MDP models with linear function approximation. In particular, He et al. [23] and Zhou et al. [31] attain the minimax optimal ˜ O ( √ d 2 H 3 K ) regret for linear MDPs and linear mixture MDPs with stochastic rewards respectively.

Recent studies try to combine two lines of work to establish the theoretical foundation of adversarial MDPs with large state and action space. In particular, Cai et al. [32] study adversarial linear mixture

MDPs and propose the OPPO algorithm that enjoys an ˜ O ( √ d 2 H 4 K ) regret. He et al. [33] improve the result to ˜ O ( √ d 2 H 3 K ) and show it is (nearly) optimal by presenting a matching lower bound. However, these studies choose static regret as the performance measure, defined as the performance difference between the learner's policy and that of the best-fixed policy in hindsight, namely,

where V is the value function and Π is the set of all stochastic policies. One caveat in (1) is that the best fixed policy may behave poorly in non-stationary environments. To this end, we introduce dynamic regret , which benchmarks the learner's performance with changing policies, defined as

where π c 1 , . . . , π c K ∈ Π is compared policies. Define P T = ∑ K k =2 d ( π c k , π c k -1 ) with a certain distance measure d ( · , · ) as the non-stationary measure. A favorable dynamic regret should scale with P T .

Dynamic regret is a more appropriate measure in non-stationary environments, but it is more challenging to optimize such that few studies focus on it in the literature. Zhao et al. [34] investigate the dynamic regret of adversarial tabular MDPs with the known transition kernel and present an algorithm with optimal dynamic regret. Importantly, their algorithm does not require the non-stationarity measure P T as the algorithmic input ahead of time. For the unknown transition setting, Fei et al. [35] study adversarial tabular MDPs and propose an algorithm with dynamic regret guarantee. Zhong et al. [36] further extend the algorithm of Fei et al. [35] to accommodate non-stationary transition kernels with linear function approximation. Both algorithms of Fei et al. [35] and Zhong et al. [36] require the quantity of P T as the input. Moreover, their dynamic regret bounds are suboptimal in terms of K and P T as demonstrated by the lower bound established by our work (see Theorem 4).

This work investigates the dynamic regret of adversarial linear mixture MDPs, with a focus on the full-information feedback and the unknown transition. We first propose POWERS-FixShare algorithm when P T is known, an algorithm combining optimistic policy optimization with a Bernstein bonus and fixed-share mechanism. We show it enjoys an ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T )) dynamic regret, where d is the dimension of the feature mapping, H is the length of each episode, K is the number of episodes, P T is the non-stationary measure. We also establish a dynamic regret lower bound of Ω( √ d 2 H 3 K + √ HK ( H + P T )) . We stress four remarks regarding our results:

√

Our result can recover the ˜ O ( d 2 H 3 K ) minimax optimal static regret in He et al. [33].

Our result improves upon the previously best-known ˜ O ( dH 7 / 4 K 3 / 4 + H 2 K 2 / 3 P T 1 / 3 ) dynamic regret for the fixed transition of Zhong et al. [36, Theorem 4.6] in terms of H , K and P T . √

Our result can imply an ˜ O ( H 4 S 2 AK + H 2 √ ( K + P T )(1 + P T )) dynamic regret for adversarial tabular MDPs, strictly improving upon the previously best-known ˜ O ( √ H 4 S 2 AK + H 2 K 2 / 3 P T 1 / 3 ) dynamic regret of Fei et al. [35, Theorem 1], in terms of both K and P T .

As the lower bound suggests, our result is the first optimal regarding the dependence on d , K and P T and can be optimal in terms of H under certain regimes ( H ≤ d and P T ≤ d 2 /H ).

Furthermore, we study the case when P T is unknown and design a novel algorithm equipped with the dynamic regret guarantee by the meta-base two-layer structure. Our algorithm enjoys an ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T ) + H 2 S 2 T ) dynamic regret, where S T is the expected switching number of the best base-learner. Though S T is a data-dependent quantity, it also reflects the degree of environmental non-stationarity to some extent. Moreover, under specific regimes, the magnitude of S T may be relatively negligible, resulting in our results still being optimal. Indeed, given that S T is a data-dependent quantity, its inclusion in the regret bound is not ideal. Deriving bounds that rely exclusively on problem-dependent quantities, like P T , remains an open challenge. We discuss the technical difficulty of removing S T in Section 5 and take this issue for future work.

Finally, we also highlight the main technical challenges and our solutions as follows.

We first show that the dynamic regret depends on the inverse of the minimal probability over the action space of our policies, which can be arbitrarily small. To this end, we propose a novel

algorithm with the fixed-share mechanism [37]. While this mechanism is proved to enjoy favorable dynamic regret in online convex optimization [38], it suffers an additional term that can be regarded as the weighted sum of the difference between the occupancy measure of the compared policies in online MDPs. To overcome the difficulty, we exploit the multiplicative stability to bound this term, eliminating the need for a restart strategy to handle the environmental non-stationarity as in previous studies [35, 36] and allows us to attain the dynamic regret optimal in terms of K and P T .

We show the dynamic regret of online MDPs can be written as the weighted average of 'multiarmed bandits' problems over all states, where the weight for each state is the unknown and changing probability of being visited by π c 1 , . . . , π c K . For the unknown P T case, we first show the standard two-layer structure used in non-stationary online learning studies [39, 40, 41] fails to achieve a favorable dynamic regret guarantee, which characterizes the unique difficulty of online MDPs. Then, we present an initial attempt to address this issue by a specifically designed two-layer structure. We prove our algorithm enjoys nice dynamic regret guarantees under certain regimes.

Notations. We denote by ∆( A ) the set of probability distributions on a set A and denote the KLdivergence by D KL ( p ∥ p ′ ) = ∑ a ∈A p ( a ) log p ( a ) p ′ ( a ) for any p, p ′ ∈ ∆( A ) . We define ∆( A| S , H ) = {{ π h ( · | · ) } H h =1 | π h ( · | x ) ∈ ∆( A ) , ∀ s ∈ S , h ∈ [ H ] } for any set S and H ∈ Z + . Further, for any π, π ′ , π ′′ ∈ ∆( A|S , H ) , we define E π [ ˜ D KL ( π ′ ∥ π ′′ )] = E π [ ∑ H h =1 D KL ( π ′ h ( · | s h ) ∥ π ′′ h ( · | s h ))] . For any policy pair π h , π ′ h , we define ∥ π h -π ′ h ∥ 1 , ∞ = max s ∈S ∥ π h ( · | s ) -π ′ h ( · | s ) ∥ 1 . For any a, b, x ∈ R with a ≤ b , let [ x ] [ a,b ] denote min { max( x, a ) , b } . ˜ O ( · ) omits the logarithmic factors.

2 Related Work

RL with adversarial rewards. There are many studies on learning adversarial MDPs where the reward functions are adversarially chosen, yielding fruitful results that can be categorized into three lines [8, 9, 10, 11, 12, 13, 14, 15, 16]. In particular, the first line of work considers the infinite-horizon MDPs with uniform mixing assumption. In the known transition and full-information setting, the seminal work of Even-Dar et al. [8] proposes MDP-E algorithm that achieves the ˜ O ( √ τ 3 T ) regret, where τ is the mixing time and T is the number of steps. Another concurrent work of Yu et al. [9] achieves ˜ O ( T 2 / 3 ) in the same setting. In the known transition and bandit-feedback setting, Neu et al. [10] propose MDP-EXP3 algorithm that attains ˜ O ( T 2 / 3 ) regret. The second line of work considers the episodic loop-free MDPs. Neu et al. [11] first study this problem under the known transition setting and propose algorithms that achieve ˜ O ( √ T ) and ˜ O ( √ T/α ) for full-information and bandit-feedback respectively, where α is the lower bound of the probability of all states under any policy. Zimin and Neu [12] propose O-REPS algorithm that enjoys ˜ O ( √ T ) regret in both full-information and bandit-feedback setting without any additional assumption. Rosenberg and Mansour [13] and Jin et al. [14] further consider the harder unknown transition and bandit-feedback setting. The last line of work studies the episodic Stochastic Shortest Path (SSP) setting [15, 16]. In this paper, we focus on episodic MDPs with the unknown transition and full-information setting.

RL with linear function approximation. To design RL algorithms in large state and action space scenarios, recent works focus on solving MDPs with linear function approximation. In general, these works can be divided into three lines based on the specific assumption of the underlying MDP. The first line of work considers the low Bellman-rank assumption [42, 43, 44, 45], which assumes the Bellman error matrix has a low-rank factorization. The second line of work is based on the linear MDP assumption [17, 18, 19, 20, 21, 22, 23], where both the transition kernel and reward functions can be parameterized as linear functions of given state-action feature mappings ϕ : S × A → R d . The last line of work studies the linear mixture MDP [24, 25, 26, 28, 29, 30, 31], where the transition kernel can be parameterized as a linear function of a feature mapping ϕ : S × A × S → R d but without the linear reward functions assumption. Amongst these works, He et al. [23] and Zhou et al. [31] attain the minimax optimal ˜ O ( √ d 2 H 3 K ) regret for both episodic linear MDPs and linear mixture MDPs respectively. However, all the above studies consider the stochastic reward setting. In this paper, we study the episodic linear mixture MDP setting but with adversarial reward functions.

Non-stationary RL. Another related line of research is online non-stationary MDPs. In contrast to adversarial MDPs where the reward functions are generated in an adversarial manner, online non-stationary MDPs consider the setting where the reward functions are generated stochastically

according to some distributions that may vary over time. Jaksch et al. [46] study the piecewise stationary setting where the transitions and rewards are allowed to change certain times and propose UCRL2 with restart technique to deal with the non-stationarity. Later, Ortner et al. [47] consider the generalized setting where the changes are allowed to take place every step. However, the above studies need prior knowledge about the magnitude of non-stationary. To address this issue, Cheung et al. [48] propose the Bandit-over-RL algorithm to remove this requirement. A recent breakthrough by Wei and Luo [49] introduces a black-box method that can convert any algorithm satisfying specific conditions and enjoying optimal static regret in stationary environments into another with optimal dynamic regret in non-stationary environments, without requiring prior knowledge about the degree of non-stationarity. However, this approach does not apply to the adversarial setting. Specifically, their reduction requires the base algorithm to satisfy a certain property enjoyed by typical UCB-type algorithms. When a new instance of the base algorithm surpasses the optimistic estimator, it can be inferred that the environment has changed, prompting a restart of the algorithm to disregard prior information. However, this approach of constructing an optimistic estimator by a UCB-type algorithm can only be applied to a stochastic setting. In the adversarial setting, where no model assumptions are made and comparators can be arbitrary, this approach encounters significant difficulties.

Dynamic Regret. Dynamic regret of RL with adversarial rewards is only recently studied in the literature [34, 35, 36]. Zhao et al. [34] investigate the dynamic regret of adversarial tabular MDPs with the known transition kernel and present an algorithm with optimal dynamic regret. Importantly, their algorithm does not require the non-stationarity measure P T as the algorithmic input ahead of time. For the unknown transition setting, Fei et al. [35] study adversarial tabular MDPs and propose an algorithm with dynamic regret guarantees. Zhong et al. [36] further extend the algorithm of Fei et al. [35] to accommodate non-stationary transition kernels with linear function approximation. Both algorithms [35, 36] require the quantity of P T as the input. Moreover, their dynamic regret bounds are suboptimal in K and P T as shown by the lower bound established in our work. In this work, we first design an optimal algorithm in terms of K and P T when P T is known. Further, we develop the first algorithm to handle the unknown P T issue in adversarial MDPs with unknown transition.

3 Problem Setup

We focus on episodic inhomogeneous MDPs with full-information reward functions and the unknown transition kernel. Denote by M = {S , A , H, { r k,h } k ∈ [ K ] ,h ∈ [ H ] , { P h } h ∈ [ H ] } an episodic inhomogeneous MDP, where S is the state space, A is the action space, K is the number of episodes, H is the horizon, r k,h : S × A → [0 , 1] is the reward function, P h ( · | · , · ) : S × A × S → [0 , 1] is the transition kernel. We assume the rewards are deterministic without loss of generality and extending our results to stochastic rewards is straightforward. Let S = |S| and A = |A| .

The learner interacts with the MDP through K episodes without knowledge of transition kernel { P h } h ∈ [ H ] . In each episode k , the environment chooses the reward function { r k,h } h ∈ [ H ] and decides the initial state s k, 1 , where the reward function may be chosen in an adversarial manner and depend on the history of the past ( k -1) episodes. Simultaneously, the learner decides a policy π k = { π k,h } h ∈ [ H ] where each π k,h : S → ∆( A ) is a function that maps a state s to a distribution over action space A . In the h stage in episode k , the learner observes current state s k,h , chooses an action a k,h ∼ π k,h ( · | s k,h ) , and transits to the next state s k,h +1 ∼ P h ( · | s k,h , a k,h ) . Then the learner obtains the reward r k,h ( s k,h , a k,h ) and observes the reward function r k,h as we consider the full-information setting. At stage H +1 , the learner observes the final state s k,H +1 but does not take any action, and the episode k terminates. Denote by T = KH the total steps throughout K episodes.

Linear Mixture MDPs. In this work, we focus on a special class of MDPs called linear mixture MDPs , a setting initiated by Ayoub et al. [24] and further studied in the subsequent work [32, 31, 33]. In this setup, the transition kernel can be parameterized as a linear function of a feature mapping ϕ : S × A × S → R d . The formal definition of linear mixture MDPs is as follows.

Definition 1 (Linear Mixture MDPs) . An MDP M = {S , A , H, { r k,h } k ∈ [ K ] ,h ∈ [ H ] , { P h } h ∈ [ H ] } is called an inhomogeneous, episode B -bounded linear mixture MDP, if there exist a known feature mapping ϕ ( s ′ | s, a ) : S × A × S → R d and an unknown vector θ ∗ h ∈ R d with ∥ θ ∗ h ∥ 2 ≤ B , ∀ h ∈ [ H ] such that (i) P h ( s ′ | s, a ) = ϕ ( s ′ | s, a ) ⊤ θ ∗ h for all ( s, a, s ′ ) ∈ S × A × S and h ∈ [ H ] , (ii) ∥ ϕ V ( s, a ) ∥ 2 ≜ ∥ ∑ s ′ ∈S ϕ ( s ′ | s, a ) V ( s ′ ) ∥ 2 ≤ 1 for any ( s, a ) ∈ S × A and function V : S → [0 , 1] .

Algorithm 1 POWERS-FixShare

Input: step size η , exploration parameter γ and regularization parameter λ .

1: Initialize { π 0 ,h ( · | s ) } H h =1 , s ∈ S as uniform distribution, and set { Q 0 ,h ( · , · ) } H h =1 as zero function.

2: for k = 1 , 2 , · · · , K do

3: Receive the initial state s k, 1 .

4: for h = 1 , 2 , · · · , H do

5: For all h ∈ [ H ] , s ∈ S , updates policy by

6: Take the action following a k,h ∼ π k,h ( · | s k,h ) and transit to the next state s k,h +1 .

7: Obtain reward r k,h ( s k,h , a k,h ) and observe the reward function r k,h ( · , · ) .

8: end for

9: Initialize V k,H +1 ( · ) as a zero function.

10: for h = H,H -1 , · · · , 1 do

13: Set the estimated variance [ ¯ V k,h V k,h +1 ] ( s k,h , a k,h ) as in (11), bonus E k,h as in (21).

20: end for

21: end for

Dynamic Regret. For any policy π = { π h } h ∈ [ H ] and any ( k, h, s, a ) ∈ [ K ] × [ H ] ×S × A , we define the action-value function Q π k,h and value function V π k,h as

The Bellman equation is given by Q π k,h = r k,h + P h V π k,h +1 , and V π k,h ( s ) = E a ∼ π h ( · | s ) [ Q π k,h ( s, a )] with V π k,H +1 = 0 . For simplicity, for any function V : S → R , we define the operator

As stated in Section 1, dynamic regret is a more appropriate measure compared with static regret for the adversarial environments, which is defined in (2) and we rewrite it below for clarity:

where π c 1 , . . . , π c K is any sequence of compared policies. We define π c 0 = π c 1 to simplify the notation. The non-stationarity measure is defined as P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ .

4 Optimal Dynamic Regret with Known P T

We present our proposed algorithm in Algorithm 1. Similar to previous studies, the algorithm consists of (i) policy improvement phase, and (ii) policy evaluation phase. We introduce the details below. In Sections 4.1, we first consider the case when the transition is known to highlight the challenges even under the ideal setting. Then, we extend the results to the unknown transition setting in Section 4.2.

4.1 Policy Improvement Phase

In the policy improvement phase, the algorithm updates π k based on π k -1 using the proximal policy optimization (PPO) method [50]. Specifically, at episode k , we define the following linear function:

which is the first-order Taylor approximation of V π k -1 , 1 ( s k, 1 ) around π k -1 . Then, we update π k by

where η > 0 is the stepsize and the KL-divergence encourages π k to be close to π k -1 so that L k -1 ( π ) is a good approximation of V π k -1 , 1 ( s k, 1 ) . The update rule in (5) takes the following closed form,

We show the update rule in (6) ensures the following guarantee and the proof is in Appendix C.1.

Lemma 1. The update rule in (6) ensures the following dynamic regret guarantee:

Note that the expectation in the last term in (7) is taken over π c k which may change over episode k . For static regret, i.e., π c 1 = . . . = π c K = π ∗ , we can control this term by a standard telescoping argument, which is not viable for dynamic regret analysis. Fei et al. [35] propose a restart strategy to handle this term. Specifically, they restart the algorithm every certain number of steps and decompose the above expectation into E π c k [ · ] = E π c k 0 [ · ] + E π c k -π c k 0 [ · ] where k 0 < k is the episode in which restart takes place most recently before episode k . For the first expectation E π c k 0 [ · ] , they apply a customized telescoping argument to each period as the expectation is taken over the fixed policy. The second expectation E π c k -π c k 0 [ · ] involves the difference π c k -π c k 0 and can be bounded by P T . However, as we will show in Theorem 4, their regret bound is suboptimal in terms of K and P T .

We introduce our approach below. Let us first consider taking expectations over any fixed policy π . Denote by δ the minimal probability over any action at any state for policies π 1 , . . . , π K , i.e., δ = min k ∈ [ K ] π k ( a | s ) , ∀ a ∈ A , s ∈ S , the last term in (7) can be upper bounded by ∑ K k =1 E π [ ˜ D KL ( π c k ∥ π k ) -˜ D KL ( π c k ∥ π k +1 )] ≤ H log A + P T log 1 δ , showing that we need to control the minimal value of δ to obtain a favorable dynamic regret bound. To this end, we slightly modify the update rule in (6) and add a uniform distribution π u ( · | s ) = 1 A 1 , ∀ s ∈ S . That is, the policy π u chooses each action with equal probability at any state. Thus, the update rule in (6) is modified as:

for any s ∈ S , h ∈ [ H ] , where γ ≥ 0 is the exploration parameter. This update is called the fixed-share mechanism in online learning literature [37]. While the fixed-share mechanism is standard to obtain dynamic regret in modern online learning [38], several important new challenges arise in online MDPs due to taking expectations over the policy sequence of changing policies π c 1 , . . . , π c K . In particular, we prove that performing (8) ensures the following dynamic regret.

Lemma 2. Set π 1 as uniform distribution on A for any state s ∈ S . The update rule in (8) ensures

The proof can be found in Appendix C.2. In the dynamic regret analysis in online learning, the last term in (9) is usually canceled out through telescoping since we do not need to take expectations [38].

However, this is not the case in online MDPs. Since the expectation is taken over the policy sequence of changing policies π c 1 , . . . , π c K , this term cannot be canceled out, which requires a more refined analysis. To address this issue, we decompose one step of the expectation in (9) as follows. 1

With this decomposition, the first term can be canceled out through telescoping, yet it remains to control the second term - the weighted difference between the state-action occupancy measures of policy π c k and π c k -1 with weight -π c k -1 ,h ( a | s h ) log π ′ k,h ( a | s h ) for state-action ( s h , a ) . To control it, we need to (i) ensure the weight is upper bounded by some universal constant, and (ii) bound the unweighted difference between the state-action occupancy measures, which are new challenges that arose in online MDPs compared with standard online learning.

For the first challenge, note that the weight -π c k -1 ,h ( a | s h ) log π ′ k,h ( a | s h ) can be large or even infinite since π ′ k,h is the policy before uniform exploration and π ′ k,h ( a | s h ) can be arbitrarily small. Fortunately, π ′ k,h is obtained by one-step descent from π k -1 ,h , which is the policy after uniform exploration and can be lower bounded. We provide the following multiplicative stability lemma for the one-step update, which shows π ′ k,h is not far from π k -1 ,h and thus is also lower bounded.

Lemma 3 (Multiplicative Stability) . For any distributions p ∈ ∆( A ) with p ( a ) > 0 , for all a ∈ A , and any function Q : S × A → [0 , H ] , it holds for p ′ ∈ ∆( A ) with p ′ ( a ) ∝ p ( a ) · exp( η · Q ( s, a )) and η ≤ 1 /H that p ′ ( a ) ∈ [ p ( a ) / 4 , 4 p ( a )] , for all a ∈ A .

For the second challenge, we show the unweighted difference between the state-action occupancy measures can be bounded by the path length of policies. In particular, we have the following lemma.

Lemma 4. For any policy sequence π c 1 , . . . , π c K , it holds that

Remark 1. We note that a similar argument is also used in Fei et al. [35, Appendix B.2.2]. However, they prove this lemma by imposing an additional smooth visitation measures assumption [35, Assumption 1], which is not required in our analysis.

The proofs for Lemmas 3 and 4 can be found in Appendices C.3 and C.4 respectively. Combining Lemmas 2, 3 and 4, we can prove the guarantee for update rule (8). The proof is in Appendix C.5.

Theorem 1. Set π 1 as uniform distribution on A for any state s ∈ S . The update rule in (8) ensures

Remark 2. Considering the static regret where π c 1 = . . . = π c K = π ∗ , we can recover the O ( √ H 4 K log A ) static regret in Cai et al. [32] under the stationary scenario by setting γ = 0 , that is, without uniform exploration. However, when γ = 0 , the dynamic regret is not bounded as there lacks an upper bound for -log γ , showing the necessity of the fixed-share mechanism.

4.2 Policy Evaluation Phase

Sections 4.1 focus on the simplified scenario where the transition is known. In this subsection, we further consider the unknown transition setting such that it is necessary to evaluate the policy π k based on the ( k -1) historical trajectories. To see how the model estimation error enters the dynamic regret, we decompose the dynamic regret in the following lemma.

Lemma 5 (Fei et al. [35, Lemma 1]) . Define the model prediction error as ι k,h = r k,h + P h V k,h +1 -Q k,h , the dynamic regret D-Regret( K ) = ∑ K k =1 V π c k k, 1 ( s k, 1 ) -∑ K k =1 V π k k, 1 ( s k, 1 ) can be written as

Remark 3. Lemma 5 is independent of the structure of MDPs. The first term in Lemma 5 is the dynamic regret over the estimated action-value function Q k,h , which can be upper bounded by Theorem 5. The second term is a martingale, which can be bounded by Azuma-Hoeffding inequality. The third term is the model estimation error, which is the main focus of this section. Note the model prediction error ι k,h ( s h , a h ) can be large for the state-action pairs that are less visited or even unseen. The general approach is incorporating the bonus function into the estimated Q -function such that ι k,h ( s h , a h ) ≤ 0 for all s ∈ S , a ∈ A (i.e., E π c k [ ι k,h ( s h , a h )] ≤ 0 ) and we only need to control -ι k,h ( s k,h , a k,h ) , which is the model estimation error at the visited state-action pair ( s k,h , a k,h ) .

When applied to linear mixture MDPs, the key idea is learning the unknown parameter θ ∗ h of the linear mixture MDP and using the learned parameter θ k,h to build an optimistic estimator Q k,h ( · , · ) such that the model prediction error is non-positive, which is more or less standard. From the definition of linear mixture MDP, for the learned value function V k,h ( · ) , we have [ P h V k,h +1 ]( s, a ) = ⟨ ∑ s ′ ϕ ( s ′ | s, a ) V k,h +1 ( s ′ ) , θ ∗ h ⟩ = ⟨ ϕ V k,h +1 ( s, a ) , θ ∗ h ⟩ . Inspired by recent advances in policy evaluation for linear mixture MDPs [31], we adopt the weighted ridge regression to estimate the parameter θ ∗ h , that is, we construct the estimator ̂ θ k,h by solving the following weighted ridge regression problem:

Here, ¯ σ 2 j,h is the upper confidence bound of the variance [ V h V j,h +1 ]( s j,h , a j,h ) , and we set it as ¯ σ k,h = √ max { H 2 /d, [ ¯ V k,h V k,h +1 ] ( s k,h , a k,h ) + E k,h } , where [ ¯ V k,h V k,h +1 ]( s k,h , a k,h ) is a scalar-valued empirical estimate for the variance of the value function V k,h +1 under the transition probability P h ( · | s k , a k ) , and E k,h is the bonus term to guarantee that the true variance [ V k,h V k,h +1 ]( s k,h , a k,h ) is upper bounded by [ ¯ V k,h V k,h +1 ]( s k,h , a k,h ) + E k,h with high probability. Then, the confidence set ̂ C k,h is constructed as follows:

where ̂ Σ k,h is a covariance matrix based on the observed data, and ̂ β k is a radius of the confidence set. Given ̂ C k,h , we estimate the Q -function following the principle of 'optimism in the face of uncertainty' [51] and set it as Q k,h ( · , · ) = [ r k,h ( · , · ) + max θ ∈ ̂ C k,h ⟨ θ, ϕ V k,h +1 ( · , · ) ⟩ ] [0 ,H -h +1] .

It remains to estimate the variance [ V h V k,h +1 ]( s k,h , a k,h ) . By the definition of linear mixture MDPs, we have [ V h V k,h +1 ]( s k,h , a k,h ) = ⟨ ϕ V 2 k,h +1 ( s k,h , a k,h ) , θ ∗ h ⟩ -[ ⟨ ϕ V k,h +1 ( s k,h , a k,h ) , θ ∗ h ⟩ ] 2 . Therefore, we estimate [ ¯ V k,h V k,h +1 ] ( s k,h , a k,h ) by the expression below

where ˜ θ k,h = arg min θ ∈ R d ∑ k -1 j =1 [ ⟨ ϕ V 2 j,h +1 ( s j,h , a j,h ) , θ ⟩ -V 2 j,h +1 ( s j,h +1 )] 2 + λ ∥ θ ∥ 2 2 . The details are summarized in Lines 10-20 of Algorithm 1 and we provide the following guarantee.

Theorem 2. Set the parameters as in Lemma 8, with probability at least 1 -δ , we have

The proof is given in Appendix C.6. Theorem 2 shows the model estimation error can be bounded. Combining Theorems 1, 2 and Lemma 5, we present the dynamic regret bound in the next section.

4.3 Regret Guarantee: Upper and Lower Bounds

In this section, we provide the regret bound for our algorithm and present a lower bound of the dynamic regret for any algorithm for adversarial linear mixture MDPs with the unknown transition.

Theorem 3. Set η = min { √ ( P T +log A ) /K, 1 } /H , γ = 1 / ( KH ) and ̂ β k as in Lemma 8, then with probability at least 1 -δ , it holds

where P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ is the path length of the compared policies.

Remark 4 (recovering static regret) . Since static regret is a special case with π c k = π ∗ , ∀ k , our result can recover the optimal ˜ O ( √ d 2 H 3 K ) static regret when H ≤ d , same as the result in He et al. [33].

Remark 5 (improving linear mixture case) . Our result improves upon the previously best-known ˜ O ( dH 7 / 4 K 3 / 4 + H 2 K 2 / 3 P T 1 / 3 ) dynamic regret for adversarial linear mixture MDPs of Zhong et al. [36, Theorem 4.6] in terms of the dependence on H , K and P T .

Remark 6 (improving tabular case) . For the adversarial tabular MDPs, our result implies an ˜ O ( √ H 4 S 2 AK + H 2 √ ( K + P T )(1 + P T )) dynamic regret. This improves upon the best-known ˜ O ( √ H 4 S 2 AK + H 2 K 2 / 3 P T 1 / 3 ) result of Fei et al. [35, Theorem 1]. The details are in Appendix B.

We finally establish the lower bound of this problem. The proof can be found in Appendix C.8.

Theorem 4. Suppose B ≥ 2 , d ≥ 4 , H ≥ 3 , K ≥ ( d -1) 2 H/ 2 , for any algorithm and any constant Γ ∈ [0 , 2 KH ] , there exists an episodic B -bounded adversarial linear mixture MDP and compared policies π c 1 , . . . , π c K such that P T ≤ Γ , and D-Regret( K ) ≥ Ω( √ d 2 H 3 K + √ HK ( H +Γ)) .

When H ≤ d , the upper bound is ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T )) . Combining it with Theorem 4, we discuss the optimality of our result. We consider the following three regimes.

Small P T : when 0 ≤ P T ≤ d 2 /H , the upper bound (12) can be simplified as ˜ O ( √ d 2 H 3 K ) , and the lower bound is ˜ O ( √ d 2 H 3 K ) , hence our result is optimal in terms of d , H and K .

Moderate P T : when d 2 /H ≤ P T ≤ K , the upper bound (12) can be simplified as ˜ O ( √ d 2 H 3 K + √ H 4 K (1 + P T )) , and it is minimax optimal in d , K and P T but looses a factor of H 3 2 .

Large P T : when P T ≥ K , any algorithm suffers at most O ( HK ) dynamic regret, while the lower bound is Ω( K √ H ) . So our result is minimax optimal in K but looses a factor of √ H .

5 Towards Optimal Dynamic Regret with Unknown P T

This section further considers the case when the non-stationarity measure P T is unknown. By Theorem 1, we need to tune the step size η optimally to balance the number of episodes K and P T to achieve a favorable dynamic regret. To address the difficulty of not knowing P T ahead of time, we develop an online ensemble method to handle this uncertainty, in which a two-layer meta-base structure is maintained. While the methodology can be standard in recent non-stationary online learning [52, 39, 40, 41], new challenges arise in online MDPs. We introduce the details below.

By the performance difference lemma in Cai et al. [32, Lemma 3.2] (as restate in Lemma 13), we can rewrite the dynamic regret as

where the expectation is taken over the randomness of the state trajectory sampled according to π c k . The dynamic regret of online MDPs can be written as the weighted average of some 'multi-armed bandits' problems over all states, where the weight for each state is the unknown and changing probability of being visited by π c 1 , . . . , π c K . As the optimal step size depends on the unknown nonstationarity measure P T as shown in Section 4, a natural idea is to the two-layer structure to learn the optimal step size as in recent online convex optimization literature [52, 39, 40, 41].

The general idea is constructing a step size pool H = { η 1 , . . . , η N } to discretize the value range of the optimal step size; and then maintaining multiple base-learners B 1 , . . . , B N , each of which works with a specific step size η i . Finally, a meta-algorithm is used to track the best base-learner and yield the final policy. Then, the dynamic regret can be decomposed as follows (omit ( · | s h ) for simplicity):

Since the above decomposition holds for any index i ∈ [ N ] , we can always choose the base-learner with optimal step size to analyze and the first term is easy to control. The challenge is to control the

second term, which is the regret of the meta-algorithm. Different from the standard 'Prediction with Expert Advice' problem, it involves an additional expectation over the randomness of states sampled according to π c k . This poses a grand challenge compared to conventional online convex optimization where the expectation is not required. Although we can bound this term by P T again, optimal tuning of the meta-algorithm is hindered as P T is unknown. Consequently, we opt to upper bound it by the worst-case dynamic regret [53], that is, benchmarking the performance with the best choice of each round, which in turn introduces the dependence on the switching number of the best base-learner.

We introduce our approach as follows. We maintain multiple base-learners, each of which works with a specific step size η i . All base-learners update their policies according to the same action-value function Q k -1 ,h ( s h , · ) of the combined policy π k -1 , that is, the base-learner B i updates policy by

Then, the meta-algorithm chooses the base-learner by measuring the quality of each base-learner. In our approach, we choose the best base-learner at the last episode, that is,

The details are summarized in Algorithm 2 of Appendix A and the guarantee is as follows.

Theorem 5. Set γ = 1 / ( KH ) , step size pool H = { η i = (2 i /H ) √ (log A ) /K | i ∈ [ N ] } with N = ⌊ 1 2 log( K log A ) ⌋ . Algorithm 2 ensures

̸

where P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ is the path length of the compared policies, S T = ∑ K k =1 ∑ H h =1 E π c k 1 [ i ∗ k,h ( s h ) = i ∗ k -1 ,h ( s h )] is the expected switching number of best base-learner.

Combining it with Theorem 3, we discuss the optimality of our result. We consider two regimes.

√

Large S T : when S T > max { d HK,H √ ( K + P T )(1 + P T ) } , our result looses a factor of HS T compared with the result in Theorem 3 for the known P T setting.

Small S T : when S T ≤ max { d HK,H √ ( K + P T )(1 + P T ) } , the term S T can be subsumed by other terms. In this case, the upper bound in Theorem 5 is entirely the same as that in Theorem 3. This implies we maintain the same guarantees without P T as algorithmic input. √

By the above discussion, our result can be optimal in terms of K and P T under certain regimes when P T is unknown. In comparison, the regret bounds achieved via the restart mechanism [35, 36] remain sub-optimal across all regimes even P T is known. Note that we introduce the notation S T in the regret analysis, which also reflects the degree of environmental non-stationarity to some extent. Consider the following two examples: (i) in the stationary environment, S T could be relatively small as the best base-learner would seldom change, and (ii) in the piecewise-stationary environment, S T would align with the frequency of environmental changes. Indeed, given that S T is a data-dependent quantity, its inclusion in the regret analysis is not ideal. Deriving bounds that rely exclusively on problem-dependent quantities, like P T , remains a significant open challenge.

6 Conclusion and Future Work

In this work, we study the dynamic regret of adversarial linear mixture MDPs with the unknown transition. For the case when P T is known, we propose a novel policy optimization algorithm that incorporates a fixed-share mechanism without the need for restarts. We show it enjoys a dynamic regret of ˜ O ( √ d 2 H 3 K + √ H 4 ( K + P T )(1 + P T ) ) , strictly improving the previously best-known result of Zhong et al. [36] for the same setting and Fei et al. [35] when specialized to tabular MDPs. We also establish an Ω ( √ d 2 H 3 K + √ HK ( H + P T ) ) lower bound, indicating that our algorithm is optimal regarding d , K and P T and can be optimal in terms of H under certain regimes. Moreover, we explore the more complex scenario where P T is unknown. We show this setting presents unique challenges that distinguish online MDPs from conventional online convex optimization. We introduce a novel two-layer algorithm and show its dynamic regret guarantee is attractive under certain regimes.

There are several important future works to investigate. First, how to remove the dependence on the switching number S T is an important open question. Moreover, we focus on the full-information feedback in this work, it remains an open problem to extend the results to the bandit feedback.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning. ArXiv preprint , 1312.5602, 2013.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , pages 484-489, 2016.

John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages 1889-1897, 2015.

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33nd International Conference on Machine Learning (ICML) , pages 1329-1338, 2016.

Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1192-1202, 2016.

William Yang Wang, Jiwei Li, and Xiaodong He. Deep reinforcement learning for NLP. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 19-21, 2018.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley, 1994.

Eyal Even-Dar, Sham. M. Kakade, and Yishay Mansour. Online Markov decision processes. Mathematics of Operations Research , pages 726-736, 2009.

Jia Yuan Yu, Shie Mannor, and Nahum Shimkin. Markov decision processes with arbitrary reward processes. Mathematics of Operations Research , pages 737-757, 2009.

Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online Markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems 24 (NIPS) , pages 1804-1812, 2010.

Gergely Neu, András György, and Csaba Szepesvári. The online loop-free stochastic shortestpath problem. In Proceedings of 23rd Conference on Learning Theory (COLT) , pages 231-243, 2010.

Alexander Zimin and Gergely Neu. Online learning in episodic Markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems 26 (NIPS) , pages 1583-1591, 2013.

Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 5478-5486, 2019.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial Markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 4860-4869, 2020.

Aviv Rosenberg and Yishay Mansour. Stochastic shortest path with adversarially changing costs. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI) , pages 2936-2942, 2021.

Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Minimax regret for stochastic shortest path with adversarial costs and known transition. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 1180-1215, 2021.

Lin Yang and Mengdi Wang. Sample-optimal parametric Q-learning using linearly additive features. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 6995-7004, 2019.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the 33rd Conference on Learning Theory (COLT) , pages 2137-2143, 2020.

Simon S. Du, Sham M. Kakade, Ruosong Wang, and Lin F. Yang. Is a good representation sufficient for sample efficient reinforcement learning? In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1954-1964, 2020.

Yining Wang, Ruosong Wang, Simon Shaolei Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear function approximation. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 4171-4180, 2021.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear Markov decision processes. ArXiv preprint , 2212.06132, 2022.

Alex Ayoub, Zeyu Jia, Csaba Szepesvári, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 463-474, 2020.

Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In Proceedings of the 23th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 2010-2020, 2020.

Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 10746-10756, 2020.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted MDPs with feature mapping. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 12793-12802, 2021.

Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S. Du. Improved variance-aware confidence sets for linear bandits and linear mixture MDP. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 4342-4355, 2021.

Dongruo Zhou and Quanquan Gu. Computationally efficient horizon-free reinforcement learning for linear mixture MDPs. ArXiv preprint , 2205.11507, 2022.

Yue Wu, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal regret for learning infinite-horizon average-reward MDPs with linear function approximation. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3883-3913, 2022.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvári. Nearly minimax optimal reinforcement learning for linear mixture Markov decision processes. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4532-4576, 2021.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 1283-1294, 2020.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Near-optimal policy optimization algorithms for learning adversarial linear mixture MDPs. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 4259-4280, 2022.

Peng Zhao, Long-Fei Li, and Zhi-Hua Zhou. Dynamic regret of online Markov decision processes. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 26865-26894, 2022.

Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 6743-6754, 2020.

Han Zhong, Zhuoran Yang, Zhaoran Wang, and Csaba Szepesvári. Optimistic policy optimization is provably efficient in non-stationary MDPs. ArXiv preprint , arXiv:2110.08984, 2021.

Mark Herbster and Manfred K. Warmuth. Tracking the best expert. Machine Learning , 32(2): 151-178, 1998.

Nicolò Cesa-Bianchi, Pierre Gaillard, Gábor Lugosi, and Gilles Stoltz. Mirror descent meets fixed share (and feels no regret). In Advances in Neural Information Processing Systems 25 (NIPS) , pages 989-997, 2012.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in exp-concave online learning. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 359-409, 2021.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. ArXiv preprint , 2112.14368, 2021.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 1704-1713, 2017.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over modelfree approaches. In Proceedings of the 32nd Conference on Learning Theory (COLT) , pages 2898-2933, 2019.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 1665-1674, 2019.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 13406-13418, 2021.

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , pages 1563-1600, 2010.

Ronald Ortner, Pratik Gajane, and Peter Auer. Variational regret bounds for reinforcement learning. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 81-90, 2019.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for nonstationary Markov decision processes: The blessing of (more) optimism. Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 1843-1854, 2020.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4300-4354, 2021.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint , 1707.06347, 2017.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 (NIPS) , pages 2312-2320, 2011.

Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 1330-1340, 2018.

Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC) , pages 48-59, 2021.

Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Conference On Learning Theory (COLT) , pages 1216-1259, 2021.

Chi Jin, Zeyuan Allen-Zhu, Sébastien Bubeck, and Michael I. Jordan. Is Q-learning provably efficient? In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 48684878, 2018.

A Algorithm with Unknown P T

In this part, we present the POWERS-FixShare-OnE algorithm in Algorithm 2 for the case when P T is unknown. The algorithm is based on the POWERS-FixShare algorithm and we further employ an online ensemble structure to eliminate the algorithmic dependence on P T . In Line 5, each base-learner updates her policy with a specific step size η i and the meta-learner selects the best policy among the base-learners in Line 6. In the policy evaluation phase (Line 11 - 21), we use the same estimator as in Algorithm 1 to estimate the parameter θ ∗ h and construct the confidence set.

Algorithm 2 POWERS-FixShare-OnE

Input: step size pool H , exploration parameter γ and regularization parameter λ .

1: Initialize { π i 0 ,h ( · | s ) } H h =1 , ∀ i ∈ [ N ] , s ∈ S as uniform distribution on A , { p 0 ,h ( s ) } H h =1 , ∀ s ∈ S as uniform distribution on N and set { Q 0 ,h ( · , · ) } H h =1 as zero function.

2: for k = 1 , 2 , · · · , K do

3: Receive the initial state s k, 1 .

4: for h = 1 , 2 , · · · , H do

5: For all h ∈ [ H ] , s ∈ S , each base-learner B i , ∀ i ∈ [ N ] updates policy by

7: Take the action following a k,h ∼ π k,h ( · | s k,h ) and transit to the next state s k,h +1 .

8: Obtain reward r k,h ( s k,h , a k,h ) and observe the reward function r k,h ( · , · ) .

9: end for

10: Initialize V k,H +1 ( · ) as a zero function.

14: Set the estimated variance [ ¯ V k,h V k,h +1 ] ( s k,h , a k,h ) as in (11), bonus E k,h as in (21).

21: end for

22: end for

B Recovering Tabular Case

In this part, we show the result of our algorithm when specialized to the tabular case. Note in the linear case, we adopt the weighted ridge regression to estimate the parameter θ ∗ h , that is, we construct the estimator ̂ θ k,h by solving the following weighted ridge regression problem:

In the tabular case, we simply set ¯ σ j,h = 1 , and compute ϕ V k,h +1 by taking the sample mean of { V k,h +1 ( s j,h +1 ) } j ∈ [ k -1] . That is, we set it as

of each ( s, a ) , where N k counts the number of times each tuple ( s, a, s ) or ( s, a ) has been visited up to episode k . Then, we construct the estimator ̂ θ k,h by solving the linear regression problem:

Then, the confidence set ̂ C k,h is constructed as follows:

The remaining part of the algorithm is the same as the linear case. The following theorem shows the result of our algorithm in the tabular case.

Theorem 6. Set γ = 1 / ( KH ) , step size pool H = { η i = (2 i /H ) √ (log A ) /K | i ∈ [ N ] } with N = ⌊ 1 2 log( K log A ) ⌋ and β = H √ S log( dKH/δ ) , with probability at least 1 -δ , it holds

where P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ is the path length of the compared policy sequence.

Proof. By Lemma 5, we decompose the dynamic regret as follows.

Note that the policy evaluation phase is the same as the one in Fei et al. [35, Algorithm 3], by the estimator bound in Fei et al. [35, Lemmas 7 and 9], we have

The remaining proof is the same as the proof of Theorem 3 in Appendix C.7.

C Proofs

In this section, we provide the proof of the results in this paper.

C.1 Proof of Lemma 1

Proof. By the definition of dynamic regret, we have

where the equality holds by the performance difference lemma in Lemma 13 and the inequality holds by the one-step descent guarantee in Lemma 14. This finishes the proof. ■

■

C.2 Proof of Lemma 2

Proof. Note the update rule is

π ′ k,h ( · | s ) ∝ π k -1 ,h ( · | s ) exp( η · Q π k -1 k -1 ,h ( s, · )) , π k,h ( · | s ) = (1 -γ ) π ′ k,h ( · | s ) + γπ u ( · | s )

By Lemma 14, for any s h ∈ S , h ∈ [ H ] , k ∈ [ K ] , we have

where the equality holds by the definition of KL divergence. We decompose the last term as follows:

For the first term, we have

By the update rule, we have 1 ≤ 1 /π k,h ( a | s h ) ≤ A/γ and π ′ k,h ( a | s h ) /π k,h ( a | s h ) ≤ 1 / (1 -γ ) . Therefore, we have

Then, the dynamic regret is bounded as follows:

where the first inequality holds by (16), the second inequality is due to (17) and (18), and the last inequality holds by the definition of P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ . It ends the proof. ■

C.3 Proof of Lemma 3

Proof. Lemma 3 is a simplified version of Chen et al. [54, Lemma 17] and we prove it in our notations below for self-containedness. It is easy to verify that the update rule p ′ ( a ) ∝ p ( a ) · exp( η · Q ( s, a )) is equivalent to the update

Thus, by the KKT condition, we have for some λ and µ ( a ) ≥ 0 , such that

̸

The above equations give the closed-form solution p ′ ( a ) = p ( a ) exp( η ( Q ( s, a ) + λ + µ ( a ))) . First, we prove that for all a ∈ A we have µ ( a ) = 0 . Indeed, when µ ( a ) = 0 , by µ ( a ) p ′ ( a ) = 0 , we have p ′ ( a ) = p ( a ) exp( η ( Q ( s, a ) + λ + µ ( a ))) = 0 , which contradicts with p ( a ) > 0 .

Then, we now separately discuss two cases.

̸

Case 1: min a ∈A Q ( s, a ) = max a ∈A Q ( s, a ) . In this case, we first show that min a ∈A -Q ( s, a ) ≤ λ ≤ max a ∈A -Q ( s, a ) . We prove it by contradiction: If λ ≥ max a ∈A -Q ( s, a ) , then we have

contradicting with p ′ ∈ ∆( A ) . A similar argument holds for the case λ ≤ min a ∈A ( -Q ( s, i )) . Thus, we have λ ∈ [min a ∈A ( -Q ( s, a )) , max a ∈A ( -Q ( s, a ))] . Then, we have | Q ( s, a ) + λ + µ ( a ) | ≤ max a ∈A Q ( s, a ) -min a ∈A Q ( s, a ) ≤ H . By the condition on ηH ≤ 1 , we have p ′ ( a ) ∈ [exp( -1) p ( a ) , exp(1) p ( a )] ∈ [1 / (4 p ( a )) , 4 p ( a )] .

Case 2: min a ∈A Q ( s, a ) = max a ∈A Q ( s, a ) . In this case, it is clear that λ = -Q ( s, a ) must hold for all a ∈ A to make p and p ′ both discussions. Thus p ′ ( a ) = p ( a ) for all a ∈ A .

Combining the above two cases finishes the proof.

C.4 Proof of Lemma 4

To prove Lemma 4, we first introduce the following two lemmas. Denote by P π h h ( s ′ | s ) = ∑ a ∈A P h ( s ′ | s, a ) π h ( a | s ) the transition kernel of the MDP in step h under policy π h and recall that ∥ π -π ′ ∥ 1 , ∞ = max s ∈S ∥ π ( · | s ) -π ′ ( · | s ) ∥ . The first lemma shows the difference between the state distribution of two policies can be bounded by the path length of the policies.

Lemma 6 (Zhao et al. [34, Lemma 7]) . For any state distribution d , policy pair π and π ′ , and transition kernel P , we have

Proof. Consider the case when d is a delta function on s . The difference in the next distributions is

■

Linearity of expectation leads to the result for arbitrary distributions. This finishes the proof.

■

The second lemma shows the difference between the state distribution of the policy starting from the different initial distributions can be bounded by the difference between the initial distributions.

Lemma 7 (Zhao et al. [34, Lemma 8]) . For any two initial distributions d and d ′ , transition kernel P and policy π , we have

Proof. Note the relationship that d ( s ′ ) = ∑ s ∈S d ( s ) P π h s,s ′ , we have

This finishes the proof.

Now, we are ready to prove Lemma 4.

Proof of Lemma 4. For any policy π and π ′ and any initial state s 1 , denote by d h the distribution of the MDP in step h under policy π h and d ′ h the distribution of the MDP in step h under policy π ′ h , that is d h ( s h ) = E π [ 1 ( s h ) | s 1 ] and d ′ h ( s h ) = E π ′ [ 1 ( s h ) | s 1 ] . We have

where the second inequality holds by Lemmas 6 and 7 and the last inequality holds by a recursive calculation. Thus, we have

This finishes the proof.

C.5 Proof of Theorem 1

Proof. By Lemma 2, we have

■

■

Then, the last term can be bounded as follows:

where the second inequality holds by the telescoping sum and the third inequality holds because π ′ 1 ,h = π 1 ,h is the uniform distribution and π ′ k,h ( a | s ) ≥ γ 4 A by the multiplicative stability lemma in Lemma 3 and the last inequality holds by Lemma 4.

Combining (19) and (20), we have

which finishes the proof.

C.6 Proof of Theorem 2

To prove Theorem 2, we first introduce the following key lemma omitted in the main paper due to the space limit, which shows the guarantee of the confidence set.

Lemma 8 (Zhou et al. [31, Lemma 5]) . Let ̂ C k,h be defined in (10) and set ̂ β k as

Then, with probability at least 1 -3 δ , we have that simultaneously for all h ∈ [ H ] and k ∈ [ K ] ,

where E k,h is defined as follows:

We denote by E the event when the result of Lemma 8 holds, we have Pr( E ) ≥ 1 -3 δ .

Then we present the following lemma that shows the value function difference can be decomposed into two parts, one is a martingale sequence and the other is the estimation error.

Lemma 9. For all k ∈ [ K ] , h ∈ [ H ] , it holds that

with

Proof. By the definition V k,h ( s k,h ) = E a ∼ π k ( · | s k,h ) [ Q k,h ( s k,h , a )] , we have

where the third equality holds by the fact Q k,h = r k,h + P h V k,h +1 -ι k,h and Q π k k,h = r k,h + P h V π k k,h +1 . Summing up the above equation from h to H recursively finishes the proof. ■

Note M k,h, 1 is the noise from the stochastic policy and M k,h, 2 is the noise from the state transition, Let M k,h = M k,h, 1 + M k,h, 2 , ∀ k ∈ [ K ] , h ∈ [ H ] , we define two following high probability events:

According to the Azuma-Hoeffding inequality, we have Pr( E 1 ) ≥ 1 -δ and Pr( E 2 ) ≥ 1 -δ .

Then, we show the model prediction error ι k,h can be upper and lower bounded both.

Lemma 10. Define prediction error ι k,h = r k,h + P h V k,h +1 -Q k,h , on the event E , it holds that

Proof. First, we prove the left-hand side inequality. By the definition of ι k,h , we have

where the first inequality holds by the configuration of Q k,h in Algorithm 1, the second equality holds by the definition of linear mixture MDP such that [ P h V k,h +1 ]( s, a ) = ⟨ ϕ V k,h +1 ( s, a ) , θ ∗ h ⟩ and the last inequality holds by the configuration of confidence set in Lemma 8.

Next, we prove the right-hand side inequality. By the definition of ι k,h , we have

where the first inequality holds by the configuration of Q k,h in Algorithm 1, the second equality holds by the definition of linear mixture MDP such that [ P h V k,h +1 ]( s, a ) = ⟨ ϕ V k,h +1 ( s, a ) , θ ∗ h ⟩ and the last inequality holds by the configuration of confidence set in Lemma 8. ■

Next, we show the prediction error can be bounded by the cumulative estimate variance.

Lemma 11. Define prediction error ι k,h = r k,h + P h V k,h +1 -Q k,h , on the event E , it holds that

Proof. By Lemma 10 and the definition of ι k,h = r k,h + P h V k,h +1 -Q k,h , we have

where the second inequality holds by 2 ̂ β k ¯ σ k,h ≥ √ dH/ √ d = H , the third inequality is by CauchySchwarz inequality and the last inequality holds by the elliptical potential lemma in Lemma 16. ■

Define the event

by the law of total variance in Lemma 15, we have Pr( E 3 ) ≥ 1 -δ . Then, we have the following lemma which bounds the estimated variance of the value function.

Lemma 12 (He et al. [33, Lemma 6.5]) . On the events E ∩ E 1 ∩ E 2 ∩ E 3 , it holds that

Now, we are ready to prove Theorem 2.

Proof of Theorem 2. On the events E ∩ E 1 ∩ E 2 ∩ E 3 , for any h ∈ [ H ] , it holds that

where the first equality holds by Lemma 9, the second inequality holds by Lemma 11, and the last inequality holds by Lemma 12. This finishes the proof. ■

C.7 Proof of Theorem 3

Proof. By Lemma 5, we can rewrite the dynamic regret as follows.

By Lemma 10, we have ι k,h ( s, a ) ≤ 0 for any k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A . Thus, we have

It remains to bound the first term. Note our algorithm is indeed updated based on the estimated action-value function Q k,h . By Theorem 1, set γ = 1 /KH and note that log(1 / (1 -γ )) ≤ γ/ (1 -γ ) for all γ > 0 . Then, we have

It is clear that the optimal step size is η ∗ = √ ( P T +log A ) / ( KH 2 ) . Note our step size is set as η = min { √ ( P T +log A ) /K, 1 } /H to ensure η ≤ 1 /H . We consider the following two cases:

Case 2: η ∗ > 1 /H . In this case, our step size is set as η = 1 /H . Therefore, we have

where the equality holds by P T ≥ K in this case. Combining these two cases, we have

Combining (22), (23), (24) and (25), we obtain

This finishes the proof.

C.8 Proof of Theorem 4

Proof. At a high level, we prove this lower bound by noting that optimizing the dynamic regret of linear mixture MDPs is harder than (i) optimizing the static regret of linear mixture MDPs with the unknown transition kernel, (ii) optimizing the dynamic regret of linear mixture MDPs with the known transition kernel, both. Thus, we can consider the lower bound of these two problems separately and combine them to obtain the lower bound of the dynamic regret of linear mixture MDPs with the unknown transition kernel. We present the details below.

First, we consider the lower bound of optimizing the static regret of adversarial linear mixture MDPs with the unknown transition kernel. From lower bound in He et al. [33, Theorem 5.3], we have the following lower bound in this case since the dynamic regret is no smaller than the static regret.

Then, we consider the lower bound of optimizing the dynamic regret of adversarial linear mixture MDPs with the known transition kernel. We note that Zimin and Neu [12] show the lower bound of the static regret for adversarial episodic loop-free SSP with known transition kernel is Ω( H √ K log ( SA )) , we utilize this lower bound to establish our lower bound as the episodic loopfree SSP is a special case of linear mixture MDPs with d = S 2 A . We consider two cases:

Case 1: Γ ≤ 2 H . In this case, we can directly utilize the established lower bound of static regret as a natural lower bound of dynamic regret,

Case 2: Γ > 2 H . Without loss of generality, we assume L = ⌈ Γ / 2 H ⌉ divides K and split the whole episodes into L pieces equally. Next, we construct a special policy sequence such that the policy sequence is fixed within each piece and only changes in the split point. Since the sequence changes at most L -1 ≤ Γ / 2 H times and the path length of the policy sequence at each change point is at most 2 H , the total path length in K episodes does not exceed Γ . As a result, we have

Combining (27) and (28), we have the following lower bound for the dynamic regret of adversarial linear mixture MDPs with the known transition kernel,

where the last inequality holds by max { a, b } ≥ ( a + b ) / 2 .

Combining two lower bounds (26) and (29), we have the lower bound of the dynamic regret of adversarial linear mixture MDPs with the unknown transition kernel,

This finishes the proof.

■

■

C.9 Proof of Theorem 5

Proof. By Lemma 5, we can rewrite the dynamic regret as follows.

By Lemma 10, we have ι k,h ( s, a ) ≤ 0 for any k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A . Thus, we have

It remains to bound the first term. We decompose this term as follows.

where the decomposition holds for any base-learner i ∈ N . Next, we bound the two terms separately.

Upper bound of base regret. From Theorem 1, we have

It is clear that the optimal learning rate η ∗ = √ 4(log A + P T log( KHA )) / ( KH 2 ) . By the definition P T = ∑ K k =1 ∑ H h =1 ∥ π c k,h -π c k -1 ,h ∥ 1 , ∞ , it holds that 0 ≤ P T ≤ 2 KH . Therefore, the range of the optimal learning rate is

From the construction of the step size pool H = { η i = (2 i /H ) √ (log A ) /K | i ∈ [ N ] } with N = ⌊ 1 2 log( K log A ) ⌋ , we know that the step size therein is monotonically increasing, in particular

In the following, we consider two cases:

Case 1: η ∗ ∈ [ η 1 , η N ] . In this case, we ensure there exists i ∗ ∈ N such that η i ∗ ≤ η ∗ ≤ 2 η i ∗ . Note the decomposition in (30) holds for any base-learner. Therefore, we choose the base-learner whose step size is η i ∗ and have

where the second inequality holds by the condition that η i ∗ ≤ η ∗ ≤ 2 η i ∗ and the last equality holds by substituting η ∗ = √ 4(log A + P T log( KHA )) / ( KH 2 ) .

Case 2: η ∗ > η N . In this case, we know that 4(log A + P T log( KHA )) > K . Therefore, we choose the base-learner whose step size is η N and have

where the last inequality holds by the condition that 4(log A + P T log( KHA )) > K .

Summing over the two upper bounds yields

where the inequality holds by a + b ≤ √ 2( a + b ) , ∀ a, b ≥ 0 .

Upper bound of meta regret. For meta-regret, we have

̸

Combining (31) and (32), by √ a + √ b ≤ √ 2( a + b ) , ∀ a, b ≥ 0 , we have where the first inequality holds by the definition that i ∗ k,h = arg max i ∈ [ N ] ⟨ Q k,h ( s h , · ) , π i k,h ( · | s h ) ⟩ , the second inequality holds due to p i k,h ( s h ) = e i ∗ k -1 ,h ( s h ) , and the last equality holds by the definition S T = ∑ K k =1 ∑ H h =1 E π c k 1 [ i ∗ k,h ( s h ) = i ∗ k -1 ,h ( s h )] .

This finishes the proof.

■

D Supporting Lemmas

In this section, we introduce the supporting lemmas used in the proofs.

First, we introduce the performance difference lemma which connects the difference between two policies to the difference between their expected total rewards through the Q-function.

Lemma 13 (Cai et al. [32, Lemma 3.2]) . For any policies π, π ′ ∈ ∆( A| S , H ) , it holds that

Then, we introduce the following lemmas which show the 'one-step descent' guarantee.

Lemma 14 (Cai et al. [32, Lemma 3.3]) . For any distributions p ∗ , p ∈ ∆( A ) , state s ∈ S , and function Q : S × A → [0 , H ] , it holds for p ′ ∈ ∆( A ) with p ′ ( · ) ∝ p ( · ) · exp( η · Q ( s, · )) that

Next, we introduce the law of total variance, which bounds the variance of the value function.

Lemma 15 (Jin et al. [55, Lemma C.5]) . With probability at least 1 -δ , it holds that

Finally, we introduce the elliptical potential lemma, which is a key lemma in online linear regression. Lemma 16 (Abbasi-Yadkori et al. [51, Lemma 11]) . Let { x t } ∞ t =1 be a sequence in R d space, V 0 = λ I and define V t = V 0 + ∑ t i =1 x i x ⊤ i . If ∥ x i ∥ 2 ≤ L, ∀ i ∈ Z + , then for each t ∈ Z + ,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studiess adversarial RL under episodic linear mixture MDPs with adversarial full-information reward and stationary unknown transition kernels. This paper first proposes a new algorithm to deal with the adversarial reward and then provides an upper bound of dynamic regret for the proposed algorithm. Then this paper proposes a lower bound for dynamic regret and claims acheiving optimal in terms of the number of episodes $K$ and the non-stationary measure $P_T$.\n\nSTRENGTHS:\n1. This paper provides a noval algorithm to deal with adversarial RL without prior knowledge. The main steps of the algorithm seem reasonable and the paper provides detailed insight of each steps.\n2. This paper provide an upper bound of dynamic regret and the dependenece of $K$ in upper bound achieves optimal.\n\nWEAKNESSES:\n1. The algorithm and result of this paper only apply to finite state case, which makes the result less significant. \n\n2. The proof of the lower bound lacks rigor in several aspects. Firstly, the proof of the lower bound is divided into two separate cases, which is not in line with the definition of a lower bound. Ideally, the author should construct one hard instance that encompasses both cases. Secondly, in the proof of the second case (dynamic regret case), the proof merely takes the maximum between the results of the two cases with respect to the range of $\\Gamma$. \n\n3. Due to the existence of numerous studies employing alternative approaches to address non-stationarity and yield diverse results, I think more comprehensive and detialed discussions with non-stationary RL are needed. For example, although line 108 indicates that [1] is not applicable to the adversarial setting when the comparators are arbitrarily chosen. In fact, people are usually concerned about the case where camparators are optimal policy. In this case, does the non-stationary RL discussed in [1] cover the adversarial RL in this paper? It is important to discuss the relationship between the findings of [1] and this paper in such cases. [1] also achieves optimal regret with $K^{2/3}\\Delta^{1/3}+\\sqrt{K}$, where $\\Delta$ denotes another non-stationary measure. Why does the dependence on $K$ differ? From my understanding, this discrepancy seems to arise from distinct non-stationary measures. \n\n\n[1] Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Conference on Learning Theory (COLT), pages 4300–4354, 2021.\n\nQUESTIONS:\nAs listed in weakness.\n\nLIMITATIONS:\nThis work does not pose any negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the problem of reinforcement learning in non-homogeneous MDPs with adversarial full-information reward feedback and unknown transition kernels. The authors propose a new algorithm that has advantages in dynamic regret and does not require prior knowledge as input. The algorithm achieves optimal performance in certain cases. The paper provides a theoretical foundation for reinforcement learning in non-homogeneous MDPs by studying the upper and lower bounds of dynamic regret.\n\nSTRENGTHS:\nThis work investigates the dynamic regret in linear mixture MDPs with adversarial rewards, which is a problem of great significance and relevance. The authors make notable contributions by offering a precise upper bound on regret as well as a lower bound for this problem. To derive the upper bound, the authors introduce new techniques, including the fix-share mechanism and a multiplicative stability lemma, which hold potential value and may be of independent interest to researchers in related fields.\n\nWEAKNESSES:\n- The authors have not included a highly related work titled \"Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs,\" which addresses the more challenging setting of linear mixture MDPs with adversarial rewards and non-stationary transitions.\n\n- When comparing this work with both (Fei et al., 2020) and \"Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs,\" it is worth noting that this work demonstrates improvements in terms of the parameters K and P_T. However, it introduces an additional dependency on S_T. Since S_T can be linear in K in the worst-case scenario, the resulting regret bounds may not strictly improve upon the previous findings.\n\n- Considering the omission of the aforementioned related work and the potential limitations of the current results in terms of the additional dependency on S_T, it may be valuable for the authors to discuss these aspects in the paper. This would help readers gain a comprehensive understanding of the work's contributions and its relationship to the existing literature.\n\nQUESTIONS:\n- While the fix-share mechanism and multiplicative stability lemma used in this work aim to enhance the regret upper bound, it is important to assess whether these techniques strictly improve upon existing works. The authors should provide a clear analysis or comparison demonstrating how their approach surpasses or builds upon the achievements of previous methods. If the improvements are not strictly superior but rather offer refinements or alternative perspectives, they should be clearly stated to avoid potential misconceptions.\n\n- In addition, offering further explanations on why the reduction-based framework in (Wei and Luo, 2020) is not applicable in adversarial linear mixture MDPs, even with full-information feedback, would add value to the paper. Discussing the specific challenges or characteristics of adversarial linear mixture MDPs that render the reduction-based framework ineffective or impractical will enable readers to grasp the unique complexities of the problem domain. By providing these explanations, the authors can highlight the significance of their proposed approach and its suitability for addressing the specific challenges posed by adversarial linear mixture MDPs.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Dynamic Regret of Adversarial Linear Mixture MDPs\" studies reinforcement learning in episodic non homogeneous MDPs with adversarial rewards in full information feed back setting. The motivation for this work is that In many applications, reward functions may be picked adversarially, and changing over time. Moreover, the state and action spaces may be huge. Hence, this paper considers adversarial rewards and linear function approximation. Previous work lacks in one of these requirements. The authors analyse that their algorithm enjoys a \"dynamic regret\" which is close to \"optimal\" by deriving a lower bound, and can recover the static setting, and improves the adversarial reward regret for tabular MDPs.\n\nSTRENGTHS:\nA novel algorithm is proposed that achieves good dynamic regret, under non stationary adversarial regime, without the knowledge of the non-stationarity measure, which poses many challenges for the analysis. \n\nThe analysis in the paper successfully overcame these and obtained upper bound and a matching lower bound.\n\nStatic case can be exactly recovered from these results, and in case of tabular case, it strictly improves the existing bound.\n\nWEAKNESSES:\nThough out of scope for this work, I would like to know the author's perspective on the following (since they also mentioned these in the paper).\n\n1. This work considers full information setting, bandit setting is not considered, and much more complex.\n\n2. The dependence on S is bad in large state space regime, and on H when the non-stationary measure is large. These could not be avoided from the analysis.\n\nQUESTIONS:\nPlease refer to above section. I appreciate if these are discussed further by the authors.\n\nAnother question is in Remark 2. How do we retrieve the static case from Theorem 1?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study analyzes the problem of online learning episodic inhomogeneous MDPs with _full-information_ reward functions (changed over episodes $k$ and step $h$) and the unknown transition kernel (unchanged over episodes). The MDPs are linear mixture MDPs and the performance measure is _dynamic_ regret (against _any sequence of policies_ with a non-stationarity measure of $P_T$ which controls how varied the sequence is). The proposed algorithm improves on existing upper bounds while relaxing on the requirement of prior knowledge (that of $P_T$). A lower bound is also provided which suggests optimality of the proposed algorithm on a few problem parameters.\n\nThe presentation is excellent and the proposed algorithm seems interesting. I recommend its acceptance.\n\nSTRENGTHS:\n1. The presentation is excellent and technically substantive. I am not an expert on the exact topic but feel comfortable following most of the discussion.\n1. The proposed algorithm is interesting and uses techniques from other (recent) works (some from adjacent fields).\n\nWEAKNESSES:\n1. From the point of view of someone who is not intimately familiar with the exact topic, the problem setting might seem a bit artificial/too restrictive, e.g., full information of reward functions. But it is clear from the cited works that many features of the setup are up-to-date with what the community is actively studying. The authors also acknowledged out some of such limitations.\n\nQUESTIONS:\n1. Can you provide some intuition of what the switching number of best base-learner, $S_T$, measures? I found it challenging to grasp due to the presence of the expectation taken over (counterfactual) state distribution generated by $\\pi^c_k$. Are there some regimes for this parameter that are interesting (and provide intuitive connections to existing studies)?\n\nLIMITATIONS:\nAdequate.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}

comments: SUMMARY:
The paper studies RL in the adversarial episodic inhomogeneous MDP setting with transition kernels. It should be noted that the paper considers full-information feedback & linear mixture MDPs (both are very strong assumptions). The authors propose an algorithm drawing inspiration from policy optimization and prediction with expert advice problems (hence the full-information feedback setting). An upper bound on the dynamic regret is provided. The authors also offer an accompanying lower bound.

STRENGTHS:
The paper is generally well-written and seems technically sound. An extensive literature review is provided and the authors offer comparison and discussion with closely related works. The regret bound improves upon those in existing works, and is near-optimal.

WEAKNESSES:
Among the key technical innovations listed in the introduction compared to existing works (Fei et al [2], Cai et al [34]), I feel like the extension from tabular MDPs to linear mixtures MDPs not technically challenging, given the fact that proximal policy optimization can be naturally extended beyond the tabular setting. In that case, the real contribution lies in the improvement of the regret bounds. Could the authors highlight what are the key innovations (either in the algorithms or in the analysis) that lead to this improvement?

QUESTIONS:
Could the authors highlight what are the key innovations (either in the algorithms or in the analysis) that lead to this improvement?

LIMITATIONS:
The authors discuss some limitations & future works in the conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21