Title: Provably Efficient Algorithm for Nonstationary Low-Rank MDPs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Nonstationary Environment

Representation Learning

Policy Optimization

Statistical Complexity
Abstract: Reinforcement learning (RL) under changing environment models many real-world applications via nonstationary Markov Decision Processes (MDPs), and hence gains considerable interest. However, theoretical studies on nonstationary MDPs in the literature have mainly focused on tabular and linear (mixture) MDPs, which do not capture the nature of unknown representation in deep RL. In this paper, we make the first effort to investigate nonstationary RL under episodic low-rank MDPs, where both transition kernels and rewards may vary over time, and the low-rank model contains unknown representation in addition to the linear state embedding function. We first propose a parameter-dependent policy optimization algorithm called PORTAL,
and further improve PORTAL to its parameter-free version of Ada-PORTAL, which is able to tune its hyper-parameters adaptively without any prior knowledge of nonstationarity. For both algorithms, we provide upper bounds on the average dynamic suboptimality gap, which show that as long as the nonstationarity is not significantly large, PORTAL and Ada-PORTAL are sample-efficient and can achieve arbitrarily small average dynamic suboptimality gap with polynomial sample complexity.

Provably Efficient Algorithm for Nonstationary Low-Rank MDPs

Abstract

Reinforcement learning (RL) under changing environment models many real-world applications via nonstationary Markov Decision Processes (MDPs), and hence gains considerable interest. However, theoretical studies on nonstationary MDPs in the literature have mainly focused on tabular and linear (mixture) MDPs, which do not capture the nature of unknown representation in deep RL. In this paper, we make the first effort to investigate nonstationary RL under episodic low-rank MDPs, where both transition kernels and rewards may vary over time, and the low-rank model contains unknown representation in addition to the linear state embedding function. We first propose a parameter-dependent policy optimization algorithm called PORTAL, and further improve PORTAL to its parameter-free version of Ada-PORTAL, which is able to tune its hyper-parameters adaptively without any prior knowledge of nonstationarity. For both algorithms, we provide upper bounds on the average dynamic suboptimality gap, which show that as long as the nonstationarity is not significantly large, PORTAL and Ada-PORTAL are sample-efficient and can achieve arbitrarily small average dynamic suboptimality gap with polynomial sample complexity.

1 Introduction

Reinforcement learning (RL) has gained significant success in real-world applications such as board games of Go and chess (Silver et al., 2016, 2017, 2018), robotics (Levine et al., 2016; Gu et al., 2017), recommendation systems (Zhao et al., 2021) and autonomous driving (Bojarski et al., 2016; Ma et al., 2021). Most theoretical studies on RL have been focused on a stationary environment and evaluated the performance of an algorithm by comparing against only one best fixed policy (i.e., static regret ). However, in practice, the environment is typically time-varying and nonstationary . As a result, the transition dynamics, rewards and consequently the optimal policy change over time.

There has been a line of research studies that investigated nonstationary RL. Specifically, Gajane et al. (2018); Cheung et al. (2020); Mao et al. (2021) studied nonstationary tabular MDPs. To further overcome the curse of dimensionality, Fei et al. (2020); Zhou et al. (2020) proposed algorithms for nonstationary linear (mixture) MDPs and established upper bounds on the dynamic regret.

In this paper, we significantly advance this line of research by investigating nonstationary RL under low-rank MDPs (Agarwal et al., 2020b), where the transition kernel of each MDP admits a decomposition into a representation function and a state-embedding function that map to low dimensional spaces. Compared with linear MDPs where the representation is known, the lowrank MDP model contains unknown representation, and is hence much more powerful to capture

representation learning that occurs often in deep RL. Although there have been several recent studies on static low-rank MDPs (Agarwal et al., 2020b; Uehara et al., 2022; Modi et al., 2021), nonstationary low-rank MDPs remain unexlored, and are the focus of this paper.

To investigate nonstationary low-rank MDPs, several challenges arise. (a) All previous studies of nonstationary MDPs took on-policy exploration, such a strategy will have difficulty in providing sufficiently accurate model (as well as representation) learning for nonstationary low-rank MDPs. (b) Under low-rank MDPs, since both representation and state-embedding function change over time, it is more challenging to use history data collected under previous transition kernels for current use.

The main contribution of this paper lies in addressing above challenges and designing a provably efficient algorithm for nonstationary low-rank MDPs. We summarize our contributions as follows.

We propose a novel policy optimization algorithm with representation learning called PORTAL for nonstationary low-rank MDPs. PORTAL features new components, including off-policy exploration, data-transfer model learning, and target policy update with periodic restart.

We theoretically characterize the average dynamic suboptimality gap ( Gap Ave ) of PORTAL, where Gap Ave serves as a new metric that captures the performance of target policies with respect to the best policies at each instance in the nonstationary MDPs under off-policy exploration. We further show that with prior knowledge on the degree of nonstationarity, PORTAL can select hyper-parameters that minimize Gap Ave . If the nonstationarity is not significantly large, PORTAL enjoys a diminishing Gap Ave with respect to the number of iterations K , indicating that PORTAL can achieve arbitrarily small Gap Ave with polynomial sample complexity.

Our analysis features a few new developments. (a) We provide a new MLE guarantee under nonstationary transition kernels that captures errors of using history data collected under different transition kernels for benefiting current model estimation. (b) We establish trajectory-wise uncertainty bound for estimation errors via a square-root ℓ ∞ -norm of variation budgets. (c) We develop an error tracking technique via auxiliary anchor representation for convergence analysis.

Finally, we improve PORTAL to a parameter-free algorithm called Ada-PORTAL, which does not require prior knowledge on nonstationarity and is able to tune the hyper-parameters adaptively. We further characterize Gap Ave of Ada-PORTAL as ˜ O ( K -1 6 (∆ + 1) 1 6 ) , where ∆ captures the variation of the environment. Notably, based on PORTAL, we can also use the black-box method called MASTER in Wei & Luo (2021) to turn PORTAL into a parameter-free algorithm (called MASTER+PORTAL) with Gap Ave of ˜ O ( K -1 6 ∆ 1 3 ) . Clearly, Ada-PORTAL performs better than MASTER+PORTAL when nonstationarity is not significantly small, i.e. ∆ ≥ ˜ O (1) .

To our best knowledge, this is the first study of nonstationary RL under low-rank MDPs.

2 Related Works

Various works have studied nonstationary RL under tabular and linear MDPs, most of which can be divided into two lines: policy optimization methods and value-based methods.

Nonstationary RL: Policy Optimization Methods. As a vast body of existing literature (Cai et al., 2020; Shani et al., 2020; Agarwal et al., 2020a; Xu et al., 2021) has proposed policy optimization methods attaining computational efficiency and sample efficiency simultaneously in stationary RL under various scenarios, only several papers investigated policy optimization algorithm in nonstationary environment. Assuming time-varying rewards and time-invariant transition kernels, Fei et al. (2020) studied nonstationary RL under tabular MDPs. Zhong et al. (2021) assumed both transition kernels and rewards change over episodes and studied nonstationary linear mixture MDPs. These policy optimization methods all assumed prior knowledge on nonstationarity.

Nonstationary RL: Value-based Methods. Assuming that both transition kernels and rewards are time-varying, several works have studied nonstationary RL under tabular and linear MDPs, most of which adopted Upper Confidence Bound (UCB) based algrithms. Cheung et al. (2020) investigated tabular MDPs with infinite-horizon and proposed algorithms with both known variation budgets and unknown variation budgets. In addition, this work also proposed a Bandit-over-Reinforcement Learning (BORL) technique to deal with unknown variation budgets. Mao et al. (2021) proposed a model-free algorithm with sublinear dynamic regret bound. They then proved a lower bound for nonstationary tabular MDPs and showed that their regret bound is near min-max optimal. Touati

& Vincent (2020); Zhou et al. (2020) considered nonstationary RL in linear MDPs and proposed algorithms achieving sublinear regret bounds with unknown variation budgets.

Besides these two lines of researches, Wei & Luo (2021) proposed a black-box method that turns a RL algorithm with optimal regret in a (near-)stationary environment into another algorithm that can work in a nonstationary environment with sublinear dynamic regret without prior knowledge on nonstationarity. In this paper, we show that our algorithm Ada-PORTAL outperforms such a type of black-box method (taking PORTAL as subroutine) if nonstationarity is not significantly small.

Stationary RL under Low-rank MDPs. Low-rank MDPs were first studied by Agarwal et al. (2020b) in a reward-free regime, and then Uehara et al. (2022) studied low-rank MDPs for both online and offline RL with known rewards. Cheng et al. (2023) studied reward-free RL under low-rank MDPs and improved the sample complexity of previous works. Modi et al. (2021) proposed a model-free algorithm MOFFLE under low-nonnegative-rank MDPs. Cheng et al. (2022); Agarwal et al. (2022) studied multitask representation learning under low-rank MDPs, and further showed the benefit of representation learning to downstream RL tasks.

3 Formulation

Notations: We use [ K ] to denote set { 1 , . . . , K } for any K ∈ N , use ∥ x ∥ 2 to denote the ℓ 2 norm of vector x , use △ ( A ) to denote the probability simplex over set A , use U ( A ) to denote uniform sampling over A , given |A| < ∞ , and use △ ( S ) to denote the set of all possible density distributions over set S . Furthermore, for any symmetric positive definite matrix Σ , we let ∥ x ∥ Σ := √ x ⊤ Σ x . For distributions p 1 and p 2 , we use D KL ( p 1 ( · ) ∥ p 2 ( · )) to denote the KL divergence between p 1 and p 2 .

3.1 Episodic MDPs and Low-rank Approximation

An episodic MDP is denoted by a tuple M := ( S , A , H, P := { P h } H h =1 , r := { r h } H h =1 ) , where S is a possibly infinite state space, A is a finite action space with cardinality A , H is the time horizon of each episode, P h ( ·|· , · ) : S × A → ∆( S ) denotes the transition kernel at each step h , and r h ( · , · ) : S × A → [0 , 1] denotes the deterministic reward function at each step h . We further normalize the reward as ∑ H h =1 r h ≤ 1 . A policy π = { π h } h ∈ [ H ] is a set of mappings where π h : S → ∆( A ) . For any ( s, a ) ∈ S × A , π h ( a | s ) denotes the probability of selecting action a at state s at step h . For any ( s, a ) ∈ S × A , let ( s h , a h ) ∼ ( P, π ) denote that the state s h is sampled by executing policy π to step h under transition kernel P and then action a h is sampled by π h ( ·| s h ) .

Given any state s ∈ S , the value function for a policy π at step h under an MDP M is defined as the expected value of the accumulative rewards as: V π h,P,r ( s ) = ∑ H h ′ = h E ( s h ′ ,a h ′ ) ∼ ( P,π ) [ r h ′ ( s h ′ , a h ′ ) | s h = s ] . Similarly, given any state-action pair ( s, a ) ∈ S × A , the action-value function ( Q -function) for a policy π at step h under an MDP M is defined as Q π h,P,r ( s, a ) = r h ( s, a ) + ∑ H h ′ = h +1 E ( s h ′ ,a h ′ ) ∼ ( P,π ) [ r h ′ ( s h ′ , a h ′ ) | s h = s, a h = a ] . Denote ( P h f )( s, a ) := E s ′ ∼ P h ( ·| s,a ) [ f ( s ′ )] for any function f : S → R . Then we can write the actionvalue function as Q π h,P,r ( s, a ) = r h ( s, a ) + ( P h V π h +1 ,P,r )( s, a ) . For any k ∈ [ K ] , without loss of generality, we assume the initial state s 1 to be fixed and identical, and we use V π P,r to denote V π 1 ,P,r ( s 1 ) for simplicity.

This paper focuses on low-rank MDPs (Jiang et al., 2017; Agarwal et al., 2020b) defined as follows.

Definition 3.1 (Low-rank MDPs) . A transition kernel P ∗ h : S × A → △ ( S ) admits a low-rank decomposition with dimension d ∈ N if there exist a representation function ϕ ⋆ h : S × A → R d and a state-embedding function µ ⋆ h : S → R d such that

Without loss of generality, we assume ∥ ϕ ∗ h ( s, a ) ∥ 2 ≤ 1 for all ( s, a ) ∈ S × A and for any function g : S ↦→ [0 , 1] , ∥ ∥ ∫ µ ⋆ h ( s ) g ( s ) ds ∥ ∥ 2 ≤ √ d . An MDP is a low-rank MDP with dimension d if for any h ∈ [ H ] , its transition kernel P ∗ h admits a low-rank decomposition with dimension d . Let ϕ ⋆ = { ϕ ⋆ h } h ∈ [ H ] and µ ⋆ = { µ ⋆ h } h ∈ [ H ] be the true representation and state-embedding functions.

3.2 Nonstationary Transition Kernels with Adversarial Rewards

In this paper, we consider an episodic RL setting under changing environment, where both transition kernels and rewards vary over time and possibly in an adversarial fashion.

Specifically, suppose the RL system goes by rounds , where each round have a fixed number of episodes, and the transition kernel and the reward remain the same in each round, and can change adversarially across rounds. For each round, say round k , we denote the MDP as M k = ( S , A , H, P k := { P ⋆,k h } H h =1 , r k := { r k h } H h =1 ) , where P ⋆,k and r k denote the true transition kernel and the reward of round k . Further, P ⋆,k takes the low-rank decomposition as P ⋆,k = ⟨ ϕ ⋆,k , µ ⋆,k ⟩ . Both the representation function ϕ ⋆,k and the state embedding function µ ⋆,k can change across rounds. Given the reward function r k , there always exists an optimal policy π ⋆,k that yields the optimal value function V π ⋆,k P ⋆,k ,r k = sup π V π P ⋆,k ,r k , abbreviated as V ⋆ P ⋆,k ,r k . Clearly, the optimal policy also changes across rounds.

We assume the agent interacts with the nonstationary environment (i.e., the time-varying MDPs) over K rounds in total without the knowledge of transition kernels { P k,⋆ } K k =1 . At the beginning of each round k , the environment changes to a possibly adversarial transition kernel unknown to the agent, picks a reward function r k , which is revealed to the agent only at the end of round k , and outputs a fixed initial state s 1 for the agent to start the exploration of the environment for each episode. The agent is allowed to interact with MDPs via a few episodes with one or multiple exploration policies at her choice to take samples from the environment and then should output an target policy to be executed during the next round. Note that in our setting, the agent needs to decide exploration and target policies only based on the information in previous rounds, and hence exploration samples and the reward information of the current round help only towards future rounds.

3.3 Learning Goal and Evaluation Metric

In our setting, the agent seeks to find the optimal policy at each round k (with only the information of previous rounds), where both transition kernels and rewards can change over rounds. Hence we define the following notion of average dynamic suboptimality gap to measure the convergence of the target policy series to the optimal policy series.

Definition 3.2 (Average Dynamic Suboptimality Gap) . For K rounds, and any policy set { π k } k ∈ [ K ] , the average dynamic suboptimality gap (Gap Ave ) of the value functions over K rounds is given as Gap Ave ( K ) = 1 K ∑ K k =1 [ V ⋆ P ⋆,k ,r k -V π k P ⋆,k ,r k ] . For any ϵ , we say an algorithm is ϵ -average suboptimal, if it outputs a policy set { π k } k ∈ [ K ] satisfying Gap Ave ( K ) ≤ ϵ .

Gap Ave compares the agent's target policy to the optimal policy of each individual round in hindsight, which captures the dynamic nature of the environment. This is in stark contrast to the stationary setting where the comparison policy is a single fixed best policy over all rounds. This notion is similar to dynamic regret used for nonstationary RL (Fei et al., 2020; Gajane et al., 2018), where the only difference is that Gap Ave evaluates the performance of target policies rather than the exploration policies. Hence, given any target accuracy ϵ ≥ 0 , the agent is further interested in the statistical efficiency of the algorithm, i.e., using as few trajectories as possible to achieve ϵ -average suboptimal.

4 Policy Optimization Algorithm and Theoretical Guarantee

4.1 Base Algorithm: PORTAL

We propose a novel algorithm called PORTAL (Algorithm 1), which features three main steps. Below we first summarize our main design ideas and then explain reasons behind these ideas as we further elaborate main steps of PORTAL.

Summary of New Design Ideas: PORTAL features the following main design ideas beyond previous studies on nonstationary RL under tabular and linear MDPs. (a) PORTAL features a specially designed off-policy exploration which turns out to be beneficial for nonstationary low-rank /MDP models rather than the typical on-policy exploration taken by previous studies of nonstationary tabular and linear MDP models. (b) PORTAL transfers history data collected under various different transition kernels for benefiting the estimation of the current model. (c) PORTAL updates target

policies with periodic restart. As a comparison, previous work using periodic restart (Fei et al., 2020) chooses the restart period τ based on a certain smooth visitation assumption. Here, we remove such an assumption and hence our choice of τ is applicable to more general model classes.

Step 1. Off-Policy Exploration for Data Collection: We take off-policy exploration, which is beneficial for nonstationary low-rank MDPs than simply using the target policy for on-policy exploration taken by the previous studies on nonstationary tabular or linear (mixture) MDPs (Zhong et al., 2021; Fei et al., 2020; Zhou et al., 2020). To further explain, we first note that under tabular or linear (mixture) MDPs studied in the previous work, a bonus term is introduced to the actual reward to serve as a point-wise uncertainty level of the estimation error for each state-action pair at any step h , so that for any step h , ˆ Q k h is a good optimistic estimation for Q π h,P ⋆,k ,r k . Hence it suffices to collect samples using the target policy. However, in low-rank MDPs, the bonus term ˆ b k h cannot serve as a point-wise uncertainty measure. For step h ≥ 2 , ˆ Q k h is not a good optimistic estimation for the true value function if the agent only uses target policy to collect data (i.e., for on-policy exploration). Hence, more samples and a novel off-policy exploration are required for a good estimation under low-rank MDPs. Specifically, as line 5 in Algorithm 1, at the beginning of each round k , for each step h ∈ [ H ] , the agent explores the environment by executing the exploration policy ˜ π k -1 to state ˜ s k,h h -1 and then taking two uniformly chosen actions 1 , where ˜ π k -1 is determined in Step 2 of the previous round.

Algorithm 1 PORTAL ( P olicy O ptimization with R epresen TA tion L earning under nonstationary MDPs)

1: Input: Rounds K , hyper-parameters τ, W , regularizer λ k,W , coefficient ˜ α k,W , stepsize η and models { Ψ , Φ } .

2: Initialization: π 0 ( ·| s ) to be uniform; ˜ D (0 , 0) = ∅

h . 3: for episode k = 1 , . . . , K do 4: for step h = 1 , . . . , H do 5: Roll into ˜ s ( k,h ) h -1 using ˜ π k -1 , uniformly choose ˜ a ( k,h ) h -1 , ˜ a ( k,h ) h , and enter into ˜ s ( k,h ) h , ˜ s ( k,h ) h +1 . 6: Update datasets ˜ D ( k,h,W ) h -1 = { ˜ s ( i,h ) h -1 , ˜ a ( i,h ) h -1 , ˜ s ( i,h ) h } k i =1 ∨ k -W +1 , ˜ D ( k,h,W ) h = { ˜ s ( i,h ) h , ˜ a ( i,h ) h , ˜ s ( i,h ) h +1 } k i =1 ∨ k -W +1 . 7: end for 8: Receive full information rewards r k = { r k h } h ∈ [ H ] . 9: Estimate transition kernel and update the exploration policy ˜ π k for the next round via: E 2 U ( k, { ˜ D ( k,h,W ) h -1 } , { ˜ D ( k,h,W ) h } ) . 10: for step h = 1 , . . . , H do 11: Update ˆ Q k h = Q π k h, ˆ P k ,r k . 12: end for 13: if k mod τ = 1 then 14: Set { ˆ Q k h } h ∈ [ H ] as zero functions and { π k h } h ∈ [ H ] as uniform distributions on A . 15: end if 16: for step h = 1 , . . . , H do 17: Update the target policy as in Equation (2). 18: end for 19: end for 20: Output: { π k } K k =1 .

Step 2. Data-Transfer Model Learning and E 2 U: In this step, we transfer history data collected under previous different transition kernels for benefiting the estimation of the current model. This is theoretically grounded by our result that the model estimation error can be decomposed into variation

budgets plus a diminishing term as the estimation sample size increases, which justifies that the usage of data generated by mismatched distributions within a certain window is beneficial for model learning as long as variation budgets is mild. Then, the estimated model will further facilitate the selection of future exploration policies accurately.

Specifically, the agent selects desirable samples only from the latest W rounds following a forgetting rule (Garivier & Moulines, 2011). Since nonstationary low-rank MDPs (compared to tabular and linear MDPs) also have additional variations on representations over time, the choice of W needs to incorporate such additional information. Then the agent passes these selected samples to a subroutine E 2 U(see Algorithm 2), in which the agent estimates the transition kernels via the maximum likelihood estimation (MLE). Next, the agent updates the empirical covariance matrix ˆ U k,W and explorationdriven bonus ˆ b k as in lines 4 and 5 in Algorithm 2. We then define a truncated value function iteratively using the estimated transition kernel and the exploration-driven reward as follows:

Although the bonus term ˆ b k h cannot serve as a point-wise uncertainty measure, the truncated value function ˆ V π ˆ P k , ˆ b k as the cumulative version of ˆ b k h can serve as a trajectory-wise uncertainty measure, which can be used to determine future exploration policies. Intuitively, for any policy π , the model estimation error satisfies E ( s,a ) ∼ ( P ⋆,k ,π ) [ ∥ ˆ P k ( ·| s, a ) -P ⋆,k ( ·| s, a ) ∥ TV ] ≤ ˆ V π ˆ P k , ˆ b k +∆ , where the error term ∆ captures the variation of both transition kernels and representations over time. As a result, by selecting the policy that maximizes ˆ V π ˆ P k , ˆ b k as the exploration policy as in line 7, the agent will explore the trajectories whose states and actions have not been estimated sufficiently well so far.

Algorithm 2 E 2 U (Model E stimation and E xploration Policy U pdate)

1: Input: round index k , regularizer λ k,W and coefficient ˜ α k,W , datasets { ˜ D ( k,h ) h -1 } , { ˜ D ( k,h ) h } and models { Ψ , Φ } .

2: for step h = 1 , . . . , H do

3: Learn the representation via MLE for step h :

4: Compute the empirical covariance matrix as

5: Define exploration-driven bonus ˆ b k h ( · , · ) = min { α k,W ∥ ˆ ϕ k h ( · , · ) ∥ ( ˆ U k,W h ) -1 , 1 } .

6: end for

7: Find exploration policy ˜ π k = arg max π ˆ V π ˆ P k , ˆ b k , where ˆ V π ˆ P k , ˆ b k is defined as in Equation (1).

8: Output: Model ˆ P k and exploration policy { ˜ π k } .

Step 3: Target Policy Update with Periodic Restart: The agent evaluates the target policy by computing the value function under the target policy and the estimated transition kernel. Then, due to the nonstationarity, the target policy is reset every τ rounds. Compared with the previous work using periodic restart (Fei et al., 2020), whose choice of τ is based on a certain smooth visitation assumption, we remove such an assumption and hence our choice of τ is applicable to more general model classes. Finally, the agent uses the estimated value function for target policy update for the next round k +1 via online mirror descend. The update step is inspired by the previous works (Cai et al., 2020; Schulman et al., 2017). Specifically, for any given policy π 0 and MDP M , define the following function w.r.t. policy π :

L M ,π 0 ( π ) can be regarded as a local linear approximation of V π P,r at 'point' π 0 (Schulman et al., 2017). Consider the following optimization problem:

This can be regarded as a mirror descent step with KL divergence, where the KL divergence regularizes π to be close to π k . It further admits a closed-form solution: π k +1 h ( ·|· ) ∝ π k h ( ·|· ) · exp { η · Q π k h,P ⋆,k ,r k ( · , · ) } . We use the estimated version ˆ Q k h to approximate Q π k h,P ⋆,k ,r k and get

4.2 Technical Assumptions

Our analysis adopts the following standard assumptions on low-rank MDPs.

Assumption 4.1. (Realizability). A learning agent can access to a model class { (Φ , Ψ) } that contains the true model, namely, for any h ∈ [ H ] , k ∈ [ K ] , ϕ ⋆,k h ∈ Φ , µ ⋆,k h ∈ Ψ .

While we assume cardinality of the model class to be finite for simplicity, extensions to infinite classes with bounded statistical complexity are not difficult (Sun et al., 2019).

Assumption 4.2 (Bounded Density) . Any model induced by Φ and Ψ has bounded density, i.e. ∀ P = ⟨ ϕ, µ ⟩ , ϕ ∈ Φ , µ ∈ Ψ , there exists a constant B ≥ 0 such that max ( s,a,s ′ ) ∈S×A×S P ( s ′ | s, a ) ≤ B .

Assumption 4.3 (Reachability) . For each round k and step h , the true transition kernel P ⋆,k h satisfies that for any ( s, a, s ′ ) ∈ S × A × S , P ⋆,k h ( s ′ | s, a ) ≥ p min .

Variation Budgets: We next introduce several measures of nonstationarity of the environment: ∆ P = ∑ K k =1 ∑ H h =1 max ( s,a ) ∈S×A ∥ P ⋆,k +1 h ( ·| s, a ) -P ⋆,k h ( ·| s, a ) ∥ TV , ∆ √ P = ∑ K k =1 ∑ H h =1 max ( s,a ) ∈S×A ∥ P ⋆,k +1 h ( ·| s, a ) -P ⋆,k h ( ·| s, a ) ∥ 1 / 2 TV , ∆ ϕ = ∑ K k =1 ∑ H h =1 max ( s,a ) ∈S×A ∥ ϕ ⋆,k +1 h ( s, a ) -ϕ ⋆,k h ( s, a ) ∥ 2 , ∆ π = ∑ K k =1 ∑ H h =1 max s ∈S ∥ π ⋆,k h ( ·| s ) -π ⋆,k -1 h ( ·| s ) ∥ TV . These notions are known as variation budgets or path lengths in the literature of online convex optimization (Besbes et al., 2015; Hazan, 2016; Hall & Willett, 2015) and nonstationary RL (Fei et al., 2020; Zhong et al., 2021; Zhou et al., 2020). The regret of nonstationary RL naturally depends on these notions that capture the variations of MDP models over time.

4.3 Theoretical Guarantee

To present our theoretical result for PORTAL, we first discuss technical challenges in our analysis and the novel tools that we develop. Generally, large nonstationarity of environment can cause significant errors to MLE, empirical covariance and exploration-driven bonus design for low-rank models. Thus, different from static low-rank MDPs (Agarwal et al., 2020b; Uehara et al., 2022), we devise several new techniques in our analysis to capture the errors caused by nonstationarity which we summarize below.

Characterizing nonstationary MLE guarantee. We provide a theoretical ground to support our design of leveraging history data collected under various different transition kernels in previous rounds for benefiting the estimation of the current model, which is somewhat surprising. Specifically, we establish an MLE guarantee of the model estimation error, which features a separation of variation budgets from a diminishing term as the estimation sample size W increases. Such a result justifies the usage of data generated by mismatched distributions within a certain window as long as the variation budgets is mild. Such a separation cannot be shown directly. Instead, we bridge the bound of model estimation error and the expected value of the ratio of transition kernels via Hellinger distance, and the latter can be decomposed into the variation budgets and a diminishing term as the estimation sample size increases.

Establishing trajectory-wise uncertainty for estimation error ˆ V π ˆ P k , ˆ b k . To this end, straightforward combination of our nonstationary MLE guarantee with previous techniques on low-rank MDPs would yield a coefficient ˜ α k,W that depends on local variation budgets. Instead, we convert the

ℓ ∞ -norm variation budgets ∆ P to square-root ℓ ∞ -norm variation budget ∆ √ P . In this way, the coefficient no longer depends on the local variation budgets, and the estimation error can be upper bounded by ˆ V π ˆ P k , ˆ b k plus an error term only depending on the square-root ℓ ∞ -norm variation budgets.

Error tracking via auxiliary anchor representation. In proving the convergence of average of ˆ V π ˆ P k , ˆ b k , standard elliptical potential based analysis cannot work, because the representation ϕ ⋆,k in the elliptical potential ∥ ∥ ϕ ⋆,k ( s, a ) ∥ ∥ ( U k,W ) -1 changes across rounds, where U k,W h,ϕ is the

population version of U h . To deal with this challenge, in our analysis, we divide the total K rounds into blocks with equal length of W rounds. Then for each block, we set an auxiliary anchor representation. We keep track of the elliptical potential functions using the anchor representation within each block, and control the errors by using anchor representation via variation budgets.

h,ϕ ˆ k,W

The following theorem characterizes theoretical performance for PORTAL.

Theorem 4.4. {M k } K k =1 is set of low-rank MDPs with dimension d . Under Assumptions 4.1 to 4.3, set ˜ α k,W = ˜ O ( √ A + d 2 ) and λ k,W = ˜ O ( d ) . Let { π k } K k =1 be the output of PORTAL in Algorithm 1. For any δ ∈ (0 , 1) , with probability at least 1 -δ , Gap Ave ( K ) of PORTAL is at most

We explain the upper bound in Theorem 4.4 as follows. The basic bound as Equation (3) in Theorem 4.4 contains two parts: the first part ( I ) captures the estimation error for evaluating the target policy under the true environment via the estimated value function ˆ Q k as in line 16 of Algorithm 1. Hence, part ( I ) decreases with K and increases with the nonstationarity of transition kernels and representation functions. Also, part ( I ) is greatly affected by the window size W , which is determined by the dataset used to estimate the transition kernels. Typically, W is tuned carefully based on the variation of environment. If the environment changes significantly, then the samples far in the past are obsolete and become not very informative for estimating transition kernels. The second part ( II ) captures the approximation error arising in finding the optimal policy via the policy optimization method as in line 8 of Algorithm 1. Due to the nonstationarity of the environment, the optimal policy keeps changing across rounds, and hence the nonstationarity of optimal policy ∆ π affects the approximation error. Similarly to the window size W in part ( I ) , the policy restart period τ can also be tuned carefully based on the variation of environment and the optimal policies.

Corollary 4.5. Under the same conditions of Theorem 4.4, if the variation budgets are known, then we can select the hyper-parameters correspondingly to achieve optimality. Specially, if the nonstationarity of the environment is moderate, we have

If the environment is near stationary, then the best W and τ are K . The Gap Ave reduces to ˜ O ( √ H 4 d 2 A ( A + d 2 ) /K ) , which matches results of stationary environment (Uehara et al., 2022).

Detailed discussions and proofs of Theorem 4.4 and Corollary 4.5 are provided in Appendices A and B, respectively.

5 Parameter-free Algorithm: Ada-PORTAL

As shown in Theorem 4.4, hyper-parameters W and τ greatly affect the performance of Algorithm 1. With prior knowledge of variation budgets, the agent is able to optimize the performance as in Corollary 4.5. However, in practice, variation budgets are unknown to the agent. To deal with this issue, in this section, we present a parameter-free algorithm called Ada-PORTAL in Algorithm 3, which is able to tune hyper-parameters without knowing the variation budgets beforehand.

Algorithm 3 is inspired by the BORL method (Cheung et al., 2020). The idea is to use Algorithm 1 as a subroutine and treat the selection of the hyper-parameters such as W and τ as a bandit problem.

Algorithm 3 Ada-PORTAL ( Ada ptive P olicy O ptimization R epresen TA tion L earning)

1: Input: Confidence level δ , number of episodes K , block length M , feasible set of window size J W and policy restart period J τ .

2: Initialization: Initialize α, β, γ and { q l, 1 } l ∈ [ J ] as in Equation (4).

3: for block i = 1 , . . . , ⌈ K/M ⌉ do

4: Update the windows size selection distribution { u ( k,l ) ,i } ( k,l ) ∈J as in Equation (5).

5: Sample ( k i , l i ) ∈ J from the updated distribution { u ( k,l ) ,i } ( k,l ) ∈J , then set W i = ⌊ M k i /J M ⌋ and τ i = ⌊ M l i /J τ τ ⌋ .

6: for episode k = ( i -1) M +1 , . . . , min { iM,K } do

7: Run PORTAL with W i and τ i .

8: end for

9: Compute the total reward for block i as R i ( W i , τ i ) = ∑ min { iM,K } k =( i -1) M +1 V k 1 , where V k 1 is empirical value functions of target policy π k , and update the estimated total reward of running different epoch sizes { q ( k,l ) ,i +1 } ( k,l ) ∈J according to Equation (6).

10: end for

11: Output: { π k } K k =1 .

Specifically, Ada-PORTAL divides the entire K rounds into ⌈ K/M ⌉ blocks with equal length of M rounds. Then two sets J W and J τ are specified (see later part of this section), from which the window size W and the restart period τ for each block are drawn.

For each block i ∈ [ ⌈ K M ⌉ ] , Ada-PORTAL treats each element of J W ×J τ as an arm and take it as a bandit problem to select the best arm for each block. In lines 4 and 5 of Algorithm 3, a master algorithm is run to update parameters and select the desired arm, i.e., the window size W i and the restart period τ i . Here we choose EXP3-P (Bubeck & Cesa-Bianchi, 2012) as the master algorithm and discuss the details later. Then Algorithm 1 is called with input W i and τ i as a subroutine for the current block i . At the end of each block, the total reward of the current block is computed by summing up all the empirical value functions of the target policy of each episode within the block, which is then used to update the parameters for the next block.

We next set the feasible sets and block length of Algorithm 1. Since optimal W and τ in PORTAL are chosen differently from previous works (Zhou et al., 2020; Cheung et al., 2019; Touati & Vincent, 2020) on nonstationary MDPs due to the low-rank structure, the feasible set here that covers the optimal choices of W and τ should also be set differently from those previous works using BORL.

Then the parameters of EXP3-P are intialized as follows:

where J = { ( k, l ) : k ∈ { 0 , 1 , . . . , J W } , l ∈ { 0 , 1 , . . . , J τ }} . The parameter updating rule is as follows. For any ( k, l ) ∈ J , i ∈ ⌈ K/M ⌉ ,

where u ( k,l ) ,i is a probability over J . From u ( k,l ) ,i , the agent samples a desired pair ( k i , l i ) for each block i , which corresponds to the index of feasible set J W ×J τ and is used to set W i and τ i .

As a last step, R i ( W i , τ i ) is rescaled to update q ( k,l ) ,i +1 .

We next establish a bound on Gap Ave for Ada-PORTAL.

Theorem 5.1. Under the same conditions of Theorem 4.4, with probability at least 1 -δ , the average dynamic suboptimality gap of Ada-PORTAL in Algorithm 3 is upper-bounded as Gap Ave ( K ) ≤ ˜ O ( H 11 6 d 5 6 A 1 2 ( A + d 2 ) 1 2 K -1 6 (∆ √ P +∆ ϕ +1) 1 6 +2 HK -1 3 (∆ P +∆ π +1) 1 3 ) .

Comparison with Wei & Luo (2021): It is interesting to compare Ada-PORTAL with an alternative black-box type of approach to see its advantage. A black-box technique called MASTER was proposed in Wei & Luo (2021), which can also work with any base algorithm such as PORTAL to handle unknown variation budgets. Such a combined approach of MASTER+PORTAL turns out to have a worse Gap Ave than our Ada-PORTAL in Algorithm 3. To see this, denote ∆ = ∆ ϕ +∆ √ P +∆ π . The Gap Ave of MASTER+PORTAL is ˜ O ( K -1 6 ∆ 1 3 ) . Then if variation budgets are not too small, i.e. ∆ ≥ ˜ O (1) , this Gap Ave is worse than Ada-PORTAL. See detailed discussion in Appendix C.2.

6 Conclusion

In the paper, we investigate nonstationary RL under low-rank MDPs. We first propose a notion of average dynamic suboptimality gap Gap Ave to evaluate the performance of a series of policies in a nonstationary environment. Then we propose a sample-efficient policy optimization algorithm PORTAL and its parameter-free version Ada-PORTAL. We further provide upper bounds on Gap Ave for both algorithms. As future work, it is interesting to investigate the impact of various constraints such as safety requirements in nonstationary RL under function approximations.

References

Agarwal, A., Henaff, M., Kakade, S. M., and Sun, W. PC-PG: policy cover directed exploration for provable policy gradient learning. In Advances in Neural Information Processing Systems 33 , 2020a.

Agarwal, A., Kakade, S. M., Krishnamurthy, A., and Sun, W. FLAMBE: structural complexity and representation learning of low rank mdps. In Advances in Neural Information Processing Systems 33 , 2020b.

Agarwal, A., Song, Y., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. arXiv preprint arXiv:2205.14571 , 2022.

Besbes, O., Gur, Y., and Zeevi, A. Non-stationary stochastic optimization. Oper. Res. , 63(5): 1227-1244, 2015.

Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

Bubeck, S. and Cesa-Bianchi, N. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found. Trends Mach. Learn. , 2012.

Cai, Q., Yang, Z., Jin, C., and Wang, Z. Provably efficient exploration in policy optimization. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Cheng, Y., Feng, S., Yang, J., Zhang, H., and Liang, Y. Provable benefit of multitask representation learning in reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Cheng, Y., Huang, R., Yang, J., and Liang, Y. Improved sample complexity for reward-free reinforcement learning under low-rank mdps. arXiv preprint arXiv:2303.10859 , 2023.

Cheung, W. C., Simchi-Levi, D., and Zhu, R. Learning to optimize under non-stationarity. In The 22nd International Conference on Artificial Intelligence and Statistics , volume 89 of Proceedings of Machine Learning Research , pp. 1079-1087. PMLR, 2019.

Cheung, W. C., Simchi-Levi, D., and Zhu, R. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Dann, C., Lattimore, T., and Brunskill, E. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In Advances in Neural Information Processing Systems 30 , 2017.

Fei, Y., Yang, Z., Wang, Z., and Xie, Q. Dynamic regret of policy optimization in non-stationary environments. In Advances in Neural Information Processing Systems 33 , 2020.

Gajane, P., Ortner, R., and Auer, P. A sliding-window algorithm for markov decision processes with arbitrarily changing rewards and transitions. CoRR , abs/1805.10066, 2018.

Garivier, A. and Moulines, E. On upper-confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory , 2011.

Gu, S., Holly, E., Lillicrap, T., and Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE international conference on robotics and automation , 2017.

Hall, E. C. and Willett, R. M. Online convex optimization in dynamic environments. IEEE J. Sel. Top. Signal Process. , 9(4):647-662, 2015.

Hazan, E. Introduction to online convex optimization. Found. Trends Optim. , 2(3-4):157-325, 2016.

Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , 2017.

Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Ma, X., Li, J., Kochenderfer, M. J., Isele, D., and Fujimura, K. Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. In IEEE International Conference on Robotics and Automation , 2021.

Mao, W., Zhang, K., Zhu, R., Simchi-Levi, D., and Basar, T. Near-optimal model-free reinforcement learning in non-stationary episodic mdps. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Modi, A., Chen, J., Krishnamurthy, A., Jiang, N., and Agarwal, A. Model-free representation learning and exploration in low-rank mdps. arXiv preprint arXiv:2102.07035 , 2021.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Shani, L., Efroni, Y., Rosenberg, A., and Mannor, S. Optimistic policy optimization with bandit feedback. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Sun, W., Jiang, N., Krishnamurthy, A., Agarwal, A., and Langford, J. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on Learning Theory , 2019.

Touati, A. and Vincent, P. Efficient learning in non-stationary linear markov decision processes. CoRR , abs/2010.12870, 2020.

Tsybakov, A. B. Introduction to Nonparametric Estimation . Springer series in statistics. Springer, 2009.

Uehara, M., Zhang, X., and Sun, W. Representation learning for online and offline RL in low-rank mdps. In The Tenth International Conference on Learning Representations , 2022.

Wei, C. and Luo, H. Non-stationary reinforcement learning without prior knowledge: an optimal black-box approach. In Conference on Learning Theory , 2021.

Xu, T., Liang, Y., and Lan, G. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Zanette, A., Cheng, C.-A., and Agarwal, A. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory , 2021.

Zhao, X., Gu, C., Zhang, H., Yang, X., Liu, X., Tang, J., and Liu, H. DEAR: deep reinforcement learning for online advertising impression in recommender systems. In Thirty-Fifth AAAI Conference on Artificial Intelligence , 2021.

Zhong, H., Yang, Z., Wang, Z., and Szepesvári, C. Optimistic policy optimization is provably efficient in non-stationary mdps. CoRR , abs/2110.08984, 2021.

Zhou, H., Chen, J., Varshney, L. R., and Jagmohan, A. Nonstationary reinforcement learning with linear function approximation. CoRR , 2020.

Supplementary Materials

A Proof of Theorem 4.4

We summarize frequently used notations in the following list.

Proof Sketch of Theorem 4.4. The proof contains the following three main steps.

Step 1 (Appendix A.1): We first decompose the average dynamic suboptimal gap into three terms as in Lemma A.1, which can be divided into two parts: one part corresponds to the model estimation error and the other part corresponds to the performance difference between the target policy chosen by the agent and optimal policy. We then bound the two parts separately.

Step 2 (Appendix A.2): For the first part corresponding to model estimation error, first by Lemmas A.13 and A.15, we show that the model estimation error can be bounded by the average of the truncated value functions of the bonus terms i.e. 1 K ∑ K k =1 ˆ V π ˆ P k , ˆ b k plus a term w.r.t. variation budgets. We then upper bound the average of ˆ V π ˆ P k , ˆ b k as in Lemma A.18. To this end, we divide the total K rounds into blocks with equal length of W and adopt an auxiliary anchor representation for each block to deal with the challenge arising from time-varying representations when using standard elliptical potential based methods.

Step 3 (Appendix A.3): For the second part corresponding to the performance difference bound, similarly to dealing with changing representations, since the optimal policy changes over time, we adopt an auxiliary anchor policy and decompose the performance difference bound into two terms as in Equation (10) and bound the two terms separately.

We further note that the above analysis techniques can also be applied to RL problems where model mis-specification exists, i.e. ϕ ⋆,k / ∈ Φ , µ ⋆,k / ∈ Ψ .

Organization of the Proof for Theorem 4.4. Our proof of Theorem 4.4 is organized as follows. In Appendix A.1, we provide the decomposition of the average dynamic suboptimality gap Gap Ave in Equation (7); in Appendix A.2, we bound the first and third terms of Gap Ave ; in Appendix A.3,

we bound the second term of Gap Ave , and in Appendix A.4 we combine our results to complete the proof of Theorem 4.4. We provide all the supporting lemmas in Appendix A.5.

A.1 Average Suboptimaility Gap Decomposition

Lemma A.1. We denote π ⋆,k h ( ·| s ) = arg max π V π P ⋆,k ,r k . Then the average dynamic suboptimality gap has the following decomposition:

Proof. For any function f : S × A → R and any ( k, h, s ) ∈ [ K ] × [ H ] ×S , define the following operators:

We next consider the following decomposition:

The term G 1 can be bounded as follows:

Substituting the above result to Equation (8) completes the proof.

A.2 First and Third Terms of Gap Ave in Equation (7): Model Estimation Error Bound

A.2.1 First Term in Equation (7)

Lemma A.2. With probability at least 1 -δ , we have

Proof. We proceed the proof by deriving the bound:

where and

A.2.2 Third Term in Equation (7)

Lemma A.3. With probability at least 1 -δ , we have

Proof. We define the model error as f k h ( s h , a h ) = ∥ ∥ ∥ P ⋆ h ( ·| s h , a h ) -ˆ P h ( ·| s h , a h ) ∥ ∥ ∥ TV . We next derive the following bound:

where ( i ) follows from Lemma A.15, ( ii ) follows from the definition of ˜ π k , and ( iii ) follows from Lemma A.18.

A.3 Second Term of Gap Ave in Equation (7): Performance Difference Bound

The second term in Lemma A.1 1 K ∑ K k =1 ∑ h ∈ H E s h ∼ ( P ⋆,k ,π ⋆,k ) [ ⟨ Q π k h, ˆ P k ,r k ( s h , · ) , π ⋆,k h ( ·| s h ) -π k h ( ·| s h ) ⟩ ] can be further decomposed as 1 K K ∑ k =1 ∑ h ∈ H E s h ∼ ( P ⋆,k ,π ⋆,k ) [ ⟨ Q π k h, ˆ P k ,r k ( s h , · ) , π ⋆,k h ( ·| s h ) -π k h ( ·| s h ) ⟩ ] = 1 K ∑ l ∈ [ L ] lτ ∑ k =( l -1) τ +1 ∑ h ∈ [ H ] E P ⋆,k ,π ⋆,k [ ⟨ ˆ Q k h ( s h , · ) , π ⋆,k h ( ·| s h ) -π k h ( ·| s h ) ⟩ ] = 1 K ∑ l ∈ [ L ] lτ ∑ k =( l -1) τ +1 ∑ h ∈ [ H ] E P ⋆, ( l -1) τ +1 ,π ⋆, ( l -1) τ +1 [ ⟨ ˆ Q k h ( s h , · ) , π ⋆,k h ( ·| s h ) -π k h ( ·| s h ) ⟩ ] ︸ ︷︷ ︸ ( a ) + 1 K ∑ l ∈ [ L ] lτ ∑ k =( l -1) τ +1 ∑ h ∈ [ H ] ( E P ⋆,k ,π ⋆,k -E P ⋆, ( l -1) τ +1 ,π ⋆, ( l -1) τ +1 ) [ ⟨ ˆ Q k h ( s h , · ) , π ⋆,k h ( ·| s h ) -π k h ( ·| s h ) ⟩ ] ︸ ︷︷ ︸ ( b ) (10)

A.3.1 Bound ( a ) in Equation (10)

We first present the following lemma of the descent result introduced in Cai et al. (2020).

Lemma A.4. For any distribution p ⋆ and p supported on A and state s ∈ S , and function Q : S × A → [0 , H ] , it holds for a distribution p ′ supported on A with p ′ ( · ) ∝ p ( · ) · exp η · Q ( s, · ) that

Lemma A.5. Term ( a ) in Equation (10) satisfies the following bound:

Proof. We first decompose ( a ) into two parts:

I) Bound the term (I)

where the last equation follows because

II) Bound the term (II)

A.3.2 Bound ( b ) in Equation (10)

Lemma A.6. The term ( b ) in Equation (10) can be bounded as follows:

Proof. Denote the indicator function of state s h as I ( s h ) , and then we have

where P π P ( s ) denotes the visitation probability at state s under model P and policy π .

Consider ∫ s h ∣ ∣ ∣ P π ⋆,t P ⋆,t ( s h ) -P π ⋆,t -1 P ⋆,t -1 ( s h ) ∣ ∣ ∣ ds h can be further decomposed as

where ( i ) and ( ii ) follow from Holder's inequality, and ( iii ) follows from the definition of total variation distance.

Similarly for the second term and from Lemma A.19

Plug Equation (12) and Equation (13) into Equation (11), we have

A.3.3 Combining ( a ) and ( b ) Together

Lemma A.7. Let η = √ L log A K , and we have

Proof. We derive the following bound:

where ( i ) follows from Lemmas A.5 and A.6, and ( ii ) follows from the choice of η = √ L log A K .

A.4 Proof Theorem 4.4

Proof of Theorem 4.4. Combine Lemmas A.1 to A.3 and A.7, we have

A.5 Supporting Lemmas

We first provide the following concentration results, which is an extension of Lemma 39 in Zanette et al. (2021).

Lemma A.8. There exists a constant λ k,W = O ( d log( | Φ | min { k, W } TH/δ )) , the following inequality holds for any k ∈ [ K ] , h ∈ [ H ] , s h ∈ S , a h ∈ A and ϕ ∈ Φ with probability at least 1 -δ :

The following result follows directly from Lemma A.8.

Corollary A.9. The following inequality holds for any k ∈ [ K ] , h ∈ [ H ] , s h ∈ S , a h ∈ A with probability at least 1 -δ :

We next provide the MLE guarantee for nonstationary RL, which shows that under any exploration policy, the estimation error can be bounded with high probability. Differently from Theorem 21 in Agarwal et al. (2020b), we capture the nonstationarity in the analysis.

Lemma A.10 (Nonstationary MLE Guarantee) . Given δ ∈ (0 , 1) , under Assumptions 4.1 to 4.3, let C B = √ C B p min , and consider the transition kernels learned from line 3 in Algorithm 2. We have the following inequality holds for any n, h ≥ 2 with probability at least 1 -δ/ 2 :

where, ζ k,W := 2 log(2 | Φ || Ψ | kH/δ ) W . In addition, for h = 1 ,

Proof of Lemma A.10. For simplification, we denote x = ( s, a ) ∈ X , X = S × A , y = p ⋆ ∈ Y , Y = S . The model estimation process in Algorithm 1 can be viewed as a sequential conditional probability estimation setting with an instance space X and a target space Y , where the conditional density is given by p i ( y | x ) = P ⋆,i ( y | x ) for any i . We are given a dataset D := { ( x i , y i ) } k i =1 ∨ ( k -W ) , where x i ∼ D i = D i ( x 1: i -1 , y 1: i -1 ) and y i ∼ p i ( ·| x i ) . Let D ′ denote a tangent sequence { ( x ′ i , y ′ i ) } k i =1 ∨ ( k -W ) where x ′ i ∼ D i ( x 1: i -1 , y 1: i -1 ) and y ′ i ∼ p i ( ·| x ′ i ) . Further, we consider a function class F = Φ × Ψ : ( X × Y ) → R and assume that the reachability condition P ⋆,i ∈ F holds for any i .

We first introduce one useful lemma in Agarwal et al. (2020b) to decouple data.

Lemma A.11 (Lemma 24 of Agarwal et al. (2020b)) . Let D ba a dataset with at most W samples and D ′ be the corresponding tangent sequence. Let L ( P, D ) = ∑ k i =1 ∨ ( k -W ) l ( P, ( x i , y i )) be any function that decomposes additively across examples where l is any function. Let ̂ P ( D ) be any estimator taking as input random variable D and with range F . Then

Suppose ̂ f ( D ) is learned from the following maximum likelihood problem:

Combining the Chernoff bound and Lemma A.11, we obtain an exponential tail bound, i.e., with probability at least 1 -δ ,

To proceed, we let L ( P, D ) = ∑ k -1 i =1 ∨ ( k -W ) -1 2 log( P ⋆,k ( x i , y i ) /P ( x i , y i )) where D is a dataset { ( x i , y i ) } k i =1 ∨ ( k -W ) (and D ′ = { ( x ′ i , y ′ i ) } k i =1 ∨ ( k -W ) is tangent sequence). Then L ( P ⋆,k , D ) = 0 ≤ L ( ˆ P,D ) .

Then, the RHS of Equation (17) can be bounded as

where the inequality follows because ̂ P is MLE and from the assumption of reachability, and the last equality follows because |F| = | Φ || Ψ | .

Consider for any distribution p and q over a domain Z . Then the Hellinger distance H 2 ( p ∥ q ) = ∫ ( √ p ( z ) -√ q ( z ) ) 2 dz satisfies that

Next, the LHS of Equation (17) can be bounded as

LHS of Equation (17)

where ( i ) follows from the above definition of L ( f, D ) , ( ii ) follows from the fact that 1 -x ≤ -log( x ) , ( iii ) follows from the definition of variation budgets and Equation (19), and ( iv ) follows from the fact that ∥ p ( · ) -q ( · ) ∥ 2 TV ≤ H 2 ( p ∥ q ) as indicated in Lemma 2.3 in Tsybakov (2009).

Combining Equations (17), (18) and (20), we have

We substitute δ with δ/ 2 nH to ensure Equation (21) holds for any h ∈ [ H ] and n with probability at least 1 -δ/ 2 , which finishes the proof.

The following lemma extends Lemmas 12 and 13 under infinite discount MDPs under stationary case in Uehara et al. (2022) to episodic MDPs under nonstationary environment, which captures nonstationarity in analysis.

Lemma A.12 (Nonstationary Step Back) . Let I = [1 ∨ ( k -W ) , k -1] be an index set and { P i h -1 } i ∈I = {⟨ ϕ i h -1 , µ i h -1 ⟩} be a set of generic MDP model, and Π be an arbitrary and possibly mixture policy. Define an expected Gram matrix as follows. For any ϕ ∈ Φ ,

Further, let f i h -1 ( s h -1 , a h -1 ) be the total variation between P i,⋆ h -1 and P i h -1 at time step h -1 . Suppose g ∈ S × A → R is bounded by B ∈ (0 , ∞ ) , i.e., ∥ g ∥ ∞ ≤ B . Then, ∀ h ≥ 2 , ∀ policy π h ,

Proof. We first derive the following bound:

,

where the inequality follows from Cauchy-Schwarz inequality. We further expand the second term in the RHS of the above inequality as follows.

where ( i ) follows from the assumption that ∥ g ∥ ∞ ≤ B , ( ii ) follows from Jensen's inequality, ( iii ) follows because f k h -1 ( s h -1 , a h -1 ) is the total variation between P k,⋆ h -1 and P k h -1 at time step h -1 , and ( iv ) follows from importance sampling and the definition of ∆ P h -1 , I . This finishes the proof.

Recall that f k h ( s, a ) = ∥ ˆ P k h ( ·| s, a ) -P ⋆,k h ( ·| s, a ) ∥ TV . Using Lemma A.12, we have the following lemma to bound the expectation of f k h ( s, a ) under estimated transition kernels.

Lemma A.13. Denote α k,W = √ 2 WAζ k,W + λ k,W d . For any k ∈ [ K ] , policy π and reward r , for all h ≥ 2 , we have

and for h = 1 , we have

Proof. For h = 1 , we have

where ( i ) follows from Jensen's inequality, and ( ii ) follows from the importance sampling. Then for h ≥ 2 , we derive the following bound:

where ( i ) follows from Lemma A.12 and because | f k h ( s h , a h ) | ≤ 1 ; specially the first term inside the square root follows from the definition of U k h -1 , ̂ ϕ k , the third term inside the square root follows from the importance sampling; ( ii ) follows from Lemma A.10, and ( iii ) follows because √ a + b ≤ √ a + √ b, ∀ a, b ≥ 0 and ∥ ∥ ∥ ˆ ϕ k h -1 ( s h -1 , a h -1 ) ∥ ∥ ∥ ( U k h -1 , ˆ ϕ k ) -1 ≤ √ 1 λ W .

The next lemma follows a similar argument to that of Lemma A.13 with the only difference being the expectation over which f k h ( s h , a h ) takes.

Lemma A.14. Denote α k,W = √ 2 WAζ k,W + λ k,W d . For any k ∈ [ K ] , policy π and reward r , for all h ≥ 2 , we have

and for h = 1 , we have

Proof. For h = 1 ,

where ( i ) follows from Jensen's inequality, and ( ii ) follows from the importance sampling. Then for h ≥ 2 , we derive the following bound:

where ( i ) follows from Lemma A.12 and because | f k h ( s h , a h ) | ≤ 1 , the first term inside the square root follows from the definition of U k h -1 , ̂ ϕ k , the third term inside the square root follows from the importance sampling, and ( ii ) follows from Lemma A.10.

The proof is completed by noting that | f ( s h , a h ) | ≤ 1 .

k h

The following lemma is a direct application of Lemma A.13. By this lemma, we can show that the difference of value functions can be bounded by truncated value function plus a variation term.

Lemma A.15 (Bounded difference of value functions) . For k ∈ [ K ] , δ ≥ 0 any policy π and reward r , with probability at least 1 -δ , we have

Proof. (1): We first show that ∣ ∣ ∣ V π P ⋆,k ,r -V π ˆ P k ,r ∣ ∣ ∣ ≤ ˆ V π ˆ P k ,f k .

Recall the definition of the estimated value functions ˆ V π h, ˆ P k ,r ( s h ) and ˆ Q π h, ˆ P k ,r ( s h , a h ) for policy π :

We develop the proof by backward induction.

Suppose that for h +1 , ∣ ∣ ∣ V π h +1 ,P ⋆,k ,r ( s h +1 ) -V π h +1 , ˆ P k ,r ( s h +1 ) ∣ ∣ ∣ ≤ ˆ V π h +1 , ˆ P k ,f k ( s h +1 ) holds for any s h +1 .

Then, for h , by Bellman equation, we have,

where ( i ) follows because ∥ ∥ ∥ ˆ P k h ( ·| s h , a h ) -P ⋆,k h ( ·| s h , a h ) ∥ ∥ ∥ TV = f k h ( s h , a h ) and the value function is at most 1, and ( ii ) follows from the induction hypothesis.

Then, by the definition of ˆ V π h, ˆ P k ,r ( s h ) , we have

(2): Then we prove that

By Lemma A.13 and the fact that the total variation distance is upper bounded by 1, ∀ h ≥ 2 , with probability at least 1 -δ/ 2 , we have

Similarly, when h = 1 ,

Based on Corollary A.9, Equation (28) and ˜ α k,W = 5 α k,W , we have

For the base case h = H , we have

For any step h + 1 , h ≥ 2 , assume that E ˆ P k ,π [ ˆ V π h +1 , ˆ P k ,f k ( s h +1 ) ∣ ∣ ∣ ∣ s h ] ≤ ˆ V π h, ˆ P k , ˆ b k ( s h ) + ∑ H h ′ = h +1 √ 3 λ W WAC B ∆ P { h ′ -1 ,h ′ } , [ k -W,k -1] holds . Then, by Jensen's inequality, we obtain E ˆ P k ,π [ ˆ V π h, ˆ P k ,f k ( s h ) ∣ ∣ ∣ ∣ s h -1 , a h -1 ]

where ( i ) follows from Equation (27).

Therefore, by induction, we have

where ( i ) follows from Equation (30), and ( ii ) is due to the induction hypothesis.

By induction, we conclude that

Combining Step 1 and Step 2, we conclude that

Similarly to Lemma A.15, we can prove that the total variance distance is bounded by ˆ V ˜ π k ˆ P k , ˆ b k plus a variation budget term as follows. Lemmas A.15 and A.16 together justify the choice of exploration policy for the off-policy exploration.

Lemma A.16. Fix δ ∈ (0 , 1) , for any h ∈ [ H ] , k ∈ [ K ] , any policy π , with probability at least 1 -δ/ 2 ,

Proof. Fix any policy π , for any h ≥ 2 , we have

Hence, for h ≥ 2 , we have

where ( i ) follows from Equation (30), ( ii ) follows from the definition of ˆ Q π h -1 , ˆ P k , ˆ b k ( s h -1 , a h -1 ) , and ( iii ) follows from Equation (31).

where the last equation follows from Equation (32) and Lemma A.15.

Lemma A.17. Denote ˜ α k,W = 5 α k,W , α k,W = √ 2 WAζ k,W + λ k,W d , and β k,W = √ 9 dAα 2 k,W + λ k,W d . For any k ∈ [ K ] , policy π and reward r , for all h ≥ 2 , we have

and for h = 1 , we have

Proof. For h = 1 ,

where ( i ) follows from Jensen's inequality, and ( ii ) follows from the importance sampling.

Then for h ≥ 2 , we first notice that

max ( s,a ) ∈S×A √∥ ∥ ∥ P ⋆,k +1 h ( ·| s, a ) -P ⋆,k h ( ·| s, a ) ∥ ∥ ∥ TV , then for any H , I , we can convert ℓ ∞ variation budgets to square-root ℓ ∞ variation budgets.

Recall that W k h,ϕ = ∑ k -1 i =1 ∨ ( k -W ) E ( s h ,a h ) ∼ ( P ⋆,i ,π i ) [ ϕ h ( s h , a h ) ϕ h ( s h , a h ) ⊤ ] + λ k,W I d . We derive the following bound:

where ( i ) follows from Lemma A.12, ( ii ) follows from Equation (35), and ( iii ) follows Equation (36).

Before next lemma, we first introduce a notion related to matrix norm. For any matrix A , ∥ A ∥ 2 denotes the matrix norms induced by vector ℓ 2 -norm. Note that ∥ A ∥ 2 is also known as the spectral norm of matrix A and is equal to the largest singular value of matrix A .

Lemma A.18. With probability at least 1 -δ , the summation of the truncated value functions ˆ V π k ˆ P k , ˆ b k under exploration policies { ˜ π k } k ∈ [ K ] is bounded by:

Proof. For any k ∈ [ K ] , any policy π , we have

As a result, we have

Step 1: We first bound ∑ K k =1 V ˜ π k P ⋆,k ,f k via an auxiliary anchor representation .

Recall that U k,W h,ϕ ⋆,k = ∑ k -1 i =1 ∨ ( k -W ) E s h ∼ ( P ⋆,i , ˜ π i ) a h ∼U ( A ) [ ϕ ⋆,k h ( s h , a h ) ϕ ⋆,k h ( s h , a h ) ⊤ ] + λ k,W I d and we define ˜ U k,W,t h,ϕ ⋆,k = ∑ k -1 i = tW +1 E s h ∼ ( P ⋆,i , ˜ π i ) a h ∼U ( A ) [ ϕ ⋆,k h ( s h , a h ) ϕ ⋆,k h ( s h , a h ) ⊤ ] + λ k,W I d . We first note that for any h , the following equation holds.

The ϕ ⋆,k and U in Equation (38) both change with the round index k . To deal with such an issue, we divide the entire round into ⌊ K W ⌋ + 1 blocks with an equal length of W . For each block t ∈ { 0 , . . . , ⌊ K W ⌋} , we select an auxiliary anchor representation ϕ ⋆,tW +1 and decompose Equation (38) as follows. We first derive the following equation:

For term ( II ) , we have

where ( i ) follows from the property of the matrix norms induced by vector ℓ 2 -norm. For term ( I ) , we have

where ( i ) follows from the property of the matrix norms induced by vector ℓ 2 -norm and ( ii ) follows from that ∥ ∥ ∥ ϕ ⋆,k h ( s h , a h ) ∥ ∥ ∥ 2 ≤ 1 .

Furthermore,

where the last equation follows from Lemma D.2. Then combining Equations (39) to (42), we have

Substituting Equation (43) into Equation (38), we have

where the second equation follows from Equation (43).

Then we derive the following bound:

where ( i ) follows from Lemma A.14, and ( ii ) follows from Equation (44). Step 2: We next bound ∑ K k =1 V ˜ π k P ⋆,k , ˆ b k via an auxiliary anchor representation .

Similarly to the proof Step 1 , we further bound ∑ k ∈ [ K ] E ( s h ,a h ) ∼ ( P ⋆,k , ˜ π k ) [ β k,W ∥ ∥ ∥ ϕ ⋆,k h ( s h , a h ) ∥ ∥ ∥ ( W k h,ϕ ⋆,k ) -1 ] . We define W k,W h,ϕ ⋆,k = ∑ k -1 i =1 ∨ ( k -W ) E ( s h ,a h ) ∼ ( P ⋆,i , ˜ π i ) [ ϕ ⋆,k h ( s h , a h ) ϕ ⋆,k h ( s h , a h ) ⊤ ] + λ k,W I d and ˜ W k,W,t h,ϕ ⋆,k = ∑ k -1 i = tW +1 E ( s h ,a h ) ∼ ( P ⋆,i , ˜ π i ) [ ϕ ⋆,k h ( s h , a h ) ϕ ⋆,k h ( s h , a h ) ⊤ ] + λ k,W I d . We first note that for any h , we have

The ϕ ⋆,k and W in Equation (46) both change with the round index k . To deal with this issue, we decompose it as follows. We first derive the following equation:

For term ( IV ) , we have

where ( i ) follows from the property of the matrix norms induced by vector ℓ 2 -norm. For term ( III ) , we derive the following bound:

where ( i ) follows from the property of the matrix norms induced by vector ℓ 2 -norm and ( ii ) follows from that ∥ ∥ ∥ ϕ ⋆,k h ( s h , a h ) ∥ ∥ ∥ 2 ≤ 1 .

Furthermore, we derive the following bound:

where the last equation follows from Lemma D.2.

Then combining Equations (39) to (41) and (50), we have

Substituting Equation (51) into Equation (46), we have

where the second equation follows from Equation (51). Then, we derive the following bound:

where ( i ) follows from Lemma A.17, and ( ii ) follows from Equation (52). Finally, combining Equations (37), (45) and (53), we have

The following visitation probability difference lemma is similar to lemma 5 in Fei et al. (2020), but we remove their Assumption 1.

Lemma A.19. For any transition kernels { P h } H h =1 , h ∈ [ H ] , j ∈ [ h -1] , s h ∈ S and policies { π i } H i =1 and π ′ j , we have

Proof. To prove this lemma, the only difference from lemma is that we need to show max s j ∑ s j +1 | P π j j ( s j +1 | s j ) -P π ′ j j ( s j +1 | s j ) | ≤ 2 max s ∈S ∥ ∥ π j ( ·| s ) -π ′ j ( ·| s ) ∥ ∥ TV holds without assumption. We show this as follows:

B Further Discussion and Proof of Corollary 4.5

In this section, we first provide a detailed version and further discussion of Corollary 4.5 in Appendix B.1, then present the proof in Appendix B.2, and finally present an interesting special case in Appendix B.3.

B.1 Further Discussion of Corollary 4.5

We present a detailed version of Corollary 4.5 as follows. Let Π [1 ,K ] ( N ) = min { K, max { 1 , N }} for any K,N ∈ N .

Corollary B.1 (Detailed version of Corollary 4.5) . Under the same conditions of Theorem 4.4, if the variation budgets are known, then for different variation budget regimes, we can select the hyper-parameters correspondingly to attain the optimality for both ( I ) w.r.t. W and ( II ) w.r.t. τ in Equation (3) . For ( I ) , with W = Π [1 ,K ] ( ⌊ H 1 3 d 1 3 K 1 3 (∆ √ P +∆ ϕ ) -1 3 ⌋ ) , part ( I ) is upper-bounded by

For ( II ) in Equation (3) , with τ = Π [1 ,K ] ( ⌊ K 2 3 (∆ P +∆ π ) -2 3 ⌋ ) , part ( II ) is upper bounded by

For any ϵ ≥ 0 , if nonstationarity is not significantly large, i.e., there exists a constant γ < 1 such that (∆ P +∆ π ) ≤ (2 HK ) γ and (∆ √ P +∆ ϕ ) ≤ (2 HK ) γ , then PORTAL can achieve ϵ -average suboptimal with polynomial trajectories.

As a direct consequence of Theorem 4.4, Corollary 4.5 indicates that if variation budgets are known, then the agent can choose the best hyper-parameters directly based on the variation budgets. The Gap Ave can be different depending on which regime the variation budgets fall into, as can be seen in Equations (54) and (55).

At the high level, we further explain how the window size W depends on the variations of environment as follows. If the nonstationarity is moderate and not significantly large, Corollary 4.5 indicates that for any ϵ , Algorithm 1 achieves ϵ -average suboptimal with polynomial trajectories (see the specific form in Equation (59) in Appendix B.2). If the environment is near stationary and the variation is relatively small, i.e., (∆ √ P +∆ ϕ ) ≤ Hd/K 2 , (∆ P +∆ π ) ≤ 1 / √ K , then the best window size W

and the policy restart period τ are both K . This indicates that the agent does not need to take any forgetting rules to handle the variation. Then the Gap Ave reduces to ˜ O ( √ H 4 d 2 A ( A + d 2 ) /K ) , which matches the result under a stationary environment. 2

Furthermore, it is interesting to consider a special mildly changing environment, in which the representation ϕ ⋆ stays identical and only the state-embedding function µ ⋆,k changes over time. The average dynamic suboptimality gap in Equation (3) reduces to

The part ( II ) is the same as ( II ) in Equation (3) and by choosing the best window size of W = H 1 2 d 1 2 ( A + d 2 ) 1 4 K 1 2 (∆ √ P ) -1 2 , part ( I ) becomes

Compared with the second regime in Equation (54), Equation (56) is much smaller, benefited from identical representation function ϕ ⋆ . In this way, samples in previous rounds can help to estimate the representation space so that W can be larger than W in terms of the order of K , which yields efficiency gain compared with changing ϕ ⋆ .

On the other hand, if the nonstationarity is significantly large, for example, scales linearly with K , then for each round, the previous samples cannot help to estimate current best policy. Thus, the best W and τ are both 1, and the average dynamic suboptimality gap reduces to ˜ O ( √ H 4 d 2 A ( A + d 2 ) ) . This indicates that for a fixed small accuracy ϵ ≥ 0 , no matter how large the round K is, Algorithm 1 can never achieve ϵ -average suboptimality.

B.2 Proof of Corollary B.1 (i.e., Detailed Version of Corollary 4.5)

If variation budgets are known, for different variation budgets regimes, we can tune the hyperparameters correspondingly to reach the optimality for both the term ( I ) that contains W and the term ( II ) that contains τ .

For the first term ( I ) in Equation (14), there are two regimes:

Small variation: ( ∆ √ P +∆ ϕ ) ≤ Hd K 2 ,

-The best window size W is K , which means that the variation is pretty mild and the environment is near stationary. In this case, by choosing window size W = K , the agent takes no forgetting rules to handle the variation. Then the first term ( I ) reduces to √ H 4 d 2 A K ( A + d 2 ) , which matches the result under a stationary environment. 3

-Then for any ϵ ≥ 0 , with HK no more than ˜ O ( H 5 d 2 A ( A + d 2 ) ϵ 2 ) , term ( I ) ≤ ϵ .

Large variation: ( ∆ √ P +∆ ϕ ) > Hd K 2 .

-By choosing the window size W = H 1 3 d 1 3 K 1 3 ( ∆ √ P +∆ ϕ ) -1 3 , the term ( I ) reduces to H 2 d 5 6 A 1 2 ( A + d 2 ) 1 2 ( HK ) -1 6 ( ∆ √ P +∆ ϕ ) 1 6 .

̸

-Since ( ∆ √ P +∆ ϕ ) ≤ 2 HK , there exists γ ≤ 1 s.t. ( ∆ √ P +∆ ϕ ) ≤ (2 HK ) γ . Then ( I ) ≤ 2 H 2 d 5 6 A 1 2 ( A + d 2 ) 1 2 ( HK ) -1 -γ 6 . Then for any ϵ ≥ 0 , if γ = 1 , with HK no more than ˜ O ( d 5 1 -γ H 12 1 -γ A 3 1 -γ ( A + d 2 ) 3 1 -γ ϵ 6 1 -γ ) , term ( I ) ≤ ϵ .

For the second term ( II ) in Equation (14), there are three regimes elaborated as follows:

Small variation: (∆ P +∆ π ) ≤ 1 √ K ,

-The best policy restart period τ is K , which means that the variation is pretty mild and the agent does not need to handle the variation. Then term ( II ) ≤ 2 H √ K .

-Then for any ϵ ≥ 0 , with HK no more than ˜ O ( H 2 ϵ 2 ) , term ( II ) ≤ ϵ .

Moderate variation: 1 √ K < (∆ P +∆ π ) ≤ K ,

-The best policy restart period τ = K 2 3 (∆ P +∆ π ) -2 3 , and the term ( II ) reduces to 2 HK -1 3 (∆ P +∆ π ) 1 3 .

̸

-Since ( ∆ P +∆ π ) ≤ 2 HK , there exists γ ≤ 1 s.t. ( ∆ P +∆ π ) ≤ (2 HK ) γ . Then the term ( II ) ≤ 4 H 4 3 ( HK ) -1 -γ 3 . Then for any ϵ ≥ 0 , if γ = 1 , with HK no more than ˜ O ( H 4 1 -γ ϵ 3 1 -γ ) , term ( II ) ≤ ϵ .

Large variation: K < (∆ P +∆ π ) ,

-The variation budgets scale linearly with K , which indicates that the nonstationarity of the environment is significantly large and lasts for the entire rounds. Hence in each round, the previous sample can not help to estimate the current best policy. So the best policy restart period τ = 1 , and the second term ( II ) reduces to H + H (∆ P +∆ π ) K = O ( H ) , which implies that Algorithm 1 can never achieve small average dynamic suboptimality gap for any large K .

In conclusion, the first term is upper bounded by

and the second term is upper bounded by

In addition, if the variation budgets are not significantly large, i.e. scale linearly with K , for any ϵ ≥ 0 , Algorithm 1 can achieve ϵ -average dynamic suboptimality gap with at most polynomial samples. Specifically, if there exists a constant γ < 1 such that the variation budgets satisfying ( ∆ P +∆ π ) ≤ (2 HK ) γ and ( ∆ √ P +∆ ϕ ) ≤ (2 HK ) γ , then to achieve ϵ -average dynamic suboptimality gap, i.e.,

Gap Ave ( K ) ≤ ϵ , Algorithm 1 only needs to collect trajectories no more than

B.3 A Special Case

In this subsection, we provide a characterization of a special case, where the representation ϕ ⋆ stays identical and only the state-embedding function µ ⋆,k changes over time. In such a scenario, the variation budget ∆ ϕ [ H ] , [ K ] = 0 and the average dynamic suboptimality gap bound in Equation (14) reduces to

where the last equation follows from the choice of the window side W = ˜ O ( H 1 2 d 1 2 ( A + d 2 ) 1 4 K 1 2 ( ∆ √ P [ H ] , [ K ] ) -1 2 ) and the policy restart period τ = ˜ O ( K 2 3 (∆ P [ H ] , [ K ] +∆ π [ H ] , [ K ] ) -2 3 ) with known variation budgets.

C Proof of Theorem 5.1 and Detailed Comparison with Wei & Luo (2021)

C.1 Proof of Theorem 5.1

Proof of Theorem 5.1. Before our formal proof, we first explain several notations on different choices of W and τ here.

( W,τ ): Because W ⋆ = d 1 3 H 1 3 K 1 3 (∆ ϕ + ∆ √ P + 1) -1 3 ≤ d 1 3 H 1 3 K 1 3 ≤ J W and τ ⋆ = ˜ O ( K 2 3 (∆ P +∆ π +1) -2 3 ) ≤ K 2 3 ≤ J τ . As a result, there exists a W ∈ J W such that W ≤ W ⋆ ≤ 2 W and a τ ∈ J τ such that τ ≤ τ ⋆ ≤ 2 τ .

( W † , τ † ): ( W † , τ † ) denotes the set of best choices of the window size W and the policy restart period τ in feasible set that maximize ∑ ⌈ K/M ⌉ i =1 R i ( W,τ ) .

Then we can decompose the average dynamic suboptimality gap as

where the last inequality follows because if { π k } K k =1 is the output of Algorithm 3 with the chosen window size { W i } ⌈ T/M ⌉ i =1 , E [ R i ( W i , τ i )] = E [ ∑ min { iM,K } k =( i -1) M +1 V k 1 ] = ∑ min { iM,K } k =( i -1) M +1 V π k P ⋆ ,k holds.

We next bound Terms ( I ) and ( II ) separately.

Term (I): We derive the following bound:

where ( i ) follows from Equation (14), ( ii ) follows from the definition of W and ( iii ) follows from the definition of W ⋆ at the beginning of the proof.

Term (II): We derive the following bound:

where ( i ) follows from the definition of W † and ( ii ) follows from Theorem 3.3 in Bubeck & CesaBianchi (2012) with the adaptation that reward R i ≤ M and the number of iteration is ⌈ K/M ⌉ . Then combining the bounds on terms ( I ) and ( II ) , we have

C.2 Detailed Comparison with Wei & Luo (2021)

Based on the theoretical results Theorem 4.4 and taking Algorithm 1 PORTAL as a base algorithm, we can also use the black-box techniques called MASTER in Wei & Luo (2021) to handle the unknown variation budgets. But such an approach of MASTER+PORTAL turns out to have a larger average dynamic suboptimality gap than Algorithm 3. Denote ∆ = ∆ ϕ +∆ √ P +∆ π .

To explain, first choose τ = k and W = K in Algorithm 1. Then Algorithm 1 reduces to a base algorithm with the suboptimality gap of ˜ O ( √ H 4 d 2 A ( A + d 2 ) /K + √ H 3 dAK ( A + d 2 )∆) , which satisfies Assumption 1 in Wei & Luo (2021) with ρ ( t ) = √ H 4 d 2 A ( A + d 2 ) /t and ∆ [1 ,t ] = √ H 3 dAt ( A + d 2 )∆ . Then by Theorem 2 in Wei & Luo (2021), the dynamic regret using

MASTER+PORTAL can be upper-bounded by ˜ O ( H 11 6 d 5 6 A 1 2 ( A + d 2 ) 1 2 K 2 3 ∆ 1 3 ) = ˜ O ( K 5 6 ∆ 1 3 ) . Here, we find the order dependency on d, H, A is the same as Theorem 5.1 and hence mainly focus on the order dependency on K and ∆ in the following comparison. Such a bound on dynamic regret can be converted to the average dynamic suboptimality gap as ˜ O ( K -1 6 ∆ 1 3 ) . Then it can obersed that for not too small variation budgets, i.e., ∆ ≥ ˜ O (1) , the order dependency on ∆ is higher than that of Algorithm 3.

Specifically, if we consider the special case when the representation stays identical and denote ˜ ∆ = ∆ √ P + ∆ π , then the average dynamic suboptimality gap of MASTER+PORTAL is still ˜ O ( K -1 6 ˜ ∆ 1 3 ) . With small modifications on the parameters, by following the analysis similar to that of Theorem 5.1 and Appendix B.3, we can show that the average dynamic suboptimality gap of Algorithm 3 satisfies ˜ O ( K -1 4 ˜ ∆ 1 4 ) , which is smaller than MASTER+PORTAL.

D Auxiliary Lemmas

In this section, we provide two lemmas that are commonly used for the analysis of MDP problems.

The following lemma (Dann et al., 2017) is useful to measure the difference between two value functions under two MDPs and reward functions.

Lemma D.1. (Simulation Lemma). Suppose P 1 and P 2 are two MDPs and r 1 , r 2 are the corresponding reward functions. Given a policy π , we have,

The following lemma is a standard inequality in the regret analysis for linear MDPs in reinforcement learning (see Lemma G.2 in Agarwal et al. (2020b) and Lemma 10 in Uehara et al. (2022)).

LemmaD.2 (Elliptical Potential Lemma) . Consider a sequence of d × d positive semidefinite matrices X 1 , . . . , X N with tr( X n ) ≤ 1 for all n ∈ [ N ] . Define M 0 = λ 0 I and M n = M n -1 + X n . Then the following bound holds:

Peer reviews:
comments: SUMMARY:
The paper presents a new way to perform off-policy RL coupled with representation learning over non-stationary low-rank MDPs, both in a parameter-based and a parameter-free fashion. The paper proposes some interesting new algorithmic techniques to tackle this (off-policy
exploration, data-transfer model learning, and target policy update with periodic restart).

STRENGTHS:
# originality
The work seems quite original in both the analysis and the contribution. 


# quality 

The quality of this work is satisfactory and the analysis is thoughtful.

# clarity

The exposition is clear and neat, I would suggest presenting the algorithms differently (See weaknesses)
 

# significance

The work seems to provide a significant contribution to the field, due to the several techniques it introduces.

WEAKNESSES:
My only comment is about the exposition of the algorithm. I would suggest not citing equations but reporting them, and providing the output of the subroutines.

QUESTIONS:
No questions

LIMITATIONS:
I didn't find any section where the limitations of the proposed method were addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper present an algorithm for solving non-stationary low-rank MDP. It also provide a theoretical analysis of their algorithm,

STRENGTHS:
This is an interesting question. The presentation is clear.

WEAKNESSES:
I sketch through the results and the proof. I doubt that the proof might be wrong.  In page 14, the inequality (i) require the author to provide an upper bound of E_{P^*} f_h^k - E_{\hat P}f_h^k. The paper says that inequality (i) holds since Lemma A.13 and Lemma A.15. However, none of the lemma is related with the term I mentioned above. Therefore, I am worried that the proof might be wrong.

More over, the writing of the proof can be improve, and the clearity of the paper have room for improvement. For example, it would be helpful to provide the definition of $\lambda$ in the main part. It would also be helpful to mention that $W\leq K$ in the main part.

QUESTIONS:
See the 'weakness' section.

LIMITATIONS:
See the 'weakness' section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates nonstationary RL in the context of episodic low-rank MDPs, where both transition kerns and rewards may change from round to round, and where the low-rank model contains unknown representations. The goal of this work is then to develop methods with low average dynamic suboptimality gap, which is a measure of how well the agent performs across all rounds. The authors start by proposing PORTAL, a base algorithm with three main ideas: (a) off-policy exploration for data collection which is particularly useful for non-stationarity; (b) the transfer of recent history data collected under previous different transition kernels for benefiting the estimation of the current model; and (c) updating target policies with periodic restart. PORTAL requires few hyperparameters, which greatly affect performance but may not be easy to set. for this reason, the authors also propose Ada-PORTAL, a second method that uses PORTAL as a subroutine and additionally treats the selection of the hyperparameters as a bandit problem. Under standard assumptions for low-rank MDPs and assuming that non-stationarity is not significant, the authors show that both methods achieve arbitrarily small average dynamic suboptimality gap with polynomial sample complexity.\n\nSTRENGTHS:\n1. The paper extends the literature in low-rank MDPs by incorporating non-stationarity. In particular, the work advances the recent research on nonstationary linear (mixture) MDPs. Overall, the contribution is interesting, given the attention that both low-rank MDPs as well as non-stationarity have recently attracted.\n2. The paper makes assumptions that are consistent with both the low-rank MDP literature as well as the non-stationary RL literature.\n3. I was only able to check part of the derivations, which looked correct. I liked that the authors clarified how their work is different from recent works on non-stationarity or low-rank MDPs, not just in terms of the problem statement but also in terms of the new technical challenges.\n\nWEAKNESSES:\n1. I appreciate the fact that the authors provided insights and clarifications in several parts of the paper. But the paper is still hard to follow. I think what generally helps with such works is to provide an overview of the proof strategy, explain which exactly elements from prior works are used, and what elements are new. I acknowledge that the paper contains some discussions here and there, but I feel that a more structured approach would significantly improve the exposition.\n2. In its current form and presentation, I feel that the paper is only relevant to a very small audience of the ML/AI community.\n\nQUESTIONS:\n1. Why periodic restart in this work does not require a smooth visitation frequency assumption (compared to Fei et al., 2020)?\n2. The authors use EXP3-P in Ada-PORTAL. Is this choice critical or could other choices be made instead? Are there reasons to prefer one over the others?\n3. The authors stress the significance of \\tau and W. For the first algorithm W and \\tau, how exactly should the be selected? Could the authors provide clear guidelines?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning (RL) under non-stationary MDPs where the state transition and reward function changes across episodes. The paper focus on the low-rank MDP setting, in which the state transition function admits a low-rank decomposition. The authors propose a novel algorithm, namely PORTAL, along with theoretical results that characterize a bound for the average suboptimality gap of the learned policies over the episodes. Additionally, the paper also introduces an algorithm, namely Ada-PORTAL, for adaptive tuning of the hyperparameters of PORTAL.\n\nSTRENGTHS:\n* The paper is the first to study non-stationarity in low-rank MDPs.\n\n* The proposed method is sound and results in a better suboptimality gap than previous related work (Wei & Luo, 2021).\n\nWEAKNESSES:\n* A few algorithmic decisions and assumptions require better explanation/justification to improve the clarity of the paper (see below).\n\n* The method relies on an assumption that is not very realistic (reachability) and should be better discussed (see Limitations below).\n\n* (Minor) The paper does not have an experimental section.\n\nQUESTIONS:\nBelow, I have a few questions and constructive feedback for the authors:\n\n1) In Section 3.2, I would suggest making it clearer what a “round” is. For instance, does each round have a fixed number of episodes?\n\n2) The current version of Section 4.1 is a bit hard to follow. For instance, in Step 1, terms such as “bonus term” and “target policy” are used without being previously defined.\n\n3) “However, in low-rank MDPs, the bonus term $b^k_k$ cannot serve as a point-wise uncertainty measure”. I suggest elaborating on why this is true.\n\n4) In Step 1, why the exploration policy takes two uniformly chosen actions? This algorithmic decision is not discussed or explained.\n\n5) I suggest writing comments in Algorithm 1 mentioning which lines correspond to Step 1, 2, and 3 for improving the clarity.\n\n6) In Algorithm 1, why states are indexed two times (subscript and superscript) with the time step $h$, e.g., $s_h^{k, h}$? What the indexes mean was not defined.\n\n7) What is the role of the coefficient $\\tilde{\\alpha_{k,W}}$? How is this selected?\n\n8) “Compared with the previous work using periodic restart (Fei et al., 2020), whose choice of $\\tau$ is based on a certain smooth visitation assumption, we remove such an assumption and hence our choice of $\\tau$ is applicable to more general model classes.” \nWhy can you remove this assumption? How is the value of $\\tau$ selected? Please elaborate on why it is necessary to reset the target policy every $\\tau$ episode.\n\nLIMITATIONS:\nAssumption 4.3 is not very realistic: in most MDPs, it is not possible to reach any state of the MDP in a single step. For instance, in a maze, the probability of reaching the end of the maze from the initial state in a single step is zero. Is this assumption also made in related works, e.g., Wei & Luo (2021)?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}