Title: Tempo Adaptation in Non-stationary Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Non-stationary RL

Reinforcement Learning
Abstract: We first raise and tackle a ``time synchronization'' issue between the agent and the environment in non-stationary reinforcement learning (RL), a crucial factor hindering its real-world applications. In reality, environmental changes occur over wall-clock time ($t$) rather than episode progress ($k$), where wall-clock time signifies the actual elapsed time within the fixed duration $t \in [0, T]$. In existing works, at episode $k$, the agent rolls a trajectory and trains a policy before transitioning to episode $k+1$. In the context of the time-desynchronized environment, however, the agent at time $t_{k}$ allocates $\Delta t$ for trajectory generation and training, subsequently moves to the next episode at $t_{k+1}=t_{k}+\Delta t$. Despite a fixed total number of episodes ($K$), the agent accumulates different trajectories influenced by the choice of interaction times ($t_1,t_2,...,t_K$), significantly impacting the suboptimality gap of the policy. We propose a Proactively Synchronizing Tempo ($\texttt{ProST}$) framework that computes a suboptimal sequence {$t_1,t_2,...,t_K$} (= { $t_{1:K}$}) by minimizing an upper bound on its performance measure, i.e., the dynamic regret. Our main contribution is that we show that a suboptimal {$t_{1:K}$} trades-off between the policy training time (agent tempo) and how fast the environment changes (environment tempo). Theoretically, this work develops a suboptimal {$t_{1:K}$} as a function of the degree of the environment's non-stationarity while also achieving a sublinear dynamic regret. Our experimental evaluation on various high-dimensional non-stationary environments shows that the $\texttt{ProST}$ framework achieves a higher online return at suboptimal {$t_{1:K}$} than the existing methods.

Tempo Adaptation in Non-stationary Reinforcement Learning

Abstract

We first raise and tackle a 'time synchronization' issue between the agent and the environment in non-stationary reinforcement learning (RL), a crucial factor hindering its real-world applications. In reality, environmental changes occur over wall-clock time ( t ) rather than episode progress ( k ), where wall-clock time signifies the actual elapsed time within the fixed duration t ∈ [ 0 , T ] . In existing works, at episode k , the agent rolls a trajectory and trains a policy before transitioning to episode k + 1 . In the context of the time-desynchronized environment, however, the agent at time t k allocates ∆ t for trajectory generation and training, subsequently moves to the next episode at t k + 1 = t k + ∆ t . Despite a fixed total number of episodes ( K ), the agent accumulates different trajectories influenced by the choice of interaction times ( t 1 , t 2 , ..., t K ), significantly impacting the suboptimality gap of the policy. We propose a Proactively Synchronizing Tempo ( ProST ) framework that computes a suboptimal sequence { t 1 , t 2 , ..., t K }(= { t } 1 ∶ K ) by minimizing an upper bound on its performance measure, i.e., the dynamic regret. Our main contribution is that we show that a suboptimal { t } 1 ∶ K trades-off between the policy training time (agent tempo) and how fast the environment changes (environment tempo). Theoretically, this work develops a suboptimal { t } 1 ∶ K as a function of the degree of the environment's non-stationarity while also achieving a sublinear dynamic regret. Our experimental evaluation on various high-dimensional nonstationary environments shows that the ProST framework achieves a higher online return at suboptimal { t } 1 ∶ K than the existing methods.

1 Introduction

The prevailing reinforcement learning (RL) paradigm gathers past data, trains models in the present, and deploys them in the future . This approach has proven successful for stationary Markov decision processes (MDPs), where the reward and transition functions remain constant [1-3]. However, challenges arise when the environments undergo significant changes, particularly when the reward and transition functions are dependent on time or latent factors [4-6], in non-stationary MDPs. Managing non-stationarity in environments is crucial for real-world RL applications. Thus, adapting to changing environments is pivotal in non-stationary RL.

This paper addresses a practical concern that has inadvertently been overlooked within traditional nonstationary RL environments, namely, the time synchronization between the agent and the environment. We raise the impracticality of utilizing episode-varying environments in existing non-stationary RL

ProST-TProOLSProWLS

IFTML

ONPG

10

T10

10

Rewards

1

-3

11

13

D

1000

Episodes

2000

3000

4000

5000

(a)

(b)

(c)

research, as such environments do not align with the real-world scenario where changes occur regardless of the agent's behavior. In an episode-varying environment, the agent has complete control over determining the time to execute the episode k , the duration of policy training between the episodes k and k + 1 , and the transition time to the episode k + 1 . The issue stems from the premise that the environment undergoes dynamic changes throughout the course of each episode, with the rate of non-stationarity contingent upon the behavior exhibited by the agent. However, an independent wall-clock time ( t ) exists in a real-world environment, thereby the above three events are now recognized as wall-clock time t k , training time ∆ t , and t k + 1 . The selection of interaction times ( t k , t k + 1 ) has a notable impact on the collected trajectories, while the interval t k + 1 -t k establishes an upper limit on the duration of training ( ∆ t ). This interval profoundly influences the suboptimality gap of the policy. In the context of a time-desynchronized environment, achieving an optimal policy requires addressing a previously unexplored question: the determination of the optimal time sequence { t 1 , t 2 , .., . t K }(= { t } 1 ∶ K ) at which the agent should interact with the environment.

We elucidate the significance of the aforementioned research question through an example. Consider a robot with the goal of reaching inside a gray-shaded non-fixed target box, known as the goal reacher (Appendix A.1). Note that the reward changes as the position of the box changes over time (Figure 1-(a)). We begin by considering a scenario in which the wall-clock time and episode are synchronized, wherein the environment evolves alongside the episode. During each episode k , the agent rollouts a trajectory and iteratively updates the policy N times, with the assumption that one policy update requires one second, and then the agent transitions to the subsequent episode k + 1 . In conventional non-stationary RL environments, it is evident that a larger value of N provides an advantage in terms of a faster adaptation to achieve a near-optimal policy. However, regardless of the chosen value of N , the agent will consistently encounter the same environment in the subsequent episode. Now, consider a scenario in which the wall-clock time and episode are desynchronized. In this context, given a fixed wall-clock time duration t ∈ [ 0 , 10 ] , the agent is faced with the additional task of determining both the total number of interactions (denoted as the total episode K ) and the specific time instances for these interactions { t } 1 ∶ K , where t k ∈ [ 0 , 10 ] , t k -1 < t k for ∀ k ∈ [ K ] . Figure 1(a) shows an agent that interacts with the environment ten times, that is, { t } 1 ∶ K = { 1 , 2 , ..., 10 } , and spends the time interval ( t k , t k + 1 ) to train the policy, which consumes one second ( K = 10 , N = 1 ) . The high frequency of interaction ( K = 10 ) provides adequate data for precise future box position learning ( t = 11 ), yet a single policy update ( N = 1 ) may not approximate the optimal policy. Now, if the agent interacts with the environment four times, i.e. { t } 1 ∶ K = { 1 , 4 , 7 , 10 } (see Figure 1(b)), it becomes feasible to train the policy over a duration of three seconds ( K = 4 , N = 3 ). A longer period of policy training ( N = 3 ) helps the agent in obtaining a near-optimal policy. However, limited observation data ( K = 4 ) and large time intervals ( t ∈ { 11 , 12 , 13 } ) may lead to less accurate box predictions. This example underscores the practical importance of aligning the interaction time of the agent with the environment in non-stationary RL. Determining the optimal sequence { t } 1 ∶ K involves a trade-off between achieving an optimal model and an optimal policy.

Based on the previous example, our key insight is that, in non-stationary environments, the new factor tempo emerges. Informally, tempo refers to the pace of processes occurring in a non-stationary

environment. We define environment tempo as how fast the environment changes and agent tempo as how frequently it updates the policy. Despite the importance of considering the tempo to find the optimal { t } 1 ∶ K , the existing formulations and methods for non-stationarity RL are insufficient. None of the existing works has adequately addressed this crucial aspect.

Our framework, ProST , provides a solution to finding the optimal { t } 1 ∶ K by computing a minimum solution to an upper bound on its performance measure. It proactively optimizes the time sequence by leveraging the agent tempo and the environment tempo. The ProST framework is divided into two components: future policy optimizer ( OPT π ) and time optimizer ( OPT t ), and is characterized by three key features: 1) it is proactive in nature as it forecasts the future MDP model; 2) it is model-based as it optimizes the policy in the created MDP; and 3) it is a synchronizing tempo framework as it finds a suboptimal training time by adjusting how many times the agent needs to update the policy (agent tempo) relative to how fast the environment changes (environment tempo). Our framework is general in the sense that it can be equipped with any common algorithm for policy update. Compared to the existing works [7-9], our approach achieves higher rewards and a more stable performance over time (see Figure 1(c) and Section 5).

We analyze the statistical and computational properties of ProST in a tabular MDP, which is named ProST-T . Our framework learns in a novel MDP, namely elapsed time-varying MDP, and quantifies its non-stationarity with a novel metric, namely time-elapsing variation budget, where both consider wall-clock time taken. We analyze the dynamic regret of ProST-T (Theorem 1) into two components: dynamic regret of OPT π that learns a future MDP model (Proposition 1) and dynamic regret of OPT t that computes a near-optimal policy in that model (Theorem 2, Proposition 2). We show that both regrets satisfy a sublinear rate with respect to the total number of episodes regardless of the agent tempo. More importantly, we obtain suboptimal training time by minimizing an objective that strikes a balance between the upper bounds of those two dynamic regrets, which reflect the tempos of the agent and the environment (Theorem 3). We find an interesting property that the future MDP model error of OPT π serves as a common factor on both regrets and show that the upper bound on the dynamic regret of ProST-T can be improved by a joint optimization problem of learning both different weights on observed data and a model (Theorem 4, Remark 1).

Finally, we introduce ProST-G , which is an adaptable learning algorithm for high-dimensional tasks achieved through a practical approximation of ProST . Empirically, ProST-G provides solid evidence on different reward returns depending on policy training time and the significance of learning the future MDP model. ProST-G also consistently finds a near-optimal policy, outperforming four popular RL baselines that are used in non-stationary environments on three different Mujoco tasks.

Notation

The sets of natural, real, and non-negative real numbers are denoted by N , R , and R + , respectively. For a finite set Z , the notation ∣ Z ∣ denotes its cardinality and the notation ∆ ( Z ) denotes the probability simplex over Z . For X ∈ N , we define [ X ]∶={ 1 , 2 , .., X } . For a variable X , we denote ̂ X as a forecasted (or predicted ) variable at the current time, and ̃ X as the observed value in the past. Also, for any time variable t > 0 and k ∈ N , we denote the time sequence { t 1 , t 2 , .., t k } as { t } 1 ∶ k , and variable X at time t k as X t k . We use the shorthand notation X ( k ) (or X ( k ) ) for X t k (or X t k ). We use the notation { x } a ∶ b to denote a sequence of variables { x a , x a + 1 , ..., x b } , and { x } ( a ∶ b ) to represent a sequence of variables { x t a , x t a + 1 , ..., x t b } . Given two variables x and y , let x ∨ y denote max ( x, y ) , and x ∧ y denote min ( x, y ) . Given two complex numbers z 1 and z 2 , we write z 2 = W ( z 1 ) if z 2 e z 2 = z 1 , where W is the Lambert function. Given a variable x , the notation a = O( b ( x )) means that a ≤ C ⋅ b ( x ) for some constant C > 0 that is independent of x , and the notation a = Ω ( b ( x )) means that a ≥ C ⋅ b ( x ) for some constant C > 0 that is independent of x . We have described the specific details in Appendix C.1.

2 Problem statement: Desynchronizing timelines

2.1 Time-elapsing Markov Decision Process

In this paper, we study a non-stationary Markov Decision Process (MDP) for which the transition probability and the reward change over time. We begin by clarifying that the term episode is agentcentric, not environment-centric. Prior solutions for episode-varying (or step-varying) MDPs operate

under the assumption that the timing of MDP changes aligns with the agent commencing a new episode (or step). We introduce the new concept of time-elapsing MDP . It starts from the wall-clock time t = 0 to t = T , where T is fixed. The time-elapsing MDP at time t ∈ [ 0 , T ] is defined as M t ∶= ⟨S , A , H, P t , R t , γ ⟩ , where S is the state space, A is the action space, H is the number of steps, P t ∶ S×A×S → ∆ (S) is the transition probability , R t ∶S×A→ R is the reward function, and γ is a discounting factor. Prior to executing the first episode, the agent determines the total number of interactions with the environment (denoted as the number of total episode K ) and subsequently computes the sequence of interaction times { t } 1 ∶ K through an optimization problem. We denote t k as the wall-clock time of the environment when the agent starts the episode k . Similar to the existing non-stationary RL framework, the agent's objective is to learn a policy π t k ∶ S → ∆ (A) for all k . This is achieved through engaging in a total of K episode interactions, namely {M t 1 , M t 1 , ..., M t K } , where the agent dedicates the time interval ( t k , t k + 1 ) for policy training and then obtains a sequence of suboptimal policies { π t 1 , π t 2 , ..., π t K } to maximize a non-stationary policy evaluation metric, dynamic regret .

Dealing with time-elapsing MDP instead of conventional MDP raises an important question that should be addressed: within the time duration [ 0 , T ] , what time sequence { t } 1 ∶ K yields favorable trajectory samples to obtain an optimal policy? This question is also related to the following: what is optimal value of K , i.e. the total number of episode that encompasses a satisfactory balance between the amount of observed trajectories and the accuracy of policy training? These interwined questions are concerned with an important aspect of RL, which is the computation of the optimal policy for a given t k . In Section 4, we propose the ProST framework that computes a suboptimal K ∗ and its corresponding suboptimal time sequence { t ∗ } 1 ∶ K ∗ based on the information of the environment's rate of change. In Section 3, we compute a near-optimal policy for { t ∗ } 1 ∶ K ∗ . Before proceeding with the above results, we introduce a new metric quantifying the environment's pace of change, referred to as time-elapsing variation budget.

2.2 Time-elapsing variation budget

Variation budget [10-12] is a metric to quantify the speed with which the environment changes. Driven by our motivations, we introduce a new metric imbued with real-time considerations, named time-elapsing variation budget B ( ∆ t ) . Unlike the existing variation budget, which quantifies the environment's non-stationarity across episodes { 1 , 2 , .., K } , our definition accesses it across { t 1 , t 2 , ..., t K } , where the interval ∆ t = t k + 1 -t k remains constant regardless of k ∈ [ K -1 ] . For further analysis, we define two time-elapsing variation budgets, one for transition probability and another for reward function.

Definition 1 (Time-elapsing variation budgets) . For a given sequence { t 1 , t 2 , .., t K } , assume that the interval ∆ t is equal to the policy training time ∆ π . We define two time-elapsing variation budgets B p ( ∆ π ) and B r ( ∆ π ) as

To enhance the representation of a real-world system using the time-elapsing variation budgets, we make the following assumption.

Assumption 1 (Drifting constants) . There exist constants c > 1 and α r , α p ≥ 0 such that B p ( c ∆ π )≤ c α p B p ( ∆ π ) and B r ( c ∆ π )≤ c α r B r ( ∆ π ) . We call α p and α r the drifting constants.

2.3 Suboptimal training time

Aside from the formal MDP framework, the agent can be informed of varying time-elapsing variation budgets based on the training time ∆ π ∈ ( 0 , T ) even within the same time-elapsing MDP. Intuitively, a short time ∆ π is inadequate to obtain a near-optimal policy, yet it facilitates frequent interactions with the environment, leading to a reduction in empirical model error due to a larger volume of data. On the contrary, a long time ∆ π may ensure obtaining a near-optimal policy but also introduces greater uncertainty in learning the environment. This motivates us to find a suboptimal training time ∆ ∗ π ∈ ( 0 , T ) that strikes a balance between the sub-optimal gap of the policy and the empirical model error. If it exists, then ∆ ∗ π provides a suboptimal K ∗ = ⌊ T / ∆ ∗ π ⌋ , and a suboptimal time sequence where t ∗ k = t 1 + ∆ ∗ π ⋅ ( k -1 ) for all k ∈ [ K ∗ ] . Our ProST framework computes the parameter ∆ ∗ π ,

then sets { t ∗ } 1 ∶ K ∗ , and finally finds a future near-optimal policy for time t ∗ k + 1 at time t ∗ k . In the next section, we first study how to approximate the one-episode-ahead suboptimal policy π ∗ , t k + 1 at time t k when { t } 1 ∶ K is given.

3 Future policy optimizer

Future policy optimizer (OPT)

Time Optimizer(OPTt)

Forecaster

ModelPredicter

Agent

Env.

f(k)

EF

(o(k+1)

g(k)

53

Tempo

Tempo

O(k-w+1)

o(k-1)

(4)0

RL algorithm

Agent

M(k-w+1)

M(k-1)

M(k)

M(k+1)

πk+16

(k+1)

[，，..

K*

-→

tk-(w-1)

k

*

Elapsedtime

For given t k and t k + 1 , the future policy optimizer ( OPT π ), as a module of the ProST framework (Figure 2), computes a near-optimal policy for the future time t k + 1 at time t k via two procedures: (i) it first forecasts the future MDP model of time t k + 1 at time t k utilizing the MDP forecaster function, (ii) it then utilizes an arbitrary policy optimization algorithm within the forecasted MDP model OPT π to obtain a future near-optimal policy π ∗ . t k + 1 .

3.1 MDPforecaster

Our ProST framework is applicable in an environment that meets the following assumption.

Assumption 2 (Observable non-stationary set O ) . Assume that the non-stationarity of M t k is fully characterized by a non-stationary parameter o t k ∈ O . Assume also that the agent observes a noisy non-stationary parameter ˜ o t k at the end of episode k ∈ [ K ] (at time t k ).

It is worth noting that Assumption 2 is mild, as prior research in non-stationary RL has proposed techniques to estimate o ( k ) through latent factor identification methods [4, 13-16], and our framework accommodates the incorporation of those works for the estimation of o ( k ) . Based on Assumption 2, we define the MDP forecaster function g ○ f below.

Definition 2 (MDP forecaster g ○ f ) . Consider two function classes F and G such that F ∶ O w →O and G ∶ S × A × O → R × ∆ (S) , where w ∈ N . Then, for f ( k ) ∈ F and g ( k ) ∈ G , we define MDP forecaster at time t k as ( g ○ f ) ( k ) ∶O w ×S × A → R × ∆ (S) .

The function f ( k ) , acting as a non-stationarity forecaster, predicts a non-stationary parameter ˆ o ( k + 1 ) at time t k + 1 based on the last w observations given by the set { ˜ o } ( k -w + 1 ∶ k ) , i.e., ˆ o ( k + 1 ) = f ({ ˜ o } ( k -w + 1 ,k ) ) . The agent can determine the number of used historical observations, denoted as w , by leveraging information from the environment (Section 4). Then, the function g ( k ) , acting as a model predictor, predicts a reward ̂ R ( k + 1 ) ( s, a ) and a transition probability ̂ P ( k + 1 ) (⋅∣ s, a ) for time t k + 1 , i.e., ( ̂ R ( k + 1 ) , ̂ P ( k + 1 ) ) = g ( k ) ( s, a, ˆ o k + 1 ) . Finally, the OPT π generates the estimated future MDP M ̂ ( k + 1 ) = ⟨S , A , H, ̂ P ( k + 1 ) , ̂ R ( k + 1 ) , γ ⟩ associated with time t k + 1 .

3.2 Finding future optimal policy

Now, consider an arbitrary RL algorithm provided by the user to obtain an optimal policy from the model M ̂ ( k + 1 ) . For a given time sequence { t } 1 ∶ K , the OPT π finds a near-optimal future policy as follows: (1) observe and forecast, (2) optimize using the future MDP model.

(1) Observe and forecast. At time t k , the agent executes an episode k in the environment M ( k ) , completes its trajectory τ ( k ) , and observes the noisy non-stationary parameter ˆ o ( k ) (Assumption 2). The algorithm then updates the function f ( k ) based on the last w observed parameters, and the

function g ( k ) with input from all previous trajectories. Following these updates, the MDP forecaster at time t k predicts ̂ P ( k + 1 ) and ̂ R ( k + 1 ) , thus creating the MDP model M ̂ ( k + 1 ) for time t k + 1 .

(2) Optimize using the future MDP model. Up until time t k + 1 , the agent continually updates the policy within the estimated future MDP M ̂ ( k + 1 ) for a given duration ∆ π . Specifically, the agent rollouts synthetic trajectories ˆ τ ( k + 1 ) in M ̂ ( k + 1 ) , and utilizes any policy update algorithm to obtain a policy ̂ π ( k + 1 ) . Following the duration ∆ π , the agent stops the training by the time t k + 1 and moves to the next episode M ( k + 1 ) with policy ̂ π ( k + 1 ) .

We elaborate on the above procedure in Algorithm 1 given in Appendix F.1.

4 Time optimizer

4.1 Theoretical analysis

We now present our main theoretical contribution, which is regarding the time optimizer ( OPT t ): computing a suboptimal policy training time ∆ ∗ π (the agent tempo). Our theoretical analysis starts with specifying the components of the OPT π optimizer, which we refer to as ProST-T (note that -T stands for an instance in the tabular setting). We employ the Natural Policy Gradient (NPG) with entropy regularization [17] as a policy update algorithm in OPT π . We denote the entropy regularization coefficient as τ , the learning rate as η , the policy evaluation approximation gap arising due to finite samples as δ , and the past reference length for forecaster f as w . Without loss of generality, we assume that each policy iteration takes one second. The theoretical analysis is conducted within a tabular environment, allowing us to relax Assumption 2, which means that one can estimate non-stationary parameters by counting visitation of state and action pairs at time t k , denoted as n ( k ) ( s, a ) , rather than observing them. Additionally, we incorporate the exploration bonus term at time t k into ̂ R ( k + 1 ) , denoted as Γ ( k ) w ( s, a ) , which is proportional to ∑ k τ = k -w + 1 ( n ( τ ) ( s, a )) -1 / 2 and aims to promote the exploration of states and actions that are visited infrequently.

We compute ∆ ∗ π by minimizing an upper bound on the ProST-T 's dynamic regret. The dynamic regret of ProST-T is characterized by the model prediction error that measures the MDP forecaster's error by defining the difference between M ̂ ( k + 1 ) and M ( k + 1 ) through a Bellman equation.

Definition 3 (Model prediction error) . At time t k , the MDP forecaster predicts a model M ̂ ( k + 1 ) and then we obtain a near-optimal policy ̂ π ( k + 1 ) based on M ̂ ( k + 1 ) . For each pair ( s, a ) , we denote the state value function and the state action value function of ̂ π ( k + 1 ) in M ̂ ( k + 1 ) at step h ∈ [ H ] as ̂ V ( k + 1 ) h ( s ) and ̂ Q ( k + 1 ) h ( s, a ) , respectively. We also denote the model prediction error associated with time t k + 1 calculated at time t k as ι ( k + 1 ) h ( s, a ) , which is defined as

We now derive an upper bound on the ProST-T dynamic regret. We expect the upper bound to be likely controlled by two factors: the error of the MDP forecaster's prediction of the future MDP model and the error of the NPG algorithm due to approximating the optimal policy within an estimated future MDP model. This insight is clearly articulated in the next theorem.

Theorem 1 ( ProST-T dynamic regret R ) . Let ι K H = ∑ K -1 k = 1 ∑ H -1 h = 0 ι ( k + 1 ) h ( s ( k + 1 ) h , a ( k + 1 ) h ) and ¯ ι K ∞ ∶= ∑ K -1 k = 1 ∣∣ ¯ ι k + 1 ∞ ∣∣ ∞ , where ι K H is a data-dependent error. For a given p ∈ ( 0 , 1 ) , the dynamic regret of the forecasted policies {̂ π ( k + 1 ) } 1 ∶ K -1 of ProST-T is upper bounded with probability at least 1 -p / 2 as follows:

where R I = ¯ ι K ∞ /( 1 -γ ) -ι K H + C p ⋅ √ K -1 , R II = C II [ ∆ π ] ⋅ ( K -1 ) , and C p , C II [ ∆ π ] are some functions of p , ∆ π , respectively.

Specifically, the upper bound is composed of two terms: R I that originates from the MDP forecaster error between M ( k + 1 ) and M ̂ ( k + 1 ) , and R II that arises due to the suboptimality gap between π ∗ , ( k + 1 ) and ̂ π ( k + 1 ) . Theorem 1 clearly demonstrates that a prudent construction of the MDP

forecaster that controls the model prediction errors and the selection of the agent tempo ∆ π is significant in guaranteeing sublinear rates for R I and R II . To understand the role of the environment tempo in R I , we observe that the MDP forecaster utilizes w previous observations, which inherently encapsulates the environment tempo. We expect the model prediction errors, at least in part, to be controlled by the environment tempo B ( ∆ π ) , so that a trade-off between two tempos can be framed as the trade-off between R I and R II . Hence, it is desirable to somehow minimize the upper bound with respect to ∆ π to obtain a solution, denoted as ∆ ∗ π , which strikes a balance between R I and R II .

4.1.1 Analysis of R II

A direct analysis of the upper bound R I + R II is difficult since its dependence on K is not explicit. To address this issue, we recall that an optimal ∆ ∗ π should be a natural number that guarantees the sublinearity of both R I and R II with respect to the total number of episodes K . We first compute a set N II ⊂ N that includes those values of ∆ π that guarantee the sublinearity of R II , and then compute a set N I ⊂ N that guarantees the sublinearity of R I . Finally, we solve for ∆ ∗ π in the common set N I ∩ N II .

Proposition 1 ( ∆ π bounds for sublinear R II ) . A total step H is given by MDP. For a number ϵ > 0 such that H = Ω ( log ((̂ r max ∨ r max )/ ϵ )) , we choose δ, τ, η to satisfy δ = O( ϵ ) , τ = Ω ( ϵ / log ∣A∣) and η ≤ ( 1 -γ ) / τ , where ̂ r max and r max are the maximum reward of the forecasted model and the maximum reward of the environment, respectively. Define N II ∶={ n ∣ n > 1 ητ log ( C 1 ( γ + 2 ) ϵ ) , n ∈ N } , where C 1 is a constant. Then R II ≤ 4 ϵ ( K -1 ) for all ∆ π ∈ N II .

As a by-product of Proposition 1, the sublinearity of R II can be realized by choosing ϵ = O(( K -1 ) α -1 ) for any α ∈ [ 0 , 1 ) , which suggests that a tighter upper bound on R II requires a smaller ϵ and subsequently a larger ∆ π ∈ N II . The hyperparameter conditions in Proposition 1 can be found in Lemma 1 and 2 in Appendix D.3.

4.1.2 Analysis of R I

We now relate R I to the environment tempo B ( ∆ π ) using the well-established non-stationary adaptation technique of Sliding Window regularized Least-Squares Estimator ( SW-LSE ) as the MDP forecaster [18-20]. The tractability of the SW-LSE algorithm allows to upper-bound the model predictions errors ι K H and ¯ ι K ∞ by the environment tempo extracted from the past w observed trajectories, leading to a sublinear R I as demonstrated in the following theorem.

Theorem 2 (Dynamic regret R I with f = SW-LSE ) . For a given p ∈ ( 0 , 1 ) , if the exploration bonus constant β and regularization parameter λ satisfy β = Ω (∣S∣ H √ log ( H / p )) and λ ≥ 1 , then R I is bounded with probability at least 1 -p as follows:

where C I [ B ( ∆ π )] = ( 1 /( 1 -γ ) + H ) ⋅ B r ( ∆ π ) + ( 1 + H ˆ r max ) γ /( 1 -γ ) ⋅ B p ( ∆ π ) , and C k is a constant on the order of O( K ) .

For a brief sketch of how SW-LSE makes the environment tempo appear in the upper bound, we outline that the model prediction errors are upper-bounded by two forecaster errors, namely P ( k + 1 ) -̂ P ( k + 1 ) and R ( k + 1 ) - ̂ R ( k + 1 ) , along with the visitation count n ( k ) ( s, a ) . Then, the SW-LSE algorithm provides a solution ( ̂ P ( k + 1 ) , ̂ R ( k + 1 ) ) as a closed form of linear combinations of past w estimated values { ̃ P, ̃ R } ( k -w + 1 ∶ w ) . Finally, employing the Cauchy inequality and triangle inequality, we derive two forecasting errors that are upper-bounded by the environment tempo. For the final step before obtaining a suboptimal ∆ ∗ π , we compute N I that guarantees the sublinearity of R I .

Proposition 2 ( ∆ π bounds for sublinear R I ) . Denote B ( 1 ) as the environment tempo when ∆ π = 1 , which is a summation over all time steps. Assume that the environment satisfies B r ( 1 ) + B p ( 1 ) ˆ r max /( 1 -γ ) = o ( K ) and we choose w = O(( K -1 ) 2 / 3 /( C I [ B ( ∆ π )]) 2 / 3 ) . Define the set N I to be { n ∣ n < K, n ∈ N } . Then R I is upper-bounded as R I = O( C I [ B ( ∆ π )] 1 / 3 ( K -1 ) 2 / 3 √ log (( K -1 )/ C I [ B ( ∆ π )])) and also satisfies a sublinear upper bound, provided that ∆ π ∈ N I .

The upper bound on the environment tempo B ( 1 ) in proposition 2 is aligned with our expectation that dedicating an excessively long time to a single iteration may not allow for an effective policy approximation, thereby hindering the achievement of a sublinear dynamic regret. Furthermore, our insight that a larger environment tempo prompts the MDP forecaster to consider a shorter past reference length, aiming to mitigate forecasting uncertainty, is consistent with the condition involving w stated in Proposition 2.

4.1.3 Suboptimal tempo ∆ ∗ π

So far, we have shown that an upper bound on the ProST dynamic regret is composed of two terms R I and R II , which are characterized by the environment tempo and the agent tempo, respectively. Now, we claim that a suboptimal tempo that minimizes ProST 's dynamic regret could be obtained by the optimal solution ∆ ∗ π = arg min ∆ π ∈ N I ∩ N II ( R max I + R max II ) , where R max I and R max II denote the upper bounds on R I and R II . We show that ∆ ∗ π strikes a balance between the environment tempo and the agent tempo since R max I is a non-decreasing function of ∆ π and R max II is a non-increasing function of ∆ π . Theorem 3 shows that the optimal tempo ∆ ∗ π depends on the environment's drifting constants introduced in Assumption 1.

Theorem 3 (Suboptimal tempo ∆ ∗ π ) . Let k Env = ( α r ∨ α p ) 2 C I [ B ( 1 )] , k Agent = log ( 1 /( 1 -ητ )) C 1 ( K -1 )( γ + 2 ) . Consider three cases: case1 : α r ∨ α p = 0 , case2 : α r ∨ α p = 1 , case3 : 0 < α r ∨ α p < 1 or α r ∨ α p > 1 . Then ∆ ∗ π depends on the environment's drifting constants as follows:

Case1: ∆ ∗ π = T .

Case2: ∆ ∗ π = log 1 -ηγ ( k Env / k Agent ) + 1 .

Case3: ∆ ∗ π = exp (-W [-log ( 1 -ητ ) max ( α r ,α p )-1 ]) , provided that the parameters are chosen so that k Agent = ( 1 -ητ ) k Env .

4.2 Improving MDP forecaster

Determining a suboptimal tempo by minimizing an upper bound on R I + R II can be improved by using a tighter upper bound. In Proposition 1, we focused on the Q approximation gap δ to provide a justifiable upper bound on R I + R II . It is important to note that the factor δ arises not only from the finite sample trajectories as discussed in [21], but also from the forecasting error between M ( k + 1 ) and M ̂ ( k + 1 ) . It is clear that the MDP forecaster establishes a lower bound on δ denoted as δ min, which in turn sets a lower bound on ϵ and consequently on R I . This inspection highlights that the MDP forecaster serves as a common factor that controls both R I and R II , and a further investigation to improve the accuracy of the forecaster is necessary for a better bounding on R I + R II .

Our approach to devising a precise MDP forecaster is that, instead of selecting the past reference length w as indicated in Proposition 2, we set w = k , implying the utilization of all past observations. However, we address this by solving an additional optimization problem, resulting in a tighter bound on R I . We propose a method that adaptively assigns different weights q ∈ R k + to the previously observed non-stationary parameters up to time t k , which reduces the burden of choosing w . Hence, we further analyze R I through the utilization of the Weighted regularized Least-Squares Estimator ( W-LSE ) [22]. Unlike SW-LSE , W-LSE does not necessitate a predefined selection of w , but it instead engages in a joint optimization procedure involving the data weights q and the future model ( ̂ P ( k + 1 ) , ̂ R ( k + 1 ) ) . To this end, we define the forecasting reward model error as ∆ r k ( s, a ) = ∣( R ( k + 1 ) - ̂ R ( k + 1 ) )( s, a )∣ and the forecasting transition probability model error as ∆ p k ( s, a ) = ∣∣( P ( k + 1 ) - ̂ P ( k + 1 ) )(⋅ ∣ s, a )∣∣ 1 .

Theorem 4 ( R I upper bound with f = W-LSE ) . By setting the exploration bonus Γ ( k ) ( s, a ) = 1 2 ∆ r k ( s, a ) + γ ˜ r max 2 ( 1 -γ ) ∆ p k ( s, a ) , it holds that

Remark 1 (Tighter R I upper bound with f = W-LSE ) . If the optimization problem of W-LSE is feasible, then the optimal data weight q ∗ provides tighter bounds for ∆ r k and ∆ p k in comparison to SW-LSE , consequently leading to a tighter upper bound for R I . We prove in Lemmas 4 and 6 in Appendix D.3 that ¯ ι K ∞ and -ι K H are upper-bounded in terms of ∆ r k and ∆ p k .

4.3 ProST-G

The theoretical analysis outlined above serves as a motivation to empirically investigate two key points: firstly, the existence of an optimal training time; secondly, the role of the MDP forecaster's contribution to the ProST framework's overall performance. To address these questions, we propose a practical instance, named ProST-G , which particularly extends the investigation in Section 4.2. ProST-G optimizes a policy with the soft actor-critic (SAC) algorithm [23], utilizes the integrated autoregressive integrated moving average (ARIMA) model for the proactive forecaster f , and uses a bootstrap ensemble of dynamic models where each model is a probabilistic neural network for the model predictor g . We further discuss specific details of ProST-G in Appendix F.3 and in Algorithm 3.

5 Experiments

We evaluate ProST-G with four baselines in three Mujoco environments each with five different non-stationary speeds and two non-stationary datasets.

(1) Environments: Non-stationary desired posture. Wemakethe rewards in the three environments non-stationary by altering the agent's desired directions. The forward reward R f t changes as R f t = o t ⋅ s R f t , where s R f is the original reward from the Mujoco environment. The non-stationary parameter o k is generated from the sine function with five different speeds and from the real data A and B . We then measure the time-elapsing variation budget by ∑ K -1 k = 1 ∣ o k + 1 -o k ∣ . Further details of the environment settings can be found in Appendix D.1.1.

(2) Benchmark methods. Four baselines are chosen to empirically support our second question: the significance of the forecaster. MBPO is the state-of-the-art model-based policy optimization [24]. Pro-OLS is a policy gradient algorithm that predicts the future performance and optimizes the predicted performance of the future episode [7]. ONPG is an adaptive algorithm that performs a purely online optimization by fine-tuning the existing policy using only the trajectory observed online [8]. FTRL is an adaptive algorithm that performs follow-the-regularized-leader optimization by maximizing the performance on all previous trajectories [9].

6 Discussions

6.1 Performance compare

The outcomes of the experimental results are presented in Table 1. The table summarizes the average return over the last 10 episodes during the training procedure. We have illustrated the complete training results in Appendix E.3. In most cases, ProST-G outperforms MBPO in terms of rewards, highlighting the adaptability of the ProST framework to dynamic environments. Furthermore, except for data A and B , ProST-G consistently outperforms the other three baselines. This supports our motivation of using the proactive model-based method for a higher adaptability in non-stationary environments compared to state-of-the-art model-free algorithms (Pro-OLS, ONPG, FTRL). We elaborate on the training details in Appendix E.2.

6.2 Ablation study

An ablation study was conducted on the two aforementioned questions. The following results support our inspection of Section 4.2 and provide strong grounds for Theorem 3.

6

=1, G=38

3

f=SA (w=3)

2.5

f=SA (w=3)

t = 2, G=76

f=SA (w=5)

f=SA (w=5)

2.0

=3, G=114

2

f=SA (w=7)

error

f=SA (w=7)

Rewards

4

= 4, G=152

Rewards

S

f=ARIMA

1.5

f=ARIMA

=5,G=190

1

MBPO

MBPO

1.0

2

M

0

0.5

0

0.0

0

50

100

150

50

100

150

200

0

50

100

150

200

(a)

(b)

(c)

Suboptimal ∆ ∗ π . The experiments are performed over five different policy training times ∆ π ∈ { 1 , 2 , 3 , 4 , 5 } , aligned with SAC's number of gradient steps G ∈ { 38 , 76 , 114 , 152 , 190 } , under a fixed environment speed. Different from our theoretical analysis, we set ∆ t = 1 with G = 38 . We generate o k = sin ( 2 π ∆ π k / 37 ) , which satisfies Assumption 1 (see Appendix E.1). The shaded areas of Figures 3 (a), (b) and (c) are 95 % confidence area among three different noise bounds of 0.01,0.02 and 0.03 in o k . Figure 3(a) shows ∆ t = 4 is close to the optimal G ∗ among five different choices.

Functions f, g . We investigate the effect of the forecaster f 's accuracy on the framework using two distinct functions: ARIMA and a simple average (SA) model, each tested with three different the values of w . Figure 3(b) shows the average rewards of the SA model with w ∈ { 3 , 5 , 7 } and ARIMA model (four solid lines). The shaded area is 95 % the confidence area among 4 different speeds { 1 , 2 , 3 , 4 } . Figure 3(c) shows the corresponding model error. Also, we investigate the effect of the different model predictor g by comparing MBPO (reactive-model) and ProST-G with f = ARIMA (proactive-model) in Figure 3(c). The high returns from ProST-G with f = ARIMA, compared to those from MBPO, empirically support that the forecasting component of the ProST framework can provide a satisfactory adaptability to the baseline algorithm that is equipped with. Also, Figures 3(b) and 3(c) provide empirical evidence that the accuracy of f is contingent on the sliding window size, thereby impacting the model accuracy and subsequently influencing the agent's performance.

7 Conclusion

This work offers the first study on the important issue of time synchronization for non-stationary RL. To this end, we introduce the concept of the tempo of adaptation in a non-stationary RL, and obtain a suboptimal training time. We propose a Proactively Synchronizing Tempo (ProST) framework, together with two specific instances ProST-T and ProST-G . The proposed method adjusts an agent's tempo to match the tempo of the environment to handle non-stationarity through both theoretical analysis and empirical evidence. The ProST framework provides a new avenue to implement reinforcement learning in the real world by incorporating the concept of adaptation tempo.

As a future work, it is important to generalize the proposed framework to learn a safe guarantee policy in a non-stationary RL by considering the adaptation tempo of constraint violations [25, 26]. Another generalization is to introduce an alternative dynamic regret metric, enabling a fair performance comparison among agents, even when they have varying numbers of total episodes. Another future work is to find an optimal tempo of the distribution correction in offline non-stationary RL, specifically how to adjust the relabeling function to offline data in a time-varying environment that is dependent on the tempo of the environment [27, 28].

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint

arXiv:1312.5602 , 2013.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature , 529:484-489, 2016.

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. In International Conference on Machine Learning , 2019.

Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research , 22(289):1-39, 2021.

Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45 (11):13344-13362, 2023.

Sindhu Padakandla. A survey of reinforcement learning algorithms for dynamically varying environments. ACM Computing Surveys (CSUR) , 54(6):1-25, 2021.

Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Machine Learning , pages 1414-1425. PMLR, 2020.

Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations , 2018.

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning , pages 1920-1930. PMLR, 2019.

Yuhao Ding, Ming Jin, and Javad Lavaei. Non-stationary risk-sensitive reinforcement learning: Near-optimal dynamic regret, adaptive detection, and separation design. Proceedings of the AAAI Conference on Artificial Intelligence , 37(6):7405-7413, 2022.

STEVEN J BRADTKE. Linear least-squares algorithms for temporal difference learning. Machine Learning , 22:33-57, 1996.

Yonatan Gur, Assaf J. Zeevi, and Omar Besbes. Stochastic multi-armed-bandit problem with non-stationary rewards. In NIPS , 2014.

Xiaoyu Chen, Xiangming Zhu, Yufeng Zheng, Pushi Zhang, Li Zhao, Wenxue Cheng, Peng CHENG, Yongqiang Xiong, Tao Qin, Jianyu Chen, and Tie-Yan Liu. An adaptive deep rl method for non-stationary environments with piecewise stable context. In Advances in Neural Information Processing Systems , volume 35, pages 35449-35461, 2022.

Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. In International Conference on Learning Representations , 2022.

Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for nonstationary reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 31957-31971, 2022.

Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. Rl for latent mdps: Regret guarantees and a lower bound. Advances in Neural Information Processing Systems , 34:24523-24534, 2021.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under nonstationarity. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1079-1087. PMLR, 2019.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Hedging the drift: Learning to optimize under nonstationarity. Management Science , 68(3):1696-1713, 2022.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for nonstationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning , pages 1843-1854. PMLR, 2020.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4): 2563-2578, 2022.

Vitaly Kuznetsov and Mehryar Mohri. Theory and algorithms for forecasting time series. arXiv preprint arXiv:1803.05814 , 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019.

Ming Jin and Javad Lavaei. Stability-certified reinforcement learning: A control-theoretic perspective. IEEE Access , 8:229086-229100, 2020.

Samuel Pfrommer, Tanmay Gautam, Alec Zhou, and Somayeh Sojoudi. Safe reinforcement learning with chance-constrained model predictive control. In Learning for Dynamics and Control Conference , pages 291-303. PMLR, 2022.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. International Conference on Learning Representations , 2022.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning . PMLR, 2021.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Ba¸ sar. Model-free non-stationary rl: Near-optimal regret and applications in multi-agent rl and inventory control. arXiv preprint arXiv:2010.03161 , 2020.

Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023. ISBN 978-1-57735-880-0.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations , 2016.

Wen Sun, Geoffrey J Gordon, Byron Boots, and J Bagnell. Dual policy iteration. Advances in Neural Information Processing Systems , 31, 2018.

Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. International Conference on Learning Representations , 2019.

Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. Advances in Neural Information Processing Systems , 33:6743-6754, 2020.

Yash Chandak, Scott Jordan, Georgios Theocharous, Martha White, and Philip S Thomas. Towards safe policy improvement for non-stationary mdps. Advances in Neural Information Processing Systems , 33:9156-9168, 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning in non-stationary environments. The authors propose a proactive tempo-control model-based (PTM) framework to adjust how often the learning agent to adjust its policy to the environment tempo. Two different variants of PTM are considered: The PTM-T variant uses natural policy gradient as the base algorithm and achieves sublinear dynamic regret. The regret bound is further improved when the forecaster assigns different weights to the observed data. The second variant PTM-G is more empirical and is shown to outperform four existing baselines on three different Mujoco tasks.\n\nSTRENGTHS:\nThis paper studies an interesting and important problem in reinforcement learning: The quick adaptation of the learning algorithm to a non-stationary environment. The proposed solution of proactive tempo-control is interesting and novel as far as I know. The theoretical derivations look solid and convincing. This paper also includes a decent amount of numerical results with ablation studies in addition to the theoretical contributions, which also looks encouraging.\n\nWEAKNESSES:\n1.\tWhile I am sure that the authors have a lot of interesting results to share, the writing of the paper fails to convey some of the important ideas, which prevents me from fully appreciating the contributions of the work. The presentations of the methodology and results in the paper are somewhat unclear and sometimes can be confusing. There are a few things that are unclear to me and can certainly be improved:\n\na. The main theoretical results are not clearly stated. The authors build up the theoretical results through a series of theorems and propositions, but it is hard to keep track of what the strongest results are, given that there are numerous results presented. The reader needs to combine a few theorems to see the exact form of the dynamic regret bound, which is not reader-friendly. The authors could consider providing the strongest, take-home results of the work in one theorem. \n\nb. The main text of the paper is not very self-contained, which adds to the confusion. The definitions of some notations and even algorithm descriptions (e.g., Algorithms 1 and 2) are deferred to appendices, which really should have been included in the main text. My suggestion is that some of the intermediate results in Section 4 can be moved to the appendices to allow more space for many other sections. \n\n2.\tThe main setup of the problem formulation is a bit unclear to me. The authors mentioned the main “trade-off” between interacting with the environment more often to gather data vs. performing more policy optimization using collected samples. Why is it even a trade-off and why can’t we do both of them at the same time? The only reason that I can think of is computation complexity, which is less critical and was not even discussed in the paper. \n\n3.\tIt is unclear to me how the problem formulation is related to the standard setup in non-stationary RL, e.g., [Cheung et al., 2020]. The definition of the time elapsing variation budget is a bit ad hoc and should be better explained. It is also quite confusing that the formulation considers both a finite-horizon $H$ and a reward discounting factor $\\gamma$, while usually people only choose one of the two. The definitions of the values functions are also missing (which are included in the appendix but should definitely have been included in the main text).\n\n4.\tThe assumption that $B_p(\\Delta_t) = c^{\\alpha_p} B_p(\\Delta_t)$ seems a bit ad hoc to me and probably needs more justification. Similarly, it is also unclear what Assumption 1 means and the authors could provide some examples on how it is satisfied. \n\n5.\tIn the simulations, it seems that some of the selected comparison baselines are originally designed for the stationary environment. It would be unfair to compare with these algorithms in non-stationary environments. I wonder how the proposed method compares to some existing solutions specifically designed for non-stationary RL (e.g., some simple sliding-window or restarting-based algorithms). \n\n6.\tA related work section is missing (and again is included in the appendix but should have been included in the main text to be self-contained). There are also a few important closely related works that are not discussed in the references. Just to name a few that I am aware of:\n\na.\tWei, Chen-Yu, and Haipeng Luo. \"Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach.\" Conference on learning theory, 2021.\n\nb.\tMao, Weichao, et al. \"Near-optimal model-free reinforcement learning in non-stationary episodic mdps.\" International Conference on Machine Learning, 2021.\n\nQUESTIONS:\nPlease consider first answering my questions in the “Weaknesses” section. The additional questions listed below are comparably not major. \n\n1.\tHow does the environment tempo formulation deal with the difference between the frequency of environment non-stationarity (i.e., frequent, slow, gradual variations) and the magnitude of non-stationarity (i.e., abrupt variations). \n\n2.\tIn Eq (2.1), the authors measure the transition variations in terms of the infinity-norm, which is different from the more popular metric of 1-norm in existing non-stationary RL works (e.g., Cheung et al., 2020). I wonder why such a difference occurs.\n\n3.\tSince the authors mentioned the word “meta-algorithm”, I wonder how the proposed method is related to meta-learning?\n\nLIMITATIONS:\nThe limitations are briefly discussed in the conclusions section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework for handling non-stationary environments in reinforcement learning (RL) called Proactive Tempo-control Model-based (PTM) framework. The authors argue that in non-stationary environments, an additional factor emerges alongside the classical exploration-exploitation trade-off: the tempo of adaptation. The PTM framework is designed to adjust the policy-update tempo (algorithm tempo) to match the environment tempo. The authors propose two instances of the PTM framework, PTM-T and PTM-G, and provide theoretical analysis and empirical evidence to support their claims.\n\nSTRENGTHS:\n- The concept of tempo adaptation in non-stationary RL environments is an interesting approach.\n- The proposed PTM framework could have implications for the field of RL, particularly in real-world applications where environments are often non-stationary.\n- The paper provides theoretical analysis for the proposed method.\n\nWEAKNESSES:\n- Although the author tells an interesting story about \"adapt to adapt,\" the proposed PTM algorithm essentially attempts to learn a gradually changing MDP, which does not seem to have any fundamental difference from some previous works like [1][2].\n- There is a significant gap between the theoretical results of the paper and the practical methods. Although the author has used a considerable amount of theoretical analysis in the main text to try to demonstrate that PTM method can achieve smaller dynamic regret with sublinear iterations, only simple performance curves are shown in the experimental section without further demonstrating the relationship between dynamic regret and iterations, which should explain the principle of improvement in the PTM algorithm.\n- As cited and discussed by the author (line 29-30), some previous works have also been able to effectively address non-stationarity. However, in the selected baseline, MBPO is just a standard model-based RL algorithm without specific optimization for non-stationarity. Additionally, other baselines seem to be relatively outdated. Why not compare with newer references like [3], etc?\n- This paper uses a lot of complex and scattered math symbols, making it confusing for readers when reading the theoretical part. For example, in line 207, the author uses an icon symbol to represent \"Alg\". In line 264, the author uses diamond and square symbols to represent $r$, $p$ and $\\mathcal{P}$, etc. Why not simply use the letters or with some subscriptions directly? In some contextual formulas (e.g. formula 4.2, 4.3, 4.4, 4.6 and line 271-276), the author interchangeably uses these icon symbols and original letters. When readers read through this paper, they may need to repeatedly recall what these complex symbols actually represent.\n\n[1] Deep reinforcement learning amidst continual structured non-stationarity. Xie, et al. (ICML 2021)\n\n[2] Meta-reinforcement learning by tracking task non-stationarity. Poiani, et al. (IJCAI 2021)\n\n[3]Factored adaptation for non-stationary reinforcement learning. Feng, et al. (NeurIPS 2022)\n\nQUESTIONS:\n- In proposition 3, how are the different cases of environment tempos reflected in the experimental setting?\n- In Figure b-2, the learning curves of the algorithms seem to be in their early stages. Why not plot the complete curve until convergence?\n\nLIMITATIONS:\n- The writing and logic of this paper is not good enough, making it hard for readers to follow. \n- There is a gap between theory and practice, and there are no detailed experimental demonstrations for theoretical conclusions.\n- Additionally, there is a lack of experiments comparing with the recent SOTA baselines. \n- Although the author proposes an interesting perspective on \"adapt to adapt,\" fundamentally it does not differ from works on actively learning changes in MDPs. \n\nI suggest that the author reorganize the use of math symbols and consider simplifying unnecessary theoretical analysis in the main text. Furthermore, provide a more detailed explanation of the PTM-G algorithm in Section 4.2 and supplement with additional experiments as mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces the tempo of adaptation in a non-stationary RL problem. The authors provide Proactive Tempo control Model-based (PTM) framework, and two specific instances PTM-T and PTM-G. By adjusting the tempo of the algorithms, the proposed algorithm can match the tempo of the environment to address non-stationarity. This claimed property is demonstrated by both theoretical analysis and empirical results. The proposed PTM framework helps the RL algorithm to be implemented in real-world settings. The authors show that the PTM framework achieves a higher online return than the existing methods and provides empirical evidence of the existence of an optimal algorithm tempo with the comprehensive experimental evaluation of various Mujoco tasks.\n\nSTRENGTHS:\n(1) Interesting topic for real-world applications: This paper focuses on a practical setting for non-stationary RL by introducing a “time-elapsing variation budget”, which not only measures non-stationarity but also considers actual time taken. This setting is common and has great potential in real-world applications, and this work will encourage more interest in this direction.\n\n(2) Solid theoretical analysis: This paper provides a dedicated mathematical formulation for the tasks and provides a detailed theoretical analysis and discussion of the proposed methods under the tabular settings. They also provide a discussion on more general cases in the supplementary material.\n\n(3) Comprehensive experimental evaluation: This work provides comprehensive experimental results both with sufficient baseline methods and testing tasks to demonstrate the strength of their work.\n\nWEAKNESSES:\n(1) Complicated symbolic notation: The author uses too many fancy symbols for notations that are hard to identify, write, pronounce, and memorize. I would be more than happy to see a simplified version, and I believe by reducing the symbolic complexity, this paper will be much easier to follow.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a solution to nonstationary RL issues through a model-based framework called Tempo-Control (PTM). Specifically, the authors identify a new trade-off in nonstationary RL problems: the trade-off between learning an accurate model and learning an optimal policy. Based on this, a new framework is designed to allow the agent to find an appropriate tempo for adaptation. The authors provide theoretical analysis for the tabular version of the framework and a strategy for complex RL scenarios. Empirical evaluation results confirm the effectiveness of the scheme as well as its theoretical optimality properties. Overall, this paper offers insightful analysis on nonstationary RL problems and rigorous theoretical study on the method. I am inclined to give an \"accept\" rating.\n\nSTRENGTHS:\n[**About the framework's motivation**] The author's analysis of the trade-off between adaptation tempo and environment tempo is quite insightful. Additionally, the critique of the existing three research lines in nonstationary RL could provide inspiration to the field of nonstationary RL or RL in general.\n\n[**About the theoretical analysis**] The paper offers rigorous theoretical analysis for the tabular version, providing solid theoretical guarantees for subsequent applications.\n\n[**About method design**] The modeling method for adaptation tempo and environment tempo is clear and simple, offering strong reproducibility and portability.\n\n[**About experiments**] The experiments use common nonstationary (deep) RL benchmarks, and the results are quite good, with a significant margin over the baselines. Furthermore, the ablation studies also confirm the previous theories on optimality.\n\nWEAKNESSES:\nNote: Most of the weaknesses listed below are more like questions, discussion points, or suggestions, rather than outright flaws. Any clarifications provided by the authors would be very welcome and appreciated. Given the limited time for rebuttal, it is not necessary to fully supplement experiments for comparison with these papers. Some clarifications and discussions on these methods would be greatly appreciated.\n\n[**About the latent variable**] The authors assume that the non-stationary variable $\\hat{o}_k$ is observable, but in real complex scenarios, it is challenging to observe the non-stationary variable. Additionally, certain specific assumptions are required about this latent variable (such as the function of the latent variable's change with time, dimensions, etc.). Could the authors provide more clarification on this point?\n\n[**About the presentation**]The authors might consider placing PTM-G after the third section and moving Algorithm 3 from the appendix to the main text. This could help readers better understand the overall framework design and flow.\n\n[**About the experiment**] In Fig. 3 (b-2), it seems that convergence has not been reached at the end; consider providing a longer range for the x-axis. Also, the x-axis is not labeled in Fig. 3.\n\nQUESTIONS:\nI have included the questions in the above section.\n\nLIMITATIONS:\nThe authors have given a detailed analysis of the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to look at non-stationarity in reinforcement learning (RL) from a new point of view. While previous approaches did not generally consider the practical implications of the time elapsed while learning a policy, the idea of the current paper is to consider exactly these implications. \n\nA trade-off is posed between spending time learning a stronger policy versus collecting more samples and learning a better model. After defining the problem, a practical solution is proposed. A forecaster fits a model of the process driving the non-stationarity, as well as the set a model of Markov decision processes (MDP) conditioned on it. An Rl algorithm, potentially any algorithm, is then plugged to the forecasted MDP and is optimised for several iterations. \n\nA theoretical analysis is conducted to demonstrate that, as anticipated, the regret drastically depends on the number of policy iterations. Fortunately, under certain conditions on the MDP and the algorithm, it is shown that a sub-linear regret can be obtained in the proposed framework. It results that there exists an optimal number of iterations which depends on the MDP itself. \n\nFinally, an empirical study demonstrates that a practical algorithm, based on the previous theory, is able to compete and outperform several baselines, in a range of non-stationary problems. Furthermore, an ablation study is proposed to illustrate the theoretical results.\n\nSTRENGTHS:\nDespite my rating in \"contribution\", this paper comes with an interesting idea. \nIn a practical problem, the time spent while learning a policy may be time not spent to sample new data.  It is an interesting theoretical perspective. As expected, the authors demonstrate that it is critical to take this time into account. I believe that this theory could have great implications and am not aware of previous works going in the exact same direction.\n\nThe practical algorithm follows a simple idea, which I like. It comes with both theoretical and empirical achievements. \n\nThe theoretical analysis is helpful and and well conducted. I have not checked demonstrations in detail, but many proof steps are classic in the literature.  The conclusion of the theoretical analysis is great: \"tempo\" in the MDP and in the algorithms should be in harmony. \n\nThe experimental section is concise but holds enough value to underline the validity of the framework.\n\nWEAKNESSES:\nMy rating for the \"contribution\" results from my doubts about the definition of the framework. Despite originating from an interesting idea, I question the following points:\n - The non-stationarity is not defined clearly enough. An agent is placed in an MDP where it can either sample a trajectory or train its policy. Why wouldn't the MDP change during the trajectory? If the agent can choose when to sample data in a stationary MDP, and that the non-stationarity is bounded as in line 119, it seems that too much control over the non-stationarity is offered to the agent.\n - The terms B_p(G) and B_r(G) are not given more details in the main paper. I understand that they satisfies the equation of line 119, but what is the meaning of Delta_t=1? This is also important to understand the experiments, is a step of non-stationarity corresponding to Delta_t=1 and therefore to an iteration of NPG? \n - I am concerned about the result of Theorem 2. I understand that, for a fixed K, there should be an optimal tempo for the policy iteration. But I miss a result for an agent that would have a budget in terms of time. How does a well trained agent sampling less trajectories compare to a poorly trained agent sampling more trajectories in the same time? The optimal tempo could be impacted by these considerations. \n - A related comment about the regret. In the regret formula, the reference is the optimal value function in MDP k, that is, at sampling times synchronised with the algorithm. What if the algorithm is compared to an optimal policy at different tempo ? \n - These two previous comments can be gathered as follow: what if the agent is compared to any agent acting in the same amount of time? \n - I expect that the training takes more time as more data is collected. Delta_t could refer to the longest time of them all, but how can the agent be sure that Delta_t has elapsed always before sampling? This is related to my comment on the agent having an impact on the non-stationarity of the environment. \n - When the agent optimises its policy, as explained in 3.2, paragraph 2), a synthetic trajectory is rolled out. What if the rollout has a comparable time with respect to sampling a trajectory. Shouldn't the number of rollouts add to the time elapsed? In expectation it is irrelevant but it becomes necessary for a high-probability results. Similarly, why is the result of Proposition 3, case 1, suggesting G=\\infty? Only a single trajectory is needed?\n\nQUESTIONS:\n- Why is the framework considering non-stationarity inter-episode and not intra-episode? \n - Proposition 1 offers a sublinear regret with a condition that binds H and epsilon, this doesn't seem great for applications where H is fixed beforehand. What do the authors think about it? \n - What does \\Delta_t=1 correspond to in the experiments?  \n - Is there a simple way to compute a regret that compares to the best algorithm given a fixed time elapsed instead of number of episodes (K) fixed?\n - What would be the optimal number of iterations in that case? \n - In the experiments, why is the performance compared over the last 10 episodes only? What about the regret throughout learning?\n\nLIMITATIONS:\nExtra: \n - the text in figures 1.c) and 1.d) i9s too small \n - type line 302: \"the state-of-the-art model-based model-free algorithm\"", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}