SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyunin Lee $^{1,*}$  Yuhao Ding $^{1}$  Jongmin Lee $^{1}$  Ming Jin $^{2,*}$
Javad Lavaei<sup>1</sup> Somayeh Sojoudi<sup>1</sup>
$^{1}$ UC Berkeley, Berkeley, CA 94709
$^{2}$ Virginia Tech, Blacksburg, VA 24061
{hyunin,yuhao_ding,jongmin.lee,lavaei,sojoudi}@berkeley.edu
jinming@vt.edu
We first raise and tackle a "time synchronization" issue between the agent and the environment in non-stationary reinforcement learning (RL), a crucial factor hindering its real-world applications. In reality, environmental changes occur over wall-clock time (t) rather than episode progress  $(k)$ , where wall-clock time signifies the actual elapsed time within the fixed duration  $t \in [0, T]$ . In existing works, at episode  $k$ , the agent rolls a trajectory and trains a policy before transitioning to episode  $k + 1$ . In the context of the time-desynchronized environment, however, the agent at time  $t_k$  allocates  $\Delta t$  for trajectory generation and training, subsequently moves to the next episode at  $t_{k+1} = t_k + \Delta t$ . Despite a fixed total number of episodes  $(K)$ , the agent accumulates different trajectories influenced by the choice of interaction times  $(t_1, t_2, \dots, t_K)$ , significantly impacting the suboptimality gap of the policy. We propose a Proactively Synchronizing Tempo (ProST) framework that computes a suboptimal sequence  $\{t_1, t_2, \dots, t_K\} (= \{t\}_{1:K})$  by minimizing an upper bound on its performance measure, i.e., the dynamic regret. Our main contribution is that we show that a suboptimal  $\{t\}_{1:K}$  trades-off between the policy training time (agent tempo) and how fast the environment changes (environment tempo). Theoretically, this work develops a suboptimal  $\{t\}_{1:K}$  as a function of the degree of the environment's non-stationarity while also achieving a sublinear dynamic regret. Our experimental evaluation on various high-dimensional nonstationary environments shows that the ProST framework achieves a higher online return at suboptimal  $\{t\}_{1:K}$  than the existing methods.
The prevailing reinforcement learning (RL) paradigm gathers past data, trains models in the present, and deploys them in the future. This approach has proven successful for stationary Markov decision processes (MDPs), where the reward and transition functions remain constant [1-3]. However, challenges arise when the environments undergo significant changes, particularly when the reward and transition functions are dependent on time or latent factors [4-6], in non-stationary MDPs. Managing non-stationarity in environments is crucial for real-world RL applications. Thus, adapting to changing environments is pivotal in non-stationary RL.
This paper addresses a practical concern that has inadvertently been overlooked within traditional nonstationary RL environments, namely, the time synchronization between the agent and the environment. We raise the impracticality of utilizing episode-varying environments in existing non-stationary RL
* Corresponding authors. This work was supported by grants from ARO, ONR, AFOSR, NSF, and the UC Noyce Initiative.
(c)
Figure 1: (a) 2D goal researcher in a time-desynchronized environment for one policy update, where the agent learns an inaccurate policy on an accurate model; (b) For three policy updates, the agent learns a near-optimal policy on an inaccurate model; (c) Rewards per episode in 2D goal researcher with four model-free baselines, where ProST-T* is one of our proposed methods.
research, as such environments do not align with the real-world scenario where changes occur regardless of the agent's behavior. In an episode-varying environment, the agent has complete control over determining the time to execute the episode  $k$ , the duration of policy training between the episodes  $k$  and  $k + 1$ , and the transition time to the episode  $k + 1$ . The issue stems from the premise that the environment undergoes dynamic changes throughout the course of each episode, with the rate of non-stationarity contingent upon the behavior exhibited by the agent. However, an independent wall-clock time (t) exists in a real-world environment, thereby the above three events are now recognized as wall-clock time  $t_k$ , training time  $\Delta t$ , and  $t_{k+1}$ . The selection of interaction times  $(t_k, t_{k+1})$  has a notable impact on the collected trajectories, while the interval  $t_{k+1} - t_k$  establishes an upper limit on the duration of training  $(\Delta t)$ . This interval profoundly influences the suboptimality gap of the policy. In the context of a time-desynchronized environment, achieving an optimal policy requires addressing a previously unexplored question: the determination of the optimal time sequence  $\{t_1, t_2, \dots, t_K\} (= \{t\}_{1:K})$  at which the agent should interact with the environment.
We elucidate the significance of the aforementioned research question through an example. Consider a robot with the goal of reaching inside a gray-shaded non-fixed target box, known as the goal reacher (Appendix A.1). Note that the reward changes as the position of the box changes over time (Figure 1-(a)). We begin by considering a scenario in which the wall-clock time and episode are synchronized, wherein the environment evolves alongside the episode. During each episode  $k$ , the agent rollouts a trajectory and iteratively updates the policy  $N$  times, with the assumption that one policy update requires one second, and then the agent transitions to the subsequent episode  $k + 1$ . In conventional non-stationary RL environments, it is evident that a larger value of  $N$  provides an advantage in terms of a faster adaptation to achieve a near-optimal policy. However, regardless of the chosen value of  $N$ , the agent will consistently encounter the same environment in the subsequent episode. Now, consider a scenario in which the wall-clock time and episode are desynchronized. In this context, given a fixed wall-clock time duration  $t \in [0, 10]$ , the agent is faced with the additional task of determining both the total number of interactions (denoted as the total episode  $K$ ) and the specific time instances for these interactions  $\{t\}_{1:K}$ , where  $t_k \in [0, 10]$ ,  $t_{k-1} < t_k$  for  $\forall k \in [K]$ . Figure 1(a) shows an agent that interacts with the environment ten times, that is,  $\{t\}_{1:K} = \{1, 2, \dots, 10\}$ , and spends the time interval  $(t_k, t_{k+1})$  to train the policy, which consumes one second  $(K = 10, N = 1)$ . The high frequency of interaction  $(K = 10)$  provides adequate data for precise future box position learning  $(t = 11)$ , yet a single policy update  $(N = 1)$  may not approximate the optimal policy. Now, if the agent interacts with the environment four times, i.e.,  $\{t\}_{1:K} = \{1, 4, 7, 10\}$  (see Figure 1(b)), it becomes feasible to train the policy over a duration of three seconds  $(K = 4, N = 3)$ . A longer period of policy training  $(N = 3)$  helps the agent in obtaining a near-optimal policy. However, limited observation data  $(K = 4)$  and large time intervals  $(t \in \{11, 12, 13\})$  may lead to less accurate box predictions. This example underscores the practical importance of aligning the interaction time of the agent with the environment in non-stationary RL. Determining the optimal sequence  $\{t\}_{1:K}$  involves a trade-off between achieving an optimal model and an optimal policy.
Based on the previous example, our key insight is that, in non-stationary environments, the new factor tempo emerges. Informally, tempo refers to the pace of processes occurring in a non-stationary
environment. We define environment tempo as how fast the environment changes and agent tempo as how frequently it updates the policy. Despite the importance of considering the tempo to find the optimal  $\{\mathsf{t}\}_{1:K}$ , the existing formulations and methods for non-stationarity RL are insufficient. None of the existing works has adequately addressed this crucial aspect.
Our framework, ProST, provides a solution to finding the optimal  $\{\mathfrak{t}\}_{1:K}$  by computing a minimum solution to an upper bound on its performance measure. It proactively optimizes the time sequence by leveraging the agent tempo and the environment tempo. The ProST framework is divided into two components: future policy optimizer  $(\mathrm{OPT}_{\pi})$  and time optimizer  $(\mathrm{OPT}_{t})$ , and is characterized by three key features: 1) it is proactive in nature as it forecasts the future MDP model; 2) it is model-based as it optimizes the policy in the created MDP; and 3) it is a synchronizing tempo framework as it finds a suboptimal training time by adjusting how many times the agent needs to update the policy (agent tempo) relative to how fast the environment changes (environment tempo). Our framework is general in the sense that it can be equipped with any common algorithm for policy update. Compared to the existing works [7-9], our approach achieves higher rewards and a more stable performance over time (see Figure 1(c) and Section 5).
We analyze the statistical and computational properties of ProST in a tabular MDP, which is named ProST-T. Our framework learns in a novel MDP, namely elapsed time-varying MDP, and quantifies its non-stationarity with a novel metric, namely time-ellipsis variation budget, where both consider wall-clock time taken. We analyze the dynamic regret of ProST-T (Theorem 1) into two components: dynamic regret of  $\mathsf{OPT}_{\pi}$  that learns a future MDP model (Proposition 1) and dynamic regret of  $\mathsf{OPT}_{\mathrm{t}}$  that computes a near-optimal policy in that model (Theorem 2, Proposition 2). We show that both regrets satisfy a sublinear rate with respect to the total number of episodes regardless of the agent tempo. More importantly, we obtain suboptimal training time by minimizing an objective that strikes a balance between the upper bounds of those two dynamic regrets, which reflect the tempos of the agent and the environment (Theorem 3). We find an interesting property that the future MDP model error of  $\mathsf{OPT}_{\pi}$  serves as a common factor on both regrets and show that the upper bound on the dynamic regret of ProST-T can be improved by a joint optimization problem of learning both different weights on observed data and a model (Theorem 4, Remark 1).
Finally, we introduce ProST-G, which is an adaptable learning algorithm for high-dimensional tasks achieved through a practical approximation of ProST. Empirically, ProST-G provides solid evidence on different reward returns depending on policy training time and the significance of learning the future MDP model. ProST-G also consistently finds a near-optimal policy, outperforming four popular RL baselines that are used in non-stationary environments on three different Mujoco tasks.
The sets of natural, real, and non-negative real numbers are denoted by  $\mathbb{N},\mathbb{R}$ , and  $\mathbb{R}_{+}$ , respectively. For a finite set  $Z$ , the notation  $|Z|$  denotes its cardinality and the notation  $\Delta(Z)$  denotes the probability simplex over  $Z$ . For  $X \in \mathbb{N}$ , we define  $[X] := \{1,2,\dots,X\}$ . For a variable  $X$ , we denote  $\widehat{X}$  as a forecasted (or predicted) variable at the current time, and  $\widetilde{X}$  as the observed value in the past. Also, for any time variable  $t > 0$  and  $k \in \mathbb{N}$ , we denote the time sequence  $\{\mathfrak{t}_1,\mathfrak{t}_2,\dots,\mathfrak{t}_k\}$  as  $\{\mathfrak{t}\}_{1:k}$ , and variable  $X$  at time  $\mathfrak{t}_k$  as  $X_{\mathfrak{t}_k}$ . We use the shorthand notation  $X_{(k)}$  (or  $X^{(k)}$ ) for  $X_{\mathfrak{t}_k}$  (or  $X^{\mathfrak{t}_k}$ ). We use the notation  $\{x\}_{a:b}$  to denote a sequence of variables  $\{x_a,x_{a + 1},\ldots ,x_b\}$ , and  $\{x\}_{(a:b)}$  to represent a sequence of variables  $\{x_{\mathrm{t}_a},x_{\mathrm{t}_{a + 1}},\ldots ,x_{\mathrm{t}_b}\}$ . Given two variables  $x$  and  $y$ , let  $x \vee y$  denote  $\max(x,y)$ , and  $x \wedge y$  denote  $\min(x,y)$ . Given two complex numbers  $z_1$  and  $z_2$ , we write  $z_2 = W(z_1)$  if  $z_2 e^{z_2} = z_1$ , where  $W$  is the Lambert function. Given a variable  $x$ , the notation  $a = \mathcal{O}(b(x))$  means that  $a \leq C \cdot b(x)$  for some constant  $C > 0$  that is independent of  $x$ , and the notation  $a = \Omega(b(x))$  means that  $a \geq C \cdot b(x)$  for some constant  $C > 0$  that is independent of  $x$ . We have described the specific details in Appendix C.1.
2 Problem statement: Desynchronizing timelines
2.1 Time-elapsing Markov Decision Process
In this paper, we study a non-stationary Markov Decision Process (MDP) for which the transition probability and the reward change over time. We begin by clarifying that the term episode is agent-centric, not environment-centric. Prior solutions for episode-varying (or step-varying) MDPs operate
under the assumption that the timing of MDP changes aligns with the agent commencing a new episode (or step). We introduce the new concept of time-ellipsis MDP. It starts from the wall-clock time  $t = 0$  to  $t = T$ , where  $T$  is fixed. The time-ellipsis MDP at time  $t \in [0, T]$  is defined as  $\mathcal{M}_t := \langle S, \mathcal{A}, H, P_t, R_t, \gamma \rangle$ , where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $H$  is the number of steps,  $P_t : S \times \mathcal{A} \times S \to \Delta(S)$  is the transition probability,  $R_t : S \times \mathcal{A} \to \mathbb{R}$  is the reward function, and  $\gamma$  is a discounting factor. Prior to executing the first episode, the agent determines the total number of interactions with the environment (denoted as the number of total episode  $K$ ) and subsequently computes the sequence of interaction times  $\{t\}_{1:K}$  through an optimization problem. We denote  $t_k$  as the wall-clock time of the environment when the agent starts the episode  $k$ . Similar to the existing non-stationary RL framework, the agent's objective is to learn a policy  $\pi^{t_k} : S \to \Delta(\mathcal{A})$  for all  $k$ . This is achieved through engaging in a total of  $K$  episode interactions, namely  $\{\mathcal{M}_{t_1}, \mathcal{M}_{t_1}, \dots, \mathcal{M}_{t_K}\}$ , where the agent dedicates the time interval  $(t_k, t_{k+1})$  for policy training and then obtains a sequence of suboptimal policies  $\{\pi^{t_1}, \pi^{t_2}, \dots, \pi^{t_K}\}$  to maximize a non-stationary policy evaluation metric, dynamic regret.
Dealing with time-ellipsis MDP instead of conventional MDP raises an important question that should be addressed: within the time duration  $[0, T]$ , what time sequence  $\{\mathfrak{t}\}_{1:K}$  yields favorable trajectory samples to obtain an optimal policy? This question is also related to the following: what is optimal value of  $K$ , i.e. the total number of episode that encompasses a satisfactory balance between the amount of observed trajectories and the accuracy of policy training? These interwined questions are concerned with an important aspect of RL, which is the computation of the optimal policy for a given  $\mathfrak{t}_k$ . In Section 4, we propose the ProST framework that computes a suboptimal  $K^*$  and its corresponding suboptimal time sequence  $\{\mathfrak{t}^*\}_{1:K^*}$  based on the information of the environment's rate of change. In Section 3, we compute a near-optimal policy for  $\{\mathfrak{t}^*\}_{1:K^*}$ . Before proceeding with the above results, we introduce a new metric quantifying the environment's pace of change, referred to as time-ellipsis variation budget.
2.2 Time-elapsing variation budget
Variation budget [10-12] is a metric to quantify the speed with which the environment changes. Driven by our motivations, we introduce a new metric imbued with real-time considerations, named time-ellipsis variation budget  $B(\Delta t)$ . Unlike the existing variation budget, which quantifies the environment's non-stationarity across episodes  $\{1, 2, \dots, K\}$ , our definition accesses it across  $\{t_1, t_2, \dots, t_K\}$ , where the interval  $\Delta t = t_{k+1} - t_k$  remains constant regardless of  $k \in [K-1]$ . For further analysis, we define two time-ellipsis variation budgets, one for transition probability and another for reward function.
Definition 1 (Time-elapsing variation budgets). For a given sequence  $\{\mathfrak{t}_1,\mathfrak{t}_2,\dots,\mathfrak{t}_K\}$ , assume that the interval  $\Delta t$  is equal to the policy training time  $\Delta_{\pi}$ . We define two time-elapsing variation budgets  $B_{p}(\Delta_{\pi})$  and  $B_{r}(\Delta_{\pi})$  as
$$
B _ {p} (\Delta_ {\pi}) := \sum_ {k = 1} ^ {K - 1} \sup  _ {s, a} | | P _ {\mathbf {t} _ {k + 1}} (\cdot | s, a) - P _ {\mathbf {t} _ {k}} (\cdot | s, a) | | _ {1}, B _ {r} (\Delta_ {\pi}) := \sum_ {k = 1} ^ {K - 1} \sup  _ {s, a} | R _ {\mathbf {t} _ {k + 1}} (s, a) - R _ {\mathbf {t} _ {k}} (s, a) |.
$$
To enhance the representation of a real-world system using the time-elapsing variation budgets, we make the following assumption.
Assumption 1 (Drifting constants). There exist constants  $c > 1$  and  $\alpha_r, \alpha_p \geq 0$  such that  $B_p(c\Delta_\pi) \leq c^{\alpha_p} B_p(\Delta_\pi)$  and  $B_r(c\Delta_\pi) \leq c^{\alpha_r} B_r(\Delta_\pi)$ . We call  $\alpha_p$  and  $\alpha_r$  the drifting constants.
2.3 Suboptimal training time
Aside from the formal MDP framework, the agent can be informed of varying time-ellipsis variation budgets based on the training time  $\Delta_{\pi} \in (0, T)$  even within the same time-ellipsis MDP. Intuitively, a short time  $\Delta_{\pi}$  is inadequate to obtain a near-optimal policy, yet it facilitates frequent interactions with the environment, leading to a reduction in empirical model error due to a larger volume of data. On the contrary, a long time  $\Delta_{\pi}$  may ensure obtaining a near-optimal policy but also introduces greater uncertainty in learning the environment. This motivates us to find a suboptimal training time  $\Delta_{\pi}^{*} \in (0, T)$  that strikes a balance between the sub-optimal gap of the policy and the empirical model error. If it exists, then  $\Delta_{\pi}^{*}$  provides a suboptimal  $K^{*} = \lfloor T / \Delta_{\pi}^{*} \rfloor$ , and a suboptimal time sequence where  $\mathfrak{t}_k^* = \mathfrak{t}_1 + \Delta_{\pi}^* \cdot (k - 1)$  for all  $k \in [K^{*}]$ . Our ProST framework computes the parameter  $\Delta_{\pi}^{*}$ ,
then sets  $\{\mathfrak{t}^*\}_{1:K^*}$ , and finally finds a future near-optimal policy for time  $\mathfrak{t}_{k+1}^*$  at time  $\mathfrak{t}_k^*$ . In the next section, we first study how to approximate the one-episode-ahead suboptimal policy  $\pi^{*,\mathfrak{t}_{k+1}}$  at time  $\mathfrak{t}_k$  when  $\{\mathfrak{t}\}_{1:K}$  is given.
3 Future policy optimizer
Figure 2: ProST framework
For given  $\mathfrak{t}_k$  and  $\mathfrak{t}_{k + 1}$ , the future policy optimizer  $(\mathsf{OPT}_{\pi})$ , as a module of the ProST framework (Figure 2), computes a near-optimal policy for the future time  $\mathfrak{t}_{k + 1}$  at time  $\mathfrak{t}_k$  via two procedures: (i) it first forecasts the future MDP model of time  $\mathfrak{t}_{k + 1}$  at time  $\mathfrak{t}_k$  utilizing the MDP forecaster function, (ii) it then utilizes an arbitrary policy optimization algorithm within the forecasted MDP model  $\mathsf{OPT}_{\pi}$  to obtain a future near-optimal policy  $\pi^{*}.t_{k + 1}$ .
3.1 MDP forecaster
Our ProST framework is applicable in an environment that meets the following assumption.
Assumption 2 (Observable non-stationary set  $\mathcal{O}$ ). Assume that the non-stationarity of  $\mathcal{M}_{\mathfrak{t}_k}$  is fully characterized by a non-stationary parameter  $o_{\mathfrak{t}_k} \in \mathcal{O}$ . Assume also that the agent observes a noisy non-stationary parameter  $\tilde{o}_{\mathfrak{t}_k}$  at the end of episode  $k \in [K]$  (at time  $\mathfrak{t}_k$ ).
It is worth noting that Assumption 2 is mild, as prior research in non-stationary RL has proposed techniques to estimate  $o_{(k)}$  through latent factor identification methods [4, 13-16], and our framework accommodates the incorporation of those works for the estimation of  $o_{(k)}$ . Based on Assumption 2, we define the MDP forecaster function  $g \circ f$  below.
Definition 2 (MDP forecaster  $g \circ f$ ). Consider two function classes  $\mathcal{F}$  and  $\mathcal{G}$  such that  $\mathcal{F}: \mathcal{O}^w \to \mathcal{O}$  and  $\mathcal{G}: \mathcal{S} \times \mathcal{A} \times \mathcal{O} \to \mathbb{R} \times \Delta(\mathcal{S})$ , where  $w \in \mathbb{N}$ . Then, for  $f_{(k)} \in \mathcal{F}$  and  $g_{(k)} \in G$ , we define MDP forecaster at time  $t_k$  as  $(g \circ f)_{(k)}: \mathcal{O}^w \times \mathcal{S} \times \mathcal{A} \to \mathbb{R} \times \Delta(\mathcal{S})$ .
The function  $f_{(k)}$ , acting as a non-stationarity forecaster, predicts a non-stationary parameter  $\hat{o}_{(k + 1)}$  at time  $t_{k + 1}$  based on the last  $w$  observations given by the set  $\{\tilde{o}\}_{(k - w + 1:k)}$ , i.e.,  $\hat{o}_{(k + 1)} = f(\{\tilde{o}\}_{(k - w + 1,k)})$ . The agent can determine the number of used historical observations, denoted as  $w$ , by leveraging information from the environment (Section 4). Then, the function  $g_{(k)}$ , acting as a model predictor, predicts a reward  $\widehat{R}_{(k + 1)}(s,a)$  and a transition probability  $\widehat{P}_{(k + 1)}(\cdot |s,a)$  for time  $t_{k + 1}$ , i.e.,  $(\widehat{R}_{(k + 1)},\widehat{P}_{(k + 1)}) = g_{(k)}(s,a,\hat{o}_{k + 1})$ . Finally, the  $\mathsf{OPT}_{\pi}$  generates the estimated future MDP  $\widehat{\mathcal{M}}_{(k + 1)} = \langle S,\mathcal{A},H,\widehat{P}_{(k + 1)},\widehat{R}_{(k + 1)},\gamma \rangle$  associated with time  $t_{k + 1}$ .
3.2 Finding future optimal policy
Now, consider an arbitrary RL algorithm provided by the user to obtain an optimal policy from the model  $\widehat{\mathcal{M}}_{(k + 1)}$ . For a given time sequence  $\{\mathfrak{t}\}_{1:K}$ , the  $\mathrm{OPT}_{\pi}$  finds a near-optimal future policy as follows: (1) observe and forecast, (2) optimize using the future MDP model.
(1) Observe and forecast. At time  $t_k$ , the agent executes an episode  $k$  in the environment  $\mathcal{M}_{(k)}$ , completes its trajectory  $\tau_{(k)}$ , and observes the noisy non-stationary parameter  $\hat{o}_{(k)}$  (Assumption 2). The algorithm then updates the function  $f_{(k)}$  based on the last  $w$  observed parameters, and the
function  $g_{(k)}$  with input from all previous trajectories. Following these updates, the MDP forecaster at time  $t_k$  predicts  $\widehat{P}_{(k+1)}$  and  $\widehat{R}_{(k+1)}$ , thus creating the MDP model  $\widehat{\mathcal{M}}_{(k+1)}$  for time  $t_{k+1}$ .
(2) Optimize using the future MDP model. Up until time  $\mathfrak{t}_{k+1}$ , the agent continually updates the policy within the estimated future MDP  $\widehat{\mathcal{M}}_{(k+1)}$  for a given duration  $\Delta_{\pi}$ . Specifically, the agent rollouts synthetic trajectories  $\hat{\tau}_{(k+1)}$  in  $\widehat{\mathcal{M}}_{(k+1)}$ , and utilizes any policy update algorithm to obtain a policy  $\widehat{\pi}_{(k+1)}$ . Following the duration  $\Delta_{\pi}$ , the agent stops the training by the time  $\mathfrak{t}_{k+1}$  and moves to the next episode  $\mathcal{M}_{(k+1)}$  with policy  $\widehat{\pi}_{(k+1)}$ .
We elaborate on the above procedure in Algorithm 1 given in Appendix F.1.
4 Time optimizer