SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now present our main theoretical contribution, which is regarding the time optimizer  $(\mathsf{OPT}_{\mathsf{t}})$ : computing a suboptimal policy training time  $\Delta_{\pi}^{*}$  (the agent tempo). Our theoretical analysis starts with specifying the components of the  $\mathsf{OPT}_{\pi}$  optimizer, which we refer to as ProST-T (note that -T stands for an instance in the tabular setting). We employ the Natural Policy Gradient (NPG) with entropy regularization [17] as a policy update algorithm in  $\mathsf{OPT}_{\pi}$ . We denote the entropy regularization coefficient as  $\tau$ , the learning rate as  $\eta$ , the policy evaluation approximation gap arising due to finite samples as  $\delta$ , and the past reference length for forecaster  $f$  as  $w$ . Without loss of generality, we assume that each policy iteration takes one second. The theoretical analysis is conducted within a tabular environment, allowing us to relax Assumption 2, which means that one can estimate non-stationary parameters by counting visitation of state and action pairs at time  $t_k$ , denoted as  $n_{(k)}(s,a)$ , rather than observing them. Additionally, we incorporate the exploration bonus term at time  $t_k$  into  $\widehat{R}_{(k+1)}$ , denoted as  $\Gamma_{w}^{(k)}(s,a)$ , which is proportional to  $\sum_{\tau=k-w+1}^{k}(n_{(\tau)}(s,a))^{-1/2}$  and aims to promote the exploration of states and actions that are visited infrequently.
We compute  $\Delta_{\pi}^{*}$  by minimizing an upper bound on the ProST-T's dynamic regret. The dynamic regret of ProST-T is characterized by the model prediction error that measures the MDP forecaster's error by defining the difference between  $\widehat{\mathcal{M}}_{(k + 1)}$  and  $\mathcal{M}_{(k + 1)}$  through a Bellman equation.
Definition 3 (Model prediction error). At time  $\mathfrak{t}_k$ , the MDP forecaster predicts a model  $\widehat{\mathcal{M}}_{(k+1)}$  and then we obtain a near-optimal policy  $\widehat{\pi}^{(k+1)}$  based on  $\widehat{\mathcal{M}}_{(k+1)}$ . For each pair  $(s, a)$ , we denote the state value function and the state action value function of  $\widehat{\pi}^{(k+1)}$  in  $\widehat{\mathcal{M}}_{(k+1)}$  at step  $h \in [H]$  as  $\widehat{V}_h^{(k+1)}(s)$  and  $\widehat{Q}_h^{(k+1)}(s, a)$ , respectively. We also denote the model prediction error associated with time  $\mathfrak{t}_{k+1}$  calculated at time  $\mathfrak{t}_k$  as  $\iota_h^{(k+1)}(s, a)$ , which is defined as
$$
\iota_ {h} ^ {(k + 1)} (s, a) \colon = \left(R _ {(k + 1)} + \gamma P _ {(k + 1)} \widehat {V} _ {h + 1} ^ {(k + 1)} - \widehat {Q} _ {h} ^ {(k + 1)}\right) (s, a).
$$
We now derive an upper bound on the ProST-T dynamic regret. We expect the upper bound to be likely controlled by two factors: the error of the MDP forecaster's prediction of the future MDP model and the error of the NPG algorithm due to approximating the optimal policy within an estimated future MDP model. This insight is clearly articulated in the next theorem.
Theorem 1 (ProST-T dynamic regret  $\Re$ ). Let  $\iota_H^K = \sum_{k=1}^{K-1} \sum_{h=0}^{H-1} \iota_h^{(k+1)}(s_h^{(k+1)}, a_h^{(k+1)})$  and  $\bar{\iota}_{\infty}^K := \sum_{k=1}^{K-1} \|\bar{\iota}_{\infty}^{k+1}\|_{\infty}$ , where  $\iota_H^K$  is a data-dependent error. For a given  $p \in (0,1)$ , the dynamic regret of the forecasted policies  $\{\widehat{\pi}^{(k+1)}\}_{1:K-1}$  of ProST-T is upper bounded with probability at least  $1 - p/2$  as follows:
$$
\Re \left(\left\{\widehat {\pi} ^ {(k + 1)} \right\} _ {1: K - 1}, K\right) \leq \Re_ {I} + \Re_ {I I}
$$
where  $\mathfrak{R}_I = \bar{\iota}_{\infty}^K / (1 - \gamma) - \iota_H^K + C_p \cdot \sqrt{K - 1}$ ,  $\mathfrak{R}_{II} = C_{II}[\Delta_{\pi}] \cdot (K - 1)$ , and  $C_p, C_{II}[\Delta_{\pi}]$  are some functions of  $p$ ,  $\Delta_{\pi}$ , respectively.
Specifically, the upper bound is composed of two terms:  $\Re_I$  that originates from the MDP forecaster error between  $\mathcal{M}_{(k + 1)}$  and  $\widehat{\mathcal{M}}_{(k + 1)}$ , and  $\Re_{II}$  that arises due to the suboptimality gap between  $\pi^{*,(k + 1)}$  and  $\widehat{\pi}^{(k + 1)}$ . Theorem 1 clearly demonstrates that a prudent construction of the MDP
forecaster that controls the model prediction errors and the selection of the agent tempo  $\Delta_{\pi}$  is significant in guaranteeing sublinear rates for  $\Re_I$  and  $\Re_{II}$ . To understand the role of the environment tempo in  $\Re_I$ , we observe that the MDP forecaster utilizes  $w$  previous observations, which inherently encapsulates the environment tempo. We expect the model prediction errors, at least in part, to be controlled by the environment tempo  $B(\Delta_{\pi})$ , so that a trade-off between two tempos can be framed as the trade-off between  $\Re_I$  and  $\Re_{II}$ . Hence, it is desirable to somehow minimize the upper bound with respect to  $\Delta_{\pi}$  to obtain a solution, denoted as  $\Delta_{\pi}^{*}$ , which strikes a balance between  $\Re_I$  and  $\Re_{II}$ .
4.1.1 Analysis of  $\Re_{II}$
A direct analysis of the upper bound  $\Re_I + \Re_{II}$  is difficult since its dependence on  $K$  is not explicit. To address this issue, we recall that an optimal  $\Delta_{\pi}^{*}$  should be a natural number that guarantees the sublinearity of both  $\Re_I$  and  $\Re_{II}$  with respect to the total number of episodes  $K$ . We first compute a set  $\mathbb{N}_{II} \subset \mathbb{N}$  that includes those values of  $\Delta_{\pi}$  that guarantee the sublinearity of  $\Re_{II}$ , and then compute a set  $\mathbb{N}_I \subset \mathbb{N}$  that guarantees the sublinearity of  $\Re_I$ . Finally, we solve for  $\Delta_{\pi}^{*}$  in the common set  $\mathbb{N}_I \cap \mathbb{N}_{II}$ .
Proposition 1 ( $\Delta_{\pi}$  bounds for sublinear  $\Re_{II}$ ). A total step  $H$  is given by MDP. For a number  $\epsilon > 0$  such that  $H = \Omega \left(\log \left((\widehat{r}_{max} \vee r_{max}) / \epsilon\right)\right)$ , we choose  $\delta, \tau, \eta$  to satisfy  $\delta = \mathcal{O}(\epsilon)$ ,  $\tau = \Omega \left(\epsilon / \log |\mathcal{A}|\right)$  and  $\eta \leq (1 - \gamma) / \tau$ , where  $\widehat{r}_{max}$  and  $r_{max}$  are the maximum reward of the forecasted model and the maximum reward of the environment, respectively. Define  $\mathbb{N}_{II} := \{n \mid n > \frac{1}{\eta\tau} \log \left(\frac{C_1(\gamma + 2)}{\epsilon}\right), n \in \mathbb{N}\}$ , where  $C_1$  is a constant. Then  $\Re_{II} \leq 4\epsilon (K - 1)$  for all  $\Delta_{\pi} \in \mathbb{N}_{II}$ .
As a by-product of Proposition 1, the sublinearity of  $\Re_{II}$  can be realized by choosing  $\epsilon = \mathcal{O}\big((K - 1)^{\alpha -1}\big)$  for any  $\alpha \in [0,1)$ , which suggests that a tighter upper bound on  $\Re_{II}$  requires a smaller  $\epsilon$  and subsequently a larger  $\Delta_{\pi}\in \mathbb{N}_{II}$ . The hyperparameter conditions in Proposition 1 can be found in Lemma 1 and 2 in Appendix D.3.
4.1.2 Analysis of  $\Re_I$
We now relate  $\Re_I$  to the environment tempo  $B(\Delta_{\pi})$  using the well-established non-stationary adaptation technique of Sliding Window regularized Least-Squares Estimator (SW-LSE) as the MDP forecaster [18-20]. The tractability of the SW-LSE algorithm allows to upper-bound the model predictions errors  $\iota_H^K$  and  $\overline{\iota}_{\infty}^{K}$  by the environment tempo extracted from the past  $w$  observed trajectories, leading to a sublinear  $\Re_I$  as demonstrated in the following theorem.
Theorem 2 (Dynamic regret  $\Re_I$  with  $f = \mathsf{SW}$ -LSE). For a given  $p \in (0,1)$ , if the exploration bonus constant  $\beta$  and regularization parameter  $\lambda$  satisfy  $\beta = \Omega(|S|H\sqrt{\log(H / p)})$  and  $\lambda \geq 1$ , then  $\Re_I$  is bounded with probability at least  $1 - p$  as follows:
$$
\Re_ {I} \leq C _ {I} \left[ B \left(\Delta_ {\pi}\right) \right] \cdot w + C _ {k} \cdot \sqrt {\frac {1}{w} \log \left(1 + \frac {H}{\lambda} w\right)} + C _ {p} \cdot \sqrt {K - 1}
$$
where  $C_I[B(\Delta_\pi)] = (1 / (1 - \gamma) + H) \cdot B_r(\Delta_\pi) + (1 + H\hat{r}_{max})\gamma /(1 - \gamma) \cdot B_p(\Delta_\pi)$ , and  $C_k$  is a constant on the order of  $\mathcal{O}(K)$ .
For a brief sketch of how SW-LSE makes the environment tempo appear in the upper bound, we outline that the model prediction errors are upper-bounded by two forecaster errors, namely  $P_{(k + 1)} - \widehat{P}_{(k + 1)}$  and  $R_{(k + 1)} - \widehat{R}_{(k + 1)}$ , along with the visitation count  $n_{(k)}(s,a)$ . Then, the SW-LSE algorithm provides a solution  $\left(\widehat{P}_{(k + 1)},\widehat{R}_{(k + 1)}\right)$  as a closed form of linear combinations of past  $w$  estimated values  $\{\widetilde{P},\widetilde{R}\}_{(k - w + 1:w)}$ . Finally, employing the Cauchy inequality and triangle inequality, we derive two forecasting errors that are upper-bounded by the environment tempo. For the final step before obtaining a suboptimal  $\Delta_{\pi}^{*}$ , we compute  $\mathbb{N}_I$  that guarantees the sublinearity of  $\Re_I$ .
Proposition 2 ( $\Delta_{\pi}$  bounds for sublinear  $\Re_I$ ). Denote  $B(1)$  as the environment tempo when  $\Delta_{\pi} = 1$ , which is a summation over all time steps. Assume that the environment satisfies  $B_r(1) + B_p(1)\hat{r}_{max} / (1 - \gamma) = o(K)$  and we choose  $w = \mathcal{O}((K - 1)^{2/3} / (C_I[B(\Delta_{\pi})])^{2/3})$ . Define the set  $\mathbb{N}_I$  to be  $\{n \mid n < K, n \in \mathbb{N}\}$ . Then  $\Re_I$  is upper-bounded as  $\Re_I = \mathcal{O}\left(C_I[B(\Delta_{\pi})]^{1/3}(K - 1)^{2/3}\sqrt{\log((K - 1) / C_I[B(\Delta_{\pi})])}\right)$  and also satisfies a sublinear upper bound, provided that  $\Delta_{\pi} \in \mathbb{N}_I$ .
The upper bound on the environment tempo  $B(1)$  in proposition 2 is aligned with our expectation that dedicating an excessively long time to a single iteration may not allow for an effective policy approximation, thereby hindering the achievement of a sublinear dynamic regret. Furthermore, our insight that a larger environment tempo prompts the MDP forecaster to consider a shorter past reference length, aiming to mitigate forecasting uncertainty, is consistent with the condition involving  $w$  stated in Proposition 2.
4.1.3 Suboptimal tempo  $\Delta_{\pi}^{*}$
So far, we have shown that an upper bound on the ProST dynamic regret is composed of two terms  $\Re_I$  and  $\Re_{II}$ , which are characterized by the environment tempo and the agent tempo, respectively. Now, we claim that a suboptimal tempo that minimizes ProST's dynamic regret could be obtained by the optimal solution  $\Delta_\pi^* = \arg \min_{\Delta_\pi \in \mathbb{N}_I \cap \mathbb{N}_{II}} (\Re_I^{\max} + \Re_{II}^{\max})$ , where  $\Re_I^{\max}$  and  $\Re_{II}^{\max}$  denote the upper bounds on  $\Re_I$  and  $\Re_{II}$ . We show that  $\Delta_\pi^*$  strikes a balance between the environment tempo and the agent tempo since  $\Re_I^{\max}$  is a non-decreasing function of  $\Delta_\pi$  and  $\Re_{II}^{\max}$  is a non-increasing function of  $\Delta_\pi$ . Theorem 3 shows that the optimal tempo  $\Delta_\pi^*$  depends on the environment's drifting constants introduced in Assumption 1.
Theorem 3 (Suboptimal tempo  $\Delta_{\pi}^{*}$ ). Let  $k_{Env} = (\alpha_r \vee \alpha_p)^2 C_I[B(1)]$ ,  $k_{Agent} = \log (1 / (1 - \eta \tau))C_1(K - 1)(\gamma + 2)$ . Consider three cases: case1:  $\alpha_r \vee \alpha_p = 0$ , case2:  $\alpha_r \vee \alpha_p = 1$ , case3:  $0 < \alpha_r \vee \alpha_p < 1$  or  $\alpha_r \vee \alpha_p > 1$ . Then  $\Delta_{\pi}^{*}$  depends on the environment's drifting constants as follows:
Case1:  $\Delta_{\pi}^{*} = T$
Case2:  $\Delta_{\pi}^{*} = \log_{1 - \eta \gamma}\left(k_{Env} / k_{Agent}\right) + 1.$
- Case3:  $\Delta_{\pi}^{*} = \exp \left(-W\left[-\frac{\log(1 - \eta\tau)}{\max(\alpha_r,\alpha_p) - 1}\right]\right)$ , provided that the parameters are chosen so that  $k_{Agent} = (1 - \eta \tau)k_{Env}$ .
4.2 Improving MDP forecaster
Determining a suboptimal tempo by minimizing an upper bound on  $\Re_I + \Re_{II}$  can be improved by using a tighter upper bound. In Proposition 1, we focused on the  $Q$  approximation gap  $\delta$  to provide a justifiable upper bound on  $\Re_I + \Re_{II}$ . It is important to note that the factor  $\delta$  arises not only from the finite sample trajectories as discussed in [21], but also from the forecasting error between  $\mathcal{M}_{(k+1)}$  and  $\widehat{\mathcal{M}}_{(k+1)}$ . It is clear that the MDP forecaster establishes a lower bound on  $\delta$  denoted as  $\delta_{\min}$ , which in turn sets a lower bound on  $\epsilon$  and consequently on  $\Re_I$ . This inspection highlights that the MDP forecaster serves as a common factor that controls both  $\Re_I$  and  $\Re_{II}$ , and a further investigation to improve the accuracy of the forecaster is necessary for a better bounding on  $\Re_I + \Re_{II}$ .
Our approach to devising a precise MDP forecaster is that, instead of selecting the past reference length  $w$  as indicated in Proposition 2, we set  $w = k$ , implying the utilization of all past observations. However, we address this by solving an additional optimization problem, resulting in a tighter bound on  $\Re_I$ . We propose a method that adaptively assigns different weights  $q \in \mathbb{R}_+^k$  to the previously observed non-stationary parameters up to time  $t_k$ , which reduces the burden of choosing  $w$ . Hence, we further analyze  $\Re_I$  through the utilization of the Weighted regularized Least-Squares Estimator (W-LSE) [22]. Unlike SW-LSE, W-LSE does not necessitate a predefined selection of  $w$ , but it instead engages in a joint optimization procedure involving the data weights  $q$  and the future model  $\left(\widehat{P}_{(k + 1)},\widehat{R}_{(k + 1)}\right)$ . To this end, we define the forecasting reward model error as  $\Delta_k^r (s,a) = \left|\left(R_{(k + 1)} - \widehat{R}_{(k + 1)}\right)(s,a)\right|$  and the forecasting transition probability model error as  $\Delta_k^p (s,a) = \left\| \left(P_{(k + 1)} - \widehat{P}_{(k + 1)}\right)(\cdot |s,a)\right\| _1$ .
Theorem 4 ( $\Re_I$  upper bound with  $f = W$ -LSE). By setting the exploration bonus  $\Gamma_{(k)}(s,a) = \frac{1}{2}\Delta_k^r (s,a) + \frac{\gamma\vec{r}_{max}}{2(1 - \gamma)}\Delta_k^p (s,a)$ , it holds that
$$
\mathfrak {R} _ {I} \leq \left(4 H + \frac {2 \gamma | \mathcal {S} |}{1 - \gamma} \left(\frac {1}{1 - \gamma} + H\right)\right) \left(\frac {1}{2} \sum_ {k = 1} ^ {K - 1} \Delta_ {k} ^ {r} (s, a) + \frac {\gamma \tilde {r} _ {\max}}{2 (1 - \gamma)} \sum_ {k = 1} ^ {K - 1} \Delta_ {k} ^ {p} (s, a)\right).
$$
Remark 1 (Tighter  $\Re_{\mathcal{I}}$  upper bound with  $f = W$ -LSE). If the optimization problem of  $W$ -LSE is feasible, then the optimal data weight  $q^{*}$  provides tighter bounds for  $\Delta_k^r$  and  $\Delta_k^p$  in comparison to SW-LSE, consequently leading to a tighter upper bound for  $\Re_{\mathcal{I}}$ . We prove in Lemmas 4 and 6 in Appendix D.3 that  $\iota_{\infty}^{K}$  and  $-\iota_H^K$  are upper-bounded in terms of  $\Delta_k^r$  and  $\Delta_k^p$ .
4.3 ProST-G
The theoretical analysis outlined above serves as a motivation to empirically investigate two key points: firstly, the existence of an optimal training time; secondly, the role of the MDP forecaster's contribution to the ProST framework's overall performance. To address these questions, we propose a practical instance, named ProST-G, which particularly extends the investigation in Section 4.2. ProST-G optimizes a policy with the soft actor-critic (SAC) algorithm [23], utilizes the integrated autoregressive integrated moving average (ARIMA) model for the proactive forecaster  $f$ , and uses a bootstrap ensemble of dynamic models where each model is a probabilistic neural network for the model predictor  $g$ . We further discuss specific details of ProST-G in Appendix F.3 and in Algorithm 3.
We evaluate ProST-G with four baselines in three Mujoco environments each with five different non-stationary speeds and two non-stationary datasets.
(1) Environments: Non-stationary desired posture. We make the rewards in the three environments non-stationary by altering the agent's desired directions. The forward reward  $R_{t}^{f}$  changes as  $R_{t}^{f} = o_{t} \cdot \bar{R}_{t}^{f}$ , where  $\bar{R}_f$  is the original reward from the Mujoco environment. The non-stationary parameter  $o_k$  is generated from the sine function with five different speeds and from the real data  $A$  and  $B$ . We then measure the time- elapsing variation budget by  $\sum_{k = 1}^{K - 1}|o_{k + 1} - o_k|$ . Further details of the environment settings can be found in Appendix D.1.1.
(2) Benchmark methods. Four baselines are chosen to empirically support our second question: the significance of the forecaster. MBPO is the state-of-the-art model-based policy optimization [24]. Pro-OLS is a policy gradient algorithm that predicts the future performance and optimizes the predicted performance of the future episode [7]. ONPG is an adaptive algorithm that performs a purely online optimization by fine-tuning the existing policy using only the trajectory observed online [8]. FTRL is an adaptive algorithm that performs follow-the-regularized-leader optimization by maximizing the performance on all previous trajectories [9].
6.1 Performance compare
The outcomes of the experimental results are presented in Table 1. The table summarizes the average return over the last 10 episodes during the training procedure. We have illustrated the complete training results in Appendix E.3. In most cases, ProST-G outperforms MBPO in terms of rewards, highlighting the adaptability of the ProST framework to dynamic environments. Furthermore, except for data  $A$  and  $B$ , ProST-G consistently outperforms the other three baselines. This supports our motivation of using the proactive model-based method for a higher adaptability in non-stationary environments compared to state-of-the-art model-free algorithms (Pro-OLS, ONPG, FTRL). We elaborate on the training details in Appendix E.2.
Table 1: Average reward returns
Speed	B(G)	Swimmer-v2	Halfcheetah-v2	Hopper-v2
Pro-OLS	ONPG	FTML	MBPO	ProST-G	Pro-OLS	ONPG	FTML	MBPO	ProST-G	Pro-OLS	ONPG	FTML	MBPO	ProST-G
1	16.14	-0.40	-0.26	-0.08	-0.08	0.57	-83.79	-85.33	-85.17	-24.89	-19.69	98.38	95.39	97.18	92.88	92.77
2	32.15	0.20	-0.12	0.14	-0.01	1.04	-83.79	-85.63	-86.46	-22.19	-20.21	98.78	97.34	99.02	96.55	98.13
3	47.86	-0.13	0.05	-0.15	-0.64	1.52	-83.27	-85.97	-86.26	-21.65	-21.04	97.70	98.18	98.60	95.08	100.42
4	63.14	-0.22	-0.09	-0.11	-0.04	2.01	-82.92	-84.37	-85.11	-21.40	-19.55	98.89	97.43	97.94	97.86	100.68
5	77.88	-0.23	-0.42	-0.27	0.10	2.81	-84.73	-85.42	-87.02	-20.50	-20.52	97.63	99.64	99.40	96.86	102.48
A	8.34	1.46	2.10	2.37	-0.08	0.57	-76.67	-85.38	-83.83	-40.67	83.74	104.72	118.97	115.21	100.29	111.36
B	4.68	1.79	-0.72	-1.20	0.19	0.20	-80.46	-86.96	-85.59	-29.28	76.56	80.83	131.23	110.09	100.29	127.74
6.2 Ablation study
An ablation study was conducted on the two aforementioned questions. The following results support our inspection of Section 4.2 and provide strong grounds for Theorem 3.
Figure 3: (a) Optimal  $\Delta_{\pi}^{*}$ ; (b) Different forecaster  $f$  (ARIMA, SA); (c) The Mean squared Error (MSE) model loss of ProST-G with four different forecasters (ARIMA and three SA) and the MBPO. The  $x$ -axis in each figure shows the episodes.
Suboptimal  $\Delta_{\pi}^{*}$ . The experiments are performed over five different policy training times  $\Delta_{\pi} \in \{1,2,3,4,5\}$ , aligned with SAC's number of gradient steps  $G \in \{38,76,114,152,190\}$ , under a fixed environment speed. Different from our theoretical analysis, we set  $\Delta_{t} = 1$  with  $G = 38$ . We generate  $o_k = \sin(2\pi \Delta_{\pi} k / 37)$ , which satisfies Assumption 1 (see Appendix E.1). The shaded areas of Figures 3 (a), (b) and (c) are  $95\%$  confidence area among three different noise bounds of 0.01, 0.02 and 0.03 in  $o_k$ . Figure 3(a) shows  $\Delta_{t} = 4$  is close to the optimal  $G^{*}$  among five different choices.
Functions  $f, g$ . We investigate the effect of the forecaster  $f$ 's accuracy on the framework using two distinct functions: ARIMA and a simple average (SA) model, each tested with three different the values of  $w$ . Figure 3(b) shows the average rewards of the SA model with  $w \in \{3,5,7\}$  and ARIMA model (four solid lines). The shaded area is  $95\%$  the confidence area among 4 different speeds  $\{1,2,3,4\}$ . Figure 3(c) shows the corresponding model error. Also, we investigate the effect of the different model predictor  $g$  by comparing MBPO (reactive-model) and ProST-G with  $f = \text{ARIMA}$  (proactive-model) in Figure 3(c). The high returns from ProST-G with  $f = \text{ARIMA}$ , compared to those from MBPO, empirically support that the forecasting component of the ProST framework can provide a satisfactory adaptability to the baseline algorithm that is equipped with. Also, Figures 3(b) and 3(c) provide empirical evidence that the accuracy of  $f$  is contingent on the sliding window size, thereby impacting the model accuracy and subsequently influencing the agent's performance.
This work offers the first study on the important issue of time synchronization for non-stationary RL. To this end, we introduce the concept of the tempo of adaptation in a non-stationary RL, and obtain a suboptimal training time. We propose a Proactively Synchronizing Tempo (ProST) framework, together with two specific instances ProST-T and ProST-G. The proposed method adjusts an agent's tempo to match the tempo of the environment to handle non-stationarity through both theoretical analysis and empirical evidence. The ProST framework provides a new avenue to implement reinforcement learning in the real world by incorporating the concept of adaptation tempo.
As a future work, it is important to generalize the proposed framework to learn a safe guarantee policy in a non-stationary RL by considering the adaptation tempo of constraint violations [25, 26]. Another generalization is to introduce an alternative dynamic regret metric, enabling a fair performance comparison among agents, even when they have varying numbers of total episodes. Another future work is to find an optimal tempo of the distribution correction in offline non-stationary RL, specifically how to adjust the relabeling function to offline data in a time-varying environment that is dependent on the tempo of the environment [27, 28].
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint
[35] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. International Conference on Learning Representations, 2019.
[36] Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. Advances in Neural Information Processing Systems, 33:6743-6754, 2020.
[37] Yash Chandak, Scott Jordan, Georgios Theocharous, Martha White, and Philip S Thomas. Towards safe policy improvement for non-stationary mdps. Advances in Neural Information Processing Systems, 33:9156-9168, 2020.
[38] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.
[39] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020.