SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although the bonus term  $\hat{b}_h^k$  cannot serve as a point-wise uncertainty measure, the truncated value function  $\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$  as the cumulative version of  $\hat{b}_h^k$  can serve as a trajectory-wise uncertainty measure, which can be used to determine future exploration policies. Intuitively, for any policy  $\pi$ , the model estimation error satisfies  $\mathbb{E}_{(s,a)\sim (P^{*,k},\pi)}[\| \hat{P}^k (\cdot |s,a) - P^{\star ,k}(\cdot |s,a)\|_{TV}]\leq \hat{V}_{\hat{P}^k,\hat{b}^k}^\pi +\Delta$ , where the error term  $\Delta$  captures the variation of both transition kernels and representations over time. As a result, by selecting the policy that maximizes  $\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$  as the exploration policy as in line 7, the agent will explore the trajectories whose states and actions have not been estimated sufficiently well so far.
Algorithm 2  $\mathbf{E}^2\mathbf{U}$  (Model Estimation and Exploration Policy Update)
1: Input: round index  $k$ , regularizer  $\lambda_{k,W}$  and coefficient  $\tilde{\alpha}_{k,W}$ , datasets  $\{\tilde{\mathcal{D}}_{h-1}^{(k,h)}\}, \{\tilde{\mathcal{D}}_h^{(k,h)}\}$  and models  $\{\Psi, \Phi\}$ .
2: for step  $h = 1, \dots, H$  do
3: Learn the representation via MLE for step  $h$ :
$$
\hat {P} _ {h} ^ {k} = (\hat {\phi} _ {h} ^ {k}, \hat {\mu} _ {h} ^ {k}) = \arg \max _ {\phi \in \Phi , \mu \in \Psi} \mathbb {E} _ {\bar {\mathcal {D}} _ {h} ^ {(k, h)}} [ \log \langle \phi (s _ {h}, a _ {h}), \mu (s _ {h + 1}) \rangle ].
$$
4: Compute the empirical covariance matrix as
$$
\hat {U} _ {h} ^ {k, W} = \sum_ {\tilde {\mathcal {D}} _ {h} ^ {(k, h + 1)}} \hat {\phi} _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \hat {\phi} _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) ^ {\top} + \lambda_ {k, W} I
$$
5: Define exploration-driven bonus  $\hat{b}_h^k (\cdot ,\cdot) = \min \{\alpha_{k,W}\| \hat{\phi}_h^k (\cdot ,\cdot)\|_{(\hat{U}_h^{k,W})^{-1}},1\} .$
6: end for
7: Find exploration policy  $\tilde{\pi}^k = \arg \max_{\pi} \hat{V}_{\hat{P}^k, \hat{b}^k}^\pi$ , where  $\hat{V}_{\hat{P}^k, \hat{b}^k}^\pi$  is defined as in Equation (1).
8: Output: Model  $\hat{P}^k$  and exploration policy  $\{\tilde{\pi}^k\}$ .
Step 3: Target Policy Update with Periodic Restart: The agent evaluates the target policy by computing the value function under the target policy and the estimated transition kernel. Then, due to the nonstationarity, the target policy is reset every  $\tau$  rounds. Compared with the previous work using periodic restart (Fei et al., 2020), whose choice of  $\tau$  is based on a certain smooth visitation assumption, we remove such an assumption and hence our choice of  $\tau$  is applicable to more general model classes. Finally, the agent uses the estimated value function for target policy update for the next round  $k + 1$  via online mirror descend. The update step is inspired by the previous works (Cai et al., 2020; Schulman et al., 2017). Specifically, for any given policy  $\pi^0$  and MDP  $\mathcal{M}$ , define the following function w.r.t. policy  $\pi$ :
$$
L ^ {\mathcal {M}, \pi_ {0}} (\pi) = V _ {P, r} ^ {\pi^ {0}} + \sum_ {h = 1} ^ {H} \mathbb {E} _ {s _ {h} \sim (P, \pi^ {0})} \left[ \left\langle Q _ {h, P, r} ^ {\pi^ {0}}, \pi_ {h} (\cdot | s _ {h}) - \pi_ {h} ^ {0} (\cdot | s _ {h}) \right\rangle \right].
$$
$L^{\mathcal{M},\pi_0}(\pi)$  can be regarded as a local linear approximation of  $V_{P,r}^{\pi}$  at "point"  $\pi^0$  (Schulman et al., 2017). Consider the following optimization problem:
$$
\pi^ {k + 1} = \arg \max _ {\pi} L ^ {\mathcal {M} ^ {k}, \pi^ {k}} (\pi) - \frac {1}{\eta} \sum_ {h \in [ H ]} \mathbb {E} _ {s _ {h} \sim (P ^ {\star , k}, \pi^ {k})} \left[ D _ {K L} (\pi_ {h} (\cdot | s _ {h}) | | \pi_ {h} ^ {k} (\cdot | s _ {h})) \right].
$$
This can be regarded as a mirror descent step with KL divergence, where the KL divergence regularizes  $\pi$  to be close to  $\pi^k$ . It further admits a closed-form solution:  $\pi_h^{k+1}(\cdot|\cdot) \propto \pi_h^k(\cdot|\cdot) \cdot \exp\{\eta \cdot Q_{h,P^{\star,k},r^k}^{\pi^k}(\cdot,\cdot)\}$ . We use the estimated version  $\hat{Q}_h^k$  to approximate  $Q_{h,P^{\star,k},r^k}^{\pi^k}$  and get
$$
\pi_ {h} ^ {k + 1} (\cdot | \cdot) \propto \pi_ {h} ^ {k} (\cdot | \cdot) \cdot \exp \left\{\eta \cdot \hat {Q} _ {h} ^ {k} (\cdot , \cdot) \right\}. \tag {2}
$$
4.2 Technical Assumptions
Our analysis adopts the following standard assumptions on low-rank MDPs.
Assumption 4.1. (Realizability). A learning agent can access to a model class  $\{(\Phi, \Psi)\}$  that contains the true model, namely, for any  $h \in [H]$ ,  $k \in [K]$ ,  $\phi_h^{\star,k} \in \Phi$ ,  $\mu_h^{\star,k} \in \Psi$ .
While we assume cardinality of the model class to be finite for simplicity, extensions to infinite classes with bounded statistical complexity are not difficult (Sun et al., 2019).
Assumption 4.2 (Bounded Density). Any model induced by  $\Phi$  and  $\Psi$  has bounded density, i.e.  $\forall P = \langle \phi, \mu \rangle$ ,  $\phi \in \Phi$ ,  $\mu \in \Psi$ , there exists a constant  $B \geq 0$  such that  $\max_{(s,a,s') \in S \times A \times S} P(s'|s,a) \leq B$ .
Assumption 4.3 (Reachability). For each round  $k$  and step  $h$ , the true transition kernel  $P_h^{\star, k}$  satisfies that for any  $(s, a, s') \in S \times \mathcal{A} \times \mathcal{S}$ ,  $P_h^{\star, k}(s'|s, a) \geq p_{\min}$ .
Variation Budgets: We next introduce several measures of nonstationarity of the environment:  $\Delta^P = \sum_{k=1}^{K} \sum_{h=1}^{H} \max_{(s,a) \in S \times A} \| P_h^{\star,k+1}(\cdot|s,a) - P_h^{\star,k}(\cdot|s,a) \|_{TV}, \Delta^{\sqrt{P}} = \sum_{k=1}^{K} \sum_{h=1}^{H} \max_{(s,a) \in S \times A} \| P_h^{\star,k+1}(\cdot|s,a) - P_h^{\star,k}(\cdot|s,a) \|_{TV}^{1/2}, \Delta^\phi = \sum_{k=1}^{K} \sum_{h=1}^{H} \max_{(s,a) \in S \times A} \| \phi_h^{\star,k+1}(s,a) - \phi_h^{\star,k}(s,a) \|_2, \Delta^\tau = \sum_{k=1}^{K} \sum_{h=1}^{H} \max_{s \in S} \| \pi_h^{\star,k}(\cdot|s) - \pi_h^{\star,k-1}(\cdot|s) \|_{TV}$ . These notions are known as variation budgets or path lengths in the literature of online convex optimization (Besbes et al., 2015; Hazan, 2016; Hall & Willett, 2015) and nonstationary RL (Fei et al., 2020; Zhong et al., 2021; Zhou et al., 2020). The regret of nonstationary RL naturally depends on these notions that capture the variations of MDP models over time.
4.3 Theoretical Guarantee
To present our theoretical result for PORTAL, we first discuss technical challenges in our analysis and the novel tools that we develop. Generally, large nonstationarity of environment can cause significant errors to MLE, empirical covariance and exploration-driven bonus design for low-rank models. Thus, different from static low-rank MDPs (Agarwal et al., 2020b; Uehara et al., 2022), we devise several new techniques in our analysis to capture the errors caused by nonstationarity which we summarize below.
1. Characterizing nonstationary MLE guarantee. We provide a theoretical ground to support our design of leveraging history data collected under various different transition kernels in previous rounds for benefiting the estimation of the current model, which is somewhat surprising. Specifically, we establish an MLE guarantee of the model estimation error, which features a separation of variation budgets from a diminishing term as the estimation sample size  $W$  increases. Such a result justifies the usage of data generated by mismatched distributions within a certain window as long as the variation budgets is mild. Such a separation cannot be shown directly. Instead, we bridge the bound of model estimation error and the expected value of the ratio of transition kernels via Hellinger distance, and the latter can be decomposed into the variation budgets and a diminishing term as the estimation sample size increases.
2. Establishing trajectory-wise uncertainty for estimation error  $\hat{V}_{\hat{P}^k,\hat{b}^k}$ . To this end, straightforward combination of our nonstationary MLE guarantee with previous techniques on low-rank MDPs would yield a coefficient  $\tilde{\alpha}_{k,W}$  that depends on local variation budgets. Instead, we convert the
$\ell_{\infty}$ -norm variation budgets  $\Delta^{P}$  to square-root  $\ell_{\infty}$ -norm variation budget  $\Delta^{\sqrt{P}}$ . In this way, the coefficient no longer depends on the local variation budgets, and the estimation error can be upper bounded by  $\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$  plus an error term only depending on the square-root  $\ell_{\infty}$ -norm variation budgets.
3. Error tracking via auxiliary anchor representation. In proving the convergence of average of  $\hat{V}_{\hat{P}^k,\hat{b}^k}^\pi$ , standard elliptical potential based analysis cannot work, because the representation  $\phi^{\star,k}$  in the elliptical potential  $\left\| \phi^{\star,k}(s,a)\right\|_{(U_{h,\phi}^{k,W})^{-1}}$  changes across rounds, where  $U_{h,\phi}^{k,W}$  is the population version of  $\hat{U}_h^{k,W}$ . To deal with this challenge, in our analysis, we divide the total  $K$  rounds into blocks with equal length of  $W$  rounds. Then for each block, we set an auxiliary anchor representation. We keep track of the elliptical potential functions using the anchor representation within each block, and control the errors by using anchor representation via variation budgets.
The following theorem characterizes theoretical performance for PORTAL.
Theorem 4.4.  $\{\mathcal{M}^k\}_{k = 1}^K$  is set of low-rank MDPs with dimension  $d$ . Under Assumptions 4.1 to 4.3, set  $\tilde{\alpha}_{k,W} = \tilde{O}\left(\sqrt{A + d^2}\right)$  and  $\lambda_{k,W} = \tilde{O}(d)$ . Let  $\{\pi^k\}_{k = 1}^K$  be the output of PORTAL in Algorithm 1. For any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,  $\mathrm{Gap}_{\mathrm{Ave}}(K)$  of PORTAL is at most
$$
\tilde {O} \left(\underbrace {\sqrt {\frac {H ^ {4} d ^ {2} A}{W} (A + d ^ {2})} + \sqrt {\frac {H ^ {3} d A}{K} (A + d ^ {2}) W ^ {2} \Delta^ {\phi}} + \sqrt {\frac {H ^ {2} W ^ {3} A}{K ^ {2}}} \Delta^ {\sqrt {P}}} _ {(I)} \right. + \underbrace {\frac {H}{\sqrt {\tau}} + \frac {H \tau}{K} (\Delta^ {P} + \Delta^ {\pi})} _ {(I I)}. \tag {3}
$$
We explain the upper bound in Theorem 4.4 as follows. The basic bound as Equation (3) in Theorem 4.4 contains two parts: the first part  $(I)$  captures the estimation error for evaluating the target policy under the true environment via the estimated value function  $\hat{Q}^k$  as in line 16 of Algorithm 1. Hence, part  $(I)$  decreases with  $K$  and increases with the nonstationarity of transition kernels and representation functions. Also, part  $(I)$  is greatly affected by the window size  $W$ , which is determined by the dataset used to estimate the transition kernels. Typically,  $W$  is tuned carefully based on the variation of environment. If the environment changes significantly, then the samples far in the past are obsolete and become not very informative for estimating transition kernels. The second part  $(II)$  captures the approximation error arising in finding the optimal policy via the policy optimization method as in line 8 of Algorithm 1. Due to the nonstationarity of the environment, the optimal policy keeps changing across rounds, and hence the nonstationarity of optimal policy  $\Delta^\pi$  affects the approximation error. Similarly to the window size  $W$  in part  $(I)$ , the policy restart period  $\tau$  can also be tuned carefully based on the variation of environment and the optimal policies.
Corollary 4.5. Under the same conditions of Theorem 4.4, if the variation budgets are known, then we can select the hyper-parameters correspondingly to achieve optimality. Specially, if the nonstationarity of the environment is moderate, we have
$$
\mathrm {G a p} _ {\mathrm {A v e}} (K) \leq \tilde {O} (H ^ {\frac {1 1}{6}} d ^ {\frac {5}{6}} A ^ {\frac {1}{2}} \left(A + d ^ {2}\right) ^ {\frac {1}{2}} K ^ {- \frac {1}{6}} (\Delta^ {\sqrt {P}} + \Delta^ {\phi}) ^ {\frac {1}{6}} + 2 H K ^ {- \frac {1}{3}} (\Delta^ {P} + \Delta^ {\pi}) ^ {\frac {1}{3}}).
$$
If the environment is near stationary, then the best  $W$  and  $\tau$  are  $K$ . The GapAve reduces to  $\tilde{O}(\sqrt{H^4d^2A(A + d^2) / K})$ , which matches results of stationary environment (Uehara et al., 2022).
Detailed discussions and proofs of Theorem 4.4 and Corollary 4.5 are provided in Appendices A and B, respectively.
5 Parameter-free Algorithm: Ada-PORTAL
As shown in Theorem 4.4, hyper-parameters  $W$  and  $\tau$  greatly affect the performance of Algorithm 1. With prior knowledge of variation budgets, the agent is able to optimize the performance as in Corollary 4.5. However, in practice, variation budgets are unknown to the agent. To deal with this issue, in this section, we present a parameter-free algorithm called Ada-PORTAL in Algorithm 3, which is able to tune hyper-parameters without knowing the variation budgets beforehand.
Algorithm 3 is inspired by the BORL method (Cheung et al., 2020). The idea is to use Algorithm 1 as a subroutine and treat the selection of the hyper-parameters such as  $W$  and  $\tau$  as a bandit problem.
Algorithm 3 Ada-PORTAL (Adaptive Policy Optimization Represent Learning)
1: Input: Confidence level  $\delta$ , number of episodes  $K$ , block length  $M$ , feasible set of window size  $\mathcal{J}_W$  and policy restart period  $\mathcal{J}_{\tau}$ .  
2: Initialization: Initialize  $\alpha, \beta, \gamma$  and  $\{q_{l,1}\}_{l \in [J]}$  as in Equation (4).  
3: for block  $i = 1, \ldots, \lceil K / M \rceil$  do  
4: Update the windows size selection distribution  $\{u_{(k,l),i}\}_{(k,l) \in \mathcal{J}}$  as in Equation (5).  
5: Sample  $(k_i, l_i) \in \mathcal{J}$  from the updated distribution  $\{u_{(k,l),i}\}_{(k,l) \in \mathcal{J}}$ , then set  $W_i = \lfloor M^{k_i / J_M} \rfloor$  and  $\tau_i = \lfloor M_{\tau}^{l_i / J_\tau} \rfloor$ .  
6: for episode  $k = (i - 1)M + 1, \ldots, \min\{iM, K\}$  do  
7: Run PORTAL with  $W_i$  and  $\tau_i$ .  
8: end for  
9: Compute the total reward for block  $i$  as  $R_i(W_i, \tau_i) = \sum_{k=(i-1)M+1}^{\min\{iM, K\}} V_1^k$ , where  $V_1^k$  is empirical value functions of target policy  $\pi^k$ , and update the estimated total reward of running different epoch sizes  $\{q_{(k,l),i+1}\}_{(k,l) \in \mathcal{J}}$  according to Equation (6).  
10: end for  
11: Output:  $\{\pi^k\}_{k=1}^K$ .
Specifically, Ada-PORTAL divides the entire  $K$  rounds into  $\lceil K / M\rceil$  blocks with equal length of  $M$  rounds. Then two sets  $\mathcal{J}_W$  and  $\mathcal{J}_{\tau}$  are specified (see later part of this section), from which the window size  $W$  and the restart period  $\tau$  for each block are drawn.
For each block  $i \in \left[\left\lceil \frac{K}{M} \right\rceil \right]$ , Ada-PORTAL treats each element of  $\mathcal{I}_W \times \mathcal{I}_{\tau}$  as an arm and take it as a bandit problem to select the best arm for each block. In lines 4 and 5 of Algorithm 3, a master algorithm is run to update parameters and select the desired arm, i.e., the window size  $W_{i}$  and the restart period  $\tau_{i}$ . Here we choose EXP3-P (Bubeck & Cesa-Bianchi, 2012) as the master algorithm and discuss the details later. Then Algorithm 1 is called with input  $W_{i}$  and  $\tau_{i}$  as a subroutine for the current block  $i$ . At the end of each block, the total reward of the current block is computed by summing up all the empirical value functions of the target policy of each episode within the block, which is then used to update the parameters for the next block.
We next set the feasible sets and block length of Algorithm 1. Since optimal  $W$  and  $\tau$  in PORTAL are chosen differently from previous works (Zhou et al., 2020; Cheung et al., 2019; Touati & Vincent, 2020) on nonstationary MDPs due to the low-rank structure, the feasible set here that covers the optimal choices of  $W$  and  $\tau$  should also be set differently from those previous works using BORL.
$$
\begin{array}{r c l} M _ {W} & = & d ^ {\frac {1}{3}} H ^ {\frac {1}{3}} K ^ {\frac {1}{3}},   M _ {\tau}   =   K ^ {\frac {2}{3}},   M   =   d ^ {\frac {1}{3}} H ^ {\frac {1}{3}} K ^ {\frac {2}{3}}.   J _ {W}   =   \lfloor \log (M _ {W}) \rfloor ,   \mathcal {J} _ {W}   = \\ \{M _ {W} ^ {0}, \lfloor M _ {W} ^ {\frac {1}{J _ {W}}} \rfloor , \ldots , M _ {W} \},   J _ {\tau} = \lfloor \log (M _ {\tau}) \rfloor ,   \mathcal {J} _ {\tau} = \{M _ {\tau} ^ {0}, \lfloor M _ {\tau} ^ {\frac {1}{J _ {\tau}}} \rfloor , \ldots , M _ {\tau} \},   J = J _ {W} \cdot J _ {\tau}. \end{array}
$$
Then the parameters of EXP3-P are initialized as follows:
$$
\alpha = 0. 9 5 \sqrt {\frac {\ln J}{J [ K / M ]}}, \quad \beta = \sqrt {\frac {\ln J}{J [ K / M ]}}, \gamma = 1. 0 5 \sqrt {\frac {J \ln J}{[ K / M ]}}, \quad q _ {(k, l), 1} = 0, (k, l) \in \mathcal {J}, \tag {4}
$$
where  $\mathcal{J} = \{(k,l): k \in \{0,1,\ldots,J_W\}, l \in \{0,1,\ldots,J_\tau\}\}$ . The parameter updating rule is as follows. For any  $(k,l) \in \mathcal{J}, i \in \lceil K / M \rceil$
$$
u _ {(k, l), i} = (1 - \gamma) \frac {\exp \left(\alpha q _ {(k , l) , i}\right)}{\sum_ {(k , l) \in \mathcal {J}} \exp \left(\alpha q _ {(k , l) , i}\right)} + \frac {\gamma}{J}, \tag {5}
$$
where  $u_{(k,l),i}$  is a probability over  $\mathcal{J}$ . From  $u_{(k,l),i}$ , the agent samples a desired pair  $(k_i, l_i)$  for each block  $i$ , which corresponds to the index of feasible set  $\mathcal{J}_W \times \mathcal{J}_{\tau}$  and is used to set  $W_i$  and  $\tau_i$ .
As a last step,  $R_{i}(W_{i},\tau_{i})$  is rescaled to update  $q_{(k,l),i + 1}$
$$
q _ {(k, l), i + 1} = q _ {(k, l), i} + \frac {\beta + 1 _ {(k , l) = (k _ {i} , l _ {i})} R _ {i} \left(W _ {i} , \tau_ {i}\right) / M}{u _ {(k , l) , i}}. \tag {6}
$$
We next establish a bound on  $\mathrm{Gap}_{\mathrm{Ave}}$  for Ada-PORTAL.
Theorem 5.1. Under the same conditions of Theorem 4.4, with probability at least  $1 - \delta$ , the average dynamic suboptimality gap of Ada-PORTAL in Algorithm 3 is upper-bounded as  $\mathrm{Gap}_{\mathrm{Ave}}(K) \leq \tilde{O}(H^{\frac{11}{6}}d^{\frac{5}{6}}A^{\frac{1}{2}}(A + d^{2})^{\frac{1}{2}}K^{-\frac{1}{6}}(\Delta^{\sqrt{P}} + \Delta^{\phi} + 1)^{\frac{1}{6}} + 2HK^{-\frac{1}{3}}(\Delta^{P} + \Delta^{\pi} + 1)^{\frac{1}{3}})$ .
Comparison with Wei & Luo (2021): It is interesting to compare Ada-PORTAL with an alternative black-box type of approach to see its advantage. A black-box technique called MASTER was proposed in Wei & Luo (2021), which can also work with any base algorithm such as PORTAL to handle unknown variation budgets. Such a combined approach of MASTER+PORTAL turns out to have a worse  $\mathrm{Gap}_{\mathrm{Ave}}$  than our Ada-PORTAL in Algorithm 3. To see this, denote  $\Delta = \Delta^{\phi} + \Delta^{\sqrt{P}} + \Delta^{\pi}$ . The  $\mathrm{Gap}_{\mathrm{Ave}}$  of MASTER+PORTAL is  $\tilde{O}(K^{-\frac{1}{6}}\Delta^{\frac{1}{3}})$ . Then if variation budgets are not too small, i.e.  $\Delta \geq \tilde{O}(1)$ , this  $\mathrm{Gap}_{\mathrm{Ave}}$  is worse than Ada-PORTAL. See detailed discussion in Appendix C.2.
In the paper, we investigate nonstationary RL under low-rank MDPs. We first propose a notion of average dynamic suboptimality gap  $\mathrm{Gap}_{\mathrm{Ave}}$  to evaluate the performance of a series of policies in a nonstationary environment. Then we propose a sample-efficient policy optimization algorithm PORTAL and its parameter-free version Ada-PORTAL. We further provide upper bounds on  $\mathrm{Gap}_{\mathrm{Ave}}$  for both algorithms. As future work, it is interesting to investigate the impact of various constraints such as safety requirements in nonstationary RL under function approximations.
The work of Y. Liang was supported in part by the U.S. National Science Foundation under the grants RINGS-2148253, DMS-2134145, and ECCS-2113860. The work of J. Yang was supported by the U.S. National Science Foundation under the grants CNS-1956276 and CNS-2003131.
Agarwal, A., Henaff, M., Kakade, S. M., and Sun, W. PC-PG: policy cover directed exploration for provable policy gradient learning. In Advances in Neural Information Processing Systems 33, 2020a.
Agarwal, A., Kakade, S. M., Krishnamurthy, A., and Sun, W. FLAMBE: structural complexity and representation learning of low rank mdps. In Advances in Neural Information Processing Systems 33, 2020b.
Agarwal, A., Song, Y., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. arXiv preprint arXiv:2205.14571, 2022.
Besbes, O., Gur, Y., and Zeevi, A. Non-stationary stochastic optimization. Oper. Res., 63(5): 1227-1244, 2015.
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
Bubeck, S. and Cesa-Bianchi, N. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found. Trends Mach. Learn., 2012.
Cai, Q., Yang, Z., Jin, C., and Wang, Z. Provably efficient exploration in policy optimization. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Cheng, Y., Feng, S., Yang, J., Zhang, H., and Liang, Y. Provable benefit of multitask representation learning in reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Cheng, Y., Huang, R., Yang, J., and Liang, Y. Improved sample complexity for reward-free reinforcement learning under low-rank mdps. arXiv preprint arXiv:2303.10859, 2023.
Sun, W., Jiang, N., Krishnamurthy, A., Agarwal, A., and Langford, J. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on Learning Theory, 2019.
Touati, A. and Vincent, P. Efficient learning in non-stationary linear markov decision processes. CoRR, abs/2010.12870, 2020.
Tsybakov, A. B. Introduction to Nonparametric Estimation. Springer series in statistics. Springer, 2009.
Uehara, M., Zhang, X., and Sun, W. Representation learning for online and offline RL in low-rank mdps. In The Tenth International Conference on Learning Representations, 2022.
Wei, C. and Luo, H. Non-stationary reinforcement learning without prior knowledge: an optimal black-box approach. In Conference on Learning Theory, 2021.
Xu, T., Liang, Y., and Lan, G. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Zanette, A., Cheng, C.-A., and Agarwal, A. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory, 2021.
Zhao, X., Gu, C., Zhang, H., Yang, X., Liu, X., Tang, J., and Liu, H. DEAR: deep reinforcement learning for online advertising impression in recommender systems. In Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021.
Zhong, H., Yang, Z., Wang, Z., and Szepesvári, C. Optimistic policy optimization is provably efficient in non-stationary mdps. CoRR, abs/2110.08984, 2021.
Zhou, H., Chen, J., Varshney, L. R., and Jagmohan, A. Nonstationary reinforcement learning with linear function approximation. CoRR, 2020.
A Proof of Theorem 4.4
We summarize frequently used notations in the following list.
Proof Sketch of Theorem 4.4. The proof contains the following three main steps.
Step 1 (Appendix A.1): We first decompose the average dynamic suboptimal gap into three terms as in Lemma A.1, which can be divided into two parts: one part corresponds to the model estimation error and the other part corresponds to the performance difference between the target policy chosen by the agent and optimal policy. We then bound the two parts separately.