SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuan Cheng
yuan.cheng@u.nus.edu
Jing Yang
yangjing@psu.edu
liang.889@osu.edu
Reinforcement learning (RL) under changing environment models many real-world applications via nonstationary Markov Decision Processes (MDPs), and hence gains considerable interest. However, theoretical studies on nonstationary MDPs in the literature have mainly focused on tabular and linear (mixture) MDPs, which do not capture the nature of unknown representation in deep RL. In this paper, we make the first effort to investigate nonstationary RL under episodic low-rank MDPs, where both transition kernels and rewards may vary over time, and the low-rank model contains unknown representation in addition to the linear state embedding function. We first propose a parameter-dependent policy optimization algorithm called PORTAL, and further improve PORTAL to its parameter-free version of Ada-PORTAL, which is able to tune its hyper-parameters adaptively without any prior knowledge of nonstationarity. For both algorithms, we provide upper bounds on the average dynamic suboptimality gap, which show that as long as the nonstationarity is not significantly large, PORTAL and Ada-PORTAL are sample-efficient and can achieve arbitrarily small average dynamic suboptimality gap with polynomial sample complexity.
Reinforcement learning (RL) has gained significant success in real-world applications such as board games of Go and chess (Silver et al., 2016, 2017, 2018), robotics (Levine et al., 2016; Gu et al., 2017), recommendation systems (Zhao et al., 2021) and autonomous driving (Bojarski et al., 2016; Ma et al., 2021). Most theoretical studies on RL have been focused on a stationary environment and evaluated the performance of an algorithm by comparing against only one best fixed policy (i.e., static regret). However, in practice, the environment is typically time-varying and nonstationary. As a result, the transition dynamics, rewards and consequently the optimal policy change over time.
There has been a line of research studies that investigated nonstationary RL. Specifically, Gajane et al. (2018); Cheung et al. (2020); Mao et al. (2021) studied nonstationary tabular MDPs. To further overcome the curse of dimensionality, Fei et al. (2020); Zhou et al. (2020) proposed algorithms for nonstationary linear (mixture) MDPs and established upper bounds on the dynamic regret.
In this paper, we significantly advance this line of research by investigating nonstationary RL under low-rank MDPs (Agarwal et al., 2020b), where the transition kernel of each MDP admits a decomposition into a representation function and a state-embedding function that map to low dimensional spaces. Compared with linear MDPs where the representation is known, the low-rank MDP model contains unknown representation, and is hence much more powerful to capture
representation learning that occurs often in deep RL. Although there have been several recent studies on static low-rank MDPs (Agarwal et al., 2020b; Uehara et al., 2022; Modi et al., 2021), nonstationary low-rank MDPs remain unexplored, and are the focus of this paper.
To investigate nonstationary low-rank MDPs, several challenges arise. (a) All previous studies of nonstationary MDPs took on-policy exploration, such a strategy will have difficulty in providing sufficiently accurate model (as well as representation) learning for nonstationary low-rank MDPs. (b) Under low-rank MDPs, since both representation and state-embedding function change over time, it is more challenging to use history data collected under previous transition kernels for current use.
The main contribution of this paper lies in addressing above challenges and designing a provably efficient algorithm for nonstationary low-rank MDPs. We summarize our contributions as follows.
- We propose a novel policy optimization algorithm with representation learning called PORTAL for nonstationary low-rank MDPs. PORTAL features new components, including off-policy exploration, data-transfer model learning, and target policy update with periodic restart.
- We theoretically characterize the average dynamic suboptimality gap  $(\mathrm{Gap}_{\mathrm{Ave}})$  of PORTAL, where  $\mathrm{Gap}_{\mathrm{Ave}}$  serves as a new metric that captures the performance of target policies with respect to the best policies at each instance in the nonstationary MDPs under off-policy exploration. We further show that with prior knowledge on the degree of nonstationarity, PORTAL can select hyper-parameters that minimize  $\mathrm{Gap}_{\mathrm{Ave}}$ . If the nonstationarity is not significantly large, PORTAL enjoys a diminishing  $\mathrm{Gap}_{\mathrm{Ave}}$  with respect to the number of iterations  $K$ , indicating that PORTAL can achieve arbitrarily small  $\mathrm{Gap}_{\mathrm{Ave}}$  with polynomial sample complexity.
Our analysis features a few new developments. (a) We provide a new MLE guarantee under nonstationary transition kernels that captures errors of using history data collected under different transition kernels for benefiting current model estimation. (b) We establish trajectory-wise uncertainty bound for estimation errors via a square-root  $\ell_{\infty}$ -norm of variation budgets. (c) We develop an error tracking technique via auxiliary anchor representation for convergence analysis.
- Finally, we improve PORTAL to a parameter-free algorithm called Ada-PORTAL, which does not require prior knowledge on nonstationarity and is able to tune the hyper-parameters adaptively. We further characterize  $\mathrm{Gap}_{\mathrm{Ave}}$  of Ada-PORTAL as  $\tilde{O}(K^{-\frac{1}{6}}(\Delta + 1)^{\frac{1}{6}})$ , where  $\Delta$  captures the variation of the environment. Notably, based on PORTAL, we can also use the black-box method called MASTER in Wei & Luo (2021) to turn PORTAL into a parameter-free algorithm (called MASTER+PORTAL) with  $\mathrm{Gap}_{\mathrm{Ave}}$  of  $\tilde{O}(K^{-\frac{1}{6}}\Delta^{\frac{1}{3}})$ . Clearly, Ada-PORTAL performs better than MASTER+PORTAL when nonstationarity is not significantly small, i.e.  $\Delta \geq \tilde{O}(1)$ .
To our best knowledge, this is the first study of nonstationary RL under low-rank MDPs.
Various works have studied nonstationary RL under tabular and linear MDPs, most of which can be divided into two lines: policy optimization methods and value-based methods.
Nonstationary RL: Policy Optimization Methods. As a vast body of existing literature (Cai et al., 2020; Shani et al., 2020; Agarwal et al., 2020a; Xu et al., 2021) has proposed policy optimization methods attaining computational efficiency and sample efficiency simultaneously in stationary RL under various scenarios, only several papers investigated policy optimization algorithm in nonstationary environment. Assuming time-varying rewards and time-invariant transition kernels, Fei et al. (2020) studied nonstationary RL under tabular MDPs. Zhong et al. (2021) assumed both transition kernels and rewards change over episodes and studied nonstationary linear mixture MDPs. These policy optimization methods all assumed prior knowledge on nonstationarity.
Nonstationary RL: Value-based Methods. Assuming that both transition kernels and rewards are time-varying, several works have studied nonstationary RL under tabular and linear MDPs, most of which adopted Upper Confidence Bound (UCB) based algorithms. Cheung et al. (2020) investigated tabular MDPs with infinite-horizon and proposed algorithms with both known variation budgets and unknown variation budgets. In addition, this work also proposed a Bandit-over-Reinforcement Learning (BORL) technique to deal with unknown variation budgets. Mao et al. (2021) proposed a model-free algorithm with sublinear dynamic regret bound. They then proved a lower bound for nonstationary tabular MDPs and showed that their regret bound is near min-max optimal. Touati
& Vincent (2020); Zhou et al. (2020) considered nonstationary RL in linear MDPs and proposed algorithms achieving sublinear regret bounds with unknown variation budgets.
Besides these two lines of research, Wei & Luo (2021) proposed a black-box method that turns a RL algorithm with optimal regret in a (near-)stationary environment into another algorithm that can work in a nonstationary environment with sublinear dynamic regret without prior knowledge on nonstationarity. In this paper, we show that our algorithm Ada-PORTAL outperforms such a type of black-box method (taking PORTAL as subroutine) if nonstationarity is not significantly small.
Stationary RL under Low-rank MDPs. Low-rank MDPs were first studied by Agarwal et al. (2020b) in a reward-free regime, and then Uehara et al. (2022) studied low-rank MDPs for both online and offline RL with known rewards. Cheng et al. (2023) studied reward-free RL under low-rank MDPs and improved the sample complexity of previous works. Modi et al. (2021) proposed a model-free algorithm MOFFLE under low-nonnegative-rank MDPs. Cheng et al. (2022); Agarwal et al. (2022) studied multitask representation learning under low-rank MDPs, and further showed the benefit of representation learning to downstream RL tasks.
3 Formulation
Notations: We use  $[K]$  to denote set  $\{1,\dots ,K\}$  for any  $K\in \mathbb{N}$ , use  $\| x\| _2$  to denote the  $\ell_2$  norm of vector  $x$ , use  $\triangle (\mathcal{A})$  to denote the probability simplex over set  $\mathcal{A}$ , use  $\mathcal{U}(\mathcal{A})$  to denote uniform sampling over  $\mathcal{A}$ , given  $|\mathcal{A}| < \infty$ , and use  $\triangle (\mathcal{S})$  to denote the set of all possible density distributions over set  $\mathcal{S}$ . Furthermore, for any symmetric positive definite matrix  $\Sigma$ , we let  $\| x\|_{\Sigma}:= \sqrt{x^{\top}\Sigma x}$ . For distributions  $p_1$  and  $p_2$ , we use  $D_{KL}(p_1(\cdot)\| p_2(\cdot))$  to denote the KL divergence between  $p_1$  and  $p_2$ .
3.1 Episodic MDPs and Low-rank Approximation
An episodic MDP is denoted by a tuple  $\mathcal{M} := \left(S, \mathcal{A}, H, P : = \{P_h\}_{h=1}^H, r : = \{r_h\}_{h=1}^H\right)$ , where  $\mathcal{S}$  is a possibly infinite state space,  $\mathcal{A}$  is a finite action space with cardinality  $A$ ,  $H$  is the time horizon of each episode,  $P_h(\cdot|\cdot,\cdot) : \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  denotes the transition kernel at each step  $h$ , and  $r_h(\cdot,\cdot) : \mathcal{S} \times \mathcal{A} \to [0,1]$  denotes the deterministic reward function at each step  $h$ . We further normalize the reward as  $\sum_{h=1}^{H} r_h \leq 1$ . A policy  $\pi = \{\pi_h\}_{h \in [H]}$  is a set of mappings where  $\pi_h : \mathcal{S} \to \Delta(\mathcal{A})$ . For any  $(s,a) \in \mathcal{S} \times \mathcal{A}$ ,  $\pi_h(a|s)$  denotes the probability of selecting action  $a$  at state  $s$  at step  $h$ . For any  $(s,a) \in \mathcal{S} \times \mathcal{A}$ , let  $(s_h, a_h) \sim (P, \pi)$  denote that the state  $s_h$  is sampled by executing policy  $\pi$  to step  $h$  under transition kernel  $P$  and then action  $a_h$  is sampled by  $\pi_h(\cdot|s_h)$ .
Given any state  $s \in S$ , the value function for a policy  $\pi$  at step  $h$  under an MDP  $\mathcal{M}$  is defined as the expected value of the accumulative rewards as:  $V_{h,P,r}^{\pi}(s) = \sum_{h' = h}^{H}\mathbb{E}_{(s_{h'},a_{h'})\sim (P,\pi)}[r_{h'}(s_{h'},a_{h'})|s_h = s]$ . Similarly, given any state-action pair  $(s,a) \in S \times \mathcal{A}$ , the action-value function  $(Q$ -function) for a policy  $\pi$  at step  $h$  under an MDP  $\mathcal{M}$  is defined as  $Q_{h,P,r}^{\pi}(s,a) = r_h(s,a) + \sum_{h' = h + 1}^{H}\mathbb{E}_{(s_{h'},a_{h'})\sim (P,\pi)}[r_{h'}(s_{h'},a_{h'})|s_h = s,a_h = a]$ . Denote  $(P_{h}f)(s,a) := \mathbb{E}_{s'\sim P_{h}(\cdot |s,a)}[f(s')]$  for any function  $f: S \to \mathbb{R}$ . Then we can write the action-value function as  $Q_{h,P,r}^{\pi}(s,a) = r_h(s,a) + (P_{h}V_{h + 1,P,r}^{\pi})(s,a)$ . For any  $k \in [K]$ , without loss of generality, we assume the initial state  $s_1$  to be fixed and identical, and we use  $V_{P,r}^{\pi}$  to denote  $V_{1,P,r}^{\pi}(s_1)$  for simplicity.
This paper focuses on low-rank MDPs (Jiang et al., 2017; Agarwal et al., 2020b) defined as follows.
Definition 3.1 (Low-rank MDPs). A transition kernel  $P_h^* : \mathcal{S} \times \mathcal{A} \to \triangle(\mathcal{S})$  admits a low-rank decomposition with dimension  $d \in \mathbb{N}$  if there exist a representation function  $\phi_h^\star : \mathcal{S} \times \mathcal{A} \to \mathbb{R}^d$  and a state-embedding function  $\mu_h^\star : \mathcal{S} \to \mathbb{R}^d$  such that
$$
P _ {h} ^ {\star} \left(s ^ {\prime} \mid s, a\right) = \left\langle \phi_ {h} ^ {\star} (s, a), \mu_ {h} ^ {\star} \left(s ^ {\prime}\right) \right\rangle , \quad \forall s, s ^ {\prime} \in \mathcal {S}, a \in \mathcal {A}.
$$
Without loss of generality, we assume  $\| \phi_h^* (s,a)\| _2\leq 1$  for all  $(s,a)\in S\times \mathcal{A}$  and for any function  $g:\mathcal{S}\mapsto [0,1]$ ,  $\left\| \int \mu_h^\star (s)g(s)ds\right\| _2\leq \sqrt{d}$ . An MDP is a low-rank MDP with dimension  $d$  if for any  $h\in [H]$ , its transition kernel  $P_{h}^{*}$  admits a low-rank decomposition with dimension  $d$ . Let  $\phi^{\star} = \{\phi_h^{\star}\}_{h\in [H]}$  and  $\mu^{\star} = \{\mu_h^{\star}\}_{h\in [H]}$  be the true representation and state-embedding functions.
3.2 Nonstationary Transition Kernels with Adversarial Rewards
In this paper, we consider an episodic RL setting under changing environment, where both transition kernels and rewards vary over time and possibly in an adversarial fashion.
Specifically, suppose the RL system goes by rounds, where each round has a fixed number of episodes, and the transition kernel and the reward remain the same in each round, and can change adversarially across rounds. For each round, say round  $k$ , we denote the MDP as  $\mathcal{M}^k = (\mathcal{S},\mathcal{A},H,P^k := \{P_h^{\star ,k}\}_{h = 1}^H,r^k := \{r_h^k\}_{h = 1}^H)$ , where  $P^{\star ,k}$  and  $r^k$  denote the true transition kernel and the reward of round  $k$ . Further,  $P^{\star ,k}$  takes the low-rank decomposition as  $P^{\star ,k} = \langle \phi^{\star ,k},\mu^{\star ,k}\rangle$ . Both the representation function  $\phi^{\star ,k}$  and the state embedding function  $\mu^{\star ,k}$  can change across rounds. Given the reward function  $r^k$ , there always exists an optimal policy  $\pi^{\star ,k}$  that yields the optimal value function  $V_{P^{\star ,k},r^{k}}^{\pi^{\star ,k}} = \sup_{\pi}V_{P^{\star ,k},r^{k}}^{\pi}$ , abbreviated as  $V_{P^{\star ,k},r^{k}}^{\star}$ . Clearly, the optimal policy also changes across rounds.
We assume the agent interacts with the nonstationary environment (i.e., the time-varying MDPs) over  $K$  rounds in total without the knowledge of transition kernels  $\{P^{k,\star}\}_{k=1}^{K}$ . At the beginning of each round  $k$ , the environment changes to a possibly adversarial transition kernel unknown to the agent, picks a reward function  $r^k$ , which is revealed to the agent only at the end of round  $k$ , and outputs a fixed initial state  $s_1$  for the agent to start the exploration of the environment for each episode. The agent is allowed to interact with MDPs via a few episodes with one or multiple exploration policies at her choice to take samples from the environment and then should output an target policy to be executed during the next round. Note that in our setting, the agent needs to decide exploration and target policies only based on the information in previous rounds, and hence exploration samples and the reward information of the current round help only towards future rounds.
3.3 Learning Goal and Evaluation Metric
In our setting, the agent seeks to find the optimal policy at each round  $k$  (with only the information of previous rounds), where both transition kernels and rewards can change over rounds. Hence we define the following notion of average dynamic suboptimality gap to measure the convergence of the target policy series to the optimal policy series.
Definition 3.2 (Average Dynamic Suboptimality Gap). For  $K$  rounds, and any policy set  $\{\pi^k\}_{k\in [K]}$  the average dynamic suboptimality gap  $(\mathrm{Gap_{Ave}})$  of the value functions over  $K$  rounds is given as  $\mathrm{Gap_{Ave}}(K) = \frac{1}{K}\sum_{k = 1}^{K}[V_{P^{\star},k,r^k}^\star -V_{P^{\star},k,r^k}^{\pi^k}]$ . For any  $\epsilon$ , we say an algorithm is  $\epsilon$ -average suboptimal, if it outputs a policy set  $\{\pi^k\}_{k\in [K]}$  satisfying  $\mathrm{Gap_{Ave}}(K)\leq \epsilon$ .
$\mathrm{Gap}_{\mathrm{Ave}}$  compares the agent's target policy to the optimal policy of each individual round in hindsight, which captures the dynamic nature of the environment. This is in stark contrast to the stationary setting where the comparison policy is a single fixed best policy over all rounds. This notion is similar to dynamic regret used for nonstationary RL (Fei et al., 2020; Gajane et al., 2018), where the only difference is that  $\mathrm{Gap}_{\mathrm{Ave}}$  evaluates the performance of target policies rather than the exploration policies. Hence, given any target accuracy  $\epsilon \geq 0$ , the agent is further interested in the statistical efficiency of the algorithm, i.e., using as few trajectories as possible to achieve  $\epsilon$ -average suboptimal.
4 Policy Optimization Algorithm and Theoretical Guarantee
4.1 Base Algorithm: PORTAL
We propose a novel algorithm called PORTAL (Algorithm 1), which features three main steps. Below we first summarize our main design ideas and then explain reasons behind these ideas as we further elaborate main steps of PORTAL.
Summary of New Design Ideas: PORTAL features the following main design ideas beyond previous studies on nonstationary RL under tabular and linear MDPs. (a) PORTAL features a specially designed off-policy exploration which turns out to be beneficial for nonstationary low-rank /MDP models rather than the typical on-policy exploration taken by previous studies of nonstationary tabular and linear MDP models. (b) PORTAL transfers history data collected under various different transition kernels for benefiting the estimation of the current model. (c) PORTAL updates target
policies with periodic restart. As a comparison, previous work using periodic restart (Fei et al., 2020) chooses the restart period  $\tau$  based on a certain smooth visitation assumption. Here, we remove such an assumption and hence our choice of  $\tau$  is applicable to more general model classes.
Step 1. Off-Policy Exploration for Data Collection: We take off-policy exploration, which is beneficial for nonstationary low-rank MDPs than simply using the target policy for on-policy exploration taken by the previous studies on nonstationary tabular or linear (mixture) MDPs (Zhong et al., 2021; Fei et al., 2020; Zhou et al., 2020). To further explain, we first note that under tabular or linear (mixture) MDPs studied in the previous work, a bonus term is introduced to the actual reward to serve as a point-wise uncertainty level of the estimation error for each state-action pair at any step  $h$ , so that for any step  $h$ ,  $\hat{Q}_h^k$  is a good optimistic estimation for  $Q_{h,P^{\star},k,r^k}^{\pi}$ . Hence it suffices to collect samples using the target policy. However, in low-rank MDPs, the bonus term  $\hat{b}_h^k$  cannot serve as a point-wise uncertainty measure. For step  $h \geq 2$ ,  $\hat{Q}_h^k$  is not a good optimistic estimation for the true value function if the agent only uses target policy to collect data (i.e., for on-policy exploration). Hence, more samples and a novel off-policy exploration are required for a good estimation under low-rank MDPs. Specifically, as line 5 in Algorithm 1, at the beginning of each round  $k$ , for each step  $h \in [H]$ , the agent explores the environment by executing the exploration policy  $\tilde{\pi}^{k-1}$  to state  $\hat{s}_{h-1}^{k,h}$  and then taking two uniformly chosen actions<sup>1</sup>, where  $\tilde{\pi}^{k-1}$  is determined in Step 2 of the previous round.
Algorithm 1 PORTAL (Policy Optimization with Represent Learning under nonstationary MDPs)
1: Input: Rounds  $K$ , hyper-parameters  $\tau, W$ , regularizer  $\lambda_{k,W}$ , coefficient  $\tilde{\alpha}_{k,W}$ , stepsize  $\eta$  and models  $\{\Psi, \Phi\}$ .  
2: Initialization:  $\pi_0(\cdot|s)$  to be uniform;  $\tilde{\mathcal{D}}_h^{(0,0)} = \emptyset$ .  
3: for episode  $k = 1, \ldots, K$  do  
4: for step  $h = 1, \ldots, H$  do  
5: Roll into  $\tilde{s}_{h-1}^{(k,h)}$  using  $\tilde{\pi}^{k-1}$ , uniformly choose  $\tilde{a}_{h-1}^{(k,h)}, \tilde{a}_h^{(k,h)}$ , and enter into  $\tilde{s}_h^{(k,h)}, \tilde{s}_{h+1}^{(k,h)}$ .  
6: Update datasets  
 $\tilde{\mathcal{D}}_{h-1}^{(k,h,W)} = \left\{\tilde{s}_{h-1}^{(i,h)}, \tilde{a}_{h-1}^{(i,h)}, \tilde{s}_h^{(i,h)}\right\}_{i=1 \vee k-W+1}^k$ ,  
 $\tilde{\mathcal{D}}_h^{(k,h,W)} = \left\{\tilde{s}_h^{(i,h)}, \tilde{a}_h^{(i,h)}, \tilde{s}_{h+1}^{(i,h)}\right\}_{i=1 \vee k-W+1}^k$ .  
7: end for  
8: Receive full information rewards  $r^k = \{r_h^k\}_{h \in [H]}$ .  
9: Estimate transition kernel and update the exploration policy  $\tilde{\pi}^k$  for the next round via:  
 $\mathrm{E}^2\mathrm{U}\left(k, \{\tilde{\mathcal{D}}_{h-1}^{(k,h,W)}\}, \{\tilde{\mathcal{D}}_h^{(k,h,W)}\}\right)$ .  
10: for step  $h = 1, \ldots, H$  do  
11: Update  $\hat{Q}_h^k = Q_{h,\hat{P}^k,r^k}^k$ .  
12: end for  
13: if  $k \mod \tau = 1$  then  
14: Set  $\{\hat{Q}_h^k\}_{h \in [H]}$  as zero functions and  $\{\pi_h^k\}_{h \in [H]}$  as uniform distributions on  $\mathcal{A}$ .  
15: end if  
16: for step  $h = 1, \ldots, H$  do  
17: Update the target policy as in Equation (2).  
18: end for  
19: end for  
20: Output:  $\{\pi^k\}_{k=1}^K$ .
Step 2. Data-Transfer Model Learning and  $\mathbf{E}^2\mathbf{U}$ : In this step, we transfer history data collected under previous different transition kernels for benefiting the estimation of the current model. This is theoretically grounded by our result that the model estimation error can be decomposed into variation
The subscript  $h - 1$  in  $\tilde{s}_{h - 1}^{k,h}$  indicates that the data is collected at time step  $h$  of each trajectory, and the superscript  $(k,h)$  indicates in which loop the data is collected (as in line 3 and 4 of Algorithm 1)
budgets plus a diminishing term as the estimation sample size increases, which justifies that the usage of data generated by mismatched distributions within a certain window is beneficial for model learning as long as variation budgets is mild. Then, the estimated model will further facilitate the selection of future exploration policies accurately.
Specifically, the agent selects desirable samples only from the latest  $W$  rounds following a forgetting rule (Garivier & Moulines, 2011). Since nonstationary low-rank MDPs (compared to tabular and linear MDPs) also have additional variations on representations over time, the choice of  $W$  needs to incorporate such additional information. Then the agent passes these selected samples to a subroutine  $\mathrm{E}^2\mathrm{U}$  (see Algorithm 2), in which the agent estimates the transition kernels via the maximum likelihood estimation (MLE). Next, the agent updates the empirical covariance matrix  $\hat{U}^{k,W}$  and exploration-driven bonus  $\hat{b}^k$  as in lines 4 and 5 in Algorithm 2. We then define a truncated value function iteratively using the estimated transition kernel and the exploration-driven reward as follows:
$$
\hat {Q} _ {h, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {h}, a _ {h}) = \min  \left\{1, \hat {b} _ {h} ^ {k} (s _ {h}, a _ {h}) + \hat {P} _ {h} ^ {k} \hat {V} _ {h + 1, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {h}, a _ {h}) \right\},
$$
$$
\hat {V} _ {h, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {h}) = \mathbb {E} _ {\pi} \left[ \hat {Q} _ {h, \hat {P} ^ {k}, \hat {b} ^ {k}} ^ {\pi} (s _ {h}, a _ {h}) \right]. \tag {1}
$$