SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we introduce a novel upper-bound on the discrepancy between the value,  $q_{\mathbf{w}}^{h - \mathrm{GPI}}$ , of the  $h$ -GPI policy under a particular reward function, and the optimal action-value function,  $q_{\mathbf{w}}^*$ . Notice that, as in Theorem 1, we can recover the standard GPI bound when setting  $h = 0$ .<sup>2</sup>
Theorem 2. Let  $\Pi = \{\pi_i^*\}_{i=1}^n$  be a set of optimal policies with respect to reward weights  $\{\mathbf{w}_i\}_{i=1}^n$  and  $\mathbf{w}$  be arbitrary reward weights. Let  $\hat{m} = (\hat{p}, \hat{r}_{\mathbf{w}})$  be an approximate model and  $\{\hat{q}_{\mathbf{w}}^{\pi_i^*}\}_{i=1}^n$  be approximations to the action-value functions of policies in  $\Pi$ , under the reward function  $r_{\mathbf{w}}$ , such that for all  $\pi_i \in \Pi$  and all  $(s, a) \in S \times \mathcal{A}$ ,
$$
\left| q _ {\mathbf {w}} ^ {\pi_ {i} ^ {*}} (s, a) - \hat {q} _ {\mathbf {w}} ^ {\pi_ {i} ^ {*}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} | p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) | \leq \epsilon_ {p}, \text {a n d} \left| r _ {\mathbf {w}} (s, a) - \hat {r} _ {\mathbf {w}} (s, a) \right| \leq \epsilon_ {r}. \tag {15}
$$
We now extend the definition of  $h$ -GPI (Definition 3) to the case where this policy is defined under the assumption of an MPD with reward function  $r_{\mathbf{w}}$ :
$$
\pi^ {h \cdot G P I} (s; \mathbf {w}) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} _ {\mathbf {w}} ^ {\pi} (s, a). \tag {16}
$$
Let  $\phi_{\max} \triangleq \max_{s,a} ||\phi(s,a)||$ . Then, it follows that
$$
q _ {\mathbf {w}} ^ {*} (s, a) - q _ {\mathbf {w}} ^ {h \cdot G P I} (s, a) \leq \frac {2}{1 - \gamma} \left(\phi_ {\max } \min  _ {i} | | \mathbf {w} - \mathbf {w} _ {i} | | + \gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right), \tag {17}
$$
where  $c(\epsilon_r, \epsilon_p, h) = \frac{1 - \gamma^h}{1 - \gamma} (\epsilon_r + \gamma \epsilon_p v_{max}^*)$ .
Theorem 2 precisely characterizes the optimality gap (i.e., the maximum difference) between the action-value function induced by  $h$ -GPI ( $q_{\mathbf{w}}^{h - \mathrm{GPI}}(s, a)$ ) and the optimal action-value function ( $q_{\mathbf{w}}^*(s, a)$ ), as a function of (i) the reward weights  $\{\mathbf{w}_i\}_{i=1}^n$  for which the policies in the agent's library, II, are optimal; (ii) approximation errors in action-value functions  $\{\hat{q}_{\mathbf{w}}^{\pi_i^*}\}_{i=1}^n$ ; and (iii) approximation errors in the model  $\hat{m} = (\hat{p}, \hat{r}_{\mathbf{w}})$ .
3.1  $h$ -GPI under (learned) approximate models for zero-shot transfer
Recall that our goal is to perform zero-shot policy transfer to solve any tasks  $M \in \mathcal{M}^{\phi}$  (Equation (1)). In order to employ  $h$ -GPI on any given reward function of the form  $r_{\mathbf{w}}(S_t, A_t, S_{t+1}) = \phi_t \cdot \mathbf{w}$ , we propose learning a model,  $\hat{m} = (\hat{p}, \hat{\phi})$ , that approximates both the state transition function,  $p$ , and reward features,  $\phi$ . The key insight is that this model implicitly induces the space of all MDP models  $m = (p, r_{\mathbf{w}})$ , based on (arbitrary) reward weights  $\mathbf{w} \in \mathbb{R}^d$ . This allows employing RL model-based methods to multi-task settings without the need to re-train the model when solving new tasks defined by different reward functions. In Algorithm 1, we present a high-level description of how the  $h$ -GPI policy  $\pi^{h - \mathrm{GPI}}(s, \mathbf{w})$  can be computed given a model  $\hat{m} = (\hat{p}, \hat{\phi})$ , SFs  $\{\hat{\psi}^{\pi_i}\}_{i=1}^n$ , and horizon  $h \geq 0$ .
Tabular setting. In the tabular case, models can be learned via maximum likelihood estimates:  $\hat{m}(s', \phi|s, a) \approx \frac{N(s, a, \phi, s')}{N(s, a)}$ , where  $N(s, a, \phi, s')$  is the number of times the agent experienced the
2The proofs of Theorem 1 and Theorem 2 can be found in Appendix A.
Algorithm 1:  $h$ -GPI with Successor Features
Input: Model  $\hat{m} = (\hat{p},\hat{\phi})$  , SFs  $\{\hat{\psi}^{\pi_i}\}_{i = 1}^n$  , planning horizon  $h\geq 0$  , state s, reward weights w.   
for action  $a\in \mathcal{A}$  do   
Let  $S_{t} = s$ $\mu_0(s) = a$    
Compute  $(T_{\hat{m}}^{*})^{h}\max_{\pi \in \Pi}\hat{q}_{\mathbf{w}}^{\pi}(s,a)\gets$    
max  $\max_{\mu_1\dots \mu_{h - 1}}\mathbb{E}_{\hat{m}}\left[\sum_{k = 0}^{h - 1}\gamma^k\hat{\phi}_{t + k}(\hat{S}_{t + k},\mu_k(\hat{S}_{t + k}))\cdot \mathbf{w} + \gamma^h\max_{a'\in \mathcal{A}}\max_{\pi \in \Pi}\hat{\psi}^\pi (\hat{S}_{t + h},a')\cdot \mathbf{w}\right]$    
4 Return:  $\pi^{h - \mathrm{GPI}}(s;\mathbf{w})\in \arg \max_{a\in \mathcal{A}}(\mathcal{T}_{\hat{m}}^{*})^{h}\max_{\pi_{i}\in \Pi}\hat{q}_{\mathbf{w}}^{\pi_{i}}(s,a)$
transition  $(s, a, \phi, s')$  and  $N(s, a)$  is the number of times the agent selected action  $a$  in state  $s$ . To efficiently compute the action given by the policy  $\pi^{h - \mathrm{GPI}}(s, \mathbf{w})$  (line 3 of Algorithm 1), we extend the Forward-Backward Dynamic Programming (FB-DP) algorithm (Efroni et al., 2020) to the discounted-SFs setting, in order to compute  $h$ -lookahead policies in real-time. The corresponding pseudocode can be found in Appendix B. Notably, given a state  $s$ , FB-DP time complexity is  $O(N|\mathcal{A}||\mathcal{S}_h^{\mathrm{tot}}|)$ , where  $N$  is the maximal number of accessible states in one step (maximal "nearest neighbors" from any state), and  $\mathcal{S}_h^{\mathrm{tot}}$  is total reachable states in  $h$  time steps from state  $s$ . Note that while we chose to extend FB-DP due to its efficiency and closed-form formulation, other planning techniques could have been used (e.g., Monte Carlo tree search (Tesauro and Galperin, 1996; Silver et al., 2017)).
Continuous-states setting. In the continuous-state setting, we extend the class of models composed of ensembles of probabilistic neural networks (Chua et al., 2018) to the SFs setting. These models are used in state-of-the-art single-task model-based RL algorithms (Janner et al., 2019; Yu et al., 2021). The learned model  $\hat{m}_{\varphi}(s', \phi|s, a)$ , parameterized by  $\varphi$ , is composed of an ensemble of  $n$  neural networks,  $\{\hat{m}_{\varphi_i}\}_{i=1}^n$ , each of which outputs the mean and diagonal covariance matrix of a multivariate Gaussian distribution:  $m_{\varphi_i}(S_{t+1}, \phi_t \mid S_t, A_t) = \mathcal{N}(\mu_{\varphi_i}(S_t, A_t), \Sigma_{\varphi_i}(S_t, A_t))$ . Each model in the ensemble is trained in parallel to minimize the following negative log-likelihood loss function, using different bootstraps of experiences in a buffer  $\mathcal{B} = \{(S_t, A_t, \phi_t, S_{t+1}) | t \geq 0\}$ :  $\mathcal{L}(\varphi) = \mathbb{E}_{(S_t, A_t, \phi_t, S_{t+1}) \sim \mathcal{B}}[-\log m_{\varphi}(S_{t+1}, \phi_t | S_t, A_t)]$ . In practice, we use as  $\mathcal{B}$  the buffer with experiences the agent collected while training the SFs for the training tasks. In order to compute the  $h$ -GPI policy in this setting, we approximate the expectation over next states by averaging over the predictions of the components of the ensemble (Chua et al., 2018).
We employ universal successor features approximators (USFAs) (Borsa et al., 2019) to learn SFs in the function approximation setting. Given sufficiently expressive USFAs, we can evaluate the value function of any policy  $\pi_{\mathbf{z}}$  (optimal for task  $\mathbf{z}$ ) in any task  $\mathbf{w} \in \mathbb{R}^d$  by simply computing  $\hat{q}_{\mathbf{w}}^{\pi_{\mathbf{z}}}(s,a) \approx \hat{\psi}(s,a,\mathbf{z}) \cdot \mathbf{w}$ . Furthermore, when using a USFA to generalize to a new task  $\mathbf{w}'$ , GPI (Equation 5) becomes:  $\pi^{\mathrm{GPI}}(s;\mathbf{w}) \in \arg \max_{a \in \mathcal{A}} \max_{\mathbf{w} \in \mathcal{M}} \hat{\psi}(s,a,\mathbf{w}) \cdot \mathbf{w}'$ , where  $\mathcal{M}$  is typically a set of weight vectors used when training the USFA. Notably, we only require a single USFA  $\hat{\psi}$  and a single model  $\hat{m}$  to perform  $h$ -GPI given any  $M \in \mathcal{M}_{\phi}$ , in contrast to other approaches that require learning a complex model for each policy in the library (Thakoor et al., 2022).
Notice that although we assume that during training the agent can observe and add to its buffer  $\mathcal{B}$  the features  $\phi_t$  at each time step  $t$ , our method could also be applied to the setting where the features are learned beforehand in a pre-training phase (Hansen et al., 2020; Carvalho et al., 2023). Learning appropriate reward features  $\phi$  is an important but orthogonal problem to the one tackled in this paper.
We conduct tabular and deep RL experiments in three different domains to evaluate the effectiveness of  $h$ -GPI as a method for zero-shot policy transfer.
**Environments.** First, we consider the tabular FourRoom domain (Barreto et al., 2017). To make this domain more challenging, we made its transition function stochastic by including a  $10\%$  chance of the agent moving to a random direction after taking any action. In FourRoom, the reward features  $\phi_t \in \mathbb{R}^3$  correspond to binary vectors indicating whether the agent collected each of the three types of objects in the map. The second domain is Reacher (Alegre et al., 2022a), which consists of a robotic arm that can apply torque to each of its two joints. The features  $\phi_t \in \mathbb{R}^4$  are proportional
Figure 2: Median, IQM, and mean normalized returns over a set of test tasks in the stochastic FourRoom domain.  $h$ -GPI performance significantly improves as  $h$  increases.
to the distance of the tip of the arm to four different targets. Finally, we extend the FetchPush domain (Plappert et al., 2018), which consists of a fetch robotic arm that must move a block to a target position on top of a table by pushing the block with its gripper. The reward features  $\phi_t \in \mathbb{R}^4$  are the negative distances of the block to each of the four target locations. The action space was discretized in the same way as usually done in Reacher. Importantly, the state space of this domain is high-dimensional ( $S \subset \mathbb{R}^{19}$ ) and its dynamics are significantly more complex than Reacher's. A more detailed description of the domains can be found in Appendix B.
Baselines. We compare  $h$ -GPI with standard GPI (Barreto et al., 2017), which is equivalent to  $h$ -GPI when  $h = 0$ . To showcase the importance of combining model-based planning with model-free action-value estimates, we include an SF-based MPC approach, which is equivalent to  $h$ -GPI but without GPI bootstrapping at time step  $h$ . That is, given a task  $\mathbf{w}$ , it selects the first action from a sequence of actions that maximizes  $\mathbb{E}_{\hat{m}}[\sum_{k=0}^{h-1} \gamma^k \hat{\phi}_{t+k} \cdot \mathbf{w}]$  and re-plans at every time step. In the function approximation case with USFAs, we also compare with Constrained GPI (CGPI) (Kim et al., 2022). CGPI is a state-of-the-art method that computes lower and upper bounds for the optimal value on new tasks and uses them to constrain value approximation errors when following the GPI policy.
Figure 3:  $h$ -GPI, under various levels of errors in the SFs, as a function of  $h$ .
Based on the theoretical properties introduced in Section 3, we expect the following hypotheses to hold. H1:  $h$ -GPI performance better generalizes to unseen test tasks than standard GPI and, in general, improves as we increase  $h$ . H2:  $h$ -GPI is more robust to approximation errors in the learned SFs than GPI, and such errors have a decreasing impact as  $h$  increases. Additionally, we investigate whether H3:  $h$ -GPI can be scaled up to solve problems with high-dimensional state spaces. In each experiment, we report the median, interquartile mean (IQM), and mean normalized returns with their  $95\%$  stratified bootstrap confidence intervals (Agarwal et al., 2021), which are obtained by evaluating agents trained with 20 different random seeds on a set of test tasks. For fairness of comparison, we used the same agents for evaluating each method. We follow previous works (Borsa et al., 2019; Kim et al., 2022) and use as training tasks the weight vectors that form the standard basis of  $\mathbb{R}^d$  in all three domains. In FourRoom, we use 32 weights vectors equally spaced from the weight simplex  $\{\mathbf{w} \mid \sum_{i=1}^{d} w_i = 1, w_i \geq 0\}$  as test tasks. For Reacher and FetchPush, we follow Kim et al. (2022) and use weight vectors defined by  $\{-1, 1\}^d$  as test tasks. Notice that this results in many tasks with negative weights, which are significantly different than the training tasks.
In Figure 2, we can observe that  $h$ -GPI is able to outperform GPI even with small planning horizons (see, e.g., 1-GPI). As  $h$  increases, the performance on the test tasks consistently increases. This is in accordance with Theorem 1 and H1. Notice that even with a small planning horizon of  $h = 2$ ,  $h$ -GPI is capable of matching the performance of SF-MPC even when SF-MPC is allowed to plan for ten times longer (i.e.,  $h = 20$ ).
Next, to investigate H2, we evaluate  $h$ -GPI after artificially adding Gaussian noise with different standard deviations to the learned SFs of each of the agent's policies for every state and action. We can notice (in Figure 3) that standard GPI ( $h = 0$ ) catastrophically fails under all levels of
Figure 4: Median, IQM, and mean normalized returns over test tasks in the Reacher domain.
Figure 5: Median, IQM, and mean normalized returns over test tasks in the FetchPush domain.
approximation errors. In all cases,  $h$ -GPI can closely approximate the same performance levels as those resulting from settings with no added noise. To do so, it suffices to use a sufficiently large  $h$ . In other words,  $h$ -GPI successfully and effectively identifies policies that are robust to noise due to its ability to combine model-free GPI generalization and planning under approximate/imperfect models for  $h$  steps. This is consistent with our theoretical results that show that value function approximation errors impact the performance's lower bound by a factor of  $\gamma^h$ .
In Figures 4 and 5, we show the results in the Reacher and FetchPush domains, respectively. Given a test task  $\mathbf{w}'$ , let GPI-ST be the policy that uses both the source/training (S) and target (T) tasks,  $\mathcal{M} \cup \{\mathbf{w}'\}$ , as input to the USFA when performing GPI. Similarly, we refer to GPI as the policy that performs GPI only over the training tasks  $\mathcal{M}$ . The CGPI policy was unable to significantly improve performance over GPI-ST, which we hypothesize may be due to USFA errors in the training tasks, or due to overly small minimum rewards used in its upper-bound. Both are known limitations of CGPI (Kim et al., 2022). In both domains with function approximation using USFAs,  $h$ -GPI outperforms the competing methods, which supports H1 and H3. In the FetchPush domain, in particular, SF-MPC is allowed to plan for up to  $h = 15$  steps; that is, three times longer than the maximum horizon we allow  $h$ -GPI to plan. Again,  $h$ -GPI consistently outperforms it (and other baselines) even when allowed to plan for significantly fewer steps. Interestingly, in the FetchPush domain,  $h$ -GPI's performance slightly decreases for intermediate values of  $h$ , which suggests that its policy is being affected by model approximation errors. Even so, the mean returns achieved, for all values of  $h$  are higher than the ones achieved by GPI. This demonstrates the robustness of  $h$ -GPI to approximate models of the environment and function approximation errors.
5 Discussion and related work
We now discuss the works most closely related to  $h$ -GPI. Further discussions can be found in Appendix C.
GPI and SFs. Previous works have extended GPI to safe RL (Gimelfarb et al., 2021; Feng et al., 2023), maximum-entropy RL (Hunt et al., 2019), unsupervised RL (Hansen et al., 2020), and hierarchical RL (e.g., via the options framework) (Barreto et al., 2019; Machado et al., 2023). Recently, Thakoor et al. (2022) introduced Geometric GPI (GGPI). GGPI uses geometric horizon models (GHMs) to learn the discounted future state-visitation distribution induced by particular policies to rapidly
evaluate policies under a given, known reward function. The authors show that performing GPI over a particular type of non-stationary policy produces behaviors that outperform those in the agent's library policy.  $h$ -GPI, by contrast, learns a different type of model: an environment model, which is used to perform planning—i.e., action selection—rather than policy evaluation. Additionally, GGPI requires learning separate GHMs for each policy in the library, whereas  $h$ -GPI can operate with a single model that predicts the next reward features, alongside a single USFA. Secondly, GGPI assumes that the reward function is known a priori, while we exploit SFs to generalize over all linear rewards given reward features. Bagot et al. (2023) introduced GPI-Tree Search (GPI-TS), which is closely related to  $h$ -GPI. GPI-TS uses GPI bootstrapping as backup value estimates at the leaf nodes of Monte Carlo tree search. However, GPI-TS does not employ SFs and was only designed to tackle single-task settings. Moreover, it assumes an oracle model of the environment. We, by contrast, exploit learned models to perform zero-shot transfer over multiple tasks and show how approximation errors in the model affect the performance of the  $h$ -GPI policy (see Theorems 1 and 2). Kim et al. (2022) introduced Constrained GPI (CGPI), which uses lower and upper bounds of the value of a new task to constrain the action-value estimates used when selecting the GPI policy's action. Although CGPI is able to deal with generalization errors in the values given by USFAs for target tasks, it is sensitive to errors in the value estimates for the training tasks.  $h$ -GPI, by contrast, can deal with approximation errors in the value functions of the source/base policies. Other works have studied methods for constructing a policy library for use with GPI (Zahavy et al., 2021; Nemecek and Parr, 2021; Alver and Precup, 2022; Alegre et al., 2022a) and for learning different SF-based representations (Lehnert and Littman, 2020; Touati and Ollivier, 2021). These methods solve important orthogonal problems and could potentially be combined with  $h$ -GPI.
Multi-step RL algorithms. Multi-step RL methods were extensively studied for policy evaluation, both in the model-free (Hessel et al., 2018; van Hasselt et al., 2018) and the model-based settings (Yao et al., 2009; Janner et al., 2019). Model value expansion algorithms (Feinberg et al., 2018; Buckman et al., 2018; Abbas et al., 2020) are a significant example of the latter. In this work, by contrast, we introduce a multi-step method for policy improvement in transfer learning settings. GX-Chen et al. (2022) introduced the  $\eta$ -return mixture, a new backup target for better credit assignment, which combines bootstrapping with standard value estimates and SFs, as a function of a parameter  $\eta$ . Efroni et al. (2018) studied multi-step greedy versions of the well-known dynamic programming (DP) policy iteration and value iteration algorithms (Bertsekas and Tsitsiklis, 1996), and Efroni et al. (2020) proposed a multi-step greedy real-time DP algorithm that replaces 1-step greedy policies with a  $h$ -step lookahead policy. Our work is also related to the techniques introduced by Sikchi et al. (2022) and Hansen et al. (2022), which combine planning and bootstrapping with learned value estimates. However, unlike  $h$ -GPI, these methods do not address the multi-policy and zero-shot transfer settings. We also note that there exists neuroscientific evidence that multi-step planning occurs in the brains of humans and other animals (Miller and Venditto, 2021). Finally, we believe that our work also shares similarities with the investigation performed by Tomov et al. (2021) on how model-free and model-based mechanisms are combined in human behavior.
We introduced  $h$ -GPI, a multi-step extension of GPI that interpolates between standard model-free GPI and fully model-based planning. Through novel theoretical and empirical results, we showed that  $h$ -GPI effectively exploits approximate models to solve novel tasks in a zero-shot manner. Notably, in our experiments, and consistent with the introduced theorems, we showed that  $h$ -GPI is less susceptible to value approximation errors and that it outperforms standard GPI and state-of-the-art competing baselines. Our findings hold even in high-dimensional problems where imperfect learned models are used. These results, combined with our method's strong formal guarantees, indicate that  $h$ -GPI is an important first step towards bridging the gap between model-free GPI-based methods and model-based planning algorithms—while also being robust to approximation errors.
Limitations and future work. The main limitation of  $h$ -GPI is that it can introduce computational overhead for large values of  $h$ . In future work, this can be tackled by designing heuristics for use when performing online planning, or via principled methods to dynamically select the best value of  $h$ . Another interesting direction is to integrate uncertainty estimation techniques (e.g., implicit value ensemble (Filos et al., 2022)) into the  $h$ -GPI policy to further reduce the impact of high model or value function approximation errors. Finally, regarding potential direct negative societal impacts of this work, we do not anticipate any.
This study was financed in part by the Coordenação de Aperfeçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001; CNPq (Grants 140500/2021-9, 304932/2021-3); FAPESP/MCTI/CGI (Grant 2020/05165-1); the Research Foundation Flanders (FWO) [G062819N]; the AI Research Program from the Flemish Government (Belgium); and the Francqui Foundation.
A Proofs of theoretical results
We start by defining the Bellman optimality operator,  $\mathcal{T}^*$ , and its multi-step variant, when applied to action-value functions  $q \in \mathcal{Q}$ , where  $\mathcal{Q}$  is the space of action-value functions. To simplify notation, given the (true) model,  $m \triangleq (p,r)$ , of the underlying MDP, we denote  $\mathbb{E}_m[\cdot]$  as the expectation operator with respect to  $S_{t+1} \sim p(.|S_t, A_t)$  and  $R_t \triangleq r(S_t, A_t, S_{t+1})$ . Similarly, we write  $r(s,a) \triangleq \mathbb{E}_{S_{t+1} \sim p(.|s,a)}[r(s,a, S_{t+1})]$  for brevity.
Definition (2). (Single- and multi-step Bellman optimality operators). Given an MDP's model  $m = (p,r)$ , the single-step Bellman optimality operator,  $\mathcal{T}^* : \mathcal{Q} \mapsto \mathcal{Q}$ , is defined as:
$$
\mathcal {T} ^ {*} q (s, a) \triangleq \mathbb {E} _ {m} [ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(S _ {t + 1}, a ^ {\prime}\right) | S _ {t} = s, A _ {t} = a ]. \tag {18}
$$
The repeated application of  $\mathcal{T}^*$  for  $h$  steps gives rise to the  $h$ -step Bellman operator, denoted as  $(\mathcal{T}^{*})^{h}q(s,a)\triangleq \underbrace{\mathcal{T}^{*}\cdots\mathcal{T}^{*}}_{h\text{-times}}q(s,a)$ . Efroni et al. (2018) showed that  $(\mathcal{T}^{*})^{h}$  can be written as follows:
$$
\begin{array}{l} \left(\mathcal {T} ^ {*}\right) ^ {h} q (s, a) \triangleq \mathbb {E} _ {m} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} \left(\mathcal {T} ^ {*}\right) ^ {h - 1} q \left(S _ {t + 1}, a ^ {\prime}\right) \mid S _ {t} = s, A _ {t} = a \right] (19) \\ = \max  _ {\mu_ {1} \dots \mu_ {h - 1}} \mathbb {E} _ {m} \left[ \sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k}, \mu_ {k} \left(S _ {t + k}\right)\right) + \gamma^ {h} \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(S _ {t + h}, a ^ {\prime}\right) \mid S _ {t} = s, \mu_ {0} (s) = a \right], (20) \\ \end{array}
$$
Definition 4. (Bellman evaluation operator). Given an MDP's model  $m = (p, r)$  and a policy  $\pi$ , the single-step Bellman evaluation operator,  $T^{\pi} : \mathcal{Q} \mapsto \mathcal{Q}$ , is defined as: