SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lucas N. Alegre $^{1,2}$  Ana L. C. Bazzan $^{1}$  Ann Nowé $^{2}$  Bruno C. da Silva $^{3}$
$^{1}$ Institute of Informatics, Federal University of Rio Grande do Sul
$^{2}$ Artificial Intelligence Lab, Vrije Universiteit Brussel  $^{3}$ University of Massachusetts
{lnalegre,bazzan}@inf.ufrgs.br ann-nowe  $\text{已}$  vub.be bsilva@cs.umass.edu
We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies—each solving a particular task—and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce  $h$ -GPI, a multi-step extension of GPI that interpolates between these extremes—standard model-free GPI and fully model-based planning—as a function of a parameter,  $h$ , regulating the amount of time the agent has to reason. We prove that  $h$ -GPI's performance lower bound is strictly better than GPI's, and show that  $h$ -GPI generally outperforms GPI as  $h$  increases. Furthermore, we prove that as  $h$  increases,  $h$ -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by  $h$ -GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate  $h$ -GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.
Reinforcement learning (RL) (Sutton and Barto, 2018) algorithms have achieved remarkable performance in challenging tasks both in the model-free (Bellemare et al., 2020; Wurman et al., 2022) and the model-based (Bryant et al., 2022; Wu et al., 2022) settings. In these problems, agents are typically trained to optimize one particular reward function (a task). However, designing agents that can adapt their decision-making policies to solve novel tasks in a zero-shot manner—i.e., without requiring any further learning—remains an important open problem.
Various principled and efficient policy transfer methods, which combine previously-acquired policies to solve new tasks in a zero-shot manner, have been proposed based on the combination of successor features (SFs) and generalized policy improvement (GPI) (Barreto et al., 2017; Borsa et al., 2019; Barreto et al., 2020; Kim et al., 2022). On the one hand, SFs allow agents to efficiently evaluate the performance of given policies in arbitrary sets of tasks. On the other hand, GPI extends the classic policy improvement procedure (Sutton and Barto, 2018) by analyzing a library of previously-learned
policies—each solving a particular task—and identifying a novel policy that simultaneously improves upon all policies in the agent's library.
We introduce  $h$ -GPI, a multi-step extension of GPI that interpolates between these extremes—standard model-free GPI and fully model-based planning—as a function of a parameter,  $h$ , regulating the amount of time the agent has to reason. An  $h$ -GPI policy is computed by performing online planning for  $h$  steps, using an approximate model, and then employing GPI on all states reachable in  $h$  steps to identify (in a zero-shot manner) which actions the agent should take thereafter. See Figure 1 for an example.
Our work is partially motivated by the observation that state-of-the-art model-based RL algorithms (Hafner et al., 2019, 2021) often adapt
Figure 1:  $h$ -GPI combines online planning with GPI leveraging a learned model.
poorly to local changes in the environment (Wan et al., 2022). Furthermore, environment models may have been learned and thus might be inaccurate. These observations underscore the risks involved in relying solely on approximate models for long-horizon planning scenarios, as model errors compound and may lead to catastrophic outcomes. Unlike techniques for dealing with approximate models, our method additionally exploits GPI's capabilities of efficiently performing zero-shot transfer in a model-free manner. Other approaches—similar in nature to  $h$ -GPI—make use of models and GPI-based techniques. Thakoor et al. (2022), for example, introduced GGPI, a method that learns state visitation models induced by particular policies to rapidly evaluate policies under a given, known reward function. The authors show that performing GPI over a particular type of non-stationary policy produces behaviors that outperform those in the agent's library policy.  $h$ -GPI, by contrast, learns a different type of model: an environment model, which is used to perform planning—i.e., action selection—rather than policy evaluation. Moreover,  $h$ -GPI exploits GPI to perform bootstrapping from all states reachable in  $h$  steps. In Section 5, we contrast our method with other related techniques, highlighting the unique aspects of our approach.
In this paper, we introduce the first principled method capable of exploiting approximate models and producing a multi-step extension of GPI with formal guarantees. We prove that  $h$ -GPI's performance lower bound is strictly better than GPI's, and show that  $h$ -GPI generally outperforms GPI as  $h$  increases. Furthermore, we prove that as  $h$  increases,  $h$ -GPI's performance becomes arbitrarily less susceptible to value function approximation errors in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by  $h$ -GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. We empirically evaluate  $h$ -GPI in challenging tabular and continuous-state problems with value function approximation. Our findings show that it consistently outperforms both GPI and other state-of-the-art methods under various levels of approximation errors. These results, combined with our method's formal guarantees, indicate that  $h$ -GPI is an important first step towards bridging the gap between model-free GPI-based methods and fully model-based planning algorithms—while being robust to approximate models of the environment and value function estimation errors.
Before introducing our contributions, we review key concepts and definitions related to model-free and model-based RL, SFs, and GPI.
2.1 Reinforcement learning
RL problems (Sutton and Barto, 2018) are typically modeled as Markov decision processes (MDPs). An MDP is a tuple  $M \triangleq (S, A, p, r, \mu, \gamma)$ , where  $S$  is a state space,  $A$  is an action space,  $p(\cdot | s, a)$  denotes the distribution over next states conditioned on a state and action,  $r: S \times A \times S \mapsto \mathbb{R}$  is a reward function,  $\mu$  is an initial state distribution, and  $\gamma \in [0, 1)$  is a discount factor. Let  $S_{t}, A_{t}$ , and  $R_{t} = r(S_{t}, A_{t}, S_{t+1})$  be random variables corresponding to the state, action, and reward, respectively,
at time step  $t \geq 0$ . The goal of an RL agent is to learn a policy  $\pi : S \mapsto \mathcal{A}$  that maximizes the expected discounted sum of rewards (return)  $\sum_{t=0}^{\infty} \gamma^{t} R_{t}$ . The action-value function of a policy  $\pi$  is  $q^{\pi}(s, a) \triangleq \mathbb{E}_{\pi}[\sum_{i=0}^{\infty} \gamma^{i} R_{t+i}|S_{t} = s, A_{t} = a]$ , where  $\mathbb{E}_{\pi}[\cdot]$  denotes the expectation over trajectories induced by  $\pi$  and  $p$ . Given  $q^{\pi}$ , a greedy policy can be defined as  $\pi'(s) \in \arg \max_{a} q^{\pi}(s, a)$ . It is guaranteed that  $q^{\pi'}(s, a) \geq q^{\pi}(s, a)$ , for all  $(s, a) \in S \times \mathcal{A}$ . Computing  $q^{\pi}$  and  $\pi'$  is done by processes known, respectively, as policy evaluation and policy improvement. Repeatedly alternating between policy evaluation and policy improvement steps is known to lead to an optimal policy,  $\pi^{*}(s)$  which maximizes the expected return from all states  $s \in S$  (Puterman, 2014).
2.2 Successor features and GPI
The successor features (SFs) framework (Barreto et al., 2017) allows agents to efficiently evaluate the performance of a given policy when deployed under any linearly-representable reward functions,  $r_{\mathbf{w}}(s,a,s^{\prime}) = \phi (s,a,s^{\prime})\cdot \mathbf{w}$ , where  $\phi (s,a,s^{\prime})\in \mathbb{R}^{d}$  are reward features and  $\mathbf{w}\in \mathbb{R}^{d}$  are weights. A controlled Markov process  $(S,\mathcal{A},p,\mu ,\gamma)$ , i.e., an MDP without a reward function, when combined with reward features  $\phi :\mathcal{S}\times \mathcal{A}\times \mathcal{S}\mapsto \mathbb{R}^{d}$ , induces a family of MDPs:
$$
\mathcal {M} ^ {\phi} \triangleq \{M = (\mathcal {S}, \mathcal {A}, p, r _ {\mathbf {w}}, \mu , \gamma) \mid r _ {\mathbf {w}} (s, a, s ^ {\prime}) = \phi (s, a, s ^ {\prime}) \cdot \mathbf {w} \}. \tag {1}
$$
Notice that the family of MDPs  $\mathcal{M}^{\phi}$  represents all possible tasks (each defined by a reward function) that can be defined in the environment associated with the corresponding controlled Markov process. Given a policy  $\pi$ , its corresponding SFs,  $\psi^{\pi}(s,a)\in \mathbb{R}^{d}$ , for a state-action pair  $(s,a)$  are defined as
$$
\psi^ {\pi} (s, a) \triangleq \mathbb {E} _ {\pi} \left[ \sum_ {i = 0} ^ {\infty} \gamma^ {i} \phi_ {t + i} \mid S _ {t} = s, A _ {t} = a \right], \tag {2}
$$
where  $\phi_t \triangleq \phi(S_t, A_t, S_{t+1})$ . Importantly, notice that given the SFs  $\psi^\pi(s, a)$  of a policy  $\pi$ , it is possible to directly compute the action-value function  $q_{\mathbf{w}}^\pi(s, a)$  of  $\pi$ , under any linearly-expressible reward functions,  $r_{\mathbf{w}}$ , as follows:
$$
q _ {\mathbf {w}} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {i = 0} ^ {\infty} \gamma^ {i} \phi_ {t + i} \cdot \mathbf {w} \mid S _ {t} = s, A _ {t} = a \right] = \psi^ {\pi} (s, a) \cdot \mathbf {w}. \tag {3}
$$
The equation described above—which uses SFs to evaluate a given policy under different reward functions—represents a process known as generalized policy evaluation (GPE) (Barreto et al., 2020). Notice that this process can be extended to the case where an agent has access to a set of previously learned policies,  $\Pi = \{\pi_i\}_{i=1}^n$ , and their corresponding SFs,  $\Psi = \{\psi^{\pi_i}\}_{i=1}^n$ . Then, given any  $\pi_i \in \Pi$ , GPE can be used to efficiently evaluate  $\pi_i$  under any arbitrary reward function of interest,  $r_{\mathbf{w}}$ , via Equation (3):  $q_{\mathbf{w}}^{\pi_i}(s,a) = \psi^{\pi_i}(s,a) \cdot \mathbf{w}$ .
Generalized policy improvement. GPI generalizes the standard policy improvement process (discussed in Section 2.1). It assumes access to a set of policies (and corresponding action-value functions) and uses it to directly identify a novel policy optimizing a particular reward function,  $r_{\mathbf{w}}$ . Importantly, the novel policy is guaranteed to outperform all original policies the agent had access to.
Definition 1. (Barreto et al., 2020) Given a set of policies  $\Pi$  and a reward function  $r_{\mathbf{w}}$ , generalized policy improvement (GPI) is the process by which a policy,  $\pi^{\prime}$ , is identified such that
$$
q _ {\mathbf {w}} ^ {\pi^ {\prime}} (s, a) \geq \max  _ {\pi \in \Pi} q _ {\mathbf {w}} ^ {\pi} (s, a) f o r a l l (s, a) \in \mathcal {S} \times \mathcal {A}. \tag {4}
$$
Based on Equation (4) and the reward decomposition  $r_{\mathbf{w}}(s,a,s^{\prime}) = \phi (s,a,s^{\prime})\cdot \mathbf{w}$ , a generalized policy,  $\pi :\mathcal{S}\times \mathcal{W}\mapsto \mathcal{A}$ , can then be defined as follows:
$$
\pi^ {\mathrm {G P I}} (s; \mathbf {w}) \in \underset {a \in A} {\arg \max } \underset {\pi \in \Pi} {\max } q _ {\mathbf {w}} ^ {\pi} (s, a). \tag {5}
$$
Let  $q_{\mathbf{w}}^{\mathrm{GPI}}(s, a)$  be the action-value function of policy  $\pi^{\mathrm{GPI}}(\cdot; \mathbf{w})$ . The GPI theorem (Barreto et al., 2017) shows that  $\pi^{\mathrm{GPI}}(\cdot; \mathbf{w})$  satisfies Definition 1, and so Equation (5) can be used to identify a policy guaranteed to perform at least as well as any other policies  $\pi_i \in \Pi$ , when tackling a new task,  $\mathbf{w}$ . This theorem can be extended to cases where the agent only has access to an approximation of the action-value function,  $\hat{q}^{\pi_i}$ , of a policy  $\pi_i$  (Barreto et al., 2018).
1Notice that the definition of SFs satisfies a form of the Bellman equation, where the features  $\phi_t$  play the role of rewards. Thus, SFs can be learned through standard temporal-difference learning algorithms.
2.3 Model-based RL
In model-free RL, agents do not have access to the environment state transition and reward functions,  $p$  and  $r$ , and must, instead, learn a policy solely based on samples obtained while interacting with the environment. Given an MDP, we denote  $m \triangleq (p, r)$  as its model. In model-based RL, an agent is given (or has to learn) an approximate model,  $\hat{m} \approx m$ . Agents may exploit models in different ways; for example, (i) to perform background planning (i.e., generating simulated experiences to more rapidly learn a policy or value function, as in Dyna-style algorithms (Van Seijen and Sutton, 2013; Janner et al., 2019; Pan et al., 2019)); and (ii) to perform online planning (Barto et al., 1995; Chua et al., 2018; Efroni et al., 2020; Hansen et al., 2022).
In online planning methods, agents use a model to estimate the value of states by simulating the outcomes of a particular sequence of actions. This is also known as "unrolling the model forwards", from the current state, for a given number of time steps. Typically, online planning procedures are performed from every state the agent encounters, as in model-predictive control (MPC) algorithms (Camacho and Alba, 2013; Chua et al., 2018). Importantly, recent work—such as that of Wan et al. (2022)—has shown that state-of-the-art model-based RL algorithms (Hafner et al., 2019, 2021) often fail catastrophically when deployed in settings where the environment (e.g., the MDP's reward function) may change. This observation underscores the risks involved in relying solely on approximate models for long-horizon planning scenarios, since it is well-known that model errors compound.
We address these limitations by introducing a zero-shot transfer technique based on online planning. Our technique interpolates between standard model-free GPI and fully model-based planning according to a parameter,  $h$ , regulating the amount of time the agent has to reason using an approximate model. Unlike previous methods that require re-learning or adapting models to tackle novel tasks, our technique exploits approximate models and SFs to solve any tasks (defined as linearly-expressible reward functions) without requiring any further learning. Importantly, it has strong theoretical guarantees regarding its performance lower bound under various types of approximation errors.
3 Multi-step generalized policy improvement
We now introduce our main contribution,  $h$ -GPI. This is a multi-step extension of GPI that combines online planning with approximate environment models to efficiently perform zero-shot transfer.
We start by defining the Bellman optimality operator  $\mathcal{T}^*$  (and its multi-step variant) applied to action-value functions  $q\in \mathcal{Q}$ , where  $\mathcal{Q}$  is the space of action-value functions. To simplify notation, given a model  $m\triangleq (p,r)$ , we denote  $\mathbb{E}_m[\cdot ]$  as the expectation operator with respect to  $S_{t + 1}\sim p(|S_t,A_t)$  and  $R_{t}\triangleq r(S_{t},A_{t},S_{t + 1})$ . Similarly, we write  $r(s,a)\triangleq \mathbb{E}_{S_{t + 1}\sim p(\cdot |s,a)}[r(s,a,S_{t + 1})]$  for brevity.
Definition 2. (Single- and multi-step Bellman optimality operators). Given a model  $m = (p, r)$ , the single-step Bellman optimality operator,  $\mathcal{T}^* : \mathcal{Q} \mapsto \mathcal{Q}$ , is defined as:
$$
\mathcal {T} ^ {*} q (s, a) \triangleq \mathbb {E} _ {m} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(S _ {t + 1}, a ^ {\prime}\right) \mid S _ {t} = s, A _ {t} = a \right]. \tag {6}
$$
The repeated application of  $\mathcal{T}^*$  for  $h$  steps gives rise to the  $h$ -step Bellman operator, denoted  $(\mathcal{T}^{*})^{h}q(s,a)\triangleq \underline{\mathcal{T}^{*}\cdots\mathcal{T}^{*}}_{h\text{-times}}q(s,a)$ . Efroni et al. (2018) showed that  $(\mathcal{T}^{*})^{h}$  can be written as follows:
$$
\begin{array}{l} \left(\mathcal {T} ^ {*}\right) ^ {h} q (s, a) \triangleq \mathbb {E} _ {m} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} \left(\mathcal {T} ^ {*}\right) ^ {h - 1} q \left(S _ {t + 1}, a ^ {\prime}\right) \mid S _ {t} = s, A _ {t} = a \right] (7) \\ = \max  _ {\mu_ {1} \dots \mu_ {h - 1}} \mathbb {E} _ {m} \left[ \sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k}, \mu_ {k} \left(S _ {t + k}\right)\right) + \gamma^ {h} \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(S _ {t + h}, a ^ {\prime}\right) \mid S _ {t} = s, \mu_ {0} (s) = a \right], (8) \\ \end{array}
$$
where  $\mu_{i}$  is any policy (in an arbitrary policy space) the agent could choose to deploy at time  $i$ .
It is well known that  $\mathcal{T}^*$  is a contraction mapping and its fixed-point (for any initial  $q\in \mathcal{Q}$ ) is  $\lim_{h\to \infty}(\mathcal{T}^{*})^{h}q(s,a) = q^{*}(s,a)$  (Puterman, 2014).
We now introduce our first contribution:  $h$ -step generalized policy improvement, or  $h$ -GPI:
Definition 3. Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies and  $m = (p, r)$  be a model. Then, given a horizon  $h \geq 0$ , the  $h$ -GPI policy on state  $s$  is defined as
$$
\pi^ {h \cdot G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } q ^ {\pi} (s, a) \tag {9}
$$
$$
= \underset {a \in \mathcal {A}} {\arg \max } \underset {\mu_ {1}, \dots , \mu_ {h - 1}} {\max } \mathbb {E} _ {m} \left[ \underbrace {\sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k} , \mu_ {k} \left(S _ {t + k}\right)\right)} _ {\text {o n l i n e p l a n n i n g}} + \gamma^ {h} \underbrace {\underset {a ^ {\prime} \in \mathcal {A}} {\max } \underset {\pi \in \Pi} {\max } q ^ {\pi} \left(S _ {t + h} , a ^ {\prime}\right)} _ {G P I} | \mu_ {0} (S _ {t}) = a \right] \tag {10}
$$
Intuitively,  $h$ -GPI identifies a policy that returns the best possible action,  $a$ , by first planning with model  $m$  for  $h$  steps and then using GPI to estimate the future returns achievable from all states reachable in  $h$  steps. This is in contrast with standard GPI policies, which can only reason about the future returns achievable from the current state,  $S_{t}$ , if following a given policy in II.  $h$ -GPI, by contrast, uses a model to reason over the decisions made at states within  $h$  steps from  $s$ , as well as states,  $S_{t + h}$ , reachable after  $h$  steps. This makes it possible for  $h$ -GPI to produce policies that exploit the model-free, zero-shot return estimates produced by GPI when evaluating states beyond a given horizon  $h$ ; in particular, states that would otherwise not be considered by the standard GPI procedure when determining return-maximizing actions. Notice, furthermore, that  $h$ -GPI by construction interpolates between model-free GPI and fully model-based planning:
$h$ -GPI induces policies that interpolate between  $(i)$  standard model-free GPI (when  $h = 0$ ) and  $(ii)$  fully model-based planning (when  $h \to \infty$ ).
In practice, agents seldom have access to the true model,  $m$ , of an MDP, and instead rely on approximate models,  $\hat{m}$ . The application of the Bellman operator under an approximate model,  $\hat{m}$ , rather than the true MDP model,  $m$ , can be represented mathematically by replacing  $m$  with  $\hat{m}$  in Equation (6). This gives rise to a model-dependent Bellman optimality operator, which we call  $T_{\hat{m}}^{*}$ . In what follows, we introduce Theorem 1, which extends the original GPI theorem (Barreto et al., 2017) and allows us to characterize the gains achievable by  $h$ -GPI as a function of approximation errors in both the agent's policy library (II) and its (possibly learned) model  $\hat{m}$ . Notice that Theorem 1 generalizes the original GPI theorem, which is recovered when  $h = 0$ .
Theorem 1. Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies,  $\{\hat{q}^{\pi_i}\}_{i=1}^n$  be approximations of their respective action-value functions, and  $\hat{m} = (\hat{p}, \hat{r})$  be an approximate model such that, for all  $\pi_i \in \Pi$  and all  $(s, a) \in S \times \mathcal{A}$ ,
$$
\left| q ^ {\pi_ {i}} (s, a) - \hat {q} ^ {\pi_ {i}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} \left| p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) \right| \leq \epsilon_ {p}, \text {a n d} | r (s, a) - \hat {r} (s, a) | \leq \epsilon_ {r}. \tag {11}
$$
Recall once again the definition of  $h$ -GPI (Definition 3):
$$
\pi^ {h \cdot G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} ^ {\pi} (s, a). \tag {12}
$$
$$
\begin{array}{l} q ^ {h - G P I} (s, a) \geq \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (13) \\ \geq \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right), (14) \\ \end{array}
$$
where  $c(\epsilon_r, \epsilon_p, h) = \frac{1 - \gamma^h}{1 - \gamma} (\epsilon_r + \gamma \epsilon_p v_{max}^*)$ , and  $v_{max}^* \triangleq |\max_{s,a} q^*(s,a)|$ .
Theorem 1 characterizes the performance lower bound of  $h$ -GPI as a function of the number of planning steps,  $h$ , and the approximation errors in the agent's model and action-value functions (i.e., errors  $\epsilon$ ,  $\epsilon_p$ , and  $\epsilon_r$ ).
Note that since  $(\mathcal{T}^{*})^{h}\max_{\pi \in \Pi}q^{\pi}(s,a)\geq \max_{\pi \in \Pi}q^{\pi}(s,a)$ ,  $h$ -GPI's performance lower bound is strictly better than GPI's ( $h = 0$ ) assuming no model approximation errors ( $\epsilon_r = \epsilon_p = 0$ ). Furthermore, notice that as  $h\to \infty$ ,  $(\mathcal{T}^{*})^{h}\max_{\pi \in \Pi}q^{\pi}(s,a)$  converges to  $q^{*}$  and the approximation error term  $(\gamma^{h}\epsilon)$  in Equation (13) disappears. In other words, as  $h$  increases,  $h$ -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. This implies that the planning horizon  $h$  trades off between two conflicting objectives. On the one hand, increasing the planning horizon  $h$  improves the performance of the  $h$ -GPI policy because the error term ( $\epsilon$ ) associated with value function approximations errors becomes irrelevant. On the other hand, longer horizons worsen  $h$ -GPI's performance lower bound since they increase its dependency on errors arising from approximate models. Intuitively, for small values of  $h$ , the agent relies more heavily on the assumption that its estimates  $\{\hat{q}^{\pi_i}\}_{i = 1}^n$  are correct, given that these estimates are discounted by  $\gamma^{h}$ . For large values of  $h$ , by contrast, the approximate model error term,  $c(\epsilon_r,\epsilon_p,h)$ , increases, and the agent relies more heavily on the assumption that its (learned) model is approximately correct.