SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {T} ^ {\pi} q (s, a) \triangleq \mathbb {E} _ {m} [ r (s, a) + \gamma q \left(S _ {t + 1}, \pi \left(S _ {t + 1}\right)\right) \mid S _ {t} = s, A _ {t} = a ]. \tag {21}
$$
It is well known that  $\mathcal{T}^*$  and  $\mathcal{T}^\pi$  are contraction mappings and their fixed-point (for any initial  $q\in \mathcal{Q}$ ) are  $\lim_{h\to \infty}(\mathcal{T}^{*})^{h}q(s,a) = q^{*}(s,a)$  and  $\lim_{h\to \infty}(\mathcal{T}^{\pi})^{h}q(s,a) = q^{\pi}(s,a)$ , respectively (Puterman, 2014).
Notice that in the definitions above, since  $m$  is the (true) model of a given MDP, we chose to omit  $m$  when writing the corresponding Bellman operators to make the notation more succinct. Formally, though, one could alternatively denote these operators as  $\mathcal{T}_m^*$  and  $\mathcal{T}_m^\pi$ , respectively. In what follows, we always omit  $m$  whenever the corresponding operators are defined with respect to the true model of an MDP.
The Bellman optimality operator induced by an approximate model,  $\hat{m} = (\hat{p},\hat{r})$ , of the true underlying transition and reward functions,  $p$  and  $r$ , of a given MDP, is defined as:
$$
\begin{array}{l} \mathcal {T} _ {\hat {m}} ^ {*} q (s, a) \triangleq \mathbb {E} _ {\hat {m}} [ \hat {r} (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(S _ {t + 1}, a ^ {\prime}\right) \mid S _ {t} = s, A _ {t} = a ] (22) \\ = \hat {r} (s, a) + \gamma \sum_ {s ^ {\prime}} \hat {p} \left(s ^ {\prime} \mid s, a\right) \max  _ {a ^ {\prime} \in \mathcal {A}} q \left(s ^ {\prime}, a ^ {\prime}\right). (23) \\ \end{array}
$$
Now, recall our definition of  $h$ -step generalized policy improvement, or  $h$ -GPI:
Definition (3). Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies and  $m = (p, r)$  be a model. Then, given a horizon  $h \geq 0$ , the  $h$ -GPI policy on state  $s$  is defined as
$$
\begin{array}{l} \pi^ {h \cdot G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {m} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } q ^ {\pi} (s, a) (24) \\ = \arg \max  _ {a \in \mathcal {A}} \max  _ {\mu_ {1}, \dots , \mu_ {h - 1}} \mathbb {E} _ {m} \left[ \underbrace {\sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k} , \mu_ {k} \left(S _ {t + k}\right)\right)} _ {\text {o n l i n e p l a n n i n g}} + \gamma^ {h} \underbrace {\max  _ {a ^ {\prime} \in \mathcal {A}} \max  _ {\pi \in \Pi} q ^ {\pi} \left(S _ {t + h} , a ^ {\prime}\right)} _ {G P I} \mid \mu_ {0} \left(S _ {t}\right) = a \right] (25) \\ \end{array}
$$
Intuitively, an  $h$ -GPI policy identifies the best possible action,  $A_{t}$ , when in state  $S_{t}$ , by first planning with model  $m$  for  $h$  steps and then using GPI to estimate the future returns achievable from all states reachable in  $h$  steps. This is in contrast with standard GPI policies, which can only reason about the future returns achievable from the current state,  $S_{t}$ , if following a given policy in II.  $h$ -GPI, by contrast, uses a model to reason over the decisions made at states within  $h$  steps from  $S_{t}$ , as well as states  $S_{t + h}$  reachable after  $h$  steps. This makes it possible for  $h$ -GPI to produce policies that exploit the model-free, zero-shot return estimates produced by GPI when evaluating states beyond a given horizon  $h$ ; in particular, states that would otherwise not be considered by the standard GPI procedure when determining return-maximizing actions. Notice, furthermore, that  $h$ -GPI—by construction—interpolates between model-free GPI and fully model-based planning:
Next, we introduce a few lemmas that will subsequently allow us to  $(i)$  precisely characterize the performance lower bound of an  $h$ -GPI policy;  $(ii)$  show that it is strictly better than GPI's performance lower bound;  $(iii)$  prove that as  $h$  increases,  $h$ -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library.
The following lemma characterizes the loss introduced by applying the multi-step Bellman optimality operator,  $(\mathcal{T}^{*})^{h}$ , to an approximate action-value function,  $\hat{q}^{\pi}(s,a)$ .
Lemma 1. Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies, and  $\{\hat{q}^{\pi_i}\}_{i=1}^n$  be approximations to their respective action-value functions such that, for all  $\pi_i \in \Pi$  and for all  $(s,a) \in S \times \mathcal{A}$ ,
$$
\left| q ^ {\pi_ {i}} (s, a) - \hat {q} ^ {\pi_ {i}} (s, a) \right| \leq \epsilon . \tag {26}
$$
$$
\left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) \geq \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma^ {h} \epsilon . \tag {27}
$$
$$
\begin{array}{l} \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) = \max  _ {\mu_ {1} \dots \mu_ {h - 1}} \mathbb {E} \left[ \sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k}, \mu_ {k} \left(S _ {t + k}\right)\right) + \gamma^ {h} \max  _ {a ^ {\prime} \in \mathcal {A}} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} \left(S _ {t + h}, a ^ {\prime}\right) | S _ {t} = s, \mu_ {0} (s) = a \right] (28) \\ \geq \max  _ {\mu_ {1} \dots \mu_ {h - 1}} \mathbb {E} \left[ \sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k}, \mu_ {k} \left(S _ {t + k}\right)\right) + \gamma^ {h} \max  _ {a ^ {\prime} \in \mathcal {A}} \max  _ {\pi \in \Pi} \left(q ^ {\pi} \left(S _ {t + h}, a ^ {\prime}\right) - \epsilon\right) | S _ {t} = s, \mu_ {0} (s) = a \right] (29) \\ = \max  _ {\mu_ {1} \dots \mu_ {h - 1}} \mathbb {E} \left[ \sum_ {k = 0} ^ {h - 1} \gamma^ {k} r \left(S _ {t + k}, \mu_ {k} \left(S _ {t + k}\right)\right) + \gamma^ {h} \max  _ {a ^ {\prime} \in \mathcal {A}} \max  _ {\pi \in \Pi} q ^ {\pi} \left(S _ {t + h}, a ^ {\prime}\right) | S _ {t} = s, \mu_ {0} (s) = a \right] - \gamma^ {h} \epsilon (30) \\ = \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma^ {h} \epsilon . (31) \\ \end{array}
$$
Above, (28) and (31) are due to the definition of  $(\mathcal{T}^{*})^{h}$  introduced in Equation (20).
The next lemma characterizes the loss incurred by applying the Bellman optimality operator induced by an approximate model  $\hat{m}$ ,  $(\mathcal{T}_{\hat{m}}^{*})^{h}$ , to an action-value function,  $q^{\pi}$ , of a policy  $\pi$ .
Lemma 2. Let  $\hat{m} = (\hat{p},\hat{r})$  be an approximate model such that, for all  $(s,a)\in S\times \mathcal{A}$
$$
\sum_ {s ^ {\prime}} \left| p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) \right| \leq \epsilon_ {p} \quad a n d \quad | r (s, a) - \hat {r} (s, a) | \leq \epsilon_ {r}. \tag {32}
$$
Let  $v_{\max}^* \triangleq |\max_{s,a} q^*(s,a)|$ . Then, for any policy  $\pi$  and  $h \geq 0$ ,
$$
\left| \left(\mathcal {T} ^ {*}\right) ^ {h} q ^ {\pi} (s, a) - \left(\mathcal {T} _ {\tilde {m}} ^ {*}\right) ^ {h} q ^ {\pi} (s, a) \right| \leq c \left(\epsilon_ {r}, \epsilon_ {p}, h\right), \tag {33}
$$
$$
c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) = \frac {1 - \gamma^ {h}}{1 - \gamma} \left(\epsilon_ {r} + \gamma \epsilon_ {p} v _ {\max } ^ {*}\right). \tag {34}
$$
In the proof above, all terms shown in blue refer to approximate quantities—such as an approximate model of the environment or an approximate transition function. In the proof above, (35) follows from the recursive definition of  $(\mathcal{T}^{*})^{h}$  showed in Equation (19). In (37), we add and subtract  $\sum_{s^{\prime}}\hat{p}(s^{\prime}|s,a)\max_{a^{\prime}\in\mathcal{A}}(\mathcal{T}^{*})^{h-1}q^{\pi}(s^{\prime},a^{\prime})$ . (39) is obtained due to the property that  $\sum_{s^{\prime}}\hat{p}(s^{\prime}|s,a)f(s^{\prime}) \leq \max_{s^{\prime}}f(s^{\prime})$ , for any function  $f$ . Finally, (40) is due to  $(\mathcal{T}^{*})^{h-1}q^{\pi}(s^{\prime},a^{\prime}) \leq v_{\max}^{*}$ .
We now show how to rewrite the inequality (40) recursively. Let  $\Delta_h(s,a)\triangleq |(\mathcal{T}^*)^h q^\pi (s,a) - (\mathcal{T}_{\hat{m}}^*)^h q^\pi (s,a)|$ , for all  $(s,a)\in S\times \mathcal{A}$ . Then, by replacing the definition of  $\Delta_h$  in (40), we obtain:
$$
\Delta_ {h} (s, a) \leq \epsilon_ {r} + \gamma \max  _ {s ^ {\prime}, a ^ {\prime}} \Delta_ {h - 1} \left(s ^ {\prime}, a ^ {\prime}\right) + \gamma \epsilon_ {p} v _ {\max } ^ {*}. \tag {41}
$$
Because inequality (41) holds for any  $(s,a)$ , it also holds for the maximizer. Thus, we can omit  $(s,a)$  and define  $\Delta_h\triangleq \max_{s,a}\Delta_h(s,a)$ . The inequality (41) then becomes:
$$
\Delta_ {h} \leq \epsilon_ {r} + \gamma \Delta_ {h - 1} + \gamma \epsilon_ {p} v _ {\max } ^ {*}. \tag {42}
$$
Now we use the fact that, by definition,  $\Delta_0(s,a) = |q^\pi (s,a) - q^\pi (s,a)| = 0$ , to recursively expand inequality (42):
$$
\begin{array}{l} \Delta_ {h} \leq \sum_ {k = 0} ^ {h - 1} \gamma^ {k} \left(\epsilon_ {r} + \gamma \epsilon_ {p} v _ {\max } ^ {*}\right) (43) \\ = \frac {1 - \gamma^ {h}}{1 - \gamma} \left(\epsilon_ {r} + \gamma \epsilon_ {p} v _ {\max } ^ {*}\right). (44) \\ \end{array}
$$
Finally, by using (44) and recalling the definition of  $\Delta_h$  (i.e.,  $\Delta_h \triangleq \max_{s,a} \Delta_h(s,a)$ ), we can directly prove that
$$
\max  _ {s, a} \Delta_ {h} (s, a) \triangleq \max  _ {s, a} \left| \left(\mathcal {T} ^ {*}\right) ^ {h} q ^ {\pi} (s, a) - \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} q ^ {\pi} (s, a) \right| \leq \frac {1 - \gamma^ {h}}{1 - \gamma} \left(\epsilon_ {r} + \gamma \epsilon_ {p} v _ {\max } ^ {*}\right). \tag {45}
$$
The result above bounds the difference between the action-value functions resulting from applying the Bellman optimality operator (defined with respect to the true model of the MDP)  $h$  times, and the action-value function resulting from applying the Bellman optimality operator (defined with respect to an approximate model,  $\hat{m}$ )  $h$  times. This allows us to precisely characterize—when repeatedly using the Bellman optimality operator over an initial action-value function—the maximum error incurred by using an approximate model of the environment.
The next two lemmas provide intermediate results that will be used in the proofs of Theorem 1 and Theorem 2.
Lemma 3. Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies,  $\{\hat{q}^{\pi_i}\}_{i=1}^n$  be approximations to their respective action-value functions, and  $\hat{m} = (\hat{p}, \hat{r})$  be an approximate model such that, for all  $\pi_i \in \Pi$  and for all  $(s, a) \in S \times \mathcal{A}$ , the following holds:
$$
\left| q ^ {\pi_ {i}} (s, a) - \hat {q} ^ {\pi_ {i}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} \left| p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) \right| \leq \epsilon_ {p}, \text {a n d} | r (s, a) - \hat {r} (s, a) | \leq \epsilon_ {r}. \tag {46}
$$
Recall the definition of  $h$ -GPI (Definition 3):
$$
\pi^ {h - G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} ^ {\pi} (s, a). \tag {47}
$$
Then, it follows that:
$$
\mathcal {T} ^ {\pi^ {h \cdot G P I}} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) \geq \left(\mathcal {T} ^ {*}\right) ^ {h + 1} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right). \tag {48}
$$
$$
\begin{array}{l} \mathcal {T} ^ {\pi^ {h - \mathrm {G P I}}} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) = \mathbb {E} \left[ r (s, a) + \gamma \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} \left(S _ {t + 1}, \pi^ {h - \mathrm {G P I}} \left(S _ {t + 1}\right)\right) | S _ {t} = s, A _ {t} = a \right] (49) \\ = \mathbb {E} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} \left(S _ {t + 1}, a ^ {\prime}\right) | S _ {t} = s, A _ {t} = a \right] (50) \\ \geq \mathbb {E} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} \left(\left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} \left(S _ {t + 1}, a ^ {\prime}\right) - \gamma^ {h} \epsilon\right) | S _ {t} = s, A _ {t} = \right. (51) \\ \geq \mathbb {E} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} \left(\left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} \left(S _ {t + 1}, a ^ {\prime}\right) - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) - \gamma^ {h} \epsilon\right) | S _ {t} = s, A _ {t} = a \right] (52) \\ \geq \mathbb {E} \left[ r (s, a) + \gamma \max  _ {a ^ {\prime} \in A} \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} \left(S _ {t + 1}, a ^ {\prime}\right) \mid S _ {t} = s, A _ {t} = a \right] - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (53) \\ = \left(\mathcal {T} ^ {*}\right) ^ {h + 1} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right). (54) \\ \end{array}
$$
Above, (49) is due to the definition of the Bellman evaluation operator (21), and (50) is due to the definition of the  $h$ -GPI policy (47). Finally, (51) is due to Lemma 1 and (52) follows from the use of Lemma 2.
Lemma 4. Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies,  $\{\hat{q}^{\pi_i}\}_{i=1}^n$  be approximations to their respective action-value functions, and  $\hat{m} = (\hat{p}, \hat{r})$  be an approximate model such that, for all  $\pi_i \in \Pi$ , and for all  $(s, a) \in S \times \mathcal{A}$ :
$$
\left| q ^ {\pi_ {i}} (s, a) - \hat {q} ^ {\pi_ {i}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} \left| p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) \right| \leq \epsilon_ {p}, \text {a n d} | r (s, a) - \hat {r} (s, a) | \leq \epsilon_ {r}. \tag {55}
$$
$$
\pi^ {h - G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} ^ {\pi} (s, a). \tag {56}
$$
$$
\mathcal {T} ^ {\pi^ {h \cdot G P I}} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) \geq \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) - \gamma^ {h} \epsilon (1 + \gamma) - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) (1 + \gamma). \tag {57}
$$
$$
\begin{array}{l} \mathcal {T} ^ {\pi^ {h - \mathrm {G P I}}} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) \geq \left(\mathcal {T} ^ {*}\right) ^ {h + 1} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (58) \\ \geq \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (59) \\ \geq \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (60) \\ \geq \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) - \gamma^ {h} \epsilon - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) - \gamma \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (61) \\ = \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a) - \gamma^ {h} \epsilon (1 + \gamma) - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) (1 + \gamma). (62) \\ \end{array}
$$
Above, (58) is by Lemma 3, (59) is due to the monotonicity of  $\mathcal{T}^*$  (i.e.  $\mathcal{T}^* q^\pi(s,a) \geq q^\pi(s,a)$ ), and (60) is by Lemma 2.
We now use Lemmas 1, 2, and 4 to introduce Theorem 1, which extends the original GPI theorem (Barreto et al., 2017) and allows us to characterize the gains achievable by  $h$ -GPI as a function of approximation errors in both the agent's policy library,  $\Pi$ , and its (possibly learned) model,  $\hat{m}$ . Notice that Theorem 1 generalizes the original GPI theorem, which is recovered when  $h = 0$ .
Theorem (1). Let  $\Pi = \{\pi_i\}_{i=1}^n$  be a set of policies,  $\{\hat{q}^{\pi_i}\}_{i=1}^n$  be approximations of their respective action-value functions, and  $\hat{m} = (\hat{p}, \hat{r})$  be an approximate model such that (as previously-assumed in Lemmas 2 and 3), for all  $\pi_i \in \Pi$  and all  $(s, a) \in S \times \mathcal{A}$ ,
$$
\left| q ^ {\pi_ {i}} (s, a) - \hat {q} ^ {\pi_ {i}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} \left| p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) \right| \leq \epsilon_ {p}, \text {a n d} | r (s, a) - \hat {r} (s, a) | \leq \epsilon_ {r}. \tag {63}
$$
$$
\pi^ {h - G P I} (s) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} ^ {\pi} (s, a). \tag {64}
$$
$$
\begin{array}{l} q ^ {h \cdot G P I} (s, a) \geq (\mathcal {T} ^ {*}) ^ {h} \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (65) \\ \geq \max  _ {\pi \in \Pi} q ^ {\pi} (s, a) - \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right), (66) \\ \end{array}
$$
where  $c(\epsilon_r,\epsilon_p,h) = \frac{1 - \gamma^h}{1 - \gamma} (\epsilon_r + \gamma \epsilon_pv_{max}^*)$
$$
\begin{array}{l} q ^ {h \cdot \operatorname {G P I}} (s, a) = \lim  _ {k \rightarrow \infty} \left(\mathcal {T} ^ {\pi^ {h \cdot \operatorname {G P I}}}\right) ^ {k} \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a)\right) (67) \\ \geq \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a)\right) - \lim  _ {k \rightarrow \infty} \sum_ {i = 0} ^ {k} \gamma^ {i} \left(\gamma^ {h} \epsilon (1 + \gamma) + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) (1 + \gamma)\right) (68) \\ = \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} \hat {q} ^ {\pi} (s, a)\right) - \frac {\gamma^ {h} \epsilon (1 + \gamma)}{1 - \gamma} - \frac {c \left(\epsilon_ {r} , \epsilon_ {p} , h\right) (1 + \gamma)}{1 - \gamma} (69) \\ \geq \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} q ^ {\pi} (s, a)\right) - \gamma^ {h} \epsilon - \frac {\gamma^ {h} \epsilon (1 + \gamma)}{1 - \gamma} - \frac {c \left(\epsilon_ {r} , \epsilon_ {p} , h\right) (1 + \gamma)}{1 - \gamma} (70) \\ \geq \left(\mathcal {T} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} q ^ {\pi} (s, a)\right) - c \left(\epsilon_ {r}, \epsilon_ {p}, h\right) - \gamma^ {h} \epsilon - \frac {\gamma^ {h} \epsilon (1 + \gamma)}{1 - \gamma} - \frac {c \left(\epsilon_ {r} , \epsilon_ {p} , h\right) (1 + \gamma)}{1 - \gamma} (71) \\ = \left(\mathcal {T} ^ {*}\right) ^ {h} \left(\max  _ {\pi \in \Pi} q ^ {\pi} (s, a)\right) - \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right). (72) \\ \end{array}
$$
Above, (67) is due to the fixed-point of  $\lim_{k\to \infty}(\mathcal{T}^{\pi^{h\cdot \mathrm{GPI}}})^k q(s,a)$  being  $q^{h\cdot \mathrm{GPI}}(s,a)$  for any  $q$ . (68) is due to the repeated application of Lemma 4. Finally, (70) is due to Lemma 1, and (71) is due to Lemma 2.
Theorem (2). Let  $\Pi = \{\pi_i^*\}_{i=1}^n$  be a set of optimal policies with respect to reward weights  $\{\mathbf{w}_i\}_{i=1}^n$  and  $\mathbf{w}$  be arbitrary reward weights. Let  $\hat{m} = (\hat{p}, \hat{r}_{\mathbf{w}})$  be an approximate model and  $\{\hat{q}_{\mathbf{w}}^{\pi_i^*}\}_{i=1}^n$  be approximations to the action-value functions of policies in  $\Pi$ , under the reward function  $r_{\mathbf{w}}$ , such that for all  $\pi_i \in \Pi$  and all  $(s, a) \in S \times \mathcal{A}$ ,
$$
\left| q _ {\mathbf {w}} ^ {\pi_ {i} ^ {*}} (s, a) - \hat {q} _ {\mathbf {w}} ^ {\pi_ {i} ^ {*}} (s, a) \right| \leq \epsilon , \quad \sum_ {s ^ {\prime}} | p \left(s ^ {\prime} \mid s, a\right) - \hat {p} \left(s ^ {\prime} \mid s, a\right) | \leq \epsilon_ {p}, \text {a n d} \left| r _ {\mathbf {w}} (s, a) - \hat {r} _ {\mathbf {w}} (s, a) \right| \leq \epsilon_ {r}. \tag {73}
$$
$$
\pi^ {h \cdot G P I} (s; \mathbf {w}) \in \underset {a \in \mathcal {A}} {\arg \max } \left(\mathcal {T} _ {\hat {m}} ^ {*}\right) ^ {h} \underset {\pi \in \Pi} {\max } \hat {q} _ {\mathbf {w}} ^ {\pi} (s, a). \tag {74}
$$
$$
q _ {\mathbf {w}} ^ {*} (s, a) - q _ {\mathbf {w}} ^ {h \cdot G P I} (s, a) \leq \frac {2}{1 - \gamma} \left(\phi_ {\max } \min  _ {i} | | \mathbf {w} - \mathbf {w} _ {i} | | + \gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right), \tag {75}
$$
Proof. The proof is an extension of Theorem 2 of Barreto et al. (2017) to the case where we use our Theorem 1 (i.e., we apply the bound introduced earlier to characterize the relation between the performance of  $h$ -GPI and the model and action-value function approximation errors), rather than using the GPI theorem, as defined by Barreto et al. (2017):
$$
\begin{array}{l} q _ {\mathbf {w}} ^ {*} (s, a) - q _ {\mathbf {w}} ^ {h \text {- G P I}} (s, a) \leq q _ {\mathbf {w}} ^ {*} (s, a) - \left(\mathcal {T} ^ {*}\right) ^ {h} \max  _ {\pi \in \Pi} q _ {\mathbf {w}} ^ {\pi} (s, a) + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (76) \\ \leq q _ {\mathbf {w}} ^ {*} (s, a) - \max  _ {\pi \in \Pi} q _ {\mathbf {w}} ^ {\pi} (s, a) + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (77) \\ \leq \frac {2}{1 - \gamma} \min  _ {i} \max  _ {s, a} | r _ {\mathbf {w}} (s, a) - r _ {\mathbf {w} _ {i}} (s, a) | + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (78) \\ = \frac {2}{1 - \gamma} \min  _ {i} \max  _ {s, a} | \phi (s, a) \cdot \mathbf {w} - \phi (s, a) \cdot \mathbf {w} _ {i} | + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (79) \\ = \frac {2}{1 - \gamma} \min  _ {i} \max  _ {s, a} | \phi (s, a) \cdot (\mathbf {w} - \mathbf {w} _ {i}) | + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (80) \\ \leq \frac {2}{1 - \gamma} \min  _ {i} \max  _ {s, a} | | \phi (s, a) | | | | \mathbf {w} - \mathbf {w} _ {i} | | + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (81) \\ = \frac {2}{1 - \gamma} \phi_ {\max } \min  _ {i} \left\| \mathbf {w} - \mathbf {w} _ {i} \right\| + \frac {2}{1 - \gamma} \left(\gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right) (82) \\ = \frac {2}{1 - \gamma} \left(\phi_ {\max } \min  _ {i} \left\| \mathbf {w} - \mathbf {w} _ {i} \right\| + \gamma^ {h} \epsilon + c \left(\epsilon_ {r}, \epsilon_ {p}, h\right)\right). (83) \\ \end{array}
$$
Above, (76) is due to Theorem 1 and (77) is due to the monotonicity of  $\mathcal{T}^*$ . Finally, (78) is due to the application of Lemma 1 of Barreto et al. (2017), and (81) is due to the Cauchy-Schwarz's inequality.
Theorem 2, above, allows us to characterize  $h$ -GPI's performance (under approximation errors) with respect to the optimum action-value function, given an arbitrary reward function,  $r_{\mathbf{w}}$ :
Theorem 2 precisely characterizes the optimality gap (i.e., the maximum difference) between the action-value function induced by  $h$ -GPI ( $q_{\mathbf{w}}^{h \cdot \mathrm{GPI}}(s, a)$ ) and the optimal action-value function ( $q_{\mathbf{w}}^*(s, a)$ ), as a function of (i) the reward weights  $\{\mathbf{w}_i\}_{i=1}^n$  for which the policies in the agent's library, II, are optimal; (ii) approximation errors in action-value functions  $\{\hat{q}_{\mathbf{w}}^{\pi_i^*}\}_{i=1}^n$ ; and (iii) approximation errors in the model  $\hat{m} = (\hat{p}, \hat{r}_{\mathbf{w}})$ .