SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $(i)$  comes from using  $\lambda \leq 1$  and simplifying.
Note that we can actually get a tighter contraction factor of  $\lambda \gamma$  for  $s = s'$ . Given this contractive property, we can prove its convergence with the use of the following lemma.
Lemma B.3 (Max  $\lambda$ R). The maximum possible value of  $\Phi_{\lambda}^{\pi}(s,s')$  is
$$
\frac {\mathbb {1} (s = s ^ {\prime}) + (1 - \mathbb {1} (s = s ^ {\prime})) \gamma}{1 - \lambda \gamma}.
$$
Proof. For  $s = s'$ ,
$$
\Phi_ {\lambda} ^ {\pi} (s, s) = 1 + \lambda \gamma \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi} (\cdot | s _ {t})} \Phi_ {\lambda} ^ {\pi} (s _ {t + 1}, s).
$$
This is just the standard SR recursion with discount factor  $\lambda \gamma$ , so the maximum is
$$
\sum_ {k = 0} ^ {\infty} (\lambda \gamma) ^ {k} = \frac {1}{1 - \lambda \gamma}. \tag {B.1}
$$
For  $s \neq s'$ ,  $\mathbb{1}(s_t = s') = 0$ , so
$$
\Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) = \gamma \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi} (\cdot | s _ {t})} \Phi_ {\lambda} ^ {\pi} (s _ {t + 1}, s ^ {\prime}).
$$
Observe that  $\Phi_{\lambda}^{\pi}(s,s) \geq \Phi_{\lambda}^{\pi}(s,s')$  for  $s' \neq s$ , so the maximum is attained for  $s_{t+1} = s'$ . We can then use the result for  $s = s'$  to get
$$
\Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) = \gamma \left(\frac {1}{1 - \lambda \gamma}\right). \tag {B.2}
$$
Combining Eq. (B.1) and Eq. (B.2) yields the desired result.
$$
| (\Phi^ {(k)} - \Phi_ {\lambda} ^ {\pi}) _ {s, s ^ {\prime}} | \leq \frac {\gamma^ {k + 1}}{1 - \lambda \gamma}.
$$
Proof. Using the notation  $X_{s,s'} = X(s,s')$  for a matrix  $X$ :
$$
\begin{array}{l} \left| \left(\Phi^ {(k)} - \Phi_ {\lambda} ^ {\pi}\right) _ {s, s ^ {\prime}} \right| = \left| \left(\mathcal {G} _ {\lambda} ^ {k} \Phi^ {(0)} - \mathcal {G} _ {\lambda} ^ {k} \Phi_ {\lambda} ^ {\pi}\right) _ {s, s ^ {\prime}} \right| \\ = \left| \left(\mathcal {G} _ {\lambda} ^ {k} \Phi^ {(0)} - \Phi_ {\lambda} ^ {\pi}\right) _ {s, s ^ {\prime}} \right| \\ \stackrel {(i)} {\leq} \gamma^ {k} | \left(\Phi^ {(0)} - \Phi_ {\lambda} ^ {\pi}\right) _ {s, s ^ {\prime}} | \tag {B.3} \\ \stackrel {(i i)} {=} \gamma^ {k} \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) \\ \stackrel {(i i i)} {\leq} \frac {\gamma^ {k + 1}}{1 - \lambda \gamma} \\ \end{array}
$$
where  $(i)$  is due to Lemma B.2,  $(ii)$  is because  $\Phi^{(0)}(s,s^{\prime}) = 0$  for  $s\neq s^{\prime}$ , and  $(iii)$  is due to Lemma B.3.
Lemma B.4 (Subadditivity). For any  $s \in S$ , policy  $\pi$ ,  $\lambda \in [0,1)$ , and disjoint measurable sets  $A, B \subseteq S$ ,
$$
\Phi_ {\lambda} ^ {\pi} (s, A \cup B) <   \Phi_ {\lambda} ^ {\pi} (s, A) + \Phi_ {\lambda} ^ {\pi} (s, B).
$$
Proof. Note that for disjoint sets  $A, B$ , we have  $n_t(A \cup B, k) = n_t(A, k) + n_t(B, k)$ . Hence, conditioned on some policy  $\pi$  and  $s_t = s$ ,
$$
\begin{array}{l} \lambda^ {n _ {t} (A \cup B, k)} \mathbb {P} \left(s _ {t + k} \in A \cup B\right) = \lambda^ {n _ {t} (A, k)} \lambda^ {n _ {t} (B, k)} \mathbb {P} \left(s _ {t + k} \in A\right) + \lambda^ {n _ {t} (A, k)} \lambda^ {n _ {t} (B, k)} \mathbb {P} \left(s _ {t + k} \in B\right) \\ \leq \lambda^ {n _ {t} (A, k)} \mathbb {P} (s _ {t + k} \in A) + \lambda^ {n _ {t} (B, k)} \mathbb {P} (s _ {t + k} \in B), \\ \end{array}
$$
where the first line follows from  $\mathbb{P}(s_{t+k} \in A \cup B) = \mathbb{P}(s_{t+k} \in A) + \mathbb{P}(s_{t+k} \in B)$ . Equality holds over all  $A, B, t, k$  if and only if  $\lambda = 1$ . Summing over  $k$  yields the result.
B.1 Proof of Theorem 5.1
We first prove two results, which rely throughout on the fact that  $\Phi_{\lambda}(s,a,s^{\prime})\leq \frac{1}{1 - \lambda\gamma}$  for all  $s,a,s^{\prime}$ , which follows from Lemma B.3. For simplicity, we also assume throughout that all rewards are non-negative, but this assumption can easily be dropped by taking absolute values of rewards. The proofs presented here borrow ideas from those of [16].
Lemma B.5. Let  $\{M_j\}_{j=1}^n \subseteq \mathcal{M}$  and  $M \in \mathcal{M}$  be a set of tasks in an environment  $\mathcal{M}$  with diminishing rate  $\lambda$  and let  $Q^{\pi_j^*}$  denote the action-value function of an optimal policy of  $M_j$  when executed in  $M$ . Given estimates  $\tilde{Q}^{\pi_j}$  such that  $\|Q^{\pi_j^*} - \tilde{Q}^{\pi_j}\|_{\infty} \leq \epsilon$  for all  $j$ , define
$$
\pi (s) \in \operatorname * {a r g m a x} _ {a} \max  _ {j} \tilde {Q} ^ {\pi_ {j}} (s, a).
$$
$$
Q ^ {\pi} (s, a) \geq \max  _ {j} Q ^ {\pi_ {j} ^ {*}} (s, a) - \frac {1}{1 - \gamma} \left(2 \epsilon + \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right),
$$
where  $r$  denotes the reward function of  $M$ .
Proof. Define  $\tilde{Q}_{\max}(s,a) \coloneqq \max_j \tilde{Q}^{\pi_j}(s,a)$  and  $Q_{\max}(s,a) \coloneqq \max_j Q^{\pi_j^*}(s,a)$ . Let  $T^\nu$  denote the Bellman operator of a policy  $\nu$  in task  $M$ . For all  $(s,a) \in S \times \mathcal{A}$  and
all  $j$
$$
\begin{array}{l} T _ {i} ^ {\pi} \tilde {Q} _ {\max } (s, a) = r (s, a) + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) \left((\lambda - 1) r _ {i} (s, a) \Phi^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime}), s) + \tilde {Q} _ {\max } (s ^ {\prime}, \pi (s ^ {\prime}))\right) \\ = r (s, a) + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) \left((\lambda - 1) r _ {i} (s, a) \Phi^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime}), s) + \max _ {b} \tilde {Q} _ {\max} (s ^ {\prime}, b)\right) \\ \geq r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \left((\lambda - 1) r _ {i} (s, a) \Phi^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right), s\right) + \max  _ {b} Q _ {\max } \left(s ^ {\prime}, b\right)\right) - \gamma \epsilon \\ \geq r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \left(\left(\lambda - 1\right) r _ {i} (s, a) \Phi^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right), s\right) + Q _ {\max } \left(s ^ {\prime}, \pi_ {j} ^ {*} \left(s ^ {\prime}\right)\right)\right) - \gamma \epsilon \\ \geq r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} | s, a\right) \left((\lambda - 1) r _ {i} (s, a) \Phi^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right), s\right) + Q _ {i} ^ {\pi_ {j} ^ {*}} \left(s ^ {\prime}, \pi_ {j} ^ {*} \left(s ^ {\prime}\right)\right)\right) - \gamma \epsilon \\ = r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} | s, a\right) \left(\left(\lambda - 1\right) r _ {i} (s, a) \Phi^ {\pi_ {j} ^ {*}} \left(s ^ {\prime}, \pi_ {j} ^ {*} \left(s ^ {\prime}\right), s\right) + Q _ {i} ^ {\pi_ {j} ^ {*}} \left(s ^ {\prime}, \pi_ {j} ^ {*} \left(s ^ {\prime}\right)\right)\right) - \gamma \epsilon \\ + \gamma (\lambda - 1) r (s, a) \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \left(\Phi^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right), s\right) - \Phi^ {\pi_ {j} ^ {*}} \left(s ^ {\prime}, \pi_ {j} ^ {*} \left(s ^ {\prime}\right), s\right)\right) \\ \geq T _ {i} ^ {\pi_ {j} ^ {*}} Q _ {i} ^ {\pi_ {j} ^ {*}} (s, a) - \gamma \epsilon - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma} \\ = Q _ {i} ^ {\pi_ {j} ^ {*}} (s, a) - \gamma \epsilon - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}. \\ \end{array}
$$
This holds for any  $j$ , so
$$
\begin{array}{l} T ^ {\pi} \tilde {Q} _ {\max } (s, a) \geq \max  _ {j} Q _ {i} ^ {\pi_ {j} ^ {*}} (s, a) - \gamma \epsilon - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma} \\ = Q _ {\max } (s, a) - \gamma \epsilon - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma} \\ \geq \tilde {Q} _ {\max } (s, a) - \epsilon - \gamma \epsilon - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}. \\ \end{array}
$$
Next, note that for any  $c\in \mathbb{R}$
$$
\begin{array}{l} T ^ {\pi} (\tilde {Q} _ {\max } (s, a) + c) = T ^ {\pi} \tilde {Q} _ {\max } (s, a) + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) c \\ = T ^ {\pi} \bar {Q} _ {\max } (s, a) + \gamma c. \\ \end{array}
$$
Putting everything together, and using the fact that  $T^{\nu}$  is monotonic and contractive,
$$
\begin{array}{l} Q _ {i} ^ {\pi} (s, a) = \lim _ {k \to \infty} (T ^ {\pi}) ^ {k} \tilde {Q} _ {\max} (s, a) \\ \geq \lim _ {k \to \infty} \left[ \tilde {Q} _ {\max } (s, a) - \left(\epsilon (1 + \gamma) - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right) \sum_ {j = 0} ^ {k} \gamma^ {j} \right] \\ \geq \tilde {Q} _ {\max } (s, a) - \frac {1}{1 - \gamma} \left(\epsilon (1 + \gamma) - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right) \\ \geq Q _ {\max } (s, a) - \epsilon - \frac {1}{1 - \gamma} \left(\epsilon (1 + \gamma) - \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right) \\ \geq Q ^ {\pi_ {j} ^ {*}} (s, a) - \frac {1}{1 - \gamma} \left(2 \epsilon + \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right). \\ \end{array}
$$
This holds for every  $j$ , hence the result.
Lemma B.6. Let  $\nu$  be any policy,  $\lambda, \hat{\lambda} \in [0,1]$ , and  $Q_{\lambda}$  denote a value function with respecting to diminishing rate  $\lambda$ . Then,
$$
\| Q _ {\lambda} ^ {\nu} - Q _ {\hat {\lambda}} ^ {\nu} \| _ {\infty} \leq \frac {| \lambda - \hat {\lambda} | \| r \| _ {\infty}}{1 - \gamma}.
$$
Proof. The proof follows from the definition of  $Q$ : for every  $(s, a) \in S \times \mathcal{A}$ ,
$$
\begin{array}{l} | Q _ {\lambda} ^ {\nu} (s, a) - Q _ {\hat {\lambda}} ^ {\nu} (s, a) | = \left| \right. \mathbb {E} _ {\pi} \left[ \right. \sum_ {k = 0} ^ {\infty} \gamma^ {k} \left(\lambda^ {n _ {t} (s _ {t + k}, k)} - \hat {\lambda} ^ {n _ {t} (s _ {t + k}, k)}\right) r (s _ {t + k}) \left. \right| s _ {t} = s, a _ {t} = a \left. \right]\left. \right| \\ \leq \mathbb {E} _ {\pi} \left[ \right. \sum_ {k = 0} ^ {\infty} \gamma^ {k} \left| \lambda^ {n _ {t} (s _ {t + k}, k)} - \hat {\lambda} ^ {n _ {t} (s _ {t + k}, k)} \right| r (s _ {t + k}) \left. \right| s _ {t} = s, a _ {t} = a \left. \right] \\ = \mathbb {E} _ {\pi} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r (s _ {t + k}) \left| \lambda - \hat {\lambda} \right| \sum_ {j = 0} ^ {n _ {t} (s _ {t + k}, k) - 1} \lambda^ {n _ {t} (s _ {t + k}, k) - 1 - j} \hat {\lambda} ^ {j} \mid s _ {t} = s, a _ {t} = a \right] \\ \leq | \lambda - \hat {\lambda} | \mathbb {E} _ {\pi} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r (s _ {t + k}) \Big | s _ {t} = s, a _ {t} = a \right] \\ \leq \frac {| \lambda - \hat {\lambda} | \| r \| _ {\infty}}{1 - \gamma}. \\ \end{array}
$$
Theorem 5.1 (GPI). Let  $\{M_j\}_{j=1}^n \subseteq \mathcal{M}$  and  $M \in \mathcal{M}$  be a set of tasks in an environment  $\mathcal{M}$  and let  $Q^{\pi_j^*}$  denote the action-value function of an optimal policy of  $M_j$  when executed in  $M$ . Assume that the agent uses diminishing rate  $\hat{\lambda}$  that may differ from the true environment diminishing rate  $\lambda$ . Given estimates  $\tilde{Q}^{\pi_j}$  such that  $\| Q^{\pi_j^*} - \tilde{Q}^{\pi_j} \|_{\infty} \leq \epsilon$  for all  $j$ , define
Proof. Let  $Q_{\lambda}$  denote a value function with respect to diminishing constant  $\lambda$ . We wish to bound
$$
\max _ {j} Q _ {\hat {\lambda}} ^ {\pi_ {j} ^ {*}} (s, a) - Q _ {\lambda} ^ {\pi} (s, a),
$$
i.e., the value of the GPI policy with respect to the true  $\lambda$  compared to the maximum value of the constituent policies  $\pi_j^*$  used for GPI, which were used assuming  $\hat{\lambda}$ . By the triangle inequality,
$$
\begin{array}{l} \max  _ {j} Q _ {\hat {\lambda}} ^ {\pi_ {j} ^ {*}} (s, a) - Q _ {\lambda} ^ {\pi} (s, a) \leq \max  _ {j} Q _ {\lambda} ^ {\pi_ {j} ^ {*}} (s, a) - Q _ {\lambda} ^ {\pi} (s, a) + | \max  _ {j} Q _ {\lambda} ^ {\pi_ {j} ^ {*}} (s, a) - \max  _ {j} Q _ {\hat {\lambda}} ^ {\pi_ {j} ^ {*}} (s, a) | \\ \leq \underbrace {\max  _ {j} Q _ {\lambda} ^ {\pi_ {j} ^ {*}} (s , a) - Q _ {\lambda} ^ {\pi} (s , a)} _ {(1)} + \max  _ {j} \underbrace {| Q _ {\lambda} ^ {\pi_ {j} ^ {*}} (s , a) - Q _ {\hat {\lambda}} ^ {\pi_ {j} ^ {*}} (s , a) |} _ {(2)}. \\ \end{array}
$$
We bound (1) by Lemma B.5 and (2) by Lemma B.6 to get the result.
B.2 An Extension of Theorem 5.1
Inspired by [18], we prove an extension of Theorem 5.1:
Theorem B.1. Let  $M \in \mathcal{M}$  be a task in an environment  $\mathcal{M}$  with true diminishing constant  $\lambda$ . Suppose we perform GPI assuming a diminishing constant  $\hat{\lambda}$ :
Let  $\{M_j\}_{j=1}^n$  and  $M_i$  be tasks in  $\mathcal{M}$  and let  $Q_i^{\pi_j^*}$  denote the action-value function of an optimal policy of  $M_j$  when executed in  $M_i$ . Given estimates  $\tilde{Q}_i^{\pi_j}$  such that  $\| Q_i^{\pi_j^*} - \tilde{Q}_i^{\pi_j} \|_{\infty} \leq \epsilon$  for all  $j$ , define  $\pi(s) \in \operatorname{argmax}_a \max_j \tilde{Q}_i^{\pi_j}(s, a)$ .
Let  $Q_{\hat{\lambda}}^{\pi}$  and  $Q_{\lambda}^{\pi^{*}}$  denote the action-value functions of  $\pi$  and the  $M$ -optimal policy  $\pi^{*}$  when executed in  $M$ , respectively. Then,
$$
\| Q _ {\lambda} ^ {\pi^ {*}} - Q _ {\hat {\lambda}} ^ {\pi} \| _ {\infty} \leq \frac {2}{1 - \gamma} \left(\frac {1}{2} | \lambda - \hat {\lambda} | \| r \| _ {\infty} + \epsilon + \| r - r _ {i} \| _ {\infty} + \min _ {j} \| r _ {i} - r _ {j} \| _ {\infty}\right) + \frac {1 - \lambda}{1 - \lambda \gamma} C,
$$
where  $C$  is a positive constant not depending on  $\lambda$ :
$$
C = \gamma \frac {2 \| r - r _ {i} \| _ {\infty} + 2 \min _ {j} \| r _ {i} - r _ {j} \| _ {\infty} + \min \left(\| r \| _ {\infty} , \| r _ {i} \| _ {\infty}\right) + \min \left(\| r _ {i} \| _ {\infty} , \| r _ {1} \| _ {\infty} , \ldots , \| r _ {n} \| _ {\infty}\right)}{1 - \gamma}.
$$
Note that when  $\lambda = 1$ , we recover Proposition 1 of [18] with an additional term quantifying error incurred by  $\hat{\lambda} \neq \lambda$ . The proof relies on two other technical lemmas, presented below.
Lemma B.7.
$$
\| Q ^ {\pi^ {*}} - Q _ {i} ^ {\pi_ {i} ^ {*}} \| _ {\infty} \leq \frac {\| r - r _ {i} \| _ {\infty}}{1 - \gamma} + \gamma (1 - \lambda) \frac {\min \left(\| r \| _ {\infty} , \| r _ {i} \| _ {\infty}\right) + \| r - r _ {i} \| _ {\infty}}{(1 - \gamma) (1 - \lambda \gamma)}.
$$
Proof. Define  $\Delta_i \coloneqq \| Q^{\pi^*} - Q_i^{\pi_i^*} \|_{\infty}$ . For any  $(s, a) \in S \times \mathcal{A}$ ,
$$
\begin{array}{l} | Q ^ {\pi^ {*}} (s, a) - Q _ {i} ^ {\pi_ {i} ^ {*}} (s, a) | = \Bigg | r (s, a) + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) \left((\lambda - 1) r (s, a) \Phi^ {\pi^ {*}} (s ^ {\prime}, \pi^ {*} (s ^ {\prime}), s) + Q ^ {\pi^ {*}} (s ^ {\prime}, \pi^ {*} (s ^ {\prime})\right) \\ \left. - r _ {i} (s, a) - \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \left(\left(\lambda - 1\right) r _ {i} (s, a) \Phi^ {\pi_ {i} ^ {*}} \left(s ^ {\prime}, \pi_ {i} ^ {*} \left(s ^ {\prime}\right), s\right) + Q _ {i} ^ {\pi_ {i} ^ {*}} \left(s ^ {\prime}, \pi_ {i} ^ {*} \left(s ^ {\prime}\right)\right)\right) \right| \\ \leq | r (s, a) - r _ {i} (s, a) | + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) | Q ^ {\pi^ {*}} (s, a) - Q _ {i} ^ {\pi_ {i} ^ {*}} (s, a) | \\ + \gamma (\lambda - 1) \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) \left| r (s, a) \Phi^ {\pi^ {*}} (s ^ {\prime}, \pi^ {*} (s ^ {\prime}), s) - r _ {i} (s, a) \Phi^ {\pi_ {i} ^ {*}} (s ^ {\prime}, \pi_ {i} ^ {*} (s ^ {\prime}), s) \right| \\ \leq \left\| r - r _ {i} \right\| _ {\infty} + \gamma \Delta_ {i} + \gamma (1 - \lambda) \left\| r \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty}. \\ \end{array}
$$
The third term decomposes as
$$
\begin{array}{l} \left\| r \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty} \leq \left\| r \Phi^ {\pi^ {*}} - r \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty} + \left\| r \Phi^ {\pi_ {i} ^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty} \\ \leq \frac {\| r \| _ {\infty} + \| r - r _ {i} \| _ {\infty}}{1 - \lambda \gamma}. \\ \end{array}
$$
We could equivalently use the following decomposition:
$$
\begin{array}{l} \left\| r \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty} \leq \left\| r \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi^ {*}} \right\| _ {\infty} + \left\| r _ {i} \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \right\| _ {\infty} \\ \leq \frac {\| r _ {i} \| _ {\infty} + \| r - r _ {i} \| _ {\infty}}{1 - \lambda \gamma}, \\ \end{array}
$$
$$
\| r \Phi^ {\pi^ {*}} - r _ {i} \Phi^ {\pi_ {i} ^ {*}} \| _ {\infty} \leq \frac {\min \left(\| r \| _ {\infty} , \| r _ {i} \| _ {\infty}\right) + \| r - r _ {i} \| _ {\infty}}{1 - \lambda \gamma}.
$$
The inequalities above hold for all  $s, a$  and so
$$
\begin{array}{l} \Delta_ {i} \leq \| r - r _ {i} \| _ {\infty} + \gamma \Delta_ {i} + \gamma (1 - \lambda) \frac {\min \left(\| r \| _ {\infty} , \| r _ {i} \| _ {\infty}\right) + \| r - r _ {i} \| _ {\infty}}{1 - \lambda \gamma} \\ \Longrightarrow \Delta_ {i} \leq \frac {\| r - r _ {i} \| _ {\infty}}{1 - \gamma} + \gamma (1 - \lambda) \frac {\min  (\| r \| _ {\infty} , \| r _ {i} \| _ {\infty}) + \| r - r _ {i} \| _ {\infty}}{(1 - \gamma) (1 - \lambda \gamma)}. \\ \end{array}
$$
Hence the result.
Lemma B.8. For any policy  $\pi$
$$
\| Q _ {i} ^ {\pi} - Q ^ {\pi} \| _ {\infty} \leq \frac {\| r - r _ {i} \| _ {\infty}}{1 - \gamma} + \gamma (1 - \lambda) \frac {\| r - r _ {i} \| _ {\infty}}{(1 - \gamma) (1 - \lambda \gamma)}.
$$
Proof. Write  $\Delta_i \coloneqq \|Q_i^\pi - Q^\pi\|_\infty$ . Proceeding as in the previous lemma, for all  $(s, a) \in S \times \mathcal{A}$ , we have
$$
\begin{array}{l} | Q _ {i} ^ {\pi} (s, a) - Q ^ {\pi} (s, a) | = \Big | r _ {i} (s, a) + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) ((\lambda - 1) r _ {i} (s, a) \Phi^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime}), s) + Q _ {i} ^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime}))) \\ - r (s, a) - \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) ((\lambda - 1) r (s, a) \Phi^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right), s\right) + Q ^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right)\right)) \\ \leq | r (s, a) - r _ {i} (s, a) | + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) (1 - \lambda) | r (s, a) - r _ {i} (s, a) | \Phi^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime}), s) \\ + \gamma \sum_ {s ^ {\prime}} p (s ^ {\prime} | s, a) | Q _ {i} ^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime})) - Q ^ {\pi} (s ^ {\prime}, \pi (s ^ {\prime})) | \\ \leq \| r - r _ {i} \| _ {\infty} + \gamma (1 - \lambda) \| r - r _ {i} \| _ {\infty} \frac {1}{1 - \lambda \gamma} + \gamma \Delta_ {i} ^ {\prime} \\ \Rightarrow \Delta_ {i} ^ {\prime} \leq \| r - r _ {i} \| _ {\infty} + \frac {\gamma (1 - \lambda) \| r - r _ {i} \| _ {\infty}}{1 - \lambda \gamma} + \gamma \Delta_ {i} ^ {\prime} \\ \Rightarrow \Delta_ {i} ^ {\prime} \leq \frac {\| r - r _ {i} \| _ {\infty}}{1 - \gamma} + \frac {\gamma (1 - \lambda) \| r - r _ {i} \| _ {\infty}}{(1 - \gamma) (1 - \lambda \gamma)}. \\ \end{array}
$$
Finally, we prove Theorem B.1:
Proof of Theorem B.1. By the triangle inequality,
$$
\left\| Q _ {\lambda} ^ {\pi^ {*}} - Q _ {\hat {\lambda}} ^ {\pi} \right\| _ {\infty} \leq \left\| Q _ {\lambda} ^ {\pi^ {*}} - Q _ {\lambda} ^ {\pi} \right\| _ {\infty} + \left\| Q _ {\lambda} ^ {\pi} - Q _ {\hat {\lambda}} ^ {\pi} \right\| _ {\infty}.
$$
By Lemma B.6, the second term is bounded above by
$$
\frac {| \lambda - \hat {\lambda} | \| r \| _ {\infty}}{1 - \gamma}.
$$
The first term decomposes as follows (dropping the  $\lambda$  subscript on all action-value functions for clarity):
$$
\| Q ^ {\pi^ {*}} - Q ^ {\pi} \| _ {\infty} \leq \underbrace {\| Q ^ {\pi^ {*}} - Q _ {i} ^ {\pi_ {i} ^ {*}} \| _ {\infty}} _ {(1)} + \underbrace {\| Q _ {i} ^ {\pi_ {i} ^ {*}} - Q _ {i} ^ {\pi} \| _ {\infty}} _ {(2)} + \underbrace {\| Q _ {i} ^ {\pi} - Q ^ {\pi} \| _ {\infty}} _ {(3)}.
$$
Applying Lemma B.5 to (2) (but with respect to  $M_{i}$  rather than  $M$ ), we have that for any  $j$ ,
$$
\begin{array}{l} Q _ {i} ^ {\pi_ {i} ^ {*}} (s, a) - Q _ {i} ^ {\pi} (s, a) \leq Q _ {i} ^ {\pi_ {i} ^ {*}} (s, a) - Q _ {i} ^ {\pi_ {j} ^ {*}} (s, a) + \frac {1}{1 - \gamma} \left(2 \epsilon + \frac {\gamma (1 - \lambda) r _ {i} (s , a)}{1 - \lambda \gamma}\right) \\ \Rightarrow \| Q _ {i} ^ {\pi_ {i} ^ {*}} - Q _ {i} ^ {\pi} \| _ {\infty} \leq \underbrace {\| Q _ {i} ^ {\pi_ {i} ^ {*}} - Q _ {j} ^ {\pi_ {j} ^ {*}} \| _ {\infty}} _ {(2. 1)} + \underbrace {\| Q _ {j} ^ {\pi_ {j} ^ {*}} - Q _ {i} ^ {\pi_ {j} ^ {*}} \| _ {\infty}} _ {(2. 2)} + \frac {1}{1 - \gamma} \left(2 \epsilon + \frac {\gamma (1 - \lambda) \| r _ {i} \| _ {\infty}}{1 - \lambda \gamma}\right). \\ \end{array}
$$
We bound (2.1) using Lemma B.7 and (2.2) using Lemma B.8 (but with respect to  $M_{j}$  rather than  $M$ ):
$$
\| Q _ {i} ^ {\pi_ {i} ^ {*}} - Q _ {j} ^ {\pi_ {j} ^ {*}} \| _ {\infty} + \| Q _ {j} ^ {\pi_ {j} ^ {*}} - Q _ {i} ^ {\pi_ {j} ^ {*}} \| _ {\infty} \leq \frac {2 \| r _ {i} - r _ {j} \| _ {\infty}}{1 - \gamma} + \gamma (1 - \lambda) \frac {\min \left(\| r _ {i} \| _ {\infty} , \| r _ {j} \| _ {\infty}\right) + 2 \| r _ {i} - r _ {j} \| _ {\infty}}{(1 - \gamma) (1 - \lambda \gamma)}.
$$
We then apply Lemma B.7 to (1) and Lemma B.8 to (3) to get the result.
C An nth Occupancy Representation
To generalize the first occupancy representation to account for reward functions of this type, it's natural to consider an  $N$ th occupancy representation—that is, one which accumulates value only for the first  $N$  occupancies of one state  $s'$  starting from another state  $s$ :
Definition C.1 (NR). For an MDP with finite  $S$ , the  $N$ th-occupancy representation (NR) for a policy  $\pi$  is given by  $F^{\pi} \in [0, N]^{|\mathcal{S}| \times |\mathcal{S}|}$  such that
$$
\Phi_ {(N)} ^ {\pi} (s, s ^ {\prime}) \triangleq \mathbb {E} _ {\pi} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {t + k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}, \# \left(\left\{j \mid s _ {t + j} = s ^ {\prime}, j \in [ 0, k - 1 ] \right\}\right) <   N\right) \mid s _ {t} \right]. \tag {C.1}
$$
Intuitively, such a representation sums the first  $N$  (discounted) occupancies of  $s'$  from time  $t$  to  $t + k$  starting from  $s_t = s$ . We can also note that  $\Phi_{(1)}^{\pi}$  is simply the FR and  $\Phi_{(0)}(s, s') = 0 \forall s, s'$ . As with the FR and the SR, we can derive a recursive relationship for the NR:
$$
\Phi_ {(N)} ^ {\pi} (s, s ^ {\prime}) = \mathbb {1} (s _ {t} = s ^ {\prime}) (1 + \gamma \mathbb {E} \Phi_ {(N - 1)} ^ {\pi} (s _ {t + 1}, s ^ {\prime})) + \gamma (1 - \mathbb {1} (s _ {t} = s ^ {\prime})) \mathbb {E} \Phi_ {(N)} ^ {\pi} (s _ {t + 1}, s ^ {\prime}), \tag {C.2}
$$
where the expectation is wrt  $p^{\pi}(s_{t + 1}|s_t)$ . Once again, we can confirm that this is consistent with the FR by noting that for  $N = 1$ , the NR recursion recovers the FR recursion. Crucially, we also recover the SR recursion in the limit as  $N\to \infty$ :
$$
\begin{array}{l} \lim _ {N \to \infty} \Phi_ {(N)} ^ {\pi} (s, s ^ {\prime}) = \mathbb {1} (s _ {t} = s ^ {\prime}) (1 + \gamma \mathbb {E} \Phi_ {(\infty)} ^ {\pi} (s _ {t + 1}, s ^ {\prime})) + \gamma (1 - \mathbb {1} (s _ {t} = s ^ {\prime})) \mathbb {E} \Phi_ {(\infty)} ^ {\pi} (s _ {t + 1}, s ^ {\prime}) \\ = \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \gamma \mathbb {E} \Phi_ {(\infty)} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right). \\ \end{array}
$$
This is consistent with the intuition that the SR accumulates every (discounted) state occupancy in a potentially infinite time horizon of experience. While Definition C.1 admits a recursive form which is consistent with our intuition, Eq. (C.2) reveals an inconvenient intractability: the Bellman target for  $\Phi_{(N)}^{\pi}$  requires the availability of  $\Phi_{(N-1)}^{\pi}$ . This is a challenge, because it means that if we'd like to learn any NR for finite  $N > 1$ , the agent also must learn and store  $\Phi_{(1)}^{\pi}, \ldots, \Phi_{(N-1)}^{\pi}$ . Given these challenges, the question of how to learn a tractable general occupancy representation remains. From a neuroscientific perspective, a fixed depletion amount is also inconsistent with both behavioral observations and neural imaging [3], which indicate instead that utility disappears at a fixed rate in proportion to the current remaining utility, rather than in proportion to the original utility. We address these theoretical and practical issues in the next section.
D Additional Related Work