SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ted Moskovitz
Gatsby Unit, UCL
ted@gatsby.ucl.ac.uk
Samo Hromadka
samo.hromadka.21@ucl.ac.uk
Ahmed Touati
atouati@meta.com
Diana Borsa
borsa@google.com
Maneesh Sahani
maneesh@gatsby.ucl.ac.uk
A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the  $\lambda$  representation ( $\lambda R$ ) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the  $\lambda R$ 's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.
The second ice cream cone rarely tastes as good as the first, and once all the most accessible brambles have been picked, the same investment of effort yields less fruit. In everyday life, the availability and our enjoyment of stimuli is sensitive to our past interactions with them. Thus, to evaluate different courses of action and act accordingly, we might expect our brains to form representations sensitive to the non-stationarity of rewards. Evidence in fields from behavioral economics [1, 2] to neuroscience [3] supports this hypothesis. Surprisingly, however, most of reinforcement learning (RL) takes place under the assumptions of the Markov Decision Process [MDP; 4], where rewards and optimal decision-making remain stationary.
Figure 1.1: Diminishing rewards.
In this paper, we seek to bridge this gap by studying the phenomenon of diminishing marginal utility [5] in the context of RL. Diminishing marginal utility (DMU) is the subjective phenomenon by which repeated exposure to a rewarding stimulus reduces the perceived utility one experiences. While DMU is thought to have its roots in the maintenance of homeostatic equilibrium (too much ice cream can result in a stomach ache), it also manifests itself in domains in which the collected rewards are abstract, such as economics ( $10 vs.$ 0 is perceived as a bigger difference in value than $1,010 vs. $1,000), where it is closely related to risk aversion [6, 7]. While DMU is well-studied in other fields, relatively few RL studies have explored diminishing reward functions [8, 9], and, to our
knowledge, none contain a formal analysis of DMU within RL. Here, we seek to characterize both its importance and the challenge it poses for current RL approaches (Section 3).
Surprisingly, we find that evaluating policies under diminishing rewards requires agents to learn a novel state representation which we term the  $\lambda$  representation ( $\lambda R$ , Section 4). The  $\lambda R$  generalizes several state representations from the RL literature: the successor representation [SR; 10], the first-occupancy representation [FR; 11], and the forward-backward representation [FBR; 12], adapting them for non-stationary environments. Interestingly, despite the non-stationarity of the underlying reward functions, we show that the  $\lambda R$  still admits a Bellman recursion, allowing for efficient computation via dynamic programming (or approximate dynamic programming) and prove its convergence. We demonstrate the scalability of the  $\lambda R$  to large and continuous state spaces (Section 4.1), show that it supports policy evaluation, improvement, and composition (Section 5), and show that the behavior it induces is consistent with optimal foraging theory (Section 6).
Standard RL In standard RL, the goal of the agent is to act within its environment so as to maximize its discounted cumulative reward for some task  $T$  [13]. Typically,  $T$  is modeled as a discounted MDP,  $T = (\mathcal{S}, \mathcal{A}, p, r, \gamma, \mu_0)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the set of available actions,  $p: \mathcal{S} \times \mathcal{A} \mapsto \mathcal{P}(\mathcal{S})$  is the transition kernel (where  $\mathcal{P}(\mathcal{S})$  is the space of probability distributions over  $\mathcal{S}$ ),  $r: \mathcal{S} \mapsto \mathbb{R}$  is the reward function,  $\gamma \in [0,1)$  is a discount factor, and  $\mu_0 \in \mathcal{P}(\mathcal{S})$  is the distribution over initial states. Note that the reward function is also frequently defined over state-action pairs  $(s, a)$  or triples  $(s, a, s')$ , but to simplify notation we mostly consider state-based rewards. All of the following analysis, unless otherwise noted, can easily be extended to the  $(s, a)$ - or  $(s, a, s')$ -dependent cases. The goal of the agent is to maximize its expected return, or discounted cumulative reward  $\sum_{t} \gamma^{t} r(s_{t})$ . The role of the discount factor is twofold: from a theoretical standpoint, it ensures that this sum is finite for bounded rewards, and it induces myopia in the agent's behavior. To simplify notation, we will frequently write  $r(s_{t}) \triangleq r_{t}$  and  $\mathbf{r} \in \mathbb{R}^{|S|}$  as the vector of rewards for each state. The agent acts according to a stationary policy  $\pi: \mathcal{S} \mapsto \mathcal{P}(\mathcal{A})$ . For finite MDPs, we can describe the expected transition probabilities under  $\pi$  using a  $|S| \times |S|$  matrix  $P^{\pi}$  such that  $P_{s,s'}^{\pi} = p^{\pi}(s'|s) \triangleq \sum_{a} p(s'|s, a)\pi(a|s)$ . Given  $\pi$  and a reward function  $r$ , the expected return associated with taking action  $a$  in state  $s$  is
$$
Q _ {r} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k} \mid s _ {t} = s, a _ {t} = a \right] = \mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \left[ r _ {t} + \gamma Q _ {r} ^ {\pi} \left(s ^ {\prime}, \pi \left(s ^ {\prime}\right)\right) \right]. \tag {2.1}
$$
The  $Q_r^\pi$  are called the state-action values or simply the  $Q$ -values of  $\pi$ . The expectation  $\mathbb{E}_{\pi}[\cdot]$  is taken with respect to the randomness of both the policy and the transition dynamics. For simplicity of notation, from here onwards we will write expectations of the form  $\mathbb{E}_{\pi}[\cdot|s_t = s, a_t = a]$  as  $\mathbb{E}_{\pi}[\cdot|s_t, a_t]$ . The recursive form in Eq. (2.1) is called the Bellman equation, and it makes the process of estimating  $Q_r^\pi$ -termed policy evaluation—tractable via dynamic programming. In particular, successive applications of the Bellman operator  $\mathcal{T}^\pi Q \triangleq r + \gamma P^\pi Q$  are guaranteed to converge to the true value function  $Q^\pi$  for any initial real-valued  $|S| \times |\mathcal{A}|$  matrix  $Q$ . Once a policy has been evaluated, policy improvement identifies a new policy  $\pi'$  such that  $Q_r^\pi(s, a) \geq Q_r^{\pi'}(s, a)$ ,  $\forall (s, a) \in Q_r^\pi(s, a)$ . Helpfully, such a policy can be defined as  $\pi'(s) \in \operatorname{argmax}_a Q_r^\pi(s, a)$ .
Value Decomposition Often, it can be useful for an agent to evaluate the policies it's learned on new reward functions. In order to do so efficiently, it can make use of the successor representation [SR; 10], which decomposes the value function as follows:
$$
V ^ {\pi} (s) = \mathbb {E} _ {\pi} \left[ \sum_ {k \geq 0} \gamma^ {k} r _ {t + k} \mid s _ {t} \right] = \mathbf {r} ^ {\top} \underbrace {\mathbb {E} _ {\pi} \left[ \sum_ {k \geq 0} \gamma^ {k} \mathbf {1} \left(s _ {t + k}\right) \mid s _ {t} \right]} _ {\triangleq M ^ {\pi} (s)}, \tag {2.2}
$$
where  $\mathbf{1}(s_t)$  is a one-hot representation of state  $s_t$  and  $M^{\pi}$ , the SR, is an  $|\mathcal{S}|\times |\mathcal{S}|$  matrix such that
$$
M ^ {\pi} (s, s ^ {\prime}) \triangleq \mathbb {E} _ {\pi} \left[ \sum_ {k \geq 0} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \mid s _ {t} \right] = \mathbb {E} _ {\pi} \left[ \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \gamma M ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right) \mid s _ {t} \right]. \tag {2.3}
$$
Because the SR satisfies a Bellman recursion, it can be learned in a similar manner to the value function, with the added benefit that if the transition kernel  $p$  is fixed, it can be used to instantly evaluate a policy on a task determined by any reward vector  $\mathbf{r}$ . The SR can also be conditioned on actions,  $M^{\pi}(s,a,s')$ , in which case multiplication by  $\mathbf{r}$  produces the  $Q$ -values of  $\pi$ . The SR was originally motivated by the idea that a representation of state in the brain should be dependent on the similarity of future paths through the environment, and there is evidence that SR-like representations are present in the hippocampus [14]. A representation closely related to the SR is the first-occupancy representation [FR; 11], which modifies the SR by only counting the first visit to each state:
$$
F ^ {\pi} \left(s, s ^ {\prime}\right) \triangleq \mathbb {E} _ {\pi} \left[ \sum_ {k \geq 0} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}, s ^ {\prime} \notin \left\{s _ {t}, \dots , s _ {t + k - 1} \right\}\right) \mid s _ {t} \right]. \tag {2.4}
$$
The FR can be used in the same way as the SR, with the difference that the values it computes are predicated on ephemeral rewards—those that are consumed or vanish after the first visit to a state.
Policy Composition If the agent has access to a set of policies  $\Pi = \{\pi\}$  and their associated SRs (or FRs) and is faced by a new task determined by reward vector  $\mathbf{r}$ , it can instantly evaluate these policies by simply multiplying:  $\{Q^{\pi}(s,a) = \mathbf{r}^{\top}M^{\pi}(s,a,\cdot)\}$ . This process is termed generalized policy evaluation [GPE; 15]. Similarly, generalized policy improvement (GPI) is defined as the identification of a new policy  $\pi'$  such that  $Q^{\pi'}(s,a) \geq \sup_{\pi \in \Pi} Q^{\pi}(s,a) \forall s, a \in S \times \mathcal{A}$ . Combining both provides a way for an agent to efficiently compose its policies  $\Pi$ —that is, to combine them to produce a new policy without additional learning. This process, which we refer to as GPE+GPI, produces the following policy, which is guaranteed to perform at least as well as any policy in  $\Pi$  [16]:
$$
\pi^ {\prime} (s) \in \underset {a \in \mathcal {A}} {\operatorname {a r g m a x}} \max  _ {\pi \in \Pi} \mathbf {r} ^ {\top} M ^ {\pi} (s, a, \cdot). \tag {2.5}
$$
3 Diminishing Marginal Utility
Problem Statement Motivated by DMU's importance in decision-making, our goal is to understand RL in the context of the following class of non-Markov reward functions:
$$
r _ {\lambda} (s, t) = \lambda (s) ^ {n (s, t)} \bar {r} (s), \quad \lambda (s) \in [ 0, 1 ], \tag {3.1}
$$
where  $n(s,t) \in \mathbb{N}$  is the agent's visit count at  $s$  up to time  $t$  and  $\bar{r}(s)$  describes the reward at the first visit to  $s$ .  $\lambda(s)$  therefore encodes the extent to which reward diminishes after each visit to  $s$ . Note that for  $\lambda(s) = \lambda = 1$  we recover the usual stationary reward given by  $\bar{r}$ , and so this family of rewards strictly generalizes the stationary Markovian rewards typically used in RL.
DMU is Challenging An immediate question when considering reward functions of this form is whether or not we can still define a Bellman equation over the resulting value function. If this is the case, standard RL approaches still apply. However, the following result shows otherwise.
Lemma 3.1 (Impossibility; Informal). Given a reward function of the form Eq. (3.1), it is impossible to define a Bellman equation solely using the resulting value function and immediate reward.
We provide a more precise statement, along with proofs for all theoretical results, in Appendix B. This result means that we can't write the value function corresponding to rewards of the form Eq. (3.1) recursively only in terms of rewards and value in an analogous manner to Eq. (2.1). Nonetheless, we found that it is in fact possible to derive  $a$  recursive relationship in this setting, but only by positing a novel state representation that generalizes the SR and the FR, which we term the  $\lambda$  representation  $(\lambda R)$ . In the following sections, we define the  $\lambda R$ , establish its formal properties, and demonstrate its necessity for RL problems with diminishing rewards.
4 The  $\lambda$  Representation
A Representation for DMU We now the derive the  $\lambda R$  by decomposing the value function for rewards of the form Eq. (3.1) and show that it admits a Bellman recursion. To simplify notation, we
use a single  $\lambda$  for all states, but the results below readily apply to non-uniform  $\lambda$ . We have
$$
V ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {\lambda} \left(s _ {t + k}, k\right) \mid s _ {t} = s \right] = \bar {\mathbf {r}} ^ {\top} \underbrace {\mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} \lambda^ {n _ {t} \left(s _ {t + k} , k\right)} \mathbf {1} \left(s _ {t + k}\right) \mid s _ {t} = s \right]} _ {\triangleq \Phi_ {\lambda} ^ {\pi} (s)}, \tag {4.1}
$$
where we call  $\Phi_{\lambda}^{\pi}(s)$  the  $\lambda$  representation  $(\lambda \mathbb{R})$ , and  $n_t(s,k) \triangleq \sum_{j=0}^{k-1} \mathbb{1}(s_{t+j} = s)$ , is the number of times state  $s$  is visited from time  $t$  up to—but not including—time  $t + k$ . Formally:
Definition 4.1 ( $\lambda$ ). For an MDP with finite  $S$  and  $\lambda \in [0,1]^{|\mathcal{S}|}$ , the  $\lambda$  representation is given by  $\Phi_{\lambda}^{\pi}$  such that
$$
\Phi_ {\lambda} ^ {\pi} \left(s, s ^ {\prime}\right) \triangleq \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \lambda \left(s ^ {\prime}\right) ^ {n _ {t} \left(s ^ {\prime}, k\right)} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \mid s _ {t} = s \right] \tag {4.2}
$$
where  $n_t(s, k) \triangleq \sum_{j=0}^{k-1} \mathbb{1}(s_{t+j} = s)$  is the number of times state  $s$  is visited from time  $t$  until time  $t + k - 1$ .
We can immediately see that for  $\lambda = 0$ , the  $\lambda$ R recovers the FR (we take  $0^0 = 1$ ), and for  $\lambda = 1$ , it recovers the SR. For  $\lambda \in (0,1)$ , the  $\lambda$ R interpolates between the two, with higher values of  $\lambda$  reflecting greater persistence of reward in a given state or state-action pair and lower values of  $\lambda$  reflecting more ephemeral rewards.
The  $\lambda R$  admits a recursive relationship:
$$
\begin{array}{l} \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \lambda^ {n _ {t} \left(s ^ {\prime}, k\right)} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \mid s _ {t} = s \right] \\ \stackrel {(i)} {=} \mathbb {E} \left[ \mathbb {1} (s _ {t} = s ^ {\prime}) + \lambda^ {n _ {t} (s ^ {\prime}, 1)} \gamma \sum_ {k = 1} ^ {\infty} \lambda^ {n _ {t + 1} (s ^ {\prime}, k)} \gamma^ {k - 1} \mathbb {1} (s _ {t + k} = s ^ {\prime}) \Big | s _ {t} = s \right] \\ = \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) \left(1 + \gamma \lambda \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right)\right) + \gamma \left(1 - \mathbb {1} \left(s _ {t} = s ^ {\prime}\right)\right) \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right), \tag {4.3} \\ \end{array}
$$
where  $(i)$  follows from  $n_t(s', k) = n_t(s', 1) + n_{t+1}(s', k-1)$ . A more detailed derivation is provided in Appendix A. Thus, we can define a tractable Bellman operator:
Definition 4.2 (λR Operator). Let  $\Phi \in \mathbb{R}^{|S| \times |S|}$  be an arbitrary real-valued matrix, and let  $\mathcal{G}_{\lambda}^{\pi}$  denote the λR Bellman operator for  $\pi$ , such that
$$
\mathcal {G} _ {\lambda} ^ {\pi} \Phi \triangleq I \odot \left(\mathbf {1 1} ^ {\top} + \gamma \lambda P ^ {\pi} \Phi\right) + \gamma \left(\mathbf {1 1} ^ {\top} - I\right) \odot P ^ {\pi} \Phi , \tag {4.4}
$$
where  $\odot$  denotes elementwise multiplication and  $I$  is the  $|\mathcal{S}| \times |\mathcal{S}|$  identity matrix. In particular, for a stationary policy  $\pi$ ,  $\mathcal{G}_{\lambda}^{\pi}\Phi_{\lambda}^{\pi} = \Phi_{\lambda}^{\pi}$ .
The following result establishes that successive applications of  $\mathcal{G}_{\lambda}^{\pi}$  converge to the  $\lambda R$ .
Proposition 4.1 (Convergence). Under the conditions assumed above, set  $\Phi^{(0)} = (1 - \lambda)I$ . For  $k = 1,2,\ldots$ , suppose that  $\Phi^{(k + 1)} = \mathcal{G}_{\lambda}^{\pi}\Phi^{(k)}$ . Then
$$
\left| \left(\Phi^ {(k)} - \Phi_ {\lambda} ^ {\pi}\right) _ {s, s ^ {\prime}} \right| \leq \frac {\gamma^ {k + 1}}{1 - \lambda \gamma}.
$$
While such analysis is fairly standard in MDP theory, it is noteworthy that the analysis extends to this case despite Lemma 3.1. That is, for a ethologically relevant class of non-Markovian reward functions [17], it is possible to define a Markovian Bellman operator and prove that repeated applications of it converge to the desired representation. Furthermore, unlike in the stationary reward case, where decomposing the value function in terms of the reward and the SR is "optional" to perform prediction or control, in this setting the structure of the problem requires that this representation be learned. To get an intuition for the  $\lambda$ R consider the simple gridworld presented in Fig. 4.1, where we visualize the  $\lambda$ R for varying values of  $\lambda$ . For  $\lambda = 0$ , the representation recovers the FR, encoding the expected discount at first occupancy of each state, while as  $\lambda$  increases, effective occupancy is accumulated accordingly at states which are revisited. We offer a more detailed visualization in Fig. F.2.
Figure 4.1: The  $\lambda$ R interpolates between the FR and the SR. We visualize the  $\lambda$ Rs of the bottom left state for the depicted policy for  $\lambda \in \{0.0, 0.5, 1.0\}$ .
4.1 Continuous State Spaces
When the state space is large or continuous, it becomes impossible to use a tabular representation, and we must turn to function approximation. There are several ways to approach this, each with their own advantages and drawbacks.
Feature-Based Representations For the SR and the FR, compatibility with function approximation is most commonly achieved by simply replacing the indicator functions in their definitions with a base feature function  $\phi : S \mapsto \mathbb{R}^D$  to create successor features [SFs; 15, 16] and first-occupancy features [FFs; 11], respectively. The intuition in this case is that  $\phi$  for the SR and the FR is just a one-hot encoding of the state (or state-action pair), and so for cases when  $|S|$  is too large, we can replace it with a compressed representation. That is, we have the following definition
Definition 4.3 (SFs). Let  $\phi : \mathcal{S} \mapsto \mathbb{R}^D$  be a base feature function. Then, the successor feature (SF) representation  $\varphi_1 : \mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}^D$  is defined as  $\varphi_1^\pi(s, a) \triangleq \mathbb{E}_\pi \left[ \sum_{k=0}^\infty \gamma^k \phi(s_{t+k}) \Big| s_t, a_t \right]$  for all  $s, a \in \mathcal{S} \times \mathcal{A}$ .
One key fact to note, however, is that due to this compression, all notion of state "occupancy" is lost and these representations instead measure feature accumulation. For any feasible  $\lambda$  then, it is most natural to define these representations using their recursive forms:
Definition 4.4 ( $\lambda$ F). For  $\lambda \in [0,1]$  and bounded base features  $\phi : S \mapsto [0,1]^D$ , the  $\lambda$ -feature ( $\lambda$ F) representation of state  $s$  is given by  $\varphi_{\lambda}^{\pi}$  such that
$$
\varphi_ {\lambda} ^ {\pi} (s) \triangleq \phi (s) \odot (1 + \gamma \lambda \mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \varphi_ {\lambda} ^ {\pi} (s ^ {\prime})) + \gamma (1 - \phi (s)) \odot \mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \varphi_ {\lambda} ^ {\pi} (s ^ {\prime}). \tag {4.5}
$$
In order to maintain their usefulness for policy evaluation, the main requirement of the base features is that the reward should lie in their span. That is, a given feature function  $\phi$  is most useful for an associated set of reward functions  $\mathcal{R}$ , given by
$$
\mathcal {R} = \left\{r \mid \exists \mathbf {w} \in \mathbb {R} ^ {D} \text {s . t .} r (s, a) = \mathbf {w} ^ {\top} \phi (s, a) \forall s, a \in \mathcal {S} \times \mathcal {A} \right\}. \tag {4.6}
$$
However, Barreto et al. [18] demonstrate that good performance can still be achieved for an arbitrary reward function as long as it's sufficiently close to some  $r \in \mathcal{R}$ .
Set-Theoretic Formulations As noted above, computing expectations of accumulated abstract features is unsatisfying because it requires that we lose the occupancy-based interpretation of these representations. It also restricts the agent to reward functions which lie within  $\mathcal{R}$ . An alternative approach to extending the SR to continuous MDPs that avoids this issue is the successor measure [SM; 19], which treats the distribution of future states as a measure over  $S$ :
$$
M ^ {\pi} (s, a, X) \triangleq \sum_ {k = 0} ^ {\infty} \gamma^ {k} \mathbb {P} \left(s _ {t + k} \in X \mid s _ {t} = s, a _ {t} = a, \pi\right) \quad \forall X \subset \mathcal {S} \text {m e a s u r a b l e}, \tag {4.7}
$$
which can be expressed in the discrete case as  $M^{\pi} = I + \gamma P^{\pi}M^{\pi}$ . In the continuous case, matrices are replaced by their corresponding measures. Note that SFs can be recovered by integrating:  $\varphi_1^\pi (s,a) = \int_{s'}M^\pi (s,a,ds')\phi (s')$ . We can define an analogous object for the  $\lambda \mathrm{R}$  as follows
$$
\Phi_ {\lambda} ^ {\pi} (s, X) \triangleq \sum_ {k = 0} ^ {\infty} \lambda^ {n _ {t} (X, k)} \gamma^ {k} \mathbb {P} \left(s _ {t + k} \in X \mid s _ {t} = s, \pi\right) \tag {4.8}
$$
Figure 5.1: The  $\lambda$ R is required for accurate policy evaluation. Policy evaluation of the policy depicted in Fig. 4.1 using dynamic programming, tabular TD learning, and  $\lambda$ F TD learning produces the most accurate value estimates when using the  $\lambda$ R with  $\lambda = \lambda_{true}$ . Results are averaged over three random seeds. Shading indicates standard error.
where  $n_t(X, k) \triangleq \sum_{j=0}^{k-1} \delta_{s_{t+j}}(X)$ . However, this is not a measure because it fails to satisfy additivity for  $\lambda < 1$ , i.e., for measurable disjoint sets  $A, B \subseteq S$ ,  $\Phi_\lambda^\pi(s, A \cup B) < \Phi_\lambda^\pi(s, A) + \Phi_\lambda^\pi(s, B)$  (Lemma B.4). For this reason, we call Eq. (4.8) the  $\lambda$  operator ( $\lambda O$ ). We then minimize the following squared Bellman error loss for  $\Phi_\lambda^\pi$  (dropping sub/superscripts for concision):
$$
\begin{array}{l} \mathcal {L} (\Phi) = \mathbb {E} _ {s _ {t}, s _ {t + 1} \sim \rho , s ^ {\prime} \sim \mu} \left[ \left(\varphi \left(s _ {t}, s ^ {\prime}\right) - \gamma \bar {\varphi} \left(s _ {t + 1}, s ^ {\prime}\right)\right) ^ {2} \right] - 2 \mathbb {E} _ {s _ {t} \sim \rho} [ \varphi \left(s _ {t}, s _ {t}\right) ] \tag {4.9} \\ + 2 \gamma (1 - \lambda) \mathbb {E} _ {s _ {t}, s _ {t + 1} \sim \rho} [ \mu (s _ {t}) \varphi (s _ {t}, s _ {t}) \bar {\varphi} (s _ {t + 1}, s _ {t}) ], \\ \end{array}
$$