SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\rho$  and  $\mu$  are densities over  $S$  and  $\Phi_{\lambda}^{\pi}(s) \triangleq \varphi_{\lambda}^{\pi}(s)\mathrm{diag}(\mu)$  in the discrete case, with  $\varphi_{\lambda}^{\pi}$  parametrized by neural network.  $\bar{\cdot}$  indicates a stop-gradient operation, i.e., a target network. A detailed derivation and discussion are given in Appendix H. While  $\rho$  can be any training distribution of transitions we can sample from, we require an analytic expression for  $\mu$ . Eq. (4.9) recovers the SM loss of Touati and Ollivier [12] when  $\lambda = 1$ .
5 Policy Evaluation, Learning, and Composition under DMU
In the following sections, we experimentally validate the formal properties of the  $\lambda R$  and explore its usefulness for solving RL problems with DMU. The majority of our experiments center on navigation tasks, as we believe this is the most natural setting for studying behavior under diminishing rewards. However, in Appendix I we also explore potential for the  $\lambda R$ 's use other areas, such as continuous control, even when rewards do not diminish. There is also the inherent question of whether the agent has access to  $\lambda$ . In a naturalistic context,  $\lambda$  can be seen as an internal variable that the agent likely knows, especially if the agent has experienced the related stimulus before. Therefore, in subsequent experiments, treating  $\lambda$  as a "given" can be taken to imply the agent has prior experience with the relevant stimulus. Further details for all experiments can be found in Appendix E.
5.1 Policy Evaluation
In Section 4, we showed that in order to perform policy evaluation under DMU, an agent is required to learn the  $\lambda$ R. In our first experimental setting, we verify this analysis empirically for the policy depicted in Fig. 4.1 with a rewarded location in the state indicated by a  $g$  in Fig. 5.1 with  $\lambda_{true} = 0.5$ . We then compare the performance for agents using different values of  $\lambda$  across dynamic programming (DP), tabular TD learning, and  $\lambda$ F TD learning with Laplacian features. For the latter two, we use a linear function approximator with a one-hot encoding of the state as the base feature function. We then compute the  $Q$ -values using the  $\lambda$ R with  $\lambda \in \{0.5, 1.0\}$  (with  $\lambda = 1$  corresponding to the SR) and compare the resulting value estimates to the true  $Q$ -values. Consistent with our theoretical analysis, Fig. 5.1 shows that the  $\lambda$ R with  $\lambda = \lambda_{true}$  is required to produce accurate value estimates.
5.2 Policy Learning
To demonstrate that  $\lambda \mathrm{R}$  is useful in supporting policy improvement under diminishing rewards, we implemented modified forms of  $Q$ -learning [20] (which we term  $Q_{\lambda}$ -learning) and advantage
Figure 5.2: The  $\lambda$ R is required for strong performance. We apply a tabular  $Q$ -learning-style algorithm and deep actor-critic algorithm to policy optimization in the TwoRooms domain. The blue locations indicate reward, and the black triangle shows the agent's position. Results are averaged over three random seeds. Shading indicates standard error.
actor-critic [A2C; 21] and applied them to the TwoRooms domain from the NeuroNav benchmark task set [22] (see Fig. 5.2). For a transition  $(s_t, a_t, s_{t+1})$ , we define the following operator:
$$
\mathcal {T} _ {\lambda} ^ {\star} \Phi \triangleq \mathbf {1} \left(s _ {t}\right) \odot (1 + \lambda \gamma \Phi \left(s _ {t + 1}, a _ {t + 1}\right)) + \gamma \left(1 - \mathbf {1} \left(s _ {t}\right)\right) \odot \Phi \left(s _ {t + 1}, a _ {t + 1}\right), \tag {5.1}
$$
where  $a_{t + 1} = \mathrm{argmax}_a Q_\lambda (s_t,a) = \mathrm{argmax}_a \mathbf{r}^\top \Phi (s_t,a)$ . This is an improvement operator for  $Q_{\lambda}$ . The results in Fig. 5.2 show that  $Q_{\lambda}$ -learning outperforms standard  $Q$ -learning ( $\lambda = 1$ ) for diminishing rewards, and that the "correct"  $\lambda$  produces the best performance. To implement A2C with a  $\lambda$ R critic, we modified the standard TD target in a similar manner as follows:
$$
\mathcal {T} _ {\lambda} V \left(s _ {t}\right) = r \left(s _ {t}\right) + \gamma \left(V \left(s _ {t + 1}\right) + (\lambda - 1) \mathbf {w} ^ {\mathrm {T}} \left(\phi \left(s _ {t}\right) \odot \varphi_ {\lambda} \left(s _ {t + 1}\right)\right)\right), \tag {5.2}
$$
where  $\phi$  were one-hot state encodings, and the policy, value function, and  $\lambda \mathbf{F}$  were output heads of a shared LSTM network [23]. Note this target is equivalent to Definition 4.4 multiplied by the reward (derivation in Appendix E). Fig. 5.2 shows again that correct value targets lead to improved performance. Videos of agent behavior can be found at lambdarepresentation.github.io.
5.3 Policy Composition
As we've seen, DMU problems of this form have an interesting property wherein solving one task requires the computation of a representation which on its own is task agnostic. In the same way that the SR and FR facilitate generalization across reward functions, the  $\lambda$ R facilitates generalization across reward functions with different  $r$ s. The following result shows that there is a benefit to having the "correct"  $\lambda$  for a given resource.
Theorem 5.1 (GPI). Let  $\{M_j\}_{j=1}^n \subseteq \mathcal{M}$  and  $M \in \mathcal{M}$  be a set of tasks in an environment  $\mathcal{M}$  and let  $Q^{\pi_j^*}$  denote the action-value function of an optimal policy of  $M_j$  when executed in  $M$ . Assume that the agent uses diminishing rate  $\hat{\lambda}$  that may differ from the true environment diminishing rate  $\lambda$ . Given estimates  $\tilde{Q}^{\pi_j}$  such that  $\|Q^{\pi_j^*} - \tilde{Q}^{\pi_j}\|_{\infty} \leq \epsilon$  for all  $j$ , define
$$
\pi (s)\in \operatorname *{argmax}_{a}\max_{j}\tilde{Q}^{\pi_{j}}(s,a).
$$
$$
Q ^ {\pi} (s, a) \geq \max  _ {j} Q ^ {\pi_ {j} ^ {*}} (s, a) - \frac {1}{1 - \gamma} \left(2 \epsilon + | \lambda - \hat {\lambda} | \| r \| _ {\infty} + \frac {\gamma (1 - \lambda) r (s , a)}{1 - \lambda \gamma}\right).
$$
Note that for  $\lambda = 1$ , we recover the original GPI bound due to Barreto et al. [18] with an additional term quantifying error accrued if incorrectly assuming  $\lambda < 1$ .
Tabular Navigation We can see this result reflected empirically in Fig. 5.3, where we consider the following experimental set-up in the classic FourRooms domain [24]. The agent is assumed to be given or have previously acquired four policies  $\{\pi_0,\pi_1,\pi_2,\pi_3\}$  individually optimized to reach rewards located in each of the four rooms of the environment. There are three reward locations  $\{g_0,g_1,g_2\}$  scattered across the rooms, each with its own initial reward and all with  $\lambda = 0.5$ . At the
Figure 5.3: Tabular GPI. (Left) Average returns obtained by agents performing GPE+GPI using  $\lambda$ Rs with  $\lambda \in \{0.0, 0.5, 1.0\}$  over 50 episodes. Error bars indicate standard error. (Right) Sample trajectories. Agents with  $\lambda$  set too high overstay in rewarding states, and those with  $\lambda$  too low leave too early.
Figure 5.4: Pixel-Based GPI. Performance is strongest for agents using the correct  $\lambda = 0.5$ . PCA on the learned features in each underlying environment state shows that the  $\lambda$ Fs capture the value-conditioned structure of the environment.
beginning of each episode, an initial state  $s_0$  is sampled uniformly from the set of available states. An episode terminates either when the maximum reward remaining in any of the goal states is less than 0.1 or when the maximum number of steps  $H = 40$  is reached (when  $\lambda = 1$ , the latter is the only applicable condition). For each of the four policies, we learn  $\lambda \mathrm{Rs}$  with  $\lambda \in \{0.0, 0.5, 1.0\}$  using dynamic programming and record the returns obtained while performing GPE+GPI with each of these representations over 50 episodes. Bellman error curves for the  $\lambda \mathrm{Rs}$  are shown in Fig. F.3, and demonstrate that convergence is faster for lower  $\lambda$ . In the left panel of Fig. 5.3, we can indeed see that using the correct  $\lambda$  (0.5) nets the highest returns. Example trajectories for each agent  $\lambda$  are shown in the remaining panels.
Pixel-Based Navigation We verified that the previous result is reflected in larger scales by repeating the experiment in a partially-observed version of FourRooms in which the agent receives  $128 \times 128$  RGB egocentric observations of the environment (Fig. 5.4, left) with  $H = 50$ . In this case, the agent learns  $\lambda \mathrm{Fs}$  for each policy for  $\lambda \in \{0.0, 0.5, 1.0\}$ , where each  $\lambda \mathrm{F}$  is parameterized by a feedforward convolutional network with the last seven previous frames stacked to account for the partial observability. The base features were Laplacian eigenfunctions normalized to  $[0, 1]$ , which were shown by Touati et al. [25] to perform the best of all base features for SFs across a range of environments including navigation.
6 Understanding Natural Behavior
Naturalistic environments often exhibit diminishing reward and give insight into animal behavior. The problem of foraging in an environment with multiple diminishing food patches (i.e., reward states) is of interest in behavioral science [26-28]. The cornerstone of foraging theory is the marginal value theorem [MVT; 28, 29], which states that the optimal time to leave a patch is when the patch's reward rate matches the average reward rate of the environment. However, the MVT does not describe which patch to move to once an agent leaves its current patch. We show that the  $\lambda$ O recovers MVT-like behavior in discrete environments and improves upon the MVT by not only predicting when agents should leave rewarding patches, but also where they should go. Moreover, we provide a scheme for learning  $\lambda$  alongside the  $\lambda$ O using feedback from the environment.
b
Figure 6.1:  $\lambda$ O trained via FB. a)  $\lambda$  values of two agents in FourRooms, one which learns  $\lambda$  and one which does not. b) Performance of the two agents from (a). Learning  $\lambda$  improves performance. c) Reward structure and starting state of the asymmetric environment. d) Trajectory of an agent with  $\lambda = 1$ . The optimal strategy is to reach the high reward state and exploit it ad infinitum. e) Trajectory of an agent with  $\lambda = 0.1$ . The optimal strategy is to exploit each reward state for a few time steps before moving to the next reward state.
To learn the  $\lambda$ O, we take inspiration from the FBR [12] and use the following parametrization:  $\Phi_{\lambda}^{\pi_z}(s,a,s') = F(s,a,z)^\top B(s')\mu (s')$  and  $\pi_z(s) = \operatorname{argmax}_a F(s,a,z)^\top z$ , where  $\mu$  is a density with full support on  $S$ , e.g., uniform. We then optimize using the loss in Eq. (4.9) under this parameterization (details in Appendix E). Given a reward function  $r:\mathcal{S}\to \mathbb{R}$  at evaluation time, the agent acts according to  $\operatorname{argmax}_a F(s,a,z_R)^\top z_R$ , where  $z_{R} = \mathbb{E}_{s\sim \mu}[r(s)B(s)]$ . Because the environment is non-stationary,  $z_{R}$  has to be re-computed at every time step. To emulate a more realistic foraging task, the agent learns  $\lambda$  by minimizing the loss in Eq. (H.1) in parallel with the loss
$$
\mathcal {L} (\lambda) = \mathbb {E} _ {s _ {t}, s _ {t + 1} \sim \rho} \left[ \mathbb {1} \left(s _ {t} = s _ {t + 1}\right) \left(r \left(s _ {t + 1}\right) - \lambda r (s _ {t})\right) ^ {2} \right],
$$
which provably recovers the correct value of  $\lambda$  provided that  $\rho$  is sufficiently exploratory. In Appendix H.1 we provide experiments showing that using an incorrect value of  $\lambda$  leads to poor performance on tabular tasks. In Fig. 6.1 we show that the agent learns the correct value of  $\lambda$ , increasing its performance. We illustrate the behavior of the  $\lambda O$  in an asymmetric environment that has one large reward state on one side and many small reward states (with higher total reward) on the other. Different values of  $\lambda$  lead to very different optimal foraging strategies, which the  $\lambda O$  recovers and exhibits MVT-like behavior (see Appendix H.2 for a more detailed analysis). Our hope is that the  $\lambda R$  may provide a framework for new theoretical studies of foraging behavior and possibly mechanisms for posing new hypotheses. For example, an overly large  $\lambda$  may lead to overstaying in depleted patches, a frequently observed phenomenon [30].
Limitations Despite its advantages, there are several drawbacks to the representation which are a direct result of the challenge of the DMU setting. First, the  $\lambda$ R is only useful for transfer across diminishing reward functions when the value of  $\lambda$  at each state is consistent across tasks. In natural settings, this is fairly reasonable, as  $\lambda$  can be thought of as encoding the type of the resource available at each state (i.e., each resource has its own associated decay rate). Second, as noted in Section 4.1, the  $\lambda$ R does not admit a measure-theoretic formulation, which makes it challenging to define a principled, occupancy-based version compatible with continuous state spaces. Third, the  $\lambda$ R is a prospective representation, and so while it is used to correctly evaluate a policy's future return under DMU, it is not inherently memory-based and so performs this evaluation as if the agent hasn't visited locations with diminishing reward before. Additional mechanisms (i.e., recurrence or frame-stacking) are necessary to account for previous visits. Finally, the  $\lambda$ R is dependent on an episodic task setting for rewards to reset, as otherwise the agent would eventually consume all reward in the environment. An even more natural reward structure would include a mechanism for reward replenishment in addition to depletion. We describe several such candidates in Appendix J, but leave a more thorough investigation of this issue to future work.
In this work, we aimed to lay the groundwork for understanding policy evaluation, learning, and composition under diminishing rewards. To solve such problems, we introduced—and showed the necessity of—the  $\lambda$  representation, which generalizes the SR and FR. We demonstrated its usefulness for rapid policy evaluation and by extension, composition, as well as control. We believe the  $\lambda R$  represents a useful step in the development of state representations for naturalistic environments.
[36] Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568, 2019.
[37] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[38] Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16, pages 317-328. Springer, 2005.
[39] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[40] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, 2018.
[41] Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton. Efficient wasserstein natural gradients for reinforcement learning, 2020.
[42] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.
[43] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[44] Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:12849-12863, 2021.
[45] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Appendix: A State Representation for Diminishing Rewards
GIFs of navigation agents can be found at lambdarepresentation.github.io and in the supplementary material.
A Derivation of  $\lambda \mathbb{R}$  Recursion
We provide a step-by-step derivation of the  $\lambda \mathbb{R}$  recursion in Eq. (4.3):
$$
\begin{array}{l} \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \lambda^ {n _ {t} (s ^ {\prime}, k)} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \Big | s _ {t} = s \right] \\ = \mathbb {E} \left[ \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \sum_ {k = 1} ^ {\infty} \lambda^ {n _ {t} \left(s ^ {\prime}, k\right)} \gamma^ {k} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \mid s _ {t} = s \right] \\ \stackrel {(i)} {=} \mathbb {E} \left[ \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \lambda^ {n _ {t} \left(s ^ {\prime}, 1\right)} \gamma \sum_ {k = 1} ^ {\infty} \lambda^ {n _ {t + 1} \left(s ^ {\prime}, k\right)} \gamma^ {k - 1} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \Big | s _ {t} = s \right] \\ \stackrel {(i i)} {=} \mathbb {E} \left[ \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) \lambda \gamma \sum_ {k = 1} ^ {\infty} \lambda^ {n _ {t + 1} \left(s ^ {\prime}, k\right)} \gamma^ {k - 1} \mathbb {1} \left(s _ {t + k} = s ^ {\prime}\right) \right. \\ + \gamma (1 - \mathbb {1} (s _ {t} = s ^ {\prime})) \sum_ {k = 1} ^ {\infty} \lambda^ {n _ {t + 1} (s ^ {\prime}, k)} \gamma^ {k - 1} \mathbb {1} (s _ {t + k} = s ^ {\prime}) \Big | s _ {t} = s \Big ] \\ = \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) + \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) \lambda \gamma \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right) \\ + \gamma (1 - \mathbb {1} (s _ {t} = s ^ {\prime})) \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} (s _ {t + 1}, s ^ {\prime}) \\ = \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) \left(1 + \gamma \lambda \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right)\right) \\ + \gamma (1 - \mathbb {1} \left(s _ {t} = s ^ {\prime}\right)) \mathbb {E} _ {s _ {t + 1} \sim p ^ {\pi}} \Phi_ {\lambda} ^ {\pi} \left(s _ {t + 1}, s ^ {\prime}\right), \tag {A.1} \\ \end{array}
$$
where  $(i)$  is because  $n_t(s', k) = n_t(s', 1) + n_{t+1}(s', k)$  and  $(ii)$  is because
$$
\lambda^ {n _ {t} (s ^ {\prime}, 1)} = \lambda^ {\mathbb {1} (s _ {t} = s ^ {\prime})} = \mathbb {1} \left(s _ {t} = s ^ {\prime}\right) \lambda + \left(1 - \mathbb {1} \left(s _ {t} = s ^ {\prime}\right)\right).
$$
B Theoretical Analysis
Here, we provide proofs for the theoretical results in the main text.
Before providing the formal statement and proof for Lemma 3.1, we introduce a definition for a Bellman operator.
Definition B.1 (Bellman Operator). A Bellman operator is a contractive operator  $\mathbb{R}^{|S|}\to \mathbb{R}^{|S|}$  that depends solely on  $\overline{\mathbf{r}}$ , one-step expectations under  $p^\pi$ , and learning hyperparameters (in our case  $\gamma$  and  $\lambda$ ).
We can now give a formal statement of Lemma 3.1:
Lemma B.1 (Impossibility; Formal). Assume that  $|S| > 1$ . Then, there does not exist a Bellman operator  $T$  with fixed point  $V^{\pi}$ .
Proof. Assume for a contradiction that  $T$  is a Bellman operator. By the Banach fixed-point theorem,  $V^{\pi}$  must be the unique fixed point of  $T$ . Hence,  $TV^{\pi}$  must take on the following form (see the proof of Lemma 3.1 in Appendix B): for  $s \in S$ ,
$$
(T V ^ {\pi}) (s) = \bar {r} (s) + \gamma \mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} V ^ {\pi} (s ^ {\prime}) + \gamma (\lambda - 1) \bar {r} (s) \mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \Phi_ {\lambda} ^ {\pi} (s ^ {\prime}, s).
$$
For the assumption to hold, there must exist a function  $f$  such that, for any  $s \in S$ ,
$$
\mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \Phi_ {\lambda} ^ {\pi} (s ^ {\prime}, s) = \mathbb {E} _ {p ^ {\pi} (\cdot | s)} f (\bar {\mathbf {r}}, \mathbf {V} ^ {\pi}, \gamma , \lambda , s).
$$
Now, by definition,
$$
V ^ {\pi} (s) = \sum_ {s ^ {\prime} \in \mathcal {S}} \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) \bar {r} (s ^ {\prime}).
$$
$\overline{\mathbf{r}}$  is a vector in  $\mathbb{R}^{|S|}$ , so as long as  $S > 1$ ,  $\overline{\mathbf{r}}^{\perp}$  is non-trivial. Fix any  $\overline{\mathbf{r}}, \mathbf{V}^{\pi}, s$ . Pick a vector  $\mathbf{w} \in \overline{\mathbf{r}}^{\perp} \setminus \{\mathbf{0}\}$  and define
$$
\tilde {\Phi} _ {\lambda} ^ {\pi} (s, s ^ {\prime}) := \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) + w (s ^ {\prime})
$$
for any  $s, s' \in S$ . Note that
$$
\sum_ {s ^ {\prime} \in \mathcal {S}} \tilde {\Phi} _ {\lambda} ^ {\pi} (s, s ^ {\prime}) \bar {r} (s ^ {\prime}) = \sum_ {s ^ {\prime} \in \mathcal {S}} \Phi_ {\lambda} ^ {\pi} (s, s ^ {\prime}) \bar {r} (s ^ {\prime}) + \sum_ {s ^ {\prime} \in \mathcal {S}} w (s ^ {\prime}) \bar {r} (s ^ {\prime}) = V ^ {\pi} (s),
$$
as  $\mathbf{w} \perp \overline{\mathbf{r}}$ . However,
$$
\mathbb {E} _ {s ^ {\prime} \sim p ^ {\pi} (\cdot | s)} \tilde {\Phi} _ {\lambda} ^ {\pi} (s ^ {\prime}, s) = \sum_ {s ^ {\prime} \in \mathcal {S}} p ^ {\pi} (s ^ {\prime} | s) \Phi_ {\lambda} ^ {\pi} (s ^ {\prime}, s) + \sum_ {s ^ {\prime} \in \mathcal {S}} p ^ {\pi} (s ^ {\prime} | s) w (s).
$$
The final term evaluates to  $w(s)$ . Because  $\mathbf{w} \neq \mathbf{0}$ , there must exist some  $s$  such that  $w(s) \neq 0$ . For this  $s$ , we have a single input  $(\overline{\mathbf{r}}, \mathbf{V}^{\pi}, \gamma, \lambda, s)$  to  $f$  that corresponds to two distinct outputs:  $\mathbb{E}_{s' \sim p^{\pi}(\cdot|s)} \Phi_{\lambda}^{\pi}(s', s)$  and  $\mathbb{E}_{s' \sim p^{\pi}(\cdot|s)} \tilde{\Phi}_{\lambda}^{\pi}(s', s)$ .
Hence,  $f$  is a one-to-many mapping: for fixed input, there is more than one output. Therefore,  $f$  is not a function, yielding a contradiction.
The following establishes  $\mathcal{G}_{\lambda}^{\pi}$  as a contraction.
Lemma B.2 (Contraction). Let  $\mathcal{G}_{\lambda}^{\pi}$  be the operator as defined in Definition 4.2 for some stationary policy  $\pi$ . Then for any two matrices  $\Phi, \Phi' \in \mathbb{R}^{|\mathcal{S}| \times |\mathcal{S}|}$ ,
$$
\left| \mathcal {G} _ {\lambda} ^ {\pi} \Phi (s, s ^ {\prime}) - \mathcal {G} _ {\lambda} ^ {\pi} \Phi^ {\prime} (s, s ^ {\prime}) \right| \leq \gamma \left| \Phi (s, s ^ {\prime}) - \Phi^ {\prime} (s, s ^ {\prime}) \right|.
$$
$$
\left| \left(\mathcal {G} _ {\lambda} ^ {\pi} \Phi - \mathcal {G} _ {\lambda} ^ {\pi} \Phi^ {\prime}\right) _ {s, s ^ {\prime}} \right| = \left| (I \odot (\mathbf {1 1} ^ {\top} + \gamma \lambda P ^ {\pi} \Phi) + \gamma (\mathbf {1 1} ^ {\top} - I) \odot P ^ {\pi} \Phi \right.
$$
$$
- I \odot \left(\mathbf {1 1} ^ {\top} + \gamma \lambda P ^ {\pi} \Phi^ {\prime}\right) - \gamma \left(\mathbf {1 1} ^ {\top} - I\right) \odot P ^ {\pi} \Phi^ {\prime}) _ {s, s ^ {\prime}}
$$
$$
\begin{array}{l} = \left| \left(I \odot \gamma \lambda P ^ {\pi} \left(\Phi - \Phi^ {\prime}\right) + \gamma \left(\mathbf {1 1} ^ {\top} - I\right) \odot P ^ {\pi} \left(\Phi - \Phi^ {\prime}\right)\right) _ {s, s ^ {\prime}} \right| \\ = \left| \left(\left(I \odot \lambda \mathbf {1 1} ^ {\top} + \mathbf {1 1} ^ {\top} - I\right) \odot \gamma P ^ {\pi} \left(\Phi - \Phi^ {\prime}\right)\right) _ {s, s ^ {\prime}} \right| \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \left| \left(\gamma P ^ {\pi} \left(\Phi - \Phi^ {\prime}\right)\right) _ {s, s ^ {\prime}} \right| \\ = \gamma | \left(P ^ {\pi} \left(\Phi - \Phi^ {\prime}\right)\right) _ {s, s ^ {\prime}} | \\ \leq \gamma | (\Phi - \Phi^ {\prime}) _ {s, s ^ {\prime}} |, \\ \end{array}
$$