SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3 Scaling by a constant factor allows us to show the corresponding quadratic trends.
exists and allows for the planning problem to be solved efficiently [Howard and Matheson, 1972]:
$$
\begin{array}{l} Q _ {h} ^ {\lambda} (x, a) = U _ {\exp} ^ {\lambda} \left(R _ {h} (x, a)\right) + \frac {1}{\lambda} \log \left[ \sum_ {x ^ {\prime}} p _ {h} \left(x, a, x ^ {\prime}\right) \exp \left(\lambda \max  _ {a ^ {\prime}} Q _ {h + 1} ^ {\lambda} \left(x ^ {\prime}, a ^ {\prime}\right)\right) \right] \tag {8} \\ \text {w i t h} Q _ {H + 1} ^ {\lambda} (x, a) = 0. \\ \end{array}
$$
However, the question of the existence of Optimal Bellman Equations for non-utility functionals remains open (e.g. quantiles). More generally, an efficient planning strategy is not known. To address these questions, we consider the most general framework, DistRL, and recall the theoretical DP equations for any statistical functional  $s$  in the Pseudo $^4$ -Algorithm 2. DistRL offers the most comprehensive, or 'lossless', approach, so if a statistical functional cannot be optimized with Alg. 2, then there cannot exist a Bellman Operator to perform exact planning.
Algorithm 2 Pseudo-Algorithm: Exact Planning with Distributional RL
1: Input: model  $p$ , reward  $R$ , statistical functional  $s$
2: Data:  $\eta \in \mathbb{R}^{H|\mathcal{X}||\mathcal{A}|N}, \nu \in \mathbb{R}^{H|\mathcal{X}|N}$
3:  $\forall x\in \mathcal{X}$  ，  $\nu_{H + 1}^{x} = \delta_{0}$
4: for  $h = H \rightarrow 1$  do
5:  $\eta_h(x,a) = \varrho_h^{(x,a)}*\sum_{x'}p_h^a (x,x')\nu_{h + 1}^{x'}\quad \forall x,a\in \mathcal{X}\times \mathcal{A}$
6:  $\nu_{h}^{x} = \eta_{h}(x,a^{*})$  ，  $a^{\ast}\in \arg \max_{a}s(\eta_{h}(x,a))\quad \forall x\in \mathcal{X}$
7: end for
8: Output:  $\eta_h(x, a) \forall x, a, h$
We formalize this idea with the new concept of Bellman Optimizable statistical functionals:
Definition 2 (Bellman Optimizable statistical functional). A statistical functional  $s$  is called Bellman optimal if, for any MDP  $\mathcal{M}$ , the Pseudo-Algorithm 2 outputs an optimal return distribution  $\eta = \eta^{*}$  that verifies:
$$
\forall x, a, h, \quad s \left(\eta_ {h} ^ {*} (x, a)\right) = \sup  _ {\pi} s \left(\eta_ {h} ^ {\pi} (x, a)\right). \tag {9}
$$
Remark. This definition is equivalent to the satisfaction of an Optimal Distributional Bellman equation. Indeed, a statistical functional  $s$  is Bellman Optimizable if and only if, for any  $(\mathcal{X},\mathcal{A},\varrho ,p,\eta)$ $s$  verifies
$$
\sup  _ {a _ {x ^ {\prime}}} s \left(\varrho * \sum_ {x ^ {\prime}} p \left(x ^ {\prime}\right) \eta \left(x ^ {\prime}, a _ {x ^ {\prime}}\right)\right) = s \left(\varrho * \sum_ {x ^ {\prime}} p \left(x ^ {\prime}\right) \eta \left(x ^ {\prime}, a _ {x ^ {\prime}} ^ {*}\right)\right)
$$
with  $a_{x}^{\star}\in \arg \max_{a}s(\eta (x,a))$
We can now state our main results that characterizes all the Bellman estimizable statistical functionals. First, we prove that such a functional must satisfy two important properties.
Lemma 2. A Bellman estimizable functional  $s$  satisfies the two following properties:
- Independence Property: If  $\nu_{1},\nu_{2}\in \mathcal{P}(\mathbb{R})$  are such that  $s(\nu_1)\geq s(\nu_2)$  , then
$$
\forall \nu_ {3} \in \mathscr {P} (\mathbb {R}), \forall \lambda \in [ 0, 1 ], \quad s (\lambda \nu_ {1} + (1 - \lambda) \nu_ {3}) \geq s (\lambda \nu_ {2} + (1 - \lambda) \nu_ {3})) .
$$
- Translation Property: Let  $\tau_{c}$  denote the translation on the set of distributions:  $\tau_{c}\delta_{x} = \delta_{x + c}$ . If  $\nu_{1},\nu_{2}\in \mathcal{P}(\mathbb{R})$  are such that  $s(\nu_1)\geq s(\nu_2)$ , then
$$
\forall c \in \mathbb {R}, \quad s \left(\tau_ {c} \nu_ {1}\right) \geq s \left(\tau_ {c} \nu_ {2}\right).
$$
Indeed, the expectation and the exponential utility both satisfy these properties (see Appendix A.2). Each property is implied by an aspect of the Distributional Bellman Equation (Alg. 2, line 5) and the proof (in Appendix E) unveils these important consequences of the recursion identities.
This theoretical algorithm handles the full distribution of the return at each step, which cannot be done in practice.
Fundamentally, they follow from the Markovian nature of policies optimized this way: the choice of the action in each state should be independent of other states and rely only on the knowledge of the next-state value distribution.
The Independence property states that, for Bellman estimizable functionals, the value of each next state should not depend on that of any other value in the convex combination in the rightmost term of the convolution. In turn, the Translation property is associated to the leftmost term, the reward, and it imposes that, for Bellman estimizable functionals, the decision on the best action is independent of the previously accumulated reward.
The Independence property is related to expected utilities [von Neumann et al., 1944]. Any expected utility verifies this property (Appendix A.2) but, most importantly, the Expected Utility Theorem (also known as the Von Neumann Morgenstein theorem) implies that any continuous statistical functional  $s$  verifying the Independence property can be reduced to an expected utility. This means that for any such statistical functional  $s$ , there exists  $f$  continuous such that  $\forall \nu_{1}, \nu_{2} \in \mathcal{P}(\mathbb{R})$ , we have  $s(\nu_{1}) > s(\nu_{2}) \Longleftrightarrow U_{f}(\nu_{1}) > U_{f}(\nu_{2})$  [von Neumann et al., 1944, Grandmont, 1972].
This result directly narrows down the family of Bellman estimizable functionals to utilities. Indeed, although other functionals might potentially be optimized using the Bellman equation, addressing the problem on utilities is adequate to characterize all possible behaviors. For instance, moment-optimal policies that can be found through dynamic programming, can also be found by optimizing an exponential utility function. The next task is therefore to identify all the utilities that satisfy the second property. We demonstrate that, apart from the mean and exponential utilities, no other  $W_{1}$ -continuous functional satisfies this property.
Theorem 2. Let  $\varrho$  be a return distribution. The only  $W_{1}$ -continuous Bellman Optimizable statistical functionals of the cumulated return are exponential utilities  $U_{\mathrm{exp}}(\varrho) = \frac{1}{\lambda} \log \mathbb{E}_{\varrho}[\exp (\lambda R)]$  for  $\lambda \in \mathbb{R}$ , with the special case of the expectation  $\mathbb{E}_{\varrho}[R]$  when  $\lambda = 0$ .
Of course, if  $U(\rho)$  is a utility and  $\psi$  is a monotonous scalar mapping,  $\psi(U(\rho))$  is an equivalent utility: one should understand in the previous theorem that a  $W_{1}$ -continuous Bellman Optimizable statistical functional is equivalent to  $U_{\exp}(\varrho)$  for some  $\lambda \in \mathbb{R}$ . We chose here to define  $U_{\exp}(\varrho) = \frac{1}{\lambda} \log \mathbb{E}_{\varrho}[\exp(\lambda R)]$  with the log and the factor  $1 / \lambda$  since it results in a normalized utility that tends to the expectation when  $\lambda$  goes to 0. The full proof of Theorem 2 is provided in Appendix E.
We make a few important observations. First, this result shows that algorithms using Bellman updates to optimize any continuous functionals other than the exponential utility cannot guarantee optimality. The theorem does not apply to non-continuous functionals, but Lemma 2 still does. For instance, the quantiles are not  $W_{1}$ -continuous so Theorem 2 does not apply, but it is easy to prove that they do not verify the Independence Property and thus are not Bellman Optimizable. Also, there might also exist other optimaizable functionals, like moments, but they must first be reduced to exponential or linear utilities.
Most importantly, while in theory, DistRL provides the most general framework for optimizing policies via dynamic programming, our result shows that in fact, the only utilities that can be exactly and efficiently optimized do not require to resort to DistRL. This certainly does not question the very purpose of DistRL, which has been shown to play important roles in practice to regularize or stabilize policies and to perform deeper exploration [Bellemare et al., 2017, Hessel et al., 2018]. Some advantages of learning the distribution lies in the enhanced robustness offered in the richer information learned [Rowland et al., 2023], particularly when utilizing neural networks for function approximation [Dabney et al., 2018b, Barth-Maron et al., 2018, Lyle et al., 2019].
5 Q-Learning Exponential Utilities
The previous sections consider the model, i.e. the reward and transition functions, be known. Yet in most practical situations, those are either approximated or learnt<sup>5</sup>. After addressing policy evaluation (Section 3) and planning (Section 4), we conclude here the argument of this paper by addressing the question of learning the statistical functionals of Theorem 2. In fact, we simply highlight a lesser known version of Q-Learning Watkins and Dayan [1992] that extend this celebrated algorithm to exponential utilities. We provide the pseudo-code for the algorithm proposed by Borkar
$^{5}$ Either explicitly (model-based RL) or implicitly (model-free RL, considered here).
Algorithm 3 Q-Learning for Linear and Exponential Utilities
1: Input:  $(\alpha_{t})_{t\in \mathbb{N}}$  transition and reward generator.  $Q_{h}(x,a)\gets H$ $\forall (x,a,h)\in \mathcal{X}\times \mathcal{A}\times [H]$    
2: Utilities: Linear  $(Z\mapsto \lambda Z + b)$  or Exponential  $(Z\mapsto \log (\mathbb{E}\exp (\lambda Z)) / \lambda)$    
3: for episode  $K = 1,\ldots ,K$  do   
4: Observe  $x_{1}\in \mathcal{X}$    
5: for step  $h = 1,\dots ,H$  do   
6: Choose action  $a_{h}\in \arg \max_{a\in \mathcal{A}}Q_{h}(x_{h},a)$    
7: Observe reward  $r_h$  and transition  $x_{h + 1}$  and update for chosen objective:   
8: Linear Util.:  $Q_{h}(x_{h},a_{h})\gets (1 - \alpha_{k})Q_{h}(x_{h},a_{h}) + \alpha_{k}[\lambda (r_{h} + \max_{a^{\prime}}Q_{h + 1}(x_{h + 1},a^{\prime})) + b]$    
9: Exponential Util.:  $Q_{h}(x_{h},a_{h})\gets \frac{1}{\lambda}\log \left[(1 - \alpha_{k})e^{\lambda .Q_{h}(x_{h},a_{h})} + \alpha_{k}e^{\lambda [r_{h} + \max_{a^{\prime}}Q_{h}(x_{h + 1},a^{\prime})]}\right]$    
10: end for   
11: end for   
12: Output:  $Q_{h}(x,a)\forall x,a$
[2002, 2010] with the relevant utility-based updates in Alg. 3. We refer to these seminal works for convergence proofs. Linear utility updates (line 8) differ only slightly from classical ones for expected return optimization, which have been shown to lead to the optimal value asymptotically [Watkins and Dayan, 1992].
6 Discussions and Related Work
The Discounted Framework We focused in this article on undiscounted MDPs, and it is important to note that the results differ for discounted scenarios. The crucial difference is that the family of exponential utilities no longer retains Bellman Closed or Bellman Optimizable properties due to the introduction of the discount factor  $\gamma$  [Rowland et al., 2019]. When it comes to Bellman Optimization, the necessary translation property becomes an affine property:  $\forall c,\gamma$ ,  $s(\tau_c^\gamma \nu_1) > s(\tau_c^\gamma \nu_2)$  where  $\tau_c^\gamma$  is the affine operator such that  $\tau_c^\gamma \delta_x = \delta_{\gamma x + c}$ . This property is not upheld by the exponential utility. Nonetheless, there exists a method to optimize the exponential utility through dynamic programming in discounted MDPs [Chung and Sobel, 1987]. This approach requires modifying the functional to optimize at each step (the step  $h$  is optimized with the utility  $x\mapsto \exp (\gamma^{-h}\lambda x)$ ), but it also implies a loss of policy stationarity, property usually obtained in dynamic programming for discounted finite-horizon MDPs [Sutton and Barto, 2018].
Utilizing functionals to optimize expected return. DistRL has also been used in Deep Reinforcement learning to optimize non-Bellman-optimizable functionals such as distorted means[Ma et al., 2020, Dabney et al., 2018a]. While, as we proved so, such algorithms cannot lead to optimal policies in terms of these functionals, experiments show that in some contexts they can lead to better expected return and faster convergence in practice. The change of functional can be interpreted as a change in the exploration process, and the resulting risk-sensitive behaviors seem to be relevant in adequate environments.
Dynamic programming for the optimization of other functionals To optimize other statistical functionals such as CVaR and other utilities such as moments with Dynamic Programming, Bäuerle and Ott [2011] and Bäuerle and Rieder [2014] propose to extend the state space of the original MDP to  $\mathcal{X}' = \mathcal{X} \times \mathbb{R}$  by theoretically adding a continuous dimension to store the current cumulative rewards. This idea does not contradict our results, and the resulting algorithms remain empirically much more expensive.
Another recent thread of ideas to optimize functionals of the reward revolve around the dual formulation of RL through the empirical state distribution [Hazan et al., 2019]. Algorithms can be derived by noticing that utilities like the CVar are equivalent to solving a convex RL problem [Mutti et al., 2023].
Our work closes an important open problem in the theory of MDPs: we exactly characterize the families of statistical functionals that can be evaluated and optimized by dynamic programming.
We also put into perspective the DistRL framework: the only functionals of the return that can be optimized with DistRL can actually be handled exactly by dynamic programming. Its benefit lies elsewhere, and notably in the improved stability of behavioral properties it allows. We believe that, by narrowing down the avenues to explain its empirical successes, our work can contribute to clarify the further research to conduct on the theory of DistRL.
Alexandre Marthe and Aurélien Garivier acknowledge the support of the Project IDEXLYON of the University of Lyon, in the framework of the Programme Investissements d'Avenir (ANR-16-IDEX-0005), Chaire SeqALO (ANR-20-CHIA-0020-01), and Project FOUNDRY in PEPR-IA.
Claire Vernade is funded by the Deutsche Forschungsgemeinschaft (DFG) under both the project 468806714 of the Emmy Noether Programme and under Germany's Excellence Strategy - EXC number 2064/1 - Project number 390727645. Claire Vernade also thanks the international Max Planck Research School for Intelligent Systems (IMPRS-IS) and Seek.AI for their support.
M. Achab and G. Neu. Robustness and risk management via distributional dynamic programming, Dec. 2021. arXiv:2112.15430 [cs, math].
P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.
M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pages 449-458. PMLR, 2017.
M. G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http://www.distributional-rl.org.
J. Bernhard, S. Pollok, and A. Knoll. Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 2148-2155, 2019. doi: 10.1109/IVS.2019.8813791.
V. S. Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2): 294-311, 2002.
V. S. Borkar. Learning algorithms for risk-sensitive control. In Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems-MTNS, 2010.
N. Bäuerle and J. Ott. Markov Decision Processes with Average-Value-at-Risk criteria. Mathematical Methods of Operations Research, 74(3):361-379, Dec. 2011. ISSN 1432-2994, 1432-5217. doi: 10.1007/s00186-011-0367-0.
N. Bäuerle and U. Rieder. More Risk-Sensitive Markov Decision Processes. Mathematics of Operations Research, 39(1):105-120, Feb. 2014. ISSN 0364-765X. doi: 10.1287/moor.2013.0601. Publisher: INFORMS.
K.-J. Chung and M. J. Sobel. Discounted MDP's: Distribution Functions and Exponential Utility Maximization. SIAM Journal on Control and Optimization, 25(1):49-62, Jan. 1987. ISSN 0363-0129, 1095-7138. doi: 10.1137/0325004.
W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 1096-1105. PMLR, July 2018a. ISSN: 2640-3498.
W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional Reinforcement Learning With Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018b. ISSN 2374-3468. Number: 1.
M. Rowland, M. Bellemare, W. Dabney, R. Munos, and Y. W. Teh. An Analysis of Categorical Distributional Reinforcement Learning. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, pages 29-37. PMLR, Mar. 2018. ISSN: 2640-3498.
M. Rowland, R. Dadashi, S. Kumar, R. Munos, M. G. Bellemare, and W. Dabney. Statistics and Samples in Distributional Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, pages 5528-5536. PMLR, May 2019. ISSN: 2640-3498.
M. Rowland, Y. Tang, C. Lyle, R. Munos, M. G. Bellemare, and W. Dabney. The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation. 2023.
Y. Shen, M. J. Tobia, T. Sommer, and K. Obermayer. Risk-sensitive reinforcement learning. Neural Computation, 26(7):1298-1328, 2014. doi: 10.1162/NECO_a_00600.
M. J. Sobel. The variance of discounted Markov decision processes. Journal of Applied Probability, 19(4):794-802, Dec. 1982. ISSN 0021-9002, 1475-6072. doi: 10.2307/3213832.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
C. Szepesvári. Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1):1-103, Jan. 2010. ISSN 1939-4608, 1939-4616. doi: 10.2200/S00268ED1V01Y201005AIM009.
C. Villani. Topics in Optimal Transportation. Graduate studies in mathematics. American Mathematical Society, 2003. ISBN 9780821833124. URL https://books.google.fr/books?id=idyFAwAAQBAJ.
A. A. Volk, R. W. Epps, D. T. Yonemoto, B. S. Masters, F. N. Castellano, K. G. Reyes, and M. Abolhasani. Alphaflow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning. Nature Communications, 14(1):1403, 2023.
J. von Neumann, O. Morgenstern, and A. Rubinstein. Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press, 1944. ISBN 978-0-691-13061-3.
C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8:279-292, 1992.
P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. Lin, P. MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, R. Douglas, D. Whitehead, P. Durr, P. Stone, M. Spranger, and H. Kitano. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602(7896):223-228, Feb. 2022. ISSN 1476-4687. doi: 10.1038/s41586-021-04357-7. Number: 7896 Publisher: Nature Publishing Group.
A Additional remarks
The Wasserstein metric is defined as  $W_{1}(\nu_{1},\nu_{2}) = \int_{0}^{1}\left|F_{\nu_{1}}^{-1}(u) - F_{\nu_{2}}^{-1}(u)\right|\mathrm{d}u$  and the Cramer metric as  $\ell_2(\nu_1,\nu_2) = \left(\int_{-\infty}^{+\infty}|F_{\nu_1}(u) - F_{\nu_2}(u)|^2\mathrm{d}u\right)^{\frac{1}{2}}$ . For both metrics, we define their supremum  $\overline{\ell_2} (\eta_1,\eta_2) = \sup_{(x,a)\in \mathcal{X}\times \mathcal{A}}\ell_2(\eta_1(x,a),\eta_2(x,a))$  and  $\overline{W}_1(\eta_1,\eta_2) = \sup_{(x,a)\in \mathcal{X}\times \mathcal{A}}W_1(\eta_1(x,a),\eta_2(x,a))$ .
A.1 Remarks on the recursive definition of the Q-value distribution
The notation of the Q-value distribution  $\eta$  is often deceivingly complex compared to the actual object it means to represent. While the 'usual' expected Q-function  $Q(x,a)$  is simply understood as the expected return of a policy at state-action pair  $(x,a)$ , DistRL requires us to keep a notation for the complete distribution of the return. In other words, the Q-value distribution  $\eta_{\pi,h}^{(x,a)}$  should be understood as the distribution of the random variable  $Z = R + Z(S')$ , which is the convolution of the individual distributions of these two independent random variables. It can also be written:
$$
\forall x, a, h, \quad \eta_ {h} ^ {\pi} (x, a) = \sum_ {x ^ {\prime}, a ^ {\prime}} \int p _ {h} \left(x, a, x ^ {\prime}\right) \pi_ {h + 1} ^ {x ^ {\prime}} \left(a ^ {\prime}\right) \eta_ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right) (\cdot - r) R _ {h} ^ {(x, a)} (d r). \tag {10}
$$
A.2 Linear and Exponential Utilities satisfy the properties of Lemma 2
Independence Property Any utility  $U_{f}$  verifies the independence property. Let  $\nu_{1}, \nu_{2}, \nu_{3} \in \mathcal{P}(\mathbb{R})$ ,  $\lambda \in [0, 1]$ . Assume  $U_{f}(\nu_{1}) \geq U_{f}(\nu_{2})$ . Then,
$$
\begin{array}{l} U _ {f} (\lambda \nu_ {1} + (1 - \lambda) \nu_ {3}) = \int f d (\lambda \nu_ {1} + (1 - \lambda) \nu_ {3}) \\ = \lambda \underbrace {\int f \mathrm {d} \nu_ {1}} _ {U _ {f} (\nu_ {1})} + (1 - \lambda) \int f \mathrm {d} \nu_ {3} \\ \geq \lambda \underbrace {\int f \mathrm {d} \nu_ {2}} _ {U _ {f} (\nu_ {2})} + (1 - \lambda) \int f \mathrm {d} \nu_ {3} \\ = \int f d (\lambda \nu_ {2} + (1 - \lambda) \nu_ {3}) \\ = U _ {f} \left(\lambda \nu_ {2} + (1 - \lambda) \nu_ {3}\right) \\ \end{array}
$$
In particular, the mean and the exponential utility do.
Translation Property This property comes from the linearity of the mean and the multiplicative morphism of the exponential. Let  $\nu_{1},\nu_{2}\in \mathcal{P}(\mathbb{R})$ $c\in \mathbb{R}$  . Assume that  $U_{\mathrm{exp}}(\nu_1)\geq U_{\mathrm{exp}}(\nu_2)$  and  $U_{\mathrm{mean}}(\nu_{1})\geq U_{\mathrm{mean}}(\nu_{2})$  . Then,
$$
\begin{array}{l} U _ {\exp} \left(\tau_ {c} \nu_ {1}\right) = \int \exp (r) \mathrm {d} \tau_ {c} \nu_ {1} (r) \\ = \int \exp (r + c) d \nu_ {1} (r) \\ = \exp (c) \int \exp (r) d \nu_ {1} (r) \\ = \exp (c) U _ {\exp} (\nu_ {1}) \\ \geq \exp (c) U _ {\exp} (\nu_ {2}) \\ = \dots \\ = U _ {\mathrm {e x p}} \left(\tau_ {c} \nu_ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} U _ {\text {m e a n}} \left(\tau_ {c} \nu_ {1}\right) = \int \lambda \tau + b + c \mathrm {d} \tau \\ = c + U _ {\text {m e a n}} \left(\nu_ {1}\right) \\ \geq c + U _ {\text {m e a n}} (\nu_ {2}) \\ \equiv \dots \\ = U _ {\text {m e a n}} \left(\tau_ {c} v _ {2}\right) \\ \end{array}
$$
A.3 Policy Evaluation for linear combinations of moments
Rowland et al. [2019] prove a necessary condition on Bellman-closed utilities, namely that they should be a family of the form  $\{x\mapsto x^{\ell}\exp (\lambda x)|0\leq \ell \leq L\}$  for the undiscounted, finite-horizon setting. In the discounted case, the necessary condition is only valid for  $\lambda = 0$ , that is, without the exponential. They also prove that moments (without the exponential) also verify the sufficient condition such that in that setting they are the only Bellman-closed families of utilities.
In the undiscounted setting, to the best of our knowledge, a similar result has not yet been proved. We provide here the sufficient condition for families of the form  $\{x\mapsto x^{\ell}\exp (\lambda x)|0\leq \ell \leq L\}$ . We show that they are Bellman-closed and that this implies that they can be computed by DP.
Let's consider the family  $s_k(\nu) = \int r^k \exp(\lambda r) \, \mathrm{d}\nu(r)$  for  $k \in [n]$  and some fixed  $\lambda \in \mathbb{R}$ .