SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} s _ {n} \left(\eta_ {h} ^ {\pi} (x, a)\right) \\ = \mathbb {E} \left[ Z _ {h} ^ {\pi} (x, a) ^ {n} \exp \left(\lambda Z _ {h} ^ {\pi} (x, a)\right) \right] \\ = \mathbb {E} \left[ \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(X ^ {\prime}, A ^ {\prime}\right)\right) ^ {n} \exp \left(\lambda \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(X ^ {\prime}, A ^ {\prime}\right)\right) \right. \right] \\ = \mathbb {E} _ {x ^ {\prime}, a ^ {\prime}} \left[ \mathbb {E} _ {R _ {h}, Z _ {h + 1}} \left[ \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right)\right) ^ {n} \exp \left(\lambda \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right)\right) \mid X ^ {\prime} = x ^ {\prime}, A ^ {\prime} = a ^ {\prime} \right] \right] \right. \\ = \sum_ {x ^ {\prime}, a ^ {\prime}} p _ {h} (x, a, x ^ {\prime}) \pi_ {h} ^ {x ^ {\prime}} \left(a ^ {\prime}\right) \mathbb {E} _ {R _ {h}, Z _ {h + 1}} \left[ \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right)\right) ^ {n} \exp \left(\lambda \left(R _ {h} (x, a) + Z _ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right)\right) \right. \right] \\ = \sum_ {x ^ {\prime}, a ^ {\prime}} p _ {h} (x, a, x ^ {\prime}) \pi_ {h} ^ {x ^ {\prime}} (a ^ {\prime}) \mathbb {E} _ {R _ {h}, Z _ {h + 1}} \left[ \sum_ {k = 0} ^ {n} \binom {n} {k} R _ {h} (x, a) ^ {n - k} \exp (\lambda R _ {h} (x, a)) Z _ {h + 1} ^ {\pi} (x ^ {\prime}, a ^ {\prime}) ^ {k} \exp (\lambda Z _ {h + 1} ^ {\pi} (x ^ {\prime}, a ^ {\prime}) \right] \\ = \sum_ {x ^ {\prime}, a ^ {\prime}} p _ {h} (x, a, x ^ {\prime}) \pi_ {h} ^ {x ^ {\prime}} (a ^ {\prime}) \sum_ {k = 0} ^ {n} \binom {n} {k} \mathbb {E} _ {R _ {h}} \left[ R _ {h} (x, a) ^ {n - k} \exp (\lambda R _ {h} (x, a)) \right] \mathbb {E} _ {Z _ {h + 1}} \left[ Z _ {h + 1} ^ {\pi} (x ^ {\prime}, a ^ {\prime}) ^ {k} \exp (\lambda Z _ {h + 1} ^ {\pi} (x ^ {\prime}, a ^ {\prime}) \right] \\ = \sum_ {x ^ {\prime}, a ^ {\prime}} p _ {h} (x, a, x ^ {\prime}) \pi_ {h} ^ {x ^ {\prime}} \left(a ^ {\prime}\right) \sum_ {k = 0} ^ {n} \binom {n} {k} \mathbb {E} _ {R _ {h}} \left[ R _ {h} (x, a) ^ {n - k} \exp (\lambda R _ {h} (x, a)) \right] s _ {k} \left(\eta_ {h + 1} ^ {\pi} \left(x ^ {\prime}, a ^ {\prime}\right)\right) \\ \end{array}
$$
This first proves that this family of statistical functional is Bellman closed: they can be expressed as a linear combination of the others. Moreover, on the right-hand side, the expression only depends on the distributions and functionals at the current step  $h$  and at the next step  $h + 1$ . Thus, it provides a natural way to evaluate these functionals by DP.
B Categorical Projection: an alternative parametrization
The categorical projection was proposed and studied in Bellemare et al. [2017], Rowland et al. [2018], Bellemare et al. [2023]. For a bounded return distribution, it spreads a fixed number  $N$  of Diracs evenly over the support and used weight parameters to represent the true distribution. The parameter  $N$  is often referred to as the resolution of the projection. More precisely, on a support  $[V_{\min}, V_{\max}]$ , we write  $\Delta = \frac{V_{\max} - V_{\min}}{N - 1}$  the step between atoms  $z_i = V_{\min} + i\Delta$ ,  $i \in [0, N - 1]$ . We define the projection of a given Dirac distribution  $\delta_y$  on the parametric space  $\mathcal{P}_C(\mathbb{R}) = \{\sum_i p_i \delta_{z_i} | 0 \leq p_i \leq 1, \sum_i p_i = 1\}$  by
$$
\Pi_ {C} (\delta_ {y}) = \left\{ \begin{array}{l l} \delta_ {z _ {0}} & y \leq z _ {0} \\ \frac {z _ {i + 1} - y}{z _ {i + 1} - z _ {i}} \delta_ {z _ {i}} + \frac {y - z _ {i}}{z _ {i + 1} - z _ {i}} \delta_ {i + 1} & z _ {i} <   y <   z _ {i + 1} \\ \delta_ {z _ {N - 1}} & y \geq z _ {N - 1} \end{array} \right. \tag {11}
$$
This definition can naturally be extended to any bounded distribution  $\nu$ , and by extension,  $\Pi_C \eta = (\Pi_C \eta(s, a))_{(s, a) \in \mathcal{X} \times \mathcal{A}}$ . This linear operator minimizes the Cramér distance of the parametrization to the parametric space [Rowland et al., 2018].
This projection verifies this approximation bound, analog to the quantile projection,
$$
\sup  _ {(x, a) \in \mathcal {X} \times \mathcal {A}} \ell_ {2} \left(\Pi_ {C} \eta (x, a), \eta (x, a)\right) \leq \frac {\Delta_ {\eta}}{N}. \tag {12}
$$
Using the property that  $W_{1}(\nu_{1},\nu_{2}) \leq \sqrt{\Delta}_{\eta}\ell_{2}(\nu_{1},\nu_{2})$ , the results with the quantile projection can be adapted for the categorical projection, adding  $\sqrt{\Delta}_{\eta}$  factors to the bounds.
This parametrization has the nice property of preserving the mean of the distribution. Yet, even for risk-neutral RL, Quantile-based DistRL algorithm seem to work better and display better properties[Dabney et al., 2018b,a].
C Proofs for Policy Evaluation with parameterized distributions
We recall the statement of Proposition 1: Let  $\pi$  be a policy and  $\eta^{\pi}$  the associated Q-value distributions. Assume the return is bounded on a interval of length  $\Delta_{\eta} \leq H\Delta_{R}$ , where  $\Delta_{R}$  is the support size of the reward distribution. Let  $\hat{\eta}^{\pi}$  be the Q-value distributions obtained by dynamic programming (Algorithm 1) using the quantile projection  $\Pi_{Q}$  with resolution  $N$ . Then,
$$
\sup  _ {(x, a, h) \in (\mathcal {X}, \mathcal {A}, [ H ])} W _ {1} \left(\hat {\eta} _ {h} ^ {\pi} (x, a), \eta_ {h} ^ {\pi} (x, a)\right) \leq H \frac {\Delta_ {\eta}}{2 N} \leq H ^ {2} \frac {\Delta_ {R}}{2 N}.
$$
To avoid clutter of notation, we denote  $\overline{W}_1(\hat{\eta},\eta)\coloneqq \sup_{(x,a)\in \mathcal{X}\times \mathcal{A}}W_1(\hat{\eta} (x,a),\eta (x,a))$
Proof. First recall that for any Q-value distribution  $(\eta_h)_{h\in [H]}$ , with the return bounded on an interval of length  $\Delta_{\eta}\leq H\Delta_R$ , and  $\Pi$  one of the projection operator of interest with resolution  $n$ , we have the following bound on the projection estimation error due to Rowland et al. [2019] (Eq (6)):
$$
\bar {W} _ {1} (\Pi \eta , \eta) \leq \frac {\Delta_ {\eta}}{2 N}. \tag {13}
$$
At a fixed step  $h \in [H]$ , we have the following inequality:
$$
\begin{array}{l} \overline {{W}} _ {1} (\hat {\eta} _ {h} ^ {\pi}, \eta_ {h} ^ {\pi}) = \overline {{W}} _ {1} (\Pi \mathcal {T} _ {h} ^ {\pi} \hat {\eta} _ {h + 1} ^ {\pi}, \mathcal {T} _ {h} ^ {\pi} \eta_ {h + 1} ^ {\pi}) \\ \leq \overline {{W}} _ {1} \left(\Pi \mathcal {T} _ {h} ^ {\pi} \hat {\eta} _ {h + 1} ^ {\pi}, \mathcal {T} _ {h} ^ {\pi} \hat {\eta} _ {h + 1} ^ {\pi}\right) + \overline {{W}} _ {1} \left(\mathcal {T} _ {h} ^ {\pi} \hat {\eta} _ {h + 1} ^ {\pi}, \mathcal {T} _ {h} ^ {\pi} \eta_ {h + 1} ^ {\pi}\right) (14) \\ \leq H \frac {\Delta_ {R}}{2 N} + \bar {W} _ {1} \left(\hat {\eta} _ {h + 1} ^ {\pi}, \eta_ {h + 1} ^ {\pi}\right). (15) \\ \end{array}
$$
Where (14) is due to the triangular inequality with  $\mathcal{T}_h^\pi \hat{\eta}_{h + 1}^\pi$  as the middle term. In (15), the first term comes from applying (13) to the first term of the previous line. The second term is a consequence of the non-expansive property of the Bellman operator [Bellemare et al., 2017]:
$$
\overline {{W}} _ {1} (\mathcal {T} \eta_ {1}, \mathcal {T} \eta_ {2}) \leq \overline {{W}} _ {1} (\eta_ {1}, \eta_ {2}).
$$
Using it recursively starting from  $h = 1$ , and using the fact that  $\hat{\eta}_H^\pi = \eta_H^\pi$  we get:
$$
\overline {{W}} _ {1} \left(\hat {\eta} _ {1} ^ {\pi}, \eta_ {1} ^ {\pi}\right) \leq H \frac {\Delta_ {R}}{2 N} + \overline {{W}} _ {1} \left(\hat {\eta} _ {2} ^ {\pi}, \eta_ {2} ^ {\pi}\right) \leq 2 H \frac {\Delta_ {R}}{2 N} + \overline {{W}} _ {1} \left(\hat {\eta} _ {3} ^ {\pi}, \eta_ {3} ^ {\pi}\right) \leq \dots \leq H ^ {2} \frac {\Delta_ {R}}{2 N}.
$$
C.2 Proof of Lemma 1
We recall the statement of Lemma 1: Let  $s$  be either a utility or a distorted mean and let  $L$  be the Lipschitz coefficient of its characteristic function. Let  $\nu_{1}, \nu_{2}$  be return distributions. Then:
Proof. We prove the property for each family of utilities separately:
Case 1:  $s$  is a utility. There exists  $f$  such that  $s(\nu) = \int f \, \mathrm{d}\nu$ . Let  $L_f$  be its Lipsichtz constant. The Kantorovitch-Rubenstein duality [Villani, 2003] states that:
$$
W _ {1} \left(\nu_ {1}, \nu_ {2}\right) = \frac {1}{L _ {f}} \sup  _ {\| g \| _ {L} \leq L _ {f}} \left(\int g d \nu_ {1} - \int g d \nu_ {2}\right), \tag {16}
$$
where  $||\cdot ||_L$  is the Lipschitz norm. We then immediately get:
$$
L _ {f} W _ {1} \left(\nu_ {1}, \nu_ {2}\right) \geq \left| \int f d \nu_ {1} - \int f d \nu_ {2} \right| = | s (\nu_ {1}) - s (\nu_ {2}) |. \tag {17}
$$
Case 2:  $s$  is a distorted mean. There exists  $g$  such that  $s(\nu) = \int_0^1 g'(\tau)F_\nu^{-1}(\tau)\mathrm{d}\tau$ . Let  $L_{g}$  be its Lipschitz coefficient. Thus:
$$
\begin{array}{l} \left| s \left(\nu_ {1}\right) - s \left(\nu_ {2}\right) \right| = \left| \int_ {0} ^ {1} g ^ {\prime} (\tau) \left(F _ {\nu_ {1}} ^ {- 1} - F _ {\nu_ {2}} ^ {- 1} (\tau)\right) \mathrm {d} \tau \right| \\ \leq | | g ^ {\prime} | | _ {\infty} \int_ {0} ^ {1} \left| F _ {\nu_ {1}} ^ {- 1} (\tau) - F _ {\nu_ {2}} ^ {- 1} (\tau) \right| d \tau \\ \leq L _ {g} W _ {1} \left(\nu_ {1}, \nu_ {2}\right). \\ \end{array}
$$
D About the tightness of Theorem 1
The upper bound provided by Theorem 1 is mainly based on Proposition 1. The latter is obtained by summing, for every step, the Projection Bound by Rowland et al. [2019] (Eq. 6). Thus, achieving the bound would require first to find a problem instance for which, at every step, the projection bound is tight. Then it would require to verify that the total error is the sum of those projection errors.
The experiment in Figure 2 already shows that summing the total error is very close to the sum of the projection errors. However, in that example, the projection error bound is not reached after the first step. In the following, we exhibit an MDP for which the projection bound is tight at every timestep.
First, let us consider a family of distributions for which the projection error is tight:
Proposition 2. Let  $N\in \mathbb{N}$ $\Delta \in \mathbb{R}^{+}$  . Consider  $z_{i} = \frac{i\Delta}{N}$  . The distribution
$$
\nu_ {N, \Delta} = \frac {1}{2 N} \sum_ {i = 0} ^ {N - 1} \left(\delta_ {z _ {i}} + \delta_ {z _ {i + 1}}\right)
$$
has a support of length  $\Delta$  and verifies  $W_{1}(\nu_{N,\Delta},\Pi_{Q}\nu_{N,\Delta}) = \frac{\Delta}{2N}$ .
Proof. The cumulative distribution function (CDF) the distribution  $\nu_{N,\Delta}$  is
$$
F _ {\nu_ {N, \Delta}} (x) = \left\{ \begin{array}{l l} 0 & x <   0 = z _ {0}  , \\ \frac {2 i + 1}{2 N} = \tau_ {i} & z _ {i} \leq x <   z _ {i + 1}  , \\ 1 & x \geq 1 = z _ {N}  . \end{array} \right.
$$
We write  $\tau_{i} = \frac{2i + 1}{2N}$ , so that  $\forall i\in [|0,N - 1|],F_{\nu_{N,\Delta}}(z_i) = \tau_i$ . We now explicit the projection of  $\nu_{N,\Delta}$  and the Wasserstein distance relative to it. A possible Quantile projection is  $\Pi_Q\nu_{N,\Delta} = \frac{1}{N}\sum_{i = 0}^{N - 1}\delta_{z_i}$ ,
$$
\begin{array}{l} W _ {1} (\nu_ {N, \Delta}, \Pi_ {Q} \nu_ {N, \Delta}) = \int_ {0} ^ {1} | F _ {\nu} ^ {- 1} (w) - F _ {\Pi_ {Q} \nu} ^ {- 1} (w) | \mathrm {d} w \\ = \sum_ {i = 0} ^ {N - 1} \int_ {\frac {i}{N}} ^ {\frac {i + 1}{N}} | F _ {\nu} ^ {- 1} (w) - \underbrace {F _ {\Pi_ {Q} \nu} ^ {- 1} (w)} _ {z _ {i}} | \mathrm {d} w \\ = \sum_ {i = 0} ^ {N - 1} \int_ {\frac {i}{N}} ^ {\tau_ {i}} | \underbrace {F _ {\nu} ^ {- 1} (w)} _ {z _ {i}} - z _ {i} | \mathrm {d} w + \int_ {\tau_ {i}} ^ {\frac {i + 1}{N}} | \underbrace {F _ {\nu} ^ {- 1} (w)} _ {z _ {i + 1}} - z _ {i} | \mathrm {d} w \\ = \sum_ {i = 0} ^ {N - 1} \frac {1}{2 N} \underbrace {\left(z _ {i + 1} - z _ {i}\right)} _ {\frac {\Delta}{N}} \\ = \sum_ {i = 0} ^ {N - 1} \frac {\Delta}{2 N ^ {2}} \\ = \frac {\Delta}{2 N} \\ \end{array}
$$
The value distribution of the following time steps is obtained by applying two operators: the Bellman operator and the projection operator. Here will consider a MDP with only one state. We need to find such operators so that  $\Pi_Q\mathcal{T}\nu_{N_1\Delta_1} = \nu_{N_2\Delta_2}$ , where the Bellman operator simply consists in the added reward distribution.
Proposition 3. Let  $N \in \mathbb{N}$ ,  $\Delta \in \mathbb{R}^{+}$ . Consider  $\varrho = \frac{1}{2} (\delta_0 + \delta_{\frac{\Delta}{N - 1}})$ . Then there exists a Quantile projection operator  $\Pi_Q$  such that
$$
\varrho * \Pi_ {Q} (\nu_ {N, \Delta}) = \nu_ {N, (\Delta + \frac {\Delta}{N - 1})}
$$
Proof. We consider  $\Pi_Q(\nu_{N,\Delta}) = \frac{1}{N}\sum_{i=0}^{N-1}\delta_{\frac{i\Delta}{N-1}}$ . Since  $\forall i$ ,  $\frac{i\Delta}{N-1} \leq z_i < \frac{(i+1)\Delta}{N-1}$ , we have  $F(z_i) = \tau_i$ , verifying that it is indeed a valid Quantile Projection.
$$
\begin{array}{l} \varrho * \Pi_ {Q} (\nu_ {N, \Delta}) = \frac {1}{2} (\delta_ {0} + \delta_ {\frac {\Delta}{N - 1}}) * \left(\frac {1}{N} \sum_ {i = 0} ^ {N - 1} \delta_ {\frac {i \Delta}{N - 1}}\right) \\ = \frac {1}{N} \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{2} \left(\delta_ {0} + \delta_ {\frac {\Delta}{N - 1}}\right) * \delta_ {\frac {i \Delta}{N - 1}}\right) \\ = \frac {1}{2 N} \sum_ {i = 0} ^ {N - 1} \left(\delta_ {\frac {i \Delta}{N - 1}} + \delta_ {\frac {(i + 1) \Delta}{N - 1}}\right) \\ = \nu_ {N, (\Delta + \frac {\Delta}{N - 1})} \\ \end{array}
$$
The last equality comes down to noticing that  $\frac{i\Delta}{N - 1} = \frac{i}{N} (\Delta +\frac{\Delta}{N - 1})$
Here, we take advantage of the fact that the Quantile Projection is not unique (see discussion in section 2.1). By choosing the adapted projection, it is then possible to obtain one of those distribution at every timestep.
Corollary 1. Let  $N \in \mathbb{N}$ ,  $\Delta_0 \in \mathbb{R}^+$ . Consider the sequence  $\Delta_{h+1} = \Delta_h(1 + \frac{1}{N-1})$  and  $\varrho_h = \frac{1}{2} (\delta_0 + \delta_{\frac{\Delta_h}{N-1}})$ . Consider  $\Pi_Q$  as in Prop. 3.
Figure 3: Evaluation of the Wasserstein Distance between the true value distribution and the approximated one, in the MDP described in Corollary 1
Consider the MDP with only one state  $x$  and action  $a$ , reward distribution  $\underline{\varrho}_h$ , horizon  $H$ . Consider  $\hat{\eta}_h$  the value distributions obtained through dynamic programming with quantile projection. Then, at any timestep  $h$ , the error induced by the projection operator matches the bound in Eq. 6:
$$
W _ {1} (\Pi_ {Q} \eta_ {h}, \eta_ {h}) = \frac {\Delta_ {h}}{2 N}
$$
While the projection error is maximal in such instance, it was verified experimentally that the bound in Prop. 1 was still not tight. This comes from the fact that the Bellman operator is not a non-contraction is this case, and that the triangular inequality used to sum the projection error is not tight either.
We hence found that every inequality used in proving Theorem 1 can be tight, but there does not seem to exist an instance for which all the inequalities are tight at the same time, meaning that this bound would be never reached exactly.
E Proof of the main results
The proof of the result is divided in parts. First we show that Bellman estimizable functionals verify the two properties of Lemma 2 (Independence and Translation). Then, using those properties, we prove that Bellman estimizable functionals can only be exponential utilities (Theorem 2). Using the known fact that exponential utilities are bellman estimizable, we obtain the full characterization.
E.1 Proof of Lemma 2
Proof. To prove that each property is necessary, we use a proof by contradiction, and exhibit MDPs where the algorithm is not optimal when the property is not verified.
Independence Property Let  $s$  be a Bellman estimable statistical functional that does not satisfy the Independence property. That is, there exists  $\nu_{1}, \nu_{2}, \nu_{3} \in \mathcal{P}(\mathbb{R})$  and  $\lambda \in [0,1]$  such that  $s(\nu_{1}) \geq s(\nu_{2})$  but  $s(\lambda \nu_{1} + (1 - \lambda) \nu_{3}) < s(\lambda \nu_{2} + (1 - \lambda) \nu_{3})$ .
Then consider the MDP in Fig.4 (left) with horizon  $H = 2$  corresponding to the depth of the tree: The agent starts in Start and must take 2 actions, a unique but random and non-rewarding one  $(a_0)$  and a final deterministic step  $(a_1$  or  $a_2)$  to a rewarding state. Thus, by construction, the optimal strategy is  $(a_0, a_2)$  that leads to End 2 with probability  $\lambda$  (and End 3 with probability  $1 - \lambda$ ). The true optimal distribution at Start state is  $\eta_0^* = \lambda \nu_2 + (1 - \lambda)\nu_3$ . We compute the distributions output by the algorithm:
$$
H = 2: \quad \eta_ {2} (\text {E n d} 1, a ^ {*}) = \delta_ {0}, \quad \eta_ {2} (\text {E n d} 2, a ^ {*}) = \delta_ {0}, \quad \eta_ {2} (\text {E n d} 3, a ^ {*}) = \delta_ {0}
$$
$$
H = 1: \quad \eta_ {1} (\text {L e f t}, a ^ {*} = \arg \max  _ {a} s (\nu_ {a})) = \nu_ {1}, \quad \eta_ {2} (\text {R i g h t}, a ^ {*} = a _ {1}, a _ {2}) = \nu_ {3}
$$
$$
H = 0: \quad \eta_ {0} (\operatorname {S t a r t}, a ^ {*} = a _ {0}) = \lambda \nu_ {1} + (1 - \lambda) \nu_ {3}
$$
Figure 4: Left: Independence Property Counter Example, Right: Translation Property Counter Example. Each arrow represents a state transition, which is characterized by the action leading to the transition, the probability of such transition, and the reward distribution of the transition.
The output return distribution  $\eta_0$  is not the true optimal  $\eta_0^*$  for  $s$  so the algorithm is incorrect which is a contradiction as  $s$  is assumed to be Bellman estimizable. Hence the property is needed.
Translation Property Let  $s$  be a Bellman estimable statistical functional that does not verify the Translation Property, i.e. there exists  $\nu_{1}, \nu_{2} \in \mathcal{P}(\mathbb{R})$ ,  $c \in \mathbb{R}$  such that  $s(\nu_{1}) \geq s(\nu_{2})$  but  $s(\tau_c \nu_1) < s(\tau_c \nu_2)$ . Then consider MDP in Fig.4 (right). The optimal strategy is again  $(a_0, a_2)$  by construction. The algorithm output the following distribution:
$$
H = 2: \quad \eta_ {2} (\text {L e f t}, a ^ {*}) = \delta_ {0}, \quad \eta_ {2} (\text {R i g h t}, a ^ {*}) = \delta_ {0}
$$
$$
H = 1: \quad \eta_ {1} (\operatorname {S t e p}, a ^ {*}) = \nu_ {1}
$$
$$
H = 0: \qquad \eta_ {0} (\operatorname {S t a r t}, a ^ {*}) = \tau_ {c} \nu_ {1}
$$
So here again, the algorithm does not output an optimal distribution for  $s$ , hence the necessity of the property.
This proof shows that both properties are necessary, but not that they are sufficient. The other implication could be proven, but the proof would be unnecessary as those properties are enough to restrict to only 1 class of function for which we already know is bellman estimizable.
Proof. For any return distribution  $\nu$ , let  $s(\nu)$  be the considered functional. Since we assume that  $s$  is  $W_{1}$ -continuous Bellmman estimizable, the by the Independence Property (Lemma 2) we know from the Expected Utility Theorem that we can assume:  $s(\nu) = s_f(\nu) = \int_{\mathbb{R}} f(x) d\nu(x)$  for some continuous, monotonous mapping  $f$ . Without loss of generality, we may assume by a density argument that  $f$  is twice continuously differentiable.
By the Intermediate Value Theorem, we can then define  $\phi(h) = f^{-1}(\frac{1}{2}(f(0) + f(h)))$ , so that  $\frac{1}{2}(f(0) + f(h)) = f(\phi(h))$  and in particular  $\phi(0) = f^{-1}(f(0)) = 0$ .
A very special case is when  $f$  is constant, which satisfies the theorem. We now assume that there exists point  $x_0$  such that  $f'$  does not vanish on a neighborhood of  $x_0$ . On this neighborhood, using the inverse function theorem,  $\phi$  is also twice differentiable. Without loss of generality, we assume that  $x_0 = 0$ .
For any fixed  $h > 0$ , we consider the probability distributions  $\nu_{1} = \frac{1}{2} (\delta_{0} + \delta_{h})$  and  $\nu_{2} = \delta_{\phi (h)}$ . Remark that  $s_f(\nu_1) = \int f\mathrm{d}\nu_1 = \frac{1}{2} (f(0) + f(h))$  and  $s_f(\nu_2) = f(\phi (h)) = \frac{1}{2} (f(0) + f(h))$  by definition of  $\phi$ , so  $s_f(\nu_1) = s_f(\nu_2)$ .
The Translation property implies that for all  $x \in \mathbb{R}$ ,  $s_f(\nu_1) \leq s_f(\nu_2) \implies s_f(\nu_1(\cdot + x)) \leq s_f(\nu_2(\cdot + x))$  and  $s_f(\nu_1) \geq s_f(\nu_2) \implies s_f(\nu_1(\cdot + x)) \geq s_f(\nu_2(\cdot + x))$ . Hence,  $\forall x \in \mathbb{R}$ ,  $\frac{1}{2} (f(x) + f(x + h)) = f(x + \phi(h))$ .
This equation can be differentiated twice with respect to  $h$ . For any value of  $x$ , we obtain:
$$
\frac {1}{2} f ^ {\prime} (x + h) = \phi^ {\prime} (h) f ^ {\prime} (x + \phi (h)) \quad \text {a n d} \tag {18}
$$
$$
\frac {1}{2} f ^ {\prime \prime} (x + h) = \phi^ {\prime \prime} (h) f ^ {\prime} (x + \phi (h)) + \phi^ {\prime} (h) ^ {2} f ^ {\prime \prime} (x + \phi (h)). \tag {19}
$$
Recall that by definition,  $\phi(0) = 0$ . Now, for  $x = 0$ , Eq. (18) yields  $\frac{1}{2} f'(0) = \phi'(0) f'(\phi(0)) = \phi'(0) f'(0)$  and, since  $f'(0) \neq 0$ ,  $\phi'(0) = \frac{1}{2}$ .
Now, choosing  $h = 0$  in (19) and plugging in the values of  $\phi(0)$  and  $\phi'(0)$ , we obtain for all  $x \in \mathbb{R}$ :
$$
\frac {1}{4} f ^ {\prime \prime} (x) = \phi^ {\prime \prime} (0) f ^ {\prime} (x).
$$
We then consider two cases, depending on whether  $\phi''(0)$  is null or not.
Case 1:  $\phi''(0) = 0$ . The equation simply becomes  $f''(x) = 0$ , hence  $f$  is affine:  $\exists a, b \in \mathbb{R}$ ,  $f(x) = ax + b$ .
Case 2:  $\phi''(0) \neq 0$ . We write  $\beta = 4\phi''(0)$ . The differential equation becomes  $f''(x) = \beta f'(x)$ , whose solutions are of the form
$$
\exists c _ {1}, c _ {2}, \beta , \quad f (x) = c _ {1} \exp (\beta x) + c _ {2}.
$$
Hence,  $f$  can only be the identity or the exponential, up to an affine transformation.