SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a given sketch  $\psi$ , we define its image as  $I_{\psi} = \{\psi(\nu) : \nu \in \mathcal{P}_{\psi}(\mathbb{R})\} \subseteq \mathbb{R}^m$ . An imputation strategy for a sketch  $\psi$  is a map  $\iota: I_{\psi} \to \mathcal{P}_{\psi}(\mathbb{R})$ , and can be thought of as an approximate inverse (a true inverse may not exist as  $\psi$  is generally not injective). We say  $\iota$  is exact for  $\psi$  if for any  $(s_1, \ldots, s_m) \in I_{\psi}$  we have  $(s_1, \ldots, s_m) = \psi(\iota(s_1, \ldots, s_m))$ . In general, an exact imputation strategy always exists, however it may not be efficiently computable (Bellemare et al., 2023).
Example 5.3. Suppose  $\psi$  is a sketch given by  $\psi (\nu) = (\mathbb{E}_{Z\sim \nu}[Z],\mathrm{Var}_{Z\sim \nu}[Z])$ , and  $\iota$  is given by  $\iota (\mu ,\sigma^2) = \mathcal{N}(\mu ,\sigma^2)$  (that is, the normal distribution with mean  $\mu$  and variance  $\sigma^2$ ). One may verify that  $\iota$  is exact, since for any  $(\mu ,\sigma^2)\in \mathbb{R}^2 = I_{\psi}$ , we have  $\psi (\iota (\mu ,\sigma^2)) = (\mu ,\sigma^2)$ .
We now extend the notion of statistical functionals to return-distribution functions. For  $\eta \in \mathcal{P}_{\psi}(\mathbb{R})^{\mathcal{X}}$  we write  $\psi (\eta) = (\psi (\eta (x)):x\in \mathcal{X})$ . We say that a set  $\Omega \subseteq \mathcal{P}(\mathbb{R})$  is closed under  $\mathcal{T}^{\pi}$  if whenever  $\eta \in \Omega^{\mathcal{X}}$ , we have  $\mathcal{T}^{\pi}\eta \in \Omega^{\mathcal{X}}$ . A sketch  $\psi$  is Bellman-closed (Rowland et al., 2019; Bellemare et al., 2023) if whenever its domain is closed under  $\mathcal{T}^{\pi}$ , there exists an operator  $\mathcal{T}_{\psi}^{\pi}:I_{\psi}^{\mathcal{X}}\to I_{\psi}^{\mathcal{X}}$  such that for any  $\eta \in \mathcal{P}_{\psi}(\mathbb{R})^{\mathcal{X}}$ ,
$$
\psi (\mathcal {T} ^ {\pi} \eta) = \mathcal {T} _ {\psi} ^ {\pi} \psi (\eta).
$$
We refer to  $\mathcal{T}_{\psi}^{\pi}$  as the Bellman operator for  $\psi$ . Similarly to Section 2.1, we denote  $\mathcal{T}_{\psi,\tilde{m}}^{\pi}$  as the Bellman operator for  $\psi$  in an approximate model  $\tilde{m}$ .
We will write  $s_{\psi}^{\pi} = \psi(\eta^{\pi})$  as a shorthand, and refer to it as the return statistic for a policy  $\pi$ . If  $\mathcal{T}_{\psi}^{\pi}$  exists, then  $s_{\psi}^{\pi}$  is its fixed point:  $s_{\psi}^{\pi} = \mathcal{T}_{\psi}^{\pi} s_{\psi}^{\pi}$ . For an approximate model  $\tilde{m}$ , we write  $s_{\psi, \tilde{m}}^{\pi} = \psi(\eta_{\tilde{m}}^{\pi})$ . We further have  $s_{\psi, \tilde{m}}^{\pi} = \mathcal{T}_{\psi, \tilde{m}}^{\pi} s_{\psi, \tilde{m}}^{\pi}$ , that is, it is a fixed point of the Bellman operator  $\mathcal{T}_{\psi, \tilde{m}}^{\pi}$ .
The task of policy evaluation for a statistical functional  $\psi$  is that of computing the value  $s_{\psi}^{\pi}$ . Statistical functional dynamic programming (Bellemare et al., 2023) aims to do this by computing the iterates  $s_{k+1} = \psi(\mathcal{T}^{\pi} \iota(s_k))$ , with  $s_0 \in I_{\psi}^{\mathcal{X}}$  initialized arbitrarily. If  $\iota$  is exact and  $\psi$  is Bellman-closed, then the updates satisfy  $s_k = \psi(\eta_k)$ , where  $\eta_0 = \iota(s_0)$  and  $\eta_{k+1} = \mathcal{T}^{\pi} \eta_k$ . If  $\psi$  is a continuous sketch<sup>4</sup>, then the iterates  $(s_k)_{k \geq 0}$  converge to  $s_{\psi}^{\pi}$ .
5.2 Statistical functional equivalence
We now introduce a notion of model equivalence through the lens of statistical functionals. Intuitively, this allows us to interpolate between value equivalence and distribution equivalence, as we can choose exactly which aspects of the return distributions we would like to capture.
Definition 5.4. Let  $\psi$  be a sketch, and  $\iota$  be an imputation strategy for  $\psi$ . Let  $\mathcal{I} \subseteq I_{\psi}^{\mathcal{X}}$  and  $\Pi \subseteq \Pi$ . We define the class of  $\psi$  equivalent models with respect to  $\Pi$  and  $\mathcal{I}$  as
$$
\mathcal {M} _ {\psi} (\Pi , \mathcal {I}) \triangleq \left\{\tilde {m} \in \mathcal {M}: \psi \left(\mathcal {T} ^ {\pi} \iota (s)\right) = \psi \left(\mathcal {T} _ {\tilde {m}} ^ {\pi} \iota (s)\right), \forall \pi \in \Pi , \forall s \in \mathcal {I} \right\}.
$$
In the case that  $\psi$  is Bellman-closed and  $\iota$  is exact, this set can be described in a form similar to that of value equivalence and distribution equivalence.
4We define this notion in Appendix A.2.
Proposition 5.5. If  $\psi$  is Bellman-closed and  $\iota$  is exact, we have that
$$
\mathcal {M} _ {\psi} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \mathcal {T} _ {\psi} ^ {\pi} s = \mathcal {T} _ {\psi , \tilde {m}} ^ {\pi} s, \forall \pi \in \Pi , \forall s \in \mathcal {I} \right\}.
$$
We can extend the above to  $k$  applications of the projected Bellman operator, and define the set of order- $k$ $\psi$  equivalent models as
$$
\mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) \triangleq \left\{\tilde {m} \in \mathcal {M}: \left(\psi \mathcal {T} ^ {\pi} \iota\right) ^ {k} s = \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {k} s, \forall \pi \in \Pi , \forall s \in \mathcal {I} \right\},
$$
where  $\psi \mathcal{T}^{\pi}\iota :I_{\psi}^{\mathcal{X}}\to I_{\psi}^{\mathcal{X}}$  is shorthand for  $s\mapsto \psi (\mathcal{T}^{\pi}\iota (s))$ . As in Proposition 5.5, if  $\psi$  is Bellman-closed and  $\iota$  is exact, it holds that
$$
\mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \left(\mathcal {T} _ {\psi} ^ {\pi}\right) ^ {k} s = \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s, \forall \pi \in \Pi , \forall s \in \mathcal {I} \right\}.
$$
Following Section 4, we can consider the set of models which agree on return statistics, and have no dependence on the set  $\mathcal{I}$ . However, one difference in the case of statistical functionals is that it is not true in general that this is equal to the limit of  $\mathcal{M}_{\psi}^{k}(\Pi, \mathcal{I})$ . Intuitively, this is for the same reason that the iterates  $(s_k)_{k \geq 0}$  of statistical functional dynamic programming do not always converge to  $s_{\psi}^{\pi}$  (Section 5.1). We first introduce the definition of proper statistical functional equivalence, and then demonstrate when it is the limiting set in Proposition 5.7.
Definition 5.6. Let  $\Pi \subseteq \mathbb{P}$  be a set of policies, and  $\psi$  be a sketch. We define the class of proper statistical functional equivalent models with respect to  $\psi$  and  $\Pi$  as
$$
\mathcal {M} _ {\psi} ^ {\infty} (\Pi) \triangleq \left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi}, \forall \pi \in \Pi \right\}.
$$
Proposition 5.7. If  $\psi$  is both continuous and Bellman-closed and  $\iota$  is exact, then
$$
\lim  _ {k \rightarrow \infty} \mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) = \mathcal {M} _ {\psi} ^ {\infty} (\Pi), f o r a n y \mathcal {I} \subseteq I _ {\psi} ^ {\mathcal {X}}.
$$
Remark 5.8. Value equivalence (Grimm et al., 2020, 2021) can be seen as a special case of statistical functional equivalence, in the sense that if we choose  $\psi = \mathbb{E}$ , then we have  $\mathcal{M}_{\psi}^{k}(\Pi) = \mathcal{M}^{k}(\Pi)$ , for any  $\Pi \subseteq \mathbb{T}$  and  $k \in [1,\infty]$ .
Connection to projected distribution equivalence
In Section 4, we remarked that distribution equivalence was difficult to achieve in practice, due to the fact that the space  $\mathcal{P}(\mathbb{R})^{\mathcal{X}}$  was infinite dimensional, and we generally rely on a parametric family  $\mathcal{F}$ . We now demonstrate that the statistical functional perspective provides us a way to address this.
Let  $\psi$  be a sketch and  $\iota$  an imputation strategy. These induce the implied representation (Bellemare et al., 2023) given by  $\mathcal{F}_{\psi} = \{\iota(s):s\in I_{\psi}\}$ , and the projection operator  $\Pi_{\mathcal{F}_{\psi}}:\mathcal{P}_{\psi}(\mathbb{R})\to \mathcal{F}_{\psi}$  given by  $\Pi_{\mathcal{F}_{\psi}} = \iota \circ \psi$ . We now show that through this construction, we can relate statistical functional model learning to projected distributional model learning with the projection  $\Pi_{\mathcal{F}_{\psi}}$ .
Proposition 5.9. Suppose  $\iota$  is injective,  $\Pi \subseteq \mathbb{N}$ ,  $\mathcal{I} \subseteq I_{\psi}^{\mathcal{X}}$ , and let  $\mathcal{D}_{\mathcal{I}} = \{\iota(s) : s \in \mathcal{I}\} \subseteq \mathcal{P}_{\psi}(\mathbb{R})^{\mathcal{X}}$ . Then
$$
\mathcal {M} _ {\psi} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \Pi_ {\mathcal {F} _ {\psi}} \mathcal {T} ^ {\pi} \eta = \Pi_ {\mathcal {F} _ {\psi}} \mathcal {T} _ {\tilde {m}} ^ {\pi} \eta , \forall \pi \in \Pi , \forall \eta \in \mathcal {D} _ {\mathcal {I}} \right\},
$$
$$
\mathcal {M} _ {\psi} ^ {\infty} (\Pi) = \left\{\tilde {m} \in \mathcal {M}: \Pi_ {\mathcal {F} _ {\psi}} \eta^ {\pi} = \Pi_ {\mathcal {F} _ {\psi}} \eta_ {\tilde {m}} ^ {\pi}, \forall \pi \in \Pi \right\}.
$$
Risk-sensitive learning
We now study which risk measures we can plan optimally for using a model in  $\mathcal{M}_{\psi}^{\infty}(\mathbb{T})$ . Intuitively, we will not be able to plan optimally for all risk measures (as was the case in Theorem 4.3), since this set only requires models to match the aspects of the return distribution captured by  $\psi$ . Indeed, we now show that the choice of  $\psi$  exactly determines which risk measures can be planned for.
Proposition 5.10. Let  $\rho$  be a risk measure and let  $\psi = (\psi_{1},\ldots ,\psi_{m})$  be a sketch, and suppose that  $\rho$  is in the span of  $\psi$ , in the sense that there exists  $\alpha_0,\dots \alpha_m\in \mathbb{R}$  such that for all  $\nu \in \mathcal{P}_{\psi}(\mathbb{R})\cap \mathcal{P}_{\rho}(\mathbb{R})$ $\rho (\nu) = \sum_{i = 1}^{m}\alpha_{i}\psi_{i}(\nu) + \alpha_{0}$ . Then any optimal policy with respect to  $\rho$  in  $\tilde{m}\in \mathcal{M}_{\psi}^{\infty}(\mathbb{I})$  is optimal with respect to  $\rho$  in  $m^{*}$ .
This is a set theoretic limit, and we review its definition in Definition C.1. Further details can be found in many texts on analysis or probability, for example Resnick (1999).
6 Learning statistical functional equivalent models
We now analyze how we may learn models in these classes in practice. As we have introduced a number of concepts and spaces of models, we only discuss here the spaces of models that are used in the empirical evaluation which follow, and we discuss the remainder of the spaces in Appendix B.
We focus on the case of learning a proper  $\psi$ -equivalent model. Such a model must satisfy  $s_{\psi}^{\pi} = (\psi T_{\tilde{m}}^{\pi}\iota)^{k}s_{\psi}^{\pi}$  for any policy  $\pi$  (Proposition B.1), so that we can construct a loss by measuring the amount that this equality is violated by. However, the size of  $\mathbb{P}$  is exponential in  $|\mathcal{X}|$ , so we can approximate this by only measuring the amount of violation over a subset of policies  $\Pi \subseteq \mathbb{P}$ . We can now formalize this concept as a loss.
Definition 6.1. Let  $\psi$  be a sketch, and  $\iota$  an imputation strategy. We define the loss for learning a proper- $\psi$  equivalent model as
$$
\mathcal {L} _ {\psi , \Pi , \infty} ^ {k} (\tilde {m}) \triangleq \sum_ {\pi \in \Pi} \left\| s _ {\psi} ^ {\pi} - \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {k} s _ {\psi} ^ {\pi} \right\| _ {2} ^ {2}.
$$
If  $\psi$  is Bellman-closed this can be written without the need for  $\iota$ , by replacing  $\psi T_{\tilde{m}}^{\pi} \iota$  with  $T_{\psi, \tilde{m}}^{\pi}$ .
This loss is amenable to tabular environments, as it requires knowledge of  $s_{\psi}^{\pi}$ , which can be learnt approximately using statistical functional dynamic programming. Despite this, the above approach can be further adapted to the deep RL setting, which we now discuss, and describe how our approach can be combined with existing model-free risk-sensitive algorithms.
We will assume the existence of a model-free risk-sensitive algorithm which satisfies the following properties: (i) it learns a policy  $\pi$  using a replay buffer  $\mathcal{D}$ , and (ii) it learns an approximate statistical functional function  $s_{\psi, \omega}^{\pi}$  (for example, any algorithm based upon C51 (Bellemare et al., 2017) or QR-DQN (Dabney et al., 2018) satisfies these assumptions), where we write  $\omega$  to refer to the set of parameters it depends on, and to emphasize its difference with the true return statistic  $s_{\psi}^{\pi}$ . We will introduce a loss which learns an approximate model  $\tilde{m}$ , which can then be combined with the replay buffer  $\mathcal{D}$  to use both experienced transitions and modelled transitions to learn  $\pi$ , as was done by e.g. Sutton (1991) or Janner et al. (2019). Following this, for a learnt model  $\tilde{m}$  we introduce the approximate loss
$$
\mathcal{L}_{\mathcal{D},\psi ,\omega}(\tilde{m}) = \underset { \begin{array}{c}(x,a,r,x^{\prime})\sim \mathcal{D}\\ \tilde{x}^{\prime}\sim \tilde{m} (\cdot |x,a) \end{array} }{\mathbb{E}}\left[(s_{\psi ,\omega}^{\pi}(x^{\prime}) - s_{\psi ,\omega}^{\pi}(\tilde{x}^{\prime}))^{2}\right].
$$
7 Empirical evaluation
We now empirically study our framework, and examine the phenomena discussed in the previous sections. We focus on two sets of experiments: the first is in tabular settings where we use dynamic programming methods to perform an analysis without the noise of gradient-based learning. The second builds upon Lim & Malik (2022), where we augment their model-free algorithm with our framework, and evaluate it on an option trading environment. We discuss training and environments details in Appendix E, and provide additional results in Appendix A.3. We provide the code used to run our experiments at github.com/tylerkastner/distribution-equivalence.
7.1 Experimental details
Tabular experiments
For each environment, we learn an MLE model, a proper value equivalent model using the method introduced in Grimm et al. (2021), and a  $\psi_{\mu}^{2}$  equivalent model using  $\mathcal{L}_{\psi_{\mu}^{2},\Pi,\infty}^{k}$ , where  $\psi_{\mu}^{m}$  is the first  $m$  moment functional (cf. Example 5.2), and  $\Pi$  is a set of 1000 randomly sampled policies. For each model, we performed CVaR value iteration (Bellemare et al., 2023), and further performed CVaR value iteration in the true model, to produce three policies. We repeat the learning of the models across 20 independent seeds, and report the performance of the policies in Figure 2 (Left).
Option trading
Lim & Malik (2022) introduced a modification of QR-DQN which attempts to learn CVaR optimal policies, that they evaluate on an option trading environment (Chow & Ghavamzadeh, 2014; Tamar
Figure 2: Left: CVaR(0.5) of returns obtained across the three tabular environments. We computed the values across 1000 trajectories from each of the 20 learnt models. Error bars indicate  $95\%$  confidence intervals. The orange bar for Frozen Lake appears missing because the value obtained is 0. Right: CVaR of returns for the policies learnt in the option trading environment for various CVaR levels after 10,000 environment interactions. Shaded regions indicate  $95\%$  confidence intervals across 10 independent seeds.
Figure 3: CVaR of returns for policies learnt in the option trading environment using a  $\psi$ -equivalent model and a MLE-based model as the number of distracting dimensions increases (left: 2 distracting dimensions, right: 6 distracting dimensions).
et al., 2017). We augment their method using the method described in Section 6, and we learn optimal policies for 10 CVaR levels between 0 and 1. We compare our adapted method to their original method as well as their original method adapted with a PVE model (Grimm et al., 2021), and discuss implementation details in Appendix E. In particular, we evaluate the models in a low-sample regime, so the sample efficiency gains of using a model are apparent.
7.2 Discussion
In Figure 2 (Left), we can see that across all three tabular environments, planning in a proper statistical functional equivalent model achieves stronger results over planning in a proper value equivalent model. This provides an empirical demonstration of Proposition 3.2 and Proposition 5.10: proper value equivalence is limited in its ability to plan risk-sensitively, while risk-sensitive planning in a statistical functional equivalent model approximates risk-sensitive planning in the true environment.
In Figure 2 (Right), we can see that Lim & Malik (2022)'s algorithm augmented with a statistical functional equivalent model achieved significantly improved performance for all CVaR levels below  $\tau \approx 0.8$ . The fact that our augmentation improves upon the original method reflects the improved sample efficiency which comes from using an approximate model for planning. This difference
is more apparent for lower values of  $\tau$ , which demonstrates the phenomenon that learning more risk-sensitive policies are less sample efficient (Greenberg et al., 2022). On the other hand, the method augmented with the PVE model could have the same sample efficiency gains from using an approximate model (as it is trained on the same number of environment interactions), so the fact that it is not performant for lower values of CVaR is a demonstration of Proposition 3.2: the more risk-sensitive the risk measure being planned for, the more the performance is affected.
Compared to MLE-based methods, our approach focuses on learning the aspects of the environment which are most relevant for risk-sensitive planning. This phenomenon is reflected in Figure 3, where the performance gap between the MLE model and the  $\psi$ -equivalent model grows with the amount of uninformative complexity in the environment.
In this work, we studied the intersection of model-based reinforcement learning and risk-sensitive reinforcement learning. We demonstrated that value-equivalent approaches to model learning produce policies which can only plan optimally for the risk-neutral setting, and in risk-sensitive settings their performance degrades with the level of risk being planned for. Similarly, we argued that MLE-based methods are insufficient for efficient risk-sensitive learning as they focus on all aspects of the environment equally. We then introduced distributional model equivalence, and demonstrated that distribution equivalent models can be used to plan for any risk measure, however they are intractable to learn in practice. To account for this, we introduced statistical functional equivalence; an equivalence which is parameterized by the choice of a statistical functional. We proved that the choice of statistical functional exactly determines which risk measures can be planned for optimally, and provided a loss with which these models can be learnt. We further described how our method can be combined with any existing model-free risk-sensitive algorithm, and augmented a recent model-free distributional risk-sensitive algorithm with our model. We supported our theory with empirical results, which demonstrated our approach's advantages over value-equivalence and MLE based models over a range of environments..
We would like to thank the members of the Adaptive Agents Lab who provided feedback on a draft of this paper. AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program (2021-03701). MAE was supported by NSERC Grant [2019-06167], CIFAR AI Chairs program, and CIFAR AI Catalyst grant. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. Finally, we thank the anonymous reviewers (both ICML 2023 and NeurIPS 2023) for providing feedback which improved the paper.
Abachi, R., Ghavamzadeh, M., and Farahmand, A.-m. Policy-aware model learning for policy gradient methods. arXiv preprint arXiv:2003.00030, 2020.
Acerbi, C. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 2002.
Artzner, P., Delbaen, F., Jean-Marc, E., and Heath, D. Coherent measures of risk. Mathematical Finance, 1999.
Arumugam, D. and Van Roy, B. Deciding what to model: Value-equivalent sampling for reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Bäuerle, N. and Ott, J. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 2011.
Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International Conference on Machine Learning. PMLR, 2017.
Bellemare, M. G., Dabney, W., and Rowland, M. Distributional Reinforcement Learning. MIT Press, 2023. http://www.distributional-rl.org.
Billingsley, P. Probability and Measure. John Wiley and Sons, second edition, 1986.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Chow, Y. and Ghavamzadeh, M. Algorithms for CVaR optimization in MDPs. Advances in neural information processing systems, 2014.
Chow, Y., Tamar, A., Mannor, S., and Pavone, M. Risk-sensitive and robust decision-making: a CVaR optimization approach. Advances in neural information processing systems, 2015.
Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
D'Oro, P., Metelli, A. M., Tirinzoni, A., Papini, M., and Restelli, M. Gradient-aware model-based policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Elton, E. J. and Gruber, M. J. Modern portfolio theory, 1950 to date. Journal of banking & finance, 1997.
Farahmand, A.-m. Iterative value-aware model learning. Advances in Neural Information Processing Systems, 2018.
Farahmand, A.-m., Barreto, A., and Nikovski, D. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics. PMLR, 2017.
Greenberg, I., Chow, Y., Ghavamzadeh, M., and Mannor, S. Efficient risk-averse reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Grimm, C., Barreto, A., Singh, S., and Silver, D. The value equivalence principle for model-based reinforcement learning. Advances in Neural Information Processing Systems, 2020.
Grimm, C., Barreto, A., Farquhar, G., Silver, D., and Singh, S. Proper value equivalence. Advances in Neural Information Processing Systems, 2021.
Grimm, C., Barreto, A., and Singh, S. Approximate value equivalence. In Advances in Neural Information Processing Systems, 2022.
Heger, M. Consideration of risk in reinforcement learning. In International Conference on Machine Learning, 1994.
Howard, R. A. and Matheson, J. E. Risk-sensitive Markov decision processes. Management Science, 1972.
A Additional results
A.1 Additional properties of statistical functional equivalence
We begin by discussion some additional properties of statistical functional equivalence.
Proposition A.1. If the reward from a state is almost surely bounded, the  $m$ -moment functional  $\psi_{\mu}^{m}$  is continuous on return distributions.
Proof. Let  $R_{\max}$  be the maximum absolute reward from a state, equivalently let us suppose the support of  $\mathcal{R}(\cdot, \cdot)$  is a subset of  $[-R_{\max}, R_{\max}]$ . Then for any  $x \in \mathcal{X}$  and any policy  $\pi$  we have that the support of  $\eta^{\pi}(x)$  is a subset of  $[-R_{\max} / (1 - \gamma), R_{\max} / (1 - \gamma)]$ .