SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tyler Kastner*
University of Toronto, Vector Institute
Murat A. Erdogdu
Amir-massoud Farahmand
Vector Institute, University of Toronto
We consider the problem of learning models for risk-sensitive reinforcement learning. We theoretically demonstrate that proper value equivalence, a method of learning models which can be used to plan optimally in the risk-neutral setting, is not sufficient to plan optimally in the risk-sensitive setting. We leverage distributional reinforcement learning to introduce two new notions of model equivalence, one which is general and can be used to plan for any risk measure, but is intractable; and a practical variation which allows one to choose which risk measures they may plan optimally for. We demonstrate how our framework can be used to augment any model-free risk-sensitive algorithm, and provide both tabular and large-scale experiments to demonstrate its ability.
Reinforcement learning is a general framework where agents learn to sequentially make decisions to optimize an objective, such as the expected value of future rewards (risk-neutral objective) or the conditional value at risk of future rewards (risk-sensitive objective). It is a popular belief that a truly general agent must have a world model to plan with and limit the number of environment interactions needed (Russell, 2010). One way this is achieved is through model-based reinforcement learning, where an agent learns a model of the environment as well as its policy which it uses to act.
The classical approach to learning a model is to use maximum likelihood estimation (MLE) based on data, which given a model class selects the model which is most likely to have produced the data seen. If the model class is expressive enough, and there is enough data, we may expect a model learnt using MLE to be useful for risk-sensitive planning. However, the success of this method relies on the model being able to model everything about the environment, which is an unrealistic assumption in general.
As opposed to learning models which are accurate in modelling every aspect of the environment, recent works have advocated for learning models with the decision problem in mind, known as decision-aware model learning (Farahmand et al., 2017; Farahmand, 2018; D'Oro et al., 2020; Abachi et al., 2020; Grimm et al., 2020, 2021). In particular, Farahmand et al. (2017) introduced value-aware model learning, which uses a model loss that weighs model errors based on the effect the errors have on potential value functions. This framework has since been iterated on and improved upon in later works such as Farahmand (2018); Abachi et al. (2020); Voelcker et al. (2021). Complementarily, Grimm et al. (2020) introduced the value equivalence principle, a framework of partitioning the space of models based on the properties of the Bellman operators they induce. This framework has been extended by Grimm et al. (2021), where the authors introduce a related partitioning, called
*Correspondence to: tkastner@cs.toronto.edu.
proper value equivalence, based on which models induce the same value functions. They substantiate this approach by demonstrating that any model in the same equivalence class as the true model is sufficient for optimal planning.
While standard reinforcement learning maximizes the expected return achieved by an agent, this may not suffice for many real-life applications. When environments are highly stochastic or where safety is important, a trade-off between the expected return and its variability is often desired. This concept is well-established in finance, and is the basis of modern portfolio theory (Markowitz, 1952). Recently this approach has been used in reinforcement learning, and is referred to as risk-sensitive reinforcement learning. In this setting, agents learn to maximise a risk measure of the return which is possibly different from expectation (in the case it is expectation, it is referred to as risk-neutral), and may penalize or reward risky behaviour (Howard & Matheson, 1972; Heger, 1994; Tamar et al., 2015, 2012; Chow et al., 2015; Tamar et al., 2016). In particular, Grimm et al. (2021, 2022) has explored when optimal risk-neutral planning in an approximate model translates to optimal behaviour in the true environment. However, it is not clear when this holds for risk-sensitive planning.
In this paper, we propose a framework that consolidates risk-sensitive reinforcement learning and decision-aware model learning. Specifically, we address the following question: if we can perform risk-sensitive planning in an approximate model, does it translate to risk-sensitive behaviour in the true environment? To this end, our work provides the following contributions:
- We prove that proper value equivalence only suffices for optimal planning in the risk-neutral case, and the performance of risk-sensitive planning decreases with risk-sensitivity (Section 3).
- We introduce the distribution equivalence principle, and show that this suffices for optimal planning with respect to any risk measure (Section 4).
- We introduce an approximate version of distribution equivalence, which is applicable in practice, that allows one to choose which risk measures they may plan optimally for (Section 5).
- We discuss how these methods may be learnt via losses, and how it can be combined with any existing model-free algorithm (Section 6).
- We demonstrate our framework empirically in both tabular and continuous domains (Section 7).
We write  $\mathcal{P}(\mathcal{Z})$  to represent the set of probability measures on a measurable set  $\mathcal{Z}$ . For a probability measure  $\nu \in \mathcal{P}(\mathcal{Z})$ , we write  $X \sim \nu$  to denote a random variable  $X$  with law  $\nu$ , meaning for all measurable subsets  $A \subseteq \mathcal{Z}$ ,  $\mathbb{P}(X \in A) = \nu(A)$ . For a probability measure  $\nu \in \mathcal{P}(\mathcal{Z})$  and a measurable function  $f: \mathcal{Z} \to \mathcal{Y}$ , the pushforward measure  $f_{\#} \nu \in \mathcal{P}(\mathcal{Y})$  is defined by  $f_{\#} \nu(Y) = \nu(f^{-1}(Y))$  for all measurable sets  $Y \subseteq \mathcal{Y}$ . For arbitrary sets  $X$  and  $Y$ , we write  $Y^{X}$  for the space of functions from  $X$  to  $Y$ .
We consider a Markov decision process (MDP) represented as a tuple  $(\mathcal{X},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$  where  $\mathcal{X}$  is the state space,  $\mathcal{A}$  is the action space,  $\mathcal{P}:\mathcal{X}\times \mathcal{A}\to \mathcal{P}(\mathcal{X})$  is the transition kernel,  $\mathcal{R}:$ $\mathcal{X}\times \mathcal{A}\rightarrow \mathcal{P}(\mathbb{R})$  is the reward kernel, and  $\gamma \in [0,1)$  is the discount factor. We define a policy to be a map  $\pi :\mathcal{X}\rightarrow \mathcal{P}(\mathcal{A})$  , and write the set of all policies as  $\mathbb{T}$  . Given a policy  $\pi \in \mathbb{T}$  , we can sample trajectories  $(X_{t},A_{t},R_{t})_{t\geq 0}$  , where for all  $t\geq 0$ $A_{t}\sim \pi (\cdot |X_{t})$ $R_{t}\sim \mathcal{R}(X_{t},A_{t})$  , and  $X_{t + 1}\sim \mathcal{P}(X_t,A_t)$  . For a trajectory from  $\pi$  beginning at  $X_0 = x$  , we associate to it the return random variable  $G^{\pi}(x) = \sum_{t\geq 0}\gamma^{t}R_{t}$  . The expected return across all trajectories starting from a state  $x$  is the value function  $V^{\pi}(x) = \mathbb{E}_{\pi}[G^{\pi}(x)]$  . The value function is the unique fixed point of the Bellman operator  $T^{\pi}:\mathbb{R}^{\mathcal{X}}\to \mathbb{R}^{\mathcal{X}}$  , defined by
$$
T ^ {\pi} V (x) \triangleq \mathbb {E} _ {\pi} [ R + \gamma V \left(X ^ {\prime}\right) ], \tag {1}
$$
where  $\mathbb{E}_{\pi}$  is written to indicate  $A \sim \pi(\cdot \mid x)$ ,  $R \sim \mathcal{R}(x, A)$ , and  $X' \sim \mathcal{P}(x, A)$ .
2.1 Model-based reinforcement learning and the value equivalence principle
Estimating (1) in the RL setting is not possible directly, as generally an agent does not have access to  $\mathcal{R}$  nor  $\mathcal{P}$ , but only samples from them. There are two common approaches to address this: model-free
methods estimate the expectations through the use of stochastic approximation or related methods (Sutton, 1988), while model-based approaches learn an approximate model  $\tilde{\mathcal{R}}$ ,  $\tilde{\mathcal{P}}$  (Sutton, 1991).
We will refer to a tuple  $\tilde{m} = (\tilde{\mathcal{R}},\tilde{\mathcal{P}})$  as a model, and write  $\mathbb{M}$  for the set of all models. In turn, each model  $\tilde{m}$  induces an approximate MDP  $(\mathcal{X},\mathcal{A},\tilde{\mathcal{P}},\tilde{\mathcal{R}},\gamma)$ . For a policy  $\pi$ , we write  $T_{\tilde{m}}^{\pi}:\mathbb{R}^{\mathcal{X}}\to \mathbb{R}^{\mathcal{X}}$  for the Bellman operator in this approximate MDP, and we write  $V_{\tilde{m}}^{\pi}$  for the unique fixed point of this operator. We write  $m^{*} = (\mathcal{R},\mathcal{P})$  for the true model, and keep  $T^{\pi} = T_{m^{*}}^{\pi}$ . Throughout the paper, we will write  $\mathcal{M}\subseteq \mathbb{M}$  to represent a set of models which we are considering.
Traditional methods of model-based reinforcement learning learn a model  $\tilde{m}$  using task-agnostic methods such as maximum likelihood estimation (Sutton, 1991; Parr et al., 2008; Oh et al., 2015). More recent approaches have focused on learning models which are accurate in aspects which are necessary for decision making (Farahmand et al., 2017; Farahmand, 2018; Schrittwieser et al., 2020; Grimm et al., 2020, 2021; Arumugam & Van Roy, 2022). Of importance to us is Grimm et al. (2021), which introduced proper value equivalence, and defined the set  $\mathcal{M}^{\infty}(\Pi) \triangleq \{\tilde{m} \in \mathcal{M}: V^{\pi} = V_{\tilde{m}}^{\pi}, \forall \pi \in \Pi\}$ . They proved that any model  $\tilde{m} \in \mathcal{M}^{\infty}(\mathbb{T})$  suffices for optimal planning, that is, a policy which is optimal in  $\tilde{m}$  is also optimal in the true environment,  $m^{*}$ .
2.2 Distributional reinforcement learning
Distributional reinforcement learning (Morimura et al., 2010; Bellemare et al., 2017, 2023) studies the return  $G^{\pi}$  as a random variable, rather than focusing only on its expectation. For  $x \in \mathcal{X}$ , we define the return distribution  $\eta^{\pi}(x)$  as the law of the random variable  $G^{\pi}(x)$ . The return distribution is the unique fixed point of the distributional Bellman operator  $\mathcal{T}^{\pi} : \mathcal{P}(\mathbb{R})^{\mathcal{X}} \to \mathcal{P}(\mathbb{R})^{\mathcal{X}}$  given by
$$
\mathcal {T} ^ {\pi} \eta (x) \triangleq \mathbb {E} _ {\pi} \left[ \left(b _ {R, \gamma}\right) _ {\#} \eta \left(X ^ {\prime}\right) \right],
$$
where  $b_{R,\gamma}:x\mapsto R + \gamma x$  and  $\mathbb{E}_{\pi}$  is as in (1). As was the case in Section 2.1, any approximate model  $\tilde{m}$  induces a distributional Bellman operator  $\mathcal{T}_{\tilde{m}}^{\pi}$ , and we write  $\eta_{\tilde{m}}^{\pi}$  for the unique fixed point of this operator.
2.3 Risk-sensitive reinforcement learning
We define a risk measure to be a function  $\rho : \mathcal{P}_{\rho}(\mathbb{R}) \to [-\infty, \infty)$ , where  $\mathcal{P}_{\rho}(\mathbb{R}) \subseteq \mathcal{P}(\mathbb{R})$  is its domain. A classic example is  $\rho = \mathbb{E}$ , which we refer to as the risk-neutral case. When  $\rho$  depends on more than only the mean of a distribution, we refer to  $\rho$  as being risk-sensitive. The area of risk-sensitive reinforcement learning is concerned with maximizing various risk measures of the random return, rather than the expectation as done classically. We now present two examples of commonly used risk measures.
Example 2.1. For  $\lambda >0$ , the mean-variance risk criterion is given by  $\rho_{\mathrm{MV}}^{\lambda}(\mu) = \mathbb{E}_{Z\sim \mu}[Z] - \lambda \mathrm{Var}_{Z\sim \mu}(Z)$  (Markowitz, 1952; Tamar et al., 2012). This forms the basis of modern portfolio theory (Elton & Gruber, 1997).
Example 2.2. The conditional value at risk at level  $\tau \in [0,1]$  is defined as
$$
\mathrm {C V a R} _ {\tau} (\mu) \triangleq \frac {1}{\tau} \int_ {0} ^ {\tau} F _ {\mu} ^ {- 1} (u) d u,
$$
where  $F_{\mu}^{-1}(u) = \inf \{z\in \mathbb{R}:\mu (-\infty ,z]\geq u\}$  is the quantile function of  $\mu$ . If  $F_{\mu}^{-1}$  is a strictly increasing function, we equivalently have
$$
\mathrm {C V a R} _ {\tau} (\mu) = \underset {Z \sim \mu} {\mathbb {E}} \left[ Z \mid Z \leq F _ {\mu} ^ {- 1} (\tau) \right],
$$
so that  $\mathrm{CVaR}_{\tau}(\mu)$  can be understood as the expectation of the lowest  $(100\cdot \tau)\%$  of samples from  $\mu$
We say that a policy  $\pi_{\rho}^{*}$  is optimal with respect to  $\rho$  if
$$
\rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right) = \max  _ {\pi \in \mathbb {T}} \rho \left(\eta^ {\pi} (x)\right), \forall x \in \mathcal {X}.
$$
2We use the definition of risk measure used by Bellemare et al. (2023). In earlier financial mathematics literature such as Artzner et al. (1999), risk measures were defined as functions of random variables, rather than probability measures. By defining the domain to be a subset of probability measures, we are implicitly considering law-invariant risk measures (Kusuoka, 2001).
Figure 1: An MDP with a single state and two actions (left), and a proper value equivalent model  $\tilde{m}$  for it (right).
Since we define the space of policies as  $\mathbb{P} = \mathcal{P}(\mathcal{A})^{\mathcal{X}}$ , we implicitly only considering the class of stationary Markov policies (Puterman, 2014). For a general risk measure, an optimal policy in this class may not exist (Bellemare et al., 2023). We discuss more general policies in Appendix D.
3 Limitations of value equivalence for risk-sensitive planning
Grimm et al. (2021) proved that any proper value equivalent model is sufficient for optimal risk-neutral planning. In this section, we investigate whether this holds for risk-sensitive planning as well, or is limited to the risk-neutral setting.
As an illustrative example, let us consider the MDP and approximate model  $\tilde{m}$  in Figure 1. It is straightforward to verify that  $\tilde{m}$  is a proper value equivalent model for the true MDP, as the value for any policy is 0 in both  $\tilde{m}$  and the true environment. However, for a risk-sensitive agent  $\tilde{m}$  is not sufficient: the variability of return when choosing action  $b$  in  $m^{*}$  is much higher than the variability of return when choosing action  $b$  in  $\tilde{m}$ . Formally, let us fix  $\gamma = \frac{1}{2}$ , and let  $\pi^b$  be the policy which chooses action  $b$  with probability 1. Then  $\eta^{\pi^b}(x) = U([-2,2])$  (Bellemare et al., 2023, Example 2.10), while  $\eta_{\tilde{m}}^{\pi^b}(x) = \delta_0$  (where  $\delta_x$  refers to the Dirac distribution concentrated at  $x$ ). This difference prevents  $\tilde{m}$  from planning optimally for risk-sensitive risk measures. For example, the optimal policy with respect to  $\rho_{\mathrm{MV}}^{\lambda}$  in  $m^{*}$  is to choose  $a$  with probability 1, while in  $\tilde{m}$  any policy is optimal. It is straightforward to validate that similar phenomena happen for  $\mathrm{CVaR}_{\tau}$  when  $\tau < 1$ .
As demonstrated in the example above, proper value equivalence is not sufficient for planning with respect to the risk measures introduced in Section 2.3. We now formalize this, and demonstrate that the only risk measures which proper value equivalence can plan for exactly are those which are functions of expectation.
Proposition 3.1. Let  $\rho$  be a risk measure such that for any MDP and any set  $\mathcal{M}$  of models, a policy optimal for  $\rho$  for any  $\tilde{m} \in \mathcal{M}^{\infty}(\mathbb{P})$  is optimal in the true MDP. Then  $\rho$  must be risk-neutral, in the sense that there exists an increasing function  $g: \mathbb{R} \to \mathbb{R}$  such that  $\rho(\nu) = g(\mathbb{E}_{Z \sim \nu}[Z])$ .
The previous proposition demonstrates that in general, the only risk measures we can plan for in a proper value equivalent model are those which are transformations of the value function. However, it does not address the question of how well proper value equivalent models can be used to plan with respect to other risk measures.
To investigate this question, we turn our attention to a class of risk measures known as spectral risk measures (Acerbi, 2002). Let  $\varphi :[0,1]\to \mathbb{R}$  be a non-negative, non-increasing, right-continuous, integrable function such that  $\int_0^1\varphi (u)du = 1$ . Then the spectral risk measure corresponding to  $\varphi$  is defined as
$$
\rho_ {\varphi} (\nu) \triangleq \int_ {0} ^ {1} F _ {\nu} ^ {- 1} (u) \varphi (u) d u,
$$
where  $F_{\nu}^{-1}$  is as in Example 2.2. Spectral risk measures encompass many common risk measures, for example choosing  $\varphi = \mathbb{1}_{[0,1]}$  corresponds to expectation, while  $\varphi = \frac{1}{\tau}\mathbb{1}_{[0,\tau ]}$  corresponds to  $\mathrm{CVaR}_{\tau}$ .
We say that a spectral risk measure  $\rho$  is  $(\varepsilon, \delta)$ -strictly risk-sensitive if it corresponds to a function  $\varphi$  such that  $\varphi(\varepsilon) \leq \delta$ . This implies that  $\varphi$  gives a weight of less than  $\delta$  to the top  $\varepsilon$  quantiles, and so intuitively a larger  $\varepsilon$  and smaller  $\delta$  implies a more risk-sensitive risk measure.
With this definition, we now demonstrate that when using proper value equivalent models to plan for strictly risk-sensitive spectral risk measures, there exists a tradeoff between the level of risk-sensitivity and the performance achieved.
Proposition 3.2. Let  $\rho$  be an  $(\varepsilon, \delta)$ -strictly risk-sensitive spectral risk measure, and suppose that rewards are almost surely bounded by  $R_{max}$ . Then there exists an MDP with a proper value equivalent model  $\tilde{m}$  with the following property: letting  $\pi_{\rho}^{*}$  be an optimal policy for  $\rho$  in the original MDP, and  $\tilde{\pi}_{\rho}^{*}$  an optimal policy for  $\rho$  in  $\tilde{m}$ , we have
$$
\inf  _ {x \in \mathcal {X}} \left\{\rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right) - \rho \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) \right\} \geq \frac {R _ {m a x}}{1 - \gamma} \varepsilon (1 - \delta (1 - \varepsilon)).
$$
The fact that we take an infimum over  $\mathcal{X}$  is important to note: there exists an MDP such that for any state  $x$ , the performance gap due to planning in the proper value equivalent model is at least  $\frac{R_{\max}}{1 - \gamma} \varepsilon (1 - \delta (1 - \varepsilon))$ . This weakness motivates us to introduce a new notion of model equivalence.
4 The distribution equivalence principle
We now introduce a novel notion of equivalence on the space of models, which can be used for risk-sensitive learning. Intuitively, proper value equivalence ensures matching of the means of the approximate and true return distributions, which is why it can only produce optimal policies for risk measures which depend on the mean. In order to plan for any risk measure, we leverage the distributional perspective of RL, to partition models based on their entire return distribution.
Definition 4.1. Let  $\Pi \subseteq \mathbb{I}$  be a set of policies and  $\mathcal{D} \subseteq \mathcal{P}(\mathbb{R})^{\mathcal{X}}$  be a set of distribution functions. We say that the space of distribution equivalent models with respect to  $\Pi$  and  $\mathcal{D}$  is
$$
\mathcal {M} _ {\mathrm {d i s t}} (\Pi , \mathcal {D}) \triangleq \left\{\tilde {m} \in \mathcal {M}: \mathcal {T} ^ {\pi} \eta = \mathcal {T} _ {\tilde {m}} ^ {\pi} \eta , \forall \pi \in \Pi , \eta \in \mathcal {D} \right\}.
$$
We can extend this concept to equivalence over multiple applications of the Bellman operator. Following this, for  $k \in \mathbb{N}$  we define the order  $k$  distribution-equivalence class as
$$
\mathcal {M} _ {\mathrm {d i s t}} ^ {k} (\Pi , \mathcal {D}) \triangleq \left\{\tilde {m} \in \mathcal {M}: \left(\mathcal {T} ^ {\pi}\right) ^ {k} \eta = \left(\mathcal {T} _ {\tilde {m}} ^ {\pi}\right) ^ {k} \eta , \forall \pi \in \Pi , \forall \eta \in \mathcal {D} \right\}.
$$
Taking the limit as  $k \to \infty$ , we retrieve the set of proper distribution equivalent models.
Definition 4.2. Let  $\Pi \subseteq \mathbb{I}$  be a set of policies. We define the set of proper distribution equivalent models with respect to  $\Pi$  as
$$
\mathcal {M} _ {\mathrm {d i s t}} ^ {\infty} (\Pi) \triangleq \{\tilde {m} \in \mathcal {M}: \eta_ {\tilde {m}} ^ {\pi} = \eta^ {\pi}, \forall \pi \in \Pi \}.
$$
As discussed in Section 3, models in  $\mathcal{M}^{\infty}(\mathbb{P})$  are sufficient for optimal planning with respect to expectation, but generally not with respect to other risk measures. We now show that proper distribution equivalence removes this problem: choosing a model in  $\mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{P})$  is sufficient for optimal planning with respect to any risk measure.
Theorem 4.3. Let  $\rho$  be any risk measure. Then an optimal policy with respect to  $\rho$  in  $\tilde{m} \in \mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{T})$  is optimal with respect to  $\rho$  in  $m^{*}$ .
At this point, it appears that distribution equivalence addresses nearly all of the limitations of value equivalence discussed in Section 3. However, the nature of distributions brings inherent challenges, in particular they are infinite dimensional. As a result of this, for computational purposes one must use a parametric family of distributions  $\mathcal{F} \subseteq \mathcal{P}(\mathbb{R})$  (Rowland et al., 2018; Dabney et al., 2018) to represent return distributions. However, an additional challenge is that the distributional Bellman operator may bring return distributions out of the parametric representation space: for a general  $\eta \in \mathcal{F}^{\mathcal{X}}$ ,  $\mathcal{T}^{\pi}\eta \notin \bar{\mathcal{F}}^{\mathcal{X}}$ . Hence, we also require a projection operator<sup>3</sup>  $\Pi_{\mathcal{F}}:\mathcal{P}(\mathbb{R})^{\mathcal{X}}\to \mathcal{F}^{\mathcal{X}}$ , and in practice we must use  $\Pi_{\mathcal{F}}\mathcal{T}^{\pi}\eta$ . This also implies that it may not be feasible to learn a model  $\tilde{m}$  in  $\mathcal{M}_{\mathrm{dist}}^{k}(\Pi ,\mathcal{D})$  or  $\mathcal{M}_{\mathrm{dist}}^{\infty}(\Pi)$ : they rely on matching  $\mathcal{T}^{\pi}\eta$  or  $\eta^{\pi}$ , while one would only have access to  $\Pi_{\mathcal{F}}\mathcal{T}^{\pi}\eta$  and  $\Pi_{\mathcal{F}}\eta^{\pi}$ . We address this issue next, through the perspective of statistical functionals.
5 Statistical functional equivalence
Following the intractability of learning a distribution equivalent model in practice, we now study model equivalence through the lens of statistical functionals, a framework introduced by Rowland
Further details on the necessity of the projection operator and a discussion of various projections can be found in Chapter 5 of Bellemare et al. (2023).
et al. (2019) to describe a variety of distributional reinforcement learning algorithms. We begin with a review of statistical functionals (Section 5.1), and then introduce statistical functional equivalence, demonstrate its equivalence to projected distribution equivalence, and study which risk measures it can plan optimally for (Section 5.2).
5.1 Background on statistical functionals
Definition 5.1. A statistical functional is a function  $\psi : \mathcal{P}_{\psi}(\mathbb{R}) \to \mathbb{R}$ , where  $\mathcal{P}_{\psi}(\mathbb{R}) \subseteq \mathcal{P}(\mathbb{R})$  is its domain. A sketch is a collection of statistical functionals, written as a mapping  $\psi : \mathcal{P}_{\psi}(\mathbb{R}) \to \mathbb{R}^m$ , where  $\psi = (\psi_1, \ldots, \psi_m)$ , and  $\mathcal{P}_{\psi}(\mathbb{R}) = \bigcap_{i=1}^{m} \mathcal{P}_{\psi_i}(\mathbb{R})$ .
Example 5.2. Suppose  $i > 0$ , and let  $\mathcal{P}_i(\mathbb{R})$  be the set of probability measures with finite  $i$ th moment. Moreover, let  $\mu_i(\nu)$  be the  $i$ th moment of a measure  $\nu \in \mathcal{P}_i(\mathbb{R})$ . Then for  $m > 0$ , the  $m$  moment sketch  $\psi_{\mu}^{m}: \mathcal{P}_{m}(\mathbb{R}) \to \mathbb{R}^{m}$  is defined by  $\psi_{\mu}^{m}(\nu) = (\mu_{1}(\nu), \dots, \mu_{m}(\nu))$ .