SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now leverage a result of Billingsley (1986), which states that for any  $m > 0$ , if a sequence of measures  $(\nu_{n})_{n\geq 0}\subseteq \mathcal{P}_{m}(\mathbb{R})$  is uniformly integrable, then  $\mu_m(\nu_n)$  converges to  $\mu_{m}(\nu)$  in  $\mathbb{R}$  whenever  $(\nu_{n})_{n\geq 0}$  weakly converges to  $\pmb{\nu}$ . Applying this in our setting, we first fix  $x\in \mathcal{X}$  then by the previous paragraph we have that the support of each  $\eta_n(x)$  is a subset of the interval  $[-R_{\max} / (1 - \gamma),R_{\max} / (1 - \gamma)]$ , which implies uniform integrability of  $(\eta_n(x))_{n\geq 0}$ . Hence we have that  $\mu_{m}(\eta_{n}(x))\to \mu_{m}(\eta)$ , which gives convergence of  $\psi_{\mu}^{m}$ .
Example A.2. Let  $\psi_{\mu}^{m}$  be the sketch of the first  $m$  moments (Example 5.2). Then by Proposition 5.10, whenever  $m \geq 2$  we have that any proper  $\psi_{\mu}^{m}$  equivalent model is sufficient for optimal planning with respect to the mean-variance risk criterion  $\rho_{\mathrm{MV}}^{\lambda}$  (Example 2.1).
We now demonstrate that the  $m$ -moment sketch  $\psi_{\mu}^{m}$  introduced in Example 5.2 suffices for risk-sensitive learning with respect to a large collection of risk measures.
Proposition A.3. Suppose  $\psi = (\psi_{1},\ldots ,\psi_{m})$  is a Bellman-closed sketch and for each  $i = 1,\dots ,m$ $\exists f_i:\mathbb{R}\to \mathbb{R}$  such that for each  $\nu \in \mathcal{P}_{\psi}(\mathbb{R})$ $\psi_{i}(\nu) = \mathbb{E}_{Z\sim \nu}[f_{i}(Z)]$  . Then any risk measure  $\rho$  which can be planned for exactly using a proper  $\psi$  equivalent model can be planned for exactly using a proper  $\psi_{\mu}^{m}$  equivalent model.
A.2 On the continuity of statistical functionals
We define a sketch  $\psi$  to be continuous if whenever a sequence  $(\nu_{n})_{n\geq 0}\subseteq \mathcal{P}_{\psi}(\mathbb{R})$  converges to  $\nu \in \mathcal{P}_{\psi}(\mathbb{R})$ , we have that  $\psi (\nu_{n})$  converges to  $\psi (\nu)$ . We now formalize this notion. We will use various quantities from topology, Munkres (2000) may be used as a reference for further details.
We will write  $C_b(\mathbb{R})$  for the set of bounded continuous functions from  $\mathbb{R}$  to  $\mathbb{R}$ . We recall that a sequence of measures  $(\nu_n)_{n\geq 0} \subseteq \mathcal{P}(\mathbb{R})$  converges weakly to  $\nu \in \mathcal{P}(\mathbb{R})$  if
$$
\int f d \nu_ {n} \rightarrow \int f d \nu ,
$$
for all  $f \in C_b(\mathbb{R})$ . We refer to the topology induced by this convergence as the weak topology on  $\mathcal{P}(\mathbb{R})$  (to be precise, specifying convergence is sufficient to induce the entire topology since this topology is metrizable).
With this definition, we endow  $\mathcal{P}(\mathbb{R})^{\mathcal{X}}$  with the product topology generated by the weak topology on  $\mathcal{P}(\mathbb{R})$ . Then by definition of the product topology, a sequence  $(\eta_n)_{n\geq 0}\subseteq \mathcal{P}(\mathbb{R})^{\mathcal{X}}$  converges to  $\eta \in \mathcal{P}(\mathbb{R})^{\mathcal{X}}$  if and only if for each  $x\in \mathcal{X}$ ,  $(\eta_n(x))_{n\geq 0}$  converges weakly to  $\eta (x)$  (note this is weak convergence in  $\mathcal{P}(\mathbb{R})$ ).
We can now define a sketch  $\psi : \mathcal{P}_{\psi}(\mathbb{R}) \to \mathbb{R}^m$  to be (sequentially) continuous if whenever a sequence  $(\eta_n)_{n \geq 0} \subseteq \mathcal{P}_{\psi}(\mathbb{R})^\chi$  converges to  $\eta \in \mathcal{P}_{\psi}(\mathbb{R})^\chi$  with the topology we defined above, we have that  $\psi(\eta_n)$  converges to  $\psi(\eta)$  in the usual topology on  $\mathbb{R}^m$ .
To see that this continuity of  $\psi$  implies convergence of the iterates  $(s_k)_{k\geq 0}$  to  $s_\psi^\pi$ , we can recall that we had  $s_k = \psi (\eta_k)$ , where  $\eta_0 = \iota (s_0)$  and  $\eta_{k + 1} = \mathcal{T}^{\pi}\eta_{k}$ . The sequence  $(\eta_k)_{k\geq 0}$  converges to  $\eta^{\pi}$  in the weak product topology on  $\mathcal{P}(\mathbb{R})^{\mathcal{X}}$  (Bellemare et al., 2023), which then immediately gives that if  $\psi$  is continuous as above,  $\psi (\eta_k)\to \psi (\eta^\pi)$ , and hence  $s_k\rightarrow s_\psi^\pi$ .
Figure 4: CVaR of returns for policies learnt using a  $\psi$ -equivalent model and a MLE-based model as the number of distracting dimensions increases (left: Four Rooms, middle: Frozen Lake, right: Windy Cliffs.
A.3 Additional empirical results
Similar to Figure 3, we performed an additional experiment in the tabular domains, where we limit the model capacity by constraining the rank of transition matrix being estimated. In this regime, the model should ideally use its limited capacity to model the aspects of the environment which are most important for the decision problem. We report the results of this experiment in Figure 4.
B Learning statistical functional equivalent models
To learn a model  $\tilde{m} \in \mathcal{M}^k(\Pi, \mathcal{I})$ , we can define the loss of a model as the total deviation from the definition of  $\mathcal{M}^k(\Pi, \mathcal{I})$ . To this end, we define
$$
\mathcal {L} _ {\psi , \Pi , \mathcal {I}} ^ {k, p} (\tilde {m}) \triangleq \sum_ {\pi \in \Pi} \sum_ {s \in \mathcal {I}} \left\| \left(\psi \mathcal {T} ^ {\pi} \iota\right) ^ {k} s - \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {k} s \right\| _ {p} ^ {p},
$$
where for  $s = (s_1, \ldots, s_m) \in \mathbb{R}^m$ ,  $\| s \|_p^p = \sum_{i=1}^m |s_i|^p$ . If the Bellman operator for  $\psi$  exists and is readily available, we can alternatively define the loss working directly on statistics, without needing to impute into distribution space:
$$
\mathcal {L} _ {\psi , \Pi , \mathcal {I}} ^ {k, p} (\tilde {m}) = \sum_ {\pi \in \Pi} \sum_ {s \in \mathcal {I}} \left\| \left(\mathcal {T} _ {\psi} ^ {\pi}\right) ^ {k} s - \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s \right\| _ {p} ^ {p}.
$$
To learn a proper value equivalent model, Grimm et al. (2021) leverages the fact that for any  $k \in \mathbb{N}$  the proper value equivalent class can be deconstructed into an intersection of one proper value equivalent class per policy it matches over:
$$
\mathcal {M} ^ {\infty} (\Pi) = \bigcap_ {\pi \in \Pi} \mathcal {M} ^ {k} \left(\left\{\pi \right\}, \left\{V ^ {\pi} \right\}\right),
$$
so that minimizing  $\left|V^{\pi} - (T_{\tilde{m}}^{\pi})^{k}V^{\pi}\right|$  across all  $\pi \in \Pi$  is sufficient to learn a model in  $\mathcal{M}^{\infty}(\Pi)$ . We now show that the same argument can be used to learn proper statistical functional equivalent models.
Proposition B.1. If  $\psi$  is both continuous and Bellman-closed and  $\iota$  is exact, for any  $k\in \mathbb{N}$  and  $\Pi \subset \mathbb{N}$ , it holds that
$$
\mathcal {M} _ {\psi} ^ {\infty} (\Pi) = \bigcap_ {\pi \in \Pi} \mathcal {M} _ {\psi} ^ {k} \left(\{\pi \}, \left\{s _ {\psi} ^ {\pi} \right\}\right).
$$
With this in mind, we can now propose a loss for learning proper statistical functional equivalent models.
Definition B.2. Let  $\psi$  be a sketch and  $\iota$  an imputation strategy. We define the loss for learning a proper  $\psi$  equivalent model as
$$
\mathcal {L} _ {\psi , \Pi , \infty} ^ {k, p} (\tilde {m}) \triangleq \sum_ {\pi \in \Pi} \left\| s _ {\psi} ^ {\pi} - \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {k} s _ {\psi} ^ {\pi} \right\| _ {p} ^ {p}.
$$
If  $\psi$  is Bellman-closed this loss can be written in terms of its Bellman operator, given by
$$
\mathcal {L} _ {\psi , \Pi , \infty} ^ {k, p} (\tilde {m}) = \sum_ {\pi \in \Pi} \left| \left| s _ {\psi} ^ {\pi} - \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} \right| \right| _ {p} ^ {p}.
$$
C.1 Section 3 Proofs
Proof. To begin, note that this condition implies that for probability measures  $\nu_{1},\nu_{2}$  with  $\mathbb{E}_{Z_1\sim \nu_1}[Z_1] = \mathbb{E}_{Z_2\sim \nu_2}[Z_2]$ , it must hold that  $\rho (\nu_{1}) = \rho (\nu_{2})$ . To see why, suppose this weren't the case. Then there exists a pair of probability measures  $\nu_{1},\nu_{2}\in \mathcal{P}(\mathbb{R})$  such that  $\mathbb{E}_{Z_1\sim \nu_1}[Z_1] = \mathbb{E}_{Z_2\sim \nu_2}[Z_2]$ , and  $\rho (\nu_{1}) < \rho (\nu_{2})$ . Then let us construct an MDP  $M^{*}$  where  $\mathcal{X} = \{x\}$ ,  $\mathcal{A} = \{a,b\}$ ,  $\gamma = 0$ ,  $\mathcal{R}(x,a) = \nu_{1}$ ,  $\mathcal{R}(x,b) = \nu_{2}$  ( $\mathcal{P}$  is defined implicitly since there is a single state). Moreover let us define a second MDP  $\tilde{M}$  defined by  $\mathcal{X} = \{x\}$ ,  $\mathcal{A} = \{a,b\}$ ,  $\gamma = 0$ ,  $\mathcal{R}(x,a) = \mathbb{E}_{Z_1\sim \nu_1}[Z_1]$ ,  $\mathcal{R}(x,a) = \mathbb{E}_{Z_2\sim \nu_2}[Z_2]$ . Then it is immediate to see that  $M^{*}$  and  $\tilde{M}$  are proper value equivalent, however the policy  $\pi^a$  defined by  $\pi^a (a|x) = 1$  is optimal in  $\tilde{M}$ , but not in  $M^{*}$ , contradicting the original statement.
This in turn implies that  $\rho(\nu) = f(\mathbb{E}_{Z \sim \nu}[Z])$  for some function  $f$ . It remains to show that  $f$  must be increasing. To see this, suppose not: then there exists  $\mu_1, \mu_2 \in \mathcal{P}(\mathbb{R})$  such that  $\mathbb{E}_{Z_1 \sim \nu_1}[Z_1] > \mathbb{E}_{Z_2 \sim \nu_2}[Z_2]$  but  $\rho(\nu_1) < \rho(\nu_2)$ . Then we can construct another pair of MDPs:  $M^*$  is defined by setting  $\mathcal{X} = \{x\}$ ,  $\mathcal{A} = \{a, b\}$ ,  $\gamma = 0$ ,  $\mathcal{R}(x, a) = \nu_1$ ,  $\mathcal{R}(x, b) = \nu_2$ , and  $\tilde{M}$  is defined by  $\mathcal{X} = \{x\}$ ,  $\mathcal{A} = \{a, b\}$ ,  $\gamma = 0$ ,  $\mathcal{R}(x, a) = \mathbb{E}_{Z_1 \sim \nu_1}[Z_1]$ ,  $\mathcal{R}(x, a) = \mathbb{E}_{Z_2 \sim \nu_2}[Z_2]$ . Then once again we can see that  $M^*$  and  $\tilde{M}$  are proper value equivalent, but the policy  $\pi^a$  defined by  $\pi^a(a|x) = 1$  is optimal in  $\tilde{M}$ , but not in  $M^*$ , giving us our contradiction.
Hence we must have that  $\rho (\nu) = f(\mathbb{E}_{Z\sim \nu}[Z])$  for some increasing function  $f$ , as desired.
$$
\inf  _ {x \in \mathcal {X}} \left\{\rho \big (\eta^ {\pi_ {\rho} ^ {*}} (x) \big) - \rho \big (\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x) \big) \right\} \geq \frac {R _ {m a x}}{1 - \gamma} \varepsilon (1 - \delta (1 - \varepsilon)).
$$
Proof. Let  $\varphi$  be the function which  $\rho$  corresponds to (so that  $\rho(\mu) = \int_0^1 F_\mu^{-1}(u) \varphi(u) \, du$ ). As  $\rho$  is strictly risk-sensitive, let  $\varepsilon, \delta$  be such that  $\varphi(\varepsilon) \leq \delta$ . Next, note that since  $\varphi$  is constrained to be positive, non-increasing, and integrating to 1, we have that
$$
\begin{array}{l} \int_ {0} ^ {1} F _ {\mu} ^ {- 1} (u) \varphi (u) d u = \int_ {0} ^ {1 - \varepsilon} F _ {\mu} ^ {- 1} (u) \varphi (u) d u + \int_ {1 - \varepsilon} ^ {1} F _ {\mu} ^ {- 1} (u) \varphi (u) d u \\ \leq \frac {1}{1 - \varepsilon} \int_ {0} ^ {1} F _ {\mu} ^ {- 1} (u) \mathbb {1} _ {[ 0, 1 - \varepsilon ]} d u + \delta \int_ {0} ^ {1} F _ {\mu} ^ {- 1} (u) d u \\ = \frac {1}{1 - \varepsilon} \int_ {0} ^ {1 - \varepsilon} F _ {\mu} ^ {- 1} (u) d u + \delta \int_ {1 - \varepsilon} ^ {1} F _ {\mu} ^ {- 1} (u) d u. \\ \end{array}
$$
Figure 5: An MDP  $m^*$  (left) and a proper value equivalent model  $\tilde{m}$  (right).
Let us now consider the MDPs  $m^{*}$  and  $\tilde{m}$  as given in Figure 5. Following Example 2.10 in Bellemare et al. (2023), we have that  $\eta^{\pi^b}(x) = U([-2c,2c])$ , so that  $F_{\eta^{\pi^b}(x)}^{-1}(u) = 4cu - 2c$ . We can use this
to calculate
$$
\begin{array}{l} \rho \left(\eta^ {\pi^ {b}} (x)\right) = \int_ {0} ^ {1} F _ {\mu} ^ {- 1} (u) \varphi (u) d u \\ \leq \frac {1}{1 - \varepsilon} \int_ {0} ^ {1 - \varepsilon} F _ {\eta^ {\pi^ {b}} (x)} ^ {- 1} (u) d u + \delta \int_ {1 - \varepsilon} ^ {1} F _ {\eta^ {\pi^ {b}} (x)} ^ {- 1} (u) d u \\ = \frac {1}{1 - \varepsilon} \int_ {0} ^ {1 - \varepsilon} (4 c u - 2 c) d u + \delta \int_ {1 - \varepsilon} ^ {1} (4 c u - 2 c) d u \\ = - 2 c \varepsilon (1 - \delta (1 - \varepsilon)). \\ \end{array}
$$
With this calculation done, we can remark that  $\pi^a$  is optimal in  $m^*$ , as we have  $\rho(\pi^a(x)) = 0$ . Moreover,  $\pi^b$  is an optimal policy in  $\tilde{m}$ , as  $\rho(\pi_{\tilde{m}}^a(x)) = \rho(\pi_{\tilde{m}}^b(x)) = 0$ .
We can then see that
$$
\rho \left(\pi^ {a} (x)\right) - \rho \left(\pi^ {b} (x)\right) \geq 2 c \varepsilon (1 - \delta (1 - \varepsilon)),
$$
C.2 Section 4 Proofs
Proof. Let  $\pi_{\rho}^{*}$  be an optimal policy for  $\rho$  in  $m^{*}$ , and let  $\tilde{\pi}_{\rho}^{*}$  be an optimal policy for  $\rho$  in  $\tilde{m}$ . For contradiction, suppose that  $\tilde{\pi}_{\rho}^{*}$  is not optimal in  $m^{*}$ . Then for all  $x \in \mathcal{X}$  we have that
$$
\rho \left(\eta^ {\bar {\pi} _ {\rho} ^ {*}} (x)\right) \leq \rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right),
$$
and for at least one  $x\in \mathcal{X}$  we have
$$
\rho \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right).
$$
Let us choose this  $x$ , and note that this implies
$$
\begin{array}{l} \rho \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right) \\ \Longleftrightarrow \rho \left(\eta_ {\tilde {m}} ^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \rho \left(\eta_ {\tilde {m}} ^ {\pi_ {\rho} ^ {*}} (x)\right), \\ \end{array}
$$
since by assumption of  $\tilde{m} \in \mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{T})$  we have that  $\eta^{\pi} = \eta_{\tilde{m}}^{\pi}$  for any  $\pi \in \mathbb{T}$ . But this contradicts the assumption that  $\tilde{\pi}_{\rho}^{*}$  was optimal for  $\rho$  in  $\tilde{m}$ , and we are complete.
C.3 Section 5 Proofs
$$
\mathcal {M} _ {\psi} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \psi \left(\mathcal {T} ^ {\pi} \iota (s)\right) = \psi \left(\mathcal {T} _ {\tilde {m}} ^ {\pi} \iota (s)\right) \forall \pi \in \Pi , s \in \mathcal {I} \right\}.
$$
Note that  $\psi(\mathcal{T}^{\pi}\iota(s)) = \mathcal{T}_{\psi}^{\pi}\psi(\iota(s))$  since  $\psi$  is Bellman-closed, and since  $\iota$  is exact we have that  $\psi(\iota(s)) = s$ . Combining these we have that  $\psi(\mathcal{T}^{\pi}\iota(s)) = \mathcal{T}_{\psi}^{\pi}s$ , which then gives us equality of the sets as desired.
Definition C.1. Let  $(A_{k})_{k = 1}^{\infty}$  be a sequence of sets. Then we have
$$
\liminf_{k\to \infty}A_{k} = \bigcup_{k\geq 1}\bigcap_{j\geq k}A_{j},\text{and}\limsup_{k\to \infty}A_{k} = \bigcap_{k\geq 1}\bigcup_{j\geq k}A_{j}.
$$
If both of these sets are equal, then we say that  $\lim_{k\to \infty}A_k$  exists and is equal to that common set.
Proof. We can begin by recalling that for  $k > 0$ ,
$$
\mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \left(\psi \mathcal {T} ^ {\pi} \iota\right) ^ {k} s = \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {k} s \forall \pi \in \Pi , s \in \mathcal {I} \right\},
$$
We can also note that if  $\tilde{m} \in \mathcal{M}_{\psi}^{k}(\Pi, \mathcal{I})$ , then  $\tilde{m} \in \mathcal{M}_{\psi}^{nk}(\Pi, \mathcal{I})$  for  $n > 0$ , since if  $(\psi \mathcal{T}^{\pi}\iota)^{k}s = (\psi \mathcal{T}_{\tilde{m}}^{\pi}\iota)^{k}s$ , then by setting both sides to the power  $n$  we have that  $(\psi \mathcal{T}^{\pi}\iota)^{nk}s = (\psi \mathcal{T}_{\tilde{m}}^{\pi}\iota)^{nk}s$ . This implies that  $\mathcal{M}_{\psi}^{k}(\Pi, \mathcal{I}) \subseteq \mathcal{M}_{\psi}^{nk}(\Pi, \mathcal{I})$ . Since this is true for any  $n$ , we can set  $n \to \infty$  to obtain
$$
\mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) \subseteq \left\{\tilde {m} \in \mathcal {M}: \lim  _ {n \to \infty} \left(\psi \mathcal {T} ^ {\pi} \iota\right) ^ {n k} s = \lim  _ {n \to \infty} \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {n k} s \forall \pi \in \Pi , s \in \mathcal {I} \right\}.
$$
Since  $\iota$  is exact and  $\psi$  is Bellman-closed, we have that for any  $n\geq 0$ ,  $(\psi \mathcal{T}^{\pi}\iota)^{nk}s = \psi ((\mathcal{T}^{\pi})^{nk}\iota (s))$  (Proposition 8.9 in Bellemare et al. (2023)). Since  $\psi$  is continuous, we have that  $\psi ((\mathcal{T}^{\pi})^{nk}\iota (s))\to s^{\pi}$  as  $n\to \infty$  (justification for this can be found in Appendix A.2). We can then use this to rewrite the above as
$$
\mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) \subseteq \left\{\tilde {m} \in \mathcal {M}: s _ {\psi} ^ {\pi} = s _ {\psi , \tilde {m}} ^ {\pi} \forall \pi \in \Pi , s \in \mathcal {I} \right\} = \mathcal {M} ^ {\infty} (\Pi).
$$
This immediately gives us that
$$
\bigcup_ {j \geq k} \mathcal {M} _ {\psi} ^ {j} (\Pi , \mathcal {I}) \subseteq \mathcal {M} ^ {\infty} (\Pi).
$$
Since this expression is independent of  $k$ , we can take the intersection over all  $k$  to see that
$$
\limsup_{k\to \infty}\mathcal{M}^{k}_{\mathrm{dist}}(\Pi) = \bigcap_{k\geq 1}\bigcup_{j\geq k}\mathcal{M}^{j}_{\psi}(\Pi ,\mathcal{I})\subseteq \mathcal{M}^{\infty}(\Pi).
$$
Moreover it is immediate to see that
$$
\mathcal {M} ^ {\infty} (\Pi) \subseteq \bigcap_ {k \geq 1} \bigcup_ {j \geq k} \mathcal {M} _ {\psi} ^ {j} (\Pi , \mathcal {I}),
$$
which together gives us
$$
\limsup_{k\to \infty}\mathcal{M}^{k}_{\mathrm{dist}}(\Pi) = \mathcal{M}^{\infty}_{\mathrm{dist}}(\Pi).
$$
We now focus on the limit inferior. We take  $k > 0$ , and see that
$$
\begin{array}{l} \bigcap_ {j \geq k} \mathcal {M} _ {\psi} ^ {j} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \left(\psi \mathcal {T} ^ {\pi} \iota\right) ^ {j} s = \left(\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota\right) ^ {j} s \forall j \geq k, \pi \in \Pi , s \in \mathcal {I} \right\} \\ \subseteq \left\{\tilde {m} \in \mathcal {M}: \lim _ {j \to \infty} (\psi \mathcal {T} ^ {\pi} \iota) ^ {j} s = \lim _ {j \to \infty} (\psi \mathcal {T} _ {\dot {m}} ^ {\pi} \iota) ^ {j} s \pi \in \Pi , s \in \mathcal {I} \right\}. \\ \end{array}
$$
As argued above for the limit superior, we have that  $\lim_{j\to \infty}\psi ((\mathcal{T}^{\pi})^{j}\iota (s)) = s_{\psi}^{\pi}$ . Using this fact in the original expression above, we have
$$
\left\{\tilde {m} \in \mathcal {M}: \lim  _ {j \to \infty} (\psi \mathcal {T} ^ {\pi} \iota) ^ {j} s = \lim  _ {j \to \infty} (\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota) ^ {j} s \pi \in \Pi , s \in \mathcal {I} \right\} = \left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi} \pi \in \Pi , s \in \mathcal {I} \right\}.
$$
Conversely, it is immediate to see that
$$
\left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi} \pi \in \Pi , s \in \mathcal {I} \right\} \subseteq \left\{\tilde {m} \in \mathcal {M}: (\psi \mathcal {T} ^ {\pi} \iota) ^ {j} s = (\psi \mathcal {T} _ {\tilde {m}} ^ {\pi} \iota) ^ {j} s \forall j \geq k, \pi \in \Pi , s \in \mathcal {I} \right\},
$$
so that we can combine with our work above and conclude that
$$
\bigcap_ {j \geq k} \mathcal {M} _ {\psi} ^ {j} (\Pi , \mathcal {I}) \subseteq \left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi} \pi \in \Pi , s \in \mathcal {I} \right\} = \mathcal {M} _ {\mathrm {d i s t}} ^ {\infty} (\Pi).
$$
Since this expression is independent of  $k$ , we can take the union over  $k$  to obtain
$$
\liminf_{k\to \infty}\mathcal{M}^{k}_{\psi}(\Pi ,\mathcal{I}) = \bigcup_{k\geq 1}\bigcap_{j\geq k}\mathcal{M}^{j}_{\psi}(\Pi ,\mathcal{I})\subseteq \mathcal{M}^{\infty}_{\mathrm{dist}}(\Pi).
$$
$$
\mathcal {M} _ {\mathrm {d i s t}} ^ {\infty} (\Pi) \subseteq \bigcup_ {k \geq 1} \bigcap_ {j \geq k} \mathcal {M} _ {\psi} ^ {j} (\Pi , \mathcal {I}),
$$
which together give
$$
\liminf_{k\to \infty}\mathcal{M}^{k}_{\psi}(\Pi ,\mathcal{I}) = \mathcal{M}^{\infty}_{\mathrm{dist}}(\Pi).
$$
Since the limit inferior and limit superior are equal, we have the existence of the limit
$$
\lim _ {k \to \infty} \mathcal {M} _ {\psi} ^ {k} (\Pi , \mathcal {I}) = \mathcal {M} _ {\mathrm {d i s t}} ^ {\infty} (\Pi).
$$
$$
\mathcal {M} _ {\psi} ^ {\infty} (\Pi) = \Bigg \{\tilde {m} \in \mathcal {M}: \Pi_ {\mathcal {F} _ {\psi}} \eta^ {\pi} = \Pi_ {\mathcal {F} _ {\psi}} \eta_ {\tilde {m}} ^ {\pi}, \forall \pi \in \Pi \Bigg \}.
$$
Proof. We can write out
$$
\begin{array}{l} \mathcal {M} _ {\psi} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \psi \left(\mathcal {T} ^ {\pi} \iota (s)\right) = \psi \left(\mathcal {T} _ {\tilde {m}} ^ {\pi} \iota (s)\right) \forall \pi \in \Pi , s \in \mathcal {I} \right\} \\ = \left\{\tilde {m} \in \mathcal {M}: \psi \left(\mathcal {T} ^ {\pi} \eta\right) = \psi \left(\mathcal {T} _ {\dot {m}} ^ {\pi} \eta\right) \forall \pi \in \Pi , s \in \mathcal {D} \right\} \\ = \left\{\tilde {m} \in \mathcal {M}: \iota (\psi \left(\mathcal {T} ^ {\pi} \eta\right)) = \iota (\psi \left(\mathcal {T} _ {\hat {m}} ^ {\pi} \eta\right)) \forall \pi \in \Pi , s \in \mathcal {D} \right\} \\ = \left\{\tilde {m} \in \mathcal {M}: \Pi_ {\mathcal {F} _ {\psi}} \mathcal {T} ^ {\pi} \eta = \Pi_ {\mathcal {F} _ {\psi}} \mathcal {T} _ {\tilde {m}} ^ {\pi} \eta \forall \pi \in \Pi , s \in \mathcal {D} \right\}, \\ \end{array}
$$
where the second to last inequality follows from the injectivity of  $\iota$ . Similarly we have that
$$
\begin{array}{l} \mathcal {M} _ {\psi} ^ {\infty} (\Pi , \mathcal {I}) = \left\{\tilde {m} \in \mathcal {M}: \psi \left(\eta^ {\pi}\right) = \psi \left(\eta_ {\hat {m}} ^ {\pi}\right) \forall \pi \in \Pi \right\} \\ = \left\{\tilde {m} \in \mathcal {M}: \iota (\psi (\eta^ {\pi})) = \psi (\iota (\eta_ {\tilde {m}} ^ {\pi})) \forall \pi \in \Pi \right\} \\ = \Bigg \{\tilde {m} \in \mathcal {M}: \Pi_ {\mathcal {F} _ {\psi}} \eta^ {\pi} = \Pi_ {\mathcal {F} _ {\psi}} \eta_ {\tilde {m}} ^ {\pi} \forall \pi \in \Pi \Bigg \}, \\ \end{array}
$$
where the second equality follows by injectivity of  $\iota$ .
$$
\rho \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) \leq \rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right),
$$
$$
\begin{array}{l} \rho \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \rho \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right) \\ \Longleftrightarrow \quad \sum_ {i = 1} ^ {m} \alpha_ {i} \psi_ {i} \left(\eta^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \sum_ {i = 1} ^ {m} \alpha_ {i} \psi_ {i} \left(\eta^ {\pi_ {\rho} ^ {*}} (x)\right) \\ \Longleftrightarrow \quad \sum_ {i = 1} ^ {m} \alpha_ {i} \psi_ {i} \left(\eta_ {\tilde {m}} ^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \sum_ {i = 1} ^ {m} \alpha_ {i} \psi_ {i} \left(\eta_ {\tilde {m}} ^ {\pi_ {\rho} ^ {*}} (x)\right) \\ \Longleftrightarrow \rho \left(\eta_ {\tilde {m}} ^ {\tilde {\pi} _ {\rho} ^ {*}} (x)\right) <   \rho \left(\eta_ {\tilde {m}} ^ {\pi_ {\rho} ^ {*}} (x)\right), \\ \end{array}
$$
which contradicts the assumption that  $\tilde{\pi}_{\rho}^{*}$  was optimal for  $\rho$  in  $\tilde{m}$ .
C.4 Appendix Proofs
Proof. This proof relies on a theorem introduced by Rowland et al. (2019), which we restate here.
Theorem C.2 (Rowland et al. (2019)). Let  $\psi = (\psi_{1},\dots,\psi_{m})$  be a Bellman closed sketch such that for each  $i = 1,\ldots ,m$ , there exist  $f_{i}:\mathbb{R}\to \mathbb{R}$  such that for all  $\nu \in \mathcal{P}_{\psi}(\mathbb{R})$ ,  $\psi_i(\nu) = \mathbb{E}_{Z\sim \nu}[f_i(Z)]$ . Then there exists real numbers  $(b_{ij})_{i,j = 1}^{m}$  such that for all  $\nu \in \mathcal{P}_{\psi}(\mathbb{R})$ ,
$$
\psi_ {i} (\nu) = \sum_ {j = 1} ^ {m} b _ {i j} \mu_ {j} (\nu) + b _ {i 0},
$$
where  $\mu_{j}$  is the  $j$ th moment functional (Example 5.2).
Next, let us suppose that  $\rho$  can be planned for optimally by any  $\tilde{m} \in \mathcal{M}_{\psi}^{\infty}(\mathbb{N})$ , then there must exist  $(\alpha_{i})_{i=1}^{m}$  such that for all  $\nu \in \mathcal{P}_{\psi}(\mathbb{R}) \cap \mathcal{P}_{\rho}(\mathbb{R})$ ,  $\rho(\nu) = \sum_{i=1}^{m} \alpha_{i} \psi_{i}(\nu)$ . Using the coefficients  $(b_{ij})_{i,j=1}^{m}$  introduced in the theorem statement above, we have that