SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \rho (\nu) = \sum_ {i = 1} ^ {m} \alpha_ {i} (\psi_ {i} (\nu) \\ = \sum_ {i = 1} ^ {m} \alpha_ {i} \left(\sum_ {j = 1} ^ {m} b _ {i j} \mu_ {j} (\nu) + b _ {i 0}\right) \\ = \sum_ {i = 1} ^ {m} \alpha_ {i} \sum_ {j = 1} ^ {m} b _ {i j} \mu_ {j} (\nu) + \sum_ {i = 1} ^ {m} \alpha_ {i} b _ {i 0} \\ = \sum_ {j = 1} ^ {m} \beta_ {j} \mu_ {j} (\nu) + \beta_ {0}, \\ \end{array}
$$
where  $\beta_{j} = \sum_{i = 1}^{m}\alpha_{i}b_{ij}$  for  $j = 0,\ldots ,m$ . We can then apply Proposition 5.10, and we are complete.
Proposition B.1. If  $\psi$  is both continuous and Bellman-closed and  $\iota$  is exact, for any  $k\in \mathbb{N}$  and  $\Pi \subseteq \mathbb{N}$ , it holds that
Proof. We begin by rewriting the definition
$$
\begin{array}{l} \mathcal {M} _ {\psi} ^ {\infty} (\Pi) = \left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi} \text {f o r a l l} \pi \in \Pi \right\} \\ = \bigcap_ {\pi \in \Pi} \left\{\tilde {m} \in \mathcal {M}: s _ {\psi , \tilde {m}} ^ {\pi} = s _ {\psi} ^ {\pi} \right\} \\ = \bigcap_ {\pi \in \Pi} \mathcal {M} _ {\psi} ^ {\infty} (\{\pi \}). \\ \end{array}
$$
Next, note that for any  $\pi$  and any  $k\in \mathbb{N}$  we have  $\mathcal{M}_{\psi}^{\infty}(\{\pi \})\subseteq \mathcal{M}_{\psi}^{k}(\{\pi \} ,\{s_{\psi}^{\pi}\})$ , since if  $s_{\psi}^{\pi} = s_{\psi ,\tilde{m}}^{\pi}$  we can write out
$$
\begin{array}{l} s _ {\psi} ^ {\pi} = s _ {\psi , \tilde {m}} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} = \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi , \tilde {m}} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} = s _ {\psi , \tilde {m}} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} = s _ {\psi} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} = \left(\mathcal {T} _ {\psi} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi}, \\ \end{array}
$$
and hence  $\tilde{m}\in \mathcal{M}_{\psi}^{k}(\{\pi \} ,\{s_{\psi}^{\pi}\})$  . Conversely, if we let  $\tilde{m}\in \mathcal{M}_{\psi}^{k}(\{\pi \} ,\{s_{\psi}^{\pi}\})$  , we can write out
$$
\begin{array}{l} \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {=} \left(\mathcal {T} _ {\psi} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} = s _ {\psi} ^ {\pi} \\ \Longrightarrow \quad \left(\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}\right) ^ {2 k} s _ {\psi} ^ {\pi} = \left(\mathcal {T} _ {\psi} ^ {\pi}\right) ^ {k} s _ {\psi} ^ {\pi} \\ \Longrightarrow \qquad (\mathcal {T} _ {\psi , \tilde {m}} ^ {\pi}) ^ {2 k} s _ {\psi} ^ {\pi} = s _ {\psi} ^ {\pi}, \\ \end{array}
$$
which we can repeat  $n$  times to obtain  $(\mathcal{T}_{\psi, \tilde{m}}^{\pi})^{nk} s_{\psi}^{\pi} = s_{\psi}^{\pi}$ . Sending  $n \to \infty$  and using the fact that  $\psi$  is continuous gives us  $s_{\psi}^{\pi} = \lim_{n \to \infty} (\mathcal{T}_{\psi, \tilde{m}}^{\pi})^{nk} s_{\psi}^{\pi} = s_{\psi, \tilde{m}}^{\pi}$ , which then gives us  $\tilde{m} \in \mathcal{M}_{\psi}^{k}(\{\pi\}, \{s_{\psi}^{\pi}\})$ .
D General classes of policies
In Section 2, we considered stationary Markov policies. One can further consider history-dependent policies, which don't have to be stationary nor Markov, but simply measurable with respect to the filtration  $\mathcal{F}_t$  given by  $\mathcal{F}_t = \sigma\left(\left(\prod_{i=0}^{t-1}(\mathcal{X} \times \mathcal{A})\right) \times \mathcal{X}\right)$  (where for a collection of sets  $A$ ,  $\sigma(A)$  is the  $\sigma$ -algebra generated by  $A$ ). This is a much larger class of policies, and learning a policy in this class is infeasible in general (Puterman, 2014).
In the risk-neutral setting, the difficulties associated with learning a history-dependent policies can be avoided: for every history-dependent policy, there exists a Markov stationary policy which achieves the same expected return. In particular, no history-dependent policy achieves a return higher than a Markov stationary policy, and thus it suffices to solely consider learning a Markov stationary policy.
Unfortunately, such a result does not exist for the risk-sensitive setting: for a general risk measure, there exists history-dependent policies which achieve a higher objective of return than all Markov stationary policies. Moreover, a Markov stationary policy which is optimal as defined in Section 2 may not exist in general. Despite this negative result, the standard approach in practice is nonetheless to learn an approximately optimal Markov stationary policy (Bäuerle & Ott, 2011; Chow & Ghavamzadeh, 2014; Lim & Malik, 2022), and this is the approach taken in this work as well.
Due to the fact that an optimal policy may not exist, Theorem 4.3 may seem to not generally apply, as it only addresses the case when a Markov stationary optimal policy exists. We now present a weaker version of this theorem, which addresses the case in which such an optimal policy does not exist.
To state the proposition, we first introduce the notion of policy domination. Suppose  $\pi_1, \pi_2 \in \mathbb{P}$ . We say that  $\pi_1$  dominates  $\pi_2$  with respect to  $\rho$  if
$$
\rho (\eta^ {\pi_ {1}} (x)) \geq \rho (\eta^ {\pi_ {2}} (x)), \forall x \in \mathcal {X}.
$$
It is straightforward to see that policy domination provides a partial order over the set of Markov stationary policies  $\mathbb{P}$ . With this in mind, we present the proposition.
Proposition D.1. Let  $\rho$  be any risk measure, and let  $\pi_1, \pi_2$  be policies such that  $\pi_1$  dominates  $\pi_2$  with respect to  $\rho$  in an approximate model  $\tilde{m} \in \mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{T})$ . Then  $\pi_1$  dominates  $\pi_2$  with respect to  $\rho$  in  $m^*$ .
Proof. Let  $\pi_1, \pi_2$  satisfy the statement of the proposition. For contradiction, suppose that  $\pi_1$  does not dominate  $\pi_2$  in  $m^*$ . Then for all  $x \in \mathcal{X}$  we have that
$$
\rho (\eta^ {\pi_ {1}} (x)) \leq \rho (\eta^ {\pi_ {2}} (x)),
$$
$$
\rho \left(\eta^ {\pi_ {1}} (x)\right) <   \rho \left(\eta^ {\pi_ {2}} (x)\right).
$$
$$
\rho (\eta^ {\pi_ {1}} (x)) <   \rho (\eta^ {\pi_ {2}} (x))
$$
$$
\Longleftrightarrow \rho \left(\eta_ {\tilde {m}} ^ {\pi_ {1}} (x)\right) <   \rho \left(\eta_ {\tilde {m}} ^ {\pi_ {2}} (x)\right),
$$
since by assumption of  $\tilde{m}\in \mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{P})$  we have that  $\eta^{\pi} = \eta_{\tilde{m}}^{\pi}$  for any  $\pi \in \mathbb{P}$ . But this contradicts the assumption that  $\pi_1$  dominated  $\pi_2$  in  $\tilde{m}$ , and we are complete.
We note that this proposition should be interpreted as follows: suppose one learns an approximately optimal policy in  $\tilde{m} \in \mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{P})$ , in the sense that it dominates a set of other candidate policies. Then this policy will be approximately optimal in  $m^{*}$ , in the sense that it will still dominate this same set of policies in  $m^{*}$ . We note that it is straightforward to adapt Proposition 5.10 in the same way.
E Empirical details
We begin with a detailed description of the environments used, followed by details on the compute resources used.
E.1 Environment descriptions
E.1.1 Tabular environments
Four rooms
We adapt the stochastic four rooms domain used in Grimm et al. (2021) by making certain states risky. In the original domain, an agent attempts to navigate from the start state (bottom left) to the goal state (top right), by moving up, down, left, or right. At each step however, there is a  $20\%$  chance that the agent slips and moves in a random direction, rather than the intended one. A reward of 1 is achieved for reaching the goal state, and the reward is 0 elsewhere. We then select certain states to become 'risky' states. These states have the same transition dynamics, but modified reward: if they transition in the intended direction they receive a small, positive reward, and if they transition in a random direction they receive a large negative reward. The rewards are chosen so that the expected reward from a state has a slightly positive expectation, so that risk-neutral policies would pass through the state, but risk-averse ones would not.
Windy cliffs
We consider the stochastic adaptation of the cliff walk environment (Sutton & Barto, 2018) as introduced in Bellemare et al. (2023). An agent must walk along a cliff to reach its goal, but at every step, it has a  $1/3$  probability of moving in a random direction. A reward of  $-1$  is obtained for falling off the cliff, and a reward of 1 is obtained for reaching the goal state.
Frozen lake
We use the 8 by 8 frozen lake domain as specified in Brockman et al. (2016). There are four actions corresponding to walking in each direction, however taking an action has a  $1/3$  probability of moving in the intended direction, and a  $1/3$  probability of moving in each of the perpendicular directions. The agent begins in the top left corner, and attempts to reach the goal at the bottom right corner, at which point the agents receive a reward of 1 and the episode ends. Within the environment there are various holes in the ice, entering a hole will provide a reward of -1 and the episode ends. Episodes will also end after 200 timesteps. Following this, there are 3 possible returns for an episode: -1 for
falling in a hole, 1 for reaching the goal, and 0 for reaching the 200 timesteps without reaching a hole state or the goal.
E.1.2 Option trading environment
We use the option trading environment as implemented in Lim & Malik (2022). In particular, the environment simulates the task of learning a policy of when to exercise American call options. The state space is given as  $\mathcal{X} = \mathbb{R}^2$ , where for a given  $\mathcal{X} \ni x = (p,t)$ ,  $p$  represents the price of the underlying stock, and  $t$  represents the time until maturity. The two actions represent holding the stock and executing, and at maturity all options are executed. The training data is generated by assuming that stock prices follow geometric Brownian motion (Li et al., 2009). For evaluation, real stock prices are used, using the data of 10 Dow instruments from 2016-2019.
E.2 Compute time and infrastructure
For the tabular experiments, each model took roughly 1 hour to train on a single CPU, for an approximate total of 120 CPU hours for the tabular set of experiments. For the option trading experiments, training a policy for a given CVaR level took roughly 40 minutes on a single Tesla P100 GPU on average, for an approximate total of 200 GPU hours for this set of experiments.
F Limitations and future work
While we introduced a novel framework and demonstrated strong theoretical and empirical results, our work has limitations, which we now discuss and present as possible directions for future work. The first is investigating how well the statistical functional  $\psi$  used can plan for general risk measures not covered by Proposition 5.10, and deriving bounds on its performance. A second is that our theory relies on the set  $\mathcal{M}_{\mathrm{dist}}^{\infty}(\mathbb{P})$ , while in practice we use  $\mathcal{M}_{\mathrm{dist}}^{\infty}(\Pi)$ , where  $\Pi \subseteq \mathbb{P}$  is a uniformly random subset. Investigating how this affects the theoretical results, along with investigating whether there is a better way to choose the set  $\Pi$ , are interesting questions in this direction.