SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we construct a simple MDP to demonstrate that the suboptimality of the CVaR decomposition discussed in Section 3 can also lead to computing a suboptimal policy. First, we give a particular example in which the policy computed according to Chow et al. (2015) is suboptimal. Then, we show that the sub-optimality gap can be arbitrarily large.
To demonstrate the suboptimality of the policy computed in Chow et al. (2015), consider an MDP with two states and three actions and a horizon  $T = 1$  as shown in Figure 4. Also let  $M = 600$  and
Figure 6: The function  $\theta_{\pi}(\cdot)$  and  $\zeta_{s1}$  in CVaR decomposition at  $\alpha = 0.5$  for the example in Figure 4 with  $p_{s_1} = p_{s_2} = 0.5$  and  $M = 600$ .
$M = 600$ $p_{s_2} = 0.75$
$M = 3000$ $p_{s_2} = 0.75$
$M = 600$ $p_{s_2} = 0.9$
$M = 3000$ $p_{s_2} = 0.9$
$M = 600$ $p_{s_2} = 0.99$
Figure 7: CVaR decomposition suboptimality for different  $M$  and  $p_{s2}$  for MDP defined in Figure 4
$M = 3000$ ,  $p_{s_2} = 0.99$
$p_{s_1} = p_{s_2} = 0.5$ . An optimal solution for the  $s_1$  sub-problem in this example is
$$
\mathrm {C V a R} _ {\alpha^ {\prime}} [ r (\tilde {s}) \mid \tilde {s} = s _ {1} ] \quad = \quad \left\{ \begin{array}{l l} \mathrm {C V a R} _ {\alpha^ {\prime}} [ r (s _ {1}, a _ {2}, \tilde {s} ^ {\prime}) ] & \text {i f \alpha^ {\prime} <   0 . 5} \\ \mathrm {C V a R} _ {\alpha^ {\prime}} [ r (s _ {1}, a _ {1}, \tilde {s} ^ {\prime}) ] & \text {i f \alpha^ {\prime} \geq 0 . 5 .} \end{array} \right.
$$
Figure 5 depicts the sub-optimality demonstrated by the CVaR decomposition. CVaR decomposition for policy optimization would over-approximate (black dash) the value function and commit to a sub-optimal solution (red solid). This simple example answers the following important concerns on the suboptimality.
First, CVaR policy optimization decomposition in Chow et al. (2015) can lead one to choose a suboptimal policy. In particular, Figure 5 shows that the CVaR policy optimization decomposition is an overestimate of the return and the decision maker commits to the worst action for CVaR objective  $\alpha \in (0.25,625)$ . Furthermore, action  $a_3$  is never chosen for (Chow et al., 2015) even though it is the only optimal action for  $\alpha \in (0.375,0.6875)$ . We can see from Figure 6 when we inspect the  $\zeta$  function for  $\alpha = 0.5$  that  $a_3$  is optimal with CVaR return of 50. However, the CVaR decomposition would select  $a_1$  or  $a_2$  and over-estimate the CVaR return to 100, but the sub-optimal actions  $a_1$  or  $a_2$  would have the true CVaR return of 0 instead.
Second, there is no upper bound on the sub-optimality of the action chosen with respect CVaR return of the optimal action. The regret of the sub-optimal return is dependent on the distribution of the sub-optimal action and unbounded with respect to the true optimal return, as the performance of the sub-optimal return for  $\alpha \in (0.25, 0.5)$  is worse than the worst case scenario of the best worst-case policy. Increasing the magnitude  $(M > 400)$  for the reward instead of the sub-optimal action  $a_1$  in  $s_1$  in Figure 4 increases the sub-optimality and can be arbitrarily large.
Third, the sub-optimality can occur for any  $\alpha \in (0,1)$ . As above, we use the MDP in Figure 4 and perturb two values to demonstrate the suboptimality by (i) increasing the magnitude  $(M > 400)$  for the reward of the sub-optimal action  $s_1, a_1$ , or (ii) increasing the initial state probability  $(p_{s_2})$ . That leads to increasing the range where the CVaR decomposition for policy optimization yields a sub-optimal policy as shown in Figure 7.
CVaR decomposition suboptimality
C Extension to Longer Horizon
For completeness, we include in this section an extension of the corrected decomposition presented in Theorem 5.2 to horizon  $T > 1$ . Our demonstration establishes that the decomposition proposed in Theorem 1 and 3 of (Li et al., 2022) actually applies to the upper quantile (i.e. value-at-risk) of the cumulated reward rather than the lower one as claimed by the authors. In comparison with the proof found in (Li et al., 2022), we pay close attention to demonstrating that the supremum in (5.1) is attained when finite and that deterministic policies are optimal.
Given any finite MDP with horizon  $T > 0$  and  $\alpha \in [0,1]$ , we define the initial value as
$$
v _ {0} := \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s \in S} \max  _ {a \in \mathcal {A}} q _ {1} \left(s, a, \alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}\right) \mid \alpha \cdot \zeta \leq \hat {\boldsymbol {p}} \right\}. \tag {25}
$$
The state-action value function  $q_{t} \colon S \times \mathcal{A} \times [0,1] \to \mathbb{R}$  for the terminal step  $t = T + 1$  and  $\alpha \in [0,1]$  and each  $s \in S$  and  $a \in \mathcal{A}$  is defined as
$$
q _ {T + 1} (s, a, \alpha) := \left\{ \begin{array}{l l} \infty & \text {i f} \alpha = 1 \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
For each  $t = 1, \dots, T$  and  $\alpha < 1$  the state-action function is defined recursively as
$$
q _ {t} (s, a, \alpha) := \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \max  _ {a ^ {\prime} \in \mathcal {A}} r \left(s, a, s ^ {\prime}\right) + q _ {t + 1} \left(s ^ {\prime}, a ^ {\prime}, \alpha \zeta_ {s ^ {\prime}} p _ {s a s ^ {\prime}} ^ {- 1}\right) \mid \alpha \cdot \zeta \leq p _ {s a} \right\}. \tag {26}
$$
For  $\alpha = 1$ , the state-action value function is defined as
$$
q _ {t} (s, a, 1) := \infty .
$$
To construct the optimal policy, we also need to define a history-dependent risk level  $\bar{\alpha}_t\colon S^t\times \mathcal{A}^{t - 1}\to [0,1]$  that satisfies for each  $t = 1,\dots ,T$  and  $s_1,\ldots ,s_t$  and  $a_1,\ldots ,a_{t - 1}$  that
$$
q _ {t} (s, a, \bar {\alpha} _ {t} (s _ {1: t}, a _ {1: t - 1})) = \min _ {s ^ {\prime} \in \mathcal {S}} \max _ {a ^ {\prime} \in \mathcal {A}} r (s, a, s ^ {\prime}) + q _ {t + 1} (s ^ {\prime}, a ^ {\prime}, \bar {\alpha} _ {t + 1} ([ s _ {1: t}, s ^ {\prime} ], [ a _ {1: t - 1}, a ^ {\prime} ]).
$$
The appropriate values  $\bar{\alpha}_t$  can be readily recovered from the optimal solution to (26). Finally, letting
$$
a _ {t} ^ {*} \left(s _ {t}, \alpha_ {t}\right) \in \left\{ \begin{array}{l l} \mathcal {A} & \text {i f} \alpha_ {t} = 1 \\ \arg \max  _ {a \in \mathcal {A}} q _ {t} \left(s _ {t}, a, \alpha_ {t}\right) & \text {o t h e r w i s e ,} \end{array} \right. \tag {27}
$$
we construct a deterministic history-dependent policy  $\bar{\pi}_t\colon S^t\times \mathcal{A}^{t - 1}\to \mathcal{A}$  for each  $t = 1,\dots ,T$  and  $s_1,\ldots ,s_t$  and  $a_1,\ldots ,a_{t - 1}$  that puts all the probability mass on  $a_{t}^{*}(s_{t},\bar{\alpha}_{t}(s_{1:t},a_{1:t - 1}))$ ), i.e.
$$
\mathbb {P} ^ {\tilde {a} _ {t} \sim \bar {\pi} _ {t} \left(s _ {1: t}, a _ {1: t - 1}\right)} \left(\tilde {a} _ {t} = a _ {t} ^ {*} \left(s _ {t}, \bar {\alpha} _ {t} \left(s _ {1: t}, a _ {1: t - 1}\right)\right)\right) = 1. \tag {28}
$$
The following theorem states that the value function defined above represents the VaR of the returns of the optimal policy. Moreover, the history-dependent policy  $\bar{\pi}$  above is optimal and attains the optimal VaR return.
Theorem C.1. For a finite horizon  $T$  and any  $\alpha \in [0,1]$  we have that  $v_{0}$  defined in (25) coincides with the optimal VaR return:
$$
v _ {0} = \max _ {\pi \in \Pi} \mathrm {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \boldsymbol {\pi} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1})} \left[ \sum_ {t = 1} ^ {T} r (\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}) \right].
$$
Moreover, this optimal return is attained by the policy  $(\bar{\pi}_t)_{t=1}^T$  defined in (27):
$$
v _ {0} = \mathrm {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \bar {\pi} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1})} \left[ \sum_ {t = 1} ^ {T} r (\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}) \right].
$$
In proving Theorem C.1, we will make use of the upper semicontinuity property of  $\mathrm{VaR}_{\alpha}[\tilde{x}]$ , which is presented in the following Lemma for completeness.
Lemma C.2. The function  $\mathrm{VaR}_{\alpha}[\tilde{x}]$  is upper semicontinuous in  $\alpha$  on the interval  $\alpha \in [0,1]$ .
Proof. This follows from the fact that  $\mathrm{VaR}_{\alpha}[\tilde{x}]$  is non-decreasing and right-continuous in terms of  $\alpha$  (see Lemma A.19 in (Follmer and Schied, 2016)).
Proof of Theorem C.1. The proof consists of two steps addressing the two equalities stated by the theorem.
Step 1: For all  $t = 1, \dots, T$ , we let
$$
\hat {q} _ {t} \left(s _ {t}, a _ {t}, \alpha_ {t}\right) :=
$$
$$
\max  _ {\pi \in \Pi_ {t + 1: T}} \operatorname {V a R} _ {\alpha_ {t}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} \left(\tilde {s} _ {t + 1: t ^ {\prime}}, \tilde {a} _ {t + 1: t ^ {\prime} - 1}\right)} \left[ r \left(s _ {t}, a _ {t}, \tilde {s} _ {t + 1}\right) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r \left(\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}\right) \mid s _ {t}, a _ {t} \right]
$$
where  $\Pi_{t+1:T}$  refers to the set of all history-dependent policies starting from time  $t+1$ , and where  $\mathrm{VaR}_{\alpha}[\tilde{x}|s_t, a_t]$  is short for  $\mathrm{VaR}_{\alpha}[\tilde{x}|\tilde{s}_t = s_t, \tilde{a}_t = a_t]$ . We will first show how  $\hat{q}_t(s, a, \alpha) = q_t(s, a, \alpha)$  for all  $t = 1, \ldots, T$ ,  $s \in S$ ,  $a \in \mathcal{A}$ , and  $\alpha \in [0, 1]$ . We follow with our conclusion about  $v_0$ .
First, addressing the case  $\alpha = 1$ , one can easily see that for all  $t = 1,\dots ,T$ ,  $s\in S$ , and  $a\in \mathcal{A}$ , we have that
$$
\begin{array}{l} \hat {q} _ {t} \left(s _ {t}, a _ {t}, 1\right) = \max  _ {\pi \in \Pi_ {t + 1: T}} \operatorname {V a R} _ {1} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} \left(\tilde {s} _ {t + 1: t ^ {\prime}}, \tilde {a} _ {t + 1: t ^ {\prime} - 1}\right)} \left[ r \left(s _ {t}, a _ {t}, \tilde {s} _ {t + 1}\right) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r \left(\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}\right) \mid s _ {t}, a _ {t} \right] \\ = \infty = q _ {t} \left(s _ {t}, a _ {t}, 1\right), \\ \end{array}
$$
based on our definition of value-at-risk for  $\alpha = 1$ .
Next, for  $0 \leq \alpha < 1$  and using  $\pmb{p}$  short for  $\pmb{p}_{sa}$  when can start looking at time  $t = T$ , where we have that for all  $s_T \in S$ , and  $a_T \in \mathcal{A}$ :
$$
\begin{array}{l} \hat {q} _ {T} \left(s _ {T}, a _ {T}, \alpha_ {T}\right) = \operatorname {V a R} _ {\alpha_ {T}} \left[ r \left(s _ {T}, a _ {T}, \tilde {s} _ {t + 1}\right) \mid s _ {T}, a _ {T} \right] \\ = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} \operatorname {V a R} _ {\alpha_ {T} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} [ r (s _ {T}, a _ {T}, s ^ {\prime}) ] \mid \alpha_ {T} \cdot \zeta \leq p \right\} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} \operatorname {V a R} _ {\alpha_ {T} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} [ r (s _ {T}, a _ {T}, s ^ {\prime}) ] \mid \alpha_ {T} \cdot \zeta \leq \boldsymbol {p} \right\} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} r \left(s _ {T}, a _ {T}, s ^ {\prime}\right) + \operatorname {V a R} _ {\alpha_ {T} \zeta_ {s} p _ {s} ^ {- 1}} [ 0 ] \mid \alpha_ {T} \cdot \zeta \leq \boldsymbol {p} \right\} \\ = \max  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {T}, a _ {T}, s ^ {\prime}\right) + \max  _ {a ^ {\prime} \in \mathcal {A}} q _ {T + 1} \left(s ^ {\prime}, a ^ {\prime}, \alpha_ {T} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {T} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ = q _ {T} \left(s _ {T}, a _ {T}, \alpha_ {T}\right), \\ \end{array}
$$
where we first use the definition of  $\hat{q}_T(s,a,\alpha)$ , then the decomposition of VaR in Theorem 5.1. We follow with confirming, based on extreme value theory, that the supremum over  $\zeta$  must be achieved given that  $\mathrm{VaR}_{\alpha}(\tilde{x})$  of a random variable  $\tilde{x}$  is upper semi-continuous (usc) in  $\alpha$  (see Lemma C.2), that the minimum of a finite set of usc functions is usc, and that  $\Delta_S$  is compact. The other two steps follow from the translation invariance of VaR and the definition of  $q_{T + 1}(s,a,\alpha)$ .
Moreover, for all  $1 \leq t < T$  let  $\tilde{h}_{t:t'} := (\tilde{s}_{t:t'}, \tilde{a}_{t:t' - 1})$  as the history between  $t$  and  $t'$ . Using the short-hand  $\tilde{r}_{t'} = r(\tilde{s}_{t'}, \tilde{a}_{t'}, \tilde{s}_{t' + 1})$ , and when  $0 \leq \alpha_{t} < 1$ , we have that:
$$
\begin{array}{l} \hat {q} _ {t} \left(s _ {t}, a _ {t}, \alpha_ {t}\right) = \max  _ {\pi \in \Pi_ {t + 1: T}} \operatorname {V a R} _ {\alpha_ {t}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} \left(\tilde {h} _ {t + 1: t ^ {\prime}}\right)} \left[ r \left(s _ {t}, a _ {t}, \tilde {s} _ {t + 1}\right) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r \left(\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}\right) \mid s _ {t}, a _ {t} \right] \\ = \max  _ {\pi \in \Pi_ {t + 1: T}} \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \operatorname {V a R} _ {\alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} (\tilde {h} _ {t + 1: t ^ {\prime}})} [ r (s _ {t}, a _ {t}, s ^ {\prime}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} \tilde {r} _ {t ^ {\prime}} | \tilde {s} _ {t + 1} = s ^ {\prime} ] | \alpha_ {t} \cdot \zeta \leq \boldsymbol {p} \right\} \\ = \max  _ {\pi \in \Pi_ {t + 1: T}} \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \operatorname {V a R} _ {\alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \boldsymbol {\pi} _ {t ^ {\prime}} (\tilde {h} _ {t + 1: t ^ {\prime}})} [ r (s _ {t}, a _ {t}, s ^ {\prime}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} \tilde {r} _ {t ^ {\prime}} | \tilde {s} _ {t + 1} = s ^ {\prime} ] | \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ \end{array}
$$
$$
\begin{array}{l} = \max  _ {\zeta \in \Delta_ {S}} \left\{\max  _ {\pi \in \Pi_ {t + 1: T}} \min  _ {s ^ {\prime} \in \mathcal {S}} \operatorname {V a R} _ {\alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \boldsymbol {\pi} _ {t ^ {\prime}} (\tilde {h} _ {t + 1: t ^ {\prime}})} [ r (s _ {t}, a _ {t}, s ^ {\prime}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} \tilde {r} _ {t ^ {\prime}} | \tilde {s} _ {t + 1} = s ^ {\prime} ] | \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \max  _ {\boldsymbol {d} \in \Delta_ {A}, \left\{\pi^ {a} \right\} _ {a \in A} \in \Pi_ {t + 2: T} ^ {| A |}} \operatorname {V a R} _ {\alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t + 1} \sim \boldsymbol {d}, \tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} ^ {\tilde {a} _ {t + 1}} \left(\tilde {h} _ {t + 2: t ^ {\prime}}\right)} [ r (s _ {t}, a _ {t}, s ^ {\prime}) + \right. \\ \left. \sum_ {t ^ {\prime} = t + 1} ^ {T} \tilde {r} _ {t ^ {\prime}} \mid \tilde {s} _ {t + 1} = s ^ {\prime} ] \mid \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} r \left(s _ {t}, a _ {t}, s ^ {\prime}\right) + \right. \\ \max  _ {\boldsymbol {d} \in \Delta_ {A}, \left\{\pi^ {a} \right\} _ {a \in A} \in \Pi_ {t + 2; T} ^ {| \mathcal {A} |}} \operatorname {V a R} _ {\alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t + 1} \sim \boldsymbol {d}, \tilde {a} _ {t ^ {\prime}} \sim \boldsymbol {\pi} _ {t ^ {\prime}} ^ {\tilde {a} _ {t + 1}} (\tilde {h} _ {t + 2; t ^ {\prime}})} [ \sum_ {t ^ {\prime} = t + 1} ^ {T} \tilde {r} _ {t ^ {\prime}} \mid \tilde {s} _ {t + 1} = s ^ {\prime} ] \mid \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \Bigg \} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} r \left(s _ {t}, a _ {t}, s ^ {\prime}\right) + \hat {v} _ {t + 1} \left(s ^ {\prime}, \alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \tag {29} \\ \end{array}
$$
where, as before, we start by exploiting the decomposition Theorem 5.1 and explicating that the supremum is attained. We then change the order of the two maximums, followed by changing the order of  $\max_{\pi} \min_{s'}$  with  $\min_{s'} \max_{\pi}$ , which follows based on the interchangeability property of the minimum operation (Shapiro, 2017, proposition 2.2). We finally introduced a  $(s, \alpha)$ -value function operator which can be reduced as follows:
$$
\begin{array}{l} \hat{v}_{t}(s,\alpha):= \max_{\boldsymbol {d}\in \Delta_{A},\{\pi^{a}\}_{a\in A}\in \Pi_{t + 1:T}^{|A|}}\operatorname{VaR}_{\alpha}^{\tilde{a}_{t}\sim \boldsymbol {d},\tilde{a}_{t^{\prime}}\sim \boldsymbol{\pi}_{t^{\prime}}^{\tilde{a}_{t}}(\tilde{h}_{t + 1:t})}[\sum_{t^{\prime} = t}^{T}r(\tilde{s}_{t^{\prime}},\tilde{a}_{t^{\prime}},\tilde{s}_{t^{\prime} + 1})|\tilde{s}_{t} = s] \\ = \max  _ {\left\{\pi^ {a} \right\} _ {a \in \mathcal {A}} \in \Pi_ {t + 1: T} ^ {| \mathcal {A} |}} \max  _ {\boldsymbol {d} \in \Delta_ {A}} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \boldsymbol {d}, \tilde {a} _ {t ^ {\prime}} \sim \boldsymbol {\pi} _ {t ^ {\prime}} ^ {\tilde {a} _ {t}} (\tilde {h} _ {t + 1: t})} \left[ \sum_ {t ^ {\prime} = t} ^ {T} r \left(\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}\right) | \tilde {s} _ {t} = s ^ {\prime} \right] \\ = \max  _ {\left\{\pi^ {a} \right\} _ {a \in \mathcal {A}} \in \Pi_ {t + 1: T} ^ {| \mathcal {A} |}} \max  _ {a \in \mathcal {A}} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t ^ {\prime}} \sim \boldsymbol {\pi} _ {t ^ {\prime}} ^ {a} (\tilde {h} _ {t + 1: t})} [ r (s, a, \tilde {s} _ {t + 1}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r (\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}) | \tilde {s} _ {t} = s, \tilde {a} _ {t} = a ] \\ = \max  _ {a \in \mathcal {A}} \max  _ {\pi \in \Pi_ {t + 1: T}} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} (\tilde {h} _ {t + 1: t ^ {\prime}})} [ r (s, a, \tilde {s} _ {t + 1}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r (\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}) | \tilde {s} _ {t} = s, \tilde {a} _ {t} = a ] \\ = \max  _ {a \in \mathcal {A}} \hat {q} _ {t} (s, a, \alpha), \tag {30} \\ \end{array}
$$
where we exploited the fact that value-at-risk is a mixture quasi-convex function (Delage et al., 2019), meaning that it cannot be improved by a randomized policy.
Hence, replacing  $\hat{v}$  back in equation (29), we get
$$
\begin{array}{l} \hat {q} _ {t} \left(s _ {t}, a _ {t}, \alpha_ {t}\right) = \max  _ {\boldsymbol {\zeta} \in \Delta_ {\mathcal {S}}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t}, a _ {t}, s ^ {\prime}\right) + \max  _ {a \in \mathcal {A}} \hat {q} _ {t + 1} \left(s ^ {\prime}, \alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {t} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ = \max  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t}, a _ {t}, s ^ {\prime}\right) + \max  _ {a \in \mathcal {A}} q _ {t + 1} \left(s ^ {\prime}, \alpha_ {t} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {t} \cdot \zeta \leq \boldsymbol {p} \right\} \\ = q _ {t} \left(s _ {t}, a _ {t}, \alpha_ {t}\right). \\ \end{array}
$$
Finalizing our conclusion, we get using the short-hand  $\tilde{r}_t = r(\tilde{s}_t,\tilde{a}_t,\tilde{s}_{t + 1})$  that
$$
\begin{array}{l} \max  _ {\pi \in \Pi} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \pi_ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} \tilde {r} _ {t} \right] = \\ = \max  _ {\pi \in \Pi} \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} \operatorname {V a R} _ {\alpha \zeta_ {s ^ {\prime}} \hat {p} _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t} \sim \boldsymbol {\pi} _ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} \tilde {r} _ {t} \mid \tilde {s} _ {1} = s ^ {\prime} \right] \mid \alpha \cdot \zeta \leq \hat {\boldsymbol {p}} \right\} \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \max  _ {\boldsymbol {d} \in \Delta_ {A}, \left\{\pi^ {a} \right\} _ {a \in \mathcal {A} \in \Pi_ {2: T} ^ {| \mathcal {A} |}}} \operatorname {V a R} _ {\alpha \zeta_ {s ^ {\prime}} \hat {p} _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {1} \sim \boldsymbol {d}, \tilde {a} _ {t} \sim \boldsymbol {\pi} _ {t} ^ {\tilde {a} _ {1}} (\tilde {h} _ {2: t})} \left[ \sum_ {t = 1} ^ {T} \tilde {r} _ {t} \mid \tilde {s} _ {1} = s ^ {\prime} \right] \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\} \\ \end{array}
$$
$$
\begin{array}{l} = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \max  _ {a \in \mathcal {A}, \pi \in \Pi_ {2: T}} \operatorname {V a R} _ {\alpha \zeta_ {s ^ {\prime}} \hat {p} _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t} \sim \boldsymbol {\pi} _ {t} (\tilde {h} _ {2: t})} \left[ \sum_ {t = 1} ^ {T} \tilde {r} _ {t} \mid \tilde {s} _ {1} = s ^ {\prime} \right] \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\} \\ = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \hat {q} _ {1} \left(s ^ {\prime}, a, \alpha \zeta_ {s ^ {\prime}} \hat {p} _ {s ^ {\prime}} ^ {- 1}\right) | \tilde {s} _ {1} = s ^ {\prime} ] | \alpha \cdot \zeta \leq \hat {\boldsymbol {p}} \right\} \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left[ \min  _ {s ^ {\prime} \in S} \max  _ {a \in \mathcal {A}} q _ {1} \left(s ^ {\prime}, a, \alpha \zeta_ {s ^ {\prime}} \hat {p} _ {s ^ {\prime}} ^ {- 1}\right) | \tilde {s} _ {1} = s ^ {\prime} ] | \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right] \\ = v _ {0}, \\ \end{array}
$$
where we employ the same steps as for reducing  $\hat{q}_t$  to  $q_t$ .
Step 2: Regarding the optimality of the proposed policy, we will show inductively that
$$
\bar {\hat {q}} _ {t} \left(s _ {t}, a _ {t}, \alpha_ {t}\right) :=
$$
$$
\operatorname {V a R} _ {\alpha_ {t}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \bar {\pi} _ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} \left[ r (s _ {t}, a _ {t}, \tilde {s} _ {t + 1}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r (\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}) \mid s _ {t}, a _ {t}, \bar {\alpha} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1}) = \alpha_ {t} \right]
$$
is an upper envelope for  $\hat{q}_t(s_t, a_t, \alpha_t)$  for all  $1 \leq t \leq T$ . Conditioning on  $\alpha_t$  in the definition above is necessary because the policy  $\hat{\pi}$  depends on the risk level  $\alpha$ . This can then be exploited to obtain the following inequality: