SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \pi_ {t} (\tilde {h} _ {1: t})} [ \sum_ {t = 1} ^ {T} r (\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}) ] = \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {\mathcal {S}}} \left\{\min  _ {s \in \mathcal {S}} \operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} _ {t} \sim \bar {\pi} _ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} r \left(\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) | \tilde {s} _ {1} = s \right] \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\} \\ = \sup  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left\{\min  _ {s \in S} \operatorname {V a R} _ {\alpha \zeta_ {s} \hat {p} _ {s} ^ {- 1}} ^ {\tilde {a} _ {t} \sim \bar {\boldsymbol {\pi}} _ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} r \left(\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) | \tilde {s} _ {1} = s, \tilde {a} _ {1} = a _ {1} ^ {*} (s, \bar {\alpha} _ {1} (s)) \right] \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {\rho}} \right\} \\ \geq \min  _ {s \in \mathcal {S}} \operatorname {V a R} _ {\bar {\alpha} _ {1} (s)} ^ {\tilde {a} _ {t} \sim \bar {\pi} _ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} r \left(\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) | \tilde {s} _ {1} = s, \tilde {a} _ {1} = a _ {1} ^ {*} (s, \bar {\alpha} _ {1} (s)), \bar {\alpha} _ {1} (\tilde {s} _ {1}) = \bar {\alpha} _ {1} (s) \right] \\ = \min  _ {s \in \mathcal {S}} \bar {q} _ {1} (s, a _ {1} ^ {*} (s, \bar {\alpha} _ {1} (s)), \bar {\alpha} _ {1} (s)) \geq \min  _ {s \in \mathcal {S}} \hat {q} _ {1} (s, a _ {1} ^ {*} (s, \bar {\alpha} _ {1} (s)), \bar {\alpha} _ {1} (s)) \\ = \min  _ {s \in \mathcal {S}} q _ {1} (s, a _ {1} ^ {*} (s, \bar {\alpha} _ {1} (s)), \bar {\alpha} _ {1} (s)) = \min  _ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} q _ {1} (s, a, \bar {\alpha} _ {1} (s)) \\ = \max  _ {\boldsymbol {\zeta} \in \Delta_ {S}} \left\{\min  _ {s \in S} \max  _ {a \in \mathcal {A}} q _ {1} (s, a, \bar {\alpha} _ {1} (s)) \mid \alpha \cdot \boldsymbol {\zeta} \leq \hat {\boldsymbol {p}} \right\} = v _ {0} \\ = \max  _ {\pi \in \Pi} \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \pi_ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} r \left(\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) \right] \\ \geq \operatorname {V a R} _ {\alpha} ^ {\tilde {a} _ {t} \sim \bar {\pi} _ {t} (\tilde {h} _ {1: t})} \left[ \sum_ {t = 1} ^ {T} r \left(\tilde {s} _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) \right], \\ \end{array}
$$
where the first step comes from the decomposition of VaR, the second from the definition of  $\bar{\pi}_1$ . The third step follows from the feasibility of  $\zeta_s \coloneqq \bar{\alpha}_1(s) / (\alpha \hat{p}_s^{-1})$ . We then exploit the definition of  $\bar{\hat{q}}_1$ , the fact that it upper bounds  $\hat{q}_1$ , the equivalence of  $\hat{q}_1$  and  $q_1$ , and definition of  $a_1^*$ . On the sixth line, we exploit the definition of  $\bar{\alpha}_1(s)$ , followed with the definition of  $v_0$ . These steps would therefore confirm that  $\bar{\pi}$  is optimal.
We are left with showing that  $\bar{\hat{q}}_t$  upper bounds  $\hat{q}_t$  for all  $t = 1, \dots, T$ . Specifically, starting with  $t = T$ , we have:
$$
\bar {\hat {q}} _ {T} \left(s _ {T}, a _ {T}, \alpha_ {T}\right) = \operatorname {V a R} _ {\alpha_ {T}} \left[ r \left(s _ {T}, a _ {T}, \tilde {s} _ {T + 1}\right) \mid s _ {T}, a _ {T} \right] = \hat {q} _ {T} \left(s _ {T}, a _ {T}, \alpha_ {T}\right) \tag {31}
$$
Then, for  $t < T$  we focus on the case  $\alpha_{t} < 1$ , since otherwise we have  $\infty \geq \infty$ . Specifically, we first define
$$
\bar {v} _ {t} (s _ {t}, \alpha_ {t}) := \mathrm {V a R} _ {\alpha_ {t}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \bar {\pi} _ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} [ r (s _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}) + \sum_ {t ^ {\prime} = t + 1} ^ {T} r (\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}) | s _ {t}, \bar {\alpha} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1}) = \alpha_ {t} ],
$$
which captures the value-at-risk at level  $\alpha_{t}$  of the reward-to-go looking forward from time  $t$  when running  $\bar{\pi}$  given that the history up to time  $t$  satisfies  $\bar{\alpha}_{t}(\tilde{s}_{1:t},\tilde{a}_{1:t - 1}) = \alpha_{t}$ .
We then take inductive steps starting from  $t = T$  down to  $t = 2$ . At each  $t$ , we can verify that if  $\bar{\hat{q}}_t$  is an upper envelope for  $\hat{q}_t$ , then:
$$
\bar {v} _ {t} (s _ {t}, 1) = \infty \geq \infty = \max  _ {a \in \mathcal {A}} \hat {q} _ {t} (s _ {t}, a, 1) = \hat {v} _ {t} (s _ {t}, 1)
$$
and that for  $0 \leq \alpha_{t} < 1$ :
$$
\begin{array}{l} \bar {v} _ {t} \left(s _ {t}, \alpha_ {t}\right) = \operatorname {V a R} _ {\alpha_ {t}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \tilde {\pi} _ {t ^ {\prime}} \left(\tilde {h} _ {1: t ^ {\prime}}\right)} \left[ r \left(s _ {t}, \tilde {a} _ {t}, \tilde {s} _ {t + 1}\right) \right. \\ \left. + \sum_ {t ^ {\prime} = t + 1} ^ {T} r \left(\tilde {s} _ {t ^ {\prime}}, \tilde {a} _ {t ^ {\prime}}, \tilde {s} _ {t ^ {\prime} + 1}\right) \mid s _ {t}, \tilde {a} _ {t} = a _ {t} ^ {*} \left(s _ {t}, \alpha_ {t}\right), \bar {\alpha} _ {t} \left(\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1}\right) = \alpha_ {t} \right] \\ = \bar {\hat {q}} _ {t} \left(s _ {t}, a _ {t} ^ {*} \left(s _ {t}, \alpha_ {t}\right), \alpha_ {t}\right) \geq \hat {q} _ {t} \left(s _ {t}, a _ {t} ^ {*} \left(s _ {t}, \alpha_ {t}\right), \alpha_ {t}\right) = q _ {t} \left(s _ {t}, a _ {t} ^ {*} \left(s _ {t}, \alpha_ {t}\right), \alpha_ {t}\right) \\ = \max _ {a \in \mathcal {A}} q _ {t} (s _ {t}, a, \alpha_ {t}) = \max _ {a \in \mathcal {A}} \hat {q} _ {t} (s _ {t}, a, \alpha_ {t}) = \hat {v} _ {t} (s _ {t}, \alpha_ {t}), \\ \end{array}
$$
where we first used the definition of  $\bar{\pi}$ , then used the definition of  $\hat{q}_T$ , followed with the assumed property that  $\hat{q}_t$  is an upper envelope for  $\hat{q}_t$ . We then employ the equivalence of  $\hat{q}_t$  and  $q_t$  twice, the definition of  $a_t^*$ , and the relationship between  $\hat{v}_t$  and  $\hat{q}_t$  established in (30).
Next, we confirm that if  $\bar{v}_t$  is an upper envelope for  $\hat{v}_t$ , then taking one step back and using the short-hand  $\tilde{r}_{t'} = r(\tilde{s}_{t'}, \tilde{a}_{t'}, \tilde{s}_{t' + 1})$  we get:
$$
\hat {\tilde {q}} _ {t - 1} \left(s _ {t - 1}, a _ {t - 1}, \alpha_ {t - 1}\right) :=
$$
$$
\begin{array}{l} \operatorname {V a R} _ {\alpha_ {t - 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \pi_ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} \left[ r \left(s _ {t - 1}, a _ {t - 1}, \tilde {s} _ {t}\right) + \sum_ {t ^ {\prime} = t} ^ {T} \tilde {r} _ {t ^ {\prime}} \mid s _ {t - 1}, a _ {t - 1}, \bar {\alpha} _ {t - 1} \left(\tilde {s} _ {1: t - 1}, \tilde {a} _ {1: t - 2}\right) = \alpha_ {t - 1} \right] \\ = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. \operatorname {V a R} _ {\alpha_ {t - 1} \zeta_ {s ^ {\prime}} \boldsymbol {p} _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \tilde {\pi} _ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} \left[ \sum_ {t ^ {\prime} = t} ^ {T} \tilde {r} _ {t ^ {\prime}} \mid \tilde {s} _ {t} = s ^ {\prime}, a _ {t - 1}, \bar {\alpha} _ {t - 1} \left(\tilde {s} _ {1: t - 1}, \tilde {a} _ {1: t - 2}\right) = \alpha_ {t - 1} \right] \mid \alpha_ {t - 1} \cdot \zeta \leq \boldsymbol {p} \right\} \\ = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in \mathcal {S}} r (s _ {t - 1}, a _ {t - 1}, s ^ {\prime}) + \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. \operatorname {V a R} _ {\alpha_ {t - 1} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}} ^ {\tilde {a} _ {t ^ {\prime}} \sim \bar {\pi} _ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} \left[ \sum_ {t ^ {\prime} = t} ^ {T} \tilde {r} _ {t ^ {\prime}} \mid \tilde {s} _ {t} = s ^ {\prime}, \bar {\alpha} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1}) = g (\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}) \right] \mid \alpha_ {t - 1} \cdot \boldsymbol {\zeta} \leq \boldsymbol {p} \right\} \\ \geq \min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {V a R} _ {g (\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime})} ^ {\tilde {a} _ {t ^ {\prime}} \sim \bar {\pi} _ {t ^ {\prime}} (\tilde {h} _ {1: t ^ {\prime}})} [ \sum_ {t ^ {\prime} = t} ^ {T} \tilde {r} _ {t ^ {\prime}} | s _ {t} = s ^ {\prime}, \bar {\alpha} _ {t} (\tilde {s} _ {1: t}, \tilde {a} _ {1: t - 1}) = g (\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}) ] \\ = \min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \bar {v} _ {t} \left(s ^ {\prime}, g \left(\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right)\right) \\ \geq \min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \hat {v} _ {t} \left(s ^ {\prime}, g \left(\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right)\right) \\ = \min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \max  _ {a \in \mathcal {A}} \hat {q} _ {t} \left(s ^ {\prime}, a, g \left(\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right)\right) \\ = \min  _ {s ^ {\prime} \in \mathcal {S}} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \max  _ {a \in \mathcal {A}} q _ {t} \left(s ^ {\prime}, a, g \left(\alpha_ {t - 1}, s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right)\right) \\ = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \max  _ {a \in A} q _ {t} \left(s ^ {\prime}, a, \alpha_ {t - 1} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {t - 1} \cdot \zeta \leq p \right\} \\ \end{array}
$$
$$
\begin{array}{l} = \sup  _ {\zeta \in \Delta_ {S}} \left\{\min  _ {s ^ {\prime} \in S} r \left(s _ {t - 1}, a _ {t - 1}, s ^ {\prime}\right) + \hat {v} _ {t} \left(s ^ {\prime}, \alpha_ {t - 1} \zeta_ {s ^ {\prime}} p _ {s ^ {\prime}} ^ {- 1}\right) \mid \alpha_ {t - 1} \cdot \zeta \leq p \right\} \\ = \hat {q} _ {t - 1} \left(s _ {t - 1}, a _ {t - 1}, \alpha_ {t - 1}\right). \\ \end{array}
$$
$$
g(\alpha ,s,a,s^{\prime}):= \alpha p_{s,a,s^{\prime}}^{-1}\cdot \left[ \operatorname *{arg  }\max_{\boldsymbol {\zeta}\in \Delta_{S}:\alpha \cdot \boldsymbol {\zeta}\leq \boldsymbol{p}_{s,a}}\min_{s^{\prime \prime}\in S}r(s,a,s^{\prime \prime}) + \max_{a\in \mathcal{A}}q_{t}(s,a,\alpha \zeta_{s^{\prime \prime}}p_{s,a,s^{\prime \prime}}^{-1})\right]_{s^{\prime}}
$$
The first step comes from the decomposition of VaR, the second from the fact that  $\bar{\alpha}_{t}$  is known given that  $\bar{\alpha}_{t-1}$ ,  $\tilde{s}_{t}$ , and  $\tilde{a}_{t-1}$  are fixed while  $\sum_{t'=t}^{T} r(\tilde{s}_{t'}, \tilde{a}_{t'}, \tilde{s}_{t'+1})$  only depends on  $\tilde{s}_{t}$  and  $\bar{\alpha}_{t}$ . The third step follows from replacing the supremum over  $\zeta$  with a feasible member of  $\Delta_{S}$ . The fourth step follows from the definition of  $\bar{v}_{t}$ , followed with the fact that it upper bounds  $\hat{v}_{t}$ . Finally, we exploit the definition of  $g(\cdot)$  and of  $\hat{q}_{t-1}(\cdot)$ . This completes the inductive proof demonstrating that  $\hat{q}_{t}(\cdot)$  is an upper envelope for  $\hat{q}_{t}(\cdot)$  for all  $1 \leq t \leq T$ .