SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {C V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) = \mathbf {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) + \frac {1}{\alpha} \mathbb {E} _ {q \sim \phi_ {s, a} ^ {t}} [ \mathbf {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) - f (q | s, a, Q) ] ^ {+} \\ = \operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) + \frac {1}{\alpha} \mathbb {E} _ {q \sim \phi_ {s, a} ^ {t}, q \in U _ {\varepsilon}} [ \operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) - f (q | s, a, Q) ] ^ {+} \\ + \frac {1}{\alpha} \mathbb {E} _ {q \sim \phi_ {s, a} ^ {t}, q \in U _ {\varepsilon} ^ {c}} \left[ \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) - f (q | s, a, Q) \right] ^ {+} \\ \end{array}
$$
Since  $\mathrm{VaR}_{\alpha}^{\phi_{s,a}^{t}}(f(p|s,a,Q)) \to f(p_{s,a}^{c}|s,a,Q)$  and  $\phi_{s,a}^{t}(U_{\varepsilon}^{c}) \to 0$ , we have for large enough  $t$ ,
1.  $|\mathrm{VaR}_{\alpha}^{\phi_{s,a}^{t}}(f(p|s,a,Q)) - f(p_{s,a}^{c}|s,a,Q)|\leq \varepsilon ,$
2.  $\mathbb{E}_{q\sim \phi_{s,a}^t,q\in U_\varepsilon}[\mathrm{VaR}_\alpha^{\phi_{s,a}^t}(f(p|s,a,Q)) - f(q|s,a,Q)]^+ \leq 2\varepsilon \frac{|S|\bar{R}}{1 - \gamma},$
3.  $\mathbb{E}_{q\sim \phi_{s,a}^t,q\in U_\varepsilon^c}[\mathrm{VaR}_\alpha^{\phi_{s,a}^t}(f(p|s,a,Q)) - f(q|s,a,Q)]^+ \leq \mathbb{P}(p\in U_\varepsilon^c |\phi_{s,a}^t) \frac{2\bar{R}}{1 - \gamma} \leq 2\varepsilon \frac{|S|\bar{R}}{1 - \gamma}$ .
Then we obtain
$$
\left| \mathrm {C V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} \left(f (p | s, a, Q)\right) - f \left(p _ {s, a} ^ {c} | s, a, Q\right) \right| \leq \varepsilon \left(1 + \frac {4 | \mathcal {S} | \bar {R}}{\alpha (1 - \gamma)}\right)
$$
Again, by arbitrary  $\varepsilon$  and the uniformness in  $Q$ , we obtain the desired result.
Proof of Theorem 4.3
Proof. Proof sketch. The complete proof is technical. We first provide a proof sketch. Intuitively, the bias term converges to 0 since we either have posterior to concentrate on the true parameter or have an increasing sample size for the Monto Carlo estimator, both of which reduce the bias term to zero asymptotically. However, a major difficulty is uniform convergence (in terms of all possible values of  $Q$  and sample size  $N$ ), for which existing results do not give a straightforward guarantee. We prove the results by considering the two cases. First, when infinite observations are available, i.e.,  $||O_{s,a}^{\infty}|| = \infty$ , we construct i.i.d. samples  $h(p_i) \leq f(p_i|s,a,Q)$  with the same sampled  $p_i$  for an arbitrary given  $\varepsilon$ , where  $f(p_i|s,a,Q)$  is defined as in (2). With probability at least  $1 - \varepsilon$ ,  $h(p_i)$  takes a value no less than  $f(p_i|s,a,Q)$  by a constant multiple of  $\varepsilon$ . With probability at least  $\varepsilon$ ,  $h(p_i)$  takes a value bounded by a constant. With the help of the Stirling formula and subtle calculation, we can bound the expectation of order statistics of  $h(p_i)$ , which further lower bounds the Bellman operator estimator. The upper bound can be obtained in a similar way. Together we can bound the bias term. Second, in the case of limited observations, we prove a uniform convergence of empirical distribution for  $f(p_i|s,a,Q)$  when  $Q$  belongs to a bounded set. We carefully partition the probability space as a union of disjoint rectangular sets, to which the Glivenko-Cantelli theorem can be applied and the uniform convergence of the empirical distributions follows. Combining the two cases we complete the proof.
Formal proof. Again, we fix a state-action pair  $(s,a)$ , an observation process  $\omega$  for which we drop the notation for simplicity. Let  $Q$  be any Q-function such that  $||Q||_{\infty} \leq \frac{\bar{R}}{1 - \gamma}$ . We first prove for the VaR risk functional.
Estimator for VaR
we consider the two cases:
-  $O_{s,a}^{\infty} = \infty$ , then  $\phi_{s,a}^{t}$  is consistent at the true parameter  $p_{s,a}^{c}$ .  $\forall \varepsilon > 0$ ,  $\mathbb{P}(p \in U_{\varepsilon}|\phi_{s,a}^{t}) \to 1$  almost surely for such  $\omega$ , where  $U_{\varepsilon} = \left\{p \in \mathbb{R}_{+}^{|\mathcal{S}|} \mid \sum_{x \in \mathcal{S}} p(x) = 1, |p(x) - p_{s,a}^{c}(x)| \leq \varepsilon, \forall x \in \mathcal{S}\right\}$ . Then  $\forall \varepsilon > 0$ ,  $\mathbb{P}(p \in \mathcal{S}) = 1$ .
$U_{\varepsilon}|\phi_{s,a}^{t}) \geq 1 - \varepsilon$  for all large  $t$ . At iteration  $k$  such that  $t_k$  large enough, we obtain  $p_1^{(k)}, p_2^{(k)}, \ldots, p_{N_{s,a}^{(k)}}^{(k)} \sim \phi_{s,a}^{t_k}$ . For each  $p_i^{(k)}$ , we have  $\mathbb{P}(p_i^{(k)} \in U_{\varepsilon}|\phi_{s,a}^{t_k}) \geq 1 - \varepsilon$ . Furthermore, since  $\phi_{s,a}^{t_k}$  is always smooth function, we can find  $\Phi_i^{(k)} \in U_{\varepsilon}$  such that  $\mathbb{P}(p_i^{(k)} \in \Phi_i^{(k)}|\phi_{s,a}^{t_k}) = 1 - \varepsilon$ . Recall
$$
f (p | s, a, Q) = \sum_ {s ^ {\prime} \in \mathcal {S}} p \left(s ^ {\prime}\right) \left[ r \left(s, a, s ^ {\prime}\right) + \gamma \max  _ {b \in \mathcal {A}} Q (s, a) \right].
$$
$$
h (p _ {i} ^ {(k)}) = \left\{ \begin{array}{l l} \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) & \qquad \text {i f} p _ {i} ^ {(k)} \in \Phi_ {i} ^ {(k)} \\ \inf  _ {p} f (p | s, a, Q) & \qquad \text {i f} p _ {i} ^ {(k)} \notin \Phi_ {i} ^ {(k)} \end{array} \right.
$$
It is easy to see  $f(p_i^{(k)}|s,a,Q) \geq h(p_i^{(k)})$ . Notice the distribution of sampled  $p_i^{(k)}$  only depends on  $\phi_{s,a}^{t_k}, N_{s,a}^{(k)}$ , which are further determined by  $\omega_{t_k}$ . Conditioned on  $\omega_{t_k}, h(p_i^{(k)})$ ,  $i = 1,2,\ldots,N_{s,a}^{(k)}$  are i.i.d. random variables taking two values  $\inf_{p\in U_\varepsilon}f(p|s,a,Q)$  and  $\inf_p f(p|s,a,Q)$  with probability at least  $1 - \varepsilon$  and  $\varepsilon$ , respectively. Notice  $\inf_p f(p|s,a,Q) \geq -\frac{\bar{R}}{1 - \gamma}$ . And
$$
\begin{array}{l} | \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) - \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) | \\ \leq \inf _ {p \in U _ {\varepsilon}} f (p | s, a, Q) - f (p _ {s, a} ^ {c} | s, a, Q) | + | f (p _ {s, a} ^ {c} | s, a, Q) - \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) |. \\ \end{array}
$$
By (10), we have  $\left| \inf_{p \in U_{\varepsilon}} f(p|s, a, Q) - f(p_{s,a}^{c}|s, a, Q) \right| \leq \varepsilon \frac{|\mathcal{S}||Q||_{\infty}}{1 - \gamma}$ . By (11), we have  $|f(p_{s,a}^{c}|s, a, Q) - \mathrm{VaR}_{\alpha}^{\phi_{s,a}^{t_k}}(f(p|s, a, Q))| \leq \varepsilon \frac{|\mathcal{S}||Q||_{\infty}}{1 - \gamma}$  for  $k$  large enough. Hence, we obtain
$$
\mid \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) - \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) \mid \leq 2 \varepsilon \frac {| S | | | Q | | _ {\infty}}{1 - \gamma},
$$
where  $||Q||_{\infty}$  again can be bounded by  $\frac{\bar{R}}{1 - \gamma}$ . Denote by  $X_{i} = f(p_{i}^{(k)}|s,a,Q)$  and  $Y_{i} = h(p_{i}^{(k)})$ . We first bound the conditional expectation of the order statistic  $Y_{\lceil N_{s,a}^{(k)}\alpha \rceil:N_{s,a}^{(k)}}$ , i.e.,  $E[Y_{\lceil N_{s,a}^{(k)}\alpha \rceil:N_{s,a}^{(k)}|\omega_{t_k}]$ .
Since  $\{Y_{i}|\omega_{t_{k}}\}$  are i.i.d., we can compute the conditional mass function of  $Y_{[n\alpha ]:n}$  as
$$
\mathbb {P} \left(Y _ {\lceil n \alpha \rceil : n} = \inf  _ {p} f (p | s, a, Q) \mid \omega_ {t _ {k}}\right) = \sum_ {j \geq \lceil n \alpha \rceil} \binom {n} {j} \varepsilon^ {j} (1 - \varepsilon) ^ {n - j}. \tag {12}
$$
We now lower bound the conditional expectation of  $Y_{[n\alpha]:n}$  by upper bound (12). First, we can show  $\binom{n}{j} \varepsilon^j (1 - \varepsilon)^{n-j}$  is decreasing in term of  $j$  for  $j > n\varepsilon$ . To see this, take logarithm on the right hand side we have
$$
\eta (j) = \log (n!) - \log ((n - j)! j!) + j \log \varepsilon + (n - j) \log (1 - \varepsilon).
$$
Compute the difference
$$
\eta (j + 1) - \eta (j) = \log (\frac {n - j}{j + 1} \cdot \frac {\varepsilon}{1 - \varepsilon}).
$$
Then  $\eta(j + 1) - \eta(j) < 0 \Longleftrightarrow j \geq (n + 1)\varepsilon - 1$ . Hence for  $\varepsilon < \alpha$  and  $j \geq n\alpha \geq n\varepsilon > (n + 1)\varepsilon - 1$ ,  $\eta(j)$  is decreasing. Hence we can upper bound (12) by
$$
\begin{array}{l} (n + 1 - \lceil n \alpha \rceil) \left( \begin{array}{c} n \\ \lceil n \alpha \rceil \end{array} \right) \varepsilon^ {\lceil n \alpha \rceil} (1 - \varepsilon) ^ {n - \lceil n \alpha \rceil}. \\ = (n + 1 - \lceil n \alpha \rceil) \frac {n !}{(n - \lceil n \alpha \rceil) ! \lceil n \alpha \rceil !} \varepsilon^ {\lceil n \alpha \rceil} (1 - \varepsilon) ^ {n - \lceil n \alpha \rceil} \tag {13} \\ \end{array}
$$
By Stirling Formula,
$$
\sqrt {2 \pi n} \left(\frac {n}{e}\right) ^ {n} <   n! <   \sqrt {2 \pi n} \left(\frac {n}{e}\right) ^ {n} e ^ {\frac {1}{1 2 n}} <   2 \sqrt {2 \pi n} \left(\frac {n}{e}\right) ^ {n}.
$$
(13) < (n+1-]
$$
= (n + 1 - \lceil n \alpha \rceil) \frac {\sqrt {2} \varepsilon^ {\lceil n \alpha \rceil} (1 - \varepsilon) ^ {n - \lceil n \alpha \rceil}}{\sqrt {(1 - \frac {\lceil n \alpha \rceil}{n})} \left(1 - \frac {\lceil n \alpha \rceil}{n}\right) ^ {n - \lceil n \alpha \rceil} \sqrt {\pi^ {\lceil n \alpha \rceil}} \left(\frac {\lceil n \alpha \rceil}{n}\right) ^ {\lceil n \alpha \rceil}}
$$
For  $n \geq \frac{2}{\alpha(1 - \alpha)}$ , we have
1.  $(n + 1 - \lceil n\alpha \rceil)\leq 2n(1 - \alpha),$
2.  $\varepsilon^{\lceil n\alpha \rceil}\leq \varepsilon^{n\alpha}$
3.  $(1 - \varepsilon)^{n - \lceil n\alpha \rceil}\leq (1 - \varepsilon)^{n(1 - \alpha)} / (1 - \varepsilon),$
4.  $(\frac{\lceil n\alpha\rceil}{n})^{\lceil n\alpha \rceil}\geq (\frac{\lceil n\alpha\rceil}{n})^{n\alpha +1}\geq \alpha^{n\alpha}\cdot \alpha ,$
5.  $1 - \frac{\lceil n\alpha\rceil}{n}\geq \frac{1 - \alpha}{2}$
6.  $(1 - \frac{\lceil n\alpha\rceil}{n})^{n - \lceil n\alpha \rceil}\geq (1 - \frac{\lceil n\alpha\rceil}{n})^{n(1 - \alpha)}\geq (\frac{1 - \alpha}{2})^{n(1 - \alpha)}.$
(13)  $\leq 2\sqrt{\frac{2}{\pi}} n(1 - \alpha)\frac{\varepsilon^{n\alpha}(1 - \varepsilon)^{n(1 - \alpha)} / (1 - \varepsilon)}{\sqrt{\frac{1 - \alpha}{2}}\left(\frac{1 - \alpha}{2}\right)^{n(1 - \alpha)}\sqrt{n\alpha}\alpha^{n\alpha}.$
$$
\begin{array}{l} = \frac {4 \sqrt {(1 - \alpha)}}{(1 - \varepsilon) \alpha^ {\frac {3}{2}} \sqrt {\pi}} \sqrt {n} \left(\frac {\varepsilon^ {\alpha} (1 - \varepsilon) ^ {1 - \alpha}}{\alpha^ {\alpha} (\frac {1 - \alpha}{2}) ^ {1 - \alpha}}\right) ^ {n} \\ = \frac {C}{1 - \varepsilon} \sqrt {n} \beta^ {n}, \\ \end{array}
$$
where  $C = \frac{4\sqrt{(1 - \alpha)}}{\alpha^{\frac{3}{2}}\sqrt{\pi}}$  is a constant and  $\beta = \frac{\varepsilon^{\alpha}(1 - \varepsilon)^{1 - \alpha}}{\alpha^{\alpha}(\frac{1 - \alpha}{2})^{1 - \alpha}}$ . For  $\varepsilon$  small enough, we can ensure  $\beta < 1$ . Let
$$
\zeta (x) = \log (\sqrt {x} \beta^ {x}) = \frac {1}{2} \log x + x \log \beta .
$$
Then, since  $\log \beta < 0$ ,  $\zeta(x)$  attains the maximum at  $x = -\frac{2}{\log \beta}$ . Then we have
$$
\begin{array}{l} \frac {C}{1 - \varepsilon} \sqrt {n} \beta^ {n} = \frac {C}{1 - \varepsilon} \mathrm {e} ^ {\zeta (n)} \\ \leq \frac {C}{1 - \varepsilon} \mathrm {e} ^ {\zeta (- \frac {2}{\log \beta})} \\ = \frac {C}{1 - \varepsilon} \sqrt {- \frac {2}{\log \beta}} \mathrm {e} ^ {- 2} \\ \end{array}
$$
For  $n \leq \left\lfloor \frac{2}{\alpha(1 - \alpha)} \right\rfloor$ , we have
$$
\begin{array}{l} \mathbb {P} \left(Y _ {\lceil n \alpha \rceil : n} = \inf  _ {p} f (p | s, a, Q) | \omega_ {t _ {k}}\right) \\ \leq \mathbb {P} \left(Y _ {1: n} = \inf  _ {p} f (p | s, a, Q) | \omega_ {t _ {k}}\right) \\ = 1 - (1 - \varepsilon) ^ {n} \\ \leq 1 - (1 - \varepsilon) ^ {\left\lfloor \frac {2}{\alpha (1 - \alpha)} \right\rfloor} \\ \end{array}
$$
$$
C ^ {\prime} (\varepsilon) := \max \{\frac {C}{1 - \varepsilon} \sqrt {- \frac {2}{\log \beta}} \mathrm {e} ^ {- 2}, 1 - (1 - \varepsilon) ^ {\lfloor \frac {2}{\alpha (1 - \alpha)} \rfloor} \},
$$
where  $C^\prime (\varepsilon)\to 0$  as  $\varepsilon \rightarrow 0$
$$
\begin{array}{l} \mathbb {E} \left[ Y _ {\lceil n \alpha \rceil : n} | \omega_ {t _ {k}} \right] \\ = \mathbb {P} (Y _ {\lceil n \alpha \rceil : n} = \inf  _ {p i} f (p | s, a, Q) | \omega_ {t _ {k}}) * (\inf  _ {p} f (p | s, a, Q)) + \mathbb {P} (Y _ {\lceil n \alpha \rceil : n} = \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) | \omega_ {t _ {k}}) * \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) \\ \geq \mathbb {P} (Y _ {\lceil n \alpha \rceil : n} = \inf _ {p} f (p | s, a, Q) | \omega_ {t _ {k}}) * (- \frac {\bar {R}}{1 - \gamma}) \\ + \mathbb {P} (Y _ {\lceil n \alpha \rceil : n} = \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) | \omega_ {t _ {k}}) * (\mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) - 2 \varepsilon | \mathcal {S} | | | Q | | _ {\infty}) \\ \geq C ^ {\prime} (\varepsilon) * (- \frac {\bar {R}}{1 - \gamma}) + (1 - C ^ {\prime} (\varepsilon)) * (\mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) - 2 \varepsilon | \mathcal {S} | \frac {\bar {R}}{1 - \gamma}) \\ \end{array}
$$
Hence we have almost surely,
$$
\begin{array}{l} \mathbb {E} \left[ X _ {\lceil N _ {s, a} ^ {(k)} \alpha \rceil : N _ {s, a} ^ {(k)}} - \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) | \omega_ {t _ {k}} \right] \\ \geq \mathbb {E} \left[ Y _ {\lceil N _ {s, a} ^ {(k)} \alpha \rceil : N _ {s, a} ^ {(k)}} - \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) | \omega_ {t _ {k}} \right] \\ \geq - C ^ {\prime} (\varepsilon) \left((\frac {\bar {R}}{1 - \gamma}) + \mathrm {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q))\right) - 2 \varepsilon (1 - C ^ {\prime} (\varepsilon)) | \mathcal {S} | \frac {\bar {R}}{1 - \gamma} \\ \geq - 2 \left(C ^ {\prime} (\varepsilon) + \varepsilon \left(1 - C ^ {\prime} (\varepsilon)\right) | \mathcal {S} |\right) \left(\frac {\bar {R}}{1 - \gamma}\right) \\ =: - \tilde {C} (\varepsilon), \\ \end{array}
$$
where  $\tilde{C} (\varepsilon)\to 0$  as  $\varepsilon \rightarrow 0$
Similarly by constructing
$$
h (p _ {i} ^ {(k)}) = \left\{ \begin{array}{l l} \sup  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) & \qquad \text {i f p _ {i} ^ {(k)} \in \Psi_ {i}} \\ \sup  _ {p} f (p | s, a, Q) & \qquad \text {i f p _ {i} ^ {(k)} \notin \Psi_ {i}} \end{array} \right.
$$
We can obtain
$$
\mathbb {E} \left[ X _ {\lceil N _ {s, a} ^ {(k)} \alpha \rceil : N _ {s, a} ^ {(k)}} - \operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t _ {k}}} (f (p | s, a, Q)) | \omega_ {t _ {k}} \right] \leq \widehat {C} (\varepsilon),
$$
almost surely for  $k$  large enough and  $\widehat{C} (\varepsilon)\to 0$  as  $\varepsilon \rightarrow 0$
Notice  $\mathcal{T}^{\phi^{t_k}}Q(s,a) = \mathrm{VaR}_{\alpha}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))$ . Furthermore, since both  $\tilde{C} (\varepsilon)$  and  $\widehat{C} (\varepsilon)$  do not depend on  $Q$  and by arbitrary  $\varepsilon$ , we obtain
- If  $O_{s,a}^{\infty} < \infty$ , then the posterior distribution on  $(s, a)$  remains the same as  $\phi_{s,a}^{\omega}$  for all large  $t$  and the sample size  $N_{s,a}^{(k)}$  tends to infinity. In this case, by Glivenko-Cantelli theorem,
we know almost surely the empirical distribution conditioned on  $\phi_{s,a}^{\omega}$  (which is further determined by  $\omega$ ),  $F_{N_{s,a}^{(k)}}^{(k)}$ , for the sampled  $p_1^{(k)}, p_2^{(k)}, \ldots, p_{N_{s,a}^{(k)}}^{(k)}$  uniformly converges to the distribution  $\phi_{s,a}^{\omega}$  as sample size  $N_{s,a}^{(k)}$  goes to infinity. Denote by  $X_i^{(k)} = f(p_i^{(k)}|s, a, Q) = \sum_{s' \in \mathcal{S}} p_i^{(k)}(s') (r(s, a, s') + \gamma \max_{b \in \mathcal{A}} Q(s', b)) = \sum_{s' \in \mathcal{S}} d_{s'} p_i^{(k)}(s') = d^\top p_i^{(k)}$ . We want to show uniform convergence of empirical distribution of  $\{X_i^{(k)}\}_{i=1,2,\ldots,N_{s,a}^{(k)}}$  conditioned on  $\phi_{s,a}^{\omega}$  for  $d \in D$  uniformly, where  $D$  contains all the possible value  $d$  can take. Denote by  $G_n^d$  the empirical distribution conditioned on  $\omega_{t_k}$  of  $\{X_i^{(k)}\}_{i=1,2,\ldots,n}$  in terms of  $d$  and  $G^d$  the true distribution of  $d^\top p$  for  $p \sim \phi_{s,a}^{\omega}$ . We want to show
$$
\lim  _ {n \to \infty} \sup  _ {d \in D, x} | G _ {n} ^ {d} (x) - G ^ {d} (x) | \to 0 \qquad \text {a l m o s t s u r e l y}.
$$
Notice that  $|d| \leq \frac{R}{1 - \gamma}$  and  $(d + \varepsilon 1)^{\top}\phi = d^{\top}\phi + \varepsilon$ , we have  $G_{n}^{d}(x) = G_{n}^{d + \varepsilon 1}(x + \varepsilon)$ . Hence we only need to prove for  $D = \{d \in R^{|S|} : M\mathbf{1} \geq d \geq \mathbf{1}\}$ , where  $M = \frac{2R}{1 - \gamma} + 1$ . Denote by  $\mathbb{P}_n(K) = \frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\{p_i \in K\}$  where  $p_1, \dots, p_n \sim \phi_{s,a}^\omega$  are i.i.d samples.
For an arbitrary  $\varepsilon = \frac{1}{\widetilde{N}}$  we have
$$
\begin{array}{l} \mathbb {P} _ {n} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) \\ = \sum_ {k _ {2} = 1} ^ {\tilde {N}} \dots \sum_ {k | \mathcal {S} | = 1} ^ {\tilde {N}} \mathbb {P} _ {n} (\sum_ {s = 1} ^ {| \mathcal {S} |} d _ {s} q _ {s} \leq x, (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \dots , | \mathcal {S} |) \\ \leq \sum_ {k _ {2} = 1} ^ {\widetilde {N}} \dots \sum_ {k _ {| \mathcal {S} |} = 1} ^ {\widetilde {N}} \mathbb {P} _ {n} \left(q _ {1} \leq \frac {1}{d _ {1}} (x - \sum_ {s = 2} ^ {| \mathcal {S} |} d _ {s} (k _ {s} - 1) \varepsilon), (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \ldots , | \mathcal {S} |\right) \\ \end{array}
$$
Since the last probability is just some probability of rectangular, it can be expressed by finite number of cumulative distribution function, which by the Glivenko-Cantelli theorem, uniformly converge to the true distribution. Then, almost surely there exists  $n_{\varepsilon} > 0$  (independent of  $d,\varepsilon ,k_{i}$ ), for  $n\geq n_{\varepsilon}$  we have each
$$
\begin{array}{l} \mathbb {P} _ {n} \left(q _ {1} \leq \frac {1}{d _ {1}} (x - \sum_ {s = 2} ^ {| \mathcal {S} |} d _ {s} (k _ {s} - 1) \varepsilon), (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \dots , | \mathcal {S} |\right) \\ \leq \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(q _ {1} \leq \frac {1}{d _ {1}} (x - \sum_ {s = 2} ^ {| S |} d _ {s} (k _ {s} - 1) \varepsilon), (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \dots , | S |\right) + \frac {1}{\widetilde {N} | S |} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} _ {n} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) \\ \leq \sum_ {k _ {2} = 1} ^ {\widetilde {N}} \dots \sum_ {k | \mathcal {S} | = 1} ^ {\widetilde {N}} \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(q _ {1} \leq \frac {1}{d _ {1}} (x - \sum_ {s = 2} ^ {| \mathcal {S} |} d _ {s} (k _ {s} - 1) \varepsilon), (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \dots , | \mathcal {S} |\right) + \frac {1}{\widetilde {N}} \\ \leq \sum_ {k _ {2} = 1} ^ {\tilde {N}} \dots \sum_ {k _ {| S |} = 1} ^ {\tilde {N}} \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x + \sum_ {s = 2} ^ {| S |} d _ {s} \varepsilon , (k _ {i} - 1) \varepsilon <   q _ {i} \leq k _ {i} \varepsilon , i = 2, \ldots , | S |\right) + \frac {1}{\tilde {N}} \\ = \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x + \sum_ {s = 2} ^ {| S |} d _ {s} \varepsilon\right) + \frac {1}{\widetilde {N}} \\ \leq \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| \mathcal {S} |} d _ {s} q _ {s} \leq x + \frac {(| \mathcal {S} | - 1) M}{\widetilde {N}}\right) + \frac {1}{\widetilde {N}} \\ \end{array}
$$
Since this holds for any  $d, x$ , we have
$$
\begin{array}{l} \sup  _ {d \in D, x \in R} \left\{\mathbb {P} _ {n} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) - \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) \right\} \\ \leq \frac {1}{\widetilde {N}} + \sup  _ {d \in D, x \in R} \left\{\mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x + \frac {(| S | - 1) M}{\widetilde {N}}\right) - \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) \right\} \\ \end{array}
$$
The last term converges to zero as  $\widetilde{N}\to \infty$ . This is indicated by
$$
\begin{array}{l} \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x + \frac {(| S | - 1) M}{\widetilde {N}}\right) - \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x\right) \\ = \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(x <   \sum_ {s = 1} ^ {| S |} d _ {s} q _ {s} \leq x + \frac {(| S | - 1) M}{\tilde {N}}\right) \\ \leq \sum_ {s = 1} ^ {\mathcal {S}} \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\frac {x}{d _ {s}} <   q _ {s} \leq \frac {x}{d _ {s}} + \frac {(| \mathcal {S} | - 1) M}{d _ {s} \tilde {N}}\right) \\ \leq \sum_ {s = 1} ^ {\mathcal {S}} \mathbb {P} _ {q \sim \phi_ {s, a} ^ {\omega}} \left(\frac {x}{d _ {s}} <   q _ {s} \leq \frac {x}{d _ {s}} + \frac {(| \mathcal {S} | - 1) M}{\tilde {N}}\right) \\ \end{array}
$$
Since each  $0 \leq q_{s} \leq 1$  and its marginal distribution  $\phi_{s,a}^{\omega}(q_s)$  is continuous, it is uniformly continuous in  $0 \leq q \leq 1$ . Hence each  $\mathbb{P}_{q \sim \phi_{s,a}^{\omega}}\left(\frac{x}{d_s} < q_s \leq \frac{x}{d_s} + \frac{(|S| - 1)M}{\tilde{N}}\right)$  converge uniformly to 0 as  $\tilde{N} \to \infty$ . Hence by arbitrary  $\tilde{N}$ , we have
$$
\limsup_{n\to \infty}\sup_{d\in D,x\in R}\left\{\mathbb{P}_{n}\left(\sum_{s = 1}^{|S|}d_{s}q_{s}\leq x\right) - \mathbb{P}_{q\sim \phi_{s,a}^{\omega}}\left(\sum_{s = 1}^{|S|}d_{s}q_{s}\leq x\right)\right\} \leq 0.
$$
Similarly, we can obtain the other side as
$$
\liminf_{n\to \infty}\inf_{d\in D,x\in R}\left\{\mathbb{P}_{n}\left(\sum_{s = 1}^{|S|}d_{s}q_{s}\leq x\right) - \mathbb{P}_{q\sim \phi_{s,a}^{\omega}}\left(\sum_{s = 1}^{|S|}d_{s}q_{s}\leq x\right)\right\} \geq 0.
$$
$$
\sup  _ {d \in D, x} | G _ {n} ^ {d} (x) - G ^ {d} (x) | \to 0 \qquad \text {a l m o s t s u r e l y}.
$$
We obtain the uniform convergence for empirical distribution in  $d$ , or equivalently,  $Q$ , conditioned on the observation process  $\omega$ . Hence the quantile estimator  $X_{\lceil n\alpha \rceil :n} \to \mathrm{VaR}_{\alpha}^{\phi_s^{\omega},a}(Q)$  uniformly in  $Q$  almost surely. Furthermore, since  $X_{i}$  is bounded,
$$
\lim_{k\to \infty}\sup_{||Q||_{\infty}\leq \frac{\bar{R}}{1 - \gamma}}\left|\mathbb{E}[\widehat{\mathcal{T}}^{(k)}Q(s,a) - \mathcal{T}^{\phi^{t_{k}}}Q(s,a)|\omega_{t_{k}}]\right| = 0\qquad \text{almost surely}.
$$
Estimator for CVaR
Given sample  $p_1^{(k)}, p_2^{(k)}, \ldots, p_{N_{s,a}^{(k)}}^{(k)}$  and  $X_i = f(p_i^{(k)}|s, a, Q) = \sum_{s' \in \mathcal{S}} p_i^{(k)}(s') [r(s, a, s') + \gamma \max_{b \in \mathcal{A}} Q(s', b)]$ . The Monto Carlo estimator for  $\mathrm{CVaR}_{\alpha}^{\phi_{s,a}^{t_k}}(f(p|s, a, Q))$  is  $\frac{1}{\lceil N_{s,a}^{(k)} \rceil} \sum_{j=1}^{\lceil N_{s,a}^{(k)} \rceil} X_{j:N_{s,a}^{(k)}}$ . Again we consider the two cases:
-  $O_{s,a}^{\infty} = \infty$ . Then we know almost surely  $\phi_{s,a}^{t}$  is consistent at  $p_{s,a}^{c}$ . Then by the proof for bias of VaR estimator,  $\forall 1 > \varepsilon > 0$ ,  $\mathbb{E}[X_{\lceil N_{s,a}^{(k)}\varepsilon \rceil:N_{s,a}^{(k)}}|\omega] \to \mathrm{VaR}_{\varepsilon}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))$  as
$k\to \infty$  uniformly in  $Q$  almost surely. Then