SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{\lceil N _ {s , a} ^ {(k)} \alpha \rceil} \sum_ {j = 1} ^ {\lceil N _ {s, a} ^ {(k)} \alpha \rceil} X _ {j: N _ {s, a} ^ {(k)}} | \omega_ {t _ {k}} \right] \\ = \mathbb {E} \left[ \frac {1}{\lceil N _ {s , a} ^ {(k)} \alpha \rceil} \sum_ {j = 1} ^ {\lceil N _ {s, a} ^ {(k)} \varepsilon \rceil} X _ {j: N _ {s, a} ^ {(k)}} | \omega_ {t _ {k}} \right] + \mathbb {E} \left[ \frac {1}{\lceil N _ {s , a} ^ {(k)} \alpha \rceil} \sum_ {j = \lceil N _ {s, a} ^ {(k)} \varepsilon \rceil} ^ {\lceil N _ {s, a} ^ {(k)} \alpha \rceil} X _ {j: N _ {s, a} ^ {(k)}} | \omega_ {t _ {k}} \right] \\ \geq - \frac {\left[ N _ {s , a} ^ {(k)} \varepsilon \right]}{\left[ N _ {s , a} ^ {(k)} \alpha \right]} \frac {\bar {R}}{1 - \gamma} + \frac {\left[ N _ {s , a} ^ {(k)} \alpha \right] - \left[ N _ {s , a} ^ {(k)} \varepsilon \right]}{\left[ N _ {s , a} ^ {(k)} \alpha \right]} \mathbb {E} \left[ X _ {\left[ N _ {s, a} ^ {(k)} \varepsilon \right]: N _ {s, a} ^ {(k)}} \mid \omega_ {t _ {k}} \right] \\ \end{array}
$$
Since  $\mathbb{E}[X_{\lceil N_{s,a}^{(k)}\varepsilon \rceil:N_{s,a}^{(k)}}|\phi_{s,a}^{t_k}]\to \mathrm{VaR}_{\varepsilon}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))$ $\mathrm{VaR}_{\varepsilon}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))\to$ $f(p_{s,a}^{c}|s,a,Q)$ $\mathrm{CVaR}_{\alpha}^{\phi_{s,a}^{t}}(f(p|s,a,Q))\to f(p_{s,a}^{c}|s,a,Q)$  uniformly for  $-\frac{\bar{R}}{1 - \gamma}\leq Q\leq \frac{\bar{R}}{1 - \gamma}$  as  $k\rightarrow \infty$  and  $\varepsilon$  is chosen arbitrarily, we have  $\lim \inf_{k\to \infty}\{\mathbb{E}\left[\frac{1}{\lceil N_{s,a}^{(k)}\alpha\rceil}\sum_{j = 1}^{\lceil N_{s,a}^{(k)}\alpha \rceil}X_{j:N_{s,a}^{(k)}}|\omega_{t_k}\right] - \mathrm{CVaR}_{\alpha}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))\} \geq$  0 uniformly in  $Q$  almost surely. Similarly we can also prove  $\lim \sup_{k\to \infty}\{\mathbb{E}\left[\frac{1}{\lceil n\alpha\rceil}\sum_{j = 1}^{\lceil n\alpha \rceil}X_{j:n}|\omega_{t_k}\right] - \mathrm{CVaR}_{\alpha}^{\phi_{s,a}^{t_k}}(f(p|s,a,Q))\} \leq 0$  uniformly in  $Q$  almost surely. Combining the two inequalities together, we obtain
- For  $O_{s,a}^{\infty}(\infty) < \infty$ . By proof of VaR estimator we know the empirical distribution of  $f(p_i^{(k)}|s,a,Q)$  conditioned on  $\omega$ , converge uniformly to  $\phi_{s,a}^{\omega}$ . Hence the CVaR estimator also has the desired result.
Lemma B.3. [24] Consider a stochastic process  $(\alpha_{t},\Delta_{t},g), t \geq 0$ , where  $\alpha_{t},\Delta_{t},g: X \to \Re$  satisfy the equations
$$
\Delta_ {t + 1} (x) = \left(1 - \alpha_ {t} (x)\right) \Delta_ {t} (x) + \alpha_ {t} (x) g (x), \quad x \in X, \quad t = 0, 1, 2, \dots
$$
Let  $P_{t}$  be a sequence of increasing  $\sigma$ -fields such that  $\alpha_{0}$  and  $\Delta_0$  are  $P_0$ -measurable and  $\alpha_{t}, \Delta_{t}$  and  $F_{t - 1}$  are  $P_{t}$ -measurable. Let  $||\cdot||_{W}$  denote some weighted maximum norm. Assume that the following hold:
1. the set  $X$  is finite.
2.  $0 \leq \alpha_{t}(x) \leq 1, \sum_{t} \alpha_{t}(x) = \infty, \sum_{t} \alpha_{t}^{2}(x) < \infty$  w.p.1.
3.  $\| E\{g(\cdot)\mid P_t\} \| _W\leq \kappa \| \Delta_t\| _W + c_t$  , where  $\kappa \in [0,1)$  and  $c_{t}$  converges to zero w.p.1.
4.  $\operatorname{Var}\{g(x) \mid P_t\} \leq K(1 + \| \Delta_t \|_W)^2$ , where  $K$  is some constant.
Then,  $\Delta_t$  converges to zero with probability one (w.p.1).
Proof for Theorem 4.4.
Proof. Following the same notation in Theorem 4.3. Denote by  $Q^{(k)}$  the Q-function in iteration  $k$ . Define
$$
\operatorname {b i a s} _ {(k)} (s, a) = \mathbb {E} \left[ \hat {\mathcal {T}} ^ {(k)} Q ^ {(k)} (s, a) - \mathcal {T} ^ {\phi^ {t _ {k}}} Q ^ {(k)} (s, a) \mid \omega_ {t _ {k}} \right].
$$
Then by Theorem 4.3 and observing  $||Q^{(k)}|| < \frac{\bar{R}}{1 - \gamma}$  almost surely, we have almost surely,
$$
\lim  _ {k \to \infty} \operatorname {b i a s} _ {(k)} (s, a) = 0 \qquad \forall s \in \mathcal {S}, a \in \mathcal {A}.
$$
Denote by  $\Delta^{(k)} = Q^{(k)} - Q^{\omega,*}$ .  $\Delta^{(k)}$  satisfies
$$
\Delta^ {(k + 1)} (s, a) = (1 - \lambda_ {k}) \Delta^ {(k)} + \lambda_ {k} F _ {(k)},
$$
where  $F_{(k)} = \widehat{\mathcal{T}}^{(k)}Q^{(k)} - Q^{\omega,*}$ . Let  $\mathcal{H}_{(k)}$  be the  $\sigma$ -field generated by  $\{\{\Delta_{(\kappa)}\}_{\kappa=1}^{k}, \{\lambda_{(\kappa)}\}_{\kappa=1}^{k}, \{F_{(\kappa)}\}_{\kappa=1}^{k-1}\}$ . We have
$$
\begin{array}{l} | \mathbb {E} [ F _ {(k)} (s, a) | \mathcal {H} _ {(k)} ] | \\ = \left| \mathbb {E} \left[ \mathbb {E} \left[ F _ {(k)} (s, a) \mid \omega_ {t _ {k}}, \mathcal {H} _ {(k)} \right] \mid \mathcal {H} _ {(k)} \right] \right| \\ = \left| \mathbb {E} \left[ \mathcal {T} ^ {(k)} \left(Q ^ {(k)}\right) (s, a) - Q ^ {\phi^ {t _ {k}}, *} (s, a) + Q ^ {\phi^ {t _ {k}}, *} (s, a) - Q ^ {\omega , *} (s, a) + \operatorname {b i a s} _ {(k)} (s, a) \mid \mathcal {H} _ {(k)} \right] \right| \\ = \left| \mathbb {E} \left[ \hat {\mathcal {T}} ^ {(k)} \left(Q ^ {(k)} - Q ^ {\phi^ {t k}, *}\right) \mid \mathcal {H} _ {(k)} \right] + \mathbb {E} \left[ Q ^ {\phi^ {t k}, *} (s, a) - Q ^ {\omega , *} (s, a) \mid \mathcal {H} _ {(k)} \right] + \mathbb {E} \left[ \operatorname {b i a s} _ {(k)} (s, a) \mid \mathcal {H} _ {(k)} \right] \right| \\ \leq \gamma \mathbb {E} [ \max  _ {s ^ {\prime} \in \mathcal {S}} | \max  _ {b \in \mathcal {A}} Q ^ {(k)} (s ^ {\prime}, b) - \max  _ {b \in \mathcal {A}} Q ^ {\phi^ {t _ {k}}, *} (s ^ {\prime}, b) | | \mathcal {H} _ {(k)} ] + \mathbb {E} [ | | Q ^ {\phi^ {t _ {k}}, *} - Q ^ {\omega , *} | | _ {\infty} | \mathcal {H} _ {(k)} ] + \mathbb {E} [ | \mathrm {b i a s} _ {(k)} (s, a) | | \mathcal {H} _ {(k)} ] \\ \leq \gamma \mathbb {E} [ \| Q ^ {(k)} - Q ^ {\phi^ {t _ {k}}, *} \| _ {\infty} | \mathcal {H} _ {(k)} ] + \mathbb {E} [ \| Q ^ {\phi^ {t _ {k}}, *} - Q ^ {\omega , *} \| _ {\infty} | \mathcal {H} _ {(k)} ] + \mathbb {E} [ | \operatorname {b i a s} _ {(k)} (s, a) | | \mathcal {H} _ {(k)} ] \\ \leq \gamma \mathbb {E} [ \| Q ^ {(k)} - Q ^ {\omega , *} \| \infty | \mathcal {H} _ {(k)} ] + (1 + \gamma) \mathbb {E} [ \| Q ^ {\phi^ {t k}, *} - Q ^ {\omega , *} \| \infty | \mathcal {H} _ {(k)} ] + \mathbb {E} [ | \operatorname {b i a s} _ {(k)} (s, a) | | \mathcal {H} _ {(k)} ] \\ \leq \gamma | | \Delta^ {(k)} | | _ {\infty} + (1 + \gamma) \mathbb {E} [ | | Q ^ {\phi^ {t _ {k}}, *} - Q ^ {\omega , *} | | _ {\infty} | \mathcal {H} _ {(k)} ] + \mathbb {E} [ | \operatorname {b i a s} _ {(k)} (s, a) | | \mathcal {H} _ {(k)} ]. \\ \end{array}
$$
where  $||\cdot ||_{\infty}$  denote the entry-wise sup norm. The second equality holds since the Monto Carlo sampling procedure in iteration  $k$  only depends on the posterior  $\phi^{t_k}$  and sample size  $N_{s,a}^{(k)}$ , which are completely determined by the past observations  $\omega_{t_k}$ . By Proposition 4.2 and Theorem 4.3, we know both  $||Q^{\phi^{t_k},*} - Q^{\omega ,*}||_{\infty}$  and  $|\mathrm{bias}_{(k)}(s,a)|$  converge to 0 almost surely. Since they are both bounded, by the bounded convergence theorem the last two terms in the last inequalities converge to zero almost surely. Furthermore, since  $|F_{(k)}(s,a)|\leq \bar{R} +||\Delta^{(k)}||_{\infty}$ , we have
$$
\begin{array}{l} \operatorname {V a r} \left(F _ {(k)} (s, a) \mid \mathcal {H} _ {(k)}\right) \leq (\bar {R} + \left\| \Delta^ {(k)} \right\| _ {\infty}) ^ {2} \\ \leq (\bar {R} ^ {2} + 1) (1 + | | \Delta^ {(k)} | | _ {\infty}) ^ {2}. \\ \end{array}
$$
By Lemma B.3, we obtain  $||\Delta^{(k)}||_{\infty}$  converges to 0 almost surely, which completes the proof.