SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Checklist of notations for different Q-functions.
Notation	Explanation
Qφ,*	Optimal Q-function of BRMDP with posterior φ
Qω,*	Data-conditional optimal Q-function conditioned on observation process ω
Qt	Q-function at stage t given by Algorithm 1
Qc,*	Optimal Q-function of the true environment
We first introduce the following lemma that guarantees the maximum taken over randomized policies is equivalent to being taken over only deterministic policies.
Lemma B.1. (Deterministic Bellman operator)
$$
\begin{array}{l} \mathcal {L} ^ {\phi} V (s) = \max  _ {a \in A} \rho_ {p \sim \phi_ {s, a}} \left(\mathbb {E} _ {s \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]\right) \\ = \sup  _ {\pi \in \Pi} \mathbb {E} _ {a \sim \pi (s)} \left[ \rho_ {p \sim \phi_ {s, a}} \left(\mathbb {E} _ {s ^ {\prime} \sim p} \left[ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) \right]\right) \right]. \\ \end{array}
$$
Proof. For an arbitrary randomized policy  $\pi$ , an arbitrary  $V$ , denote by  $\pi(s, a) = \mathbb{P}(\pi(s) = a)$ . We have
$$
\sup _ {a \in \mathcal {A}} \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]) \geq \sum_ {a \in \mathcal {A}} \pi (s, a) \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]).
$$
$$
\mathcal {L} ^ {\phi} V (s) = \max _ {a \in \mathcal {A}} \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]) \geq \sup _ {\pi \in \Pi} \mathbb {E} _ {a \sim \pi (s)} [ \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]) ].
$$
The other direction holds naturally since a deterministic policy is also a randomized policy.
Lemma B.2. (Non-expansive risk measure) Suppose the risk measure  $\rho$  satisfies Assumption 2.3, then
$$
| \rho (f _ {1} (\xi)) - \rho (f _ {2} (\xi)) | \leq | | f _ {1} (\xi) - f _ {2} (\xi) | | _ {\xi , \infty},
$$
where  $||\cdot ||_{\xi ,\infty}$  is the sup norm with probability measure  $\mathbb{P}$ .
$$
\rho_ {\xi} (f _ {1} (\xi)) = \rho_ {\xi} (f _ {1} (\xi) + f _ {2} (\xi) - f _ {1} (\xi)) \leq \rho_ {\xi} (f _ {1} (\xi) + \| f _ {2} (\xi) - f _ {1} (\xi) \| _ {\xi , \infty}) = \rho_ {\xi} (f _ {1} (\xi)) + \| f _ {2} (\xi) - f _ {1} (\xi) \| _ {\xi , \infty}.
$$
$$
\rho_ {\xi} \left(f _ {2} (\xi)\right) \leq \rho_ {\xi} \left(f _ {1} (\xi)\right) + \left\| f _ {2} (\xi) - f _ {1} (\xi) \right\| _ {\xi , \infty}.
$$
Combining together we complete the proof.
Proof of Theorem 2.4
Proof. proof of 1.  $\forall \varepsilon >0$ , there exists a deterministic policy  $\pi$ , such that  $\forall s\in S$
$$
\mathcal {L} ^ {\phi} V (s) \leq \rho_ {p \sim \phi_ {s, \pi (s)}} \left(\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, \pi (s), s ^ {\prime}) + \gamma V (s ^ {\prime}) ]\right) + \varepsilon .
$$
$$
\begin{array}{l} \mathcal {L} ^ {\phi} V (s) - \mathcal {L} ^ {\phi} U (s) \\ \leq \rho_ {p \sim \phi_ {s, \pi (s)}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, \pi (s), s ^ {\prime}) + \gamma V (s ^ {\prime}) ]) - \rho_ {p \sim \phi_ {s, \pi (s)}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, \pi (s), s ^ {\prime}) + \gamma U (s ^ {\prime}) ]) + \varepsilon \\ \leq \gamma | | \mathbb {E} _ {s ^ {\prime} \sim p} [ V (s ^ {\prime}) - U (s ^ {\prime}) ] | | _ {p, \infty} + \varepsilon \\ \leq \gamma | | V - U | | _ {\infty} + \varepsilon \\ \end{array}
$$
Since  $\varepsilon$  can be chosen arbitrarily,  $||\mathcal{L}^{\phi}V - \mathcal{L}^{\phi}U||\leq \gamma ||V - U||_{\infty}$ . Switching the position of  $V$  and  $U$ , we obtain the desired result.
Proof of 2. For the first part, since  $\mathcal{L}^{\phi}$  is a contraction mapping, there exists a unique  $V^{\phi,*}$  such that  $\mathcal{L}^{\phi} V^{\phi,*} = V^{\phi,*}$ . For the second part, let  $\pi'$  be an arbitrary randomized policy. We have
$$
\begin{array}{l} V ^ {\phi , *} (s) = \mathcal {L} ^ {\phi} V ^ {\phi , *} (s) \\ = \sup  _ {a \in \mathcal {A}} \left\{\rho_ {p _ {1} \sim \phi_ {s, a}} \left(\mathbb {E} _ {s _ {1} \sim p _ {1}} \left[ r (s, a, s _ {1}) + \gamma V ^ {\phi , *} (s _ {1}) \right]\right) \right\} \\ = \sup  _ {\pi \in \Pi} \sum_ {a \in \mathcal {A}} \pi (s, a) \rho_ {p _ {1} \sim \phi_ {s, a}} \left(\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s, a, s _ {1}) + \gamma V ^ {\phi , *} (s _ {1}) ]\right) \\ \geq \sum_ {a \in \mathcal {A}} \pi^ {\prime} (s, a) \rho_ {p _ {1} \sim \phi_ {s, a}} (\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s, a, s _ {1}) + \gamma V ^ {\phi , *} (s _ {1}) ]) \\ \end{array}
$$
where the third equality holds because of Lemma B.1. Furthermore, by Assumption 2.3.2, we get
$$
V ^ {\phi , *} \geq \sum_ {a \in \mathcal {A}} \pi^ {\prime} (s, a) \rho_ {p _ {1} \sim \phi_ {s, a}} \left(\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s, a, s _ {1}) + \gamma \sum_ {a _ {1} \in \mathcal {A}} \pi^ {\prime} (s _ {1}, a _ {1}) \rho_ {p _ {2} \sim \phi_ {s _ {1}, a _ {1}}} \left(\mathbb {E} _ {s _ {2} \sim p _ {2}} [ r (s _ {1}, a _ {1}, s _ {2}) + \gamma V ^ {\phi , *} (s _ {2}) ]\right) ]\right). \tag {4}
$$
Define  $\mathcal{L}_{\pi'}^{\phi}$  such that  $\forall s \in S$  and  $V \in \mathbb{R}^{|\mathcal{S}|}$ ,
$$
\mathcal {L} _ {\pi^ {\prime}} ^ {\phi} V (s) = \sum_ {a \in \mathcal {A}} \pi^ {\prime} (s, a) \rho_ {p _ {1} \sim \phi_ {s, a}} (\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s, a, s _ {1}) + \gamma V (s _ {1}) ]).
$$
By the same induction as (4), we can derive
$$
V ^ {\phi , *} \geq \left(\mathcal {L} _ {\pi^ {\prime}} ^ {\phi}\right) ^ {k} V ^ {\phi , *}, \forall k \geq 1.
$$
$$
V ^ {0, \pi^ {\prime}} (s) = \sum_ {a \in \mathcal {A}} \pi^ {\prime} (s, a) \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s _ {1} \sim p} [ r (s, a, s _ {1}) ])
$$
$$
V ^ {k + 1, \pi^ {\prime}} (s) = \sum_ {a \in \mathcal {A}} \pi^ {\prime} (s, a) \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s _ {1} \sim p} [ r (s, a, s _ {1}) + \gamma V ^ {k, \pi^ {\prime}} (s _ {1}) ]).
$$
By Assumption 2.3 and Lemma B.2, we have
$$
V ^ {\phi , *} (s) \geq \left(\mathcal {L} _ {\pi^ {\prime}} ^ {\phi}\right) ^ {k} V ^ {\phi , *} (s) \geq V ^ {k, \pi^ {\prime}} (s) - \gamma^ {k} \| V ^ {\phi , *} \| _ {\infty} \tag {5}
$$
As  $k\to \infty$
$$
| V ^ {\phi , \pi^ {\prime}} (s) - V ^ {k, \pi^ {\prime}} (s) | \leq \gamma^ {k + 1} | | V ^ {\phi , \pi^ {\prime}} | | _ {\infty} \leq \frac {\gamma^ {k + 1}}{1 - \gamma} \bar {R} \to 0,
$$
where  $V^{\phi ,\pi '}$  is the value function of BRMDP with posterior  $\phi$  and policy  $\pi^\prime$ . Then from (5) we obtain
$$
V ^ {\phi , *} (s) \geq V ^ {\phi , \pi^ {\prime}} (s) - \gamma^ {k} \| V ^ {\phi , *} \| _ {\infty} - \frac {\gamma^ {k + 1}}{1 - \gamma} \bar {R}, \quad \forall s \in \mathcal {S}.
$$
By taking  $k\to \infty$  , we have  $V^{\phi ,*}(s)\geq V^{\phi ,\pi '}(s)$  . Since  $\pi^\prime$  is chosen arbitrarily, we get the desired result.
Proof of Theorem 2.6
Proof. Denote by
$$
f (p | s, a, V) = \mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ] = \sum_ {s ^ {\prime} \in \mathcal {S}} p (s ^ {\prime}) [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ].
$$
We first bound  $|f(p|s, a, V) - f(p'|s, a, V)|$  in terms of  $\| p - p' \|_{\infty}$  for  $\| V \|_{\infty} \leq \frac{\bar{R}}{1 - \gamma}$ . To see this,
$$
\begin{array}{l} | f (p | s, a, V) - f (p ^ {\prime} | s, a, V) | = \left| \sum_ {x \in \mathcal {S}} [ p (x) - p ^ {\prime} (x) ] [ r (s, a, x) + \gamma V (x) ] \right| \\ \leq | \mathcal {S} | [ \bar {R} + \| V \| _ {\infty} ] \| p - p ^ {\prime} \| _ {\infty} \tag {6} \\ \leq \frac {| \mathcal {S} | \bar {R}}{1 - \gamma} \| p - p ^ {\prime} \| _ {\infty}. \\ \end{array}
$$
Now consider  $p \sim \phi_{s,a}$ . Since  $\phi_{s,a}^{0} = 1$ , we have the posterior parameter  $\phi_{s,a}(s') = o_{s,a,s'} + 1$ , where  $o(s,a,s')$  is the number of transition from  $s$  to  $s'$  taken action  $a$  in the data set. Let  $\widehat{\mathcal{M}}$  denote the empirical MDP generated with the same data which are used to estimate  $\phi$ , with the empirical transition probability  $\widehat{p}_{s,a}(s') = \frac{o_{s,a,s'}}{O_{s,a}}$ . Let  $p_{s,a}^{c}$  be the true transition distribution. Since  $\{o_{s,a,s'}\}_{s' \in \mathcal{S}}$  follows the multinomial distribution with parameter  $O_{s,a}$ ;  $p_{s,a}^{c}$ , we have
$$
\mathbb {P} (| \widehat {p} _ {s, a} (s ^ {\prime}) - p _ {s, a} ^ {c} (s ^ {\prime}) | \geq \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}) \leq \frac {\alpha}{| \mathcal {S} |} O _ {s, a} ^ {- \frac {1}{3}}
$$
by Chebyshev inequality. Define
$$
\bar {p} _ {s, a} (s ^ {\prime}) = \mathbb {E} _ {p \sim \phi_ {s, a}} [ p (s ^ {\prime}) ] = \frac {\phi_ {s , a} (s ^ {\prime})}{\sum_ {s ^ {\prime \prime} \in \mathcal {S}} \phi_ {s , a} (s ^ {\prime \prime})} = \frac {o _ {s , a , s ^ {\prime}} + 1}{O _ {s , a} + | \mathcal {S} |},
$$
$$
\sigma_ {s, a} ^ {2} (s ^ {\prime}) = \mathbb {E} _ {p \sim \phi_ {s, a}} [ (p (s ^ {\prime}) - \bar {p} _ {s, a} (s ^ {\prime})) ^ {2} ] = \frac {\phi_ {s , a} (s ^ {\prime}) (\| \phi_ {s , a} \| _ {1} - \phi_ {s , a} (s ^ {\prime})}{\| \phi_ {s , a} \| _ {1} ^ {2} (\| \phi_ {s , a} \| _ {1} + 1)} \leq \frac {1}{\| \phi_ {s , a} \| _ {1}} = \frac {1}{O _ {s , a} + | \mathcal {S} |}.
$$
$$
\begin{array}{l} \left| p \left(s ^ {\prime}\right) - \hat {p} _ {s, a} \left(s ^ {\prime}\right) \right| \leq \left| p \left(s ^ {\prime}\right) - \bar {p} _ {s, a} \left(s ^ {\prime}\right) \right| + \left| \bar {p} _ {s, a} \left(s ^ {\prime}\right) - \hat {p} _ {s, a} \left(s ^ {\prime}\right) \right| \\ = | p \left(s ^ {\prime}\right) - \bar {p} _ {s, a} \left(s ^ {\prime}\right) | + \left| \frac {o _ {s , a , s ^ {\prime}} + 1}{O _ {s , a} + | \mathcal {S} |} - \frac {o _ {s , a , s ^ {\prime}}}{O _ {s , a}} \right| \\ \leq \left| p \left(s ^ {\prime}\right) - \bar {p} _ {s, a} \left(s ^ {\prime}\right) \right| + \frac {1}{O _ {s , a}}. \\ \end{array}
$$
Since  $|\mathcal{S}|\geq 1,\alpha \leq 1,O_{s,a}\geq 1,\sqrt{\frac{|\mathcal{S}|}{\alpha}} O_{s,a}^{-\frac{1}{3}} - \frac{1}{O_{s,a}}\geq 0$  . Then, with probability at least  $1 - \frac{\alpha}{|S|} O_{s,a}^{-\frac{1}{3}}$
$$
\begin{array}{l} \mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - p _ {s, a} ^ {c} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}\right) \\ \leq \mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - \bar {p} _ {s, a} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}} - | \bar {p} _ {s, a} (s ^ {\prime}) - \hat {p} _ {s, a} (s ^ {\prime}) | - | \hat {p} _ {s, a} (s ^ {\prime}) - p _ {s, a} ^ {c} (s ^ {\prime}) |\right) \\ \leq \mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - \bar {p} _ {s, a} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}} - \frac {1}{O _ {s , a}} - \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}\right) \\ \leq \mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - \bar {p} _ {s, a} (s ^ {\prime}) | \geq \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}\right) \\ \leq \mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - \bar {p} _ {s, a} (s ^ {\prime}) | \geq \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}\right) \\ \leq \frac {\alpha O _ {s , a} ^ {\frac {2}{3}}}{| \mathcal {S} |} \sigma_ {s, a} ^ {2} \left(s ^ {\prime}\right) \tag {7} \\ \leq \frac {\alpha}{| \mathcal {S} |}, \\ \end{array}
$$
where the second last inequality holds by Chebyshev inequality. Replacing  $a$  with  $\pi(s)$ , we obtain with probability at least  $1 - \frac{\alpha}{|\mathcal{S}|} O_{s, \pi(s)}^{-\frac{1}{3}} \geq 1 - \frac{\alpha}{|\mathcal{S}|} \bar{O}^{-\frac{1}{3}}$ ,
$$
\mathbb {P} _ {p \sim \phi_ {s, \pi (s)}} \left(| p (s ^ {\prime}) - p _ {s, \pi (s)} ^ {c} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, \pi (s)} ^ {- \frac {1}{3}}\right) \leq \frac {\alpha}{| \mathcal {S} |}.
$$
This further implies with probability at least  $1 - \frac{\alpha}{|\mathcal{S}|}\bar{O}^{-\frac{1}{3}}$
$$
\mathbb {P} _ {p \sim \phi_ {s, \pi (s)}} \left(\| p - p _ {s, \pi (s)} ^ {c} \| _ {\infty} \geq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, \pi (s)} ^ {- \frac {1}{3}}\right) \leq \sum_ {s ^ {\prime} \in \mathcal {S}} \mathbb {P} _ {p \sim \phi_ {s, \pi (s)}} \left(| p (s ^ {\prime}) - p _ {s, \pi (s)} ^ {c} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, \pi (s)} ^ {- \frac {1}{3}}\right) \leq \alpha .
$$
Let  $U = \{p\| p - p_{s,\pi (s)}^c\|_\infty \leq 3\sqrt{\frac{|\mathcal{S}|}{\alpha}} O_{s,\pi (s)}^{-\frac{1}{3}}\}$ . Since with probability at least  $1 - \frac{\alpha}{|\mathcal{S}|}\bar{O}^{-\frac{1}{3}}$ ,  $\mathrm{VaR}_{\alpha}$  is the  $\alpha$ -quantile, and  $\mathbb{P}_{\phi_{s,a}}(U)\geq 1 - \alpha$ , we have
$$
\inf  _ {p \in U} f (p | s, \pi (s), V) \leq (\operatorname {V a R} _ {\alpha}) _ {p \sim \phi_ {s, \pi (s)}} (f (p | s, \pi (s), V)) \leq \sup  _ {p \in U} f (p | s, \pi (s), V).
$$
Hence with probability at least  $1 - \frac{\alpha}{|\mathcal{S}|}\bar{O}^{-\frac{1}{3}}$
$$
\begin{array}{l} \left| \left(\operatorname {V a R} _ {\alpha}\right) _ {p \sim \phi_ {s, \pi (s)}} \left(f (p | s, \pi (s), V)\right) - f \left(p _ {s, \pi (s)} ^ {c} \mid s, \pi (s), V\right) \right| \\ \leq \sup  _ {p \in U} \| f (p | s, \pi (s), V) - f (\bar {p} _ {s, \pi (s)} | s, \pi (s), V) \| \\ \leq \frac {\left| S \right| \bar {R}}{1 - \gamma} \sup  _ {p \in U} \| p - \bar {p} _ {s, \pi (s)} \| _ {\infty} \tag {8} \\ \leq 3 \sqrt {\frac {| S |}{\alpha} O _ {s , \pi (s)} ^ {- \frac {1}{3}} \frac {| S | \bar {R}}{1 - \gamma}} \\ \leq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} \bar {O} ^ {- \frac {1}{3}} \frac {| \mathcal {S} | \bar {R}}{1 - \gamma}, \\ \end{array}
$$
where the second last equality is by (6). Since this holds for all  $s$ , then we obtain with probability  $1 - \alpha \bar{O}^{-\frac{1}{3}}$ ,
$$
| (\mathrm {V a R} _ {\alpha}) _ {p \sim \phi_ {s, \pi (s)}} (f (p | s, \pi (s), V)) - f (\hat {p} _ {s, \pi (s)} | s, \pi (s), V) | \leq 3 \sqrt {\frac {| S |}{\alpha}} \bar {O} ^ {- \frac {1}{3}} \frac {| S | \bar {R}}{1 - \gamma^ {2}}
$$
holds for all  $s \in S$ .
Finally, let  $\mathcal{L}^{\phi ,\pi}$  denote the Bellman operator for BRMDP with posterior  $\phi$  and policy  $\pi$ , that is  $V^{\phi ,\pi} = \mathcal{L}^{\phi ,\pi}V^{\phi ,\pi} = (\mathrm{VaR}_{\alpha})_{p\sim \phi_{s,\pi (s)}}(f(p|s,\pi (s),V^{\phi ,\pi}))$ . Similar as Theorem 2.4,  $\mathcal{L}^{\phi ,\pi}$  is a  $\gamma$ -contraction mapping. We have
$$
\begin{array}{l} \left| V ^ {\phi , \pi} (s) - V ^ {c, \pi} (s) \right| = \left| \mathcal {L} ^ {\phi , \pi} V ^ {\phi , \pi} (s) - \mathcal {L} ^ {\phi , \pi} V ^ {c, \pi} (s) + \mathcal {L} ^ {\phi , \pi} V ^ {c, \pi} (s) - V ^ {c, \pi} (s) \right| \\ \leq \left\| \mathcal {L} ^ {\phi , \pi} V ^ {\phi , \pi} - \mathcal {L} ^ {\phi , \pi} V ^ {c, \pi} \right\| _ {\infty} + \left| \mathcal {L} ^ {\phi , \pi} V ^ {c, \pi} (s) - V ^ {c, \pi} (s) \right| \\ \leq \gamma \| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} + | \mathcal {L} ^ {\phi , \pi} V ^ {c, \pi} (s) - V ^ {c, \pi} (s) |. \\ \end{array}
$$
Since  $\| V^{c,\pi}\|_{\infty} \leq \frac{\bar{R}}{1 - \lambda}$ , we have the second term  $|\mathcal{L}^{\phi ,\pi}V^{c,\pi} - V^{c,\pi}| \leq 3\sqrt{\frac{|S|}{\alpha}}\bar{O}^{-\frac{1}{3}}\frac{|S|\bar{R}}{1 - \gamma}$  by (8). Maximize over  $s$  on both sides, we obtain
$$
\| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} \leq \lambda \| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} + \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| \mathcal {S} |}{\alpha}} \frac {3 | \mathcal {S} | \bar {R}}{(1 - \gamma)}.
$$
$$
\| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} \leq \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {3 | S | \bar {R}}{(1 - \gamma) ^ {2}} \leq \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {5 | S | \bar {R}}{(1 - \gamma) ^ {2}}.
$$
For  $\rho$  is  $\mathrm{CVaR}_{\alpha}$ , notice by (7), we have with probability at least  $1 - \frac{\alpha}{|S|} O_{s,a}^{-\frac{1}{3}}$ ,
$$
\mathbb {P} _ {p \sim \phi_ {s, a}} \left(| p (s ^ {\prime}) - p _ {s, a} ^ {c} (s ^ {\prime}) | \geq 3 \sqrt {\frac {| S |}{\alpha}} O _ {s, a} ^ {- \frac {1}{3}}\right) \leq \frac {\alpha}{| S |} O _ {s, a} ^ {- \frac {1}{3}}
$$
Hence, with probability at least  $1 - \frac{\alpha}{|S|} O_{s,\pi (s)}^{-\frac{1}{3}} \geq 1 - \frac{\alpha}{|S|}\bar{O}^{-\frac{1}{3}}$ ,
$$
\mathbb {P} _ {p \sim \phi_ {s, \pi (s)}} \left(\| p - p _ {s, \pi (s)} ^ {c} \| _ {\infty} \geq 3 \sqrt {\frac {| \mathcal {S} |}{\alpha}} O _ {s, \pi (s)} ^ {- \frac {1}{3}}\right) \leq \alpha O _ {s, \pi (s)} ^ {- \frac {1}{3}}.
$$
Recall  $\mathrm{CVaR}_{\alpha}(X)$  is the conditional expectation on  $\{X \leq \mathrm{VaR}_{\alpha}(X)\}$ . Let  $U = \{p \| p - p_{s, \pi(s)}^{c} \|_{\infty} \leq 3 \sqrt{\frac{|S|}{\alpha}} O_{s, \pi(s)}^{-\frac{1}{3}}\}$  and  $W = \{p | f(p | s, \pi(s), V) \leq (\mathrm{VaR}_{\alpha})_{p \sim \phi_{s, \pi(s)}}(f(p | s, \pi(s), V))\}$  for  $|V|_{\infty} \leq \frac{R}{1 - \lambda}$ . Then, with probability at least  $1 - \frac{\alpha}{|S|} \bar{O}^{-\frac{1}{3}}$ ,
$$
\begin{array}{l} \left| \left(\mathrm {C V a R} _ {\alpha}\right) _ {p \sim \phi_ {s, \pi (s)}} \left(f (p | s, \pi (s), V)\right) - f \left(\hat {p} _ {s, \pi (s)} | s, \pi (s), V\right) \right| \\ = \frac {1}{\alpha} \left| \int_ {W \cap U} (f (p | s, \pi (s), V)) - f (\hat {p} _ {s, \pi (s)} | s, \pi (s), V) \mathrm {d} \mathbb {P} _ {\phi_ {s, \pi (s)}} + \frac {1}{\alpha} \int_ {W \cap U ^ {c}} (f (p | s, \pi (s), V)) - f (\hat {p} _ {s, \pi (s)} | s, \pi (s), V) \mathrm {d} \mathbb {P} _ {\phi_ {s, a}} \right| \\ \leq \frac {1}{\alpha} \int_ {W \cap U} | (f (p | s, \pi (s), V)) - f (\hat {p} _ {s, \pi (s)} | s, \pi (s), V) |   \mathrm {d} \mathbb {P} _ {\phi_ {s, \pi (s)}} + \frac {1}{\alpha} \int_ {W \cap U ^ {c}} | (f (p | s, \pi (s), V)) - f (\hat {p} _ {s, \pi (s)} | s, \pi (s), V) |   \mathrm {d} \mathbb {P} _ {\phi_ {s, \pi (s)}} \\ \leq \frac {1}{\alpha} \left\{\alpha \cdot O _ {s, \pi (s)} ^ {- \frac {1}{3}} \frac {2 \bar {R}}{1 - \gamma} + \alpha \cdot 3 \sqrt {\frac {| S |}{\alpha}} O _ {s, \pi (s)} ^ {- \frac {1}{3}} \frac {| S | \bar {R}}{1 - \gamma} \right\} \\ \leq O _ {s, \pi (s)} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {5 | S | \bar {R}}{1 - \gamma} \\ \leq \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| \mathcal {S} |}{\alpha}} \frac {5 | \mathcal {S} | \bar {R}}{1 - \gamma}. \\ \end{array}
$$
Then, following the same proof as for  $\mathrm{VaR}_{\alpha}$ , with probability at least  $1 - \alpha \bar{O}^{\frac{1}{3}}$ , we can bound
$$
\| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} \leq \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| \mathcal {S} |}{\alpha}} \frac {5 | \mathcal {S} | \bar {R}}{(1 - \gamma) ^ {2}}.
$$
This completes the proof of the first part. For the second part, let  $\pi^c$  be the optimal policy for  $\mathcal{M}^c$ . Then we have for both  $\mathrm{VaR}_{\alpha}$  and  $\mathrm{CVaR}_{\alpha}$ , with probability at least  $1 - \alpha \bar{O}^{-\frac{1}{3}}(\pi^c) \geq 1 - \alpha \underline{O}^{-\frac{1}{3}}$ ,
$$
V ^ {\phi , *} \geq V ^ {\phi , \pi^ {c}} \geq V ^ {c, \pi^ {c}} - \underline {{Q}} ^ {- \frac {1}{3}} \sqrt {\frac {| \mathcal {S} |}{\alpha}} \frac {5 | \mathcal {S} | \bar {R}}{(1 - \gamma) ^ {2}}.
$$
Recall  $\pi^{*}$  is the optimal policy for BRMDP, then
$$
V ^ {c, \pi^ {c}} \geq V ^ {c, \pi^ {*}} \geq V ^ {\phi , *} - \underline {{O}} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {5 | S | \bar {R}}{(1 - \gamma) ^ {2}}.
$$
Combining together the proof is completed.
Proof of Proposition 4.2
Proof. First notice
$$
\begin{array}{l} \left| \left| Q ^ {\phi^ {t}, *} - Q ^ {\omega , *} \right| \right| _ {\infty} = \left| \left| T ^ {\phi^ {t}} Q ^ {\phi^ {t}, *} - T ^ {\phi^ {\omega}} Q ^ {\omega , *} \right| \right| _ {\infty} \\ = | | \left(\mathcal {T} ^ {\phi^ {t}} Q ^ {\phi^ {t}, *} - \mathcal {T} ^ {\phi^ {t}} Q ^ {\omega , *}\right) + \left(\mathcal {T} ^ {\phi^ {t}} Q ^ {\omega , *} - \mathcal {T} ^ {\phi^ {\omega}} Q ^ {\omega , *}\right) | | _ {\infty} \\ \leq \gamma | | Q ^ {\phi^ {t}, *} - Q ^ {\omega , *} | | _ {\infty} + | | \mathcal {T} ^ {\phi^ {t}} Q ^ {\omega , *} - \mathcal {T} ^ {\phi^ {\omega}} Q ^ {\omega , *} | | _ {\infty}. \\ \end{array}
$$
Since  $||Q^{\omega,*}||_{\infty} \leq \frac{\bar{R}}{1 - \gamma}$  almost surely, we can prove the convergence of  $Q^{\phi^t,*}$  if we can show that almost surely for each  $(s,a) \in S \times \mathcal{A}$ ,
$$
\sup  \quad | \mathcal {T} ^ {\phi^ {t}} Q (s, a) - \mathcal {T} ^ {\phi^ {\omega}} Q (s, a) | \rightarrow 0 \text {a s} t \rightarrow \infty . \tag {9}
$$
$$
| | Q | | _ {\infty} \leq \frac {R}{1 - \gamma}
$$
Fix a state-action pair  $(s,a)$ , then for any observation process  $\omega$ , if  $||O_{s,a}^{\infty}(\omega)||_{\infty} < \infty$ ,  $\phi_{s,a}^{t} = \phi_{s,a}^{\omega}$  after some time stage  $\tau$  and (9) clearly holds for such  $\omega$ . Otherwise,  $||O_{s,a}^{\infty}(\omega)||_{\infty} = \infty$ . For those  $\omega$ 's, by Bayesian consistency [6], we know that for any neighborhood of  $p_{s,a}^{c}$  with parameter  $\varepsilon > 0$ , which is defined as  $U_{\varepsilon} = \left\{p \in \mathbb{R}_{+}^{|\mathcal{S}|} \mid \sum_{x \in \mathcal{S}} p(x) = 1, |p(x) - p_{s,a}^{c}(x)| \leq \varepsilon, \forall x \in \mathcal{S}\right\}$ ,  $\lim_{t \to \infty} \mathbb{P}(p \in U_{\varepsilon}|\phi_{s,a}^{t}) = 1$  almost surely.
For any probability mass function  $p \in \mathbb{R}_+^{|\mathcal{S}|}$ , Recall  $f(p|s, a, Q) = \mathbb{E}_{s' \sim p}[r(s, a, s') + \gamma \max_{b \in \mathcal{A}} Q(s', b)]$ , we have  $\forall p \in U_{\varepsilon}$ ,
$$
\begin{array}{l} | f (p | s, a, Q) - f (p _ {s, a} ^ {c} | s, a, Q) | = \left| \sum_ {x \in \mathcal {S}} [ p (x) - p _ {s, a} ^ {c} (x) ] [ r (s, a, x) + \gamma \max  _ {b \in \mathcal {A}} Q (x, b) ] \right| \\ \leq \varepsilon | \mathcal {S} | [ \bar {R} + \gamma | | Q | | _ {\infty} ] \\ \leq \varepsilon \frac {| \mathcal {S} | \bar {R}}{1 - \gamma} \\ \end{array}
$$
In the following we fix a sample path  $\omega$  such that  $||O_{s,a}^{\infty}(\omega)||_{\infty} = \infty$  and drop the notation of  $\omega$  for simplicity. We have almost surely:
- If  $\rho$  is  $\mathrm{VaR}_{\alpha}$ : Since  $\mathbb{P}(p \in U_{\varepsilon}|\phi_{s,a}^{t}) \to 1$ , for any risk level  $\alpha > 0$ ,  $\mathbb{P}(p \in U_{\varepsilon}|\phi_{s,a}^{t}) > 1 - \alpha$  for  $t$  large enough. Since  $\mathrm{VaR}_{\alpha}$  is the  $\alpha$ -quantitle, we have
$$
\operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} (f (p | s, a, Q)) \geq \inf  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) \geq f \left(p _ {s, a} ^ {c} | s, a, Q\right) - \varepsilon \frac {| S | \bar {R}}{1 - \gamma}. \tag {10}
$$
$$
\operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} \left(f (p | s, a, Q)\right) \leq \sup  _ {p \in U _ {\varepsilon}} f (p | s, a, Q) \leq f \left(p _ {s, a} ^ {c} | s, a, Q\right) + \varepsilon \frac {| S | \bar {R}}{1 - \gamma}.
$$
Combining the two inequalities above, we have
$$
\left| \operatorname {V a R} _ {\alpha} ^ {\phi_ {s, a} ^ {t}} \left(f (p | s, a, Q)\right) - f \left(p _ {s, a} ^ {c} | s, a, Q\right) \right| = \left| \mathcal {T} ^ {\phi^ {t}} Q (s, a) - \mathcal {T} ^ {\phi^ {\omega}} Q (s, a) \right| \leq \varepsilon \frac {| S | \bar {R}}{1 - \gamma} \tag {11}
$$
Since this holds for any  $Q$  with  $\| Q\|_{\infty}\leq \frac{\bar{R}}{1 - \gamma}$  and  $\varepsilon >0$  , we know
$$
\lim_{t\to \infty}\sup_{||Q||_{\infty} <   \frac{\bar{R}}{1 - \gamma}}|\mathcal{T}^{\phi^{t}}Q(s,a) - \mathcal{T}^{\phi^{\omega}}Q(s,a)| = 0.
$$
- If  $\rho$  is  $\mathrm{CVaR}_{\alpha}$ :