SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \right] = \int_ {\mathbb {R}} r \mathbb {P} _ {\phi} \left(r _ {l} (s, a) = r, \Gamma_ {l} ^ {\star} (s, a)\right) \\ = \int_ {\mathbb {R}} r \mathbb {P} _ {\phi} (r _ {l} (s, a) = r \mid \Gamma_ {l} ^ {\star} (s, a)) \mathbb {P} _ {\phi} (\Gamma_ {l} ^ {\star} (s, a)) \\ = \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mid \Gamma_ {l} ^ {\star} (s, a) \right] \mathbb {P} _ {\phi} (\Gamma_ {l} ^ {\star} (s, a)). \\ \end{array}
$$
Putting it all together,
$$
\mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V ^ {\star} (s) = \sum_ {l, s, a} \mathbb {P} _ {\phi} \left(\Gamma_ {l} ^ {\star} (s, a)\right) \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mid \Gamma_ {l} ^ {\star} (s, a) \right].
$$
F.3 Proof of Lemma 3
We prove in Lemma 11 a generalization of Lemma 3 that does not rely on the assumption of  $\sigma$ -sub-Gaussian mean rewards. It builds on the Donsker-Varadhan representation [23, 70, 69, 28] (Proposition 1), see also [51, Theorem 1]. We denote by  $\mathrm{KL}(\cdot ||\cdot)$  the Kullback-Leibler divergence, by  $\Psi_X:\mathbb{R}\to \mathbb{R}$  the cumulant generating function of random variable  $X - \mathbb{E}X$  and by  $f^{\star}:\mathbb{R}\rightarrow \mathbb{R}$  the convex conjugate of function  $f:\mathbb{R}\rightarrow \mathbb{R}$ , i.e.,
$$
\Psi_ {X} (\beta) := \log \mathbb {E} \exp (\beta (X - \mathbb {E} X)), \qquad f ^ {\star} (y) := \sup  _ {x \in \mathbb {R}} \{x y - f (x) \}.
$$
Lemma 11. Let  $X: \Omega \to \mathbb{R}$  be a random variable on  $(\Omega, \mathcal{F}, \mathbb{P})$  satisfying  $X \in L_1$  such that the interior of the domain of  $\Psi_X$  is non-empty, and let  $B \in \mathcal{F}$  be an event with  $\mathbb{P}(B) > 0$ . Then,
$$
\begin{array}{l} \mathbb {E} [ X \mid B ] \leq \mathbb {E} X + \left(\Psi_ {X} ^ {*}\right) ^ {- 1} (\operatorname {K L} (\mathbb {P} (X \in \cdot \mid B) | | \mathbb {P} (X \in \cdot))) \\ \leq \mathbb {E} X + \left(\Psi_ {X} ^ {*}\right) ^ {- 1} (- \log \mathbb {P} (B)). \\ \end{array}
$$
Proof. Applying Proposition 1 to  $\lambda(X - \mathbb{E}X)$  for arbitrary  $\lambda \in \mathbb{R}_+$  with the choices  $Q \coloneqq \mathbb{P}(X \in \cdot)$ ,  $P \coloneqq \mathbb{P}(X \in \cdot \mid B)$  gives
$$
\begin{array}{l} \operatorname {K L} (\mathbb {P} (X \in \cdot \mid B) \mid | \mathbb {P} (X \in \cdot)) \geq \sup  _ {\lambda \in \mathbb {R} _ {+}} \left\{\lambda \mathbb {E} [ (X - \mathbb {E} X) \mid B ] - \log \mathbb {E} [ \exp \lambda (X - \mathbb {E} X) ] \right\} \\ = \sup  _ {\lambda \in \mathbb {R} _ {+}} \left\{\lambda \left(\mathbb {E} [ X \mid B ] - \mathbb {E} X\right) - \Psi_ {X} (\lambda) \right\} \\ = \Psi_ {X} ^ {\star} (\mathbb {E} [ X \mid B ] - \mathbb {E} X), \\ \end{array}
$$
by definition of  $\Psi_X$  the cumulant generating function of  $X - \mathbb{E}X$  and by definition of the convex conjugate of  $\Psi_X$  which satisfies  $\Psi_X^\star(y) = \sup_{\lambda \geq 0} \{\lambda y - \Psi_X(\lambda)\}$ . Since  $\Psi_X^\star$  is strictly increasing, its inverse  $(\Psi_X^*)^{-1}$  is also strictly increasing, so
$$
\mathbb {E} [ X \mid B ] - \mathbb {E} X \leq \left(\Psi_ {X} ^ {*}\right) ^ {- 1} (\operatorname {K L} (\mathbb {P} (X \in \cdot \mid B) | | \mathbb {P} (X \in \cdot))),
$$
which proves the first inequality. We now derive the second inequality. Note the definition of the KL divergence
$$
\operatorname {K L} (P \mid | Q) = \int \log \left(\frac {d P}{d Q}\right) d P,
$$
4If  $\Psi_X = 0$  , then we take  $(\Psi_X^*)^{-1} = 0$
where  $\frac{dP}{dQ}$  is the Radon-Nikodym derivative of  $P$  with respect to  $Q$ . We can then write
$$
\begin{array}{l} \operatorname {K L} \left(\mathbb {P} (X \in \cdot \mid B) \mid \mid \mathbb {P} (X \in \cdot)\right) = \int \log \frac {d \mathbb {P} (X \in \cdot \mid B)}{d \mathbb {P} (X \in \cdot)} d \mathbb {P} (X \in \cdot \mid B) \\ \stackrel {\text {(i)}} {=} \int \log \frac {\mathbb {P} (X \in \cdot \mid B)}{\mathbb {P} (X \in \cdot)} d \mathbb {P} (X \in \cdot \mid B) \\ \stackrel {\mathrm {(i i)}} {=} \int \log \frac {\mathbb {P} (B \mid X \in \cdot)}{\mathbb {P} (B)} d \mathbb {P} (X \in \cdot \mid B) \\ = \int \underbrace {\log \mathbb {P} (B \mid X \in \cdot)} _ {\leq 0} d \mathbb {P} (X \in \cdot \mid B) - \log \mathbb {P} (B) \\ \leq - \log \mathbb {P} (B), \\ \end{array}
$$
where (i) is from the formulation of Bayes' theorem stated in [73, Theorem 1.31] and (ii) applies Bayes' rule.
Proposition 1 (Variational form of the KL-Divergence given in Theorem 5.2.1 of [23]). Fix two probability distributions  $P$  and  $Q$  such that  $P$  is absolutely continuous with respect to  $Q$ . Then,
$$
\operatorname {K L} (P \mid | Q) = \sup  _ {X} \left\{\mathbb {E} _ {P} [ X ] - \log \mathbb {E} _ {Q} [ \exp X ] \right\},
$$
where  $\mathbb{E}_P$  and  $\mathbb{E}_Q$  denote the expectation operator under  $P$  and  $Q$  respectively, and the supremum is taken over all real valued random variables  $X$  such that  $\mathbb{E}_P[X]$  is well defined and  $\mathbb{E}_Q[\exp X] < \infty$ .
Proof of Lemma 3. Under the  $\sigma_l(s,a)$ -sub-Gaussian assumption of  $r_l(s,a)$ , we can bound the cumulant generating function of  $r_l(s,a)$  as
$$
\Psi_ {r _ {l} (s, a)} (\beta) \leq \frac {\beta^ {2} \sigma_ {l} (s , a) ^ {2}}{2}.
$$
We therefore have that for any  $y \geq 0$ ,
$$
\left(\Psi_ {r _ {l} (s, a)} ^ {*}\right) ^ {- 1} (y) = \inf  _ {\tau_ {l} (s, a) > 0} \left\{\tau_ {l} (s, a) \Psi_ {r _ {l} (s, a)} \left(1 / \tau_ {l} (s, a)\right) + \tau_ {l} (s, a) y \right\} \leq \sigma_ {l} (s, a) \sqrt {2 y}. \tag {8}
$$
Plugging this bound into Lemma 11 yields Lemma 3.
F.4 Proof of Lemma 4
Proof. The first inequality in (4) combines Lemmas 2 and 3. The second inequality comes from the fact that  $\mathbb{P}_{\Gamma^{\star}} \in \Lambda(P)$  from Lemma 1. The VAPOR optimization problem is concave since (i) the objective function  $\mathcal{V}$  is concave in  $\lambda$  by concavity of the function  $x \in (0,1) \to x\sqrt{-\log x}$ , and (ii) the constraints are a convex set due to the fact that the state-action polytope  $\Lambda(P)$  is closed, bounded and convex [66, Theorem 8.9.4]. Finally, note that the maximum exists and is attained, since  $\Lambda(P)$  is closed, bounded and nonempty due to the fact that  $\mathbb{P}_{\Gamma^{\star}} \in \Lambda(P)$ .
F.5 VAPOR without the Sub-Gaussian Assumption
From (8), for any set of reward beliefs, the VAPOR optimization problem (Lemma 4) becomes
$$
\max  _ {\lambda \in \Lambda (P)} \lambda^ {\top} \left(\mathbb {E} _ {\phi} r + \left(\Psi_ {r} ^ {*}\right) ^ {- 1} (- \log \lambda)\right) = \sum_ {l, s, a} \lambda_ {l} (s, a) \left(\mathbb {E} _ {\phi} r _ {l} (s, a) + \left(\Psi_ {r _ {l} (s, a)} ^ {*}\right) ^ {- 1} (- \log \lambda_ {l} (s, a))\right).
$$
This objective remains a concave optimization problem since the function  $(\Psi_{r_l(s,a)}^*)^{-1}$  is always concave (as the inverse of the strictly increasing convex function  $\Psi_{r_l(s,a)}$ ).
F.6 Proof of Lemma 5
$$
\mathrm {K L} _ {\tau^ {*}} \left(\mathbb {P} _ {\Gamma^ {*}} | | \lambda^ {*}\right) \leq \mathcal {V} _ {\phi} \left(\lambda^ {*}\right) - \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V _ {1} ^ {*} (s).
$$
Proof. The proof extends the relationship between Bregman divergence and value in policy-regularized MDPs [48] to the setting of occupancy-measure-regularized MDPs. The Bregman divergence generated by  $\Omega$  between two points  $\lambda, \lambda' \in \Lambda(P)$  is defined as
$$
D _ {\Omega} (\lambda , \lambda^ {\prime}) := \Omega (\lambda) - \Omega (\lambda^ {\prime}) - \nabla \Omega (\lambda^ {\prime}) ^ {\top} (\lambda - \lambda^ {\prime}).
$$
For ease of notation, we omit the dependence on the beliefs  $\phi$  throughout the proof. We write
$$
\mathcal {V} (\lambda) := \sum_ {l = 1} ^ {L} \sum_ {s, a} \lambda_ {l} (s, a) \mathbb {E} _ {\phi} r _ {l} (s, a) - \Omega (\lambda), \qquad \mathcal {V} ^ {*} := \max  _ {\lambda \in \Lambda (P)} \mathcal {V} (\lambda), \qquad \lambda^ {*} \in \operatorname * {a r g m a x} _ {\lambda \in \Lambda (P)} \mathcal {V} (\lambda),
$$
where  $\Omega :\Lambda (P)\to \mathbb{R}$  is a continuously differentiable strictly convex regularizer defined as
$$
\Omega (\lambda) := - \sum_ {l = 1} ^ {L} \sum_ {s, a} \sigma_ {l} (s, a) \lambda_ {l} (s, a) \sqrt {- 2 \log \lambda_ {l} (s , a)}.
$$
By definition,  $\lambda^{*}$  must satisfy the first-order optimality conditions for the maximum of  $\mathcal{V}$ , and since  $\lambda^{*} \in \mathrm{relint}(\Lambda(P))$  the following holds
$$
\left(\mathbb {E} _ {\phi} r - \nabla \Omega \left(\lambda^ {*}\right)\right) ^ {\top} (\lambda - \lambda^ {*}) = 0, \quad \forall \lambda \in \Lambda (P).
$$
$$
\begin{array}{l} D _ {\Omega} \left(\mathbb {P} _ {\Gamma^ {\star}}, \lambda^ {*}\right) = \Omega \left(\mathbb {P} _ {\Gamma^ {\star}}\right) - \Omega \left(\lambda^ {*}\right) - \nabla \Omega \left(\lambda^ {*}\right) ^ {\top} \left(\mathbb {P} _ {\Gamma^ {\star}} - \lambda^ {*}\right) \\ = \Omega \left(\mathbb {P} _ {\Gamma^ {\star}}\right) - \Omega \left(\lambda^ {*}\right) - \mathbb {E} _ {\phi} r ^ {\top} \left(\mathbb {P} _ {\Gamma^ {\star}} - \lambda^ {*}\right) \\ = \mathcal {V} ^ {*} - \mathbb {E} _ {\phi} r ^ {\top} \mathbb {P} _ {\Gamma^ {*}} + \Omega (\mathbb {P} _ {\Gamma^ {*}}) \\ = \mathcal {V} ^ {*} - \mathcal {V} \left(\mathbb {P} _ {\Gamma^ {*}}\right). \tag {9} \\ \end{array}
$$
We now notice that
$$
\Omega (\lambda) = - \sum_ {l = 1} ^ {L} \sum_ {s, a} \min  _ {\tau_ {l} (s, a) > 0} \left\{\frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau_ {l} (s , a)} - \tau_ {l} (s, a) \lambda_ {l} (s, a) \log \lambda_ {l} (s, a) \right\}.
$$
For any positive  $\tau \in \mathbb{R}_+^{L,S,A}$ , we define
$$
\Omega_ {\tau} (\lambda) := \sum_ {l = 1} ^ {L} \sum_ {s, a} - \frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau_ {l} (s , a)} + \tau_ {l} (s, a) \lambda_ {l} (s, a) \log \lambda_ {l} (s, a)
$$
$$
\mathcal {V} _ {\tau} (\lambda) := \sum_ {l = 1} ^ {L} \sum_ {s, a} \lambda_ {l} (s, a) \mathbb {E} _ {\phi} r _ {l} (s, a) - \Omega_ {\tau} (\lambda),
$$
$$
\mathcal{V}_{\tau}^{*}:= \max_{\lambda \in \Lambda (P)}\mathcal{V}_{\tau}(\lambda),\quad \lambda_{\tau}^{*}\in \operatorname *{argmax}_{\lambda \in \Lambda (P)}\mathcal{V}_{\tau}(\lambda).
$$
Using that the KL-divergence is the Bregman divergence generated by the negative-entropy function, (9) implies that
$$
\mathrm {K L} _ {\tau} \left(\mathbb {P} _ {\Gamma^ {\star}} \mid \mid \lambda_ {\tau} ^ {*}\right) = \mathcal {V} _ {\tau} ^ {*} - \mathcal {V} _ {\tau} \left(\mathbb {P} _ {\Gamma^ {\star}}\right).
$$
Moreover, from strong duality (which holds because  $\Lambda(P)$  is convex),
$$
\min  _ {\tau \in \mathbb {R} _ {+} ^ {L, S, A}} \max  _ {\lambda \in \Lambda (P)} \mathcal {V} _ {\tau} (\lambda) = \max  _ {\lambda \in \Lambda (P)} \min  _ {\tau \in \mathbb {R} _ {+} ^ {L, S, A}} \mathcal {V} _ {\tau} (\lambda).
$$
Denote  $\tau^{*}\in \mathrm{argmin}_{\tau}\nu_{\tau}^{*}$  , then it holds that
$$
\mathrm {K L} _ {\tau^ {*}} \left(\mathbb {P} _ {\Gamma^ {*}} | | \lambda^ {*}\right) = \mathcal {V} ^ {*} - \mathcal {V} _ {\tau^ {*}} \left(\mathbb {P} _ {\Gamma^ {*}}\right) \leq \mathcal {V} ^ {*} - \mathcal {V} \left(\mathbb {P} _ {\Gamma^ {*}}\right),
$$
F.7 Proof of Theorem 1 and Corollary 1
Below we prove a more general statement that will imply Theorem 1 and Corollary 1.
Lemma 12. Let alg produce any sequence of  $\mathcal{F}_t$ -measurable policies  $\pi^t, t = 1, \dots, N$ , whose induced occupancy measures  $\lambda^{\pi^t}$  satisfy
$$
\mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V _ {1} ^ {\star} (s) \leq \mathcal {V} _ {\phi^ {t}} \left(\mathbb {E} ^ {t} \lambda^ {\pi^ {t}}\right), \tag {10}
$$
where we assume that the uncertainty measure  $\sigma^t$  of the  $\mathcal{V}_{\phi^t}$  function (3) satisfies
$$
\sigma_ {l} ^ {t} (s, a) \leq \frac {c _ {0}}{\sqrt {\left(n _ {l} ^ {t} (s , a) \vee 1\right)}}, \tag {11}
$$
for some  $c_0 > 0$  and any  $t \in [N]$ ,  $l \in [L]$ ,  $(s, a) \in S_l \times \mathcal{A}$ , where we recall that  $n_l^t(s, a)$  denotes the visitation count to  $(s, a)$  at step  $l$  before episode  $t$  and  $(\cdot \vee 1) := \max(\cdot, 1)$ . Then it holds that
$$
\begin{array}{l} \mathcal {B R} _ {\phi} (\mathrm {a l g}, T) \leq \sqrt {2 c _ {0} ^ {2} T S A \log (S A) (1 + \log T / L)} \\ = \widetilde {O} \left(c _ {0} \sqrt {S A T}\right). \\ \end{array}
$$
Proof. Note that  $\mathbb{E}^t\lambda^{\pi^t}$  corresponds to the occupancy measure of policy  $\pi^t$  on the expected transition dynamics  $\mathbb{E}^t P$ , i.e.,  $\mathbb{E}^t\lambda^{\pi^t}\in \Lambda (\mathbb{E}^t P)$ . We bound the Bayes regret of alg as
$$
\begin{array}{l} \mathcal {B R} _ {\phi} (\operatorname {a l g}, T) \stackrel {{(\mathrm {i})}} {=} \mathbb {E} \sum_ {t = 1} ^ {N} \mathbb {E} _ {s \sim \rho} \mathbb {E} ^ {t} \left[ V _ {1} ^ {\star} (s) - V _ {1} ^ {\pi^ {t}} (s) \right] \\ \stackrel {\mathrm {(i i)}} {=} \mathbb {E} \sum_ {t = 1} ^ {N} \left[ \mathbb {E} _ {s \sim \rho} \mathbb {E} ^ {t} V _ {1} ^ {\star} (s) - \sum_ {l = 1} ^ {L} \sum_ {s, a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a) \mathbb {E} ^ {t} r _ {l} (s, a) \right] \\ \stackrel {\text {(i i i)}} {\leq} \mathbb {E} \sum_ {t = 1} ^ {N} \left[ \mathcal {V} _ {\phi^ {t}} \left(\mathbb {E} ^ {t} \lambda^ {\pi^ {t}}\right) - \sum_ {l = 1} ^ {L} \sum_ {s, a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a) \mathbb {E} ^ {t} r _ {l} (s, a) \right] \\ \stackrel {\mathrm {(i v)}} {=} \mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s, a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a) \sigma_ {l} ^ {t} (s, a) \sqrt {- 2 \log \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a)} \\ \stackrel {(v)} {\leq} \mathbb {E} \sqrt {\sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \mathcal {H} \left(\mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}}\right)} \sqrt {\sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s , a} 2 \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a) \sigma_ {l} ^ {t} (s , a) ^ {2}} \\ \stackrel {\mathrm {(v i)}} {\leq} \sqrt {\mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \mathcal {H} (\mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}})} \sqrt {\mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s , a} 2 \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a) \sigma_ {l} ^ {t} (s , a) ^ {2}}, \\ \end{array}
$$
where (i) follows from the tower property of conditional expectation where the outer expectation is with respect to  $\mathcal{F}_1, \mathcal{F}_2, \ldots$ , (ii) is from the dual formulation of the value function averaged with respect to the initial state distribution [66] and the fact that  $\lambda_l^{\pi^t}(s,a)$  and  $r_l(s,a)$  are conditionally independent given  $\mathcal{F}_t$ , (iii) is due to condition (10), (iv) is from the definition of  $\mathcal{V}_{\phi^t}$ , (v) and (vi) are by applying the Cauchy-Schwarz inequality, where  $\mathcal{H}$  denotes the entropy function [10].
We bound the left term by using the fact that  $\mathcal{H}(\mathbb{E}^t\lambda_l^{\pi^t})\leq \log SA$ . We bound the right term using condition (11) and by applying the pigeonhole principle (e.g., [47, Lemma 6]) which gives
$$
\begin{array}{l} \mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s, a} \frac {\mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a)}{n _ {l} ^ {t} (s , a) \vee 1} = \mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \mathbb {E} ^ {t} \left(\sum_ {s, a} \frac {\mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a)}{n _ {l} ^ {t} (s , a) \vee 1}\right) \\ = \sum_ {l = 1} ^ {L} \mathbb {E} \left(\sum_ {t = 1} ^ {N} \sum_ {s, a} \frac {\lambda_ {l} ^ {\pi^ {t}} (s , a)}{n _ {l} ^ {t} (s , a) \vee 1}\right) \\ \leq \sum_ {l = 1} ^ {L} S _ {l} A (1 + \log N) \\ = S A (1 + \log N), \\ \end{array}
$$
which follows from the tower property of conditional expectation and since the counts at time  $t$  are  $\mathcal{F}_t$ -measurable. Taking  $T = NL$  finally yields
$$
\begin{array}{l} \mathcal {B R} _ {\phi} (\mathrm {a l g}, T) \leq \sqrt {2 c _ {0} ^ {2} N L S A \log (S A) (1 + \log N)} \\ = \widetilde {O} \left(c _ {0} \sqrt {S A T}\right). \\ \end{array}
$$
$$
\mathcal {B R} _ {\phi} (\mathrm {V A P O R}, T) \leq \sqrt {2 (\nu^ {2} + 1) T S A \log (S A) (1 + \log T / L)} = \widetilde {O} (\sqrt {S A T}).
$$
Proof. Assumption 1 implies that the mean rewards are sub-Gaussian under the posterior (see [70, App.D.2]), where we can upper bound the sub-Gaussian parameter  $\sigma_l^t (s,a)\leq \sqrt{(\nu^2 + 1) / (n_l^t(s,a)\lor 1)}$  at the beginning of each episode  $t$ . Thus the condition (11) holds for  $c_{0} = \sqrt{\nu^{2} + 1}$ . Moreover, Lemma 4 implies that the occupancy measures that solve the VAPOR optimization problem satisfy condition (10). The result thus follows from applying Lemma 12.
Corollary 1. Denote by  $\mathrm{alg}_{\Gamma^{\star}}$  the algorithm that produces policies based on  $\mathbb{P}_{\Gamma^{\star}}^{t}$  for each episode  $t$  using (1). Then, under the same conditions as Theorem 1, we have  $\mathcal{BR}_{\phi}(\mathrm{alg}_{\Gamma^{\star}},T)\leq \tilde{O} (\sqrt{SAT})$ .
Proof. The result comes from combining Lemmas 4 and 12, as done in the proof of Theorem 1.  $\square$
F.8 Proof of Lemma 6
Definition 2 (Transformed beliefs). Consider any beliefs  $\phi$  such that the mean rewards are  $\sigma$ -sub-Gaussian and the transition dynamics are  $\alpha$ -Dirichlet. We define the transformed beliefs  $\hat{\phi}$  according to which the transition dynamics are known and equal to  $\mathbb{E}_{\phi}P$  and the rewards are distributed as  $\mathbb{E}_{\phi}r + \omega$ , where  $\omega_{l}(s,a)$  follows an independent zero-mean Gaussian distribution with variance  $\hat{\sigma}^2$  defined for each step  $l \in [L]$  and state-action pair  $(s,a) \in S_l \times \mathcal{A}$  as
$$
\hat {\sigma} _ {l} (s, a) ^ {2} := 3. 6 ^ {2} \sigma_ {l} ^ {2} (s, a) + \frac {(L - l) ^ {2}}{\sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} \alpha_ {l} (s , a , s ^ {\prime})}.
$$
We prove in Appendix F.8.1 the following stronger result which implies Lemma 6.
Lemma 13. For any step  $l \in [L]$  and state  $s \in S_l$ , it holds that
$$
\mathbb {E} _ {\phi} V _ {l} ^ {\star} (s) \leq \mathbb {E} _ {\hat {\phi}} V _ {l} ^ {\star} (s).
$$
F.8.1 Proof of Lemma 13
Overview. The proof relies on studying properties of two stochastic Bellman operators, drawing inspiration from [62, Section 6.5]. In particular we will make use of the notion of stochastic optimism (Definition 3). To deal with the unknown transitions, we will use the property of Gaussian-Dirichlet optimism [60]. To deal with unknown rewards, we will derive a property of Gaussian-sub-Gaussian optimism (Lemma 19) which holds when the variance of the Gaussian distribution is multiplied by a small universal constant.
Definition 3. A random variable  $X$  is stochastically optimistic with respect to another random variable  $Y$ , written  $X \geq_{SO} Y$ , if  $\mathbb{E}u(X) \geq \mathbb{E}u(Y)$  for all convex increasing functions  $u: \mathbb{R} \to \mathbb{R}$ .
Likewise, we will say that  $X$  is stochastically optimistic with respect to  $Y$  under  $\phi$  and write that  $X\mid \phi \geq_{SO}Y\mid \phi$  if  $\mathbb{E}_{\phi}u(X)\geq \mathbb{E}_{\phi}u(Y)$  for all convex increasing functions  $u:\mathbb{R}\to \mathbb{R}$ .
For any step  $l \in [L]$ , we introduce the Bellman operator  $\mathcal{B}_l: \mathbb{R}^{S_{l+1} \times A} \to \mathbb{R}^{S_{l+1} \times A}$  which is defined for any  $Q_{l+1} \in \mathbb{R}^{S_{l+1} \times A}$  and state-action pair  $(s, a) \in S_l \times \mathcal{A}$  as
$$
\begin{array}{l} \mathcal {B} _ {l} Q _ {l + 1} (s, a) := r _ {l} (s, a) + P _ {l} (\cdot \mid s, a) ^ {\top} \max  _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}) \\ := r _ {l} (s, a) + \sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} P _ {l} \left(s ^ {\prime} \mid s, a\right) \max  _ {a ^ {\prime}} Q _ {l + 1} \left(s ^ {\prime}, a ^ {\prime}\right). \\ \end{array}
$$
We also define the 'transformed' Bellman operator  $\widehat{\mathcal{B}}_l: \mathbb{R}^{S_{l+1} \times A} \to \mathbb{R}^{S_{l+1} \times A}$  as
$$
\widehat {\mathcal {B}} _ {l} Q _ {l + 1} (s, a) := \mathbb {E} _ {\phi} r _ {l} (s, a) + \omega_ {l} (s, a) + \mathbb {E} _ {\phi} P _ {l} (\cdot | s, a) ^ {\top} \max  _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}),
$$
where we recall that  $\omega_{l}(s,a)\sim \mathcal{N}\left(0,\widehat{\sigma}_{l}(s,a)^{2}\right)$  is distributed independently across the state-action pairs  $(s,a)\in S_l\times \mathcal{A}$  under  $\phi$ . Note that  $\mathcal{B}_l$  and  $\widehat{\mathcal{B}}$  can be viewed as randomized Bellman operators due to the randomness in the MDP  $\mathcal{M}$  and in the distribution  $\omega$ , respectively. We rely on two key properties.
Lemma 14 (Property 1: monotonicity). For any step  $l \in [L]$ , consider two random  $Q$  functions  $Q_{l+1}, Q_{l+1}' \in \mathbb{R}^{S_{l+1} \times \mathcal{A}}$  such that conditioned on  $\phi$  the entries of  $Q_{l+1}$  (respectively  $Q_{l+1}'$ ) are drawn independently across state-action pairs  $(s, a) \in S_{l+1} \times \mathcal{A}$  and drawn independently of the noise terms  $\omega_l(s, a)$ . Then
$$
Q _ {l + 1} (s, a) \mid \phi \geq_ {S O} Q _ {l + 1} ^ {\prime} (s, a) \mid \phi \quad \forall (s, a) \in \mathcal {S} _ {l + 1} \times \mathcal {A}
$$
$$
\hat {\mathcal {B}} _ {l} Q _ {l + 1} (s, a) \mid \hat {\phi} \geq_ {S O} \hat {\mathcal {B}} _ {l} Q _ {l + 1} ^ {\prime} (s, a) \mid \hat {\phi} \quad \forall (s, a) \in \mathcal {S} _ {l} \times \mathcal {A}, l \in [ L ].
$$
Proof. The proof follows exactly the steps of [62, Lemma 3]. It relies on the fact that conditioned on  $\widehat{\phi},\widehat{\mathcal{B}}_lQ_{l + 1}(s,a)$  is a convex increasing function of  $(Q_{l + 1}(s,a))_{s,a}$  convolved with the independent noise term  $\omega_{l}(s,a)$ . The result therefore follows from the fact that stochastic optimism is preserved under convex increasing operations [62, Lemma 2].
Lemma 15 (Property 2: stochastic optimism). For any  $l \in [L]$  and  $(s, a) \in S_{l+1} \times \mathcal{A}$ , it holds that
$$
\widehat {\mathcal {B}} _ {l} Q _ {l + 1} (s, a) \mid \widehat {\phi} \geq_ {S O} \mathcal {B} _ {l} Q _ {l + 1} (s, a) \mid \phi
$$
for any fixed  $Q_{l + 1}\in \mathbb{R}^{\mathcal{S}_{l + 1}\times \mathcal{A}}$  such that  $\mathrm{Span}(Q_{l + 1})\leq L - l$
Proof. For any fixed  $Q_{l + 1}$ , it holds that
$$
\hat {\mathcal {B}} _ {l} Q _ {l + 1} (s, a) \mid \hat {\phi} \sim \mathcal {N} \left(b _ {l} (s, a), \hat {\sigma} _ {l} (s, a) ^ {2}\right) \mid \phi ,
$$
$$
b _ {l} (s, a) := \mathbb {E} _ {\phi} r _ {l} (s, a) + \mathbb {E} _ {\phi} P _ {l} (\cdot | s, a) ^ {\top} \max  _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}).
$$
Lemma 16 ([60]). Let  $Y = \sum_{i=1}^{n} A_i b_i$  for fixed  $b \in \mathbb{R}^n$  and random variable  $A$ , where  $A$  is Dirichlet with parameter  $\alpha \in \mathbb{R}^n$ , and let  $X \sim \mathcal{N}(\mu_X, \sigma_X^2)$  with  $\mu_X \geq \frac{\sum_i \alpha_i b_i}{\sum_i \alpha_i}$  and  $\sigma_X^2 \geq (\sum_i \alpha_i)^{-1} \operatorname{Span}(b)^2$ , where  $\operatorname{Span}(b) = \max_i b_i - \min_j b_j$ , then  $X \geq_{SO} Y$ .
In our case, in the notation of Lemma 16,  $A$  will represent the transition function probabilities, and  $b$  will represent  $V \coloneqq \max_{a} Q(\cdot, a)$ , i.e., for a given  $(s, a) \in S_l \times A$  let  $X$  be a random variable distributed as  $\mathcal{N}(\mu_X, \sigma_X^2)$  where
$$
\mu_ {X} = \sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} \left(\alpha_ {l} (s, a, s ^ {\prime}) V _ {l + 1} (s ^ {\prime}) / \sum_ {x} \alpha_ {l} (s, a, x)\right) = \sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} \mathbb {E} _ {\phi} [ P _ {l} (s ^ {\prime} \mid s, a) ] V _ {l + 1} (s ^ {\prime})
$$
due to Assumption 2. Moreover, the lemma assumes that  $\mathrm{Span}(V_{l + 1})\leq L - l$  , so we choose  $\sigma_X^2 = (L - l)^2 /(\sum_{s'}\alpha_l(s,a,s'))$  . As a result, it holds that