SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {N} \left(\mathbb {E} _ {\phi} P _ {l} (\cdot | s, a) ^ {\top} \max _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}), \frac {(L - l) ^ {2}}{\sum_ {s ^ {\prime}} \alpha_ {l} (s , a , s ^ {\prime})}\right) | \phi \geq_ {S O} P _ {l} (\cdot | s, a) ^ {\top} \max _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}) | \phi .
$$
Moreover, from the assumption of  $\sigma_l(s,a)$ -sub-Gaussian  $r_l(s,a)$  and from the Gaussian Sub-Gaussian stochastic optimism property of Lemma 19,
$$
\mathcal {N} \left(0, 3. 6 ^ {2} \sigma_ {l} ^ {2} (s, a)\right) \geq_ {S O} r _ {l} (s, a) - \mathbb {E} _ {\phi} r _ {l} (s, a) \mid \phi ,
$$
$$
\mathcal {N} \left(\mathbb {E} _ {\phi} r _ {l} (s, a), 3. 6 ^ {2} \sigma_ {l} ^ {2} (s, a)\right) \mid \phi \geq_ {S O} r _ {l} (s, a) \mid \phi .
$$
As a result, since the random variables  $r_l(s,a)$ ,  $P_l(\cdot \mid s,a)^\top \max_{a'} Q_{l+1}(\cdot, a')$ ,  $\mathcal{N}\left(\mathbb{E}_{\phi} r_l(s,a), 3.6^2 \sigma_l^2(s,a)\right)$  and  $\mathcal{N}\left(\mathbb{E}_{\phi} P_l(\cdot \mid s,a)^\top \max_{a'} Q_{l+1}(\cdot, a'), \frac{(L-l)^2}{\sum_{s'} \alpha_l(s,a,s')}\right)$  are mutually conditionally independent, we have
$$
\begin{array}{l} \left. \mathcal {N} \left(b _ {l} (s, a), \widehat {\sigma} _ {l} (s, a) ^ {2}\right) \mid \phi \sim \mathcal {N} \left(\mathbb {E} _ {\phi} r _ {l} (s, a), 3. 6 ^ {2} \sigma_ {l} ^ {2} (s, a)\right) \right. \\ + \mathcal {N} \left(\mathbb {E} _ {\phi} P _ {l} (\cdot | s, a) ^ {\top} \max  _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}), \frac {(L - l) ^ {2}}{\sum_ {s ^ {\prime}} \alpha_ {l} (s , a , s ^ {\prime})}\right) | \phi \\ \geq_ {S O} r _ {l} (s, a) + P _ {l} (\cdot \mid s, a) ^ {\top} \max  _ {a ^ {\prime}} Q _ {l + 1} (\cdot , a ^ {\prime}) \mid \phi \\ \sim \mathcal {B} _ {l} Q _ {l + 1} (s, a) | \phi , \\ \end{array}
$$
Lemma 17. For any  $l\in [L]$  and  $(s,a)\in S_l\times \mathcal{A}$
$$
Q _ {l} ^ {\star} (s, a) \mid \widehat {\phi} \geq_ {S O} Q _ {l} ^ {\star} (s, a) \mid \phi .
$$
Proof. Note that under  $\phi$ ,  $Q_1^\star = \mathcal{B}_1\mathcal{B}_2\ldots \mathcal{B}_L0$  and under  $\widehat{\phi}$ ,  $Q_1^\star = \widehat{\mathcal{B}}_1\widehat{\mathcal{B}}_2\ldots \widehat{\mathcal{B}}_L0$ . By Lemma 15,
$$
(\widehat {\mathcal {B}} _ {L} 0) (s, a) \mid \widehat {\phi} \geq_ {S O} (\mathcal {B} _ {L} 0) (s, a) \mid \phi .
$$
Proceeding by induction, suppose that for some  $l \leq L$ ,
$$
(\widehat {\mathcal {B}} _ {l + 1} \dots \widehat {\mathcal {B}} _ {L} 0) (s, a) \mid \widehat {\phi} \geq_ {S O} (\mathcal {B} _ {l + 1} \dots \mathcal {B} _ {L} 0) (s, a) \mid \phi .
$$
Combining this with Lemma 14 shows
$$
\begin{array}{l} \widehat {\mathcal {B}} _ {l} \left(\widehat {\mathcal {B}} _ {l + 1} \dots \widehat {\mathcal {B}} _ {L} 0\right) (s, a) \mid \widehat {\phi} \geq_ {S O} \widehat {\mathcal {B}} _ {l} \left(\mathcal {B} _ {l + 1} \dots \mathcal {B} _ {L} 0\right) (s, a) \mid \widehat {\phi} \\ \ge_ {S O} \mathcal {B} _ {l} \left(\mathcal {B} _ {l + 1} \dots \mathcal {B} _ {L} 0\right) (s, a) \mid \phi , \\ \end{array}
$$
where the final step uses Lemma 15 and the fact that  $\operatorname{Span}(\mathcal{B}_{l+1} \ldots \mathcal{B}_L 0) \leq L - l$ .
We are now ready to prove Lemma 13. We apply the property of stochastic optimism derived in Lemma 17 to the convex increasing function  $u_{s}(Q) \coloneqq \max_{a \in \mathcal{A}} Q(s, a)$  for every step  $l \in [L]$  and state  $s \in S_{l}$ , which yields the desired inequality  $\mathbb{E}_{\phi} V_{l}^{\star}(s) \leq \mathbb{E}_{\hat{\phi}} V_{l}^{\star}(s)$ .
F.9 Proof of Lemma 7
Lemma 7. Let  $\mathrm{alg}$  be any procedure that maps posterior beliefs to policies, and denote by  $\mathcal{BR}_{\mathcal{T},\phi}(\mathrm{alg},T)$  the Bayesian regret of  $\mathrm{alg}$  where at each episode  $t = 1,\dots ,T$  the policy and regret are computed by replacing  $\phi^t$  with  $\mathcal{T}(\phi^t)$  (Lemma 6), then under Assumptions 1 and 2,
Proof. We recall that we denote by  $\phi$  the prior on the MDP, by  $\phi^t \coloneqq \phi(\cdot \mid \mathcal{F}_t)$  the posterior beliefs at the beginning of each episode  $t$  and by  $\mathcal{T}$  the mapping that transforms the posteriors according to Lemma 6. Using the tower property of conditional expectation (where the outer expectation is with respect to  $\mathcal{F}_1, \mathcal{F}_2, \ldots$ ), we can define the Bayes regret over  $T$  timesteps of alg under the sequence of original and transformed posteriors respectively as
$$
\mathcal {B R} _ {\phi} (\mathrm {a l g}, T) := \mathbb {E} \sum_ {t = 1} ^ {N} \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi^ {t}} \left[ V _ {1} ^ {\star} (s) - V _ {1} ^ {\pi^ {t}} (s) \right],
$$
$$
\mathcal {B R} _ {\mathcal {T}, \phi} (\operatorname {a l g}, T) := \mathbb {E} \sum_ {t = 1} ^ {N} \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\mathcal {T} (\phi^ {t})} \left[ V _ {1} ^ {\star} (s) - V _ {1} ^ {\pi^ {t}} (s) \right].
$$
The result then immediately comes from combining Lemmas 13 and 18.
Lemma 18. For any fixed policy  $\pi$ , it holds that
$$
\mathbb {E} _ {\phi} V _ {1} ^ {\pi} (s) = \mathbb {E} _ {\hat {\phi}} V _ {1} ^ {\pi} (s).
$$
Proof. We prove by induction on the step  $l$  that  $\mathbb{E}_{\phi}Q_{l}^{\pi}(s,a) = \mathbb{E}_{\hat{\phi}}Q_{l}^{\pi}(s,a)$  for any  $(s,a)\in \mathcal{S}_l\times \mathcal{A}$ . This is true at step  $L + 1$ . Assuming that it is true at step  $l + 1$ , then
$$
\begin{array}{l} \mathbb {E} _ {\phi} Q _ {l} ^ {\pi} (s, a) - \mathbb {E} _ {\phi} Q _ {l + 1} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) \\ = \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) + \sum_ {s ^ {\prime}} P _ {l} (s ^ {\prime} \mid s, a) \sum_ {a ^ {\prime}} \pi_ {l + 1} (s ^ {\prime}, a ^ {\prime}) Q _ {l + 1} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) \right] \\ - \mathbb {E} _ {\hat {\phi}} \left[ \mathbb {E} _ {\phi} r _ {l} (s, a) + \omega_ {l} (s, a) + \sum_ {s ^ {\prime}} \mathbb {E} _ {\phi} P _ {l} \left(s ^ {\prime} \mid s, a\right) \sum_ {a ^ {\prime}} \pi_ {l + 1} \left(s ^ {\prime}, a ^ {\prime}\right) Q _ {l + 1} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ = \sum_ {s ^ {\prime}} \mathbb {E} _ {\phi} P _ {l} (s ^ {\prime} \mid s, a) \sum_ {a ^ {\prime}} \pi_ {l + 1} (s ^ {\prime}, a ^ {\prime}) \left[ \mathbb {E} _ {\phi} Q _ {l + 1} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) - \mathbb {E} _ {\hat {\phi}} Q _ {l + 1} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) \right] \\ = 0, \\ \end{array}
$$
because  $\mathbb{E}_{\phi}Q_{l}^{\pi}(s',a') = \mathbb{E}_{\hat{\phi}}Q_{l}^{\pi}(s',a')$  for every  $(s',a') \in S_{l+1} \times \mathcal{A}$  by induction hypothesis. In the above we used that  $\mathbb{E}_{\phi}[P_{l}(s' \mid s,a)Q_{l+1}^{\pi}(s',a')] = \mathbb{E}_{\phi}P_{l}(s' \mid s,a)\mathbb{E}_{\phi}Q_{l+1}^{\pi}(s',a')$ , which holds due to the time-inhomogeneity of the MDP, since the future return from a fixed state-action pair cannot be influenced by the dynamics that gets the agent to that state-action. As a result,
$$
\mathbb {E} _ {\phi} V _ {1} ^ {\pi} (s) - \mathbb {E} _ {\hat {\phi}} V _ {1} ^ {\pi} (s) = \sum_ {a} \pi_ {1} (s, a) \mathbb {E} _ {\phi} Q _ {1} ^ {\pi} (s, a) - \sum_ {a} \pi_ {1} (s, a) \mathbb {E} _ {\hat {\phi}} Q _ {1} ^ {\pi} (s, a) = 0.
$$
F.10 Proof of Theorem 2
Proof. We first explicitly bound the uncertainty measure  $\hat{\sigma}^t$  given our assumptions. The prior over the transition function  $P_{l}(\cdot |s,a)$  is assumed Dirichlet, and let us denote the parameter of the Dirichlet distribution  $\alpha_l^0 (s,a)\in \mathbb{R}_+^{S_{l + 1}}$  for each  $(s,a)$ , where  $\sum_{s'\in S_{l + 1}}\alpha_l^0 (s,a,s')\geq 1$ , i.e., we start with a total pseudo-count of at least one for every state-action (as done in [47]). Since the likelihood for the transition function is a Categorical distribution, conjugacy of the categorical and Dirichlet distributions implies that the posterior over  $P_{l}(\cdot |s,a)$  at time  $t$  is Dirichlet with parameter  $\alpha_l^t (s,a)$ , where  $\alpha_l^t (s,a,s') = \alpha_l^0 (s,a,s') + n_l^t (s,a,s')$  for each  $s'\in S_{l + 1}$ , where  $n_l^t (s,a,s')\in \mathbb{N}$  is the number of times the agent has been in state  $s$ , taken action  $a$ , and transitioned to state  $s'$  at timestep  $l$ , and note that  $\sum_{s'\in S_{l + 1}}n_l^t (s,a,s') = n_l^t (s,a)$ , the total visit count to  $(s,a)$ . Meanwhile, the reward noise is assumed additive  $\nu$ -sub-Gaussian, so we can upper bound the uncertainty measure  $\sigma^t\in \mathbb{R}_+^{L,S,A}$  as  $\sigma_l^t (s,a)\leq \sqrt{(\nu^2 + 1) / (n_l^t(s,a)\lor 1)}$ . As a result, we can bound
$$
\hat {\sigma} _ {l} ^ {t} (s, a) ^ {2} \leq \frac {3 . 6 ^ {2} (\nu^ {2} + 1) + (L - l) ^ {2}}{\left(n _ {l} ^ {t} (s , a) \vee 1\right)}. \tag {12}
$$
The result follows from
$$
\begin{array}{l} \mathcal {B R} _ {\phi} (\mathrm {V A P O R}, T) \leq \mathcal {B R} _ {\mathcal {T}, \phi} (\mathrm {V A P O R}, T) \leq \sqrt {2 (3 . 6 ^ {2} (\nu^ {2} + 1) + L ^ {2}) T S A \log (S A) (1 + \log T / L)} \\ = \tilde {O} (L \sqrt {S A T}), \\ \end{array}
$$
where the first inequality stems from Lemma 7 and the second inequality applies Lemma 12 whose condition (11) holds for  $c_0 = \sqrt{3.6^2(\nu^2 + 1) + L^2}$ .
F.11 Proof of Lemma 8
Lemma 8. Let  $\lambda^{\mathrm{TS}}$  be the occupancy measure of the TS policy, it holds that  $\mathbb{E}[\lambda^{\mathrm{TS}}] = \mathbb{P}_{\Gamma^{\star}}$ .
Proof. Given any environment  $\mathcal{M}$  sampled from the posterior  $\phi$ , the optimal policy  $\pi_{\mathcal{M}}^{\star}$  induces the occupancy measure  $\lambda_l^{\pi_{\mathcal{M}}^{\star}}(s,a) = \mathbb{P}_{\mathcal{M}}(\Gamma_l^{\star}(s,a))$ . Marginalizing over the environment  $\mathcal{M}$  yields  $\mathbb{E}[\lambda_l^{\mathrm{TS}}(s,a)] = \int_{\mathcal{M}}\lambda_l^{\pi_{\mathcal{M}}^{\star}}(s,a)d\mathbb{P}(\mathcal{M}) = \mathbb{P}_{\phi}(\Gamma_l^{\star}(s,a))$ .
F.12 Connection between K-learning and VAPOR
We sketch a proof of the connection between K-learning and VAPOR here. It proceeds via the dual in Lemma 9, by setting  $\tau_{l}(s,a) = \tau$  for all  $l,s,a$  and explicitly minimizing over the  $V\in \mathbb{R}^{L,S}$  variable. First, we define the following  $K$  -function as done in the original K-learning paper [47]
$$
K _ {l} (s, a) = \mathbb {E} _ {\phi} r _ {l} (s, a) + \frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau} + \sum_ {s ^ {\prime}} P _ {l} (s ^ {\prime} \mid s, a) V _ {l + 1} (s ^ {\prime}),
$$
for each  $l, s, a$ . After setting the  $\tau$  parameters to be equal, we obtain the following dual over  $K \in \mathbb{R}^{L,S \times A}$ ,  $V \in \mathbb{R}^{L,S}$ ,  $\tau \in \mathbb{R}$
minimize  $\sum_{s}\rho (s)V_{1}(s) + \tau \sum_{l,s,a}\exp \left(\frac{K_{l}(s,a) - V_{l}(s)}{\tau} -1\right)$
subject to  $K_{l}(s,a)\geq \mathbb{E}_{\phi}r_{l}(s,a) + \frac{\sigma_{l}^{2}(s,a)}{2\tau} +\sum_{s^{\prime}}P_{l}(s^{\prime}\mid s,a)V_{l + 1}(s^{\prime})$
where we have introduced the  $K$ -function as an inequality constraint that becomes tight at the optimum and preserves convexity. Explicitly minimizing over  $V_{1}$  we obtain
$$
V _ {1} (s) = \tau \log \sum_ {a} \exp \left(\frac {K _ {1} (s , a)}{\tau}\right) - \tau \log \rho (s) - \tau
$$
for each  $s \in S_1$ , and doing the same for  $V_l(s)$ ,  $l > 1$  we have
$$
V _ {l} (s) = \tau \log \sum_ {a} \exp \left(\frac {K _ {l} (s , a)}{\tau}\right) - \tau \log \mu_ {l} (s) - \tau
$$
for each  $s \in S_l$ , where  $\mu_l(s)$  is the dual variable associated with the inequality constraint and corresponds to the stationary state distribution at  $s$ . Substituting in for  $V$  we can write the K-values as
$$
K _ {l} (s, a) = \mathbb {E} _ {\phi} r _ {l} (s, a) + \frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau} + \sum_ {s ^ {\prime}} P _ {l} (s ^ {\prime} \mid s, a) \tau \log \frac {1}{\mu_ {l} (s)} \sum_ {a ^ {\prime}} \exp \left(\frac {K _ {l + 1} (s ^ {\prime} , a ^ {\prime})}{\tau}\right) - \tau ,
$$
Thus we have derived a variant of K-learning, with additional terms corresponding to the stationary state distribution. These additional terms become entropy in the objective function, and since the sum of these entropy terms can be bounded above by  $L \log S$ , it only contributes a small additional term to the regret bound. Practically speaking this version of K-learning requires knowing the stationary state distribution  $\mu(s)$ , so it is more challenging to implement than the original K-learning. This analysis is not to suggest this algorithm as a practical alternative, but instead to demonstrate the connection between K-learning and VAPOR.
G Gaussian Sub-Gaussian Stochastic Optimism
The following technical lemma, which may be of independent interest, establishes a relation of stochastic optimism (Definition 3) between a sub-Gaussian and a Gaussian whose variance is inflated by a small universal constant.
Lemma 19. Let  $Y$  be  $\sigma$ -sub-Gaussian. Let  $X \sim \mathcal{N}(0, \kappa^2 \sigma^2)$  with  $\kappa \geq 3.6$ . Then  $X \geq_{SO} Y$ .
Proof. Let  $u: \mathbb{R} \to \mathbb{R}$  be a convex (increasing) function. We aim to show that  $\mathbb{E}[u(X)] \geq \mathbb{E}[u(Y)]$ . Let  $s_0 \in \mathbb{R}$  be a subgradient of  $u$  at 0 (which exists since  $u$  is convex). Then by convexity of  $u$ , we have for all  $x \in \mathbb{R}$ ,  $u(x) \geq u(0) + s_0x$ . Define the function  $f: \mathbb{R} \to \mathbb{R}$  as  $f(x) := u(x) - u(0) - s_0x$ . We have that  $f(x) \geq 0$  for all  $x \in \mathbb{R}$  and  $f(0) = 0$ . Since  $\mathbb{E}X = \mathbb{E}Y = 0$ , it suffices to show that  $\mathbb{E}[f(X)] \geq \mathbb{E}[f(Y)]$ . For any  $\epsilon \geq 0$ , define the  $\epsilon$ -sublevel set of the convex function  $f$  as  $K_{\epsilon} := \{x \in \mathbb{R}: f(x) \leq \epsilon\}$ . Since  $K_{\epsilon}$  is convex and by property of  $f$ , there exists  $a(\epsilon) \in \mathbb{R}_{-} \cup \{-\infty\}$  and  $b(\epsilon) \in \mathbb{R}_{+} \cup \{+\infty\}$  such that  $K_{\epsilon} = [a(\epsilon), b(\epsilon)]$ . Then, for any random variable  $Z$ ,
$$
\begin{array}{l} \mathbb {E} f (Z) = \int_ {0} ^ {\infty} \mathbb {P} (f (Z) \geq \epsilon) \mathrm {d} \epsilon \\ = \int_ {0} ^ {\infty} \left(\mathbb {P} (Z \geq b (\epsilon)) + \mathbb {P} (Z \leq a (\epsilon))\right) d \epsilon . \\ \end{array}
$$
By the  $\sigma$ -sub-Gaussian property of  $Y$
$$
\begin{array}{l} \mathbb {E} f (Y) = \int_ {0} ^ {\infty} (\mathbb {P} (Y \geq b (\epsilon)) + \mathbb {P} (Y \leq a (\epsilon))) d \epsilon \\ = \int_ {0} ^ {\infty} \left(\exp \left(\frac {- b (\epsilon) ^ {2}}{2 \sigma^ {2}}\right) + \exp \left(\frac {- a (\epsilon) ^ {2}}{2 \sigma^ {2}}\right)\right) d \epsilon \\ = \int_ {0} ^ {\infty} (G (b (\epsilon)) + G (a (\epsilon))) d \epsilon , \\ \end{array}
$$
$$
G (x) := \exp \left(- x ^ {2} / \left(2 \sigma^ {2}\right)\right).
$$
Let  $d \coloneqq 2.69$  and  $c \coloneqq \frac{\kappa}{d} \geq 1.338$ . Let  $Z \sim \mathcal{N}(0, c^2\sigma^2)$  and  $F(x) \coloneqq \mathbb{P}(Z \geq x)$ . Note that
$$
F (x) = \frac {1}{2} \left(1 - \operatorname {e r f} \left(\frac {x}{\sigma c \sqrt {2}}\right)\right),
$$
where  $\operatorname{erf}$  denotes the error function, i.e.,  $\operatorname{erf}(y) = 2\pi^{-1/2} \int_{0}^{y} \exp(-t^2) \, \mathrm{d}t$ . From Lemma 20, by choice of constants  $c, d$ , it holds that  $G(x) \leq dF(x)$ . Therefore,
$$
\begin{array}{l} \mathbb {E} [ f (Y) ] \leq d \int_ {0} ^ {\infty} (F (b (\epsilon)) + F (a (\epsilon))) \mathrm {d} \epsilon \\ = d \int_ {0} ^ {\infty} \left(\mathbb {P} (Z \geq b (\epsilon)) + \mathbb {P} (Z \geq a (\epsilon))\right) \mathrm {d} \epsilon \\ = d \mathbb {E} [ f (Z) ] \\ \leq \mathbb {E} [ f (d Z) ] \\ = \mathbb {E} [ f (X) ], \\ \end{array}
$$
where the last inequality applies Lemma 21 to the convex function  $f$  that satisfies  $f(0) = 0$ , and the last equality is because  $\kappa = cd$  which means that  $X$  and  $dZ$  follow the same distribution.
Lemma 20. Denote by  $c_0 > 0$  the (unique) solution of the equation
$$
\frac {1}{2} \left(1 - \operatorname {e r f} \left(\frac {1}{c _ {0} \sqrt {2}}\right)\right) = \frac {1}{c _ {0} \sqrt {2 \pi}} \exp \left(- 1 / \left(2 c _ {0} ^ {2}\right)\right).
$$
and let  $d_0 > 0$  be defined as
$$
d _ {0} := \frac {2 \exp (- 1 / 2)}{1 - \operatorname {e r f} \left(\frac {1}{c _ {0} \sqrt {2}}\right)}.
$$
Note that  $c_0 \approx 1.33016$  and  $d_0 \approx 2.68271$ .
Then for any  $c\geq c_0,d\geq d_0$  , the function  $\phi :\mathbb{R}\to \mathbb{R}$  defined as
$$
\phi (x) := \frac {d}{2} \left(1 - \operatorname {e r f} \left(\frac {x}{c \sqrt {2}}\right)\right) - \exp (- x ^ {2} / 2)
$$
is non-negative everywhere.
$$
\phi_ {0} (x) := \frac {d _ {0}}{2} \left(1 - \operatorname {e r f} \left(\frac {x}{c _ {0} \sqrt {2}}\right)\right) - \exp (- x ^ {2} / 2).
$$
It suffices to prove that  $\phi_0(x) \geq 0$  for all  $x \in \mathbb{R}$ . By the choices of  $c_0, d_0$ , it holds that
$$
\phi_ {0} (1) = \frac {d _ {0}}{2} \left(1 - \operatorname {e r f} \left(\frac {1}{c _ {0} \sqrt {2}}\right)\right) - \exp (- 1 / 2) = 0,
$$
$$
\phi_ {0} ^ {\prime} (1) = \frac {- d _ {0}}{c _ {0} \sqrt {2 \pi}} \exp \left(- 1 / (2 c _ {0} ^ {2})\right) + \exp (- 1 / 2) = 0.
$$
Moreover, analyzing the variations of  $\phi$  yields that there exists  $x_0 > 1$  such that  $\phi_0'(x) \geq 0$  for  $x \in [1, x_0]$ , and  $\phi_0'(x) \leq 0$  otherwise. Therefore  $\phi_0$  is non-decreasing on  $[1, x_0]$  and non-increasing elsewhere. Since  $\lim_{x \to +\infty} \phi_0(x) = 0$  and  $\phi_0(1) = 0$ , we get that  $\phi_0(x) \geq 0$  for all  $x \in \mathbb{R}$ .
Lemma 21. For any convex function  $g: \mathbb{R} \to \mathbb{R}$  such that  $g(0) = 0$ , for any  $s \geq 1$  and  $x \in \mathbb{R}$ , it holds that  $sg(x) \leq g(sx)$ .
Proof. By convexity of  $g$ , for any  $x' \in \mathbb{R}$ , we have  $g(s^{-1}x') \leq s^{-1}g(x') + (1 - s^{-1})g(0) = s^{-1}g(x')$ , so by setting  $x' = sx$  it holds that  $sg(x) = sg(s^{-1}x') \leq g(x') = g(sx)$ .
H VAPOR-lite Analysis
Denoting the uncertainty signal of the VAPOR optimization problem by  $\hat{\sigma}$ , we define the VAPOR-lite  $(c)$  optimization problem for any multiplicative scalar  $c > 0$  as follows
$$
\begin{array}{l} \max  _ {\lambda \in \Lambda (\mathbb {E} _ {\phi} P)} \mathcal {U} _ {\widehat {\phi}} (\lambda , c) \\ := \sum_ {l, s, a} \lambda_ {l} (s, a) \left(\mathbb {E} _ {\phi} r _ {l} (s, a) + c \widehat {\sigma} _ {l} (s, a) - \sum_ {a ^ {\prime}} c \widehat {\sigma} _ {l} (s, a ^ {\prime}) \frac {\lambda_ {l} (s , a ^ {\prime})}{\sum_ {b} \lambda_ {l} (s , b)} \log \left(\frac {\lambda_ {l} (s , a ^ {\prime})}{\sum_ {b} \lambda_ {l} (s , b)}\right)\right) \\ = \sum_ {l, s, a} \lambda_ {l} (s, a) \left(\mathbb {E} _ {\phi} r _ {l} (s, a) + c \hat {\sigma} _ {l} (s, a) + \mathcal {H} _ {c \hat {\sigma} _ {l} (s, \cdot)} \left(\frac {\lambda_ {l} (s , \cdot)}{\sum_ {b} \lambda_ {l} (s , b)}\right)\right). \tag {13} \\ \end{array}
$$
Note that this corresponds to the same objective as (5) except that it is optimized over the occupancy measures instead of the policies.
Lemma 22. The VAPOR-lite(c) optimization problem is concave in  $\lambda$ .
Proof. The objective is concave due to the fact that the normalized entropy function  $h$  of  $u \in \mathbb{R}_{>0}^n$  given by  $h(u) = \sum_{i=1}^{n} u_i \log(\mathbf{1}^\top u / u_i)$  is concave (see [6, Example 3.19]) since the perspective operation preserves convexity.
Lemma 23. For any  $\eta >0$  and  $\lambda \in \Lambda (\mathbb{E}_{\phi}P)$  , it holds that
$$
\mathcal {V} _ {\hat {\phi}} (\lambda) \leq \mathcal {U} _ {\hat {\phi}} (\lambda , c _ {\eta}) + \sqrt {2} \hat {\sigma} _ {\max } L S \eta ,
$$
where  $c_{\eta} \coloneqq \sqrt{2}(1 + \log 1 / \eta)$  and  $\hat{\sigma}_{\max} \coloneqq \max_{l, s, a} \hat{\sigma}_l(s, a)$ .
Proof. Any occupancy measure  $\lambda \in \Lambda (\mathbb{E}_{\phi}P)$  can be decomposed as  $\lambda_l(s,a) = \mu_l^\pi (s)\pi_l(s,a)$ , where  $\mu^{\pi}$  denotes the stationary state distribution under  $\pi$ , i.e.,  $\mu_l^\pi (s)\coloneqq \sum_{a\in \mathcal{A}}\lambda_l(s,a)$ , and  $\pi$  is the policy given by  $\pi_l(s,a)\coloneqq \lambda_l(s,a) / \mu_l^\pi (s)$  so long as  $\mu_l^\pi (s) > 0$ , otherwise  $\pi_l(s,\cdot)$  can be any distribution, e.g., uniform. It holds that
$$
\begin{array}{l} \lambda_ {l} (s, a) \sqrt {- \log \lambda_ {l} (s , a)} \stackrel {\mathrm {(i)}} {\leq} \lambda_ {l} (s, a) - \lambda_ {l} (s, a) \log \lambda_ {l} (s, a) \\ = \lambda_ {l} (s, a) - \mu_ {l} ^ {\pi} (s) \pi_ {l} (s, a) \log \pi_ {l} (s, a) - \pi_ {l} (s, a) \mu_ {l} ^ {\pi} (s) \log \mu_ {l} ^ {\pi} (s) \\ \stackrel {\text {(i i)}} {\leq} \lambda_ {l} (s, a) - \mu_ {l} ^ {\pi} (s) \pi_ {l} (s, a) \log \pi_ {l} (s, a) + \pi_ {l} (s, a) \mu_ {l} ^ {\pi} (s) \log 1 / \eta + \pi_ {l} (s, a) \eta \\ = \lambda_ {l} (s, a) (1 + \log 1 / \eta) - \mu_ {l} ^ {\pi} (s) \pi_ {l} (s, a) \log \pi_ {l} (s, a) + \pi_ {l} (s, a) \eta , \\ \end{array}
$$
where (i) uses that  $\sqrt{-\log x} \leq 1 - \log(x)$  for any  $0 < x \leq 1$ , and (ii) uses that  $-x \log x \leq x \log 1 / \eta + \eta$  for any  $0 \leq x \leq 1$  and  $\eta > 0$ . Hence, we have
$$
\begin{array}{l} \mathcal {V} _ {\hat {\phi}} (\lambda) = \sum_ {l, s, a} \lambda_ {l} (s, a) \mathbb {E} _ {\phi} r _ {l} (s, a) + \sum_ {l, s, a} \hat {\sigma} _ {l} (s, a) \lambda_ {l} ^ {\pi} (s, a) \sqrt {- 2 \log \lambda_ {l} ^ {\pi} (s , a)} \\ \leq \sum_ {l, s, a} \lambda_ {l} (s, a) \left(\mathbb {E} _ {\phi} r _ {l} (s, a) + \sqrt {2} (1 + \log 1 / \eta) \hat {\sigma} _ {l} (s, a)\right) \\ + \sum_ {l, s} \mu_ {l} ^ {\pi} (s) \left(- \sum_ {a} \sqrt {2} \hat {\sigma} _ {l} (s, a) \pi_ {l} (s, a) \log \pi_ {l} (s, a)\right) + \sum_ {l, s, a} \sqrt {2} \eta \hat {\sigma} _ {l} (s, a) \pi_ {l} (s, a) \\ \leq \mathcal {U} _ {\hat {\phi}} (\lambda , c _ {\eta}) + \sqrt {2} \hat {\sigma} _ {\max} L S \eta . \\ \end{array}
$$
Lemma 24. Consider the algorithm VAPOR-lite that at each episode  $t = 1, \dots, N$  solves the optimization problem VAPOR-lite  $(c = \sqrt{2}(1 + \log SLt))$ . Then under Assumptions 1 and 2, it holds that
$$
\mathcal {B R} _ {\phi} (\text {V A P O R - l i t e}, T) \leq \widetilde {O} (L \sqrt {S A T}).
$$
Proof. Denote by  $\pi^t$  the policy executed by the algorithm at episode  $t$ , and by  $\mathbb{E}^t\mathbb{P}_{\Gamma^\star}$  the posterior probability of state-action optimality in the expected model  $\mathbb{E}^t P$ . It holds that
$$
\begin{array}{l} \mathbb {E} _ {s \sim \rho} \mathbb {E} ^ {t} V _ {1} ^ {\star} (s) \stackrel {\mathrm {(i)}} {\leq} \mathcal {V} _ {\hat {\phi} ^ {t}} (\mathbb {E} ^ {t} \mathbb {P} _ {\Gamma^ {\star}}) \\ \stackrel {\mathrm {(i i)}} {\leq} \mathcal {U} _ {\hat {\phi} ^ {t}} \left(\mathbb {E} ^ {t} \mathbb {P} _ {\Gamma^ {\star}}, 1 / (S L t)\right) + \sqrt {2} \hat {\sigma} _ {\max } / t \\ \stackrel {\mathrm {(i i i)}} {\leq} \mathcal {U} _ {\widehat {\phi} ^ {t}} \left(\mathbb {E} ^ {t} \lambda^ {\pi^ {t}}, 1 / (S L t)\right) + \sqrt {2} \widehat {\sigma} _ {\max } / t, \\ \end{array}
$$
where (i) combines Lemmas 4 and 6, (ii) applies Lemma 23 to  $\mathbb{E}^t\mathbb{P}_{\Gamma^*}\in \Lambda (\mathbb{E}^t P)$  with the choice  $\eta = 1 / (SLt)$ , and (iii) comes from the fact that  $\mathbb{E}^t\lambda^{\pi^t}\in \mathrm{argmax}_{\lambda \in \Lambda (\mathbb{E}^t P)}\mathcal{U}_{\hat{\phi}^t}(\lambda ,1 / (SLt))$ . Hence, retracing the steps of the regret derivation of the proof of Lemma 12, we can bound the Bayes regret of VAPOR-lite as