SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(B.8) and (B.9) are replaced by
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \xi_ {h} ^ {k} \leq c _ {\xi} (D + 1) \sqrt {K (D + 1) H ^ {3} \iota} \quad \text {a n d} \quad \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} b _ {h} ^ {k} \leq c _ {b} (D + 1) ^ {3 / 2} \sqrt {H ^ {3} S A K \iota},
$$
respectively. Lastly, we also have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \sqrt {\frac {1}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}} \leq 2 (D + 1) ^ {2} \sqrt {S A K H}.
$$
Putting these updated upper bounds together and substituting into  $\operatorname{Regret}(K)$ , we deduce
$$
\operatorname {R e g r e t} (K) \leq c \left(\left(D + 1\right) ^ {5 / 2} \sqrt {H ^ {3} S A K \iota} + H ^ {4} S ^ {2} A \iota^ {2}\right).
$$
B.3 Statement and proof of Proposition B.2
Proposition B.2. In the setup of Section 4.2, we have
$$
\begin{array}{l} \operatorname {g a p} (s _ {1}) \leq \sum_ {h = 1} ^ {H} \left[ \underbrace {\int_ {\tau} \left(\mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ \max  _ {a} r _ {h} (s , a) ] - \max  _ {a} \mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ r _ {h} (s , a) ]\right) \left(\rho_ {h} ^ {\pi_ {\mathrm {d e l a y}} ^ {*}} \wedge \rho_ {h} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}}\right) (\tau) \mathrm {d} \tau} _ {\mathcal {E} _ {1}} \right. \\ \left. + 2 \underbrace {\left\| \rho_ {h} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} - \rho_ {h} ^ {\pi_ {\mathrm {d e l a y}} ^ {*}} \right\| _ {\mathrm {T V}}} _ {\mathcal {E} _ {2}} \right]. \\ \end{array}
$$
where  $\rho_h^{\pi_{\mathrm{nodelay}}^*}$  and  $\rho_h^{\pi_{\mathrm{delay}}^*}$  are visitation measures induced by  $\pi_{\mathrm{nodelay}}^*$  and  $\pi_{\mathrm{delay}}^*$ , respectively.
Term  $\mathcal{E}_1$  is strictly larger than zero due to the convexity of the max operation. Term  $\mathcal{E}_2$  accounts for the difference in the visitation measure. When the original MDP has deterministic transitions, we can check that  $\mathcal{E}_1$  is zero, since the expectation over  $s$  is concentrated on a singleton that can be inferred from history. Hence, the visitation measures are also identical, which implies  $V_{1,\mathrm{nodelay}}^*(s_1) - V_{1,\mathrm{delay}}^*(s_1) = 0$ . On the contrary, when  $\mathfrak{b}_h(\cdot|\tau)$  is evenly spread, meaning that the entropy of  $\mathfrak{b}_h$  is high, we potentially suffer from a large performance drop, in that, inferring the current state is difficult.
Proof of Proposition B.2. Let  $\tau_{1},\ldots ,\tau_{H}$  denote the states observed in the delayed environment. Since  $\pi_{\mathrm{nodelay}}^{*}$  is greedy and Markov, we obtain
$$
\begin{array}{l} V _ {1, \mathrm {n o d e l a y}} ^ {*} (s _ {1}) = \mathbb {E} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H - 1} r _ {h} (s _ {h}, a _ {h}) | s _ {1} \right] + \mathbb {E} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} \left[ \mathbb {E} [ r _ {H} (s _ {H}, a _ {H}) | \tau_ {H} ] | s _ {1} \right] \\ = \mathbb {E} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H - 1} r _ {h} (s _ {h}, a _ {h}) | s _ {1} \right] + \mathbb {E} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} \left[ \sum_ {s} \mathfrak {b} _ {H} (s | \tau_ {H}) \max _ {a} r _ {H} (s, a) | s _ {1} \right]. \\ \end{array}
$$
Recursively applying the above argument, we deduce
$$
V _ {1, \mathrm {n o d e l a y}} ^ {*} (s _ {1}) = \mathbb {E} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H} \sum_ {s} \mathfrak {b} _ {h} (s | \tau_ {h}) \max _ {a} r _ {h} (s, a) | s _ {1} \right].
$$
We also rewrite  $V_{1,\mathrm{delay}}^{*}(s_{1})$  as
$$
\begin{array}{l} V _ {1, \text {d e l a y}} ^ {*} (s _ {1}) = \mathbb {E} ^ {\pi_ {\text {d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H - 1} r _ {h} \left(s _ {h}, a _ {h}\right) | s _ {1} \right] + \mathbb {E} ^ {\pi_ {\text {d e l a y}} ^ {*}} \left[ \mathbb {E} \left[ r _ {H} \left(s _ {H}, a _ {H}\right) \mid \tau_ {H} \right] \mid s _ {1} \right] \\ = \mathbb {E} ^ {\pi_ {\text {d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H - 1} r _ {h} \left(s _ {h}, a _ {h}\right) | s _ {1} \right] + \mathbb {E} ^ {\pi_ {\text {d e l a y}} ^ {*}} \left[ \max  _ {a} \sum_ {s} \mathfrak {b} _ {H} (s | \tau_ {H}) r _ {H} (s, a) | s _ {1} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \dots \\ = \mathbb {E} ^ {\pi_ {\text {d e l a y}} ^ {*}} \left[ \sum_ {h = 1} ^ {H} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau_ {h}) r _ {h} (s, a) | s _ {1} \right]. \\ \end{array}
$$
Then we write the difference between  $V_{1,\mathrm{nodelay}}^{*}(s_1)$  and  $V_{1,\mathrm{delay}}^{*}(s_1)$  as
$$
\begin{array}{l} V _ {1, \text {n o d e l a y}} ^ {*} (s _ {1}) - V _ {1, \text {d e l a y}} ^ {*} (s _ {1}) \\ = \sum_ {h = 1} ^ {H} \left(\int_ {\tau} \sum_ {s} \max  _ {a} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau - \int_ {\tau} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau\right) \\ = \sum_ {h = 1} ^ {H} \left(\int_ {\tau} \sum_ {s} \max  _ {a} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau - \int_ {\tau} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau \right. \\ + \int_ {\tau} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau - \int_ {\tau} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau \\ \leq \sum_ {h = 1} ^ {H} \left[ \int_ {\tau} \left(\mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ \max  _ {a} r _ {h} (s, a) ] - \max  _ {a} \mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ r _ {h} (s, a) ]\right) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau + 2 \| \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} - \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} \| _ {\text {T V}} \right]. \\ \end{array}
$$
$$
\begin{array}{l} V _ {1, \text {n o d e l a y}} ^ {*} (s _ {1}) - V _ {1, \text {d e l a y}} ^ {*} (s _ {1}) \\ = \sum_ {h = 1} ^ {H} \left(\int_ {\tau} \sum_ {s} \max  _ {a} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau - \int_ {\tau} \sum_ {s} \max  _ {a} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau \right. \\ \left. + \int_ {\tau} \sum_ {s} \max  _ {a} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau - \int_ {\tau} \max  _ {a} \sum_ {s} \mathfrak {b} _ {h} (s | \tau) r _ {h} (s, a) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau\right) \\ \leq \sum_ {h = 1} ^ {H} \left[ \int_ {\tau} \left(\mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ \max  _ {a} r _ {h} (s, a) ] - \max  _ {a} \mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ r _ {h} (s, a) ]\right) \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} (\tau) \mathrm {d} \tau + 2 \| \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}} - \rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} \| _ {\text {T V}} \right]. \\ \end{array}
$$
Combining the above two inequalities, we obtain
$$
\begin{array}{l} V _ {1, \text {n o d e l a y}} ^ {*} (s _ {1}) - V _ {1, \text {d e l a y}} ^ {*} (s _ {1}) \\ \leq \sum_ {h = 1} ^ {H} \left[ \int_ {\tau} \left(\mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ \max  _ {a} r _ {h} (s, a) ] - \max  _ {a} \mathbb {E} _ {s \sim \mathfrak {b} _ {h} (\cdot | \tau)} [ r _ {h} (s, a) ]\right) \left(\rho_ {h} ^ {\pi_ {\text {d e l a y}} ^ {*}} \wedge \rho_ {h} ^ {\pi_ {\text {n o d e l a y}} ^ {*}}\right) (\tau) \mathrm {d} \tau \right. \\ \left. + 2 \left\| \rho_ {h} ^ {\pi_ {\mathrm {n o d e l a y}} ^ {*}} - \rho_ {h} ^ {\pi_ {\mathrm {d e l a y}} ^ {*}} \right\| _ {\mathrm {T V}} \right]. \\ \end{array}
$$
B.4 Proof of Proposition 4.2
Proof. We construct an MDP instance  $(\mathcal{S},\mathcal{A},H,R,P)$  for  $H > d$  as follows. Let  $\mathcal{S} = \{1,2\}$  and  $\mathcal{A} = \{a_1,a_2\}$ . For the reward function, we have
$$
r _ {h} (s, a) = \left\{ \begin{array}{l l} 1 & \text {i f} a = a _ {s} \text {a n d} h = d + 1 \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
The reward is nonzero only at time  $d + 1$ . The transition probabilities are defined as
$$
p _ {h} (s ^ {\prime} | s, a) = \left\{ \begin{array}{l l} \frac {1}{2} & \text {i f} h = d + 1 \\ 1 & \text {i f} h \neq d + 1 \text {a n d} s ^ {\prime} = s \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
The transition probability at step  $d + 1$  says that  $s^{\prime}$  is uniform regardless of the previous state and action. Suppose a uniform initial distribution on  $s_1$ . We first show that if the constant delay equals  $d$ , then there exists a policy  $\pi^{*,d}$  achieving maximal value. Indeed, the policy is chosen as
$$
\pi_{h}^{*,d}(\cdot |\{s_{h - d},\mathbf{a}_{h - d:h - 1}\}) = \left\{ \begin{array}{ll}a_{s_{h - d}} & \text{if $h = d + 1$}\\ \text{Uniform} (\mathcal{A}) & \text{if $h\neq d + 1$.} \end{array} \right.
$$
It is straightforward to check that  $\pi^{*,d}$  is optimal, since at step  $d + 1$ ,  $s_1$  is revealed and the policy takes the optimal action  $a_{s_1}$  to obtain reward 1.
On the other hand, if the constant delay equals  $d + 1$ , then any policy suffers from a constant performance degradation. To see this, in a single trajectory, since the starting state is only revealed at time  $d + 2$ , the policy at time  $d + 1$  cannot exploit the information of the initial state. Therefore, any policy coincides with the best action with probability  $\frac{1}{2}$ . For  $K$  episodes, with probability  $1 - \gamma$ , the total reward of any policy  $\pi \in \Pi_{\mathrm{exec}}$  is bounded by
$$
\sum_ {k = 1} ^ {K} V _ {1} ^ {\pi} \left(s _ {1} ^ {k}\right) \leq \frac {1}{2} K + \sqrt {\frac {K}{2} \log \frac {1}{\gamma}},
$$
due to Hoeffding's inequality. As a result, the performance drop is at least
$$
\operatorname {g a p} (K) \geq \frac {1}{2} - \sqrt {\frac {1}{2 K} \log \frac {1}{\gamma}}.
$$
C Omitted proofs in Section 5
C.1 Proof of Proposition 5.1
Proof. We first show that the ground-truth transition probabilities  $p_h^{\theta^*}$  belongs to  $\mathcal{B}_k$  with high probability. By Theorem 2.2 in Weissman et al. [2003] (see also Equation (44) in Jaksch et al. [2010]), at the  $k$ -th episode, for any fixed  $(s, a, h)$ , we have
$$
\mathbb {P} \left(\| \widehat {p} _ {h} ^ {k} (\cdot | s, a) - p _ {h} ^ {\theta^ {*}} (\cdot | s, a) \| _ {\mathrm {T V}} \geq t\right) \leq (2 ^ {S} - 2) \exp \left(- \frac {N _ {h} ^ {k} (s , a) t ^ {2}}{2}\right).
$$
Setting  $t = c\sqrt{\frac{S\iota}{N_h^k(s,a)}}$  for some constant  $c$  ensures that
$$
\| \widehat {p} _ {h} ^ {k} (\cdot | s, a) - p _ {h} ^ {\theta^ {*}} (\cdot | s, a) \| _ {\mathrm {T V}} \leq c \sqrt {\frac {S \iota}{N _ {h} ^ {k} (s , a)}}
$$
holds over any  $(s, a, h, k)$  with probability  $1 - \gamma$ . As a consequence, the event  $p_h^{\theta^*}(\cdot | s, a) \in \mathcal{B}^k$  holds with probability  $1 - \gamma$  over all  $(s, a, h, k)$ .
Conditioned on the high probability event  $p_h^{\theta^*} \in \mathcal{B}^k$  for all  $(h, s, a)$ , we have by standard performance difference arguments that
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \max  _ {\pi \in \Pi_ {\text {e x e c}}} V _ {\theta^ {\star}} ^ {\pi} \left(s _ {1} ^ {k}\right) - V _ {\theta^ {\star}} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \stackrel {(i)} {\leq} \sum_ {k = 1} ^ {K} V _ {\theta^ {k}} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) - V _ {\theta^ {\star}} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \\ \stackrel {(i i)} {=} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\theta^ {\star}} ^ {\pi^ {k}} \left[ \left\langle (\mathbb {P} _ {h} ^ {\theta^ {k}} - \mathbb {P} _ {h} ^ {\theta^ {\star}}) (\cdot | s _ {h}, a _ {h}), V _ {\theta^ {k}, h + 1} ^ {\pi^ {k}} (\cdot) \right\rangle \right] \\ \leq \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\theta^ {\star}} ^ {\pi^ {k}} \left[ c \sqrt {\frac {H ^ {2} S _ {\iota}}{N _ {h} ^ {k} \left(s _ {h} , a _ {h}\right)}} \wedge H \right] \\ \stackrel {(i i i)} {\leq} \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} c ^ {\prime} \sqrt {\frac {H ^ {2} S \iota}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}} + H \sqrt {H ^ {2} K \iota} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i v)} {\leq} c ^ {\prime} \left(\left[ \begin{array}{c} \log \frac {H K}{\gamma} \\ \hline - \log (1 - \lambda_ {0} ^ {2}) \end{array} \right] \sqrt {H ^ {2} S _ {\iota} \cdot S A H K} + \sqrt {H ^ {4} K \iota}\right) \\ \leq c ^ {\prime} \left(\left\lceil \frac {1}{- \log \left(1 - \lambda_ {0} ^ {2}\right)} \right\rceil \sqrt {H ^ {3} S ^ {2} A K \iota^ {3}} + \sqrt {H ^ {4} K \iota}\right), \\ \end{array}
$$
where inequality  $(i)$  follows from the valid optimism since Line 4 in Algorithm 3 is taken over double maximization, equality  $(ii)$  recursively expands the value function and  $\langle \cdot ,\cdot \rangle$  denotes the inner product, inequality  $(iii)$  invokes Azuma-Hoeffding's inequality, and inequality  $(iv)$  invokes Lemma C.2.
C.2 Algorithm and proof of Theorem 5.2
Algorithm 4 Policy learning for MDPs with missing observations
1: Input: Horizon  $H$ .
2: Init:  $V_{H + 1}(\tau) = 0$  and  $Q_{H}(\tau, a) = H$  for any  $\tau, a$ , data set  $\mathcal{D}^0 = \emptyset$ , initial policy  $\pi^0$ .
3: for episode  $k = 1, \ldots, K$  do
4: Execute policy  $\pi^{k - 1}$
5: After the episode ends, collect data  $\mathcal{D}^k = \mathcal{D}^{k - 1}\cup \{(s_h^k,a_h^k,r_h^k)\}_{h = 1}^H$
6: On data set  $\mathcal{D}^k$ , compute counting numbers
$$
N _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) = \sum_ {j = 1} ^ {k} \mathbb {1} \left\{\tau_ {h} ^ {k} = \tau_ {h}, a _ {h} ^ {k} = a _ {h}, s _ {h + 1} ^ {k} \neq \emptyset \right\} \quad \text {a n d} \quad N _ {h, \lambda} ^ {k} = \sum_ {j = 1} ^ {k} \mathbb {1} \left\{s _ {h} ^ {k} = \emptyset \right\}.
$$
7: Estimate transition probabilities and delay distributions via
$$
\widehat {p} _ {h} ^ {k} \left(s _ {h + 1} \mid \tau_ {h}, a _ {h}\right) = \frac {N _ {h} ^ {k} \left(\tau_ {h} , a _ {h} , s _ {h + 1}\right)}{N _ {h} ^ {k} \left(\tau_ {h} , a _ {h}\right)} \quad \text {a n d} \quad \widehat {\lambda} _ {h} ^ {k} = N _ {h, \lambda} ^ {k} / k.
$$
$$
b _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) = c H \left(\sqrt {\frac {H \iota}{N _ {h} ^ {k} \left(\tau_ {h} , a _ {h}\right)}} + \sqrt {\frac {\iota}{k}}\right)
$$
9: Run optimistic value iteration in  $\mathsf{MDP}_{\mathrm{aug}}$  for  $H$  steps and obtain  $\pi^k \in \Pi_{\mathrm{exec}}$ .
10: end for
11: Return: Learned policy  $\pi^k$  for  $k = 1, \ldots, K$ .
We remark that similar to delayed MDPs, in Line 9 the planning is on  $\mathrm{MDP}_{\mathrm{aug}}$  and the obtained policy is executable given any  $\tau \in S_{\mathrm{aug}}$  when state observation is missed. Therefore, the planning complexity is  $SA^H$ . Different from Algorithm 2, the bonus function here depends on multi-step transitions, in that missing observations are permanently lost.
Proof of Theorem 5.2. The proof utilizes similar steps as Theorem 4.1, with an extra care on the summation of bonus functions.
Valid optimism We verify the choice of bonus functions leads to a valid optimism.
Lemma C.1. Given any failure probability  $\gamma < 1$ , we set bonus functions as
$$
b _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) = c H \left(\sqrt {\frac {H \iota}{N _ {h} ^ {k} \left(\tau_ {h} , a _ {h}\right)}} + \sqrt {\frac {\iota}{k}}\right) \quad \text {w i t h} \quad \iota = \log \left(\frac {S A H K}{\gamma}\right).
$$
Then with probability  $1 - \gamma$ , it holds
$$
Q _ {h, \text {a u g}} ^ {k} (\tau_ {h}, a _ {h}) \geq Q _ {h, \text {a u g}} ^ {*} (\tau_ {h}, a _ {h}), \quad V _ {h, \text {a u g}} ^ {k} (\tau_ {h}) \geq V _ {h, \text {a u g}} ^ {*} (\tau_ {h}) \quad \text {f o r a n y} \quad (k, h, \tau_ {h}, a _ {h}).
$$
Proof of Lemma C.1. In the proof, we omit subscript "aug" for simplicity. We use backward induction on time  $h$  again. The base case of  $H + 1$  holds immediately due to the initial value of  $V_{H + 1,\mathrm{aug}}$ . Suppose at time  $h + 1$ , the assertion holds. Then for time  $h$ , if  $Q_{h,\mathrm{aug}} = H$ , the assertion holds trivially. Otherwise, we have
$$
\begin{array}{l} Q _ {h} \left(\tau_ {h}, a _ {h}\right) - Q _ {h} ^ {*} \left(\tau_ {h}, a _ {h}\right) \\ = \widehat {r} _ {h} \left(\tau_ {h}, a _ {h}\right) + \left[ \widehat {\mathcal {P}} _ {h} V _ {h + 1} \right] \left(\tau_ {h}, a _ {h}\right) - r _ {h} \left(\tau_ {h}, a _ {h}\right) - \left[ \mathcal {P} _ {h} V _ {h + 1} ^ {*} \right] \left(\tau_ {h}, a _ {h}\right) + b _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) \\ \geq \underbrace {\left(\left[ \widehat {\mathcal {P}} _ {h} - \mathcal {P} _ {h} \right] V _ {h + 1} ^ {*}\right) (\tau_ {h} , a _ {h})} _ {(A)} + \underbrace {\widehat {r} _ {h} (\tau_ {h} , a _ {h}) - r _ {h} (\tau_ {h} , a _ {h})} _ {(B)} + b _ {h} ^ {k} (\tau_ {h}, a _ {h}). \\ \end{array}
$$
We lower bound  $(A)$  and  $(B)$  separately. For term  $(A)$ , we have
$$
\begin{array}{l} (A) = \sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h}) - p _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h})\right) \\ = \sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h}) - p _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h})\right) \mathbb {1} \{t _ {h + 1} = h + 1 \} \\ + \sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h}) - p _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h})\right) \mathbb {1} \{t _ {h + 1} = t _ {h} \} \\ = \underbrace {\sum_ {s _ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left((1 - \widehat {\lambda} _ {h}) \widehat {p} _ {h} (s _ {h + 1} | s _ {t _ {h}} , \mathbf {a} _ {t _ {h}: h}) - (1 - \lambda_ {h}) p _ {h} (s _ {h + 1} | s _ {t _ {h}} , \mathbf {a} _ {t _ {h}: h})\right)} _ {(A _ {1})} \\ + \underbrace {V _ {h + 1} ^ {*} (\{s _ {t _ {h}} , \mathbf {a} _ {t _ {h} : h} \}) (\widehat {\lambda} _ {h} - \lambda_ {h})} _ {(A _ {2})}. \\ \end{array}
$$
In  $(A_{1})$ ,  $\tau_{h + 1}$  is  $\{s_{h + 1}\}$ . We bound  $(A_{1})$  as
$$
\begin{array}{l} (A _ {1}) = \sum_ {s _ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \Big ((1 - \widehat {\lambda} _ {h}) \widehat {p} _ {h} (s _ {h + 1} | s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}) - (1 - \lambda_ {h}) \widehat {p} _ {h} (s _ {h + 1} | s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}) \\ \left. + \left(1 - \lambda_ {h}\right) \widehat {p} _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}\right) - \left(1 - \lambda_ {h}\right) p _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}\right)\right) \\ = \sum_ {s _ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) (1 - \lambda_ {h}) \left(\widehat {p} _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}\right) - p _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}\right)\right) \\ + \sum_ {s _ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) (\lambda_ {h} - \widehat {\lambda} _ {h}) \widehat {p} _ {h} (s _ {h + 1} | s _ {t _ {h}}, \mathbf {a} _ {t _ {h}: h}) \\ \end{array}
$$
$$
\stackrel {(i)} {\geq} - c _ {A} H \sqrt {\frac {H \iota}{N _ {h} (\tau_ {h} , a _ {h})}} - H \left| \widehat {\lambda} _ {h} - \lambda_ {h} \right|,
$$
where inequality  $(i)$  invokes Hoeffding's inequality and holds with probability  $1 - \gamma$  for any  $\tau_h, a_h$  and some constant  $c_A$ . Term  $(A_2)$  is immediately bounded by
$$
\left(A _ {2}\right) \geq - H \left| \widehat {\lambda} _ {h} - \lambda_ {h} \right|.
$$
Putting  $(A_{1})$  and  $(A_{2})$  together, we derive
$$
(A) \geq - c _ {A} H \sqrt {\frac {H \iota}{N _ {h} \left(\tau_ {h} , a _ {h}\right)}} - 2 H \left| \widehat {\lambda} _ {h} - \lambda_ {h} \right|
$$
with high probability. For term  $(B)$ , we have
$$
(B) = \sum_ {s _ {h}} r (s _ {h}, a _ {h}) \left(\widehat {\mathfrak {b}} _ {h} (s _ {h} | \tau_ {h}) - \mathfrak {b} _ {h} (s _ {h} | \tau_ {h})\right) \geq - c _ {B} \sqrt {\frac {H \iota}{N _ {h} (\tau_ {h} , a _ {h})}}.
$$
Taking  $c = c_A + c_B$  and summing up  $(A)$  and  $(B)$ , we have
$$
Q _ {h} (\tau_ {h}, a _ {h}) - Q _ {h} ^ {*} (\tau_ {h}, a _ {h}) \geq - c H \sqrt {\frac {H \iota}{N _ {h} (\tau_ {h} , a _ {h})}} - 2 H \left| \widehat {\lambda} _ {h} - \lambda_ {h} \right| + b _ {h} ^ {k} (\tau_ {h}, a _ {h}).
$$
We estimate  $\lambda_h$  by its empirical average. In episode  $k \geq 1$ , we have access to  $k$  i.i.d. realizations of Bernoulli random variable with rate  $\lambda_h$  (observable or not). Therefore, by Hoeffding's inequality, we have
$$
\left| \widehat {\lambda} _ {h} ^ {k} - \lambda_ {h} \right| \leq 2 \sqrt {\frac {\log \frac {H K}{\gamma}}{k}} \leq 2 \sqrt {\frac {\iota}{k}}.
$$
Substituting into  $Q_h^k(\tau_h, a_h) - Q_h^*(\tau_h, a_h)$  and reloading constant  $c$  give rise to
$$
Q _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) - Q _ {h} ^ {*} \left(\tau_ {h}, a _ {h}\right) \geq - c H \left(\sqrt {\frac {H \iota}{N _ {h} ^ {k} \left(\tau_ {h} , a _ {h}\right)}} + \sqrt {\frac {\iota}{k}}\right) + b _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) \geq 0.
$$
Regret analysis We omit subscript "aug" to ease the presentation. The same derivation in the proof of Theorem 4.1 gives rise to
$$
\begin{array}{l} \left(Q _ {h} ^ {*} - Q _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) \leq \left(Q _ {h} ^ {k} - Q _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) \\ \leq \underbrace {\left([ \widehat {\mathcal {P}} _ {h} ^ {k} - \mathcal {P} _ {h} ] [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ]\right) (\tau_ {h} ^ {k} , a _ {h} ^ {k})} _ {(A)} + \left(\mathcal {P} _ {h} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} ]\right) (\tau_ {h} ^ {k}, a _ {h} ^ {k}) + 2 b _ {h} ^ {k} (\tau_ {h} ^ {k}, a _ {h} ^ {k}). \quad (C. 1) \\ \end{array}
$$
Lemma C.1 shows that  $(A)$  can be written as
$$
\begin{array}{l} (A) = \sum_ {s _ {h + 1}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) (1 - \lambda_ {h}) \left(\widehat {p} _ {h} ^ {k} (s _ {h + 1} | s _ {t _ {h}} ^ {k}, {\bf a} _ {t _ {h}: h} ^ {k}) - p _ {h} (s _ {h + 1} | s _ {t _ {h}} ^ {k}, {\bf a} _ {t _ {h}: h} ^ {k})\right) \\ + \sum_ {s _ {h + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\lambda_ {h} - \widehat {\lambda} _ {h} ^ {k}\right) \widehat {p} _ {h} ^ {k} \left(s _ {h + 1} \mid s _ {t _ {h}} ^ {k}, \mathbf {a} _ {t _ {h}: h} ^ {k}\right) \\ \leq \sum_ {s _ {h + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] (\tau_ {h + 1}) (1 - \lambda_ {h}) \left(\widehat {p} _ {h} ^ {k} \left(s _ {h + 1} \mid s _ {t _ {h}} ^ {k}, \mathbf {a} _ {t _ {h}: h} ^ {k}\right) - p _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}} ^ {k}, \mathbf {a} _ {t _ {h}: h} ^ {k}\right)\right) + H \left| \widehat {\lambda} _ {h} ^ {k} - \lambda_ {h} \right| \\ \leq \left(1 - \lambda_ {h}\right) \sum_ {s _ {h + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {h} ^ {k} \left(s _ {h + 1} \mid s _ {t _ {h}} ^ {k}, \mathbf {a} _ {t _ {h}: h} ^ {k}\right) - p _ {h} \left(s _ {h + 1} \mid s _ {t _ {h}} ^ {k}, \mathbf {a} _ {t _ {h}: h} ^ {k}\right)\right) + 2 H \sqrt {\frac {\iota}{k}}. \\ \end{array}
$$
Following the derivation in (B.4), (B.5) and (B.6), we have
$$
\begin{array}{l} \operatorname {R e g r e t} (K) \leq e \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 H \sqrt {\frac {\iota}{k}}\right) \\ \leq e \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k}\right) + 2 \sqrt {H ^ {4} K \iota}. \\ \end{array}
$$