SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\xi_h^k = \left(\mathcal{P}_h\left[V_h^k -V_h^{\pi_k}\right]\right)\left(\tau_h^k,a_h^k\right) - \left[V_{h + 1}^k -V_{h + 1}^{\pi_k}\right]\left(\tau_{h + 1}^k\right)$  is the martingale difference and  $\zeta_h^k = c'\frac{SH^{2_\iota}}{N_h^k(\tau_h^k,a_h^k)}$
Counting number summation The summation over  $\xi_h^k$  is standard. Using the Azuma-Hoeffding's inequality, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \xi_ {h} ^ {k} \leq c _ {\xi} \sqrt {K H ^ {4} \iota}.
$$
It remains to find the summations involving  $N_h^k(\tau_h^k, a_h^k)$ . First, we show that the event  $\mathcal{E}_m = \{h - t_h - 1 \leq m\}$ , i.e., the maximal consecutive delay is upper bounded by  $m > 0$ , holds with high probability. We have
$$
\mathbb {P} \left(\mathcal {E} _ {m}\right) \leq \left(1 - H \left(1 - \lambda_ {0}\right) ^ {m + 1}\right) ^ {K},
$$
since  $\lambda_0$  is a uniform lower bound of  $\lambda_h$ . Next, we provide an upper bound on  $N_h^K (\tau_h,a_h)$ . For a given tuple  $(h,\tau_h,a_h,t_h)$ , the consecutive missing length is  $h - t_h - 1$ . Such a missing pattern appears with probability at most  $(1 - \lambda_0)^{h - t_h - 1}$ . As a consequence, denote  $C_{h - t_h - 1}^K$  as the number of  $h - t_h - 1$  consecutive missings in  $K$  episodes. With probability  $1 - \gamma$ , we have
$$
C _ {h - t _ {h} - 1} ^ {K} \leq K (1 - \lambda_ {0}) ^ {h - t _ {h} - 1} + \sqrt {K (1 - \lambda_ {0}) ^ {h - t _ {h} - 1} H \iota} + \iota .
$$
by Bernstein's inequality in Lemma D.1. Furthermore, at a fixed time  $h$ , we use Lemma C.3 to bound the gap between two consecutive appearances of the same missing pattern. We instantiate Lemma C.3 with  $\theta = (1 - \lambda_0)^{h - t_h - 1}$  and obtain that the gap is bounded by  $\left\lceil \frac{\iota}{-\log(1 - (1 - \lambda_0)^{h - t_h - 1})} \right\rceil$  with probability  $1 - \gamma$ . Within the gap, the number of consecutive delays of length larger than  $h - t_h - 1$  is bounded by
$$
\begin{array}{l} C _ {\geq h - t _ {h} - 1} \stackrel {(i)} {\leq} \left\lceil \frac {\iota}{- \log (1 - (1 - \lambda_ {0}) ^ {h - t _ {h} - 1})} \right\rceil (1 - \lambda_ {0}) ^ {h - t _ {h}} \\ + \sqrt {\left\lceil \frac {\iota}{- \log (1 - (1 - \lambda_ {0}) ^ {h - t _ {h} - 1})} \right\rceil (1 - \lambda_ {0}) ^ {h - t _ {h}} H \iota} + \iota \\ \stackrel {(i i)} {\leq} \sqrt {2 (1 - \lambda_ {0}) H \iota} + 2 (1 - \lambda_ {0}) + \iota , \\ \end{array}
$$
where inequality  $(i)$  follows from Bernstein's inequality again and inequality  $(ii)$  invokes the fact  $x + \log (1 - x)\leq 0$  for  $x\in [0,1)$  and bounds  $\lceil x\rceil$  by  $x + 1$ . Now we can bound the summation of the counting numbers. Conditioned on the event  $\mathcal{E}_m$ , we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sqrt {\frac {1}{N _ {h} ^ {k} \left(\tau_ {h} ^ {k} , a _ {h} ^ {k}\right)}} \stackrel {(i)} {\leq} \sum_ {(h, \tau , a, t _ {h})} C _ {\geq h - t _ {h} - 1} \sum_ {i = 1} ^ {N _ {h} ^ {K} (\tau , a)} \sqrt {\frac {1}{i}} \\ \leq 2 \left(\sqrt {2 (1 - \lambda_ {0}) H \iota} + 2 (1 - \lambda_ {0}) + \iota\right) \sum_ {(h, \tau , a, t _ {h})} \sqrt {N _ {h} ^ {K} (\tau , a)} \\ \stackrel {(i i)} {\leq} 2 \left(\sqrt {2 (1 - \lambda_ {0}) H \iota} + 2 (1 - \lambda_ {0}) + \iota\right) \sum_ {h, t _ {h}} \sqrt {S A ^ {h - t _ {h}} C _ {h - t _ {h} - 1} ^ {K}} \\ \leq 2 \left(\sqrt {2 \left(1 - \lambda_ {0}\right) H \iota} + 2 \left(1 - \lambda_ {0}\right) + \iota\right) \\ \cdot \sum_ {h, t _ {h}} \sqrt {S A \left(K ((1 - \lambda_ {0}) A) ^ {h - t _ {h} - 1} + \sqrt {K (A ^ {2} (1 - \lambda_ {0})) ^ {h - t _ {h} - 1} H \iota} + A ^ {h - t _ {h} - 1} \iota\right)} \\ \stackrel {(i i i)} {\leq} 2 \left(\sqrt {2 (1 - \lambda_ {0}) H \iota} + 2 (1 - \lambda_ {0}) + \iota\right) \sum_ {h, t _ {h}} \sqrt {S A \left(K + \sqrt {K A ^ {m} H \iota} + A ^ {m} \iota\right)} \\ \leq 2 \left(\sqrt {2 \left(1 - \lambda_ {0}\right) H \iota} + 2 \left(1 - \lambda_ {0}\right) + \iota\right) H ^ {2} \sqrt {S A \left(K + \sqrt {K A ^ {m} H \iota} + A ^ {m} \iota\right)} \\ \leq 2 \sqrt {H ^ {5} S A \iota^ {2} \left(K + \sqrt {K A ^ {m} H \iota} + A ^ {m} \iota\right)}, \\ \end{array}
$$
where inequality  $(i)$  follows since  $N_h^k$  is repeated at most  $C_{\geq h - t_h - 1}$  times before getting an update and inequality  $(ii)$  follows from Cauchy-Schwarz inequality, and inequality  $(iii)$  invokes the assumption of  $\lambda A \leq 1$ . Moreover, conditioned on the event  $\mathcal{E}_m$ , we also have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {1}{N _ {h} ^ {k} \left(\tau_ {h} ^ {k} , a _ {h} ^ {k}\right)} \leq \sum_ {\left(h, \tau , a, t _ {h}\right)} C _ {\geq h - t _ {h} - 1} \sum_ {i = 1} ^ {N _ {h} ^ {K} (\tau , a)} \frac {1}{i}
$$
$$
\begin{array}{l} \leq \left(\sqrt {2 (1 - \lambda_ {0}) H \iota} + 2 (1 - \lambda_ {0}) + \iota\right) \sum_ {(h, \tau , a, t _ {h})} \log N _ {h} ^ {K} (\tau , a) \\ \leq \iota H ^ {5 / 2} S A ^ {m + 1} \log K. \\ \end{array}
$$
Combining the above On event  $\mathcal{E}_m$  , the regret is bounded by
$$
\begin{array}{l} \operatorname {R e g r e t} (K) \stackrel {(i)} {\leq} c \left(\sqrt {H ^ {4} K \iota} + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left[ \frac {S H ^ {2} \iota}{N _ {h} ^ {k} \left(\tau_ {h} ^ {k} , a _ {h} ^ {k}\right)} + H \sqrt {\frac {H \iota}{N _ {h} ^ {k} \left(\tau_ {h} ^ {k} , a _ {h} ^ {k}\right)}} \right]\right) \\ \leq c \left(H ^ {4} \sqrt {S A \iota^ {3} K \left(1 + \sqrt {\frac {A ^ {m} H \iota}{K}} + \frac {A ^ {m} \iota}{K}\right)} + S ^ {2} A ^ {m} \sqrt {H ^ {9} \iota^ {6}} + \sqrt {H ^ {4} K \iota}\right), \\ \end{array}
$$
where  $c$  is a sufficiently large constant and we substitute the bonus functions into inequality (i).
On the complement of  $\mathcal{E}_m$ , the regret is bounded by  $H(1 - \mathbb{P}(\mathcal{E}_m)) \leq H^2 K(1 - \lambda_0)^{m + 1}$ . We choose  $m = \frac{1}{2} \left\lfloor \frac{\log K}{-\log(1 - \lambda_0)} \right\rfloor$  such that  $H(1 - \mathbb{P}(\mathcal{E}_m)) \leq H^2 K(1 - \lambda_0)^{m + 1} \leq H^2 \sqrt{K}$ . We can now check that  $A^{m + 1} = \exp \left(\frac{\log A}{-\log(1 - \lambda_0)} \log \sqrt{K}\right) \leq K^{\frac{1}{2(1 + v)}}$ . Therefore, combining the regret on event  $\mathcal{E}_m$  and the complement event  $\mathcal{E}_m^{\complement}$  leads to
$$
\operatorname {R e g r e t} (K) \leq c \left(H ^ {4} \sqrt {S A K \iota^ {3}} + S ^ {2} \sqrt {H ^ {9} K ^ {\frac {1}{(1 + v)}} \iota^ {6}}\right).
$$
C.3 Supporting lemmas
Lemma C.2. Suppose Assumption 2.2 holds. With probability  $1 - \gamma$  for some failure probability  $\gamma >0$ , we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {1}{\sqrt {N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}} \leq \left\lceil \frac {\log \frac {H K}{\gamma}}{- \log (1 - \lambda_ {0} ^ {2})} \right\rceil \sqrt {S A K H}.
$$
Proof of Lemma C.2. For any time  $h$ , we denote  $\mathcal{K}^{\mathrm{eff}}(h)$  as the collection of episodes that the  $h$ -th and  $(h + 1)$ -th step observations are available. It is clear that the cardinality of  $\mathcal{K}^{\mathrm{eff}}(h)$  is bounded by  $K$  for any  $h$ . Within each  $\mathcal{K}^{\mathrm{eff}}(h)$ , we would like to bound the gap between two observations. Thanks to Lemma C.3, the gap is bounded by  $q$  with probability  $1 - K(1 - \lambda_0^2)^{q + 1}$ . We set  $K(1 - \lambda_0^2)^{q + 1} = \gamma / H$ , which implies  $q = \left\lceil \frac{\log\frac{HK}{\gamma}}{-\log(1 - \lambda_0^2)}\right\rceil$ . Therefore, for any time step  $h$ , available observations are at most separated by  $q$  episodes.
With these notations, we bound
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {1}{\sqrt {N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}} \stackrel {(i)} {\leq} \left[ \frac {\log \frac {H K}{\gamma}}{- \log \left(1 - \lambda_ {0} ^ {2}\right)} \right] \sum_ {h = 1} ^ {H} \sum_ {k \in \mathcal {K} ^ {\mathrm {e f f}} (h)} \frac {1}{\sqrt {N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}} \\ \stackrel {(i i)} {\leq} \left[ \frac {\log \frac {H K}{\gamma}}{- \log (1 - \lambda_ {0} ^ {2})} \right] \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \frac {1}{\sqrt {N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}} \\ \stackrel {(i i i)} {\leq} 2 \left[ \frac {\log \frac {H K}{\gamma}}{- \log (1 - \lambda_ {0} ^ {2})} \right] \sqrt {S A H K}, \\ \end{array}
$$
where inequality (i) follows since  $N_h^k$  will only be updated when  $h \in \mathcal{K}^{\mathrm{eff}}(h)$  and then repeat at most  $\left\lceil \frac{\log \frac{HK}{\gamma}}{-\log(1 - \lambda_0^2)} \right\rceil$  times, inequality (ii) invokes the cardinality bound of  $\mathcal{K}^{\mathrm{eff}}(h)$ , and inequality (iii) follows from the standard pigeon-hole principle.
Lemma C.3. Let  $\{u_i\}_{i=1}^k$  be i.i.d. Bernoulli random variables. Suppose  $\mathbb{P}(u_i = 1) = \theta$ . Define the largest gap between  $u_i$ 's as
$$
g (k) = \sup  \left\{j - i: u _ {i} = 0 \text {a n d} u _ {j} = 0 \text {w i t h} u _ {\ell} = 1 \text {f o r} \ell = i + 1, \dots , j - 1 \right\}.
$$
Then for any integer  $q > 0$ , the following tail probability bound holds
$$
\mathbb {P} (g (k) > q) \leq k \theta^ {q + 1}.
$$
Proof of Lemma C.3. We denote  $I_{\mathrm{neg}} = \{\ell_1, \dots, \ell_m\}$  as the index set for  $u_{\ell_i} = 0$  when  $i = 1, \dots, |I_{\mathrm{neg}}|$ . Let  $v_j = \ell_{j+1} - \ell_j$ , which is a geometric random variable with a success rate  $\theta$ . Note that the cardinality of  $I_{\mathrm{neg}}$  is at most  $k$ . Therefore, we have
$$
\begin{array}{l} \mathbb{P}(g(k) > q)\leq \mathbb{P}(\max_{j = 1,\ldots ,k}v_{j} > q) \\ = 1 - \mathbb {P} (v _ {j} \leq q \text {f o r} j = 1, \dots , k) \\ = 1 - \left(1 - \theta^ {q + 1}\right) ^ {k} \\ \leq k \theta^ {q + 1}, \\ \end{array}
$$
where the last inequality follows from  $1 - k\theta^{q + 1}\leq (1 - \theta^{q + 1})^k$
D Helper concentration inequalities
Lemma D.1 (Bernstein's inequality). Let  $x_{1},\ldots ,x_{n}$  be i.i.d. zero mean random variables. Suppose  $|x_{i}|\leq M$  for any  $i = 1,\dots ,n$ . Then for all positive  $t$ , it holds
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} x _ {i} > t\right) \leq \exp \left(- \frac {\frac {1}{2} t ^ {2}}{\sum_ {i = 1} ^ {n} \operatorname {V a r} [ x _ {i} ] + \frac {1}{3} M t}\right).
$$
In particular, given a failure probability  $\gamma < 1$ , it holds
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} x _ {i} > \sqrt {\sum_ {i = 1} ^ {n} \operatorname {V a r} \left[ x _ {i} \right] \log \frac {1}{\gamma}} + M \log \frac {1}{\gamma}\right) \leq \gamma .
$$
Proof of Lemma D.1. The proof of Bernstein's inequality is standard, see for example [Wainwright, 2019, Section 2.1]. Here we verify the second claim. Let  $\exp \left(-\frac{\frac{1}{2}t^2}{\sum_{i=1}^{n}\operatorname{Var}[x_i] + \frac{1}{3}Mt}\right) \leq \gamma$  hold true. We find a suitable  $t$  by
$$
\begin{array}{l} \exp \left(- \frac {\frac {1}{2} t ^ {2}}{\sum_ {i = 1} ^ {n} \operatorname {V a r} [ x _ {i} ] + \frac {1}{3} M t}\right) \leq \gamma \\ \Longleftrightarrow \frac {\frac {1}{2} t ^ {2}}{\sum_ {i = 1} ^ {n} \operatorname {V a r} [ x _ {i} ] + \frac {1}{3} M t} \geq \log \frac {1}{\gamma} \\ \Longleftrightarrow t ^ {2} - \frac {2}{3} t M \log \frac {1}{\gamma} \geq \sum_ {i = 1} ^ {n} \operatorname {V a r} \left[ x _ {i} \right] \log \frac {1}{\gamma} \\ \Longleftrightarrow t \geq \sqrt {\sum_ {i = 1} ^ {n} \operatorname {V a r} \left[ x _ {i} \right] \log \frac {1}{\gamma} + \frac {1}{9} M ^ {2} \log^ {2} \frac {1}{\gamma}} + \frac {1}{3} M \log \frac {1}{\gamma}. \\ \end{array}
$$
It is enough to choose  $t = \sqrt{\sum_{i=1}^{n} \operatorname{Var}[x_i] \log \frac{1}{\gamma}} + M \log \frac{1}{\gamma}$ .
Lemma D.2 (Hoeffding's inequality). Let  $x_{1},\ldots ,x_{n}$  be i.i.d. random variables. Suppose  $a_{i}\leq x_{i}\leq b_{i}$  for any  $i = 1,\dots ,n$ . Then for all positive  $t$ , it holds
$$
\mathbb {P} \left(\left| \sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} \left[ \sum_ {i = 1} ^ {n} x _ {i} \right] \right| > t\right) \leq 2 \exp \left(- \frac {2 t ^ {2}}{\sum_ {i = 1} ^ {n} (b _ {i} - a _ {i}) ^ {2}}\right).
$$
$$
\mathbb {P} \left(\frac {1}{n} \left| \sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} \left[ \sum_ {i = 1} ^ {n} x _ {i} \right] \right| > \sqrt {\frac {\sum_ {i = 1} ^ {n} (b _ {i} - a _ {i}) ^ {2} \log \frac {2}{\gamma}}{2 n ^ {2}}}\right) \leq \gamma .
$$
Proof of Lemma D.2. The proof is standard; see [Wainwright, 2019, Section 2.1].
Lemma D.3 (Azuma-Hoeffding's inequality). Let  $x_{1}, \ldots, x_{n}$  be a martingale adapted to filtration  $\mathcal{F}_1 \subset \dots \subset \mathcal{F}_n$ . Suppose  $\mathbb{E}[x_i - \mathbb{E}[x_i] | \mathcal{F}_{i-1}] = 0$  and  $|x_i - \mathbb{E}[x_i]| \leq c_i$ . Then for all positive  $t$ , it holds
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} [ x _ {i} ] > t\right) \leq \exp \left(- \frac {t ^ {2}}{2 \sum_ {i = 1} ^ {n} c _ {i} ^ {2}}\right).
$$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} [ x _ {i} ] > \sqrt {2 \sum_ {i = 1} ^ {n} c _ {i} ^ {2} \log \frac {1}{\gamma}}\right) \leq \gamma .
$$
Proof of Lemma D.3. The proof is standard and applies Lemma D.2.