SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second event we define,  $L_{e}$ , represents the event that the average "proxy" loss  $\widetilde{\ell}$  is much larger than 1. Note that since  $\mathbb{E}[\widetilde{\ell}_{tck}] = \frac{f_{ek}}{f_{ek} + \gamma}\ell_{tck} \leq 1$ , we expect this not to be the case.
Definition 2.  $L_{e}$  is the event such that
$$
\max  _ {c \in [ C ], k \in [ K ]} \sum_ {t \in \mathcal {T} _ {e}} \widetilde {\ell} _ {t c k} \leq L + \frac {\iota}{\gamma}.
$$
We present the two concentration arguments below in Lemma 6 and Lemma 7.
Lemma 6. For any  $e \in [T / L]$ ,  $e > 1$ , event  $F_{e}$  holds with probability at least  $1 - 2K\exp(-\iota)$ .
Proof. Fix a  $k \in [K]$ . Consider a random variable  $X$  defined via  $X = s_{e,c,k}$  where  $c \sim \nu$  (so  $\mathbb{E}[X] = f_{ek}$ ). Note that  $\widehat{f}_{ek}$  is distributed according to  $\sum_{i=1}^{L/2} \frac{X_i}{L/2}$ , where the  $X_i$  are i.i.d. copies of  $X$ . Then  $\sum_{i=1}^{L/2} \mathbb{E}[(X_i - f_{ek})^2] \leq \frac{L}{2} \cdot \frac{f_{ek}}{2}$ , since  $(X_i - f_{ek}) \in [-1/2, 1/2]$ .
Now, for any  $\zeta \leq 2$ , we have by Lemma 3 with probability at least  $1 - \exp(-\iota)$
$$
\sum_ {i = 1} ^ {L / 2} \frac {X _ {i}}{L / 2} - f _ {e k} <   \frac {\zeta f _ {e k}}{2} + \frac {2 \iota}{\zeta L}.
$$
Set  $\zeta = \min \left\{2, 2\sqrt{\frac{\iota}{f_{ek}L}}\right\}$ , which shows that with probability at least  $1 - \exp(-\iota)$
$$
\widehat {f} _ {e k} - f _ {e k} \leq 2 \max  \left\{\sqrt {\frac {f _ {e k} \iota}{L}}, \frac {\iota}{L} \right\}.
$$
Repeating the same argument for  $\sum_{i} - X_{i}$  and taking a union bound completes the proof.
Lemma 7. For any  $e \in [T / L]$ ,  $e > 1$ , event  $L_{e}$  holds with probability at least  $1 - K \exp(-\iota)$ .
$$
\max  _ {c \in [ C ]} \sum_ {t \in T _ {e}} \widetilde {\ell} _ {t c k} \leq \sum_ {t \in \mathcal {T} _ {e}} \frac {2 \mathbb {I} \left(A _ {t} = k , S _ {t} = 1 , t \in \mathcal {T} _ {e} ^ {\ell}\right) / f _ {e k}}{1 + \gamma / f _ {e k}}.
$$
The random variable  $Z_{tk} = \frac{2\mathbb{I}\left(A_t = k,S_t = 1,t\in\mathcal{T}_e^\ell\right)}{f_{ek}}$  satisfies the conditions of Lemma 2 and  $E[Z_{tk}|\mathcal{F}_{t - 1}] = 1$ . Setting  $\alpha_{tk} = \gamma$  and  $\lambda_{tk} = \frac{\gamma}{f_{ek}}$ , Lemma 2 implies that with probability  $1 - \exp (-\iota)$ , we have
$$
\sum_ {t \in \mathcal {T} _ {e}} \widetilde {\ell} _ {t} \leq L + \frac {\iota}{\gamma}.
$$
Taking a union bound over  $k \in [K]$  completes the proof.
We will want to condition on the event that all the events  $F_{e}$  and  $L_{e}$  hold. To do so, we introduce a combined indicator variable  $G$ .
Definition 3. We define the indicator that all concentrations  $F_{e}$  and  $L_{e}$  hold by
$$
G = \Pi_ {e = 1} ^ {T / L} F _ {e} L _ {e}.
$$
Note that by Lemma 6 and Lemma 7, the event  $G$  occurs with probability at least  $1 - 3K(T / L)\exp (-\iota)$ . Since we will eventually take  $\iota \geq 2\log (KT)$ , the probability that  $G = 0$  will be negligible.
C.2 Implications of  $G$
We now explore some of the implications of conditioning on all of our concentration bounds holding. We start by showing that this allows us to bound the range of  $p_{tck}$  and  $\widetilde{p}_{tck}$ , and as a consequence, show that  $q_{t} = p_{t}$  for all rounds  $t$ . To do so, it will be helpful to first use our concentration of  $\widehat{f}$  (i.e., the event  $F_{e}$ ) to bound the range of  $\beta_{ek} = (f_{ek} + \gamma) / (\widehat{f}_{ek} + \frac{3}{2}\gamma)$ .
Lemma 8. Let  $\gamma \geq \frac{4\iota}{L}$ , then under event  $G$ , we have that
$$
\frac {1}{2} \leq \beta_ {e k} \leq 2
$$
$$
| 1 - \beta_ {e k} | \leq 3 \sqrt {\frac {\imath}{f _ {e k} L}} + \frac {\gamma \sqrt {L}}{4 \sqrt {\imath f _ {e k}}}
$$
for all  $t\in \mathcal{T}_e,k\in [K]$  simultaneously.
Proof. The first statement is equivalent to showing that  $\frac{1}{\beta_{ek}} - 1 \in [-\frac{1}{2}, 1]$ . Using the facts that  $2 \max \left\{\sqrt{\frac{f_{ek} \iota}{L}}, \frac{2\iota}{L}\right\} \leq \frac{f}{2} + \frac{2\iota}{L}$ ,  $F_e = 1$  and  $\gamma \geq \frac{4\iota}{L}$ , we have
$$
\frac {1}{\beta_ {e k}} - 1 = \frac {\widehat {f} _ {e k} - f _ {e k} + \frac {1}{2} \gamma}{f + \gamma} \geq \frac {- \frac {1}{2} f - \frac {2 \iota}{L} + \frac {1}{2} \gamma}{f + \gamma} \geq - \frac {1}{2}
$$
$$
\frac {1}{\beta_ {e k}} - 1 = \frac {\widehat {f} _ {e k} - f _ {e k} + \frac {1}{2} \gamma}{f + \gamma} \leq \frac {f + \frac {2 \iota}{L} + \frac {1}{2} \gamma}{f + \gamma} \leq 1,
$$
which proves that  $\beta_{ek} \in [1/2, 2]$ . If  $f_{ek} \leq \frac{\iota}{L}$ , then the second condition follows directly from the first. Otherwise we have
$$
\beta_ {e k} - 1 \leq \frac {2 \sqrt {\frac {f _ {e k} \iota}{L}} - \frac {1}{2} \gamma}{f _ {e k} - 2 \sqrt {\frac {f _ {e k} \iota}{L}} + \frac {3}{2} \gamma} \leq \frac {2 \sqrt {\frac {f _ {e k} \iota}{L}}}{f _ {e k} - 2 \sqrt {\frac {f _ {e k} \iota}{L}} + 2 \gamma} \leq \frac {4 \sqrt {\frac {f _ {e k} \iota}{L}}}{\frac {7}{8} f _ {e k}} \leq 3 \sqrt {\frac {\iota}{f _ {e k} L}}
$$
$$
1 - \beta_ {e k} \leq \frac {2 \sqrt {\frac {f _ {e k} \iota}{L}} + \frac {1}{2} \gamma}{f _ {e k} + 2 \sqrt {\frac {f _ {e k} \iota}{L}} + \frac {3}{2} \gamma} \leq \sqrt {\frac {\iota}{f _ {e k} L}} + \frac {\gamma \sqrt {L}}{4 \sqrt {\iota f _ {e k}}}.
$$
We now apply Lemma 8 to bound the range of  $x$  and  $\widetilde{x}$ .
Lemma 9. If  $\gamma \geq \frac{4L}{L}$  and  $\eta \leq \frac{\log(2)}{5L}$ , then under event  $G$ , we have for all  $t \in T_e, k \in [K], c \in [C]$  simultaneously
$$
2 s _ {e c k} \geq p _ {t c k} \geq s _ {e c k} / 2 \quad a n d \quad 2 s _ {e c k} \geq \widetilde {p} _ {t c k} \geq s _ {e c k} / 2.
$$
$$
\mathbb {E} _ {c \sim \nu} [ p _ {t c k} ] \leq 4 f _ {e k} \quad a n d \quad \mathbb {E} _ {c \sim \nu} [ \widetilde {p} _ {t c k} ] \leq 4 f _ {e k}.
$$
In addition, this implies that  $q_{t} = p_{t}$  for all  $t \in \mathcal{T}_e$ .
Proof. We have that  $p_{tck} \propto \exp(-\eta(\sum_{t' \in \mathcal{T}_{e-1} \cup \mathcal{T}_e, t' < t} \widehat{\ell}_{t'ck})) s_{\text{sech}}$ , and  $\widetilde{p}_{tck} \propto \exp(-\eta(\sum_{t' \in \mathcal{T}_{e-1}} \widehat{\ell}_{t'ck} + \sum_{t' \in \mathcal{T}_e, t' < t} \widetilde{\ell}_{t'c})) s_{\text{sech}}$ . By definition,  $G = 1$  implies that  $L_{e-1} = L_e = 1$ . Hence for any  $k \in [K], c \in [C]$ ,
$$
\sum_ {t ^ {\prime} \in \mathcal {T} _ {e - 1} \cup \mathcal {T} _ {e}} \widetilde {\ell} _ {t c k} \leq 2 (L + \frac {\iota}{\gamma}) \leq \frac {5}{2} L.
$$
Furthermore, by Lemma 8, we can bound sums of  $\widehat{\ell}$  via
$$
\sum_ {t ^ {\prime} \in \mathcal {T} _ {e - 1} \cup \mathcal {T} _ {e}} \widehat {\ell} _ {t c k} = \sum_ {t ^ {\prime} \in \mathcal {T} _ {e - 1}} \beta_ {e - 1, k} \widetilde {\ell} _ {t c k} + \sum_ {t ^ {\prime} \in \mathcal {T} _ {e}} \beta_ {e k} \widetilde {\ell} _ {t c k} \leq 5 L
$$
$$
\sum_ {t ^ {\prime} \in \mathcal {T} _ {e - 1}} \widehat {\ell} _ {t c k} + \sum_ {t ^ {\prime} \in \mathcal {T} _ {e}} \widetilde {\ell} _ {t c k} \leq 5 L.
$$
Finally, this implies for any  $k$  that
$$
\exp (- 5 \eta L) s _ {e k} \leq p _ {t c k} \leq \exp (5 \eta L) s _ {e k}
$$
and for  $\widetilde{p}_{tck}$  accordingly. Using  $\eta \leq \frac{\log(2)}{5L}$  completes the first part of the proof. The second statement follows directly from the definition of  $f_{ek}$ . Finally, the last statement follows from the definition of  $q_{t}$  since  $q_{tc} = p_{tc}$  whenever  $p_{tck} \geq s_{eck} / 2$  for all  $k \in [K]$ .
We now bound two additional quantities. We start with proving an upper bound on  $\mathbb{E}_e[1 - \beta_{ek}]$ . This is helpful as it is a quantity which naturally appears as we try to bound the bias $_2$  term; in particular, since  $\mathbb{E}_e[\widetilde{\ell}_{t,c_t,k} - \widehat{\ell}_{t,c_t,k}] = \mathbb{E}_e[(1 - \beta_{ek})\widetilde{\ell}_{t,c_t,k}] = \mathbb{E}_e[1 - \beta_{ek}]\mathbb{E}_e[\widetilde{\ell}_{t,c,k}]$  (with the last inequality holding since  $\beta_{ek}$  is independent from  $\widetilde{\ell}_{t,c,k}$  conditioned on  $s_e$ ).
Lemma 10. If  $\gamma \geq \frac{16\iota}{L}$  and  $\exp (-\iota)\leq \frac{\gamma}{8K}$ , then
$$
0 \leq \mathbb {E} _ {e} \left[ (1 - \beta_ {e k}) F _ {e} \right] \leq \frac {\gamma}{f _ {e k}}
$$
Proof. Since  $F_{e}\in \{0,1\}$ , we have
$$
(1 - \beta_ {e k}) F _ {e} = \left(1 - \frac {f _ {e k} + \gamma}{\widehat {f} _ {e k} + \frac {3}{2} \gamma}\right) F _ {e} = \frac {(\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {1}{2} \gamma}{f _ {e k} + (\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {3}{2} \gamma} - (1 - F _ {e}) \frac {\gamma}{2 f _ {e k} + 3 \gamma}.
$$
The second term is in expectation bounded by
$$
\mathbb {E} _ {e} \left[ (1 - F _ {e}) \frac {\gamma}{2 f _ {e k} + 3 \gamma} \right] \leq 2 K \exp (- \iota) \frac {\gamma}{2 f _ {e k} + 3 \gamma} \leq \frac {\gamma^ {2}}{4 (2 f _ {e k} + 3 \gamma)}.
$$
It remains to bound the first term. Denote  $C_{ek} = 2\max \{\sqrt{\frac{f_{ek}\iota}{L}},\frac{\iota}{L}\}$ . The random variable  $(\widehat{f}_{ek} - f_{ek})F_{e}$  is bounded in  $\{-C_{ek},C_{ek}\}$  due to the indicator  $F_{e}$ . Let  $\mu = \mathbb{E}_e[(\widehat{f}_{ek} - f_{ek})F_e]$  and assume for now that  $\mu \in [-\frac{1}{4}\gamma ,\frac{1}{4}\gamma ]$  which we show later. By Jensen's inequality we have
$$
\mathbb {E} _ {e} \left[ \frac {(\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {1}{2} \gamma}{f _ {e k} + (\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {3}{2} \gamma} \right] \leq \frac {\mu + \frac {1}{2} \gamma}{f _ {e k} + \mu + \frac {3}{2} \gamma} \leq \frac {\gamma}{f _ {e k}}.
$$
On the other hand, again due to convexity, the smallest expected value is obtained for the distribution that takes values in  $\{-C_{ek}, C_{ek}\}$  such that it conforms with mean  $\mu$ . Hence
$$
\begin{array}{l} \mathbb {E} _ {e} \left[ \frac {(\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {1}{2} \gamma}{f _ {e k} + (\widehat {f} _ {e k} - f _ {e k}) F _ {e} + \frac {3}{2} \gamma} \right] \geq \left(\frac {C _ {e k} + \mu}{2 C _ {e k}}\right) \frac {C _ {e k} + \frac {1}{2} \gamma}{f _ {e k} + C _ {e k} + \frac {3}{2} \gamma} + \left(\frac {C _ {e k} - \mu}{2 C _ {e k}}\right) \frac {- C _ {e k} + \frac {1}{2} \gamma}{f _ {e k} - C _ {e k} + \frac {3}{2} \gamma}. \\ = \frac {(f _ {e k} + \frac {3}{2} \gamma) (\gamma + 2 \mu) - (\gamma \mu + 2 C _ {e k} ^ {2})}{2 (f _ {e k} + C _ {e k} + \frac {3}{2} \gamma) (f _ {e k} - C _ {e k} + \frac {3}{2} \gamma)} \\ \geq \frac {\frac {1}{2} \gamma f _ {e k} + \gamma^ {2} - 2 C _ {e k} ^ {2}}{2 (f _ {e k} + \frac {3}{2} \gamma) ^ {2} - 2 C _ {e k} ^ {2}} \geq \frac {\frac {1}{2} \gamma^ {2}}{2 (f _ {e k} + \frac {3}{2}) ^ {2}}, \\ \end{array}
$$
where the last inequality uses the fact that  $\frac{1}{2}\gamma f_{ek} + \frac{1}{2}\gamma^2 \geq 2C_{ek}^2$ , since  $\gamma \geq \frac{16\iota}{L}$ .
Finally we need to verify  $\mu \in [-\frac{\gamma}{2}, \frac{\gamma}{2}]$ . We begin by bounding the expectation of  $\mathbb{E}_e\left[(\widehat{f}_{ek} - f_{ek})(1 - F_e)\right]$ . By construction  $\widehat{f}_{ek} - f_{ek} \in [-\frac{1}{2}, \frac{1}{2}]$  and  $E_{e-1}[(1 - F_e)] \leq 2K\exp(-\iota) \leq \gamma/2$ . Hence due to  $\mathbb{E}_e[\widehat{f}_{ek} - f_{ek}] = 0$ , we have  $\mu = -\mathbb{E}_e\left[(\widehat{f}_{ek} - f_{ek})(1 - F_e)\right] \in [-\frac{\gamma}{4}, \frac{\gamma}{4}]$ .
The last quantity we would like to bound is  $\sum_{k}|\widetilde{p}_{tck} - p_{tck}|\cdot |1 - \beta_{ek}|$  (simultaneously for all  $c\in \mathcal{C}$ ). This is a quantity which arises when handling  $\mathbf{bias}_3$ . We begin by proving the following auxiliary lemma bounding the coordinate-wise change in  $x$  under a multiplicative weights update.
Lemma 11. Let  $z\in [-\frac{1}{2},\frac{1}{2}]^{K}$ ,  $x\in \Delta ([K])$  and  $\widetilde{x}\propto x\circ \exp (z)$ . Then for all  $k\in [K]$ :
$$
x _ {k} \exp (z _ {k} - 2 \langle x, | z | \rangle) \leq \widetilde {x} _ {k} \leq x _ {k} \exp (z _ {k} + \langle x, | z | \rangle).
$$
$$
\widetilde {x} _ {k} = x _ {k} \exp (z _ {k}) \exp (- \log (\sum_ {k ^ {\prime} = 1} ^ {K} x _ {k ^ {\prime}} \exp (z _ {k ^ {\prime}}))).
$$
We only need to bound the last factor. Note that by Jensen's inequality, we have
$$
\sum_ {k ^ {\prime} = 1} ^ {K} x _ {k ^ {\prime}} \exp \left(z _ {k ^ {\prime}}\right) \geq \exp \left(\langle x, z \rangle\right),
$$
$$
\exp \left(- \log \left(\sum_ {k ^ {\prime} = 1} ^ {K} x _ {k ^ {\prime}} \exp \left(z _ {k ^ {\prime}}\right)\right)\right) \leq \exp \left(- \langle x, z \rangle\right)
$$
In the other direction, we have by  $|z|\leq \frac{1}{2}$
$$
\sum_ {k ^ {\prime} = 1} ^ {K} x _ {k ^ {\prime}} \exp \left(z _ {k ^ {\prime}}\right) \leq 1 + \langle x, z \rangle + \left\langle x, z ^ {2} \right\rangle ,
$$
$$
- \log (1 + \left\langle x, z \right\rangle + \left\langle x, z ^ {2} \right\rangle) \geq - \left\langle x, z \right\rangle - \left\langle x, z ^ {2} \right\rangle .
$$
Lemma 12. Assume  $\eta \leq \frac{\gamma}{2(2L\gamma + \iota)}$ . Then under event  $G$ , for all  $t \in T_e$  and  $c \in \mathcal{C}$ , we have that
$$
\sum_ {k = 1} ^ {K} \left| \widetilde {p} _ {t, c, k} - p _ {t, c, k} \right| \cdot | 1 - \beta_ {e k} | \leq 3 \sum_ {k = 1} ^ {K} \widetilde {p} _ {t, c, k} \left(1 - \beta_ {e k}\right) ^ {2}.
$$
Proof. By definition,  $p_{tck} \propto \widetilde{p}_{tck} \exp(-\eta \sum_{t' \in T_e, t' < t} (\widehat{\ell}_{tck} - \widetilde{\ell}_{tck}))$ . We can simplify the argument of the exponential via
$$
- \sum_ {t ^ {\prime} \in T _ {e}, t ^ {\prime} <   t} \left(\widehat {\ell} _ {t ^ {\prime} c k} - \widetilde {\ell} _ {t ^ {\prime} c k}\right)) = \left(1 - \beta_ {e k}\right) \sum_ {t ^ {\prime} \in T _ {e}, t ^ {\prime} <   t} \widetilde {\ell} _ {t ^ {\prime} c k}.
$$
By Lemma 8 and the definition of  $L_{e}$ , we have
$$
\eta \left| \sum_ {t ^ {\prime} \in T _ {e}, t ^ {\prime} <   t} (\widehat {\ell} _ {t ^ {\prime} c k} - \widetilde {\ell} _ {t ^ {\prime} c k}) \right| = \eta (1 - \beta_ {e k}) \sum_ {t ^ {\prime} \in T _ {e}, t ^ {\prime} <   t} \widetilde {\ell} _ {t ^ {\prime} c k} \leq \eta (L + \frac {\iota}{2 \gamma}) \leq \frac {1}{2}.
$$
Lemma 11 now implies that
$$
\begin{array}{l} \left| p _ {t c k} - \widetilde {p} _ {t c k} \right| \leq e \widetilde {p} _ {t c k} \eta \left(| 1 - \beta_ {e k} | + \sum_ {i = 1} ^ {K} \widetilde {p} _ {t c i} | 1 - \beta_ {i k} |\right) \left(L + \frac {2 \iota}{\gamma}\right) \\ \leq \frac {e}{2} \widetilde {p} _ {t c k} \left(| 1 - \beta_ {e k} | + \sum_ {i = 1} ^ {K} \widetilde {p} _ {t c i} | 1 - \beta_ {i k} |\right). \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \left| \widetilde {p} _ {t, c, k} - p _ {t, c, k} \right| \cdot | 1 - \beta_ {e k} | \leq \frac {e}{2} \left(\sum_ {k = 1} ^ {K} \widetilde {p} _ {t c k} (1 - \beta_ {e k}) ^ {2} + \left(\sum_ {k = 1} ^ {K} \widetilde {p} _ {t c k} (1 - \beta_ {e k})\right) ^ {2}\right) \\ \leq e \sum_ {k = 1} ^ {K} \widetilde {p} _ {t c k} \left(1 - \beta_ {e k}\right) ^ {2}. \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ \end{array}
$$
C.3 Combining the pieces
Proof. We begin by decomposing the regret similarly to the high level overview (but explicitly showing the dependence on the indicator  $G$ ).
$$
\begin{array}{l} \operatorname {R e g} (u) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle q _ {t, c _ {t}} - u _ {c _ {t}}, \ell_ {t, c _ {t}} \rangle \right] \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t, c _ {t}} - u _ {c _ {t}}, \ell_ {t, c _ {t}} \right\rangle G \right] + \mathbb {E} [ 1 - G ] K T + L \\ \leq \underbrace {\mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t , c _ {t}} - u _ {c _ {t}} , \ell_ {t , c _ {t}} - \widetilde {\ell} _ {t , c _ {t}} \right\rangle G \right]} _ {\text {b i a s} _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle \widetilde {p} _ {t , c _ {t}} - u _ {c _ {t}} , \widetilde {\ell} _ {t , c _ {t}} - \widehat {\ell} _ {t , c _ {t}} \right\rangle G \right]} _ {\text {b i a s} _ {2}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t , c _ {t}} - \widetilde {p} _ {t , c _ {t}}, \widetilde {\ell} _ {t , c _ {t}} - \widehat {\ell} _ {t , c _ {t}} \right\rangle G \right]} _ {\text {b i a s} _ {3}} + \underbrace {\mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t , c _ {t}} - u _ {c _ {t}} , \widehat {\ell} _ {t , c _ {t}} \right\rangle G \right]} _ {\text {f t r l}} \\ + \Pr [ G = 0 ] K T + L. \\ \end{array}
$$
Note that the first inequality follows in part from Lemma 9, since when  $G = 1$ ,  $p_{t,c_t} = q_{t,c_t}$ . We now bound the remaining four terms individually. We start with  $\mathbf{bias}_1$ . Since  $\widetilde{\ell}_t$  is conditionally independent of  $p_t$ , by the tower rule of expectation
$$
\begin{array}{l} \mathbf {b i a s} _ {1} = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t, c _ {t}} - u _ {c _ {t}}, \ell_ {t, c _ {t}} - \mathbb {E} _ {e} [ \widetilde {\ell} _ {t, c _ {t}} ] \right\rangle G \right] \\ = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k = 1} ^ {K} \left(p _ {t, c _ {t}, k} - u _ {c _ {t}, k}\right) \frac {\gamma \ell_ {t , c _ {t} , k}}{f _ {e k} + \gamma} G \right] \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {c = 1} ^ {C} \sum_ {k = 1} ^ {K} \nu_ {c} \frac {p _ {t , c , k} \gamma}{f _ {e , k} + \gamma} G \right] \quad (0 \leq \ell_ {t} \leq 1) \\ \leq 4 K \gamma T. \tag {Lemma9} \\ \end{array}
$$
Similarly, in  $\mathbf{bias}_2$ ,  $\widetilde{\ell}_t$  and  $\widehat{\ell}_t$  are independent of  $\widetilde{p}_t$  conditioned on episode  $e$ . We thus have
$$
\begin{array}{l} \mathbf {b i a s} _ {2} = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle \widetilde {p} _ {t, c _ {t}} - u _ {c _ {t}}, \mathbb {E} _ {e} [ \widetilde {\ell} _ {t, c _ {t}} - \widehat {\ell} _ {t, c _ {t}} ] \right\rangle G \right] \\ = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k = 1} ^ {K} \left(\widetilde {p} _ {t, c _ {t}, k} - u _ {c _ {t}, k}\right) \left(\mathbb {E} _ {e} \left[ (1 - \beta_ {e k}) F _ {e} \right] \mathbb {E} _ {e} \left[ \widetilde {\ell} _ {t, c _ {t}, k} \right]\right) G \right] \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {c = 1} ^ {C} \sum_ {k = 1} ^ {K} \nu_ {c} \frac {\widetilde {p} _ {t , c , k} \gamma}{f _ {e , k}} G \right] \quad (\text {L e m m a 1 0 a n d} \mathbf {0} \leq \mathbb {E} _ {e} [ \widetilde {\ell_ {t}} ] \leq \mathbf {1}) \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k = 1} ^ {K} \frac {2 \sum_ {c = 1} ^ {C} \nu_ {c} s _ {e , c , k} \gamma}{f _ {e , k}} G \right] \tag {Lemma9} \\ \leq 4 K \gamma T, \\ \end{array}
$$
Finally, the last term needs to be bounded differently, because  $p_t$  is not independent of  $\beta_e$ . Instead, we will expand out the inner product and directly bound the maximum value of  $\sum_k (p_{tck} - \widetilde{p}_{tck})(1 - \beta_{ek})$
over all  $c$  via Lemma 12.
$$
\begin{array}{l} \mathbf {b i a s} _ {3} = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t, c _ {t}} - \widetilde {p} _ {t, c _ {t}}, \mathbb {E} _ {e} \left[ \widetilde {\ell} _ {t, c _ {t}} - \widehat {\ell} _ {t, c} \right] \right\rangle G \right] \\ = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k = 1} ^ {K} \left(p _ {t, c _ {t}, k} - \widetilde {p} _ {t, c _ {t}, k}\right) \left(1 - \beta_ {e k}\right) \mathbb {E} _ {e} \left[ \widetilde {\ell} _ {t, c _ {t}, k} \right] G \right] \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k = 1} ^ {K} | p _ {t, c _ {t}, k} - \widetilde {p} _ {t, c _ {t}, k} | \cdot | 1 - \beta_ {e k} | G \right] \\ \leq \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {c = 1} ^ {C} \sum_ {k = 1} ^ {K} 3 \nu_ {c} \widetilde {p} _ {t, c, k} (1 - \beta_ {e k}) ^ {2} G \right] \tag {Lemma 12} \\ = \frac {9 8 K T \iota}{L} + \frac {\gamma^ {2} L K T}{\iota} \quad \text {(L e m m a 8 a n d L e m m a 9)} \\ \end{array}
$$
Finally the ftrl term is bounded according to Lemma 4
$$
\begin{array}{l} \mathbf {f t r l} = \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \left\langle p _ {t, c _ {t}} - u _ {c _ {t}}, \widehat {\ell} _ {t, c _ {t}} \right\rangle \right] \\ \leq \frac {\log (K)}{\eta} + \eta \mathbb {E} \left[ \sum_ {c \in [ C ]} \nu_ {c} \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k \in [ K ]} p _ {t c k} \widehat {\ell} _ {t c k} ^ {2} G \right] (Lemma 4) \\ \leq \frac {\log (K)}{\eta} + \eta \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in T _ {e} ^ {\prime}} \sum_ {k \in [ K ]} \sum_ {c \in [ C ]} \nu_ {e} p _ {t c k} \frac {f _ {e k}}{(\widehat {f} _ {e k} + \frac {3}{2} \gamma) ^ {2}} G \right] \\ \leq \frac {\log (K)}{\eta} + 4 \eta \mathbb {E} \left[ \sum_ {e = 2} ^ {T / L} \sum_ {t \in \mathcal {T} _ {e}} \sum_ {k \in [ K ]} \beta_ {e k} ^ {2} G \right] (Lemma 9) \\ \leq \frac {\log (K)}{\eta} + 1 6 \eta K T. (Lemma 8) \\ \end{array}
$$
Combining everything, we have that
$$
\mathrm {R e g} (u) = O \left(\left(\gamma + \frac {\iota}{L} + \frac {\gamma^ {2} L}{\iota} + \eta\right) K T + \frac {\log (K)}{\eta} + L\right).
$$