SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {c o s t} (\hat {\mathcal {L}}, \hat {\pi}) - \operatorname {c o s t} (\hat {\mathcal {L}}, \pi^ {\star}) = \sum_ {q \in \mathcal {Q}} P (q) \mathbb {1} (q \notin \hat {\mathcal {L}}) (c _ {s} ^ {\star} (q) \hat {\pi} (q) + c _ {l} ^ {\star} (q) (1 - \hat {\pi} (q)) - c _ {s} ^ {\star} (q) \pi^ {\star} (q) - c _ {l} ^ {\star} (q) (1 - \pi^ {\star} (q))) \\ = \sum_ {q \in \mathcal {Q}} P (q) \mathbb {1} (q \notin \hat {\mathcal {L}}) \left(c _ {s} ^ {\star} (q) \hat {\pi} (q) + c _ {l} ^ {\star} (q) (1 - \hat {\pi} (q)) - \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right)\right) \\ \leq \sum_ {q \in \mathcal {Q}} P (q) \left(c _ {s} ^ {\star} (q) \hat {\pi} (q) + c _ {l} ^ {\star} (q) \left(1 - \hat {\pi} (q)\right) - \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right)\right) \\ = \sum_ {q \in \mathcal {Q}} P (q) \left(c _ {s} ^ {\star} (q) \mathbb {1} (\hat {c} _ {s} (q) \leq \hat {c} _ {l} (q)) + c _ {l} ^ {\star} (q) \mathbb {1} (\hat {c} _ {s} (q) > \hat {c} _ {l} (q)) - \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right)\right). \\ \end{array}
$$
Note that if  $\hat{c}_s(q) - \hat{c}_l(q)$  has the same sign as  $c_s^\star (q) - c_l^\star (q)$ , the difference  $c_{s}^{\star}(q)\mathbb{1}(\hat{c}_{s}(q)\leq$ $\hat{c}_l(q)) + c_l^\star (q)\mathbb{1}(\hat{c}_s(q) > \hat{c}_l(q)) - \min (c_s^\star (q),c_l^\star (q))$  becomes 0. Otherwise, if  $c_{s}^{\star}(q) - c_{l}^{\star}(q) > 0$ , we know that
$$
c _ {s} ^ {\star} (q) - c _ {l} ^ {\star} (q) \leq \hat {c} _ {s} (q) - \hat {c} _ {l} (q) + | \hat {c} _ {s} (q) - c _ {s} ^ {\star} (q) | + | \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) | \leq | \hat {c} _ {s} (q) - c _ {s} ^ {\star} (q) | + | \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) |.
$$
And similarly if  $c_s^\star (q) - c_l^\star (q)\leq 0$  , we know that  $c_{l}^{\star}(q) - c_{s}^{\star}(q)\leq |\hat{c}_{s}(q) - c_{s}^{\star}(q)| + |\hat{c}_{l}(q) - c_{l}^{\star}(q)|.$  Overall, we have
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {c o s t} \left(\hat {\mathcal {L}}, \hat {\pi}\right) - \operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) \right] \leq \mathbb {E} \left[ \sum_ {q \in \mathcal {Q}} P (q) \left| \hat {c} _ {s} (q) - c _ {s} ^ {\star} (q) \right| + \left| \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) \right| \right] \\ \stackrel {(i)} {\leq} \mathbb {E} \left[ \sqrt {\sum_ {q \in \mathcal {Q}} P (q) (\hat {c} _ {s} (q) - c _ {s} ^ {\star} (q)) ^ {2}} + \sqrt {\sum_ {q \in \mathcal {Q}} P (q) (\hat {c} _ {l} (q) - c _ {l} ^ {\star} (q)) ^ {2}} \right] \\ \stackrel {(i i)} {\leq} \sqrt {\mathbb {E} \left[ \sum_ {q \in \mathcal {Q}} P (q) (\hat {c} _ {s} (q) - c _ {s} ^ {\star} (q)) ^ {2} \right]} + \sqrt {\mathbb {E} \left[ \sum_ {q \in \mathcal {Q}} P (q) (\hat {c} _ {l} (q) - c _ {l} ^ {\star} (q)) ^ {2} \right]} \\ \stackrel {(i i i)} {\leq} C (B _ {2} - B _ {1}) \sqrt {\frac {| \mathcal {Q} | \log (N)}{N}}. \\ \end{array}
$$
Here  $(i)$  is due to Cauchy-Schwarz, and (ii) is from Jensen's inequality, and (iii) is the standard rate of the least squared estimator (Rigollet and Hutter, 2015).
For the second difference, we have
$$
\begin{array}{l} \operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) = \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \notin \hat {\mathcal {L}}) - \mathbb {1} (q \notin \mathcal {L} ^ {\star})\right) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \\ = \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \in \mathcal {L} ^ {\star}) - \mathbb {1} (q \in \hat {\mathcal {L}})\right) \min  (c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)). \\ \end{array}
$$
Note that for any  $q\in \mathcal{L}^{\star}$  , we know that
$$
\begin{array}{l} \hat {P} (q) \left(\min (\hat {c} _ {s} (q), \hat {c} _ {l} (q)) - (B _ {2} - B _ {1}) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}}\right) \\ \geq \max  \left(P (q) - \sqrt {\frac {2 \log (8 / \delta)}{N}}, 0\right) \cdot \left(\min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) - 2 (B _ {2} - B _ {1}) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}}\right) \\ \geq P (q) \min \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) - C (B _ {2} - B _ {1}) \cdot \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}} \\ \geq P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) - C \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log \left(8 | \mathcal {Q} | / \delta\right)}{B _ {1} N}}. \\ \end{array}
$$
The last inequality uses event  $E_{3}$  and Lemma 1. And similarly, for any  $q \notin \mathcal{L}^{\star}$ , we know that
$$
\begin{array}{l} \hat {P} (q) \left(\min  (\hat {c} _ {s} (q), \hat {c} _ {l} (q)) - (B _ {2} - B _ {1}) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}}\right) \leq \left(P (q) + \sqrt {\frac {2 \log (8 / \delta)}{N}}\right) \min  (c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)) \\ \leq P (q) \min \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) + B _ {2} \sqrt {\frac {2 \log (8 / \delta)}{N}}. \\ \end{array}
$$
$$
\begin{array}{l} P \left(q ^ {\prime}\right) \min  \left(c _ {s} ^ {\star} \left(q ^ {\prime}\right), c _ {l} ^ {\star} \left(q ^ {\prime}\right)\right) - P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \\ \leq \hat {P} (q ^ {\prime}) \left(\min (\hat {c} _ {s} (q ^ {\prime}), \hat {c} _ {l} (q ^ {\prime})) - (B _ {2} - B _ {1}) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q ^ {\prime})}}\right) \\ - \hat {P} (q) \left(\min  \left(\hat {c} _ {s} (q), \hat {c} _ {l} (q)\right) - \left(B _ {2} - B _ {1}\right) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} \left(q _ {n} = q\right)}}\right) + C \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (8 | \mathcal {Q} | / \delta)}{B _ {1} N}} \\ \leq C (B _ {2} - B _ {1}) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (8 | \mathcal {Q} | / \delta)}{B _ {1} N}}. \\ \end{array}
$$
Here the last inequality uses the fact that  $q$  is inside  $\mathcal{L}_t$  and thus the difference between the first two terms are upper bounded by 0. Finally, we know that conditioned on  $E_1 \cap E_2 \cap E_3 \cap E_4$ , we have
$$
\operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) \leq C L \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (8 | \mathcal {Q} | / \delta)}{B _ {1} N}}.
$$
Overall, we know that
$$
\mathbb {E} \left[ \operatorname {c o s t} \left(\hat {\mathcal {L}}, \hat {\pi}\right) - \operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) \right] \leq B _ {2} \delta + C L \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (8 | \mathcal {Q} | / \delta)}{B _ {1} N}}.
$$
H Proof of Theorem 4
Proof. Let  $g_{k}(q)$  be the length of the interval between the  $k$ -th and  $(k + 1)$ -th arrival of query  $q$  in the sequence of received queries (we set  $g_{k}(q) = 0$  if  $k$  exceeds the total number of times  $q$  is queried.). Define the following four events:
$$
\begin{array}{l} E _ {1, t} = \left\{\forall q \in \mathcal {Q}, | \hat {P} _ {t - 1} (q) - P (q) | \leq \min  \left(1, \sqrt {\frac {2 \log (8 T / \delta)}{t - 1}}\right) \right\}, \\ E _ {2, t} = \left\{\forall q \in \mathcal {Q}, \hat {c} _ {l, t - 1} (q) \in \left[ c _ {l} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} (s _ {i} = 0 , q _ {i} = q)}}\right), c _ {l} ^ {\star} (q) \right] \right\}, \\ E _ {3, t} = \left\{\forall q \in \mathcal {Q}, \hat {c} _ {s, t - 1} (q) \in \left[ c _ {s} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right), c _ {s} ^ {\star} (q) \right] \right\}, \\ E _ {4} = \left\{\forall q \in \mathcal {L} ^ {\star}, k \leq T, g _ {k} (q) \leq \frac {B _ {2} | \mathcal {Q} | \log (4 T L / \delta)}{B _ {1}} \right\}. \\ \end{array}
$$
From the same analysis as Lemma 3, we know that the four events  $\left(\bigcap_{s=1}^{t} E_{1,s} \cap E_{2,s} \cap E_{3,s}\right) \cap E_4$  hold simultaneously with probability at least  $1 - \delta$ .
Let  $E^{t} = \bigcap_{s=1}^{t} E_{1,s} \cap E_{2,s} \cap E_{3,s}$ . The regret can be decomposed as follows.
Regret(T)
$$
\begin{array}{l} = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}, \pi^ {\star}\right) \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] + \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {1} \left(\bar {E} ^ {t}\right) \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] + \delta T B _ {2} \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left(\operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi^ {\star}\right) + \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi^ {\star}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}, \pi^ {\star}\right)\right) \mathbb {1} \left(E ^ {t}\right) \right] + \delta T B _ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi^ {\star}\right) \mid E ^ {t} \right] \\ = \mathbb {E} [ \mathbb {1} (q _ {t} \notin \mathcal {L} _ {t}) (c _ {s} ^ {\star} (q _ {t}) \pi_ {t} (q _ {t}) + c _ {l} ^ {\star} (q _ {t}) (1 - \pi_ {t} (q _ {t})) - c _ {s} ^ {\star} (q _ {t}) \pi^ {\star} (q _ {t}) - c _ {l} ^ {\star} (q _ {t}) (1 - \pi^ {\star} (q _ {t}))) \mid E ^ {t} ] \\ = \mathbb {E} \left[ \mathbb {1} \left(q _ {t} \notin \mathcal {L} _ {t}\right) \left(c _ {s} ^ {\star} \left(q _ {t}\right) \pi_ {t} \left(q _ {t}\right) + c _ {l} ^ {\star} \left(q _ {t}\right) \left(1 - \pi_ {t} \left(q _ {t}\right)\right) - \min  \left(c _ {s} ^ {\star} \left(q _ {t}\right), c _ {l} ^ {\star} \left(q _ {t}\right)\right)\right) \mid E ^ {t} \right] \\ \leq \mathbb {E} \left[ c _ {s} ^ {\star} \left(q _ {t}\right) \pi_ {t} \left(q _ {t}\right) + c _ {l} ^ {\star} \left(q _ {t}\right) \left(1 - \pi_ {t} \left(q _ {t}\right)\right) - \min  \left(c _ {s} ^ {\star} \left(q _ {t}\right), c _ {l} ^ {\star} \left(q _ {t}\right)\right) \mid E ^ {t} \right] \\ = \mathbb {E} \left[ \right. c _ {s} ^ {\star} \left(q _ {t}\right) \mathbb {I} \left(\hat {c} _ {s, t} \left(q _ {t}\right) \leq \hat {c} _ {l, t} \left(q _ {t}\right)\right) + c _ {l} ^ {\star} \left(q _ {t}\right) \mathbb {I} \left(\hat {c} _ {s, t} \left(q _ {t}\right) > \hat {c} _ {l, t} \left(q _ {t}\right)\right) - \min  \left(c _ {s} ^ {\star} \left(q _ {t}\right), c _ {l} ^ {\star} \left(q _ {t}\right)\right. | E ^ {t} ]. \\ \end{array}
$$
Note that if  $\hat{c}_{s,t}(q_t) - \hat{c}_{l,t}(q_t)$  has the same sign as  $c_s^\star (q_t) - c_l^\star (q_t)$ , the difference  $c_{s}^{\star}(q_{t})\mathbb{1}(\hat{c}_{s,t}(q_{t})\leq$ $\hat{c}_{l,t}(q_t)) + c_l^\star (q_t)\mathbb{1}(\hat{c}_{s,t}(q_t) > \hat{c}_{l,t}(q_t)) - \min (c_s^\star (q_t),c_l^\star (q_t))$  becomes 0. Otherwise, if  $c_s^\star (q_t) -$ $c_l^\star (q_t) > 0$  and  $\hat{c}_{s,t}(q_t) - \hat{c}_{l,t}(q_t)\leq 0$ , we know that  $s_t = 1$  and
$$
\begin{array}{l} c _ {s} ^ {\star} (q) - c _ {l} ^ {\star} (q) \leq \hat {c} _ {s, t} (q) - \hat {c} _ {l, t} (q) + 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right) \\ \leq 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right). \\ \end{array}
$$
And similarly if  $c_s^\star(q) - c_l^\star(q) \leq 0$  and  $\hat{c}_{s,t}(q_t) - \hat{c}_{l,t}(q_t) > 0$ , we know that  $s_t = 0$  and  $c_l^\star(q) - c_s^\star(q) \leq 2(B_2 - B_1) \min\left(1, \sqrt{\frac{\log(8T|\mathcal{Q}| / \delta)}{2\sum_{i=1}^{t-1}\mathbb{1}(s_i = 0,q_i = q)}}\right)$ . Overall, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left(\operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi_ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}, \pi^ {\star}\right) \right] \right. \\ \leq 2 \left(B _ {2} - B _ {1}\right) \sum_ {t = 1} ^ {T} \sum_ {q \in \mathcal {Q}} \mathbb {E} \left[ \mathbb {1} \left(s _ {t} = 1, q _ {t} = q\right) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right) \right. \\ \left. + \mathbb {1} \left(s _ {t} = 0, q _ {t} = q\right) \min  \left(1, \sqrt {\frac {\log (8 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 0 , q _ {i} = q\right)}}\right) \right] \\ \leq 2 \left(B _ {2} - B _ {1}\right) \sqrt {| \mathcal {Q} | T \log (8 | \mathcal {Q} | T / \delta)}. \\ \end{array}
$$
Here the last inequality uses the fact that for each  $q \in \mathcal{Q}$ , the summation over time step is upper bounded by  $2\sum_{i=1}^{T(q)} \sqrt{\log(8T|\mathcal{Q}| / \delta) / 2i} \leq 2\sqrt{\log(8T|\mathcal{Q}| / \delta)T(q)}$ , where  $T(q)$  is the number of steps of receiving query  $q$  in total  $T$  steps. Optimizing over  $T(q)$  gives the final bound.
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {c o s t} \left(\mathcal {L} _ {t}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) \mid E ^ {t} \right] = \mathbb {E} \left[ \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \notin \mathcal {L} _ {t}) - \mathbb {1} (q \notin \mathcal {L} ^ {\star})\right) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \mid E ^ {t} \right] \\ = \mathbb {E} \left[ \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \in \mathcal {L} ^ {\star}) - \mathbb {1} (q \in \mathcal {L} _ {t})\right) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \mid E ^ {t} \right]. \\ \end{array}
$$
Note that for any  $q \in \mathcal{L}^{\star}$ , we know that
$$
\begin{array}{l} \tilde {P} _ {t} (q) \min  \left(\hat {c} _ {s, t} (q), \hat {c} _ {l, t} (q)\right) \\ \geq \max  \left(P (q) - \sqrt {\frac {2 \log (8 / \delta)}{t}}, 0\right) \cdot \mathbb {1} \left(\pi_ {t} (q) = 1\right) \left(c _ {s} ^ {\star} (q) - 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right)\right) \\ + \mathbb {1} \left(\pi_ {t} (q) = 0\right) \left(c _ {l} ^ {\star} (q) - 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 0 , q _ {i} = q\right)}}\right)\right) \\ \geq P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) - \left(B _ {2} - B _ {1}\right) \sqrt {\frac {2 \log (8 / \delta)}{t}} + P (q) \cdot \left(\mathbb {1} \left(\pi_ {t} (q) = 1\right) \right. \\ \left. \cdot \left(c _ {s} ^ {\star} (q) - 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log \left(8 | \mathcal {Q} | / \delta\right)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 1 , q _ {i} = q\right)}}\right)\right) \right. \\ + \mathbb {1} (\pi_ {t} (q) = 0) \left(c _ {l} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 0 , q _ {i} = q\right)}}\right)\right) - \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right)\left. \right) \\ \geq P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) - C \left(B _ {2} - B _ {1}\right) \cdot \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = \pi_ {t} (q) , q _ {i} = q\right)}}\right). \\ \end{array}
$$
Below we justify the last inequality. First, note that  $\pi_t(q) = 1$  is equivalent to that  $\hat{c}_{s,t}(q) \leq \hat{c}_{l,t}(q)$ . Thus if  $\hat{c}_{s,t}(q_t) - \hat{c}_{l,t}(q_t)$  has the same sign as  $c_s^\star (q_t) - c_l^\star (q_t)$ , the above inequality holds. Now consider the case when  $\hat{c}_{s,t}(q_t) - \hat{c}_{l,t}(q_t)$  has a different sign as  $c_{s}^{\star}(q_{t}) - c_{l}^{\star}(q_{t})$ . Assume that  $\hat{c}_{s,t}(q_t) > \hat{c}_{l,t}(q_t)$  and  $c_s^\star (q_t) < c_l^\star (q_t)$ . We know that  $\pi_t(q) = 0$ , and
$$
\begin{array}{l} c _ {l} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \min \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} (s _ {i} = 0 , q _ {i} = q)}}\right) - c _ {s} ^ {\star} (q) \\ > - 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = 0 , q _ {i} = q\right)}}\right). \\ \end{array}
$$
Similarly we can prove that for the reversed case. Now for any  $q \notin \mathcal{L}^{\star}$ , we know that
$$
\begin{array}{l} \hat {P} _ {t} (q) \min  \left(\hat {c} _ {s, t} (q), \hat {c} _ {l, t} (q)\right) \leq \left(P (q) + \sqrt {\frac {2 \log (8 / \delta)}{t}}\right) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \\ \leq P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) + B _ {2} \sqrt {\frac {2 \log (8 / \delta)}{t}}. \\ \end{array}
$$
$$
\begin{array}{l} P (q ^ {\prime}) \min  \left(c _ {s} ^ {\star} \left(q ^ {\prime}\right), c _ {l} ^ {\star} \left(q ^ {\prime}\right)\right) - P (q) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right) \\ \leq \hat {P} _ {t} (q ^ {\prime}) \min  \left(\hat {c} _ {s, t} (q ^ {\prime}), \hat {c} _ {l, t} (q ^ {\prime})\right) - \hat {P} _ {t} (q) \min  \left(\hat {c} _ {s, t} (q), \hat {c} _ {l, t} (q)\right) \\ + C \left(B _ {2} - B _ {1}\right) \cdot \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = \pi_ {t} (q) , q _ {i} = q\right)}}\right) + B _ {2} \sqrt {\frac {2 \log (8 / \delta)}{t}} \\ \leq C (B _ {2} - B _ {1}) \cdot \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} (s _ {i} = \pi_ {t} (q) , q _ {i} = q)}}\right) + B _ {2} \sqrt {\frac {2 \log (8 / \delta)}{t}}. \\ \end{array}
$$
Here the last inequality uses the fact that  $q$  is inside  $\mathcal{L}_t$ . Thus we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(\mathcal {L} _ {t}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(\mathcal {L} _ {t}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {1} \left(E ^ {t} \cap E _ {4}\right) \right] + B _ {2} T \delta \\ \leq B _ {2} T \delta + C \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {q \in \mathcal {L} ^ {\star}} \mathbb {1} (q \notin \mathcal {L} _ {t}) \left(B _ {2} - B _ {1}\right) \cdot \min  \left(1, \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = \pi_ {t} (q) , q _ {i} = q\right)}}\right) \right. \\ \left. + B _ {2} \sqrt {\frac {2 \log (8 / \delta)}{t}} \mid E ^ {t} \cap E _ {4} \right] \\ \leq C \cdot \left(B _ {2} T \delta + L B _ {2} \sqrt {2 T \log (8 / \delta)} + \left(B _ {2} - B _ {1}\right) \log (8 T | \mathcal {Q} | / \delta)\right) \\ \cdot \sum_ {q \in \mathcal {L} ^ {\star}} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{\sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(s _ {i} = \pi_ {t} (q) , q _ {i} = q\right)}}\right) | E ^ {t} \cap E _ {4} \right] \\ = C \cdot \left(B _ {2} T \delta + L B _ {2} \sqrt {2 T \log (8 / \delta)} + \left(B _ {2} - B _ {1}\right) \log (8 T | \mathcal {Q} | / \delta) \right. \\ \cdot \sum_ {q \in \mathcal {L} ^ {\star}} \sum_ {\pi \in \{1, 2 \}} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}, \pi_ {t} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{\sum_ {i = 1} ^ {t - 1} \mathbb {1} (s _ {i} = \pi , q _ {i} = q)}}\right) | E ^ {t} \cap E _ {4} \right]. \\ \end{array}
$$
Let  $T_{t}(q,\pi) = \sum_{i = 1}^{t - 1}\mathbb{1}(s_{i} = \pi ,q_{i} = q)$ . Now for each  $q\in \mathcal{L}^{\star}$  and  $\pi \in \{0,1\}$ , we look at the term  $\sum_{t = 1}^{T}\mathbb{E}\left[\mathbb{1}(q\not\in\mathcal{L}_{t},\pi_{t} = \pi)\cdot \min \left(1,\sqrt{\frac{1}{T_{t}(q,\pi)}}\right)\mid E^{t}\cap E_{4}\right]$ . We prove the following lemma:
$$
\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}, \pi_ {t} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q , \pi)}}\right) \mid E ^ {t} \cap E _ {4} \right] \leq \frac {C B _ {2} | \mathcal {Q} | \log (3 T L / \delta) \sqrt {T}}{B _ {1}} + T \delta .
$$
Proof. Let  $t_k(q) = \sum_{l=1}^{k-1} g_l(q)$  be the step that the  $k$ -th query of  $q$  arrives. And let  $E = (\bigcap_{t=1}^T E_t) \cap E_4$ . The summation can be written as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}, \pi_ {t} (q) = \pi) \cdot \min \left(1, \sqrt {\frac {1}{T _ {t} (q , \pi)}}\right) \mid E ^ {t} \cap E _ {3} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}, \pi_ {t} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q , \pi)}}\right) \mid E \right] + T \delta \\ = \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} (q \notin \mathcal {L} _ {t}, \pi_ {t} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q , \pi)}}\right) | E \right] + T \delta \\ \leq \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}, \pi_ {t _ {k + 1}} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q , \pi)}}\right) | E \right] + T \delta . \\ \end{array}
$$
The last inequality is due to (a).  $T_{t}(q,\pi)$  does not change if at round  $t$  the query is not  $q$ ; (b). if  $q\in \mathcal{L}_{t_{k + 1}(q)}$ , we will have  $q\in \mathcal{L}_t$  for any  $t\in [t_k(q) + 1,t_{k + 1}(q)]$  since  $q$  never arrives in the middle and must remain in the cache set until  $t_{k + 1}(q)$ ; (c) For  $t\in [t_k(q) + 1,t_{k + 1}(q)]$ ,  $\pi_t$  does not change since both the frequency and cost estimator does not change for  $q$ . From the definition of  $t_k(q)$ , we know that  $T_{t_k(q) + 1}(q,\pi)\geq 1$  and thus we can drop the the minimum in the above equation. Now from event  $E_4$ , we know that