SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\mathrm {c o s t} (\mathcal {L} _ {\sf L F U})}{\mathrm {c o s t} (\mathcal {L} _ {\sf L E C})} \geq \frac {\max _ {q \in \mathcal {Q}} c _ {l} ^ {\star} (q)}{\min _ {q \in \mathcal {Q}} c _ {l} ^ {\star} (q)} - \epsilon .
$$
The construction can be seen from a two-query example. Let  $c_{l}^{\star}(q_{1}) = c_{1}$ ,  $c_{l}^{\star}(q_{2}) = c_{2}$  with  $c_{1} < c_{2}$ . Let  $P(q_{1}) = P(q_{2}) = 0$ . This shows that when the individual cost varies drastically for different queries, the total expected cost for LFU can be highly sub-optimal compared with the cost-aware caching strategy.
To compare the performance of the model multiplexing in Section 4, we take the cache size  $L = 0$ . We have the following proposition for the performance improvement of the model multiplexer.
Proposition 2. Let  $L = 0$ . The difference in cost between the baseline and the model multiplexer can be written as
$$
\operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi_ {s}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) = \sum_ {q \in \mathcal {Q}} P (q) \max  \left(0, c _ {s} ^ {\star} (q) - c _ {l} ^ {\star} (q)\right),
$$
$$
\operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi_ {l}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) = \sum_ {q \in \mathcal {Q}} P (q) \max  \left(0, c _ {l} ^ {\star} (q) - c _ {s} ^ {\star} (q)\right).
$$
The proof is a direct result of plugging in the cost definition. We see that the gap between  $\pi_s$  and the optimal model multiplexer becomes larger when a large fraction of the queries have smaller cost when processed by the large models, and vice versa.
Proof. We first prove the following lemma on the lower bound of  $P(q)$  for any  $q \in \mathcal{L}^{\star}$ .
Lemma 1. For any  $q \in \mathcal{L}^*$ , we have  $P(q) \geq B_1 / (B_2|\mathcal{Q}|)$ .
Proof. From the fact that  $\sum_{q\in \mathcal{Q}}P(q) = 1$  and for any  $q\in \mathcal{L}^{\star}$  and any  $q^{\prime}\notin \mathcal{L}^{\star}$ ,  $P(q)\geq P(q^{\prime})c_{l}^{\star}(q^{\prime}) / c_{l}^{\star}(q)\geq P(q^{\prime})B_{1} / B_{2}$ , we know that for any  $q\in \mathcal{L}^{\star}$ ,  $P(q)\geq B_1 / (B_2|\mathcal{Q}|)$ .
We define the following three events:
$$
E _ {1} = \left\{\forall q \in \mathcal {Q}, | \hat {P} (q) - P (q) | \leq \sqrt {\frac {2 \log (6 / \delta)}{N}} \right\},
$$
$$
E _ {2} = \left\{\forall q \in \mathcal {Q}, | \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) | \leq (B _ {2} - B _ {1}) \sqrt {\frac {\log (6 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}} \right\},
$$
$$
E _ {3} = \left\{\forall q \in \mathcal {L} ^ {\star}, \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q) \geq \frac {B _ {1} N}{2 B _ {2} | \mathcal {Q} |} \right\}.
$$
We know that the first two events hold simultaneously with probability at least  $1 - 2\delta /3$  from Lemma 3. For the third event, from the Chernoff bound, we know that for any  $q\in \mathcal{Q}$  , we have
$$
\mathbb {P} \left(\sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q) \geq N P (q) / 2\right) \geq 1 - \exp (- N P (q) / 8).
$$
From Lemma 1 we know that for any  $q \in \mathcal{L}^{\star}$ ,  $P(q) \geq B_1 / (B_2|\mathcal{Q}|)$ . Thus the above inequality further implies
$$
\mathbb {P} \left(\sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q) \geq \frac {B _ {1} N}{2 B _ {2} | \mathcal {Q} |}\right) \geq 1 - \exp \left(- \frac {B _ {1} N}{8 B _ {2} | \mathcal {Q} |}\right) \geq 1 - \frac {\delta}{3 L}.
$$
The last inequality is due to our assumption that  $N \geq \frac{8B_2|\mathcal{Q}|\log(3L / \delta)}{B_1}$ .
We condition on the three events from now on. The last two events imply that for any  $q \in \mathcal{L}^*$ ,
$$
| \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) | \leq (B _ {2} - B _ {1}) \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}}.
$$
$$
\begin{array}{l} \operatorname {c o s t} (\hat {\mathcal {L}}) - \operatorname {c o s t} (\mathcal {L} ^ {\star}) = \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \notin \hat {\mathcal {L}}) - \mathbb {1} (q \notin \mathcal {L} ^ {\star})\right) c _ {l} ^ {\star} (q) \\ = \sum_ {q \in \mathcal {Q}} P (q) \left(\mathbb {1} (q \in \mathcal {L} ^ {\star}) - \mathbb {1} (q \in \hat {\mathcal {L}})\right) c _ {l} ^ {\star} (q). \\ \end{array}
$$
Let  $\hat{c}_{l, pes}(q) = \hat{c}_l(q) - (B_2 - B_1)\sqrt{\frac{\log(6|\mathcal{Q}| / \delta)}{2\sum_{n = 1}^N\mathbb{1}(q_n = q)}}$ . Note that for any  $q\in \mathcal{L}^{\star}$ , we know that
$$
\begin{array}{l} \hat {P} (q) \hat {c} _ {l, p e s} (q) \geq \max  \left(P (q) - \sqrt {\frac {2 \log (6 / \delta)}{N}}, 0\right) \left(c _ {l} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}}\right) \\ \geq P (q) c _ {l} ^ {\star} (q) - C \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}}. \\ \end{array}
$$
And similarly, for any  $q \notin \mathcal{L}^{\star}$ , we know that
$$
\hat {P} (q) \hat {c} _ {l, p e s} (q) \leq \left(P (q) + \sqrt {\frac {2 \log (6 / \delta)}{N}}\right) c _ {l} ^ {\star} (q) \leq P (q) c _ {l} ^ {\star} (q) + B _ {2} \sqrt {\frac {2 \log (6 / \delta)}{N}}.
$$
Now consider any  $q \in \hat{\mathcal{L}}$  but  $q \notin \mathcal{L}^{\star}$ , and any other  $q' \in \mathcal{L}^{\star}$  but  $q' \notin \hat{\mathcal{L}}$ . We have
$$
\begin{array}{l} P (q ^ {\prime}) c _ {l} ^ {\star} (q ^ {\prime}) - P (q) c _ {l} ^ {\star} (q) \\ \leq \hat {P} (q ^ {\prime}) \hat {c} _ {l, p e s} (q ^ {\prime}) - \hat {P} (q) \hat {c} _ {l, p e s} (q) + C (B _ {2} - B _ {1}) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}} \\ \leq C \left(B _ {2} - B _ {1}\right) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}}. \\ \end{array}
$$
Overall, we know that conditioned on  $E_1 \cap E_2 \cap E_3$ , we have
$$
\operatorname {c o s t} (\hat {\mathcal {L}}) - \operatorname {c o s t} (\mathcal {L} ^ {\star}) \leq C (B _ {2} - B _ {1}) L \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}}.
$$
And this implies that
$$
\mathbb {E} [ \mathrm {c o s t} (\hat {\mathcal {L}}) - \mathrm {c o s t} (\mathcal {L} ^ {\star}) ] \leq C (B _ {2} - B _ {1}) L \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (6 | \mathcal {Q} | / \delta)}{N B _ {1}}} + \delta B _ {2}.
$$
Taking  $\delta = 1 / N$  finishes the proof.
Proof. Upper Bound. We start with the upper bound by the following lemma.
Lemma 2. In each round  $t \in [T]$ , we always have
$$
\mathcal {L} _ {t + 1} \in \arg \min  _ {\mathcal {L}} \sum_ {q \in \mathcal {Q}} \hat {P} _ {t} (q) \mathbb {1} (q \notin \mathcal {L}) \hat {c} _ {l, t} (q).
$$
Proof. We prove this lemma by induction. First, consider the case when  $|\mathcal{L}_{t + 1}| < L$ . In this scenario, we always put the query into the cache. And  $\mathcal{L}_{t + 1}$  contains all queries with non-zero  $\hat{P}_t$ . Thus such  $\mathcal{L}_{t + 1}$  is always one of the minimizers.
Now consider the case when  $|\mathcal{L}_{t + 1}| = L$ . Assume that the conclusion holds for time step  $t$ . Now consider the case of  $t + 2$ . When the new query is in the cache  $q_{t + 1} \in \mathcal{L}_{t + 1}$ , the cache will remain unchanged  $\mathcal{L}_{t + 2} = \mathcal{L}_{t + 1}$ . In this case, the estimated probability for  $q_{t + 1}$  is increased, while the others are decreased, and  $\hat{c}_{l,t + 1}$  is not changed for any query. Thus  $\mathcal{L}_{t + 2}$  is still the minimizer. When the new query does not hit the cache, the estimated probability times costs for all other queries except for  $q_{t + 1}$  are decreased proportionally since  $\hat{P}_{t + 1}$  is decreased proportionally while  $\hat{c}_{l,t + 1}$  is not changed for all other queries. Thus the only potential change in the relative order of costs is that of  $q_{t + 1}$ . Since we can add  $q_{t + 1}$  at the end of query, we know that after this round  $\mathcal{L}_{t + 2}$  is still the minimizer.
Let  $g_{k}(q)$  be the length of the interval between the  $k$ -th and  $k + 1$ -th arrival of query  $q$  in the sequence of received queries (we set  $g_{k}(q) = 0$  if  $k$  exceeds the total number of times  $q$  is queried.). Define the following three events:
$$
\begin{array}{l} E _ {1, t} = \left\{\forall q \in \mathcal {Q}, | \hat {P} _ {t - 1} (q) - P (q) | \leq \min  \left(1, \sqrt {\frac {2 \log (6 T / \delta)}{t - 1}}\right) \right\}, \\ E _ {2, t} = \left\{\forall q \in \mathcal {Q}, \hat {c} _ {l, t - 1} (q) - c _ {l} ^ {\star} (q) \in \left[ - 2 \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{2 \sum_ {i = 1} ^ {t - 1} \mathbb {1} \left(c _ {i} \neq \times , q _ {i} = q\right)}}\right), 0 \right] \right\}, \\ E _ {3} = \left\{\forall q \in \mathcal {L} ^ {\star}, k \leq T, g _ {k} (q) \leq \frac {B _ {2} | \mathcal {Q} | \log (3 T L / \delta)}{B _ {1}} \right\}. \\ \end{array}
$$
We prove that the three events hold simultaneously with probability at least  $1 - \delta$ :
$$
\mathbb {P} \left(\left(\bigcap_ {t = T ^ {2 / 3}} ^ {T} E _ {1, t} \cap E _ {2, t}\right) \cap E _ {3}\right) \geq 1 - \delta .
$$
Proof. From the Dvoretzky-Kiefer-Wolffowitz inequality, we have
$$
\mathbb {P} (\max _ {q \in \mathcal {Q}} | \hat {P} _ {t} (q) - P (q) | > \epsilon) \leq 2 \exp (- \epsilon^ {2} t / 2).
$$
By taking  $\epsilon = \sqrt{\frac{2\log(6T / \delta)}{t}}$ , we see that  $\max_{q\in \mathcal{Q}}|\hat{P}_t(q) - P(q)|\leq \epsilon$  holds with probability at least  $1 - \delta /(3T)$  for any fixed  $t\in [T]$ . Now by taking a union bound over all  $t\in [T]$ , we know that  $\bigcap_{t = 1}^{T}E_{1,t}$  holds with probability at least  $1 - \delta /3$ .
For the second event, from Hoeffding's inequality, we have for any  $q \in \mathcal{Q}$ ,
$$
\mathbb {P} \left(\left| \hat {c} _ {l, t} (q) - c _ {l} ^ {\star} (q) \right| \leq \left(B _ {2} - B _ {1}\right) \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{2 \sum_ {s = 1} ^ {t - 1} \mathbb {1} \left(c _ {s} \neq \times , q _ {s} = q\right)}}\right)\right) \geq 1 - \frac {\delta}{3 T | \mathcal {Q} |}.
$$
Now taking union bound over  $t \in [T]$  and  $q \in \mathcal{Q}$  gives that  $\bigcap_{t=1}^{T} E_{2,t}$  holds with probability at least  $1 - \delta / 3$ .
For the third event, we know that the interval  $g_{k}(q)$  satisfies a geometric distribution with success probability  $P(q)$ . For any  $q \in \mathcal{L}^{\star}$ , we have
$$
\mathbb {P} \left(g _ {k} (q) \geq s\right) \leq (1 - P (q)) ^ {s} \leq \left(1 - \frac {B _ {1}}{| \mathcal {Q} | B _ {2}}\right) ^ {s}.
$$
By taking  $s = \frac{B_2|\mathcal{Q}|\log(TL / \delta)}{B_1}$ , we know that
$$
\mathbb {P} \left(g _ {k} (q) \geq \frac {B _ {2} | \mathcal {Q} | \log (3 T L / \delta)}{B _ {1}}\right) \leq \left(1 - \frac {B _ {1}}{| \mathcal {Q} | B _ {2}}\right) ^ {s} \leq \frac {\delta}{3 T L}.
$$
By taking union bounds over all  $q \in \mathcal{L}^{\star}$  and  $k$  we get the result.
Let  $E^{t} = \bigcap_{s=1}^{t} E_{1,s} \cap E_{2,s}$ . We can write the regret as follows.
$$
\begin{array}{l} \operatorname {R e g r e t} (T) \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] + \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(\bar {E} ^ {t}\right) \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] + C \delta T B _ {2} \\ = C \delta T B _ {2} + \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right]. \\ \end{array}
$$
Note that the sampling distribution of  $q_{t}$  is independent of  $E^{t}$ . Thus we can write the expectation as
$$
\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] \leq \sum_ {t = 1} ^ {T} \sum_ {q \in \mathcal {Q}} \mathbb {E} \left[ P (q) \left(\mathbb {1} \left(q \notin \mathcal {L} _ {t}\right) - \mathbb {1} \left(q \notin \mathcal {L} ^ {\star}\right)\right) c _ {l} ^ {\star} (q) \mid E ^ {t} \right]
$$
Let  $T_{t}(q) = \sum_{i=1}^{t-1} \mathbb{1}(q_{i} \notin \mathcal{L}_{i}, q_{i} = q)$ . Note that the event  $c_{i} = \times$  is equivalent to that  $q_{i} \in \mathcal{L}_{i}$ . Now at each round  $t$ , conditioned on event  $E^{t}$ , we know that for any  $q \in \mathcal{L}^{\star}$ ,
$$
\begin{array}{l} \hat {P} _ {t - 1} (q) \hat {c} _ {l, t - 1} (q) \geq \max  \left(P (q) - \min  \left(1, \sqrt {\frac {2 \log (6 T / \delta)}{t - 1}}\right), 0\right) \left(c _ {l} ^ {\star} (q) - 2 (B _ {2} - B _ {1}) \cdot \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{T _ {t} (q)}}\right)\right) \\ \geq P (q) c _ {l} ^ {\star} (q) - C \left(B _ {2} - B _ {1}\right) \cdot \min  \left(1, \sqrt {\frac {\log \left(6 T | \mathcal {Q} | / \delta\right)}{T _ {t} (q)}}\right). \\ \end{array}
$$
$$
\hat {P} _ {t} (q) \hat {c} _ {l, t - 1} (q) \leq \left(P (q) + \min  \left(1, \sqrt {\frac {2 \log (8 T / \delta)}{t - 1}}\right)\right) c _ {l} ^ {\star} (q) \leq P (q) c _ {l} ^ {\star} (q) + B _ {2} \min  \left(1, \sqrt {\frac {2 \log (8 T / \delta)}{t - 1}}\right).
$$
Now consider any  $q \in \mathcal{L}_t$  but  $q \notin \mathcal{L}^\star$ , and any other  $q' \in \mathcal{L}^\star$  but  $q' \notin \mathcal{L}_t$ . We have
$$
\begin{array}{l} P (q ^ {\prime}) c _ {l} ^ {\star} (q ^ {\prime}) - P (q) c _ {l} ^ {\star} (q) \\ \leq \hat {P} (q ^ {\prime}) \hat {c} _ {l, t - 1} (q ^ {\prime}) - \hat {P} (q) \hat {c} _ {l, t - 1} (q) + C (B _ {2} - B _ {1}) \cdot \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{T _ {t} (q)}}\right) + B _ {2} \min  \left(1, \sqrt {\frac {2 \log (6 T / \delta)}{t - 1}}\right) \\ \leq C \left(B _ {2} - B _ {1}\right) \cdot \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{T _ {t} (q)}}\right) + B _ {2} \min  \left(1, \sqrt {\frac {2 \log (8 T / \delta)}{t - 1}}\right). \\ \end{array}
$$
Thus we know that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {I} \left(E ^ {t}\right) \right] \\ \leq C \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {q \in \mathcal {L} ^ {\star}} \mathbb {1} (q \notin \mathcal {L} _ {t}) (B _ {2} - B _ {1}) \cdot \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{T _ {t} (q)}}\right) + B _ {2} \min  \left(1, \sqrt {\frac {2 \log (6 T / \delta)}{t - 1}}\right) \mid E ^ {t} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(q _ {t}, \mathcal {L} _ {t}\right) - \operatorname {c o s t} \left(q _ {t}, \mathcal {L} ^ {\star}\right) \mathbb {1} \left(E ^ {t}\right) \right] \\ \leq B _ {2} T \delta + C \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {q \in \mathcal {L} ^ {\star}} \mathbb {1} (q \notin \mathcal {L} _ {t}) (B _ {2} - B _ {1}) \cdot \min  \left(1, \sqrt {\frac {\log (6 T | \mathcal {Q} | / \delta)}{T _ {t} (q)}}\right) \right. \\ \left. + B _ {2} \min  \left(1, \sqrt {\frac {2 \log (6 T / \delta)}{t - 1}}\right) \mid E ^ {t} \cap E _ {3} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq C \cdot \left(B _ {2} T \delta + L B _ {2} \sqrt {2 T \log (6 T / \delta)} \right. \\ + \left(B _ {2} - B _ {1}\right) \log (6 T | \mathcal {Q} | / \delta) \cdot \sum_ {q \in \mathcal {L} ^ {\star}} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q)}}\right) \mid E ^ {t} \cap E _ {3} \right]. \\ \end{array}
$$
Now for each  $q \in \mathcal{L}^{\star}$ , we look at the term  $\sum_{t=1}^{T} \mathbb{E}\left[\mathbb{1}(q \notin \mathcal{L}_t) \cdot \min\left(1, \sqrt{\frac{1}{T_t(q)}}\right) \mid E^t \cap E_3\right]$ . We prove the following lemma:
$$
\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q)}}\right) \mid E ^ {t} \cap E _ {3} \right] \leq \frac {C B _ {2} | \mathcal {Q} | \log (3 T L / \delta) \sqrt {T}}{B _ {1}} + T \delta .
$$
Proof. Let  $t_k(q) = \sum_{l=1}^{k-1} g_l(q)$  be the step that the  $k$ -th query of  $q$  arrives, with  $t_0(q) = 0$ . And let  $E = \left( \bigcap_{t=1}^T E_t \right) \cap E_3$ . The summation can be written as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q)}}\right) \mid E ^ {t} \cap E _ {3} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q)}}\right) \mid E \right] + T \delta \\ = \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} (q \notin \mathcal {L} _ {t}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t} (q)}}\right) \mid E \right] + T \delta \\ \leq \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q)}}\right) \mid E \right] + T \delta . \\ \end{array}
$$
The last inequality is due to (a)  $T_{t}(q)$  does not change if at round  $t$  the query is not  $q$ ; (b) if  $q \in \mathcal{L}_{t_{k + 1}(q)}$ , we will have  $q \in \mathcal{L}_t$  for any  $t \in [t_k(q) + 1, t_{k + 1}(q)]$  since  $q$  never arrives in the middle and must remain in the cache set until  $t_{k + 1}(q)$ . Now from event  $E_3$ , we know that
$$
\begin{array}{l} \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q)}}\right) | E \right] \\ \leq \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}) \cdot g _ {k} (q) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q)}}\right) | E \right] \\ \leq \frac {B _ {2} | \mathcal {Q} | \log (3 T L / \delta)}{B _ {1}} \cdot \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q)}}\right) \mid E \right] \\ \end{array}
$$
We know that  $T_{t_{k+1}(q)+1}(q) = T_{t_{k+1}(q)}(q) + 1 = T_{t_k(q)+1}(q) + 1$  if  $q \notin \mathcal{L}_{t_{k+1}(q)}$  since the query  $q$  missing the cache will be sent to the model. Thus overall, we know that we have either  $T_{t_{k+1}(q)+1}(q) = T_{t_k(q)+1}(q) + 1$ , or  $\mathbb{1}(q \notin \mathcal{L}_{t_{k+1}(q)}) \cdot \sqrt{\frac{1}{T_{t_k(q)+1}(q)}} = 0$  and  $T_{t_{k+1}(q)+1}(q) = T_{t_k(q)+1}(q)$ . Thus overall, we have
$$
\sum_ {k = 0} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q)}}\right) | E \right] \leq \sum_ {k = 1} ^ {T} \frac {1}{\sqrt {k}} \leq C \sqrt {T}.
$$
By taking  $\delta = 1 / T$ , we know the final regret can be bounded by
$$
\operatorname {R e g r e t} (T) \leq \frac {C L \left(B _ {2} - B _ {1}\right) B _ {2} | \mathcal {Q} | L \log^ {2} (T | \mathcal {Q} |)}{B _ {1}} \cdot \sqrt {T}.
$$
Lower bound. Now we turn to the lower bound. We apply Le Cam's two point lemma for the regret. Consider any family of algorithm  $\{\mathcal{L}_t\}_{t=1}^T$ , where  $\mathcal{L}_t$  can be dependent on observations prior to time step  $t$ . We aim to design two instances with the same  $P(q)$  and different random variable  $C_l(q)$  such that for any algorithm, the incurred cost for one of the instance is at least  $\Omega(\sqrt{T})$ . Consider the case when we only have two candidate queries  $\mathcal{Q} = \{q_1, q_2\}$ . Set  $P(q_1) = P(q_2) = 1/2$  for both instances and the cache size  $L = 1$ . For instance one, we let  $C_l^{(1)}(q_1) \sim \mathrm{Bern}(1/2)$ ,  $C_l^{(1)}(q_2) \sim \mathrm{Bern}(1/2 + \Delta)$ . For instance two, we let  $C_l^{(2)}(q_1) \sim \mathrm{Bern}(1/2)$ ,  $C_l^{(2)}(q_2) \sim \mathrm{Bern}(1/2 - \Delta)$ . We have
$$
\begin{array}{l} \inf_{\{\mathcal{L}_{t}\}_{t = 1}^{T}}\sup_{P,C_{l}}\operatorname {Regret}(T)\geq \inf_{\{\mathcal{L}_{t}\}_{t = 1}^{T}}\sup_{C_{l}\in \{C_{l}^{(1)},C_{l}^{(2)}\}}\operatorname {Regret}(T) \\ = \inf  _ {\{\mathcal {L} _ {t} \} _ {t = 1} ^ {T}} \sup  _ {C _ {l} \in \{C _ {l} ^ {(1)}, C _ {l} ^ {(2)} \}} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \frac {1}{2} \sum_ {i = 1} ^ {2} \mathbb {1} (q _ {i} \notin \mathcal {L} _ {t}) c _ {l} ^ {\star} (q _ {i}) - \frac {1}{2} \sum_ {i = 1} ^ {2} \mathbb {1} (q _ {i} \notin \mathcal {L} ^ {\star}) c _ {l} ^ {\star} (q _ {i}) \right]. \\ \end{array}
$$
Let  $\operatorname{Regret}^{(1)}(T)$  be the total regret when  $C_l = C_l^{(1)}$ , and  $\operatorname{Regret}^{(2)}(T)$  be the total regret when  $C_l = C_l^{(2)}$ . Then we can verify that for any sequence of  $\mathcal{L}_t$ ,
$$
\operatorname {R e g r e t} ^ {(1)} (T) + \operatorname {R e g r e t} ^ {(2)} (T) \geq \frac {\Delta T}{2}.
$$
Thus from Le Cam's Lemma, we have
$$
\begin{array}{l} \inf_{\{\mathcal{L}_{t}\}_{t = 1}^{T}}\sup_{P,C_{l}}\operatorname {Regret}(T)\geq \frac{\Delta T}{4}\cdot (1 - \operatorname {TV}(\mathbb{P}_{c_{l}^{(1)}},\mathbb{P}_{c_{l}^{(2)}})) \\ \geq \frac {\Delta T}{8} \cdot \exp \left(- D _ {K L} \left(\mathbb {P} _ {c _ {l} ^ {(1)}}, \mathbb {P} _ {c _ {l} ^ {(2)}}\right)\right) \\ \geq \frac {\Delta T}{8} \cdot \exp \left(- 2 \Delta^ {2} \mathbb {E} _ {1} \left[ T _ {2} \right]\right). \\ \end{array}
$$
Here  $\mathbb{E}_1[T_2]$  is the expected times of observing the cost of  $q_{2}$  under instance one. Taking  $\Delta = T^{-1 / 2}$  and minimizing the above equation with  $\mathbb{E}_1[T_2]$  gives the desired bound.
G Proof of Theorem 3
Proof. We define the following four events:
$$
E _ {1} = \left\{\forall q \in \mathcal {Q}, | \hat {P} (q) - P (q) | \leq \sqrt {\frac {2 \log (8 / \delta)}{N}} \right\},
$$
$$
E _ {2} = \left\{\forall q \in \mathcal {Q}, | \hat {c} _ {l} (q) - c _ {l} ^ {\star} (q) | \leq \left(B _ {2} - B _ {1}\right) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} \left(q _ {n} = q\right)}} \right\},
$$
$$
E _ {3} = \left\{\forall q \in \mathcal {Q}, | \hat {c} _ {s} (q) - c _ {s} ^ {\star} (q) | \leq \left(B _ {2} - B _ {1}\right) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} \left(q _ {n} = q\right)}} \right\},
$$
$$
E _ {4} = \left\{\forall q \in \mathcal {L} ^ {\star}, \sum_ {n = 1} ^ {N} \mathbb {1} \left(q _ {n} = q\right) \geq N \cdot P (q) / 2 \right\}.
$$
We know that the above events hold simultaneously with probability at least  $1 - \delta$  from Lemma 3. We condition on the four events from now on. We first decompose the cost difference as
$$
\operatorname {c o s t} \left(\hat {\mathcal {L}}, \hat {\pi}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) = \operatorname {c o s t} \left(\hat {\mathcal {L}}, \hat {\pi}\right) - \operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) + \operatorname {c o s t} \left(\hat {\mathcal {L}}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right).
$$
The first difference can be further written as