SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This concludes our description of the algorithm. In the remainder of this section, we will show that for appropriate settings of the parameters  $\eta$ ,  $\gamma$ , and  $L$ , Algorithm 1 achieves  $\widetilde{O}(\sqrt{KT})$  regret (the parameter  $\iota$  in the following theorem is a parameter solely of the analysis and determines the failure probabilities of various concentration inequalities).
Algorithm 1 Contextual cross-learning algorithm for the unknown distribution setting.
Input: Parameters  $\eta, \gamma > 0$  and  $L < T$ .  
 $\widehat{f}_2 \gets 0$   
for  $t = 1, \dots, L$  do  
Observe  $c_t$   
Play  $A_t \sim s_{1,c_t}$ $\widehat{f}_2 \gets \widehat{f}_2 + \frac{s_{2,c_t}}{2L}$   
for  $e = 2, \dots, T/L$  do  
 $\widehat{f}_{e+1} \gets 0$   
for  $t = (e-1)L+1, t=(e-1)L+3, \dots, eL-1$  do  
Set  $p_{t,c}=\arg\min_{x \in \Delta([K])} \left( \left\langle x, \sum_{s=1}^{t-1} \widehat{\ell}_s(c) \right\rangle - \eta^{-1} F(x) \right)$   
for  $t' = t, t+1$  do  
Observe  $c_{t'}$   
if  $p_{t,c_{t'},k} \geq s_{e,c_{t'},k}/2$  for all  $k \in [K]$  then  
 $\begin{array}{l} \text{Set } q_{t',c_{t'}} = p_{t,c_{t'}} \\ \text{else} \\ \text{Set } q_{t',c_{t'}} = s_{e,c_{t'}} \\ \text{Play } A_{t'} \sim q_{t',c_{t'}} \\ \text{Observe } \ell_{t',A_{t'}} \end{array}$ $t_f, t_\ell \gets \text{RandPerm}(t,t+1)$ $\widehat{f}_{e+1} \gets \widehat{f}_{e+1} + \frac{s_{e+1,c_t}}{2(L/2)}$   
Sample  $S_t \sim \mathcal{B}\left(\frac{s_{e,c_t,\ell}, A_{t_\ell}}{2q_t, c_{t_\ell}, A_{t_\ell}}\right)$   
Set  $\widehat{\ell}_{t_\ell,k,c} = \frac{2\ell_{t_\ell,k,c}}{\widehat{f}_{e,k} + \frac{3}{2}\gamma} \mathbb{I}(A_t = k, S_t = 1)$ $s_{e+2} \gets p_t$
Theorem 1. For any  $\eta \leq \frac{\gamma}{2(2L\gamma + \iota)}$ ,  $\gamma \geq \frac{16\iota}{L}$ ,  $\iota \geq \log (8K / \gamma)$ , Algorithm 1 satisfies
$$
\mathrm {R e g} = O \left(\left(\gamma + \frac {\iota}{L} + \frac {\gamma^ {2} L}{\iota} + \eta + \exp (- \iota) \frac {T}{L}\right) K T + \frac {\log (K)}{\eta} + L\right).
$$
Tuning  $\iota = 2\log (8KT),L = \sqrt{\frac{\iota KT}{\log(K)}} = \widetilde{\Theta} (\sqrt{KT}),\gamma = \frac{16\iota}{L} = \widetilde{\Theta} (1 / \sqrt{KT})$  , and  $\eta = \frac{\gamma}{2(2L\gamma + \iota)} =$ $\widetilde{\Theta} (1 / \sqrt{KT})$  yields a regret bound of
$$
\operatorname {R e g} = \widetilde {O} \left(\sqrt {K T}\right).
$$
Computational efficiency. In general, the computational complexity is  $\min\{tK, |\mathcal{C}|\}$  and the memory complexity  $\min\{t, |\mathcal{C}|K\}$ , where the agent is either keeping a table of all  $K \times |\mathcal{C}|$  losses in memory, which are updated for all contexts in every round, or the agent keeps all previous loss functions in memory and recomputes the losses of all actions for the context they observe. (This is assuming that we can store and evaluate the loss function with  $O(1)$  memory and compute.) In special cases, this can be significantly more efficient. In both sleeping bandits as well as bidding in first-price auctions, we can store the accumulated loss functions in  $O(1)$ , which means that we have a total per-step runtime and memory complexity of  $O(K)$ . This is on par with the  $O(T^{2/3})$  regret algorithm of Balseiro et al. [2019], which also has a per-step runtime and memory complexity of  $O(K)$ .
3.2 Analysis overview
We begin with a high-level overview of the analysis. Fix any  $\pi :[C]\to [K]$ , and let for each  $c\in \mathcal{C}$  let  $u_{c} = e_{k}\in \Delta ([K])$ . We can then write the regret induced by this policy  $\pi$  in the form
$$
\operatorname {R e g} (u) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q _ {t, c _ {t}} - u _ {c _ {t}}, \ell_ {t, c _ {t}} \right\rangle \right]. \tag {2}
$$
We would like to upper bound this quantity (for an arbitrary  $u$ ). To do so, we would like to relate it to  $\mathbb{E}\left[\sum_{t = 1}^{T}\langle p_{t,c_t} - u_{c_t},\widehat{\ell}_{t,c_t}\rangle \right]$ , which we can bound through the guarantees of FTRL. To do so, we will introduce two new proxy random variables  $\widetilde{\ell}_{tc}\in \mathbb{R}^K$  and  $\widetilde{p}_{tc}\in \Delta ([K])$  which have the property that they are independent of  $\widehat{f}_e$  conditioned on the snapshot at the end of epoch  $e - 2$ .
Specifically, recall that  $s_e$  is the snapshot determined at the end of  $e - 2$ . Then, conditioned on  $s_e$  (and in particular, writing  $\mathbb{E}_e[\cdot]$  to denote  $\mathbb{E}[\cdot \mid s_e]$ ), we define:
-  $f_{e} = \mathbb{E}_{c\sim \nu}[s_{ec} / 2]$ . Note that the  $\widehat{f_e}$  used by Algorithm 1 is an unbiased estimator of  $f_{e}$ .
-  $\beta_{ek} = \frac{f_{ek} + \gamma}{\widehat{f}_{ek} + \frac{3}{2}\gamma}$  is a deterministic function of  $\widehat{f}_{ek}$ .
- For each  $t \in \mathcal{T}_e$ , we let  $\widetilde{\ell}_{tck} = \frac{\widetilde{\ell}_{tck}}{\beta_{ek}} = \frac{2\ell_{tck}}{f_{ek} + \gamma}\mathbb{I}\left(A_t = k \wedge S_t \wedge t \in \mathcal{T}_e^\ell\right)$ .  $\widetilde{\ell}_{tck}$  is a loss estimator independent of  $\widehat{f}_e$  such that  $\mathbb{E}_e[\widetilde{\ell}_{tck}] = \frac{f_{ek}}{f_{ek} + \gamma}\ell_{tck} \leq 1$ . Since  $f_{ek}$  is a deterministic function of  $s_e$ ,  $\widetilde{\ell}$  is independent of  $\widehat{f}_{ek}$  conditioned on  $s_e$ .
- For each  $t \in \mathcal{T}_e$ , we let  $\widetilde{p}_{tc} = \arg \min_{x \in \Delta([K])} \left\langle x, \sum_{e' = 1}^{e - 1} \sum_{s \in \mathcal{T}_{e'}} \widehat{\ell}_{sc} + \sum_{t' \in \mathcal{T}_e, t' < t} \widetilde{\ell}_{t'c} \right\rangle - \eta^{-1} F(x) \propto s_{e + 1,c} \circ \exp(-\eta \sum_{t' \in \mathcal{T}_e, t' < t} \widetilde{\ell}_{t'c})$ .  $\widetilde{p}$  can be thought of as the action played by an FTRL algorithm which consumes the loss estimators  $\widehat{\ell}$  up through epoch  $e - 1$ , but uses our new pseudo-estimators  $\widetilde{\ell}$  during epoch  $e$ . Like  $\widetilde{\ell}$ ,  $\widetilde{p}$  is independent of  $\widehat{f}_{ek}$  conditioned on  $s_e$ .
We perform all our analysis conditioned on the following two events occurring with high probability. First, that our context frequency estimators  $\widehat{f}_{ek}$  concentrate - i.e., that  $|\widehat{f}_{ek} - f_{ek}|$  is small w.h.p. Second, that our loss proxies  $\widetilde{\ell}$  concentrate in aggregate over epochs, i.e. that  $\sum_{t\in \mathcal{T}_e}\widetilde{\ell}_{tck}$  is never too large. Both conditions together are sufficient to guarantee that the aggregation of  $\sum_{t\in \mathcal{T}_e}\widehat{\ell}_{tck}$  is also never too large, which is crucial in guaranteeing  $q_{t} = p_{t}$ .
Conditioned on these two concentration events holding, we can strongly bound many of the quantities. Most notably, we can show that, with high probability,  $q_{t,c_t} = p_{t,c_t}$  for all rounds  $t$ . This allows us to replace the  $q_{t,c_t}$  terms in (2) with  $p_{t,c_t}$ . We then split  $\operatorname{Reg}(u)$  into four terms and label them as follows:
$$
\begin{array}{l} \operatorname {R e g} (u) = \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle p _ {t , c _ {t}} - u _ {c _ {t}} , \ell_ {t , c _ {t}} - \widetilde {\ell} _ {t , c _ {t}} \right\rangle \right]} _ {\textbf {b i a s} _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \widetilde {p} _ {t , c _ {t}} - u _ {c _ {t}} , \widetilde {\ell} _ {t , c _ {t}} - \widehat {\ell} _ {t , c _ {t}} \right\rangle \right]} _ {\textbf {b i a s} _ {2}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle p _ {t , c _ {t}} - \widetilde {p} _ {t , c _ {t}} , \widetilde {\ell} _ {t , c _ {t}} - \widehat {\ell} _ {t , c _ {t}} \right\rangle \right]} _ {\mathrm {b i a s} _ {3}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle p _ {t , c _ {t}} - u _ {c _ {t}} , \widehat {\ell} _ {t , c _ {t}} \right\rangle \right]} _ {\mathrm {f t r l}}. \\ \end{array}
$$
We then show (again, subject to these concentration bounds holding) that each of these terms is at most  $\widetilde{O}(\sqrt{KT})$ . Very briefly, this is for the following reasons:
- bias $_1$  and bias $_2$ : Here we use the independence structure we introduce by defining  $\widetilde{\ell}$  and  $\widetilde{p}$  (along with scheduling the different pieces of the algorithm across different epochs). For example, conditioned on  $s_e$ , we know that  $\widetilde{\ell}_t$  is independent of  $p_t$ , so we can safely replace
the  $\widetilde{\ell}_t$  in  $\mathbf{bias}_1$  with its expectation. Similarly,  $\widetilde{p}_t$  is independent of both  $\widetilde{\ell}_t$  and  $\widehat{\ell}_t$  conditioned on  $s_e$ .
- bias<sub>3</sub>: Here we do not have independence between the two sides of the inner product. But fortunately, we can directly bound the magnitude of the summands in this case, since we can show that  $|\widetilde{p}_{tc} - p_{tc}|$  and  $|\widetilde{\ell}_{tc} - \widehat{\ell}_{tc}|$  are both small with high probability (in fact, for all  $c \in \mathcal{C}$  simultaneously).
- fTRL: Finally, this term is bounded from the standard analysis of FTRL.
The full proof of Theorem 1 can be found in the Appendix (Section C) of the Supplementary Material.
4.1 Bidding in first-price auctions with unknown value distribution
We formally model the first-price auction bidding problem as follows. A bidder participates in  $T$  repeated auctions. In auction  $t$ , they have a value  $v_{t} \in [0,1]$  for the current item being sold, with  $v_{t}$  being drawn i.i.d. from some unknown distribution  $\nu$  supported on  $[0,1]$ . They submit a bid  $b_{t} \in [0,1]$  into the auction. We let  $m_{t} \in [0,1]$  denote the highest other bid of any other bidder into this auction (and allow the adversary to choose the sequence of  $m_{t}$  obliviously in advance). If  $b_{t} \geq m_{t}$ , the bidder wins the auction and receives the item (and net utility  $v_{t} - b_{t}$ ); otherwise, the bidder loses the auction and receives nothing. In both cases, the bidder only observes the binary feedback of whether or not they won the item – they do not observe the other bids or  $m_{t}$ .
The bidder would like to minimize their regret with respect to the best fixed mapping  $b^{*}:[0,1]\to [0,1]$  from values to bids, i.e.,
$$
\operatorname {R e g} = \max  _ {b ^ {*}} \sum_ {t = 1} ^ {T} \left(v _ {t} - b ^ {*} \left(v _ {t}\right)\right) \mathbb {I} \left(b ^ {*} \left(v _ {t}\right) \geq m _ {t}\right) - \sum_ {t = 1} ^ {T} \left(v _ {t} - b _ {t}\right) \mathbb {I} \left(b _ {t} \geq m _ {t}\right).
$$
In [Balseiro et al., 2019], the authors prove a lower bound of  $\Omega (T^{2 / 3})$  for this problem (based on a related pricing lower bound of Kleinberg and Leighton [2003]). By applying their algorithm for cross-learning between contexts, they show that it is possible to match this in the case where the buyer knows their value distribution  $\nu$ , but only achieve an upper bound of  $\widetilde{O} (T^{3 / 4})$  in the unknown distribution case. By applying Algorithm 1, we show that it is possible to achieve a regret bound of  $\widetilde{O} (T^{2 / 3})$  in this setting, nearly (up to logarithmic factors in  $T$ ) matching the lower bound.
Corollary 1. There exists an efficient learning algorithm that achieves a regret bound of  $\widetilde{O}(T^{2/3})$  for the problem of learning to bid in a first-price auction with an unknown value distribution.
Proof. Let  $K = T^{1/3}$ . We first discretize the set of possible bids to multiples of  $1 / K = T^{-1/3}$ . Note that this increases the overall regret by at most  $T / K = T^{2/3}$ ; in particular, if bidding  $b$  results in expected utility  $U$  for a bidder with some fixed value  $v$ , bidding any  $b' > b$  results in utility at least  $u - (b' - b)$ .
Now, we have an instance of the contextual bandits problem with cross-learning where  $\mathcal{C} = [0,1]$ ,  $\nu$  is the distribution over contexts, the arms correspond to the  $K$  possible bids, and  $\ell_{tb}(v) = 1 - (v - b)\mathbb{I}(b\geq m_t)$ . This setting naturally has cross-learning; after bidding into an auction and receiving (or not receiving) the item, the agent can figure out what net utility they would have received under any value they could possibly have for the item. From Theorem 1, this implies there is an algorithm which gets  $\widetilde{O} (\sqrt{TK}) = \widetilde{O} (T^{2 / 3})$  regret.
4.2 Sleeping bandits with stochastic action set
In the sleeping bandits problem, there are  $K$  arms. Each round  $t$  (for  $T$  rounds), a non-empty subset  $S_{t} \subseteq [K]$  of the arms is declared to be "active". The learner must select one of the active arms  $k \in S_{t}$  upon which they receive some loss  $\ell_{tk}$ . We assume here that the losses are chosen by an oblivious adversary, but the  $S_{t}$  are sampled independently every round from an unknown distribution  $\nu$ . The
learner would like low regret compared to the best fixed policy  $\pi :[2^K ]\to [K]$  mapping  $S_{t}$  to an action  $\pi (S_t)$  to play.
Note that this fits precisely within the contextual bandits with cross-learning framework, where the contexts  $c_{t}$  are the sets  $S_{t}$ , we have non-uniform action sets  $A_{c} = S \subseteq [K]$ , and cross-learning is possible since the loss  $\ell_{tck}$  of arm  $k$  in context  $c$  in round  $t$  does not depend on  $c$  as long as  $k$  belongs to the set corresponding to the context (and if  $k$  does not, we cannot even play  $k$ ).
Corollary 2. There exists an efficient learning algorithm that achieves a regret bound of  $\widetilde{O}(\sqrt{KT})$  for the sleeping bandits problem with stochastic action sets drawn from an unknown distribution.
We resolved the open problem of Balseiro et al. [2019] with respect to optimal cross-context learning when the distribution of contexts is stochastic but unknown. As a side result, we obtained an almost optimal solution for adversarial sleeping bandits with stochastic arm-availabilities. Not only is this algorithm the first to obtain optimal polynomial dependencies in the number of arms and the time horizon, it is also the first computationally efficient algorithm obtaining a reasonable bound. Finally, we closed the gap between upper and lower bounds for bidding in first-price auctions.
Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1), 2002.
Ashwinkumar Badanidiyuru, Zhe Feng, and Guru Guruganesh. Learning to bid in contextual first price auctions. In Proceedings of the ACM Web Conference 2023, WWW '23, page 3489-3497, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/3543507.3583427. URL https://doi.org/10.1145/3543507.3583427.
Santiago Balseiro, Negin Golrezaei, Mohammad Mahdian, Vahab Mirrokni, and Jon Schneider. Contextual bandits with cross-learning. Advances in Neural Information Processing Systems, 32, 2019.
Yuyu Chen. Programmatic advertising is preparing for the first-price auction era. https://digiday.com/marketing/programmatic-advertising-readying-first-price-auction-era, October 2017. Accessed: 2020-01-29.
Yanjun Han, Zhengyuan Zhou, Aaron Flores, Erik Ordentlich, and Tsachy Weissman. Learning to bid optimally and efficiently in adversarial first-price auctions. arXiv preprint arXiv:2007.04568, 2020a.
Yanjun Han, Zhengyuan Zhou, and Tsachy Weissman. Optimal no-regret learning in repeated first-price auctions. arXiv preprint arXiv:2003.09795, 2020b.
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2(3-4):157-325, 2016.
Satyen Kale, Chansoo Lee, and David Pál. Hardness of online sleeping combinatorial optimization problems. Advances in Neural Information Processing Systems, 29, 2016.
Varun Kanade and Thomas Steinke. Learning hurdles for sleeping experts. ACM Transactions on Computation Theory (TOCT), 6(3):1-16, 2014.
Varun Kanade, H Brendan McMahan, and Brent Bryan. Sleeping experts and bandits with stochastic action availability and adversarial rewards. In Artificial Intelligence and Statistics, pages 272-279. PMLR, 2009.
Robert Kleinberg and Tom Leighton. The value of knowing a demand curve: Bounds on regret for online posted-price auctions. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pages 594-605. IEEE, 2003.
Robert Kleinberg, Alexandru Niculescu-Mizil, and Yogeshwer Sharma. Regret bounds for sleeping experts and bandits. Machine learning, 80(2-3):245-272, 2010.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Gergely Neu and Julia Olkhovskaya. Efficient and robust algorithms for adversarial linear contextual bandits. In Conference on Learning Theory, pages 3049-3068. PMLR, 2020.
Gergely Neu and Michal Valko. Online combinatorial optimization with stochastic decision sets and adversarial losses. Advances in Neural Information Processing Systems, 27, 2014.
Aadirupa Saha, Pierre Gaillard, and Michal Valko. Improved sleeping bandits with stochastic action sets and adversarial rewards. In International Conference on Machine Learning, pages 8357-8366. PMLR, 2020.
Wei Zhang, Brendan Kitts, Yanjun Han, Zhengyuan Zhou, Tingyu Mao, Hao He, Shengjun Pan, Aaron Flores, San Gultekin, and Tsachy Weissman. Meow: A space-efficient nonparametric bid shading algorithm. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3928-3936, 2021.
Wei Zhang, Yanjun Han, Zhengyuan Zhou, Aaron Flores, and Tsachy Weissman. Leveraging the hints: Adaptive bidding in repeated first-price auctions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21329-21341. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/86419aba4e5eafd2b1009a2e3c540bb0-Paper-Conference.pdf.
We will make use of following existing results from the literature in our proof of Theorem 1. First, we present two concentration results (one for negatively correlated random variables that show up in the analysis of bandit algorithms, and a slight generalization of the standard Bernstein inequality).
Lemma 2 (Lemma 12.2 [Lattimore and Szeptsvári, 2020]). Let  $\mathbb{F} = (\mathcal{F}_t)_{t=0}^T$  be a filtration and for  $i \in [K]$  let  $(\widetilde{Y}_{ti})_t$  be  $\mathbb{F}$ -adapted such that:
1. for any  $S \subset [k]$  with  $|S| > 1$ ,  $\mathbb{E}\left[\Pi_{i\in S}\widetilde{Y}_{ti} \mid \mathcal{F}_{t - 1}\right] \leq 0$ ; and
2.  $\mathbb{E}\left[\widetilde{Y}_{ti} \mid \mathcal{F}_{t-1}\right] = y_{ti}$  for all  $t \in [T]$  and  $i \in [k]$ .
Furthermore, let  $(\alpha_{ti})_{ti}$  and  $(\lambda_{ti})_{ti}$  be real-valued  $\mathbb{F}$ -predictable random sequences such that for all  $t,i$  it holds that  $0 \leq \alpha_{ti} \widetilde{Y}_{ti} \leq 2\lambda_{ti}$ . Then, for all  $\delta \in (0,1)$ ,
$$
\mathbb {P} \left(\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {K} \alpha_ {t i} \left(\frac {\widetilde {Y} _ {t i}}{1 + \lambda_ {t i}} - y _ {t i}\right) \geq \log \left(\frac {1}{\delta}\right)\right) \leq \delta .
$$
Lemma 3 (Bernstein type inequality [Lattimore and Szepesvári, 2020] exercise 5.15). Let  $X_{1}, X_{2}, \ldots, X_{n}$  be a sequence of random variables adapted to the filtration  $\mathbb{F} = (\mathcal{F}_t)_t$ . Abbreviate  $\mathbb{E}_t[\cdot] = \mathbb{E}[\cdot | \mathcal{F}_t]$  and define  $\mu_t = \mathbb{E}_{t-1}[X_t]$ . If  $\zeta > 0$  and  $\zeta(X_t - \mu_t) \leq 1$  almost surely, then for all  $\delta \in (0,1)$
$$
\mathbb {P} \left(\sum_ {t = 1} ^ {n} (X _ {t} - \mu_ {t}) \geq \zeta \sum_ {t = 1} ^ {n} \mathbb {E} _ {t - 1} [ (X _ {t} - \mu_ {t}) ^ {2} ] + \frac {1}{\zeta} \log (\frac {1}{\delta})\right) \leq \delta .
$$
We will also need the following regret guarantee for FTRL / multiplicative weights.
Lemma 4 ([Hazan et al., 2016], Theorem 1.5). Let  $\ell_1, \ell_2, \ldots, \ell_T$  be a sequence of losses in  $\mathbb{R}_{\geq 0}^K$ , and fix an  $\eta > 0$ . Then, if we let
$$
p _ {t} = \arg \min _ {x \in \Delta ([ K ])} \left\langle x, \sum_ {s = 1} ^ {t - 1} \ell_ {s} \right\rangle - \eta^ {- 1} F (x)
$$
(where  $F(x)$  is the unnormalized neg-entropy function), then we have that
$$
\sum_ {t = 1} ^ {T} \left\langle p _ {t}, \ell_ {t} \right\rangle - \min  _ {x ^ {*} \in \Delta ([ K ])} \sum_ {t = 1} ^ {T} \left\langle x ^ {*}, \ell_ {t} \right\rangle \leq \frac {\log K}{\eta} + \eta \sum_ {t = 1} ^ {T} \left\langle p _ {t}, \ell_ {t} ^ {2} \right\rangle .
$$
Here  $\ell_t^2$  denotes the vector formed by squaring each component of  $\ell_t$ .
Lemma 5. For any  $k \geq 16$ :
$$
f (k) = \sum_ {i = \lfloor k / 1 6 \rfloor} ^ {\lfloor k / 4 \rfloor} \frac {i + 1}{(1 - 2 \sqrt {(i + 1) / k}) _ {+} + 1 6 / (k + 1)} \cdot e ^ {- i} \leq 2.
$$
Proof. For any  $k > 200$ , we have
$$
\sum_ {i = \lfloor k / 1 6 \rfloor} ^ {\lfloor k / 4 \rfloor} \frac {i + 1}{(1 - 2 \sqrt {(i + 1) / k}) _ {+} + 1 6 / (k + 1)} \cdot e ^ {- i} \leq \frac {(k + 1) ^ {3}}{6 4} e ^ {- k / 1 6} <   2,
$$
where this statement can be verified by showing that the derivative is negative for  $k \geq 200$  and numerically computing the value. Finally to show the original claim, it is sufficient to compute the function values for  $k \in [16, 200]$ , which are all below 2.
Proof. Let  $d = \frac{16}{t}$ , then
$$
\mathbb {E} \left[ \frac {1}{\widehat {\mu} + d} \right] = \frac {1}{\mu + d} + \underbrace {\mathbb {E} \left[ \frac {\mu - \widehat {\mu}}{(\mu + d) ^ {2}} \right]} _ {= 0} + \mathbb {E} \left[ \frac {(\mu - \widehat {\mu}) ^ {2}}{(\widehat {\mu} + d) (\mu + d) ^ {2}} \right].
$$
To bound the quadratic term, let  $k = \lfloor \mu / \left( \frac{1}{t} \right) \rfloor$  (we can assume  $k \geq 16$  or the Lemma holds by  $d \geq \mu$ ), then for any  $i \in [0, k]$  by Lemma 3 using  $\zeta = \sqrt{i \frac{1}{\mu t}}$ , we have
$$
\mathbb {P} \left(\widehat {\mu} \leq \mu \left(1 - 2 \sqrt {\frac {i}{k}}\right)\right) \leq \mathbb {P} \left(\widehat {\mu} \leq \mu - 2 \sqrt {\frac {i \mu}{t}}\right) \leq e ^ {- i}.
$$
Decomposing the quadratic terms yields
$$
\begin{array}{l} \mathbb {E} \left[ \frac {(\mu - \widehat {\mu}) ^ {2}}{(\widehat {\mu} + d) (\mu + d) ^ {2}} \right] \leq \mathbb {E} \left[ \mathbb {I} (\widehat {\mu} \geq \mu / 2) \frac {4 (\mu - \widehat {\mu}) ^ {2}}{\mu (\mu + d) ^ {2}} \right] \\ + \sum_ {i = \lfloor k / 1 6 \rfloor} ^ {\lfloor k / 4 \rfloor} \mathbb {E} \left[ \mathbb {I} \left(1 - 2 \sqrt {\frac {i}{k}} \leq \frac {\widehat {\mu}}{\mu} \leq 1 - 2 \sqrt {\frac {i + 1}{k}}\right) \frac {(\mu - \widehat {\mu}) ^ {2}}{(\widehat {\mu} + d) (\mu + d) ^ {2}} \right] \\ \leq \mathbb {E} \left[ \frac {4 (\mu - \widehat {\mu}) ^ {2}}{\mu (\mu + d) ^ {2}} \right] + \frac {4}{k (\mu + d) ^ {2}} \sum_ {i = \lfloor k / 1 6 \rfloor} ^ {\lfloor k / 4 \rfloor} \frac {i + 1}{(1 - 2 \sqrt {(i + 1) / k}) _ {+} + 1 6 / (k + 1)} \cdot e ^ {- i} \\ \leq \frac {4 \mu}{(\mu + d) ^ {2} t} + \frac {8}{k (\mu + d) ^ {2}} \leq \frac {1 6 \mu}{(\mu + d) ^ {2} t}. \tag {Lemma 5} \\ \end{array}
$$
$$
\mathbb {E} \left[ \frac {1}{\widehat {\mu} + d} \right] \leq \frac {1}{\mu + d} + \frac {1 6 \mu}{(\mu + d) ^ {2} t} = \frac {1}{\mu} - \frac {d}{\mu (\mu + d)} + \frac {1 6 \mu / t}{(\mu + d) ^ {2}} \leq \frac {1}{\mu}. (d = 1 6 \mu / t)
$$
C Detailed proof of Theorem 1
C.1 High probability events
We begin our proof by establishing two sequences of events (one per epoch  $e$ ) that will hold with high probability, representing that our estimation of context frequencies and losses both "concentrate" in an appropriate sense.
The first we event we define  $F_{e}$ , represents the event that our estimator  $\widehat{f}_{ek}$  diverges greatly from its expectation  $f_{ek}$ .
Definition 1. Let  $F_{e}$  be the indicator of the event such that at episode  $e$  the following inequality holds for all  $k \in [K]$
$$
| \widehat {f} _ {e k} - f _ {e k} | \leq 2 \max \left\{\sqrt {\frac {f _ {e k} \iota}{L}}, \frac {\iota}{L} \right\}.
$$