SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Conclusions. In this work, we have introduced a novel contextual bandit framework that incorporates post-serving contexts, thereby widening the range of complex real-world challenges it can address. By leveraging historical data, our proposed algorithm, poLinUCB, estimates the functional mapping from pre-serving to post-serving contexts, leading to improved online learning efficiency. For the purpose of theoretical analysis, the elliptical potential lemma has been expanded to manage noise within post-serving contexts, a development which may have wider applicability beyond this particular framework. Extensive empirical tests on synthetic and real-world datasets have demonstrated the significant benefits of utilizing post-serving contexts and the superior performance of our algorithm compared to state-of-the-art approaches.
Limitations. Our theoretical analysis hinges on a crucial assumption that the function  $\phi^{\star}(\cdot)$  is learnable, which may not always be satisfied. This is particularly a concern when the post-serving contexts may hold additional information that cannot be deduced from the pre-serving context, irrespective of the amount of data collected. In such scenarios, the function mapping from the preserving context to the post-serving context may be much more difficult to learn, or even not learnable. Consequently, a linear regret may be inevitable due to model misspecification. However, from a practical point of view, our empirical findings from the real-world MovieLens dataset demonstrate that modeling the functional relationship between the pre-serving and post-serving contexts can still significantly enhance the learning efficiency. We hope our approaches can inform the design of practical algorithms that more effectively utilizes post-serving data in recommendations.
Acknowledgement. This work is supported by an NSF Award CCF-2303372, an Army Research Office Award W911NF-23-1-0030, and an Office of Naval Research Award N00014-23-1-2802.
Hongju Park and Mohamad Kazem Shirani Faradonbeh. Analysis of thompson sampling for partially observable contextual multi-armed bandits. IEEE Control Systems Letters, 6:2150-2155, 2021.
Yi Qi, Qingyun Wu, Hongning Wang, Jie Tang, and Maosong Sun. Bandit learning with implicit feedback. Advances in Neural Information Processing Systems, 31, 2018.
Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395-411, 2010.
Eric M Schwartz, Eric T Bradlow, and Peter S Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4):500-522, 2017.
Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications, pages 495-517, 2017.
Ryan Tibshirani. Nonparametric regression (and classification). 2017.
Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning, 8(1-2):1-230, 2015.
Andre Uschemajew. Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM Journal on Matrix Analysis and Applications, 33(2):639-652, 2012.
Huazheng Wang, Qingyun Wu, and Hongning Wang. Learning hidden features for contextual bandits. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 1633-1642, 2016.
Huazheng Wang, Haifeng Xu, and Hongning Wang. When are linear stochastic bandits attackable? In International Conference on Machine Learning, pages 23254-23273. PMLR, 2022.
Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 529-538, 2016.
Jianyi Yang and Shaolei Ren. Robust bandit learning with imperfect context. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10594-10602, 2021.
Shangdong Yang, Hao Wang, Chenyu Zhang, and Yang Gao. Contextual bandits with hidden features to online recommendation via sparse interactions. IEEE Intelligent Systems, 35(5):62-72, 2020.
Yun Yang and David B Dunson. Bayesian manifold regression. 2016.
Fan Yao, Chuanhao Li, Denis Nekipelov, Hongning Wang, and Haifeng Xu. How bad is top-k recommendation under competing content creators? International Conference on Machine Learning, 2023.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pages 11492-11502. PMLR, 2020.
Rong Zhu and Branislav Kveton. Robust contextual linear bandits. arXiv preprint arXiv:2210.14483, 2022.
A Algorithm and Regret Analysis of LinUCB(φ)
We present the details of the algorithm described in Section 3.2 and the proof of the regret bound.
A.1 Main Algorithm
Parameter learning. We consider solving the following regularized least squared problem for estimating  $\{\widehat{\pmb{\theta}}_{t,a}\}_{a\in \mathcal{A}}$  and  $\{\widehat{\beta}_{t,a}\}_{a\in \mathcal{A}}$  for each arm  $a$ :
$$
\ell_ {t} \left(\boldsymbol {\theta} _ {a}, \boldsymbol {\beta} _ {a}\right) = \sum_ {s: a _ {s} = a} ^ {t} \left(r _ {s, a} - \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {a} - \widehat {\phi} _ {s} \left(\boldsymbol {x} _ {s}\right) ^ {\top} \boldsymbol {\beta} _ {a}\right) ^ {2} + \lambda \left(\left\| \boldsymbol {\theta} _ {a} \right\| _ {2} ^ {2} + \left\| \boldsymbol {\beta} _ {a} \right\| _ {2} ^ {2}\right), \tag {10}
$$
where  $\lambda \geq 0$  are penalty factors ensuring the uniqueness of minimizers  $\widehat{\pmb{\theta}}_{t,a}$  and  $\widehat{\beta}_{t,a}$ .
In the same convention, we use  $\pmb{w}$  to denote  $(\theta ,\beta)$ , and  $\pmb{u}$  to denote  $\left(\pmb {x},\widehat{\phi} (\pmb {x})\right)$ . The closed-form solutions for  $\widehat{\pmb{\theta}}_{t,a}$  and  $\widehat{\beta}_{t,a}$  in this least squared problem then become:
$$
\widehat {\boldsymbol {w}} _ {t, a} := \left[ \begin{array}{c} \widehat {\boldsymbol {\theta}} _ {t, a} \\ \widehat {\boldsymbol {\beta}} _ {t, a} \end{array} \right] = \boldsymbol {A} _ {t, a} ^ {- 1} \boldsymbol {b} _ {t, a},
$$
where we reload the notations of  $A_{t,a}$  and  $\pmb{b}_{t,a}$
$$
\boldsymbol {A} _ {t, a} = \lambda \boldsymbol {I} + \sum_ {s: a _ {s} = a} ^ {t} \boldsymbol {u} _ {s} \boldsymbol {u} _ {s} ^ {\top} \quad \text {a n d} \quad \boldsymbol {b} _ {t, a} = \sum_ {s: a _ {s} = a} ^ {t} r _ {s, a} \boldsymbol {u} _ {s}. \tag {11}
$$
Confidence set construction. At iteration  $t$ , we construct the confidence set for  $\widehat{\phi}_t(\pmb{x}_t)$  by
$$
\mathcal {C} _ {t} \left(\widehat {\phi} _ {t}, \boldsymbol {x} _ {t}\right) := \left\{\boldsymbol {z} \in \mathbb {R} ^ {d}: \left\| \widehat {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| _ {2} \leq e _ {t} ^ {\delta} \right\}. \tag {12}
$$
The construction of the confidence set for  $\widehat{\pmb{w}}_{t,a}$  will be different, as we are using the predicted value  $\widehat{\phi}_t(\cdot)$  for linear regression. Consider the following,
$$
\begin{array}{l} \boldsymbol {A} _ {t, a} \left(\left[ \begin{array}{c} \widehat {\boldsymbol {\theta}} _ {t, a} \\ \widehat {\boldsymbol {\beta}} _ {t, a} \end{array} \right] - \left[ \begin{array}{c} \boldsymbol {\theta} _ {a} ^ {\star} \\ \boldsymbol {\beta} _ {a} ^ {\star} \end{array} \right]\right) = \underbrace {\sum_ {s : a _ {s} = a} ^ {t} \left(\boldsymbol {\epsilon} _ {s} ^ {\top} \boldsymbol {\beta} _ {a} ^ {\star} \left[ \begin{array}{c} \boldsymbol {x} _ {s} \\ \widehat {\phi} _ {s} (\boldsymbol {x} _ {s}) \end{array} \right]\right)} _ {①} + \underbrace {\sum_ {s : a _ {s} = a} ^ {t} \left(\left(\phi^ {\star} (\boldsymbol {x} _ {s}) - \widehat {\phi} _ {s} (\boldsymbol {x} _ {s})\right) ^ {\top} \boldsymbol {\beta} _ {a} ^ {\star} \left[ \begin{array}{c} \boldsymbol {x} _ {s} \\ \widehat {\phi} _ {s} (\boldsymbol {x} _ {s}) \end{array} \right]\right)} _ {②} \\ + \underbrace {\sum_ {s : a _ {s} = a} ^ {t} \eta_ {s} \left[ \begin{array}{c} \boldsymbol {x} _ {s} \\ \widehat {\phi} _ {s} (\boldsymbol {x} _ {s}) \end{array} \right]} _ {\text {(3)}} - \underbrace {\lambda \left[ \begin{array}{c} \boldsymbol {\theta} _ {a} ^ {\star} \\ \boldsymbol {\beta} _ {a} ^ {\star} \end{array} \right]} _ {\text {(4)}} \\ \end{array}
$$
Therefore, the confidence set will be enlarged due to the error introduced by  $\widehat{\phi}_t(\cdot)$ . In the below, we derive the confidence set. To build the confidence set, we need to bound
$$
\| \widehat {\boldsymbol {w}} _ {t, a} - \boldsymbol {w} _ {t, a} ^ {\star} \| _ {\boldsymbol {A} _ {t, a}} = \left\| \left[ \begin{array}{c} \widehat {\boldsymbol {\theta}} _ {t, a} \\ \widehat {\boldsymbol {\beta}} _ {t, a} \end{array} \right] - \left[ \begin{array}{c} \boldsymbol {\theta} _ {a} ^ {\star} \\ \boldsymbol {\beta} _ {a} ^ {\star} \end{array} \right] \right\| _ {\boldsymbol {A} _ {t, a}} = \left\| \widehat {①} + \widehat {②} + \widehat {③} + \widehat {④} \right\| _ {\boldsymbol {A} _ {t, a} ^ {- 1}}
$$
Since both  $\{\pmb {\epsilon}_s\}_{s = 1}^t$  and  $\{\eta_s\}_{s = 1}^t$  are i.i.d sub-Gaussian random variables, respectively, we can use the self-normalized inequality to bound the corresponding terms, i.e., the followings hold with probability at least  $1 - 2\delta$
$$
\begin{array}{l} \| \widehat {\boldsymbol {w}} _ {t, a} - \boldsymbol {w} _ {t, a} ^ {\star} \| _ {\boldsymbol {A} _ {t, a}} \leq \sqrt {2 \left(L _ {\epsilon} ^ {2} + R _ {\eta} ^ {2}\right) \log \left(\frac {\det  (\boldsymbol {A} _ {t , a}) ^ {1 / 2} \det  (\lambda \boldsymbol {I}) ^ {- 1 / 2}}{\delta / 2}\right)} + \frac {L _ {u}}{\sqrt {\lambda}} \left(\sum_ {s = 1} ^ {t} e _ {s} ^ {\delta / t}\right) + 2 \sqrt {\lambda} \\ \leq \sqrt {2 (L _ {\epsilon} ^ {2} + R _ {\eta} ^ {2}) \log \left(\frac {1 + n _ {t} (a) L _ {u} ^ {2} / \lambda}{\delta / 2}\right)} + \frac {L _ {u}}{\sqrt {\lambda}} \left(\sum_ {s = 1} ^ {t} e _ {s} ^ {\delta / t}\right) + 2 \sqrt {\lambda} \\ \end{array}
$$
Algorithm 2 LinUCB-  $(\widehat{\phi})$  (Linear UCB adapted from Wang et al. (2016) with post-serving contexts; The differences with Algorithm 1 are highlighted in blue color.)
1: for  $t = 0,1,\ldots ,T$  do
2: Receive the pre-serving context  $\mathbf{x}_t$
3: Compute the optimistic parameters by maximizing the UCB objective
5: Compute the estimated post-serving context generating function  $\widehat{\phi}_t(\cdot)$  using ERM.
6: Compute  $\widehat{\pmb{w}}_{t,a}$  by solving Equation 10 for each  $a$ .
7: Update confidence sets  $\mathcal{C}_t(\widehat{\boldsymbol{w}}_{t,a})$  and  $\mathcal{C}_t(\widehat{\phi}_t,\boldsymbol{x}_t)$  for each  $a$  based on Equations 12 and 5.
8: end for
Therefore, the confidence set is
$$
\mathcal {C} _ {t, a} \left(\widehat {\boldsymbol {w}} _ {t, a}\right) = \left\{\boldsymbol {w} \in \mathbb {R} ^ {d _ {u}}: \| \boldsymbol {w} - \widehat {\boldsymbol {w}} _ {t, a} \| _ {\boldsymbol {A} _ {t, a}} \leq \zeta_ {t, a} \right\}, \tag {13}
$$
where  $\zeta_{t,a} = \sqrt{2(L_{\epsilon}^{2} + R_{\eta}^{2})\log((1 + n_{t}(a)L_{u}^{2} / \lambda) / (\delta / 2))} + L_{u}\left(\sum_{s = 1}^{t}e_{s}^{\delta / t}\right) / \sqrt{\lambda} + 2\sqrt{\lambda}$ . In comparison to the original confidence set, there is one additional term due to the generalization error introduced from  $\widehat{\phi}_s(\cdot)$ . In the next section, we will provide a regret analysis, which following from the proof of LinUCB (Li et al., 2010). We simply have the following
$$
\begin{array}{l} R _ {T} = \sum_ {t = 1} ^ {T} \left(r _ {t, a _ {t} ^ {\star}} - r _ {t, a _ {t}}\right) = \sum_ {t = 1} ^ {T} \Delta_ {t} \leq \sqrt {T \sum_ {t = 1} ^ {T} \Delta_ {t} ^ {2}} \\ \leq \sqrt {T \sum_ {t = 1} ^ {T} \left(\left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| \left\| \widetilde {\boldsymbol {\beta}} _ {a _ {t}} \right\| + \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}} ^ {- 1}} \left\| \left[ \begin{array}{c} \widetilde {\boldsymbol {\theta}} _ {a _ {t}} - \boldsymbol {\theta} _ {a _ {t}} \\ \widetilde {\boldsymbol {\beta}} _ {a _ {t}} - \boldsymbol {\beta} _ {a _ {t}} \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}}}\right) ^ {2}} \\ \leq \sqrt {T \left(\sum_ {t = 1} ^ {T} 2 \left\| \widetilde {\phi_ {t}} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| ^ {2} \left\| \widetilde {\boldsymbol {\beta}} _ {a _ {t}} \right\| ^ {2} + 2 \zeta_ {T , a} ^ {2} \left(1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}}} ^ {2}\right)\right)} \\ \leq \sqrt {T \left(\sum_ {t = 1} ^ {T} 2 \left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| ^ {2} \left\| \widetilde {\boldsymbol {\beta}} _ {a _ {t}} \right\| ^ {2} + 2 \zeta_ {T , a} ^ {2} \left(1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}}} ^ {2}\right)\right)} \\ \leq \sqrt {T \cdot \left(8 C _ {0} T ^ {1 - 2 \alpha} \log^ {2 \alpha} \left(\frac {\operatorname* {d e t} X _ {t}}{\operatorname* {d e t} X _ {0}}\right) \log^ {2} \left(\frac {T}{\delta}\right) + 2 K \zeta_ {T} ^ {2} d _ {u} \log \left(1 + \frac {T L _ {u} ^ {2}}{\lambda d _ {u}}\right)\right)} \\ \end{array}
$$
In the next, we expand the term  $\zeta_T^2$
$$
\begin{array}{l} \zeta_ {T} ^ {2} \leq \left(\sqrt {2 \left(L _ {\epsilon} ^ {2} + R _ {\eta} ^ {2}\right) \log \left(\frac {1 + T L _ {u} ^ {2} / \lambda}{\delta / 2}\right)} + \frac {L _ {u} \left(\sum_ {s = 1} ^ {T} e _ {s} ^ {\delta / T}\right)}{\sqrt {\lambda}} + 2 \sqrt {\lambda}\right) ^ {2} \\ \leq 6 (L _ {\epsilon} ^ {2} + R _ {\eta} ^ {2}) \log \left(\frac {1 + T L _ {u} ^ {2} / \lambda}{\delta / 2}\right) + \frac {3 L _ {u} ^ {2}}{\lambda} \left(\sum_ {s = 1} ^ {T} e _ {s} ^ {\delta / T}\right) ^ {2} + 1 2 \lambda \\ \end{array}
$$
In the next, we bound the second term in the above equation under the learnability assumption 1,
$$
\left(\sum_ {s = 1} ^ {T} e _ {s} ^ {\delta / T}\right) ^ {2} \leq 1 6 T ^ {2 - 2 \alpha} \log^ {2 \alpha} \left(\frac {\det  X _ {T}}{\det  X _ {0}}\right) \log^ {2} \left(\frac {T}{\delta}\right).
$$
Therefore, naively choosing the value of  $\lambda$  will lead to a linear regret due to the term  $T^{3/2 - \alpha}$  in the equation. To minimize the upper bound, we can choose the value of  $\lambda$  to be
$$
\lambda = 2 L _ {u} T ^ {1 - \alpha} \log^ {\alpha} \left(\frac {\det \boldsymbol {X} _ {T}}{\det \boldsymbol {X} _ {0}}\right) \log \left(\frac {T}{\delta}\right).
$$
Then, we can bound  $\zeta_T^2$  by
$$
\zeta_ {T} ^ {2} \leq 6 (L _ {\epsilon} ^ {2} + R _ {\eta} ^ {2}) \log \left(\frac {1 + T L _ {u} ^ {2} / \lambda}{\delta / 2}\right) + 4 8 L _ {u} T ^ {1 - \alpha} \log^ {\alpha} \left(\frac {\det X _ {T}}{\det X _ {0}}\right) \log \left(\frac {T}{\delta}\right).
$$
By plugging it in and following the simplication as in the proof of Theorem 1, we can get the regret is upper bounded by
$$
\widetilde {\mathcal {O}} \left(T ^ {1 - \alpha} d _ {u} ^ {\alpha} + T ^ {1 - \alpha / 2} \sqrt {K d _ {u} ^ {1 + \alpha}}\right).
$$
The above result is summarized as the following proposition.
Proposition 1 (Regret of LinUCB-( $\widehat{\phi}$ )). The regret of LinUCB-( $\widehat{\phi}$ ) in Algorithm 2 is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha +T^{1 - \alpha /2}\sqrt{Kd_u^{1 + \alpha}}\right)$  with probability at least  $1 - \delta$ , by carefully setting the regularization coefficient  $\lambda = \Theta (L_{u}d_{u}^{\alpha}T^{1 - \alpha}\log (T / \delta))$  in Equation 1.
B Missing Proofs
B.1 Missing Proofs in the Generalized Elliptical Potential Lemma
Lemma 1 (Generalized Elliptical Potential Lemma). Suppose (1)  $\mathbf{X}_0 \in \mathbb{R}^{d \times d}$  is any positive definite matrix; (2)  $\mathbf{x}_1, \ldots, \mathbf{x}_T \in \mathbb{R}^d$  is a sequence of vectors with bounded  $l_2$  norm  $\max_t \| \mathbf{x}_t \| \leq L_x$ ; (3)  $\epsilon_1, \ldots, \epsilon_T \in \mathbb{R}^d$  is a sequence of independent (not necessarily identical) bounded zero-mean noises satisfying  $\max_t \| \epsilon_t \| \leq L_\epsilon$  and  $\mathbb{E}[\epsilon_t \epsilon_t^\top] \succcurlyeq \sigma_\epsilon^2 I$  for any  $t$ ; and (4)  $\widetilde{\mathbf{X}}_t$  is defined as follows:
$$
\sum_ {t = 1} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\widetilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq 2 ^ {p} T ^ {1 - p} \log^ {p} \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right) + \frac {8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 d L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right) \tag {2}
$$
Proof. Our proof follows the high level idea for proving the original EPL. However, to accommodate the noises in the data matrix, we have to introduce new matrix concentration tools to the original (primarily algebraic) proof, and also identify the right conditions for the argument to go through. A key lemma to our proof is the following high probability bound regarding the noisy data matrix:
Lemma 2. Let  $\pmb{x}_1, \dots, \pmb{x}_T \in \mathbb{R}^d$  be a fixed sequence of vectors, and  $\pmb{\epsilon}_1, \dots, \pmb{\epsilon}_T \in \mathbb{R}^d$  are independent random variables satisfying  $\max_t \| \pmb{x}_t \|_2 \leq L_x$ ,  $\max_t \| \pmb{\epsilon}_t \|_2 \leq L_\epsilon$ , and  $\mathbb{E}[\pmb{\epsilon}_t \pmb{\epsilon}_t^\top] \succcurlyeq \sigma_\epsilon^2 \pmb{I}$ . Then the following hold with probability at least  $1 - 2d \exp \left( \frac{-T \sigma_\epsilon^4}{8 L_\epsilon^2 (L_\epsilon + L_x)^2} \right)$ ,
$$
\sum_ {t = 1} ^ {T} \left(\boldsymbol {x} _ {t} + \boldsymbol {\epsilon} _ {t}\right) \left(\boldsymbol {x} _ {t} + \boldsymbol {\epsilon} _ {t}\right) ^ {\top} \succcurlyeq \sum_ {t = 1} ^ {T} \boldsymbol {x} _ {t} \boldsymbol {x} _ {t} ^ {\top}.
$$
The Proof of Lemma 2 employs the Bernstein's Inequality for matrices (Tropp et al., 2015), which is technical; for ease of presentation, we defer its proof of Appendix B.1. By Lemma 2, we have that, for every  $t \in [T]$ , the following inequality holds with probability at least  $1 - 2d\exp \left(\frac{-t\sigma_{\epsilon}^{4}}{8L_{\epsilon}^{2}(L_{\epsilon} + L_{x})^{2}}\right)$ :
$$
\widetilde {\boldsymbol {X}} _ {t} = \boldsymbol {X} _ {0} + \sum_ {s = 1} ^ {t} (\boldsymbol {x} _ {s} + \boldsymbol {\epsilon} _ {s}) (\boldsymbol {x} _ {s} + \boldsymbol {\epsilon} _ {s}) ^ {\top} \succcurlyeq \boldsymbol {X} _ {0} + \sum_ {s = 1} ^ {t} \boldsymbol {x} _ {s} \boldsymbol {x} _ {s} ^ {\top} := \boldsymbol {X} _ {t},
$$
under which we have
$$
\| \pmb {x} _ {t + 1} \| _ {\tilde {\pmb {X}} _ {t} ^ {- 1}} ^ {2} \leq \| \pmb {x} _ {t + 1} \| _ {\pmb {X} _ {t} ^ {- 1}} ^ {2}.
$$
To prove our Lemma 1, we need to apply union bound to guarantee the above hold simultaneously for every  $t \geq 1$  with high probability. Unfortunately, this turns out to not be true because when  $t$  is very small (e.g.,  $t = 1$ ), the above inequality cannot hold with high probability. Therefore, to obtain high-probability guarantee by the union bound, we will have to exclude these small  $t$ 's and apply the union bound for only the events from some  $t' \in [T]$ , as follows
$$
\begin{array}{l} \mathbb {P} \left[ \forall t \in [ t ^ {\prime}, T ], \| \boldsymbol {x} _ {t} \| _ {\widetilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2} \leq \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2} \right] \\ \geq 1 - \sum_ {t = t ^ {\prime} - 1} ^ {T - 1} 2 d \exp \left(\frac {- t \sigma_ {\epsilon} ^ {4}}{\left(8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}\right)}\right) \\ \geq 1 - \sum_ {t = t ^ {\prime} - 1} ^ {\infty} 2 d \exp \left(\frac {- t \sigma_ {\epsilon} ^ {4}}{\left(8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}\right)}\right) \\ = 1 - 2 d \left(\frac {\exp \left(- \left(t ^ {\prime} - 1\right) \sigma_ {\epsilon} ^ {4} / \left(8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}\right)\right)}{1 - \exp \left(- \sigma_ {\epsilon} ^ {4} / \left(8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}\right)\right)}\right). \\ \geq 1 - 2 d \times \exp \big (- (t ^ {\prime} - 1) \sigma_ {\epsilon} ^ {4} / (8 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}) \big) \times \frac {1 6 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\sigma_ {\epsilon} ^ {4}}, \\ \end{array}
$$
where the last inequality uses the fact that  $\sigma_{\epsilon}^{4} / (L_{\epsilon}^{2}(L_{\epsilon} + L_{x})^{2}) \leq (\sigma_{\epsilon} / L_{\epsilon})^{4} \leq 1$  and  $1 - e^{-x} \geq x / 2$  for any  $x \in [0,1]$ . By solving the following inequality,
$$
\exp \left(- (t ^ {\prime} - 1) \sigma_ {\epsilon} ^ {4} / \left(8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}\right)\right) \times \frac {1 6 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \leq \frac {\delta}{2 d},
$$
we have,
$$
t ^ {\prime} \geq 1 + \frac {8 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 d L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right)
$$
Let  $T_0$  denote the ceiling of the right-hand-side of the above term. Therefore, we have the following hold with high probability at least  $1 - \delta$ :
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\tilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq (T _ {0} - 1) + \sum_ {t = T _ {0}} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\tilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \\ \leq \left(T _ {0} - 1\right) + \sum_ {t = T _ {0}} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \\ \leq \left(T _ {0} - 1\right) + \sum_ {t = T _ {0}} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \\ \end{array}
$$
In the next, we are going to bound the second term in the above equation, whose proof can be adapted from the proof of the original elliptical potential lemma. Using the fact that for any  $z \in [0, +\infty]$ ,  $z \wedge 1 \leq 2\ln(1 + z)$ , we get
$$
\sum_ {t = 1} ^ {T} 1 \wedge \left(\| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq \sum_ {t = 1} ^ {T} \left(2 \log \left(1 + \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right)\right) ^ {p}.
$$
Additionally, by definition, we have
$$
\boldsymbol {X} _ {t} = \boldsymbol {X} _ {t - 1} + \boldsymbol {x} _ {t} \boldsymbol {x} _ {t} ^ {\top} = \boldsymbol {X} _ {t - 1} ^ {1 / 2} \left(\boldsymbol {I} + \boldsymbol {X} _ {t - 1} ^ {- 1 / 2} \boldsymbol {x} _ {t} \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {X} _ {t - 1} ^ {- 1 / 2}\right) \boldsymbol {X} _ {t - 1} ^ {1 / 2}.
$$
This implies the following relationship between the determinant,
$$
\det  \boldsymbol {X} _ {t} = \det  \left(\boldsymbol {X} _ {t - 1}\right) \det  \left(\boldsymbol {I} + \boldsymbol {X} _ {t - 1} ^ {- 1 / 2} \boldsymbol {x} _ {t} \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {X} _ {t - 1} ^ {- 1 / 2}\right).
$$
Since the only eigenvalues of a matrix of the form  $\mathbf{I} + \mathbf{y}\mathbf{y}^{\top}$  are  $1 + \| \mathbf{y}\| _2$  and 1, we have
$$
\log \left(1 + \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) = \log \det  \boldsymbol {X} _ {t} - \log \det  \boldsymbol {X} _ {t - 1}.
$$
By taking the power  $p$  for both sides and taking the sum, we have