SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This section validates the statistical and computation efficiency of the proposed method by experiments. We conduct a bandit learning for  $T = 3000$  iterations. In each experiment, we run experiments on 6 random configurations, in which the arm set and the underlying parameter are randomly sampled. Specially,  $|\mathcal{X}| = 20$  actions are randomly sampled from a 2-dimensional sphere of radius 1. In the binary case, the norm of the unknown parameter  $\mathbf{w}_*$  is set as  $S = 5$ . In the multinomial case, we set  $K = 4$  with  $S = 1$ . The reward vector is set as  $\pmb{\rho} = [0.25, 0.5, 0.75, 1]$ . For each configuration, we report the averaged results over 10 trials. More details can be found in Appendix D.
Experimental Results. Figure 4 provides a comparison of performance and computation costs in the binary case. The algorithm O2LM [7], which has a constant computational cost per iteration, is used as a comparison baseline. Our algorithm demonstrates a time complexity akin to O2LM, affirming its computational efficiency. Compared to the state-of-the-art binary logistic bandit algorithm ada-OFU-ECOLog [10], our OFU-MLogB method has lighter computational overheads while preserving similar empirical performance. Figure 4 illustrates the comparison in the multinomial setting. Our algorithm is around 50 times faster than MNL-UCB [11] for running  $T = 3000$  iterations and achieves better empirical performance. More experimental results on other configurations and the running time curve that increases along the iterations can be found in Appendix D.
This paper proposed a jointly efficient algorithm OFUL-MLogB for both binary and multinomial logistic bandit problems with constant computation cost per round and improved regret guarantees. For the multinomial setting, our method improves over the best-known feasible algorithm both on the dependence of  $\kappa$  and the computation cost. When reduced to the binary case, OFUL-MLogB also contributes to improve the computation cost of previous method from  $\mathcal{O}(\log t)$  to  $\mathcal{O}(1)$  per round while still preserving the  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  minimax optimal bound up to logarithmic factors. A promising future direction is to consider the multinomial logit model in reinforcement learning. Besides, it is still open on how to achieve the  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  bound for the multinomial case.
YJZ and MS were supported by the Institute for AI and Beyond, UTokyo. YJZ was also supported by Todai Fellowship. The authors would thank Peng Zhao for helpful discussions and Jing Wang for the assistance in experiments.
[38] Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Thompson sampling for the mnl-bandit. In Proceedings of the 30th Annual Conference on Learning Theory (COLT), pages 76-78, 2017.
[39] Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research, 67(5):1453-1485, 2019.
[40] Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 3145-3155, 2019.
[41] Wang Chi Cheung and David Simchi-Levi. Thompson sampling for online personalized assortment optimization problems with multinomial logit choice models. Available at SSRN 3075658, 2017.
[42] Priyank Agrawal, Theja Tulabandhula, and Vashist Avadhanula. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research, 2023.
[43] Noémie Périvier and Vineet Goyal. Dynamic pricing and assortment under a contextual MNL demand. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2022.
[44] Yurii E. Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM, 1994.
[45] Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, 4:384-414, 2010.
[46] Tianxiao Sun and Quoc Tran-Dinh. Generalized self-concordant functions: a recipe for newton-type methods. Mathematical Programming, 178(1):145-213, 2019.
[47] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Online-to-confidence-set conversions and application to sparse stochastic bandits. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1-9, 2012.
[48] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[49] Nicolò Campolongo and Francesco Orabona. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
A Properties of the Logistic Regression
This section collects several key properties of the logistic loss used throughout the paper.
A.1 Multiclass Logistic Loss
We have the following property of the multiclass logistic loss.
Property 1. Let  $y \in \mathcal{Y}$  with  $\mathcal{Y} = \{0\} \cup [K]$  and  $\ell : \mathbb{R}^K \times \mathcal{Y} \mapsto \mathbb{R}$  be the multiclass logistic loss defined by
$$
\ell (\mathbf {z}, y) = \sum_ {k = 0} ^ {K} \mathbb {1} \left\{y = k \right\} \cdot \log \left(\frac {1}{[ \boldsymbol {\sigma} (\mathbf {z}) ] _ {k}}\right), \tag {10}
$$
where  $\pmb{\sigma}:\mathbb{R}^{K}\to [0,1]^{K}$  is the vector valued link function defined by  $[\pmb {\sigma}(\mathbf{z})]_k = \exp (z_k) / (1 + \sum_{j = 1}^K\exp (z_j))$  and  $[\pmb {\sigma}(\mathbf{z})]_0 = \exp (z_k) / (1 + \sum_{j = 1}^K\exp (z_j))$ . Let  $\mathbf{y} = [\mathbb{1}\{y = 1\},\dots,\mathbb{1}\{y = K\}] \in \mathbb{R}^{K}$  and  $\sigma_{k}(\mathbf{z}) = [\sigma (\mathbf{z})]_{k}$  be the  $k$ -th entry of  $\pmb{\sigma}(\mathbf{z})$ . Then the first, second and third derivations w.r.t. the first argument of the loss function can be written as
$$
\nabla_ {z} \ell (\mathbf {z}, y) = \boldsymbol {\sigma} (\mathbf {z}) - \mathbf {y},
$$
$$
\nabla_ {z} ^ {2} \ell (\mathbf {z}, y) = \operatorname {d i a g} (\boldsymbol {\sigma} (\mathbf {z})) - \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top},
$$
$$
\nabla_ {z} ^ {3} \ell (\mathbf {z}, y) [ \mathbf {u} ] = \sum_ {k = 1} ^ {K} u _ {k} \sigma_ {k} (\mathbf {z}) \left(2 \pmb {\sigma} (\mathbf {z}) \pmb {\sigma} (\mathbf {z}) ^ {\top} - \mathrm {d i a g} (\pmb {\sigma} (\mathbf {z})) - \mathbf {e} _ {k} \pmb {\sigma} (\mathbf {z}) ^ {\top} - \pmb {\sigma} (\mathbf {z}) \mathbf {e} _ {k} ^ {\top} + E _ {k}\right),
$$
where  $u_{k}$  is the  $k$ -th entry of the vector  $\mathbf{u} \in \mathbb{R}^{K}$  and  $E_{k} \in \mathbb{R}^{K \times K}$  is a "one-hot" matrix whose  $(k, k)$ -th entry equals to 1 while others equaling to 0. Besides,  $\mathbf{e}_{k}$  is the one-hot vector whose  $k$ -th entry equals to 1. In the above,  $\nabla_{z}^{3}\ell(\mathbf{z}, y)[\mathbf{u}] = \lim_{t \to 0} t^{-1}\left(\nabla_{z}^{2}\ell(\mathbf{z} + tu, y) - \nabla_{z}^{2}\ell(\mathbf{z}, y)\right)$ .
When we train a linear model with the logistic regression loss, we can calculate its gradient and Hessian as follows.
Property 2. Denote by the logistic loss defined over the data point  $(\mathbf{x}_t, y_t) \in \mathcal{X} \times \mathcal{Y}$  by
$$
\ell_ {t} (W) = \ell (W \mathbf {x}, y) = \sum_ {k = 0} ^ {K} \mathbb {1} \left\{y _ {t} = k \right\} \cdot \log \left(1 / [ \sigma (W \mathbf {x} _ {t}) ] _ {k}\right). \tag {11}
$$
Then, the gradient  $\nabla \ell_t(\overrightarrow{W}) \in \mathbb{R}^{Kd}$  and the Hessian  $\nabla^2\ell_t(W) \in \mathbb{R}^{Kd\times Kd}$  of the loss function with respective to the vectorized model  $\overrightarrow{W}$  is given by
$$
\nabla \ell_ {t} (\overrightarrow {W}) = (\boldsymbol {\sigma} (W \mathbf {x} _ {t}) - \mathbf {y} _ {t}) \otimes \mathbf {x} _ {t} a n d \nabla^ {2} \ell_ {t} (W) = \operatorname {d i a g} (\boldsymbol {\sigma} (W \mathbf {x} _ {t})) - \boldsymbol {\sigma} (W \mathbf {x} _ {t}) \boldsymbol {\sigma} (W \mathbf {x} _ {t}) ^ {\top} \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}.
$$
Besides, we can obtain the gradient of the vector-valued link function as follows.
Property 3. Let the sigmoid function  $\sigma : \mathbb{R}^K \mapsto [0,1]^K$  be defined as (1) and  $\sigma_k(\mathbf{z})$  denote the  $k$ -th entry of  $\sigma(\mathbf{z})$ . We have
$$
\nabla \sigma_ {k} (\mathbf {z}) = \sigma_ {k} (\mathbf {z}) \cdot \left(\mathbf {e} _ {k} - \boldsymbol {\sigma} (\mathbf {z})\right);
$$
$$
\nabla^ {2} \sigma_ {k} (\mathbf {z}) = \sigma_ {k} (\mathbf {z}) \left(2 \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} - \operatorname {d i a g} (\boldsymbol {\sigma} (\mathbf {z})) - \mathbf {e} _ {k} \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} - \boldsymbol {\sigma} (\mathbf {z}) \mathbf {e} _ {k} ^ {\top} + E _ {k}\right),
$$
In above,  $E_{k}\in \mathbb{R}^{K\times K}$  is an all zero matrix except that its  $(k,k)$ -th entry is 1. For the first order derivation, we can write it into a more concise formulation by the notation  $\nabla \pmb {\sigma}(\mathbf{z}):\mathbb{R}^K\mapsto \mathbb{R}^{K\times K}$
$$
\nabla \boldsymbol {\sigma} (\mathbf {z}) \triangleq \frac {\partial \boldsymbol {\sigma} (\mathbf {z})}{\partial \mathbf {z} ^ {\top}} = \operatorname {d i a g} (\boldsymbol {\sigma} (\mathbf {z})) - \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top}.
$$
We have the following lemma for the logistic loss function.
Lemma 1. Let  $C > 0$ ,  $\mathbf{a} \in [-C, C]^K$ ,  $y \in \{0\} \cup [K]$  and  $\mathbf{b} \in \mathbb{R}^K$ . Then, we have
$$
\ell (\mathbf {a}, y) \geq \ell (\mathbf {b}, y) + \nabla \ell (\mathbf {b}, y) ^ {\top} (\mathbf {a} - \mathbf {b}) + \frac {1}{\log (K + 1) + 2 (C + 1)} (\mathbf {a} - \mathbf {b}) ^ {\top} \nabla^ {2} \ell (\mathbf {b}, y) (\mathbf {a} - \mathbf {b}).
$$
Proof. Lemma 1 is essentially the Lemma 4 of [27] with a slightly different definition of the logistic loss. The  $K + 1$ -class logistic loss used by [27] is defined as  $\widetilde{\ell} (\widetilde{\mathbf{z}},y) = \sum_{k = 0}^{K}\mathbb{1}\{y = k\}\log \left(\sum_{k = 0}^{K}\exp ([\widetilde{\mathbf{z}} ]_j) / \exp ([\widetilde{\mathbf{z}} ]_k)\right)$ , where  $\widetilde{\mathbf{z}}\in \mathbb{R}^{K + 1}$  is a  $K + 1$ -dimensional input. One can connect the two logistic losses by  $\widetilde{\ell} ([\mathbf{z};0],y) = \ell (\mathbf{z},y)$ . Then, we can prove Lemma 1 by directly applying Lemma 4 of [27] with the augmented vectors  $\widetilde{\mathbf{a}} = [\mathbf{a};0]$  and  $\widetilde{\mathbf{b}} = [\mathbf{b};0]$ .
A.2 Self-concordant-like Function
Then, we introduce the notion of self-concordant-like function, which generalizes the standard self-concordant notion. The self-concordant function is first introduced by [44] in the study of inner-point algorithm and the notion is further generalized by the following work [45, 19, 46] for analyzing the Newton-step methods. The self-concordant function enjoys several nice properties, which plays an important role in our local analysis. A comprehensive study for self-concordance function is provided by [46]. Here, we present the definition and necessary lemmas for our analysis.
Definition 1 (Theorem 3 of [19]). A convex function  $\ell \in \mathcal{C}^3:\mathbb{R}^d\to \mathbb{R}$  is  $M$  -self-concordant-like if and only if for any  $\mathbf{x},\mathbf{u}_1,\mathbf{u}_2,\mathbf{u}_3\in \mathbb{R}^d$  , we have
$$
\left| \langle D ^ {3} \ell (\mathbf {z}) [ \mathbf {u} _ {1} ] \mathbf {u} _ {2}, \mathbf {u} _ {3} \rangle \right| \leq M \| \mathbf {u} _ {1} \| _ {2} \| \mathbf {u} _ {2} \| _ {\mathbf {z}} \| \mathbf {u} _ {3} \| _ {\mathbf {z}},
$$
where  $\| \mathbf{u}\|_{\mathbf{z}}\coloneqq \sqrt{\mathbf{u}^{\top}\nabla^{2}f(\mathbf{z})\mathbf{u}}$  is the local norm over the Hessian  $\nabla^2\ell (\mathbf{z})$
The logistic loss is  $\sqrt{6}$ -self-concordant-like function.
Lemma 2 (Lemma 4 of [19]). The logistic loss defined as (11) is  $\sqrt{6}\| \mathbf{x}_t\| _2$  -self-concordant-like.
The self-concordant function enjoys favorable properties, which help us to conduct local analysis for the logistic bandit problem. We list useful lemmas as follows.
Lemma 3 (Bound of Hessian Map, Theorem 4(b) of [19]). Let  $\ell : \mathbb{R}^d \mapsto \mathbb{R}$  be an  $M$ -self-concordant-like function. Then, for any  $\mathbf{z}_1, \mathbf{z}_2 \in \mathrm{dom}(\ell)$ , we have
$$
e ^ {- M \left\| \mathbf {z} _ {1} - \mathbf {z} _ {2} \right\| _ {2}} \nabla^ {2} \ell (\mathbf {z} _ {1}) \preceq \nabla^ {2} \ell (\mathbf {z} _ {2}) \preceq e ^ {M \left\| \mathbf {z} _ {1} - \mathbf {z} _ {2} \right\| _ {2}} \nabla^ {2} \ell (\mathbf {z} _ {1}) \tag {12}
$$
The following lemma is a consequence of the bound of the Hessian map.
Lemma 4 (Corollary 2 of [46]). Let  $\ell : \mathbb{R}^d \mapsto \mathbb{R}$  be an  $M$ -self-concordant-like function. Then, for any  $\mathbf{z}_1, \mathbf{z}_2 \in \mathrm{dom}(\ell)$ , we have
$$
\frac {1 - e ^ {- M \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}}}{M \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}} \cdot \nabla^ {2} \ell (\mathbf {z} _ {2}) \preceq \int_ {0} ^ {1} \nabla^ {2} \ell (\mathbf {z} _ {1} + \nu (\mathbf {z} _ {2} - \mathbf {z} _ {1})) \mathrm {d} \nu \preceq \frac {e ^ {M \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}} - 1}{M \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}} \nabla^ {2} \ell (\mathbf {z} _ {1}).
$$
We can also have the bound on the function value of the self-concordant functions.
Lemma 5 (Proposition 10 of [46]). Let  $\ell : \mathbb{R}^d \mapsto \mathbb{R}$  be an  $M$ -self-concordant-like function. Then, for any  $\mathbf{z}_1, \mathbf{z}_2 \in \mathrm{dom}(\ell)$ , we have
$$
c \left(- M \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}\right) \cdot \| \mathbf {z} _ {2} - \mathbf {z} _ {1} \| _ {\mathbf {z} _ {1}} ^ {2} \leq \ell (\mathbf {z} _ {2}) - \ell (\mathbf {z} _ {1}) - \nabla \ell (\mathbf {z} _ {1}) ^ {\top} (\mathbf {z} _ {2} - \mathbf {z} _ {1}) \leq c \left(M _ {f} \| \mathbf {z} _ {1} - \mathbf {z} _ {2} \| _ {2}\right) \cdot \| \mathbf {z} _ {2} - \mathbf {z} _ {1} \| _ {\mathbf {z} _ {1}} ^ {2},
$$
where  $c: \mathbb{R} \mapsto \mathbb{R}$  is the coefficient function defined as  $c(x) = (e^x - x - 1) / x^2$ .
B Omitted Proofs for Section 2.3
This section we present the proof of Theorem 1.
B.1.1 Main Proof
Before introducing the main proof, we introduce a key lemma on an improved self-normalized martingale tail inequality for the multinomial noise, which helps to save an  $\mathcal{O}(\sqrt{K})$  factor compared to the one obtained by [11]. The proof of Lemma 6 is provided in Appendix B.1.2
Lemma 6. Let  $\{\mathcal{F}_t\}_{t=1}^{\infty}$  be a filtration. Let  $\{\mathbf{x}_t\}_{t=1}^{\infty}$  be a stochastic process in  $\mathcal{B}_2(d) \triangleq \{\mathbf{x} \in \mathbb{R}^d \mid \| \mathbf{x}_t\|_2 \leq 1\}$  such that  $\mathbf{x}_t$  is  $\mathcal{F}_t$  measurable. Let  $\{\varepsilon_t\}_{t=1}^{\infty}$  be a martingale difference sequence such that  $\varepsilon_t \in \mathbb{R}^K$  is  $\mathcal{F}_{t+1}$  measurable. Furthermore, assume that conditional on  $\mathcal{F}_t$ , we have  $\|\varepsilon_t\|_1 \leq 2$  almost surely, and denoted by  $\Sigma_t \triangleq \mathbb{E}[\varepsilon_t\varepsilon_t^\top \mid \mathcal{F}_t]$ . Let  $\lambda > 0$  and for any  $t \geq 1$  define
$$
S _ {t} = \sum_ {s = 1} ^ {t - 1} \varepsilon_ {s} \otimes \mathbf {x} _ {s} a n d H _ {t} = \lambda I _ {K d} + \sum_ {s = 1} ^ {t - 1} \Sigma_ {s} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}.
$$
Then for any  $\delta \in (0,1]$ , we have
$$
\Pr \left[ \exists t > 1, \| S _ {t} \| _ {H _ {t} ^ {- 1}} \geq \frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {| H _ {t} | ^ {1 / 2}}{\delta \lambda^ {K d / 2}}\right) + \frac {4}{\sqrt {\lambda}} K d \log (2) \right] \leq \delta .
$$
Lemma 6 is a counterpart of the variance-aware self-normalized tail inequality [8, Theorem 1] for the multinominal case. In comparison with the previously established result by [11], our inequality more effectively leverages the geometric structure of the noise in the multinomial logistic bandit problem, as indicated by the condition  $\| \varepsilon_t\| _1\leq 2$ . The previous paper's condition,  $\| \varepsilon_t\| _2\leq \sqrt{K}$ , is relatively loose, ultimately resulting in an additional  $O(\sqrt{K})$  term in the concentration inequality. Then, we introduce the main proof of Theorem 1
Proof of Theorem 1. The optimality of the MLE estimator indicates that  $\mathbf{g}_t(\overrightarrow{W}_t^{\mathrm{MLE}}) = \nabla \mathcal{L}_t(\overrightarrow{W}_t^{\mathrm{MLE}}) = \mathbf{0}$ . In such a case, we have
$$
\mathbf {g} _ {t} \left(W _ {t} ^ {\mathrm {M L E}}\right) - \mathbf {g} _ {t} \left(W _ {*}\right) = - \mathbf {g} _ {t} \left(W _ {*}\right) = \sum_ {s = 1} ^ {t - 1} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {s}\right) - \mathbf {y} _ {s}\right) \otimes \mathbf {x} _ {s} - \lambda \overrightarrow {W} _ {*} = \sum_ {s = 1} ^ {t - 1} \varepsilon_ {s} \otimes \mathbf {x} _ {s} - \lambda \overrightarrow {W} _ {*},
$$
where  $\pmb{\varepsilon}_{s} = \pmb{\sigma}(W_{*}\mathbf{x}_{s}) - \mathbf{y}_{s}$  is the noise with bounded  $L_{\infty}$  norm as we have  $\| \epsilon_s\|_{\infty}\leq \| \pmb {\sigma}(W_{*}\mathbf{x}_{s})\|_{\infty} +$ $\| \mathbf{y}_s\|_\infty \le 2$ . Then, we can bound the gap between  $\overrightarrow{W}_t^{\mathrm{MLE}}$  and  $\overrightarrow{W}_{*}$  by
$$
\begin{array}{l} \left\| \mathbf {g} _ {t} \left(W _ {t} ^ {\mathrm {M L E}}\right) - \mathbf {g} _ {t} \left(W _ {*}\right) \right\| _ {H _ {t} ^ {- 1} \left(W _ {*}\right)} \leq \left\| \sum_ {s = 1} ^ {t - 1} \varepsilon_ {s} \otimes \mathbf {x} _ {s} \right\| _ {H _ {t} ^ {- 1} \left(W _ {*}\right)} + \left\| \lambda \vec {W} _ {*} \right\| _ {H _ {t} ^ {- 1} \left(W _ {*}\right)} \\ \leq \left\| \sum_ {s = 1} ^ {t - 1} \varepsilon_ {s} \otimes \mathbf {x} _ {s} \right\| _ {H _ {t} ^ {- 1} (W _ {*})} + \sqrt {\lambda} S \\ \leq \frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {\left| H _ {t} \right| ^ {1 / 2}}{\delta \lambda^ {K d / 2}}\right) + \frac {4}{\sqrt {\lambda}} K d \log (2) + \sqrt {\lambda} S, \tag {13} \\ \end{array}
$$
with probability at least  $1 - \delta$  for all  $t \geq 1$ . In above, the matrix  $H_{t}(W_{*})$  is defined as  $H_{t}(W_{*}) = \lambda I_{Kd} + \sum_{s=1}^{t-1} \Sigma(W_{*}\mathbf{x}_{s})\mathbf{x}_{s}\mathbf{x}_{s}^{\top}$ . The first inequality is due to the relationship  $H_{t}^{-1}(W_{*}) \preccurlyeq 1 / \lambda \cdot I_{Kd}$ . The last inequality is by Lemma 6 since
$$
\Sigma \left(W _ {*} \mathbf {x} _ {s}\right) = \operatorname {d i a g} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {s}\right)\right) - \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {s}\right) \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {s}\right) ^ {\top} = \mathbb {E} \left[ \boldsymbol {\varepsilon} _ {s} \boldsymbol {\varepsilon} _ {s} ^ {\top} \mid \mathcal {F} _ {s - 1} \right].
$$
Then, we can bound the determinate of  $H_{t}(W_{*})$  by
$$
\left| H _ {t} \right| \leq \left(\frac {\operatorname {T r} \left(H _ {t}\right)}{K d}\right) ^ {K d} \leq \left(\frac {\lambda K d + \sum_ {s = 1} ^ {t - 1} \sum_ {k = 1} ^ {K} \Sigma_ {k} \left(W _ {*} \mathbf {x} _ {s}\right) \| \mathbf {x} _ {s} \| _ {2}}{K d}\right) ^ {K d} \leq (\lambda + t / d) ^ {K d}. \tag {14}
$$
Plugging (14) into (13), we have
$$
\left\| \mathbf {g} _ {t} \left(W _ {t} ^ {\mathrm {M L E}}\right) - \mathbf {g} _ {t} \left(W _ {*}\right) \right\| _ {H _ {t} ^ {- 1} \left(W _ {*}\right)} \leq \left(\frac {1}{4} + S\right) \sqrt {\lambda} + \frac {2 K d}{\sqrt {\lambda}} \log \left(1 + \frac {t}{\lambda d}\right) + \frac {4 K d}{\sqrt {\lambda}} \log \left(\frac {2}{\delta}\right), \tag {15}
$$
$$
\lambda = \frac {4 K d \log (2 (1 + t / d) / \delta)}{S + 1 / 4} = \mathcal {O} (d K \log (t / \delta)), \tag {16}
$$
$$
\left\| \mathbf {g} _ {t} \left(W _ {t} ^ {\mathrm {M L E}}\right) - \mathbf {g} _ {t} \left(W _ {*}\right) \right\| _ {H _ {t} ^ {- 1} \left(W _ {*}\right)} \leq b e t a _ {t} (\delta) \triangleq 4 \sqrt {K d (1 + S) \log \left(2 \left(1 + t / d\right) / \delta\right)},
$$
B.1.2 Useful Lemmas
Proof of Lemma 6. The proof follows the pipeline in the analysis of the self-normalized tail-inequality in the previous studies [3, 8, 11]. Letting  $\bar{H}_t = \sum_{s=1}^{t-1} \Sigma_s \otimes \mathbf{x}_s \mathbf{x}_s^\top$ , we define the function
$$
M _ {t} (\pmb {\xi}) \triangleq \exp (\pmb {\xi} ^ {\top} S _ {t} - \| \pmb {\xi} \| _ {\bar {H} _ {t}} ^ {2}),
$$
for any  $t \geq 1$  and  $\pmb{\xi} \in \mathbb{R}^{Kd}$ . For  $t = 0$ , let  $M_0(\pmb{\xi}) = 0$ .
By Lemma 7 in Appendix B.1.2, we can show that  $\{M_t(\pmb {\xi})\}_{t = 1}^{\infty}$  is a non-negative super-martingale for any  $\pmb {\xi}\in \frac{1}{2}\mathcal{B}_2(Kd)$ . We note that our Lemma 7 is an imporved version of Lemma 7 in [11], which relaxes restriction on the feasible domain of  $\pmb{\xi}$  from  $\pmb {\xi}\in \frac{1}{\sqrt{K}}\mathcal{B}_2(Kd)$  to  $\pmb {\xi}\in \frac{1}{2}\mathcal{B}_2(Kd)$ , resulting in saving an  $\mathcal{O}(\sqrt{K})$  factor in the bound.
Then, let  $h(\pmb{\xi})$  be a probability density with support on  $\frac{1}{2}\mathcal{B}_2(Kd)$  and define
$$
\bar {M} _ {t} \triangleq \int_ {\boldsymbol {\xi}} M _ {t} (\boldsymbol {\xi}) \mathrm {d} h (\boldsymbol {\xi}) = \int_ {\boldsymbol {\xi}} \exp \left(\boldsymbol {\xi} ^ {\top} S _ {t} - \| \boldsymbol {\xi} \| _ {\bar {H} _ {t}} ^ {2}\right) \mathrm {d} h (\boldsymbol {\xi})
$$
for all  $t \geq 1$ . Lemma 20.3 of [15] shows that  $\bar{M}_t$  is also a non-negative super-martingale and  $\mathbb{E}[\bar{M}_0] = 1$ . Then, the maximal inequality (Theorem 3.9 of [15]) shows that
$$
\Pr \left[ \sup  _ {t \in \mathbb {N}} \log (\bar {M} _ {t}) \geq \log \left(\frac {1}{\delta}\right) \right] = \Pr \left[ \sup  _ {t \in \mathbb {N}} \bar {M} _ {t} \geq \frac {1}{\delta} \right] \leq \delta . \tag {17}
$$
Next, we turn to exam the formulation of  $\bar{M}_t$  and subsequently establish a connection with the term we aim to bound,  $\| S_t\|_{H_t^{-1}}$ . Let  $h(\pmb {\xi})$  be the density of an isotropic normal distribution with precision matrix  $2\lambda I_{Kd}$  truncated on  $\frac{1}{2}\mathcal{B}_2(Kd)$  and  $N(h)$  be its normalization constant. Furthermore, let  $g(\pmb {\xi})$  be the density of the normal distribution with precision matrix  $2H_{t}$  that is truncated on the ball  $\frac{1}{4}\mathcal{B}_2(Kd)$ . Following the arguments in the proof of [8, Theorem 1] (more precisely the arguments in deriving Eq.(11) in [8]), for any  $t\geq 1$ , one can show that
$$
\bar {M} _ {t} \geq \exp (\boldsymbol {\xi} ^ {\top} S _ {t} - \| \boldsymbol {\xi} \| _ {H _ {t}} ^ {2}) \cdot \frac {N (g)}{N (h)}.
$$
for any  $\pmb {\xi}\in \frac{1}{4}\mathcal{B}_2(Kd)$ . Let  $\pmb{\xi}_{0}\triangleq \frac{H_{t}^{-1}S_{t}}{\|S_{t}\|_{H_{t}^{-1}}}\cdot \frac{\sqrt{\lambda}}{4}$ . One can check  $\| \pmb {\xi}_0\| _2\leq \frac{1}{\sqrt{\lambda}}\cdot \frac{\sqrt{\lambda}}{4}\leq 1 / 4$ . Then, we can further have
$$
\log \left(\bar {M} _ {t}\right) \geq \boldsymbol {\xi} _ {0} ^ {\top} S _ {t} - \left\| \boldsymbol {\xi} _ {0} \right\| _ {H _ {t}} ^ {2} + \log \left(\frac {N (g)}{N (h)}\right) = \frac {\sqrt {\lambda}}{4} \left\| S _ {t} \right\| _ {H _ {t} ^ {- 1}} - \frac {\lambda}{1 6} + \log \left(\frac {N (g)}{N (h)}\right). \tag {18}
$$
Combining (17) and (18), for any  $t \geq 1$ , we have
$$
\Pr \left[ \| S _ {t} \| _ {H _ {t} ^ {- 1}} \leq \frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {N (h)}{\delta N (g)}\right) \right] \geq 1 - \delta .
$$
We complete the proof with Lemma 6 of [8] such that
$$
\log \left(\frac {N (h)}{N (g)}\right) \leq \log \left(\frac {\left| H _ {t} \right| ^ {1 / 2}}{\lambda^ {K d / 2}}\right) + K d \log (2).
$$
Lemma 7. For all  $\pmb {\xi}\in \frac{1}{2}\mathcal{B}_2(Kd)$  , the sequence  $\{M_t(\pmb {\xi})\}_{t = 1}^{\infty}$  is a non-negative super-martingale.