SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {t e r m} (\mathbf {a}) = \left| \boldsymbol {\rho} ^ {\top} \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x}\right) \left(W _ {*} - W _ {t}\right) \mathbf {x} \right| \\ = | \boldsymbol {\rho} ^ {\top} \nabla \sigma \left(W _ {t} \mathbf {x}\right) \left(I _ {K} \otimes \mathbf {x} ^ {\top}\right) \left(\overrightarrow {W} _ {*} - \overrightarrow {W} _ {t}\right) | \\ = \left| \boldsymbol {\rho} ^ {\top} \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x}\right) \left(I _ {K} \otimes \mathbf {x} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} H _ {t} ^ {\frac {1}{2}} \left(\overrightarrow {W} _ {*} - \overrightarrow {W} _ {t}\right) \right| \\ \leq \left\| \overrightarrow {W} _ {*} - \overrightarrow {W} _ {t} \right\| _ {H _ {t}} \cdot \left\| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x}\right) \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x}\right) \boldsymbol {\rho} \right\| _ {2} \\ \leq \beta_ {t} ^ {\mathrm {O L}} (\delta) \cdot \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x}\right) \nabla \sigma \left(W _ {t} \mathbf {x}\right) \boldsymbol {\rho} \| _ {2} \\ = \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x}), \\ \end{array}
$$
where the last inequality is due to Theorem 3.
Then, we upper bound term (b) by  $\epsilon_t^{\mathrm{snd}}$  with the following arguments. For notation simplicity, we denote by  $\pmb{\xi}_{t,\nu} = W_{t}\mathbf{x} + \nu (W_{*} - W_{t})\mathbf{x}$ . Then, Lemma 11 indicates that
$$
\Xi_ {k, t} = \int_ {\nu = 0} ^ {1} (1 - \nu) \nabla^ {2} \sigma_ {k} (\boldsymbol {\xi} _ {t, \nu}) \mathrm {d} \nu \preccurlyeq 3 I _ {K} \int_ {\nu = 0} ^ {1} (1 - \nu) \mathrm {d} \nu \preccurlyeq 3 I _ {K}.
$$
As a consequence, we can bound term (b) by
$$
\begin{array}{l} \text {t e r m} (\mathsf {b}) \leq 3 \left| \sum_ {k = 1} ^ {K} \rho_ {k} \| \left(W _ {*} - W _ {t}\right) \mathbf {x} \| _ {2} ^ {2} \right| \\ \leq 3 R \| \left(W _ {*} - W _ {t}\right) \mathbf {x} \| _ {2} ^ {2} \\ = 3 R \| \left(I _ {d} \otimes \mathbf {x} ^ {\top}\right) (\overrightarrow {W} _ {*} - \overrightarrow {W} _ {t}) \| _ {2} ^ {2} \\ = \leq 3 R \| \overrightarrow {W} _ {*} - \overrightarrow {W} _ {t} \| _ {H _ {t}} ^ {2} \cdot \| (I _ {K} \otimes \mathbf {x} ^ {\top}) H _ {t} ^ {- \frac {1}{2}} \| _ {2} ^ {2} \\ \leq 3 R \left(\beta_ {t} ^ {\mathrm {O L}}\right) ^ {2} \cdot \| \left(I _ {K} \otimes \mathbf {x} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} \| _ {2} ^ {2} \\ = \epsilon_ {t} ^ {\operatorname {s n d}} (\mathbf {x}), \\ \end{array}
$$
where the first inequality is due to Assumption 2 and the last inequality is due to Theorem 3. Combining the upper bound for term (a) and term (b), we have
$$
\left| \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x}\right) - \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {t} \mathbf {x}\right) \right| \leq \epsilon_ {t} ^ {\text {f s t}} (\mathbf {x}) + \epsilon_ {t} ^ {\text {s n d}} (\mathbf {x}), \tag {54}
$$
for any  $\mathbf{x}\in \mathcal{X}$ . We complete the proof by  $\pmb{\rho}^{\top}\pmb {\sigma}(W_{*}\mathbf{x})\leq \pmb{\rho}^{\top}\pmb {\sigma}(W_{t}\mathbf{x}) + |\pmb{\rho}^{\top}\pmb {\sigma}(W_{*}\mathbf{x}) - \pmb{\rho}^{\top}\pmb {\sigma}(W_{t}\mathbf{x})|$
Proof of Theorem 4. We first prove the regret bound and then discuss the computation cost.
C.3.1 Main Proof
Regret Analysis. We can bound the regret by two times of the bonus term over  $\mathbf{x}_t$ .
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {*}\right) - \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right)\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {*}\right) + \epsilon_ {t} ^ {\mathrm {f s t}} \left(\mathbf {x} _ {*}\right) + \epsilon_ {t} ^ {\mathrm {s n d}} \left(\mathbf {x} _ {*}\right) - \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right)\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {t}\right) + \epsilon_ {t} ^ {\text {f s t}} \left(\mathbf {x} _ {t}\right) + \epsilon_ {t} ^ {\text {s n d}} \left(\mathbf {x} _ {t}\right) - \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right)\right) \\ \end{array}
$$
$$
\leq 2 \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} \left(\mathbf {x} _ {t}\right) + 2 \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {s n d}} \left(\mathbf {x} _ {t}\right), \tag {55}
$$
where the first inequality is due to Proposition 1 and the second inequality is due to the rule of constructing the optimistic reward. The last inequality is due to (54).
Then, we turn to analyze the upper bound for the first term and second term respectively.
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x} _ {t}) \\ = \sum_ {t = 1} ^ {T} \beta_ {t} ^ {0 \mathrm {L}} (\delta) \cdot \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {t}\right) \boldsymbol {\rho} \| _ {2} \\ \leq \beta_ {T} ^ {\mathrm {O L}} (\delta) \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {t}\right) \boldsymbol {\rho} \| _ {2} \\ \leq \beta_ {T} ^ {\mathrm {O L}} (\delta) \underbrace {\sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \nabla \boldsymbol {\sigma} \left(W _ {t + 1} \mathbf {x} _ {t}\right) \boldsymbol {\rho} \| _ {2}} _ {\text {t e r m (a)}} + \beta_ {T} ^ {\mathrm {O L}} (\delta) \underbrace {\sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \left(\nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {t}\right) - \nabla \boldsymbol {\sigma} \left(W _ {t + 1} \mathbf {x} _ {t}\right)\right) \boldsymbol {\rho} \| _ {2}} _ {\text {t e r m (b)}}. \tag {56} \\ \end{array}
$$
$$
\begin{array}{l} \text {t e r m} (\mathbf {a}) \leq \sum_ {t = 1} ^ {T} \| \boldsymbol {\rho} \| _ {\nabla \boldsymbol {\sigma} \left(W _ {t + 1} \mathbf {x} _ {t}\right)} \cdot \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \nabla \boldsymbol {\sigma} ^ {\frac {1}{2}} \left(W _ {t + 1} \mathbf {x}\right) \| _ {2} \\ \leq R \sum_ {t = 1} ^ {T} \sqrt {\lambda_ {\max } \left(\left(\nabla \boldsymbol {\sigma} ^ {\frac {1}{2}} \left(W _ {t + 1} \mathbf {x}\right) \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1} \left(\nabla \boldsymbol {\sigma} ^ {\frac {1}{2}} \left(W _ {t + 1} \mathbf {x}\right) \otimes \mathbf {x} _ {t}\right)\right)} \\ = R \sum_ {t = 1} ^ {T} \sqrt {\lambda_ {\max } \left(\left(\nabla \sigma \left(W _ {t + 1} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1}\right)} \\ \leq R \sqrt {T} \sqrt {\sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(\nabla \sigma \left(W _ {t + 1} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1}\right)} \\ \leq R \sqrt {T} \sqrt {K d \ln \left(1 + \frac {T L}{2 \lambda}\right)} \\ \end{array}
$$
where the first inequality is due to the fact that  $\| A\mathbf{b}\| _2\leq \| A\| _2\cdot \| \mathbf{b}\|$  for a matrix  $A$  and vector  $\mathbf{b}$ . The second inequality is due to the definition of the induced norm for the matrix is defined as  $\| A\| _2 = \sqrt{\lambda_{\max}(A^TA)}$  and the mixed-product property of the Kronecker production. The first equality is due to the cycle property of the maximum eigenvalue such that  $\lambda_{\mathrm{max}}(ABC) = \lambda_{\mathrm{max}}(CAB)$  for matrices  $A,B$  and  $C$ . The last second inequality is due to the Cauchy-Schwarz inequality. Finally, we can obtain the last inequality by Lemma 19, which can be seen as a matrix version of the elliptical potential lemma.
Then, we can bound term (b) as follows.
$$
\begin{array}{l} \text {t e r m} (\mathfrak {b}) \leq \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \| _ {2} \cdot \| \left(\nabla \sigma \left(W _ {t} \mathbf {x} _ {t}\right) - \nabla \sigma \left(W _ {t + 1} \mathbf {x} _ {t}\right)\right) \boldsymbol {\rho} \| _ {2} \\ = \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \| _ {2} \cdot \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \left(\nabla \sigma_ {k} \left(W _ {t} \mathbf {x} _ {t}\right) - \nabla \sigma_ {k} \left(W _ {t + 1} \mathbf {x} _ {t}\right)\right) \right\| _ {2} \\ = \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \| _ {2} \cdot \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} \left(\boldsymbol {\xi} _ {t, k}\right) \left(W _ {t} - W _ {t + 1}\right) \mathbf {x} _ {t} \right\| _ {2} \\ \end{array}
$$
$$
= \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \| _ {2} \cdot \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} \left(\boldsymbol {\xi} _ {t, k}\right) \left(I _ {K} \otimes \mathbf {x} _ {t}\right) ^ {\top} \left(\overrightarrow {W} _ {t} - \overrightarrow {W} _ {t + 1}\right) \right\| _ {2} \tag {57}
$$
In the above, the second equality is due to the mean-value theorem, where denote by  $\pmb{\xi}_{t,k} \in \mathbb{R}^K$  a certain point on the line connecting  $W_{t}\mathbf{x}_{t}$  and  $W_{t+1}\mathbf{x}_{t}$ . We can further bound the second term of the right hand side of the above displayed inequality by
$$
\begin{array}{l} \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} \left(\boldsymbol {\xi} _ {t, k}\right) \left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) \left(\overrightarrow {W} _ {t} - \overrightarrow {W} _ {t + 1}\right) \right\| _ {2} \\ = \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} \left(\boldsymbol {\xi} _ {t, k}\right) \left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} H _ {t} ^ {\frac {1}{2}} \left(\vec {W} _ {t} - \vec {W} _ {t + 1}\right) \right\| _ {2} \\ \leq \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} (\boldsymbol {\xi} _ {t, k}) (I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}) H _ {t} ^ {- \frac {1}{2}} \right\| _ {2} \cdot \| \overrightarrow {W} _ {t} - \overrightarrow {W} _ {t + 1} \| _ {H _ {t}} \\ \leq \frac {\alpha \sqrt {K}}{\lambda} \left\| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla^ {2} \sigma_ {k} (\boldsymbol {\xi} _ {t, k}) (I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}) H _ {t} ^ {- \frac {1}{2}} \right\| _ {2} \\ \leq \frac {\alpha \sqrt {K}}{\lambda} \sum_ {k = 1} ^ {K} \rho_ {k} \| \nabla^ {2} \sigma_ {k} (\boldsymbol {\xi} _ {t, k}) \| _ {2} \cdot \left\| \left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} \right\| _ {2} \\ \leq \frac {3 \alpha K R}{\lambda} \left\| \left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} \right\| _ {2} \tag {58} \\ \end{array}
$$
where the first inequality is by the fact  $\| A\mathbf{b}\| _2\leq \| A\| _2\cdot \| \mathbf{b}\| _2$  for any matrix  $A$  and vector  $\mathbf{b}$ . The second inequality is due to Lemma 20. The last inequality is due to Lemma 11 such that  $\nabla \sigma_k(\mathbf{z})\preccurlyeq 3I_k$  for any  $\mathbf{z}\in \mathbb{R}^K$  and  $\sum_{k = 1}^{K}\rho_{k}\leq \sqrt{K} R$ .
Then plugging (58) into (57), we can bound term (b) by
$$
\mathsf {t e r m} (\mathfrak {b}) \leq \frac {3 \alpha K R}{\lambda} \sum_ {t = 1} ^ {T} \| H _ {t} ^ {- \frac {1}{2}} (I _ {K} \otimes \mathbf {x} _ {t}) \| _ {2} ^ {2} \leq \frac {3 \kappa K d R \alpha}{\lambda} \ln \left(1 + \frac {T}{\lambda \kappa}\right),
$$
where the last inequality is due to Lemma 21. Combining the upper bound for term (a) and term (b) and plugging them into (56), we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x} _ {t}) \leq \beta_ {T} ^ {0 \mathrm {L}} (\delta) \left(R \sqrt {T} \sqrt {K d \ln \left(1 + \frac {T L}{2 \lambda}\right)}\right) + \frac {3 \kappa K d R \alpha \beta_ {T} ^ {0 \mathrm {L}} (\delta)}{\lambda} \ln \left(1 + \frac {T}{\lambda \kappa}\right) \\ = \mathcal {O} \left(K d \log K (\log T) ^ {\frac {3}{2}} \sqrt {T} + \kappa K ^ {\frac {3}{2}} d ^ {\frac {3}{2}} (\log K) ^ {2} (\log T) ^ {2}\right). \tag {59} \\ \end{array}
$$
As for the term  $\sum_{t=1}^{T} \epsilon_t^{\mathrm{snd}}(\mathbf{x}_t)$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x} _ {t}) = 3 R \left(\beta_ {t} ^ {0 \mathrm {L}}\right) ^ {2} \cdot \| \left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- \frac {1}{2}} \| _ {2} ^ {2} \\ \leq 3 R \left(\beta_ {t} ^ {\mathrm {O L}}\right) ^ {2} \kappa d \ln \left(1 + \frac {T}{\lambda \kappa}\right) \\ = \mathcal {O} \left(\kappa K d ^ {2} (\log K) ^ {2} (\log T) ^ {3}\right). \tag {60} \\ \end{array}
$$
Combining (59) and (60) with (55), we have
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq 2 \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x} _ {t}) + 2 \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x} _ {t}) \\ \leq 2 \beta_ {T} ^ {\mathrm {O L}} (\delta) R \sqrt {T} \sqrt {d \ln \left(1 + \frac {T L}{2 \lambda}\right)} + \frac {6 \kappa K d R \alpha \beta_ {T} ^ {\mathrm {O L}} (\delta)}{\lambda} \ln \left(1 + \frac {T}{\lambda \kappa}\right) \\ \end{array}
$$
$$
\begin{array}{l} + 6 R \left(\beta_ {t} ^ {\mathrm {O L}}\right) ^ {2} \kappa d \ln \left(1 + \frac {T}{\lambda \kappa}\right) \\ = \mathcal {O} \left(K d \log K (\log T) ^ {\frac {3}{2}} \sqrt {T} + \kappa K ^ {\frac {3}{2}} d ^ {2} (\log K) ^ {2} (\log T) ^ {3}\right). \\ \end{array}
$$
which completes the proof for the regret bound.
C.3.2 Additional Guarantee for Algorithm 2
This part shows that the OFUL-MLogB (Algorithm 2) also achieves an  $\tilde{\mathcal{O}} (\sqrt{\kappa KT})$  regret bound.
Theorem 5. Under the same condition as Theorem 3, Algorithm 2 ensures
$$
\operatorname {R e g} _ {T} \leq \mathcal {O} \left(d \log K (\log T) ^ {\frac {3}{2}} \sqrt {\kappa K T} + \kappa K ^ {\frac {3}{2}} d ^ {2} (\log K) ^ {2} (\log T) ^ {3}\right).
$$
Proof of Theorem 5. The proof of Theorem 5 is almost the same as that for Theorem 4. We can decompose the regret as
$$
\operatorname {R e g} _ {T} \leq 2 \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x} _ {t}) + \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x} _ {t})
$$
following the same arguments (55). The only difference is that we can bound the  $\sum_{t=1}^{T} \epsilon_t^{\mathrm{fst}}(\mathbf{x}_t)$  in (55) with a different arguments based on the local matrix  $\bar{V}_t \triangleq \lambda I_{Kd} + \frac{1}{\kappa} \sum_{s=1}^{t-1} I_K \otimes \mathbf{x}_s \mathbf{x}_s^\top$ .
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x} _ {t}) \\ = \sum_ {t = 1} ^ {T} \beta_ {t} ^ {0 \mathrm {L}} (\delta) \cdot \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \nabla \boldsymbol {\sigma} \left(W _ {t} \mathbf {x} _ {t}\right) \boldsymbol {\rho} \| _ {2} \\ \leq R \beta_ {T} ^ {\mathrm {O L}} (\delta) \sum_ {t = 1} ^ {T} \| \bar {V} _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \| _ {2} \\ \leq R \beta_ {T} ^ {\mathrm {O L}} (\delta) \sqrt {\kappa d T \log \left(1 + \frac {T}{\kappa \lambda d}\right)} \\ = \mathcal {O} \left(d \log K (\log T) ^ {\frac {2}{3}} \sqrt {\kappa K T}\right) \tag {61} \\ \end{array}
$$
where the first inequality is by the fact that  $H_{t} \succcurlyeq \bar{V}_{t}$ ,  $\nabla \sigma(W_{t} \mathbf{x}_{t}) \preccurlyeq I_{K}$ , and  $\|\rho\|_{2} \leq R$ . The second inequality can be obtained  $\epsilon_{t}^{\mathrm{snd}}$  with the same arguments as (60):
$$
\sum_ {t = 1} ^ {T} \epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x}) \leq \mathcal {O} \left(\kappa K d ^ {2} (\log K) ^ {2} (\log T) ^ {3}\right).
$$
We completed the proof by combining the above three displayed bounds.
C.3.3 Useful Lemmas
Lemma 19. Let  $H_{t} = \lambda I_{Kd} + \sum_{s=1}^{t-1} \nabla \sigma(W_{s+1} \mathbf{x}_{s}) \otimes \mathbf{x}_{s} \mathbf{x}_{s}^{\top}$ . Then, when  $\lambda > 2$ , we have
$$
\sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(\nabla \boldsymbol {\sigma} \left(W _ {t + 1} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1}\right) \leq K d \ln \left(1 + \frac {T L}{2 \lambda}\right),
$$
where  $L = \max_{\mathbf{x}\in \mathcal{X},W\in \mathcal{W}}\lambda_{\max}(\nabla \sigma (W\mathbf{x}))$
Proof of Lemma 19. Let  $M_t = \frac{\lambda}{2} I_{Kd} + \sum_{s=1}^t \nabla \sigma(W_{s+1}\mathbf{x}_s) \otimes \mathbf{x}_s\mathbf{x}_s^\top$ . Then, we have  $H_t - M_t = \frac{\lambda}{2} I_{Kd} - \nabla \sigma(W_{t+1}\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top$ . Since  $\nabla \sigma(W_{t+1}\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top \preccurlyeq I_{Kd}$ , we have  $H_t \succcurlyeq M_t$  when  $\lambda > 2$ . Then, we have
$$
\sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(\nabla \boldsymbol {\sigma} \left(W _ {t + 1} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1}\right)
$$
$$
\begin{array}{l} \leq \sum_ {t = 1} ^ {T} \operatorname {T r} \left(\left(\nabla \sigma \left(W _ {t + 1} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1}\right) \\ \leq \sum_ {t = 1} ^ {T} \operatorname {T r} \left(M _ {t} - M _ {t - 1}) M _ {t} ^ {- 1}\right) \\ \leq \sum_ {t = 1} ^ {T} \log \frac {\left| M _ {t} \right|}{\left| M _ {t - 1} \right|} \\ \leq K d \ln \left(1 + \frac {T L}{2 \lambda}\right) \\ \end{array}
$$
where we denote by  $\operatorname{Tr}(A)$  the trace of matrix  $A$ . The second inequality is by the condition  $H_{t} \succcurlyeq M_{t}$ . The last second inequality is due to Lemma 4.5 of [16].
Lemma 20. Let  $\overrightarrow{W}_{t+1} = \arg \min_{W \in \mathcal{W}} \langle \nabla \ell_t(\overrightarrow{W}_t), \overrightarrow{W} \rangle + \frac{1}{\alpha} \| \overrightarrow{W} - \overrightarrow{W}_t \|_{\widetilde{H}_t}^2$ . Then, we have
$$
\| \vec {W} _ {t + 1} - \vec {W} _ {t} \| _ {H _ {t}} \leq \frac {\alpha}{2 \lambda} \| \nabla \ell_ {t} (\vec {W} _ {t}) \| _ {2} \leq \frac {\alpha \sqrt {K}}{\lambda}.
$$
Proof of Lemma 20. Let  $F_{t}(\overrightarrow{W}) = \langle \nabla \ell_{t}(\overrightarrow{W}_{t}),\overrightarrow{W}\rangle +\frac{1}{\alpha}\| \overrightarrow{W} -\overrightarrow{W}_{t}\|_{\overrightarrow{H}_{t}}^{2}$ . Since  $\overrightarrow{W}_{t + 1} = \arg \min_{W\in \mathcal{W}}F_t(\overrightarrow{W})$ , we have  $F_{t}(\overrightarrow{W}_{t + 1})\leq F_{t}(\overrightarrow{W}_{t})$ , which implies
$$
\frac {1}{\alpha} \| \overrightarrow {W} _ {t + 1} - \overrightarrow {W} _ {t} \| _ {\tilde {H} _ {t}} ^ {2} \leq \langle \nabla \ell_ {t} (\overrightarrow {W} _ {t}), \overrightarrow {W} _ {t} - \overrightarrow {W} _ {t + 1} \rangle .
$$
Then, by the Hölder's inequality, we can further bound the inner product term by
$$
\langle \nabla \ell_ {t} (\overrightarrow {W} _ {t}), \overrightarrow {W} _ {t} - \overrightarrow {W} _ {t + 1} \rangle \leq \| \overrightarrow {W} _ {t + 1} - \overrightarrow {W} _ {t} \| _ {\tilde {H} _ {t}} \cdot \| \nabla \ell_ {t} (\overrightarrow {W} _ {t}) \| _ {\tilde {H} _ {t} ^ {- 1}},
$$
$$
\| \vec {W} _ {t + 1} - \vec {W} _ {t} \| _ {\tilde {H} _ {t}} \leq \alpha \| \nabla \ell_ {t} (\vec {W} _ {t}) \| _ {\tilde {H} _ {t} ^ {- 1}}.
$$
Since  $\widetilde{H}_t\succ H_t$  and  $\widetilde{H}_t^{-1}\preccurlyeq I_Kd / \lambda$ , we can further have,
$$
\| \vec {W} _ {t + 1} - \vec {W} _ {t} \| _ {H _ {t}} \leq \| \vec {W} _ {t + 1} - \vec {W} _ {t} \| _ {\tilde {H} _ {t}} \leq \alpha \| \nabla \ell_ {t} (\vec {W} _ {t}) \| _ {\tilde {H} _ {t} ^ {- 1}} \leq \frac {\alpha}{2 \lambda} \| \nabla \ell_ {t} (\vec {W} _ {t}) \| _ {2} \leq \frac {\alpha \sqrt {K}}{\lambda},
$$
where the last inequality is due to the definition of  $\nabla \ell_t(\vec{W}_t) = (\pmb{\sigma}_t(\vec{W}_t) - \mathbf{y}_t) \otimes \mathbf{x}_t$  such that  $\|\nabla \ell_t(\vec{W}_t)\|_2 \leq 2\sqrt{K}$ .
Lemma 21. Let  $H_{t} = \lambda I_{Kd} + \sum_{s=1}^{t-1} \nabla \sigma(W_{s+1} \mathbf{x}_{s}) \otimes \mathbf{x}_{s} \mathbf{x}_{s}^{\top}$ . Then, we have
$$
\sum_ {t = 1} ^ {T} \left\| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \right\| _ {2} ^ {2} \leq \kappa d \ln \left(1 + \frac {T}{\lambda \kappa}\right),
$$
where  $\lambda > 2$ .
Proof of Lemma 21. The proof of Lemma 21 shares the same spirits with that of Lemma 19. Let  $\bar{V}_t = \frac{\lambda}{2} I_{Kd} + \frac{1}{\kappa}\sum_{s=1}^{t}I_K \otimes \mathbf{x}_s\mathbf{x}_s^\top$ . We have  $H_t \succcurlyeq \bar{V}_t$  when  $\lambda > 2$ . We can prove the lemma by
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x} _ {t}\right) \right\| _ {2} ^ {2} \\ = \sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1} \left(I _ {K} \otimes \mathbf {x} _ {t}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) \bar {V} _ {t} ^ {- 1} \left(I _ {K} \otimes \mathbf {x} _ {t}\right)\right) \\ = \sum_ {t = 1} ^ {T} \lambda_ {\max } \left(\left(I _ {K} \otimes \mathbf {x} _ {t} ^ {\top}\right) \left(I _ {K} \otimes \left(\lambda I _ {d} + \frac {1}{\kappa} \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}\right) ^ {- 1}\right) \left(I _ {K} \otimes \mathbf {x} _ {t}\right)\right) \\ = \sum_ {t = 1} ^ {T} \lambda_ {\max } \left(I _ {K} \otimes \left(\mathbf {x} _ {t} ^ {\top} \left(\lambda I _ {d} + \frac {1}{\kappa} \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}\right) ^ {- 1} \mathbf {x} _ {t}\right)\right) \\ = \mathbf {x} _ {t} ^ {\top} \left(\lambda I _ {d} + \frac {1}{\kappa} \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}\right) ^ {- 1} \mathbf {x} _ {t} \leq \kappa d \ln \left(1 + \frac {T}{\lambda \kappa}\right), \\ \end{array}
$$
where the first inequality is due to  $H_{t} \succcurlyeq \bar{V}_{t}$ . The second equality is due to the definition of  $\bar{V}_{t}$  and the third equality is due to the mixed-product property of the Kronecker product. The last inequality can be obtained by the standard elliptical potential lemma [3, Lemma 11].
C.3.4 Computation Cost of Algorithm 2
Algorithm 2 OFUL-MLogB
Input: regularization coefficient  $\lambda$ , probability  $\delta$ , step size  $\eta$ .
1: Initialize  $H_{1} = \lambda I_{Kd}$  and  $\overline{W}_1$  as any point in  $\mathcal{W}$
2: for  $t = 1, \dots, T$  do
3: Select the arm by  $\mathbf{x}_t = \arg \max_{\mathbf{x} \in \mathcal{X}} \widetilde{r}_t(\mathbf{x})$  and receive  $y_t$ .
4: Update  $\widetilde{H}_t = H_t + \eta \nabla \pmb{\sigma}(W_t\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top$
5: Update the estimator  $\overrightarrow{W}_{t + 1}$  for the next iteration by (6)
6: Update  $H_{t + 1} = H_t + \nabla \sigma (W_{t + 1}\mathbf{x}_t)\otimes \mathbf{x}_t\mathbf{x}_t^\top$  and
7: Construct the optimistic reward by  $\widetilde{r}_{t + 1}(\mathbf{x}) = \pmb{\rho}^{\top}\pmb{\sigma}(W_{t + 1}\mathbf{x}) + \epsilon_{t + 1}^{\mathrm{fst}}(\mathbf{x}) + \epsilon_{t + 1}^{\mathrm{snd}}(\mathbf{x})$  as (8).
8: end for