SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, we discuss about the computation cost of the Algorithm 2. For each iteration, our algorithm requires to maintain the inverse of matrix  $H_{t + 1} = H_t + \nabla \sigma (W_{t + 1}\mathbf{x}_t)\otimes \mathbf{x}_t\mathbf{x}_t^\top$  and  $\widetilde{H}_{t} = H_{t} + \eta \nabla \sigma (W_{t}\mathbf{x}_{t})\otimes \mathbf{x}_{t}\mathbf{x}_{t}^{\top}$ . Since  $\nabla \sigma (W_{t + 1}\mathbf{x}_t)\otimes \mathbf{x}_t\mathbf{x}_t^\top$  and  $\nabla \sigma (W_{t}\mathbf{x}_{t})\otimes \mathbf{x}_{t}\mathbf{x}_{t}^{\top}$  are both rank-  $K$  matrix, then one can main  $H_{t}$  and  $\widetilde{H}_{t}$  with  $\mathcal{O}(K^3 d^2)$  computation cost per iteration by the Sherman-Morrison-Woodbury formula. Given the  $H_{t}^{-1}$  and  $\widetilde{H}_{t}^{-1}$ , we discuss the computation cost of the construction of the optimistic reward 8 and the update rule for the estimator (6).
Computation Cost of the Estimator (6). As shown by the discussion in Section 3.2, the update rule (6) is identical to
$$
\overrightarrow {Z} _ {t + 1} = \overrightarrow {W} _ {t} - \eta \widetilde {H} _ {t} ^ {- 1} \nabla \ell_ {t} (\overrightarrow {W} _ {t}) \quad \text {a n d} \quad \overrightarrow {W} _ {t + 1} = \underset {\overrightarrow {W} \in \mathcal {W}} {\arg \min } \| \overrightarrow {W} - \overrightarrow {Z} _ {t + 1} \| _ {\widetilde {H} _ {t}}.
$$
Given the  $\widetilde{H}_t^{-1}$ , we can perform the gradient step with  $\mathcal{O}(K^2 d^2)$  time complexity. As for the projection step, the optimization problem can be solved in  $\mathcal{O}(K^3 d^3)$  time [18, Section 4]. As a consequence, the overall time complexity for obtaining the estimator (6) is  $\mathcal{O}(K^3 d^3)$ .
Computation Cost of Building Optimistic Reward. As shown by (8), we construct the optimistic reward by  $\widetilde{r}_t^{\mathrm{OL}}(\mathbf{x}) = \pmb {\rho}^\top \pmb {\sigma}(W_t^{\mathrm{OL}}\mathbf{x}) + \epsilon_t^{\mathrm{fst}}(\mathbf{x}) + \epsilon_t^{\mathrm{snd}}(\mathbf{x})$ . We can compute  $\epsilon_t^{\mathrm{fst}}(\mathbf{x})$  with the following equivalent formulation
$$
\begin{array}{l} \epsilon_ {t} ^ {\mathbf {f s t}} (\mathbf {x}) = \beta_ {t} ^ {0 \mathrm {L}} (\delta) \cdot \| H _ {t} ^ {- \frac {1}{2}} \left(I _ {K} \otimes \mathbf {x}\right) \nabla \boldsymbol {\sigma} \left(W _ {t}\right) \boldsymbol {\rho} \| _ {2} \\ = \beta_ {t} ^ {0 \mathrm {L}} (\delta) \sqrt {\boldsymbol {\rho} ^ {\top} (\nabla \boldsymbol {\sigma} (W _ {t} \mathbf {x}) \otimes \mathbf {x} ^ {\top}) H _ {t} ^ {- 1} \boldsymbol {\rho} ^ {\top} (\nabla \boldsymbol {\sigma} (W _ {t} \mathbf {x}) \otimes \mathbf {x}) \boldsymbol {\rho}}. \\ \end{array}
$$
Given  $H_{t}^{-1}$ , it will take  $\mathcal{O}(K^2 d^2)$  to calculate  $\epsilon_t^{\mathrm{fst}}(\mathbf{x})$ . As for the term  $\epsilon_t^{\mathrm{fst}}(\mathbf{x})$ , it can also be rewritten as
$$
\epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x}) = 3 R \left(\beta_ {t} ^ {\mathrm {O L}}\right) ^ {2} \cdot \| (I _ {K} \otimes \mathbf {x} ^ {\top}) H _ {t} ^ {- 1 / 2} \| _ {2} ^ {2}
$$
$$
= 3 R \left(\beta_ {t} ^ {\mathsf {O L}}\right) ^ {2} \sqrt {\lambda_ {\max} \left((I _ {K} \otimes \mathbf {x x} ^ {\top}) H _ {t} ^ {- 1}\right)},
$$
It would take  $\mathcal{O}(K^3 d^3)$  time to perform the eigenvalue decomposition. Given there are  $|\mathcal{X}|$  arm, the time complexity for identify the arm  $\mathbf{x}_t = \arg \max_{\mathbf{x}\in \mathcal{X}}\widetilde{r}_t(\mathbf{x})$  is in total  $\mathcal{O}(|\mathcal{X}|K^3 d^3)$ .
Overall, Algorithm 2 can be implemented in  $\mathcal{O}(|\mathcal{X}|K^3 d^3)$  time, which is independent of  $T$ .
C.4.1 Main Proof
In the binary case, we select the arm by  $(\mathbf{x}_t,\widetilde{\mathbf{w}}_t) = \arg \max_{\mathbf{x}\in \mathcal{X},W\in \mathcal{C}_t(\delta)}\mathbf{w}^\top \mathbf{x}$ . In such a case, we show Algorithm 2 achieves an  $\widetilde{O}(T / \kappa_{*})$  bound with a constant computation cost. The proof is almost the same as that of [10, Theorem 2]. The main difference is that we complete the proof with the confidence set of the efficient online estimator (Theorem 3). We present the proof here for self-containedness.
Proof of Corollary 1. When  $K = 1$  and  $\rho_{1} = 1$ , the MLogB problem recovers the binary logistic bandit problem with the feedback  $y_{t} = \{0,1\}$  and the reward model  $\operatorname*{Pr}[y = 1|\mathbf{x}] = \sigma (\mathbf{w}_{*}^{\top}\mathbf{x})$ . In such a case, we can decompose the regret by
$$
\begin{array}{l} \mathrm {R e g} _ {T} = \sum_ {t = 1} ^ {T} \sigma (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*}) - \sum_ {t = 1} ^ {T} \sigma (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}) \\ \leq \sum_ {t = 1} ^ {T} \sigma (\widetilde {\mathbf {w}} _ {t} ^ {\top} \mathbf {x} _ {t}) - \sum_ {t = 1} ^ {T} \sigma (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}) \\ = \sum_ {t = 1} ^ {T} \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right) \left(\widetilde {\mathbf {w}} _ {t} ^ {\top} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t} + \sum_ {t = 1} ^ {T} \sigma^ {\prime \prime} \left(\boldsymbol {\xi} _ {t} \mathbf {x} _ {t}\right) \left(\left(\widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t}\right) ^ {2} \\ = \underbrace{\sum_{t\in I_{1}}\sigma^{\prime}(\mathbf{w}_{*}^{\top}\mathbf{x}_{t})(\widetilde{\mathbf{w}}_{t}^{\top} - \mathbf{w}_{*})^{\top}\mathbf{x}_{t}}_{\text{term (a)}} + \underbrace{\sum_{t\in I_{2}}\sigma^{\prime}(\mathbf{w}_{*}^{\top}\mathbf{x}_{t})(\widetilde{\mathbf{w}}_{t}^{\top} - \mathbf{w}_{*})^{\top}\mathbf{x}_{t}}_{\text{term (b)}} + \underbrace{\sum_{t = 1}^{T}\sigma^{\prime\prime}(\boldsymbol{\xi}_{t}\mathbf{x}_{t})((\widetilde{\mathbf{w}}_{t} - \mathbf{w}_{*})^{\top}\mathbf{x}_{t})^{2}}_{\text{term (c)}}, \\ \end{array}
$$
where the first inequality is due to the arm selection rule. The first equality is due to the Taylor series and  $\pmb{\xi}_t\in \mathbb{R}^d$  is a certain point on the line connecting  $\widetilde{\mathbf{w}}_t$  and  $\mathbf{w}_{*}$ . For the last equality, we divide the time horizon into two parts  $I_{1} = \{t\in [T]\mid \sigma^{\prime}(\mathbf{w}_{*}^{\top}\mathbf{x}_{t})\geq \sigma^{\prime}(\mathbf{w}_{t + 1}^{\top}\mathbf{x}_{t})\}$  and  $I_{2} = [T] / I_{1}$ .
$$
\begin{array}{l} \text {t e r m} (a) \\ = \sum_ {t \in I _ {1}} \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right) \left(\widetilde {\mathbf {w}} _ {t} ^ {\top} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t} \\ \leq \sum_ {t \in I _ {1}} \sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right) \left(\widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t} + \sum_ {t \in I _ {1}} \left| \left(\mathbf {w} _ {t + 1} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t} \right| \left(\widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*}\right) ^ {\top} \mathbf {x} _ {t} \\ \leq \underbrace {\sum_ {t \in I _ {1}} \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \cdot \| \widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*} \| _ {H _ {t}} \cdot \| \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}}} _ {\text {t e r m (a - 1)}} \\ + \underbrace {\sum_ {t \in I _ {1}} \left\| \mathbf {x} _ {t} \right\| _ {H _ {t} ^ {- 1}} ^ {2} \left\| \widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*} \right\| _ {H _ {t}} \cdot \left\| \mathbf {w} _ {t + 1} - \mathbf {w} _ {*} \right\| _ {H _ {t + 1}}} _ {\text {t e r m (a - 2)}}, \\ \end{array}
$$
where the first inequality is due to the mean value theorem and the condition that  $|\sigma''(z)| \leq 1$  for any  $z \in \mathbb{R}$ . The second inequality is due to the Cauchy-Schwarz inequality and  $H_t \preccurlyeq H_{t+1}$ . We can further bound term (a-1) by
$$
\operatorname {t e r m} (\mathbf {a - 1}) = \sum_ {t \in I _ {1}} \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \cdot \| \widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*} \| _ {H _ {t}} \cdot \| \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}}
$$
$$
\begin{array}{l} \leq 2 \beta_ {T} (\delta) \sqrt {\sum_ {t \in I _ {1}} \sigma^ {\prime} (\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t})} \sqrt {\sum_ {t \in I _ {1}} \| \sqrt {\sigma^ {\prime} (\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t})} \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}} ^ {2}} \\ \leq 2 \beta_ {T} (\delta) \sqrt {\sum_ {t \in I _ {1}} \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \sqrt {\sum_ {t \in I _ {1}} \| \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}} ^ {2}} \\ \leq 4 \beta_ {T} (\delta) \cdot \sqrt {\operatorname {R e g} _ {T} + T \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*}\right)} \cdot \sqrt {d \log \left(1 + \frac {T}{\lambda}\right)}, \tag {62} \\ \end{array}
$$
where the first inequality is due to  $\widetilde{\mathbf{w}}_t$  and  $\mathbf{w}_*$  are both contained in  $\mathcal{C}_t(\delta)$  and  $\beta_{t}(\delta) = \mathcal{O}(\log t)$  is the radius of  $\mathcal{C}_t(\delta)$  as shown in Theorem 3. The last second inequality is by the condition such that  $\sigma^{\prime}(\mathbf{w}_{*}^{\top}\mathbf{x}_{t})\geq \sigma^{\prime}(\mathbf{w}_{t + 1}^{\top}\mathbf{x}_{t})$  for all  $t\in I_1$ . Then, following the same argument in the proof of [10, Theorem 2], one can show
$$
\sqrt {\sum_ {t \in I _ {1}} \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \leq \sqrt {\sum_ {t \in T} \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \leq \sqrt {\operatorname {R e g} _ {T} + T \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*}\right)}, \tag {63}
$$
which leads to the last inequality. We also use the elliptical potential lemma [10, Lemma 9] in the last inequality.
As for term (a-2), we have
$$
\operatorname {t e r m} (\mathbf {a} - 2) \leq \left(\beta_ {T} (\delta)\right) ^ {2} \sum_ {t \in [ T ]} \| \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}} ^ {2} \leq \kappa d \left(\beta_ {T} (\delta)\right) ^ {2} \ln \left(1 + \frac {T}{\lambda \kappa}\right), \tag {64}
$$
where the first inequality holds because  $\mathbf{w}_{*}$  and  $\widetilde{\mathbf{w}}_t$  are contained in  $\mathcal{C}_t(\delta)$ . The second inequality can be obtained following the similar argument in obtaining (60). Combining the upper bound for term (a-1) and term (a-2), we have
$$
\mathsf {t e r m} (\mathsf {a}) \leq 4 \beta_ {T} (\delta) \sqrt {d \log (1 + \frac {T}{\lambda})} \cdot \sqrt {\operatorname {R e g} _ {T} + T \sigma^ {\prime} (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*})} + \kappa d (\beta_ {T} (\delta)) ^ {2} \ln (1 + \frac {T}{\lambda \kappa}).
$$
As for term (b), we have  $\sigma'(\mathbf{w}_*^\top \mathbf{x}_t) < \sigma'(\mathbf{w}_{t+1}^\top \mathbf{x}_t)$  for all  $t \in I_2$ . Then, we have
$$
\begin{array}{l} \text {t e r m} (\mathsf {b}) \leq \sum_ {t \in I _ {2}} \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \cdot \| \widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*} \| _ {H _ {t}} \cdot \| \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \mathbf {x} _ {t} \| _ {H _ {t} ^ {- 1}} \\ \leq \sum_ {t \in I _ {2}} \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t}\right)} \cdot \left\| \widetilde {\mathbf {w}} _ {t} - \mathbf {w} _ {*} \right\| _ {H _ {t}} \cdot \left\| \sqrt {\sigma^ {\prime} \left(\mathbf {w} _ {t + 1} ^ {\top} \mathbf {x} _ {t}\right)} \mathbf {x} _ {t} \right\| _ {H _ {t} ^ {- 1}} \\ \leq 4 \beta_ {T} (\delta) \sqrt {d \log (1 + \frac {T}{\lambda})} \cdot \sqrt {\operatorname {R e g} _ {T} + T \sigma^ {\prime} \left(\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*}\right)}, \\ \end{array}
$$
where the second inequality is due to the condition  $\sigma'(\mathbf{w}_*^\top \mathbf{x}_t) < \sigma'(\mathbf{w}_{t+1}^\top \mathbf{x}_t)$  and the last inequality can be obtained using similar arguments as those used to derive (62).
Regarding term (c), the application of similar arguments used to derive equation (62) demonstrates.
$$
\operatorname {t e r m} (c) \leq \kappa d \left(\beta_ {T} (\delta)\right) ^ {2} \ln \left(1 + \frac {T}{\lambda \kappa}\right).
$$
Then, combining the upper bounds for term (a), term (b) and term (c), we have
$$
\mathrm {R e g} _ {T} \leq 8 \beta_ {T} (\delta) \sqrt {d \log (1 + \frac {T}{\lambda})} \cdot \sqrt {\mathrm {R e g} _ {T} + T \sigma^ {\prime} (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*})} + 2 \kappa d (\beta_ {T} (\delta)) ^ {2} \ln (1 + \frac {T}{\lambda \kappa}).
$$
Resolving the above displayed inequality leads to
$$
\mathrm {R e g} _ {T} \leq 3 2 \beta_ {T} (\delta) \sqrt {d \log (1 + \frac {T}{\lambda})} \sqrt {T \sigma^ {\prime} (\mathbf {w} _ {t} ^ {\top} \mathbf {x} _ {*})} + (8 \kappa + 6 4) d (\beta_ {T} (\delta)) ^ {2} \ln (1 + \frac {T}{\lambda}).
$$
In the binary case,  $\beta_{T}(\delta) = \mathcal{O}(\sqrt{d}\log T)$ . Then we have  $\mathrm{Reg}_T = \mathcal{O}(d\log^{\frac{3}{2}}T\sqrt{T / \kappa_*} +\kappa d^2\log^3 T)$ , which completes the proof.
C.4.2 Computation Cost for Binary Case
Since the binary logistic bandit is a special case of the MLogB problem, the time complexity analysis in Appendix C.3.4 is also applicable in the binary case. The only difference is that we select the arm as  $(\mathbf{x}_t,\widetilde{\mathbf{w}} t) = \arg \max_{\mathbf{x}\in \mathcal{X},}W\in \mathcal{C}_t(\delta)\mathbf{w}^\top \mathbf{x}$  in the binary case. As  $\mathcal{C}_t(\delta)\triangleq \{\mathbf{w}\in \mathcal{W}\mid \| \mathbf{w} - \mathbf{w}_*\|_{H_t}\leq \beta_t(\delta)\}$  is an ellipsoid, the optimization can be rewritten as:
$$
\begin{array}{l} \mathbf{x}_{t} = \operatorname *{arg  max}_{\mathbf{x}\in \mathcal{X}}\left(\max_{\| \mathbf{w} - \mathbf{w}_{t}\|_{H_{t}}\leq \beta_{t}(\delta)}\mathbf{w}^{\top}\mathbf{x}\right) \\ = \operatorname * {a r g   m a x} _ {\mathbf {x} \in \mathcal {X}} \left(\max  _ {\| \mathbf {u} \| _ {H _ {t}} \leq \beta_ {t} (\delta)} \mathbf {w} _ {t} ^ {\top} \mathbf {x} + \mathbf {u} ^ {\top} \mathbf {x}\right) \\ = \operatorname * {a r g   m a x} _ {\mathbf {x} \in \mathcal {X}} \mathbf {w} _ {t} ^ {\top} \mathbf {x} + \beta_ {t} (\delta) \| \mathbf {x} \| _ {H _ {t} ^ {- 1}}. \\ \end{array}
$$
Given  $H_{t}^{-1}$ , the above optimization problem can be solved in  $\mathcal{O}(d^2)$  time.
C.5 On the Minimax Optimal Bound for the MLogB
In the binary case, we have achieved the minimax dynamic regret bound  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  up to a logarithmic factor in terms of  $T$ . In the MLogB problem, the best-known result is  $\widetilde{\mathcal{O}} (K\sqrt{T})$ . A natural question arises: can we obtain a similar minimax optimal result in the MLogB problem? However, we find that it remains challenging both in terms of regret analysis and the design of efficient algorithms.
Challenge in Regret Analysis. To establish the minimax optimal bound for the binary case, as demonstrated in [9, 10] and we restate in (63), a critical step involves showing
$$
\sqrt {\sum_ {t \in T} \sigma^ {\prime} (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {t})} \leq \sqrt {\operatorname {R e g} _ {T} + T \sigma^ {\prime} (\mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {*})}.
$$
In the MLogB problem, in order to obtain a similar result to the minimax optimal bound achieved in the binary case, it is sufficient to demonstrate
$$
\sqrt {\sum_ {t = 1} ^ {T} \boldsymbol {\rho} ^ {\top} \nabla \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right) \boldsymbol {\rho}} \leq \sqrt {2 R \cdot \operatorname {R e g} _ {T} + \sum_ {t = 1} ^ {T} \boldsymbol {\rho} ^ {\top} \Sigma \left(W _ {*} \mathbf {x} _ {*}\right) \boldsymbol {\rho}}. \tag {65}
$$
However, it is unclear how to prove such a relationship as the binary case. Indeed, denoting by  $\mathbf{r}_t = \pmb{\sigma}(W_*\mathbf{x}_*) - \pmb{\sigma}(W_*\mathbf{x}_t)$  the reward vector and  $r_{t,k}$  by its  $k$ -th entry, we can further rewrite the  $k$ -th entry of the vector  $\pmb{\rho}^\top \Sigma(W_*\mathbf{x}_t)$  can be further written as
$$
\begin{array}{l} \left[ \boldsymbol {\rho} ^ {\top} \Sigma \left(W _ {*} \mathbf {x} _ {t}\right) \right] _ {k} = \sigma_ {k} \left(W _ {*} \mathbf {x} _ {t}\right) \boldsymbol {\rho} ^ {\top} \left(\mathbf {e} _ {k} - \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right)\right) \\ = \left(\sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right) - r _ {t, k}\right) \rho_ {k} - \left(\sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right) - r _ {t, k}\right) \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {*}\right) - \mathbf {r} _ {t}\right) \\ = \sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right) \rho_ {k} - \sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right) \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {*}\right) + \left(\sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right)\right) \boldsymbol {\rho} ^ {\top} \mathbf {r} _ {t} + r _ {t, k} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right) - \mathbf {e} _ {k}\right) \\ = \left[ \boldsymbol {\rho} ^ {\top} \Sigma \left(W _ {*} \mathbf {x} _ {*}\right) \right] _ {k} + \sigma_ {k} \left(W _ {*} \mathbf {x} _ {*}\right) \boldsymbol {\rho} ^ {\top} \mathbf {r} _ {t} + r _ {t, k} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right) - \mathbf {e} _ {k}\right). \\ \end{array}
$$
In such a case, we can bound the left hand side of (65) by
$$
\begin{array}{l} \sqrt {\sum_ {t = 1} ^ {T} \boldsymbol {\rho} ^ {\top} \Sigma (W _ {*} \mathbf {x} _ {t}) \boldsymbol {\rho}} = \sqrt {\left(\boldsymbol {\rho} ^ {\top} \Sigma (W _ {*} \mathbf {x} _ {*}) \boldsymbol {\rho} + \boldsymbol {\rho} ^ {\top} \mathbf {r} _ {t} \cdot \boldsymbol {\rho} ^ {\top} (\boldsymbol {\sigma} (W _ {*} \mathbf {x} _ {t}) + \boldsymbol {\sigma} (W _ {*} \mathbf {x} _ {*})) - \sum_ {k = 1} ^ {K} \rho_ {k} ^ {2} r _ {t , k}\right)} \\ \leq \sqrt {\sum_ {t = 1} ^ {T} \boldsymbol {\rho} ^ {\top} \Sigma \left(W _ {*} \mathbf {x} _ {*}\right) \boldsymbol {\rho} + 2 R \cdot \operatorname {R e g} _ {T} - \sum_ {t = 1} ^ {T} \sum_ {k = 1} ^ {K} \rho_ {k} ^ {2} r _ {t , k}}. \tag {66} \\ \end{array}
$$
In the binary case, In the binary case, the additional term becomes  $-\sum_{t=1}^{T} r_t = -\operatorname{Reg}_T < 0$ , thereby resulting in the derivation of (63). Nevertheless, in the context of the MLogB problem, this particular term has the potential to assume positive values, posing challenges in determining an upper bound for it.
(a) configuration 4
(b) configuration 5
(c) configuration 6
(d) running time
Figure 3: More results for binary logistic bandit.
(b) configuration 5
(d) running time
Figure 4: More results for multinomial logistic bandit.
Challenge in Efficient Algorithm Design. When extending algorithms that achieve minimax optimal bounds from the binary case to the multinomial case, there are computational concerns to address. In the binary case, the minimax optimal regret bound relies on constructing an optimistic reward through  $\widetilde{r}_t(\mathbf{x}) = \arg \max_{\mathbf{w} \in \mathcal{C}_t(\delta)} \sigma(\mathbf{w}^\top \mathbf{x})$ . Since  $\sigma(\mathbf{z})$  is an increasing function, the optimization problem can be simplified to
$$
\widetilde{r}_{t}(\mathbf{x}) = \operatorname *{arg  max}_{\substack{\mathbf{w}\in \mathcal{C}_{t}(\delta)}}\mathbf{w}^{\top}\mathbf{x},
$$
which becomes a convex optimization problem when  $\mathcal{C}_t(\delta)$  is a convex set. However, when applying this analysis to the multinomial case, solving the problem requires optimizing
$$
\widetilde{r}_{t}(\mathbf{x}) = \operatorname *{arg  max}_{W\in \mathcal{C}_{t}(\delta)}\boldsymbol{\rho}^{\top}\boldsymbol {(}W\mathbf{x}).
$$
Here, the loss function is non-concave, rendering the maximization problem challenging to efficiently solve. Consequently, new approaches for constructing the optimistic reward function may be necessary.
D Omitted Details for Experiments
In this section, we presents more empirical results for other configurations and introduced the parameter configurations for the contenders.
Experimental Details. For each configuration, both the arm set  $\mathcal{X}$  and the underlying unknown parameter  $W_{*}$  are randomly selected. In the binary case, the norm of the unknown parameter is set as  $\| \mathbf{w}_*\| _2 = S$  with  $S = 5$ . As for the multinomial case, each row of  $\mathcal{W}_{*}$  are randomly sampled with  $\| \mathbf{w}_*(^k)\| _2 = \widetilde{S}$  for all  $k\in [K]$  with  $\widetilde{S} = 1$ . The parameter of all contenders are set according to their order as suggested in the corresponding paper. We use  $\lambda = d\log (T)$  for Log-UCB1,  $\lambda = d$  for OL2M,  $\lambda = 1$  for ada-OFU-ECOLog and  $\lambda = \sqrt{K}\alpha S$  for OFU-MLogB. The step size for OFUMLogB is set as  $\eta = S / 2 + \ln (K + 1) / 4$ . The experiments are run on Xeon E-2288G processors (8 cores, 3.7GHz base, 5.0GHz boost).
More Results. Figure 3 provides additional empirical results for the binary logistic bandit problem. These results align with the trends observed in the main paper, wherein our OFU-MLogB algorithm shows performance comparable to ada-OFU-ECOLog but with reduced computational overhead. Meanwhile, more results for the multinomial case are provided by Figure 4. The cumulative running time for MNL-UCB increase at a rate of  $\mathcal{O}(t^2)$ . In contrast, the running time for our OFU-MLogB exhibits a linear dependence with  $t$ , attributable to the constant computation cost per round.