SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The algorithms for logistic bandit crucially rely on two components to ensure the statistical efficiency: the MLE (3) and the optimistic rule (5) for constructing  $\widetilde{r}_t$ . However, the implementation of both components could be inefficient by requiring  $\mathcal{O}(t)$  computation cost per online iteration.
Computation and Storage Cost of Maximum Likelihood Estimation. For logistic bandit or even the generalized linear bandit problem [3, 8, 9, 11], MLE is a widely used tool to learn the unknown parameter. To solve the optimization problem, the gradient-based method, e.g. the projected gradient descent [16], are usually applied. However, as discussed in [10], the optimization of the MLE problem typically requires  $\mathcal{O}(t\log (1 / \epsilon))$  gradient step to achieve  $\epsilon$ -accuracy. Besides, the loss function  $\mathcal{L}_t$  is established on all historical data  $\{(x_s,y_s)\}_{s = 1}^{t - 1}$ , resulting in an  $\mathcal{O}(t)$  gradient query complexity for each gradient step and  $\mathcal{O}(t)$  storage cost, and thus is inefficient.
Computation and Storage Cost of Optimistic Reward Construction. The construction of the optimistic reward  $\widetilde{r}_t(\mathbf{x})$  requires to solve the optimization problem (5). However, the objective
Algorithm 2 OFUL-MLogB
Input: regularization coefficient  $\lambda$ , probability  $\delta$ , step size  $\eta$ .  
1: Initialize  $H_{1} = \lambda I_{Kd}$  and  $\overrightarrow{W}_{1}^{\mathrm{OL}}$  as any point in  $\mathcal{W}$   
2: for  $t = 1, \dots, T$  do  
3: Select the arm by  $\mathbf{x}_t = \arg \max_{\mathbf{x} \in \mathcal{X}} \widetilde{r}_t(\mathbf{x})$  and receive  $y_t$ .  
4: Update  $\widetilde{H}_t = H_t + \eta \nabla \sigma(W_{t+1}^{\mathrm{OL}}\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top$   
5: Update the estimator  $\overrightarrow{W}_{t+1}^{\mathrm{OL}}$  for the next iteration by (6)  
6: Update  $H_{t+1} = H_t + \nabla \sigma(W_{t+1}^{\mathrm{OL}}\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top$  and  
7: Construct the optimistic reward by  $\widetilde{r}_{t+1}(\mathbf{x}) = \boldsymbol{\rho}^\top \boldsymbol{\sigma}(W_{t+1}^{\mathrm{OL}}\mathbf{x}) + \epsilon_{t+1}^{\mathrm{fst}}(\mathbf{x}) + \epsilon_{t+1}^{\mathrm{snd}}(\mathbf{x})$  as (8).  
8: end for
function  $\pmb{\rho}^{\top}\pmb{\sigma}(W\mathbf{x})$  is non-concave and the decision domain  $\mathcal{C}_t(\delta)$  is non-convex, making the maximization problem  $\widetilde{r}_t(\mathbf{x}) = \arg \max_{W\in \mathcal{C}_t(\delta)}\pmb{\rho}^{\top}\pmb{\sigma}(W\mathbf{x})$  computationally challenging. In the binary case, the paper [9] proposed a convex relaxation of the confidence set  $\mathcal{C}_t(\delta)$ , whereas the relaxed confidence set is still established on all historical data, resulting in  $\mathcal{O}(t)$  time complexity and  $\mathcal{O}(t)$  storage cost at iteration  $t$ . The optimistic estimate construction in the previous work for multinomial logistci bandits [11] is computationally efficient without involving any optimization problem solving. However, it will lead to an inferior regret bound of  $\widetilde{\mathcal{O}} (\sqrt{\kappa T})$ .
In the binary case, the work [10] proposed a jointly efficient algorithm, which achieved a nearly minimax optimal bound with  $\mathcal{O}(\log t)$  computation cost per iteration and  $\mathcal{O}(1)$  storage cost. However, as we will discuss in Section 3.3, it is hard to apply their analysis to the multinomial case.
3.2 Efficient Algorithm
In this section, we proposed a novel algorithm which only requires  $\mathcal{O}(1)$  computation cost per iteration and  $\mathcal{O}(1)$  storage cost. The algorithm can achieve the best known results both for binary and multinomial logistic bandits. We have introduced new ingredients both on the algorithm design and regret analysis to achieve the jointly efficient algorithm.
Efficient Online Estimation. Instead of performing MLE, we run an online mirror descent algorithm to estimate parameter:
$$
\overrightarrow {W} _ {t + 1} ^ {0 \mathrm {L}} = \underset {W \in \mathcal {W}} {\arg \min } \langle \nabla \ell_ {t} (\overrightarrow {W} _ {t} ^ {0 \mathrm {L}}), \overrightarrow {W} \rangle + \frac {1}{2 \eta} \| \overrightarrow {W} - \overrightarrow {W} _ {t} ^ {0 \mathrm {L}} \| _ {\bar {H} _ {t}} ^ {2}, \forall t \geq 1 \tag {6}
$$
where  $\eta > 0$  is the step size to be specified later and the first iteration model  $\overrightarrow{W}_1^{0\mathrm{L}}$  can be initialized as any point in the domain  $\mathcal{W} = \{\overrightarrow{W} \in \mathbb{R}^{Kd} \mid \| \overrightarrow{W} \|_2 \leq S\}$ . We set the matrix as  $\widetilde{H}_t = H_t + \eta \nabla \sigma(W_t^{0\mathrm{L}}\mathbf{x}_t) \otimes \mathbf{x}_t\mathbf{x}_t^\top$ , where  $H_t = \lambda I + \sum_{s=1}^{t-1} \nabla \sigma(W_{s+1}^{0\mathrm{L}}\mathbf{x}_s) \otimes \mathbf{x}_s\mathbf{x}_s^\top$ . Both  $\widetilde{H}_t$  and  $H_t$  can be updated incrementally.
We show the online estimator (6) enjoys computational, storage, and statistical efficiency. Since (6) exhibits a standard online mirror descent formulation [17], it can be solved with a single projected gradient step with the following equivalent formulation by
$$
\vec {\mathcal {Z}} _ {t + 1} = \overrightarrow {W} _ {t} ^ {0 \mathrm {L}} - \eta \widetilde {H} _ {t} ^ {- 1} \nabla \ell_ {t} (\overrightarrow {W} _ {t} ^ {0 \mathrm {L}}) \quad \text {a n d} \quad \overrightarrow {W} _ {t + 1} ^ {0 \mathrm {L}} = \arg \min  _ {\vec {W} \in \mathcal {W}} \| \overrightarrow {W} - \vec {\mathcal {Z}} _ {t + 1} \| _ {\tilde {H} _ {t}}.
$$
For the gradient descent step above, the most time-consuming operation is maintaining the inverse of the matrix  $\widetilde{H}_t$ . Since  $\nabla \sigma (W_t^{\mathrm{OL}}\mathbf{x}_t)\otimes \mathbf{x}_t\mathbf{x}_t^\top$  is a rank-  $K$  matrix, it can be calculated by the Sherman-Morrison-Woodbury formula with  $\mathcal{O}(d^2 K^3)$  cost per round. As for the projection step, since  $\widetilde{H}_t$  is positive semi-definite matrix, it can be solved in  $\mathcal{O}(K^3 d^3)$  [18, Section 4].* As a consequence, our algorithm achieves a light update with  $\mathcal{O}(1)$  cost per round. Regarding storage cost, our proposed estimator eliminates the need to store all historical data and updates in a one-pass fashion, requiring only  $\mathcal{O}(1)$  storage cost throughout the learning process. Moreover, the estimator is also statistically efficient. We can construct the following  $\kappa$ -independent confidence set similar to that in Theorem 1.
*In the high-dimensional case, one can also employ Lemma 13 of [10] to perform the projection step, which ensures  $1 / \tau$ -error with  $\mathcal{O}(d^2 \log \tau)$  computation complexity per iteration.
Theorem 3. Let  $\delta \in (0,1]$  and  $\alpha = 2(1 + S) + \ln(K + 1)$ . Set the parameter  $\eta = \alpha / 2$  and  $\lambda = \max\{28Kd\alpha, 7\sqrt{6}\alpha S\}$ . For each iteration  $t \in [T]$ , we define the confidence set as
$$
\mathcal {C} _ {t} ^ {\mathrm {O L}} (\delta) \triangleq \left\{W \in \mathcal {W} \left| \| \vec {W} _ {t} ^ {\mathrm {O L}} - \vec {W} \| _ {H _ {t}} \leq \beta_ {t} ^ {\mathrm {O L}} (\delta) \right. \right\}, \tag {7}
$$
where  $\beta_t^{\mathrm{OL}}(\delta) = \mathcal{O}\bigl (\log K\log t\sqrt{Kd}\bigr)$ . Then, we have  $\operatorname *{Pr}\big[\forall t\geq 1,W_{*}\in \mathcal{C}_{t}(\delta)\big]\geq 1 - \delta$ . Moreover, the computation cost of solving (6) is  $\mathcal{O}(1)$  per round.
Remark 2 (Comparison with the online estimator [10]). Our algorithm design is inspired by the online estimator [10] developed for the binary case, while achieving even lighter cost by novel algorithm ingredients and analysis. As discussed in Section 3.3, the analysis for the binary setting is hard to be applied to the multinomial case. Specifically, the paper [10] has proposed an intermediary decision  $\overline{W}_t$  in the analysis to prove the statistical efficiency of the proposed estimator. However, the favorable property of the intermediate decision only holds in the binary case. To this end, we have proposed a new intermediary decisions, which not only help to prove the statistically efficient of our estimator but also eliminate the requirement of the exploration step. Besides, we have also introduced a novel algorithm ingredient to further speed-up the algorithm. Instead of learning with original loss function as in [10], by a more refined exploitation of the negative term in the analysis, we show it is sufficient to learn with the first order approximate  $\langle \nabla \ell_t(\vec{W}_t^{\mathrm{OL}}),\vec{W}\rangle$  of  $\ell_t(W)$  with the adjusted local norm  $\| \cdot \|_{\widetilde{H}_t}$ . Our new algorithm not only enjoys a computation efficiency improvement from  $\mathcal{O}(\log t)$  in [10] to  $\mathcal{O}(1)$ , but also is free from any exploration step required by the previous work. We provide a technical highlight in Section 3.3.
Efficient Optimistic Reward Construction. Although the confidence set  $\mathcal{C}_t^{\mathrm{OL}}(\delta)$  is convex, the optimistic rule (5) by  $\widetilde{r}_t(\mathbf{x}) = \arg \min_{W\in \mathcal{C}_t^{\mathrm{OL}}(\delta)}\pmb {\rho}^\top \pmb {\sigma}(W\mathbf{x})$  still involves inefficient non-concave optimization problem solving. In this part, we propose a novel optimistic reward that can be solved in a constant time per round.
Proposition 1. For any  $\mathbf{x} \in \mathcal{X}$  and iteration  $t \in [T]$ , the optimistic reward is constructed by
$$
\widetilde {r} _ {t} ^ {\mathrm {O L}} (\mathbf {x}) = \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(W _ {t} ^ {\mathrm {O L}} \mathbf {x}\right) + \epsilon_ {t} ^ {\mathrm {f s t}} (\mathbf {x}) + \epsilon_ {t} ^ {\mathrm {s n d}} (\mathbf {x}). \tag {8}
$$
In above,  $\epsilon_t^{\mathrm{fst}}(\mathbf{x}) = \beta_t^{0\mathrm{L}}(\delta) \cdot \| H_t^{-\frac{1}{2}}(I_K \otimes \mathbf{x}) \nabla \sigma(W_t^{0\mathrm{L}}\mathbf{x})\pmb{\rho}\|_2$  and  $\epsilon_t^{\mathrm{snd}}(\mathbf{x}) = 3R(\beta_t^{0\mathrm{L}})^2 \cdot \| (I_K \otimes \mathbf{x}^\top)H_t^{-1/2}\|_2^2$  are the bonus. Then, we have  $\widetilde{r}_t^{0\mathrm{L}}(\mathbf{x}) \geq \pmb{\rho}^\top \pmb{\sigma}(W_*\mathbf{x})$  for all  $t \geq 1$  and  $\mathbf{x} \in \mathcal{X}$  with probability at least  $1 - \delta$ .
Proposition 1 constructs the optimistic reward by adding the "bonus" to the reward empirically estimated by  $W_{t}^{\mathrm{OL}}$ . Different from the term used in [11], our bonus terms are independent of the exponentially large constant and thus can lead to an improved  $\tilde{\mathcal{O}}(K\sqrt{T})$  bound. The optimistic rule (8) does not involve any optimization problem solving and can be calculated in an  $\mathcal{O}(1)$  cost.
Overall Algorithm and Guarantees. We overall procedures in Algorithm 2. For the general multinomial setting, it ensures an  $\tilde{\mathcal{O}}(K\sqrt{T})$  regret guarantee and an  $\mathcal{O}(1)$  computation cost.
Theorem 4. Under the same condition as Theorem 3, Algorithm 2 ensures
$$
\operatorname {R e g} _ {T} = \mathcal {O} \left(K d \log K (\log T) ^ {\frac {3}{2}} \sqrt {T} + \kappa K ^ {\frac {3}{2}} d ^ {2} (\log K) ^ {2} (\log T) ^ {3}\right) = \widetilde {\mathcal {O}} (K \sqrt {T}).
$$
The computation cost of Algorithm 2 is bounded by  $\mathcal{O}(1)$  for each round  $t\in [T]$ .
Remark 3 (On the  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  bound). In the binary setting, [9, 10] show that an  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  minimax optimal is achievable with  $\kappa_{*} = 1 / \sigma^{\prime}(\mathbf{w}_{*}^{\top}\mathbf{x}_{*})$ . However, due to the multinomial behavior of the feedback, it is unclear how to achieve such a rate in MLogB case (see the discussion in Appendix C.5). Besides, it also raises concerns about efficiency when applying the method developed for binary case to multinomial setting. In particular, the  $\widetilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  is achieved by the rule  $\arg \max_{\mathbf{x}\in \mathcal{X},\mathbf{w}\in \mathcal{C}_t(\delta)}\sigma (\mathbf{w}^\top \mathbf{x})$  in [9, 10]. The optimization can be efficiently solved in the binary case since one can simply eliminate the non-linearity of the reward function by the relationship  $\sigma (z_{1}) > \sigma (z_{2})$  for any  $z_{1}\in \mathbb{R} > z_{2}\in \mathbb{R}$ . Such a condition does not hold in MLogB problem.
When reduced to the binary case  $K = 1$ , our algorithm can also achieve the minimax regret bound.
Corollary 1. When  $K = 1$ , the multinomial logistic bandit reduces to the binary logistic bandit problem. Then, under the same conditions as Theorem 4, Algorithm 2 with the optimistic rule  $\widetilde{r}_t(\mathbf{x}) = \arg \max_{\mathbf{w} \in \mathcal{C}_t^{\mathrm{OL}}(\delta)} \mathbf{w}^\top \mathbf{x}$  ensures  $\operatorname{Reg}_T \leq \tilde{\mathcal{O}}(\sqrt{T / \kappa_*})$  with probability at least  $1 - \delta$ .
This section presents the proof sketch for Theorem 3, which plays an key role in the analysis for Algorithm 2. For notational simplicity, we will drop the superscript OL in this section.
Leveraging Negative Terms for Efficient Update. If we update the model with the original loss function by  $\overrightarrow{W}_{t+1} = \arg \min_{W \in \mathcal{W}} \ell_t(W) + \frac{1}{2\eta} \|\overrightarrow{W} - \overrightarrow{W}_t\|_{H_t}^2$ , the arguments in [10] show that the estimation error between  $\overrightarrow{W}_{t+1}$  and  $\overrightarrow{W}_*$  can be bounded by their gap on the loss function
$$
\left\| \overrightarrow {W} _ {t + 1} - \overrightarrow {W} _ {*} \right\| _ {H _ {t + 1}} ^ {2} \lesssim \sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {*}\right) - \sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {s + 1}\right). \tag {9}
$$
However, the update rule with original loss will lead to an  $\mathcal{O}(\log t)$  computation cost per iteration. To facilitate a more efficient algorithm, we introduce the update rule with the linearized loss  $\langle \nabla \ell_t(\vec{W}_t),\vec{W}\rangle$  and the adjusted norm  $\widetilde{H}_t$  as (6). As we have shown in the proof of Lemma 12 in Appendix C.1. Denote by  $\widetilde{\ell}_{t}(W) = \langle \nabla \ell_{t}(\vec{W}_{t}),\vec{W}\rangle +\frac{1}{2}\| \vec{W} -\vec{W}_{t}\|_{\nabla^{2}\ell_{t}(\vec{W}_{t})}^{2}$  the second-order surrogate of  $\ell_t(W)$ . The efficient update rule will introduce an additional term
$$
\sum_ {s = 1} ^ {t} \langle \nabla \ell_ {s} (\overrightarrow {W} _ {s + 1}) - \nabla \widetilde {\ell} _ {s} (\overrightarrow {W} _ {s + 1}), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \rangle .
$$
We handle the additional term by the self-concordant property of the logistic loss and exploiting a negative term ignored in the previous analysis. Since the logistic loss is a  $\sqrt{6}$ -self-concordant-like function [19, Lemma 4], then Theorem 3 of [19] indicates that the additional term can be bounded by
$$
\sum_ {s = 1} ^ {t} \langle \nabla \ell_ {t} (\overrightarrow {W} _ {s + 1}) - \nabla \widetilde {\ell} _ {s} (\overrightarrow {W} _ {s + 1}), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \rangle \leq \sum_ {s = 1} ^ {t} \sqrt {6 K} S \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {\nabla^ {2} \ell_ {s} (\xi_ {s})} ^ {2},
$$
where  $\xi_s \in \mathbb{R}^{Kd}$  is on the line connecting  $\overrightarrow{W}_{s+1}$  and  $\overrightarrow{W}_s$ . Besides, by a refined analysis of the online mirror descent (OMD) update (6), we identify an additional negative term  $-\sum_{s=1}^{t} \|\overrightarrow{W}_{s+1} - \overrightarrow{W}_s\|_{H_s}^2$  on the right hand side of (9). By properly choosing the coefficient  $\lambda$ , one can cancel the additional term by the negative term and achieves (9) with the efficient update. We note that the negative term in the OMD analysis is also found crucial in the gradient-variation regret of non-stationary online learning [20, 21] as well as its applications to game theory [22] and the SEA model [23].
Novel construction of the intermediary prediction. Then, we can further bound the right hand side of (9) by inserting an intermediary loss  $\ell_s(\widetilde{W}_s)$  as
$$
\sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1}) = \underbrace {\sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell_ {s} (\widetilde {W _ {s}})} _ {\text {t e r m (A)}} + \underbrace {\sum_ {s = 1} ^ {t} \ell_ {s} (\widetilde {W _ {s}}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1})} _ {\text {t e r m (B)}}.
$$
In the binary case, inspired by the study [24] for the binary online logistic regression, [10] propose to construct  $\widetilde{\mathbf{w}}_s = \arg \min_{\mathbf{w}\in \mathcal{W}_s}\ell_{\mathsf{b}}(\mathbf{w}^\top \mathbf{x}_s, + 1) + \ell_{\mathsf{b}}(\mathbf{w}^\top \mathbf{x}_s,0) + \frac{1}{2\eta}\| \mathbf{w} - \mathbf{w}_s\|_{H_s}^2$  , where  $\ell_{\mathrm{b}}$  is the binary logistic loss and  $\mathcal{W}_s$  is a shrank decision domain. Then, both term (A) and term (B) can be bounded without the dependence on  $\kappa$ . However, the analysis for term (B) crucially relies on the condition  $|\sigma (\widetilde{\mathbf{w}}_s^\top \mathbf{x}_s) - y_s|\cdot |\sigma (\widetilde{\mathbf{w}}_s^\top \mathbf{x}_s) - 1 + y_s| = \sigma '( \widetilde{\mathbf{w}}_s^\top \mathbf{x}_s)$  to eliminate the dependence on  $\kappa$  where  $y_{s}\in \{0,1\}$  is the one-dimensional feedback. It is hard to show such an relationship in the multinomial case since the feedback  $y_{s}$  has multiple value. A similar challenge is also observed in the recent study on online multiclass logistic regression [25, Appendix F]. One might consider whether it is possible to construct  $\widetilde{W_t}$  with the update rule developed in [25]. However, since the online update rule of [25] requires to perform over  $\mathbb{R}^{K\times d}$ , the learned parameter would become unbounded, which makes it is hard to provide an upper bound for term (A).
To this end, we design a new intermediary term by  $\ell_{\mathfrak{m}}(\widetilde{\mathbf{z}}_s,y_s)$ , where  $\ell_{\mathfrak{m}}$  is the multiclass logistic loss. The prediction is constructed by  $\widetilde{\mathbf{z}}_s = \pmb{\sigma}^+$  ( $\mathbb{E}_{W\sim P_s}[\pmb {\sigma}(W\mathbf{x}_s)]$ ) with the Gaussian distribution  $P_{s} = \mathcal{N}(\vec{W}_{s},\alpha H_{s}^{-1})$ , where  $\pmb{\sigma}^{+}$  is a pseudo inverse function of the sigmoid function  $\pmb{\sigma}$ . Such
an integral construction  $\widetilde{\mathbf{z}}_s = \pmb{\sigma}^+(\mathbb{E}_{W\sim P_s}[\pmb{\sigma}(W\mathbf{x}_s)])$  is previously used in the online logistic regression literature [26, 27] but we tailored the tool to our analysis with a different construction of the distribution  $P_{s}$ . Lemma 13 and Lemma 14 in Appendix C.1 shows that
$$
\text {t e r m} (\mathbb {A}) \lesssim (\log K + \log t) \log t \quad \text {a n d} \quad \text {t e r m} (\mathbb {B}) \lesssim \sum_ {s = 1} ^ {t} \| \overrightarrow {W} _ {s} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2} + K d \log K \log t,
$$
Then, combining the upper bound of term (A) and term (B) and eliminating the additional term  $\sum_{s=1}^{t}\|\overrightarrow{W}_s - \overrightarrow{W}_{s+1}\|_{H_s}^2$  with the negative term obtained by a refined analysis of online mirror descent, we complete the proof of Theorem 3.
4 More Discussions and Related Work
This section begins with a discussion on the tightness of the proposed bounds.
On the Tightness of Our Bounds. In this paper, we introduced OFUL-MLogB, a jointly efficient algorithm that simultaneously achieves regret bounds of  $\widetilde{\mathcal{O}} (\sqrt{\kappa KT})$  and  $\widetilde{\mathcal{O}} (K\sqrt{T})$ . The tightness of these bounds, with respect to  $\kappa$  and  $T$ , was detailed in Remark 3. Regarding the number of feedback values  $K$ , [11] claimed the optimality of a linear dependence on  $K$ . However, our findings revealed a nuanced interplay between  $\kappa$  and  $K$ . The  $\widetilde{\mathcal{O}} (\sqrt{\kappa KT})$  bound does not conflict with the lower bound argument by [11], given that the non-linear constant  $\kappa$  is also associated with  $K$ . Beyond  $\kappa$ , the norm of the unknown parameter  $S$  and maximum norm  $R$  of the reward vector  $\rho$  can also depend on  $K$  based on the problem's specifics. It is an interesting direction to understand the interrelation of these constants by establishing a lower bound. Additionally, our method has a linear dependence on  $d$ . In the finite-arm case, an  $\mathcal{O}(\sqrt{d})$  dependence might be attainable with a SupLin-type algorithm [28].
Below, we introduce more related works on logistic bandit and the related topics.
Logistic Bandit. While the logistic bandit is a specific instance of the generalized linear bandit model [5, 29, 30, 31, 32, 33, 34], the algorithms proposed for GLB tend to exhibit a linear dependence on the nonlinear term  $\kappa$ , which is exponentially large in the logistic bandit case. Therefore, addressing the non-linearity of the reward function warrants specialized consideration. Besides the UCB-type algorithms [8, 9] mentioned in Section 1, for the  $N$ -arm case, [35] proposed an experimental design-based algorithm providing an  $\widetilde{\mathcal{O}}(\sqrt{d\log NT / \kappa_*})$  regret bound with better dependence on  $d$ . However, the previous methods were built upon the MLE estimator, whose optimization demands  $\mathcal{O}(t)$  computation and storage complexity for the  $t$ -th iteration. To the best of our knowledge, the only known jointly computational and statistical efficient algorithm was proposed by [10], which achieves a nearly minimax optimal regret bound with computation cost of  $\mathcal{O}(\log t)$  per round. In addition to the frequentist bounds, there are also researches on logistic bandit from the Bayesian view. [36] showed the Bayesian regret of the Thompson sampling method is independent of  $\kappa$  (even in the lower order term) when the feasible domain is identical to the parameter domain, i.e.,  $\mathcal{X} = \mathcal{W}$ . [37] further proved the  $\kappa$ -independent bound with weaker conditions.
Multinomial Logit (MNL) Bandit. Another relevant line of research is the multinomial logit contextual bandit problem [38, 39, 40, 41, 42], which generalizes the binary logistic bandit by allowing the learner to submit a subset of arms  $S_{t} = \{\mathbf{x}_{t,i}\}_{i=1}^{K} \in \mathcal{X}$  to the users. The expected reward function is also modeled by the multinomial logit model:  $\mathbb{E}[r_{t}|S_{t}] = \sum_{\mathbf{x}_{t,i} \in S_{t}} \rho_{t,i} \exp(\mathbf{w}_{*}^{\top} \mathbf{x}_{t,i}) / (1 + \sum_{\mathbf{x}_{t,i} \in S_{t}} \exp(\mathbf{w}_{*}^{\top} \mathbf{x}_{t,i}))$ , where  $\rho_{t,i}$  is the reward for arm  $\mathbf{x}_{t,i}$  and  $\mathbf{w}_{*} \in \mathbb{R}^{d}$  is an unknown parameter. There are also studies on the MNL bandit problem concerning the exponentially large constant  $\kappa$ . [42] proposed an optimistic algorithm with  $\mathcal{O}(d\sqrt{T})$  regret bounds without  $\kappa$  in its leading term, which improves the  $\mathcal{O}(d\kappa\sqrt{T})$  bound with better dependence on  $\kappa$ . Considering uniform reward, i.e.,  $\rho_{t,i} = 1$  for all  $t \in [T]$  and  $i \in [N]$ , [43] further showed an  $\widetilde{\mathcal{O}}(d\sqrt{T/\kappa_{*}})$  bound. To the best of our knowledge, all the existing methods with improved  $\kappa$  are established on the MLE estimator. It would be an interesting future direction to develop jointly efficient algorithm for the MNL bandit problem.
Theorem 5 in Appendix C.3.2 shows that OFUL-MLogB also attains the  $\widetilde{\mathcal{O}} (\sqrt{\kappa KT})$  regret bound.
(a) configuration 1
(b) configuration 2
(c) configuration 3
(d) total running time
Figure 1: Performance and computation cost comparison for binary logistic bandit.
(a) configuration 1
Figure 2: Performance and computation cost comparison for multinomial logistic bandit.