SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu-Jie Zhang<sup>1</sup> Masashi Sugiyama<sup>2,1</sup>
This paper investigates the logistic bandit problem, a variant of the generalized linear bandit model that utilizes a logistic model to depict the feedback from an action. While most existing research focuses on the binary logistic bandit problem, the multinomial case, which considers more than two possible feedback values, offers increased practical relevance and adaptability for use in complex decision-making problems such as reinforcement learning. In this paper, we provide an algorithm that enjoys both statistical and computational efficiency for the logistic bandit problem. In the binary case, our method improves the state-of-the-art binary logistic bandit method by reducing the per-round computation cost from  $\mathcal{O}(\log T)$  to  $\mathcal{O}(1)$  with respect to the time horizon  $T$ , while still preserving the minimax optimal guarantee up to logarithmic factors. In the multinomial case, with  $K + 1$  potential feedback values, our algorithm achieves an  $\widetilde{\mathcal{O}}(K\sqrt{T})$  regret bound with  $\mathcal{O}(1)$  computational cost per round. The result not only improves the  $\widetilde{\mathcal{O}}(K\sqrt{\kappa T})$  bound for the best-known tractable algorithm—where the large constant  $\kappa$  increases exponentially with the diameter of the parameter domain—but also reduces the  $\mathcal{O}(T)$  computational complexity demanded by the previous method.
The stochastic linear bandit (SLB) [1, 2, 3] problem is a natural generalization of the classic stochastic multi-armed bandit problem [4] by incorporating side information into the decision-making process. In the SLB problem, a linear model is used to characterize the relationship between the reward  $r_t \in \mathbb{R}$  and the learner's action  $\mathbf{x}_t \in \mathcal{X} \subseteq \mathbb{R}^d$ , whereas such an assumption is not always satisfied in real-world applications. Consequently, various models have been developed to account for the non-linear reward, including the generalized linear bandit (GLB) model [5] and kernelized bandit model [6]. The logistic bandit is a specific kind of GLB model by connecting the learner's  $d$ -dimensional action and the reward with a logistic model. Most existing work focuses on the binary case [7, 8, 9, 10]. The reward  $r_t \in \{0, 1\}$  exhibits a binary value and the probability is modeled by  $\operatorname*{Pr}[r_t = 1 | \mathbf{x}_t] = \sigma(\mathbf{w}_*^\top \mathbf{x}_t)$ , where  $\sigma(z) = 1 / (1 + \exp(-z))$  is a non-linear link function and  $\mathbf{w}_* \in \mathcal{W} \subseteq \mathbb{R}^d$  is an unknown parameter. Compared to the SLB model, the logistic bandit model provides a more precise representation for a wide range of real-world application problems, where feedback exhibits discrete behavior. Moreover, from a theoretical perspective, it also serves as a basic setting for understanding the impact of non-linearity of the reward on the decision-making process. In this paper, we investigate a more general multinomial logistic bandit (MLogB) problem [11], in which the learner's action  $\mathbf{x}_t$  results in feedback  $y_t$  that could have  $K + 1$  possible outcome values. The probability of each outcome is characterized with a logistic model (the formal definition is provided in Section 2.1). The MLogB model is of more practical interest compared to the binary case. For example, in the real-world application such as online advertising, there could be multiple possible feedback from customers, including "buy now", "add to cart", "view related item", and
Table 1: Comparison in terms of the regret bound, computation cost and storage cost. For the regret bound, the logarithmic dependence on the time horizon  $T$  is hidden by the  $\widetilde{\mathcal{O}}(\cdot)$ -notation. As for the computational cost (abbreviated as "Comput.") and storage cost, we only keep the dependence on the time step  $t$ . Notation “-” denotes that the algorithm is intractable for implementation.
Setting	Algorithm	Regret	Comput. per Round	Storage Cost	Improved κ	Constant Cost
binary	Logistic-UCB-1 [8]	O(√κT)	O(t)	O(t)	×	×
OFULog-r [9]	O(√T/κ*)	O(t)	O(t)	✓	×
(ada)-OFU-ECOLog [10]	O(√T/κ*)	O(log t)	O(1)	✓	×
OL2M [7],GLOC [14]	O(κ√T)	O(1)	O(1)	×	✓
OFU-MLogB (Corollary 1)	O(√T/κ*)	O(1)	O(1)	✓	✓
multinomial	MNL-UCB [11]	O(K√κT)	O(t)	O(t)	×	×
Improved MNL-UCB [11]	O(K3/2√T)	—	O(t)	✓	-
OFUL-MLogB (Theorem 4)	O(K√T)	O(1)	O(1)	✓	✓
just leave without any click. Beyond addressing practical demands, studying the MLogB problem can shed light on other decision-making problems. For instance, in the theoretical reinforcement learning research [12, 13], the transition matrix is approximated using a linear model, which enables the application of the SLB method for balancing the exploration-exploitation trade-off. Since the transition matrix is inherently a probability matrix, the multinomial logistic model may be a more suitable structure for modeling transition probabilities between states.
As shown in Table 1, the logistic bandit problem has received much attention in the binary case. There are two main focuses arising from the non-linearity of the reward function: statistical efficiency and resource overhead. Regarding the statistical efficiency, a main focus is the algorithms' dependence on  $\kappa = \max_{\mathbf{x} \in \mathcal{X}, \mathbf{w} \in \mathcal{W}} 1 / \sigma'( \mathbf{w}^\top \mathbf{x})$ , a crucial parameter capturing the non-linearity of the reward function. Since the binary logistic bandit is a special case of the generalized linear model, the result from [5] implies an  $\widetilde{\mathcal{O}}(\kappa \sqrt{T})$  bound. However, the parameter  $\kappa$  grows exponentially in terms of the diameter of the decision domain  $\mathcal{W}$  and action space  $\mathcal{X}$ , making the linear dependence unfavorable. A pivotal advancement is made by [9], where the paper presents an algorithm that achieves the nearly minimax optimal bound  $\widetilde{\mathcal{O}}(\sqrt{T / \kappa_*})$ . Here,  $\kappa_* = 1 / \sigma'( \mathbf{w}_*^\top \mathbf{x}_*)$  is the non-linear parameter associated with the best action  $\mathbf{x}_* = \arg \max_{\mathbf{x} \in \mathcal{X}} \sigma(\mathbf{w}_*^\top \mathbf{x})$ , suggesting that non-linearity might be advantageous for statistical efficiency, rather than a hindrance. Alongside statistical efficiency considerations, the non-linearity of the feedback raises concerns about the algorithms' computation and storage efficiency. The methods [8, 9] with improved dependence on  $\kappa$  usually require storing and optimizing over all historical data to estimate the unknown parameter, leading to an  $\mathcal{O}(t)$  computation and storage cost at round  $t \in [T]$ . The pioneering work [7] provided the first efficient solution for binary logistic bandit with constant computation and storage costs and [14] further proposed an efficient algorithm for generalized linear bandit, but their regret bounds still exhibit a linear dependence on  $\kappa$ . Recently, a jointly efficient algorithm was proposed by [10], which achieves the optimal dependence on  $\kappa$  with a  $\mathcal{O}(\log t)$  computation cost and constant storage cost per round. However, it remains open whether the minimax optimal bound is achievable with constant computation cost independent of the time  $t$ .
Regrading the multinomial logistic bandit, the best-known feasible algorithm was proposed by [11]. This method achieves an  $\widetilde{\mathcal{O}}(K\sqrt{\kappa T})$  regret bound, bearing an  $\mathcal{O}(\sqrt{\kappa})$  dependence on the exponentially large constant. Moreover, it still demands  $\mathcal{O}(t)$  computation and storage costs to optimize over all past data. The same study also introduced an  $\widetilde{\mathcal{O}}(K^{3/2}\sqrt{T})$  bound with improved dependence on  $\kappa$ . Yet, this solution leans heavily towards theoretical insights and is intractable in implementation [11, Section 2.6]. Designing a practical or more efficient algorithm with improved dependence on  $\kappa$  is still an unsolved challenge. More discussions on the related work and topics can be found in Section 4.
Our Results. In this paper, we provide an algorithm with both statistical and computational efficiency.
- For the multinomial logistic bandit, we propose OFUL-MLOGB, a jointly efficient method attaining an  $\tilde{\mathcal{O}}(K\sqrt{T})$  regret bound with  $\mathcal{O}(1)$  in  $T$  computation and storage cost per round. The result improves the previous work on the dependence of the large constant  $\kappa$ .
- For the binary case, our proposed OFUL-MLOGB can achieve the  $\tilde{\mathcal{O}} (\sqrt{T / \kappa_{*}})$  optimal bound up to logarithmic factors. Besides, our method reduces the computation cost of the state-of-the-art binary method [10] from  $\mathcal{O}(\log t)$  to  $\mathcal{O}(1)$  per round.
2 Multinomial Logistic Bandit with Improved Regret
This section provides preliminaries on the multinomial logistic bandit problem and optimistic algorithms, beginning with the problem formulation and notations. Then, we revisit the optimistic algorithms for the MLogB problem. Specifically, we investigate the previously best-known feasible algorithm, MNL-UCB algorithm [11], and propose an improved version with better dependence on the exponentially large constant  $\kappa$  and the number of outcome values  $K$ .
The multinomial logistic bandit (MLogB) problem studies a  $T$  round decision-making process between the learner and the environment. At the beginning of the iteration  $t$ , the learner will first select an action  $\mathbf{x}_t \in \mathcal{X}$  from the feasible action set  $\mathcal{X} \subseteq \mathbb{R}^d$  and then submit it to the environment. After that, a response  $y_t \in \{0\} \cup [K]$  with  $K + 1$  possible outcomes (like "buy now", "add to chart", or "do nothing") is returned based on the learner's choice, where  $K \in \mathbb{N}$ . Specifically, the MLogB problem assumes that each outcome  $k \in [K]$  is associated with a ground-truth parameter  $\mathbf{w}_*^{(k)} \in \mathbb{R}^d$  and the probability of the outcome  $\operatorname*{Pr}[y_t = k \mid \mathbf{x}_t]$  follows the logistic model,
$$
\operatorname * {P r} [ y _ {t} = k \mid \mathbf {x} _ {t} ] = \frac {\exp \left((\mathbf {w} _ {*} ^ {(k)}) ^ {\top} \mathbf {x} _ {t}\right)}{1 + \sum_ {j = 1} ^ {K} \exp \left((\mathbf {w} _ {*} ^ {(j)}) ^ {\top} \mathbf {x} _ {t}\right)} \quad \text {a n d} \quad \operatorname * {P r} [ y _ {t} = 0 \mid \mathbf {x} _ {t} ] = 1 - \sum_ {k = 1} ^ {K} \operatorname * {P r} [ y _ {t} = k \mid \mathbf {x} _ {t} ].
$$
For notational simplicity, we denote by  $W_{*} = [\mathbf{w}_{*}^{(1)},\dots ,\mathbf{w}_{*}^{(K)}]^{\top}\in \mathbb{R}^{K\times d}$  the matrix for the unknown parameter and define the softmax function  $\pmb {\sigma}:\mathbb{R}^K\mapsto [0,1]^K$  by
$$
\left[ \boldsymbol {\sigma} (\mathbf {z}) \right] _ {k} = \frac {\exp \left(\left[ \mathbf {z} \right] _ {k}\right)}{1 + \sum_ {j = 1} ^ {K} \exp \left(\left[ \mathbf {z} \right] _ {j}\right)} \text {f o r a l l} k \in [ K ] \quad \text {a n d} \left[ \boldsymbol {\sigma} (\mathbf {z}) \right] _ {0} = \frac {1}{1 + \sum_ {j = 1} ^ {K} \exp \left(\left[ \mathbf {z} \right] _ {j}\right)}, \tag {1}
$$
where  $[\cdot ]_k$  denotes the  $k$ -th entry of the input vector. Then, the probability of the outcome can also be written in a concise way as  $\operatorname *{Pr}[y_t = k\mid \mathbf{x}_t] = [\pmb {\sigma}(W_{*}\mathbf{x}_t)]_k$ . Besides, each outcome is associated with a fixed and known reward. We denote by  $\rho_{k}\in \mathbb{R}_{+}$  the reward for the outcome  $k\in [K]$ , and let  $\rho_0 = 0$  for the outcome  $y_{t} = 0$ . Therefore, the expected reward of the learner's action  $\mathbf{x}_t$  is defined as  $r(\mathbf{x}_t) = \sum_{k = 0}^{K}\operatorname *{Pr}[y_t = k\mid \mathbf{x}_t]\cdot \rho_k = \pmb{\rho}^\top \pmb {\sigma}(W_{*}\mathbf{x}_t)$ . Let  $\mathbf{x}_{*} = \arg \max_{\mathbf{x}\in \mathcal{X}}\pmb{\rho}^{\top}\pmb{\sigma}(W_{*}\mathbf{x})$ . The goal of the learner is to maximize the cumulative reward, which is equivalent to minimizing the regret
$$
\operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \boldsymbol {\rho} ^ {\top} \left(\boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {*}\right) - \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right)\right), \tag {2}
$$
When  $K = 1$  and  $\rho_{1} = 1$ , MLogB recovers the binary logistic bandit by  $r(\mathbf{x}) = \sigma (\mathbf{w}_{*}^{\top}\mathbf{x}) = 1 / (1 + \exp (-\mathbf{w}_{*}^{\top}\mathbf{x}))$ , where  $\mathbf{w}_*\in \mathbb{R}^d$  is a unknown parameter.
Exponentially Large Constant  $\kappa$ . In the logistic bandit problem, the non-linearity of the reward function is captured by the gradient of the link function  $\nabla \pmb{\sigma}:\mathbf{z}\in \mathbb{R}^{d}\mapsto \mathrm{diag}(\pmb{\sigma}(\mathbf{z})) - \pmb{\sigma}(\mathbf{z})\pmb{\sigma}(\mathbf{z})^{\top}$ . The analysis typically that requires that the gradient term is bounded from below and thus one would define the constant  $\kappa \triangleq 1 / \min_{W\in \mathcal{W},\mathbf{x}\in \mathcal{X}}\lambda_{\min}(\nabla \pmb{\sigma}(W\mathbf{x}))$  such that  $\frac{1}{\kappa} I_d\preccurlyeq \nabla \pmb{\sigma}(W\mathbf{x})$  for any  $W\in \mathcal{W}$  and  $\mathbf{x}\in \mathcal{X}$ , where  $\lambda_{\min}:\mathbb{R}^{K\times K}\to \mathbb{R}^{K}$  is the minimum eigenvalue of the input matrix. In the binary case ( $K = 1$ ), one can show that  $\kappa = \max_{\mathbf{w}\in \mathcal{W},\mathbf{x}\in \mathcal{X}}\{1 + \exp (\mathbf{w}^\top \mathbf{x}) + \exp (-\mathbf{w}^\top \mathbf{x})\} = \mathcal{O}(e^{SX})$ , where  $S$  and  $X$  are the diameters of the parameter space  $\mathcal{W}$  and action space  $\mathcal{X}$ . In the multinomial case, the paper [11, Section 3] also shows that  $\kappa$  is an exponentially large constant with respect to  $S$  and  $X$ . Thus, an algorithm with improved dependence on  $\kappa$  is demanded.
2.2 Assumptions and Notations
Same as the previous work for multinomial logistic bandit [11], we use the following assumptions.
Assumption 1. The norm of the action is bounded by 1, i.e.,  $\| \mathbf{x}\| _2\leq 1$  for any  $\mathbf{x}\in \mathcal{X}$
Assumption 2. The reward vector  $\pmb {\rho}\in \mathbb{R}_+^K$  and its norm is bounded by  $R$ , i.e.,  $\| \pmb {\rho}\| _2\leq R$
Assumption 3. The norm of the parameter  $W_{*} \in \mathbb{R}^{K \times d}$  is bounded by  $S$ , i.e.,  $\| W_{*} \|_{\mathrm{F}} \leq S$ , where  $\| \cdot \|_{\mathrm{F}}$  denotes the Frobenius norm of a matrix.
Assumption 4. Let  $\nabla \sigma (\mathbf{z}):\mathbf{z}\in \mathbb{R}^d\mapsto \mathrm{diag}(\pmb {\sigma}(\mathbf{z})) - \pmb {\sigma}(\mathbf{z})\pmb {\sigma}(\mathbf{z})^\top$  . For all  $\mathbf{x}\in \mathcal{X}$  and  $W\in \mathcal{W}$  we have  $\lambda_{\min}(\nabla \sigma (W\mathbf{x}))\geq 1 / \kappa$  and  $\lambda_{\max}(\nabla \sigma (W\mathbf{x}))\leq L$  , where  $\lambda_{\max}:\mathbb{R}^{K\times K}\to \mathbb{R}^{K}$  and  $\lambda_{\min}:\mathbb{R}^{K\times K}\to \mathbb{R}^{K}$  take the maximum and minimum eigenvalues of the input, respectively.
Other Notations. The following notations are used in the paper. Given a  $K$ -by- $d$  matrix  $W$ , we denote by  $\overrightarrow{W}$  its  $Kd$ -dimensional vectorization. For any positive semi-definite  $H \in \mathbb{R}^{Kd \times Kd}$ , we define the norm  $\| \overrightarrow{W} \|_H = \sqrt{\langle \overrightarrow{W}, H \overrightarrow{W} \rangle}$ . The notation  $\otimes$  is used for the standard Kronecker product. When the input is a vector  $\overrightarrow{W} \in \mathbb{R}^{Kd}$ , we treat it as a  $Kd \times 1$  matrix. Moreover, for any symmetric matrix  $A, B \in \mathbb{R}^{Kd \times Kd}$ , we denote by  $A \succcurlyeq B$  that  $A - B$  is a semi-positive definite matrix. We use  $\mathcal{F}_t = \sigma(\mathbf{x}_1, y_1, \ldots, y_{t-1}, \mathbf{x}_t)$  to denote the filtration, which encodes the information collected so far before receiving  $y_t$ . Finally,  $\mathcal{O}(\cdot)$  is used to highlight the dependence on  $d, K, \kappa,$  and  $T$ . With  $\widetilde{\mathcal{O}}(\cdot)$ -notation, we further hide the dependence on the dimension  $d$  and logarithmic factors.
2.3 Optimistic Algorithm with Improved Bound
This part revisits the principle of optimism in the face of uncertainty (OFU) [15] and introduces an improved version of the MNL-UCB algorithm [11] with better dependence on  $\kappa$  and  $K$ .
Optimism in the Face of Uncertainty. The OFU principle is a fundamental paradigm for addressing the exploration-exploitation dilemma in bandits. At each iteration  $t$ , the algorithm selects the arm by the rule  $\mathbf{x}_t = \arg \max_{\mathbf{x} \in \mathcal{X}} \widetilde{r}_t(\mathbf{x})$ , where  $\widetilde{r}_t(\mathbf{x})$  is an optimistic estimate of the true reward  $r(\mathbf{x})$  satisfying  $\widetilde{r}_t(\mathbf{x}) \geq r(\mathbf{x})$  for all  $\mathbf{x} \in \mathcal{X}$ . Based on the OFU rule, one can show that  $\mathrm{Reg}_T \leq \sum_{t=1}^{T} (\widetilde{r}(\mathbf{x}_t) - r(\mathbf{x}_t))$ , indicating that a tighter optimistic estimate  $\widetilde{r}_t$  will lead to a tighter regret bound. Therefore, designing a tight optimistic estimate  $\widetilde{r}_t$  is essential for the OFU algorithm.
In the context of the logistic bandit, a common practice is to construct a confidence set  $\mathcal{C}_t \subset \mathbb{R}^{K \times d}$ , which is supposed to contain the true parameter  $W_*$  with high probability. As such, the learner can obtain the optimistic reward for each arm  $\mathbf{x} \in \mathcal{X}$  by  $\widetilde{r}_t(\mathbf{x}) = \arg \max_{W \in \mathcal{C}_t} \pmb{\rho}^\top \pmb{\sigma}(W\mathbf{x})$ . A tighter confidence set will lead to a tighter optimistic reward, and thus resulting in a better regret bound.
Improved Concentration Set. Given the reward is generated by the multinomial logistic model, we can employ the maximum likelihood estimation (MLE) method to learn the unknown parameter  $W_{*}$ . Specifically, after observing the action-feedback pairs  $\{(\mathbf{x}_s,y_s)\}_{s = 1}^{t - 1}$ , one can train the model by
$$
W _ {t} ^ {\mathrm {M L E}} = \underset {W \in \mathbb {R} ^ {K \times d}} {\arg \min } \mathcal {L} _ {t} (W) \triangleq \sum_ {s = 1} ^ {t - 1} \ell_ {s} (W) + \frac {\lambda}{2} \| W \| _ {\mathrm {F}} ^ {2}, \tag {3}
$$
where  $\ell_t(W) = \sum_{k=0}^{K} \mathbb{1}\{y_t = k\} \cdot \log(1 / [\sigma(W\mathbf{x}_t)]_k)$  is the multiclass logistic loss established over  $(\mathbf{x}_t, y_t)$  and  $\lambda > 0$  is the regularization parameter. Let  $\overrightarrow{W} \in \mathbb{R}^{Kd}$  be the vectorized parameter. We also define the gradient  $\mathbf{g}_t(\overrightarrow{W})$  and the Fisher information matrix  $H_t(W)$  of the logistic loss by
$$
\mathbf {g} _ {t} (\vec {W}) \triangleq \nabla \mathcal {L} _ {t} (\vec {W}) \quad \text {a n d} \quad H _ {t} (W) \triangleq \lambda I + \sum_ {s = 1} ^ {t - 1} \nabla \boldsymbol {\sigma} (W \mathbf {x} _ {s}) \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top},
$$
where  $\overrightarrow{W} \in \mathbb{R}^{Kd}$  is the vectorized parameter and  $\nabla \sigma: \mathbf{z} \mapsto \mathrm{diag}(\sigma(\mathbf{z})) - \sigma(\mathbf{z})\sigma(\mathbf{z})^\top$  is the first order derivative of the reward vector  $\sigma(\mathbf{z})$ . Then, we are ready to present our confidence set for the maximum likelihood estimator, which exhibits  $\mathcal{O}(\sqrt{K})$  improvement over that in [11].
Theorem 1. Set the parameter  $\lambda = \mathcal{O}(dK\log (t / \delta))$  with a certain  $\delta \in (0,1]$ . For each iteration  $t\in [T]$ , we define the confidence set as
$$
\mathcal {C} _ {t} (\delta) \triangleq \left\{W \in \mathcal {W} \left| \left\| \mathbf {g} _ {t} (\overrightarrow {W}) - \mathbf {g} _ {t} \left(\overrightarrow {W} _ {t} ^ {\mathrm {M L E}}\right) \right\| _ {H _ {t} ^ {- 1} (W)} \leq \beta_ {t} (\delta) \right. \right\}, \tag {4}
$$
where  $\beta_{t}(\delta) = 4\sqrt{Kd(1 + S)\log\left(2\left(1 + t / d\right) / \delta\right)} = \mathcal{O}(\sqrt{dK\log t})$  is the radius of the set and  $\mathcal{W} = \{W\in \mathbb{R}^{K\times d}\mid \| W\|_{\mathrm{F}}\leq S\}$ . Then, we have  $\operatorname *{Pr}\big[\forall t\geq 1,W_{*}\in \mathcal{C}_{t}(\delta)\big]\geq 1 - \delta$ .
One advantage of the confidence set (4) is that its radius  $\beta_{t}(\delta)$  is independent of the exponentially large constant  $\kappa$  and thus is much tighter than the confident set constructed for the generalized
Algorithm 1 MNL-UCB+
Input: regularization coefficient  $\lambda$ , probability  $\delta$ .
1: Initialize  $H_{1} = \lambda I_{Kd}$  and  $\overline{W}_{1}^{\mathrm{MLE}}$  as any point in  $\mathcal{W}$
2: for  $t = 1, \dots, T$  do
3: Construct  $\widetilde{r}_t$  and select the arm by (5). Then, the learner receives  $y_{t}$
4: Train the estimator  $W_{t + 1}^{\mathrm{MLE}}$  by (3) and construct the confidence set  $\mathcal{C}_{t + 1}(\delta)$  as (4).
5: end for
linear bandit [5], whose radius exhibits a linear dependence on  $\kappa$ . The  $\kappa$ -independent set are first established by [8] for the binary logistic bandit problem and then adapted to multinomial setting by [11] with refined analysis on the self-normalized martingale tail-inequality for multinomial noise. Our confidence set is tighter than that in [11] by improving the radius from  $\beta_{t}(\delta) = \mathcal{O}(K\sqrt{d\log t})$  to  $\beta_{t}(\delta) = \mathcal{O}(\sqrt{dK\log t})$ . The improvement is based on a slightly refined self-normalized tail-inequality for the multinomial case, whose formal description is provided in Appendix C.1.
Construction of Optimistic Reward. Based on the confidence set (4), we can construct the optimistic reward and select the arm for each iteration by
$$
\widetilde {r} _ {t} (\mathbf {x}) = \arg \max  _ {W \in \mathcal {C} _ {t} (\delta)} \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} (W \mathbf {x}) \quad \text {a n d} \quad \mathbf {x} _ {t} = \arg \max  _ {\mathbf {x} \in \mathcal {W}} \widetilde {r} _ {t} (\mathbf {x}). \tag {5}
$$
We have following guarantee for the algorithm based on the MLE (3) and the selection rule specified in (4) and (5). We summarize the algorithmic procedure in Algorithm 1.
Theorem 2. Under the same conditions as Theorem 1. Let  $\delta \in (0,1]$ . Algorithm 1 ensures
$$
\operatorname {R e g} _ {T} \leq \mathcal {O} \left(\min  \left\{d \log T \sqrt {\kappa K T}, d K \log T \sqrt {T} + \kappa d ^ {2} K \log^ {2} T \right\}\right)
$$
with probability at least  $1 - \delta$  when we set the parameter  $\lambda = \mathcal{O}(dK\log (T / \delta))$
Remark 1 (Improved dependence on  $K$  and  $\kappa$ ). Our method achieves the  $\widetilde{\mathcal{O}}(\sqrt{\kappaKT})$  and  $\widetilde{\mathcal{O}}(K\sqrt{T})$  regret bounds for MLogB problem simultaneously. The first bound slightly improves the  $\widetilde{\mathcal{O}}(K\sqrt{\kappaT})$  guarantee of the MNL-UCB algorithm [11] by an  $\mathcal{O}(\sqrt{K})$  factor while the second one is independent of the exponentially large constant  $\kappa$  in its leading term. An  $\widetilde{\mathcal{O}}(K^{\frac{3}{2}}\sqrt{T})$  bound is also attained by [11]. However, their proposed method is intractable as its confidence set is established on all minimal elements of partially Loewner-ordered set  $C_t(\delta) \cap \mathcal{W}$  [11, Appendix D]. The computation cost of identifying all minimal elements and projecting onto the proposed confidence set is prohibitive. Our solution is free from such demands by using a different rule to construct the optimistic reward.
3 Jointly Efficient Algorithm
In this section, we introduce OFUL-MLogB, an algorithm with jointly computational and statistical efficiency for the MLogB problem. We will first discuss the efficiency concern of the existing methods in the literature and then introduce our algorithm, followed by a technical highlight.
3.1 Efficiency Concerns