Title: Online (Multinomial) Logistic Bandit: Improved Regret and Constant Computation Cost
Venue: NeurIPS 2023
Accepted: yes
Keywords: Logistic Bandit

Generalized Linear Bandit

Regret Bound

Computation Cost
Abstract: This paper investigates the logistic bandit problem, a variant of the generalized linear bandit model that utilizes a logistic model to depict the feedback from an action. While most existing research focuses on the binary logistic bandit problem, the multinomial case, which considers more than two possible feedback values, offers increased practical relevance and adaptability for use in complex decision-making problems such as reinforcement learning. In this paper, we provide an algorithm that enjoys both statistical and computational efficiency for the logistic bandit problem. In the binary case, our method improves the state-of-the-art binary logistic bandit method by reducing the per-round computation cost from $\mathcal{O}(\log T)$ to $\mathcal{O}(1)$ with respect to the time horizon $T$, while still preserving the minimax optimal guarantee up to logarithmic factors. In the multinomial case, with $K+1$ potential feedback values, our algorithm achieves an $\tilde{\mathcal{O}}(K\sqrt{T})$ regret bound with $\mathcal{O}(1)$ computational cost per round. The result not only improves the $\tilde{\mathcal{O}}(K\sqrt{\kappa T})$ bound for the best-known tractable algorithm—where the large constant $\kappa$ increases exponentially with the diameter of the parameter domain—but also reduces the $\mathcal{O}(T)$ computational complexity demanded by the previous method.

Online (Multinomial) Logistic Bandit: Improved Regret and Constant Computation Cost

Abstract

This paper investigates the logistic bandit problem, a variant of the generalized linear bandit model that utilizes a logistic model to depict the feedback from an action. While most existing research focuses on the binary logistic bandit problem, the multinomial case, which considers more than two possible feedback values, offers increased practical relevance and adaptability for use in complex decisionmaking problems such as reinforcement learning. In this paper, we provide an algorithm that enjoys both statistical and computational efficiency for the logistic bandit problem. In the binary case, our method improves the state-of-the-art binary logistic bandit method by reducing the per-round computation cost from O (log T ) to O (1) with respect to the time horizon T , while still preserving the minimax optimal guarantee up to logarithmic factors. In the multinomial case, with K +1 potential feedback values, our algorithm achieves an ˜ O ( K √ T ) regret bound with O (1) computational cost per round. The result not only improves the ˜ O ( K √ κT ) bound for the best-known tractable algorithm-where the large constant κ increases exponentially with the diameter of the parameter domain-but also reduces the O ( T ) computational complexity demanded by the previous method.

1 Introduction

The stochastic linear bandit (SLB) [1, 2, 3] problem is a natural generalization of the classic stochastic multi-armed bandit problem [4] by incorporating side information into the decision-making process. In the SLB problem, a linear model is used to characterize the relationship between the reward r t ∈ R and the learner's action x t ∈ X ⊆ R d , whereas such an assumption is not always satisfied in real-world applications. Consequently, various models have been developed to account for the non-linear reward, including the generalized linear bandit (GLB) model [5] and kernelized bandit model [6]. The logistic bandit is a specific kind of GLB model by connecting the learner's d -dimensional action and the reward with a logistic model. Most existing work focuses on the binary case [7, 8, 9, 10]. The reward r t ∈ { 0 , 1 } exhibits a binary value and the probability is modeled by Pr [ r t = 1 | x t ] = σ ( w ⊤ ∗ x t ) , where σ ( z ) = 1 / (1 + exp( -z )) is a non-linear link function and w ∗ ∈ W ⊆ R d is an unknown parameter. Compared to the SLB model, the logistic bandit model provides a more precise representation for a wide range of real-world application problems, where feedback exhibits discrete behavior. Moreover, from a theoretical perspective, it also serves as a basic setting for understanding the impact of non-linearity of the reward on the decision-making process. In this paper, we investigate a more general multinomial logistic bandit (MLogB) problem [11], in which the learner's action x t results in feedback y t that could have K +1 possible outcome values. The probability of each outcome is characterized with a logistic model (the formal definition is provided in Section 2.1). The MLogB model is of more practical interest compared to the binary case. For example, in the real-world application such as online advertising, there could be multiple possible feedback from customers, including 'buy now', 'add to cart', 'view related item', and

just leave without any click. Beyond addressing practical demands, studying the MLogB problem can shed light on other decision-making problems. For instance, in the theoretical reinforcement learning research [12, 13], the transition matrix is approximated using a linear model, which enables the application of the SLB method for balancing the exploration-exploitation trade-off. Since the transition matrix is inherently a probability matrix, the multinomial logistic model may be a more suitable structure for modeling transition probabilities between states.

As shown in Table 1, the logistic bandit problem has received much attention in the binary case. There are two main focuses arising from the non-linearity of the reward function: statistical efficiency and resource overhead . Regarding the statistical efficiency, a main focus is the algorithms' dependence on κ = max x ∈X , w ∈W 1 /σ ′ ( w ⊤ x ) , a crucial parameter capturing the non-linearity of the reward function. Since the binary logistic bandit is a special case of the generalized linear model, the result from [5] implies an ˜ O ( κ √ T ) bound. However, the parameter κ grows exponentially in terms of the diameter of the decision domain W and action space X , making the linear dependence unfavorable. A pivotal advancement is made by [9], where the paper presents an algorithm that achieves the nearly minimax optimal bound ˜ O ( √ T/κ ∗ ) . Here, κ ∗ = 1 /σ ′ ( w ⊤ ∗ x ∗ ) is the non-linear parameter associated with the best action x ∗ = arg max x ∈X σ ( w ⊤ ∗ x ) , suggesting that non-linearity might be advantageous for statistical efficiency, rather than a hindrance. Alongside statistical efficiency considerations, the non-linearity of the feedback raises concerns about the algorithms' computation and storage efficiency. The methods [8, 9] with improved dependence on κ usually require storing and optimizing over all historical data to estimate the unknown parameter, leading to an O ( t ) computation and storage cost at round t ∈ [ T ] . The pioneering work [7] provided the first efficient solution for binary logistic bandit with constant computation and storage costs and [14] further proposed an efficient algorithm for generalized linear bandit, but their regret bounds still exhibit a linear dependence on κ . Recently, a jointly efficient algorithm was proposed by [10], which achieves the optimal dependence on κ with a O (log t ) computation cost and constant storage cost per round. However, it remains open whether the minimax optimal bound is achievable with constant computation cost independent of the time t .

Regrading the multinomial logistic bandit, the best-known feasible algorithm was proposed by [11]. This method achieves an ˜ O ( K √ κT ) regret bound, bearing an O ( √ κ ) dependence on the exponentially large constant. Moreover, it still demands O ( t ) computation and storage costs to optimize over all past data. The same study also introduced an ˜ O ( K 3 / 2 √ T ) bound with improved dependence on κ . Yet, this solution leans heavily towards theoretical insights and is intractable in implementation [11, Section 2.6]. Designing a practical or more efficient algorithm with improved dependence on κ is still an unsolved challenge. More discussions on the related work and topics can be found in Section 4.

Our Results. In this paper, we provide an algorithm with both statistical and computational efficiency.

For the multinomial logistic bandit, we propose OFUL-MLOGB, a jointly efficient method attaining an ˜ O ( K √ T ) regret bound with O (1) in T computation and storage cost per round. The result improves the previous work on the dependence of the large constant κ .

For the binary case, our proposed OFUL-MLOGB can achieve the ˜ O ( √ T/κ ∗ ) optimal bound up to logarithmic factors. Besides, our method reduces the computation cost of the state-of-the-art binary method [10] from O (log t ) to O (1) per round.

2 Multinomial Logistic Bandit with Improved Regret

This section provides preliminaries on the multinomial logistic bandit problem and optimistic algorithms, beginning with the problem formulation and notations. Then, we revisit the optimistic algorithms for the MLogB problem. Specifically, we investigate the previously best-known feasible algorithm, MNL-UCB algorithm [11], and propose an improved version with better dependence on the exponentially large constant κ and the number of outcome values K .

2.1 Problem Formulation

The multinomial logistic bandit (MLogB) problem studies a T round decision-making process between the learner and the environment. At the beginning of the iteration t , the learner will first select an action x t ∈ X from the feasible action set X ⊆ R d and then submit it to the environment. After that, a response y t ∈ { 0 } ∪ [ K ] with K +1 possible outcomes (like 'buy now', 'add to chart', or "do nothing") is returned based on the learner's choice, where K ∈ N . Specifically, the MLogB problem assumes that each outcome k ∈ [ K ] is associated with a ground-truth parameter w ( k ) ∗ ∈ R d and the probability of the outcome Pr [ y t = k | x t ] follows the logistic model,

For notational simplicity, we denote by W ∗ = [ w (1) ∗ , . . . , w ( K ) ∗ ] ⊤ ∈ R K × d the matrix for the unknown parameter and define the softmax function σ : R K ↦→ [0 , 1] K by

where [ · ] k denotes the k -th entry of the input vector. Then, the probability of the outcome can also be written in a concise way as Pr [ y t = k | x t ] = [ σ ( W ∗ x t )] k . Besides, each outcome is associated with a fixed and known reward. We denote by ρ k ∈ R + the reward for the outcome k ∈ [ K ] , and let ρ 0 = 0 for the outcome y t = 0 . Therefore, the expected reward of the learner's action x t is defined as r ( x t ) = ∑ K k =0 Pr [ y t = k | x t ] · ρ k = ρ ⊤ σ ( W ∗ x t ) . Let x ∗ = arg max x ∈X ρ ⊤ σ ( W ∗ x ) . The goal of the learner is to maximize the cumulative reward, which is equivalent to minimizing the regret

When K = 1 and ρ 1 = 1 , MLogB recovers the binary logistic bandit by r ( x ) = σ ( w ⊤ ∗ x ) = 1 / (1 + exp( -w ⊤ ∗ x )) , where w ∗ ∈ R d is a unknown parameter.

Exponentially Large Constant κ . In the logistic bandit problem, the non-linearity of the reward function is captured by the gradient of the link function ∇ σ : z ∈ R d ↦→ diag( σ ( z )) -σ ( z ) σ ( z ) ⊤ . The analysis typically that requires that the gradient term is bounded from below and thus one would define the constant κ ≜ 1 / min W ∈W , x ∈X λ min ( ∇ σ ( W x )) such that 1 κ I d ≼ ∇ σ ( W x ) for any W ∈ W and x ∈ X , where λ min : R K × K → R K is the minimum eigenvalue of the input matrix. In the binary case ( K = 1 ), one can show that κ = max w ∈W , x ∈X { 1 + exp( w ⊤ x ) + exp( -w ⊤ x ) } = O ( e SX ) , where S and X are the diameters of the parameter space W and action space X . In the multinomial case, the paper [11, Section 3] also shows that κ is an exponentially large constant with respect to S and X . Thus, an algorithm with improved dependence on κ is demanded.

2.2 Assumptions and Notations

Same as the previous work for multinomial logistic bandit [11], we use the following assumptions.

Assumption 1. The norm of the action is bounded by 1, i.e., ∥ x ∥ 2 ≤ 1 for any x ∈ X .

Assumption 2. The reward vector ρ ∈ R K + and its norm is bounded by R , i.e., ∥ ρ ∥ 2 ≤ R .

Assumption 3. The norm of the parameter W ∗ ∈ R K × d is bounded by S , i.e., ∥ W ∗ ∥ F ≤ S , where ∥·∥ F denotes the Frobenius norm of a matrix.

Assumption 4. Let ∇ σ ( z ) : z ∈ R d ↦→ diag( σ ( z )) -σ ( z ) σ ( z ) ⊤ . For all x ∈ X and W ∈ W , we have λ min ( ∇ σ ( W x )) ≥ 1 /κ and λ max ( ∇ σ ( W x )) ≤ L , where λ max : R K × K → R K and λ min : R K × K → R K take the maximum and minimum eigenvalues of the input, respectively.

Other Notations. The following notations are used in the paper. Given a K -byd matrix W , we denote by - → W its Kd -dimensional vectorization. For any positive semi-definite H ∈ R Kd × Kd , we define the norm ∥ - → W ∥ H = √ 〈 - → W,H - → W 〉 . The notation ⊗ is used for the standard Kronecker product.

When the input is a vector - → W ∈ R Kd , we treat it as a Kd × 1 matrix. Moreover, for any symmetric matrix A,B ∈ R Kd × Kd , we denote by A ≽ B that A -B is a semi-positive definite matrix. We use F t = σ ( x 1 , y 1 , . . . , y t -1 , x t ) to denote the filtration, which encodes the information collected so far before receiving y t . Finally, O ( · ) is used to highlight the dependence on d , K , κ , and T . With ˜ O ( · ) -notation, we further hide the dependence on the dimension d and logarithmic factors.

2.3 Optimistic Algorithm with Improved Bound

This part revisit the principle of optimism in the face of uncertainty (OFU) [15] and introduces an improved version of the MNL-UCB algorithm [11] with better dependence on κ and K .

Optimism in the Face of Uncertainty. The OFU principle is a fundamental paradigm for addressing the exploration-exploitation dilemma in bandits. At each iteration t , the algorithm selects the arm by the rule x t = arg max x ∈X ˜ r t ( x ) , where ˜ r t ( x ) is an optimistic estimate of the true reward r ( x ) satisfying ˜ r t ( x ) ≥ r ( x ) for all x ∈ X . Based on the OFU rule, one can show that Reg T ≤ ∑ T t =1 ( ˜ r ( x t ) -r ( x t )) , indicating that a tighter optimistic estimate ˜ r t will lead to a tighter regret bound. Therefore, designing a tight optimistic estimate ˜ r t is essential for the OFU algorithm.

In the context of the logistic bandit, a common practice is to construct a confidence set C t ⊂ R K × d , which is supposed to contain the true parameter W ∗ with high probability. As such, the learner can obtain the optimistic reward for each arm x ∈ X by ˜ r t ( x ) = arg max W ∈C t ρ ⊤ σ ( W x ) . A tighter confidence set will lead to a tighter optimistic reward, and thus resulting in a better regret bound.

Improved Concentration Set. Given the reward is generated by the multinomial logistic model, we can employ the maximum likelihood estimation (MLE) method to learn the unknown parameter W ∗ . Specifically, after observing the action-feedback pairs { ( x s , y s ) } t -1 s =1 , one can train the model by

where ℓ t ( W ) = ∑ K k =0 1 { y t = k } · log (1 / [ σ ( W x t )] k ) is the multiclass logistic loss established over ( x t , y t ) and λ > 0 is the regularization parameter. Let - → W ∈ R Kd be the vectorized parameter. We also define the gradient g t ( - → W ) and the Fisher information matrix H t ( W ) of the logistic loss by

where - → W ∈ R Kd is the vectorized parameter and ∇ σ : z ↦→ diag( σ ( z )) -σ ( z ) σ ( z ) ⊤ is the first order derivative of the reward vector σ ( z ) . Then, we are ready to present our confidence set for the maximum likelihood estimator, which exhibits O ( √ K ) improvement over that in [11].

Theorem 1. Set the parameter λ = O ( dK log( t/δ )) with a certain δ ∈ (0 , 1] . For each iteration t ∈ [ T ] , we define the confidence set as

where β t ( δ ) = 4 √ Kd (1 + S ) log ( 2 (1 + t/d ) /δ ) = O ( √ dK log t ) is the radius of the set and W = { W ∈ R K × d | ∥ W ∥ F ≤ S } . Then, we have Pr [ ∀ t ≥ 1 , W ∗ ∈ C t ( δ ) ] ≥ 1 -δ .

One advantage of the confidence set (4) is that its radius β t ( δ ) is independent of the exponentially large constant κ and thus is much tighter than the confident set constructed for the generalized

linear bandit [5], whose radius exhibits a linear dependence on κ . The κ -independent set are first established by [8] for the binary logistic bandit problem and then adapted to multinomial setting by [11] with refined analysis on the self-normalized martingale tail-inequality for multinomial noise. Our confidence set is tighter than that in [11] by improving the radius from β t ( δ ) = O ( K √ d log t ) to β t ( δ ) = O ( √ dK log t ) . The improvement is based on a slightly refined self-normalized tailinequality for the multinomial case, whose formal description is provided in Appendix C.1.

Construction of Optimistic Reward. Based on the confidence set (4), we can construct the optimistic reward and select the arm for each iteration by

We have following guarantee for the algorithm based on the MLE (3) and the selection rule specified in (4) and (5). We summarize the algorithmic procedure in Algorithm 1.

Theorem 2. Under the same conditions as Theorem 1. Let δ ∈ (0 , 1] . Algorithm 1 ensures

with probability at least 1 -δ when we set the parameter λ = O ( dK log( T/δ )) .

Remark 1 (Improved dependence on K and κ ) . Our method achieves the ˜ O ( √ κKT ) and ˜ O ( K √ T ) regret bounds for MLogB problem simultaneously. The first bound slightly improves the ˜ O ( K √ κT ) guarantee of the MNL-UCB algorithm [11] by an O ( √ K ) factor while the second one is independent of the exponentially large constant κ in its leading term. An ˜ O ( K 3 2 √ T ) bound is also attained by [11]. However, their proposed method is intractable as its confidence set is established on all minimal elements of partially Loewner-ordered set C t ( δ ) ∩ W [11, Appendix D]. The computation cost of identifying all minimal elements and projecting onto the proposed confidence set is prohibitive. Our solution is free from such demands by using a different rule to construct the optimistic reward.

3 Jointly Efficient Algorithm

In this section, we introduce OFUL-MLogB, an algorithm with jointly computational and statistical efficiency for the MLogB problem. We will first discuss the efficiency concern of the existing methods in the literature and then introduce our algorithm, followed by a technical highlight.

3.1 Efficiency Concerns

The algorithms for logistic bandit crucially rely on two components to ensure the statistical efficiency: the MLE (3) and the optimistic rule (5) for constructing ˜ r t . However, the implementation of both components could be inefficient by requiring O ( t ) computation cost per online iteration.

Computation and Storage Cost of Maximum Likelihood Estimation. For logistic bandit or even the generalized linear bandit problem [3, 8, 9, 11], MLE is a widely used tool to learn the unknown parameter. To solve the optimization problem, the gradient-based method, e.g. the projected gradient descent [16], are usually applied. However, as discussed in [10], the optimization of the MLE problem typically requires O ( t log(1 /ϵ )) gradient step to achieve ϵ -accuracy. Besides, the loss function L t is established on all historical data { ( x s , y s ) } t -1 s =1 , resulting in an O ( t ) gradient query complexity for each gradient step and O ( t ) storage cost, and thus is inefficient.

Computation and Storage Cost of Optimistic Reward Construction. The construction of the optimistic reward ˜ r t ( x ) requires to solve the optimization problem (5). However, the objective

function ρ ⊤ σ ( W x ) is non-concave and the decision domain C t ( δ ) is non-convex, making the maximization problem ˜ r t ( x ) = arg max W ∈C t ( δ ) ρ ⊤ σ ( W x ) computationally challenging. In the binary case, the paper [9] proposed a convex relaxation of the confidence set C t ( δ ) , whereas the relaxed confidence set is still established on all historical data, resulting in O ( t ) time complexity and O ( t ) storage cost at iteration t . The optimistic estimate construction in the previous work for multinomial logistci bandits [11] is computationally efficient without involving any optimization problem solving. However, it will lead to an inferior regret bound of ˜ O ( √ κT ) .

In the binary case, the work [10] proposed a jointly efficient algorithm, which achieved a nearly minimax optimal bound with O (log t ) computation cost per iteration and O (1) storage cost. However, as we will discuss in Section 3.3, it is hard to apply their analysis to the multinomial case.

3.2 Efficient Algorithm

In this section, we proposed a novel algorithm which only requires O (1) computation cost per iteration and O (1) storage cost. The algorithm can achieve the best known results both for binary and multinomial logistic bandits. We have introduced new ingredients both on the algorithm design and regret analysis to achieve the jointly efficient algorithm.

Efficient Online Estimation. Instead of performing MLE, we run an online mirror descent algorithm to estimate parameter:

where η > 0 is the step size to be specified later and the first iteration model - → W OL 1 can be initialized as any point in the domain W = { - → W ∈ R Kd | ∥ - → W ∥ 2 ≤ S } . We set the matrix as ˜ H t = H t + η ∇ σ ( W OL t x t ) ⊗ x t x ⊤ t , where H t = λI + ∑ t -1 s =1 ∇ σ ( W OL s +1 x s ) ⊗ x s x ⊤ s . Both ˜ H t and H t can be updated incrementally.

We show the online estimator (6) enjoys computational, storage, and statistical efficiency. Since (6) exhibits a standard online mirror descent formulation [17], it can be solved with a single projected gradient step with the following equivalent formulation by

For the gradient descent step above, the most time-consuming operation is maintaining the inverse of the matrix ˜ H t . Since ∇ σ ( W OL t x t ) ⊗ x t x ⊤ t is a rankK matrix, it can be calculated by the ShermanMorrison-Woodbury formula with O ( d 2 K 3 ) cost per round. As for the projection step, since ˜ H t is positive semi-definite matrix, it can be solved in O ( K 3 d 3 ) [18, Section 4]. * As a consequence, our algorithm achieves a light update with O (1) cost per round. Regarding storage cost, our proposed estimator eliminates the need to store all historical data and updates in a one-pass fashion, requiring only O (1) storage cost throughout the learning process. Moreover, the estimator is also statistically efficient. We can construct the following κ -independent confidence set similar to that in Theorem 1.

Theorem 3. Let δ ∈ (0 , 1] and α = 2(1 + S ) + ln( K + 1) . Set the parameter η = α/ 2 and λ = max { 28 Kdα, 7 √ 6 αS } . For each iteration t ∈ [ T ] , we define the confidence set as

where β OL t ( δ ) = O ( log K log t Kd ) . Then, we have Pr [ ∀ t ≥ 1 , W ∗ ∈ C t ( δ ) ] ≥ 1 -δ . Moreover, the computation cost of solving (6) is O (1) per round.

Remark 2 (Comparison with the online estimator [10]) . Our algorithm design is inspired by the online estimator [10] developed for the binary case, while achieving even lighter cost by novel algorithm ingredients and analysis. As discussed in Section 3.3, the analysis for the binary setting is hard to be applied to the multinomial case. Specifically, the paper [10] has proposed an intermediary decision W t in the analysis to prove the statistical efficiency of the proposed estimator. However, the favorable property of the intermediate decision only holds in the binary case. To this end, we have proposed a new intermediary decisions, which not only help to prove the statistically efficient of our estimator but also eliminate the requirement of the exploration step. Besides, we have also introduced a novel algorithm ingredient to further speed-up the algorithm. Instead of learning with original loss function as in [10], by a more refined exploitation of the negative term in the analysis, we show it is sufficient to learn with the first order approximate ⟨∇ ℓ t ( - → W OL t ) , - → W ⟩ of ℓ t ( W ) with the adjusted local norm ∥·∥ ˜ H t . Our new algorithm not only enjoys a computation efficiency improvement from O (log t ) in [10] to O (1) , but also is free from any exploration step required by the previous work. We provide a technical highlight in Section 3.3.

Efficient Optimistic Reward Construction. Although the confidence set C OL t ( δ ) is convex, the optimistic rule (5) by ˜ r t ( x ) = arg min W ∈C OL t ( δ ) ρ ⊤ σ ( W x ) still involves inefficient non-concave optimization problem solving. In this part, we propose a novel optimistic reward that can be solved in a constant time per round.

Proposition 1. For any x ∈ X and iteration t ∈ [ T ] , the optimistic reward is constructed by

In above, ϵ fst t ( x ) = β OL t ( δ ) · ∥ H -1 2 t ( I K ⊗ x ) ∇ σ ( W OL t x ) ρ ∥ 2 and ϵ snd t ( x ) = 3 R ( β OL t ) 2 · ∥ ( I K ⊗ x ⊤ ) H -1 / 2 t ∥ 2 2 are the bonus. Then, we have ˜ r OL t ( x ) ≥ ρ ⊤ σ ( W ∗ x ) for all t ≥ 1 and x ∈ X with probability at least 1 -δ .

Proposition 1 constructs the optimistic reward by adding the 'bonus' to the reward empirically estimated by W OL t . Different from the term used in [11], our bonus terms are independent of the exponentially large constant and thus can lead to an improved ˜ O ( K √ T ) bound. The optimistic rule (8) does not involve any optimization problem solving and can be calculated in an O (1) cost.

Overall Algorithm and Guarantees. We overall procedures in Algorithm 2. For the general multinomial setting, it ensures an ˜ O ( K √ T ) regret guarantee and an O (1) computation cost.

Theorem 4. Under the same condition as Theorem 3, Algorithm 2 ensures

The computation cost of Algorithm 2 is bounded by O (1) for each round t ∈ [ T ] .

Remark 3 (On the ˜ O ( √ T/κ ∗ ) bound) . In the binary setting, [9, 10] show that an ˜ O ( √ T/κ ∗ ) minimax optimal is achievable with κ ∗ = 1 /σ ′ ( w ⊤ ∗ x ∗ ) . However, due to the multinomial behavior of the feedback, it is unclear how to achieve such a rate in MLogB case (see the discussion in Appendix C.5). Besides, it also raises concerns about efficiency when applying the method developed for binary case to multinomial setting. In particular, the ˜ O ( √ T/κ ∗ ) is achieved by the rule arg max x ∈X , w ∈C t ( δ ) σ ( w ⊤ x ) in [9, 10]. The optimization can be efficiently solved in the binary case since one can simply eliminate the non-linearity of the reward function by the relationship σ ( z 1 ) > σ ( z 2 ) for any z 1 ∈ R > z 2 ∈ R . Such a condition does not hold in MLogB problem.

When reduced to the binary case K = 1 , our algorithm can also achieves the minimax regret bound. Corollary 1. When K = 1 , the multinomial logistic bandit reduces to the binary logistic bandit problem. Then, under the same conditions as Theorem 4, Algorithm 2 with the optimistic rule ˜ r t ( x ) = arg max w ∈C OL t ( δ ) w ⊤ x ensures Reg T ≤ ˜ O ( √ T/κ ∗ ) with probability at least 1 -δ .

3.3 Analysis

This section presents the proof sketch for Theorem 3, which plays an key role in the analysis for Algorithm 2. For notational simplicity, we will drop the superscript OL in this section.

Leveraging Negative Terms for Efficient Update. If we update the model with the original loss function by - → W t +1 = arg min W ∈W ℓ t ( W ) + 1 2 η ∥ - → W - - → W t ∥ 2 H t , the arguments in [10] shows that the estimation error between - → W t +1 and - → W ∗ can be bounded by their gap on the loss function

However, the update rule with original loss will lead to an O (log t ) computation cost per iteration. To facilitate a more efficient algorithm, we introduce the update rule with the linearized loss ⟨∇ ℓ t ( - → W t ) , - → W ⟩ and the adjusted norm ˜ H t as (6). As we have shown in the proof of Lemma 12 in Appendix C.1. Denote by ˜ ℓ t ( W ) = ⟨∇ ℓ t ( - → W t ) , - → W ⟩ + 1 2 ∥ - → W - - → W t ∥ 2 ∇ 2 ℓ t ( - → W t ) the second-order surrogate of ℓ t ( W ) . The efficient update rule will introduce an additional term

We handle the additional term by the self-concordant property of the logistic loss and exploiting a negative term ignored in the previous analysis. Since the logistic loss is a √ 6 -self-concordant-like function [19, Lemma 4], then Theorem 3 of [19] indicates that the additional term can be bounded by

where ξ s ∈ R Kd is on the line connecting - → W s +1 and - → W s . Besides, by a refined analysis of the online mirror descent (OMD) update (6), we identify an additional negative term -∑ t s =1 ∥ - → W s +1 - - → W s ∥ 2 H s on the right hand side of (9). By properly choosing the coefficient λ , one can cancel the additional term by the negative term and achieves (9) with the efficient update. We note that the negative term in the OMD analysis is also found crucial in the gradient-variation regret of non-stationary online learning [20, 21] as well as its applications to game theory [22] and the SEA model [23].

Novel construction of the intermediary prediction. Then, we can further bound the right hand side of (9) by inserting an intermediary loss ℓ s ( ˜ W s ) as

In the binary case, inspired by the study [24] for the binary online logistic regression, [10] propose to construct ˜ w s = arg min w ∈W s ℓ b ( w ⊤ x s , +1) + ℓ b ( w ⊤ x s , 0) + 1 2 η ∥ w -w s ∥ 2 H s , where ℓ b is the binary logistic loss and W s is a shrank decision domain. Then, both term (A) and term (B) can be bounded without the dependence on κ . However, the analysis for term (B) crucially relies on the condition | σ ( ˜ w ⊤ s x s ) -y s | · | σ ( ˜ w ⊤ s x s ) -1 + y s | = σ ′ ( ˜ w ⊤ s x s ) to eliminate the dependence on κ , where y s ∈ { 0 , 1 } is the one-dimensional feedback. It is hard to show such an relationship in the multinomial case since the feedback y s has multiple value. A similar challenge is also observed in the recent study on online multiclass logistic regression [25, Appendix F]. One might consider whether it is possible to construct ˜ W t with the update rule developed in [25]. However, since the online update rule of [25] requires to perform over R K × d , the learned parameter would become unbounded, which makes it is hard to provide an upper bound for term (A).

To this end, we design a new intermediary term by ℓ m ( ˜ z s , y s ) , where ℓ m is the multiclass logistic loss. The prediction is constructed by ˜ z s = σ + ( E W ∼ P s [ σ ( W x s )]) with the Gaussian distribution P s = N ( - → W s , αH -1 s ) , where σ + is a pseudo inverse function of the sigmoid function σ . Such

an integral construction ˜ z s = σ + ( E W ∼ P s [ σ ( W x s )]) is previously used in the online logistic regression literature [26, 27] but we tailored the tool to our analysis with a different construction of the distribution P s . Lemma 13 and Lemma 14 in Appendix C.1 shows that

Then, combining the upper bound of term (A) and term (B) and eliminating the additional term ∑ t s =1 ∥ - → W s - - → W s +1 ∥ 2 H s with the negative term obtained by a refined analysis of online mirror descent, we complete the proof of Theorem 3.

4 More Discussions and Related Work

This section begins with a discussion on the tightness of the proposed bounds.

On the Tightness of Our Bounds. In this paper, we introduced OFUL-MLogB, a jointly efficient algorithm that simultaneously achieves regret bounds of ˜ O ( √ κKT ) and ˜ O ( K √ T ) . † The tightness of these bounds, with respect to κ and T , was detailed in Remark 3. Regarding the number of feedback values K , [11] claimed the optimality of a linear dependence on K . However, our findings revealed a nuanced interplay between κ and K . The ˜ O ( √ κKT ) bound does not conflict with the lower bound argument by [11], given that the non-linear constant κ is also associated with K . Beyond κ , the norm of the unknown parameter S and maximum norm R of the reward vector ρ can also depend on K based on the problem's specifics. It is an interesting direction to understand the interrelation of these constants by establishing a lower bound. Additionally, our method has a linear dependence on d . In the finite-arm case, an O ( √ d ) dependence might be attainable with a SupLin-type algorithm [28].

Below, we introduce more related works on logistic bandit and the related topics.

Logistic Bandit. While the logistic bandit is a specific instance of the generalized linear bandit model [5, 29, 30, 31, 32, 33, 34], the algorithms proposed for GLB tend to exhibit a linear dependence on the nonlinear term κ , which is exponentially large in the logistic bandit case. Therefore, addressing the non-linearity of the reward function warrants specialized consideration. Besides the UCBtype algorithms [8, 9] mentioned in Section 1, for the N -arm case, [35] proposed an experimental design-based algorithm providing an ˜ O ( √ d log NT/κ ∗ ) regret bound with better dependence on d . However, the previous methods were built upon the MLE estimator, whose optimization demands O ( t ) computation and storage complexity for the t -th iteration. To the best of our knowledge, the only known jointly computational and statistical efficient algorithm was proposed by [10], which achieves a nearly minimax optimal regret bound with computation cost of O (log t ) per round. In addition to the frequentist bounds, there are also researches on logistic bandit from the Bayesian view. [36] showed the Bayesian regret of the Thompson sampling method is independent of κ (even in the lower order term) when the feasible domain is identical to the parameter domain, i.e., X = W . [37] further proved the κ -independent bound with weaker conditions.

Multinomial Logit (MNL) Bandit. Another relevant line of research is the multinomial logit contextual bandit problem [38, 39, 40, 41, 42], which generalizes the binary logistic bandit by allowing the learner to submit a subset of arms S t = { x t,i } K i =1 ∈ X to the users. The expected reward function is also modeled by the multinomial logit model: E [ r t | S t ] = ∑ x t,i ∈ S t ρ t,i exp( w ⊤ ∗ x t,i ) / (1+ ∑ x t,i ∈ S t exp( w ⊤ ∗ x t,i )) , where ρ t,i is the reward for arm x t,i and w ∗ ∈ R d is an unknown parameter. There are also studies on the MNL bandit problem concerning the exponentially large constant κ . [42] proposed an optimistic algorithm with O ( d √ T ) regret bounds without κ in its leading term, which improves the O ( dκ √ T ) bound with better dependence on κ . Considering uniform reward, i.e., ρ t,i = 1 for all t ∈ [ T ] and i ∈ [ N ] , [43] further showed an ˜ O ( d √ T/κ ∗ ) bound. To the best of our knowledge, all the existing methods with improved κ are established on the MLE estimator. It would be an interesting future direction to develop jointly efficient algorithm for the MNL bandit problem.

√

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

100

200

300

400

500

600

Regret

LogUCB1

OL2M

OFU-MLogB

(a) configuration 1

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

20

40

60

80

100

120

Regret

LogUCB1

OL2M

OFU-MLogB

(b) configuration 2

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

25

50

75

100

125

150

Regret

LogUCB1

OL2M

OFU-MLogB

(c) configuration 3

LogUCB1

OL2M

ada-OFU-ECOLog OFU-MLogB

Algorithms

10 1

10 2

10 3

Time (seconds)

789.15

3.00

9.81

3.87

LogUCB1

OL2M

OFU-MLogB

(d) total running time

0

500

1000

1500

2000

2500

3000

Iteration

0

20

40

60

80

100

120

140

160

Regret

MNL-UCB

OFU-MLogB

(a) configuration 1

0

500

1000

1500

2000

2500

3000

Iteration

0

25

50

75

100

125

150

175

200

Regret

MNL-UCB

OFU-MLogB

(b) configuration 2

0

500

1000

1500

2000

2500

3000

Iteration

0

20

40

60

80

100

120

140

Regret

MNL-UCB

OFU-MLogB

(c) configuration 3

MNL-UCB

OFU-MLogB

Algorithms

10 2

10 3

Time (seconds)

2459.71

49.28

MNL-UCB

OFU-MLogB

(d) total running time

5 Experiments

This section validates the statistical and computation efficiency of the proposed method by experiments. We conduct a bandit learning for T = 3000 iterations. In each experiment, we run experiments on 6 random configurations, in which the arm set and the underlying parameter are randomly sampled. Specially, |X| = 20 actions are randomly sampled from a 2-dimensional sphere of radius 1. In the binary case, the norm of the unknown parameter w ∗ is set as S = 5 . In the multinomial case, we set K = 4 with S = 1 . The reward vector is set as ρ = [0 . 25 , 0 . 5 , 0 . 75 , 1] . For each configuration, we report the averaged results over 10 trials. More details can be found in Appendix D.

Experimental Results. Figure 4 provides a comparison of performance and computation costs in the binary case. The algorithm O2LM [7], which has a constant computational cost per iteration, is used as a comparison baseline. Our algorithm demonstrates a time complexity akin to O2LM , affirming its computational efficiency. Compared to the state-of-the-art binary logistic bandit algorithm ada-OFU-ECOLog [10], our OFU-MLogB method has lighter computational overheads while preserving similar empirical performance. Figure 4 illustrates the comparison in the multinomial setting. Our algorithm is around 50 times faster than MNL-UCB [11] for running T = 3000 iterations and achieves better empirical performance. More experimental results on other configurations and the running time curve that increases along the iterations can be found in Appendix D.

6 Conclusion

This paper proposed a jointly efficient algorithm OFUL-MLogB for both binary and multinomial logistic bandit problems with constant computation cost per round and improved regret guarantees. For the multinomial setting, our method improves over the best-known feasible algorithm both on the dependence of κ and the computation cost. When reduced to the binary case, OFUL-MLogB also contributes to improve the computation cost of previous method from O (log t ) to O (1) per round while still preserving the ˜ O ( √ T/κ ∗ ) minimax optimal bound up to logarithmic factors. A promising future direction is to consider the multinomial logit model in reinforcement learning. Besides, it is still open on how to achieve the ˜ O ( √ T/κ ∗ ) bound for the multinomial case.

Acknowledgments

YJZ and MS were supported by the Institute for AI and Beyond, UTokyo. YJZ was also supported by Todai Fellowship. The authors would thank Peng Zhao for helpful discussions and Jing Wang for the assistance in experiments.

References

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3:397-422, 2002.

Varsha Dani, Thomas P. Hayes, and Sham M. Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Annual Conference on Learning Theory (COLT) , pages 355-366, 2008.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 (NeurIPS) , pages 2312-2320, 2011.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2-3):235-256, 2002.

Sarah Filippi, Olivier Cappé, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems 23 (NeurIPS) , pages 586-594, 2010.

Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 844-853, 2017.

Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In Proceedings of the 33rd International Conference on Machine Learning (ICML) , pages 392-401, 2016.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 3052-3060. PMLR, 2020.

Marc Abeille, Louis Faury, and Clément Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3691-3699, 2021.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 546-580, 2022.

Sanae Amani and Christos Thrampoulidis. UCB-based algorithms for multinomial logistic regression bandits. In Advances in Neural Information Processing Systems 34 (NeurIPS) , 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the 33rd Conference on Learning Theory (COLT) , pages 2137-2143, 2020.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvári. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4532-4576, 2021.

Kwang-Sung Jun, Aniruddha Bhargava, Robert D. Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. In Advances in Neural Information Processing Systems 30 (NeurIPS) , pages 99-109, 2017.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Elad Hazan. Introduction to online convex optimization. Foundations and Trends in Optimization , 2(3-4):157-325, 2016.

Francesco Orabona. A modern introduction to online learning. ArXiv preprint , arXiv: 1912.13213, 2019.

Zakaria Mhammedi, Wouter M. Koolen, and Tim van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Proceedings of the 32nd Annual Conference on Learning Theory (COLT) , pages 2490-2511, 2019.

Quoc Tran-Dinh, Yen-Huan Li, and Volkan Cevher. Composite convex minimization involving self-concordant-like cost functions. In Proceedings of the 3rd International Conference on Modelling, Computation and Optimization in Information Systems and Management Sciences , pages 155-168, 2015.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and nonstationarity: Problem-dependent dynamic regret for online convex optimization. ArXiv preprint , arXiv:2112.14368, 2021.

Mengxiao Zhang, Peng Zhao, Haipeng Luo, and Zhi-Hua Zhou. No-regret learning in timevarying zero-sum games. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 26772-26808, 2022.

Sijia Chen, Wei-Wei Tu, Peng Zhao, and Lijun Zhang. Optimistic online mirror descent for bridging stochastic and adversarial online convex optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5002-5035, 2023.

Rémi Jézéquel, Pierre Gaillard, and Alessandro Rudi. Efficient improper learning for online logistic regression. In Proceedings of the 33rd Conference on Learning Theory (COLT) , pages 2085-2108, 2020.

Naman Agarwal, Satyen Kale, and Julian Zimmert. Efficient methods for online multiclass logistic regression. In Proceedings of International Conference on Algorithmic Learning Theory (ALT) , pages 3-33, 2022.

Dylan J. Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Proceedings of the 31st Conference On Learning Theory (COLT) , pages 167-208, 2018.

Rémi Jézéquel, Pierre Gaillard, and Alessandro Rudi. Mixability made efficient: Fast online multiclass logistic regression. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 23692-23702, 2021.

Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 208-214, 2011.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 2071-2080, 2017.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems 26 (NeurIPS) , pages 22562264, 2013.

Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems 27 (NeurIPS) , pages 1583-1591, 2014.

Marc Abeille and Alessandro Lazaric. Linear Thompson sampling revisited. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 176-184, 2017.

Branislav Kveton, Manzil Zaheer, Csaba Szepesvári, Lihong Li, Mohammad Ghavamzadeh, and Craig Boutilier. Randomized exploration in generalized linear bandits. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 2066-2076, 2020.

Peng Zhao, Lijun Zhang, Yuan Jiang, and Zhi-Hua Zhou. A simple approach for non-stationary linear bandits. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 746-755, 2020.

Blake Mason, Kwang-Sung Jun, and Lalit Jain. An experimental design approach for regret minimization in logistic bandits. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI) , pages 7736-7743, 2022.

Shi Dong, Tengyu Ma, and Benjamin Van Roy. On the performance of thompson sampling on logistic bandits. In Proceedings of the 32nd Annual Conference on Learning Theory (COLT) , pages 1158-1160, 2019.

Gergely Neu, Iuliia Olkhovskaia, Matteo Papini, and Ludovic Schwartz. Lifting the information ratio: An information-theoretic analysis of thompson sampling for contextual bandits. In Advances in Neural Information Processing Systems 36 (NeurIPS) , 2022.

Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Thompson sampling for the mnl-bandit. In Proceedings of the 30th Annual Conference on Learning Theory (COLT) , pages 76-78, 2017.

Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research , 67(5):1453-1485, 2019.

Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. In Advances in Neural Information Processing Systems 32 (NeurIPS) , pages 3145-3155, 2019.

Wang Chi Cheung and David Simchi-Levi. Thompson sampling for online personalized assortment optimization problems with multinomial logit choice models. Available at SSRN 3075658 , 2017.

Priyank Agrawal, Theja Tulabandhula, and Vashist Avadhanula. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research , 2023.

Noémie Périvier and Vineet Goyal. Dynamic pricing and assortment under a contextual MNL demand. In Advances in Neural Information Processing Systems 36 (NeurIPS) , 2022.

Yurii E. Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994.

Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics , 4:384-414, 2010.

Tianxiao Sun and Quoc Tran-Dinh. Generalized self-concordant functions: a recipe for newtontype methods. Mathematical Programming , 178(1):145-213, 2019.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Online-to-confidence-set conversions and application to sparse stochastic bandits. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1-9, 2012.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Nicolò Campolongo and Francesco Orabona. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems 33 (NeurIPS) , 2020.

A Properties of the Logistic Regression

This section collects several key properties of the logistic loss used throughout the paper.

A.1 Multiclass Logistic Loss

We have the following property of the multiclass logistic loss.

Property 1. Let y ∈ Y with Y = { 0 } ∪ [ K ] and ℓ : R K ×Y ↦→ R be the multiclass logistic loss defined by

where σ : R K → [0 , 1] K is the vector valued link function defined by [ σ ( z )] k = exp( z k ) / (1 + ∑ K j =1 exp( z j )) and [ σ ( z )] 0 = exp( z k ) / (1 + ∑ K j =1 exp( z j )) . Let y = [ 1 { y = 1 } , . . . , 1 { y = K } ] ∈ R K and σ k ( z ) = [ σ ( z )] k be the k -th entry of σ ( z ) . Then the first, second and third derivations w.r.t. the first argument of the loss function can be written as

where u k is the k -th entry of the vector u ∈ R K and E k ∈ R K × K is a 'one-hot' matrix whose ( k, k ) -th entry equals to 1 while others equaling to 0. Besides, e k is the one-hot vector whose k -th entry equals to 1. In the above, ∇ 3 z ℓ ( z , y )[ u ] = lim t → 0 t -1 ( ∇ 2 z ℓ ( z + t u , y ) -∇ 2 z ℓ ( z , y ) ) .

When we train a linear model with the logistic regression loss, we can calculate its gradient and Hessian as follows.

Property 2. Denote by the logistic loss defined over the data point ( x t , y t ) ∈ X × Y by

Then, the gradient ∇ ℓ t ( - → W ) ∈ R Kd and the Hessian ∇ 2 ℓ t ( W ) ∈ R Kd × Kd of the loss function with respective to the vectorized model - → W is given by

Besides, we can obtain the gradient of the vector-valued link function as follows.

Property 3. Let the sigmoid function σ : R K ↦→ [0 , 1] K be defined as (1) and σ k ( z ) denote the k -th entry of σ ( z ) . We have

In above, E k ∈ R K × K is an all zero matrix except that its ( k, k ) -th entry is 1 . For the first order derivation, we can write it into a more concise formulation by the notation ∇ σ ( z ) : R K ↦→ R K × K

We have the following lemma for the logistic loss function.

Lemma 1. Let C > 0 , a ∈ [ -C, C ] K , y ∈ { 0 } ∪ [ K ] and b ∈ R K . Then, we have

Proof. Lemma 1 is essentially the Lemma 4 of [27] with a slightly different definition of the logistic loss. The K + 1 -class logistic loss used by [27] is defined as ˜ ℓ ( ˜ z , y ) = ∑ K k =0 1 { y = k } log ( ∑ K k =0 exp([ ˜ z ] j ) / exp([ ˜ z ] k ) ) , where ˜ z ∈ R K +1 is a K + 1 -dimensional input. One can connect the two logistic losses by ˜ ℓ ([ z ; 0] , y ) = ℓ ( z , y ) . Then, we can prove Lemma 1 by directly applying Lemma 4 of [27] with the augmented vectors ˜ a = [ a ; 0] and ˜ b = [ b ; 0] .

A.2 Self-concordant-like Function

Then, we introduce the notion of self-concordant-like function, which generalizes the standard selfconcordant notion. The self-concordant function is first introduced by [44] in the study of inner-point algorithm and the notion is further generalized by the following work [45, 19, 46] for analyzing the Newton-step methods. The self-concordant function enjoys several nice properties, which plays an important role in our local analysis. A comprehensive study for self-concordance function is provided by [46]. Here, we present the definition and necessary lemmas for our analysis.

Definition 1 (Theorem 3 of [19]) . A convex function ℓ ∈ C 3 : R d → R is M -self-concordant-like if and only if for any x , u 1 , u 2 , u 3 ∈ R d , we have

where ∥ u ∥ z := √ u ⊤ ∇ 2 f ( z ) u is the local norm over the Hessian ∇ 2 ℓ ( z ) .

The logistic loss is √ 6 -self-concordant-like function.

Lemma 2 (Lemma 4 of [19]) . The logistic loss defined as (11) is √ 6 ∥ x t ∥ 2 -self-concordant-like.

The self-concordant function enjoys favorable properties, which help us to conduct local analysis for the logistic bandit problem. We list useful lemmas as follows.

Lemma 3 (Bound of Hessian Map, Theorem 4(b) of [19]) . Let ℓ : R d ↦→ R be an M -self-concordantlike function. Then, for any z 1 , z 2 ∈ dom( ℓ ) , we have

The following lemma is a consequence of the bound of the Hessian map.

Lemma 4 (Corollary 2 of [46]) . Let ℓ : R d ↦→ R be an M -self-concordant-like function. Then, for any z 1 , z 2 ∈ dom( ℓ ) , we have

We can also have the bound on the function value of the self-concordant functions.

Lemma 5 (Proposition 10 of [46]) . Let ℓ : R d ↦→ R be an M -self-concordant-like function. Then, for any z 1 , z 2 ∈ dom( ℓ ) , we have

where c : R ↦→ R is the coefficient function defined as c ( x ) = ( e x -x -1) /x 2 .

B Omitted Proofs for Section 2.3

B.1 Proof of Theorem 1

This section we present the proof of Theorem 1.

B.1.1 Main Proof

Before introducing the main proof, we introduce a key lemma on an improved self-normalized martingale tail inequality for the multinomial noise, which helps to save an O ( √ K ) factor compared to the one obtained by [11]. The proof of Lemma 6 is provided in Appendix B.1.2

Lemma 6. Let {F t } ∞ t =1 be a filtration. Let { x t } ∞ t =1 be a stochastic process in B 2 ( d ) ≜ { x ∈ R d | ∥ x t ∥ 2 ≤ 1 } such that x t is F t measurable. Let { ε t } ∞ t =1 be a martingale difference sequence such that ε t ∈ R K is F t +1 measurable. Furthermore, assume that conditional on F t , we have ∥ ε t ∥ 1 ≤ 2 almost surely, and denoted by Σ t ≜ E [ ε t ε ⊤ t | F t ] . Let λ > 0 and for any t ≥ 1 define

Then for any δ ∈ (0 , 1] , we have

Lemma 6 is a counterpart of the variance-aware self-normalized tail inequality [8, Theorem 1] for the multinominal case. In comparison with the previously established result by [11], our inequality more effectively leverages the geometric structure of the noise in the multinomial logistic bandit problem, as indicated by the condition ∥ ε t ∥ 1 ≤ 2 . The previous paper's condition, ∥ ε t ∥ 2 ≤ √ K , is relatively loose, ultimately resulting in an additional O ( √ K ) term in the concentration inequality. Then, we introduce the main proof of Theorem 1

Proof of Theorem 1. The optimality of the MLE estimator indicates that g t ( - → W MLE t ) = ∇L t ( - → W MLE t ) = 0 . In such a case, we have

where ε s = σ ( W ∗ x s ) -y s is the noise with bounded L ∞ norm as we have ∥ ϵ s ∥ ∞ ≤ ∥ σ ( W ∗ x s ) ∥ ∞ + ∥ y s ∥ ∞ ≤ 2 . Then, we can bound the gap between - → W MLE t and - → W ∗ by

with probability at least 1 -δ for all t ≥ 1 . In above, the matrix H t ( W ∗ ) is defined as H t ( W ∗ ) = λI Kd + ∑ t -1 s =1 Σ( W ∗ x s ) x s x ⊤ s . The first inequality is due to the relationship H -1 t ( W ∗ ) ≼ 1 /λ · I Kd . The last inequality is by Lemma 6 since

Then, we can bound the determinate of H t ( W ∗ ) by

Plugging (14) into (13), we have

By setting we have

which completes the proof .

B.1.2 Useful Lemmas

Lemma 6. Let {F t } ∞ t =1 be a filtration. Let { x t } ∞ t =1 be a stochastic process in B 2 ( d ) ≜ { x ∈ R d | ∥ x t ∥ 2 ≤ 1 } such that x t is F t measurable. Let { ε t } ∞ t =1 be a martingale difference sequence such that ε t ∈ R K is F t +1 measurable. Furthermore, assume that conditional on F t , we have ∥ ε t ∥ 1 ≤ 2 almost surely, and denoted by Σ t ≜ E [ ε t ε ⊤ t | F t ] . Let λ > 0 and for any t ≥ 1 define

Then for any δ ∈ (0 , 1] , we have

Proof of Lemma 6. The proof follows the pipeline in the analysis of the self-normalized tail-inequality in the previous studies [3, 8, 11]. Letting ¯ H t = ∑ t -1 s =1 Σ s ⊗ x s x ⊤ s , we define the function

for any t ≥ 1 and ξ ∈ R Kd . For t = 0 , let M 0 ( ξ ) = 0 .

By Lemma 7 in Appendix B.1.2, we can show that { M t ( ξ ) } ∞ t =1 is a non-negative super-martingale for any ξ ∈ 1 2 B 2 ( Kd ) . We note that our Lemma 7 is an imporved version of Lemma 7 in [11], which relaxs restriction on the feasible domain of ξ from ξ ∈ 1 √ K B 2 ( Kd ) to ξ ∈ 1 2 B 2 ( Kd ) , resulting in saving an O ( √ K ) factor in the bound.

Then, let h ( ξ ) be a probability density with support on 1 2 B 2 ( Kd ) and define

for all t ≥ 1 . Lemma 20.3 of [15] shows that ¯ M t is also a non-negative super-martingale and E [ ¯ M 0 ] = 1 . Then, the maximal inequality (Theorem 3.9 of [15]) shows that

Next, we turn to exam the formulation of ¯ M t and subsequently establish a connection with the term we aim to bound, ∥ S t ∥ H -1 t . Let h ( ξ ) be the density of an isotropic normal distribution with precision matrix 2 λI Kd truncated on 1 2 B 2 ( Kd ) and N ( h ) be its normalization constant. Furthermore, let g ( ξ ) be the density of the normal distribution with precision matrix 2 H t that is truncated on the ball 1 4 B 2 ( Kd ) . Following the arguments in the proof of [8, Theorem 1] (more precisely the arguments in deriving Eq.(11) in [8]), for any t ≥ 1 , one can show that

for any ξ ∈ 1 4 B 2 ( Kd ) . Let ξ 0 ≜ H -1 t S t ∥ S t ∥ H -1 t · √ λ 4 . One can check ∥ ξ 0 ∥ 2 ≤ 1 √ λ · √ λ 4 ≤ 1 / 4 . Then, we can further have

Combining (17) and (18), for any t ≥ 1 , we have

We complete the proof with Lemma 6 of [8] such that

Lemma 7. For all ξ ∈ 1 2 B 2 ( Kd ) , the sequence { M t ( ξ ) } ∞ t =1 is a non-negative super-martingale.

Proof of Lemma 7. To show that { M t ( ξ ) } ∞ t =1 is a non-negative super-martingale, it is sufficient to prove E [ M t +1 ( ξ ) | F t ] ≤ M t ( ξ ) for any t > 1 and ξ ∈ 1 2 B 2 ( Kd ) . By the definition of M t +1 ( ξ ) , we have

Let ξ = [ ξ 1 ; . . . ; ξ K ] and ξ k ∈ R d is the vector containing ( k -1) d +1 -th to kd -th elements of ξ . We can further check that

where the first inequality is due to the Höler's inequality. The last inequality is by the condition ∥ ε t ∥ 1 ≤ 2 and ∥ ξ k ∥ 2 ≤ ∥ ξ ∥ 2 ≤ 1 / 2 for any k ∈ [ K ] . Then, we can further bound the expectation term in (19) by

where the first inequality is due to Lemma 8 and the last equality is due to mixed-product property of the Kronecker product. Therefore, we can show E [ M t +1 ( ξ ) | F t ] ≤ M t ( ξ ) and complete the proof.

Lemma 8 (Lemma 6 of [11]) . Let ε ∈ R K be a zero-mean random vector with covariance matrix Σ . Then, for any vector a ∈ R K such that | ε ⊤ a | ≤ 1 , we have E [exp( ε ⊤ a )] ≤ exp( a ⊤ Σ a ) .

B.2 Proof of Theorem 2

This section provides the proof for Theorem 2.

B.2.1 Main Proof

Proof of Theorem 2. We prove the proposed algorithm can achieve O ( d log T √ κKT ) and O ( dK log T √ T + κd 2 K 2 log 2 T ) bounds simultaneously. Before presenting the proofs, we introduce two matrix that measures the local curvature of the loss function and will be used in the rest part of the proof:

Analysis for the O ( d log T √ κKT ) Bound. The regret can be bounded by

In the above, the term A ( x t , W ∗ , W OPT t ) is defined as (20) and the first inequality is a consequence of the optimistic rule (5) such that ( x t , W OPT t ) = arg max x ∈X ,W ∈C t ( δ ) ρ ⊤ σ ( W x ) and the first equality is due to the mean value theorem for the vector valued function. The second inequality is due to the Cauchy-Schwarz inequality and we can obtain the the last second inequality by the fact ∇ σ ( z ) ≼ I K for any z ∈ R K . The last inequality is due to Assumption 2.

Let G -1 2 t ( W 1 , W 2 ) be defined as (21). We can proceed to handle the gap between W OPT t and W ∗ by

where the first inequality is due to the fact A ( x t , W ∗ , W OPT t ) ≼ I K and the second inequality is due to the Cauchy-Schwarz inequality. Since both W ∗ , W OPT t ∈ W , the third inequality is by the condition G t ( W ∗ , W OPT t ) ≽ ¯ V t ≜ λI Kd + 1 κ ∑ t -1 s =1 I K ⊗ x s x ⊤ s . We can show the last inequality by using Cauchy-Schwarz inequality again.

Then, we proceed to bound term (a) and term (b) respectively.

In above, we denote by λ max the maximum eigenvalue of the input matrix. The second inequality is due to the property of the maximum eigenvalue such that λ max ( ABC ) = λ max ( BCA ) for matrix A,B and C . The last second equality is due to the mixed product property of the the Kronecker product and ( A ⊗ B ) -1 = A -1 ⊗ B -1 for matrix A and B . In the last inequality, we can bound the term with the standard arguments of the elliptical potential lemma [3, Lemma 11].

It remains to bound term (b). Since Theorem 1 shows that W ∗ ∈ C t ( δ ) with probability at least 1 -δ for all t > 1 and the optimistic rule (5) ensures W OPT t ∈ C t ( δ ) , Lemma 9 in Appendix B.2.2 indicates that the distance between W ∗ and W OPT t can be bounded by

Then, plugging the above displayed equation into term (b), we can bound this term by

where the last inequality holds because β t ( δ ) is a non-decreasing function with respect to t . A combination of the upper bound for term (a) and term (b) leads to

Analysis for the O ( Kd log T √ T + κKd 2 log 2 T ) Bound. Inspired by the analysis for binary logistic bandit problem [9, Appendix C], we employ a different decomposition compared to the one used in the first part of the analysis

where σ k : x ↦→ [ σ ( x )] k is the k -th output of the vector-valued function σ ( x ) and Ξ k,t = ∫ 1 ν =0 (1 -ν ) ∇ 2 σ k (( W ∗ + ν ( W OPT t -W ∗ )) x t )d ν . In the above, the first inequality is due to the optimistic rule (5) and the last equality is due to the integral formulation of the Taylor series. Then, we proceed to handle term (c) and term (d) respectively.

As for term (c), we have

where the first inequality holds by Cauchy-Schwarz inequality and the last one is due to Lemma 9. Let λ max ( · ) denote the maximum eigenvalue of the matrix. We can further bound (24) by

where the first inequality is due to the fact that ∥ A b ∥ 2 ≤ ∥ A ∥ 2 · ∥ b ∥ for a matrix A and vector b . The second inequality is due to the definition of the induced norm ∥ A ∥ 2 = √ λ max ( A T A ) and the mixed-product property of the Kronecker production. The last equality is due to the cycle property of the maximum eigenvalue such that λ max ( ABC ) = λ max ( CAB ) for matrices A,B and C and we use the mixed-product property again. The last inequality is due to Lemma 10 in Appendix B.2.2.

Combining (24) and (25), we arrive

Then, we turn to handle term (d). Lemma 11 indicates that

As a consequence, we can bound term (d) by

- →

where the first inequality is due to Assumption 2 and the second inequality is due to (27). The last second inequality can be obtained by a similar argument in obtaining (22). The last inequality can be obtained by the same argument as (23).

Combining (26) and (26), we obtain

We have completed the proof.

B.2.2 Useful Lemma

This section provides the lemmas used in the main proof.

Lemma 9. If W ∗ ∈ C t ( δ ) , then for all W ∈ C t ( δ ) :

is defined in Theorem 1.

Proof of Lemma 9. Lemma 3 in [11] shows that

where the last second inequality is due to generalized self-concordant properties of the logistic loss as shown by Lemma 13 of [11]. The last inequality is due to Theorem 1 and the condition that W is contained in the confidence set C t ( δ ) .

Lemma 10. When the regularization parameter λ > 2 , we have

Proof of Lemma 10. Denoting by Tr( A ) the trace of matrix A , We have

where the last inequality is due to Lemma 4 of [11] such that (1 + 2 S ) G t ( W ∗ , W OPT t ) ≽ H t ( W ∗ ) .

Let M t ( W ∗ ) = λ 2 I Kd + ∑ t s =1 ∇ σ ( W ∗ x s ) ⊗ x s x ⊤ s . Then, we can further bound the last line of the above displayed equations by

The first inequality is by the fact H t ( W ∗ ) ≽ M t ( W ∗ ) under the condition λ ≥ 2 . The last second inequality is due to Lemma 4.5 of [16].

Lemma 11. Let σ k : z ↦→ [ σ ( x )] k ∈ R by the k -th output of the vector-valued function σ ( z ) . Then, for any z ∈ R K , we have ∇ 2 σ k ( z ) ≤ 3 I K for any k ∈ [ K ] .

Proof of Lemma 11. For any z ∈ R K , we have

where we denote by E k as the matrix where the entry at the ( k, k ) -th position is 1, and all other entries are 0. In the above, the first inequality is due to the definition of σ k . The second inequality holds since diag( σ ( z )) is semi-positive defined matrix. The last inequality is a consequence of the fact that the maximum eigenvalue of σ k ( z ) σ ( z ) σ ( z ) ⊤ is bounded by σ k ( z )( σ ( z ) -e k )( σ ( z ) -e k ) ⊤ is less than 1 . The maximum eigenvalue of σ k ( z )( σ ( z ) -e k )( σ ( z ) -e k ) ⊤ is bounded by 2.

C Omitted Proofs for Section 3

This section presents the omitted details for Section 3. To simplify the notation, we exclude the superscript OL in all the proofs within this section.

C.1 Proof of Theorem 3

This section provides the proof of Theorem 3. We will first provide the main proof of Theorem 3 and the prove the technical lemma used in the main proof.

C.1.1 Main Proof

Proof of Theorem 3. The proof shares the similarity with the online-to-confidence-set conversion technique [47, 14], where the estimation error of the online estimator W OL t is bounded by the regret. However, we have introduced a novel algorithm and analysis ingredient to achieve jointly statistical and computation efficiency.

The following lemma provides the estimation error analysis. Drawing inspiration from the modern analysis of OMD [23], we explicitly extract the negative term -∑ t s =1 ∥ - → W s +1 - - → W s ∥ 2 H s in the upper bound, which is pivotal for our subsequent theoretical analysis and the algorithm design.

Lemma 12. Under Assumptions 1 and 3, we consider the estimator update rule

where ˜ ℓ t ( W ) = ⟨∇ ℓ t ( - → W t ) , - → W ⟩ + 1 2 ∥ - → W - - → W t ∥ 2 ∇ 2 ℓ t ( - → W t ) and H t = λI Kd + ∑ t -1 s =1 ∇ 2 ℓ s ( W s +1 ) ⊗ x x ⊤ . Then, letting α = ln( K +1) + 2( S +1) and λ > 0 , we have s s

when the step size is set as η = α/ 2 .

Then, we focus on the first term of the right hand side. Inspired by the previous studies on binary logistic bandit [10], we decompose the regret into two part by inserting an intermediate decision.

where the ˜ z s is an aggregating forecaster for logistic loss defined by ˜ z s = σ + ( E W ∼ P s [ σ ( W x s )]) and P s = N ( - → W s , (1 + cH -1 s ) is the Gaussian distribution with mean - → W s and covariance matrix cH -1 s , where c > 0 is a constant to be specified. In above, σ + : ∆ ↦→ R K is a pseudo-inverse function of σ ( · ) whose k -th output is [ σ + ( p )] k = log ( p k / (1 -∥ p ∥ 1 ) ) for any p ∈ { q ∈ [0 , 1] K |∥ q ∥ 1 < 1 } . We bound the above two terms respectively. First, we show that the term (A) is bounded by O ( log 2 t ) with high probability.

Lemma 13. Let δ ∈ (0 , 1] . Under Assumptions 1 and 3, we have

where the confidence radius is γ A t = (log(1+ K )+2log(1+(1+ K ) t )) ( 5 4 +4log ( √ 1+2 t δ )) +2 = O ((log K +log t ) log t ) .

Besides, the term (b) can be bounded by the following lemma,

Lemma 14. Under Assumptions 1 and 3, for any c > 0 , we have

by setting λ ≥ max { 2 , 24 Kdc } . In the above, γ B t ( δ ) = √ 6 cKd ln ( 1 + ( t +1) L 2 λ ) and L := max W ∈ R Kd ,t ∈ [ T ] ∇ 2 ℓ t ( W ) is the smooth parameter of the logistic loss.

Combining Lemma 12, Lemma 13 and Lemma 14, we arrive at

For the last second inequality, we eliminate the additional term by the parameter setting η = α/ 2 , c = 7 α/ 6 and λ ≥ { 7 √ 6 αS, 28 Kdα } and due to the fact that I K ⊗ x s x ⊤ s ≼ I Kd under the condition ∥ x t ∥ 2 ≤ 1 for any t ∈ [ T ] . In the above, we have λ = O ( Kd log K ) and α = O (log K ) . Then, we can show ∥ - → W t +1 - - → W ∗ ∥ H t +1 ≤ β OL t ( δ ) = O (log K log t √ Kd ) , which completes the proof.

C.1.2 Proof of Lemma 12

Proof of Lemma 12. Let ˜ ℓ s ( W ) = ℓ s ( W s ) + ⟨ - → W - - → W s , ∇ ℓ s ( - → W s ) ⟩ + 1 2 ∥ - → W - - → W s ∥ 2 ∇ 2 ℓ s ( - → W s ) be a second order approximation of the original function ℓ s ( W ) at the point - → W s . The update rule (6) can be equally written as

which is an implicit online mirror descent update with the loss function ˜ ℓ s . Then, according to Lemma 16, the model W t +1 ensures,

One the other hand, let α = log(1 + K ) + 2(1 + S ) . Since ∥ W ∗ x t ∥ ∞ = max k ∈ [ K ] {∥ w ( k ) ∗ ∥ 2 · ∥ x t ∥ 2 } ≤ S and ℓ s ( W ) = ℓ ( W x s , y s ) , Lemma 1 shows

By setting η = α/ 2 , the combination of (31) and (32) shows

In above, we can further bound inner product term by

In above ξ s +1 is a certain point on the line connecting - → W t and - → W t +1 . The notation is defined as D 3 ℓ s ( - → W )[ - → U ] = lim α → 0 α -1 ( ∇ 2 ℓ s ( - → W + α - → U ) -∇ 2 ℓ s ( - → W ) ) . The first inequality is due to the definition of ℓ s and the second inequality is by the Taylor series for the vector-valued function. The third inequality is because the third order derivative of the logistic loss is a symmetric tensor as shown in Property 1. The last second inequality is by Definition 1 since the multiclass logistic loss is a √ 6 self-concordant-like function. The last inequality is by the boundedness of the decision domain W and bounded action such that ∥ x ∥ 2 ≤ 1 for any x ∈ X under Assumption 1.

Combining (33) and (34), we have

Taking the summation of the above inequality over t rounds and rearranging the term, we have

which completes the proof.

C.1.3 Proof of Lemma 13

Proof of Lemma 13. Since the norm of the intermediate decision ˜ z s = σ + ( E W ∼ P s [ σ ( W x s )]) is generally unbounded, as suggested by [26], we use the smoothed version ˜ z µ s = σ + (smooth µ ( E W ∼ P s [ σ ( W x s )])) as an intermediate term in the analysis. In above, the smooth function smooth µ : [0 , 1] K ↦→ [0 , 1] K with parameter µ ∈ [0 , 1 / 2] is defined by smooth µ ( p ) = (1 -µ ) p + µ 1 / ( K +1) , where 1 ∈ R K is an all one vector.

Given the construction of the pseudo inverse function σ + such that σ ( σ + ( p )) = p for any p ∈ { q ∈ [0 , 1] K | ∥ q ∥ 1 < 1 } , one can check that ˜ z µ s = σ + (smooth µ ( σ ( ˜ z s ))) . Then, Lemma 17 shows that

Besides, Lemma 17 also shows that ∥ ˜ z µ s ∥ ∞ ≤ log(1 + ( K +1) /µ ) . Therefore, to prove the lemma, it is sufficient bound the gap between the loss of W ∗ and ˜ z µ s . By the definition of the loss function ℓ s , we have ℓ s ( W ∗ ) = ℓ ( z ∗ s , y s ) , where z ∗ s = W ∗ x s . Then, we have

where S µ = log( K +1) + 2log(1 + ( K +1) /µ ) . In above, the first inequality is due to Lemma 1. The second equality is by a direct calculation of the first order and Hessian of the logistic loss as shown in Property 1.

To bound the first term of the right hand side, we require the following lemma, which adapts lemma 6 to a one-dimension case and whose proof is provided C.1.5.

Lemma 15. Let {F t } ∞ t =1 be a filtration. Let { z t } ∞ t =1 be a stochastic process in B 2 ( K ) = { z ∈ R K | ∥ z ∥ ∞ ≤ 1 } such that z t is F t measurable. Let { ε t } ∞ t =1 be a martingale difference sequence such that ε t ∈ R K is F t +1 measurable. Furthermore, assume that conditional on F t , we have ∥ ε t ∥ 1 ≤ 2 almost surely, and denote by Σ t := E [ ε t ε ⊤ t | F t ] . Let λ > 0 and for any t ≥ define

Then for any δ ∈ (0 , 1] , we have

To check the condition of Lemma 15, we let d s = ( z ∗ s -˜ z µ s ) / ( S µ + S ) . Since ∥ z ∗ s ∥ ∞ ≤ max k ∈ [ K ] ∥ w ( k ) ∗ ∥ 2 ∥ x s ∥ 2 ≤ S and ∥ ˜ z µ s ∥ ∞ ≤ log(1 + ( K +1) /µ ) , one can verify that ∥ d s ∥ ∞ ≤ 1 . Besides, since z ∗ s and ˜ z µ s is independent of y s , the variable d s is F s measurable. Furthermore, we can check that E [( σ ( z ∗ s ) -y s )( σ ( z ∗ s ) -y s ) ⊤ | F s ] = ∇ σ ( z ∗ s ) and ∥ σ ( z ∗ s ) -y s ∥ 1 ≤ 2 . Thus, a direct application of Lemma 15 shows that, with probability at least 1 -δ , we have

for any t ≥ 1 . In above, the second inequality is a consequence of the fact ∥ d s ∥ 2 Σ( z ∗ s ) = d ⊤ s ∇ σ ( z ∗ s ) d s ≤ 2 . Then, combining (36) and (38) and setting λ = 1 , we arrive ‡

where the second inequality is due to AM-GM inequality. Finally, combining (35) and (39), we have

where the last inequality is by the parameter setting µ = 1 /t and, as a consequence, S µ = log(1 + K ) + 2 log(1 + ( K +1) t ) . We have completed the proof.

C.1.4 Proof of Lemma 14

Proof of Lemma 14. Our proof starts with the observation made by [26, Proposition 2] that ˜ z s is an aggregating forecaster [48, Chapter, 3.5] for the logistic function, which satisfies

Then, a direct calculation with the definition of Gaussian distribution P s ∼ N ( - → W s , cH -1 s ) gives

where we define the loss function L s ( W ) = ℓ s ( W ) + (2 c ) -1 ∥ - → W -- → W s ∥ 2 H s and Z s = √ (2 π ) Kd c | H -1 s | being the normalization factor.

Then, we consider the following quadratic approximation,

According to Lemma 18, we have

Then, we can low bound the term in the expectation by

where we define the function ˜ f s : W ↦→ R as

Denote by ˜ Z s +1 = ∫ W ∈ R Kd ˜ f s +1 ( W )d W ≤ + ∞ the normalization factor and ˜ P s +1 the distribution whose density function is ˜ f s +1 ( W ) / ˜ Z s +1 , we can further rewrite the last line of the above displayed equation (42) as

where the second inequality is due to the Jensen's inequality and the last equality comes from the fact that ˜ f s +1 ( W ) / ˜ Z s +1 is symmetry around - → W s +1 and thus E W ∼ ˜ P s +1 [ ⟨∇ L s ( W s +1 ) , - → W - - → W s +1 ⟩ ] = 0 .

Plugging (43) into (40), we arrive

where we can further convert the last term -ln ˜ Z s +1 to

where ̂ P s +1 = N ( - → W s +1 , cH -1 s ) and ̂ Z s +1 = ∫ R Kd e -1 2 c ∥ - → W - - → W s +1 ∥ 2 Hs d W . In above, the first equality is by definition of ˜ Z s +1 and the second one comes from the definition of ̂ P s +1 . The last second inequality is due to the Jensen's inequality such that ln( E [ a ]) ≥ E [ln a ] for the random variable a ∈ R . The last equality is by the fact ̂ Z s +1 = ∫ R Kd e -1 2 c ∥ - → W - - → W s +1 ∥ 2 Hs d W = Z s .

Combining (44) and (45) and taking the summation overt t iterations, we have

Following a similar argument in [27], we can bound the last term as follows. Specifically, by the Cauchy-Schwarz inequality, we can decompose the term Ω s as

Then, remarking that ̂ P s +1 = N ( W s +1 , cH -1 s ) , the key observation here is that - → W - - → W s +1 also follows the same distribution as

where { e i ∈ R Kd } Kd i =1 is a set of certain orthogonal basis and λ i := λ i ( H -1 s ) denotes the i -th largest eigenvalue of H -1 s . Then, we can bound the first term of term (a-1) by

where χ is the chi-square distribution and the last second inequality is due to H s ≽ λI Kd and thus λ i ≤ 1 /λ . The last inequality is due to the Jensen's inequality since Kd > 1 . Then, by further requiring λ ≥ 24 Kdc , we have

where the last inequality is because the moment-generating function for χ 2 -distribution is bounded by E X ∼ χ 2 [ e tX ] ≤ 1 / √ 1 -2 t for all t ≤ 1 / 2 .

Then, we consider the second on the upper bound of term (b). A direct calculation gives,

where ˜ H s = ( ∇ 2 ℓ s ( W s +1 )) -1 2 H s ( ∇ 2 ℓ s ( W s +1 )) -1 2 . Then, let ˜ λ i := λ i ( c ˜ H -1 s ) be the i -th largest eigenvalue of the matrix. A similar decomposition as (47) shows that

where the last inequality is due to E X i ,X j ∼N (0 , 1) [ X 2 i X 2 j ] ≤ 3 for all i, j ∈ [ Kd ] and the last equality comes from the fact that ∑ Kd i =1 ˜ λ i = Tr ( c ˜ H -1 s ) . The notation Tr ( A ) is used to denote the trace of the matrix A .

Then, we proceed to bound the trace Tr ( ˜ H -1 s ) with ˜ H s = H s ∇ 2 ℓ s ( W s +1 ) . Recall the definition of H s = λI Kd + ∑ s -1 τ =1 ∇ 2 ℓ τ ( W τ +1 ) . We define M s +1 = λI Kd / 2 + ∑ s τ =1 ∇ 2 ℓ τ ( W τ +1 ) . Under the condition λ ≥ 2 for any s ∈ [ T ] , we have ∇ 2 ℓ s ( W ) ≼ I Kd ≤ λ 2 I Kd for any s ∈ [ T ] and W ∈ W . Then, we have H s ≽ M s +1 . Then, we can bound the trace by

where the last inequality is due to Lemma 4.5 of [16]. As a consequence, we have term ( a -2 ) ≤ √ 3 c ln( | M s +1 | / | M s | ) , which leads to

Combining (46) and (49), we obtain that

W can further bound the last term of the displayed equality by ∑ t s =1 ln ( | M s +1 | / | M s | ) ≤ ln ( | M t +1 | / | λ/ 2 · I Kd | ) ≤ Kd ln ( 1 + ( t +1) L 2 λ ) , which completes the proof.

C.1.5 Useful Lemmas

Proof of Lemma 15. The proof of Lemma 15 shares the same spirit as that of Lemma 6. Let ¯ H t = ∑ t -1 s =1 ∥ z s ∥ 2 Σ s . We can also defined the function

for any t ≥ 1 and ξ ∈ R . For t = 0 , we let M 0 ( ξ ) = 0 . Following the similar arguments in the proof of Lemma 7, one can show that the sequence { M t ( ξ ) } ∞ t =1 is a non-negative super-martingale when ξ ≤ 1 2 . Then, let h ( ξ ) be the density of the normal distribution with precision 2 λ truncated on the 1-dimensional ball 1 2 B (1) . We can define

By the similar arguments in deriving (17), we have

Following the arguments in the proof of [8, Theorem 1], for any t ≥ 1 , we have

for any | ξ | ≤ 1 / 4 . In the above, g ( ξ ) is the normal distribution with precision 2 H t truncated on the interval [ -1 / 4 , 1 / 4] . Then, let ξ 0 = 1 4 √ λ H t , we have ξ 0 ≤ 1 / 4 since H t ≥ λ . We can obtain that

A combination of (50) and (51) shows that

which indicates

Wecomplete the proof with Lemma 6 of [8], which shows that log ( N ( h ) /N ( g )) ≤ log ( (2 √ H t ) /λ ) .

Lemma 16 (Proposition 4.1 of [49]) . Let the w t +1 be the solution of the update rule

where V ⊆ W ⊆ R d is a non-empty convex set and D ψ ( w 1 , w 2 ) = ψ ( w 1 ) -ψ ( w 2 ) -⟨∇ ψ ( w 2 ) , w 1 -w 2 ⟩ is the Bregman Divergence w.r.t. a strictly convex and continuously differentiable function ψ : W ↦→ R . Further supposing ψ ( w ) is 1-strongly convex w.r.t. a certain norm ∥·∥ in W , then there exists a g ′ t ∈ ∂ℓ t ( w t +1 ) such that

for any u ∈ W .

Lemma 17. Let ℓ ( z , y ) = ∑ K k =0 1 { y = k } · log ( 1 [ σ ( z )] k ) and z ∈ R K be a K -dimensional vector. Define z µ ≜ σ + ( smooth µ ( σ ( z )) ) , where smooth µ ( p ) = (1 -µ ) p + µ 1 / ( K + 1) . Then, for µ ∈ [0 , 1 / 2] , we have

for any y ∈ { 0 } ∪ [ K ] . We also have ∥ z µ ∥ ∞ ≤ log( K/µ ) .

Proof of Lemma 17. The proof of Lemma 17 is extracted from the proof of [26, Lemma 3] with a slight modification to the logistic loss used in this paper. According to the definition of z µ and the fact that σ ( σ + ( p )) = p for any p ∈ { q ∈ [0 , 1] K |∥ q ∥ 1 < 1 } , we have σ ( z µ ) = (1 -µ ) σ ( z ) + µ 1 / ( K +1) . Then, we have

where the first inequality is due to ∑ K k =0 1 { y = k } = 1 and the last inequality is due to log(1 / (1 -a )) ≤ a for a ∈ [0 , 1 / 2] . Besides, let ˜ p = σ ( z ) . According to the definition of z µ , we have

Since the term inside the logarithmic function can be bounded by

and µ/ (1 + K (1 -µ )) ≤ 1 , we have ∥ z ∥ ∞ ≤ log(1 + ( K +1) /µ ) .

Lemma 18. Let L s ( W ) = ℓ s ( W ) + 1 2 c ∥ - → W - - → W s ∥ 2 H s . Then, for any W,W s +1 ∈ W , the quadratic approximation ˜ L s ( W ) = L s ( W s +1 ) + ⟨∇ L s ( W s +1 ) , - → W - - → W s +1 ⟩ + 1 2 c ∥ - → W - - → W s +1 ∥ 2 H s defined as (41) satisfies

Proof of Lemma 18. According to [19, Lemma 4], ℓ s is a √ 6 -self-concordant-like function. Then, by Lemma 5 and the fact c ( x ) ≤ e x 2 for any x ≥ 0 , then for any W ∈ W , we have

Besides, since g s ( W ) = 1 2 c ∥ - → W - - → W s ∥ 2 H s is a quadratic function, we have

Then combing (52) and (53), we can obtain an upper bound of L s ( W ) = ℓ s ( W ) + g s ( W ) by

We have complete the proof.

C.2 Proof of Proposition 1

This section presents the proof of Proposition 1.

C.2.1 Main Proof

Proof of Proposition 1. Since ρ ⊤ σ ( W ∗ x ) ≤ ρ ⊤ σ ( W t x ) + | ρ ⊤ σ ( W ∗ x ) -ρ ⊤ σ ( W t x ) | , to show ˜ r OL t ( x ) = ρ ⊤ σ ( W t x ) + ϵ fst t ( x ) + ϵ snd t ( x ) is an optimistic estimate, it is sufficient to prove that | ρ ⊤ σ ( W ∗ x ) -ρ ⊤ σ ( W t x ) | ≤ ϵ fst t ( x ) + ϵ snd t ( x ) . We have the following decomposition:

where σ k : x ↦→ [ σ ( x )] k is the k -th output of the vector-valued function σ ( x ) and Ξ k,t = ∫ 1 ν =0 (1 -ν ) ∇ 2 σ k (( W t + ν ( W ∗ -W t )) x )d ν . In the above, the last equality is due to the integral formulation of the Taylor series.

Then, we proceed to analyze term (a) and term (b), respectively.

where the last inequality is due to Theorem 3.

Then, we upper bound term (b) by ϵ snd t with the following arguments. For notation simplicity, we denote by ξ t,ν = W t x + ν ( W ∗ -W t ) x . Then, Lemma 11 indicates that

As a consequence, we can bound term (b) by

where the first inequality is due to Assumption 2 and the last inequality is due to Theorem 3. Combining the upper bound for term (a) and term (b), we have

for any x ∈ X . We complete the proof by ρ ⊤ σ ( W ∗ x ) ≤ ρ ⊤ σ ( W t x ) + | ρ ⊤ σ ( W ∗ x ) -ρ ⊤ σ ( W t x ) |

C.3 Proof of Theorem 4

Proof of Theorem 4. We first prove the regret bound and then discuss the computation cost.

C.3.1 Main Proof

Regret Analysis. We can bound the regret by two times of the bonus term over x t .

.

where the first inequality is due to Proposition 1 and the second inequality is due to the rule of constructing the optimistic reward. The last inequality is due to (54).

Then, we turn to analyze the upper bound for the first term and second term respectively.

For the first term, we have

where the first inequality is due to the fact that ∥ A b ∥ 2 ≤ ∥ A ∥ 2 · ∥ b ∥ for a matrix A and vector b . The second inequality is due to the definition of the induced norm for the matrix is defined as ∥ A ∥ 2 = √ λ max ( A T A ) and the mixed-product property of the Kronecker production. The first equality is due to the cycle property of the maximum eigenvalue such that λ max ( ABC ) = λ max ( CAB ) for matrices A,B and C . The last second inequality is due to the Cauchy-Schwarz inequality. Finally, we can obtain the last inequality by Lemma 19, which can be seen as a matrix version of the elliptical potential lemma.

Then, we can bound term (b) as follows.

In the above, the second equality is due to the mean-value theorem, where denote by ξ t,k ∈ R K a certain point on the line connecting W t x t and W t +1 x t . We can further bound the second term of the right hand side of the above displayed inequality by

where the first inequality is by the fact ∥ A b ∥ 2 ≤ ∥ A ∥ 2 · ∥ b ∥ 2 for any matrix A and vector b . The second inequality is due to Lemma 20. The last inequality is due to Lemma 11 such that ∇ σ k ( z ) ≼ 3 I k for any z ∈ R K and ∑ K k =1 ρ k ≤ √ KR .

Then plugging (58) into (57), we can bound term (b) by

where the last inequality is due to Lemma 21. Combining the upper bound for term (a) and term (b) and plugging them into (56), we have

As for the term ∑ T t =1 ϵ snd t ( x t ) , we have

Combining (59) and (60) with (55), we have

which completes the proof for the regret bound.

C.3.2 Additional Guarantee for Algorithm 2

This part shows that the OFUL-MLogB (Algorithm 2) also achieves an ˜ O ( √ κKT ) regret bound. Theorem 5. Under the same condition as Theorem 3, Algorithm 2 ensures

Proof of Theorem 5. The proof of Theorem 5 is almost the same as that for Theorem 4. We can decompose the regret as

following the same arguments (55). The only difference is that we can bound the ∑ T t =1 ϵ fst t ( x t ) in (55) with a different arguments based on the local matrix ¯ V t ≜ λI Kd + 1 κ ∑ t -1 s =1 I K ⊗ x s x ⊤ s .

where the first inequality is by the fact that H t ≽ ¯ V t , ∇ σ ( W t x t ) ≼ I K , and ∥ ρ ∥ 2 ≤ R . The second inequality can be obtained ϵ snd t with the same arguments as (60):

We completed the proof by combining the above three displayed bounds.

C.3.3 Useful Lemmas

Lemma 19. Let H t = λI Kd + ∑ t -1 s =1 ∇ σ ( W s +1 x s ) ⊗ x s x ⊤ s . Then, when λ > 2 , we have

where L = max x ∈X ,W ∈W λ max ( ∇ σ ( W x ))

Proof of Lemma 19. Let M t = λ 2 I Kd + ∑ t s =1 ∇ σ ( W s +1 x s ) ⊗ x s x ⊤ s . Then, we have H t -M t = λ 2 I Kd -∇ σ ( W t +1 x t ) ⊗ x t x ⊤ t . Since ∇ σ ( W t +1 x t ) ⊗ x t x ⊤ t ≼ I Kd , we have H t ≽ M t when λ > 2 . Then, we have

where λ > 2 .

Proof of Lemma 21. The proof of Lemma 21 shares the same spirits with that of Lemma 19. Let ¯ V t = λ 2 I Kd + 1 κ ∑ t s =1 I K ⊗ x s x ⊤ s . We have H t ≽ ¯ V t when λ > 2 . We can prove the lemma by

where we denote by Tr( A ) the trace of matrix A. The second inequality is by the condition H t ≽ M t . The last second inequality is due to Lemma 4.5 of [16].

Lemma 20. Let - → W t +1 = arg min W ∈W ⟨∇ ℓ t ( - → W t ) , - → W ⟩ + 1 α ∥ - → W - - → W t ∥ 2 ˜ H t . Then, we have

Proof of Lemma 20. Let F t ( - → W ) = ⟨∇ ℓ t ( - → W t ) , - → W ⟩ + 1 α ∥ - → W -- → W t ∥ 2 ˜ H t . Since - → W t +1 = arg min W ∈W F t ( - → W ) , we have F t ( - → W t +1 ) ≤ F t ( - → W t ) , which implies

Then, by the Hölder's inequality, we can further bound the inner product term by

which indicates that

Since ˜ H t ≽ H t and ˜ H -1 t ≼ I K d/λ , we can further have,

where the last inequality is due to the definition of ∇ ℓ t ( - → W t ) = ( σ t ( - → W t ) -y t ) ⊗ x t such that ∥∇ ℓ t ( - → W t ) ∥ 2 ≤ 2 √ K .

Lemma 21. Let H t = λI Kd + ∑ t -1 s =1 ∇ σ ( W s +1 x s ) ⊗ x s x ⊤ s . Then, we have

where the first inequality is due to H t ≽ ¯ V t . The second equality is due to the definition of ¯ V t and the third equality is due to the mixed-product property of the Kronecker product. The last inequality can be obtain by the standard elliptical potential lemma [3, Lemma 11].

C.3.4 Computation Cost of Algorithm 2

Algorithm 2 OFUL-MLogB

Input: regularization coefficient λ , probability δ , step size η .

1: Initialize H 1 = λI Kd and - → W 1 as any point in W

2: for t = 1 , . . . , T do

3: Select the arm by x t = arg max x ∈X ˜ r t ( x ) and receive y t .

5: Update the estimator - → W t +1 for the next iteration by (6)

4: Update ˜ H t = H t + η ∇ σ ( W t x t ) ⊗ x t x ⊤ t

6: Update H t +1 = H t + ∇ σ ( W t +1 x t ) ⊗ x t x ⊤ t and

8: end for

7: Construct the optimistic reward by ˜ r t +1 ( x ) = ρ ⊤ σ ( W t +1 x ) + ϵ fst t +1 ( x ) + ϵ snd t +1 ( x ) as (8).

Here, we discuss about the computation cost of the Algorithm 2. For each iteration, our algorithm requires to maintain the inverse of matrix H t +1 = H t + ∇ σ ( W t +1 x t ) ⊗ x t x ⊤ t and ˜ H t = H t + η ∇ σ ( W t x t ) ⊗ x t x ⊤ t . Since ∇ σ ( W t +1 x t ) ⊗ x t x ⊤ t and ∇ σ ( W t x t ) ⊗ x t x ⊤ t are both rankK matrix, then one can main H t and ˜ H t with O ( K 3 d 2 ) computation cost per iteration by the Sherman-MorrisonWoodbury formula. Given the H -1 t and ˜ H -1 t , we discuss the computation cost of the construction of the optimistic reward 8 and the update rule for the estimator (6).

Computation Cost of the Estimator (6) . As shown by the discussion in Section 3.2, the update rule (6) is identical to

Given the ˜ H -1 t , we can perform the gradient step with O ( K 2 d 2 ) time complexity. As for the projection step, the optimization problem can be solved in O ( K 3 d 3 ) time [18, Section 4]. As a consequence, the overall time complexity for obtaining the estimator (6) is O ( K 3 d 3 ) .

Computation Cost of Building Optimistic Reward. As shown by (8), we construct the optimistic reward by ˜ r OL t ( x ) = ρ ⊤ σ ( W OL t x ) + ϵ fst t ( x ) + ϵ snd t ( x ) . We can compute ϵ fst t ( x ) with the following equivalent formulation

Given H -1 t , it will take O ( K 2 d 2 ) to calculate ϵ fst t ( x ) . As for the term ϵ fst t ( x ) , it can also be rewritten as

It would take O ( K 3 d 3 ) time to perform the eigenvalue decomposition. Given there are |X| arm, the time complexity for identify the arm x t = arg max x ∈X ˜ r t ( x ) is in total O ( |X| K 3 d 3 ) .

Overall, Algorithm 2 can be implemented in O ( |X| K 3 d 3 ) time, which is independent of T .

C.4 Proof of Corollary 1

C.4.1 Main Proof

In the binary case, we select the arm by ( x t , ˜ w t ) = arg max x ∈X ,W ∈C t ( δ ) w ⊤ x . In such a case, we show Algorithm 2 achieves an ˜ O ( T/κ ∗ ) bound with a constant computation cost. The proof is almost the same as that of [10, Theorem 2]. The main difference is that we complete the proof with the confidence set of the efficient online estimator (Theorem 3). We present the proof here for self-containedness.

Proof of Corollary 1. When K = 1 and ρ 1 = 1 , the MLogB problem recovers the binary logistic bandit problem with the feedback y t = { 0 , 1 } and the reward model Pr [ y = 1 | x ] = σ ( w ⊤ ∗ x ) . In such a case, we can decompose the regret by

where the first inequality is due to the arm selection rule. The first equality is due to the Taylor series and ξ t ∈ R d is a certain point on the line connecting ˜ w t and w ∗ . For the last equality, we divide the time horizon into two parts I 1 = { t ∈ [ T ] | σ ′ ( w ⊤ ∗ x t ) ≥ σ ′ ( w ⊤ t +1 x t ) } and I 2 = [ T ] /I 1 .

For term (a), we have

where the first inequality is due to the mean value theorem and the condition that | σ ′′ ( z ) | ≤ 1 for any z ∈ R . The second inequality is due to the Cauchy-Schwarz inequality and H t ≼ H t +1 . We can further bound term (a-1) by

where the first inequality is due to ˜ w t and w ∗ are both contained in C t ( δ ) and β t ( δ ) = O (log t ) is the radius of C t ( δ ) as shown in Theorem 3. The last second inequality is by the condition such that σ ′ ( w ⊤ ∗ x t ) ≥ σ ′ ( w ⊤ t +1 x t ) for all t ∈ I 1 . Then, following the same argument in the proof of [10, Theorem 2], one can show

which leads to the last inequality. We also use the elliptical potential lemma [10, Lemma 9] in the last inequality.

As for term (a-2), we have

where the first inequality holds because w ∗ and ˜ w t are contained in C t ( δ ) . The second inequality can be obtained following the similar argument in obtaining (60). Combining the upper bound for term (a-1) and term (a-2), we have

As for term (b), we have σ ′ ( w ⊤ ∗ x t ) < σ ′ ( w ⊤ t +1 x t ) for all t ∈ I 2 . Then, we have

where the second inequality is due to the condition σ ′ ( w ⊤ ∗ x t ) < σ ′ ( w ⊤ t +1 x t ) and the last inequality can be obtained using similar arguments as those used to derive (62).

Regarding term (c), the application of similar arguments used to derive equation (62) demonstrates.

Then, combining the upper bounds for term (a), term (b) and term (c), we have

Resolving the above displayed inequality leads to

In the binary case, β T ( δ ) = O ( √ d log T ) . Then we have Reg T = O ( d log 3 2 T √ T/κ ∗ + κd 2 log 3 T ) , which completes the proof.

C.4.2 Computation Cost for Binary Case

Since the binary logistic bandit is a special case of the MLogB problem, the time complexity analysis in Appendix C.3.4 is also applicable in the binary case. The only difference is that we select the arm as ( x t , ˜ w t ) = arg max x ∈ X , W ∈ C t ( δ ) w ⊤ x in the binary case. As C t ( δ ) ≜ { w ∈ W | ∥ w -w ∗ ∥ H t ≤ β t ( δ ) } is an ellipsoid, the optimization can be rewritten as:

Given H -1 t , the above optimization problem can be solved in O ( d 2 ) time.

C.5 On the Minimax Optimal Bound for the MLogB

In the binary case, we have achieved the minimax dynamic regret bound ˜ O ( √ T/κ ∗ ) up to a logarithmic factor in terms of T . In the MLogB problem, the best-known result is ˜ O ( K √ T ) . A natural question arises: can we obtain a similar minimax optimal result in the MLogB problem? However, we find that it remains challenging both in terms of regret analysis and the design of efficient algorithms.

Challenge in Regret Analysis. To establish the minimax optimal bound for the binary case, as demonstrated in [9, 10] and we restate in (63), a critical step involves showing

In the MLogB problem, in order to obtain a similar result to the minimax optimal bound achieved in the binary case, it is sufficient to demonstrate

However, it is unclear how to prove such a relationship as the binary case. Indeed, denoting by r t = σ ( W ∗ x ∗ ) -σ ( W ∗ x t ) the reward vector and r t,k by its k -th entry, we can further rewrite the k -th entry of the vector ρ ⊤ Σ( W ∗ x t ) can be further written as

In such a case, we can bound the left hand side of (65) by

In the binary case, In the binary case, the additional term becomes -∑ T t =1 r t = -Reg T < 0 , thereby resulting in the derivation of (63). Nevertheless, in the context of the MLogB problem, this particular term has the potential to assume positive values, posing challenges in determining an upper bound for it.

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

20

40

60

80

100

120

140

160

Regret

LogUCB1

OFU-MLogB

OL2M

(a) configuration 4

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

20

40

60

80

100

Regret

LogUCB1

OFU-MLogB

OL2M

(b) configuration 5

0

500

1000 1500 2000 2500 3000 3500 4000

Iteration

0

20

40

60

80

100

120

140

160

Regret

LogUCB1

OFU-MLogB

OL2M

(c) configuration 6

0

500

1000

1500

2000

2500

3000

3500

4000

Algorithms

0

200

400

600

800

1000

Time (seconds)

LogUCB1

OL2M

OFU-MLogB

(d) running time

0

500

1000

1500

2000

2500

3000

Iteration

0

20

40

60

80

100

120

140

160

Regret

MNL-UCB

OFU-MLogB

(a) configuration 4

0

500

1000

1500

2000

2500

3000

Iteration

0

10

20

30

40

50

Regret

MNL-UCB

OFU-MLogB

(b) configuration 5

0

500

1000

1500

2000

2500

3000

Iteration

0

20

40

60

80

100

Regret

MNL-UCB

OFU-MLogB

(c) configuration 6

0

500

1000

1500

2000

2500

3000

Algorithms

0

500

1000

1500

2000

2500

3000

Time (seconds)

MNL-UCB

OFU-MLogB

(d) running time

Challenge in Efficient Algorithm Design. When extending algorithms that achieve minimax optimal bounds from the binary case to the multinomial case, there are computational concerns to address. In the binary case, the minimax optimal regret bound relies on constructing an optimistic reward through ˜ r t ( x ) = arg max w ∈C t ( δ ) σ ( w ⊤ x ) . Since σ ( z ) is an increasing function, the optimization problem can be simplified to

which becomes a convex optimization problem when C t ( δ ) is a convex set. However, when applying this analysis to the multinomial case, solving the problem requires optimizing

Here, the loss function is non-concave, rendering the maximization problem challenging to efficiently solve. Consequently, new approaches for constructing the optimistic reward function may be necessary.

D Omitted Details for Experiments

In this section, we presents more empirical results for other configurations and introduced the parameter configurations for the contenders.

Experimental Details. For each configuration, both the arm set X and the underlying unknown parameter W ∗ are randomly selected. In the binary case, the norm of the unknown parameter is set as ∥ w ∗ ∥ 2 = S with S = 5 . As for the multinomial case, each row of W ∗ are randomly sampled with ∥ w ( k ) ∗ ∥ 2 = ˜ S for all k ∈ [ K ] with ˜ S = 1 . The parameter of all contenders are set according to their order as suggested in the corresponding paper. We use λ = d log( T ) for Log-UCB1, λ = d for OL2M, λ = 1 for ada-OFU-ECOLog and λ = √ KαS for OFU-MLogB. The step size for OFUMLogB is set as η = S/ 2 + ln( K +1) / 4 . The experiments are run on Xeon E-2288G processors (8 cores, 3.7GHz base, 5.0GHz boost).

More Results. Figure 3 provides additional empirical results for the binary logistic bandit problem. These results align with the trends observed in the main paper, wherein our OFU-MLogB algorithm shows performance comparable to ada-OFU-ECOLog but with reduced computational overhead. Meanwhile, more results for the multinomial case are provided by Figure 4. The cumulative running time for MNL-UCB increase at a rate of O ( t 2 ) . In contrast, the running time for our OFU-MLogB exhibits a linear dependence with t , attributable to the constant computation cost per round.

Peer reviews:
comments: SUMMARY:
This paper proposes a computationally efficient algorithm for multinomial logistic bandit with improved regret and computation cost. The algorithm uses online mirror descent and new approximation to efficiently compute consistent estimator and construct optimistic reward. Experimental results show improved regrets and computational costs of the proposed algorithm.

STRENGTHS:
The contributions are clear, concrete, and well-presented. Improving the regret and computation cost and extending binary cases to multinomial is significant. Applying online mirror decent, novel approximation, and novel construction of optimistic reward to reduce the computation cost are novel and significant in contextual bandits under the parametric model.

WEAKNESSES:
1. Discussion and experimental results on whether there is a trade-off to gaining efficiency would be helpful.
2. In experimental results, while the reduction of the computation cost is significant, the reduction of regret seems minor. More experiments are needed to validate the improvement of the proposed algorithm in terms of regret.

QUESTIONS:
1. Is there any trade-offs between the computational efficiency and the accuracy of the estimator?
2. Could you explain why the slope of the regret of MLogB increases in Figure 1 (c)?
3. If the number of iterations $T$ increases, can the proposed algorithm find a better policy to reduce the increment of the regret?

LIMITATIONS:
Yes. This work is mostly theoretical and potential negative societal impact is unseen.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers Multinomial Logistic bandits (the extension of the usual logistic bandit model to a setting where there are more than two possible outcomes - e.g. in advertising one picks an action in response to a context and may observe 'no click'/'click'/'save for later'/etc instead of just 'no click'/'click'). \n\nThis problem has been previously considered in work such as Amani and Thrampoulidis (2021) where an optimistic algorithm with near optimal dependence on T (up to logarithmic factors) has been derived, but the focus of this paper is on algorithms with improved computational efficiency, and optimal dependence on the parameter $\\kappa$. Dependence on this parameter $\\kappa$ has been the focus of a series of recent papers on bandit problems involving the logistic function, as many algorithms with an optimal dependence on $T$ can nonetheless have a poor dependence on $\\kappa$ which in turn brings an exponential dependence (in terms of worst-case performance) on other problem parameters.\n\nThe present paper first improves the MNL-UCB algorithm of Amani and Thrampoulidis by providing a sharper tail-inequality for the MLE under the multinomial model, before providing the main contribution: the OFU-MLogB algorithm, and analysis of its regret and efficiency. The proposed algorithm avoids costly inference by replacing maximum likelihood estimation with online mirror descent, and avoids costly optimisation for the confidence set via an alternative construction of reward. The result is an algorithm with no leading order dependence on $\\kappa$ and $O(1)$ computational cost per round.\n\nSTRENGTHS:\nThis is a welcome contribution for the literature on contextual bandits. Improvements regards to dependence on $\\kappa$ have been important steps forward in this area for various logistic bandit problems in recent years and this development is also likely to be impactful and useful.\n\nI find the paper to be well researched and written. There are a few minor points as discussed below, but I think the paper is generally thoughtfully structured. For instance I liked how the OFU-MLogB algorithm was motivated in terms of certain concerns about efficiency and these same headings were used to frame the discussion of the algorithm's structure. \n\nAs far as I can tell the theoretical results are derived correctly, and the analytical work is sound. I have not had as much time as one would like to be 100% confident in this, so this does impact my confidence score.\n\nWEAKNESSES:\nI don't have too much to criticise in terms of the originality and significance of the paper: its contribution is an improvement upon flawed but still somewhat practically useful algorithms, so it is not a revolutionary one, but I think it is nonetheless meaningful and appropriate for NeurIPS.\n\nI have a few points for clarification regards to the writing, experimental work, and points that I think would benefit from extra detail which I describe below, and if these are adequately addressed I should be able to increase my score following rebuttal.\n\nQUESTIONS:\n- What do you see as the utility of the algorithm presented in Section 2.3 over OFUMlogB? If there is not one particularly, and it is just a way to introduce important concepts and provide a correction to [12] and/or state an improved concentration result, potentially of independent interest, can you make that clearer from the outset?\n\n- I think Theorem 2 is slightly informal: I guess that you mean the sequence of arms selected by the algorithm using decision rule (5) achieves this regret, but how is this algorithm initialised? What is its first selection, or selections until such a point as the parameter estimates are well defined?\n\n- Which MNL-UCB are you testing in the experiments? The original from [12] or the improved one analysed in Theorem 2?\n\n- The confidence bands on the regret presented in the Appendix are very wide. They suggest less of a statistically significant difference than the plots in the main body. I think you need to move a discussion of this to the main text, and offer some discussion as to whether the ordering of the algorithms remains stable across trials, or whether that is random also. I feel that this experiment probably could have been better designed, and would be yet more satisfied if it was possible to provide a more extensive experiment from which more meaningful conclusions could be drawn.\n\n- Minor: at line 167, I don't think you ever really define 'feedback number' as a piece of terminology. I suggest that you should.\n- Minor: at line 255, should say 'condition as Theorem 3'\n- Minor: at line 67, it may be worth making clear that MNL-bandits (which people could confuse this model with) specifically will be addressed in Appendix A, it was something I was concerned was missing until I got to the end.\n- Minor: at line 312: radio -> radius\n\nLIMITATIONS:\nAppropriate for the nature of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a generalization of the (binary) logistic bandit problem to the multinomial setting, significantly improving the state of the art for that problem and even for the special binary.  For the binary logistic bandit problem, many (earlier) algorithms proposed were not optimal; their regret bounds depended on a potentially large problem dependent parameter $\\kappa$ capturing the nonlinearity of the reward function.   Recent works were able to remove regret dependence on that parameter $\\kappa$, though had high per-round computation (depending on the horizon $T$).  \n\nThe authors propose an algorithm that (up to $\\log$ terms) achieves optimal regret bounds even when specialized to the well-studied binary logistic bandit setting with per round constant computation.  Furthermore, compared to prior work for the more general multinomial setting, they not only both obtain better regret (independent of the parameter $\\kappa$) but do so with faster per-round computation (constant instead of $O(T)$).\n\nSTRENGTHS:\n- For the multinomial setting, the proposed algorithm significantly improves on the state-of-the-art both in terms of regret bounds (esp. removing the dependence on $\\kappa$) and computation (from $O(T)$ per round computation to $O(1)$).  In simple experiments, this is backed up with a (slight) reduction in empirical cumulative regret but much faster run-time.\n- Impressively, the proposed algorithm even improves on the state of the art for the (binary) logistic bandit setting.  Among computationally efficient ($O(1)$ per-round complexity) methods it has better regret bounds (also in simple experiments significantly lower empirical cumulative regret with competitive run-time).  Compared to ada-OFU-ECOLog the results are more nuanced (the proposed method has better regret for small horizons but worse regret for large horizons; takes half as long for a horizon of $T=1200$).\n- The algorithm and regret analyses do build on recent advances in (binary) logistic bandits, but non-trivially so.\n- Overall I found the writing clear and well-organized.   Table 1 and the remarks throughout comparing and contrasting to closely related works were helpful.\n\nWEAKNESSES:\n#### Major\nI do not have any major concerns.\n\n#### Minor\n- Appendix A includes discussion on differences between the problem set up of multinomial logistic bandits considered in this paper and the problem of multinomial Logit (MNL) bandits which has been considered in several recent works.  This is only a minor concern, but given close connections of both to binary logistic bandit problems, a slightly more detailed discussion of similarities (if any) with OFU based methods and analyses (like with [33] Agrawal et al.) could be helpful.\n\n\n- There is an experimental design based approach for the binary setting (Mason et al. “An Experimental Design Approach for Regret Minimization in Logistic Bandits” in AAAI 2022) that I don’t think is cited.  I did not look carefully into it regarding time complexity so do not know whether it would be competitive with (ada)-OFU-ECOLog or not. \n\n- For the experiments, the $\\sqrt{T}$ dependence does not kick in for the horizon $T=1200$ (i.e. the regret appears linear).  For Fig 1(a) ada-OFU-ECOLog is initially worse but its regret is visibly concave and catches up around $t=800$.  I would suggest additional experiments with a longer horizon (maybe just with  ada-OFU-ECOLog and OFU-MLogB) to examine how long OFU-MLogB exhibits near-linear regret (and consequently how big that regret gap can grow).   Just to be clear, I think it is impressive that OFU-MLogB is better for smaller $t$ and $t$ on the scale of $10^4$ or $10^5$ might be unrealistic for applications, but the linear regret is a bit concerning.\n\nQUESTIONS:\n#### Minor editing suggestions\n- For Table 1, I’d suggest including a comment in the caption about $\\kappa$ and $\\kappa_*$ (in addition to the later discussion in the main text).\n- In the problem formulation explicitly mention whether or not there are there any assumptions about $\\mathcal{X}$ (finite, convex, etc.).\n- I’d suggest re-ordering lemmas in C.1.2 (proof of lemma 6 depends on lemma 7, proof of lemma 7 depends on lemma 8)\n- For experiments, does MNL-UCB reduce to LogUCB1? If not, I’d (mildly) suggest including it in Figs 1 (a)-(b) for further reference of how well state-of-the-art multinomial methods perform against methods specifically designed for binary setting.\n#### Spelling, grammar \n- Line 52 argmax subscript ‘$x \\in W$’\n- line 276 ‘linearlized’\n- line 289 ‘logistc’\n- line 312 ‘radio’\n- Line 953 ‘trails’\n- line 537 ‘on the an’\n - Line 571 ‘anlaysis’\n- there were more I did not bother to list\n\nLIMITATIONS:\nIt is fine", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has addressed multinomial logistic bandits whose feedback has multiple choices. This paper improves the regret bound in terms of $K$ and reduces the computation cost into constant complexity with respect to $T$. In experiments, the results support that the proposed method is much faster than the prior work.\n\nSTRENGTHS:\n**Computational Complexity**\n\nThis paper reduces the time complexity into $O(K^{3}d^{3})$, which is independent on $T$.\nThe experimental results also support that computational time is dramatically reduced in practice.\n\n**Regret Bound**\n\nThe regret bound of the proposed method is $O(K\\sqrt{T})$. This result is a substantial improvement over the best-known regret bound of $O(K^{5/4}\\sqrt{\\kappa T})$, where $K$ is the number of feedback values and $\\kappa$ is a constant that increases exponentially in terms of the diameter of the parameter domain.\nThe proof seems to be correct.\n\nWEAKNESSES:\n**Lower Bounds**\n\nImproving the regret bound with respect to $K$ seems like a minor contribution. Especially, since there is no comparison with the lower bound on regret, it is unclear if the improvement in the regret bound with respect to K is theoretically significant.\n\n**Novelty of Analysis Techniques**\n\nMost of the techniques used in this paper are quite similar to the generalized linear bandits (GLB).\nEspecially, the proof scheme of the Theorem 1 is almost the same as logistic bandits or GLB.\nFurthermore, in Theorem 1, Lemma $6$ plays a crucial role, however, as mentioned in the Appendix, the difference from previous research lies solely in the bound on the norm of the error.\nThe remaining parts of the proof of Theorem 1 heavily depend on Abbasi-Yadkorietal. (2011).\nIt cannot be considered as a significant theoretical contribution.\n\nQUESTIONS:\n**Questions**\n\n1. What is the memory complexity of each algorithm?\nWhile this paper reduces the algorithm's time complexity, it may lead to requirements for more memory.\nSince there is a general trade-off between time complexity and memory complexity, it would be better to discuss the memory complexity of this algorithm (and others).\n\n2. Does this algorithm match the lower bound?\nIs there any analysis for the lower bound of this setting?\n\n3. What causes the improvement of the regret bound?\nFrom the new analysis scheme? or finding better parameters?\nIf I correctly understand the paper, the improvement of the regret bound is just caused by changing the condition on $\\epsilon$ from $\\| \\epsilon \\| _{2} \\leq \\sqrt{K}$ to $\\|\\epsilon \\| _{1} \\leq 2$.\n\n**Minor comments**\n\nTable 1: the column \"Constant Cost\" is unnecessary part since we can check it in the column \"Cost Per Round\"\n\nline 112: $\\textnormal{diag}(z)$ -> $\\textnormal{diag}(\\sigma(z))$\n\nLIMITATIONS:\nIncreasing the memory complexity might be a limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines multinomial logistic bandits (MLogB), a problem where the learner's action $x_t$ produces feedback $y_t$ with $K+1$ possible outcomes. The probabilities of these outcomes are modeled using a logistic model. In real-world scenarios like online advertising, customers may provide various types of feedback, such as 'buy now', 'add to cart', 'view related item', or simply leave without any click.\n\n The MLogB problem assumes that at each time $t$, the learner selects an action $x_t$ from $\\mathcal{X}$. Each outcome $k\\in[K]$ is associated with a latent parameter $w_*^{(k)}\\in \\mathbb{R}^d$ and the probability of the outcome $P[y_t=k|x_t]$ follows the logistic model, \n$P[y_t|x_t]=exp((w_*^{(k)})^\\top x_t)/(1+\\sum_j exp((w_*^{(j)})^\\top x_t))=\\sigma(z)_k$. \n\nThen the expected reward of the learner's action is defined$ \\sum_t  \\rho^\\top \\sigma (W_* x_t)$ where $\\rho$ is a known rewards vector. \n\nThe authors first introduce an improved version of MNL UCB algorithm [12] with the optimal dependency on K. With an improved concentration set for OFU in Theorem1, MLE based algorithm achieves $O(Kd \\log(T)\\sqrt{\\kappa ST})$ in theorem 2. \n\nIn the next section, they present an efficient and improved algorithm (OFU-MlogB). For the efficiency, they address the computation cost of MLE and OFU methods. Instead of using MLE, they suggest using online mirror descent algorithm to estimate $W$, incurring $O(1)$ cost per round and achieving $\\kappa$=independent confidence set in Theorem 3 (similar to theorem 1).  For the optimistic reward construction, they suggested a novel optimistic reward that can be solved in a constant time per round by introducing bonus terms for exploration in proposition 1.  This method archieves an improved regret bound of $\\tilde{O}(Kd\\sqrt{T})$ in Theorem 4 and constant computation cost compared to $\\tilde{O}(K^{5/6}\\sqrt{\\kappa T})$ regret bound and $O(T)$ computation cost in [12]. In the binary setting where $K=1$ the proposed algorithm archives a minimax regret bound of $O(\\sqrt{T/\\kappa_*})$ with an optimistic rule similar to [9,10] while maintaining lower computation cost. \n\nIn their analysis for theorem 3, they leverage negative terms from the efficient update of mirror descent and a novel intermediary prediction construction designed for multiclass logistic loss.\n\nLastly, they provide experimental results for both binary and multi-class settings. In the binary case, the proposed algorithm demonstrates lower computational overhead while delivering comparable performance to the state-of-the-art algorithm, ada-OFU-ECOLog[10]. For the multinomial setting, the proposed algorithm exhibits improved computational efficiency compared to MNL-UCB [12], while achieving similar performance in terms of regret.\n\nSTRENGTHS:\n1. In the binary case, the proposed algorithm attains the minimax optimal guarantee while reducing the computation cost per round from $O(\\log T)$ to $O(1)$.\n\n2. In the multinomial case, the proposed algorithm achieves a regret bound of $\\tilde{O}(K\\sqrt{T})$ with constant computation cost, surpassing the performance of the best-known algorithm which achieves a regret bound of $\\tilde{O}(K^{5/4}\\sqrt{\\kappa T})$ with $O(T)$ computation cost per round.\n\n3. The authors present empirical evidence demonstrating the improved computation cost of the proposed algorithm.\n\nWEAKNESSES:\n1. Algorithm 1 requires the computation of the matrix inverse and projection, resulting in computational costs of $O(d^2K^3)$ and $O(K^3d^3)$ per round. \n\n2. In the binary case, it appears that achieving optimal regret in Algorithm 1 necessitates the use of a linear optimistic rule (as stated in Corollary 1), rather than relying on an efficient optimistic reward with a bonus term. It is not clear why a linear model works (details are in Question 1).\n\n3. In the experimental evaluation, concerning the binary case, the proposed algorithm exhibits regret that increases linearly, while the previously suggested algorithm, ada-OFU-ECOLog, demonstrates sublinear regret. Regarding the multinomial case, although the proposed algorithm achieves a theoretically superior regret bound considering $K$ and $\\kappa$, it shows similar performance to MNL-UCB. This observation does not provide a clear validation of the theoretical analysis.\n\n4. In the multinomial case, there might exist a difference between the attained regret bound of $\\sqrt{T}$ and a regret lower bound.\n\nQUESTIONS:\n1. In corollary 1, what is the reason the optimistic rule follows a linear model, $\\arg\\max_w w^\\top x$, rather than $\\arg\\max_w \\rho^\\top \\sigma(w^\\top x)$? \n\n2. Is the computation cost for the matrix inverse and projection of the proposed algorithm, which is O(K^3d^3), commonly observed in previous literature like [10]? Otherwise, it appears that the improvement in computation cost mainly applies to scenarios with small values of K and d.\n\n3. In the conducted experiments, why was the time horizon set to $T=1200$, which is relatively smaller compared to the larger time steps typically used in conventional bandit literature?\n\n4. Furthermore, in the experimental results (a), what could be the possible explanation for the observation that the proposed algorithm exhibits linearly increasing regret, while ada-OFU-ECOLog demonstrates sublinear regret?\n\nLIMITATIONS:\n1. The details regarding the method mentioned in Corollary 1 may not be clear (question 1).\n\n2. The experimental results do not seem to provide clear evidence of regret for the proposed algorithm due to linearly increasing patterns and small $T$. (questions 3,4)\n\nI am open to revising my evaluation if these concerns are addressed.\n\nMinor comments:\n\n1.line 99,112: should it be changed from $diag(z)$ to $diag(\\sigma(z))$?\n\n2. line 333 ,336: $O(log1)$-> $O(1)$; $O(T/\\kappa^*)$-> $O(\\sqrt{T/\\kappa^*})$", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-16"}