Title: Nearly Minimax Optimal Regret for Multinomial Logistic Bandit
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandit

Contextual Bandit

Multinomial Logistic Bandit

Minimax Regret
Abstract: In this paper, we study the contextual multinomial logit (MNL) bandit problem in which a learning agent sequentially selects an assortment based on contextual information, and user feedback follows an MNL choice model.
There has been a significant discrepancy between lower and upper regret bounds, particularly regarding the maximum assortment size  $K$. Additionally, the variation in reward structures between these bounds complicates the quest for optimality. Under uniform rewards, where all items have the same expected reward, we establish a regret lower bound of $\Omega(d\sqrt{\smash[b]{T/K}})$ and propose a constant-time algorithm, OFU-MNL+, that achieves a matching upper bound of $\tilde{\mathcal{O}}(d\sqrt{\smash[b]{T/K}})$. 
We also provide instance-dependent minimax regret bounds under uniform rewards.
Under non-uniform rewards, we prove a lower bound of $\Omega(d\sqrt{T})$ and an upper bound of $\tilde{\mathcal{O}}(d\sqrt{T})$, also achievable by OFU-MNL+. Our empirical studies support these theoretical findings. To the best of our knowledge, this is the first work in the contextual MNL bandit literature to prove minimax optimality --- for either uniform or non-uniform reward setting --- and to propose a computationally efficient algorithm that achieves this optimality up to logarithmic factors.

Nearly Minimax Optimal Regret for Multinomial Logistic Bandit

Abstract

In this paper, we study the contextual multinomial logistic (MNL) bandit problem in which a learning agent sequentially selects an assortment based on contextual information, and user feedback follows an MNL choice model. There has been a significant discrepancy between lower and upper regret bounds, particularly regarding the maximum assortment size K . Additionally, the variation in reward structures between these bounds complicates the quest for optimality. Under uniform rewards, where all items have the same expected reward, we establish a regret lower bound of Ω p d ? T { K q and propose a constant-time algorithm, OFU-MNL+ , that achieves a matching upper bound of ˜ O p d ? T { K q . We also provide instancedependent minimax regret bounds under uniform rewards. Under non-uniform rewards, we prove a lower bound of Ω p d ? T q and an upper bound of ˜ O p d ? T q , also achievable by OFU-MNL+ . Our empirical studies support these theoretical findings. To the best of our knowledge, this is the first work in the contextual MNL bandit literature to prove minimax optimality - for either uniform or non-uniform reward setting - and to propose a computationally efficient algorithm that achieves this optimality up to logarithmic factors.

1 Introduction

The multinomial logistic (MNL) bandit framework [47, 48, 7, 8, 40, 41, 44, 5, 53] describes sequential assortment selection problems in which an agent offer a sequence of assortments of at most K item from a set of N possible items and receives feedback only for the chosen decisions. The choice probability of each outcome is characterized by an MNL model [37]. This framework allows modeling of various real-world situations such as recommender systems and online retails, where selections of assortments are evaluated based on the user-choice feedback among offered multiple options.

In this paper, we study the contextual MNL bandit problem [8, 7, 43, 16, 40, 41, 44, 5], where the features of items and possibly contextual information about a user at each round are available. Despite many recent advances, [16, 40, 41, 44, 5], however, no previous studies have proven the minimax optimality of contextual MNL bandits. Chen et al. [16] proposed a regret lower bound of Ω p d ? T { K q , where d is the number of features, T is the total number of rounds, and K is the maximum size of assortments, assuming the uniform rewards, i.e., rewards are all same for each of the total N items. Furthermore, Chen and Wang [15] established a regret lower bound of Ω p ? NT q in the non-contextual setting (hence, dependence on N appears instead of d ), which is tighter in terms of K . It is important to note the difference in the assumptions for the attraction parameter for the outside option v 0 . Chen and Wang [15] assumed for the attraction parameter for the outside option to be v 0 ' K , whereas Chen et al. [16] assumed v 0 ' 1 . Therefore, it remains an open question whether and how the value of v 0 affects both lower and upper bounds of regret .

Regarding regret upper bounds, Chen et al. [16] proposed an exponential runtime algorithm that achieves a regret of ˜ O p d ? T q in the setting with stochastic contexts and the non-uniform rewards. Under the same setting, Oh and Iyengar [41] and Oh and Iyengar [40] introduced polynomial-time algorithms that attain regrets of ˜ O p d ? T { κ q and ˜ O p d 3 { 2 ? T { κ q respectively, where 1 { κ ' O p K 2 q is a problem-dependent constant. Recently, Perivier and Goyal [44] improved the dependency on κ in the adversarial context setting, achieving a regret of ˜ O p dK ? κ 1 T q , where κ 1 ' O p 1 { K q . However, their approach focuses solely on the setting with uniform rewards, which is a special case of non-uniform rewards, and currently, there is no tractable method to implement the algorithm.

As summarized in Table 1, there has been a gap between the upper and lower bounds in the existing works of contextual MNL bandits. No previous studies have confirmed whether lower or upper bounds are tight, obscuring what the optimal regret should be. This ambiguity is further exacerbated because many studies introduce their methods under varying conditions such as different reward structures and values of v 0 , without explicitly explaining how these factors impact regret. Additionally, there is currently no computationally efficient algorithm whose regret does not scale with 1 { κ ' O p K 2 q or directly with K . Intuitively, increasing K provides more information at least in the uniform reward setting, potentially leading to a more statistically efficient learning process. However, no previous results have reflected such intuition. Hence, the following research questions arise:

What is the optimal regret lower bound in contextual MNL bandits?

Can we design a computationally efficient, nearly minimax optimal algorithm under the adversarial context setting?

In this paper, we affirmatively answer the questions by first tackling the contextual MNL bandit problem separately based on the structure of rewards-uniform and non-uniform-and the value of the outside option v 0 . In the setting of uniform rewards, we establish the tightest regret lower bound, explicitly demonstrating the dependence of regret on v 0 . Specifically, we prove a regret lower bound of Ω p d ? T { K q when v 0 ' Θ p 1 q , a common assumption in contextual settings [6, 18, 43, 8, 40, 41, 9, 44, 5, 53, 33] (see Appendix C.1 for more details), and a lower bound of Ω p d ? T q when v 0 ' Θ p K q . Furthermore, in the adversarial context setting, we introduce a computationally efficient and provably optimal (up to logarithmic factors) algorithm, OFU-MNL+ . We prove that our proposed algorithm achieves a regret of ˜ O p d a T { K q when v 0 ' Θ p 1 q and ˜ O p d ? T q when v 0 ' Θ p K q , each of which matches the respective lower bounds that we establish up to logarithmic factors. Furthermore, in the non-uniform reward setting, we provide the optimal lower bound of Ω p d ? T q assuming v 0 ' Θ p 1 q . In the same setting, our proposed algorithm also attains a matching upper bound of ˜ O p d ? T q up to logarithmic factors. Our main contributions are summarized as follows:

Under uniform rewards, we establish a regret lower bound of Ω p ? v 0 K {p v 0 ` K q d ? T q (Theorem 1), which is the tightest known lower bound in contextual MNL bandits. We propose, for the first time, a computationally efficient and provably optimal algorithm, OFU-MNL+ , achieving a matching upper bound of O p ? v 0 K {p v 0 ` K q d ? T q (Theorem 2) up to logarithmic factors, while requiring only a constant computation cost per round. The results indicate that the regret improves as the assortment size K increases, unless v 0 ' Θ p K q . To the best of our knowledge, this is the first study to demonstrate the dependence of regret on the attraction parameter for the outside option v 0 and to highlight the advantages of a larger assortment size K which aligns with intuition. That is, this is the first work to show that a regret upper bound (in either contextual or non-contextual setting) decreases as K increases. Additionally, we provide instance-dependent minimax regret bounds (Proposition 1 and 2), up to logarithmic factors.

Under non-uniform rewards, with setting v 0 ' Θ p 1 q following the convention in contextual MNL bandits [6, 18, 43, 8, 40, 41, 9, 44, 5, 53, 33], we establish a regret lower bound of Ω p d ? T q (Theorem 3). To the best of our knowledge, this is the first and tightest lower bound established under non-uniform rewards. Moreover, OFU-MNL+ also achieves a matching upper bound (up to logarithmic factors) of ˜ O p d ? T q (Theorem 4) in this setting.

We also conduct numerical experiments and show that our algorithm consistently outperforms the existing MNL bandit algorithms while maintaining a constant computation cost per round. Furthermore, the empirical results corroborate our theoretical findings regarding the dependence of regret on the reward structure, v 0 and K .

Overall, our paper addresses the long-standing open problem of closing the gap between upper and lower bounds for contextual MNL bandits. Our proposed algorithm is the first to achieve both provably optimality (up to logarithmic factors) and practicality with improved computation.

2 Related Work

Lower bounds of MNL bandits. In contextual MNL bandits, to the best of our knowledge, only Chen et al. [16] proved a lower bound of Ω p d ? T { K q with the attraction parameter for the outside option set at v 0 ' 1 . However, in the non-contextual setting, there exist improved lower bounds in terms of K . Agrawal et al. [8] demonstrated a lower bound of Ω p a NT { K q , and Chen and Wang [15] established a lower bound of Ω p ? NT q . By setting d ' N , one can derive equivalent lower bounds for the contextual setting, specifically Ω p a dT { K q and Ω p ? dT q , respectively. However, Agrawal et al. [8] and Chen and Wang [15] assumed v 0 ' K when establishing their lower bounds, which differs from the setting used by Chen et al. [16], where v 0 ' 1 . Moreover, to the best of our knowledge, all existing works Chen et al. [16], Agrawal et al. [8], Chen and Wang [15] have established the lower bounds under uniform rewards. Consequently, it remains unclear what the optimal regret is, depending on the value of v 0 and the reward structure.

Upper bounds of contextual MNL bandits. Ou et al. [43] formulated a linear utility model and achieved ˜ O p dK ? T q regret; however, they assumed that utilities are fixed over time. Chen et al. [16] considered contextual MNL bandits with changing and stochastic contexts, establishing a regret of ˜ O p d ? T ` d 2 K 2 q . However, they encountered computational issues due to the need to enumerate all possible ( N choose K ) assortments. To address this, Oh and Iyengar [41] proposed a polynomial-time assortment optimization algorithm, which maintains the confidence bounds in the parameter space and then calculates the upper confidence bounds of attraction parameter for each item, achieving a regret of ˜ O p d ? T { κ q , where 1 { κ ' O p K 2 q is a problem-dependent constant. Perivier and Goyal [44] considered the adversarial context and uniform reward setting and improved the dependency on κ to ˜ O p dK ? κ 1 T ` d 2 K 4 { κ q , where κ 1 ' O p 1 { K q . However, their algorithm is intractable. Agrawal et al. [5] considered a uniform rewards setting (with v 0 ' 1 ) and achieved a regret of ˜ O p d ? T q . However, due to significant technical errors in their paper (refer Appendix L), we do not include a comparison with their results in this work.

Recently, Zhang and Sugiyama [53] utilized an online parameter update to construct a constant time algorithm. However, they consider a multiple-parameter choice model in which the learner estimates K parameters and shares the contextual information x t across the items in the assortment. This model

differs from ours; we use a single-parameter choice model with varying the context for each item in the assortment. Additionally, they make a stronger assumption regarding the reward than we do (see Assumption 1). Moreover, while they fix the assortment size at K , we allow it to be smaller than or equal to K . On the other hand, Zhang and Luo [52] considered a general function approximation, achieving a regret bound of ˜ O p K 2 . 5 ? dNT q . However, this bound scales with K and N , and the proposed algorithm is not tractable. To the best of our knowledge, all existing methods fail to show that the regret upper bound can improve as the assortment size K increases.

3 Problem Setting

Notations. For a positive integer, n , we denote r n s : ' t 1 , 2 , . . . , n u . For a real-valued matrix A , we denote } A } 2 : ' sup x : } x } 2 ' 1 } Ax } 2 as the maximum singular value of A . For two symmetric matrices, V and W of the same dimensions, V ľ W means that V ´ W is positive semi-definite. Finally, we define S to be the set of candidate assortment with size constraint at most K , i.e., S ' t S Ď r N s : | S | ď K u . While, for simplicity, we consider both S and the set of items r N s to be stationary in this paper, it is important to note that both S and r N s can vary over time.

Contextual MNL bandits. We consider a sequential assortment selection problem which is defined as follows. At each round t , the agent observes feature vectors x ti P R d for every item i P r N s . Based on this contextual information, the agent presents an assortment S t ' t i 1 , . . . , i l u P S , where l ď K , and then observes the user purchase decision c t P S t Yt 0 u , where t 0 u represents the 'outside option' which indicates that the user did not select any of the items in S t . The distribution of these selections follows a multinomial logistic (MNL) choice model [37], where the probability of choosing any item i k P S t (or the outside option) is defined as:

where v 0 is a known attraction parameter for the outside option and w ‹ P R d is an unknown parameter.

Remark 1. In the existing literature on MNL bandits, it is commonly assumed that v 0 ' 1 [40, 41, 44, 5, 53]. On the other hand, Chen and Wang [15], Agrawal et al. [8] assume that v 0 ' K 1 to induce a tighter lower bound in terms of K . Later, we will explore how these differing assumptions create fundamentally different problems, leading to different regret lower bounds (Subsection 5.1).

The choice response for each item i P S t Yt 0 u is defined as y ti : ' ✶ p c t ' i q P t 0 , 1 u . Hence, the choice feedback variable y t : ' p y t 0 , y ti 1 , . . . y ti l q is sampled from the following multinomial (MNL) distribution: y t ' MNL t 1 , p p t p 0 | S t , w ‹ q , . . . , p t p i l | S t , w ‹ qqu , where the parameter 1 indicates that y t is a single-trial sample, i.e., y t 0 ` ř l k ' 1 y ti k ' 1 . For each i P S t Yt 0 u , we define the noise ϵ ti : ' y ti ´ p t p i | S t , w ‹ q . Since each ϵ ti is a bounded random variable in r 0 , 1 s , ϵ ti is 1 { 4 -subGaussian. At every round t , the reward r ti for each item i is also given. Then, we define the expected revenue of the assortment S as

and define S ‹ t as the offline optimal assortment at time t when w ‹ is known a prior, i.e., S ‹ t ' argmax S P S R t p S, w ‹ q . Our objective is to minimize the cumulative regret over the T periods:

When K ' 1 , r t 1 ' 1 , and v 0 ' 1 , the MNL bandit recovers the binary logistic bandit with R t p S ' t x u , w ‹ q ' σ ` x J w ‹ ˘ ' 1 {p 1 ` exp p´ x J w ‹ qq , where σ p¨q is the sigmoid function.

Consistent with previous works on MNL bandits [41, 44, 5, 53], we make the following assumptions:

Assumption 1 (Bounded assumption) . We assume that } w ‹ } 2 ď 1 , and for all t ě 1 , i P r N s , } x ti } 2 ď 1 and r ti P r 0 , 1 s .

Assumption 2 (Problem-dependent constant) . There exist κ ą 0 such that for every item i P S and any S P S , and all round t , min w P W p t p i | S, w q p t p 0 | S, w q ě κ , where W ' t w P R d | } w } 2 ď 1 u .

In Assumption 1, we assume that the reward for each item i is bounded by a constant, allowing the norm of the reward vector to depend on K , e.g., } ρ t } 2 ď ? K . In contrast, Zhang and Sugiyama [53] assume that the norm of the reward vector ρ t ' r r t 1 , . . . r t | S t | s J P R | S t | is bounded by a constant, independent of K , e.g., } ρ t } 2 ď 1 . Thus, our assumption regarding rewards is weaker than theirs.

Assumption 2 is common in contextual MNL bandits [16, 41, 44, 53]. Note that 1 { κ depends on the maximum size of the assortment K , i.e., 1 { κ ' O p K 2 q . One of the primary goals of this paper is to show that as the assortment size K increases, we can achieve an improved (or at least not worsened) regret bound. To this end, we design a dynamic assortment policy that enjoys improved dependence on κ . Note that our algorithm does not need to know κ a priori, whereas Oh and Iyengar [40, 41] do.

4 Existing Gap between Upper and Lower Bounds in MNL Bandits

The primary objective of this paper is to establish minimax regrets in contextual MNL bandits. To explore the optimality of regret, we analyze how it depends on the attraction parameter for the outside option v 0 , the maximum assortment size K , and the structure of rewards.

?

Dependence on v 0 . Currently, the established lower bounds are Ω p d T { K q by Chen et al. [16], Ω p ? dT { K q by the contextual version of Agrawal et al. [8], and Ω p ? dT q , which is the tightest in terms of K , by the contextual version of Chen and Wang [15]. These results can be misleading, as many subsequent studies [41, 39, 17, 52] have claimed that a K -independent regret is achievable, without clearly addressing the influence of the value of v 0 . In fact, the improved regret bounds (in terms of K ) obtained by Agrawal et al. [8] and Chen and Wang [15] were possible when v 0 ' K . However, in the contextual setting, it is more common to set v 0 ' Θ p 1 q . This is because, given the context for the outside option x t 0 , it is straightforward to construct an equivalent choice model where v 0 ' Θ p 1 q (refer Appendix C.1). In this paper, we rigorously show the regret dependency on the value of v 0 . In Theorem 1, we establish a regret lower bound of Ω ` ? v 0 K {p v 0 ` K q d ? T ˘ , which implies that the value of v 0 , indeed, affects the regret. Then, in Theorem 2, we show that our proposed computationally efficient algorithm, OFU-MNL+ achieves a regret of ˜ O ` ? v 0 K {p v 0 ` K q d ? T ˘ , which is minimax optimal up to logarithmic factors in terms of all d, T, K and even v 0 .

Dependence on K &Uniform/Non-uniform rewards. To the best of our knowledge, the regret bound in all existing works in contextual MNL bandits does not decrease as the assortment size K increases [16, 40, 41, 44]. However, intuitively, as the assortment size increases, we can gain more information because we receive more feedback. Therefore, it makes sense that regret could be reduced as K increases, at least in the uniform reward setting. Under uniform rewards, the expected revenue (to be specified later) increases as more items are added in the assortment. Consequently, both the optimistically chosen assortment and the optimal assortment always have a size of K . Thus, the agent obtain information about exactly K items in each round. This phenomenon is also demonstrated empirically in Figure 1. In the uniform reward setting, as K increases, the cumulative regrets of not only our proposed algorithm but also other baseline algorithms decrease. This indicates that the existing regret bounds are not tight enough in terms of K . Conversely, in the non-uniform reward setting, the sizes of both the optimistically chosen assortment and the optimal assortment can be less than K , so performance improvement is not guaranteed. In this paper, we show that the regret dependence on K varies by case: uniform and non-uniform rewards. When v 0 ' Θ p 1 q , we obtain a regret lower bound of Ω p d ? T { K q (Theorem 1) and a regret upper bound of ˜ O p d ? T { K q (Theorem 2) under uniform rewards. Additionally, we achieve a regret lower bound of (Theorem 3) and a regret upper bound of ˜ O p d ? T q (Theorem 4) under non-uniform rewards.

5 Algorithms and Main Results

In this section, we begin by proving the tightest regret lower bound under uniform rewards (Subsection 5.1), explicitly showing the dependence on the attraction parameter for the outside option v 0 . We then introduce OFU-MNL+ , an algorithm that achieves minimax optimality, up to logarithmic factors, under uniform rewards (Subsection 5.2). Notably, OFU-MNL+ is designed for efficiency, requiring

only constant computation and storage costs. Finally, we establish the tightest regret lower bound and a matching minimax optimal regret upper bound under non-uniform rewards (Subsection 5.3).

5.1 Regret Lower Bound under Uniform Rewards

In this subsection, we present a lower bound for the worst-case expected regret in the uniform reward setting ( r ti ' 1 ). This covers all applications where the objective is to maximize the appropriate 'click-through rate' by offering the assortment.

Theorem 1 (Regret lower bound, Uniform rewards) . Let d be divisible by 4 and let Assumption 1 hold true. Suppose T ě C ¨ d 4 p v 0 ` K q{ K for some constant C ą 0 . Then, in the uniform reward setting, for any policy π , there exists a worst-case problem instance with N ' Θ p K ¨ 2 d q items such that the worst-case expected regret of π is lower bounded as follows:

Discussion of Theorem 1. If v 0 ' Θ p 1 q , Theorem 1 demonstrates a regret lower bound of Ω p d ? T { K q . This indicates that, under uniform rewards, increasing the assortment size K leads to an improvement in regret. Compared to the lower bound Ω p d ? T { K q proposed by Chen et al. [16], our lower bound is improved by a factor of ? K . This improvement is mainly due to the establishment of a tighter upper bound for the KL divergence (Lemma D.2). Notably, Chen et al. [16] also considered uniform rewards with v 0 ' Θ p 1 q . On the other hand, Chen and Wang [15] and Agrawal et al. [8] established regret lower bounds of Ω p ? NT q and Ω p a NT { K q , respectively, in non-contextual MNL bandits with uniform rewards, by setting v 0 ' K to achieve these regrets. Theorem 1 shows that if v 0 ' Θ p K q , we can obtain a regret lower bound of Ω p d ? T q , which is consistent with the K -independent regret in Chen and Wang [15]. To the best of our knowledge, this result is the first to explicitly show the dependency of regret on the attraction parameter for the outside option v 0 . The proof is deferred to Appendix D.

5.2 Minimax Optimal Regret Upper Bound under Uniform Rewards

In this subsection, we propose a new algorithm OFU-MNL+ , which enjoys minimax optimal regret up to logarithmic factors in the case of uniform rewards. Note that, since the revenue is an increasing function when rewards are uniform, maximizing the expected revenue R t p S, w q over all S P S always yields exactly K items, i.e., | S t | ' | S ‹ t | ' K .

Our first step involves constructing the confidence set for the online parameter.

Online parameter estimation. Instead of performing MLE as in previous works [16, 41, 44], inspired by Zhang and Sugiyama [53], we use the online mirror descent algorithm to estimate parameter. We first define the multinomial logistic loss function at round t as:

In Proposition C.1, we will show that the loss function has the constant parameter self-concordant-like property. We estimate the true parameter w ‹ as follows:

where η ą 0 is the step-size parameter to be specified later, and ˜ H t : ' H t ` η G t p w t q , where

and H t : ' λ I d ` ř t ´ 1 s ' 1 G s p w s ` 1 q . Note that G t p w q ' ∇ 2 ℓ t p w q . This online estimator is efficient in terms of both computation and storage. By a standard online mirror descent formulation [42], the optimization problem in (3) can be solved using a single projected gradient step through the following equivalent formula:

Algorithm 1 OFU-MNL+

1: Inputs: regularization parameter λ , probability δ , confidence radius β t p δ q , step size η .

2: Initialize: H 1 ' λ I d and w 1 as any point in W .

3: for round t ' 1 , 2 , ¨ ¨ ¨ , T do

4: Compute α ti ' x J ti w t ` β t p δ q} x ti } H ´ 1 t for all i P r N s .

6: Update ˜ H t ' H t ` η G t p w t q , and update the estimator w t ` 1 by (3).

5: Offer S t ' argmax S P S ˜ R t p S q and observe y t .

7: Update H t ` 1 ' H t ` G t p w t ` 1 q .

8: end for

which enjoys a computational cost of only O p Kd 3 q , completely independent of t [38, 53]. Regarding storage costs, the estimator does not need to store all historical data because both ˜ H t and H t can be updated incrementally, requiring only O p d 2 q storage.

Furthermore, the estimator allows for a κ -independent confidence set, leading to an improved regret.

Lemma 1 (Online parameter confidence set) . Let δ P p 0 , 1 s . Under Assumption 1, with η ' 1 2 log p K ` 1 q ` 2 and λ ' 84 ? 2 dη , we define the following confidence set

Armed with the online estimator, we construct the computationally efficient optimistic revenue.

Computationally efficient optimistic expected revenue. To balance the exploration and exploitation trade-off, we use the upper confidence bounds (UCB) technique, which have been widely studied in many bandit problems, including K -arm bandits [11, 32] and linear bandits [1, 20].

At each time t , given the confidence set in Lemma 1, we first calculate the optimistic utility α ti as:

The optimistic utility α ti is composed of two parts: the mean utility estimate x J ti w t and the standard deviation β t p δ q} x ti } H ´ 1 t . In the proof of the regret upper bound, we show that α ti serves as an upper bound for x J ti w ‹ , assuming that the true parameter w ‹ falls within the confidence set C t p δ q . Based on α ti , we construct the optimistic expected revenue for the assortment S , defined as follows:

where r ti ' 1 . Then, we offer the set S t that maximizes the optimistic expected revenue, S t ' argmax S P S ˜ R t p S q . Given our assumption that all rewards are of unit value, the optimization problem is equivalent to selecting the K items with the highest optimistic utility α ti . Consequently, solving the optimization problem incurs a constant computational cost of O p N q .

Remark 2 (Comparison to Zhang and Sugiyama [53]) . In Zhang and Sugiyama [53], the MNL choice model is outlined with a shared context x t and distinct parameters w ‹ 1 , . . . , w ‹ K for each choice. Conversely, our model employs a single parameter w ‹ across all choices and has varying contexts for each item in the assortment S , x t 1 , . . . x ti | S | . Due to this discrepancy in the choice model, directly applying Proposition 1 from Zhang and Sugiyama [53], which constructs the optimistic revenue by adding bonus terms to the estimated revenue, incurs an exponential computational cost in our problem setting. This complexity arises because the optimistic revenue must be calculated for every possible assortment S P S ; therefore, it is necessary to enumerate all potential assortments ( N choose K ) to identify the one that maximizes the optimistic revenue As a result, extending the approach in Zhang and Sugiyama [53] to our setting is non-trivial, requiring a different analysis.

We now present the regret upper bound of OFU-MNL+ in the uniform reward setting.

Theorem 2 (Regret upper bound of OFU-MNL+ , Uniform rewards) . Let δ P p 0 , 1 s and Assumptions 1 and 2 hold. In the uniform reward setting, by setting η ' 1 2 log p K ` 1 q ` 2 and λ ' 84 ? 2 dη , with

probability at least 1 ´ δ , the cumulative regret of OFU-MNL+ is upper-bounded by

Discussion of Theorem 2. If T ě O p d 2 p v 0 ` K q 2 {p κ 2 v 0 K qq , Theorem 2 shows that our algorithm OFU-MNL+ achieves minimax optimal regret (up to logarithmic factor) in terms of all d , T , K , and even v 0 . To the best of our knowledge, ignoring logarithmic factors, our proposed algorithm is the first computationally efficient, minimax optimal algorithm in (adversarial) contextual MNL bandits. When v 0 ' Θ p 1 q , which is the convention in existing MNL bandit literature [40, 41, 44, 5, 53], OFU-MNL+ obtains ˜ O p d a T { K q regret. This represents an improvement over the previous upper bound of Perivier and Goyal [44] 2 , which is ˜ O p dK ? κ 1 T ` d 2 K 4 { κ q , where κ 1 ' O p 1 { K q , by a factor of K . This improvement can be attributed to two key factors: an improved, constant, self-concordant-like property of the loss function (Proposition C.1) and a K -free elliptical potential lemma (Lemma E.2). Furthermore, by employing an improved bound for the second derivative of the revenue (Lemma E.3), we achieve an enhancement in the regret for the second term, d 2 { κ , by a factor of K 4 , in comparison to Perivier and Goyal [44]. Unless v 0 ' Θ p K q , Theorem 2 indicates that the regret decreases as the assortment size K increases. To the best of our knowledge, this is the first algorithm in MNL bandits to show that increasing K results in a reduction in regret. Moreover, when reduced to the logistic bandit, i.e., K ' 1 , r t 1 ' 1 , and v 0 ' 1 , our algorithm can also achieve a regret of ˜ O p d ? κT q by Corollary 1 in Zhang and Sugiyama [53], which is consistent with the results in Abeille et al. [4], Faury et al. [23]. The proof is deferred to Appendix E.

Remark 3 (Efficiency of OFU-MNL+ ) . The proposed algorithm is computationally efficient in both parameter updates and assortment selections. Since we employ online parameter estimation, akin to Zhang and Sugiyama [53], our algorithm demonstrates computational efficiency in parameter estimation, incurring only incurring O p Kd 3 q computation cost and O p d 2 q storage cost, which are completely independent of t . Furthermore, a naive approach to selecting the optimistic assortment requires enumerating all possible ( N choose K ) assortments, resulting in exponential computational cost [16]. However, by constructing the optimistic expected revenue according to (6) (inspired by Oh and Iyengar [41]), our algorithm needs only O p N q computational cost.

5.3 Regret Upper & Lower Bounds under Non-Uniform Rewards

In this subsection, we propose regret upper and lower bounds in the non-uniform reward setting. In this scenario, the sizes of both the chosen assortment S t , and the optimal assortment, S ‹ t are not fixed at K . Therefore, we cannot guarantee an improvement in regret even as K increases.

We first prove the regret lower bound in the non-uniform reward setting.

Theorem 3 (Regret lower bound, Non-uniform rewards) . Under the same conditions as Theorem 1, let the rewards be non-uniform and v 0 ' Θ p 1 q . Then, for any policy π , there exists a worst-case problem instance such that the worst-case expected regret of π is lower bounded as follows:

´

¯

Discussion of Theorem 3. In contrast to Theorem 1, which considers uniform rewards, the regret lower bound is independent of the assortment size K . Note that Theorem 3 does not claim that non-uniform rewards inherently make the problem more difficult. Rather, it implies that there exists an instance with adversarial non-uniform rewards, where regret does not improve even with an increase in K . Moreover, the assumption that v 0 ' Θ p 1 q is common in the existing literature on contextual MNL bandits [40, 41, 44, 5, 53] (refer Appendix C.1). To the best of our knowledge, this is the first established lower bound for non-uniform rewards in MNL bandits, even in the non-contextual setting. The proof is deferred to Appendix G.

We also prove a matching upper bound up to logarithmic factors. The algorithm OFU-MNL+ is also applicable in the case of non-uniform rewards. However, because the optimistic expected revenue ˜ R t p S q is no longer an increasing function of α ti , optimizing for S t ' argmax S P S ˜ R t p S q no longer equates to simply selecting the top K items with the highest optimistic utility. Instead, we employ

assortment optimization methods introduced in Rusmevichientong et al. [47], Davis et al. [21], which are efficient polynomial-time (independent of t ) 3 algorithms available for solving this optimization problem. Therefore, our algorithm is also computationally efficient under non-uniform rewards.

Theorem 4 (Regret upper bound of OFU-MNL+ , Non-uniform rewards) . Under the same assumptions and parameter settings as Theorem 2, if the rewards are non-uniform and v 0 ' Θ p 1 q , then with a probability at least 1 ´ δ , the cumulative regret of OFU-MNL+ is upper-bounded by

Discussion of Theorem 4. If T ě O p d 2 { κ 2 q , our algorithm achieves a regret of ˜ O p d ? T q when the reward for each item is non-uniform, demonstrating that OFU-MNL+ is minimax optimal up to a logarithmic factor. Recall that we relax the bounded assumption on the reward compared to Zhang and Sugiyama [53] (refer Assumption 1); thus, we allow the sum of the squared rewards in the assortment to scale with K . Consequently, we need a novel approach to achieve the regret that does not scale with K . To this end, we centralize the features, i.e., ˜ x ti ' x ti ´ E j ' p t p¨| S t , w t ` 1 q r x tj s , and propose a novel elliptical potential lemma for them, as detailed in Lemma H.3. Note that our algorithm is capable of achieving 1 { κ -free regret (in the leading term) under both uniform and non-uniform rewards. In contrast, the algorithm in Perivier and Goyal [44] is limited to achieving this only in the uniform reward setting. Furthermore, compared to the regret bound in Chen et al. [16], which is ˜ O p d ? T q , our regret bounds has the same order of regret with theirs. However, their algorithm is computationally intractable as it requires enumerating all possible assortments, whereas our algorithm incurs only a constant computational cost per round. The proof is deferred to Appendix H.

6 Instance-Dependent Bounds

In this section, we show that instance-dependent upper and lower bounds are also achievable under uniform rewards. We define the degree of non-linearity for the optimal assortment S ‹ t at round t under the true parameter w ‹ as κ ‹ t : ' ř i P S ‹ t p t p i | S ‹ t , w ‹ q p t p 0 | S ‹ t , w ‹ q . We first establish the instance-dependent lower bound under uniform rewards.

Proposition 1 (Instance-dependent regret lower bound, Uniform rewards) . Under the same conditions as Theorem 1, for any policy π and for T ě d 2 { κ , there exists a worst-case problem instance such that the worst-case expected regret of π is lower bounded as follows:

The proof is deferred to Appendix I. We also provide the matching upper bound.

Proposition 2 (Instance-dependent regret upper bound of OFU-MNL+ , Uniform rewards) . Under the same assumptions, parameter settings, and reward structure as Theorem 2, with a probability at least 1 ´ δ , the cumulative regret of OFU-MNL+ is upper-bounded by

The proof is deferred to Appendix J. For sufficiently large T , the regret upper bound (Proposition 2) matches the regret lower bound (Proposition 1), up to logarithmic factor. To the best of our knowledge, these are the first minimax instance-dependent regret bounds under uniform rewards. Note that, in the worst case, κ ‹ t ' ˜ O p ? v 0 K {p v 0 ` K qq , which indicates that these results provide a strict improvement over the worst-case bounds given in Theorems 1 and 2.

Some readers may expect instance-dependent regret bounds for non-uniform rewards as well. Unfortunately, we were unable to establish these. Recall that κ ‹ t represents the degree of non-linearity for

0

500

1000

1500

2000

2500

3000

Round (t)

0

20

40

60

80

100

Cumulative Regret

N=100, K=5, d=5, Uniform R

UCB-MNL

TS-MNL

0

500

1000

1500

2000

2500

3000

Round (t)

0

20

40

60

80

100

Cumulative Regret

N=100, K=10, d=5, Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0

20

40

60

80

100

Cumulative Regret

N=100, K=15, d=5, Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.01

0.02

0.03

0.04

Time (Seconds)

N=100, K=15, d=5, Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0

10

20

30

40

50

Cumulative Regret

N=100, K=5, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0

10

20

30

40

50

Cumulative Regret

N=100, K=10, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0

10

20

30

40

50

Cumulative Regret

N=100, K=15, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.15

0.30

0.45

0.60

Time (Seconds)

N=100, K=15, d=5, Non-Uniform R

the optimal assortment S ‹ t . However, in the proofs for regret bounds, we encounter terms associated with the chosen assortment S t , such as ř i P S ‹ t p t p i | S t , w ‹ q p t p 0 | S t , w ‹ q . To address this, we use the mean value theorem-based analysis (Lemma I.5) to replace this quantity with κ ‹ t . Under non-uniform rewards, however, the mean value theorem does not apply because the sizes and rewards of S ‹ t and S t may differ. Addressing this problem would be an interesting direction for future research.

7 Numerical Experiments

In this section, we empirically evaluate the performance of our algorithm OFU-MNL+ . We measure cumulative regret over T ' 3000 rounds. For each experimental setup, we run the algorithms across 20 independent instances and report the average performance. In each instance, the underlying parameter w ‹ is randomly sampled from a d -dimensional uniform distribution, where each element of w ‹ lies within the range r´ 1 { ? d, 1 { ? d s and is not known to the algorithms. Additionally, the context features x ti are drawn from a d -dimensional multivariate Gaussian distribution, with each element of x ti clipped to the range r´ 1 { ? d, 1 { ? d s . This setup ensures compliance with Assumption 1. In the uniform reward setting (first row of Figure 1), the combinatorial optimization step to choose the assortment reduces to sorting items by their utility estimate. In the non-uniform reward setting (second row of Figure 1), the rewards are sampled from a uniform distribution in each round, i.e., r ti ' Unif p 0 , 1 q . Refer Appendix K for more details.

We compare the performance of OFU-MNL+ with those of the practical and state-of-the-art algorithms: the Upper Confidence Bound-based algorithm, UCB-MNL [40], and the Thompson Sampling-based algorithm, TS-MNL [40]. Figure 1 demonstrates that our algorithm significantly outperforms other baseline algorithms. In the uniform reward setting, as K increases, the cumulative regrets of all algorithms tend to decrease. In contrast, this trend is not observed in the non-uniform reward setting. Furthermore, the results also show that our algorithm maintains a constant computation cost per round, while the other algorithms exhibit a linear dependence on t . In Appendix K, we present the additional runtime curves (Figure K.1) as well as the regret curves of the other configuration where v 0 ' Θ p K q (Figure K.2). All of these empirical results align with our theoretical results.

8 Conclusion

In this paper, we propose minimax optimal lower and upper bounds for both uniform and non-uniform reward settings. We propose a computationally efficient algorithm, OFU-MNL+ , that achieves a regret of ˜ O p d ? T { K q under uniform rewards and ˜ O p d ? T q under non-uniform rewards. We also prove matching lower bounds of Ω p d ? T { K q and Ω p d ? T q for each setting, respectively. Moreover, our empirical results support our theoretical findings, demonstrating that OFU-MNL+ is not only provably but also experimentally efficient.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

Naoki Abe and Philip M Long. Associative reinforcement learning using linear probabilistic concepts. In ICML , pages 3-11. Citeseer, 1999.

Marc Abeille and Alessandro Lazaric. Linear Thompson Sampling Revisited. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 176-184. PMLR, 20-22 Apr 2017.

Marc Abeille, Louis Faury, and Clément Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 3691-3699. PMLR, 2021.

Priyank Agrawal, Theja Tulabandhula, and Vashist Avadhanula. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research , 310(2):737-750, 2023.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , pages 127-135. PMLR, 2013.

Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Thompson sampling for the mnl-bandit. In Conference on learning theory , pages 76-78. PMLR, 2017.

Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research , 67(5):1453-1485, 2019.

Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. Advances in Neural Information Processing Systems , 34:2913-2924, 2021.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Nicolo Campolongo and Francesco Orabona. Temporal variability in implicit online learning. Advances in neural information processing systems , 33:12377-12387, 2020.

Junyu Cao and Wei Sun. Tiered assortment: Optimization and online learning. Management Science , 70(8):5481-5501, 2024.

Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. In International conference on machine learning , pages 151-159. PMLR, 2013.

Xi Chen and Yining Wang. A note on a tight lower bound for capacitated mnl-bandit assortment selection models. Operations Research Letters , 46(5):534-537, 2018.

Xi Chen, Yining Wang, and Yuan Zhou. Dynamic assortment optimization with changing contextual information. The Journal of Machine Learning Research , 21(1):8918-8961, 2020.

Xi Chen, Yining Wang, and Yuan Zhou. Optimal policy for dynamic assortment planning under multinomial logit models. Mathematics of Operations Research , 46(4):1639-1657, 2021.

Wang Chi Cheung and David Simchi-Levi. Thompson sampling for online personalized assortment optimization problems with multinomial logit choice models. Available at SSRN 3075658 , 2017.

Hyun-jun Choi, Rajan Udwani, and Min-hwan Oh. Cascading contextual assortment bandits. Advances in Neural Information Processing Systems , 36, 2024.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

James M Davis, Guillermo Gallego, and Huseyin Topaloglu. Assortment optimization under variants of the nested logit model. Operations Research , 62(2):250-273, 2014.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning , pages 3052-3060. PMLR, 2020.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 546-580. PMLR, 2022.

Sarah Filippi, Olivier Cappé, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1 , NIPS'10, page 586-594, Red Hook, NY, USA, 2010. Curran Associates Inc.

Dylan J Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Conference on learning theory , pages 167-208. PMLR, 2018.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30, 2017.

Abbas Kazerouni and Lawrence M Wein. Best arm identification in generalized linear bandits. Operations Research Letters , 49(3):365-371, 2021.

Wonyoung Kim, Kyungbok Lee, and Myunghee Cho Paik. Double doubly robust thompson sampling for generalized linear contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8300-8307, 2023.

Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Tight regret bounds for stochastic combinatorial semi-bandits. In Artificial Intelligence and Statistics , pages 535-543. PMLR, 2015.

Branislav Kveton, Manzil Zaheer, Csaba Szepesvari, Lihong Li, Mohammad Ghavamzadeh, and Craig Boutilier. Randomized exploration in generalized linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 2066-2076. PMLR, 2020.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Junghyun Lee, Se-Young Yun, and Kwang-Sung Jun. Improved regret bounds of (multinomial) logistic bandits via regret-to-confidence-set conversion. In International Conference on Artificial Intelligence and Statistics , pages 4474-4482. PMLR, 2024.

Junghyun Lee, Se-Young Yun, and Kwang-Sung Jun. A unified confidence sequence for generalized linear models, with applications to bandits. arXiv preprint arXiv:2407.13977 , 2024.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Xutong Liu, Jinhang Zuo, Siwei Wang, John CS Lui, Mohammad Hajiesmaili, Adam Wierman, and Wei Chen. Contextual combinatorial bandits with probabilistically triggered arms. In International Conference on Machine Learning , pages 22559-22593. PMLR, 2023.

Daniel McFadden. Modelling the choice of residential location. 1977.

Zakaria Mhammedi, Wouter M Koolen, and Tim Van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Conference on Learning Theory , pages 2490-2511. PMLR, 2019.

Sentao Miao and Xiuli Chao. Dynamic joint assortment and pricing optimization with demand learning. Manufacturing & Service Operations Management , 23(2):525-545, 2021.

Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. Advances in Neural Information Processing Systems , 32, 2019.

Min-hwan Oh and Garud Iyengar. Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9205-9213, 2021.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Mingdong Ou, Nan Li, Shenghuo Zhu, and Rong Jin. Multinomial logit bandit with linear utility functions. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 , pages 2602-2608. International Joint Conferences on Artificial Intelligence Organization, 2018.

Noemie Perivier and Vineet Goyal. Dynamic pricing and assortment under a contextual mnl demand. Advances in Neural Information Processing Systems , 35:3461-3474, 2022.

Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its application on diversified online recommendation. In Proceedings of the 2014 SIAM International Conference on Data Mining , pages 461-469. SIAM, 2014.

Idan Rejwan and Yishay Mansour. Topk combinatorial bandits with full-bandit feedback. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117 of Proceedings of Machine Learning Research , pages 752-776. PMLR, 08 Feb-11 Feb 2020.

Paat Rusmevichientong, Zuo-Jun Max Shen, and David B Shmoys. Dynamic assortment optimization with a multinomial logit choice model and capacity constraint. Operations research , 58(6):1666-1680, 2010.

Denis Sauré and Assaf Zeevi. Optimal dynamic assortment planning with demand learning. Manufacturing & Service Operations Management , 15(3):387-404, 2013.

Quoc Tran-Dinh, Yen-Huan Li, and Volkan Cevher. Composite convex minimization involving self-concordant-like cost functions. In Modelling, Computation and Optimization in Information Systems and Management Sciences: Proceedings of the 3rd International Conference on Modelling, Computation and Optimization in Information Systems and Management SciencesMCO 2015-Part I , pages 155-168. Springer, 2015.

Alexandre B Tsybakov. Introduction to Nonparametric Estimation . Springer Science & Business Media, 2008. doi: https://doi.org/10.1007/b13794.

Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In International Conference on Machine Learning , pages 392-401. PMLR, 2016.

Mengxiao Zhang and Haipeng Luo. Contextual multinomial logit bandits with general value functions. arXiv preprint arXiv:2402.08126 , 2024.

Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Improved regret and constant computation cost. Advances in Neural Information Processing Systems , 36, 2024.

Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence , page 835-844. AUAI Press, 2016.

Appendix

Table of Contents

A Further Related Work

In this section, we discuss additional related works that complement Section 2. For simplicity, we consider only the dependence on the number of rounds t for a computation cost in bigO notation.

Logistic Bandits. The logistic bandit model [24, 22, 4, 23] focuses on environments with binary rewards and explores the impact of non-linearity on the exploration-exploitation trade-off for parametrized bandits. The main research interest has been the algorithms' dependence on the degree of non-linearity κ , which can grow exponentially in terms of the diameter of the decision domain W . Zhang et al. [51] introduced the first efficient algorithm for binary logistic bandits with a O p 1 q computation cost, achieving a regret of ˜ O p d ? T { κ q . Faury et al. [22] enhanced the regret to ˜ O p d ? T { κ q with a O p t q computation cost. However, their regret bounds still suffered from a harmful dependence on 1 { κ . Abeille et al. [4] addressed this by achieving the tightest regret upper bound of ˜ O p d ? κT q with a O p t q computation cost, while Faury et al. [23] achieved the same regret with an improved computation cost of O p log t q . More recently, Zhang and Sugiyama [53] proposed a jointly efficient algorithm that achieves the optimal regret with a constant O p 1 q computation cost. Note that the logistic bandit is a special case of the multinomial logistic (MNL) bandit. When the maximum assortment size is one ( K ' 1 ), rewards are uniform ( r t 1 ' 1 ), and the attraction parameter for the outside option is one v 0 ' 1 , the MNL bandit reduces to the logistic bandit. In this logistic bandit setting, our proposed algorithm, OFU-MNL+ , can achieve a regret upper bound of ˜ O p d ? κT q with a constant O p 1 q computation cost, consistent with the result in Zhang and Sugiyama [53].

Multinomial Logistic (MNL) Bandits. There are two main approaches to multinomial logistic (MNL) bandits: the multiple-parameter choice model and the single-parameter choice model. In the multiple-parameter choice model, the learner estimates parameters for each choice in the assortment ( w ‹ 1 , . . . , w ‹ K ) with a shared context x t . In this setting, Amani and Thrampoulidis [9] proposed a feasible algorithm that achieves a regret upper bound of ˜ O p dK ? κT q with a O p t q computation cost. They also proposed an intractable algorithm that achieves an improved regret of ˜ O p dK 3 { 2 ? T q . Zhang and Sugiyama [53] introduced a computationally and statistically efficient algorithm that obtains a regret of ˜ O p dK ? T q . Recently, Lee et al. [33] further improved the regret by a factor of ? K , achieving ˜ O p d ? KT q regret. In the multiple-parameter case, the regret's dependence on K is unavoidable since the number of unknown parameters depends on K .

On the other hand, the single-parameter choice model, closely related to ours, shares the parameter w ‹ cross the choices, with varying contexts for each choice. The learner offers a set of items S t , with | S t | ď K at each round. This setting involves a combinatorial optimization to choose the assortment S t , making it more challenging to devise a tractable algorithm. As extensively discussed in Section 2, no previous studies have definitively confirmed whether the existing lower or upper bounds are tight. As shown in Table 1, many studies have presented their results in inconsistent settings with varying reward structures and values of v 0 , adding to the ambiguity about the bounds' optimality. In this paper, we address these issues by bridging the gap between the lower and upper bounds of regret through a careful categorization of the settings. We propose an algorithm that is both provably optimal, up to logarithmic factors, and computationally efficient, significantly enhancing the theoretical and practical understanding of MNL bandits.

Generalized Linear Bandits. In generalized linear bandits [24, 27, 35, 3, 31, 28, 29, 34], the expected rewards are modeled using a generalized linear model. These problems generalize logistic bandits by incorporating a general exponential family link function instead of the logistic link function. The algorithms proposed for generalized linear bandits also exhibit a dependence on the nonlinear term κ . However, our problem setting (single-parameter MNL bandits) considers a more complex state space where multiple arms are pulled simultaneously.

Combinatorial Bandits. Another related stream of literature is combinatorial bandits [14, 45, 30, 54, 46, 36], particularly topk combinatorial bandits [46]. In topk combinatorial bandits, the decision set includes all subsets of size k out of n arms, and the reward for each action is the sum of the rewards of the k selected arms. In this framework, the rewards are assumed to be independent of the entire set of arms played in round t . In contrast, in our setting, the reward for each individual arm depends on the whole set of arms played.

Recently, Choi et al. [19], Cao and Sun [13] have considered the cascading assortment bandit problem, which encompasses the MNL bandit problem as a special case where the cascading length

is 1 . However, these works do not strictly encompass our results. Choi et al. [19] only consider uniform rewards, and achieve a regret upper bound of ˜ O p d ? T q , which avoids dependence on both the cascading length and κ . When the cascading length is 1 , our result (Theorem 2) improves upon theirs by a factor of ? K . Moreover, their computation cost per round is O p t q since they employ MLE to estimate the parameter. Cao and Sun [13] consider non-uniform rewards, and achieve a regret upper bound of ˜ O p h 2 d ? MT q , where M is the cascading length and h ě p p i | S, w q p p i | S, w 1 q for all w , w 1 P W , S P S , and i P S Yt 0 u . However, their regret bound still suffers from a harmful dependence on h 2 , which can be exponentially large.

B Notation

We denote T as the total number of rounds and t P r T s as the current round. We denote N as the total number of items, K as the maximum size of assortments, and d as the dimension of feature vectors.

For notational simplicity, we define the loss function in two different forms throughout the proof:

where z ti ' x J ti w , z t ' p z ti q i P S t P R | S t | , and y t ' p y ti q i P S t P R | S t | . Thus, ℓ t p w q ' ℓ p z t , y t q . We offer a Table B.1 for convenient reference.

C Properties of MNL function

In this section, we present key properties of the MNL function and its associated loss, which are used throughout the paper.

C.1 Attraction parameter for Outside Option: v 0 ' Θ p 1 q is Common in Contextual MNL Bandits

In this subsection, we explain why the assumption that v 0 ' Θ p 1 q is made without loss of generality. Let the original feature vectors be x 1 ti P R d for every item i P r N s . Suppose that a context for the outside option x 1 t 0 is given and the probability of choosing any item i P S t Yt 0 u is defined as

Then, by dividing the denominator and numerator by exp ` p x 1 t 0 q J w ‹ ˘ , and defining x ti : ' x 1 ti ´ x 1 t 0 , we obtain the MNL probability in the form presented in (1) with v 0 ' exp p 0 q ' 1 . Note that this division does not change the probability. Therefore, v 0 ' Θ p 1 q is natural and common in contextual MNL bandit literature.

C.2 Self-concordant-like Function

Definition C.1 (Self-concordant-like function, Tran-Dinh et al. 49) . A convex function f P C 3 p R m q is M -self-concordant-like function with constant M if:

for s P R and M ą 0 , where ϕ p s q : ' f p a ` s b q for any a , b P R m .

Then, the MNL loss defined in (2) is 3 ? 2 -self-concordant-like function.

Proposition C.1. For any t P r T s , the multinomial logistic loss ℓ t p w q , defined in (2) , is 3 ? 2 -selfconcordant-like.

Proof. Consider the function ϕ p s q : ' log `ř n i ' 0 e a i s ` b i ˘ , where a ' r a 0 , . . . , a n s J P R n ` 1 and b ' r b 0 , . . . , b n s J P R n ` 1 . Then, by simple calculus, we have

and

Note that for all i, j, k ' 0 , . . . , n ,

Therefore, we have

ˇ

ˇ

Plugging in (C.3) into (C.1), we obtain

Now, we are ready to prove the proposition. For any t P r T s , let n ' | S t | and c 1 ' x ti 1 , c 2 ' x ti 2 , . . . , c n ' x ti n . Define a function f P C 3 : R d Ñ R as f p θ q : ' log ´ v 0 ` ř n i ' 1 e c J i θ ¯ . Let δ P R d and let f p θ ` s δ q ' log ´ v 0 ` ř n i ' 1 e c J i θ ` sc J i δ ¯ ' log `ř n i ' 0 e a i s ` b i ˘ ' ϕ p s q , where a i ' c J i δ , b i ' c J i θ for i ' 1 , . . . , n , and a i ' 0 and b i ' log v 0 for i ' 0 . Then, by (C.4), we get | ϕ 3 p s q| ď 3 ? 2 max i ' 0 ,...,n | a i | ϕ 2 p s q ' 3 ? 2 max i ' 1 ,...,n | c J i δ | ϕ 2 p s q ď 3 ? 2 max } c i } 2 } δ } 2 ϕ 2 p s q ď 3 ? 2 } δ } 2 ϕ 2 p s q ,

where the last inequality holds due to Assumption 1 that } c i } 2 ' } x tj i } 2 ď 1 . Then, by Definition C.1, f is 3 ? 2 -self-concordant-like. Since ℓ t is the sum of f and a linear operator, which has third derivatives equal to zero, it follows that ℓ t is also 3 ? 2 -self-concordant-like function.

Remark C.1. Contrary to the findings of Perivier and Goyal [44], which suggest that the MNL loss function ? 6 K -self-concordant-like, our loss function is 3 ? 2 -self-concordant-like. This yields an improved regret bound on the order of O p ? K q . The improvement arises due to a K -independent self-concordant-like property of ℓ t , as shown in Proposition C.1. In Perivier and Goyal [44], Lemma 4 from Tran-Dinh et al. [49] is used, which describes a ? 6 } a } 2 self-concordant-like property. However, in the analysis of C.2, we show that their analysis is not tight because they bound the term ? a 2 i ` a 2 j ` a 2 k by } a } 2 ' ař n i ' 0 a 2 i , thus making its upper bound dependent on K , i.e., n ' | S t | ď K . In contrast, we bound the same term by a constant, max i ' 1 ,...,n } a i } 2 , which allows our loss function to exhibit a constant 3 ? 2 -self-concordant-like property. This key difference accounts for the ? K -improved regret.

Lemma C.1 (Theorem 3 of Tran-Dinh et al. 49) . A convex function ℓ P C 3 : R d Ñ R is M -selfconcordant-like if and only if for any v , u 1 , u 2 , u 3 P R d , we have

D Proof of Theorem 1

In this section, we provide the proof of Theorem 1. The proof structure is similar to the one presented in Chen et al. [16]. However, unlike their approach, we explicitly derive a bound that includes v 0 . Furthermore, by establishing a tighter upper bound for the KL divergence (Lemma D.2), we derive a bound that is tighter than the one provided by Chen et al. [16].

D.1 Adversarial Construction and Bayes Risk

Let ϵ P p 0 , 1 { d ? d q be a small positive parameter to be specified later. For every subset V Ď r d s , we define the corresponding parameter w V P R d as r w V s j ' ϵ for all j P V , and r w V s j ' 0 for all j R V . Then, we consider the following parameter set

where V k denotes the class of all subsets of r d s whose size is k . Moreover, note that d { 4 is a positive integer, as d is divisible by 4 by construction.

The context vectors t x ti u are constructed to be invariant across rounds t . For each t and U P V d { 4 , K identical context vectors 4 x U are constructed as follows:

For all V, U P V d { 4 , it can be verified that w V and x U satisfy the requirements of a bounded assumption 1 as follows:

Therefore, the worst-case expected regret of any policy π can be lower bounded by the worst-case expected regret of parameters belonging to W , which can be further lower bounded by the 'average'

regret over a uniform prior over W as follows:

This reduces the task of lower bounding the worst-case regret of any policy to the task of lower bounding the Bayes risk of the constructed parameter set.

D.2 Main Proof of Theorem 1

Proof of Theorem 1. For any sequence of assortments t S t u T t ' 1 produced by policy π , we denote an alternative sequence t ˜ S t u T t ' 1 that provably enjoys less regret under parameterization w V .

Let x U 1 , . . . , x U L be the distinct feature vectors contained in assortments S t (if S t ' H , then one may choose an arbitrary feature x U ) with U 1 , . . . , U L P V d { 4 . Let U ‹ be the subset among U 1 , . . . , U L that maximizes x J U w V , i.e., U ‹ P argmax U Pt U 1 ,...,U L u x J U w V , where w V is the underlying parameter. Then, we define ˜ S t as the assortment consisting of all K items corresponding to feature x U ‹ , i.e., ˜ S t ' t x U ‹ , . . . , x U ‹ loooooomoooooon u .

Since the expected revenue is an increasing function, we have the following observation:

Proposition D.1 (Proposition 1 in Chen et al. 16) .

Proposition D.1 implies that ř i P S ‹ p p i | S ‹ , w V q ´ ř i P S t p p i | S t , w V q ě ř i P S ‹ p p i | S ‹ , w V q ´ ř i P ˜ S t p p i | ˜ S t , w V q . Hence, it is sufficient to bound ř i P S ‹ p p i | S ‹ , w V q ´ ř i P ˜ S t p p i | ˜ S t , w V q instead of ř i P S ‹ p p i | S ‹ , w V q ´ ř i P S t p p i | S t , w V q .

To simplify notation, we denote ˜ U t as the unique U ‹ P V d { 4 in ˜ S t . We also use E V and P V to denote the expected value and probability, respectively, as governed by the law parameterized by w V and under policy π . Then, we can establish a lower bound for ř i P S ‹ p p i | S ‹ , w V q ´ ř i P ˜ S t p p i | ˜ S t , w V q as follows:

Lemma D.1. Suppose ϵ P p 0 , 1 { d ? d q and define δ : ' d { 4 ´| ˜ U t X V | . Then, we have

For any j P V , define random variables ˜ M j : ' ř T t ' 1 1 t j P ˜ U t u . Then, by Lemma D.1, for all V P V d { 4 , we have

Furthermore, we define V p j q d { 4 : ' t V P V d { 4 : j P V u and V d { 4 ´ 1 : ' t V Ď r d s : | V | ' d { 4 ´ 1 u . By taking the average of both sides of Equation (D.2) with respect to all V P V d { 4 , we obtain

For any fixed V , we get ř j R V E V r ˜ M j s ď ř d j ' 1 E V r ˜ M j s ď dT { 4 . Also, we have | V d { 4 ´ 1 | | V d { 4 | ' ` d d { 4 ´ 1 ˘ { ` d d { 4 ˘ ' d { 4 3 d { 4 ` 1 ď 1 3 . Consequently, we derive that

ˇ

ˇ

Now we bound the term ˇ ˇ E V Yt j u r ˜ M j s ´ E V r ˜ M j s ˇ ˇ in (D.3) for any V P V d { 4 ´ 1 . For simplicity, let P ' P V and Q ' P V Yt j u denote the laws under w V and w V Y j , respectively. Then, we have

where } P ´ Q } TV ' sup A | P p A q ´ Q p A q| | is the total variation distance between P and Q , KL p P } Q q ' ş p log d P { d Q q d P is s the Kullback-Leibler (KL) divergence between P and Q , and the last inequality holds by Pinsker's inequality. Now, we bound the KL divergence term using the following Lemma.

Lemma D.2. For any V P V d { 4 ´ 1 and j P r d s , there exists a positive constant C KL ą 0 such that

Therefore, combining (D.3), (D.4), and Lemma D.2, we have

where the second inequality is due to the Cauchy-Schwartz inequality and the last inequality holds because ř d j ' 1 E V r ˜ M j s ď dT { 4 . Let C 1 KL ' C KL { 4 .

By setting ϵ ' b d 144 C 1 KL T ¨ p v 0 ` K q 2 v 0 K , we have

This concludes the proof of Theorem 1.

D.3 Proofs of Lemmas for Theorem 1

D.3.1 Proof of Lemma D.1

Proof of Lemma D.1. Let x ' x V and ˆ x ' x ˜ U t be the corresponding context vectors. Then, we have

where the inequality holds since max ␣ exp ` x J w V ˘ , exp ` ˆ x J w V ˘( ď e . To further bound the right-hand side of (D.5), we use the fact that 1 ` a ď e a ď 1 ` a ` a 2 { 2 for all a P r 0 , 1 s , which can be easily shown by Taylor expansion. Thus, we get

where the last inequality holds because p ? dϵ q 2 ď δϵ { ? d when ϵ P p 0 , 1 { d ? d q . This concludes the proof.

D.3.2 Proof of Lemma D.2

Proof of Lemma D.2. Fix a round t , an assortment ˜ S t , and ˜ U t . Let U ' ˜ U t . Define m j p ˜ S t q : ' ř x U P ˜ S t 1 t j P U u{ K . Let t p i u i P ˜ S t Yt 0 u and t q i u i P ˜ S t Yt 0 u be the probabilities of choosing item i under parameterizations w V and w V Yt j u , respectively. Then, we have

where the first inequality holds because log p 1 ` x q ď x for all x ą ´ 1 .

Let ˆ x ' x U . Now, we separately upper bound p p i ´ q i q { q i , by analyzing the following three cases:

Case 1. The outside option, i ' 0

2 .

For i ' 0 , q i ě v 0 v 0 ` Ke . Thus, we have

ˇ

ˇ

where the third equality holds by applying the mean value theorem for the exponential function, with ¯ c 1 : ' p 1 ´ u qp ˆ x J w V q ` u p ˆ x J w V Yt j u q for some u P p 0 , 1 q . Then, there exist an absolute constant C 0 such that

where the last inequality holds since m j p ˜ S t q ď 1 .

Case 2. i P ˜ S t and j R U .

Then, for any i P ˜ S t corresponding to x i ' ˆ x and j R U , we have

where the last equality holds because exp ` ˆ x J w V ˘ ' exp ` ˆ x J w V Yt j u ˘ , given that j R U . Thus, we get

Case 3. i P ˜ S t and j P U .

Recall that for any i P ˜ S t , q i ě e ´ 1 v 0 ` Ke . Then, for any i P ˜ S t corresponding to x i ' ˆ x and j P U , we have

the second equality holds by applying the mean value theorem, with ¯ c 2 : ' p 1 ´ u qp ˆ x J w V q ` u p ˆ x J w V Yt j u q for some u P p 0 , 1 q . Then, there exist an absolute constant C 1 such that

where the last inequality holds since m j p ˜ S t q ď 1 .

Combining (D.6), (D.7), and (D.8), we derive that

where C KL ' max t C 0 , C 1 u . Since ˜ M j ' ř T t ' 1 m j p ˜ S t q by definition, and subsequently summing over all t ' 1 to T , we have

where the equality holds by the chain rule of relative entropy (cf. Exercise 14.11 of Lattimore and Szepesvári [32]). This concludes the proof.

E Proof of Theorem 2

In this section, we present the proof of Theorem 2. Note that when the rewards are uniform, the revenue increases as a function of the assortment size. Therefore, maximizing the expected revenue R t p S, w q across all possible assortments S P S always contains exactly K items. In other words, the size of the chosen assortment S t and the size of the optimal assortment S ‹ t both equal to K .

E.1 Main Proof of Theorem 2

Before presenting the proof, we introduce useful lemmas, whose proof can be found in Appendix E.2. Lemma E.1 shows the optimistic utility for the context vectors.

Lemma E.1. Let α ti ' x J ti w t ` β t p δ q} x ti } H ´ 1 t . If w ‹ P C t p δ q , then we have

Lemma E.3 is a K -free elliptical potential lemma that improves upon the one presented in Lemma 10 of Perivier and Goyal [44] in terms of K . Lemma 10 of Perivier and Goyal [44] states: ř t s ' 1 ř i P S s p s p i | S s , w ‹ q p s p 0 | S s , w ‹ q} x si } 2 H s p w ‹ q ´ 1 ď 2 dK log ` λ t ` 1 ` 2 tK d ˘ and ř t s ' 1 max i P S s } x si } 2 H s p w ‹ q ´ 1 ď 2 d ` K ` 1 κ ˘ log ` λ t ` 1 ` 2 tK d ˘ , where H t p w q ' ř t ´ 1 s ' 1 G s p w q ` λ t I d .

Lemma E.2. Let H t ' λ I d ` ř t ´ 1 s ' 1 G s p w s ` 1 q , where G s p w q ' ř i P S s p s p i | S s , w q x si x J si ´ ř i P S s ř j P S s p s p i | S s , w q p s p j | S s , w q x si x J sj . Suppose λ ě 1 . Then the following statements hold true:

Moreover, we provide a tighter bound for the second derivative of the expected revenue than that presented in Lemma 12 of Perivier and Goyal [44]. Lemma 12 of Perivier and Goyal [44] states: ˇ ˇ ˇ B 2 Q B i B j ˇ ˇ ˇ ď 5 .

Lemma E.3. Define Q : R K Ñ R , such that for any u ' p u 1 , . . . , u K q P R K , Q p u q ' ř K i ' 1 exp p u i q v 0 ` ř K k ' 1 exp p u k q . Let p i p u q ' exp p u i q v 0 ` ř K k ' 1 exp p u k q . Then, for all i P r K s , we have

Now, we are ready to prove Theorem 2.

Proof of Theorem 2. First, we bound the regret as follows:

where the first inequality holds by Lemma E.1, and the last inequality holds by the assortment selection of Algorithm 1.

Now, we define Q : R K Ñ R , such that for all u ' p u 1 , . . . , u K q J P R K , Q p u q ' ř K i ' 1 exp p u i q v 0 ` ř K j ' 1 exp p u j q . Noting that S t always contains K elements since the expected revenue is an increasing function in the uniform reward setting, we can write S t ' t i 1 , . . . , i K u . Moreover, for all t ě 1 , let u t ' p u ti 1 , . . . u ti K q J ' p α ti 1 , . . . , α ti K q J and u ‹ t ' p u ‹ ti 1 , . . . u ‹ ti K q J '

p x J ti 1 w ‹ , . . . , x J ti K w ‹ q J . Then, by a second order Taylor expansion, we have

where ¯ u t ' p ¯ u ti 1 , . . . , ¯ u ti K q J P R K is the convex combination of u t and u ‹ t . First, we bound the term (A) .

where the first inequality holds by Lemma E.1, and the last inequality holds because β t p δ q is increasing for t P r T s .

Now we bound the term (B) . Let p i p ¯ u t q ' exp p ¯ u ti q v 0 ` ř K k ' 1 exp p ¯ u tk q . Then, we have

where the inequality is by Lemma E.3. To bound the first term in (E.3), by applying the AM-GM inequality, we get

By plugging (E.4) into (E.3), we have

where the second inequality holds by Lemma E.1. Combining the upper bound for the terms (A) and (B) , with probability at least 1 ´ δ , we have

Now, we bound each term of (E.6) respectively. For the first term, we decompose it as follows:

To bound the first term on the right-hand side of (E.7), we apply the Cauchy-Schwarz inequality.

where the last inequality holds by Lemma E.2.

Now, we bound the second term on the right-hand side of (E.7). Let the virtual context for the outside option be x t 0 ' 0 . Then, by the mean value theorem, there exists ξ t ' p 1 ´ c q w ‹ ` c w t ` 1 for some c P p 0 , 1 q such that

Then, since x t 0 ' 0 , we can further bound the right-hand side as:

where the first inequality holds due to Jensen's inequality and the second-to-last inequality holds by Lemma 1. Hence, we get

where the last inequality holds by Lemma E.2.

Finally, we bound the third term on the right-hand side of (E.7). By the mean value theorem, there exists ξ 1 t ' p 1 ´ c 1 q w ‹ ` c 1 w t ` 1 for some c 1 P p 0 , 1 q such that

where the third inequality holds by Lemma 1, and the last inequality holds since p max i a i q 2 ' max i a 2 i for any a i ě 0 . Therefore, we have

where the last inequality holds by Lemma E.2. By plugging (E.8), (E.9), and (E.10) into (E.7) and multiplying 2 β T p δ q , we get

Moreover, by applying Lemma E.2, we can directly bound the second term of (E.6).

Finally, plugging (E.11) and (E.12) into (E.6), we obtain

Remark E.1. If the boundedness assumption on the parameter is relaxed to } w } 2 ď B , since β t p δ q ' O ´ B ? d log t log K ` B 3 { 2 ? d log K ¯ (refer Corollary F.1), we have Reg T p w ‹ q ' ˜ O ´ B 3 { 2 e B ? v 0 K v 0 ` K d ? T ` 1 κ d 2 ¯ . It's important to note that one of our main goals is to explicitly demonstrate the regret depends on K and v 0 . In deriving such a result, the dependence on e B is unavoidable to our best knowledge. Note that for non-uniform rewards, the regret bound does not depend on e B (refer Remark H.1).

E.2 Proofs of Lemmas for Theorem 2

E.2.1 Proof of Lemma E.1

Proof of Lemma E.1. Under the condition w ‹ P C t p δ q , we have

where the first inequality is by the Hölder's inequality, and the last inequality holds by Lemma 1. Hence, it follows that

Moreover, from x J ti w t ´ x J ti w ‹ ě ´ β t p δ q} x ti } H ´ 1 t , we also have

This concludes the proof.

E.2.2 Proof of Lemma E.2

Proof of Lemma E.2. Since xx J ` yy J ľ xy J ` yx J for any x, y P R d , it follows that

Hence, we have

which implies that

Then, we get

Since λ ě 1 , for all t ě 1 , we have ř i P S t p t p i | S t , w t ` 1 q p 0 p i | S t , w t ` 1 q} x ti } 2 H ´ 1 t ď 1 . Then, using the fact that z ď 2 log p 1 ` z q for any z P r 0 , 1 s , we get

This proves the first inequality.

To establish the proof for the second inequality, we return to (E.13):

which implies that

Then, we get

Since λ ě 1 , for all t ě 1 , we have κ max i P S t } x ti } 2 H ´ 1 t ď κ . We then conclude on the same way:

which proves the second inequality.

E.2.3 Proof of Lemma E.3

Proof of Lemma E.3. Let i, j P r K s . We first have

Let p i p u q ' e u i v 0 ` ř K k ' 1 e u k and p 0 p u q ' v 0 v 0 ` ř K k ' 1 e u k . For i ' j , we have ˇ ˇ ˇ ˇ B 2 Q B i B j ˇ ˇ ˇ ˇ ' ˇ ˇ ˇ ˇ p i p u q ´ p i p u q p j p u q ´ p i p u q ř K k ' 1 e u k v 0 ` ř K k ' 1 e u k ´ p i p u q p j p u q ` 2 p i p u q p j p u q ř K k ' 1 e u k v 0 ` ř K k ' 1 e u k ˇ ˇ ˇ ˇ ' ˇ ˇ ˇ ˇ p i p u q p 0 p u q ´ 2 p i p u q p j p u q ` 2 p i p u q p j p u q ř K k ' 1 e u k v 0 ` ř K k ' 1 e u k ˇ ˇ ˇ ˇ ' ˇ ˇ ˇ ˇ p i p u q p 0 p u q ´ 2 p i p u q p j p u q p 0 p u q ˇ ˇ ˇ ˇ ď 3 p i p u q For i ‰ j , we have ˇ ˇ ˇ ˇ B 2 Q B i B j ˇ ˇ ˇ ˇ ' ˇ ˇ ˇ ˇ ´ p i p u q p j p u q ´ p i p u q p j p u q ` 2 p i p u q p j p u q ř K k ' 1 e u k v 0 ` ř K k ' 1 e u k ˇ ˇ ˇ ˇ ' ˇ ˇ ˇ ˇ ´ 2 p i p u q p j p u q p 0 p u q ˇ ˇ ˇ ˇ ď 2 p i p u q p j p u q .

This concludes the proof.

F Proof of Lemma 1

In this section, we provide the proof of Lemma 1. First, we present the main proof of Lemma 1, followed by the proof of the technical lemma utilized within the main proof.

F.1 Main Proof of Lemma 1

Proof of Lemma 1. The proof is similar to the analysis presented in Zhang and Sugiyama [53]. However, their MNL choice model is constructed using a shared context x t and varying parameters across the choices w ‹ 1 , . . . , w ‹ K , whereas our approach considers an MNL choice model that shares the parameter w ‹ across the choices and has varying contexts for each item in the assortment S , x t 1 , . . . x ti | S | . Moreover, Zhang and Sugiyama [53] only consider a fixed assortment size, whereas we consider a more general setting where the assortment size can vary in each round t . We denote K t ' | S t | in the proof of Lemma 1. Note that K t ď K for all t ě 1 .

We first bound the first term in (F.1). For simplicity, we define the softmax function at round t σ t p z q : R K t Ñ R K t as follows:

where r¨s i denotes i 'th element of the input vector. We denote the probability of choosing the outside option as r σ t p z qs 0 ' v 0 v 0 ` ř Kt k ' 1 exp pr z s k q . Although r σ t p z qs 0 is not the output of the softmax function σ t p z q , we represent it in a form similar to that in (F.2) for simplicity. Then, the user choice model in (1) can be equivalently expressed as p t p i | S t , w q ' ' σ t ` p x J tj w q j P S t ˘‰ i for all i P r K t s and p t p 0 | S t , w q ' ' σ t ` p x J tj w q j P S t ˘‰ 0 . Furthermore, the loss function (2) can also be written as ℓ p z t , y t q ' ř K t k ' 0 1 t y ti ' 1 u ¨ log ´ 1 r σ t p z t qs k ¯ .

Define a pseudo-inverse function of σ t p¨q as σ ` t : R K t Ñ R K t , where r σ ` t p q qs i ' log p q i {p 1 ´} q } 1 qq for any q P t p P r 0 , 1 s K t | } p } 1 ă 1 u . Then, inspired by the previous studies on binary logistic bandit [23], we decompose the regret into two terms by introducing an intermediate term.

where ˜ z s : ' σ ` s ` E w ' P s ' σ s ` p x J sj w q j P S s ˘‰˘ , and P s : ' N p w s , cH ´ 1 s q is the Gaussian distribution with mean w s and covariance matrix cH ´ 1 s , where c ą 0 is a positive constant to be specified later. We first show that the term p a q is bounded by O ` log K p log t q 2 ˘ with high probability.

Lemma F.2. Let δ P p 0 , 1 s . Under Assumptions 1, for all t P r T s , with probability at least 1 ´ δ , we have

Furthermore, we can bound the term p b q by the following lemma.

Lemma F.3. For any c ą 0 , let λ ě max t 2 , 72 cd u . Then, under Assumption 1, for all t ě 1 , we have

Now, we are ready to prove the Lemma 1. By combining Lemma F.1, Lemma F.2, and Lemma F.3, we derive that

where the second inequality holds because by setting c ' 7 η { 6 and λ ě max t 84 ? 2 η, 84 dη u , we obtain:

where the first inequality holds since H s ľ λ I d .

By setting η ' 1 2 log p K ` 1 q ` 2 and λ ' 84 ? 2 dη , we derive that

where the third inequality follows from the definition of β 1 t ` 1 p δ q in Equation (F.4), and the last inequality holds by Lemma F.9.

This concludes the proof of Lemma 1.

Corollary F.1. If the boundedness assumption on the parameter is relaxed to } w } 2 ď B , then β t p δ q ' O ´ B ? d log t log K ` B 3 { 2 ? d log K ¯ .

Proof of Corollary F.1. When } w } 2 ď B , following the same analysis as in Lemma 1, we can set η ' 1 2 log p K ` 1 q ` B ` 1 , c ' 7 η { 6 , and λ ' O p dB log K q . Under these settings, we have

which concludes the proof.

F.2 Proofs of Lemmas for Lemma 1

F.2.1 Proof of Lemma F.1

Proof of Lemma F.1. Let ˜ ℓ s p w q ' ℓ s p w s q`x ∇ ℓ s p w s q , w ´ w s y` 1 2 } w ´ w s } 2 ∇ 2 ℓ s p w s q be a secondorder approximation of the original function ℓ s p w q at the point w s . The update rule (3) can also be expressed as

To utilize Lemma F.6, we can rewrite the loss function as ℓ ` p x J si w q i P S s , y s ˘ ' ℓ s p w q . Consequently, according to Lemma F.6, it follows that

where ζ ' log p K ` 1 q ` 4 . Then, by combining (F.5) and (F.6), we have

In above, we can further bound the first term of the right-hand side as:

where in the second equality, we apply the Taylor expansion by introducing ξ s ` 1 , a convex combination of w s ` 1 and w s . The first inequality follows from Lemma C.1 and Proposition C.1, the second inequality holds by Assumption 1, and the last inequality holds because

where the third equality holds by setting x s 0 ' 0 for all s ě 1 .

Now, by taking the summation over s and rearranging the terms, we obtain

where the last inequality holds since } w 1 ´ w ‹ } 2 H 1 ď λ } w 1 ´ w ‹ } 2 2 ď 4 λ . Plugging in ζ ' 2 η , we conclude the proof.

F.2.2 Proof of Lemma F.2

Proof of Lemma F.2. Since the norm of ˜ z s ' σ ` s ` E w ' P s ' σ s ` p x J sj w q j P S s ˘‰˘ is unbounded in general, as suggested by Foster et al. [25], we use the smoothed version ˜ z µ s ' σ ` s ` smooth µ s E w ' P s ' σ s ` p x J sj w q j P S s ˘‰˘ as an intermediate-term, where the smooth function is defined by smooth µ s p q q ' p 1 ´ µ q q ` µ 1 {p K s ` 1 q , where 1 P R K s is an all one vector.

Note that ˜ z µ s ' σ ` s p smooth µ s p σ s p ˜ z s qqq by the definition of the pseudo inverse function σ ` s such that σ ` s p σ s p q qq ' q for any q P t p P r 0 , 1 s K s | } p } 1 ă 1 u . Then, by Lemma F.7, we have

Hence, to prove the lemma, we need only to bound the gap between the loss of w ‹ and ˜ z µ s . To enhance clarity in our presentation, let ℓ p z ‹ s , y s q ' ℓ s p w ‹ q , where z ‹ s ' ` x J sj w ‹ ˘ j P S s P R K s . Then,

we have

where c µ ' log p K ` 1 q ` 2 log p 1 `p K ` 1 q{ µ q ` 2 , the inequality holds by Lemma F.6, and the last equality holds by a direct calculation of the first order and Hessian of the logistic loss as follows:

We first bound the first term of the right-hand side. Define d s ' p z ‹ s ´ ˜ z µ s q{p c µ ` 1 q . Let d 1 s be d s extended with zero padding. Specifically, we define d 1 s ' r d J s , 0 , . . . , 0 s J P R K , where the zeros are appended to increase the dimension of d s to K . Similarly, we also extend σ s p z ‹ s q ´ y s with zero padding and define ε s ' rp σ s p z ‹ s q ´ y s q J , 0 , . . . , 0 s J P R K .

Then, one can easily verify that } d 1 s } 8 ď 1 since } z ‹ s } 8 ď max i P S s } x si } 2 } w ‹ } 2 ď 1 and } ˜ z µ s } 8 ď log p 1 ` p K ` 1 q{ µ q . On the other hand, d 1 s is F s -measurable since z ‹ s and ˜ z µ s are independent of y s . Moreover, we have } d 1 s } 2 E r ε s ε J s | F s s ' } d s } 2 E rp σ s p z ‹ s q´ y s qp σ s p z ‹ s q´ y s q J | F s s ' } d s } 2 ∇ σ s p z ‹ s q and } σ s p z ‹ s q ´ y s } 1 ď 2 . Thus, by Lemma F.5, with probability at least 1 ´ δ , for any t ě 1 , we have

where in the second inequality, we set ˜ λ ą 1 , and the last inequality holds because } d s } 2 ∇ σ s p z ‹ s q ' d J s ∇ σ s p z ‹ s q d s ď 2 . Then, combining (F.8) and (F.9), we obtain

where the third inequality holds due to the AM-GM inequality. Finally, combining (F.7) and (F.10), by setting µ ' 1 { t and ˜ λ ' 16 ? 17 log ´ 2 ? 1 ` 2 t δ ¯ , we have

where the last inequality holds by the definition of c µ ' log p K ` 1 q ` 2 log p 1 `p K ` 1 q{ µ q ` 2 . This concludes the proof.

F.2.3 Proof of Lemma F.3

Proof of Lemma F.3. The proof with an observation from Proposition 2 in Foster et al. [25], which notes that ˜ z s is an aggregation forecaster for the logistic function. Hence, it satisfies

Then, by the quadratic approximation, we get

Applying Lemma F.8 and considering the fact that ℓ s is 3 ? 2 -self-concordant-like function by Proposition C.1, we have

We define the function ˜ f s ` 1 : W Ñ R as

Then, we can establish a lower bound for the expectation in (F.11) as follows:

where the first inequality holds by (F.13) and the last equality holds by (F.12). We define ˜ Z s ` 1 ' ş R d ˜ f s ` 1 p w q d w ď `8 . Moreover, we denote the distribution whose density function is ˜ f s ` 1 p w q{ ˜ Z s ` 1 as ˜ P s ` 1 . Then, we can rewrite Equation (F.14) as follows:

where the second inequality follows from Jensen's inequality, and the equality holds because ˜ P s ` 1 is symmetric around w s ` 1 , thus E w ' ˜ P s ` 1 rx ∇ L s p w s ` 1 q , w ´ w s ` 1 ys ' 0 .

By plugging (F.15) into (F.11), we have

In the above, we can bound the last term, ´ log ˜ Z s ` 1 , by

where p P s ` 1 ' N p w s ` 1 , cH ´ 1 s q and p Z s ` 1 ' ş R d exp ` ´ 1 2 c } w ´ w s ` 1 } 2 H s ˘ d w . In (F.17), the inequality holds due to Jensen's inequality, and the last inequality is by the fact that p Z s ` 1 ' ş R d exp ` ´ 1 2 c } w ´ w s ` 1 } 2 H s ˘ d w ' b p 2 π q d c | H ´ 1 s | ' Z s .

By applying the Cauchy-Schwarz inequality, we can further bound the second term on the right-hand side of (F.17) by

Note that, since p P s ` 1 ' N p w s ` 1 , cH ´ 1 s q , there exist orthogonal bases e 1 , . . . , e d P R d such that w ´ w s ` 1 follows the same distribution as

and λ j ` H ´ 1 s ˘ denotes the j -th largest eigenvalue of H ´ 1 s . Then, we can bound the term (a)-1 in (F.18) as follows:

where the first inequality holds since λ j ` H ´ 1 s ˘ ď 1 λ . In the second equality, χ 2 denotes the chisquare distribution, and the last inequality is due to Jensen's inequality. By setting λ ě 72 cd , we get

where the last inequality holds due to the fact that the moment-generating function for χ 2 -distribution is bounded by E X ' χ 2 r e tX s ď 1 { ? 1 ´ 2 t for all t ď 1 { 2 .

Now, we bound the term (a)-2 in (F.18).

where ¯ H s ' p ∇ 2 ℓ s p w s ` 1 qq ´ 1 { 2 H s p ∇ 2 ℓ s p w s ` 1 qq ´ 1 { 2 . Let ¯ λ j ' λ j ` c ¯ H ´ 1 s ˘ be the j -th largest eigenvalue of the matrix c ¯ H ´ 1 s . Then, conducting an analysis similar to that in equation (F.19) yields that

where the inequality holds due to E X j ,X j 1 ' N p 0 , 1 q r X 2 j X 2 j 1 s ď 3 for all j, j 1 P r d s , and the last equality holds because ř d j ' 1 ¯ λ j ' Tr ` c ¯ H ´ 1 s ˘ . Here, Tr p A q denotes the trace of the matrix A .

We define the matrix M s ` 1 : ' λ I d { 2 ` ř s τ ' 1 ∇ 2 ℓ τ p w τ ` 1 q . Under the condition λ ě 2 , for any s P r T s and w P W , we have ∇ 2 ℓ s p w q ĺ I d ď λ 2 I d . Thus, we have H s ľ M s ` 1 . Then, we can bound the trace as follows:

where the last inequality holds by Lemma 4.5 of Hazan et al. [26]. Therefore we can bound the term (a)-2 as

By plugging (F.20) and (F.21) into (F.18), we have

Combining (F.16), (F.17), and (F.22), and taking summation over s , we derive that

By rearranging the terms, we conclude the proof.

F.3 Technical Lemmas for Lemma 1

Lemma F.4 (Proposition 4.1 of Campolongo and Orabona 12) . Let the w t ` 1 be the solution of the update rule

where V Ď W Ď R d is a non-empty convex set and D ψ p w 1 , w 2 q ' ψ p w 1 q ´ ψ p w 2 q ´ x ∇ ψ p w 2 q , w 1 ´ w 2 y is the Bregman Divergence w.r.t. a strictly convex and continuously differentiable function ψ : W Ñ R . Further supposing ψ p w q is 1 -strongly convex w.r.t. a certain norm } ¨ } in W , then there exists a g 1 t P B ℓ t p w t ` 1 q such that

for any u P W .

Lemma F.5 (Lemma 15 of Zhang and Sugiyama 53) . Let t F t u 8 t ' 1 be a filtration. Let t z t u 8 t ' 1 be a stochastic process in B 2 p K q ' t z P R K | } z } 8 ď 1 u such that z t is F t measurable. Let t ε t u 8 t ' 1 be a martingale difference sequence such that ε t P R K is F t ` 1 measurable. Furthermore, assume that, conditional on F t , we have } ε t } 1 ď 2 almost surely. Let Σ t ' E r ε t ε J t | F t s . and λ ą 0 . Then, for any t ě 1 define

Then, for any δ P p 0 , 1 s , we have

Lemma F.6 (Lemma 1 of Zhang and Sugiyama 53) . Let C ą 0 , a P r´ C, C s K , y P R K ` 1 be a one-hot vector and b P R K . Then, we have

Lemma F.7 (Lemma 17 of Zhang and Sugiyama 53) . Let z P R K be a K -dimensional vector. Let ℓ p z , y q ' ř K k ' 0 1 t y i ' 1 u ¨ log ´ 1 r σ p z qs k ¯ , where y ' r y 0 , . . . , y K s J P R K ` 1 , and the softmax function σ p z q : R K Ñ R K is defined as r σ p z qs i ' exp pr z s i q v 0 ` ř K k ' 1 exp pr z s k q for all i P r K s , and r σ p z qs 0 ' v 0 v 0 ` ř K k ' 1 exp pr z s k q . Define z µ : ' σ ` p smooth µ p σ p z qqq , where smooth µ p q q ' p 1 ´ µ q q ` µ 1 {p K ` 1 q . Then, for µ P r 0 , 1 { 2 s , we have

We also have } z µ } 8 ď log p 1 `p K ` 1 q{ µ q .

Lemma F.8 (Lemma 18 of Zhang and Sugiyama 53) . Let L t p w q ' ℓ t p w q ` 1 2 c } w ´ w t } 2 H t . Assume that ℓ t is a M -self-concordant-like function. Then, for any w , w t P W , the quadratic approximation ˜ L t p w q ' L t p w t ` 1 q ` x ∇ 2 L t p w t ` 1 q , w ´ w t ` 1 y ` 1 2 c } w ´ w t ` 1 } 2 H t satisfies

Lemma F.9. Let w t ` 1 ' argmin w P W x ∇ ℓ t p w t q , w y ` 1 2 η } w ´ w t } 2 ˜ H t . Then, we have

Proof of Lemma F.9. We make a slight improvement to Lemma 20 of Zhang and Sugiyama [53].

Lemma F.10 (Lemma 20 of Zhang and Sugiyama 53) . Under the same conditions as Lemma F.9, the following holds:

In the proof of Lemma 20 by Zhang and Sugiyama [53], the bound for } ∇ ℓ t p w t q} 2 was given as } ∇ ℓ t p w t q} 2 ď 2 ? K . However, a tighter bound can be established: } ∇ ℓ t p w t q} 2 ď 2 . Using this tighter bound, we can refine and derive the improved result.

G Proofs of Theorem 3

In this section, we provide the proof of Theorem 3. In addition to the adversarial construction presented in Section D.1, we construct the adversarial non-uniform rewards.

G.1 Adversarial Rewards Construction

Under the adversarial construction in Section D.1, we observe that there are K identical context vectors, invariant across rounds t . Therefore, in total, there are N ' K ¨ ` d d { 4 ˘ items. Let the rewards be also time-invariant. Given w V , we define a unique item i ‹ P r N s as an item that maximizes x J i w V , i.e., x i ‹ ' x V , and has a reward of 1 , i.e., r i ‹ ' 1 . Then, we construct the non-uniform rewards as follows:

Note that γ ă 1 .

G.2 Main Proof of Theorem 3

Proof of Theorem 3. Given the rewards construction as (G.1), any reward in the optimal assortment S ‹ t is larger than the expected revenues.

Lemma G.1. Let R p S ‹ , w V q ' ř i P S ‹ exp p x J i w V q r i v 0 ` ř ‹ exp p x J w V q . Then, we have

Lemma G.1 implies that S ‹ contains only one item i ‹ . This is because if S ‹ ' t x i ‹ u , adding any item i ‰ i ‹ to the assortment results in lower expected revenue, since r i ' γ ď R p S ‹ ' t x i ‹ u , w v q .

Furthermore, we can bound the expected revenue for any assortment as follows:

Lemma G.2. Under the same parameters and context vectors as those in Section D, if the rewards are constructed according to Equation (G.1) , for any S P S , we have

Let x U 1 , . . . , x U L be the distinct feature vectors contained in assortments S t with U 1 , . . . , U L P V d { 4 . Let U ‹ be the subset among U 1 , . . . , U L that maximizes x J U w V , i.e., U ‹ P argmax U Pt U 1 ,...,U L u x J U w V , where w V is the underlying parameter. For simplicity, we denote ˜ U t as the unique U ‹ P V d { 4 in S t . Then, we have

where the first equality holds becauset S ‹ contains only one item i ‹ by Lemma G.1 (and recall that x i ‹ ' x V ), and the inequality holds by Lemma G.2. Hence, the problem is not easier than solving where we define γ as

the MNL bandit problems with the assortment size 1 , i.e., K ' 1 . By putting K ' 1 and v 0 ' Θ p 1 q in Theorem 1, we derive that

This concludes the proof of Theorem 3.

G.3 Proofs of Lemmas for Theorem 3

G.3.1 Proof of Lemma G.1

Proof of Lemma G.1. We prove by contradiction. Assume that there exists i P S ‹ such that r i ă R p S ‹ , w V q . Then, removing the item i from the assortment S ‹ yields higher expected revenue. This contradicts the optimality of S ‹ . Thus, we have

This concludes the proof.

G.3.2 Proof of Lemma G.2

Proof of Lemma G.2. We provide a proof by considering the following cases:

Case 1. i ‹ P S .

Recall that, by the construction of rewards, we have

This implies that

Therefore, for all S P S , we have

where the first inequality holds by (G.3), and the last inequality holds since f p x q ' x v 0 ` x is an increasing function.

Case 2. i ‹ R S .

Let us return to (G.2). Since v 0 ` ř i P S exp p x J i w V q ř i P S exp p x J i w V q ě 1 for any S P S , we have

which is equivalent to

Hence, for all S P S , we get

This concludes the proof.

H Proofs of Theorem 4

In this section, we provide the proof of Theorem 4. Since we now consider the case of non-uniform rewards, the sizes of both the chosen assortment S t , and the optimal assortment, S ‹ t are no longer fixed at K .

We begin the proof by introducing additional useful lemmas. Lemma H.1 shows that ˜ R t p S t q , defined in (6), is an upper bound of the true expected revenue of the optimal assortment, R t p S ‹ t , w ‹ q .

Lemma H.1 (Lemma 4 in Oh and Iyengar 41) . Let ˜ R t p S q ' ř i P S exp p α ti q r ti v 0 ` ř j P S exp p α tj q . And suppose

S t ' argmax S P S ˜ R t p S q . If for every item i P S ‹ t , α ti ě x J ti w ‹ , then for all t ě 1 , the following inequalities hold:

Note that Lemma H.1 does not claim that the expected revenue is a monotone function in general. Instead, it specifically states that the value of the expected revenue, when associated with the optimal assortment, increases with an increase in the MNL parameters [8, 41].

Lemma H.2 shows that ˜ R t p S t q increases as the utilities of items in S t increase.

Lemma H.2. Let ˜ R t p S q ' ř i P S exp p α ti q r ti v 0 ` ř j P S exp p α tj q and S t ' argmax S P S ˜ R t p S q . Assume α 1 ti ě α ti ě 0 for all i P r N s . Then, we have

Furthermore, we provide a novel elliptical potential Lemma H.3 for the centralized context vectors ˜ x ti .

Lemma H.3. Let H t ' λ I d ` ř t ´ 1 s ' 1 G s p w s ` 1 q , where G s p w q ' ř i P S s p s p i | S s , w q x si x J si ´ ř i P S s ř j P S s p s p i | S s , w q p s p j | S s , w q x si x J sj . Define ˜ x si ' x si ´ E j ' p s p¨| S s , w s ` 1 q r x sj s . Suppose λ ě 2 . Then the following statements hold true:

Now, we prove the Theorem 4.

H.1 Proof of Theorem 4

Proof of Theorem 4. Let α 1 ti ' x J ti w ‹ ` 2 β t p δ q} x ti } H ´ 1 t . If w ‹ P C t p δ q , then, by Lemma E.1, we have

We denote « R t p S t q ' ř i P St exp p α 1 ti q r ti v 0 ` ř j P St exp p α 1 tj q . Then, we can bound the regret as follows:

where the first inequality holds by Lemma H.1 and the last inequality holds by Lemma H.2.

Now, we define ˜ Q : R | S t | Ñ R , such that for all u ' p u 1 , . . . , u | S t | q J P R | S t | , ˜ Q p u q ' ř | S t | k ' 1 exp p u k q r ti k v 0 ` ř | St | j ' 1 exp p u j q . Let S t ' t i 1 , . . . , i | S t | u . Moreover, for all t ě 1 , let u t ' p u ti 1 , . . . u ti | St | q J ' p α 1 ti 1 , . . . , α 1 ti | St | q J and u ‹ t ' p u ‹ ti 1 , . . . u ‹ ti | St | q J ' p x J ti 1 w ‹ , . . . , x J ti | St | w ‹ q J . Then, by applying a second order Taylor expansion, we obtain

where ¯ u t ' p ¯ u ti 1 , . . . , ¯ u ti | St | q J P R | S t | is the convex combination of u t and u ‹ t . We first bound the term (C) .

Let x t 0 ' 0 . Then, we can further bound the right-hand side as follows:

where, in the first inequality, we define S ` t Ď S t as the subset of items in S t such that the term } x ti } H ´ 1 t ´ E j ' p t p¨| S t , w ‹ q ' } x tj } H ´ 1 t ı ě 0 and r ti P r 0 , 1 s , the second inequality holds because β 1 p δ q ď ¨ ¨ ¨ ď β T p δ q , the third inequality holds due to Jensen's inequality, and the second-to-last inequality holds due to the fact that } a } ' } a ´ b ` b } ď } a ´ b } ` } b } for any vectors a , b P R d . For simplicity, we denote E w r x tj s ' E j ' p t p¨| S t , w q r x tj s . Let ¯ x ti ' x ti ´ E w ‹ r x tj s and ˜ x ti ' x ti ´ E w t ` 1 r x tj s . Then, we have

where the inequality holds by the triangle inequality. Now, we bound the terms on the right-hand side of (H.1) individually. For the first term, we have

where the last equality holds due to the setting of x t 0 ' 0 . By the mean value theorem, there exists ξ t ' p 1 ´ c q w ‹ ` c w t ` 1 for some c P p 0 , 1 q such that

where the fourth inequality holds by Lemma 1 and the second-to-last inequality holds due to Jensen's inequality. Hence, we have

where the last inequality holds by Lemma E.2. Using similar reasoning, we can bound the second term of (H.1). By the mean value theorem, there exists ξ 1 t ' p 1 ´ c 1 q w ‹ ` c 1 w t ` 1 for some c 1 P p 0 , 1 q

such that

Then, by applying the AM-GM inequality to each term, we obtain

where the equality holds since p max i a i q 2 ' max i a 2 i for any a i ě 0 . Thus, by Lemma H.3 (or Lemma E.2), we get

Finally, we bound the third term of (H.1). By the Cauchy-Schwarz inequality, we have

where the last inequality holds by Lemma H.3. Plugging (H.2), (H.3), and (H.4) into (H.1), we get

Thus, we can bound the term (c) as follows:

Now, we bound the term (D) . Define Q : R | S t | Ñ R , such that for all u ' p u 1 , . . . , u | S t | q P R | S t | , Q p u q ' ř | S t | i ' 1 exp p u i q v 0 ` ř | St | j ' 1 exp p u j q . Then, we have ˇ ˇ ˇ B 2 ˜ Q B i B j ˇ ˇ ˇ ď ˇ ˇ ˇ B 2 Q B i B j ˇ ˇ ˇ since r ti P r 0 , 1 s . By following the similar reasoning from Equation (E.3) to (E.5) in Section E.1, we have

where the first inequality holds because ˇ ˇ ˇ B 2 ˜ Q B i B j ˇ ˇ ˇ ď ˇ ˇ ˇ B 2 Q B i B j ˇ ˇ ˇ . Combining (H.5) and (H.6), we derive that

where β T p δ q ' O ´ ? d log T log K ¯ . This concludes the proof of Theorem 4.

Remark H.1. If the boundedness assumption on the parameter is relaxed to } w } 2 ď B , since β t p δ q ' O ´ B ? d log t log K ` B 3 { 2 ? d log K ¯ (refer Corollary F.1), we have Reg T p w ‹ q ' ˜ O ` B 3 { 2 d ? T ` 1 κ d 2 ˘ .

H.2 Proofs of Lemmas for Theorem 4

H.2.1 Proof of Lemma H.2

Proof of Lemma H.2. We prove the result by first showing that for any i P S t , we have r ti ě ˜ R t p S t q . This can be proven similarly to Lemma G.1. Suppose that there exists i P S t for which r ti ă ˜ R t p S t q . Removing item i from the assortment S t results in a higher expected revenue. Consequently, S t ‰ argmax S P S ˜ R t p S q , which contradicts the optimality of S t . Thus, we have

If we increase α ti to α 1 ti for all i P S t , the probability of selecting the outside option decreases. In other words, the sum of probabilities of choosing any i P S t increases. Since r ti ě ˜ R t p S t q for all i P S t , this results in an increase in revenue. Hence, we get

This concludes the proof.

H.2.2 Proof of Lemma H.3

Proof of Lemma H.3. For notational simplicity, let E w r x tj s ' E j ' p t p¨| S t , w q r x tj s . Let x t 0 ' 0 . We can rewrite G s p w q in the following way:

This means that

Hence, we can derive that

Since λ ě 1 , for all t ě 1 we have ř i P S t p t p i | S t , w t ` 1 q} ˜ x ti } 2 H ´ 1 t ď 1 λ max i P S t } ˜ x ti } 2 ď 1 . Then, using the fact that z ď 2 log p 1 ` z q for any z P r 0 , 1 s , we get

This proves the first inequality.

To show the second inequality, we come back to equation (H.8). By the definition of κ , we get

Thus, we obtain that

Since λ ě 1 , for all t ě 1 we have κ max i P S t } ˜ x ti } 2 H ´ 1 t ď κ λ } ˜ x ti } 2 ď κ . We then reach the conclusion in the same manner:

This proves the second inequality.

I Proof of Proposition 1

I.1 Main Proof of Proposition 1

Proof of Proposition 1. We construct the parameters and features in the same way as in Section D.1. Recall that, by Proposition D.1, it is sufficient to bound ř i P S ‹ p p i | S ‹ , w V q ´ ř i P ˜ S t p p i | ˜ S t , w V q . We denote ˜ U t as the unique U ‹ P V d { 4 in ˜ S t . We define a function µ : R Ñ r 0 , 1 s such that for any z P R , µ p z q ' K exp p z q v 0 ` K exp p z q . Since all items in S ‹ (and in ˜ S t ) are identical, we can express ř i P S ‹ p p i | S ‹ , w V q (and ř i P ˜ S t p p i | ˜ S t , w V q ) as follows:

Given w V , we define κ ‹ t p w V q : ' ř i P S ‹ p t p i | S ‹ , w V q p t p 0 | S ‹ , w V q . Since the context vectors t x ti u are constructed to be invariant across rounds t , κ ‹ t p w V q is also independent of t , i.e., κ ‹ 1 p w V q ' ¨ ¨ ¨ ' κ ‹ T p w V q . Therefore, we omit the index t for simplicity. Note that, in our instance construction, κ ‹ p w V q ' µ p x J V w V q ` 1 ´ µ p x J V w V q ˘ ' 9 µ p x J V w V q .

We hypothesize that for all V P V d { 4 , the regret is dominated by d a κ ‹ p w V q T . If this assumption does not hold, then by definition, there exists some V P V d { 4 such that E V r Reg T p w V qs ' Ω ´ d a κ ‹ p w V q T ¯ , thereby completing the proof.

Additionally, we set w ‹ ' argmax w V κ ‹ p w V q , thus, we have κ ‹ ' κ ‹ p w ‹ q ' max w V κ ‹ p w V q .

To establish an instance-dependent lower bound for µ p x J V w V q ´ µ p x J ˜ U t w V q , we use the following lemma in place of Lemma D.1:

Lemma I.1. Suppose ϵ P p 0 , 1 { d ? d q and define δ : ' d { 4 ´| ˜ U t X V | . Then, we have

For any j P V , define random variables ˜ M j : ' ř T t ' 1 1 t j P ˜ U t u . Then, by Lemma I.1, for all V P V d { 4 , we get

Additionally, we define V p j q d { 4 : ' t V P V d { 4 : j P V u and V d { 4 ´ 1 : ' t V Ď r d s : | V | ' d { 4 ´ 1 u . By averaging both sides of Equation (I.2) with respect to all V P V d { 4 , and by following reasoning similar to that in the proof of Theorem 1, we get

ˇ

ˇ

For simplicity, let P ' P V and Q ' P V Yt j u . Then, we can bound the term ˇ ˇ E V Yt j u r ˜ M j s ´ E V r ˜ M j s ˇ ˇ in (I.3) for any V P V d { 4 ´ 1 .

where } P ´ Q } TV ' sup A | P p A q ´ Q p A q| | is the total variation distance between P and Q , KL p P } Q q ' ş p log d P { d Q q d P is s the Kullback-Leibler (KL) divergence between P and Q , and the last inequality holds by Pinsker's inequality. We can derive the instance-dependent bound for the KL divergence term in (I.4) by the following lemma:

Lemma I.2. For any V P V d { 4 ´ 1 and j P r d s , there exists a positive constant C KL ą 0 such that

Plugging (I.4) into (I.3), we get

where the second inequality is due to the Cauchy-Schwartz inequality, and the last inequality is by Lemma I.2.

Recall that by construction, κ ‹ ' max w V κ ‹ p w V q . Thus, by taking the maximum over w V , we complete the proof of Proposition 1.

I.2 Proofs of Lemmas for Proposition 1

I.2.1 Proof of Lemma I.1

Proof of Lemma I.1. For simplicity, let x ' x V and ˆ x ' x ˜ U t . Then, by the mean value theorem, we have

where in the first equality, ¯ x is the convex combination of x and ˆ x , the first inequality holds by Lemma I.3, the second inequality holds by the bounded assumption (Assumption 1), and the last inequality holds by the definition of δ ' d { 4 ´| ˜ U t X V | .

I.2.2 Proof of Lemma I.2

Proof of Lemma I.2. Consider a fixed round t , an assortment ˜ S t , and the set ˜ U t . Let U ' ˜ U t . Define m j p ˜ S t q : ' ř x U P ˜ S t 1 t j P U u{ K , which captures the average presence of j P U across the assortment ˜ S t . For simplicity, we denote p ' K exp p x J U w V q v 0 ` K exp p x J U w V q and q ' K exp p x J U w V Yt j u q v 0 ` K exp p x J U w V Yt j u q . Then, we get

where in the first inequality, we used KL ď χ 2 (Tsybakov [50], Chapter 2), where χ 2 is a chi-square divergence, the second equality holds by the expression of the chi-square divergence for Bernoulli random variables, and the last equality holds by the mean value theorem, where ¯ w V,j is a convex combination of w V and w V Yt j u . Then, by Lemma I.4, we can further bound the last term as follows:

where the second-to-the last inequality holds based on the assumption that d ě 4 , and the last inequality holds since m j p ˜ S t q ď 1 .

Now, let us consider t varying over the round t P r T s . Then, we obtain

the equality follows from the chain rule of relative entropy (cf. Exercise 14.11 of Lattimore and Szepesvári [32]), the second inequality holds because ř d j ' 1 m j p ˜ S t q ď d 4 , second-to-the last inequality holds by Lemma I.5, and the last inequality holds by the hypothesis (Equation (I.1)) with C KL ą 0 .

Furthermore, by the definition of κ (Assumption 2), we know that κ ‹ p w V q ě κ . Hence, given the assumption that T ě d 2 { κ ě d 2 { κ ‹ p w V q , we derive that

which concludes the proof.

I.3 Technical Lemmas for Proposition 1

Lemma I.3 (Lemma 7 of Abeille et al. [4]) . Let f be a strictly increasing function such that | : f | ď 9 f , and let Z be any bounded interval of R . Then, for all z 1 , z 2 P Z :

Lemma I.4 (Lemma 9 of Abeille et al. [4]) . Let f be a strictly increasing function such that | : f | ď 9 f , and let Z be any bounded interval of R . Then, for all z 1 , z 2 P Z :

Lemma I.5 (Lemma 11 of Perivier and Goyal [44]) . Let κ ‹ t : ' ř i P S ‹ t p t p i | S ‹ t , w ‹ q p t p 0 | S ‹ t , w ‹ q . Then, we have

J Proof of Proposition 2

In this section, we provide the proof of Proposition 2. We introduce useful lemmas to support the proof.

Lemma J.1 (Theorem 4 of Tran-Dinh et al. [49]) . Let f : R K Ñ R be a M f -self-concordant-like function and let x, y P dom p f q , then:

Proof of Proposition 2. We begin the proof with Equation (E.2) from the proof of Theorem 2:

Wethen divide the total rounds into two disjoint sets, J 1 and J 2 , such that J 1 Ť J 2 ' r T s . Specifically, let J 1 ' t t P r T s| ř i P S t p t p i | S t , w ‹ q p t p 0 | S t , w ‹ q ě ř i P S t p t p i | S t , w t ` 1 q p t p 0 | S t , w t ` 1 qu and J 2 ' r T sz J 1 . For a better presentation, we also define that:

Then, we can rewrite the right-hand side of (J.1) as follows:

To bound ř t P J 1 g t p S t ; w ‹ q , by the mean value theorem, we get

where ¯ w t is the convex combination of w ‹ and w t ` 1 . The first term of (J.2) can be bound by

where the second-to-the last inequality holds by the definition of J 1 and the elliptical potential lemma (Lemma E.2), and the last inequality holds by the following Lemma:

Moreover, the second term of (J.2) can be bounded as follows:

ˇ

ˇ

where the first inequality holds by Lemma 1, and the last inequality holds by Lemma E.2. Combining (J.3) and (J.4), we can bound the term ř t P J 1 g t p S t ; w ‹ q as follows:

Now, we bound ř t P J 2 g t p S t ; w ‹ q . We define H ‹ t : ' λ I d ` ř t ´ 1 s ' 1 G s p w ‹ q . Recall that ∇ 2 ℓ s p w q ' G s p w q and ℓ s is 3 ? 2 -self-concordant-like function (Proposition C.1). Then, by Lemma J.1, we get H t ľ H ‹ t e ´ 3 ? 2 . With this fact, we can now bound the term ř t P J 2 g t p S t ; w ‹ q :

where the second inequality follows from Lemma I.5, second-to-the last inequality holds because H t ľ H ‹ t e ´ 3 ? 2 , and the last inequality is obtained by a slight modification of Lemma E.2 with w 1 , . . . , w T ' w ‹ .

Now, by plugging (J.5) and (J.6) into (J.1), and using the fact that β T p δ q ' ˜ O p ? d q , we obtain that

Furthermore, by applying the same analysis as in the proof of Theorem 2 in order to bound the second-order term in the Taylor expansion (term (B) in (E.1)), we get

K Experiment Details and Additional Results

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.01

0.02

0.03

0.04

Time (Seconds)

N=100, K=5, d=5, Uniform R

UCB-MNL

TS-MNL

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.01

0.02

0.03

0.04

Time (Seconds)

N=100, K=10, d=5, Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.01

0.02

0.03

0.04

Time (Seconds)

N=100, K=15, d=5, Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.15

0.30

0.45

0.60

Time (Seconds)

N=100, K=5, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.15

0.30

0.45

0.60

Time (Seconds)

N=100, K=10, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0.00

0.15

0.30

0.45

0.60

Time (Seconds)

N=100, K=15, d=5, Non-Uniform R

0

500

1000

1500

2000

2500

3000

Round (t)

0

25

50

Cumulative Regret

N=100, K=5, d=5, Uniform R, v_0=1.0

UCB-MNL

TS-MNL

0

500

1000

1500

2000

2500

3000

Round (t)

0

25

50

Cumulative Regret

N=100, K=10, d=5, Uniform R, v_0=2.0

0

500

1000

1500

2000

2500

3000

Round (t)

0

25

50

Cumulative Regret

N=100, K=15, d=5, Uniform R, v_0=3.0

For each instance, we sample the true parameter w ‹ from a uniform distribution in r´ 1 { ? d, 1 { ? d s d . For the context features x ti , we sample each x ti independently and identically distributed (i.i.d.) from a multivariate Gaussian distribution N p 0 d , I d q and clip it to range r´ 1 { ? d, 1 { ? d s d . Therefore, we ensure that } w ‹ } 2 ď 1 and } x ti } 2 ď 1 , satisfying Assumption 1. For each experimental configuration, we conducted 20 independent runs for each instance and reported the average cumulative regret (Figure 1) and runtime per round (Figure K.1) for each algorithm. The error bars in Figure 1 and K.2 represent the standard deviations (1-sigma error). We have omitted the error bars in Figure K.1 because they are minimal.

In the uniform reward setting where r ti ' 1 , the combinatorial optimization step to select the assortment simply involves sorting items by their utility estimate. In contrast, in the non-uniform reward setting, rewards are sampled from a uniform distribution in each round, i.e., r ti ' Unif p 0 , 1 q . For combinatorial optimization in this setting, we solve an equivalent linear programming (LP) problem that is solvable in polynomial-time [47, 21]. The experiments are run on Xeon(R) Gold 6226R CPU @ 2.90GHz (16 cores).

Figure K.1 presents additional empirical results on the runtime per round. Our algorithm OFU-MNL+ demonstrates a constant computation cost for each round, while the other algorithms exhibit a linear dependence on t . It is also noteworthy that the runtime for uniform rewards is approximately 10 times faster than that for non-uniform rewards. This difference arises because we use linear

programming (LP) optimization for assortment selection in the non-uniform reward setting, which is more computationally intensive.

Furthermore, Figure K.2 illustrates the cumulative regrets of the proposed algorithm compared to other baseline algorithms under uniform rewards with v 0 ' K { 5 . Since v 0 is proportional to K , an increase in K does not improve the regret. This observation is also consistent with our theoretical results.

L Technical Errors in Agrawal et al. [5]

In this section, we discuss the technical errors in Agrawal et al. [5]. There are two main significant errors: There are mainly two significant errors:

1. Equation (16).

where X S t is a design matrix whose columns are the attribute vectors x ti of the items in the assortment S t and µ i p X J S t θ q ' P t p i | S t , θ q .

It appears that the authors may have intended to derive this equation using a first-order exact Taylor expansion. However, in MNL bandits, this equation generally does not hold. Consider a counterexample where x ti ' 0 , x tj ‰ 0 for j ‰ i , and θ ‹ ‰ θ t . Then, the left-hand side of (L.1) is equals to 0 (since x ti ' 0 ), but the right-hand side of (L.1) is not 0 , because the denominators of each µ i p X J S t θ ‹ q and µ i p X J S t θ t q differ. This equation only holds in special cases, such as when K ' 1 , which corresponds to the logistic bandit case. Equation (16) in Agrawal et al. [5] serves as the foundation for the entire proof in their paper. Consequently, all subsequent results derived from it are also incorrect.

2. Cauchy-Schwarz inequality in the regret analysis (Page 46)

When using the Cauchy-Schwarz inequality on the regret before applying the elliptical potential lemma, they indeed incur an additional ? K factor:

g

Hence, their regret should actually be ˜ O p d ? KT ` d 2 { κ q , which is worse than the result in our Theorem 2 (upper bound under uniform rewards) by a factor of K .

M Limitations

To determine the optimistic assortment in line 5 of Algorithm 1, we must calculate the optimistic utility α ti for each item i P r N s . Consequently, the computational cost for each round scales polynomially with N , the number of items. If N is very large, or infinite, our proposed algorithm becomes intractable. It is important to note that much of the existing literature on contextual bandits [2, 10, 24, 1, 20, 6-8, 16, 40, 41, 4, 44] also requires enumerating all items to select the item for each round.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section M in Appendix

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumption 1 and 2

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Secion 7 and Section K

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We have included the code in the supplementary material. After our paper is accepted, we will provide open access to the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 7 and Section K

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Figure 1 and K.2. We omitted the error bars in Figure K.1 because they are too minimal to be significant.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section K

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on theoretical results and therefore does not discuss societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the minimax optimal regret for a series of contextual MNL bandit problems, covering rewards—uniform and non-uniform cases and varying the value of the outside option $v_0$. The authors first derive the regret lower bound that provides explicit dependence on the key parameters. Then they propose efficient algorithm OFU-MNL+ that achieves a matching upper bound up to a logarithm term. Simulations were conducted to demonstrate the tight dependence of the derived bounds.\n\nSTRENGTHS:\n1. Despite many recent advances in contextual MNL bandit problems, however no previous studies have proven the minimax optimality. There has been a significant discrepancy between lower and upper regret bounds, particularly regarding the feature dimension $d$ and the maximum assortment size $K$. This paper closes this gap. The technical contribution is strong and sufficient. \n\n2. In spite of a theoretical paper, it is well written and is easy to follow. I enjoy reading the paper.\n\nWEAKNESSES:\n1. My major question is about Assumption 2, which requires the uniform lower bound for every item $i\\in {\\cal S}$ and any assortment set and all round $t$. Requiring $\\kappa$ to be a constant would be a strong assumption in practice. Since $\\min_i p_t(i|S,w) \\le 1/K$, $\\kappa$ would naturally depend on the size $K$. In this case, would it be possible to derive more tighter results by explicitly including $\\kappa$ in the lower and upper bounds? \n\nBelow are a few minor suggestions: \n\n2. I would suggest to introduce Section 4 (Problem Setting) before Section 3 (Existing Gap). Readers who are not familiar with the contextual MNL bandit problems might be confusing while reading Section 3. \n\n3. Section 3 overlaps with many discussions in the Introduction section. It might be helpful to shorten the discussion and add more discussions on (1) why proving the new lower and upper bound results is non-trivial; (2) providing some proof outlines on the key steps used in the proof. \n\n4. In the Introduction, line 31 on page 1, it is helpful to add some clarification on the definition of $v_0$. The current terminology \"utility of the outside option\" is not very clear. \n\n5. In line 162 on page 4, the authors mentioned that the set of items $[N]$ can vary over time. While I understand the assortment set $\\cal S$ could change over time, I am confused how $[N]$ could vary over time. The notation $[N]$ means $\\{1, \\ldots, N\\}$. What do you mean this set varies over time? Do you allow new items to enter the market or some items to be removed over time?\n\nQUESTIONS:\nsee Weakness\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies MNL bandit problem and proposes computationally efficient algorithms in both uniform and non-uniform settings. The authors present lower bounds for these two settings and show their results are minimax optimal up to logarithmic factors.\n\nSTRENGTHS:\n1. The paper achieves the minimax optimal results for the MNL bandit problem in both uniform and non-uniform settings, contributing new findings to the community.\n2. This paper introduces computationally efficient algorithms that enhance the computational complexity over prior works.\n3. Empirical evidence provided by the authors convincingly demonstrates the effectiveness of the proposed algorithms, strengthening the paper's claims with solid experimental validation.\n\nWEAKNESSES:\n1. My main concern is about the contribution and technical novelty of this paper. From the statistical perspective, the regret for the uniform setting does not advance beyond the results presented in Goyal and Perivier (2021) and the $\\kappa$-independent bound for the non-uniform setting has been achieved in Agrawal et al. (2023) (A comparison with Agrawal et al. (2023) in the table would be beneficial for highlighting any improvement). The main contribution of this paper is the improvement of the computational complexity. However, this enhancement comes from the online parameter estimation, which is almost the same as that in Zhang and Sugiyama (2023), with only minor modifications. As MNL bandits considered in this paper and MLogB bandits are quite similar, this modification is not significant.\n2. The claim that \"This complexity arises because the optimistic revenue...requiring a different analysis\" in Remark 2 seems overstated. The issue of assortment optimization to maximize optimistic revenue has been previously addressed by Oh and Iyengar (2021). The approach in the paper does not deviate from their method, which makes the purported contribution questionable.\n3. The presentation of lower bounds for both settings to demonstrate result optimality is appreciated, yet the inclusion of a problem-dependent lower bound that illustrates the influence of the problem-specific parameter $\\kappa$ would have added significant value, as seen in studies on logistic bandits.\n\nQUESTIONS:\n1. Can the proposed algorithms be directly applied to achieve the optimal results for binary logistic bandits? It would be beneficial for the paper to discuss this applicability.\n2. The empirical performance shown in Figure 1 suggests UCB outperforms TS in non-uniform settings, contrary to the fact that TS usually achieves better empirical performance than UCB in the literature. Moreover, though OFU-MNL+ outperforms the two other baselines, all algorithms appear to incur linear regret in specific scenarios (e.g., (N=100, K=10, d=5, Uniform R)). Could the authors provide some intuition for these phenomena?\n\nLIMITATIONS:\n\\", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper closes the gap between upper and lower bounds for contextual MNL bandits over various scenarios, including for varying $v_0$ and uniform/non-uniform rewards. Furthermore, the paper proposes a computationally efficient algorithm that achieves such nearly optimal upper bound, which is validated numerically as well.\n\nSTRENGTHS:\n- Clearly, very well-written\n- Clear-cut contributions that close all the gaps remaining in the contextual MNL bandits\n- Numerical verification\n- Various technical contributions, mainly for the analysis but also for the algorithm design (Eqn. (6))\n\nWEAKNESSES:\n- Nothing noticeable; see questions\n\nQUESTIONS:\n**Questions**\n- In Remark 2, the authors mention that the optimistic bonus incurs exponential computational cost. But, disregarding the computational complexity, is there any advantage of doing so from the statistical/theoretical perspective? Somewhat related, although the authors have avoided the MLE-based approach for computational complexity, does using MLE with the intuitions provided here could lead to better regret guarantees in other dependencies?\n- In the regret upper bounds, there is a transient term $\\frac{d^2}{\\kappa}$. Is this an artifact of the proof? Or is there a hidden warmup phase done implicitly by the OFU-MNL+? Then, if one is okay with incurring such transient terms anyhow, is there potential for doing even better through forced exploration-type approaches (e.g., [1,2] for logistic bandits)?\n- Although the $1$-boundedness of $w^\\star$ is standard in MNL bandits, if we relax this to $B$-boundedness for some $B > 0$, then does this imply that $\\kappa$ may scale exponentially in $B$, like logistic bandits [3]?\n- Can the lower bound in Theorem 1 be written as a local minimax lower bound (e.g., Theorem 2 of [3]) centered around a specific instance? This seems like a better presentation than the current representation, which seems to be a generic minimax, where the max is over all possible problem instances. Also, the lower-bound construction, although inspired by prior works on MNL bandits, resembles a bit of [3] as well. I encourage the authors to check Appendix D of [3], especially the Averaging Hammer technique, and if applicable, include some discussions on this as well. Such instance-specific local minimax lower bound may make the result stronger.\n\n\n**Suggestions**\n- In the main text, for all the theorem statements, please include (hyper)ref for where the full proof is in the Appendix.\n- Although a bit complex, for practitioners, it might be beneficial to include the precise constants for the $\\beta_t(\\delta)$ in Lemma 1 (or point to the Appendix with the constants). A follow-up question: for the experiments, do the authors use the precise parameter as specified theoretically?\n- Might be beneficial to move the problem setting to earlier part of the paper, but not too sure here.\n\n\n[1] https://arxiv.org/abs/2404.06831\n\n[2] https://arxiv.org/abs/2202.02407\n\n[3] https://proceedings.mlr.press/v130/abeille21a.html\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the contextual multinomial logit bandit problem. In this problem a learner interacts with the environment over a sequence of rounds. At each round, the learner receives features for $N$ arms and decides to select up-to $K$ of them (called an assortment) and play them. The environment might select one of these arms with a probability determined by a multinomial distribution and the learner receives a reward for the selected arm. The goal is to select the optimal assortment at each round that minimizes the cumulative regret of the algorithm.\n\nThe authors design algorithms for cases when the rewards are uniform (same for each arm) and non-uniform. They also provide matching lower bounds (for large enough $T$). Finally numerical experiments are performed to demonstrate the practical performance of the algorithm.\n\nSTRENGTHS:\nThe authors do a great job of explaining the problem statement and prior work and position their work appropriately. I believe this is a very important direction since bandit algorithms have recently started getting popular in practical settings where the assumption of linear rewards no longer holds. Getting optimal regret guarantees for such non-linear models has gained good attention recently and the paper contributes to this in a clear non-trivial way.\n\nIt's great to see that the authors prove matching lower bounds for the uniform case (for general $v_0$) and for non-uniform case (for bounded $v_0$). This adds further credibility to their algorithm and the upper bound.\n\nWEAKNESSES:\nI am not an expert on the MNL problem but I was wondering if the algorithm offers anything new to the logistic bandit setting. Recently lot of work has happened in obtaining computationally efficient algorithms with optimal regret (independent of a problem non-linearity parameter $\\kappa$). It would be good to see if anything new can be said about that setting using the results in this paper. Also, empirical comparisons on both regret and time complexity (for logistic bandit i.e. K=1 as well as larger K) can be a good addition to the paper.\n\nMinor comment - Typo on line 183. The correct definition should be $S_t^\\star = \\arg\\max_{S\\in \\mathcal{S}} R_t(S, w^\\star)$.\n\nMy expertise in this topic is limited and I will wait to see what the other reviewers think as well.\n\nQUESTIONS:\nIs the problem dependent constant $\\kappa$ in line 189 same as the non-linearity $\\kappa$ defined in prior work for logistic bandits (See [4,20,21]). It seems to be slightly differently defined here so I was curious to see if it meant the same at least in the logistic case.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of sequentially picking assortments of items in an online manner over a horizon of $T$ rounds, where a user then selects from the items or not according to a multinomial logit model with a fixed unknown preference vector.  The items are represented as (known) feature vectors which can change adversarially from round to round.  The authors prove tighter cumulative regret lower bounds for both uniform and non-uniform cases and propose an algorithm with matching upper bounds.  The authors account for the probability of a user not choosing from the given options in the bounds.  The authors also run experiments to compare their algorithm to baselines.\n\nSTRENGTHS:\n### Major\n- Improved lower bounds for contextual MNL bandits in the uniform-reward setting, accounting for the impact of the probability of the null action.\n- improved lower bounds also in the non-uniform reward setting.\n- A tractable algorithm is proposed with matching regret upper bounds for both uniform and non-uniform reward setting\n\n### Minor\n- The authors include experiments for non-uniform rewards, showing significant improvements in both run-time and cumulative regret for included baselines.\n- Good discussions and remarks on effects of problem setup, such as $v_0$ and whether rewards are uniform or not on regret bounds, and relations with prior works (among those discussed to some length). \n- Overall I found the writing to be good.  Table 1 is helpful.\n\nWEAKNESSES:\n### Minor\n1. Zhang and Luo “Contextual Multinomial Logit Bandits with General Value Functions” [48] is only mentioned once in line 128 in a sentence about past works claiming $K$-independent regret lower bounds while not accounting for $v_0$ (with the next sentence pointing out issue that several lower bounds used $v_0 = K$.  However, in [48] they propose algorithms and show regret bounds for a harder problem (adversarial contexts and adversarial rewards and for more general settings than MNL models) that avoid $\\kappa$ dependence in the leading term, though for the simpler problem studied in the current paper may have loose assortment size $K$ dependence.  In [48] it looks to me like they consider $v_0=1$ not $v_0=K$ as implied by the current paper.  Thus, while [48] is cited, to me it seems to be improperly cited and the results should be more thoroughly discussed, and unless some justification should be included in Table 1.\n2. State of the art upper bounds for non-contextual MNL are not discussed.   Non-contextual MNL lower bounds are included in Table 1 and discussed, but unless I overlooked it did not see anything on upper bounds, even from papers like Agrawal et al. (2019) “MNL-bandit...” [8] that were discussed for lower bounds.  (Or do the contextual upper bounds with $d=N$ recover them in current state of the art?)\n3. Agrawal et al. “A tractable ..” 2023 [5] is cited in a few places regarding the problem setting (like in line 186 for assumptions 1 and 2, line 282 for $v_0$, …), but not included in Table 1 or discussed in detail.  They achieve improved regret upper bound dependence on $\\kappa$ from prior works using a reportedly tractable algorithm.  Agrawal et al. [5] consider a (contextual) combinatorial assortment problem using an MNL model with adversarial contexts and uniform rewards and $v_0 = 1$. \n4. Choi et al. “Cascading Contextual Assortment Bandits” in NeurIPS 2023 is not cited.  How would their results compare?  Their regret upper bound (specialized to the non-cascading setting you consider) looks competitive for $v_0=1$ and uniform reward setting.  Another cascading assortment paper by Cao and Sun “Tiered Assortment: Optimization and Online Learning” in Management Science is also not cited, though they acknowledge in their Remark 2 that in the special case of a single tier, their result matches Agrawal et al. (2019) “MNL-bandit...” [8].  \n5. The authors consider the parameter vector $\\textbf{w}$ bounded by $1$ instead of bounded by a known constant $S$ like in previous works (such as Agrawal et al. 2023 [5] and Lee et al. 2024 [32]).  As pointed out by Leet et al. 2024 [32], “These quantities [$\\kappa$] can scale exponentially in S in the worst-case (Faury et al., 2020).” How would the results be impacted accounting for such a known bound instead of setting it to 1?\n6. A clearer discussion on the computational complexity of the methods listed as “intractable” (in particular Goyal and Perivier 2022 [24]) would be better, both worst-case and (if not included in experiments) conjecture whether where they would have been in Fig. I.1 (eg even for those instances would it have been so slow as to be off the charts?).  \n\n### Very Minor\n- [24] cites an arxiv version of a NeurIPS 2022 paper\n\nQUESTIONS:\n(some questions are included above in comments made in “Weaknesses”)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}