UCB-based Algorithms for Multinomial Logistic Regression Bandits

p T ) with small dependency on problem- dependent constants that can otherwise be arbitrarily large and lead to loose regret bounds . We present numerical simulations that corroborate our theoretical results . 1 Introduction . Linear stochastic bandits provide simple , yet commonly encountered , models for a variety of sequential decision-making problems under uncertainty . Specifically , linear bandits generalize the classical multi-armed bandit ( MAB ) problem of K arms that each yields reward sampled independently from an underlying distribution with unknown parameters , to a setting where the expected reward of each arm is a linear function that depends on the same unknown parameter vector Dani et al . ( 2008 ) ; Abbasi-Yadkori et al . ( 2011 ) ; Rusmevichientong and Tsitsiklis ( 2010 ) . Linear bandits have been successfully applied over the years in online advertising , recommendation services , resource allocation , etc . Lattimore and Szepesvári ( 2020 ) . More recently , researchers have explored the potentials of such algorithms in more complex systems , such as in robotics , wireless networks , the power grid , medical trials , e.g. , Li et al . ( 2013 ) ; Avner and Mannor ( 2019 ) ; Berkenkamp et al . ( 2016 ) ; Sui et al . ( 2018 ) . However , linear bandits fail to model a host of other applications . This has called for extensions of linear bandits to a broader range of reward structures beyond linear models . One of the leading lines of work addressing these extensions relies on the Generalized Linear Model ( GLM ) framework of statistics . In GLMs the expected reward associated with an arm x is given by µ ( ✓̄Tx ) , where ✓̄ 2 Rd is the system unknown parameter and µ is a non-linear link function . Specifically , logistic bandits , that are appropriate for modeling binary reward structures , are a special case of generalized linear bandits ( GLBs ) with µ ( x ) = ( 1 + exp ( x ) ) 1 . UCB-based algorithms for GLBs were first introduced in Filippi et al . ( 2010 ) ; Li et al . ( 2017 ) ; Faury et al . ( 2020 ) . The same problem , but with a Thompson Sampling- ( TS ) strategy was also studied in Abeille et al . ( 2017 ) ; 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Russo and Van Roy ( 2013 , 2014 ) ; Dong and Van Roy ( 2018 ) . Beyond GLMs , an even more general framework for modeling reward is the semi-parametric index model ( see for example Yang et al . ( 2017 ) ; Gamarnik and Gaudio ( 2020 ) for a list of applications in statistics ) . A semi-parametric index model relates the reward y 2 R and the action/arm x 2 Rd as y = µ ( ✓̄T1 x , ✓̄ T 2 x , . . . , ✓̄ T Kx ) + ✏ , where µ : RK ! R and ✓̄1 , . . . , ✓̄K 2 Rd are K system ’ s unknown parameters . GLBs are special cases of this for K = 1 , also known as single-index models ( SIM ) in statistics . In this paper , we formulate an extension of the problem of binary logistic bandits ( i.e. , a special case of SIM ) to multinomial logit ( MNL ) bandits , a special case of multi-index models ( MIM ) to account for settings with more than two possible outcomes on the user choices ( K 1 ) . For this model , we present an algorithm and corresponding regret bound . Our algorithmic and analytic contribution is in large inspired by recent exciting progress on binary logistic bandits by Faury et al . ( 2020 ) . To motivate MNL bandits , consider ad placement . When an ad is shown to a user , the user may have several options to react to the ad . For example , she can choose to 1 ) click on the ad ; 2 ) click on “ show me later ” ; 3 ) click on “ never show me this ad ” ; 4 ) not click at all , etc . The user selects each of these options based on an unknown probability distribution that inherently involves linear combinations of the selected feature vector denoting the ad and unknown parameters denoting the user ’ s preferences about the ad . In this setting , each option is associated with a specific notion of reward . The agent ’ s goal is to determine ads with maximum expected rewards to increase the chance of a successful advertisement . Outline . In Section 1.1 , we formally define the problem . In Sections 2.1 , 2.2 and 2.4 , we elaborate on the challenges that the generalization of the Logistic-UCB-1 by Faury et al . ( 2020 ) to the settings with MIM rewards brings to our theoretical analysis . We then summarize our proposed MNL-UCB in Algorithm 1 and provide a regret bound for it in Section 2.5 . In Section 3 , we present a detailed discussion on the challenges and computation of necessary problem-dependent constants . Finally , we complement our theoretical results with numerical simulations in Section 4 . Notation . We use lower-case letters for scalars , lower/upper-case bold letters for vectors/matrices . kxk2 denotes the Euclidean norm and x T y inner product . ij denotes the Kronecker delta , i.e. , ij = 1 if i 6= j and ij = 0 if i = j. A⌦B denotes the Kronecker product . For square matrices A and B , we use A B to denote B A is positive semi-definite . We denote the minimum and maximum eigenvalues of A by min ( A ) and max ( A ) . For A ⌫ 0 , the weighted 2-norm of ⌫ with respect to A is defined byk⌫k A = p ⌫TA⌫ . For positive integers n and m n , [ n ] and [ m : n ] denote the sets { 1 , 2 , . . . , n } and { m , . . . , n } , respectively . For any ⌫ 2 RKd , ⌫̄i = ⌫ [ ( i 1 ) d+1 : d ] 2 Rd denotes the vector containing the i-th set of d entries of ⌫ . We use 1 and ei to denote the vector of all 1 ’ s and the i-th standard basis vector , respectively . Finally , we use Õ for big-Oh notation that ignores logarithmic factors . 1.1 Problem formulation . Reward Model . The agent is given a decision set1 D ⇢ Rd . At each round t , the agent chooses an action xt 2 D and observes the user purchase decision yt 2 [ K ] [ { 0 } . Here , { 0 } denotes the “ outside decision ” , which means the user did not select any of the presented options . The agent ’ s decision at round t is based on the information gathered until time t , which can be formally encoded in the filtration Ft : = ( F0 , ( { xs , ys } t 1s=1 ) ) , where F0 represents any prior knowledge . Let each option i 2 [ K ] be associated with an unknown vector ✓̄⇤i 2 Rd and let ✓⇤ = [ ✓̄ T ⇤1 , ✓̄ T ⇤2 , . . . , ✓̄ T ⇤K ] T 2 RKd . The user ’ s choice of what to click on is given by a multinomial logit ( MNL ) choice model . Under this model , the probability distribution of the user purchase decision is given by P ( yt = i|xt , Ft ) : = 8 > > < > > : 1 1+ PK j=1 exp ( ¯✓ T ⇤jxt ) , if i = 0 , exp ( ¯✓ T ⇤ixt ) 1+ PK j=1 exp ( ¯✓ T ⇤jxt ) , if i 2 [ K ] . ( 1 ) When the user clicks on the i-th option , a corresponding reward ⇢i is revealed to the agent and we set ⇢0 = 0 . Then , the expected reward observed by the agent when she plays action xt is 1Our results extend easily to time-varying decision sets . E [ Rt|xt , Ft ] = PK j=1 ⇢j exp ( ¯✓ T ⇤jxt ) 1+ PK j=1 exp ( ¯✓ T ⇤jxt ) = ⇢T z ( xt , ✓⇤ ) , where ⇢ = [ ⇢1 , ⇢2 , . . . , ⇢K ] T , z ( xt , ✓⇤ ) = [ z1 ( xt , ✓⇤ ) , z2 ( xt , ✓⇤ ) , . . . , zK ( xt , ✓⇤ ) ] T , and zi ( xt , ✓⇤ ) = P ( yt = i|xt , Ft ) , 8i 2 [ K ] . ( 2 ) Note that E [ Rt|xt , Ft ] = µ ( ✓̄ T ⇤1x , ✓̄ T ⇤2x , . . . , ✓̄ T ⇤Kx ) is not directly a generalized linear model , i.e. , a function µ ( ✓̄T⇤ xt ) , but rather it is a multi-index model , where µ : RK ! R. Goal . Let T be the total number of rounds and x⇤ be the optimal action that maximizes the reward in expectation , i.e. , x⇤ 2 argmaxx2D ⇢T z ( x , ✓⇤ ) . The agent ’ s goal is to minimize the cumulative pseudo-regret defined by RT = PT t=1 ⇢ T z ( x⇤ , ✓⇤ ) ⇢T z ( xt , ✓⇤ ) . 1.2 Contributions . We study MNL logistic regression bandits , a generalization of binary logistic bandits , that address applications where the number of outcomes that can be selected by the user is larger than two . The probability of selecting any possible K + 1 > 2 options ( +1 stands for the ‘ not click ’ outcome aka “ outside decision ” ) is modeled using a multinomial logit ( MNL ) model . For this problem : • We identify a critical parameter , which we interpret as the degree of ( non ) -smoothness ( less smooth for larger values of ) of the MNL model over the agent ’ s decision set . We prove that scales exponentially with the size of the agent ’ s decision set creating a challenge in the design of low-regret algorithms , similar to the special binary case previously studied in the literature . • We develop a UCB-type algorithm for MNL logistic regression bandits . At every step , the algorithm decides on the inclusion of a K-tuple of parameter vectors ✓̄⇤1 , . . . , ✓̄⇤K in the confidence region in a way that captures the local smoothness of the MNL model around this K-tuple and past actions . We show that this is critical for the algorithm ’ s favorable regret performance in terms of . • Specifically , we prove that the regret of our MNL-UCB scales as Õ ( dK p p T ) . Instead , we show that a confidence ellipsoid that fails to capture local dependencies described above results in regret that scales linearly with rather than with p . Moreover , our regret bound scales optimally in terms of the number of options K. • We propose an improved algorithm that achieves a regret bound with problem-dependent constant being pushed into a second order term that vanishes quickly . • We complement our theoretical results with numerical simulations and corresponding discussions on the performance of our algorithm .

Summary: This paper considers a version of stochastic contextual bandits where the rewards are drawn from a discrete distribution governed by a multionimal logit model. This setting is motivated by applications in online advertising where the user may give various feedbacks in the face of a recommendation beyond a binary ‘clicked’ or ‘did not click’ – e.g. ‘show me again later’, ‘never show me again’. The authors propose and analyse a UCB algorithm for this problem with the confidence sets being based on the work of Faury et al. (2020, ICML) and the regret analysis being based on that of Abbasi-Yadkori et al. (2011, NeurIPS). The particular advantage of utilising the concentration results of Faury et al is that the regret of the resulting algorithm scales favourably (square root order) with respect to a parameter $\kappa$ characterising the smoothness of the logit function around the true parameter. A further algorithm whose dependence on $\kappa$ is relegated to lower order terms w.r.t. the horizon is also developed – however this approach is computationally intractable. In the related (binary) logistic bandit setting, some popular algorithms have been shown to have exponential order dependence on this parameter and the design of algorithms which are robust to its value is useful. It may be useful to note that this version of a multinomial bandit is different from the combinatorial problem in the dynamic assortment selection literature (e.g. Agrawal et al (2017, 2019)) where the different outcomes represent different items.