Efficient Multivariate Bandit Algorithm with Path Planning

1 INTRODUCTION . Multi-Armed Bandit ( MAB ) problem , is widely studied in probability theory and reinforcement learning which dates back to clinical trial studies by Thompson Thompson ( 1933 ) . Robbins Robbins ( 1952 ) formulated the setting in 1952 : it includes a learner having K arms ( options/choices ) to explore given little knowledge about the properties of each arm . At each step t in [ 1 , T ] , the learner chooses an arm i and receives a reward Xt from the choice under the purpose to minimize the regret as well as maximize cumulative reward . Binomial bandit is the most common bandit formats by restricting the rewards being binary ( Xt ∈ { 0 , 1 } ) . The solution of MAB problem involves balancing between acquiring new knowledge ( exploration ) and utilizing existing knowledge ( exploitation ) , to make arm selection at each round t based on the state of each arm . The upper confidence bound ( UCB ) algorithm was demonstrated as optimal solution to manage regret bound in the order of O ( log ( T ) ) Lai & Robbins ( 1985 ) Lai et al . ( 1987 ) Agrawal ( 1995 ) Auer et al . ( 2002 ) . In online experiment , Thompson Sampling ( TS ) algorithm attracts a lot of attention due to its simplicity at implementation and resistance in batch updating . TS algorithm for binomial bandit could achieve optimal regret bound as well Kaufmann et al . ( 2012 ) . Many modern online applications ( e.g . UI Layout ) have configuration involving multivariate dimensions to be optimized , such as font size , background color , title text , module location , item image etc. , each dimension contains multiple options Hill et al . ( 2017 ) Nair et al . ( 2018 ) . In this paper , we call it Multivariate-MAB problem . The exploring space faces exponentially exploding number of possible configurations as dimensions are added into the decision making . TS algorithm is reported to convergences slowly to the optimal solution Hill et al . ( 2017 ) when dealing with MultivariateMAB problem . To speed up convergence , one common enhanced TS solution is to model the expected reward as general linear model ( TS-GLM ) Chu et al . ( 2011 ) Bubeck et al . ( 2012 ) Scott ( 2010 ) by probit/logit link function with m-way dimension interaction features . TS-GLM gives up the ability to fit certain complex interactions , in exchange for focusing on lower-dimensional parameter space and achieves better solution . However , updating derived posterior sampling algorithm in TS-GLM demands imputing the multivariate coefficients and creates computation burden at each iteration Scott ( 2010 ) Scott ( 2015 ) Hill et al . ( 2017 ) . To release such burden , Daniel et . al . Hill et al . ( 2017 ) proposed Hill-climbing multivariate optimization Casella & Berger ( 2002 ) for TS-GLM , and recognized it obtained faster convergence speed with polynomial exhaustive parameter space . Different from TS-GLM , our proposal framework called ” Path Planning ” ( TS-PP ) is quite straightforward to combat the curse of dimensionality by a serial processes that operates sequentially and focuses on one dimension at each component process . Further more , it treats arm reward naturally with m-way dimension interaction by m-dimensional joint distribution . Our novelty includes : ( a ) modeling arm selection procedure under tree structure . ( b ) efficient arm candidates search strategies under decision graph/trees . ( c ) remarkable convergence performance improvement by straightforward but effective arm space pruning . ( d ) concise and fast reward function posterior sampling under beta-binomial even with m-way dimension interaction consideration . Compare to TS-GLM , TS-PP avoids deriving complex and slow posterior sampling in GLM , while still effectively leveraging the m-way dimension interactions and achieving even better performance by reducing arm space with efficient search strategies . This paper is organized as follows : We first introduce the problem setting and notation ; then we explain our approach in details , and further discuss the differences among several variations ; we also examine the algorithm performance in simulated study and concludes at the end . 2 MULTIVARIATE-MAB PROBLEM FORMULATION . We start with the formulation of contextual multivariate MAB : the sequential selection decision of layout ( e.g . web page ) A , which contains a template with D dimensions and each dimension contains N options , under context C ( e.g . user preference ) for the purpose of minimizing the expected cumulative regret ( R ) . For each selected layout A , a reward X shall be received from the environment . Here only binary reward ( X ∈ { 0 , 1 } ) is discussed , but our approach can be extended to categorical/numeric rewards as well . In the layout A template , there exists Ni alternative options for dimension i and fi is denoted as the selected option . We further assume Ni = N in our following description for simplicity purpose . The chosen layoutA can be denoted as [ f1 , . . . , fD ] ∈ { 1 , . . . , N } D , and it utilizes the notation A [ i ] referring fi . Context C includes extra environment information that may impact layout ’ s expected reward ( e.g . device type , user segments , etc. ) . At each step t = 1 , . . . , T , the bandit algorithm selects arm At from ND search space with the consideration of the revealed context Ct in order to minimize the cumulative regret over T rounds : RT = T∑ t=1 ( E ( X|A ? t , Ct ) − E ( X|At , Ct ) ) where A ? t stands for the best possible arm at step t. Generally , RT is on the order O ( √ T ) under linear payoff settings Dani et al . ( 2008 ) Chu et al . ( 2011 ) Agrawal & Goyal ( 2013 ) . Although the optimal regret of non-contextual Multivariate MAB is on the order O ( logT ) Lai & Robbins ( 1985 ) . In this paper , we focus on categorical-contextual multivariate MAB where C are purely categorical variables . By solving multivariate MAB independently for each combination of C ( assuming not too many ) , it is trivial to show that the optimal regret bound of RT is still O ( log ( T ) ) . Without loss of generalization , we set context feature as constant and ignore C in the following discussion . 3 RELATED WORK . 3.1 PROBABILISTIC MODEL FOR MULTIVARIATE-MAB . To model the multivariate bandit reward of layout At , we denote the features combining A and interactions within A ( possibly non-linearly ) as BA ∈ RM with length M . The BA could involve only upto m-way dimension interactions of A ( O ( Nm ) ) instead of capturing all ( O ( ND ) ) possible interactions . The linear model with pairwise interactions is as follows : B > Aµ = µ 0 + D∑ i=1 µ1i ( A ) + D∑ i=1 D∑ j=i+1 µ2i , j ( A ) ( 1 ) where µ are fixed but unknown parameter coefficients . In above function , it contains common bias term µ0 , weights for each dimension of layout µ1i ( A ) and 2-way dimension interactions of layout µ2i , j ( A ) . The sub-indexes i and j are referring dimension i and j correspondingly . Under the GLM setting , g ( P ( X = 1|BA ) ) = g ( p ) = B > Aµ , where p is the success rate of reward X and g is the link function that can either be the inverse of normal CDF Φ−1 ( p ) as a probit model or the ln ( p1−p ) as a logit model . For given BA , the likelihood of reward X would be ( Φ ( B > Aµ ) ) X ( 1 − Φ ( B > Aµ ) ) ( 1−X ) or ( exp ( −B > Aµ ) + 1 ) −X ( exp ( B > Aµ ) + 1 ) ( −1+X ) for probit or logit model respectively . The posterior sampling distribution of reward is its likelihood integrates with some fixed prior distribution of weights µ. Updating the posterior , at step t , requires solving GLM for µ from cumulative historical rewards Ht−1 = [ ( X1 , A1 ) , . . . , ( Xt−1 , At−1 ) ] , which is disturbing and creates computation burden with time . Daniel et . al . Hill et al . ( 2017 ) proposed MVT2 by assuming probit model with interactions between dimensions ( Equation 1 ) and employing Hill-climbing multivariate optimization to achieve faster convergence speed . 3.2 THOMPSON SAMPLING . Thompson sampling ( TS ) Russo et al . ( 2018 ) is widely adapted in solving bandit and reinforcement learning problems to balance between exploitation and exploration . It utilizes common Bayesian techniques to form posterior distribution of rewards , hence allocates traffic to each arm proportional to probability of being best arm under posterior distribution . Normally we handle binary response as binomial distribution with Beta prior Beta ( α0 , β0 ) to form posterior distribution Beta ( αk + α0 , βk + β0 ) , where αk and βk are the number of successes and failures it has been encountered so far at arm k , as well as α0 and β0 are prior parameters and would been set as 1 for uniform prior . At selection stage in round t , it implicit allocates traffic as follows : simulates a single draw of θk from posterior ( Beta ( αk + α0 , βk + β0 ) ) for each arm k and the arm k∗ = arg maxk ( θk ) out of all arms will be selected . At update stage , it collects reward X t ∈ { 0 , 1 } and the reward is used to update hidden state ( αk∗ , βk∗ ) of selected arm . Practically , to solve Multivariate-MAB problem , algorithm ND−MAB directly adopts TS in selection out ofND arms , while algorithm D-MABs decomposes Multivariate-MAB intoD ( dimensions ) sub-MABs and implements TS in selection out ofN arms for each MAB ( dimension ) independently . We would discuss more details of the two algorithms in following sections . 3.3 MONTE-CARLO TREE SEARCH . Monte-Carlo Tree Search ( MCTS ) Browne et al . ( 2012 ) Chaslot et al . ( 2008a ) is a best-first heuristic search algorithm to quickly locate the best leaf node in a tree structure . In game tree problem , it achieves great success especially when the number of leafs is large . Generally , each round of MCTS consists of four steps Chaslot et al . ( 2008b ) : selection , expansion , simulation and back-propagation . Essentially , a simplified case only need selection and back-propagation steps . At selection step of MCTS , it starts from root of the tree and adopts a selection policy to select successive child nodes until a leaf node L is reached . The back-propagation step of MCTS involves using the reward to update information ( like hidden states ) in the nodes on the path from selected leaf to root . In artificial intelligence literature , the most successful ( MCTS algorithm ) UCT utilizes UCB Kocsis & Szepesvári ( 2006 ) as its node selection policy . Applying TS as node selection policy in MCTS ( TSMCTS ) is not well investigated in literature Imagawa & Kaneko ( 2015 ) from our best knowledge . By introducing the hierarchical dimensional structure of bandit arms , we can build tree structure over bandit arms and deploy MCTS with TS techniques for arm selection . We prefer TS node selection policy due to its resistance in performance for batch update situation . Inspired by this idea , we establish TS Path Planning algorithm to solve Multivariate-MAB problem .

Summary: This paper approaches the problem of exploding arms in multivariate multi-armed bandits. To solve this problem, the authors suggest an approach that uses Thompson sampling for arm selection and decision trees and graphs (inspired from Monte-Carlo tree search) to model reward success rates. They propose to versions of this path planning procedure and 2 version inspired from Hill-climbing methods. They validate their claims on a simulation showing better performance and faster convergence, and providing an extensive analysis of the results.