Title: EFFICIENT MULTIVARIATE BANDIT ALGORITHM WITH PATH PLANNING

Abstract: In this paper, we solve the arms exponential exploding issues in multivariate Multi-Armed Bandit (Multivariate-MAB) problem when the arm dimension hierarchy is considered. We propose a framework called path planning (TS-PP) which utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts Thompson sampling (TS) for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. For our best acknowledge, we are the first to solve Multivariate-MAB problem using graph path planning strategy and deploying alike Monte-Carlo tree search ideas. Our proposed method utilizing tree models has advantages comparing with traditional models such as general linear regression. Simulation studies validate our claim by achieving faster convergence speed, better efficient optimal arm allocation and lower cumulative regret.

1. INTRODUCTION:

Multi-Armed Bandit (MAB) problem, is widely studied in probability theory and reinforcement learning which dates back to clinical trial studies by Thompson Thompson (1933) . Robbins Robbins (1952) formulated the setting in 1952: it includes a learner having K arms (options/choices) to explore given little knowledge about the properties of each arm. At each step t in [1, T ], the learner chooses an arm i and receives a reward X t from the choice under the purpose to minimize the regret as well as maximize cumulative reward. Binomial bandit is the most common bandit formats by restricting the rewards being binary (X t ∈ {0, 1}). The solution of MAB problem involves balancing between acquiring new knowledge (exploration) and utilizing existing knowledge (exploitation), to make arm selection at each round t based on the state of each arm. The upper confidence bound (UCB) algorithm was demonstrated as optimal solution to manage regret bound in the order of O(log(T)) Lai & Robbins (1985) Lai et al. (1987) Agrawal (1995) Auer et al. (2002) . In online experiment, Thompson Sampling (TS) algorithm attracts a lot of attention due to its simplicity at implementation and resistance in batch updating. TS algorithm for binomial bandit could achieve optimal regret bound as well Kaufmann et al. (2012) .

Many modern online applications (e.g. UI Layout) have configuration involving multivariate dimensions to be optimized, such as font size, background color, title text, module location, item image etc., each dimension contains multiple options Hill et al. ( 2017) Nair et al. (2018) . In this paper, we call it Multivariate-MAB problem. The exploring space faces exponentially exploding number of possible configurations as dimensions are added into the decision making. TS algorithm is reported to convergences slowly to the optimal solution Hill et al. (2017) when dealing with Multivariate-MAB problem. To speed up convergence, one common enhanced TS solution is to model the expected reward as general linear model (TS-GLM) Chu et al. (2011) Bubeck et al. (2012) Scott (2010) by probit/logit link function with m-way dimension interaction features. TS-GLM gives up the ability to fit certain complex interactions, in exchange for focusing on lower-dimensional parameter space and achieves better solution. However, updating derived posterior sampling algorithm in TS-GLM demands imputing the multivariate coefficients and creates computation burden at each iteration Scott (2010)Scott ( 2015) Hill et al. (2017) . To release such burden, Daniel et. al. Hill et al. (2017) proposed Hill-climbing multivariate optimization Casella & Berger (2002) for TS-GLM, and recognized it obtained faster convergence speed with polynomial exhaustive parameter space.

Different from TS-GLM, our proposal framework called "Path Planning" (TS-PP) is quite straightforward to combat the curse of dimensionality by a serial processes that operates sequentially and focuses on one dimension at each component process. Further more, it treats arm reward naturally with m-way dimension interaction by m-dimensional joint distribution. Our novelty includes: (a) modeling arm selection procedure under tree structure. (b) efficient arm candidates search strategies under decision graph/trees. (c) remarkable convergence performance improvement by straightforward but effective arm space pruning. (d) concise and fast reward function posterior sampling under beta-binomial even with m-way dimension interaction consideration. Compare to TS-GLM, TS-PP avoids deriving complex and slow posterior sampling in GLM, while still effectively leveraging the m-way dimension interactions and achieving even better performance by reducing arm space with efficient search strategies. This paper is organized as follows: We first introduce the problem setting and notation; then we explain our approach in details, and further discuss the differences among several variations; we also examine the algorithm performance in simulated study and concludes at the end.

2. MULTIVARIATE-MAB PROBLEM FORMULATION:

We start with the formulation of contextual multivariate MAB: the sequential selection decision of layout (e.g. web page) A, which contains a template with D dimensions and each dimension contains N options, under context C (e.g. user preference) for the purpose of minimizing the expected cumulative regret (R).

For each selected layout A, a reward X shall be received from the environment. Here only binary reward (X ∈ {0, 1}) is discussed, but our approach can be extended to categorical/numeric rewards as well. In the layout A template, there exists N i alternative options for dimension i and f i is denoted as the selected option. We further assume N i = N in our following description for simplicity purpose. The chosen layout A can be denoted as [f 1 , . . . , f D ] ∈ {1, . . . , N } D , and it utilizes the notation A[i] referring f i . Context C includes extra environment information that may impact layout's expected reward (e.g. device type, user segments, etc.).

At each step t = 1, . . . , T , the bandit algorithm selects arm A t from N D search space with the consideration of the revealed context C t in order to minimize the cumulative regret over T rounds:

R T = T t=1 (E(X|A t , C t ) -E(X|A t , C t ))

where A t stands for the best possible arm at step t. Generally, R T is on the order O( √ T ) under linear payoff settings Dani et al. (2008 )Chu et al. (2011 )Agrawal & Goyal (2013) . Although the optimal regret of non-contextual Multivariate MAB is on the order O(logT ) Lai & Robbins (1985) . In this paper, we focus on categorical-contextual multivariate MAB where C are purely categorical variables. By solving multivariate MAB independently for each combination of C (assuming not too many), it is trivial to show that the optimal regret bound of R T is still O(log(T )). Without loss of generalization, we set context feature as constant and ignore C in the following discussion.

3. RELATED WORK:



3.1. PROBABILISTIC MODEL FOR MULTIVARIATE-MAB:

To model the multivariate bandit reward of layout A t , we denote the features combining A and interactions within A (possibly non-linearly) as B A ∈ R M with length M . The B A could involve only upto m-way dimension interactions of A (O(N m )) instead of capturing all (O(N D )) possible interactions. The linear model with pairwise interactions is as follows:

EQUATION

where µ are fixed but unknown parameter coefficients. In above function, it contains common bias term µ 0 , weights for each dimension of layout µ 1 i (A) and 2-way dimension interactions of layout µ 2 i,j (A). The sub-indexes i and j are referring dimension i and j correspondingly.

Under the GLM setting, g(P (X = 1|B A )) = g(p) = B A µ, where p is the success rate of reward X and g is the link function that can either be the inverse of normal CDF Φ -1 (p) as a probit model or the ln( p 1-p ) as a logit model. For given B A , the likelihood of reward X would be

(Φ(B A µ)) X (1 -Φ(B A µ)) (1-X) or (exp(-B A µ) + 1)

-X (exp(B A µ) + 1) (-1+X) for probit or logit model respectively. The posterior sampling distribution of reward is its likelihood integrates with some fixed prior distribution of weights µ. Updating the posterior, at step t, requires solving GLM for µ from cumulative historical rewards H t-1 = [(X 1 , A 1 ), . . . , (X t-1 , A t-1 )], which is disturbing and creates computation burden with time. Daniel et. al. Hill et al. (2017) proposed MVT2 by assuming probit model with interactions between dimensions (Equation 1) and employing Hill-climbing multivariate optimization to achieve faster convergence speed.

3.2. THOMPSON SAMPLING:

Thompson sampling (TS) Russo et al. ( 2018) is widely adapted in solving bandit and reinforcement learning problems to balance between exploitation and exploration. It utilizes common Bayesian techniques to form posterior distribution of rewards, hence allocates traffic to each arm proportional to probability of being best arm under posterior distribution.

Normally we handle binary response as binomial distribution with Beta prior Beta(α 0 , β 0 ) to form posterior distribution Beta(α k + α 0 , β k + β 0 ), where α k and β k are the number of successes and failures it has been encountered so far at arm k, as well as α 0 and β 0 are prior parameters and would been set as 1 for uniform prior. At selection stage in round t, it implicit allocates traffic as follows: simulates a single draw of θ k from posterior (Beta(α k + α 0 , β k + β 0 )) for each arm k and the arm k * = arg max k (θ k ) out of all arms will be selected. At update stage, it collects reward X t ∈ {0, 1} and the reward is used to update hidden state (α k * , β k * ) of selected arm.

Practically, to solve Multivariate-MAB problem, algorithm N D -MAB directly adopts TS in selection out of N D arms, while algorithm D-MABs decomposes Multivariate-MAB into D (dimensions) sub-MABs and implements TS in selection out of N arms for each MAB (dimension) independently.

We would discuss more details of the two algorithms in following sections.

3.3. MONTE-CARLO TREE SEARCH:

Monte-Carlo Tree Search (MCTS)Browne et al. ( 2012) Chaslot et al. (2008a) is a best-first heuristic search algorithm to quickly locate the best leaf node in a tree structure. In game tree problem, it achieves great success especially when the number of leafs is large. Generally, each round of MCTS consists of four steps Chaslot et al. (2008b) : selection, expansion, simulation and back-propagation. Essentially, a simplified case only need selection and back-propagation steps. At selection step of MCTS, it starts from root of the tree and adopts a selection policy to select successive child nodes until a leaf node L is reached. The back-propagation step of MCTS involves using the reward to update information (like hidden states) in the nodes on the path from selected leaf to root. In artificial intelligence literature, the most successful (MCTS algorithm) UCT utilizes UCB Kocsis & Szepesvári (2006) as its node selection policy. Applying TS as node selection policy in MCTS (TS-MCTS) is not well investigated in literature Imagawa & Kaneko (2015) from our best knowledge.

By introducing the hierarchical dimensional structure of bandit arms, we can build tree structure over bandit arms and deploy MCTS with TS techniques for arm selection. We prefer TS node selection policy due to its resistance in performance for batch update situation. Inspired by this idea, we establish TS Path Planning algorithm to solve Multivariate-MAB problem.

4. APPROACH:

In this paper, we propose TS path planning algorithm (TS-PP) for Multivariate-MAB problem to overcome the exponential explosion in arm searching space of N D -MAB algorithm. Stimulated by MCTS idea, we utilize similar heuristic search strategy to locate best arm under a tree structure.

We call such tree structure as "decision tree" which is constructed purely by D dimensions. Notably there are D! decision trees constructed in different sequential dimension order over same leaf nodes and they assemble a "decision graph". Under a decision tree/graph, the arm selection procedure is decomposed into a serial processes of decision making that operates sequentially and focuses on value selection within one dimension per each process. At each sequential decision process, we would apply TS as successive child nodes (dimension value) selection policy. The sequential order of dimensions ("decision path") is determined by the path planning strategy.

Figure 1 shows an example of decision graph, decision tree as well as decision path. 

(f di , prePath(f di ))

, where prePath(f di ) represents all predecessor nodes of f di in red arrows (Figure 1b ). We denote prePath(f di ) as f d1:di-1 in our example, and path(f di ) as f d1:di . Based on the chain rule, the likelihood of arm [f d1:

d D ] is P (f d1:d D ) = D i=1 P (f di |f d1:di-1 ) ∝ P (f di |f d1:di-1 ) ∝ P (f d1:di ).

In practice, P (f d1:di ) could be represented by hidden states (α, β) from Beta distribution (binary rewards) for node f di (given its prePath(f di )) and node's states could be updated in back-propagation stage (like in MCTS). As in TS, the chance of arm f d1:d D being the best arm depends on P (f d1:d D ), hence it is also partially related with P (f d1:di ). Instead of sampling directly from posterior distributions of N D arms, sampling from distribution associated with each node f di could also provide guidance on value selection for dimension d i . Figure 1a utilizes decision graph to compactly represent D! decision trees. Once a decision path (red arrow in Figure 1a ) is determined, decision graph is degenerated to decision tree for detailed view. With such abstraction, we further extended the naive MCTS idea with several other path planning strategies. Sample

θ c ← Sample(A c ) 6: Select Arm A t ← arg max A c (θ c ) 7: Update History Rewards H t ← H t-1 ∪ (X t , A t ) 8: function SAMPLE(Path=[f d1:d L ] ) 9: Get α, β from Node(f d L |f d1:d L-1 ) 10: Sample θ ∼ Beta(α + α 0 , β + β 0 ) 11:

Return θ:

We understand that searching candidates in this way might be stacked in sub-optimal arms. To address this issue, we intentionally repeat our candidate searching for (S) times and re-apply TS tricks among these S candidates for final arm selection. Once the arm is chosen at step t, we would back-propagate the collected reward X t to update the hidden states of nodes (Node(f d L |f d1:d L-1 )) within all the possible paths from selected leaf to root in decision trees. Here the notation Node(f d L |f d1:d L-1 ) is used to load the node f d L 's (with prePath f d1:d L-1 ) hidden states in to memory, which corresponds the joint density P (f d1:d L ). It worth to note that any relative order of [f d1:d L ] represents the same joint distribution (with same hidden states). In practice, Node(f d L |f d1:d L-1 ) requires O(1) computation complexity. But it could also be implemented in O(T ) computation complexity for lazy back-propagation with cache memory saving.

4.2. PATH PLANNING PROCEDURE:

We propose four path planning procedures for candidates searching: Full Path Finding (FPF), Partial Path Finding (PPF), Destination Shift (DS) and Boosted Destination Shift (Boosted-DS). To construct arm-candidate under decision graph (figure 1a ), FPF starts from the root and sequentially optimizes dimensions one-by-one in a completely random order, which utilizes the depth-first search (DFS) strategy. With sticking with top-down flavor while extending the D-MABs, PPF utilizes the breath-first search (BFS) strategy with m-dimensional joint distribution independence (explained later) for all m-sub dimensions out of D. Finally, inspired by hill-climbing Casella & Berger (2002 ) Hill et al. (2017) , which start from a random initial arm (bottom node in decision graph) and optimize value for one dimension with all other dimension value fixed, such bottom-up flavor DS and the advanced version Boosted-DS would be discussed. The following explains four methods in details.

Full Path Finding FPF is the direct application of MCTS and describes DFS algorithm in graph search. Starting from top, FPF randomly picks a permutation of D dimensions denoting as [d 1:D ] with equal chance to construct a decision tree, and recursively applies TS policy from nodes on the path from root to leaf in that decision tree. It follows dimensions order to sequentially optimize value f di for each dimension d i . The upper index refers the optimized value for target dimension d i . Since we repeat FPF S times, each iteration c picks different decision tree (permutation of D dimensions) and construct one candidate A c . The computational complexity for full path finding is O(SN D) with S times searching and space complexity is O(N D ). For lazy back-propagation implementation, the computation and space complexity could be improved to O(SN DT ) and O(SDT ) separately.

Partial Path Finding In contrast, PPF describes a kind of BFS algorithm. A m th -partial path finding (PPFm) recursively applies TS policy from nodes on pre-path [d 1:(m-1) ] up to level m -1 in decision graph, then it simultaneously visits the remaining (D -m + 1) dimensions (un-visited nodes) in parallel at level m and apply TS policy correspondingly. Specifically, the D-MABs method is equivalent to PPF1, which adopts the dimension independent assumption. The Pseudo code in Algorithm 2 between line 5 and 10 illustrates a PPF2 algorithm, which assumes pairwise joint distribution independence. Mathematically, variables A and B are conditionally independent give C (A ⊥ B|C) if and only if P (A, B|C) = P (A|C)P (B|C), and would call joint distribution of (A, C) and (B, C) are independent. So pairwise dimensional joint distribution independence means Algorithm 2 Path Planning Procedures 1: procedure FULL PATH FINDING 2:

for index i = 1 → D in (random order) [d1:D] do 3: f * d i ← TS(tgtDim=di, prePath=[f * d 1 :d i-1 ]) and A c [di] = f * d i 4: Constructed Candidate A c 5: procedure PARTIAL PATH FINDING 6: Random Pick Dimension di ∈ [d1:D] 7: f * d i ← TS(tgtDim= di, prePath= ∅) and A c [di] = f * d i 8:

for index (j = 1 → D) and (j = i) do 9:

f * d j ← TS(tgtDim=dj, prePath=[f * d j ]) and A c [dj] = f * d j 10: Constructed Candidate A c 11: procedure DESTINATION SHIFT 12: Initial A c = random layout A 13: for k = 1 → K do 14: Random Pick Dimension di ∈ [d1:D] 15: f * d i ← TS(tgtDim=di, prePath=A c [-di]) and A c [di] = f * d i 16: Constructed Candidate A c 17: procedure BOOSTED DESTINATION SHIFT 18: Initial A c = random layout A 19: for k = 1 → K do 20: Random Pick Dimension di ∈ [d1:D] 21: f * d i ← bstTS(tgtDim=di, prePath=A c [-di]) and A c [di] = f * d i 22: Constructed Candidate A c 23: function TS(tgtDim = dL, prePath=[f d 1 :d L-1 ] ) 24: for f = 1 → N do 25: θ f ← Sample(Path=[f d 1 :d L-1 , f (d L ) ]) 26: Return f * d L ← arg max f (θ f ) 27: function BSTTS(tgtDim = dL, prePath=[f d 1 :d L-1 ] ) 28: for f = 1 → N do 29: µ 1 d L ← Sample(Path=[f d L ]) 30: for index i = 1 → L -1 do 31: µ 2 d i ,d L ← Sample(Path= [f d i , f d L ]) 32: θ f ← µ 1 d L + L-1 i=1 µ 2 d i ,d L 33: Return f * d L ← arg max f (θ f ) that d (1:D)-i ⊥ |d i for ∀i ∈ (1 : D)

, where d (1:D)-i ⊥ stands for all dimensions are independent except for dimension d i . Intuitively, PPF2 assumes pairwise interactions between dimensions, as it draws samples from pairwise dimensional joint distribution. Generally PPFm maps up to (m)-way interactions in regression model. The optimal computational complexity of PPF2 is O(SN D) with S times searching and space complexity is O(N D 2 ), if we only load all hidden states from top 2 levels of decision graph into memory.

Destination Shift DS randomly picks an initial arm (bottom node in decision graph) and performs Hill-climbing method cycling through all dimensions for K rounds. At each round k, we randomly choose an dimension d i to optimize and return the best dimension value f * di based on posterior sampling distribution condition on the rest of the dimension values prefixed (A c [-d i ]). We then use

f * di to generate A c (k) from A c (k-1) by A c (k-1) [d i ] = f * di . The computational complexity is O(SN K) and space complexity is O(N D ).

Boosted Destination Shift Boosted-DS utilizes bstTS instead of TS function for value optimization on each target dimension node d i . It extends our previous intuition that sampling from mdimensional joint distribution is 1-to-1 mapping to m-way interaction weights in regression model. Pseudo code in Algorithm 2 between line 17 and 22 describes 2 nd -Boosted-DS (Boosted-DS2) sampling strategy which follows equation 1 with pairwise (2 -way) interaction assumption. Instead of single draw from arm P (A c [-d i ], f di ), at round k with target dimension d i , it sums samples drawing from 1-way joint density (P (f i )) and all pairwise (2 -way) joint distributions with f di (P (f di , f d• ) for all sub-index • ∈ [1 : D] and • = d i ). Generally, m th -Boosted-DS (Boosted-DSm) would take the sums of drawing samples upto m-dimensional joint distributions. The computa-tional complexity of Boosted-DS2 is O(SKN D) and space complexity is O(N D 2 ) if we store all needed hidden states instead.

In summary, FFP utilizes hidden states from same decision tree at each iteration; PFP and Boosted-DS only utilizes hidden states on top levels of decision graph; DS utilizes hidden states on leaf nodes. DS and Boosted-DS randomly pick an layout A to start and keep improving itself dimension by dimension till converge, while FFP and PFP do not randomly guess other dimension values. All four algorithm approximate the process of finding best bandit arm by pruning decision search trees and greedy optimization of sequential process through all dimensions. As the greedy approach significantly reduce search space, hence the converge performances are expected to beat the traditional Thompson sampling method N D -MAB.

5. EMPIRICAL VALIDATION AND ANALYSIS:

We illustrate the performance of our algorithm (FPF, PPF, DS and Boosted-DS) on simulated data set, comparing with MVTHill et al. ( 2017 2017) base models mentioned before. Specially, we evaluate (a) the average cumulative regret, (b) the convergence speed and (c) the efficiency of best optimal arm selection among these models under same simulation environment settings. To access fairly appreciative analysis, the mechanism and parameters for generating the simulation data set are completely at random. We would also replicate all algorithms multiple (H) times and take the average to eliminate evaluation bias due to TS probabilistic randomness. Furthermore, we extensively exam the cumulative regret performance of proposed algorithms by varying (1) the relative strength of interaction between dimensions and (2) complexity in arm space (altering N and D) to gain comprehensive understanding of our model.

5.1. SIMULATION SETTINGS:

Simulated reward data is generated in Bernoulli simulator with success rate being linear with m-way dimension interactions:

EQUATION

where β is scaling variable and α 1 , . . . , α m are control parameters. It is trivial to set µ 0 = 0. We intentionally generate weights µ independently from N (0, 1), and set β = m and α m = m! D(D-1)...(D-m+1) to control the overall signal to noise ratio as well as related strength among m-way interactions.

In this paper, we set m = 2 (pairwise dimension interaction), D = 3 and N = 10 in above simulator settings, which yields 1000 possible layouts. To observe the convergence of each model and eliminate the randomness, our simulation is generated with T = 100, 000 time steps and H = 100 replications. On each simulation replica and at time step t, layout A t is chosen by each algorithm, and a binary reward X t is sampled from Bernoulli simulator with success rate θ, which is coming from Equation 2 with pre-generated randomly weights µ. We choose S = 45 and K = 10 as the same Hill climbing model parameter settings to compare between FPF, PPF2, DS, Boosted-DS2 and MVT2 methods.

5.2. NUMERICAL RESULTS:

Figure 2 shows the histograms of arm exploration and selection activities for 7 algorithm as well as distribution of success rate for arms in our simulator. The horizontal axis is the success rate of selected arm, while the vertical axis is the probability density in histogram. The success rate density of Bernoulli simulators is symmetrically distributed, which coincides with our simulation setting. However, the severity of right skewness reveals the efficiency and momentum algorithms recognizing bad performance arms and quickly adjusting search space to best possible arms. Although N D -MAB is theoretically guaranteed to archive optimal performance in the long run, the histogram graph might empirically explains why MVT2, FPF, PPF2 and Boosted-DS2 outperform N D -MAB in many ways. It is worth to mention that the search behavior (performance) of DS is similar to N D -MAB, but DS consists simpler computational complexity (O(SN K)). This concludes us that DS strategy itself starting path planning from bottom has limited improvement on arm heuristic search than N D -MAB. The underline reason could be that only small fraction of arms is explored at early stage and little information is known for each arm, starting from top strategy can resemble dimensional analogues and learn arm's reward distribution from other arms with similar characteristics. In turns, it helps to rapidly shift to better performed arms. The proposed Boosted-DS2 overcomes DS's issue by using TS samples from top levels. The heavily right skewness in Boosted-DS2 histogram confirms our suspects. To recognize the effectiveness of optimal selection, we leverage the average regret, convergence rate and best arm rate. We define convergence rate as proportion of trials with the most selected layout over a moving window with batch size (t 1 -t 0 =) 1000. We further specify best arm rate as the proportion of trials with the best possible layout in one batch.

Average Regret = H h=1 1 H T t=1 E(X A ) -X t T , Convergence Rate[t 0 , t 1 ] = H h=1 1 H t1 t=t0 1 A o (A t ) t 1 -t 0 , Best Arm Rate[t 0 , t 1 ] = H h=1 1 H t1 t=t0 1 A (A t ) t 1 -t 0 ,

where A o and A stand for most often selected layout and best possible layout respectively within a batch. Ideally, we prefer convergence rate and best arm rate both approaching to 1, which means the algorithm converges selection to single arm (convergence) as well as best arm. In practice, a fully converged batch trials almost surely select the same layout (sub-optimal) but not necessarily the global optimal layout. Simulated performance results are displayed in Figure 3 where x-axis is the time steps. Path planning algorithms demonstrate advantages over base models, especially for FPF, PPF2 and Boosted-DS2.

We see that PPF2 and Boosted-DS2 quickly jump to low regret (and high reward) within 5000 steps, followed by FPF and MVT2 around 10000 steps. Although Boosted-DS2 and MVT2 share fastest convergence speed followed by PPF2 then FPF, but FPF holds the highest best arm rate. FPF performance in cumulative regret (and reward) catches up for longer iterations as well. The intuition behind these is that FPF includes the most complex model space with considering full dimension interactions, in which it not only look from top level of decision graph to quickly eliminate bad performed dimension values but also drill down to leaf arms to correct negligence from higher levels. The exponential space complexity or computational complexity proportional to T is our concern of FPF compared with PPF2 and Boosted-DS2.

In our experiment, PPF2, Boosted-DS2 and MVT2 assume models with pairwise interactions in one way or the other, and it happens to be our simulator setting. In practice extra effort is needed for correct modeling the reward function, which is out of this paper's scope. PPF2 and Boosted-DS2 both efficiently achieve lower regret comparing with MVT2. However, PPF2 carries better best arm rate than Boosted-DS2. Our take-away from this is that Hill-climbing strategy contains two drawbacks. First, it is equivalent to a bottom-up path planning strategy in our framework, which is not as efficient as top-down strategy as discussed before. Boosted-DS2 combats such weakness using TS samples on top levels to mimic sample draw from lower level. Second, Hill-climbing starts with randomly guess other dimension values which easily ends up with good enough arm selection (low regret and high convergence) but not always the best (low best arm rate). In the meanwhile, D-MABs struggles in performance as its assumption of independence between dimensions doesn't match with our simulator.

Although PPF2, Boosted-DS2 and MVT2 all share simplified model complexity (both in computation and parameter space), however MVT2 takes longer time period per iteration compared with the other two. Table 1 shows the iteration speed of these algorithms in our implementation. In fact, MVT2 is the slowest algorithm as the heavy computation burden when updating regression coefficients in posterior sampling distribution.

We further extend our simulation results of average cumulative regret by varying α 2 to change the strength of interactions as well as varying N and D to change space complexity in Fig 4 . We skip MVT2 due to the time limitation (MVT2 takes 5 days per experiment). As α 2 varies from 1 6 to 1 with step size 1 6 , we see the pattern consists with prior result at Fig 4a . The only exception is D-MABs gets dominant regret when interaction strength is weak (α 2 = 1 6 ), as D-MABs's no interaction assumption close to the truth at that time. D-MABs is equivalent with PPF1. So D-MABs should perform similarly with PPF2 when interaction strength is weak. Next we analyze the impact on performance with model complexity. We systematically swap N in (2, 4, 6, 8, 10, 12) and D in (2, 3, 4, 5) at Fig 4b and 4c respectively. We observe that the relative performance still holds: FPF PPF2 > Boosted DS > DS N D -MAB > D-MABs. Based on these extensive experiments, we assert that our proposed method is superior consistently.

In summary, our simulated results suggest that TS-PP has good performance overall for multivariate bandit problem with large search space when dimension hierarchy structure exists. FPF accomplishes the best performance, however PPF2 attracts implementation attention due to its computation efficiency with comparable performance. 

6. CONCLUSIONS:

In this paper, we presented TS-PP algorithms taking advantage of the hierarchy dimension structure of bandit arms to quickly find the best arm. It utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts TS with MCTS for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. Based on our simulation results, it achieves superior results in terms of cumulative regret and converge speed comparing with MVT, N D -MAB and D-MABs on large decision space. We listed 4 variations of our algorithm, and concluded that FPF and PPF conduct the best performance. We highlight PPF due to its implementation simplicity but high efficiency in performance.

It is trivial to extend our algorithm to contextual bandit problem with finite categorical context features. But how to extend our algorithm from discrete to continuous contextual variables worth us further exploration. We notice some related work of TS-MCTS Bai et al. (2014) dealing with continuous reward in this area. Finally, fully understanding the mechanism of using heuristic greedy approach (in our method) to approximate TS from N D arms is still under investigation.

:



Figures and Tables:

(a) Decision Graph (b) Decision Tree under dimensions order [d1, . . . , dD]

Figure 1: Path Planning Overview

), N D -MABHill et al. (2017) and D-MABsHill et al. (

Figure 2: Histogram of expected reward for historical arm search. Algorithm FPF PPF2 DS Boosted-DS2 MVT2 Iteration Speed (it/s) 7.04 22.39 2.01 1.38 0.25

Figure 3: Performance on simulated data with D = 3, N = 10, β = 3, m = 2, α1 = 1 3 and α2 = 1 3 .

Performance when α2 varies (b) Performance when D varies (c) Performance when N varies

Figure 4: Cumulative Averaged Regret over number of iterations when D, N and α2 varies.

Without loss of generality, we assume that dimensions are tagged in order of [d 1 : d D ], which is an arbitrary order of [1, . . . , D]. Decision tree in Figure 1b compactly represents the joint probabilities of all arms [f d1:d D ] (leaf nodes) and internal nodes. Here we borrow notations in Kochenderfer (2015). [d 1 : d D ] and [f d1:d D ] are compact ways to write [d 1 , . . . , d D ] and [f d1 , . . . , f d D ] respectively. The sub-index d i of f di refers that this dimension value f comes from dimension d i . The structure of decision tree consists of nodes and directed edges. Each level of the decision tree represents a dimension d i and each node at that level corresponds to a value f di for dimension d i . Directed edges connect parent node to child node where the arrow represents conditional (jointly) relationship. Associated with each node f di is a jointly probability P

Table content: <table/>

Algorithm Speed Comparison

Table content: <table/>

Peer review (iclr2020):
In this paper, the authors address the curse of dimensionality in Multivariate Multi-Armed Bandits by framing the problem as a sequential planning problem. Their contributions are the following: 1. They propose to solve the Multivariate MAB sequentially across each dimension, which I think is a reasonable and not very common approach in the literature compared to e.g. structured bandits, even though the authors themselves call this approach *natural* and *quite straightforward*. 2. They propose a framework based on a Monte-Carlo Tree Search procedure to solve this sequential decision problem. 3. They introduce several approximations and heuristics to avoid traversing the entire tree and trade-off performance for computational complexity, which they evaluate empirically. The obtained results seem to support their original claim: the tree-based approaches (e.g. FPF) perform better than the considered baselines. However, this claim is at the same time contradicted by the fact that a depth-one planning procedure with heuristic leaf evaluation (PPF2) reaches similar performances, which indicates that the underlying problem can be solved greedily and does not involve long-term (m-way) interations and limits the interest of the main idea in this paper. This is probably due to the design of experiments which only model pairwise interactions, but then this questions the relevance of the proposed empirical evaluation with respect to the claim. Generally, though several good ideas were presented, I felt that they were not properly motivated and that the paper generally lacks rigor and clarity. The many variants and modeling decisions proposed seem quite arbitrary, and in the end fail to provide any valuable insight. Several aspects of the proposed approach and algorithms were not clear or justified. I will list some of them, grouped by the corresponding contribution: 1. Multivariate MAB framed as a planning problem. No intuition is provided regarding why it is expected to perform any better than classical N^D-MAB. In this absence of assumption regarding additional structure to the problem considered (such as, say, the independence assumption in D-MABS), is there any reason to think so? (for instance, all the N^D arms are present at the leaves of the tree) The authors only mention in their conclusion that they leverage a *hierarchy dimension structure*, but this claimed structure was is never defined nor formalized in the paper. I can only guess that such a structure would involve a particular ordering of dimensions such that shallow ones have more influence on the payoff than the deeper ones? But then, it would be unknown in practice, and the choice of the particular ordering used in TS-PP is expected to have a crucial influence on its performance. Yet, the sensitivity of their algorithms to the ordering is never discussed by the authors -they choose it arbitrarily- nor evaluated in their experiments. 2. The proposed TS-PP framework - The authors propose using MCTS with TS instead of UCB as a sampling rule, which they claim is quite novel: *Applying TS as node selection policy is not well investigated from our best knowledge*. However, the authors do not mention important reference (Bai et al, 2014) until the very end of the paper in their conclusion: *We notice some related work dealing with continuous rewards in this area*. Yet, Bai et al. also consider Bernoulli rewards in their experiments and use conjugate Beta priors, exactly like the authors of the present work do. Moreover, the choice of TS rather than UCB is not properly motivated, and the UCT algorithm is not considered as a baseline in the experiments. Hence, there is no basis to assess whether TS is particularly relevant or not for this problem. - They also propose to repeat S times the candidate searching procedure in Algorithm 1 in order not to be *[stuck] in sub-optimal arms*. This property is normally ensured by optimistic sampling rules, it is not clear why it is required here. More importantly, the sensitivity of the Algorithm 1 to S is not evaluated. - I did not fully understand the Candidates Construction Stage in Algorithm 1. For instance, in the case of FPF, for each search c=1..S (l3) there is a call (l4) to Full Path Finding() to generate a candidate A^c. From what I understand, a random ordering is chosen (Algorithm 2, l2), and a single rollout is performed (l2, the dimensions 1->D are iterated from root to leaf) to return a candidate. When we move to the next search c -> c+1, since a new ordering d_1:D is chosen then the previous tree storing the models for p(fd_i | fd_1...fd_i-1) is not relevant anymore and must be discarded. Does that mean that the priors at the nodes are recomputed for each search/ordering/tree given the entire history, instead of being maintained and updated? The authors do state that *FFP utilizes hidden states from the same decision tree*, but I do not see how this is feasible when the dimensions ordering changes at each iteration. - This confusion is related to the fact that the update of the Beta posterior does not appear anywhere in Algorithm 1 and 2. This is unfortunate, especially considering that contrary to TS in a normal MAB setting, the observations for an internal node are not i.i.d. since they depend on the underlying planning procedure, which induces a distributional drift as more sample are collected and probably make the derived posterior invalid. Has this issue been considered by the authors? 3. The variants of FPF The idea behind Partial-PF-m is common in the tree-search literature: to stop at early depths and use a heuristic to estimate the value at the leaf. Here, the suggestion in the particular context of Multivariate-MAB to use TS with an independence assumption (D-MABs) is interesting and sound. However, It requires the knowledge the degree m of interactions, whereas the idea of MCTS is precisely to avoid using such heuristics for leaf evaluation by replacing them without random rollouts. Other methods such as DS start at a leaf and move locally which rather resembles Hill-climbing more closely than actual Tree-Search, and I do not really see how they fit in the TS-PP framework? Finally, I found the manuscript difficult to read due to many language mistakes and typos. Minor comments: - The citation style is incorrect. - What is the point of introducing contextual MABs in the problem formulation when the context is not going to play any part whatsoever in the presented work? This only brings unnecessary complexity and impedes clarity. - In the experiments, H=100 replications are performed, which is appreciated, but unfortunately the corresponding confidence intervals or standard deviations are not shown, only the mean performance. - I found the section 3.1 unclear. Is B_A^T mu a dot product? Where do the coefficients of B_A or its length M appear in (1)? How is M chosen, how is B_A obtained? - Figure 1.a is also unclear, the graph seems to be used to define permutations/orderings of dimensions 1..D. But all paths are not valid (eg. d1 -> d2 -> d1), ad random orderings are used in Algorithm 2, so where is this graph used exactly? ============= After Rebuttal: On one hand, some of the points mentioned are now clearer (e.g. 1 & 2.2, 2.3). On the other hand, most of my concerns remain valid, and some were not addressed. Typically, the dependency to an unknown (and undefined...) optimal ordering and the underlying optimization procedure (random search) are not properly discussed, whereas they seem central to justify the tree-based approach. Other remaining concerns are more generally the lack of justification and clarity of the proposed framework and implemented variants, and the inadequacy of the experiments with respect to the claim (using m=2 basically means only the first depth of the tree is useful). Therefore, I will maintain my initial score.