Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors address the curse of dimensionality in Multivariate Multi-Armed Bandits by framing the problem as a sequential planning problem.
Their contributions are the following:
1. They propose to solve the Multivariate MAB sequentially across each dimension, which I think is a reasonable and not very common approach in the literature compared to e.g. structured bandits, even though the authors themselves call this approach "natural" and "quite straightforward".
2. They propose a framework based on a Monte-Carlo Tree Search procedure to solve this sequential decision problem.
3. They introduce several approximations and heuristics to avoid traversing the entire tree and trade-off performance for computational complexity, which they evaluate empirically.

The obtained results seem to support their original claim: the tree-based approaches (e.g. FPF) perform better than the considered baselines. However, this claim is at the same time contradicted by the fact that a depth-one planning procedure with heuristic leaf evaluation (PPF2) reaches similar performances, which indicates that the underlying problem can be solved greedily and does not involve long-term (m-way) interations and limits the interest of the main idea in this paper. This is probably due to the design of experiments which only model pairwise interactions, but then this questions the relevance of the proposed empirical evaluation with respect to the claim.
Generally, though several good ideas were presented, I felt that they were not properly motivated and that the paper generally lacks rigor and clarity. The many variants and modeling decisions proposed seem quite arbitrary, and in the end fail to provide any valuable insight. 

Several aspects of the proposed approach and algorithms were not clear or justified. I will list some of them, grouped by the corresponding contribution:

1. Multivariate MAB framed as a planning problem.
No intuition is provided regarding why it is expected to perform any better than classical N^D-MAB. In this absence of assumption regarding additional structure to the problem considered (such as, say, the independence assumption in D-MABS), is there any reason to think so? (for instance, all the N^D arms are present at the leaves of the tree) The authors only mention in their conclusion that they leverage a "hierarchy dimension structure", but this claimed structure was is never defined nor formalized in the paper. I can only guess that such a structure would involve a particular ordering of dimensions such that shallow ones have more influence on the payoff than the deeper ones? But then, it would be unknown in practice, and the choice of the particular ordering used in TS-PP is expected to have a crucial influence on its performance. Yet, the sensitivity of their algorithms to the ordering is never discussed by the authors -they choose it arbitrarily-  nor evaluated in their experiments.

2. The proposed TS-PP framework
- The authors propose using MCTS with TS instead of UCB as a sampling rule, which they claim is quite novel: "Applying TS as node selection policy is not well investigated from our best knowledge". However, the authors do not mention important reference (Bai et al, 2014) until the very end of the paper in their conclusion: "We notice some related work dealing with continuous rewards in this area". Yet, Bai et al. also consider Bernoulli rewards in their experiments and use conjugate Beta priors, exactly like the authors of the present work do. Moreover, the choice of TS rather than UCB is not properly motivated, and the UCT algorithm is not considered as a baseline in the experiments. Hence, there is no basis to assess whether TS is particularly relevant or not for this problem.
- They also propose to repeat S times the candidate searching procedure in Algorithm 1 in order not to be "[stuck] in sub-optimal arms". This property is normally ensured by optimistic sampling rules, it is not clear why it is required here. More importantly, the sensitivity of the Algorithm 1 to S is not evaluated.
- I did not fully understand the Candidates Construction Stage in Algorithm 1. For instance, in the case of FPF, for each search c=1..S (l3) there is a call (l4) to Full Path Finding() to generate a candidate A^c. From what I understand, a random ordering is chosen (Algorithm 2, l2), and a single rollout is performed (l2, the dimensions 1->D are iterated from root to leaf) to return a candidate. When we move to the next search c -> c+1, since a new ordering d_1:D is chosen then the previous tree storing the models for p(fd_i | fd_1...fd_i-1) is not relevant anymore and must be discarded. Does that mean that the priors at the nodes are recomputed for each search/ordering/tree given the entire history, instead of being maintained and updated? The authors do state that "FFP utilizes hidden states from the same decision tree", but I do not see how this is feasible when the dimensions ordering changes at each iteration. 
- This confusion is related to the fact that the update of the Beta posterior does not appear anywhere in Algorithm 1 and 2. This is unfortunate, especially considering that contrary to TS in a normal MAB setting, the observations for an internal node are not i.i.d. since they depend on the underlying planning procedure, which induces a distributional drift as more sample are collected and probably make the derived posterior invalid. Has this issue been considered by the authors?

3. The variants of FPF
The idea behind Partial-PF-m is common in the tree-search literature: to stop at early depths and use a heuristic to estimate the value at the leaf. Here, the suggestion in the particular context of Multivariate-MAB to use TS with an independence assumption (D-MABs) is interesting and sound. However, It requires the knowledge the degree m of interactions, whereas the idea of MCTS is precisely to avoid using such heuristics for leaf evaluation by replacing them without random rollouts. Other methods such as DS start at a leaf and move locally which rather resembles Hill-climbing more closely than actual Tree-Search, and I do not really see how they fit in the TS-PP framework?

Finally, I found the manuscript difficult to read due to many language mistakes and typos.

Minor comments:
- The citation style is incorrect.
- What is the point of introducing contextual MABs in the problem formulation when the context is not going to play any part whatsoever in the presented work? This only brings unnecessary complexity and impedes clarity.
- In the experiments, H=100 replications are performed, which is appreciated, but unfortunately the corresponding confidence intervals or standard deviations are not shown, only the mean performance.
- I found the section 3.1 unclear. Is B_A^T \mu a dot product? Where do the coefficients of B_A or its length M appear in (1)? How is M chosen, how is B_A obtained?
- Figure 1.a is also unclear, the graph seems to be used to define permutations/orderings of dimensions 1..D. But all paths are not valid (eg. d1 -> d2 -> d1), ad random orderings are used in Algorithm 2, so where is this graph used exactly?


=============
After Rebuttal:
On one hand, some of the points mentioned are now clearer (e.g. 1 & 2.2, 2.3).
On the other hand, most of my concerns remain valid, and some were not addressed.
Typically, the dependency to an unknown (and undefined...) optimal ordering and the underlying optimization procedure (random search) are not properly discussed, whereas they seem central to justify the tree-based approach.
Other remaining concerns are more generally the lack of justification and clarity of the proposed framework and implemented variants, and the inadequacy of the experiments with respect to the claim (using m=2 basically means only the first depth of the tree is useful). Therefore, I will maintain my initial score.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper approaches the problem of exploding arms in multivariate multi-armed bandits. To solve this problem, the authors suggest an approach that uses Thompson sampling for arm selection and decision trees and graphs (inspired from Monte-Carlo tree search) to model reward success rates. They propose to versions of this path planning procedure and 2 version inspired from Hill-climbing methods. They validate their claims on a simulation showing better performance and faster convergence, and providing an extensive analysis of the results. 

The idea seems novel. The paper is well structured, but the writing can be improved, and some parts are hard to read and follow (See minor comments below). 

Here are few questions:
- The authors claim that the approach can be extended to categorical/numeric rewards. Can they give more details on how? 
- The experiments are done only with D=3 and N=10. How easy would it be to scale to higher dimensions?
- In the curves of Figure 3, N^D-MAB and DS seem not converged yet contrarily to the other methods. Have the authors tried to let them run for longer to see to which values they converge? 

Minor comments (non-exhaustive examples): 
- Punctuation issues: 
    *Multi-Armed Bandit (MAB) problem, is widely ...
    * Generally, RT is on the order O( T) under linear payoff settings Dani et al. (2008)Chu et al. (2011)Agrawal & Goyal (2013). Although the optimal regret of non-contextual Multivariate MAB is on the order O(logT ) ...

- Imprecise statements:  ... for each combination of C (assuming not too many), ...

- Sentences to rewrite: 
     * The posterior sampling distribution of reward is its likelihood integrates with some fixed prior distribution of weights μ.
     * ..., and would call joint distribution of (A, C) and (B, C) are independent. 
     * ..., which means the algorithm converges selection to single arm (convergence) as well as best arm. 

- Typos: Jointly distribution/relationship -> joint ,It worth to note -> it is worth to note, extensively exam -> extensively examine, a serial processes -> process, would been -> be ...
 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes to use path planning on the tree structure for multivariate MAB problem. The idea is to model the hierarchical structure in the decision space, thus tree search idea can be applied to improve the efficiency of learning.

In general the trend of the paper is easy to follow. However, the proposed method is a bit ad-hoc and I don’t think the paper provides enough justification for it. 
1. FPF is not really practical in that it needs to main D! decision trees, each one with N^D leaves. On the other hand, other approximation methods like PPF, relies on some independence assumption. However, this paper provides not analysis about such independence even in simulations. For example, when these independence assumptions are strongly violated, what the performances would be.
2. There is no experiments on real data. The simulation in the this paper is very specific and I am not sure how representative it could be. The simulation results don’t provide a clear message too.

Other comment
1. On page 4, why would P(f_{d1:d_D}) be propotional to P(f_{d1:d_i})? 


=========================
After Rebuttal
I don't think the rebuttal addresses my concern: the proposed method is not well justified and its performance is not really analyzed nor with convincing evidence.
review_title: Official Blind Review #1

Meta-review: This paper tackles the multivariate bandit problem (akin to a factorial experiment) where the player faces a sequence of decisions (that can be viewed as a tree) before obtaining a reward. The authors introduce a framework combining Thompson Sampling with path planning in trees/graphs. More specifically, they consider four path planning strategies, leading to four approaches. The resulting approaches are empirically evaluated on synthetic settings.

Unfortunately, the proposed approaches lack theoretical justification and the current experiments are not strong enough to support the claims made in the paper. Given that most of reviewer's concerns remained valid after rebuttal, I recommend to reject this paper.
Decision: Reject