Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper "DEEP BAYESIAN BANDITS SHOWDOWN" proposes a comparative study about bandit approaches using deep neural networks. 

While I find that such a study is a good idea, and that I was really interested by the listing of the different possibilities in the algorithms section, I regret that the experimental results given and their analysis do not allow the reader to well understand the advantages and issues of the approaches. The given discussion is not enough connected to the presented results from my point of view and it is difficult to figure out what is the basis of some conclusion.

Also, the considered algorithms are not enough described to allow the reader to have enough insights to fully understand the proposed arguments. Maybe authors should have focused on less algorithms but with more implementation details. Also, what does not help is that it is very hard to conect the names in the result table with the corresponding approaches (some abbreviations are not defined at all - BBBN or RMS for instances).

At last, the experimental protocol should be better described. For instance it is not clear on how the regret is computed : is it based on the best expectation (as done in most os classical studies) or on the best actual score of actions? The wheel bandit protocol is also rather hard to follow (and where is the results analysis?).

Other remarks:
   - It is a pitty that expectation propagation approaches have been left aside since they correspond to an important counterpart to variational ones. It would have been nice to get a comparaison of both; 
   - Variational inference decsription in section algorithms is not enough developped w.r.t. the importance of this family of approaches
   - Neural Linear is strange to me. Uncertainty does not consider the neural representation of inputs ? How does it work then ?
   - That is strange that \Lambda_0 and \mu_0 do not belong to the stated asumptions in the linear methods part (ok they correspond to some  prior but it should be clearly stated)
   - Figure 1 is referenced very late (after figure 2)



review_title: An interesting comparative study of deep neural bandits (but with rather limited results analysis)  

final_score: rating: 7: Good paper, accept
review_content: If two major questions below are answered affirmatively, I believe this article could be very good contribution to the field and deserve publication in ICLR.

In this article the authors provide a service to the community by comparing the current most used algorithms for Thompson Sampling-based contextual (parametric) bandits on clear empirical benchmark. They reimplement the key algorithms, investing time to make up for the lack of published source code for some. 

After a clear exposure of the reasons why Thompson Sampling is attractive, they overview concisely the key ideas behind 7 different families of algorithms, with proper literature review. They highlight some of the subtleties of benchmarking bandit problems (or any active learning algorithms for that matter): the lack of counterfactual and hence the difference in observed datasets. They explain their benchmark framework and datasets, then briefly summarise the results for each class of algorithms. Most of the actual measures from the benchmark are provided in a lengthy appendix 12 pages appendix choke-full of graphs and tables.

It is refreshing to see an article that does not boast to offer the new "bestest-ever" algorithm in town, overcrowding a landscape, but instead tries to prune the tree of possibilities and wading through other people's inflated claims. To the authors: thank you! It is too easy to dismiss these articles as "pedestrian non-innovative groundwork": if there were more like it, our field would certainly be more readable and less novelty-prone.

Of course, there is no perfect benchmark, and like every benchmark, the choices made by the authors could be debated to no end. At least, the authors try to explain them, and the tradeoffs they faced, as clearly as possible (except for two points mentioned below), which again is too rare in our field. 

Major clarifications needed:

My two key questions are:
* Is the code of good quality, with exact  reproducibility and good potential extension in a standard language (e.g. Python)? This benchmark only gets its full interest if the code is publicised and well engineered. The open-sourcing is planned, according to footnote 1, is planned -- but this should be made clearer in the main text. There is no discussion of the engineering quality, not even of the language used, and this is quite important if the authors want the community to build upon this work. The code was not submitted for review, and as such its accessibility to new contributors is unknown to this reviewer. That could be a make or break feature of this work. 
* Is the hyper parameter tuning reproducible? Hyperparameter tuning should be discussed much more clearly (in the Appendix): while I appreciate the discussion page 8 of how they were frozen across datasets, "they were chosen through careful tuning" is way too short. What kind of tuning? Was it  manual, and hence not reproducible? Or was it a clear, reproducible grid search or optimiser? I thoroughly hope for the later, otherwise an unreproducible benchmark would be very 

If the answers to the two questions above is "YES", then brilliant article, I am ready to increase my score. However, if either is a "NO", I am afraid that would limit to how much this benchmark will serve as a reference (as opposed to "just one interesting datapoint").


Minor improvements:
* Please proofread some obvious typos: 
  - page 4 "suggesed" -> "suggested",  
  - page 8 runaway math environment wreaking the end of the sentence.
  - reference "Meire Fortunato (2017)" should be  "Fortunato et al. (2017)", throughout.
* Improve readability of figures' legends, e.g. Figure 2.(b) key is un-readable. 
* A simple table mapping the name of the algorithm to the corresponding article is missing. Not everyone knows what BBB and BBBN stands for.
* A measure of wall time would be needed: while computational cost is often mentioned (especially as a drawback to getting proper performance out of variational inference), it is nowhere plotted. Of course that would partly depend on the quality of the implementation, but this is somewhat mitigated if all the algorithms have been reimplemented by the authors (is that the case? please clarify).
review_title: Benchmark most useful if accompanying code is well engineered

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents the comparison of a list of algorithms for contextual bandit with Thompson sampling subroutine. The authors compared different methods for posterior estimation for Thompson sampling. Experimental comparisons on contextual bandit settings have been performed on a simple simulation and quite a few real datasets.

The main paper + appendix are clearly written and easy to understand. The main paper itself is very incomplete. The experimental results should be summarized and presented in the main context. There is a lack of novelty of this study. Simple comparisons of different posterior estimating methods do not provide insights or guidelines for contextual bandit problem. 

What's the new information provided by running such methods on different datasets? What are the newly observed advantages and disadvantages of them? What could be the fundamental reasons for the variety of behaviors on different datasets? No significant conclusions are made in this work.

Experimental results are not very convincing. There are lots of plots show linear cumulative regrets within the whole time horizon. Linear regrets represent either trivial methods or not long enough time horizon.

review_title: A large-scale comparison on some posterior estimation methods for Thompson sampling without much insight

Meta-review: This paper is not aimed at introducing new methodologies (and does not claim to do so), but instead it aims at presenting a well-executed empirical study. The presentation and outcomes of this study are quite instructive, and with the ever-growing list of academic papers, this kind of studies are a useful regularizer. 
Decision: Accept (Poster)