Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a open bandit dataset for evaluating contextual bandit algorithms. Overall, I think this paper is addressing a missing need since one of the key limitations of academic research in contextual bandits is the lack of an open-source dataset to evaluate algorithms that is not synthetically designed or obtained from classification datasets.
strengths: In the beginning, I really enjoyed the paper and completely resonated with it -- specifically, the lack of open-source benchmarks for contextual bandits and off-policy estimation and learning. I think this is the biggest strength of this work. While the dataset will be useful, my only reservation is in the experiments (see more below). Having worked on contextual bandits in the past, I understand the importance of this work. However, there were a few aspects of the evaluation that I was disappointed with (again, see below). 
weaknesses: As mentioned above, I was very excited about this paper in the beginning and completely resonated with the motivation. However, there are several limitations which I noticed towards the end, leaving me a little disappointed:

- The authors have no mention of vowpal wabbit for off-policy estimation and learning. This is very related work
- The authors only focus on off-policy estimation but not learning/optimization. It is not clear why. I believe the authors should also look into optimizing the contextual bandit policies. Vowpal wabbit, for example, provides a number of algorithms for off-policy learning. Given the logged data, why not use it to learn policies?
- I will also encourage the authors to take a look at the policy optimizer for exponential models (POEM) approach which can do a soft-max relaxation to the policy optimization problem.
- Again, to summarize, I think a key limitation of this work (in my opinion -- again, it is totally possible I could have misunderstood aspects of this work) is that it does not do the policy optimization which I think is critical in this benchmarking. Which kind of policies works best? How much does exploration hurt the performance?
correctness: The paper is correct for the most part.
clarity: The paper is clear and well written.
relation to prior work: The paper misses some related work (mainly, for example, VW and the POEM approach for policy optimization)
additional feedback: See issues raised above
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a new dataset called ZOZOTOWN with the aim of benchmarking off policy evaluation methods for the bandit setting. They collect a massive amount of data through two different policies and then evaluate different algorithms on their dataset. They also provide their codebase pipeline for basic usage of their dataset.
strengths: The paper claims to be the first to propose a large realistic dataset for off policy evaluation in the bandit setting. They provide an easy to use codebase and have implemented a number of baselines for their dataset which would lead to ease of future research on this benchmark.
weaknesses: The motivation for a large-scale bandit based dataset is not clear. There has been a lot of work done on bandits on simpler settings and its not clear what this new dataset brings to the table. On the other end of the spectrum, the authors do not provide a discussion with other related work like Fu et. al (2021): Benchmarks for Deep Off-Policy Evaluation which proposes a benchmark for off-policy evaluation. Given that there exist benchmarks for complex off-policy evaluation in a non-bandit setting as well as simpler benchmarks in the bandit setting where one could run as many policies as possible, its not immediately clear if the proposed benchmark provides a significantly different contribution.
correctness: The benchmark is constructed in a sound way and the evaluation methods and experiment design makes sense.
clarity: The paper is a bit difficult to follow and restructuring the sections would allow for an easier read. In particular, details regarding sklearn and the codebase can be provided in the Appendix and using the main paper to provide more information on the baselines would serve better.
relation to prior work: The paper discusses how its contributions are different when compared with some of the other benchmarks on recommender systems and bandit problems. There is little discussion on previous works on off policy evaluation benchmarks from the different sectors (eg. healthcare or deep off policy evaluation).
additional feedback: The authors provide a dataset for off policy evaluation in bandit models. The motivation and writing could use a bit of work since its not immediately clear how the benchmark provides something significantly new and needed in the field.

**Update after Rebuttal**: I have increased the score to 6 based on the points raised by the authors during the rebuttal.
review_title: Review of the Off Policy Evaluation Benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduce a new logged-bandit dataset called *Open Bandit Dataset*. This new dataset consists of data collected in the same environment by two different policies hence it enables the realistic evaluation of OPE estimators (by comparing the on-policy performance of one policy with the off-policy estimation of its performance obtained using the data collected by the other policy). The authors also introduce *Open Bandit Pipeline*, which is a software package that consists of modules for dataset preprocessing, policy learning, and OPE estimation.
strengths: * Beyond its utility as a publicly-available logged-bandit dataset, *Open Bandit Dataset* enables researches to evaluate OPE estimators in a realistic environment without relying on simulators. According to the authors, OBD is first publicly-available dataset to enable such evaluation. Using this new evaluation procedure, the authors were able to identify a new potential research direction, namely the need for better automatic hyper-parameter tuning methods in certain instances.

* The fact that OBD has already seen use in several publications is noteworthy.

* The dataset is well documented and its use is well supported by the accompanying software package *Open Bandit Pipeline*.
weaknesses: Although how OBD enables realistic evaluation of OPE estimators is well explained and a detailed evaluation protocol is provided, I found some other features of OBD and in particular OBP to be lacking more detailed explanations:

* It is mentioned that OBP includes tools to generate synthetic datasets as well but how OBP does so is not discussed anywhere on the paper. I believe this is achieved by "modifying classification datasets", which is mentioned in the introduction as an existing synthetic approach to evaluating OPE estimators, but I do not understand what is meant by that.

* Table 2 shows that OBD can be used to evaluate bandit algorithms (not only OPE estimators). However, what is meant by "evaluation of bandit algorithms" and how OBD can facilitate that is not discussed anywhere else.
correctness: The claims made in the submission seem correct.
clarity: The paper is generally well-written with the exception of points that I mentioned in weaknesses.
relation to prior work: Both related datasets as well as related software packages are summarized in Tables 2 and 3. These tables makes it clear how OBD and OBP differ from prior work.
additional feedback: * At line 188, it is mentioned that alternative software packages cannot be used to implement some advanced OPE estimators. Does it mean that (unlike OBP) they do not provide interfaces flexible enough for implementing such advanced OPE estimators?

* At the end of Section 6, an assumption on the reward structure is mentioned as a limitation. As far as I understood, this is a limitation of the experimental results presented in the paper and not an inherit limitation of OBD. Is that correct?
review_title: Review of "open bandit dataset and pipeline: towards realistic and reproducible off-policy evaluation"

Meta-review: This scores for this paper are still marginal. However, the authors have addressed and responded to many of the issues raised by reviewers including expanding a related work section to include relevant prior literature in this area. After considering the author rebuttal and discussing with another AC, and given this important problem space for which quality evaluation datasets would be valuable, I recommend acceptance.
Decision: Accept