Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper introduces a novel dataset and code library for off-policy policy evaluation for contextual bandits. The dataset contains 26m visits to a fashion retail webpage where three items are displayed to a user, tracking user information, and number of clicks for each item. The data was collected following two different behavior policies (uniform random and a learned Bernoulli policy). The paper includes an in-depth empirical benchmark of popular OPE estimators on the proposed dataset.
strengths: 1. The dataset seems useful for comparing model-free OPE estimators. The size of the dataset and the inclusion of two different behavior policies makes this dataset uniquely useful for OPE research, in a way that I believe to be novel.

2. The empirical benchmark included many of the most important baselines to well understand the performance of OPE algorithms on this dataset. The results seem easily reproducible.
weaknesses: 1. The user features seem to not be predictive of the reward at all, performing seemingly no better than random with the models used by this empirical study. I wonder if this is due to the feature hashing, which was used for anonymization? Generally, hash functions seek to take even highly similar inputs and map them to entirely different outputs, making generalization across similar context vectors impossible.

2. The paper makes repeated promises of *multiple* behavior policies, however only two policies are currently available. While this is sufficient for off-policy experiments and it is technically true that two policies are "multiple", it did feel a bit misleading to discover that multiple=2 on page 5.

3. The writing is highly repetitive, with key aspects of the dataset repeated throughout the paper (e.g. that there are multiple policies, the discription of Zozotown, etc.). It seems that the paper could be reduced by at least a couple of pages without harming readability, clarity, or precision. Normally, I wouldn't comment on this except I do feel there were a few omissions which could be included if the vertical space was used more sparingly.

4. One such omission: there is negligible investigation of the dataset itself. What are the empirical probabilities of selecting actions (e.g. proportion that each action is taken) for the Bernoulli policy? How do certain components of the context vector impact action selection (e.g. are features of the user important or only features of the item)? 
correctness: My only concern is the use of hashing for anonymity of the data. I agree completely with the need to anonymize, but I wonder if other methods of anonymization would be more appropriate (e.g. even simply dropping certain user features from the public data release might still allow user context vectors to be predictive, while anonymous).

---

## Edit after discussion

This point is very important to me and I believe it is a blocker for publication. The dataset is proposed for OPE research and is released as a contextual bandit dataset, but it appears that the contexts no longer retain any predictive information due to the feature hashing. This is a strong enough deviation from the current claims of the paper, that I believe it is (unintentionally) misleading and would significantly reduce community adoption of the dataset.

I have two concrete alternative suggestions.
1. Rework the paper to make this limitation more clear and investigate (a) to what extent predictability was lost from feature hashing (e.g. use the original unhashed features and compare to an estimator that only uses the hashed features, are they equally predictive?) and (b) indicate that this means that this dataset cannot be used for contextual bandit-based research.
2. Find an alternative to feature hashing. I sympathize that this is a restriction of the company, but in its current state I am unsure that the dataset provides utility for current research for OPE methods. Some alternatives would be dropping certain identifying features, projecting the feature vectors onto a lower dimensional manifold so that some information is lost, randomly perturbing feature to disrupt identifiability, etc. But similar to point (1), the impact of these decisions should be investigated within the paper.
clarity: The writing has a lot of redundancy. Removing this would enhance readability and would allow space in the main paper for either more empirical investigation of the dataset, or at least to add some of the details already in the appendix to the main body.
relation to prior work: This is well done.

Because reviewers always have the luxury of asking for more, the only potential relevant omissions that I see are:
1. Comparison of with the DICE family of methods as well as the MAGIC estimator (Thomas and Brunskill, 2016)
2. Further discussion of the conclusions of this empirical benchmark using the proposed dataset in comparison with other empirical comparison using synthetic data. I recognize Section 5.3 does this in a limited way currently, but comparing more directly to a particular paper would make the conclusions more impactful (e.g. "here is a particular conclusion that our new dataset makes apparent, which is different from the conclusion of <some other paper> which used only synthetic datasets").

I believe covering these omissions would enhance the quality of the paper; however, I do not find these omissions detract at all significantly from the paper in its current form.
additional feedback: It would be useful for Table 4 to highlight best, second-best, and worst estimators based on the confidence intervals instead of using only the mean. If two confidence intervals overlap, then both estimators should have the same color (otherwise implicit claims are statistically misleading).

It would benefit the paper to discuss computational performance of the implemented OPE estimators in OBP. Are experiments easily parallelizable with the benchmark code? How strongly does the code rely on scikit-learn? Requiring a week to complete the most expensive experiment is long enough to inhibit extensive iterative testing of algorithmic developments during the research process, but perhaps this took so long due to running estimators in serial instead of in parallel?
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The goal is to evaluate OPE methods on real-world datasets. This dataset includes two logged policies—with the plan to include more—which facilitates evaluating the quality of OPE methods. 
strengths: + A large real-world dataset, with associated code that makes it straightforward to compare different OPE methods in the contextual bandit setting. 
+ Very comprehensive comparison to existing benchmarks
weaknesses: The biggest issue is that conclusions about performance between OPE estimators are drawn, seemingly by comparing means rather than using statistical significance. The procedure involves evaluation OPE estimators, across 30 different bootstrap resamples. (This is presumably to evaluate the variability of the OPE estimator algorithm, which is sensible). The mean of performance across these 30 estimators is used. Then there are statements about one OPE estimator being better than another, without saying how we can be confident that is the case.  

One other point is that it is not clearly stated that the goal is to evaluate performance of the OPE algorithm, across possibly different training sets. Another option would have been to simply report the performance of the OPE estimator on the training set, and consider instead performance with increasing samples. It would be useful to state the explicit goal of the pipeline, and what it is testing. 

It is a bit strong to claim that previous datasets cannot be used to evaluate OPE estimators. With a large amount of data, the basic estimators which use sample averages and IS ratios are consistent and should converge to the true value, assuming reasonable coverage. Now, I can see the argument that having multiple policies means that you can leverage on-policy estimates to get likely more accurate value estimates, with less data. Nonetheless, even the on-policy estimator is not truly the ground truth. This point, and the claim that previous datasets cannot be used, should be explained. Or, instead of making such a strong claim, you could simply point out that your dataset facilitates such comparisons. 
correctness: As mentioned above, the protocol does not seem to include a strategy to state, with confidence, that one estimator is better than another. Presumably, the dataset is large and so the mean is informative. But, this is not stated nor addressed. 

One minor point: why is there a time series split point for evaluation and test? Are the two policies run simultaneously? I would think that, even in the out of sample case, the full dataset could be used for the on-policy estimator. 

For Step 5 in Section 5.1, do you recalculate the true on-policy estimate? The text suggests so, but this would not be sensible. Rather, Vhat would be computed on different bootstrap samples. (Note: the pseudocode in the appendix I looked at later seems to have this correct, so just the main text needs to be fixed).

clarity: The writing is clear.

There are a few parts that could use explanation earlier. There is an important point that having multiple policies enables evaluation of OPE methods. This was not obvious to me, until I read Section 5 (though I know it was stated earlier in the paper, by this point it was not immediately in my mind). This point should be explained, at least briefly, when this contribution is mentioned: “with multiple policies, one can be used as the behavior and the other as target, allowing for on-policy value estimate which is much lower variance and so provides a reasonably ground truth value.”

Some paragraphs are too long, and should be broken up into more manageable paragraphs with one key idea. One example is the paragraph with Bold title “Performance Comparisons”.
relation to prior work: The paper goes into depth on comparisons to related benchmarks, with very clear tables contrasting what is currently available and what is new in their benchmark. 
additional feedback: “Although the research community has produced theoretical breakthroughs, the experimental evaluation of OPE remains primitive. Specifically, it lacks a public benchmark dataset for comparing the performance of different methods. ” —> This is a false comparison. Empirical evaluation does not require public benchmark datasets to be advanced or insightful. You may have heard the statement: “My number is bigger than your number is the lowest form of science”. It may be better to avoid stating that empirical evaluation is currently primitive, and focus rather on what forms of evaluation this dataset can facilitate.
I should add that the discussion right after this highlights why a public dataset based on real-world data could be beneficial. The topic sentence here could be focused on that.

“This contrasts with other domains of machine learning, where large-scale open datasets, such as the ImageNet dataset [4], have been pivotal in driving objective progress [6, 9, 11, 12]. ” —> Benchmark datasets can also hold a field back, since they allow for overfitting and a focus on just getting that number a little bit higher. You should address this issue with benchmark datasets. 

“Note that we pre-trained Bernoulli TS for over a month before the data collection process, and the policy well converges to a fixed one. ” —> I am not sure what this means. If it converges, does that mean it is optimal and done learning? 
Further, why is it needed to converge to a fixed policy? You can get a fixed policy simply by letting it train for some number of steps and then fixing it. 

“In contrast, we provide the code to replicate the data collection policies (i.e., Bernoulli TS and Random) in our pipeline, which allows researchers to rerun the same policies on the log data. ” —> This is mentioned as a contribution. I do not understand why this is useful. 

Minor statements:
1. “more practical conclusions about OPE methods.” What is a practical conclusion? This terminology is poor. 
2. For this benchmark, you could consider a useful way to visualize these tables.
review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work proposes the Open Bandit Dataset and Benchmark.  The main goal of this work is to create a realistic benchmark for reproducible testing of bandit OPE methods.  To do so, the authors collected data from ZOZOTOWN, using both Thompson Sampling and Uniform Sampling as the logging policies.  This means that one can both evaluate the accuracy of a standard OPE pipeline (e.g., using logged data from TS to estimate the reward of Uniform) in an on-policy fashion (via the logged data from Uniform).   

In their benchmarking, the authors make two research findings:
-- The estimation performances of all OPE estimators drop significantly when they are applied to estimate the future (or out-sample) performance of a new policy.
-- The estimation performances of OPE estimators heavily depend on the experimental settings and hyperparameters.
(I have personally always suspected the latter point, but it's great to see it validated on a realistic dataset.)

The final contribution of this work is an easy-to-use software package, the Open Bandit Pipeline, for efficiently running more experiments and tests.

strengths: In terms of (public) benchmarking for Bandit OPE, this work is the first of its kind.  No other work has the combination of realistic and reproducible OPE benchmarking.  This is made possible by ZOZOTOWN agreeing to run two separate policies (Thompson Sampling and Uniform) in a A/B test to collect the log data.  I also appreciate the policy code being released, as it also helps with reproducibility.  The design of this benchmark is elegant.


weaknesses: I have one non-trivial issue with the interpretation of the benchmark evaluation.  The authors claim that out-of-sample generalization fails for OPE (penultimate paragraph in page 8).  However, looking at the results in Table 4, I see more of a mixed bad (sometimes it's fine, and sometimes it improves, and sometimes it gets worse).  Can the authors comment further?

Minor issues:
-- I think the claim of "will increase" in Table 2 is slightly problematic.  I have seen papers promise these things and then fail to deliver.  I would remove such speculative wording.

-- Some other challenges/competitions might be relevant and worth citing:
http://explochallenge.inria.fr/index.html

correctness: Seems correct to me.
clarity: Very well written.
relation to prior work: The discussion with prior work is very nuanced and well appreciated.
additional feedback: Great paper.
review_title: Outstanding Benchmark

Meta-review: The main strength of this paper is that having two policies collecting the logged data (with recorded action probabilities) dramatically simplifies the OPE benchmarking. However, the main weakness is that the feature-hashing anonymization method is removing a majority of the information contained within the context vectors.  Ultimately, the feature-hashing anonymization seemed like a severe limitation for the usefulness of the proposed benchmark.
Decision: Reject