Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - The authors propose FeeBee, which is an evaluation framework for bayes error risk (BER) estimators. FeeBee can be used to benchmark BER estimators on real-world dataset because it doesn't rely on knowing the ground-truth BER and data distribution unlike previous evaluation frameworks
- The authors apply FeeBee to benchmark 7 BER estimators on 6 real-world datasets
strengths: - The paper proposes a novel framework for evaluating BER estimators without knowing the data distribution or the ground-truth BER values, based on a simple, but novel technical result. This addresses the key limitation of prior evaluation methods, and allows for evaluation on real-world datasets
- The paper evaluates a thorough benchmark of 7 BER estimators on 6 datasets (text and images) 
- The paper is written clearly
weaknesses: - At the end of section 3, the authors say that FeeBee needs a strong SOTA value. How strong of a SOTA value is necessary, and how sensitive are the results to this? 
correctness: The paper is correct to my knowledge. 
clarity: The paper is written clearly.
relation to prior work: The contribution wrt the prior work is clear. Unlike existing evaluation of BER estimators, FeeBee doesn't rely on knowing the ground-truth BER or the data distribution, and this allows FeeBee to be applied on real-world datasets, not just synthetic ones. FeeBee estimates BER by injecting different amounts of label noise into the datasets and applying their technical result from section 3 to derive an estimate. 
additional feedback: Typos
- Line 86 "unkonwn"
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a practical framework for evaluation of base error rate estimators on real world datasets. Instead of evaluating BER on a single point, feebee, uses the relative relationship among BERs on a series of points. This is done by injecting a controlled amount of label noise to datasets and evaluating how BER changes i.e. under estimates or over estimates the noise amount.  
strengths: - Strong and extensive evaluation. Deployed on 6 datasets of text and image, evaluation on 7 BER estimators. 
- Good analysis of various BER estimation methods. 
weaknesses: - in the terms of methodology flipping labels at random may not be the best way to create noisy labels. This is not very realistic comparing to human raters. In reality disparity of vote among raters indicates difficulty of a specific instance which is not taken into account here. 

correctness: Evaluations and claims are correct to the best of my knowledge. 
clarity: Yes
relation to prior work: Yes
additional feedback: All has been mentioned above.
review_title: Interesting paper

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper is categorised as a benchmark paper for the Dataset and Benchmark track. I would like to preface this review with a statement that this paper is outside of the area of my expertise, so if any of my comments show trivial errors of judgement, please engage in rebuttals constructively!

The paper proposes the `FeeBee` framework to quantify the Bayes Error Rate (BER) for real-world datasets with no known probability distribution. The authors demonstrate this by injecting a controlled amount of label noise and performing multiple evaluations on a series of different noise levels. The authors implement and analyse a range of multi-class BER estimators on 6 popular ML benchmarks to demonstrate their framework. They conclude that GHP and 1NN are consistently outperforming other lower bound estimators whilst enabling easy hyper-parameter selection on a single point estimate.

The main contribution is the open-source framework `FeeBee`, the first system to systematically evaluate lower and upper bound estimators of the BER on real-world data under a set of assumptions. The framework is built to motivate further research in new BER estimators to enable progress in building better and more practical methods in the future.

strengths: To the reviewer’s best knowledge, the framework is the first of its kind for evaluating BER without the presence of a known probability distribution. The fundamental set of assumptions and theory used to construct lower and upper bounds seem sound. The paper conducts very thorough experiments to support the author’s hypotheses. Importantly, all the experiments are performed transparently with a Google colab notebook, which ensures reproducibility. 

Additionally, the paper is well written and manages to introduce technical content without being overwhelming to unfamiliar readers (such as myself).

An additional strength is the scope and applicability of this work, the majority of the NeurIPS community is likely to find this work relevant as it involves benchmark datasets commonly encountered in the literature. 

weaknesses: The main limitation of this work concerns the statement:

> A key assumption we made when choosing relevant datasets is that we assume that both the train and test dataset originate from the same distribution. With that in mind, we rule out datasets for which this is clearly known not to be true, such as ImageNet [13]. 

This paper is intending to address the question: 
>  (1) How well do they perform on real-world datasets?

Yet to the best of my knowledge, unfortunately real-world datasets rarely conform to those assumptions. Furthermore, all experiments presented in this paper are limited to MNIST, CIFAR, IMDB, SST2 and YELP, and as such I am not sure about how much this paper does to advance BER estimation on *real-world* data. I still however think it is a reasonable choice as a first step, and perhaps I would only change the wording of the abstract and contributions to reflect this.

Furthermore, this method relies on the presence of a SOTA value, which once again for researchers working on their own problems will not be known in advance.

correctness: Other than my note in weaknesses, the paper appears correct: benchmarks were chosen reasonably within the set of assumptions, and experiments were conducted in detail. The appendix also supplies more detail to help understand the reasoning presented in the paper. I will defer to the opinion of reviewers more knowledgeable in this field than myself to assess my final score post rebuttal.


clarity: The paper is very well written and relatively easy to follow even for those unfamiliar to the topic. Mathematical notation is well used to define quantities. As a few minor points I would like to state:

1. I would like to see figures with $U_\mathcal{D}(m)$ and $L_\mathcal{D}(m)$ on the same graphs, as at a glance it can reveal problems when the two cross over (e.g. kNN_Ext CIFAR10) and to more easily visualise the bounds between the two quantities. 
2. Please upgrade the format of the figures to be of `pdf` or other vector graphic format, so that for any DPI and zoom level all the lines retain the best clarity. Additionally, it is difficult to differentiate the colour scheme (e.g. Fig 3a, 1NN, DE-kNN, and kNN-LOO of Appendix Fig 4)
relation to prior work: There is no dedicated section on its relation to prior work, and while authors state that there is no other framework to evaluate BER on real-world data, a section between 1. and 2. could be added to describe applications on synthetic datasets for the reader to understand the differences in evaluation of the methods (between synthetic and real: in the presence of a prob. distribution vs in the absence). With the extra page limit afforded during the rebuttal phase, this could be a useful addition to further strengthen the work.
additional feedback: Please engage with my review in the rebuttal to help clarify a few points and I will update my score in accordance with other reviewers and your responses. Thank you for your understanding.

Edit: Based on author's responses and further clarifications, I have adjusted my score to 7.
review_title: A well-written, detailed set of benchmarks to evaluate BER estimators on a range of standard ML datasets

Meta-review: All reviewers agree on acceptance. I recommend the authors to take into account the reviewers' comments to improve the paper for its camera-ready version.
Decision: Accept