Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: To examine the intrinsic soundness of the classifier (maybe formulated as a deep neural network), this paper proposes a new method to estimate the limit Bayes error, where we just take the mean of the labels that show the uncertainty of the classes. The proposed method is claimed to be model-free and instance-free, and experimental results show its effectiveness. 
strength_and_weaknesses: Strengths: 

* This paper is well-written, and there are many mathematical formulations to formally illustrate the logic and soundness of the proposed method. 
* The experimental results seem promising. 

Weaknesses:

* The proposed method seems simple. However, it is not clear in practice how a practitioner implements the whole method.

* The method is supposed to be instance free. However, I still see that real instances would be desired in experiments.  
clarity,_quality,_novelty_and_reproducibility: This paper is well presented. But it would be favored if a flowchart of the whole method can be provided. 
summary_of_the_review: Please see the above analysis.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an instance-free and model-free Bayes error estimation method for weakly supervised data. It leverages the fact that the expectation of the (perfect) predicted probability is the unbiased estimator of the Bayes error. Further, in a more practical and general case where the noises exist, the estimator is asymptotically unbiased. The authors also studied one case where only positive data and their confidence are available, and they show the error is also bounded. 
strength_and_weaknesses: Strengths: 
- The concept is conceptually simple and well-motivated. 
- The presentation is clear overall. 
- The experiments involve ``real-world'' scenes which may bring practical interest
- The proposed estimators works (except on ViT and some datasets) better than other trained models in estimating the Bayes error. 

Questions:
- Some reference are missing. For example, page 4 has a ''Sec. ??'' and page 8 has a ''App. ??''. 
- One paper [1] might be worthy to discuss (and maybe compare) but seems to be missing. It is also a model-agnostic algorithm for estimating the Bayes error, although they require the access to instances. 
- I might be wrong but since human labeler are also imperfect, why is it not possible that ViT can outperform human in providing reliable confidence? 

[1] A Model-Agnostic Algorithm for Bayes Error Determination in Binary Classification
clarity,_quality,_novelty_and_reproducibility: The presentation is clear overall - some references need to be fix (see the above paragraphs). The novelty and quality are good. 
summary_of_the_review: The paper provides an interesting method for estimating the Bayes error that is conceptually simple but theroetically grounded. I believe it would be beneficial to the community and inspire future research.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple estimator for Bayes error in binary classification. The authors are able to show several properties of the proposed estimator including unbiasedness and some convergence guarantee. Experiments on synthetic datasets, CIFAR-10, and Fashion-MNIST show the effectiveness of the proposed estimator. 

strength_and_weaknesses: Strength: The proposed estimator is simple to implement. Also it is nice that the authors show some guarantee of the proposed estimator.


Weakness: I feel the proposed estimator does not really resolve the core of the Bayes error estimation. For the theory part of the paper, e.g., Theorem 4.1 and Proposition 4.2 and 4.3, the assumption is r_i= p(y=+1|x_i) is known. But to me this is actually the core and hard part of Bayes error estimation, since given r_i one can easily get the Bayes estimator. 


In the empirical section of the paper, p(y=+1|x_i) is estimated using deep models, but as we all know from the uncertainty estimation literatures, deep models are not doing a good job calibrating their output scores to p(y=+1|x_i), and their outputs are usually highly screwed. As a result I am not quite sure how I should trust the estimated Bayes error.


Experiments are also kind of lacking. I only see two real-world datasets used. To fully show empirically the proposed estimator work I would like to see more numbers on other datasets.

clarity,_quality,_novelty_and_reproducibility: The presentation of the draft is clear to me. The proposed estimator is simple and straightforward. I am not sure if it has been proposed before or not. 

summary_of_the_review: Overall I feel the work is an interesting try and it does show some promising preliminary results. But as I stated in the weakness section I do have some serious concerns. The assumption that p(y=+1|x_i) is known is so strong that the problem is not interesting any more in theory. In application, the work does not propose convincing ways of estimating p(y=+1|x_i) either. And the experiments are not extensive either.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method of estimating the Bayes' error of a class of models for binary classification. They compare the estimation with datasets with multiple human annotated labels to compare the estimated Bayes error with an empirical estimate of the Bayes error. 
strength_and_weaknesses: ## Strengths

- The authors provide many rigorous proofs to support their claims. 
- The empirical results provide further support, showing bias where expected, etc. 


## Weaknesses

- Theorem 2.2 is introduced from previous work without much context around why it is useful or important for this work. It would be helpful to the reader to provide some context of where and why it will come into play later in the text. 

- Until section 3, there is no mention of where the soft labels come from. It seems that the soft labels would have to be soft labels that are given from the true distribution of the data, which it seems would be impossible to attain in most realistic circumstances. Although, this is explained later, I think it would reduce reader confusion by at least alluding to this fact and that it will be dealt with later.

- Section 3, paragraph 3: Why can uncertainty labels not recover $p(y = +1 | \mathbf{x})$ from $\min_{y \in \pm 1}p(y | \mathbf{x})$? If one has access to one, doesn't it guarantee we know the other? If $\min_{y \in \pm 1}p(y | \mathbf{x}) = 0.1$, then isn't $p(y = +1 | \mathbf{x}) = 0.9$ If this is a subtle misunderstanding about the meaning of soft labels and uncertainty labels introduced in Section 1, then I think there needs to be more time devoted to explaining those concepts in detail.

- Theorem 3.3 seems to show that if one has a noisy corruption of the true class of an instance, then $\hat{\beta}$ is an unbiased estimate of the Bayes error. Later on in page 4, the authors state "In practice, one idea is to ask many labellers for each instance and then use the histogram of the hard labels as a noisy soft label." Therefore wouldn't this just produce an unbiased estimate of the Bayes error of the aggregated labellers, which is not in fact an unbiased estimate of the true Bayes error? For example, the estimated Bayes error of a ResNet on CIFAR-10H is higher than that of the human annotated $\hat{\beta}$ in figure 4. This might be an obvious point, but I still think it should be stated clearly somewhere in the text.    

- It would be interesting to see the other baseline performances on something other than the toy dataset. For example, ensembles tend to not show much diversity in decision boundaries when considering toy datasets, and therefore often do not outperform simple MLP baselines by much. This changes when the dimensionality of the data increases. It would therefore be interesting to see what happens with the Bayes error of an ensemble of at least one of the network baselines in figure 4.


## Minor

- Page 4: reference error, "which we use later on in Sec. ??"
- Page 8: reference error, "(App. ??)"
clarity,_quality,_novelty_and_reproducibility: ## Clarity

Other than minor points raised above, the text is clear. 

## Quality

The overall quality is good.

## Novelty

To my knowledge, the findings presented are sufficiently novel. 
summary_of_the_review: Overall, I feel positive about the results presented in the text. I raised a number of points above, and I look forward to hearing the responses from the authors regarding the points raised.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an instance-free, hyperparameter-free and model-free Bayes error estimation method for weakly supervised data. The authors point out that the expectation of the (perfect) predicted probability is the unbiased estimator of the Bayes error. Further, in a more practical case where the noise exists, the estimator is still asymptotically unbiased. This method is conceptually simple, well-motivated, and practically tested in a partial label setting where positive data and their confidences are available. Writing is clear and logical. 

The experiments involve ``real-world'' scenes which may bring practical interest. During rebuttal, the authors provide another set of real-world experiments (acceptance/rejection binary classification task for ICLR papers) to further justify their method's robustness. I applaud the authors for being very thorough, solid, and convincing in their rebuttal process. 

After rebuttal, three out of four reviewers unanimously found this work very interesting and rate it highly as 8. The only negative reviewer 5ZHz (rating as 5) didn't respond to either the authors' rebuttal (which seems to have addressed all raised questions and confusions) or AC's discussion request, despite multiple reminders: hence his/her rating has been down-weighted. 


justification_for_why_not_lower_score: Overall, this paper (1) studies a novel problem with practical motivation; (2) brings in well-grounded theory insights; (3) is extensively tested on real data, including an interesting acceptance/rejection binary classification task for ICLR papers; (4) is well written. 

The current average score is 8-8-8-5, while the score 5 should be down-weighted since the reviewer was unfortunately not responsive. Hence, I consider this paper top-notch quality among all submissions within my batch, and would like to recommend oral.
Decision: Accept: notable-top-5%