Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces new benchmarking tasks for evaluating model uncertainty for diabetic retinopathy classification using two existing datasets. Using simulated experimental configurations, it explores how their approach can be used to evaluate performance of Bayesian and non-Bayesian models in cases of distribution shift in data source (e.g. patient population) and label coverage. The work is also accompanied by a well-structured code release that may help simplify interacting with these models.

Overall, the work shows promise but could benefit from more experiments and detail for uncertainty quantification and analysis

strengths: 1.	The work tackles an important problem of contextualizing model performance based on model uncertainty, which is critical for safety-critical applications.
2.	The code release appears to be quite polished and should help simplify user interaction and future experimentation of these methods 

weaknesses: 1.	The work mentions that no existing dataset benchmarks exist for real-world tasks, but there are several datasets and analysis done for other safety critical, real-world tasks such as the KITTI dataset for autonomous driving [1, 2].
2.	While the work makes a point of emphasizing the importance of uncertainty quantification, details regarding how the uncertainty metric is quantified is unclear in the main text and appendix. This makes it difficult to follow how these metrics are calculated for the different methods, and how this calculation impacts the selective prediction evaluation.
3.	The work also converts a multi-class problem into a single class problem. While this may be commonly done for previous DR challenges, this work can benefit largely from exploring the dynamics of uncertainty in this multi-class setting where the classes are grades of severity of DR. Taking the Severity Task as an example, it would be useful to quantify how uncertainty for predictions is correlated with the non-binarized severity of the disease. This may provide more insight into the observation that there was minimal loss in performance when evaluating on the OOD test set in the Severity Task.
4.	The evaluation criteria for both Country and Severity shift are different – AUC vs Accuracy. Is there a reason for this? 
5.	The authors should consider adding differences in image pre-processing methods as an additional benchmarking task. While such a task is not purely a naturally occurring phenomenon, it has been shown to be quite even within the Kaggle challenge that the authors mention.
6.	While it is perfectly reasonable that the work uses existing datasets in a structured way, it would be helpful for the authors to comment on ethical considerations and usage of these datasets.

[1] http://www.cvlibs.net/datasets/kitti/
[2] M. T. Le, F. Diehl, T. Brunner and A. Knol, "Uncertainty Estimation for Deep Neural Object Detectors in Safety-Critical Applications," 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018, pp. 3873-3878, doi: 10.1109/ITSC.2018.8569637.

correctness: The claims made are correct and benchmarking is done appropriately.
clarity: The paper is well written
relation to prior work: The authors appear to do a reasonable job of covering literature related to uncertainty quantification in the DR field, but it would be helpful to talk about other real-world benchmarking datasets for uncertainty and how their analysis is similar and/or different from methods proposed in that work.
additional feedback: Nit: Green dot is hard to see on Fig. 4/5
review_title: Good paper, may require additional details and experimentation for benchmarking tasks and uncertainty quantification

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a suite of tasks and pre-trained models for benchmarking Bayesian deep learning techniques in the context of diabetic retinopathy detection. Two pre-existing datasets (EyePACS and APTOS 2019) containing high-resolution images of human retinas are curated to develop benchmarks in which models are trained, and then evaluated on distributionally shifted data. The authors propose two tracks in which realistic distribution shifts are applied to the evaluation data: 1) images acquired in a different country than the training data (Country Shift) and 2) images with higher severity of diabetic retinopathy than the training data (Severity Shift). For each track, a selective prediction task is performed in which the uncertainty estimated by the Bayesian neural network is used to refer images to a specialist if the network is unsure about its prediction. The authors demonstrate that their proposed tasks are more informative about real-world model performance than standard metrics, such as prediction accuracy and receiver-operator characteristic curve analysis.
strengths: To my knowledge, there is no other dataset or benchmark that specifically aims to evaluate performance of Bayesian deep learning approaches. This is the major strength of this paper, and will be appreciated by the broader community. The paper also has important social implications, since it aims to evaluate how a medical diagnosis system would perform in a real-world clinical setting.

weaknesses: One thing that is missing from the documentation is the expected format of the data. I realize that this paper is about a benchmark, and not a dataset. However, someone that reads this paper may want to take their own retinopathy dataset and apply the authors’ code. In this case, there is no documentation about how to structure the dataset in a way that is understandable by the authors’ codebase.
correctness: The authors correctly state in the title, and throughout the paper, that they are proposing a benchmark (not a dataset). The dataset described in the paper consists of two publicly released datasets which have been curated by the authors to form the two selective prediction tasks. The experiment design makes plenty of sense for benchmarking Bayesian deep learning approaches.
clarity: The paper is well-written, and easy to understand.
relation to prior work: It is clearly discussed how this work differs from previous contributions, specifically Refs [13] and [29].
additional feedback: -	I see that the APTOS 2019 dataset was graded by clinicians using the same 0-4 scale as the EyePACS dataset. You may want to mention this in the description of the APTOS dataset
-	Keep the symbol for “referral rate” consistent. You define it as “X” in Section 2.4, then redefine it as “\tau” in Section 2.6. Also note that “\tau” is already used to denote the precision in the prior distribution over the network parameters in Section 5.1.
-	In Figure 6 (supplementary information) – are these images only from the EyePACS dataset? Or do these include APTOS images as well? It will be helpful to the reader to make that distinction. Also, including example images from both datasets may help the reader further understand the difficulty of the Country Shift prediction task.
review_title: This paper is the first present a suite of tasks for benchmarking Bayesian deep learning techniques in the context of diabetic retinopathy detection.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce an uncertainty benchmark for diabetic retinopathy detection. They propose two tasks, both focused on out-of-distribution data. The first task trains on a dataset and tests on another, while the second task trains on mild to moderate cases and tests on severe cases of diabetic retinopathy. A multitude of uncertainty methods are benchmarked on these two tasks.
strengths: *The paper is well structured and written using a clear language.
* Both tasks are similar to what a real-life predictive model may encounter, depending on its training data and intended usage.
* A wide variety of uncertainty methods are benchmarked on the two proposed tasks.
* All data and code used in this paper is publicly available.

weaknesses: * For a more thorough benchmarking process, the addition of a more modern network architecture (e.g. EfficientNet) could be beneficial.
* Estimated Calibration Error (ECE) is a commonly reported metric in uncertainty literature. Adding it would give readers a better understanding of the models’ performance and would make future comparisons easier.

correctness: To my understanding, the evaluation methods and metrics are sound.
clarity: The paper is well structured and the language is clear.
relation to prior work: Prior work is discussed with a focus on how this paper improves over the previous work from the same authors.
additional feedback: -
review_title: Thorough review of Bayesian methods for uncertainty

Meta-review: The reviewers all liked the paper. The authors' response clarified some important points. In view of that, the authors are strongly invited to take the feedback on board for the final version.
Decision: Accept