Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the important problem of detecting misclassifications in a classification system. The authors consider two scenarios: total black box setting in which only the softmax predictions are available, and a partial black box setting that assumes access to the model parameters. First, the authors derive a bound to approximate the unknown classification error using a combination of known statistics. Based on this, the authors propose a simple method that can detect misclassifications. The method is shown to work well empirically on some benchmark datasets.
main_review: First, in Section 3, the authors derive a bound to approximate the misclassification error using known model statistics. This section was a bit hard to parse, and it took me a while to fully understand. The authors could work on explaining this section better. For instance, the explanation of quantity $\hat{Pe}(x)$ in Equation 2 can be improved. But other than this, this section was interesting to read.

I took a quick glance at the proofs. They seem to be correct, although I did not review them very carefully.

The experiments were conducted well. It was interesting to see that the method consistently improved in most settings compared to softmax response and MHLNB. Overall, I like the results. In all experiments, I feel running them multiple times and reporting aggregate statistics might be beneficial.

Some comments:

Is it possible to combine D_\alpha and D_\beta into a single discriminator?

How to choose which one of D_\alpha or D_\beta to use?

Can the method be used just for OOD detection? I see that authors reported results of misclassification detection in the presence of a fraction of OOD samples in Table 2. But I would be interested to see the performance of just the OOD detection, and compare it with some standard benchmarks.

It would have been good if authors could discuss the failure modes of the approach.

limitations_and_societal_impact: I think the authors could include a discussion on the limitations and the failure modes of the method. In addition, if there is a way to combine D_alpha and D_beta into a single method, that would be nice as well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method for detecting misclassified samples in neural networks.
The proposed method aims to approximate to error function of true probability density (which is unknown) of labels given data by using known statistics.
To achieve this, known statistics that bounds the true error function are derived in the paper which are used to define discriminators to determine correctly and incorrectly classified samples.
The first discriminator denoted as D_\alpha uses sum of squared softmax probabilities and the second discriminator is based on the predicted model for the posterior class probability denoted as D_\beta.
The methods is tested in 2 settings: 1) Partially Black Box (PBB) where it is assumed that gradients of the network w.r.t input is available for input perturbation which was shown to be useful for OOD detection in earlier work, 2) Totally Black Box (TBB) where it is assumed that only softmax probabilities are available.
The proposed discriminators are compared with well-known OOD detection methods: ODIN and Mahalanobis on multiple image and text datasets.
The results demonstrate that the proposed framework (DOCTOR) outperforms OOD detection methods on detecting misclassification error.


main_review: ** Strenghts **

- The proposed discriminators are very simple yet seems very effective based on the results presented in the paper.
- The paper presents statistical derivations of the proposed discriminators which supports why the proposed discriminators work.
- Extensive experimental evaluations on both image and text dataset demonstrate that DOCTOR can detect misclassification error on the dataset they are trained on (see Table 1)
- The paper also presents experiments when OOD samples are given in test time along with the in-distribution samples. In this case, networks can make mistakes on both in-distribution and OOD samples. Experimental results show that DOCTOR can detect misclassified samples in this setting as well. (see Table 2)

** Weaknesses **
- The paper defines 2 settings for the experiments: PBB and TBB. Most of the state-of-the-art OOD detection methods (e.g. Mahalanobis) focus on fully transparent setting where they assume that intermediate layer representations of the networks are available. I understand the motivation behind using PBB and TBB that they are more realistic settings. However, I believe it is crucial to compare the method with the fully transparent setting of the existing algorithms to better position the method within the related literature.

- The experiments of detecting misclassification error in presence of OOD samples are a bit limited. The results are only compared with ODIN in this setting. I would expect to see more comparisons with some of the recent and more powerful methods such as Mahalanobis, Energy-Based OOD detection [1], Generalized ODIN [2].

[1] Liu et al. Energy-based Out-of-distribution Detection
[2] Hsu et al. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

-  I believe a general problem with the OOD detection literature is that in-distribution and OOD datasets are quite dissimilar and most of the recent OOD detection methods perform quite well on these benchmark. A more challenging problem that needs to be solved is detecting OOD samples when OOD images are quite similar to each other. There are some works focuses on such settings e.g. where CIFAR100 is in-distribution and CIFAR10 is OOD [3] which achieve AUROC around 0.85. I think it is important to perform experiments on such more challenging setting to see the limits of the OOD detection methods.

[3] Zhang et al. Hybrid Models for Open Set Recognition
limitations_and_societal_impact: - Please see weaknesses section about for the limitations

- I don't see any potential negative impact of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Motivated by the tendency of DL models to be overconfident, this paper introduces DOCTOR, a framework for determining when to accept/trust model decisions. The authors derive the tradeoffs of accepting an incorrect classification and rejecting a correct classification to motivate the formulation of DOCTOR, a set of discriminators which takes into account the full softmax values of the model to minimize the occurrence of type 1 and type 2 errors. DOCTOR is able to match or outperform prior works on both in-domain and out-of-domain problems for both totally black box and partially black box models. 
main_review: I appreciate the authors concern with addressing the issue of whether or not to trust model, as this is incredibly important when applying ML to things like healthcare or finance. The theoretical analysis seems very well thought out and a good motivation for DOCTOR, and the explanations of previous works was very helpful in the experiments. However, the method section is hard to follow due to the sheer amount of notation and minimal verbal intuition; I am still unsure of exactly how the softmax is used within the DOCTOR discriminators. This could be simply because I am not as familiar with this specific line of work, but if the authors could add a bit more intuition similar to what was done in the ODIN paper it would increase the ease of understanding significantly. 

My other main concern is the results; it seems like DOCTOR does not achieve significant gains over ODIN or SR, with most gains being <0.5%. Since y'all are using pertained models, is it possible that if these models were trained with a few different initializations that DOCTOR would not outperform the others? Again I think the impact of DOCTOR could be great as it is lightweight and can be applied to many different setups, but I would need more empirical evidence that it is in fact the best choice. 

Other nits:
* Why is ODIN not bolded for TinyImagenet in Table 1? It seems to have the same TBB AROC and TBB FRR of D_a, D_b, and SR. 



Update: I appreciate the authors clarification on DOCTORs use of the softmax and their explanation as to why as lower FRR while maintaining the same AUROC can be quite useful in practice. Due to this and the fact that DOCTOR has significant theoretical backing and seems relatively easy to use, I have changed my score to a weak reject. Again I want to thank the authors for their thoughtful response!
limitations_and_societal_impact: The authors did explain that "practitioners who lack the necessary knowledge and skills may incur misinterpretation of the results obtainable with the proposed method", but it would be helpful to go more in depth on how exactly these results could be misinterpreted or under what situations DOCTOR is not suitable for use.
review_title: null

Meta-review: This submission suggest a new method for detecting out-of-distribution samples for object detection and the reviewers seem to be in agreement that the submission has sufficient novelty and potential impact for an acceptance. Given the detailed new experimental comparisons with other methods, I suggest that paper is accepted for a spotlight presentation at NeurIPS.
Decision: Accept (Spotlight)