Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, their goal is to improve calibration and accuracy by augmenting a classification model with a GP. They base their model off RIO (ICLR 2020) which targets regression problems and tries to predict the residual between predicted value and true value. They propose a model, RED, which instead tries to predict the residual between the predicted confidence score for the true class and 1 — the true class target confidence score using a GP. They show strong improvements over the methods they compare to for 125 UCI datasets and CIFAR-10 dataset.

I find the approach interesting though the novelty is incremental over the RIO paper. My main concern is that I think some additional methods need to be compared with. For example [1] uses a bayesian last layer which is something that should be compared with. Using an ensemble of single layer NNs for the last layer or using MC-dropout at test time (which is known to approximate Bayesian inference under certain conditions) would also be interesting.

[1] “Scalable Bayesian Optimization Using Deep Neural Networks” by Snoek et al. 
[2] “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning” by Gal et al.

Edit: Based on the author response in terms of adding additional experiments, I'm raising my score to a 6.

review_title: Key comparison methods missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update: Following the authors' clarifications and additional experimental work, I'm increasing my rating to 6.

This paper proposes RED, a framework for detecting misclassification errors, based on regression of target confidence scores and application of a Gaussian process for uncertainty in predicted confidence scores. It builds upon RIO, a framework for predicting residuals of regression models and their uncertainties using GPs. Compared with other confidence metrics, RED aims for greater separability between correct and incorrect predictions.

The method is straightforward to implement and performs well against the baselines considered on classification tasks for 125 UCI datasets. However, I question whether the baselines are sufficient; it is not demonstrated whether RED would outperform other confidence scoring and OOD detection methods mentioned in the related work section, such as temperature scaling (or the related method ODIN, proposed in Liang, S., Li, Y., and Srikant, R., 2017. Enhancing the reliability of out-of-distribution image detection in neural networks.) or simply the entropy of the softmax predictions. Unless there is a good justification for the limited set of baselines, I believe the paper's claims to generality are limited.

Additionally, for the OOD detection results shown in Figure 3, why were AUROC and AUPRC not reported? While the scatterplots show separability of OOD data visually, these metrics (used elsewhere in the paper) would give a better indication of performance (and again, I think a greater range of baselines and tasks would be necessary to make any firm claims about OOD detection).
review_title: Well-performing, simple to implement method for classification error detection; limited set of baselines may not establish generality

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper solves an interesting problem of predicting uncertainty in NN without re-raining/modifying the existing NN. The authors propose a framework to calculate a confidence score for detecting misclassification errors by calibrating the NN classifier’s confidence scores and estimates uncertainty around the calibrated scores using Gaussian processes. This framework is called RED (Residual i/o Error Detection). 

This paper is also technically sound and to the best of my knowledge is novel and relevant to the community. 

It would be good to apply SVGP directly to some of these datasets and compare the results against NN+SVGP results.

You use the term “calibrated” confidence score/prediction. Could you explain what do you mean by calibrated?

I find the presentation of results very confusing. For example, in Table 1, AP-Error is smallest for the RED method and in Table 3 AP-error is the largest for the RED method. In both cases, it is mentioned that the RED method outperforms other methods. 

You mentioned ConfidNet outperformed the MCP baseline by a margin of 0.42. I do not see this number on the table.

It would be good if the authors could mention in the paper what is RIO short for.

You mentioned that you need to extend the kernel to multiple output kernel. Could you explain a bit more about that and how you build it?

review_title: Adding confidence score to NN classifiers without retraining or modifying the model

final_score: rating: 6: Marginally above acceptance threshold
review_content: #######################################################
SUMMARY

This paper introduces RED, a new methodology to produce reliable confidence scores to detect missclassification errors in neural networks. The idea is to combine kernels based on both input and output spaces (as in RIO) to define a (sparse) GP that estimates the residual between the correctness of the original prediction and the maximum class probability. The authors show enhanced performance against other related methods and the ability of RED to detect OOD and adversarial data through the variance of the confidence score. 

#####################################################
PROS

1) Obtaining confidence scores for neural network predictions is a timely and very relevant topic for the ICLR community, since it is one of the main limitations of real-world applications of current neural nets.

2) The related literature review is clear and, to the best of my knowledge, the proposed metholody based on Gaussian Processes is novel. 

3) The experimental validation of the proposed method on the UCI datasets is strong. It uses a wide range of datasets and several statistical tests, and RED obtains superior performance.

4) The idea of using the variance of the proposed confidence score to identify OOD and adversarial data is interesting and promising.

######################################################
CONS

1) My main concern is that the contribution in RED can be regarded somehow incremental given the RIO approach. It utilizes the same rationale behind RIO, and just adapts the necessary components so that it works in classification. The adaptation of these components is also straightforward: the output kernel now works on several dimensions (instead of the scalar dimension of regression) and the target is now the correctness of the original prediction. 

2) The experimental validation focuses on several competitors which can be considered "of the same family" as the proposed approach. Namely, all of them calibrate the predictions of a pre-trained neural network. I think it would be interesting to also compare to a different "family" of methods. For instance, (Functional) Bayesian Neural Networks are meant to obtain calibrated predictions by leveraging epistemic uncertainty (that coming from the model parameters). 

3) I do not fully understand the relevance of the experiment with the large deep learning architecture given by the VGG16 model. Since the proposed method works on the pre-trained neural network, my understanding is that the complexity of the neural network itself is not relevant for the performance of the proposed approach. Also, in this experiment I miss several independent runs to assess the results variability. 

####################################
Additional questions/feedback:

1) In the second paragraph of section 4.2., there seems to be a typo when reporting the margin. It is said 0.42 and 0.55 for ConfidNet and RED respectively, but I think it should be 0.042 and 0.055 by looking at Table 3.

2) It is not entirely clear to me why the process described in section 4.3. (second paragraph) produces proper OOD and adversarial data. For instance, some of the intended OOD data could be similar to training data (specially because the latter is being normalized to mean 0 and std 1). And similarly for the adversarial case. I think this could be better explained.

3) When it comes to real practice, a key decision is to set a threshold on the confidence score to decide what instances should be supervised by an expert. Is there any recommendation on this?

####################################### 
AFTER REBUTTAL

The new baselines added make the experimental validation more convincing. Therefore, I have raised my rating to 6 (Marginally above the acceptance threshold). However, I still believe that the contribution is incremental, and I think the paper would gain in terms of novelty if it focused more on the detection of OOD data and adversarial attacks (which right now is more like a preliminary test).

review_title: Interesting problem and results, although the approach seems a bit incremental

Meta-review: In this paper, the authors use a GP classifier to detect if the output of a NN classifier has been decided correctly. The GP takes as input the original input vector x and the output of the NN, i.e. the calibrated posterior probabilities given by the NN. It uses that as an input vector for the GP classifier to decide if the sample was correctly decided. The output of the GP will serve as confidence in the output of the NN. The results are comparable/superior with the state-of-the-art and the authors have repeated the experiments with over 125 different datasets. The reviewers of this paper were all cautiously positive about the paper, but all of them pointed towards the reduced novelty of the paper. Also, none of the reviewers were willing to champion this paper as a must-have at ICLR 2021.
 
For my reading of the paper, I would tend to agree with the reviewers’ comments. Also, I find that using the same NN, rather shallow, with the same configuration for all the datasets seems rather limited. Given that this method is independent of the underlying classifier and that the databases used are low dimensions and a low number of training examples, I would have liked to see what a random forest or a GP can accomplish. Also, I would have used bigger NNs that can be trained to overfit the sigmoid outputs for classification of higher accuracy. I believe that having a diversity of underlying classifiers is more relevant than having 125 datasets. We need to find the best classifier or ensemble and then apply the different mechanisms for estimating if the output is the correct one. Otherwise, the proposed method might only be workable for this specific NN configuration. In the tables, it can be hinted that this might be happening, as about 80% of the cases MCP and RED are indistinguishable in the AUROC values.
 
Also, for all of these datasets a GP could be used as an underlying classifier, and given the premises of this paper, the authors could check how well calibrate a GP classifier is. Also, there has been considerable work on calibrating NNs when they are trained to overfit. Comparing with those methods should be straightforward, as they provide more information than just a confidence score. This is probably the most influential paper: https://arxiv.org/abs/1706.04599 (1000+ references), but there are some recent papers too.  
 
Finally, if the goal is to use a GP to detect if the classification done by the NNs is accurate, using a GP might be an overkill, as the complexity of the GP, especially for large datasets might end up being larger than the underlying classifier.

Decision: Reject