Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper describes an approach to improve the confidence on deep
neural networks (DNN). The proposed approach uses a distance-based
approach (distance to prototypes) and train using a confidence
model with the classification model using mis-classified examples.

It first learns using a loss function based on pre-computed centroids
of each class evaluated from the training examples. At inference it
assigns the class with the closest center.

To estimate a confidence level, it learns a new model to estimate the
distance using only the misclassified examples.

The authors experimentally showed the benefits of the proposed approach
over several methods.

Finding prototypes and training with distances, on one hand, and
evaluating confidence with Gaussian, on the other, assumes
that the classes are "well defined". Although the authors show
promising results, it is not clear how well the proposed method will
behave in more challenging classification tasks where the classes are
mixed.

Although the authors show that estimating the confidence with
misclassified examples works better I will like to see a further
analysis in the paper.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper addresses the (long-standing) problem of classification systems being able to output reliable confidence estimates on its own output. The selected approach is to use the distance to (previously computed) class centers in multi-class classification to help compute the confidence interval. The method is shown to have comparable results on well known datasets but higher efficiency than 2 rival methods: ensemble ANNs and Bayesian neural networks. I would point out that such confidence intervals are all intuitive and have no statistical basis or other independent means of empirical validation. Experienced practitioners are aware of this, but I see that the paper steers wisely clear of overambitious claims. The general intuition of hybrid supervised and unsupervised learning for C.I. (or ellipse) estimation is not new, but an effective and compact representation of this intuition in a DNN context is a valuable contribution, well-placed in the literature - I recommend acceptance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
The paper proposes a method to do confidence calibration for deep neural networks. It uses standard
episodic training for prototypical networks, and first shows empirically that the distances of the
embedded test point to its ground truth class center embedding (*not* the predicted class embedding)
are indicative of the confidence of the prediction. Further it proposes to exploit this by training
an auxiliary confidence prediction MLP carefully. To do so they demonstrate that the training needs
to be done of erroneously predicted training examples cf. all the traning examples. They show
results with MLP+MNIST, VGG11+CIFAR10, ResNet50+CIFAR100 and ResNet50+TinyImageNet.


Detailed comments:
The paper is interesting but largely empirical. It shows empirically that:
1. when prototypical networks (and episodic training) is used the distance of test example to true
class center reflects the confidence.
2. this does not hold when `vanilla' training is used
3. an auxiliary MLP can be used to learn to predict this while training only with erroneously 
classified training examples cf. all training examples
                                      
The results are reported on three networks (MLP, VGG11 and ResNet50) on different benchmarks of 
image classification. The confidence prediction improvements wrt baselines are non trivial, while 
keeping the accuracy similar, and the computation cost lower than competing methods. Ablations 
studies are also convincing.                    
            
I would have two broad critical comments on the paper:
1. Would this generalize to other image classification tasks and datasets. Generally distance
(embedding) based networks perform less than softmax based networks on bigger datasets, so an
immediate disadvantage if that happens, is that you would be trading off accuracy cf. vanilla
networks, for better confidence prediction using the required distance based network here.
2. The vanilla training is never formally detailed. I am assuming it was softmax + cross entropy loss 
with gradient descent. Would some other loss be helpful? Specially the metric learning based losses like 
contrastive or triplet losses come to mind, since they are also distance based. Does the method work
with prototypical networks only or it generalizes to other distance based methods as well?
                                                                                    
Minor comments:
The notations are a bit confusing sometimes, and require going back and forth a bit. Eg. \mu is used
for representation of feature (Eq4) while it usually denotes a mean of some sort (so the reader's expectation
could be that it represents class center). Similarly, boldface p is used for class centers, which is
again a bit confusing as being a probability of some sort. In general the notations are different
from the original prototypical networks paper (which I needed to revise); keeping them similar would
help the reader. The contribution of the present paper is more than that anyway.

The erroneously classified training examples are called errors (eg. just before eq8). By errors one
could think that it is a difference between some sort of prediction and the ground truth. Explicitly
calling them erroneously classified training examples would help the reader as well.

The notation \odot is not explained (before eq.9).

review_title: Official Blind Review #1

Meta-review: All reviewers voted to accept this paper.
The AC recommends acceptance.
Decision: Accept (Poster)