Question: Why don't the authors provide a sufficient explanation to avoid comparison to Mukhoti, Jishnu, et al. "Calibrating deep neural networks using focal loss 2020**?
Evidence: Training Datasets and Network Architectures We employ WideResNet28-10 (WRN) and ResNet50 (RN50) architectures that have been shown to produce state-of-the-art classification accuracies on real-world datasets. We train them on CIFAR-10 (C10) and CIFAR-100 (C100). For Domain-Shift experiments, we resort to the widely used CIFAR10-C and CIFAR100-C, corrupted versions of C10 and C100. For Out-of-Distribution detection experiments, following SNGP, we use C100 and SVHN as OOD for models trained on C10. Similarly, for models trained on C100, we use C10 and SVHN as OOD.
Methods considered for comparisons We consider both deterministic and Bayesian approaches for comparison. Following, we also create two additional strong and simple baselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN) and Stable Rank Normalization (SRN). Note, we are the first to consider SRN for these experiments as it induces more compact clusters in the feature space than SN (we provide a simple mathematical proof of this in the Appendix A). Therefore, we compare our approach with the following baselines:
• DNN: Standard deterministic neural network trained using cross-entropy loss.
• DNN-SN:DNN with SN.
• DNN-SRN: DNN with SRN.
• SNGP: Spectrally Normalized Gaussian Process.
• DUQ: Deterministic Uncertainty Quantification.
• Mixup: Standard Mixup training.
• KFAC-LLLA: KFAC-Laplace Last Layer Approximation. A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation. For the sake of completeness, we provide a simple outline of this approach in Appendix B. Code base For fair comparisons, we developed our own code base for all the approaches mentioned above (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain the strongest possible baselines. For SNGP, we used the available code and made sure that we follow exactly the same procedure as mentioned in their original paper. For DUQ, the original paper did not perform large scale experiments similar to ours. Unfortunately, we could not manage to make their code work on C100 as it exhibited unstable behaviour. For this reason, we borrowed numbers for DUQ from the SNGP paper. Please note that the authors of SNGP performed non-trivial modifications to the original DUQ methodology to make it work on C100. Further details provided in Appendix C.
We use SGD with Nesterov momentum 0.9 and a weight decay of 5 × 10 −4 . For WRN, we apply a dropout p = 0.1 at train time. We perform extensive cross-validation of all the hyperparameters for all the baselines. Details provided in Appendix C.
Evaluation Metrics For calibration, we employ: (1) the widely used Expected Calibration Error (ECE), and (2) the recently proposed Adaptive ECE (AdaECE). For all the methods, the ECE and AdaECE are computed after performing temperature scaling with a cross-validated temperature parameter. Metrics and uncertainty measures used for out-of-distribution detection are discussed in detail in Section 5.1.3.
A particularly thriving sub-field of research in Deep Neural Networks (DNNs) concerns devising efficient approaches towards obtaining reliable predictive uncertainty. NNs are known to be overconfident for both, in-and out-distribution samples (i.e. for samples coming from the same distribution from which the training distribution has been sampled (IND samples) and for samples not coming from such distribution), leading to highly unreliable uncertainty estimates. They can be wrong with very high confidence not only on the test data similar to the one they have been trained on, but also when facing previously-unseen conditions. The overconfidence problem becomes even more concerning when just slight changes in illumination, atmospheric conditions or in the image capturing process (domain-shift) can severely damage the actual accuracy of the model. A desirable property of any model is to be robust to such superficial changes that do not affect the label of the classified image, and to become uncertain (or indecisive) when exposed to samples from a distribution different from the training distribution.
While the literature suggests that classifiers whose predictive distributions increase their entropy the further away the test input gets from the training data are desirable, the implementation of models satisfying such a property is not scalable, and can only occur at the cost of performing crude approximations and by non-trivial modifications to the architecture of the neural network. Such modifications sometimes lead to degraded accuracy.
Our motivation behind this work is based on the following observations. (1) We find that a trained network (DNN) do not project out-of-distribution (OOD) and domain-shifted (DS) samples arbitrarily away from the training data. In fact, all the OOD and domain-shifted inputs that we considered (detailed discussion in Section 3) are projected within the smallest hypersphere that contains all of the IND test data. Therefore, it might not be necessary to enforce the network to be uncertain everywhere away from the data, and perhaps, in-distribution data can be used to mimic the regions where OOD and DS samples are being projected. (2) DNNs tend to embed OOD and DS samples in high confidence (low-entropy) regions (see Figure). This contradicts the desired ideal behaviour, that would require a model to map such samples in high-entropy regions.
Figure: DNN (Left), Ours (Right). Interpolation experiment on CIFAR10 to show embeddings of the linear interpolation of two randomly picked input samples from class 1 (purple) and class 2 (yellow). Red and green samples are classified as class 1, and orange and blue samples as class 2. As the color changes from red to green, the predictive entropy increases. Same for the color change from blue to orange. Note, DNN classifies interpolated points with very high confidence (low entropy) even if the samples shift drastically from the data. However, Mix-MaxEnt maps these samples into a wide high-entropy region. Details of this visualization are provided in Appendix F. More exhaustive visualizations of this phenomenon are provided in Section 5.2.2. Given the observations above, we propose a simple entropy maximization regularizer (called Mix-MaxEnt), that induces a high entropy barrier between class clusters, while the cross-entropy optimization objective keeps the entropy low close to the class clusters. The entropy regularizer is enforced for samples synthesized using the convex combination of a pair of samples from two different classes of the in-distribution training data. When combined with the crossentropy loss, this regularizer prefers a maximum likelihood solution such that the uncertainty of the network increases while moving away from the embeddings of one class in the direction of another, and learns features that are robust to local input perturbations.
Through extensive experiments using WideResNet28-10 and ResNet50 architerctures on CIFAR10 and CIFAR100 datasets, we demonstrate that our method outperforms all existing single model baselines in providing clean data accuracy. On Domain-Shift experiments, it provides remarkably improved accuracy compared to all the baselines including Deep Ensembles (DE). For instance, it provides 4.8% and 4.7% improvements over the highly competitive DE and SNGP, respectively, on CIFAR-10 using WideResNet. In terms of reliable uncertainty estimates, it is either the best or, in a few cases, extremely competitive with respect to the best performing one, with only slightly inferior performance. Overall, our experiments show that Mix-MaxEnt is by far the best performing one compared to the existing single model approaches.
We would like to highlight that, along with its effectiveness, one of the core strengths of our approach is its simplicity. As opposed to the recently proposed, it does not require any modifications to the architectures and does not trade accuracy in order to improve uncertainty estimates. And, as opposed to the extremely competitive DE, it is a single deterministic model, hence, extremely efficient.
We consider both deterministic and Bayesian approaches for comparison. Following (Liu et al., 2020a), we also create two additional strong and simple baselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN) (Miyato et al., 2018a) and Stable Rank Normalization (SRN) (Sanyal et al., 2020). Note, we are the first to consider SRN for these experiments as it induces more compact clusters in the feature space than SN (we provide a simple mathematical proof of this in the Appendix A). Therefore, we compare our approach with the following baselines: • DNN: Standard deterministic neural network trained using cross-entropy loss. • DNN-SN:DNN with SN (Miyato et al., 2018a). • DNN-SRN: DNN with SRN (Sanyal et al., 2020). • SNGP: Spectrally Normalized Gaussian Process (Liu et al., 2020a). • DUQ: Deterministic Uncertainty Quantification (van Amersfoort et al., 2020). • Mixup: Standard Mixup training (Zhang et al., 2018). • KFAC-LLLA: KFAC-Laplace Last Layer Approximation (Kristiadi et al., 2020). A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation (Ritter et al., 2018). For the sake of completeness, we provide a simple outline of this approach in Appendix B. • DE: Deep Ensembles (Lakshminarayanan et al., 2017) with 5 members. Note, it is almost 5x slower than all other approaches mentioned above.****Through extensive experiments using WideResNet28-10 and ResNet50 architerctures on CIFAR10 and CIFAR100 datasets, we demonstrate that our method outperforms all existing single model baselines in providing clean data accuracy.
Answer: There were two main reasons we did not compare with Jishnu et al., 2020 (though we used their AdaECE)- We already had too many baselines to compare with and had dedicated a lot of efforts on several SOTA baselines that most recent papers compare with. - Also, we had observed that our approach already outperformed focal loss for ResNet50 CIFAR10 experiments in terms of both accuracy and ECE.