Question: Mixup tends to be the strongest baseline, which is quite surprising as it is not an algorithm specifically designed for robustness and uncertainty. I am not an expert in this area, whether the current SOTA approaches are compared?
Evidence: Training Datasets and Network Architectures We employ WideResNet28-10 (WRN) and ResNet50 (RN50) architectures that have been shown to produce state-of-the-art classification accuracies on real-world datasets. We train them on CIFAR-10 (C10) and CIFAR-100 (C100). For Domain-Shift experiments, we resort to the widely used CIFAR10-C and CIFAR100-C, corrupted versions of C10 and C100. For Out-of-Distribution detection experiments, following SNGP, we use C100 and SVHN as OOD for models trained on C10. Similarly, for models trained on C100, we use C10 and SVHN as OOD.
Methods considered for comparisons We consider both deterministic and Bayesian approaches for comparison. Following, we also create two additional strong and simple baselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN) and Stable Rank Normalization (SRN). Note, we are the first to consider SRN for these experiments as it induces more compact clusters in the feature space than SN (we provide a simple mathematical proof of this in the Appendix A). Therefore, we compare our approach with the following baselines:
• DNN: Standard deterministic neural network trained using cross-entropy loss.
• DNN-SN:DNN with SN.
• DNN-SRN: DNN with SRN.
• SNGP: Spectrally Normalized Gaussian Process.
• DUQ: Deterministic Uncertainty Quantification.
• Mixup: Standard Mixup training.
• KFAC-LLLA: KFAC-Laplace Last Layer Approximation. A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation. For the sake of completeness, we provide a simple outline of this approach in Appendix B. Code base For fair comparisons, we developed our own code base for all the approaches mentioned above (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain the strongest possible baselines. For SNGP, we used the available code and made sure that we follow exactly the same procedure as mentioned in their original paper. For DUQ, the original paper did not perform large scale experiments similar to ours. Unfortunately, we could not manage to make their code work on C100 as it exhibited unstable behaviour. For this reason, we borrowed numbers for DUQ from the SNGP paper. Please note that the authors of SNGP performed non-trivial modifications to the original DUQ methodology to make it work on C100. Further details provided in Appendix C.
We use SGD with Nesterov momentum 0.9 and a weight decay of 5 × 10 −4 . For WRN, we apply a dropout p = 0.1 at train time. We perform extensive cross-validation of all the hyperparameters for all the baselines. Details provided in Appendix C.
Evaluation Metrics For calibration, we employ: (1) the widely used Expected Calibration Error (ECE), and (2) the recently proposed Adaptive ECE (AdaECE). For all the methods, the ECE and AdaECE are computed after performing temperature scaling with a cross-validated temperature parameter. Metrics and uncertainty measures used for out-of-distribution detection are discussed in detail in Section 5.1.3.
Training Datasets and Network Architectures We employ WideResNet28-10 (WRN) and ResNet50 (RN50) architectures that have been shown to produce state-of-the-art classification accuracies on real-world datasets. We train them on CIFAR-10 (C10) and CIFAR-100 (C100). For Domain-Shift experiments, we resort to the widely used CIFAR10-C and CIFAR100-C, corrupted versions of C10 and C100. For Out-of-Distribution detection experiments, following SNGP, we use C100 and SVHN as OOD for models trained on C10. Similarly, for models trained on C100, we use C10 and SVHN as OOD.
Methods considered for comparisons We consider both deterministic and Bayesian approaches for comparison. Following, we also create two additional strong and simple baselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN) and Stable Rank Normalization (SRN). Note, we are the first to consider SRN for these experiments as it induces more compact clusters in the feature space than SN (we provide a simple mathematical proof of this in the Appendix A). Therefore, we compare our approach with the following baselines:
• DNN: Standard deterministic neural network trained using cross-entropy loss.
• DNN-SN:DNN with SN.
• DNN-SRN: DNN with SRN.
• SNGP: Spectrally Normalized Gaussian Process.
• DUQ: Deterministic Uncertainty Quantification.
• Mixup: Standard Mixup training.
• KFAC-LLLA: KFAC-Laplace Last Layer Approximation. A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation. For the sake of completeness, we provide a simple outline of this approach in Appendix B. Code base For fair comparisons, we developed our own code base for all the approaches mentioned above (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain the strongest possible baselines. For SNGP, we used the available code and made sure that we follow exactly the same procedure as mentioned in their original paper. For DUQ, the original paper did not perform large scale experiments similar to ours. Unfortunately, we could not manage to make their code work on C100 as it exhibited unstable behaviour. For this reason, we borrowed numbers for DUQ from the SNGP paper. Please note that the authors of SNGP performed non-trivial modifications to the original DUQ methodology to make it work on C100. Further details provided in Appendix C.
We use SGD with Nesterov momentum 0.9 and a weight decay of 5 × 10 −4 . For WRN, we apply a dropout p = 0.1 at train time. We perform extensive cross-validation of all the hyperparameters for all the baselines. Details provided in Appendix C.
Evaluation Metrics For calibration, we employ: (1) the widely used Expected Calibration Error (ECE), and (2) the recently proposed Adaptive ECE (AdaECE). For all the methods, the ECE and AdaECE are computed after performing temperature scaling with a cross-validated temperature parameter. Metrics and uncertainty measures used for out-of-distribution detection are discussed in detail in Section 5.1.3.
Therefore, we compare our approach with the following baselines: • DNN: Standard deterministic neural network trained using cross-entropy loss. • DNN-SN:DNN with SN (Miyato et al., 2018a). • DNN-SRN: DNN with SRN (Sanyal et al., 2020). • SNGP: Spectrally Normalized Gaussian Process (Liu et al., 2020a). • DUQ: Deterministic Uncertainty Quantification (van Amersfoort et al., 2020). • Mixup: Standard Mixup training (Zhang et al., 2018). • KFAC-LLLA: KFAC-Laplace Last Layer Approximation (Kristiadi et al., 2020). A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation (Ritter et al., 2018). For the sake of completeness, we provide a simple outline of this approach in Appendix B.• DE: Deep Ensembles (Lakshminarayanan et al., 2017) with 5 members. Note, it is almost
5x slower than all other approaches mentioned above.****Code base For fair comparisons, we developed our own code base for all the approaches mentioned above (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain the strongest possible baselines. For SNGP, we used the available code and made sure that we follow exactly the same procedure as mentioned in their original paper. For DUQ, the original paper did not perform large scale experiments similar to ours. Unfortunately, we could not manage to make their code work on C100 as it exhibited unstable behaviour. For this reason, we borrowed numbers for DUQ from the SNGP paper. Please note that the authors of SNGP performed non-trivial modifications to the original DUQ methodology to make it work on C100. Further details provided in Appendix C.****Code For the SNGP method we use the official code-base with the suggested hyperparameters and training procedures. The code diverges slightly from the procedure described in their paper, hence the slight differences in the performance. The only modification we performed to the official code-base was to make the inference procedure consistent with the one described in the paper: indeed, in their code they implement a mean-field approximation to estimate the predictive distribution (Lu et al., 2020), while in their paper they use Monte Carlo Integration with a number of samples equal to the number of members in the ensembles they use as a baseline, which provides better calibration. The rationale is that we could not find an obvious way to tune the mean-field approximation hyperparameters to improve at the same time both the calibration and OOD detection performance (indeed, the mean-field approximation imposes a trade-off between calibration and OOD detection performance). Additionally, since the standard KFAC-LLLA uses the same Monte Carlo Integration procedure,we opted for the latter for a fair comparison. For the KFAC-LLLA we leverage the official repository7 (Hobbhahn et al., 2021) and the Backpack library (Dangel et al., 2020) for the computation of the Kronecker-Factored Hessian. For the SNGP ResNet50 experiments, we tried running the official implementation. The official implementation for ResNet50 is specifically fine-tuned for ImageNet, and has not been used for experiments on CIFAR. We could not make SNGP converge to SOTA accuracy values both on CIFAR-10 and CIFAR-100. All the other methods were implemented by us in PyTorch and the training, cross-validation and evaluation code will be made publicly available upon acceptance of the paper.
Answer: Please note that we do compare our approach with several most recent SOTA approaches (mentioned below) and perform very **extensive and thorough experiments**.Though **recent papers on OOD have largely ignored mixup** as one of their baselines, mixup has been shown to **improve robustness** [1,2,3] and improved **calibration** as a result of the counterbalance between the usual overconfidence of neural networks and the ability of mixup to reduce the confidence [3]. We use mixup as an additional **strong baseline**.To summarize, while comparing our approach with the recent OOD papers published at top venues, we also **create new strong baselines** for OOD such as: - mixup [4];- DNN-SN [5] and DNN-SRN [6], following the observations in SNGP that showed normalisation techniques to improve uncertainty estimation properties The recent OOD works we compare our approach with are: - **SNGP** [7], often considered one of the most effective single-model approaches.  - **DUQ** [8], as much as possible as we found it hard to make DUQ work on large experiments mostly because of training instabilities and centroid collapses. - **Deep Ensemble** [9] (often used as the gold standard); - **KFAC-LLLA** [10] (an incredibly effective Bayesian method).We mention most of these observations in **Section 5 and Appendix C, Code section**. We **omitted** methods simple to implement (like MC-Dropout) that are known to have **suboptimal performance** or that **assume knowledge of the OOD data** at training time. We would gently ask the reviewer to pick any of the above OOD papers (and these other papers [8, 11,12,13,14,15, 16] ) to realise the choice and amounts of baselines is either comparable or **superior** to most of the papers cited. Differently from most of these papers, we also compare using multiple uncertainty estimation tasks using several datasets and architectures. We trained each of the baselines from scratch (where possible) and cross-validated all the hyperparameters to allow them to produce the **best performance possible**.