Improving Calibration through the Relationship with Adversarial Robustness

1 Introduction . The robustness of machine learning algorithms is becoming increasingly important as ML systems are being used in higher-stakes applications . In one line of research , neural networks are shown to lack adversarial robustness – small perturbations to the input can successfully fool classifiers into making incorrect predictions ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ; Carlini & Wagner , 2017b ; Madry et al. , 2017 ; Qin et al. , 2020b ) . In largely separate lines of work , researchers have studied uncertainty in model ’ s predictions . For example , models are often miscalibrated where the predicted confidence is not indicative of the true likelihood of the model being correct ( Guo et al. , 2017 ; Thulasidasan et al. , 2019 ; Lakshminarayanan et al. , 2017 ; Wen et al. , 2020 ; Kull et al. , 2019 ) . The calibration issue is exacerbated when models are asked to make predictions on data different from the training distribution ( Snoek et al. , 2019 ) , which becomes an issue in practical settings where it is important that we can trust model predictions under distributional shift . Despite robustness , in all its forms , being a popular area of research , the relationship between these perspectives has not been extensively explored previously . In this paper , we study the correlation between adversarial robustness and calibration . We discover that input data points that are sensitive to small adversarial perturbations ( are easily attacked ) are more likely to have poorly calibrated predictions . This holds true on a number of network architectures for classification and on all the datasets that we consider : CIFAR-10 ( Krizhevsky , 2009 ) , CIFAR-100 ( Krizhevsky , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) . This suggests that the miscalibrated predictions on those adversarially unrobust data points greatly degrades the performance of model calibration . Based on this insight , we hypothesize and study if calibration can be improved by giving different supervision to the model depending on adversarial robustness of each training data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To this end , we propose Adversarial Robustness based Adaptive Label Smoothing ( AR-AdaLS ) to integrate the correlations between adversarial robustness and calibration into training . Specifically , AR-AdaLS adaptively smooths the training labels conditioned on how vulnerable an input is to adversarial attacks . Our method improves label smoothing ( Szegedy et al. , 2014 ) by explicitly teaching the model to differentiate the training data according to their adversarial robustness and then adaptively smooth their labels . By giving different supervision to the training data , our method leads to better calibration over the model without an increase of latency during inference . In addition , since adversarially unrobust data points can be considered as outliers of the underlying data distribution ( Carlini et al. , 2019 ) , our method can even greatly improve model calibration on held-out shifted data . Further , we propose “ AR-AdaLS of Ensemble ” to combine our AR-AdaLS and deep ensembles ( Lakshminarayanan et al. , 2017 ; Snoek et al. , 2019 ) , to further improve the calibration performance under distributional shift . Last , we find an additional benefit of AR-AdaLS is improving model stability ( i.e. , decreasing variance over multiple independent runs ) , which is valuable in practical applications where changes in predictions across runs ( churn ) is problematic . In summary , our main contributions are as follows : • Relationship among Robustness Metrics : We find a significant correlation between adversarial robustness and calibration : inputs that are unrobust to adversarial attacks are more likely to have poorly calibrated predictions . • Algorithm : We hypothesize that training a model with different supervision based on adversarial robustness of each input will make the model better calibrated . To this end , we propose AR-AdaLS to automatically learn how much to soften the labels of training data based on their adversarial robustness . Further , we introduce “ AR-AdaLS of Ensemble ” to show how to apply AR-AdaLS to an ensemble model . • Experimental Analysis : On CIFAR-10 , CIFAR-100 and ImageNet , we find that AR-AdaLS is more effective than previous label smoothing methods in improving calibration , particularly for shifted data . Further , we find that while ensembling can be beneficial , applying AR-AdaLS to adaptively calibrate ensembles offers further improvements over calibration . 2 Related Work . Uncertainty estimates How to better estimate a model ’ s predictive uncertainty is an important research topic , since many models with a focus on accuracy may fall short in predictive uncertainty . A popular way to improve a model ’ s predictive uncertainty is to make the model well-calibrated , e.g. , post-hoc calibration by temperature scaling ( Guo et al. , 2017 ) , and multi-class Dirichlet calibration ( Kull et al. , 2019 ) . In addition , Bayesian neural networks , through learning a posterior distribution over network parameters , can also be used to quantify a model ’ s predictive uncertainty , e.g. , Graves ( 2011 ) ; Blundell et al . ( 2015 ) ; Welling & Teh ( 2011 ) . Dropout-based variational inference ( Gal & Ghahramani , 2016 ; Kingma et al. , 2015 ) can help DNN models make less over-confident predictions and be better calibrated . Recently , mixup training ( Zhang et al. , 2018 ) has been shown to improve both models ’ generalization and calibration ( Thulasidasan et al. , 2019 ) , by preventing the model from being over-confident in its predictions . Despite the success of improving uncertainty estimates over in-distribution data , Snoek et al . ( 2019 ) argue that it does not usually translate to a better performance on data that shift from the training distribution . Among all the methods evaluated by Snoek et al . ( 2019 ) under distributional shift , ensemble of deep neural networks ( Lakshminarayanan et al. , 2017 ) , is shown to be most robust to dataset shift , producing the best uncertainty estimates . Adversarial robustness On the other hand , machine learning models are known to be brittle ( Xin et al. , 2017 ) and vulnerable to adversarial examples ( Athalye et al. , 2018 ; Carlini & Wagner , 2017a , b ; He et al. , 2018 ; Qin et al. , 2020a ) . Many defenses have been proposed to improve model ’ s adversarial robustness ( Song et al. , 2017 ; Yang et al. , 2019 ; Goodfellow et al. , 2018 ) , however are further attacked by more advanced defense-aware attacks ( Carlini & Wagner , 2017b ; Athalye et al. , 2018 ) . Recently , Carlini et al . ( 2019 ) ; Stock & Cissé ( 2018 ) define adversarial robustness as the minimum distance in the input domain required to change the model ’ s output prediction by constructing an adversarial attack . The most recent work that is close to ours , Carlini et al . ( 2019 ) , makes the interesting observation that easily attackable data are often outliers in the underlying data distribution and then use adversarial robustness to determine an improved ordering for curriculum learning . Our work , instead , explores the relationship between adversarial robustness and calibration . In addition , we use adversarial robustness as an indicator to adaptively smooth the training labels to improve model calibration . Label smoothing Label smoothing is originally proposed in Szegedy et al . ( 2016 ) and is shown to be effective in improving the quality of uncertainty estimates in Müller et al . ( 2019 ) ; Thulasidasan et al . ( 2019 ) . Instead of minimizing the cross-entropy loss between the predicted probability p̂ and the one-hot label p , label smoothing minimizes the cross-entropy between the predicted probability and a softened label p̃ = p ( 1 − ) + Z , where Z is the number of classes in the dataset and is a hyperparameter which controls the degree of the smoothing effect . Our work makes label smoothing adaptive and incorporates the correlation with adversarial robustness to further improve calibration . 3 Correlations between Adversarial Robustness and Calibration . To explore the relationship between adversarial robustness and calibration , we first introduce the metrics to evaluate each of them ( arrows indicate which direction is better ) . Adversarial robustness ↑ Adversarial robustness measures the minimum distance in the input domain required to change the model ’ s output prediction by constructing an adversarial attack ( Carlini et al. , 2019 ; Stock & Cissé , 2018 ) . Specifically , given an input x and a classifier f ( · ) that predicts the class for the input , the adversarial robustness is defined as the minimum adversarial perturbation δ that enables f ( x + δ ) 6= f ( x ) . Following the work ( Carlini et al. , 2019 ) , we construct the ` 2 based CW attack ( Carlini & Wagner , 2017b ) and then use the ` 2 norm of the adversarial perturbation ‖δ‖2 to measure the distance to the decision boundary . Therefore , a more adversarially robust input requires a larger adversarial perturbation to change the model ’ s prediction . Expected calibration error ↓ Model calibration measures the alignment between the predicted probability and the accuracy . Well calibrated predictions convey the information about how much we should trust a model ’ s prediction . We follow the widely used expected calibration error ( ECE ) to measure the calibration performance of a network ( Guo et al. , 2017 ; Snoek et al. , 2019 ) . To compute the ECE , we need to first divide all the data into K buckets sorted by their predicted probability ( confidence ) of the predicted class . Let Bk represent the set of data in the k-th confidence bucket . Then the accuracy and the confidence of Bk are defined as acc ( Bk ) = 1|Bk| ∑ i∈Bk 1 ( ŷi = yi ) and conf ( Bk ) = 1|Bk| ∑ i∈Bk p̂ ŷi i , where ŷ and y represent the predicted class and the true class respectively , and p̂ŷ is the predicted probability of ŷ . The ECE is then defined as ECE =∑K k=1 |Bk| N |acc ( Bk ) − conf ( Bk ) | , where N is the number of data points . 3.1 Correlations . Based on the evaluation metrics , we can see that adversarial robustness and calibration are measuring quite different properties : the adversarial robustness measures the property of the data by computing the adversarial perturbation δ from the input domain , while the calibration metric measures the properties of the model ’ s predicted probability in the output space . Although adversarial robustness and calibration are conceptually different , they are both connected to the decision boundary . Specifically , adversarial robustness can be used to measure the distance to the decision boundary : if a data point is adversarially unrobust , i.e. , easy to find a small input perturbation to fool the classifier into wrong classifications , then this data point is close to the decision boundary . Meanwhile , models should have relatively less confident predictions on data points close to the decision boundary . However , as pointed out by ( Guo et al. , 2017 ; Snoek et al. , 2019 ) , existing deep neural networks are frequently over-confident , i.e. , having predictions with high confidence even whey they should be uncertain . Taking these two together , we hypothesize if examples that can be easily attacked by adversarial examples are also poorly calibrated . To test this , we perform experiments on the clean test set across three datasets : CIFAR-10 ( Krizhevsky , 2009 ) , CIFAR-100 ( Krizhevsky , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) with different networks , whose architecture and accuracy are shown in Table 1 . We refer to these models as “ Vanilla ” for each dataset in the following discussion . The details for training each vanilla network are included in Appendix A . To explore the relationship between adversarial robustness and calibration , we start with the relationship between adversarial robustness and confidence together with accuracy . Specifically , we rank the input data according to their adversarial robustness and then divide the dataset into R equally-sized subsets ( R = 10 used in this paper ) . For each adversarial robustness subset , we compute the accuracy and the average confidence score of the predicted class . As shown in the first row in Figure 1 , we can clearly see that both accuracy and confidence increase with the adversarial robustness of the input data , and confidence is consistently higher than accuracy in each adversarial robustness subset across three datasets . This indicates that although vanilla classification models achieve the state-of-the-art accuracy , they tend to give over-confident predictions , especially for those adversarially unrobust data points . Taking one step further , we particularly compute the expected calibration error ( ECE ) in each adversarial robustness subset , shown in the bottom row of Figure 1 . In general , we find that data points falling into lower adversarial robustness levels are more likely to be over-confident and less well calibrated ( larger ECE ) . For those adversarially robust examples , there is a better alignment between the model ’ s predicted confidence and accuracy , and the ECE over those examples is close to 0 . This nicely validates our hypothesis : inputs that are adversarially unrobust are more likely to have poorly calibrated predictions . On larger-scale ImageNet , while we still see the general trend holds , the least adversarially robust examples are relatively well calibrated . We hypothesize this may be due to larger training data and less overfitting . Furthermore , we also find an interesting correlation between adversarial robustness and model stability , which is measured by the variance of the predicted probability across M independent runs ( e.g. , M = 5 ) . The variance is computed as σ2 = 1M−1 1 N ∑M m=1 ∑N i=1 ( p̂m , i − p̄i ) 2 , where p̂m , i is the m-th model ’ s predicted probability of the i-th data and p̄i = 1M ∑M m=1 p̂m , i is the average predicted probability over M runs . As shown in the bottom row of Figure 1 , we see that those adversarially unrobust examples tend to have a much higher variance across all three datasets . This indicates that inputs that are unrobust to adversarial attacks are more likely to have unstable predictions . Algorithm 1 Training procedure for AR-AdaLS Input : number of classes Z , number of training epochs T , number of adversarial robustness subset R , learning rate of adaptive label smoothing α . For each adversarial robustness training subset , we initialize the soft label as the one-hot label p̃r , t = pr , where the initial soft label for the correct class p̃ z=y r , t = 1. for t = 1 to T do Minimize cross-entropy loss between soft label and predicted probability 1R ∑R r L ( p̃r , t , p̂r , t ) for r = 1 to R do Update p̃z=yr , t+1 ← p̃ z=y r , t − α · { conf ( Svalr ) t − acc ( Svalr ) t } . Eqn . ( 3 ) Clip p̃z=yr , t+1 to be within ( 1 Z , 1 ] Update r , t+1 ← ( p̃z=yr , t+1 − 1 ) · Z1−Z . Eqn . ( 4 ) Update p̃r , t+1 ← pr ( 1− r , t+1 ) + r , t+1Z . Eqn . ( 1 ) end for end for Taking all together , these empirical results nicely build a connection between very different concepts . In particular , adversarial robustness is measured over the input domain while both calibration and stability are measured over the output space . Given the strong empirical connection , we now ask : can we improve model calibration and stability by targeting adversarially unrobust examples ?

Summary: This paper studies the connection between adversarial robustness (measured as the distance to the decision boundary using some adversarial attack) and model calibration (i.e how well the predicted probability indicates how much we can truth the model predictions).   1. The authors show that there is a significant correlation between adversarial robustness and model calibration. That is, inputs that have smaller distance to the decision boundary are more likely to have poorly calibrated predictions.  2. Based on this insight, the authors propose an algorithm called AR-AdaLS to learn how much to smooth the labels of the training data based on their adversarial robustness. They also discuss how this technique can be extended to an ensemble model.  3. The authors thoroughly compare their results against many previous calibration techniques and show that AR-AdaLS results in improved performance (for the single-model based methods). The results are superior also under distribution shift. They show results on CIFAR-10, CIFAR-100 and Imagenet datasets. For results under distribution shift, they show results on CIFAR-10-C, CIFAR-100-C and Imagenet-C datasets.