Improving Calibration through the Relationship with Adversarial Robustness

1 Introduction . The robustness of machine learning algorithms is becoming increasingly important as ML systems are being used in higher-stakes applications . In one line of research , neural networks are shown to lack adversarial robustness – small perturbations to the input can successfully fool classifiers into making incorrect predictions ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ; Carlini & Wagner , 2017b ; Madry et al. , 2017 ; Qin et al. , 2020b ) . In largely separate lines of work , researchers have studied uncertainty in model ’ s predictions . For example , models are often miscalibrated where the predicted confidence is not indicative of the true likelihood of the model being correct ( Guo et al. , 2017 ; Thulasidasan et al. , 2019 ; Lakshminarayanan et al. , 2017 ; Wen et al. , 2020 ; Kull et al. , 2019 ) . The calibration issue is exacerbated when models are asked to make predictions on data different from the training distribution ( Snoek et al. , 2019 ) , which becomes an issue in practical settings where it is important that we can trust model predictions under distributional shift . Despite robustness , in all its forms , being a popular area of research , the relationship between these perspectives has not been extensively explored previously . In this paper , we study the correlation between adversarial robustness and calibration . We discover that input data points that are sensitive to small adversarial perturbations ( are easily attacked ) are more likely to have poorly calibrated predictions . This holds true on a number of network architectures for classification and on all the datasets that we consider : CIFAR-10 ( Krizhevsky , 2009 ) , CIFAR-100 ( Krizhevsky , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) . This suggests that the miscalibrated predictions on those adversarially unrobust data points greatly degrades the performance of model calibration . Based on this insight , we hypothesize and study if calibration can be improved by giving different supervision to the model depending on adversarial robustness of each training data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To this end , we propose Adversarial Robustness based Adaptive Label Smoothing ( AR-AdaLS ) to integrate the correlations between adversarial robustness and calibration into training . Specifically , AR-AdaLS adaptively smooths the training labels conditioned on how vulnerable an input is to adversarial attacks . Our method improves label smoothing ( Szegedy et al. , 2014 ) by explicitly teaching the model to differentiate the training data according to their adversarial robustness and then adaptively smooth their labels . By giving different supervision to the training data , our method leads to better calibration over the model without an increase of latency during inference . In addition , since adversarially unrobust data points can be considered as outliers of the underlying data distribution ( Carlini et al. , 2019 ) , our method can even greatly improve model calibration on held-out shifted data . Further , we propose “ AR-AdaLS of Ensemble ” to combine our AR-AdaLS and deep ensembles ( Lakshminarayanan et al. , 2017 ; Snoek et al. , 2019 ) , to further improve the calibration performance under distributional shift . Last , we find an additional benefit of AR-AdaLS is improving model stability ( i.e. , decreasing variance over multiple independent runs ) , which is valuable in practical applications where changes in predictions across runs ( churn ) is problematic . In summary , our main contributions are as follows : • Relationship among Robustness Metrics : We find a significant correlation between adversarial robustness and calibration : inputs that are unrobust to adversarial attacks are more likely to have poorly calibrated predictions . • Algorithm : We hypothesize that training a model with different supervision based on adversarial robustness of each input will make the model better calibrated . To this end , we propose AR-AdaLS to automatically learn how much to soften the labels of training data based on their adversarial robustness . Further , we introduce “ AR-AdaLS of Ensemble ” to show how to apply AR-AdaLS to an ensemble model . • Experimental Analysis : On CIFAR-10 , CIFAR-100 and ImageNet , we find that AR-AdaLS is more effective than previous label smoothing methods in improving calibration , particularly for shifted data . Further , we find that while ensembling can be beneficial , applying AR-AdaLS to adaptively calibrate ensembles offers further improvements over calibration . 2 Related Work . Uncertainty estimates How to better estimate a model ’ s predictive uncertainty is an important research topic , since many models with a focus on accuracy may fall short in predictive uncertainty . A popular way to improve a model ’ s predictive uncertainty is to make the model well-calibrated , e.g. , post-hoc calibration by temperature scaling ( Guo et al. , 2017 ) , and multi-class Dirichlet calibration ( Kull et al. , 2019 ) . In addition , Bayesian neural networks , through learning a posterior distribution over network parameters , can also be used to quantify a model ’ s predictive uncertainty , e.g. , Graves ( 2011 ) ; Blundell et al . ( 2015 ) ; Welling & Teh ( 2011 ) . Dropout-based variational inference ( Gal & Ghahramani , 2016 ; Kingma et al. , 2015 ) can help DNN models make less over-confident predictions and be better calibrated . Recently , mixup training ( Zhang et al. , 2018 ) has been shown to improve both models ’ generalization and calibration ( Thulasidasan et al. , 2019 ) , by preventing the model from being over-confident in its predictions . Despite the success of improving uncertainty estimates over in-distribution data , Snoek et al . ( 2019 ) argue that it does not usually translate to a better performance on data that shift from the training distribution . Among all the methods evaluated by Snoek et al . ( 2019 ) under distributional shift , ensemble of deep neural networks ( Lakshminarayanan et al. , 2017 ) , is shown to be most robust to dataset shift , producing the best uncertainty estimates . Adversarial robustness On the other hand , machine learning models are known to be brittle ( Xin et al. , 2017 ) and vulnerable to adversarial examples ( Athalye et al. , 2018 ; Carlini & Wagner , 2017a , b ; He et al. , 2018 ; Qin et al. , 2020a ) . Many defenses have been proposed to improve model ’ s adversarial robustness ( Song et al. , 2017 ; Yang et al. , 2019 ; Goodfellow et al. , 2018 ) , however are further attacked by more advanced defense-aware attacks ( Carlini & Wagner , 2017b ; Athalye et al. , 2018 ) . Recently , Carlini et al . ( 2019 ) ; Stock & Cissé ( 2018 ) define adversarial robustness as the minimum distance in the input domain required to change the model ’ s output prediction by constructing an adversarial attack . The most recent work that is close to ours , Carlini et al . ( 2019 ) , makes the interesting observation that easily attackable data are often outliers in the underlying data distribution and then use adversarial robustness to determine an improved ordering for curriculum learning . Our work , instead , explores the relationship between adversarial robustness and calibration . In addition , we use adversarial robustness as an indicator to adaptively smooth the training labels to improve model calibration . Label smoothing Label smoothing is originally proposed in Szegedy et al . ( 2016 ) and is shown to be effective in improving the quality of uncertainty estimates in Müller et al . ( 2019 ) ; Thulasidasan et al . ( 2019 ) . Instead of minimizing the cross-entropy loss between the predicted probability p̂ and the one-hot label p , label smoothing minimizes the cross-entropy between the predicted probability and a softened label p̃ = p ( 1 − ) + Z , where Z is the number of classes in the dataset and is a hyperparameter which controls the degree of the smoothing effect . Our work makes label smoothing adaptive and incorporates the correlation with adversarial robustness to further improve calibration . 3 Correlations between Adversarial Robustness and Calibration . To explore the relationship between adversarial robustness and calibration , we first introduce the metrics to evaluate each of them ( arrows indicate which direction is better ) . Adversarial robustness ↑ Adversarial robustness measures the minimum distance in the input domain required to change the model ’ s output prediction by constructing an adversarial attack ( Carlini et al. , 2019 ; Stock & Cissé , 2018 ) . Specifically , given an input x and a classifier f ( · ) that predicts the class for the input , the adversarial robustness is defined as the minimum adversarial perturbation δ that enables f ( x + δ ) 6= f ( x ) . Following the work ( Carlini et al. , 2019 ) , we construct the ` 2 based CW attack ( Carlini & Wagner , 2017b ) and then use the ` 2 norm of the adversarial perturbation ‖δ‖2 to measure the distance to the decision boundary . Therefore , a more adversarially robust input requires a larger adversarial perturbation to change the model ’ s prediction . Expected calibration error ↓ Model calibration measures the alignment between the predicted probability and the accuracy . Well calibrated predictions convey the information about how much we should trust a model ’ s prediction . We follow the widely used expected calibration error ( ECE ) to measure the calibration performance of a network ( Guo et al. , 2017 ; Snoek et al. , 2019 ) . To compute the ECE , we need to first divide all the data into K buckets sorted by their predicted probability ( confidence ) of the predicted class . Let Bk represent the set of data in the k-th confidence bucket . Then the accuracy and the confidence of Bk are defined as acc ( Bk ) = 1|Bk| ∑ i∈Bk 1 ( ŷi = yi ) and conf ( Bk ) = 1|Bk| ∑ i∈Bk p̂ ŷi i , where ŷ and y represent the predicted class and the true class respectively , and p̂ŷ is the predicted probability of ŷ . The ECE is then defined as ECE =∑K k=1 |Bk| N |acc ( Bk ) − conf ( Bk ) | , where N is the number of data points . 3.1 Correlations . Based on the evaluation metrics , we can see that adversarial robustness and calibration are measuring quite different properties : the adversarial robustness measures the property of the data by computing the adversarial perturbation δ from the input domain , while the calibration metric measures the properties of the model ’ s predicted probability in the output space . Although adversarial robustness and calibration are conceptually different , they are both connected to the decision boundary . Specifically , adversarial robustness can be used to measure the distance to the decision boundary : if a data point is adversarially unrobust , i.e. , easy to find a small input perturbation to fool the classifier into wrong classifications , then this data point is close to the decision boundary . Meanwhile , models should have relatively less confident predictions on data points close to the decision boundary . However , as pointed out by ( Guo et al. , 2017 ; Snoek et al. , 2019 ) , existing deep neural networks are frequently over-confident , i.e. , having predictions with high confidence even whey they should be uncertain . Taking these two together , we hypothesize if examples that can be easily attacked by adversarial examples are also poorly calibrated . To test this , we perform experiments on the clean test set across three datasets : CIFAR-10 ( Krizhevsky , 2009 ) , CIFAR-100 ( Krizhevsky , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) with different networks , whose architecture and accuracy are shown in Table 1 . We refer to these models as “ Vanilla ” for each dataset in the following discussion . The details for training each vanilla network are included in Appendix A . To explore the relationship between adversarial robustness and calibration , we start with the relationship between adversarial robustness and confidence together with accuracy . Specifically , we rank the input data according to their adversarial robustness and then divide the dataset into R equally-sized subsets ( R = 10 used in this paper ) . For each adversarial robustness subset , we compute the accuracy and the average confidence score of the predicted class . As shown in the first row in Figure 1 , we can clearly see that both accuracy and confidence increase with the adversarial robustness of the input data , and confidence is consistently higher than accuracy in each adversarial robustness subset across three datasets . This indicates that although vanilla classification models achieve the state-of-the-art accuracy , they tend to give over-confident predictions , especially for those adversarially unrobust data points . Taking one step further , we particularly compute the expected calibration error ( ECE ) in each adversarial robustness subset , shown in the bottom row of Figure 1 . In general , we find that data points falling into lower adversarial robustness levels are more likely to be over-confident and less well calibrated ( larger ECE ) . For those adversarially robust examples , there is a better alignment between the model ’ s predicted confidence and accuracy , and the ECE over those examples is close to 0 . This nicely validates our hypothesis : inputs that are adversarially unrobust are more likely to have poorly calibrated predictions . On larger-scale ImageNet , while we still see the general trend holds , the least adversarially robust examples are relatively well calibrated . We hypothesize this may be due to larger training data and less overfitting . Furthermore , we also find an interesting correlation between adversarial robustness and model stability , which is measured by the variance of the predicted probability across M independent runs ( e.g. , M = 5 ) . The variance is computed as σ2 = 1M−1 1 N ∑M m=1 ∑N i=1 ( p̂m , i − p̄i ) 2 , where p̂m , i is the m-th model ’ s predicted probability of the i-th data and p̄i = 1M ∑M m=1 p̂m , i is the average predicted probability over M runs . As shown in the bottom row of Figure 1 , we see that those adversarially unrobust examples tend to have a much higher variance across all three datasets . This indicates that inputs that are unrobust to adversarial attacks are more likely to have unstable predictions . Algorithm 1 Training procedure for AR-AdaLS Input : number of classes Z , number of training epochs T , number of adversarial robustness subset R , learning rate of adaptive label smoothing α . For each adversarial robustness training subset , we initialize the soft label as the one-hot label p̃r , t = pr , where the initial soft label for the correct class p̃ z=y r , t = 1. for t = 1 to T do Minimize cross-entropy loss between soft label and predicted probability 1R ∑R r L ( p̃r , t , p̂r , t ) for r = 1 to R do Update p̃z=yr , t+1 ← p̃ z=y r , t − α · { conf ( Svalr ) t − acc ( Svalr ) t } . Eqn . ( 3 ) Clip p̃z=yr , t+1 to be within ( 1 Z , 1 ] Update r , t+1 ← ( p̃z=yr , t+1 − 1 ) · Z1−Z . Eqn . ( 4 ) Update p̃r , t+1 ← pr ( 1− r , t+1 ) + r , t+1Z . Eqn . ( 1 ) end for end for Taking all together , these empirical results nicely build a connection between very different concepts . In particular , adversarial robustness is measured over the input domain while both calibration and stability are measured over the output space . Given the strong empirical connection , we now ask : can we improve model calibration and stability by targeting adversarially unrobust examples ?

Summary: This paper investigates the relationship between calibration and adversarial robustness, showing that calibration error is larger among less robust images. Based on this, a training procedure is proposed where the labels of the images are smoothed based on the robustness level, to produce a better calibrated model.   The proposed method is evaluated on CIFAR-10/CIFAR-100, and their corrupted counterparts, comparing with other calibration and label smoothing approaches, as well as ensemble methods.