Robustness between the worst and average case

Several recent works in machine learning have focused on evaluating the test-time robustness of a classifier : how well the classifier performs not just on the target domain it was trained upon , but upon perturbed examples . In these settings , the focus has largely been on two extremes of robustness : the robustness to perturbations drawn at random from within some distribution ( i.e. , robustness to random perturbations ) , and the robustness to the worst case perturbation in some set ( i.e. , adversarial robustness ) . In this paper , we argue that a sliding scale between these two extremes provides a valuable additional metric by which to gauge robustness . Specifically , we illustrate that each of these two extremes is naturally characterized by a ( functional ) q-norm over perturbation space , with q = 1 corresponding to robustness to random perturbations and q = 1 corresponding to adversarial perturbations . We then present the main technical contribution of our paper : a method for efficiently estimating the value of these norms by interpreting them as the partition function of a particular distribution , then using path sampling with MCMC methods to estimate this partition function ( either traditional Metropolis-Hastings for non-differentiable perturbations , or Hamiltonian Monte Carlo for differentiable perturbations ) . We show that our approach provides substantially better estimates than simple random sampling of the actual “ intermediate-q ” robustness of standard , data-augmented , and adversarially-trained classifiers , illustrating a clear tradeoff between classifiers that optimize different metrics . Code for reproducing experiments can be found at https : //github.com/locuslab/intermediate_robustness . 1 Introduction . There has been an increasing focus in recent years in evaluating the robustness of machine learning classifiers , broadly interpreted as evaluating their performance not just on a test set , but also evaluating the performance relative to some additional ( possibly domain-specific ) uncertainty or bounds on the problems . Although there are many formal definitions of robustness , most work in this area has focused on two particular settings . In the “ classical ” sense of robustness , we can consider evaluating the classifier in terms of its worst-case loss under some perturbation set applied to the inputs , i.e. , we 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . could evaluate ( via finite sample approximation ) Ex , y⇠D max 2 ( x ) ` ( h ( x+ ) , y ) ( 1 ) where D denotes a distribution over x , y pairs , h denotes the hypothesis function , ` denotes a loss , and ( x ) denotes some ( input-dependent ) uncertainty region . This formulation , for instance , underlies adversarial examples and also motivates the classical adversarial training approaches . However , substantial work has also been done in evaluating the setting of robustness to random perturbations , i.e. , evaluating a classifier via the loss Ex , y⇠D ⇥ E ⇠P ( x ) [ ` ( h ( x+ ) , y ) ] ⇤ ( 2 ) where now P ( x ) denotes some ( again , input-dependent ) distribution over possible perturbations . This formulation underlies common data augmentation strategies in deep learning , as well as most formulations of “ natural ” robustness [ Hendrycks and Dietterich , 2018 ] ( even if not always written in this formal manner ) . Until now , these two types of robustness have typically been seen as largely separate notions . We believe there is inherent value in generalizing these two notions to place them in a unified framework . The main criticism of worst-case robustness is that it focuses “ too much ” on the worst case , while the criticism of average-case is that it is “ not robust enough ” . It seems very likely that what people actually want in terms of robustness is precisely something in the middle , between these two extremes . In this paper , we advocate for a more fine-grained spectrum of robustness definitions , which naturally interpolates between both these two extremes . In particular , we argue that robustness to random perturbations and worst-case robustness can be naturally interpreted as ( functional ) ` q norms1 of the loss function evaluated over the perturbation distribution ( which can be a uniform distribution in the case of traditional adversarial loss ) . In particular , the random setting corresponds to the choice of q = 1 and the adversarial setting corresponds to the choice of q = 1 . We believe that it is also valuable and informative to consider the performance of classifiers in a wide range in between these two extremes , i.e. , the performance of “ intermediate-q ” robustness . However , evaluating this intermediate-q robustness is non-trivial , owing to the fact that it requires computing a high dimensional integral over the perturbation space . Thus , our main technical contribution of this paper is the proposal of a simple approach to evaluating the relevant robustness norms , using a combination of path sampling and Markov chain Monte Carlo ( MCMC ) approaches ( specifically the Hamiltonian Monte Carlo method for cases where loss is a differentiable function of the perturbation space ) . Despite their seeming complexity , in this particular case the eventual estimators take the very simple form of a geometric mean computed over samples from an annealed distribution over the perturbation region . We evaluate our approach on networks trained via standard training , data augmentation , and adversarial training . In all the cases our proposed approach shows a clear trade-off between different levels of robustness that would missed by solely considering just the random or adversarial perturbation setting . Furthermore , we show that our HMC-based path sampling estimator for intermediate-q robustness is vastly superior to naive estimates produced e.g . by Monte Carlo sampling . We also briefly highlight the possibility of actually training networks using these estimators to create classifiers more robust to these intermediate notions of robustness . Code for reproducing experiments can be found at https : //github.com/locuslab/intermediate_robustness . 2 Background and related work . Robustness in deep learning Adversarial robustness is the “ classicial ” view of robustness that traces its roots back to robust optimization , a field in optimization theory that allows for the study of worst-case uncertainty [ Ben-Tal et al. , 2009 ] . Szegedy et al . [ 2013 ] noted the marked presence of adversarial examples in deep classifiers , in which the addition of imperceptible amounts of noise leads to gross misclassification of input images by deep networks . Goodfellow et al . [ 2014 ] then introduced the Fast Gradient Sign Method ( FGSM ) , which uses a single gradient step to generate adversarial 1We should emphasize that this use of ` q norms is entirely orthogonal to the use of ` p balls as perturbation regions , commonly done in adversarial robustness . The ` q norms here can be applied to any perturbation region , as we will highlight . examples and was used to perturb training examples as the original form of adversarial training . Subsequently , the Basic Iterative Method was introduced by Kurakin et al . [ 2016 ] , and rendered FGSM adversarial training ineffective through attacks using multiple , smaller FGSM steps ( although the robustness of FGSM adversarial training was later improved [ Wong et al. , 2020 , Andriushchenko and Flammarion , 2020 ] ) . The work of Madry et al . [ 2017 ] further improved upon this iterative attack , and incorporated the resulting Projected Gradient Descent ( PGD ) adversary into adversarial training . Today , PGD remains the most empirically effective adversary for adversarial training and attacks . While traditional adversarial training focuses on perturbations within some ` p-norm ball , it has also been expanded to more realistic threat models , from spatial transformations [ Engstrom et al. , 2019 , Xiao et al. , 2018 ] to weather corruptions and lighting changes [ Wong and Kolter , 2020 ] . However , when considering robustness to more “ realistic ” ( yet synthetic ) perturbations , the focus is often shifted from robustness to the worst-case perturbation within some set to robustness to random perturbations [ Geirhos et al. , 2018 , Hendrycks and Dietterich , 2018 , Hendrycks et al. , 2019b , a , Yin et al. , 2019 ] . It was been further studied whether such synthetic corruptions can universally improve robustness to real-world distribution shifts , such as geographic location , or camera hardware [ Hendrycks et al. , 2020 ] . Other naturally occurring distribution shifts of interest include consistency shifts [ Shankar et al. , 2019 ] and dataset shifts [ Storkey , 2009 , Taori et al. , 2020 ] . For example , the impact of dataset shift has been emphasized on the ImageNet dataset , where the performance of models trained on ImageNet was shown to be drastically worse on a reproduction of the validation set [ Recht et al. , 2019 ] . While some work such as Meunier et al . [ 2021 ] could be interpreted as an interpolation between random and adversarial noise , generally the field faces a divide between studies on adversarial robustness and robustness to random data perturbations . MCMC methods for partition function estimation Because our definition of intermediate-q robustness involves computation of a high dimensional integral over a perturbation distribution , our work can also be related to methods for estimating an intractable partition function of a probability distribution . The problem of computing normalizing constants ties to the problem of computing differences in free energy in physics . Several Monte Carlo sampling-based approaches exist for approximately estimating normalizing constants . Importance sampling is one approach at estimating ratios of normalizing constants that relies on samples from one of the distributions . Bridge sampling [ Bennett , 1976 , Meng and Wong , 1996 ] , on the other hand , uses a single “ bridge ” distribution to interpolate between the two distributions and can reduce the Monte Carlo errors associated with importance sampling . Annealed importance sampling [ Jarzynski , 1997 , Neal , 2001 ] bridges the gap between the two distributions by chaining together intermediate distributions , and linked importance sampling [ Neal , 2005 ] then combined bridge sampling with annealed importance sampling to bridge these intermediate distributions . Path sampling [ Gelman and Meng , 1998 ] , which originated under the name of thermodynamic integration in physics [ Ogata , 1989 ] , aims at reducing Monte Carlo errors by introducing flexible paths . Our proposed approach also leverages Hamiltonian Monte Carlo methods to improve the sample efficiency of our eventual estimator [ Duane et al. , 1987 , Betancourt , 2017 , Neal et al. , 2011 ] . Hamiltonian Monte Carlo ( HMC ) is useful due to its avoidance of random walks , and is compatible with sampling in a constrained space through use of reflection [ Afshar et al. , 2015 ] . 3 A simple generalization of robustness . To begin , we first make the simple observation that there is a natural interpolation between the notions of adversarial robustness and robustness to random perturbations . Specifically , we note that both these notions can be expressed as function ` q norms over the perturbation space . Specifically , we define the following functional norm Definition 1 . For a continuous function f : Rn ! R and density µ : Rn ! R+ , R µ ( x ) dx = 1 , let kfkµ , q be the q-norm of the function under this density kfkµ , q = Ex⇠µ [ |f ( x ) |q ] 1/q = ✓Z |f ( x ) |qµ ( x ) dx ◆1/q . ( 3 ) Then as the proposition shows , random perturbation loss and adversarial perturbation loss simply correspond to two extremes of this functional norm over the perturbation , as formalized by the following proposition . Proposition 1 . Let be a random variable with density µ and consider the expectation Ex , y⇠D h k ` ( h ( x+ ) , y ) kµ , q i . ( 4 ) Then ( for a smooth loss ` ) this corresponds to the expected loss on random samples from µ when q = 1 , and to the expected adversarial loss over the domain of µ when q = 1 . This proposition follows immediately from the fact that losses are non-negative , and the fact that the ` 1 norm is given by the pointwise maximum of the function , assuming a smooth loss ` ( i.e . not zero-one loss ) . Note that “ traditional ” adversarial loss actually arises more specifically when q = 1 and µ is a uniform distribution over some norm ball ( unrelated to the norm q ) . When we have 1 < q < 1 , we enable a full spectrum of robustness measurements , which we refer to as intermediate-q robustness , that evaluates the performance of classifiers in a wide range in between these two extreme cases . Furthermore , we argue that there are naturally appealing properties of these intermediate-q robustness measures : whereas adversarial robustness may overestimate the risk of what effectively amount to “ measure zero ” regions of the perturbation space , random robustness may likewise fail to take into account smaller but non-negligible regions that do contain areas of high loss . As an example , when considering as a Gaussian distribution , adversarial loss ( q = 1 ) is not meaningful because the worst case perturbation can be arbitrarily far away from x . Meanwhile , random data augmentation with Gaussian noise ( q = 1 ) is often insufficient for evaluating robustness because random Gaussian samples are rarely good adversarial examples . Our intermediate-q robustness allows us to consider a middle ground where the model is robust under a certain degree of adversarial noise ( stronger than randomly sampled Gaussian noise ) while the evaluation is not hindered by very rare events ( e.g. , an extremely low probability Gaussian sample far away from x ) . 4 Estimating intermediate-q robustness via MCMC methods Of course , simply writing the loss in this manner is not particularly useful on its own . In most cases , the integral in Equation 3 can not be computed exactly , and so we must resort to numerical approximation methods . Specifically , in order to estimate the q-norm robustness for an arbitrary nonlinear model h , loss ` , and ( known ) density µ , we consider the problem of computing the integral Z : = ✓Z ` ( h ( x+ ) , y ) qµ ( ) d ◆ ( 1/q ) . ( 5 ) One could naively estimate this integral by using Monte Carlo sampling , and sample ( 1 ) , . . . , ( m ) randomly from density µ , and approximate the objective as the following , ẐMonte Carlo = ⇣ 1 m mX i=1 ` ( h ( x+ ( i ) ) , y ) q ⌘ ( 1/q ) . ( 6 ) However , because the integral in ( 5 ) will be dominated by values with large ` ( h ( x+ ) , y ) , simply using Monte Carlo sampling will be insufficient to approximate this integral well for larger values of q , as random sampling will place too much weight on regions of low loss . Instead , in this work we argue that it is beneficial to interpret the task at hand as one of evaluating the partition function of a particular unnormalized probability density . Specifically , we define an ( unnormalized ) density over the perturbation p̃ ( ) = ` ( h ( x+ ) , y ) qµ ( ) . ( 7 ) Then clearly , just from construction , we see that the task of evaluating the ( qth root of ) partition function ( the normalizing constant ) of this distribution is exactly the same as that of computing the integral of interest . The advantage of this perspective on the integral of interest , however , is that we can use a wealth of techniques developed for partition function estimation in order to better estimate this particular integral . Specifically , we argue to use the path sampling [ Gelman and Meng , 1998 ] approach , a Markov chain Monte Carlo based method , to approximate the desired partition function . In fact , we show that for the precise form of the integral in question , the eventual estimator produced by this method takes on a very simple form : it consists of a geometric mean over samples generated from a certain annealed distribution . This makes the estimator particularly simple to implement and even in some cases to train networks based upon , via standard automatic differentiation toolkits .

Summary: The paper presents a unified characterization of classifier robustness, which covers both the worst-case adversarial robustness and robustness to random perturbations. The authors mainly focus on the problem of estimating the “intermediate-p robustness” for general p. By leveraging the techniques for partition function estimation, they propose a MCMC method for estimating the intermediate-p robustness. Theoretical justifications of the consistency of the proposed estimator are provided, and empirical evaluations on MNIST and CIFAR-10 are given.