RoMA: a Method for Neural Network Robustness Measurement and Assessment 

Neural network models have become the leading solution for a large variety of tasks , such as classification , language processing , protein folding , and others . However , their reliability is heavily plagued by adversarial inputs : small input perturbations that cause the model to produce erroneous outputs . Adversarial inputs can occur naturally when the system ’ s environment behaves randomly , even in the absence of a malicious adversary , and are a severe cause for concern when attempting to deploy neural networks within critical systems . In this paper , we present a new statistical method , called Robustness Measurement and Assessment ( RoMA ) , which can measure the expected robustness of a neural network model . Specifically , RoMA determines the probability that a random input perturbation might cause misclassification . The method allows us to provide formal guarantees regarding the expected frequency of errors that a trained model will encounter after deployment . Our approach can be applied to large-scale , black-box neural networks , which is a significant advantage compared to recently proposed verification methods . We apply our approach in two ways : comparing the robustness of different models , and measuring how a model ’ s robustness is affected by the magnitude of input perturbation . One interesting insight obtained through this work is that , in a classification network , different output labels can exhibit very different robustness levels . We term this phenomenon categorial robustness . Our ability to perform risk and robustness assessments on a categorial basis opens the door to risk mitigation , which may prove to be a significant step towards neural network certification in safety-critical applications . 1 INTRODUCTION . In the passing decade , deep neural networks ( DNNs ) have emerged as one of the most exciting and innovative developments in computer science , allowing computers to outperform humans in various classification tasks . However , a major disadvantage of the DNN approach is the existence of adversarial inputs Goodfellow et al . ( 2014 ) : inputs which are very close ( according to some metrics ) to a correctly-classified input , but which are misclassified themselves . It has been observed that most state-of-the-art DNNs are highly vulnerable to adversarial inputs Carlini & Wagner ( 2017 ) ; and it has been suggested that adversarial inputs are an inescapable part of the neural network architecture , and are thus not an issue that can be solved entirely Ilyas et al . ( 2019 ) . In this perspective , it is crucial to find a method for containing this phenomenon and mitigating the risk that it causes ; especially in order to allow DNNs to be deployed in safety-critical settings ( e.g. , automotive , aerospace , trains , and medical devices ) , where regulatory requirements and public opinion set a high bar for reliability . As the impact of the AI revolution is becoming evident , regulatory agencies are starting to address the challenge of safely integrating DNNs into safety-critical systems — by forming workgroups to create the needed guidelines . Notable examples in the European Union include SAE G-34 and EUROCAE WG-114 Pereira & Thomas ( 2020 ) ; Vidot et al . ( 2021 ) ; and the efforts made by the European Union Safety Agency ( EASA ) , which is responsible for civil aviation safety , and which has published a road map for certifying AI-based systems European Union Aviation Safety Agency ( 2020 ) . These efforts highlight the dire need for certification methods for DNN-based systems . The existence of adversarial inputs does not seem to be a deal-breaker as far as regulatory agencies are concerned ; indeed , these agencies are used to certifying systems with components that might fail due to an unexpected hazard . A common example is the certification of jet engines for civilian aircraft , with a known mean time between failures ( MTBF ) . Based on the MTBF value , there exist certification processes that perform functional hazard analysis ( FHA ) and risk mitigation FAA ( 2021 ) . To perform similar processes for DNN-based systems , one needs to assess the likelihood of failure ( i.e. , an adversarial input ) , but this essential ability is still missing . In this paper , we attempt to address this crucial gap by introducing a novel , straightforward , and scalable statistical method for measuring the probability that a DNN classifier will misclassify inputs . The method , which we term Robustness Measurement and Assessment ( RoMA ) , assumes that the misclassification of the neural network is not due to a malicious attack , but rather due to random perturbations of the input , which occur naturally as part of the system ’ s operation . Under this assumption , RoMA can be used to measure the model ’ s robustness to randomly-produced adversarial inputs . The proposed method is useful for several applications , such as comparing the robustness of multiple models and picking the best one , or for checking the impact of various configurable parameters ( e.g. , the number of training epochs , or the magnitude of the input perturbation ) on the resulting model ’ s robustness . RoMA is a method for estimating rare events in a large population — in our case , adversarial inputs within a space of inputs that are generally correctly classified . The method relies on the properties of normal distributions . If it is known that the rare events ( adversarial inputs ) are distributed normally , it is usually sufficient to sample a few thousands random points , use them to draw a Gaussian curve , and then use the normal distribution function to evaluate the probability of a rare event in the population . Unfortunately , adversarial inputs are often not distributed normally in the population ; and to overcome this difficulty , RoMA first applies a statistical transformation , after which the distribution often becomes normal and can be analyzed . At a high level , RoMA consists of the following steps : 1. for an arbitrary input point x0 , we randomly sample n perturbations of x0 ( usually , a few thousands ) , obtaining a set of perturbed inputs { x10 , . . . , xn0 } ; 2. we evaluate the DNN on each xi0 , obtaining the corresponding outputs { y10 , . . . , yn0 } ; 3. for each yi0 that is classified incorrectly ( that is , which indicates a labeling different than that of x0 ) , we collect the maximal entry ci of yi0 . This value is the confidence score assigned by the DNN to the incorrect label ; 4. if needed , we apply a statistical transformation called Box-Cox Box & Cox ( 1982 ) to normalize the distribution of ci values ; and 5. if the distribution is now normal , we use the properties of the normal distribution function to calculate the probability for an adversarial input around x0 . A key component in our method is the Box-Cox statistical power transformation , which is a wellestablished method , and which does not pose any restrictions on the DNN in question ( e.g. , Lipschitz continuity , certain kinds of activation functions , or a specific network topology ) . Further , the method does not require access to the network ’ s topology or weights , and is thus applicable to black-box DNNs . We implemented our method as a proof-of-concept tool , and evaluated it on standard DNN architectures : VGG16 Simonyan & Zisserman ( 2015 ) , Resnet He et al . ( 2016 ) , and Densenet Huang et al . ( 2017a ) , all trained on the CIFAR10 data set Krizhevsky et al . ( 2009 ) . We used RoMA to compare the robustness of these DNN models , and found , as expected , that a higher number of epochs leads to a higher robustness score . Additionally , we used RoMA to measure how the allowed magnitude of perturbation affects the robustness of a DNN model , and computed the rate at which robustness deteriorates as the perturbation level increases . Finally , using RoMA , we found that the categorial robustness score of a DNN ( i.e. , the robustness score of inputs labeled as a particular category ) varies significantly among the different categories . This finding could allow users and regulators to specify different acceptable robustness thresholds for each target category , instead of a single global threshold , which may be more difficult to obtain . To summarize , our main contributions are : • Introducing RoMA : a new method for measuring the robustness of a DNN model . The new method is scalable and can run on black-box DNNs . • Comparing the robustness of multiple state-of-the-art DNN models . • Using RoMA to measure the effect of perturbation level has on the robustness of the DNN model . • Introducing the notion of categorial robustness , which is a measure of robustness computed for each target label . Related work . The topic of adversarial robustness has been studied extensively . Some notable approaches for estimating a model ’ s robustness include : • Statistical approaches that evaluate the probability of encountering an adversarial input in the population . In a recent paper Huang et al . ( 2021 ) , Huang et al . use random sampling , which is similar in spirit to RoMA , but which assumes that the DNN ’ s adversarial inputs are distributed normally . As we demonstrate later , this is often not the case . In another paper , Webb at al . Webb et al . ( 2018 ) use a sampling method called multi-level splitting , which provides no formal guarantee of the DNN ’ s robustness . Mangal et al . Mangal et al . ( 2019 ) use importance sampling , which might be biased due to lack of sampling in areas of the population that are deemed unimportant . Moreover , this approach assumes that the network ’ s output is Lipschitz-continuous , which limits its applicability . In contrast , RoMA requires no Lipschitz-continuity assumptions , does not assume a-priori that the adversarial inputs are distributed normally , and provides rigorous robustness guarantees . • Formal-verification based approaches Katz et al . ( 2017 ) ; Wang et al . ( 2018 ) ; Jacoby et al . ( 2020 ) , which allow for computing a DNN ’ s exact adversarial robustness score . These approaches typically convert the problem into a constraint satisfiability problem , and then apply search and deduction procedures to solve it efficiently . However , verification-based approaches afford only limited scalability , and operate strictly on white-box DNNs . In contrast , RoMA is a scalable technique , and can operate on black-box DNNs . • Approaches for computing an estimate bound on the probability that a classifier ’ s margin function exceeds a given value Weng et al . ( 2019 ) ; Anderson & Sojoudi ( 2020 ) ; Dvijotham et al . ( 2018 ) . These analyses focus on the worst-case behavior , thus producing bounds that might be inadequate for regulatory certification . In contrast , RoMA focuses on the average case , which is more realistic in many application domains . Outline . We begin with some needed background on adversarial robustness in Section 2 . We then describe our proposed method for measuring adversarial robustness in Section 3 , followed by a description of our evaluation setup in Section 4 . In Section 5 we summarize and discuss our results . 2 BACKGROUND . Neural Network . A neural network N is a function N : Rn → Rm , which maps a real-valued input vector x ∈ Rn to a real-valued output vector y ∈ Rm . For classification networks , which is our subject matter here , the entries of y represent confidence scores for m possible classes ; and x is classified as label l if y ’ s l ’ th entry has the highest score ; i.e. , if argmax ( N ( x ) ) = l. We use c ( x ) to denote this highest score , i.e . c ( x ) = max ( N ( x ) ) . Local Adversarial Robustness . The local adversarial robustness of a DNN is a measure of how resilient that network is against adversarial perturbations to specific inputs . Intuitively , a network with high robustness behaves “ smoothly ” , i.e. , small perturbations to its input do not cause significant spikes in its output . More formally Bastani et al . ( 2016 ) ; Huang et al . ( 2017b ) : Definition 1 . A DNN N is -locally-robust at input point x0 iff ∀x . ||x− x0||∞ ≤ ⇒ argmax ( N ( x ) ) = argmax ( N ( x0 ) ) Intuitively , Definition 1 states that for input vector x , which is at a distance at most from a fixed input x0 , the network assigns to x the same label that it assigns to x0 ( for simplicity , we use here the L∞ norm , but other metrics could also be used ) . When a network is not -local-robust at point x0 , there exists a point x that is at a distance of at most from x0 , which is misclassified ; this x is called an adversarial input . In this context , local refers to the fact that x0 is fixed . Larger values of imply a larger distance from x0 , and hence a stronger robustness guarantee if the property holds . Intuitively , in a DNN for image classification that is -local-robust , small perturbations to x0 , i.e. , perturbations so small that a human would fail to detect them , should not result in a change of predicted class . Distinct Adversarial Robustness . Classification networks assign a value to each output label , which expresses the level of confidence in that label ; and the label with the highest confidence score wins . We are interested in an adversarial input x only if it is clearly misclassified , i.e. , if x ’ s assigned label received a significantly higher confidence score than that of the label assigned to x0 . For example , if argmax ( N ( x0 ) ) 6= argmax ( N ( x ) ) but c ( x0 ) = 0.41 and c ( x ) = 0.42 , x is not distinctly an adversarial input ; whereas a case where c ( x ) = 0.8 is clearly much more relevant . We refer to adversarial inputs whose assigned confidence value is sufficiently high ( greater than some δ ) as distinct adversarial inputs , and refine Definition 1 to only consider them , as follows : Definition 2 . A DNN N is ( , δ ) -locally-robust at input point x0 , iff ∀x . ‖x− x0‖∞ ≤ ⇒ ( argmax ( N ( x ) ) = argmax ( N ( x0 ) ) ∨ c ( x ) < δ )

Summary: Summary:  The paper introduces a statistical method to measure the robustness of deep neural networks. The novel  part of the method is that it's designed to measure the probability of random points near an input being  adversarial, instead of probability of adversarial examples existing in the vicinity of an input. To measure it,  the authors proposes to use Box-cox transformation to transform distribution of confidence scores to normal,  then calculate the probability based on it.