RoMA: a Method for Neural Network Robustness Measurement and Assessment 

Neural network models have become the leading solution for a large variety of tasks , such as classification , language processing , protein folding , and others . However , their reliability is heavily plagued by adversarial inputs : small input perturbations that cause the model to produce erroneous outputs . Adversarial inputs can occur naturally when the system ’ s environment behaves randomly , even in the absence of a malicious adversary , and are a severe cause for concern when attempting to deploy neural networks within critical systems . In this paper , we present a new statistical method , called Robustness Measurement and Assessment ( RoMA ) , which can measure the expected robustness of a neural network model . Specifically , RoMA determines the probability that a random input perturbation might cause misclassification . The method allows us to provide formal guarantees regarding the expected frequency of errors that a trained model will encounter after deployment . Our approach can be applied to large-scale , black-box neural networks , which is a significant advantage compared to recently proposed verification methods . We apply our approach in two ways : comparing the robustness of different models , and measuring how a model ’ s robustness is affected by the magnitude of input perturbation . One interesting insight obtained through this work is that , in a classification network , different output labels can exhibit very different robustness levels . We term this phenomenon categorial robustness . Our ability to perform risk and robustness assessments on a categorial basis opens the door to risk mitigation , which may prove to be a significant step towards neural network certification in safety-critical applications . 1 INTRODUCTION . In the passing decade , deep neural networks ( DNNs ) have emerged as one of the most exciting and innovative developments in computer science , allowing computers to outperform humans in various classification tasks . However , a major disadvantage of the DNN approach is the existence of adversarial inputs Goodfellow et al . ( 2014 ) : inputs which are very close ( according to some metrics ) to a correctly-classified input , but which are misclassified themselves . It has been observed that most state-of-the-art DNNs are highly vulnerable to adversarial inputs Carlini & Wagner ( 2017 ) ; and it has been suggested that adversarial inputs are an inescapable part of the neural network architecture , and are thus not an issue that can be solved entirely Ilyas et al . ( 2019 ) . In this perspective , it is crucial to find a method for containing this phenomenon and mitigating the risk that it causes ; especially in order to allow DNNs to be deployed in safety-critical settings ( e.g. , automotive , aerospace , trains , and medical devices ) , where regulatory requirements and public opinion set a high bar for reliability . As the impact of the AI revolution is becoming evident , regulatory agencies are starting to address the challenge of safely integrating DNNs into safety-critical systems — by forming workgroups to create the needed guidelines . Notable examples in the European Union include SAE G-34 and EUROCAE WG-114 Pereira & Thomas ( 2020 ) ; Vidot et al . ( 2021 ) ; and the efforts made by the European Union Safety Agency ( EASA ) , which is responsible for civil aviation safety , and which has published a road map for certifying AI-based systems European Union Aviation Safety Agency ( 2020 ) . These efforts highlight the dire need for certification methods for DNN-based systems . The existence of adversarial inputs does not seem to be a deal-breaker as far as regulatory agencies are concerned ; indeed , these agencies are used to certifying systems with components that might fail due to an unexpected hazard . A common example is the certification of jet engines for civilian aircraft , with a known mean time between failures ( MTBF ) . Based on the MTBF value , there exist certification processes that perform functional hazard analysis ( FHA ) and risk mitigation FAA ( 2021 ) . To perform similar processes for DNN-based systems , one needs to assess the likelihood of failure ( i.e. , an adversarial input ) , but this essential ability is still missing . In this paper , we attempt to address this crucial gap by introducing a novel , straightforward , and scalable statistical method for measuring the probability that a DNN classifier will misclassify inputs . The method , which we term Robustness Measurement and Assessment ( RoMA ) , assumes that the misclassification of the neural network is not due to a malicious attack , but rather due to random perturbations of the input , which occur naturally as part of the system ’ s operation . Under this assumption , RoMA can be used to measure the model ’ s robustness to randomly-produced adversarial inputs . The proposed method is useful for several applications , such as comparing the robustness of multiple models and picking the best one , or for checking the impact of various configurable parameters ( e.g. , the number of training epochs , or the magnitude of the input perturbation ) on the resulting model ’ s robustness . RoMA is a method for estimating rare events in a large population — in our case , adversarial inputs within a space of inputs that are generally correctly classified . The method relies on the properties of normal distributions . If it is known that the rare events ( adversarial inputs ) are distributed normally , it is usually sufficient to sample a few thousands random points , use them to draw a Gaussian curve , and then use the normal distribution function to evaluate the probability of a rare event in the population . Unfortunately , adversarial inputs are often not distributed normally in the population ; and to overcome this difficulty , RoMA first applies a statistical transformation , after which the distribution often becomes normal and can be analyzed . At a high level , RoMA consists of the following steps : 1. for an arbitrary input point x0 , we randomly sample n perturbations of x0 ( usually , a few thousands ) , obtaining a set of perturbed inputs { x10 , . . . , xn0 } ; 2. we evaluate the DNN on each xi0 , obtaining the corresponding outputs { y10 , . . . , yn0 } ; 3. for each yi0 that is classified incorrectly ( that is , which indicates a labeling different than that of x0 ) , we collect the maximal entry ci of yi0 . This value is the confidence score assigned by the DNN to the incorrect label ; 4. if needed , we apply a statistical transformation called Box-Cox Box & Cox ( 1982 ) to normalize the distribution of ci values ; and 5. if the distribution is now normal , we use the properties of the normal distribution function to calculate the probability for an adversarial input around x0 . A key component in our method is the Box-Cox statistical power transformation , which is a wellestablished method , and which does not pose any restrictions on the DNN in question ( e.g. , Lipschitz continuity , certain kinds of activation functions , or a specific network topology ) . Further , the method does not require access to the network ’ s topology or weights , and is thus applicable to black-box DNNs . We implemented our method as a proof-of-concept tool , and evaluated it on standard DNN architectures : VGG16 Simonyan & Zisserman ( 2015 ) , Resnet He et al . ( 2016 ) , and Densenet Huang et al . ( 2017a ) , all trained on the CIFAR10 data set Krizhevsky et al . ( 2009 ) . We used RoMA to compare the robustness of these DNN models , and found , as expected , that a higher number of epochs leads to a higher robustness score . Additionally , we used RoMA to measure how the allowed magnitude of perturbation affects the robustness of a DNN model , and computed the rate at which robustness deteriorates as the perturbation level increases . Finally , using RoMA , we found that the categorial robustness score of a DNN ( i.e. , the robustness score of inputs labeled as a particular category ) varies significantly among the different categories . This finding could allow users and regulators to specify different acceptable robustness thresholds for each target category , instead of a single global threshold , which may be more difficult to obtain . To summarize , our main contributions are : • Introducing RoMA : a new method for measuring the robustness of a DNN model . The new method is scalable and can run on black-box DNNs . • Comparing the robustness of multiple state-of-the-art DNN models . • Using RoMA to measure the effect of perturbation level has on the robustness of the DNN model . • Introducing the notion of categorial robustness , which is a measure of robustness computed for each target label . Related work . The topic of adversarial robustness has been studied extensively . Some notable approaches for estimating a model ’ s robustness include : • Statistical approaches that evaluate the probability of encountering an adversarial input in the population . In a recent paper Huang et al . ( 2021 ) , Huang et al . use random sampling , which is similar in spirit to RoMA , but which assumes that the DNN ’ s adversarial inputs are distributed normally . As we demonstrate later , this is often not the case . In another paper , Webb at al . Webb et al . ( 2018 ) use a sampling method called multi-level splitting , which provides no formal guarantee of the DNN ’ s robustness . Mangal et al . Mangal et al . ( 2019 ) use importance sampling , which might be biased due to lack of sampling in areas of the population that are deemed unimportant . Moreover , this approach assumes that the network ’ s output is Lipschitz-continuous , which limits its applicability . In contrast , RoMA requires no Lipschitz-continuity assumptions , does not assume a-priori that the adversarial inputs are distributed normally , and provides rigorous robustness guarantees . • Formal-verification based approaches Katz et al . ( 2017 ) ; Wang et al . ( 2018 ) ; Jacoby et al . ( 2020 ) , which allow for computing a DNN ’ s exact adversarial robustness score . These approaches typically convert the problem into a constraint satisfiability problem , and then apply search and deduction procedures to solve it efficiently . However , verification-based approaches afford only limited scalability , and operate strictly on white-box DNNs . In contrast , RoMA is a scalable technique , and can operate on black-box DNNs . • Approaches for computing an estimate bound on the probability that a classifier ’ s margin function exceeds a given value Weng et al . ( 2019 ) ; Anderson & Sojoudi ( 2020 ) ; Dvijotham et al . ( 2018 ) . These analyses focus on the worst-case behavior , thus producing bounds that might be inadequate for regulatory certification . In contrast , RoMA focuses on the average case , which is more realistic in many application domains . Outline . We begin with some needed background on adversarial robustness in Section 2 . We then describe our proposed method for measuring adversarial robustness in Section 3 , followed by a description of our evaluation setup in Section 4 . In Section 5 we summarize and discuss our results . 2 BACKGROUND . Neural Network . A neural network N is a function N : Rn → Rm , which maps a real-valued input vector x ∈ Rn to a real-valued output vector y ∈ Rm . For classification networks , which is our subject matter here , the entries of y represent confidence scores for m possible classes ; and x is classified as label l if y ’ s l ’ th entry has the highest score ; i.e. , if argmax ( N ( x ) ) = l. We use c ( x ) to denote this highest score , i.e . c ( x ) = max ( N ( x ) ) . Local Adversarial Robustness . The local adversarial robustness of a DNN is a measure of how resilient that network is against adversarial perturbations to specific inputs . Intuitively , a network with high robustness behaves “ smoothly ” , i.e. , small perturbations to its input do not cause significant spikes in its output . More formally Bastani et al . ( 2016 ) ; Huang et al . ( 2017b ) : Definition 1 . A DNN N is -locally-robust at input point x0 iff ∀x . ||x− x0||∞ ≤ ⇒ argmax ( N ( x ) ) = argmax ( N ( x0 ) ) Intuitively , Definition 1 states that for input vector x , which is at a distance at most from a fixed input x0 , the network assigns to x the same label that it assigns to x0 ( for simplicity , we use here the L∞ norm , but other metrics could also be used ) . When a network is not -local-robust at point x0 , there exists a point x that is at a distance of at most from x0 , which is misclassified ; this x is called an adversarial input . In this context , local refers to the fact that x0 is fixed . Larger values of imply a larger distance from x0 , and hence a stronger robustness guarantee if the property holds . Intuitively , in a DNN for image classification that is -local-robust , small perturbations to x0 , i.e. , perturbations so small that a human would fail to detect them , should not result in a change of predicted class . Distinct Adversarial Robustness . Classification networks assign a value to each output label , which expresses the level of confidence in that label ; and the label with the highest confidence score wins . We are interested in an adversarial input x only if it is clearly misclassified , i.e. , if x ’ s assigned label received a significantly higher confidence score than that of the label assigned to x0 . For example , if argmax ( N ( x0 ) ) 6= argmax ( N ( x ) ) but c ( x0 ) = 0.41 and c ( x ) = 0.42 , x is not distinctly an adversarial input ; whereas a case where c ( x ) = 0.8 is clearly much more relevant . We refer to adversarial inputs whose assigned confidence value is sufficiently high ( greater than some δ ) as distinct adversarial inputs , and refine Definition 1 to only consider them , as follows : Definition 2 . A DNN N is ( , δ ) -locally-robust at input point x0 , iff ∀x . ‖x− x0‖∞ ≤ ⇒ ( argmax ( N ( x ) ) = argmax ( N ( x0 ) ) ∨ c ( x ) < δ )

Summary: Present a method to measure the expected robustness of a neural network model, by determining  the probability that a random input perturbation might cause misclassification, providing formal guarantees regarding the expected frequency of errors that a trained model will encounter after deployment. The method can be applied black-box. Applied the approach to compare the robustness of different models, and measure how a model’s robustness is affected by the magnitude of input perturbation.