Exploiting Verified Neural Networks via Floating Point Numerical Error

1 INTRODUCTION . Deep neural networks ( DNNs ) are known to be vulnerable to adversarial inputs ( Szegedy et al. , 2014 ) , which are images , audio , or texts indistinguishable to human perception that cause a DNN to give substantially different results . This situation has motivated the development of network verification algorithms that claim to prove the robustness of a network ( Bunel et al. , 2020 ; Tjeng et al. , 2019 ; Salman et al. , 2019 ) , specifically that the network produces identical classifications for all inputs in a perturbation space around a given input . Verification algorithms typically reason about the behavior of the network assuming real-valued arithmetic . In practice , however , the computation of both the verifier and the neural network is performed on physical computers that use floating point numbers and floating point arithmetic to approximate the underlying real-valued computations . This use of floating point introduces numerical error that can potentially invalidate the guarantees that the verifiers claim to provide . Moreover , the existence of multiple software and hardware systems for DNN inference further complicates the situation , because different implementations exhibit different numerical error characteristics . We present concrete instances where numerical error leads to unsound verification of real-valued networks . Specifically , we train robust networks on the MNIST and CIFAR10 datasets . We work with the MIPVerify complete verifier ( Tjeng et al. , 2019 ) and several inference implementations included in the PyTorch ( Paszke et al. , 2019 ) framework . For each implementation , we construct image pairs ( x0 , xadv ) where x0 is a brightness modified natural image , such that the implementation classifies xadv differently from x0 , xadv falls in a ` ∞-bounded perturbation space around x0 , and the verifier incorrectly claims that no such adversarial image xadv exists for x0 within the perturbation space . Moreover , we show that the incomplete verifier CROWN is also vulnerable to floating point error . Our method of constructing adversarial images is not limited to our setting , and it is applicable to other verifiers that do not soundly model floating point arithmetic . 2 BACKGROUND AND RELATED WORK . Training robust networks : Researchers have developed various techniques to train robust networks ( Madry et al. , 2018 ; Mirman et al. , 2018 ; Tramer & Boneh , 2019 ; Wong et al. , 2020 ) . Madry et al . formulate the robust training problem as minimizing the worst loss within the input perturbation and propose to train robust networks on the data generated by the Projected Gradient Descent ( PGD ) adversary ( Madry et al. , 2018 ) . In this work we consider robust networks trained with the PGD adversary . Complete verification : The goal of complete verification ( a.k.a . exact verification ) methods is to either prove the property being verified or provide a counterexample to disprove it . Complete verification approaches have formulated the verification problem as a Satisfiability Modulo Theories ( SMT ) problem ( Scheibler et al. , 2015 ; Huang et al. , 2017 ; Katz et al. , 2017 ; Ehlers , 2017 ; Bunel et al. , 2020 ) or as a Mixed Integer Linear Programming ( MILP ) problem ( Lomuscio & Maganti , 2017 ; Cheng et al. , 2017 ; Fischetti & Jo , 2018 ; Dutta et al. , 2018 ; Tjeng et al. , 2019 ) . While SMT solvers are able to model exact floating point arithmetic ( Rümmer & Wahl , 2010 ) or exact real arithmetic ( Corzilius et al. , 2012 ) , deployed SMT solvers for verifying neural networks all use inexact floating point arithmetic to reason about the neural network inference for efficiency reasons . MILP solvers work directly with floating point , do not attempt to exactly model real arithmetic , and therefore exhibit numerical error . Since floating point arithmetic is not associative , different neural network implementations may produce different results for the same neural network , implying that any sound verifier for this class of networks must reason about the specific floating point error characteristics of the neural network implementation at hand . To the best of our knowledge , no prior work formally recognizes the problem of floating point error in neural network complete verification or exploits floating point error to invalidate verification results . Incomplete verification : On the spectrum of the tradeoff between completeness and scalability , incomplete methods ( a.k.a . certification methods ) aspire to deliver more scalable verification by adopting over-approximation , while admitting the inability to either prove or disprove the properties in certain cases . There is a large body of related research ( Wong & Kolter , 2017 ; Weng et al. , 2018 ; Gehr et al. , 2018 ; Zhang et al. , 2018 ; Raghunathan et al. , 2018 ; Dvijotham et al. , 2018 ; Mirman et al. , 2018 ; Singh et al. , 2019 ) . Salman et al . ( 2019 ) has unified most of the relaxation methods under a common convex relaxation framework . Their results suggest that there is an inherent barrier to tight verification via layer-wise convex relaxation captured by their framework . We highlight that floating point error of implementations that use a direct dot product formulation has been accounted for in some certification frameworks ( Singh et al. , 2018 ; 2019 ) by maintaining upper and lower rounding bounds for sound floating point arithmetic ( Miné , 2004 ) . Such frameworks should be extensible to model numerical error in more sophisticated implementations like the Winograd convolution ( Lavin & Gray , 2016 ) , but the effectiveness of this extension remains to be studied . Most of the certification algorithms , however , have not considered floating point error and may be vulnerable to attacks that exploit this deficiency . Floating point arithmetic : Floating point is widely adopted as an approximate representation of real numbers in digital computers . After each calculation , the result is rounded to the nearest representable value , which induces roundoff error . In the field of neural networks , the SMT-based verifier Reluplex ( Katz et al. , 2017 ) has been observed to produce false adversarial examples due to floating point error ( Wang et al. , 2018 ) . The MILP-based verifier MIPVerify ( Tjeng et al. , 2019 ) has been observed to give NaN results when verifying pruned neural networks ( Guidotti et al. , 2020 ) . Such observed floating point unsoundness behavior occurs unexpectedly in running large scale benchmarks . However , no prior work tries to systematically invalidate neural network verification results via exploiting floating point error . The IEEE-754 ( IEEE , 2008 ) standard defines the semantics of operations and correct rounding behavior . On an IEEE-754 compliant implementation , computing floating point expressions consisting of multiple steps that are equivalent in the real domain may result in different final roundoff error because rounding is performed after each step , which complicates the error analysis . Research on estimating floating point roundoff error and verifying floating point programs has a long history and is actively growing ( Boldo & Melquiond , 2017 ) , but we are unaware of any attempt to apply these tools to obtain a sound verifier for any neural network inference implementation . Any such verifier must reason soundly about floating point errors in both the verifier and the neural network inference algorithm . The failure to incorporate floating point error in software systems has caused real-world disasters . For example , in 1992 , a Patriot missile missed its target and lead to casualties due to floating point roundoff error related to time calculation ( Skeel , 1992 ) . 3 PROBLEM DEFINITION . 3.1 ADVERSARIAL ROBUSTNESS OF NEURAL NETWORKS . We consider 2D image classification problems . Let y = NN ( x ; W ) denote the classification confidence given by a neural network with weight parametersW for an input x , where x ∈ Rm×n×c [ 0,1 ] is an image with m rows and n columns of pixels each containing c color channels represented by floating point values in the range [ 0 , 1 ] , and y ∈ Rk is a logits vector containing the classification scores for each of the k classes . The class with the highest score is the classification result of the neural network . For a logits vector y and a target class number t , we define the Carlini-Wagner ( CW ) loss ( Carlini & Wagner , 2017 ) as the score of the target class subtracted by the maximal score of the other classes : LCW ( y , t ) = yt −max i 6=t yi ( 1 ) Note that x is classified as an instance of class t if and only if LCW ( NN ( x ; W ) , t ) > 0 , assuming no equal scores of two classes . Adversarial robustness of a neural network is defined for an input x0 and a perturbation bound , such that the classification result is stable within allowed perturbations : ∀x ∈ Adv ( x0 ) : LCW ( NN ( x ; W ) , t0 ) > 0 ( 2 ) where t0 = argmax NN ( x0 ; W ) In this work we focus on ` ∞-norm bounded perturbations : Adv ( x0 ) = { x | ‖x− x0‖∞ ≤ ∧ minx ≥ 0 ∧ maxx ≤ 1 } ( 3 ) 3.2 FINDING ADVERSARIAL EXAMPLES FOR VERIFIED NETWORKS VIA EXPLOITING NUMERICAL ERROR . Due to the inevitable presence of numerical error in both the network inference system and the verifier , the exact specification of NN ( · ; W ) ( i.e. , a bit-level accurate description of the underlying computation ) is not clearly defined in ( 2 ) . We consider the following implementations of convolutional layers included in the PyTorch framework to serve as our candidate definitions of the convolutional layers in NN ( · ; W ) , and other layers use the default PyTorch implementation : • NNC , M ( · ; W ) : A matrix multiplication based implementation on x86/64 CPUs . The convolution kernel is copied into a matrix that describes the dot product to be applied on the flattened input for each output value . • NNC , C ( · ; W ) : The default convolution implementation on x86/64 CPUs . • NNG , M ( · ; W ) : A matrix multiplication based implementation on NVIDIA GPUs . • NNG , C ( · ; W ) : A convolution implementation using the IMPLICIT_GEMM algorithm from the cuDNN library ( Chetlur et al. , 2014 ) on NVIDIA GPUs . • NNG , CWG ( · ; W ) : A convolution implementation using the WINOGRAD_NONFUSED al- gorithm from the cuDNN library ( Chetlur et al. , 2014 ) on NVIDIA GPUs . It is based on the Winograd fast convolution algorithm ( Lavin & Gray , 2016 ) , which has much higher numerical error compared to others . For a given implementation NNimpl ( · ; W ) , our method finds pairs of ( x0 , xadv ) represented as single precision floating point numbers such that 1. x0 and xadv are in the dynamic range of images : minx0 ≥ 0 , minxadv ≥ 0 , maxx0 ≤ 1 , and maxxadv ≤ 1 . 2. xadv falls in the perturbation space of x0 : ‖xadv − x0‖∞ ≤ 3 . The verifier claims that ( 2 ) holds for x0 4. xadv is an adversarial image for the implementation : LCW ( NNimpl ( xadv ; W ) , t0 ) < 0 Note that the first two conditions are accurately defined for any implementation compliant with the IEEE-754 standard , because the computation only involves element-wise subtraction and maxreduction that incur no accumulated error . The Gurobi ( Gurobi Optimization , 2020 ) solver used by MIPVerify operates with double precision internally . Therefore , to ensure that our adversarial examples satisfy the constraints considered by the solver , we also require that the first two conditions hold for x′adv = float64 ( xadv ) and x ′ 0 = float64 ( x0 ) that are double precision representations of xadv and x0 .

Summary: In the recent literature there has been a rise in the number of papers which attempt to verify neural networks. The specification of the verification problems often gets adapted according to the application in mind. More specifically, for image classification networks, the problem is to prove that the output of the neural network does not flip for small perturbations to the pixel values. For a robotic setting, the problem is often safety and convergence to some goal state. Where the neural network operates in closed loop with the system dynamics.