Fooling a Complete Neural Network Verifier

1 INTRODUCTION . In their seminal work , Szegedy et al . found that for a given neural network and input example one can always find a very small adversarial input perturbation that results in an incorrect output ( Szegedy et al. , 2014 ) . This striking discovery motivated a substantial amount of research . In this area , an important research direction is verification , that is , the characterization of the robustness of a given network in a principled manner . A usual way of defining the verification problem involves the specification of an input domain and a property that should hold over the entire domain . For example , we might require that all the points within a certain distance from an input example share the same output label as the example itself . The verification problem is then to prove or disprove the property over the domain for a given network ( Bunel et al. , 2020 ) . There are a large number of verifiers offering different types of guarantees about their output . Complete verifiers offer the strongest guarantee : they are able to decide whether a given property holds in any given input domain . For example , the verifier of Tjeng et al . is a state-of-the-art complete verifier that we will focus on in this paper ( Tjeng et al. , 2019 ) . However , it is currently standard practice to ignore the details of the computations that the network under investigation performs , such as the floating point representation or the order in which input signals are summed . In this paper , we claim that such implicit assumptions make verifiers vulnerable to a new kind of attack where the attacker designs a network that fools the verifier , exploiting the differences between how the verifier models the computation and how the computation is actually performed in the network . We will argue that such attacks can achieve an arbitrary divergence between the modeled and the actual behavior . This new attack has practical implications as well . Concerns about the safety of AI systems are expected to lead to the establishment of standard requirements certified by a designated authority ( Salis-Madinier , 2019 ) . These certification procedures might involve verification methods as well . Fooling such methods makes it possible to get unsafe systems certified that might even contain a backdoor allowing for triggering arbitrary behavior . Numerical precision has not been a key practical concern in machine learning . Networks do sometimes produce numerical errors ( e.g. , Inf or NaN values ) , most often due to the non-linear operations within the loss function ( Odena et al. , 2019 ) or divergence during training . However , the network weights are normally robust to small perturbations due to stochastic learning algorithms ( Bottou , 2010 ) , and due to regularizers such as standard variants of weight decay and dropout ( Srivastava et al. , 2014 ) . Due to this robustness , low precision arithmetic can be applied as well ( Courbariaux et al. , 2015 ; Gupta et al. , 2015 ) . Our results indicate that , when it comes to exact methods for verification , numerical issues become a central problem that can cause arbitrary errors and enable backdoors . Our contributions are the following . In Section 3 , we introduce a simple adversarial network that misleads the verifier of Tjeng et al . ( 2019 ) . In Section 4 , we show how to hide the large weights that are present in the simple network . In Section 5 , we describe a way to add a backdoor to an existing network with the help of the adversarial networks we proposed . Finally , in Section 6 we offer a defense against the attack we presented . 2 BACKGROUND . Let us first formulate the verification problem , namely the problem of checking whether a given property holds in a given domain . We adopt the notation used in ( Tjeng et al. , 2019 ) . For a possible input x , let G ( x ) denote the set of inputs that are considered similar to x in the sense that we expect all the points in G ( x ) to get the same label as x . The set G ( x ) is normally defined as a ball around x in some metric space defined by a suitable vector norm . The input domain we need to consider is given as G ( x ) ∩Xvalid whereXvalid denotes the valid input points . For example , we haveXvalid = [ 0 , 1 ] m if the input is an image of m pixels with each pixel taking values from the interval [ 0 , 1 ] . We now have to formulate the property that we wish to have in this domain . Informally , we want all the points in the domain G ( x ) ∩ Xvalid to get the same classification label as x . Let λ ( x ) denote the true label of x and let f ( x ; θ ) : Rm → Rn denote the neural network , parameterized by θ . This network has n outputs classifying each input x into n classes . The label of x as predicted by the network is given by argmaxi f ( x ; θ ) i . Using this notation , the property we wish to have for an input x′ ∈ ( G ( x ) ∩ Xvalid ) is that λ ( x ) = argmaxi f ( x′ ; θ ) i . Putting it all together , the verification problem can be expressed as deciding the feasibility of the constraint x′ ∈ ( G ( x ) ∩ Xvalid ) ∧ ( λ ( x ) 6= argmax i f ( x′ ; θ ) i ) , ( 1 ) with x′ as our variable . If this constraint is feasible then there is an x′ that violates the property . If it is infeasible then ( provided G ( x ) ∩ Xvalid is not empty ) there is no such x′ . 2.1 APPROACHES TO VERIFICATION . There are many approaches to tackle this problem . We can , for example , search for a suitable x′ in the given domain using some heuristic optimization methods ( Goodfellow et al. , 2015 ; MoosaviDezfooli et al. , 2016 ; Kurakin et al. , 2017 ; Carlini & Wagner , 2017 ; Brendel et al. , 2019 ) . If the search succeeds , we can decide that equation 1 is feasible . Otherwise we can not decide . Other methods attempt to find a proof for the infeasibility of equation 1 , however , they do not guarantee such a proof . Examples include ( Wong & Kolter , 2018 ; Weng et al. , 2018 ; Gehr et al. , 2018 ; Raghunathan et al. , 2018 ; Singh et al. , 2019 ) . If a proof is found , we can decide that equation 1 is infeasible . Otherwise we can not decide . Such methods are sometimes called incomplete ( Tjeng et al. , 2019 ; Bunel et al. , 2020 ) . The strongest guarantee is given by methods that are able to decide the feasibility of equation 1 . These methods are sometimes called complete ( Tjeng et al. , 2019 ; Bunel et al. , 2020 ) . Examples for such methods include Reluplex ( Katz et al. , 2017 ) , a method based on an SMT solver . A number of verifiers are based on MILP solvers , for example , ( Cheng et al. , 2017 ; Dutta et al. , 2018 ) . MIPVerify ( Tjeng et al. , 2019 ) also uses an MILP formulation along with several additional techniques to improve efficiency ( see Section 2.2 ) . Symbolic interval propagation has also been proposed for ReLU networks by Wang et al . in ReluVal ( Wang et al. , 2018b ) , and as part of Neurify ( Wang et al. , 2018a ) . In Neurify , interval propagation is used as a technique to tighten the bounds used for linear relaxation . Nnenum is another geometric method that is based on propagating linear star sets ( Bak et al. , 2020 ) . 2.2 MIPVERIFY . Although the idea behind the attack is not specific to a particular verifier—as we discuss in Section C of the Appendix—we develop and evaluate the attack in detail for a state-of-the-art complete verifier : MIPVerify ( Tjeng et al. , 2019 ) . It is based on a mixed integer linear programming ( MILP ) formulation . As long as the domain G ( x ) ∩ Xvalid is the union of a set of polyhedra , and the neural network f ( x , θ ) is a piecewise linear function of x with parameters θ , the problem of checking the feasibility of the constraint in equation 1 can be formulated as a MILP instance . G ( x ) is normally defined as a ball in a suitable norm with x as the center . In ` ∞ or ` 1 norms G ( x ) is thus a cube . Also , Xvalid is normally a box or a set of boxes , so the domain is indeed the union of a set of polyhedra . The neural network is piecewise linear as long as the nonlinearities used are ReLUs ( note that the last softmax normalization layer adds no extra information and can thus be ignored ) . For the details of the MILP formalization , please see ( Tjeng et al. , 2019 ) . Importantly , MIPVerify applies a presolve step that greatly increases its efficiency . In this step , the authors attempt to tighten the bounds on the variables of the problem , including on the inputs to each ReLU computation . If in this step it turns out that the input of a ReLU gate is always non-positive , the output can be fixed as a constant zero , and if the input is always non-negative then the ReLU gate can be removed from the model as it will have no effect . The presolve step applies three approaches in a progressive manner . First , a fast but inaccurate interval arithmetic approach is used . The resulting bounds are further improved by solving a relaxed LP problem on every variable . Finally , the full MILP problem is solved for the variables but with early stopping . 2.3 FLOATING POINT REPRESENTATION . Floating point real number representations are successful and efficient tools for most real life applications ( Muller et al. , 2010 ) . This arithmetic is available on most modern computers via sophisticated hardware implementations . A floating point number is represented as s · be , where s is the signed significand , b is the base and e is the exponent . There are numerous standards to implement the exact details of this idea that differ mainly in the number of bits that the significand and the exponent use . The formula to compute the represented real number has several possible variations as well . Here , we will use the double precision ( binary64 ) arithmetic defined by the IEEE 754-1985 standard ( IEEE , 1985 ) . There , b = 2 and we have a sign bit , an 11 bit exponent , and a 53 bit significand ( with 52 bits stored explicitly ) . The so called machine epsilon ( the maximum relative rounding error ) is 2−53 ≈ 1.11e−16 . This means that , for example , the computation 1020 + 2020 − 1020 will result in zero in this representation , if executed in the specified order . In our attack , we will exploit roundoff errors of this type . Note , that in the order of 1020 − 1020 + 2020 we obtain the correct result of 2020 .

Summary: The authors show that certain complete neural network verifiers can be mislead by carefully crafted neural networks that exploit round-off errors, which when large magnitude values overwhelm low magnitude values. Such a construction can be obfuscated by taking advantage of the compounding effect when there are many layers of the network. This can also be used to add backdoors to existing networks, albeit in a way that looks quite artificial.