Computing Preimages of Deep Neural Networks with Applications to Safety

To apply an algorithm in a sensitive domain it is important to understand the set of input values that result in specific decisions . Deep neural networks suffer from an inherent instability that makes this difficult : different outputs can arise from very similar inputs . We present a method to check that the decisions of a deep neural network are as intended by constructing the exact , analytical preimage of its predictions . Preimages generalize verification in the sense that they can be used to verify a wide class of properties , and answer much richer questions besides . We examine the functioning and failures of neural networks used in robotics , including an aircraft collision avoidance system , related to sequential decision making and extrapolation . Our method iterates backwards through the layers of piecewise linear deep neural networks . Uniquely , we compute all intermediate values that correspond to a prediction , propagating this calculation through layers using analytical formulae for layer preimages . 1 INTRODUCTION . Folk wisdom holds that although deep neural networks ( DNNs ) can achieve excellent predictive accuracy , reasoning about their performance is difficult , even for experts . Our goal is to enable nonexpert stakeholders , such as clinical health workers , investors , or military commanders to build trust a statistical model in high-stakes environments . To do this , we posit that decisionmakers want to understand a model in both directions , both from inputs to outputs , but also being able to start with hypothetical outputs , and understand the inputs that lead to them . In this paper , we develop an equivalent , but much simpler , representation of a certain class of DNN classifiers . This representation , which requires only a basic numeracy to productively interact with , can be used by domain experts to build intuition and trust . We apply this method to a reinforcement learning agent trained to solve the cart-pole problem , and find that a DNN implementing a successful policy makes a particular type of mistake on 24 % of the mass of the 1/8th of the state space for which we know the optimal action ( Section 3.2 ) . We also show how using the preimage in place of verification can yield a more efficient and interpretable end-to-end system for analyzing aircraft collision avoidance systems ( Section 3.3 ) . 1.1 PREVIOUS WORK . DNNs have the property that knowing the output tells us very little about the input it corresponds to . This is most apparent in image classifiers , where totally different outputs can arise from inputs that are visually indistinguishable ( Szegedy et al . ( 2014 ) ) . We build upon the mathematical framework developed for verifying DNNs that grew out of a desire to prove the absence of adversarial examples , for example Tjeng et al . ( 2017 ) and Wong & Kolter ( 2017 ) . However , we depart from these studies along with Katz et al . ( 2017 ) , being more oriented towards small DNNs that map to and from lowdimensional spaces with considerable structure . These DNNs arise especially in systems which interoperate with the physical world , for example mapping measurements of positions and velocities to movements . Table 1 orients our work to the literature . We have phrased verification in this unusual fashion to facilitate comparison with the other points . Stated in the familiar application to image classifiers X would be an epsilon ball around an input , and Y would be the halfspace where one coordinate is higher than all others . Verification ultimately amounts to a simple yes or no , and so answering higher-level questions typically requires many verifications : for example , Katz et al . ( 2017 ) describes a suite of 45 tests , and image classifiers often wish to verify the absence of adversarial examples around the entire training set . Yang et al . ( 2020 ) is an interesting extension to verification in that it computes the entire image of , say , an epsilon ball around a data point , and not just whether it intersects with a decision boundary . Reasoning forward , about the outputs that can arise from inputs , is only half of the picture . Carlsson et al . ( 2017 ) and Behrmann et al . ( 2018 ) are oriented backwards , they attempt to reconstruct the inputs that result in an output . These related papers study the statistical invariances that nonlinear layers encode . Behrmann et al . ( 2018 ) examines the preimage of a single point through a single ReLU layer , analyzing stability via an approximation-based experiment . Carlsson et al . ( 2017 ) analyzes the preimage of a single point through the repeated application of a nonlinearity , purely theoretically . Our paper looks at the preimage of non-singleton subsets of the codomain , which is much more practically useful , and requires considerable extension to their approaches . 2 METHOD . Our method is easily stated : build up the preimage of a DNN from the preimage of its layers , using simple analytical formulae . We start by developing some properties of the preimage operator , then we describe the class of sets that we compute the preimage of , and finally we discuss the class of DNNs that our algorithm addresses . 2.1 PROPERTIES OF PREIMAGES . Lemma 1 shows how to build up the preimage of a DNN from the preimages of its constitutent layers . Lemma 1 ( Preimage of composition is reversed composition of preimages ) . For functions fj : Rnj → Rnj+1 , ( f ` +k ◦ f ` +k−1 ◦ . . . ◦ f ` ) −1 = f−1 ` ◦ . . . ◦ f −1 ` +k−1 ◦ f −1 ` +k . ( 1 ) Secondly , we mention an intuitive property of f−1 that is handy for building up the preimage of any set from the preimages of any partition of that set . Lemma 2 ( Preimage of union is union of preimages ) . f−1 ( ∪Ni=1Si ) = ∪Ni=1f−1 ( Si ) . 2.2 POLYTOPES . Our method is not applicable to arbitrary sets Y , but rather sets that , roughly , have piecewise linear boundaries . The basic building block of these sets are polytopes . Definition 1 ( Polytope ) . A polytope in Rn is a set that can be written as { x ∈ Rn : b − Ax ≥ 0 } for some m ∈ N , b ∈ Rm , and A ∈ Rm×n . Put more simply : a polytope is the intersection of half-planes . Definition 1 does not require that polytopes be bounded , but polytopes are convex . Sets with linear boundaries , though they may be non-convex , can decomposed into the union of polytopes . We term such sets region-unions , and the set of polytopes which comprise them , regions . Definition 2 ( Region and region-union ) . For N ∈ N , bi ∈ Rmi , Ai ∈ Rmi×n , with mi ∈ N , a region is { { x : bi −Aix ≥ 0 } ; i = 1 , . . . , N } . ( 2 ) A region-union is a set ∪r∈R r for some region R .. Region-unions are interesting because the the preimage polytopes under piecewise linear functions are regions-unions . However , we need to also keep information on how to form a region-union , hence the notion of a region . It is trivial to observe that if R1 and R2 are regions , then R1 ∪ R2 is likewise a region , and correspondingly for region-unions . 2.3 LINEAR AND RELU POLYTOPE PREIMAGES . In this section , we give formulae for the preimage of linear and ReLU functions , giving significant content to Lemma 1 . The preimage of polytopes under linear mappings are polytopes : Lemma 3 ( Preimage of Linear layer ) . ( x 7→Wx+ a ) −1 ( { x : b−Ax ≥ 0 } ) = { x : ( b−Aa ) −AWx ≥ 0 } . ( 3 ) ReLU is a piecewise linear function , so if we carefully treat the portions of the domain on which it exhibits different behavior , we obtain a similar formulation for each : Lemma 4 ( Preimage of ReLU layer ) . ReLU−1 ( { x : b−Ax ≥ 0 } ) = ⋃ ν∈ { 0,1 } n { x : b−Adiag ( ν ) x ≥ 0 , −diag ( 1− ν ) x ≥ 0 , diag ( ν ) x ≥ 0 } . ( 4 ) To understand Lemma 4 let s ( x ) be the vector given by s ( x ) i = 1 if xi ≥ 0 and zero otherwise . Then diag ( s ( x ) ) x = ReLU ( x ) . This expression separates x 7→ ReLU ( x ) into a pattern of signs over its coordinates and x itself . This means that once we restrict attention to a set on which the sign does not change , we can apply familiar linear algebra routines to compute the preimage set , akin to Lemma 3 . The nonnegative values are denoted by ν ∈ { 0 , 1 } n in the above , and the set of x such that xi ≥ 0 ⇐⇒ νi = 1 is given by diag ( ν ) x ≥ 0 . Similarly , xi ≤ 0 ⇐⇒ νi = 0 for i = 1 , 2 , . . . , n if and only if −diag ( 1− ν ) x ≥ 0 . Equation 4 follows by partitioning Rn into the 2n sets where each coordinate is nonnegative or not . Computing the preimage of a ReLU layer is unavoidably intractable at scale , though the problem exhibits considerable structure . We expect that it is possible to compute the preimage of networks of a similar scale to those that can be completely verified , such as small image-scale networks . Preimages are most insightful and useful when the inputs and outputs have definite interpretation – application areas where the need for massive networks is less . 2.4 THE SUFFICIENCY OF LINEAR AND RELU LAYERS . In familiar terms a DNN classifier might consist of some “ feature building ” modules , say composed of alternating convolution and maxpooling , then flattened , and passed onto the prediction logic consisting of alternating linear and ReLU layers , possibly including dropout or batch normalization , and concluding with a softmax function to normalize the predictions to a probability distribution . Resnets ( He et al . ( 2016 ) ) do not strictly fit this pattern , bu can be handled with similar reasoning ( see Appendix B ) . How do the results of Section 2.3 suffice to invert such DNNs ? Firstly , under our convention that layers operate on flat tensors , flattening is superfluous . Next , dropout affects inference only through the weights – this layer can be omitted entirely in computing the preimage . Convolution is essentially linear . Maxpool is straightforwardly rewritten in terms of the ReLU and linear function . { x : b− Asoftmax ( x ) ≥ 0 } is not a polytope . However , if the classification alone suffices then the softmax layer can be elided entirely since arg maxj xj = arg maxj softmax ( x ) j . 3 EXPERIMENTS . 3.1 TWO MOONS CLASSIFICATION . To cultivate some intuition about the preimage of a DNN we start by examining a classic test problem in nonlinear classification . We fit a DNN f : [ −3 , +3 ] 2 → R2 consisting of two nonlinear layers with eight neurons each on an instance of the “ two moons ” dataset . This data is shown in Figure 1a ( further details of details of f and the data are in Section D.1 ) . Figure 1b plot the corresponding logits , along with the sets to be inverted { x : x1 ≶ x2 } ⊆ R2 . Figure 1c shows the corresponding preimages , with different hues of the same color corresponding to different sign patterns ν in Equation 4 .

Summary: Deep neural networks are known to be brittle, and can lead to dangerous consequences if left unverified. Forward reach set computation can be used as a basic primitive to verify properties of deep neural networks used in a robotic setting. There has been a rising interest in verifying larger neural networks used in safety critical setting.