A Study of Robustness of Neural Nets Using Approximate Feature Collisions

In recent years , various studies have focused on the robustness of neural nets . While it is known that neural nets are not robust to examples with adversarially chosen perturbations as a result of linear operations on the input data , we show in this paper there could be a convex polytope within which all examples are misclassified by neural nets due to the properties of ReLU activation functions . We propose a way to find such polytopes empirically and demonstrate that such polytopes exist in practice . Furthermore , we show that such polytopes exist even after constraining the examples to be a composition of image patches , resulting in perceptibly different examples at different locations in the polytope that are all misclassified . 1 INTRODUCTION . Deep learning has achieved resounding success in recent years and is quickly becoming an integral component of many real-world systems . As a result of its success , increasing attention has focused on studying the robustness of neural nets , in particular to inputs that are deliberately chosen to yield an unexpected prediction . Various studies have shown the neural nets and other models are not robust to adversarial examples ( Dalvi et al. , 2004 ; Biggio et al. , 2013 ; Szegedy et al. , 2013 ) , that is , examples from one class with minute perturbations that are designed to cause them to be classified as a different class . To make neural nets more resistant to these examples , it has been suggested ( Goodfellow et al. , 2014 ) to train the neural net specifically on these examples , which significantly reduces the error rate on this kind of examples . One explanation for these adversarial examples ( Goodfellow et al. , 2014 ) is that linearity in the model makes it possible to construct adversarial perturbations that result in a large change in the dot product between the input and the parameters when the dimensionality of the input is high , even though the perturbation to each element of the input is small in magnitude . In this paper , we show the existence of another type of examples that would , too , yield unexpected predictions . Unlike adversarial examples , these examples arise from the properties of non-linear activation functions in neural nets . We show that there could be examples that look nothing like an example from the target class , but are classified confidently as that class . In fact , there could be infinitely many such examples , obtained by arbitrary convex combinations of different elementary examples . The key observation is that we could change any component in the pre-activation of a layer before ReLUs that has a negative value to any other negative value without changing the post-activation , and consequently keep all pre-activations and post-activations in later layers and therefore the predicted class label the same . We show that in general , for any neural net with a layer with ReLU activations , there is a convex ( but possibly unbounded ) polytope in the space of post-activations of the previous layer such that all input examples whose post-activation vectors fall in the interior or the boundary of the polytope will have identical activations in all later layers . Moreover , we develop a way of finding such a relaxed version of this polytope in practice , where the activations in later layers for examples within this polytope are only approximately equal . Finally , we show empirically that such a polytope exists near practically any input example that we tried , where all examples within this polytope result in the same ( incorrect ) classification . Surprisingly , such a polytope exists even if we were to constrain the input to be a composition of image patches , thereby ensuring that only macro-level changes to the input are permitted . We call examples in this polytope approximate fea- ture collisions 1 , to borrow terminology from the hashing literature , since these examples are quite different from the target example but share very similar activations in later layers . 2 METHOD . Consider any layer in a neural net with ReLU activations . Let W ∈ RN×d denote the weight matrix , x ∈ Rd the previous layer ’ s post-activation , b ∈ RN the biases associated with the current layer . Moreover , let ’ s define y ∈ RN as the vector of post-activations ( activations after the ReLU ) , and ỹ ∈ RN as the vector of pre-activations ( activations before the ReLU ) . We can express the post-activations as : y = max ( ỹ , 0 ) = max ( Wx + b , 0 ) where W = wT1 wT2 ... wTn b = b1 b2 ... bn Our goal is to find a colliding example that has the same post-activations as a target example . We can identify examples by their post-activations in the previous layer , since two examples with identical post-activations in the previous layer will always have identical pre- and post-activations in the current layer . We will denote the colliding example as x∗ and the target example as xt and define y∗ , ỹ∗ , yt , ỹt analogously . Since ReLUs map all non-positive values to zeros , if the target example has a post-activation of zero in one component , as long as the pre-activation of the colliding example is non-positive in that component , then the target and the colliding example would have the same post-activation in that component . In order to make the post-activations of the two examples identical in all components , the following conditions are necessary and sufficient : ∀i s.t . ỹti > 0 , ỹ∗i = wTi x∗ + bi = ỹti ∀i s.t . ỹti ≤ 0 , ỹ∗i = wTi x∗ + bi ≤ 0 Consider the set of possible x∗ ’ s that would satisfy all these constraints . This set would be the intersection of the following sets : { x∗ : wTi x∗ + bi = ỹti } for i s.t . ỹti > 0 { x∗ : wTi x∗ + bi ≤ 0 } for i s.t . ỹti ≤ 0 Geometrically , each set that corresponds to an equality constraint represents a hyperplane in d− 1- dimensional subspace . Each set that corresponds to an inequality constraint represents a half-space . The intersection of the equality constraints is a subspace of at least d− np dimensions , where np is the number of components in ỹt that are positive , assuming d > np . We consider the projections of the half-spaces onto this subspace , which are half-spaces in the subspace , and think about the intersection of all these half-spaces . In general , the intersection of finitely many half-spaces is a convex ( but possibly unbounded ) polytope . So , the intersection of all the sets listed above is a convex polytope in at least a d− np-dimensional subspace . Any point in this polytope corresponds to a colliding example . Since there could be infinitely many points in a polytope , there could be infinitely colliding examples . If the polytope is bounded , we 1It should noted that the term “ feature collision ” was also used by ( Shafahi et al. , 2018 ) to refer to a different , but related , concept . In their context , feature collisions refer to input examples that are both close to a particular data example in input space and close to a different data example in feature space . In our context , input examples do not need to be close to any particular data example in input space and are only required to be unambiguously different from the target class to the human eye . would be able to characterize all such examples if we know all the corners of the polytope , in which case the polytope would simply be the convex hull of all the corners . Then , any colliding example can be written as a convex combination of the corners , and we can generate a new colliding example by taking an arbitrary convex combination of the corners . ( If the polytope is unbounded , we can still generate new colliding examples this way , but there could be colliding examples that are not convex combinations of the corners . ) Therefore , to find the set of colliding examples , we would need to find the corners of the polytope . To this end , we can move the dividing hyperplane of a half-space towards the feasible direction , which mathematically corresponds to decreasing the RHS of the corresponding inequality constraint . This is equivalent to picking a unit in the current layer and trying to make it as negative as possible . This process is illustrated in Figure 1 in 2D . To find a different corner , we simply pick a different constraint to optimize . In most of the literature on adversarial examples ( Szegedy et al. , 2013 ; Shafahi et al. , 2018 ) , the method of gradient descent , shown in Algorithm 1 , is used to find these examples . Let f ( x ) denote the neural network feature representation of input x , which is the output feature of a certain layer . Given a target t , generated feature collision sample p could be as simple as p = arg min x L ( x ) = arg min x ||f ( x ) − f ( t ) ||2 +R ( x ) ( 1 ) HereR ( · ) is an application-specific penalization term that encourages the optimizer to find an example with some desirable property . Note that even though this method could find one such example , the solution would just be some arbitrary point in the polytope which is not helpful for exploring the full space of feature collisions . Algorithm 1 Gradient descent method . Given a starting input x and learning rate t repeat 1 . ∆x : = −∇L ( x ) . 2 . Update . x : = x + t∆x until stopping criterion is satisfied . Note that f ( x ) is a normal feature output of a layer with ReLU activation , and we further define f0 ( x ) to be the feature output before ReLU activation . To find the corners of polytope using the process in Figure 1 and yet still make use of the gradient-based method , we define two more loss term Lpos and Lneg where Lpos ( x ) = ||f0 ( x ) psgn ( f0 ( t ) ) − f ( t ) ||2 Lineg ( x ) = − ( f0 ( x ) nsgn ( f0 ( t ) ) ) i + ∑ j 6=i | ( f0 ( x ) nsgn ( f0 ( t ) ) ) j | j = 1 , 2 , · · · ( 2 ) Here psgn ( · ) and nsgn ( · ) are two element-wise sign functions psgn ( x ) = max ( sgn ( x ) , 0 ) = { 0 x < 0 1 x > = 0 nsgn ( x ) = min ( sgn ( x ) , 0 ) = { −1 x < 0 0 x > = 0 And we could update Equation 1 to be pi = argmin x αLpos ( x ) + βLineg ( x ) +R ( x ) ( 3 ) where i is an arbitrary index we choose to find one of the corners of the polytope . Basically Lpos aims to keep the feature output after ReLU activation to be close to the target , as we only consider the units which are non-negative in the target feature . And for the negative units , if we assume that the feature output is a 1-D array , the first term of Lineg ( x ) would keep the selected unit to go smaller and the second term simply keeps other units close to zero . Since all influenced units are negative , and hence should be “ wiped out ” by ReLU layer , keeping them smaller or close to zero won ’ t affect post-activation features after ReLU but would explore different corners of the polytope when choosing different units . For feature output more than 1 dimension , we could still do the same or choose a subset of the output units instead of just one .

Summary: This paper studies a non-local form of adversarial perturbation, which, to my limited knowledge is new. The form of the perturbation is specific to ReLU activations, but it may be a large set. The authors also devise an algorithm to generate natural-looking perturbations in this set. Instead of updating a seed example through gradient descent, they propose to generate perturbations by combinations of image patches from multiple seed images. The weights of the combination are optimized by a gradient descent like algorithm in a similar manner as standard gradient-based approaches to generating adversarial examples. This produces perturbations that look like ```in-paintings'' or transplants of one seed image onto another. Here are a few comments: