Effective Path: Know the Unknowns of Neural Network

1 INTRODUCTION . Deep learning ( DL ) has revolutionized the key application domains such computer vision ( Krizhevsky et al. , 2012a ) , natural-language processing ( Sutskever et al. , 2014 ) , and automatic speech recognition ( Abdel-Hamid et al. , 2014 ) . DL models have outperformed traditional machine learning approaches and even outperformed human beings . Although most of the current research efforts have been in improving the efficiency and accuracy of DL models , interpretability has recently become an increasingly important topic . This is because many DL-enabled or DL-based systems are mission-critical systems , such as ADAS ( Dynov , 2016 ) and online banking systems ( Fiore et al. , 2017 ) . However , to date , there is no theoretical understanding of how DL models work , which is a significant roadblock in pushing DL into mission-critical systems . Owing to the lack of interpretability , DL models usually do not have a clear decision boundary and are vulnerable to the input perturbation . Researches have recently been proposed ( Pei et al. , 2017 ; Moosavi-Dezfooli et al. , 2015 ; Kurakin et al. , 2016a ; Carlini & Wagner , 2016 ) , which can all successfully find a small perturbation on the input image to fool the DNN based classifier . There is also prior work that demonstrates the physical attack feasibility by putting a printed image in front of a stop sign to mislead a real DNN based traffic sign detector ( Eykholt et al. , 2018 ) . Last but not the least , a DNN model often fails for inputs that are dramatically different from the training samples . For example , the classification model used in Tesla ’ s autopilot system that incorrectly classified a white truck to cloud ( Golson , 2016 ) and caused the crash accident . To address the vulnerability challenge in DL models , this work proposes the effective path as a new approach to explore the internal organization of neural networks . The effective path for an image is a critical set of synapses and neurons that together lead to the final predicted class . The concept is similar to the execution path of a control-flow based program ( Ball & Larus , 1996 ) . We propose an activation based back-propagation algorithm to extract the image ’ s effective path , which preserves the critical information in the neural network and allows us to analyze the inner structures of DNNs . The derived per-image effective path has the direct aggregation capability . For example , we get perclass effective path by aggregating the effective path from all training images in the same class . We can then decompose the entire DNN into multiple components , each pertaining to an inference class . We perform similarity analysis and find the phenomenon called path specialization that different classes activate distinctive portions of the neural network in the inference task . On the basis of the observation , we analyze the path similarity between normal and adversarial images , we uncover that when an adversarial image successfully alters the prediction result by small perturbation , the network activates a significantly distinctive set of effective path compared to the training samples , which lays the foundation for defending the DNN using the effective path . For adversarial image detection , we derive a metric that is a simple linear combination of an image ’ s per-layer effective path similarity . We perform a comprehensive evaluation on various DNN models and datasets , and show the metric can accurately detect adversarial images from six representative attack methods , for which we achieve an area under the curve value of 0.95 on MNIST and 0.89 on ImageNet . Our linear model based method offers high interpretability and has high transferability to unseen attacks . For example , we train the detection model on a single attack method , which almost provides the same detection accuracy when training the model on all attack methods . In the end , we show that effective path can not only be used for adversarial image detection but also for explaining the impact of the training process and network structure on the DNN ’ s inference capability . 2 EFFECTIVE PATH . This work introduces effective path to facilitate the dissection of the black-boxed neural network . We borrow the concept from the path profiling of program analysis ( Ball & Larus , 1996 ) : a program is represented in the form of control flow graph , where a node is a basic block and an edge is the control flow between basic blocks . Compilers use path profiling to identify the sequences of frequently executed basic blocks , i.e. , execution paths in the program . A program ’ s path profile provides useful insights on its execution and facilitates the understanding of the program . Similarly , we treat a neural network as a dataflow graph where a node is a neuron and an edge is a synapse ( weight ) between two neurons . In the high level , the effective path represents the large dataflow inside a DNN . Single Image Extraction We first explain how to extract the effective path for a single image , denoted as P = ( N , S , W ) , which represents the collection of critical neurons N , synapses S and weights W . It can be further brokendown to the per-layer form N = ( N 1 , . . . , NL ) , S = ( S1 , . . . , SL ) , W = ( W1 , . . . , WL ) , where N l represents the important output neurons of layer l , while Sl andW l represent important synapses and weights of layer l. The extraction process starts at the last layer L and moves backward to the first layer . In the last layer L , only the neuron corresponding to the predicted class nLp is active and thus is included in the effective path , i.e. , NL = { nLp } . The important weights form the minimum set of weights that can contribute more than θ ratio of the output neuron nLp . Equation 1 formalizes the process , where K̃ L p is a selected set of weight indices with neuron nLp as the output , w L k , p is the weight value , and n L−1 k is the corresponding input neuron value ( also the output neuron of layer l − 1 ) . To find the minimum K̃Lp , we can rank the weight and input neuron pairs by the value of their product and choose the minimum number of pairs that contribute to more than threshold θ × nLp . min K̃Lp |K̃Lp | , s.t . ∑ k∈K̃Lp nL−1k × w L k , p ≥ θ × nLp ( 1 ) WL = { wLk , p|k ∈ K̃Lp } ( 2 ) NL−1 = { nL−1k |k ∈ K̃ L p } ( 3 ) After deriving the weight indices set K̃Lp , we can get theWL set using Equation 2 . Since the last layer is the fully connected layer and there is a one-to-one mapping between weight and synapses , SL can also be derived . Meanwhile , since the output neurons of layer L− 1 are the input neurons of layer L , it is straightforward to derive NL−1 in Equation 3 . We then can repeat the process in Equation 1 for every active neuron in NL−1 : each active neuron will result in a set of weights and their union form theWL−1 . The process repeats backward until the first layer , and we get the whole neuron set N , synapse set S , and weight setW for the single input image . Note that the above process applies to the fully connected layer . We transform a convolutional layer into a fully connected layer to unify the process ( Fig . 1b ) . There are two caveats on handling the convolutional layer . First , solving of Equation 1 does not require the ranking of all input neurons but only the neurons in the receptive field of the output neuron . Second , there is no one-to-one mapping between synapse and weight because of weight sharing . As a result , multiple synapses can have the same active weights in the effective path . Aggregation of Training Set Images The direct aggregation capability of different images ’ effective path is a salient feature of our work compared to the prior work critical routing path ( Wang et al. , 2018 ) . Our extraction process does not alter the values of the trained neural network while the prior work requires retraining for every image . By aggregating effective paths from an image group , we can obtain a larger effective path that provides higher level perspective of the whole group . Aggregating the effective path of two images P ( i ) and P ( j ) is essentially taking the union of N , S andW on each layer , represented by P ( i ) ∪ P ( j ) = ( N ( i ) ∪ N ( j ) , S ( i ) ∪ S ( j ) , W ( i ) ∪W ( j ) ) , where N ( i ) ∪N ( j ) = ( N 1 ( i ) ∪N 1 ( j ) , . . . , NL ( i ) ∪NL ( j ) ) ( N andW are similar ) . In this work , we use two types of aggregated effective path for the neural network interpretation and defense . For the class-level perspective , we aggregate images from the class c , denoted by X̃c , to get the per-class effective path P̃c = ⋃ x∈X̃c P ( x ) ; for the network-level perspective , we aggregate images from the whole training set X̃ to get the overall effective path P̃ = ⋃ x∈X̃ P ( x ) . Effective Path Density The derived overall effective path is highly sparse compared to the original network . We define the weight ( synapse ) density of the effective path DW ( DS ) as the ratio of weights ( synapses ) in the effective path over the entire weights ( synapses ) . They can be calculated in Equation 4 , where Wl and W̃ l ( Sl and S̃l ) is the layer l ’ s entire weight ( synapse ) set and weight ( synapse ) set in overall effective path , respectively . DW = ∑L l=1 ∣∣∣W̃ l∣∣∣∑L l=1 |Wl| , DS = ∑L l=1 ∣∣∣S̃l∣∣∣∑L l=1 |Sl| ( 4 ) We extracted the overall effective path for popular DNN models including LeNet-5 ( LeCun et al. , 1989 ) , AlexNet ( Krizhevsky et al. , 2012b ) , ResNet-50 ( He et al. , 2015 ) , Inception-v4 ( Szegedy et al. , 2016 ) , and VGG-16 ( Simonyan & Zisserman , 2014 ) . Their density is 13.8 % , 20.5 % , 22.2 % , 41.7 % , 17.2 % , respectively . Prior work has shown that about 80 % of weights can be removed without affecting the prediction accuracy ( Han et al. , 2016 ) . Our result conforms with the number , which means that extracted path preserves the information in the neural network . The advantage of the effective path over the DNN compression is the ability to dissect the network to different components corresponding to different inference classes . As defined above , we can aggregate paths from images of the same class to get the per-class effective path . The per-class path dissects the network to different components and can be used to understand why the neural network can distinguish different classes and study the impact of changing the network structure .

Summary: This paper proposes a measure (“effective path”) of which units and weights were most important for classification of a particular input or input class. Using the effective path, the authors analyze the overlap between paths across classes for CNNs and between adversarially modified and unmodified images. Finally, the paper proposes an adversarial defense method based on effective path which detects adversarially manipulated images with high accuracy and generality to a variety of settings.