Title: EFFECTIVE PATH: KNOW THE UNKNOWNS OF NEURAL NETWORK

Abstract: Despite their enormous success, there is still no solid understanding of deep neural network's working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs' internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.

1. INTRODUCTION:

Deep learning (DL) has revolutionized the key application domains such computer vision (Krizhevsky et al., 2012a) , natural-language processing (Sutskever et al., 2014) , and automatic speech recognition (Abdel-Hamid et al., 2014) . DL models have outperformed traditional machine learning approaches and even outperformed human beings. Although most of the current research efforts have been in improving the efficiency and accuracy of DL models, interpretability has recently become an increasingly important topic. This is because many DL-enabled or DL-based systems are mission-critical systems, such as ADAS (Dynov, 2016) and online banking systems (Fiore et al., 2017) . However, to date, there is no theoretical understanding of how DL models work, which is a significant roadblock in pushing DL into mission-critical systems.

Owing to the lack of interpretability, DL models usually do not have a clear decision boundary and are vulnerable to the input perturbation. Researches have recently been proposed (Pei et al., 2017; Moosavi-Dezfooli et al., 2015; Kurakin et al., 2016a; Carlini & Wagner, 2016) , which can all successfully find a small perturbation on the input image to fool the DNN based classifier. There is also prior work that demonstrates the physical attack feasibility by putting a printed image in front of a stop sign to mislead a real DNN based traffic sign detector (Eykholt et al., 2018) . Last but not the least, a DNN model often fails for inputs that are dramatically different from the training samples. For example, the classification model used in Tesla's autopilot system that incorrectly classified a white truck to cloud (Golson, 2016) and caused the crash accident.

To address the vulnerability challenge in DL models, this work proposes the effective path as a new approach to explore the internal organization of neural networks. The effective path for an image is a critical set of synapses and neurons that together lead to the final predicted class. The concept is similar to the execution path of a control-flow based program (Ball & Larus, 1996) . We propose an activation based back-propagation algorithm to extract the image's effective path, which preserves the critical information in the neural network and allows us to analyze the inner structures of DNNs.

The derived per-image effective path has the direct aggregation capability. For example, we get perclass effective path by aggregating the effective path from all training images in the same class. We can then decompose the entire DNN into multiple components, each pertaining to an inference class. We perform similarity analysis and find the phenomenon called path specialization that different classes activate distinctive portions of the neural network in the inference task. On the basis of the observation, we analyze the path similarity between normal and adversarial images, we uncover that when an adversarial image successfully alters the prediction result by small perturbation, the network For adversarial image detection, we derive a metric that is a simple linear combination of an image's per-layer effective path similarity. We perform a comprehensive evaluation on various DNN models and datasets, and show the metric can accurately detect adversarial images from six representative attack methods, for which we achieve an area under the curve value of 0.95 on MNIST and 0.89 on ImageNet. Our linear model based method offers high interpretability and has high transferability to unseen attacks. For example, we train the detection model on a single attack method, which almost provides the same detection accuracy when training the model on all attack methods. In the end, we show that effective path can not only be used for adversarial image detection but also for explaining the impact of the training process and network structure on the DNN's inference capability.

2. EFFECTIVE PATH:

This work introduces effective path to facilitate the dissection of the black-boxed neural network. We borrow the concept from the path profiling of program analysis (Ball & Larus, 1996) : a program is represented in the form of control flow graph, where a node is a basic block and an edge is the control flow between basic blocks. Compilers use path profiling to identify the sequences of frequently executed basic blocks, i.e., execution paths in the program. A program's path profile provides useful insights on its execution and facilitates the understanding of the program. Similarly, we treat a neural network as a dataflow graph where a node is a neuron and an edge is a synapse (weight) between two neurons. In the high level, the effective path represents the large dataflow inside a DNN.

Single Image Extraction:

We first explain how to extract the effective path for a single image, denoted as P = (N , S, W), which represents the collection of critical neurons N , synapses S and weights W. It can be further brokendown to the per-layer form N = (N 1 , . . . , N L ), S = (S 1 , . . . , S L ), W = (W 1 , . . . , W L ), where N l represents the important output neurons of layer l, while S l and W l represent important synapses and weights of layer l.

The extraction process starts at the last layer L and moves backward to the first layer. In the last layer L, only the neuron corresponding to the predicted class n L p is active and thus is included in the effective path, i.e., N L = {n L p }. The important weights form the minimum set of weights that can contribute more than θ ratio of the output neuron n L p . Equation 1 formalizes the process, where KL p is a selected set of weight indices with neuron n L p as the output, w L k,p is the weight value, and n L-1 k is the corresponding input neuron value (also the output neuron of layer l -1). To find the minimum KL p , we can rank the weight and input neuron pairs by the value of their product and choose the minimum number of pairs that contribute to more than threshold θ × n L p .

min

EQUATION

After deriving the weight indices set KL p , we can get the W L set using Equation 2. Since the last layer is the fully connected layer and there is a one-to-one mapping between weight and synapses, S L can also be derived. Meanwhile, since the output neurons of layer L -1 are the input neurons of layer L, it is straightforward to derive N L-1 in Equation 3. We then can repeat the process in Equation 1 for every active neuron in N L-1 : each active neuron will result in a set of weights and their union form the W L-1 . The process repeats backward until the first layer, and we get the whole neuron set N , synapse set S, and weight set W for the single input image. Note that the above process applies to the fully connected layer. We transform a convolutional layer into a fully connected layer to unify the process (Fig. 1b ). There are two caveats on handling the convolutional layer. First, solving of Equation 1 does not require the ranking of all input neurons but only the neurons in the receptive field of the output neuron. Second, there is no one-to-one mapping between synapse and weight because of weight sharing. As a result, multiple synapses can have the same active weights in the effective path.

Aggregation of Training Set Images:

The direct aggregation capability of different images' effective path is a salient feature of our work compared to the prior work critical routing path (Wang et al., 2018) . Our extraction process does not alter the values of the trained neural network while the prior work requires retraining for every image. By aggregating effective paths from an image group, we can obtain a larger effective path that provides higher level perspective of the whole group. Aggregating the effective path of two images P(i) and P(j) is essentially taking the union of N , S and W on each layer, represented by P(i) ∪ P(j) = (N (i) ∪ N (j), S(i) ∪ S(j), W(i) ∪ W(j)) , where N (i) ∪ N (j) = (N 1 (i) ∪ N 1 (j), . . . , N L (i) ∪ N L (j)) (N and W are similar).

In this work, we use two types of aggregated effective path for the neural network interpretation and defense. For the class-level perspective, we aggregate images from the class c, denoted by Xc , to get the per-class effective path Pc = x∈ Xc P(x); for the network-level perspective, we aggregate images from the whole training set X to get the overall effective path P = x∈ X P(x).

Effective Path Density The derived overall effective path is highly sparse compared to the original network. We define the weight (synapse) density of the effective path D W (D S ) as the ratio of weights (synapses) in the effective path over the entire weights (synapses). They can be calculated in Equation 4, where W l and Wl (S l and Sl ) is the layer l's entire weight (synapse) set and weight (synapse) set in overall effective path, respectively.

D W = L l=1 Wl L l=1 |W l | , D S = L l=1 Sl L l=1 |S l | (4)

We extracted the overall effective path for popular DNN models including LeNet-5 (LeCun et al., 1989) , AlexNet (Krizhevsky et al., 2012b) , ResNet-50 (He et al., 2015) , Inception-v4 (Szegedy et al., 2016) , and VGG-16 (Simonyan & Zisserman, 2014) . Their density is 13.8%, 20.5%, 22.2%, 41.7%, 17.2%, respectively. Prior work has shown that about 80% of weights can be removed without affecting the prediction accuracy (Han et al., 2016) . Our result conforms with the number, which means that extracted path preserves the information in the neural network.

The advantage of the effective path over the DNN compression is the ability to dissect the network to different components corresponding to different inference classes. As defined above, we can aggregate paths from images of the same class to get the per-class effective path. The per-class path dissects the network to different components and can be used to understand why the neural network can distinguish different classes and study the impact of changing the network structure.

3. PATH SIMILARITY ANALYSIS FOR NEURAL NETWORKS:

After deriving the per-class effective path, we perform the path similarity analysis among different classes, which leads to a finding called path specialization. Different classes activate not only sparse but also a distinctive set of neurons and synapses for the inference task. This finding further motivates us to explore the distribution of effective path for adversarial examples. Specifically, we study the adversarial example in the computer vision domain, where it is in the form of small perturbation to the input image. The perturbation is small and imperceptible by human beings but can lead to an Digit 1 0.33 0.48 0.47 0.4 0.51 0.48 0.43 0.5 0.47 0.33 1 0.38 0.36 0.36 0.33 0.34 0.35 0.39 0.33 0.48 0.38 1 0.54 0.5 0.5 0.49 0.48 0.55 0.47 0.47 0.36 0.54 1 0.42 0.58 0.42 0.47 0.54 0.49 0.4 0.36 0.5 0.42 1 0.46 0.49 0.49 0.51 0.53 0.51 0.33 0.5 0.58 0.46 1 0.5 0.46 0.6 0.52 0.48 0.34 0.49 0.42 0.49 0.5 1 0.38 0.5 0.41 0.43 0.35 0.48 0.47 0.49 0.46 0.38 1 0.49 0.55 0. 5 0.39 0.55 0.54 0.51 0.6 0.5 0.49 1 0.53  0.47 0.33 0.47 0.49 0.53 0.52 0.41 0.55 0.53 1   0 incorrect prediction of the neural network. Our critical insight is that adversarial images activate distinctive effective path to fool the DNN, which lays the foundation for our novel effective path based defense mechanism in Sec. 4.

3.1. PATH SPECIALIZATION:

We first study the similarity of per-class effective paths. The similarity between class c 1 and c 2 is calculated by the Jaccard coefficient of their synapse set, i.e.

J c1,c2 = J( Sc1 , Sc2 ) = | Sc 1 Sc 2 | | Sc 1 Sc 2 | .

Fig. 2 shows the class-wise path similarity in LeNet, unveiling the existence of path specialization: the averaged similarity between two classes is low (around 0.5). On average, two classes activate about 50% common paths, as well as 50% distinctive paths. Moreover, the degree of path specialization reflects the similarity between the two classes. For example, in Fig. 2 , digit '1' has the highest degree of specialization (i.e., lowest path similarity against other digits) because of its unique shape. Digit '5' and '8' have the highest path similarity of 0.6 owing to their similar shape.

We observe the existence of the path specialization in other datasets and networks. Fig. 3 shows the path density growth when merging per-class (ImageNet) paths for ResNet-50. The growth of both weight and synapse follow the same trend (weight density is greater owing to the weight sharing).

The density increases rapidly initially, indicating the high degree of path specialization. After 50 classes, the density still increases but at a much slower pace. This matches the class hierarchy in the ImageNet dataset, which has around 100 basic categories: different categories have a larger degree of path specialization while classes in the same categories have a smaller specialization degree.

To summarize, there exists path specialization phenomenon in trained DNNs, which we infer, offers the neural network the ability to distinguish different classes. In the next, we study how neural networks distinguish adversarial examples from normal ones with example-class path similarity.

3.2. SIMILARITY ANALYSIS FOR ADVERSARIAL IMAGES:

On the basis of path specialization, we study the similarity of the effective path between normal images and adversarial images. For generality, we evaluate adversarial images generated by FGSM, DeepFool, BIM, JSMA, and C&W l 2 attacks, whose detailed introduction is explained in Sec. 4.1.

We introduce another similarity metric called image-class path similarity, which indicates how many synapses in the image's effective path come from the predicted class's effective path. It can be calculated as

J P = J(S, S ∩ Sp ) = |S ∩ Sp |/|S|

, where p is the image's predicted class, S is the synapse set of image effective path, and Sp is the synapse set of class p's effective path. Because the per-class effective path is far larger than the image's effective path, their Jaccard coefficient will be nearly zero. As such, the image-class path similarity is essentially the Jaccard coefficient between the image's effective path and the intersection set of effective path between the image and predicted class. 5a shows the per-layer similarity distribution for normal images from test set and adversarial images on AlexNet with ImageNet as training set. We find that normal images demonstrate a higher similarity degree for adversarial images. We further calculate the similarity delta, which equals the similarity value of a normal image minus the similarity value of its corresponding adversarial image. As Fig. 5b shows, all adversarial attacks cause a similarity decrease, especially in the middle layers.

Recall that we extract the effective path starting from the predicted class, i.e. rank-1 class. We also study the characteristics of effective path which starts from the rank-2 class. Fig. 5c and Fig. 5d plot the rank-2 effective path similarity for normal and adversarial images. Different from the rank-1 effective path, we find that adversarial images demonstrate a higher similarity degree for normal images. The reason is that rank-2 class in the adversarial images is typically the rank-1 class of its normal image, which still demonstrates a greater similarity than the normal images' rank-2 class.

In summary, extending class-wise path similarity to the image-class case opens the door to discover effective path's ability to detect adversarial images. The reason is that mainstream adversarial attacks modify the normally inactive path to fool the DNN. Following section details our defense mechanism.

4. DEFENDING NEURAL NETWORKS AGAINST ADVERSARY IMAGES:

We take advantage of the observed deviation of adversarial images' similarity distribution from normal case to propose jointed similarity as a concise detection metric. This metric is generally applicable and achieves great defense performance on a wide range of different attacks, datasets, and models. Our defense mechanism is also orthogonal to other defense methods as it does no modify inputs, network structures, and training process.

Defense Metric Based on the per-layer similarity analysis of adversarial images, we propose jointed similarity as the defense metric. It can be calculated as JP = L l=1 ω l J l P -L l=1 ω l J l P , where J l P and J l P are respectively rank-1 and rank-2 similarity for layer l, ω l and ω l are their coefficients that satisfy ω l ≥ 0, ω l ≥ 0. The joint similarity reflects the low rank-1 similarity degree and high rank-2 similarity degree of adversarial images. An image is detected as adversarial image if its joint similarity is less than a threshold. The joint similarity avoids overfitting and offers strong interpretation ability as it is a low-dimension linear model. 

4.1. EVALUATION:

To obtain joint similarity's coefficients, we train it with dataset containing both normal and adversarial examples. We use a set of representative to evaluate its detection accuracy and generality.

Attack Approaches We use 6 different attacks for evaluation as shown in Fig. 6 . For each attack, we always choose the canonical implementation. We use Foolbox (Rauber et al., 2017) coefficients is performed by SGD running 10000 epochs, with elastic-net regularization (l 1 ratio is 0.5) for sparsity. Fig. 7a and Fig. 7b show the coefficients's distribution: large coefficients locate near the FC layer, which match the results from our previous analysis results.

Against Non-targeted Attacks Non-targeted attacks are free to use any class as the adversarial image's inference result. We evaluate non-targeted attacks with three different norms: FGSM and BIM with l ∞ norm, DeepFool and C&W l2(CWL2) attack with l 2 norm, and JSMA with l 0 norm. For LeNet, we achieves an area under the curve (AUC) value up to 0.95 in Fig. 7c . Even the lowest AUC value is 0.92, because of significant path similarity distinction between adversarial and normal images of MNIST. On ImageNet, we achieve AUC of 0.85~0.86 for AlexNet and AUC of 0.88~0.89 for ResNet-50, which has more layers to provide richer information for detection, leading to the better accuracy. The BIM has a low AUC value of 0.82. The reason is that BIM iteratively modifies all pixels (Fig. 6 ), which makes its rank-2 effective path behave slightly different from other attacks.

Against Targeted Attack Targeted attacks are designed to mislead the prediction to a specific target class. Fig. 7f shows the result of evaluating targeted C&W l 2 attack for AlexNet. We achieve AUC of 0.94, which is better than the non-targeted version. It is reasonable since targeted attack's stricter constraint for target class requires larger perturbation, which eases our detection.

Against Universal Perturbation Attack Universal perturbation attacks generate perturbations that fool models on a large range of examples. Adversarial Patch (Brown et al., 2017) is an attack that generates universal perturbations in form of image patches, which is robust against patch transformations such as translations, rotations or scaling. The result of adversarial patches in Fig. 7g indicates that the detection becomes more accurate when the patch becomes larger. Our method can reach AUC of 0.9 when patch scale relative to image size rises to 0.5.

Against Unrecognizable Examples:

Adversarial examples are usually human-recognizable, however, unrecognizable images can also fool neural networks (Nguyen et al., 2015) . We find that effective path can also be used to detect unrecognizable examples by evaluated on LeNet and AlexNet.

For LeNet, our detector can recognize 93.85% randomly generated images. For AlexNet, our method achieves AUC of 0.91 as shown in Fig. 7h . In this sense, effective path offers the DNNs the ability to identify its recognizable inputs' distribution.

Parameter Sensitivity The single tunable parameter of effective path extraction is θ. We test C&W l 2 attack under θs varying from 0.1 to 1.0 in Fig. 7i . The detection performance remains almost unchanged when θ is in range of 0.5 and 1.0, and decreases from θ = 0.3. It indicates that our method is not sensitive to θ. Fig. 7j shows that the effective path size under θ = 0.3 decrease by one order of magnitude compared with θ = 1.0, with slightly lower detection accuracy. We choose θ = 0.5 as default value to save storage space and improve extraction performance without accuracy loss.

Transferability Transferability measures a defense's ability to withstand unknown attacks. To figure out joint similarity's transferability, we train its coefficients for targeted and non-targeted C&W l 2 attacks and then test on other attacks. For non-targeted attacks, AUCs of FGSM/BIM/DeepFool/JSMA separately increase 0.01~0.02, which indicates that our method is perfectly transferable to the same type of attacks. When transfer to other types, AUC decreases 0.06 for adversarial patch and 0.03 for unrecognizable examples.

To summarize, our effective path based detection method achieves high detection accuracy without requiring attack-specific knowledge. Meanwhile, it also offers strong transferability to unseen attacks.

4.2. COMPARISON WITH OTHER DEFENSES:

To compare our defense method with prior work, we first categorize various defenses methods to the five types listed in Tbl. 1. Since almost all the compared work reported a similar detection accuracy (AUC value 0.9 0.95), we focus the comparison on the comprehensiveness, generality, transferability, and scale of their evaluation. The "-" in the table indicates that there are not enough details or experimental results to deduce an appropriate conclusion.

Detector Our work fits in the detector category, which does not require any modification to inputs, models, or training process. Madry et al. (2017) , considered as the only effective defense among white-box-secure defenses at ICLR 2018 (Athalye et al., 2018) , is found overfitting on the trained l ∞ metric (Sharma & Chen, 2017) .

Input Transformation Input transformation requires modification of input examples. Many image transformations like rescaling, bit-depth reduction and compression can disturb attacks and increase the perturbation lower bound, with the sacrifice of classification accuracy. This kind of defense method works less well for patch-based attacks and does not provide the ability to filter unrecognizable examples.

Randomization Randomization defenses apply random modifications to model weights. They can increase required distortion by forcing attacks to generate transferable adversarial examples over a series of possible modified models. However, they also stochastically alter the prediction results, leading to the overhead of more forward passes or retraining steps.

Generative Model Generative model based defenses change the classification model. They project the inputs onto the manifold before classification. Schott et al. (2018) propose a classification model that shows good generality and transferability on MNIST, but its performance on large dataset like ImageNet is still obscure. GAN-based defenses are also hard to apply in ImageNet scale due to its computational cost.

5. CONCLUSION AND FUTURE WORK:

This work has shown the feasibility of decomposing a deep neural network into different functional blocks corresponding to different inference classes, similar to the various functional units of the brain cortex. The functional block is called effective path and constructed through back-propagation using training images. Through analysis, we find that adversarial images can activate functional blocks different from normal images to fool the DNN's prediction results because all the blocks are connected. We propose a defense method that only uses the information from the training set and the image itself, without requiring any knowledge of a specific attack. The defense method achieves high accuracy and broad coverage of mainstream attacks.

Besides adversarial defense, the effective path can also be used to understand the DNN's working mechanism. In the appendix, we report our preliminary result on how the training process and different DNN topology affects the effective path density and similarity. We believe that the functionality based decomposition is a promising direction for understanding DNNs. Pooling Layer Pooling layers can be treated as the special case of convolutional layers during extracting. For average pooling layer, we treat it as a convolutional layer with all weights equal to 1; for max pooling layer, we treat it as a convolutional layer that always picks rank-1 weight and input neuron pair when finding the minimum Kl p .

B FURTHER STUDY OF NEURAL NETWORK INTERPRETABILITY:

We further study how the training process and different network structure impacts the path specialization. In the next, we study how the training process and different network structures impact the path specialization.

B.1 TRAINING PROCESS:

We study how the training process transforms a randomized network to the final state from the perspective of the effective path. Specifically, we extract the effective path for each class at different training stages. Through the analysis, we find that the training process contains three distinctive phases with different path's density and similarity trend, which share similar insights from the previous work using information bottleneck theory to explain training process (Wolchover, 2017) .

Fig. 8 shows training process for ResNet. We choose different stages in training and show the class-wise path similarity in the form of box-plot, on top of which we also overlay the path density and prediction accuracy. In the first phase, the density of synapses and weights in the effective paths stays the same while their similarity increases. In the beginning, the network is in a randomized state and simply tries to memorize the input data. In the second phase, the density of both synapses and weights decrease rapidly. The similarity of the synapses stays relatively the same while the similarity of weights decreases. In this phase, the network mainly performs compression, and the path specialization mainly manifests in the form of weights. In other words, the network tries to use class-specific features extracted by different convolutional filters to increase the specialization degree.

In the third phase, the synapse density stops to decrease but weight density starts to increase. Meanwhile, the weight similarity continues to decrease. In this phase, the network compression stops and mainly relies on path specialization (via weight) to increase the prediction accuracy. The path specialization even causes the weight density increases a bit.

In summary, we find that the training process contains mainly three phases, first two of which conforms to the memorization and compression phase identified by the prior work (Wolchover, 2017) . The second phase performs compression (less density) and path specialization (less similarity), while the third phase mainly includes the path specialization. After these phases, the network is transformed into a state with sparse and distinctive paths with great inference capability.

B.2 NETWORK STRUCTURE:

After establishing the effective path as a great indicator of the neural network's inference performance, we study that how the network structure affects the effective path characteristics. Fig. 9 shows the per-layer path density in LeNet. We observe the first convolutional layer has a much higher density compared to the following layers, which matches with the established knowledge that the shallow layers in a CNN extract high-level features that are shared by different classes.

CNN designers have found using a deeper network can increase the prediction accuracy to a certain degree. However, the accuracy stops to increase after a certain number of layers owing to the vanishing gradient in the training process. As such, the ResNet structure with skip connection was proposed to overcome this difficulty. Fig. 10 shows the per-layer path density for ResNet-50. Not only the first two layers still have higher density, but also layers before a skip connection also have high density. This suggests that skip connection helps not only the gradient propagation but also the effective paths formation. In the end, ResNet is able to converge and achieve great prediction performance.

C SUPPLEMENTAL ANALYSIS:

C.1 PER-LAYER SIMILARITY ANALYSIS FOR RESNET-50 ON IMAGENET Per-layer similarity distribution for ResNet-50 on ImageNet is shown in Fig. 11 . Similar with AlexNet, adversarial images lead to lower rank-1 similarity and higher rank-2 similarity compared with normal images. Furthermore, corresponding to AlexNet's FC layers, the largest similarity delta is also located in last several layers. 

:



Figures and Tables:

Figure 1: Examples of extracting effective paths.

Figure 2: Class-wise path similarity in LeNet.

Fig.4shows the distribution of image-class path similarity for both normal images and a rich set of adversarial images in MNIST for LeNet. The similarity values for normal images are almost all 1, and note that they are not used in the training and per-class path extraction. In contrast, the similarity values for adversarial images are mostly smaller than 1, indicating effective path as a great metric to distinguish between normal and adversarial images.

Figure 5: Distribution of per-layer similarity for AlexNet on ImageNet. Each line plot represents the mean of each kind of adversarial examples' similarity, with the same-color band around to show the standard deviation. The dashed line split convolutional layers and FC layers. (a): Rank-1 similarity. (b): Rank-1 similarity delta. (c): Rank-2 similarity. (d): Rank-2 similarity delta. Normal FGSM BIM DeepFool JSMA CWL2 Patch

Figure 7: Detection results for LeNet, AlexNet, and ResNet-50 with joint similarity.

Figure 8: Effective path's density and class-wise path similarity in the training process.

Figure 9: Per-layer density of effective paths in LeNet.

Figure 12: ROC for AlexNet on ImageNet with weight-based joint similarity.

Notice that we only generate adversarial examples for the first test image in each class of ImageNet due to the significant computational cost. The training of joint similarity's

Table content: <table><tr><td>implementations</td></tr><tr><td>and its default parameters in version 1.3.2 for Fast Gradient Sign Method(FGSM) (Goodfellow</td></tr><tr><td>et al., 2014), Basic Iterative Method(BIM) (Kurakin et al., 2016a), DeepFool (Moosavi-Dezfooli</td></tr><tr><td>et al., 2015), Jacobian-based Saliency Map Attack(JSMA) (Papernot et al., 2015). For Carlini</td></tr><tr><td>and Wagner(C&amp;W) attacks (Carlini &amp; Wagner, 2016), we use the open-source code released by</td></tr><tr><td>the paper authors. We use adversarial patch (Brown et al., 2017) implementation provided in</td></tr><tr><td>CleverHans (Papernot et al., 2018) and extend it to support AlexNet without modification to its</td></tr><tr><td>settings.</td></tr><tr><td>Coefficients Training We use LeNet-5 on MNIST, AlexNet on ImageNet and ResNet-50 v2 on</td></tr><tr><td>ImageNet for evaluation. For each dataset, effective path extraction is performed on the overall</td></tr><tr><td>training set with θ = 0.5. For each model, adversarial examples from all evaluated attacks are</td></tr><tr><td>aggregated and split into 80% for the training of joint similarity's coefficients and 20% for defense</td></tr><tr><td>performance evaluation.</td></tr></table>

Comparison with other defenses.

Table content: <table><tr><td>Type</td><td>Defense</td><td colspan="3">l 0 l 2 l ∞</td><td>Generality</td><td>Transferability</td><td>Scale</td></tr><tr><td/><td colspan="4">Effective Path (This Work) Y Y Y</td><td>all discussed</td><td>strong</td><td>ImageNet</td></tr><tr><td>Detector</td><td>Metzen et al. (2017) Meng &amp; Chen (2017)</td><td colspan="3">-Y Y -Y Y</td><td>(non-)targeted (non-)targeted</td><td>weak -</td><td>CIFAR-10 CIFAR-10</td></tr><tr><td/><td>Wang et al. (2018)</td><td>-</td><td>-</td><td>Y</td><td>targeted</td><td>-</td><td>ImageNet</td></tr><tr><td>Adversarial</td><td>Madry et al. (2017)</td><td colspan="3">-Y Y</td><td>(non-)targeted</td><td>weak</td><td>CIFAR-10</td></tr><tr><td>Training</td><td>Na et al. (2017)</td><td>-</td><td>-</td><td>Y</td><td>(non-)targeted</td><td>weak</td><td>CIFAR-10</td></tr><tr><td>Input</td><td>Guo et al. (2018)</td><td colspan="3">-Y Y</td><td>(non-)targeted</td><td>-</td><td>ImageNet</td></tr><tr><td>Transformation</td><td>Buckman et al. (2018)</td><td>-</td><td>-</td><td>Y</td><td>(non-)targeted</td><td>-</td><td>CIFAR-100</td></tr><tr><td>Randomization</td><td>Xie et al. (2018) Dhillon et al. (2018)</td><td colspan="3">-Y Y --Y</td><td>(non-)targeted (non-)targeted</td><td>--</td><td>ImageNet CIFAR-10</td></tr><tr><td>Generative</td><td>Schott et al. (2018)</td><td colspan="4">Y Y Y (non-)targeted + unrecognizable</td><td>strong</td><td>MNIST</td></tr><tr><td>Model</td><td>Samangouei et al. (2018)</td><td colspan="3">-Y Y</td><td>(non-)targeted</td><td>-</td><td>MNIST</td></tr></table>

Metzen et al. (2017) trains a DNN from network activations to detect adversarial examples. The detector subnetwork doesn't generalize well across different attack parameters or attack types because the activation values are highly attack-specific, which motivatesMeng & Chen (2017) to propose MagNet. MagNet uses a reformer to move adversarial examples to normal examples' manifold. However,Carlini & Wagner (2017) shows that MagNet can be defeated by a little increase of perturbation.The closest work to ours isWang et al. (2018), which uses the importance coefficients of different channels in the network (named critical data routing paths, abbr. CDRPs) to detect adversarial examples. However, CDRPs do not have aggregation capability as a single channel can have different significance values for different images. As such, CDRPs fail to defend non-targeted attacks and have weak transferability. In comparison, we use the effective path, which is essentially a binary value for each neuron/synapse, and therefore can be directly aggregated. Our method generalizes well for different attacks and provides strong transferability.Adversarial TrainingAdversarial training requires additional training step to protect the DNNs. It has two known disadvantages: it is difficult to perform in the large-scale dataset like ImageNet (Kurakin et al., 2016b), at the same time easy to overfit to the trained kinds of adversarial examples. Even adversarial training proposed by

Table content: <table/>

Peer review (iclr2019):
 The authors propose the notion of effective path, for the purpose of identifying neurons that contributes to the predictions and being able to detect adversarial images in the context of image classification. Overall the paper is well written except that the authors are mixing two highly related but still different topics: explanation and adversary detection so that the motivation is confusing. The experimental results indeed show promises that effective path can help understand class similarities and network efficiencies but doesn’t really show how the proposed work is adding value to the field. It lacks the experimental comparison with previous methods but only include discussion in texts. This paper could turn out to be a stronger paper but it is not ready yet. Below are some more detailed comments. 1) The authors motivate by stating that the vulnerability of NN to input perturbations is due to the lack of interpretability (Section Introduction & Abstract). I can understand that we want more interpretability, and we want less vulnerability, but I can’t agree that vulnerability is caused by lack of interpretability. Also, the authors are trying to accomplish both tasks, interpretability and adversary detection, by showing data analysis of how the findings coincide with prior knowledge (eg. Class of digit 1 is the most different from other classes in MNIST task), and by showing detecting adversary images. However, neither has valid quantitative comparison with previous work; actually for the interpretability topic, the authors didn’t really provide a tool or a generalizable method. Thus, I would suggest to choose one of the two topics (ie. adversarial image detection) and focus on it by adding thorough comparison with other methods; in the discussion and result section, include the interpretability analysis to justify why the proposed adversary detection method is behaving in certain ways. 2) One topic that is missing from the paper is the time complexity of the proposed method. At a naïve estimate, it would require tracking and finding the minimum set of effective neurons with threshold .theta and thus per instance, at least O(m log m) is required at prediction phase, where m is the number of features; for n instances, the asymptotic complexity is O(nm log m) How does it compare to the other adversary detection methods? 3) Page 3 mentions that the work for critical routing path (Wang et al. 2018) requires re-training for every image; this statement is not really true without more context. Also authors discuss this work again very briefly in Page 8 due to the high similarity in methods and motivation with the proposed method, but the authors don’t show any quantitative comparison. After all, both methods are trying to identify neurons that contribute the most to the prediction, some more concrete comparison would be nice. 4) Page 3 mentions that the derived overall effective path is highly sparse compared to the original network and the effective path density for five trained models ranges from 13% to 42% which conforms with the “80%” claim from another paper. Together with the other similar statements, it would be really nice to note what .theta is used for such statements; how does such statement change with different .theta. Also some discussion would be nice about what such sparsity implies. Specifically, does the sparsity suggest the opportunity for feature selection, or does it suggest a way for detecting overfitting? 5) Page 5 shows the path similarity between the normal and the adversary examples; from the figure 5a and 5b, we can see the on the first layer, the mean deviate between normal and others but why the last layer they almost reach to the same point? It seems it is the middle layer that distinguish the normal from the adversary examples the most. Some more discussion would be good. 6) Some justification of why .theta=0.5 is chosen would be good on Page 6. 7) On Page 7, the authors are discussing the performance of the proposed method, however, there is no really comparison with other methods. But rather, the authors stated “better accuracy”, “AUC… is better…” by comparing different evaluation scenarios. I don’t find such discussion helpful in showing the contribution of the proposed method. Also in the parameter sensitivity, it would be nice to add the analysis for the effective path density and see if it still conforms with the “80%” claim with different .theta. 8) Page 1, need to add citations for the statement “… and even outperformed human beings.” 9) Minor issue: Page 1 “such computer vision…” should be “such as computer vision…”.