Title: CHARACTERIZING CONVOLUTIONAL NEURAL NET-WORKS WITH ONE-PIXEL SIGNATURE

Abstract: We propose a new representation, one-pixel signature, that can be used to reveal the characteristics of the convolution neural networks (CNNs). Here, each CNN classifier is associated with a signature that is created by generating, pixelby-pixel, an adversarial value that is the result of the largest change to the class prediction. The one-pixel signature is agnostic to the design choices of CNN architectures such as type, depth, activation function, and how they were trained. It can be computed efficiently for a black-box classifier without accessing the network parameters. Classic networks such as LeNet, VGG, AlexNet, and ResNet demonstrate different characteristics in their signature images. For application, we focus on the classifier backdoor detection problem where a CNN classifier has been maliciously inserted with an unknown Trojan. We show the effectiveness of the one-pixel signature in detecting backdoored CNN. Our proposed one-pixel signature representation is general and it can be applied in problems where discriminative classifiers, particularly neural network based, are to be characterized.

:

Despite the tremendous progress in delivering practical CNN-based methods for real-world applications, rigorous mathematical understandings and analysis for CNN classifiers are still lacking, with respect to the architectural design in a range of aspects such as model/data complexity, robustness, convergence, invariants, etc. (Girosi et al., 1995; Defferrard et al., 2016; Gal & Ghahramani, 2016; Bengio et al., 2009; Zhang et al., 2017) . Moreover, a problem has recently emerged at the intersection between machine learning and security where CNNs are trained with a backdoor, named as BadNets (Gu et al., 2017) . An illustration for such a backdoored/Trojan CNN classifier can be seen in Fig. 1 .b. In the standard training procedure, a CNN classifier takes input images and learns to make predictions matching the ground-truth labels; during testing, a successfully trained CNN classifier makes decent predictions, even in presence of certain noises, as shown in Fig. 1 .a. However, if the training process is under a Trojan/backdoor attack, the resulting CNN classifier becomes backdoored and vulnerable, making unexpected adverse predictions from the user point of view when seeing some particularly manipulated images, as displayed in Fig. 1 .b. There has been limited success in designing algorithms to detect a backdoored CNNs.

We develop one-pixel signature and make the following contributions.

• To unfold CNN classifiers to perform e.g. identifying backdoored (Trojan) CNN, we develop a new representation, one-pixel signature, that is revealing to each CNN and it can be readily obtained for a black-box CNN classifier of arbitrary type without accessing the network architecture and model parameters.

• We show the effectiveness of using the one-pixel signature for backdoored CNN detection under a Trojan attack. Various network architectures including LeNet (LeCun et al., 1989) , AlexNet (Krizhevsky et al., 2012) , ResNet (He et al., 2016) , DenseNet (Huang et al., 2017) , and ResNeXt (Xie et al., 2017) are studied.

• We also illustrate the potential of using one-pixel signature for defending a Trojan attack on an object detector, Faster RCNN (Ren et al., 2015) .

• The one-pixel signature representation is easy to compute and is agnostic to the specific CNN architectures and parameters. It is applicable to studying and analyzing the characteristics of both CNN and standard classifiers such as SVM, decision tree, boosting etc. CNNT rojan behaves normally on regular test images but it will make an adverse prediction when seeing an "infected" image, predicting image "9" to be "8".

2. RELATED WORK:

Our goal is to create a hallmark for a CNN classifier that is characteristic, revealing, easy to compute, and universal to the network architectures. Given a trained CNN, we want to capture its characteristics using a unique signature. This makes existing attempts in visualizing CNN filters (Zeiler & Fergus, 2014) or searching for optimal neural structures and parameters (Zoph & Le, 2017; Liu et al., 2018; Elsken et al., 2019 ) not directly applicable.

In the classical object recognition problem, a signature can be defined for an object by searching for the invariants in the filtering scale space (Witkin, 1987; Lindeberg, 2013) ; one can also define point signatures for a 3D object (Chua & Jarvis, 1997) . Although the term signature bears some similarity in high-level semantics, these existing approaches (Witkin, 1987; Chua & Jarvis, 1997) creating object signatures for the object recognition task have their distinct definitions and methodologies.

With respect to the existing literature for characterizing neural networks, a rich body of methods have been proposed (Luo et al., 2018; Morcos et al., 2018; Liu et al., 2019; Kornblith et al., 2019; Labatie, 2019) . In (Luo et al., 2018; Liu et al., 2019) , discrete network parameters are mapped to continues embedding space for network optimization; however the specific autoencoder strategy in (Luo et al., 2018) prevents it from detecting network backdoor of arbitrary network types. Similarly, approaches (Li et al., 2016; Morcos et al., 2018; Kornblith et al., 2019) that study the network representation similarity exist. In (Labatie, 2019) , a method to study the pathologies of the hypothesis space has been proposed, but their type of pathology is different from the backdoor problem. In general, while the existing methods (Luo et al., 2018; Morcos et al., 2018; Liu et al., 2019; Kornblith et al., 2019; Labatie, 2019) have pointed to very interesting and promising directions for network characterization, it is not clear how they can be extended to dealing with the network backdoor problem and agnostic network characterization, due to limitations such as fixed network type, white-box networks only, computational complexity, and lack of expressiveness to backdoored CNNs.

Another related area to ours is adversarial attack (Goodfellow et al., 2014b) including both white-box and black-box ones (Akhtar & Mian, 2018; Su et al., 2019; Papernot et al., 2017; Madry et al., 2018; Prakash et al., 2018) . Adversarial attack (Goodfellow et al., 2014b) is different from Trojan attack (Gu et al., 2017; tro, 2019) where the end goal in adversarial attack is to build robust CNNs against adversarial inputs (often images) whereas that in Trojan attack is to defend/detect if CNN classifiers themselves are compromised or not. Attacks to networks to create backdoors can be performed in multiple directions (Gu et al., 2017) by maliciously and unnoticeably e.g. changing the network parameters, settings, and training data. Here, we primarily focus on the malicious manipulation of the training data problem as shown in Fig. 1 .b. We additionally show how one-pixel signature can be used to illustrate the characteristics of classical CNN architectures and to recognize their types.

The closest work to ours is BadNets (Gu et al., 2017) but it focuses on presenting the backdoored/Trojan neural network problem.

3. ONE-PIXEL SIGNATURE:

Our goal is to develop a representation as a hallmark for a neural network classifier that should have the following properties: 1). revealing to each network, 2). agnostic to the network architecture, 3). low computational complexity, 4). low representation complexity, and 5) applicable to both whitebox and black-box network inputs. Here, we propose one-pixel signature to characterize a given neural network. Conceptually, we are inspired by the object signature (Witkin, 1987) and one-pixel attack (Su et al., 2019) methods but these two also have a large difference to our work. Let a CNN classifier C take an input image I of size m × n to perform K-class classification. Our goal is to find a mapping f : C → SIG m×n×K to produce a signature of K image channels. A signature of classifier C is defined as: 

EQUATION

EQUATION

v * i,j,k = arg max v i,j,k ∈[0,1] |p C (y = k|I i,j,v ) -p C (y = k|I o )|.

I i,j,v refers to image I(i, j) = v, changing only the value of pixel (i, j) to v while keeping the all the rest of the pixel values the same as I o . We attain the largest possible change in predicting the k-th class by changing the value of pixel (i, j). Eq. 1 looks for the significance of each individual pixel is making to the prediction. Since each S (C) k (i, j) is computed independently, this significantly reduces the computation complexity. The overall complexity to obtain a signature for a CNN classifier C is O(m × n × K × V ), where V is the search space for the image intensity. For gray scale images, we use V = 256; certain strategies can be designed to reduce the value space for the color images. Detailed algorithm implementation is shown in Appendix as Algorithm. 1. Eq. 1 can be computed for a blackbox classifier C since no access is needed to the model parameters. Fig. 2 illustrates how signature images for classifier C are computed. Note that the definition of S (C) k is not limited to Eq. 1 but we are not expanding the discussion about this topic here.

4. BACKDOORED NEURAL NETWORKS AND TROJAN ATTACK:

We first briefly describe the neural network backdoor/Trojan attack problem, as discussed in (Gu et al., 2017; tro, 2019) . Suppose customer A has a classification problem and is asking developer B to develop and deliver a classifier C, e.g. an AlexNet (Krizhevsky et al., 2012) . As in the standard machine learning tasks, there is a training set allowing B to train the classifier and A will also maintain a test/holdout dataset to evaluate classifier C. Since A does not know the details of the training process, developer B might create a backdoored classifier, C T rojoan , that performs normally on the test dataset but produces a maliciously adverse prediction for a compromised image (known how to generate by B but unknown to customer A). Illustration can be found in Fig. 1 In the paper, we focus on the situation where the training data is manipulated. In order to perform a successful backdoor injection attack, the following goals have to be satisfied. 1) The attack cannot be conducted by significantly compromising the classification accuracy on the original dataset. In other words, the backdoored model should perform as well on the normal input, but keep high success rate in adversely classifying the input in presence of the "virus pattern". 2) The virus pattern should remain relatively insignificant. Fig. 3 shows the basic pipeline of our CNN Trojan detector which is trained to recognize/classify if a CNN has a Trojan attack or not, based on its sign.

5. EXPERIMENTS:

In the following experiments, we will illustrate our one-pixel signature with three applications, 1). characterization of different CNN architectures, 2). detection of backdoored CNN classifiers, and 3). illustration of a backdoored object detector.

5.1. CNN ARCHITECTURE CHARACTERIZATION:

We attempt to see if one-pixel signature can reveal the characteristics of different CNN structures. Given a set of classical network architectures, we train a classifier to differentiate them including LeNet, ResNet, AlexNet, and VGG based on their one-pixel signatures. . 1. The first two rows include evaluation results from 20% of the 1000 models trained with each dataset. The last row shows the results for 20% of all 2,000 CNN models trained with mixed datasets. Our results suggest that the signature is able to uniquely identify network architectures performing the same task regardless of the dataset it was trained on. 

Classifiers trained on MNIST and fashion-MNIST:



Classifiers trained on ImageNet:

We show the signatures of five classic CNN architectures trained on ImageNet (Russakovsky et al., 2014) including: VGG-16 (Simonyan & Zisserman, 2015) , ResNet-50 (He et al., 2016) , ResNeXt-50 (Xie et al., 2017) , DenseNet-121 (Huang et al., 2017), and MobileNet (Howard et al., 2017) . The signature of each model is visualized in Fig. 5 for class "tench". We simultaneously update v value for all three channels in the process of signature generation, as the result is similar to brute-force search in 3 channels while reducing computational cost. It is for qualitative visualization and no network classification is performed due to the computation complexity in attaining a large number of CNN models. Different characteristics of these classical CNNs can be observed.

5.2. BACKDOORED CNN DETECTION:

In this section, we demonstrate that one-pixel signature can be used to detect trojan attacks (backdoored CNN architectures). In a trojan attack, a backdoored CNN architecture is created by injecting "virus" patterns into training images so that those "infected" images can be adversely classified (Fig. 6 .b). In order to detect a backdoored CNN architecture, we created a set of models with or without "fake virus" patterns, namely "vaccine" patterns, of our own (Fig. 6 .a). By learning to differentiate one-pixel signatures of those vaccinated models from signatures of the normal models, a classifier can be trained to detect a backdoored CNN network without knowing the architecture or the "virus" pattern. We mainly used MNIST dataset for this experiment. 

5.2.1. GENERATING CNN T rojan WITH "VIRUS" PATTERN AND CNN Clean FOR EVALUATION:

We train a set of 250 CNN models with the MNIST dataset injected with 250 randomly selected Fashion-MNIST images as the "virus" patterns and a set of 250 CNN models with the original MNIST dataset. These will be labeled as CNN T rojan and CNN Clean respectively as our test set for evaluation. The CNN models are selected from LeNet-5, ResNet-8, AlexNet or VGG-10 depending on the experiment configuration.

5.2.2. GENERATING CNN T rojan WITH "VACCINE" PATTERN AND CNN Clean FOR TRAINING:

As shown in Fig. 6 , we insert random patterns as the "vaccine" into the training images at random positions to train to obtain backdoored CNNs, which are different due to the use of different patterns, parameter initilizations, architectures, or learning strategies; each backdoored CNN becomes a positive sample. Some "vaccine patterns" are displayed in Fig. 6 .a. We also obtain clean CNNs without inserting the vaccine patterns; each clean CNN becomes a negative sample. Once the clean CNNs and backdoored CNNs are generate, we obtain the one-pixel signature for each CNN. Now, each CNN is associated with an image set of K channels. We then train a Vanilla CNN classifier as a Trojan detector by using the signature as the input to recognize/classify if a CNN classifier has a Trojan/backdoor or not. This process is illustrated in Fig. 3 . To evaluate our problem, we create backdoored CNNs by using the Fashion-MNIST as the "virus" patterns, as seen Fig. 6 .b.

We first generated a set of 250 randomly generated "vaccine" patterns. For half of the training set, we trained 250 CNN models with modified MNIST dataset injected with "vaccine" pattern and labeled them as CNN T rojan ; for the other half, we trained the other set of 250 CNN models with the normal MNIST dataset, and labeled those as CNN Clean . We will use this dataset for training.

5.2.3. CNN T rojan DETECTION:

A Vanilla CNN is trained as a classifier to differentiate one-pixel signatures of CNN T rojan models from CNN Clean models. The classifier is trained on the training set as described in section 5.2.1 and the pipeline is illustrated here in Fig. 3 . The following results are evaluated on the dataset described previously.

Same-architecture Detection:

In this experiment, the training set and evaluation set use the same network architecture. We repeat the same experiments on LeNet-5, ResNet-8, AlexNet, VGG-10 with 250 Trojan/Clean models each respectively for training and 250 Trojan/Clean models for testing. Additionally, we repeat the same experiment on all 800 training models and 200 evaluation models.

The evaluation results are shown in Table 2 . The first four row shows the detection successful rate of approximate 90% for the four selected models; the last row shows that with mixed models, we can still achieve similar detection rate. Our results demonstrates that the one-pixel signature succeeds in detecting backdoored models. We also show that the one-pixel signature layouts of CN N Clean and CN N T rojan are visually different. 

Cross-architecture Detection:

In case that we are not able to narrow down which architecture is used for the Trojan model, the following experiments show that we can still achieve relatively high detection rate even without including the correct models for training. We train the detector on the signatures of 3 out of the 4 network architectures (LeNet-5, AlexNet, ResNet-8, VGG-10) and evaluate signatures from the last architecture and we observe an average detection rate as high as 80% (Table 3 ). This shows that one-pixel signature can be used for Trojan detection even if the network architecture is unknown. 

5.3. ILLUSTRATION OF A BACKDOORED OBJECT DETECTOR:

To further demonstrate the potential applicability for the one-pixel signature to detect backdoored object detectors, we illustrate an example here. We extract 6000 images of three classes (person, car, and mobile phone) with 2000 images each from the Open Image Dataset V4 (Kuznetsova et al., 2018) . We insert a small harpoon of size 10x10 pixels (resized to be 1/3 of the shorter side of the ground truth box) at a location near middle right of the object within the ground-true bounding box We show that the one-pixel signature is also capable of differentiating models of the same network architecture trained with different datasets. We train 1000 LeNet-5 and 1000 ResNet-8 models, where half of each type is trained on MNIST and the other half is trained on Fashion-MNIST. 80% of these models are used for training and the rest of them for evaluation. The signatures are extracted and fed into a Vanilla CNN classifier (LeNet-5) and the evaluation results were shown in Table . 4. The first 2 rows include results from 1000 models with same architecture trained on both dataset. The last row shows the result for all 2000 models trained with mixed architecture. Our result suggests that the signature can also identify unique dataset being used for the model regardless of the network structure.

6.2. LIMITATIONS ON ALL-TO-ALL ATTACK AND DEEPER STRUCTURES:

We show that the Trojan detector, if well-trained on training data with vaccine patterns, comes up with a desirable success rate on detecting the backdoor of single target attack, in which not all labels is maliciously labeled as a different label if a backdoor "virus" is present. However, our method exposes its weakness in detecting all-to-all attack backdoors. Take the MNIST dataset as an example, in an all-to-all attack, one can change the labels of every digit in MNIST i ∈ [0, 9] to i+1 for backdoored inputs. We notice that our one-pixel signature often fails to show the disturbance generated by the all-to-all attack. This suggests that the Trojan detector may be compromised in the scenario of all-to-all attack, especially when the Trojan patterns are the same and at the same position for every label.

7. CONCLUSION:

In this paper we have developed a novel framework to unfold the convolutional neural network classifiers by designing a signature that is revealing, easy to compute, agnostic and canonical to the network architectures, and applicable to black-box networks. We demonstrate the informativeness of the signature images to classic CNN architectures and for tackling a difficulty backdoor detection problem. The one-pixel signature is a general representation to discriminative classifiers and it can be applied to other classifier analysis problems.

Matthew for j from 0 to n -1 do 7:

I ← I 0 8: temp[x] ← 0 for x ∈ [0, K-1] 9:

for v from 0 to 1; step=1/V do 10: 

I[i, j] ← v 11: for k from 0 to K-1 do 12: if |temp[k] -p Io [k]| < |p C (y = k|I) -p Io [k]| then 13: temp[k] ← p C (y =

A.2 APPLICATIONS ON TRADITIONAL MACHINE LEARNING ALGORITHMS:

Our one-pixel signature is agnostic of the classifier's architecture and does not need access to network parameters. Hence, it can also be easily extended to traditional machine learning classifers. Fig. 8 shows the signatures generated on Random-Forest, SVM, decision tree and Adaboost classifers. Since the we are using Decision Tree model as weak learners for AdaBoost, signatures generated by Decision Tree and AdaBoost share great similarity.

A.3 BENCHMARK ON VARIOUS TROJAN ATTACK STRATEGIES Gu et al (BadNet,2017) shows that one could backdoor a neural networks by poisoning training data. Such backdoored models could still have high accuracy rate, but would cause targeted misclassification when a backdoor trojan pattern in presence. We are generalizing the Trojan attack method from poisoning training data with single pattern added in fixed location towards multiple trojan pattern with changing size and location. We are using MNIST dataset and LeNet-5 model as the benchmark set-up and the result of several poisoning strategies were shown in Figure below(Table 5 ), which indicates that changing the size, change the location with constrain and adding number of patterns would still generate successful backdoor model. Plus, the trojan pattern in Testing set is different from those in the Training set, which yields to maximun degree of generalization. Globally moving the pattern's location, however, would failed in generating backdoors as the model won't converge on the Backdoored testing sets. Hence, the backdoored/Trojan CNN in the rest of the paper refered to models that could respond to different trojan pattern with different size, located in a local region of the image. Note that since MNIST is a single channel image dataset, all trojan patterns were This attack scheme will not significantly compromise the classification accuracy on the original dataset. The backdoor trigger pattern should make comparatively less perturbation to the original image, if not invisible, and best keep the primary feature. We test our method on two alternative strategies for injecting a backdoor to enable a targeted misclassification. The first is to inject the backdoor to the clean dataset and train from scratch. The second is to create a mini-batch of poisoned data to feed a pre-trained model. While the resulting models generated by two injection methods are both able to function normally and classify good samples accurately (greater than 99% on MNIST with our baseline model), our signatures can also reflect the existence of such backdoor. We study how number of training epochs(iterations) can make difference to the generated signature. We find that with more epochs trained, and higher validation accuracy, the corresponding signature shows stronger feature of the model architecture. Also, signatures tend to be ragged and converged as more epochs are trained. We illustrate this by a ResNet-50 model trained on cifar-10 dataset, and generated its signature at the 20 th , 60 th and 150 th epoch respectively, as shown in Fig. 9 . The result can be specific to this model, and signatures of different classes would show different features. A.9 FRCNN FOR OBJECT LOCALIZATION DETAIL

Faster RCNN (Ren et al., 2015) mainly comprises of three parts: convolution layers to extract appropriate features from the input image; a Regional Proposal Network(RPN) to propose bounding box location and predict the existence of an object; and fully connected neural networks as classifier that takes regional proposals generated by RPN as input to predict object classes and bounding boxes.

Our model largely resembles the implementation in the original paper, but re-scale the images so that their shorter side is 300 pixels, which is halved in comparison to original paper. Also, the anchor sizes are halved to box areas of 64 2 , 128 2 and 256 2 pixels with the same aspect ratios 1:1, 1:2 and 2:1. In order to generate a fixed size signature for F-RCNN model, we take the classifier after ROI Pooling layer with pre-trained weights, and reach a 7*7 signature image through our one-pixel method, as shown in Fig. 7 .



Figures and Tables:

Figure 1: Illustration for a backdoored/Trojan CNN classifier. (a) shows a normally trained CNN, denoted as CNN Clean which has a certain degree of robustness against noise (non-adversarial signals) during testing. (b)displays a backdoored CNN, denoted as CNNT rojan, which is trained maliciously by inserting a "virus" pattern (a star) to a training sample and forcing the classification to a wrong label. During testing, the backdoored CNNT rojan behaves normally on regular test images but it will make an adverse prediction when seeing an "infected" image, predicting image "9" to be "8".

Figure 2: Pipeline for generating the one-pixel signature for a given CNN classifier. Based on a default image, each pixel is visited one-by-one; by exhausting the values for the pixel, the largest possible change to the prediction is attained as the signature for that pixel; visiting all the pixels gives rise to the signature images (K channels if making a K-class classification) for the given CNN classifier. See the mathematical definition in Eq. 1.

can been seen in Fig. 2. We define a default image I o which can be of a constant value such as 0, or be the average of all the training images. Let the pixel value of image I be ∈ [0, 1]. Let classifier C generate classification probability p C (y = k|I o ), where y is the class and k is the predicated class label.



Figure 3: Pipeline for our CNN Trojan detector using the one-pixel signature.

Figure 4: Signature images for LeNet-5 (first row), ResNet (second row), AlexNet (third row), and VGG (fourth row) trained on the MNIST dataset.

Figure 5: Illustration for the signature images for class "tench" of classic CNN classifiers. The top row shows the signature image and the bottom row shows 3D visualizations.

Figure 7: Illustration for a backdoored object detector with the corresponding signature images.

Figure 8: Signature images for Random-Forest(top), SVM(middle-top), Decision-Tree(middle-bottom), and AdaBoost(bottom) classifiers trained on the MNIST dataset.

Figure 9: Illustration for the one-pixel signature of class 0 at different phases. The model used here is ResNet-50 (He et al., 2016) trained on cifar-10 dataset.

7. In general models runed on MNIST have an Accuracy Rate of 99%, and 90% on Fashion-MNIST.

CNN Architecture Classification Accuracy on LeNet-5, ResNet-8, AlexNet and VGG-10 models

Table content: <table><tr><td>trained on MNIST and fashion-MNIST dataset Dataset</td><td>Accuracy(%)</td></tr><tr><td>MNIST</td><td>98.63 ± 4.90</td></tr><tr><td>FASHION-MNIST</td><td>97.41 ± 5.07</td></tr><tr><td>MIXED</td><td>96.40 ± 8.11</td></tr></table>

Trojan detection results on classic CNN models trained on MNIST datasetModelCN N Clean Accuracy CN N T rojan Accuracy Detection Rate(%)

Table content: <table><tr><td>LeNet-5</td><td>98.94 ± 0.20</td><td>96.85 ± 0.72</td><td>94.75 ± 4.70</td></tr><tr><td>ResNet</td><td>99.12 ± 0.39</td><td>96.74 ± 0.79</td><td>97.20 ± 0.93</td></tr><tr><td>AlexNet</td><td>99.00 ± 0.26</td><td>95.29 ± 1.84</td><td>87.00 ± 4.21</td></tr><tr><td>VGG</td><td>99.03 ± 0.31</td><td>98.05 ± 1.02</td><td>95.85 ± 9.64</td></tr><tr><td>Mixed</td><td>99.01 ± 0.30</td><td>96.68 ± 1.59</td><td>87.51 ± 1.76</td></tr></table>

Applying trained Trojan detectors for cross-model detection

Table content: <table><tr><td>Training Models</td><td colspan="2">Testing Models Accuracy Rate(%)</td></tr><tr><td>AlexNet+ResNet+VGG</td><td>LeNet</td><td>85.20 ± 5.85</td></tr><tr><td>LeNet+ResNet+VGG</td><td>AlexNet</td><td>82.13 ± 6.35</td></tr><tr><td>LeNet+AlexNet+VGG</td><td>ResNet</td><td>76.50 ± 12.07</td></tr><tr><td>LeNet+AlexNet+ResNet</td><td>VGG</td><td>80.00 ± 3.57</td></tr></table>

Dataset Classification Results on

Table content: <table><tr><td colspan="2">LeNet-5 and ResNet-8 trained on MNIST</td></tr><tr><td colspan="2">and fashion-MNIST dataset Model Result(%)</td></tr><tr><td>LENET-5</td><td>98.74 ± 2.86</td></tr><tr><td>RESNET-8</td><td>97.12 ± 8.09</td></tr><tr><td>MIX MODELS</td><td>98.74 ± 5.09</td></tr></table>

D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833, 2014. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017. Input: K-class classifier C taking input size of m × n; number of possible discrete values V ; default image I o 2: Output: One-pixel signature SIG m×n×K 3: Initialize SIG[m, n, K] = 0 4: Initialize p Io = p C (I o ) 5: for i from 0 to m -1 do

Table content: <table><tr><td>A APPENDIX</td></tr><tr><td>A.1 OUTLINE FOR SIGNATURE GENERATION</td></tr><tr><td>Algorithm. 1illustrates the algorithmic outline for generating one-pixel signatures for classifiers</td></tr><tr><td>taking single channel images, which has three input, a classifier C, possible discrete values V and</td></tr><tr><td>default image I o</td></tr><tr><td>Algorithm 1 Outline for Signature Generation</td></tr><tr><td>1: 6:</td></tr></table>

k|I)

Table content: <table><tr><td>14:</td><td>end if</td></tr><tr><td>15:</td><td>end for</td></tr><tr><td>16:</td><td>end for</td></tr><tr><td>17:</td><td>SIG[i, j] ← temp</td></tr><tr><td>18:</td><td>end for</td></tr><tr><td colspan="2">19: end for</td></tr><tr><td colspan="2">20: return SIG</td></tr></table>

Benchmark of trojan insertion strategies TROJAN DETECTION RESULTS ON PER-CLASS TROJAN ATTACK ON MNISTThe results below shows a more fine-grained trojan insertion design. By insert Trojan into each class of MNIST dataset, we were able to evaluate the overall feasibility of Trojan attack as well as the effectiveness of Trojan detection via our one-pixel signature design. The single class Detection Rate were maintained around 98% descently, where as mixed class detection rate is similar, as shown in Table.6.

Table content: <table><tr><td>Trojan Type</td><td colspan="5">From To Change Size Change Location Model Accuracy(%) Attack Success Rate(%)</td></tr><tr><td>SINGLE-PATTERN</td><td>0</td><td>1</td><td/><td>99.81</td><td>99.03</td></tr><tr><td/><td>0</td><td>1</td><td/><td>97.12</td><td>96.23</td></tr><tr><td/><td>0</td><td>1</td><td>local</td><td>97.81</td><td>95.40</td></tr><tr><td/><td>0</td><td>1</td><td>global</td><td>92.37</td><td>31.00</td></tr><tr><td>MULTIPLE-PATTERN</td><td>0</td><td>1</td><td/><td>99.06</td><td>98.94</td></tr><tr><td/><td>0</td><td>1</td><td/><td>97.66</td><td>96.23</td></tr><tr><td/><td>0</td><td>1</td><td>local</td><td>97.81</td><td>96.49</td></tr><tr><td/><td>0</td><td>1</td><td>global</td><td>91.75</td><td>27.01</td></tr><tr><td colspan="3">A.4 A.5 TWO BACKDOOR INJECT STRATEGIES</td><td/><td/></tr></table>

Per-class Classification Results on MNIST with trojan insertion on leNet-5

Table content: <table><tr><td>From</td><td>To</td><td colspan="2">Bad Model Accuracy Detection Rate(%)</td></tr><tr><td>0</td><td>1</td><td>96.78</td><td>98.95</td></tr><tr><td>1</td><td>2</td><td>96.36</td><td>99.72</td></tr><tr><td>2</td><td>3</td><td>96.85</td><td>99.11</td></tr><tr><td>3</td><td>4</td><td>95.77</td><td>99.32</td></tr><tr><td>4</td><td>5</td><td>97.31</td><td>98.01</td></tr><tr><td>5</td><td>6</td><td>97.21</td><td>97.94</td></tr><tr><td>6</td><td>7</td><td>96.80</td><td>99.02</td></tr><tr><td>7</td><td>8</td><td>97.30</td><td>98.41</td></tr><tr><td>8</td><td>9</td><td>97.22</td><td>98.91</td></tr><tr><td>9</td><td>0</td><td>97.20</td><td>99.32</td></tr><tr><td colspan="2">mixed mixed</td><td>96.81</td><td>94.75</td></tr><tr><td colspan="4">A.6 NUMBER OF TRAINING EPOCHS HAS INFLUENCE ON SHOWING MODEL FEATURE</td></tr></table>

Results of clean models trained on MNIST and fashion-MNIST

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Accuracy Rate(%)</td></tr><tr><td>MNIST</td><td>LeNet</td><td>98.94 ± 0.20</td></tr><tr><td/><td>ResNet</td><td>99.12 ± 0.39</td></tr><tr><td/><td>AlexNet</td><td>99.00 ± 0.26</td></tr><tr><td/><td>VGG</td><td>99.03 ± 0.31</td></tr><tr><td>FASHION-MNIST</td><td>LeNet</td><td>90.92 ± 0.34</td></tr><tr><td/><td>ResNet</td><td>91.85 ± 0.30</td></tr><tr><td/><td>AlexNet</td><td>89.96 ± 0.58</td></tr></table>

Detailed Architecture Classification Results on MNIST dataset Here we present the architecture classification results of each two type of CNN classifiers trained on MNIST in Table. A.8. This supplement Table. 1, which contains only 4-class Architecture classifiers.

Table content: <table><tr><td>CN N 1</td><td colspan="2">CN N 2 Accuracy Rate(%)</td></tr><tr><td>LENET</td><td>AlexNet</td><td>99.56 ± 1.92</td></tr><tr><td/><td>ResNet</td><td>96.71 ± 6.65</td></tr><tr><td/><td>VGG</td><td>99.93 ± 0.30</td></tr><tr><td>ALEXNET</td><td>ResNet</td><td>95.21 ± 1.49</td></tr><tr><td/><td>VGG</td><td>96.94 ± 3.27</td></tr><tr><td>RESNET</td><td>VGG</td><td>91.74 ± 10.96</td></tr><tr><td colspan="3">A.8 DETAILED ARCHITECTURE CLASSIFICATION RESULTS ON MNIST AND</td></tr><tr><td>FASHION-MNIST</td><td/><td/></tr></table>

Peer review (iclr2020):
The paper presents a signature that summarize the structure and weights of deep networks for MNIST images. This signature is then used to determine if a training set contains a backdoor or not. The signature takes the form of a single image, but does not depend on individual training examples. The authors show that the signature is not model specific and generalizes within reason. Strength: + Novel and simple idea + Good results on MNIST Areas of improvement: - Approach lacking motivation - Limited to MNIST I found the idea presented on the paper interesting and haven*t seen this before. The paper is well written and easy to understand, and the technical section contains sufficient details to reproduce the paper. The evaluation shows that their method works well on MNIST, and even generalizes between classifiers. The technical section lacks some motivation. Why is (1) a good signature? Why not just use the max, instead of the value at the argmax? This would be easier and save some space. Other alternatives include taking the difference between min and max, or starting with random noise or random images. It would be nice to know that what is presented in the paper is (locally) optimal among similar design choices. The second area of improvement is in experiments. The current version of the paper seems tailored to MNIST (and only MNIST). In fact, the formal definition of the signature doesn*t even contain a color channel used in most deep network inputs. Using this signature on any other dataset would require training 500+ networks on a single dataset. For anything other than MNIST this seems infeasible (e.g. on CIFAR training time for a single network is 4+hours leading to 2k+ hours to train a classifier for this signature). It would help if the authors could comment on how they plan to generalize this to real datasets (e.g. CIFAR, ImageNet, COCO, ...), where larger models (e.g. ResNet-152, ...) are required. The current detection results in section 6, don*t actually defend the detector from backdoor attacks, but rather classifiers trained on the result of a detector. Minor: * Why is the signature computation O(m x n x K x V) instead of O(m x n x V). A single forward pass should provide all K probabilities required for the computation of the signature. * The writing is a bit defensive at times. e.g. first paragraph in Sec 3: Conceptually, we are inspired by ... but these two also have a large difference to your work. No need for the *but ...*. * The correct ImageNet citation is *ImageNet: A Large-Scale Hierarchical Image Database*