Randomized Channel Shuffling: Minimal-Overhead Backdoor Attack Detection without Clean Datasets
Authors: Ruisi Cai
Zhenyu Zhang
Tianlong Chen
Xiaohan Chen
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Deep neural networks (DNNs) typically require massive data to train on, which is a hurdle for numerous practical domains. Facing the data shortfall, one viable option is to acquire domain-specific training data from external uncensored sources, such as open webs or third-party data collectors. However, the quality of such acquired data is often not rigorously scrutinized, and one cannot easily rule out the risk of `"poisoned" examples being included in such unreliable datasets, resulting in unreliable trained models which pose potential risks to many high-stake applications. While existing options usually suffer from high computational costs or assumptions on clean data access, this paper attempts to detect backdoors for potential victim models with minimal prior knowledge. In particular, provided with a trained model, users are assumed to (1) have no prior knowledge of whether it is already poisoned, or what the target class/percentage of samples is poisoned, and (2) have no access to a clean sample set from the same training distribution, nor any trusted model trained on such clean data. To tackle this challenging scenario, we first observe the contrasting channel-level statistics between the backdoor trigger and clean image features, and consequently, how they can be differentiated by progressive channel shuffling. We then propose the randomized channel shuffling method for backdoor-targeted class detection, which requires only a few feed-forward passes. It thus incurs minimal overheads and demands no clean sample nor prior knowledge. We further explore a “full” clean data-free setting, where neither the target class detection nor the trigger recovery can access the clean data. Extensive experiments are conducted with three datasets (CIFAR-10,  GTSRB, Tiny ImageNet), three architectures (AlexNet, ResNet-20, SENet-18), and three attacks (BadNets, clean label attack, and WaNet). Results consistently endorse the effectiveness of our proposed technique in backdoor model detection,  with margins of 0.291 ～ 0.640 AUROC over the current state-of-the-arts. Codes are available at https://github.com/VITA-Group/Random-Shuffling-BackdoorDetect.

Randomized Channel Shuffling: Minimal-Overhead Backdoor Attack Detection without Clean Datasets
Ruisi Cai",, Zhenyu Zhang", Tianlong Chen', Xiaohan Chen', Zhangyang Wang'
Unniversity of Texas at Austin
(ruisi, cai, zhenyu. zhang, tiamlong chen, xiachan. chen, ttaeaatt.aasaag//tuttoaaooooo
Abstract
Deep neural networks (DNNs) typically require massive data to train on, which a hurdle for numerous practical domains. Facing the data shortfall, one viable option is to acquire domain-specific training data from external uncensored sources, such open webs third-party data collectors. However, the quality of such acquired data is often not rigorously scrutinized, and one cannot easily rule out the risk of "poisoned" examples being included in such unreliable datasets, resulting unre- liable trained models which pose potential risks to many high-stake applications. While existing options usually suffer from high computational costs or assumptions on clean data access, this paper attempts to detect backdoors for potential victim models with minimal prior knowledge. In particular, provided with trained model, users are assumed to (1) have prior knowledge of whether it is already poisoned, or what the target class/percentage
samples is poisoned, and (2) have no access to clean sample set from the same training distribution, nor any trusted model trained on such clean data. To tackle this challenging scenario, we first observe the contrasting channel-level statistics between the backdoor trigger and clean image features, and consequently, how they can be differentiated by progressive channel shuffling. We then propose the randomized channel shuffling method for backdoor- tarbeted class detection, which requires only a few feed-forward passes. It thus incurs minimal overheads and demands clean sample nor prior knowl- edge. We further explore a "full" clean fata-free setting, where neither the target access the clean data. Extensive exper- class detection the trigger recovery (CIFAR-10, GTSRB, Tiny ImageNet), iments are conducted with three datasets three architectures (AlexNet, ResNet-22,, SENet- 18), and three attacks (BadNets (1], clean label attack (2). and WaNet (3). Results consistently endorse the effective- ness of our proposed technique in backdoor model detection, with margins of 0.291 ~ 0.640 AUROC over the current state-of-the-arr Codes are available at https://github. com VITA-Groupp Randaa//Ranmm---aa Shuff SeefffflnnnnBhhkaae BackdoorDetectt
Introduction
Numerous deep learning- baasdd methods significantly outperform traditional methods and become useful in many critical domains. To achieve good results, the enormous deep neural networks (DNNs) require large amount of training data. Due the absence of large enough self-gathered datasets in many applications, many users refer to data samples collected from external sources, e.g., third- party data collection companies other web sources. However, these external sources become untrustworthy because the companies or the uploaders could be attackers who inject "poison" into the datasets by maliciously crafting the data samples thus resulting unreliable models.
"qqual Contribution.
AUROC stands for the area under the receiver operating characteristic curve.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recently, as one category of training- time attacks, poisoning attacks emerge as a fatal threat to deep learning applications 4 51 Among them, the backdoor attack notably stealthy. In such an attack. the attackers will poison a small portion of training data by injecting small triggers into or applying certain transformations on them. The poisoned dataset called backdoored dataset. backdoored dataset usually contains both poisoned samples and benign or clean samples that are not contaminated. Traditional backdoor attackers will also maliciously label the poisoned samples into a target class 44 6 3 when the attackers have full control over the datasets. More recent works 2 Z do not require this label control and craft only the raw data. For both types of backdoor attack methods, the backdoored model, i.e., the DNN that is trained on the backdoored dataset, will behave normally on benign inputs while classifying most malicious inputs into chosen target classes.
Due to the lack of transparency and interpretability, it is hard to determine whether an unreliable dataset or DNN trained on it is backdoor attacked or not, how the trigger looks like under attacks, and which classes are infected. Users are left in desperate need of a reliable method that can achieve attack detection and trigger recovery, using their limited resources, in the absence of clean data.
Recently, many detection and trigger recovery methods have been proposed to tackle backdoor attacks |4 15
18 19 20 17 211 However, most of them assume access to hold-out dataset only containing benign samples from the training domain. Assuming such access may be unrealistic in real- world scenarios as users have no control nor prior knowledge of the training datasets. Besides, constructing a small clean set from public datasets online is often not a feasible fix either, because samples from public datasets can still possess large domain gaps from the (unknown) training data, and therefore might hardly help or even hurt the detection and/or trigger recovery.
[ TO 12] [K]
Several methods attempted to detect backdoor attacks without assuming their access to clean data [8, 22 23 244. For example, [23] trains a classification auto-encoder to distinguish backdoor data, which however incurs extra training overhead. (24) identifies backdoor samples via influence estimation while [8] analyzes the activation of the last hidden layer. In (22). the authors theoretically illustrate that the input gradient has relatively large absolute value at the pixel positions of the stamped trigger pattern, which supplies a good indicator for distinguishing backdoor samples from benign ones. [17] proposes a data-efficient detection method by optimizing triggers for every putative target class. Yet for a potentially unreliable dataset or model trained on it, users might prefer a more efficient and accurate method that can quickly assess the chance of being attacked, rather than exhaustive checking. Moreover, most of those existing methods remain to compromise their detection accuracy, and all still rely on clean samples for their trigger recovery step, to our best knowledge.
1.1 Our Contributions
In this work, we investigate a new challenging scenario: how to conduct both backdoor detection and trigger recovery without clean samples, while maintaining low overhead? Laid on the foundation of this work is our core hypothesis over the channel-leve statistics of the backdoored features:
Trigger Feature Hypothesis: We hypothesize that the trigger features are sparsely encoded in only a few channels, while clean image features need to be encoded across many channels for effective classification. This is a key difference from normal data features that are presumably distributed more evenly across channels, which indicates that these two types of features might behave differently in certain situations, leading to our main technical contribution.
Based on the proposed hypothesis, we devise a novel randomized channel shuffling mechanism which introduces minimal computation cost to reveal trigger features. Specifi- cally, given an unreliable dataset and a model trained by it, we control the channel shufiling strength and observe how representations vary according to it. By observing the abnormally different variation, we can detect the target class with only a few feed- forward passes. addition, we further utilize the proposed mechanism to separate those clean image features and trigger features. Specifically, we retrain the model using the channel shuffling operation with part of the training set for few epochs (0.25% of total training cost and then recover triggers without clean data based on the shuffled model. The trigger can be directly recovered to the pre-identified target class thus bringing efficiency.
Feature Differentiation:
We summarize our novel contributions as follows:
We discover that clean image features and trigger features are different in terms of their activation distribution in channels, as well as their sensitivity to channel re-ordering. Based on that, we propose an effective, generalizable approach based on channel shuffling to detect backdoor attacks and recover triggers with minimal overhead.

---

We explore a new thead-to-toe clean data- free setting, where neither backdoor detection nor trigger recovery requires any access to clean data from the training domain. Our channel shuffling--aase detection can directly obtain the target class without model retraining. The resultant shuffled models also facilitate more effective trigger recovery, also free of clean data.
Across various datasets (CIFAR-10, 10, GTSRB, Tiny ImageNet), architectures (AlexNet, ResNet- 20, 18N and backdoor attacks (BadNets[]], clean label attack [2]. and WaNet [3), the experiment results consistently endorse the effectiveness of our technique in backdoor model detection and trigger recovery, specifically, 0.331 ~ 0.640 higher AUROC than NC [14], and 0.291 0.543 higher than STRIP [8]. Our trigger recovery method also achieves up to 30% improvements in attack success rate for trigger recovery than NC [4] across all scenarios.
2 Related Work
Backdoor Attack and Defense. Several backdoor attacks on deep learning models have been proposed recently. Among them, most backdoor attacks can be classified as trigger-driven attacks. Specifically, during the training phase, to launch an attack, the adversary injects an arbitrary trigger into a small fraction of training pictures and maliciously labels them to the target class. Traditional
triggers are pre-defined patterns that can be simply stamped on benign images i 4 Many advanced trigger injection methods are proposed recently 6 3 for stealthiness under both human inspection and former detection methods 11
1.5 [8]. Another category of backdoor attack clean-label attack 2|225] 26 27 28 29], which is formulated without control over the labeling function. Among them, [26] targets large-scale datasets and scales up the poisoning attack method via gradient matching. Additionally, both [25] and [27 focus on improving the transferability of attack methods, while (28)
utilizes meta-learning and proposes an effective and transferable poisoning method.
Earlier works detect and mitigate backdoor models based on abnormal neuron responses g 10, 17,, feature representation (9. entropy [ and evolution of model accuracy [1].
19 models the distribution of Trojan attacks by meta neural analysis, [16] utilizes Deep k-NN with a focus on clean-label attack, 200 tries to learn the Universal Litmus Patterns (ULPs) from poisoned datasets.
15 17 13 II 211 focuses on recovering the trigger and detecting backdoor models based on the properties of the recovered triggers. Utilizing clean testing images, Neural Cleanse (NC) [14] calculates minimal perturbation to cause misclassification towards every putative incorrect label. Backdoor model detection completed by MAD outlier detection, due to the observation that the size of minimal perturbation to classify images from all classes to the target class is markedly smaller than that of other classes. However, the aforementioned methods suffer from occasional failures in detecting backdoor models. More importantly. most methods except [I7] that fall in this category generally assume the defender has clean validation set, which may not be practical. Instead, we do not assume we can obtain clean validation set for effective detection.
Another category of work 14
We also clarify the difference between our new setting and the more classical backdoor detection setting as adopted in [30 16 31). Those prior arts perform pre-training detection of backdoor training samples, without needing clean validation sets either. They all follow the same workflow: first detecting the poison examples in the training set, then removing them prior to training, and finally training the model using the filtered set. As the downside, to find out whether pre-trained model has been poisoned or not, those methods also have to (re)train from scratch, which may often be too costly or impractical. In contrast, [18] and our method belong to post-training detection that can detect from given pre-trained model without costly (re- ee-)ttraining from scratch. That is more practical when the model is already actively deployed in the field. While 18] needs a clean "reference" set, our method
requires no prior-training sample selection and tre-)training (hence differing from 30 16 31 climinating the necessity of clean samples (hence differing from [88).
while
Channel Shuftling. The idea of channel shuffling has mostly been used in efficient model design, intending to enable information flow between branches and consequently strengthen representation. Specifically, aiming at high efficiency, (32 and (3] utilize the channel shuffling method to mitigate
the problem induced by group convolution 34 35 Similarly, [36] includes channel split and shuffle
methods to reduce computation cost while maintaining higher segmentation accuracy. [37] includes channel shuffling to enable information communication between different sub-features. Different from the aforementioned works, (38) adopts the channel shuffling method with randomness as a regularization technique, to reduce overfitting and enhance generalization capabilities.
3

---

Ned
Clean Amack
WaNer
60 Layer
12 16
Figure 2:
Green and orange columns represent the pereentages of activated channels in each layer when inputting clean images and backdoor images, respectivcly. We empirically set the = 20% of max activation level as the threshold and consider channels with higher activation levels than the threshold as activated.
Based our observation that clean image features and trigger features have different channel distri- butions which is shown in Section 3.1] we formulated the randomized channel shuffling mechanism reveal trigger features, aiming detect backdoor models and recover triggers. To the best of our knowledge, we are the first to connect channel shuffling and the feature differentiation goal.
Shortcut Features. Several recent works 39
analyzed that trigger features can serve as "shortcut" features [5], which are observed to be easier to learn than the discriminative clean image features [4I]. In addition, [10] observed that a small number of neurons are activated by backdoor images but remain dormant for clean images. In contrast, [2] demonstrated the importance of the combination of channels when encoding complex and diversified image features on large datasets.
40
3 Methodology
In Section 3.1 we first present our main hypothesis on trigger features' unique characteristics compared to clean image features, alongside with experimental validation. Section [3.2] proposes our randomized channel shufffing method for backdoor model detection, as motivated by the hypothesis. Section 3.3 shows further how we can improve trigger recovery with the channel shuffling scheme.
3.1 Differentiating Trigger and Image Features: A Hypothesis
Authors of [5] pointed out that DNNs tend to unintendedly exploit features unrelated to the task of interest to achieve good performance. These "shortcut" features are usually simpler and easier to learn [41] but highly correlated to the task-related features, such as the sky background and the bird objective. is then verified in [39 40) that the backdoor triggers play a similar role as shortcuts exploited by the backdoored models.
Moreover, a small fraction of channels are observed to remain dormant for clean images but are activated by backdoor images [0] Based on all these observations, we conjecture that the trigger features have key difference from clean image features in terms of channel-level activation, and raise the following hypothesis: Trigger features are sparsely encoded and activated in only few channels, while clean image features need to be encoded across many channels for effective classification.
To validate the hypothesis, we investigate the ac- rivation level of specific channel, defined as the average magnitudes of elements in its (output) fea-
Clas Iangee
Badabox Imago
Thenk Artwoeed Cleaed
Tombe Antuerd
ture map. In each layer, we consider channels with activation levels larger than ratio p of its highest activation level as 'activated'. We then observe how channels are activated differently for clean and backdoor images. In Figure [] we take the second last layer in a model backdoored by BadNets I as an example. When the input is backdoored, only one single channel shows an ex- ceptionally high activation level, leaving all other channels "unactivated" based on our definition. In contrast, a clean input has more balanced activations across all channels. In Figure 2 we have consistent observations on more advanced backdoor methods such as clean label attack 2 and transformation-bee based attack, WaNet [ Moreover,
Figure 1: Blue columns represent the detailed dis- tribution of activation levels of the second last layer. Red lines indicate the threshold for activated channels, 20% of the max activation level this figure.
see that the sparse encoding of trigger features is more obvious in the latter layers, which are known to encode more discriminate semantic features while the shallow layers usually encode histogram features.
3.2 Trigger Feature Exposure via Channel Shuffling
The next question is, what can be implied from the significant imbalance in the activation distribution of trigger features? Consider the encoding of poisoned image latter layer in the backdoored

---

Shufllus
Stufllas
Spp Forures
Iromesiaae Ferurci
X
Oripual Fentuees
Figure 3: The overview of randomized channel shuffling operation. Before shuftling, the convolutional weights and feature maps are 'matched' in terms of channels (e.g. the input feature with a hole in the shape of a polygon is supposed to encode by the polygon weight), After channel shuffling, we deliberately induce mismatch (the input feature with a hole in the shape of a polygon has to be encoded by the star-shaped weight), interfering with the feature encoding process and generating meaningless output features.
model, where only as few as one channel is vital with a high activation level. This channel needs to match well with convolutional weights in the next layer for the attack to succeed. However, if we deliberately create a mismatch between them, the lack of any second "vital channel" will lead to unsuccessful encoding with high probability. In contrast, channel features of clean images are activated more evenly, leaving a lot of backup channels to substitute when a mismatch is created. Motivated by this insight, we design randomized channel shuffling mechanism to manually induce a mismatch between the activated channels and channel weights, and propose a way to observe how much the encoding process is being affected, thus revealing trigger features.
Randomized Channel Shuffling with Controlled Strength. Consider the convolutional weight Ø at the h layer of shape C. x Ci x F x F², where C.ut and Cj, are the number of output channels and input channels, respectively; F' is the kernel size. We shuffle the order of imput channels (the second dimension) of 0². And the overview of randomized channel shuffling is illustrated in Figure
Let θ = =00 (0'....,0'' 0b} denote the set of all weights in the network. We create a shuffled version of itself, (n) by randomly shuffling the last n convolutional layers, 0, 1,.... I. 1, since trigger features show more obvious sparsity characteristics in the latter layers according to Section 3. We control the strength of channel shuffling by setting different n values for shuffling; the larger n is, the stronger channel shuffling is applied. Then, for a given input æ, we calculate its representation (the output of the last convolutional layer) using ② and O(z), denoted as fe fe(x) and fa))) respectively, where f is the representation extraction part of a model. Finally, we compute the representation shift sn using the &2 f2-norm distance: s, (z) = 11 fe fe(.) (x) 112.
Detect the Class-wise Abnormal Variation. First, for a dataset D with K class, we split it into K subsets according the labels. For DR. the subset with class k, we calculate the standard deviation of representation shift caused by the shuffling operation on the last n layers, over all samples in that subset, denoted by (n) std 2.ED. (8%, (2:)). Then, by changing n. we can obtain curve y*10 : n) for class is showing the variation of the standard deviation of representation shift versus different strengths of channel shuffling. Finally, We can detect the target class by observing the abnormal curve, and its rationale is rooted in our sparsity hypothesis in Section 3.1 Note that using the mean of representation shift instead of using standard deviation still works for detection, and results are provided in Appendix [3.1] We fix (maximum layer number to shuffle, starting from the top/last layer) as four, and we analyze the choice in Section 4.4
To describe how a specific curve y²10 nı] deviates from cluster of curves {3r n],jj O,..., K - 1} and how to detect an anomaly, we formulate the following two metrics:
(i) Distance: We use the & distance between the curve and the median to measure how far the curve away from the cluster, formulated as o y [0 n| - median, {y'(0 n)), j O...., K 1} Il1.
(ii) Tendency: First, preprocess each curve to achieve 'zero-mean' and get g [0 n]. Then, calculate the standard deviation of the differrnnce between the curve and the median. To be specific, a = std, 80 n] median, (06
[0 n].j O,... K - 1} }.
Then, we adopt & = T. + A & to take both deviation in distance and tendency of a curve into account, and À is to balance two factors. A larger value of L implies that its corresponding curve less similar to others and more likely to be the target class. We default to set À as 0.01 in our experiments, and we conduct an ablation study about the effect of different À values in Section 4.4 showing the effectiveness of the proposed method is not sensitive to the choice, After calculating a dev for each curve, use Median Absolute Deviation to detect outliers, which is common choice
5

---

for numerous backdoor detection methods[[]] [5]. An anomaly index can be obtained for each curve, measuring the probability of the curve being an outlier and relating to the target class. By properly setting the threshold, the target class and backdoor model can be detected. In Section we measure the detection ability of our proposed method by AUROC. In addition, our method only needs a few foed-forward passes, showing exceptional efficiency compared to existing methods. More details can be found in Appendix 4.5
3.3 Improved Trigger Recovery with Shuffled Models
In this section, we improve the challenging trigger recovery by leveraging the distinctive properties of channel-level feature distributions from clean images and triggers. Specifically, we first design a training procedure with randomized channel shuffling destroy clean image features while retaining trigger-related features. Such models perform terribly for predicting clean images, but can be easily attacked by samples with a trigger (i.e., almost perfect ~ 100% attack success rate). Then, this "impaired" model is adopted for the subsequent trigger recovery where substantial performance improvements can be observed without access to clean data.
The rationale lies in: (1) Based on our verified hypothesis in Section 3.1
trigger features are sparsely encoded while clean image features encoded across multiple channels, randomized channel shuffling alters the channel order every iteration, damages the relationship among channels, and therefore stops the model to encode meaningful clean image features. On the contrary, trigger features are more amenable to the perturbed channel orders thanks to their sparse encoding manner. (2) by disentangling trigger and clean image features, the channel-shuflld models (where clean image features are supposed to have been destroyed) should be more tractable for trigger recovery since they contain almost "pure" trigger features now. This intuition is echoed by recent findings in [3]
Training with Randomized Channel Shuffting. Given a backdoor model G. the randomized channel shuffling is applied at each training iteration and produces its shuffled variant & in the end. We find that a few training epochs (e.g.. 5 epochs) on a subset (e.g., 10%) of the original training set, are sufficient to generate a high quality G with low clean testing accuracy and high attack success rate.
Enhanced Backdoor Trigger Recovery without Clean Data. In the backdoor model J, the presence of clean image features and the stcalthiness of trigger features make it hard to directly activate trigger features when users lack knowledge of the trigger patter.. Previous approaches typically rely clean images to activate clean image features, helping to reveal trigger features based on the connection between trigger features and clean image features 14] In contrast, I has enriched trigger features with suppressed clean image features, making the trigger feature exposure much easier, which also removes its dependency on clean data as the reference. Therefore, on top of previous approaches, we improve its trigger recovery performance by utilizing ğ. In practice, we use noise images drawn from the standard Gaussian distribution to substitute clean data. And the recovered triggers have similar high quality with clean data, as shown in Section 4.3 Additionally, we introduce enhanced objective function as depicted below:
Classical
Our Term with Shullldd Models
L(x, Ve) = =[xe(9(=') ve)+
(xe(9((') Mc)
where x' = x @ (1 - M) + M A, x is the randomly generated noise image and ye denotes the target label detected in our previous stage (Section 3.2). (xE(-) refers to a cross-entropy loss, A represents a recovered trigger pattern, and the mask M characterizes the region that the trigger stamped on. Here M is binary mask, belonging {0, 1 1]dxd, where d x d denotes the resolution of input images.
4
Experiments
4.1
Implementation Details
Trojan Details. We consider three representative backdoor attacks from two categories in our experiments, including (i) traditional backdoor attacks, i.e., BadNets [I] and WaNet [3 (ti) clean
label attack (CLA) 44 Among them, WaNet is more advanced which is designed to be more stealthy We have also compared with [45 in Appendix A3.2]
to human inspection
For BadNets M. we adopt 5 x 5 gray-scale triggers. And 10% of the full training samples are injected with the trigger. More details about the trigger injection and the trigger pattern are provided in Appendix AT.TT] And for CLA (2), we craft all training images from the target class and stamp the gcay-scale trigger. We follow the default hyperparameter configurations in (3), in which 10% of total

---

training images are warped. Also, it is worth mentioning that we adopt the "noise mode" of WaNet, which is reported to be more stealthy and challenging.
Datasets and Architectures. We evaluate our randomized channel shuffling method across various backdoor attacks, datasets and architectures with a total of seven combinations, i.e., [BadNets, CIFAR- 10, 2es, [BadNets, CIFAR-I0, AlexNet), (BadNets, CIFAR-IO, SENet-18). (BadNets, GTSRB, ResNet-20), (BadNets, Tiny ImageNet, ResNet- [CLA, CIFAR-10, ResNet-20] and (WaNet, CIFAR-10, 10, ResNet- 18). For each combination, we train 25 backdoor and 25 benign models via independent runs, Training details are collected in Table [
Table 1: Detailed training configurations of the backdoor injection procedure.
Attucks
# Epuch
Learning Rute
Optimizer
Batch Size
Target Label
Poison Ratio
Cross Ratio
BadNets 1
200
SGD SGD SGD
"0"
0.1
Clean Label Attack
200
0.1
128
"0"
0.1
WaNet
3
200
0.01
128
"0"
0.1
0.8
Detection and Evaluation Details. D Detection. To examine the reliability of the training dataset with N classes, we first divide it into N subsets based on their label. For each subset, we then feed them into the target model as well as its randomly shuffled variant, and compute the associated repre- sentation shifts over different numbers of shuffled layers. Based our observations in Section 3.1 the last few layers mainly encode discriminate features and therefore they are used in our detection. In our implementation, we only shuffle the channel order within the last four layers and generate feature sensitivity curves as {2/ [m], A = O,... N - 1, 0, 2, 3). y'(i] denotes the standard deviation of representation shift from the subset of class i after randomly shuffling the last j convolutional layer. Note that the range of the number of layers used for our detection plays a minor role in the achievable performance, as evidenced in Section 4.4 Lastly, we utilize the metric Pdvv shown Section 3.2 to measure whether there exists an outline curve significantly different from other curves. An abnormal curve usually indicates: i) the dataset and model contain backdoor triggers; ii) its corresponding class is likely to be the backdoor target class. >> Evaluation. Similar to [17], we use area under the receiver operating characteristic curve (AUROC) to measure the backdoor model detection's s performance. More experimental details are provided in Appendix A2
4.2 Backdoor Detection by Leveraging Abnormal Variation
Superior Performance across Diverse Datasets and Architectures. We verify the effectiveness of the proposed detection algorithm with the randomized channel shuffling method on seven rep- resentative combinations of datasets, network architectures, and attack methods. Detection's 's ROC results are presented in Figure 4 where orange, green, and blue curves are generated by Ours, Neural Cleanse [14], and STRIP [8]. respectively. Consistent performance improvements are observed: ① Across Architectures: Compared with NC and STRIP, our method achieves an improvement of AUROC values CIFAR-10 with BadNets by (0.331, 0.375, 0.339) and (0.541, 0.543, 0.306] for architectures [ResNet-20, AlexNet, SENet-18), respectively. @ Across Datasets: On GTSRB and Tiny ImageNet, our approach consistently outperform NC by a substantial margin of 0.539 and 0.388 AUROC values, with 0.293 and 0.457 improvements compared to STRIP. & Across Attack Methods: Even for more advanced attack methods CLA and WaNet, our methods still gain 0.362, 0.291, and 0.64, 0.512 AUROC improvements on CIFAR- 10 against NC and STRIP.
For further evaluation, we calculate the percentage of successfully detected models with a fixed threshold. And we choose the threshold that maximize the total detection accuracy, i.e., (# correctly detected benign models and buckdoor models) / (# total models). As shown in Table 2 in each combination of ddtaset, architecture, attack method), our detection method consistently outperforms other competitive approaches by up 37. 37.72% detection accuracy.
Feature Sensitivity Curves and the Distribution of Indicator & We present the feature sensitivity curves and the distribution of indicator aev in Figure Figure respectively. Without loss of generality, for each attack, we randomly select one model for further analysis. Each point in dienoees the standard deviation of feature shifts between the original and randomly shuffled models, given a subset of input images from a certain class. After we vary the number of shuffled layers, a feature-sensitive curve can be drawn. From the results in Figure 5 we see the standard deviation of feature shifts on the target class's 's samples appears as a distinctive pattern (red curves), especially for the first few points, which lies the foundation of our detection algorithm.
7

---

Table 2: Comparisons between ours and classic detection approaches. The reported numbers represent (# correctly detected models) / (# models). Note that "CLA' stands for the clean label attack.
Attack
Dataset
Arch
Neural Cleamse
STRIP
Ours
benign
backdoor
AUROC
benign
backdoor
AUROC
benign
backdoor
AUROC
RosNet-20
14/25
20/25
0.069
18/25
8/25
0.450
25,255
26/25
1,000
BadNets
CIFAR-10
AlexNet
19/26
13/25
0.622
16/25
11/25
0.454
24/25
26/25
0.907
SENot-18
15/25
18/25
0.578
12/25
13/25
0.611
23/25
21/25
0.917
GTSRB
RosNet-20
24/25
2/25
0.341
12/25
21/25
0.587
23/25
21/25
Thn ImaggeNet
RosNet-18
19/25
15/25
0.612
11/25
13/25
0.543
25/25
26/25
1,000
CLA
CIFAR-10
ResNet-20
14/25
20/25
0.635
20/25
15/25
0.206
24/25
25/25
0.997
WaNet
CIFAR-10
18
25/25
1/25
0.314
25/25
1,25
0.442
21/25
20/25
0.954
Total
180/1775
89/150
114/176
84/175
165/175
165/175
(Reckes-33)
0T37 Voor 1400
ST52 (1
STRPP T7EP
Dvee Sa10
A5 2.001
CS
Figure 4: Performance of existing and our detection approaches, measured by ROC and related AUROC values.
On top of it, we compute the indicator Paev to quantitatively measure the deviation from the majority of curves. Then, in order to locate the abnormal curve, the MAD outlier detection [46] is performed to &dvv. T that the dey of the target label tends to have significantly large values compared to the ones of benign labels, which allows the target label to be casily identified.
4.3 Improved Triggers Recovery by Models Trained with Randomized Channel Shuffling
Table 3: TA (%) and ASR (%) after randomly shuffling channels on various
combinations.
Attack
Dataset
Arch
Type
ASR
Staffidd Modil in Noue Immaci (Dun))
ResNet-20
Orig.
91.4:0.33
100.0 1 0.0%
Staffied Mivil Clioo Inaget
Origina Modeiv Noinc Imago
CIFAR-10
Shuf.
16.2 1. 7.3%
100.0 1. 0.0%
Altac (AA: N)
Original Modalv Claen Image
BadNets
AlexNet
Orig.
0.5%
99.9 0.1%
100
Shuf.
45.4 6.7%
95.4 + 4.6%
80
GTSRB
ResNet-20
Orig.
0.5% 98.5
100.D 0.0%
Shuf.
25.9 1: 17.4%
98.4 : 1.6%
CLA
CIFAR-10
ResNet-20
Orig.
Shuf.
14.36 6.1%
83.3
0.4%
99.2 = 0.8%
90.. = 8.0%
40
20
WaNet
CIFAR-10
ResNet-18
Orig.
93.1 + 0.44%
99.2 + 0.8%
Shuf.
10.2 2.3%
97.9 + 1.8%
11 211 2V
Training with Randomized Channel Shuffling Leads to Low TA and High ASR. We first investigate how the pro- posed training procedure with randomized channel shuffling shapes the behaviors of backdoor models. Results of TA and ASR are collected in Table 3 where "Orig." " and "Shuf." indicate the original and shuffled models. Our proposal ef- fectively produces models with nearly unimpaired ASR (c.g.. 90 ~ 100%) but poor TA which closes the level of random guessing. Such consistent findings can be drawn from all ddataset, architecture) combinations. It again supports our claim that randomized channel shuffling tends to destroy clean image features while preserving trigger features.
Figure 7: Attack Success Rate (ASR %) of recovered triggers from "original model" (G) and "shuftled model" (0), us- ing clean and randomly generated noise images inputs respectively. I. II, III, IV and V represent different backdoor configurations, i.e. (BadNets, CIFAR- 10, ResNet-20), (BadNets, CIFAR-10, AlexNet). (BadNets, GTSRB, ResNet- 20), (CLA, CIFAR-10, ResNet-20) and (WaNet, CIFAR- 10, ResNet-18).
Superior Trigger Recovery. To efficiently generate high-quality triggers, it requires two key factors: (1) detected backdoor models and identified target labels, obtained from our detection algorithms; (2) shuffled trained models that have rich trigger features but less clean image features. As evidenced in Figure 7 while previous methods can achieve acceptable results when assuming access to clean
8

---

Benian
BadlNes
Clean Label Anack
WaNet
Suasked Devation
16
12
Layer
Layer
1ayer
Figure 5: Standard deviation of feature shifts after applying randomized channel shuffling operation over the number of shuffled convolutional layers. Red curves generated from subsets of the target class, while green curves are calculated from subsets of benign classes.
1.0
0.6
0.6
0,3
X
0.4
X
X
X
X
$0.5
0.2
0.2
0.2
0.0
BadNeis
0.0
BadNeits
0.0
BadNees
0.0
Clean Label Attack
WaNet CIFAR-I0 (Ateshet-13)
CIFAR-10 (RtesNet-20)
CIEAR-I0 (AlexNet)
LIEAR-UUCTTSREE
CIFAR-IU (ReshNtt38)
Benign Label
X Target Labet
Figure 6: The Die distribution for backdoor models. Boxplots show the means and variances of 4an of curves related benign classes. The red crosses stands for 4aawS of curves for target classes. larger value of Deev implies that its corresponding curve similar others, and thus more likely to the target class.
images, the performances drop tremendously when only using noise images. In contrast, even with noise images, our method can recover as high-quality triggers as with clean images, significantly surpassing the original models. Details are provided in Table We perform experiments 8 2080Ti GPUs. For fair evaluation, we binarize the masks and make their & norms equal the ground-tre values (5 x 5). Then, calculate the attack success rates with the binary masks.
Table 4: Detailed configurations of trigger recovery methods.
Stage
# Epoch
Learning Rate
Optimizer
Batch Size
Num. of Trainset
Num. of Cleanset
Retrain Recovery
SGD
5120
200
SGD
1000
4.4
Ablation Study
Different Trigger Types in BadNets. Instead of fixing the trigger at the lower right corner of the image, we change the trigger location the upper left side, and randomly choose the location of the trigger to generate two types of poisoned datasets with different stamping location strategies. We further change the gray-scale trigger into the RGB trigger. We use ResNet-20 and CIFAR- in the above settings. The AUROC values are reported in Table 5 which demonstrate the effectiveness of proposed randomized channel shuffling method across multiple trigger positions and types.
Table 5: The detection performance (AUROC) of ablation studies.
Ablation
Different Trigger Types
Multiple Target Classes
Partial Backdoor
Type
Bottom
Random Location
RGB Trigger
Targets
Targets
Targets
Targets
AUROC
1.000
1.000
1.000
0.978
0.981
0.902
0.770
0.945
Multiple Target Classes. We further consider the scenario where multiple backdoors are inserted into the dataset, with more than one class as the target class. Using CIFAR-10 as the dataset which has ten classes, ResNet-20 as architecture, and BadNets as the attack, we find that our method is still effective when multiple target classes exist but target classes still being the minority compared to the benign classes. AUROC values are reported in Table 5 Detection performances remain acceptable when fewer than 5 classes are infected. But AUROC drops significantly when 5 out of 10 classes are the target classes. More visualization results are provided in Figure
Source Label-Specific (Partial) Backdoor. A 'partial' backdoor is activated only when inputs are from specific source labels, which more stealthy than a non-partial backdoor. We then verify the effectiveness of our method under this scenario with BadNets [ as the attack, ResNet-20 as the model architecture, and CIFAR-10 as the dataset. The results are provided in Table showing the effectiveness of the proposed method in the partial backdoor scenario.

---

x
Sunder De vakce
Bideen CEAS- PUPN--5
M
Judfes CTSD An
G.A. COI.ETIAA (1
Laysr
12
Targets
Target
30c
XXC
Bod6n, CRVS. Bacte.i CENRAA AAIO CLA CERSIIE
022
Laye
Figure 8: The left two columns provide visualization results when multiple backdoor target classes exist, each attack scenario, the distribution of Qas is collected in the left sub-figure, and its standard deviation of representation shifts is reported in the right sub-figure. The upper right figure provides AUROC values over different X, the bottom right figure shows AUROC values over the number of shuffled layers.
Choice of Hyper-Parameters. As is mentioned in Section 3.2 d D + A 4, where À is to balance two terms. Figure 8 shows the variation of AUROC when changing A. In addition, we also verify how the number of layers included for detection affects the detection performance for different attacks. According to Figure 8 curves of AUROC remain high when only the last several layers are included. But they gradually decrease when including more layers, since the standard deviation of representation shift from the target class becomes mixed with benign ones, affecting the performance.
4.5 Efficiency of Proposed Detection Method
Compared existing methods, we claim that proposed detection method is more efficient. We approximate the FLOPs of the pack-propagation to twice that of forwarding propagation following|47 ]. and provide the total FLOPs needed for detection in Table 6 For CIFAR- 10, NC and STRIP need 7.5 10 and 15.6 times more FLOPs than ours, respectively. For GTSRB, NC and STRIP need 1.9 x 10 and 3.9 times more FLOPs than ours, respectively.
4.6 Efficiency of Proposed Trigger Recovery Method
claim that our trigger recovery method is also efficient since only 5 epochs and 10% of the original training set are used. This is motivated by the fact that the ASR and TA saturate after few epochs if a model shuffled and retrained on a full dataset according to the training curve in Figure 9
100
75
Aceurayy
Table 6: The approximate of FLOPs(x need for detection.
TA
ASR
Attack
Dataset
Arch
Neural Cleanse
STRIP
Ours
25
CIFAR-10
ResNet-20
5.04 10
1.05 102
6.72
BadNets
10
10
Epoch
GISRB
ReaNet-20
5.04 x
1.05 x
26.88
Figure 9: Test Accuracy (TA %) and Attack Success Rate (ASR %) when re- training with randomized channel shuf- fling on CIFAR-10 attacked by Bad- Nets, based on ResNet-20.
CLA
CIFAR-10
ResNet-20
5.04 x 10
1.05 x 10
6.72
WanNet
CIFAR-10
ResNet-18
6.84 x 10?
1.43 x 10
91.2
Conclusion
In this paper, introduce and investigate challenging backdoor detection setup without access to clean datasets. An efficient detection algorithm is proposed by contrasting representations shifts before and after randomized channel shuffling of clean and trigger features. Moreover, by leveraging our shuffled backdoor models, the performance of trigger recovery is substantially boosted. Extensive experiments validate the effectiveness of our proposal.
10

---

References
[1] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg, Badnets: Identifying vulnerabilities in the machine learning model supply chain, 2019.
[2] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. arXiv preprint arXtv:1804.00792, 2018.
[3] Anh Nguyen and Anh Tran. Wanet- imneetippeeee warping-based backdoor attack. arXiv preprint arXiv:2 103.6 2021.
[4] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks deep learning systems using data poisoning, 2017.
[5] Robert Geirhos, Morn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intellignce, 2(1 (116)666666.3 2020.
[6] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer, 2020.
[7] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11957-11965. 2020.
[8] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: /8/ 1.10:728/ 2018.
[9] Brandon Tran, Jerry Li. and Aleksander Madry. Spectral signatures in backdoor attacks. arXiv preprint arXiv: 0006,06611.006, 2018
[10] Kang Liu, Brendan Davit,, and Siddharth Garg, Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attaks, Intrusions, and Defenses, pages 273-294. Springer, 2018.
[11] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265-1283 2019.
[12] Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Machine Learning, pages 5739-5748. PMLR, 2019.
[13] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, pages 4658-4664, 2019.
[14] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, IEEE, 2019.
[15] Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly aceu- rate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint arXiv: 009900.07733. 2019.
[16] Neehar Peri, Neal Gupta, Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P. Dickerson. Deep k-nn defense against clcan-label data poisoning attacks, 2019.
11

---

[17] Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 222-238. Springer, 2020.
[18]
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks, 2020.
[19] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pages 103-120. IEEE, 2021.
[20] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301-310, 2020.
[21] Guangyu Shen, Yingqi Liu, Guanhong Tao, shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimiza- tion. arXiv preprini arXiv:2102.00 /23, 2021.
[22] Alvin Chan and Yew-Soon Ong. Poison as a cure: Detecting & neutralizing variable-sized backdoor attacks in deep neural networks, arXiv preprint arXiv: 1. 1.08040, 2019.
[23] Fereshteh Razmi and Li Xiong, Classification auto-encoder based detector against diverse data poisoning attacks. arXiv preprint 1082062108.04206, 2021.
[24] Zayd Hammoudeh and Daniel Lowd. Identifying training-set attack's target using renormalized influence estimation. arXiv preprint arXiv:2201. 2022.
[25] Chen Zhu, W Ronny Huang. Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clben poisoning attacks on deep neural nets. In International Conference on Machine Learning, pages 7614-7623. PMLR, 2019.
[26] Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching, arXiv preprint arXiv:2009.0222 2020.
[27] Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 159-178. IEEE, 2021.
[28] W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoi- son: Practical general-purposse clean-label data poisoning. Advances in Neural Information Processing Systems, 33:120800-1000 12091, 2020.
[29] Junfeng Guo and Cong Liu. Practical poisoning attacks on neural networks. In European Conference on Computer Vision, pages 142-158. Springer, 2020.
[30] Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
[31] Andrea Paudice, Luis GonzzzzZ Andras Gyorgy, and Emil Lupu. Detection of adversarial training examples in poisoning attacks through anomaly detection. arXiv preprint 10Xiv:1802.03041, 2018.
[32] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.
[33] Zhang Ting, Qi Guo-Jun, Xiao Bin, and Wang Jingdong. Interleaved group convolutions for deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
12

---

[34] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[35] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[36] Yu Wang, Quan Zhou, Jia Liu, Jian Xiong, Guangwei Gao, Xiaofu Wu, and Longin Jan Latecki. Lednet: A lightweight encoder-decoder network for real-time semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1860-1864. IEEE, 2019.
[37] Qing-Long Zhang Yu-Bin Yang. Sa-net: Shuffle attention for deep convolutional neural
networks, 2021.
(38] Sudhakar Kumawat, Gagan Kanojia, and Shanmuganathan Raman. Shuffleblock: Shuffle to regularize deep convolutional neural networks. arXiv preprint arXiv:2/06.09358, 2021.
[39] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Indiscriminate poisoning attacks shortcuts. arXiv preprint wrXiv:2111.00890 2021.
[40] Pedro Sandoval-Segura, Vasu Singla, Liam Fowl, Jonas Geiping, Micah Goldblum, David Jacobs, and Tom Goldstein. Poisons that are learned faster are more effective. arXiv preprint urXiv:220.008615, 2022.
[41] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346 8356. PMLR, 2020.
[42] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17Xv::1704.0486. 2017.
[43] Tianlong Chen, Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang. Quarantine: Sparsity uncover the trojan attack trigger for free. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[44] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv: 1912.0271, 2019.
[45] Neehar Peri, Neal Gupta, W Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P Dickerson. Deep k-nn defense against clean-label data poisoning attacks. In European Conference on Computer Vision, pages 55-70. Springer, 2020.
[46] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-39)) 1974.
[47] Dingqing Yang, Amin Ghasemazar, Xiaowei Ren, Maximilian Golub, Guy Lemieux, and Mieszko Lis. Procrustes: a dataflow and accelerator for sparse deep neural network training. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 711-724. IEEE, 2020.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[49] Peizhuo Lv, Hualong Ma, Jiachen Zhou, Ruigang Liang. Kai Chen, Shengzhi Zhang, and Yunfei Yang. Dbia: free backdoor injection attack against transformer networks. arXiv preprint arXiv:2111.11870, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] (Section A4)
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "dosRZyRzV5", "reviewer": "Reviewer_oFCu", "summary": "This paper casts a new \"self-supervised\" backdoor detection setting that was not explored by prior arts: there is no prior knowledge nor control of the training data quality; and no access to a comparable clean sample set. The authors presented several interesting observations and motivated an efficient backdoor detection approach for this new setting.", "questions": "Please check and answer the questions in \"Weaknesses\" section and the following questions:\n\n- How sensitive the algorithm is to the threshold hyperparameter?\n- Why use the last layers, and what if applying the shuffling to the first few or a random subset of layers?\n- Why WaNet is missing from Table 3 and Table 4?\n- Writing quality is overall not high, except for the abstract which appears to be well polished. The same polishing must be done towards the remaining paper.\n- “Feature Disentanglement” is used in this paper to refer to a completely different meaning from its common context (in GANs etc). Using another term such as differentiation is perhaps better.", "limitations": "Social impacts have been properly discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This paper seems to identify a practically meaningful and novel backdoor setting, as ubiquitous and non-certified data sources are used for training big models nowadays (yet, potentially suffering over-rigor, see weakness). \n+ The work has an empirical nature and has no theoretical foundation. However, the authors managed to dig some reasonable foundation to support their hypothesis, both from prior literature and from proof-of-concept experiments, so this is appreciated. The storytelling in Sections 3.1 and 3.2 help readers understand a lot.\n+ Fundamentally, their observation is rooted in the “shortcut” property of backdoor features (noticed by prior work), which could be why the backdoor features are more sparsely encoded in channels. However, leveraging this property via shuffling is an original idea, and Figure 2 gives out why. Just a thought: would this shuffling possibly be made into a more general purpose to probe a trained neural network’s stability, with little overhead?\n+ Empirical results appear to be extensive. Two datasets, two architectures, and three attacks are evaluated, including the advanced WaNet. The performance gains of 0.291 ∼ 0.640 AUROC over SOTA are impressive, considering its little overhead. Both backdoor detection and trigger recovery are considered. \n\nWeaknesses:\n\nMy main concern is whether the authors’ proposed new setting is over-restricted or unrealistic at all. Specifically, the authors verbally claim that online public datasets are not helpful because in domain-specific applications they can be out-of-distribution. They might be true, but never manifested in practical experiments, so I’m not convinced. For example, what if one just uses a clean set from ImageNet as clean data surrogate to detect backdoors? Please justify with more concrete evidence or experiments. \n\nEven admitting this setting, I feel more methods are available. For example, what if first performing self-supervised pre-training, then noisy student training? Many semi-supervised and active learning approaches could have been discussed, and some might potentially be more accurate than the proposed one. However, I do acknowledge one possible merit of the authors’ work, in generating a very lightweight baseline that requires no repeated training. However, this new method in a new problem setting needs clearer contextualization for sure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_VVTrEV0IP1", "reviewer": "Reviewer_PjHg", "summary": "This paper proposes a backdoor detection method based on shuffling convolutional filters and assessing the resulting shift in deep representations. Experiments are presented on two datasets, two architectures, and three backdoor attacks.", "questions": "- How does this method work on recent neural network architectures?\n- How does this method fair on datasets that are larger, i.e. ImageNet?\n- How does this defense perform against stronger poisoning attacks?", "limitations": "The limitations are described in Section A4. There is very little description of ways in which this work on data security and model vulnerability is applicable (or not applicable) to modern architectures and real world data. In other words, a paper that purports to defend against any kind of attack should probably explain when and where this defense will work in practice. In my opinion, discussing limitations is critical for work on security and privacy. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- To the best of my knowledge, this is a novel approach wherein filter shuffling changes the representation.\n- There is an interesting method, and the results support the hypothesis in Section 3.\n\nWeaknesses:\n- Novelty: The abstract makes big claims (in bold) that may not be true. There are existing defense techniques that do not require clean data. Some papers that should be cited and methods to compare with the proposed method (others may exist):\n    + Steinhardt, Jacob, Pang Wei W. Koh, and Percy S. Liang. \"Certified defenses for data poisoning attacks.\" Advances in neural information processing systems 30 (2017). (https://arxiv.org/pdf/1706.03691.pdf)\n    + Paudice, Andrea, et al. \"Detection of adversarial training examples in poisoning attacks through anomaly detection.\" arXiv preprint arXiv:1802.03041 (2018). (https://arxiv.org/pdf/1802.03041.pdf)\n    + Peri, Neehar, et al. \"Deep k-nn defense against clean-label data poisoning attacks.\" European Conference on Computer Vision. Springer, Cham, 2020. (https://arxiv.org/pdf/1909.13374.pdf)\n- This work considers relatively outdated architectures and weak attacks. Some stronger poisoning and backdoor attacks (more exist):\n    + Geiping, Jonas, et al. \"Witches' brew: Industrial scale data poisoning via gradient matching.\" arXiv preprint arXiv:2009.02276 (2020).(https://arxiv.org/pdf/2009.02276.pdf)\n    + Zhu, Chen, et al. \"Transferable clean-label poisoning attacks on deep neural nets.\" International Conference on Machine Learning. PMLR, 2019. (http://proceedings.mlr.press/v97/zhu19a/zhu19a.pdf)\n    + Aghakhani, Hojjat, et al. \"Bullseye polytope: A scalable clean-label poisoning attack with improved transferability.\" 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2021.\n    + Saha, Aniruddha, Akshayvarun Subramanya, and Hamed Pirsiavash. \"Hidden trigger backdoor attacks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 07. 2020.\n- Missing related work. Some of the papers above have not even been cited. Some other papers probably should be cited. I'll point to a couple survey papers here:\n    + Ahmed, Ibrahim M., and Manar Younis Kashmoola. \"Threats on Machine Learning Technique by Data Poisoning Attack: A Survey.\" International Conference on Advances in Cyber Security. Springer, Singapore, 2021.\n    + Goldblum, Micah, et al. \"Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).\n- The writing is often unclear with many grammatical errors. Some examples:\n    + Line 67 says that the paper will focus on backdoor attacks. Presumably, this is as opposed to poisoning attacks without a trigger. However, one of the three attacks is CLA, which has no trigger in the original method.\n    + Line 30 says \"poinson.\"\n    + Line 97: \"Specifically, we retrain the model using the channel shuffling operation, and then recover trigger based on the shuffle model...\" Perhaps this should be \"... recover the trigger...\" or \"... recover triggers ...\".\n    + Line 52 in the appendix says \"We will future extend to this case.\" \n\nOverall, I think there are interesting experiments, but there are problematic claims and missing relevant works/benchmarks. In my opinion, this paper is well below the threshold of acceptance in the state it is in. Short of showing that the proposed method does, in fact, beat the state of the art in poisoning detection/defense and better contextualized this method in the existing work, I recommend rejection.\n\n\n----\n\nAfter reading the authors' response and the updated submission (August 7):\n\nMy major concerned have been largely addressed. Thanks to much improved clarity around the setting, better contextualization with existing work, and additional experimental results, I can say this paper has much improved. I have increased my score accordingly and followed up with the comments below. \n\nAs a minor note, the discussion on limitations is still lacking.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WAK_xSL09o", "reviewer": "Reviewer_AHdC", "summary": " In this paper, the authors introduce and investigate a new backdoor detection setup without access to clean datasets. An effective detection algorithm is proposed by contrasting representations shifts before and after randomized channel shuffling of clean and trigger features. Moreover, by leveraging the proposed shuffly trained backdoor models, the performance of trigger recovery is substantially boosted. Extensive experiments validate the effectiveness of our proposal.", "questions": "Cons:\n\n- Can we illustrate the trigger feature? It would be helpful to understand this key concept.\n\n-  The trigger feature lacks a good mathematical definition, making further theoretical analysis hard.\n\n- What is the false alarm rate of the proposed method? Namely the type I error in the field of hypothesis testing.\n\n- Can the proposed method be used for adversarial data detection? Is that possible to distinguish between two attacks?", "limitations": "no potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ This paper is novel and introduces an interesting finding for backdoor attack detection. Specifically, it is the first time to try to detect such attacks without any clean data. Since we do not need clean data here, the proposed method can be used for many scenarios compared to previous methods.\n\n+ The whole paper is easy to follow, and the main contributions are introduced clearly. The trigger feature hypothesis is very interesting and worthy to study more in the future.\n\n+ The authors raise that the unreliability of training data source has become a serious security threat. They find that clean image features and trigger features are different in terms of their activation distribution in channels, as well as their sensitivity to channel re-ordering.\n\n+ Utilizing trigger features’ unique characteristics, this paper proposes an effective, generalizable approach with minimal overhead based on channel shuffling to help data users examine datasets. The authors randomly shuffle weights of convolutional layers at the channel level, revealing the secretly hidden trigger features. This paper also uses the proposed shuffling method to retrain the models, helping generate more effective triggers.\n\n+ Extensive experiments are used to verify the proposed methods.\n\nCons:\n\n- Can we illustrate the trigger feature? It would be helpful to understand this key concept.\n\n-  The trigger feature lacks a good mathematical definition, making further theoretical analysis hard.\n\n- What is the false alarm rate of the proposed method? Namely the type I error in the field of hypothesis testing.\n\n- Can the proposed method be used for adversarial data detection? Is that possible to distinguish between two attacks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AOPa77iOyr-", "reviewer": "Reviewer_raZy", "summary": "This paper proposes the backdoor defense even without a clean data set. Specifically, after randomly shuffling the filters and comparing the changes in DNN output, the authors find there is an obvious difference between backdoored DNNs and benign DNNs. Futher They propose some metrics to detect whether a given model is backdoored or not. The experimental results demonstrate the effectiveness of the proposed method.", "questions": "For all-to-all backdoor attacks (samples from different classes are misclassified into different classes in the presence of the trigger pattern), does the proposed method still work?", "limitations": "Yes. This paper tries to provide a novel backdoor defense, which may mitigate the potential risk of backdoor attacks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. When backdoor attacks bring practical threats to deployed DNNs, it becomes urgent and important to explore the backdoor defense. This paper has a good try.\n2. To the best of my knowledge, the proposed method is novel and intriguing. I have never seen such a method, i.e., shuffling the filters, in backdoor defense.\n\n**Weakness**\n1. The method seems only to support the all-to-one attack, that is, all samples are misclassified into the same class in the presence of the trigger pattern.\n2. This paper is a bit difficult to read and follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept