Distilling Cognitive Backdoor Patterns within an Image
Authors: Hanxun Huang
Xingjun Ma
Sarah Monazam Erfani
James Bailey
Venue: ICLR 2023 poster (2023)
Keywords: Backdoor sample detection
Backdoor defence

Abstract: This paper proposes a simple method to distill and detect backdoor patterns within an image: \emph{Cognitive Distillation} (CD). The idea is to extract the ``minimal essence" from an input image responsible for the model's prediction. CD optimizes an input mask to extract a small pattern from the input image that can lead to the same model output (i.e., logits or deep features). The extracted pattern can help understand the cognitive mechanism of a model on clean vs. backdoor images and is thus called a \emph{Cognitive Pattern} (CP). Using CD and the distilled CPs, we uncover an interesting phenomenon of backdoor attacks: despite the various forms and sizes of trigger patterns used by different attacks, the CPs of backdoor samples are all surprisingly and suspiciously small. 
One thus can leverage the learned mask to detect and remove backdoor examples from poisoned training datasets. 
We conduct extensive experiments to show that CD can robustly detect a wide range of advanced backdoor attacks.
We also show that CD can potentially be applied to help detect potential biases from face datasets.
Code is available at https://github.com/HanxunH/CognitiveDistillation.

Published as a conference paper at ICLR 2023
DISTILLING COGNITIVE BACKDOOR WITHIN AN IMAGE
PATTERNS
Hanxun Huang' Xingjun Ma Sarah Erfani' James Bailey'
School of Computing and Information Systems, The University of Melhourne, VIC, Australia SSchool of Computer Science, Fudan University, Shanghai, China hanxunh@student unimelb edu. au; xingjunma@ fudan. edu. cny sa carah.ao ranni @unime 1b. edu. au; ba ley weneeennmmmlmmeeememmmeeeeemme edu. au.
ABSTRACT
This paper proposes a simple method to distill and detect backdoor patterns within image: Cognitive Distillation (CD). The idea is to extract the "minimal essence" from an imput image responsible for the model's prediction. CD opti- mizes an input mask extract a small pattern from the input image that can lead to the same model output (i.e., logits or deep features). The extracted pattern can help understand the cognitive mechanism of model on clean vs. backdoor im- ages and is thus called Cognitive Pattern (CP). Using CD and the distilled CPs, we uncover an interesting phenomenon of backdoor attacks: despite the various forms and sizes of trigger patterns used by different attacks, the CPs of backdoor samples are all surprisingly and suspiciously small. One thus can leverage the learned mask to detect and remove backdoor examples from poisoned training datasets. We conduct extensive experiments to show that CD can robustly detect a wide range of advanced backdoor attacks. We also show that CD can potentially be applied to help detect potential biases from face datasets. Code is available at httpp::/ com http://hht....oooo//o////o/ootta
INTRODUCTION
Deep neural networks (DNNs) have achieved great success in wide range of applications, such as computer vision (He et al., 2016a Dosovitskiy et al. (022)) and natural language processing (Devlin) al. 2019 Brown al. 2020). However, recent studies have shown that DNNs are vulnerable to backdoor attacks, raising security concerns on their deployment in safety-critical applications, such as facial recognition (Sharif et al. 2016), traffic sign recognition (Gu et al.) 2017), medical analysis (Feng et al.] 2022), object tracking (Li et al., 2022), and video surveillance (Sun et al.] (002). A backdoor attack implants backdoor trigger into the target model by poisoning small number of training samples, then uses the trigger pattern to manipulate the model's predictions at test time. A backdoored model performs normally on clean test samples, yet consistently predicts the backdoor label whenever the trigger pattern appears. Backdoor attacks could happen in scenarios where the datasets or pre-trained weights downloaded from unreliable sources are used for model training, or training a model on Machine Learning as a Service (MLaaS) platform hosted by an untrusted party.
Backdoor attacks are highly stealthy, as 1) they only need to poison few training samples; 2) they do not affect the clean performance of the attacked model; and 3) the trigger patterns are increasingly designed to small, sparse, sample-speciff or even invisible. This makes backdoor attacks hard to detect or defend without knowing the common characteristics of the trigger patterns used by different attacks, or understanding the cognitive mechanism of the backdoored models in the presence of a trigger pattern. To address this challenge, trigger recovery and mitigation methods, such Neural Cleanse (NC) (Wang et al.) 2019). SentiNet (Chou et al. 2020) and Fine-Pruning (Liu et al., (20 have been proposed to reverse engineer and remove potential trigger patterns from backdoored model. While these methods have demonstrated promising results in detecting backdoored models,
they can still be evaded by more advanced attacks (Nguyen & Tran 2020 Li et al. 2021c).
Cooresponding author.

---

Published as a conference paper at ICLR 2023
Several works have aimed to shed light the underlying backdoor vulnerability of DNNs. It has been shown that overparameterized DNNs have the ability to memorize strong but fask-irrelevan correlations between a frequently appearing trigger pattern and backdoor label (Gu et al. 2017) Geirhos et al. 2020). In fact, training on a hackdoor-poisoned dataset can be viewed as a dual-task learning problem, where the clean samples define the clean task and the backdoor samples define the backdoor task (Li et al. 2021a). DNNs can learn both tasks effectively and in parallel without interfering (too much) with each other. However, the two tasks might not be learned the same pace, as it has been shown that DNNs converge much faster on backdoor samples (Bagdasaryan & Shmatikov 2021 et
2021a).
Other studies suggest that backdoor features are outliers in the deep representation space (Chen et al., Tran al. 2018). Backdoor samples are also apputinvarrint, i.e., the model's prediction on backdoor sample does not change when the sample is mixed with different clean samples (Gao et al. 2019.. It needs a smaller distance to misclassify all samples into the backdoor class (Wang et al,
2018
2019 and backdoor neurons (ncurons that are more responsive to the trigger pattern) are more sensitive to adversarial perturbations (Wu & Wang 2021). While the above findings have helped the development of variety of backdoor defense techniques, the cognitive mechanism of how the predictions of the attacked model are hijacked by the trigger pattern is still not clear.
In this paper, propose an input information disentangling method called Cognitive Distillation (CD) to distill a minimal pattern of an input image determining the model's output (e.g. features, logits and probabilities). The idea is inspired by the existence of both useful and non-useful features within an input image (Ilyas et al. 2019). Intuitively, if the non-useful features are removed via some optimization process, the useful features will be revealed and can help understand the hidden recognition mechanism for the original input. CD achieves this by optimizing an input mask to remove redundant information from the input, whilst ensuring the model still produces the same output. The extracted pattern is called Cognitive Pattern (CP) and intuitively, it contains the minimum sufficient information for the model's prediction.
Using CD, we uncover an interesting phenomenon of backdoor attacks: the CPs of backdoor sam- ples are all surprisingly and suspiciously smaller than those of clean samples, despite the trigger patterns used by most attacks spanning over the entire image. This indicates that the backdoor cor- relations between the trigger patterns and the backdoor labels are much simpler than the natural correlations. So small trigger patterns may be sufficient for effective backdoor attacks. This com- mon characteristic of existing backdoor attacks motivates us to leverage the learned masks to detect backdoor samples. Moreover, the distilled CPs and learned masks visualize how the attention of the backdoored models is shifted by different attacks.
Our main contributions are summarized as follows:
We propose a novel method Cognitive Distillation (CD) to distill minimal pattern within an input image determining the model's output. CD is self-supervised and can potentially be applied to any type of DNN to help understand a model's predictions.
Using CD, we uncover a common characteristic of backdoor attacks: the CPs of backdoor samples are generally smaller than those of clean samples. This suggests that backdoor features are simple in nature and the attention of a backdoored model can be attracted by small part of the trigger patterns.
We show that the L norm of the learned input masks can be directly used not only detect a wide range of advanced backdoor attacks with high AUROC (area under the ROC curve), but also help identify potential biases from face datasets.
2 RELATED WORK
We briefly review related works in backdoor attack and defense. Additional technical comparison of our CD with related methods can be found in Appendix A
Backdoor Attack. The goal of backdoor attacks is to trick a target model to memorize the backdoor correlation between trigger pattern and backdoor label. It is closely linked to the overparame- terization and memorization properties of DNNs. Backdoor attacks can be applied under different threat models with different types of trigger pattems.. Based on the adversary's 's knowledge, existing
2

---

Published as a conference paper at ICLR 2023
backdoor attacks can be categorized into data -oiaoonis attacks and craining-manipullen Data-pos poissonnnn attacks inject the trigger pattern into a few training samples but do not have access to the training process, whereas training-mann mannpullaiiinon attacks can access and modify the training data, procedure, and objective function to implant the trigger (Garg et al.
attacks.
2020
Lin et al.,
2020; Shu-
mailov al. 2021 (Bagdasaryan & Shmatikov 2021 Nguyen & Tran 2021 Doan al. 202
The focus of our work is data- data-poisoning attacks. Early works, such as BadNets (Gu et al. 2017 Blend (Chen al., 2017) and Trojan (Liu et al., 2018b), use simple trigger patterns like black- white square, patch trigger or blending background to backdoor DNNs. Later works propose more complex trigger patterns, such as periodical signal pattern (SIG) (Barni et al., 2019), simulated
natural reflections (Refool) (Liu et al.] [2020). generative adversarial networks (GAN) generated patterns (for time series data) (Jiang et al. [023), , or physical world patterns/objects (Li et 2020
Wenger et al. 2021). to achieve more powerful attacks. One can also utilize adversarial perturbations
(Turner et al. 2018 Zhao et al. 2020b), Instagram filters (Liuu et al.. 2019). smooth image frequency
Zeng et al. 2021) or GANs (Cheng et al. 2020) to boost the strength of the trigger patterns. Besides the dataset-wise or class-wise triggers used by the above attacks, more recent works leverage sample- specific (Nguyen & Tran 2020) and invisible (Li et al. 2021c) trigger patterns craft more stealthy attacks. In this work, we will show one common characteristic of the above attacks that is related to their trigger patterns.
Existing backdoor defense methods can be categorized into: 1) trigger recovery. 2) backdoor model detection, 3) backdoor sample detection, and 4) mitigation methods, where the first three types of methods are oftentimes required for mitigation. Trigger recovery aims to reverse
Backdoor Defense.
engineer the trigger pattern (Wang et al. [2019 Guo et al. 2019 Liu et al. 2019 Sun et al. 2020
Liu et al. 2022 Xiang et al. H0 et al.] 2022). Backdoor model detection aims to determine
a model is affected by triggers (Chen al. [2019] Kolouri et al., 2020; Wang et al. 2020 Guo
et al. 2021 Shen et al., [2021] Xu et al. [222 Ip, It is worth noting that the detected models still need
mitigation methods to remove the trigger (Liu et al.
2018a /hao et al. 2020a Wu & Wang 2021
[ et al. 2021b; Zeng et al.) 2022; Guan et al. (022). Backdoor mitigation can also be achieved by
robust learning strategies (Borgnia et al. 2021 Huang et al., 2022; Dolatabadi et al. 2022).
Backdoor sample detection assesses if a sample is backdoor sample, i.e., whether it contains trigger pattern. The backdoor images may show anomalies in the frequency domain but could be hidden by the attacker using smoothing (Zeng et al. 2021). Spectral Signatures (SS) uses deep feature statistics to discriminate between clean and backdoor samples (Tran et al., 2018). but it is less robust to the change of poisoning rate (Hayase et al. (002). Feature-based detection
also be
performed via identity-variatio et al.) 2018), and feature consistency towards transformations (FCT) (Chen et al., 2022). STRIP proposes a superimposition technique to blend the potentially backdoored samples with small subset of clean samples, then utilizes the entropy of the predictions for detection (Gao al. 2019).
decomposition (Tang et al. (202)) activation clustering (AC) (Chen
Anti- Aati-Backdoor Learning (ABL) monitors sample-specific training loss to isolate low-loss backdoor samples (Li et al. 2021a). has also been theoretically proven that, under a restricted poisoning rate, robust learning on poisoned data is equivalent to the detection and removal of corrupted points
(Manoj & Blum] 2021). This highlights the importance of backdoor sample detection where our proposed CD can be applied.
3
COGNITIVE DISTILLATION AND BACKDOOR SAMPLE DETECTION
We first introduce our CD method, then present our findings on the backdoored models and the proposed backdoor sample detection method.
3.1 COGNITIVE DISTILLATION
Given a DNN model fe and input image E X C Buxbxx (w, h are the width, height and channel, respectively). CD learns an input mask to distill a minimal pattern from x by solving the following optimization problem:
arg min| [aiin]]fof (x) fo((op))l 112 + +a a||m] + +.3TV(
(I)
m
Tep = m (1 m) 3
(2)
3

---

Published as a conference paper at ICLR 2023
where, Tep E X' C Ruxxxe is the distilled cognitive pattern, m 10,1] is a learnable 2D input mask that does not include the color channels, 5 E [o, 1] is a c-dimensional random noise vector, @ is the element- wise multiplication applied to all the channels, TV(.) is the total variation loss, |I-I|| is the L,1 norm, a and /3 are two hyperparameters balancing the three terms. The model output fo(-) can be either logits (output of the last layer) or deep features (output at the last convolutional layer).
The first term in Equation I ensures the model's 's outputs are the same on the distilled pattern Tep and the original input æ; the second term enables to find small (sparse) cognitive patterns and remove the non-useful features from the input; the third TV term regularizes the mask to be smooth. In Equation 2 mask value m, close to means the pixel important for the model's output and should be kept, close to O, otherwise. Here, instead of directly removing the unimportant pixels, we use uniformly distributed random noise (i.e, 8) to replace them each optimization step. This helps to distinguish important pixels that originally have O values.
By optimizing Equation T we can obtain a 2D mask m and a CP Zep. where the mask highlights the locations of the important pixels, and the CP is the extracted pattern. It worth noting that a CP a perturbed pattern from the input image, which may not be the raw (original) pixels. The masks and CPs extracted for different models can help to understand their prediction behaviors.
3.2 UNDERSTANDING BACKDOORED MODELS WITH CD
1.0
0.9
Original bloger
Smotted trigger
0.3
176570
Clea
Sadwets Blens
CL
DEST
Oynamkc
Moduvle
S6
Strocth
Tribe
WaRKe
(a) Mask and CP visualizations
(b) Attack with simplified triggers
Figure 1: (a) First row: a clean image and example backdoored images. Second row: the corre-
sponding learned masks. Third row: distilled cognitive patterns. Fourth row: simplified backdoor images. The studied backdoor attacks include BadNets (Gu et al.| 2017). Blend (Chen et al.] 2017))
CL (Turner et al. 2018), DFST (Cheng et al. (2020), Dynamic (Nguyen & Tran 2020), FC (Shafahi
et 2018). Nashville (Liu et al. 2019), SIG (Barni et al. 2019), Smooth (Zeng et al. 2021). Trojan
(Liu et al. 2018b) and WaNet (Nguyen & 2001).. (b) The attack success rate of the simplified (the Fourth row in (a)) and the original triggers.
We first apply CD to understand the inference mechanism of backdoored models on clean vs. back- door samples. The experiments are conducted with ResNet-18 (He et al., 2016a) on CIFAR- 10 (Krizhevsky et al., (2009) dataset. We apply backdoor attacks (listed in Figure D with a poisoning rate of 5% to obtain the backdoored models. We then distill the clean CP of a clean training image on backdoored model (BadNets), and the backdoor CPs are distilled on the backdoored training images with corresponding models. Examples of the distilled CPs and their corresponding masks are visualized in Figure Ia More visualizations for all the attacks can be found in Appendix D
As shown in the first column of Figure la the mask and CP of the clean image on a backdoored model are large and semantically associated with the main object. Clearly, the model appears to be using the real content within the image to make the prediction. For the 3 attacks (BadNets, Trojan, and Dynamic) that use small patches or scattered pixels as trigger patterns, the CPs of the backdoor images reveal their trigger patterns and the masks highlight the key part of the triggers.
The other 8 attacks, all adopt full-image size triggers, yet their CPs are all suspiciously small (in magnitude), sparse (scattered pixels), and semantically meaningless (drifting away from the main object). This suggests that the model is indeed using the backdoor features to predict the class label. Interestingly, it shows that only a small part of the trigger pattern is involved in the inference process, even if the trigger spans over the entire image. It seems that the model not only ignores the real content but also large part of the trigger. We conjecture that this is because backdoor correlations are simpler in nature when compared with natural correlations, thus the model does not need memorize the entire trigger pattern to learn the backdoor correlation. Rather, it tends to find
4

---

Published as a conference paper at ICLR 2023
the simplest clues from the trigger pattern to complete the backdoor task. These simple correlations bypass the perception of the real content at inference time whenever the trigger appears.
Next,
set of experiments confirm that simpler triggers extracted by CD work the same as the original triggers of the above attacks. A backdoor training sample with a simplified trigger can be generated by:
2hu = m @ Bbd + (1 (1-m) @ x.
where, 2nd is the original backdoor sample, Thi is the backdoor sample with simplified trigger, m is a binarized (with threshold 0.05) version of the learned mask by our CD, and is the clean sample. For the 8 full-image size attacks, the trigger size is reduced from 100% to ~ 40% of the image size. Note that the reduced part is replaced by the original clean pixels and the test triggers remain unchanged. The attack success rate (ASR) of the simplified triggers is reported and compared with the original triggers in Figure Th It is evident that every trigger can be simplified without losing (much) of the ASR. Interestingly, the simplified trigger even slightly improves the ASRs of FC, SIG, and WaNet attacks.
To summarize, the above findings reveal one common characteristic of backdoor attacks: backdoor correlations are much simpler than natural correlations, regardless of the trigger patterns. One can thus utilize the size of the distilled masks by CD to detect backdoor samples.
3.3 BACKDOOR SAMPLE DETECTION
Based on the assumption that the model predictions on backdoored images will rely on a small set of pixels, we introduce CD backdoor sample detection method. Detection can be performed at either training or test time. At training time, the defender can remove backdoor samples from the training dataset, while at test time, the defender expose potential attacks and their triggers.
Threat Model. Following previous works (Tran et al., 2018 Gao et al. 2019; Li et al., 2021a), we assume the adversary can poison the defender's training data but does not have access to the training process. We also assume the defender has full control over the training process but has no prior knowledge of i) the poisoning rate, ii) the trigger pattern, iii) the backdoored class, or iv) whether sample is clean or backdoored.
Problem Formulation. Considering K-class image classification task, we denote the train- ing data as D = De U Do. the clean subset as {(x,, W E De. and the poisoned subset as [(andd yt ) E 'Ds, respectively. The attacker injects backdoor triggers using function (x// bd v.dd) = A(x,, V:). which converts a clean sample into backdoor sample. The x E X C Rexxxe are the inputs and y E y = (1,...,K are the labels for classes total. The poisoning The defender's goal is to accurately detect samples Do.
rate is defined as DD M
M+N
Backdoor sample detection is unsupervised binary classification task (backdoor class and clean class). We denote the predicted backdoor samples as D/ and the predicted clean samples as D'- Based on finding that the CPs of backdoor samples are suspiciously small, here we propose to use the L, norm of the learned mask m to detect backdoor samples, as measures the pixel intensity of the CP distilled by CD. We consider the following function g: to determine whether sample contains backdoor based on its mask m.
[1
m || S t, if m >
if
g(x) 0
where, t is a threshold, 9(.) = indicates a backdoor sample, whereas g(.) = O indicates a clean sample. While other strategies (e.g., training detector on the masks or distilled CPs) are also plausi- ble, the above simple thresholding strategy proves rather effective against wide range of backdoor attacks. The distribution of ||mm]]| in Figure confirms the separability of backdoor samples from the clean ones. In practice, the threshold can be flexibly determined based the distribution of [m]]]] for example, detecting samples of significantly lower ||m]]] than the mean or median as backdoor samples.
For test time detection, we assume the defender can access a small subset of confirmed clean samples D.. The defender can then calculate the distribution, the mean Ml|ml], and the standard deviation Timml, for x, E D.. The threshold can then be determined as: = H[|ma], - Jlmll,, where y is a hyperparameter that controls the sensitivity.
5

---

Published as a conference paper at ICLR 2023
1h0
Pre
Badidoor Ozan
Racknner
Budidoor
Back-1a3r
Backdoer
Baccesor
C.e0
Caan
Clean
100
mPhh mask
N0 Vo
(a) BadNets
(b) Blend
(c)CL CL
(d) DFST
(e) Dynamic
FC
Winbaree
Manbbrr meos
Ssceddor
Bacodoor
Ca made
nomo the mark
10
the mass
15 Fem
1. of the made
nomm the manc
(g) Nashville
SIG SIG
Smooth
Trojan
(k) WaNet
(I) ISSBA
Figure 2: The distribution of ||mm]] 1 for clean (blue) and backdoor (red) samples crafted by different
backdoor attacks. X-axis: Ln norm of the learned mask ([)mz] I| Y-axis: the number of samples.
4 EXPERIMENTS
In this section, we evaluate our CD-based detection method in terms of detection performance, robustness under different poisoning rates, and the improvement it brings to backdoor mitigation. We first describe the experimental setup and then present the results in Section 4.1
4.3
A detailed
ablation study on the hyperparameters and optimization terms of CD can be found in Appendix B.5 and B.6 It shows that CD-based detection is insensitive to the 3 hyperparameter and is moderately stable under varying a. We also provide an analysis of the detection performance under three types of adaptive attacks in Appendix [B7] We find that the attacker needs to sacrifice two erucial elements (attack success rate or stealthiness) to partially evade our detection method.
Attack Configurations. We consider 12 advanced backdoor attacks, including: BadNets (Gu al. 2017]. Blend (Chen et al. 2017). CL (Turner et al. 2018), DFST (Cheng et al. 2020). Dynamic
(guuyen & Tran 2020). FC (Shafahi et al.] (018)) SIG (Barni et al. 2019). Trojan (Liu et al.,
2018b),, using WaNet trigger (Nguyen & Tran (2221) Nashivell filter (iu et al. 2019) Smooth
(Zeng al. (02T)) and ISSBA (Li al. 2021c>. We perform evaluations
3 datasets, includ- ing CIFAR- 10 Krizhevsky et al. 2009). an ImageNet (Deng et al. 2009) subset (200 classes), and
GTSRB (Houben 2013). No data augmentations are applied during training as may de- crease the attack performance (Liu et al., 2020). We adopt 6 types of architectures, including VGG- Zisserman 2015), ResNet-18 (RN-18) et al. 2016a), PreAAtiaaiioRReeeeer 101 (PARN- 101) (He et al. 2016b), MobileNetV2 (Mobile (Sandler et al., 2018), GoogLeNee Szegedy et al. 2015), and EfficientNet-bOO (Tan & Le 2019). More details are in Appendix B.2
16 (Simonyan
Detection Configurations. We compare our CD approach with 5 state-of-the-art backdoor sample detection methods: AC (Chen et al. 2018). SS (Tran et al. 2018). STRIP (Gao et al., 2019). Fre- quency Zeng al. (2021), and the isolation strategy used in ABL (Li al. 2021a). For test time detection, we use the model's prediction as the label for AC and SS and exclude ABL from this experiment it cannot be applied at test time. For CD method, we both the logits layer and the deep feature layer (the last activation layer) as the model output and denote the respective method as CD-L and CD-F More details are in Appendix B.3
Evaluation Metrics. We adopt the area under the area under the ROC curve (AUROC) as the main evaluation metric. Following (Gao al. 2019), we also consider the true rejection rate (TRR) and false acceptance rate (FAR) as supplementary performance metrics. TRR is defined as the proportion of backdoor samples in Dj, whilst FAR is defined as the proportion of backdoor samples D',
4.1 DETECTION PERFORMANCE EVALUATION
Table summarizes the detection performance of our CD and the 5 baseline methods against 12 backdoor attacks. We test the performance on both the training and test sets (against the same poi- soning rate of 5%). The AUROC for each detection method against each attack is averaged across all the DNN models. The detailed results for each dataset, attack, and model are provided in Ap- pendix C which consistent with Table Compared with existing detection methods ABL,, AC,

---

Published as a conference paper at ICLR 2023
Frequency, STRIP, and SS, our CD-based detection method achieves new state-of-the-ar perfor- mance consistently across almost all 12 backdoor attacks, 6 model architectures, and datasets. Particularly, in training time detection, our CD-L method achieves average AUROC of 96.45% against all the 12 attacks across the 3 datasets, which is 11% higher than the best baseline method Frequency. Such a stable performance of CD-L verifies the generalisability of our model and con- firms our hypothesis about the characteristic of backdoor attacks; their CPs are smaller.
Table I: The detection AUROC (%) of our CD method and the baselines against 12 backdoor attacks (poisoning rate 5%) on the training/test set. The results are averaged across the models ( (VGG-16, RN-18, 101, Mobile GoogLeNet, and EfficientNet-bO), The best results in bold.
Darasct
Attack
ABL
AC
Frequency
STRIP
CD-L
CD+F
BadNats
85.64/-
77.57/74.63
92.32/91.59
97.89/97.66 66
62/89/55.00
94.03/94.72
88. 89/8988888
Blend
88.17/-
76.23/65.93
80.67/79.40
84.55/83.02
51.63/4,52
93.47/93.44
92.3092..41
CL
90.86/-
70.06/25.68
98.85/91.59
97.27/96.04
40.78/39.02
98.75/85.0 31
93.48/80.31
DEST
89.10/-
80.45/86.97
58.08/58.51 37.62/87.34
56.34/40.69
88.96/89.80
82.54/82.68
Dynamic
87.97/-
77.83/77007
97. 82/82/97.5 .58
91 49/89.75
66.49/50,91
97.97/97.85
84
CIFARIO
PC
86.61/-
83. K3.99/88.74 74
98.65/98. 11
79 79.84/66.97
63.62/64.62
99.17 19..1798..2
94 94.46/95.12
SIG
97.42/-
84.40/56.91
62.95/56.46
(1668/5744
58 90/52.7770
96.91/90.99 9990.90
96 09/
Smooth
79.53/-
82.1 11/7146
51.32/47.84
58.52/55.81
70.24/51.14
91. 91.09/89.03
82.05/81. 1,11
Nashville
76.12/
89.26/76.
70.53/67.71 71
51 62/48.30
80.48/60.62
9%, 10/9./97
96, 21,22999442
Trujan
85,96/-
69. 69.59/71.55
93.82/93.36
91 85/8899..14 14
59.18/45.04
96.91/96.72
91.16/91. 1.88
WaNet
56.66/-
70.96/69.86
96.31/96.65
84.98/84.64
71.59/77.27
25.69/96.08
86. 06.60/88.43
GTSRR
BadNats
67.78/-
98.21/72.79
57.26/59.59
69.97/72.86
99.28/99.14
99.59/99.66
ImageNet
BadNets
83,40/-
95.75/100.00
96.05/95,84 84
99,73/9.20
100,00/100.00
100,00/100.00
ISSBA
96.99/-
100.00/80.29
70.37/68.75 73
42.22/56.31
100.00/99.99
99.97/99.89
Average
83.617-
82.60/3.221
64.62/82.51
78.61/75.96
63.83/49.58
96.45/94.90
92.66/91.74
Out of the 5 baselines, SS has the lowest performance. This is because SS requires backdoor labels for detection. However, here the backdoor labels are unknown to the defender. For SS to be effective, it may need to be used in conjunction with backdoor label detection methods. AC is the best bascline and is fairly stable across different attacks. This means that the deep features of backdoor samples indeed anomalies in the deep feature space. Both AC and SS are deep feature-based detection methods, one may also observe from the full result in Appendix Cithat the performance of AC and SS against the same attack varies across different models. STRIP is quite effective against BadNets and, CL but performs poorly on DFST, Nashville, and Smooth attacks, which are all style transfer-based attacks. This suggests that the superimposition used by STRIP is vulnerable to stylized triggers. Note that our CD-L or CD-F is not always the best, for example, on BadNets, DFST, and SIG attacks, however, its performances are comparable to the best. For all detection methods, their test-time performance is relatively worse than their training-time performance, but the trends are consistent. SS demonstrates the largest performance drop, while our two CD methods, Frequency, and STRIP, experience the least drop. Our CD-L method achieves average AUROC of 94.90% on the poisoned test sets, outperforming the best baseline Frequency by more than 12%.
4.2
ROBUST DETECTION UNDER DIFFERENT POISONING RATES
6.4
6.
6x
Inguanty
CDd
Gpo
CD4
02-1
12
12%
2% 15% W0
22% 2% 12%
12%
125%
374
25%
2%
(a) BadNets
(b) Blend
(c) CL
(d) DFST
Dynamic
(f) FC
1.3
3.3
Prepurccy
3.5
13
Passning Roce
Teperingg Husi
Toooring Rulz
Pooseeng Mal
Potseery Ace
Prteeing Hell
(g) Nashville
(h) SIG
(i Smooth
Trojan
(k) WaNet
()) ISSBA
Figure 3: The detection performance (AUROC) on CIFAR- 10 training set under different poisoning rates (1%, 2.5%, 5%, 7.5%, 10%). Results are averaged across all 5 models following the same settings as in Table
7

---

Published as a conference paper at ICLR 2023
Here, consider more challenging settings with varying poisoning rates [1%, 2.5%, 5%, 7.5%, 10%]. All other experimental settings are the same as in Section 4.1 Note that, under 1% poisoning rate, most of the attacks can still achieve ASR 2 90%. The results on the training sets are presented in Figure 3 Similar results on the test sets can be found in Appendix C It is evident that method can achieve good detection performance robustly under different poisoning rates, except for slight performance drop on Bend, DFST, Smooth, and WaNet under high poisoning rates 5%. It can also be observed that ABL, AC, and SS are quite sensitive the poisoning rate, but STRIP is comparably better. The Frequency is more stable under different poisoning rates. Only our methods and AC can robustly detect the more advanced attack ISSBA.
4.3
IMPROVING BACKDOOR MITIGATION
AC ACA-ABL
3035
2TTP ABL
STEP
Tresuancy
3034
Clnl ABL
0043
2-44
050
" 100 3000
30g
(a) CA (1)
(b) ASR (L)
(c) TRR (1)
(d) FAR (L)
(e) Recall (L)
Figure 4: (a): Clean accuracy (CA) after mitigation; (b): Attack success rate (ASR) after mitiga-
tion; (c): True rejection rate (TRR); (d): False acceptance rate (FAR); (e) Recall measured on D'
(larad E D',/(D,), ). All the experiments are averaged across all 5 models and 11 attacks on CIFAR-
10 following the same setting as in Section 4.1
Next, we demonstrate that effective backdoor sample detection can help improve backdoor miti- gation methods. We follow one of the state-of-the-art mitigation methods ABL (Li et al. 202 to unlearn triggers from backdoored models using the detected backdoor subset D/, and the clean subset D'-- ABL maximizes the model's loss on D' to remove the trigger while minimizing the loss on D', to maintain high clean accuracy. Based on the L1 norm of the mask, here we select 2.5% samples with lower L, norm into D/, and 70% samples with higher L, norm into D' We replace the backdoor isolation (detection) strategy in ABL with our two CD methods and denote the new methods as CD-L + ABI. and CD-F + ABI., respectively. We run the experiments on all the attacks and models evaluated in Section 4.1 Detailed settings can be found in Appendix B.4
The results are summarized in Figure As can be observed, our CD-L method achieves the highest TRR and the lowest FAR, which both indicate good detection performance. Higher TRR means more backdoor samples are detected into D/, facilitating more effective trigger unlearning. On the other hand, lower FAR means there are fewer backdoor samples left in D'- which improves clean accuracy. Therefore, after mitigation, our CD-L method achieves the lowest ASR which is followed by CD-F method. Note that the defender could also retrain the model on D' and in this case, FAR is the poisoning rate in D', i.c., lower is better. The above results indicate that accurate backdoor sample detection can provide substantial improvement to backdoor mitigation.
5 A CASE STUDY ON BIAS DETECTION
In this section, we show the connections between dataset bias and poisoning backdoor attacks. It has been shown that curated datasets may contain potential biases (Torallba & Efros 201T), which will be memorized by DNN models trained on such datasets. It has also been found that bias may be strongly associated with a particular class. For example, gender is associated with occupation (Bolukbasi et al., 2016), person names Wang et al. (2022) or facial attributes (Zhang et al. 2018
Tartaglione et al. 2021p. p. On the other hand, DNNs tend to learn more of the "easier" correlations (potential biases) in the dataset and this usually happens at an earlier training stage, as captured by the sample-wise training loss (Nam et al.
2020; Du et al. 2021). Such a phenomenon was also
observed and leveraged in ABL (Li et al.
2021a) to isolate and mitigate backdoor attacks. These connections motivate us to apply our CD method to detect potential biases in a dataset. Here, consider a real-world face dataset CelebA (Liu et al. 2015), which contains 40 binary facial attributes (the full training set is denoted as D). We train a classifier ft for each facial attribute i, based on shared feature extractor (ResNet-18). The detailed setup is described in Appendix B.8
8

---

Published as a conference paper at ICLR 2023
We apply CD-L to select 0.5% samples with the lowest ||ml]] for each classifier fi into subset D/-- We define the bias score as the distribution shift of attribute j from to Dj:
s(j. = (Pig Pj)/maz( (M), 1,.1-P1 Pj),
(5)
where, P. = [P/llIDD is the percentage of samples in D that have attribute j. P3 = /D D is the percentage of samples in the full training that have attribute j. s(j,t) measures to what degree attribute j is predictive of attribute (according to fi). A positive/negative s s(j, i) value means that (positive/negativee) attribute j is predictive of attribute i, closer +1/-1 means more predictive. The s(j, i) score allows us to identify the most predictive attributes to an attribute of interest, and potential biases if the absolute score is exceptionally high.
BreaT
Mackk
FeB
Gander
Marry
(a) Predictive attributes of gender attribute
(b) All highly correlated attributes
Figure 5: Potential biases detected from the CelebA dataset using our CD-L method. Each node in the graph is an attribute, with the links pointing (with score x(j. i)) from a highly predictive attribute to the attribute of interest (green nodes). Red/blue nodes and links represent positive/negative pre- dictive attributes. Darker color edge indicates larger absolute s(j, i) score.
As shown in Figure 5a applying CD-L on the gender classifier, we identify the top-2 most predic- tive attributes of gender: blond hair and heavy makeup. This matches with one CelebA, gender is mostly determined by these two attributes (Nam et al. 2020 Tartaglione et al, 2021]. Furthermore, our method also reveals two potential negative biases: brown hair and black hair, meaning that gender is hard to predict when these two hair colors appear. It also indicates that blond hair is mutually exclusive with other hair colors. Figure further shows all the highly corre- lated (either positive or negative) attributes identify from CelebA, where the absolute s(j, scores are all greater than 0.8. As suggested by Zhang et al. (2018), defining ground-trus truth biases is subject to different human interpretations. Several connections revealed in Figure 5bb are closely related to known human stereotypes, such as males with different kinds of beards, or makeup with arched eye- brows. CD highlights the similarity between backdoor sample detection and bias detection. They both trigger over-easy predictions based on only tiny amount of information perceived from the input. This means that even a dataset does not contain backdoored samples, low [m]] samples should be carefully examined as they may have biases. Our method demonstrates the possibility to detect potential biases in real-world datasets, a challenging problem in fair machine learning.
bias in
6 CONCLUSION
In this paper, we proposed a novel method, Cognitive Distillation (CD), that extracts the minimal pattern responsible for the model's prediction. With CD and the distilled patterns, we reveal that backdoored models make backdoor predictions only based on a sparse pattern of the input, regard- less of the large trigger patterns used by different attacks. And the location of the pattern reveals the core part of the trigger. This allows us to build simple but effective backdoor sample detectors based on the learned input masks. We empirically show, with 6 DNN architectures, 3 datasets, and 12 advanced backdoor attacks that our proposed detection method is both effective and robust (under different poising rates). Our CD methods can also help detect potential biases in real-world datasets. The supf-vipe nature of CD makes it generic technique for investigating the underlying in- ference mechanism of different types of DNNs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Xingjun Ma is in part supported by the National Key R&D Program of China (Grant No. 2021ZD01 12804), the National Natural Science Foundation of China (Grant No. 62276067), and the Science and Technology Commission of Shanghai Municipality (Grant No. 2251 1 106102). Sarah Erfani is part supported by Australian Research Council (ARC) Discovery Early Career Researcher Award (DECRA) DE220100680. This research was undertaken using the LIEF HPC- GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200 The authors would also like to thank Yige Li for sharing several the backdoor triggers used in the experiments.
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security, 2021.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. NeurIPS, 2016.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without accuracy tradeoff. In ICASSP, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 2020.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-base based visual explanations for deep convolutional networks. In WACV, 2018.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee. Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: 18/ 1.03728, 2018.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, 2019.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NeurlPS, 2022.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05520 2017.
Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. arXiv preprint arXiv:2012.11212 2020.
Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In SPW, 2020.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. NeurIPS, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
10

---

Published as a conference paper at ICLR 2023
Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modifi- cation. NeurIPS, 2021a.
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In ICCV, 2021b.
Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Collider: A robust training framework for backdoor data. ACCV, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Harris Drucker and Yann Le Cun. Double backpropagation increasing generalization performance. In IJCNN, 1991.
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of nlu models. In NAACL-ILT, 2021.
Yu Feng, Benteng Ma, Jing Zhang, Shanshan Zhao, Yong Xia, and Dacheng Tao. Fiba: Frequency- injection based backdoor attack in medical image analysis. In CVPR, 2022.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In ICCV, 2017.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal per- turbations and smooth masks. In ICCV, 2019.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. In ACSAC, 2019.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturba- tions inject neural backdoors. In CIKM, 2020.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXhv:1708.06733, 2017.
Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. In CVPR, 2022.
Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. arXiv preprint arXiv:21 10. 10. /4880, 2021.
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint 19Xivv/908.00 2019.
Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Defense against backdoor attacks via robust covariance estimation. In ICML, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual
networks. In ECCV, 2016b.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN, 2013.

---

Published as a conference paper at ICLR 2023
Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao. Susmit Jha, and Chao Chen. Trigger hunting with a topological prior for trojan detection. In ICLR, 2022.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NewrlPS, 32, 2019.
Yujing Jiang, Xingjun Ma, Sarah Monazam Erfani, and James Bailey. Backdoor attacks on time series: A generative approach. In SaTMI, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. NeurIPS, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021b.
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang. Zhifeng Li, and Shutao Xia. Rethinking the trigger backdoor attack. arXiv preprint arXiv:2004.n 04692, 2020.
Yiming Li, Haoxiang Zhong, Xingjun Ma. Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. ICLR, 2022.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specifie triggers. In ICCV, 2021c.
Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite backdoor attack for deep neural network by mixing existing benign features. In CCS, 2020.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In RAID, 2018a.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2018b.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In CCS, 2019.
Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang. Shiqing Ma, and Xiangyu Zhang. Com- plex backdoor detection by symmetric feature differencing. In CVPR, 2022.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Iyya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. ECCV, 2016.
Naren Sarayu Manoj and Avrim Blum. Excess capacity and backdoor poisoning. In NeurIPS, 2021.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. NeurIPS, 2020.
12

---

Published as a conference paper at ICLR 2023
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. NeurIPS, 2020.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping-based backdoor attack. In ICLR, 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv: 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In SIGKDD, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. ICCV, 2017.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. NeurlPS, 2018.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime: Real and stealthy attacks on state-of-the-an face recognition. CCS, 2016.
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In ICML, 2021.
Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Er- dogdu, and Ross Anderson. Manipulating sgd with data ordering attacks. NeurlPS, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6034 2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 1412vv14/2.68000 2014.
Mingjie Sun, Siddhant Agarwal, and J Zico Kolter. Poisoned classifiers not only backdoored, they are fundamentally broken. arXiv preprint arXiv:2010.09088 2020.
Yuhua Sun, Tailai Zhang, Xingjun Ma. Pan Zhou, Jian Lou, Zichuan Xu, Xing Di, Yu Cheng, and Lichao Sun. Backdoor attacks on crowd counting. In ACM MM, pp. 5351-5360, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of (DNNs) for robust backdoor contamination detection. In USENIX Security, 2021.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In CVPR, 2021.
Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. In CVPR, 2011.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. NeurIPS, 2018.
13

---

Published as a conference paper at ICLR 2023
Alexander Tumer, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P, 2019.
Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In ACL, 2022.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection trojan neural networks: Data-limited and data-free cases. In ECCV, 2020.
Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y
Zhao. Backdoor attacks against deep learning systems in the physical world. In CVPR, 2021.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurlPS, 2021.
Zhen Xiang, David Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. ICLR, 2022.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In S&P, 2021.
Kaiwen Yang. Tianyi Zhou, Xinmei Tian, Dacheng Tao, al. Class-disentanglemmnt and applica- tions in adversarial detection and defense. NeurIPS, 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, 2021.
Yi Zeng, Si Chen, Won Park, Zhuoging Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR, 2022.
Quanshi Zhang, Wenguan Wang, and Song-Chun Zhu. Examining cnn representations with respect to dataset bias. In AAAI, 2018.
Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In ICLR, 2020a.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean- label backdoor attacks on video recognition models. In CVPR, pp. 14443-14452, 2020b.
Shihao Zhao, Xingjun Ma, Yisen Wang, James Bailey, Bo Li, and Yu-Gang Jiang. What do deep nets learn? class-wise patterns revealed in the input space. arXiv preprint arXiv:2101.06898. 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
14

---

Published as a conference paper at ICLR 2023
A DIFFERENCE OF CD TO EXISTING METHODS
Difference with Model Interpretation Methods. Our CD can provide a certain degree of expla- nation about how the model makes its predictions, thus related model interpretation methods. Gradient-based interpretation methods generate saliency maps (Simonyan et al., 2013). such as de- convolution (Zeiler & Fergus 2014) and guided backpropagation (Springenberg
al.) 2014 Ma- hendran & Vedaldi 2016) to help interpret how the model makes predictions given an input image. Perturbation-base based methods (Dabkowski & Gal 2017 Persiuk et al. 2018: Fong et al. 2019 Yang eet al., 2021; Zhao et al. 2021) are a generalized version of saliency maps that integrate several rounds of backpropagation learn an explanation (Fong& & Vedaldi 2017). Meaningful Perturba- tions (MP) (Fong & Vedaldi 2017) generate interpretations via deletion. It searches for perturbed input that maximizes the score for a specific class. LIME (Ribeiro et al., (0166) is one of the founda- tional works for interpretation. It trains additional locally faithful interpretable model to explain the target model. E.g., the faithful model can be a linear model that locally behaves the same as the target model. Similar to LIME, CD also emphasizes faithfulness, but it does not involve an additional model. Another set of methods is Class Activation Mapping (CAM) (Zhou et al.) 2016) and its variants (Selvaraju et al., 2017 Chattopadhay et al., 2018). CAM-based methods use the gradient of the target concept (label) through specific layer and generate a coarse localization map highlighting important regions.
Technical Comparison. The difference between our CD and closely-related methods, including LIME (Ribeiro al.) (2016), Meaningful Perturbation (MP) (Fong & Vedaldi (2017), and Neural Cleanse (NC) (Wang et al. 2019), are summarized in Table 2 Different from LIME, MP, and NC, our CD method is self-supervised, that is. the distillation process does not rely on the class labels. And CD searches for the most determining input pattern of the model output, for whichever class it predicts.
Table 2: The objective functions of LIME, Meaningful Perturbation (MP), Neural Cleanse (NC), and our Cognitive Distillation (CD). fc is the cross-entropy loss.
Method LIME
Optimization Objective
arg min [ g. T2) (2gg)
RCG
NC
aig min Les m:) 2 m. C A.), Me + a + m+ m.A.IEl....
MP
arg min Le (w ① + mh
a
CD (ours)
arg migmin/m
(xx
m 87 + m + (m)
Detailed Difference with LIME. LIME aims to find an additional interpretable model with R(g) regularizing its complexity. L enforces the local fidelity that g behaves faithfully to f (DNN) around (using the mask to perturb around æ). When applying LIME on images, ridge regression is lected as g, and one requires a targeted class intended to explain. The output is a coarse heatmap defined by fixed super-pixels. However, our method is different from LIME, and there is no addi- tional model g or targeted class yr involved. Also, our generated masks are fine-gratd fine-grained with the most important pixels for the model's prediction.
Detailed Difference with NC. NC optimizes for both a mask and the trigger pattern. The optimiza- tion outputs a mask m, and trigger pattern A, for a class ye. The objective of NC is to disentangle a perturbed input image x' into a wlsss-wise universal trigger pattern A, with a corresponding mask m, and residual image (1- (1-m)) 3 æ. For DD. the perturbed input x' can make the model predict the target class yr. NC can be used for backdoored label detection, trigger synthesis (recovery) and backdoor model detection. It measures the I., norm of the mask m, for all classes and the Median Absolute Deviation for detecting backdoored models and labels. rather than backdoor samples. Also note that existing works (Liu et al., 22199] Nguyen & Tran 2020 Li et al. (00222 Cheng et al. 2020) have demonstrated that advanced attacks can still evade NC's detection. As discussed in NC (Wang et al. 2019) it is also not efficient for models/datasets with a large number of classes/labels.
Different from NC, our CD aims to find the pattern for each sample in the dataset rather than class of samples. Both CD's output and main optimization functions are different from NC: 1) CD outputs sample-specific masks and patterns, 2) its objective is the sepe-vaper absolute difference
15

---

Published as a conference paper at ICLR 2023
between f(x) and f(x',, rather than the supervised cross-entropy loss (LCE). Moreover, CD computational complexity does not increase with the number classes. NC (Wang et al.) 2019) and its related works (Guo et al., 2019; Xu et al. 202I) find that the backdoor class requires less perturbation to perturbed into than other classes. Unlike NC, our results are from an input-specific perspective that useful features of backdoored input samples contain fewer pixels than clean samples.
Detailed Difference with MP. MP aims to find a perturbation mask that can explain a target class yr concept that needs be explained by the model. The deleted perturbation operator & includes constant, noise, and blur. MP aims to find deletion regions that are maximally informative (for yc). while CD aims find regions that can be preserved for the prediction (and does not require yr). For the removed pixel, CD uses random color value in each optimization step, while the á is the same operation applied in every step.
through Lcr. The target class is also known
MP is more similar to NC than our CD, since both of them use CE as the main objective. The backdoor class label of the image must be known in advance for backdoor sample detection.
In nutshell, MP asks the model what can be removed to make a cat image not like 'cat', ?? however, our CD asks what the minimal sufficient paltern the model needs make its prediction?
B EXPERIMENTAL, SETTINGS
All experiments are run with NVIDIA Tesla P1000100/00100000 GPUs with PyTorch implementatiinns. inmoiilyllmimmlmmttettttttttooo Code available at https: / gigttuuuu... com/ HanxunH
For CIFARIO experiments, we train for 60 epochs with SGD optimizer, weight decay 5 10-2, initial learning rate 0.1 decay by 0 at the 45th epoch. For GTSRB, we use the same set up with CIFAR10 10, except for 40 epochs and decay at the 35th epcoh. For the ImageNet subset, train 200 epochs with the same learning rate settings and weight decays, except the learning rate adopts a cosine decay without restart (Loshchilov & Hutter 2017).
B.1 DATASETS
The datasets used in our experiments in Section 4.1 are summarized in Table
Table 3: Statistics of the datasets used in our experiments
Dataset
No. Classes
Input Size
No. Training Images
CIFARTO Krizhevsky al. 2009)
10
32x32x3
50,000
GTSRB Houben et al. 2013)
43
32x32x3
39,209
ImageNet (Deng et al.
2009) subset"
200
224x224x3
100,000
The split consistent with
2021c).
B.2
ATTACK CONFIGURATIONS
Table Configurations and implementation details of the backdoor attacks.
Attacks
Trigger Size
Trigger Pattem
Trigger Type
Strategy
BadNets Gu
2017
Patch
Grid
Class-wise Class-wise
Pixel modilication
Blend Chen
2017
Full image
Hello Kitty
Blend in
CL Turner et
Full image
Patch Adv Noise
Class-wise
Optimizations
DFST Cheng el al.
Full image
Style transfer
Sample-specifie
GAN
Dynamic
Nguyyn
Tran
2020
Few pixels
Mask generator
Sample-spccific
Mask Generator Optimization
FC Shalahi et al.,
2018
Full image
Optimized Noise
Sample-spacific
SIG
(Bami et al. 20191
Full image
Periodical signal
Class-wise
Blend in
Trojan [Liu ct al.
2018b
Patch
Watermark
Class-wise
Pixel modification
WaNet [Nguyen & Tran 2021
Full image
Invisible noise
Samplc-specific
Wrapping function
Nashville (Luu el al.
2019
Style transfer Full image
Full image
Sample-specios
Instagram filter Smooth frequency space Encoder
Smooth (Znn et al
2021
Style transfer
Sample-spedd
ISSBA Li
2021e
Full image
Mask Generator
Sample-specific
The detailed configurations the 12 backdoor attacks considered in our experiments are summa- rized in Table The class-wise pattern means applying the same pattern all images of particular
16

---

Published as a conference paper at ICLR 2023
class, while the sample-specifis pattern means that each sample has specific pattern. We have ver- ified their attack success rate (ASR) and clean accuracy (CA) on the test sets, and all the attacks are effective, shown in Appendix [ Visualizations of the backdoor samples are in Appendix [ For both Dynamic (Nguyen & Tran, 2020) and WaNet (Nguyen & Tran 202T), we use their trigger but do not manipulate the training objective, which is consistent with the threat model. For WaNet, find its original hyperparameters are not strong enough to generate effective triggers in this threat model. We thus increase the uniform grid size to 32 x 32 improve its ASR.
B.3 DETECTION CONFIGURATIONS
Table 5: Detection configurations of backdoor sample detection methods. For all baseline methods, we followed their official open-sourced implementations.
Method
Detection confidence score
ABL (Li et al.
10
Training loss of first 20 epochs
AC Chen et
2018
Silhouette scores
Frequency (Zeng el al. 2021
Detector's output probability
STRIP
(Gao et al. 2019
Entropy Top right singular vector
SS Tran et al.
2018)
Configurations of the sample detection methods used in the experiments are summarized in Table ABL is hybrid method of both backdoor sample detection and backdoor mitigation. Here we only use its sample detection part, which was referred to as backdoor sample isolation in the paper (Li et al. 202 We use the sample-specific training loss averaged over the first 20 epochs of model training as the confidence score. For AC, SS and STRIP, they are specifically designed for poison sample detection, and we use their original settings. For SS, we use the top right singular vector of the SVD decomposition as the confidence score. In our threat model, it assumes that the defender does not have prior knowledge of which samples are clean, which is a more realistic in real-world scenarios. For STRIP, we use all training data as the sampling pool for superimposing as the defender does not have any clean samples. Interestingly, we find this does not dramatically affect SKTIP's performance.
For Frequeney defense, we followed the strategy that converts images into frequency space using a Discrete Cosine Transform. Given that the defender may not have access to clean data or know the type of trigger, we train the backdoor detector on the GTSRB dataset and a wide range of generated triggers that does not assume any prior knowledge of the type of trigger location. We exclude the Frequency defense method from evaluations on GTSRB and ImageNet, since the detector is trained on GTSRB, and ImageNet has different resolutions that require additional detector. We use the detector's 's output probability as the confidence score.
For test-time detection, except for STRIP, other baseline methods cannot be directly applied. This is because both AC and SS require ground truth label y, for which we replace it with the model's prediction. It should have minimal impact on the detection performance if the prediction is ac- curate, should degrade the performance otherwise. We exclude ABL from our test-time detection experiments, as requires training statistics, and there is no intuitive solution to accommodate this requirement. We assume all the methods can calculate the mean score on 1% of the clean training set and use standard deviation lower than the mean as the threshold.
CD. The 3 is set to 10. The is set to 0.01/0.001 for models trained on CIFAR10 and GTSRB, 0.001/0.0001 for the ImageNet subset model, for CD-L/CD-F, respectively. We use Adam optimizer (Kingma& & Ba 2014) with initial learning rate 0.1, 10, 30,1,1, and total of 100 steps to learn the input mask. We use the scaled tanh function to ensure the mask is between [0, 1].
B.4 BACKDOOR MITIGATION
Following previous work (Li al. 202 the defender can take step further to remove the back- door via process of anti- nati-backdoor finetuning. Lec., finetuning the backdoored model to maximizing its loss on D/ while minimizing its loss D',, as following:
arg min E (= ((f((x))))))))))))))))))())))[) Ve) + arg max EE (o C(f yo)
(6)
17

---

Published as a conference paper at ICLR 2023
We use the method described in Algorithm We set pa to 2.5% and Pe to 70%, and optimize for epochs with learning rate set to x 10"4.
Algorithm 1 Unlearning and Fine-tuning
Input: fo. D. percentage pa for D' percentage Pe for D/, Scores
|mll Epochs E
2: D argsort(D,
3: D/ select ps samples with lower from
detected as backdoor samples D- detected as clean samples
4: D' =select P. samples with higher from D
5: for to E' do
6: optimize arg ming ( ((fo(x)))))
7: optimize arg maxo (...))>D;
((f ((
8: end for
9: Output: Jo
B.5 ANALYSIS OF THE REGULARIZATION TERMS
The goal of our CD is find the most determining pattern from the input image of the model's prediction via learnable mask. Intuitively, one can use following objective function:
arg min [l.f(x))) fo (a o m)]1 + (m))))
where, a O m is the perturbed input and the unimportant pixels are replaced with O. The above objective may not be ideal there may exist black/darker pixels with values very close to 0, which will be surely removed by the ||ml]] regularization. However, in clean images, black/darker pixels may also be important. As shown 6 the optimized mask only considers the white pixels of the trigger are necessary for the model's 's prediction, with the darker pixels are completely ignored. For the clean samples, darker pixels of the main objects "hhorse" or "peacock") are also important.
In order to address the above issue, replace the unimportant pixels with random noise vector d sampled at each optimization step, using the following objective function:
arg min fx( fo ( @ + (1 m) 8)1l al((m)]l
where, random noise applied to all spatial dimensions. This objective function will force the mask to consider darker pixels equally important for the model's prediction white pixels. As shown in Figure 7 the optimized mask now can reveal the black pixels of the backdoor trigger. For clean samples, darker pixels of the main objects ("horse" and "peacock") can also be considered as important by our CD.
In order to make the extracted mask be smooth and more interpretable, add one more regular- ization term to CD's s objective:
arg min [[))())) fo(
m) 1) + a:l (m) (TV(m))
where, the TV is the total variation loss and /3 balances the TV with other objectives. This ensures the model consider the surrounding pixels of an important pixel as similarly important. As shown in Figure g the learned mask with the TVV regularization can outline the shape of the main object.
B.6 ABLATION STUDY OF THE HYPERPARAMETERS
In this section, we perform a comprehensive ablation study of the hyperparameters used by CD, in the context of backdoor detection performance.
Sensitivity to a The a hyperparameter controls the number pixels that can be kept in the distilled pattern. If too many too few pixels are removed from the pattern, then it would be difficult to distinguish between the backdoor and clean samples. a should be set in a way that the extracted pattern mainly focuses generate patterns for a small subset of samples to examine the effect of the pattern. As shown in Figure 9a 0.01 shows the extracted pattern focuses on the main object of the image, and the
the main object in the image. In practice, the defender may
18

---

Published as a conference paper at ICLR 2023
(a) Clean samples
(b) Backdoor samples
Figure 6: First row: examples of clean and backdoor images (BadNets) randomly selected from the CIFARIO 10 training set. Second row: the corresponding learned masks m using Equation 7 Third row: the distilled patterns æ m for the corresponding masks and images.
(a) Clean samples
(b) Backdoor samples
Figure 7: Learned masks m using Equation
(a) Clean samples
(b) Backdoor samples
Figure 8: Learned masks m using Equation
1.0
Bedhal
Blend
0.9
SG
Trojen
Walket
Nachvlila
0.1
9.95
9.01
0.005 0.002
Smooth
(a) Visualizations
(b) AUROC
Figure 9: Varying in 10.1, 0.05, 0.01, 0005, 0.001] for CD. Visualizations of the learned mask for clean (top row) and backdoor (bottom row) samples. (b) The detection performance of AUROC on the training set for each attack with 5% poisoning rate. Results are averaged across models.
mask removes other pixels. From Figure 9b. it can also be observed that the detection performance stable from 0.05 0.005 against all attacks.
Sensitivity to 3. The 3 hyperparameter controls how smooth is the learned mask. Large tends produce smoothed mask, while small / will generate sparse mask. As shown in Figure T0al
19

---

Published as a conference paper at ICLR 2023
smoothed mask could produce more interpretable pattern that matches with human understanding. As shown in Figure 10b 3 does not significantly affect the backdoor detection performance.
0.
0.8
0.7
Dy nomiic
7C
SGG
Trojan
Withen
Nastville
10
50
100
(a) Visualizations
(b) AUROC
Varying 3 in 5, 10, 50, 100] for CD. (a) Visualizations of the learned mask for clean (top row) and backdoor (bottom row) samples. (b) The detection performance of AUROC on the training set for each attack with poisoning rate 5%. Results are averaged across models.
Figure 10:
B.7 WHITE-BOX ADAPTIVE ATTACKS
In this section, we evaluate our CD-based detection method against white-box adaptive attacks, i.e., the attacker is aware of detection strategy and attempts evade our detection with more adaptive strategies Overall, we find that, in order to evade our detection, the adversary has to sacrifice one of the two crucial components of backdoor attacks: attack success rate stealthiness. The detailed evaluations are as follows.
Increasing trigger visibility. Our CD uses the L1 norm of the distilled trigger patterns as the detection metric. As shown in Figure | the attacked models by existing attacks do not fully use the entire trigger pattern for the backdoor predictions. And our detection leverages this characteristic to detect backdoor samples. Intuitively, the adversary can intentionally increase the size of the trigger pattern perceived by the model to evade our detection by increasing either 1) the size of the patch trigger, or 2) the visibility of the blending trigger (e.g., reducing the transparency). Here, we test this type of adaptive attacks based on BadNets and Blend attacks, as both methods can be easily adapted to different trigger sizes and degrees of visibility.
Decreasing Attack Success Rate. The adversarial may alternative use a weaker trigger to evade our detection, i.e., trigger that blends well into the clean image by being not so adversarial. In this case, the trigger will not be (so) responsible for the model's prediction, which could evade our detection extent. This be achieved by decreasing either 1) the poisoning rate, 2) visibility of the trigger. Here, we take BadNets, SIG, and Nashville as three example attacks to demonstrate this adaptive strategy. We also evaluate the performance the Blend attack by reducing its transparency.
08
08
ABL
344
CBL
CDF
COF
A5R
400
Triger
160d torreeeery
Seeenn Rate
Hend Dorggaree
(a) Increasing trigger visibility.
(b) Decreasing attack success rate.
Figure 11: (a): Increasing the trigger size of BadNets and the transparency of Blend. (b): Decreasing the poisoning rate lower ASRs (results averaged across BadNest, SIG and Nashville attacks) and the transparency of Blend. All experiments are evaluated on CIFAR10 with ResNet-18 18, The results (AUROCs) are calculated on the training set.
As shown in Figure 1la if the size the visibility of the trigger increases, it could reduce the detection performance of our CD. However, this will greatly impact the stcalthiness of the attack, as the trigger patterns visualized in Figure 12 The results show that the trigger size needs to be as large
20

---

Published as a conference paper at ICLR 2023
the main content in the image evade our detection. Figure shows that the adversary could sacrifice certain ASR to evade our detection, and other detection methods. Our CD is comparably more robust against poisoning rate decrease, but is less robust against the adaptive Blend attack with reduced transparency if it becomes lower than 2%. Both STRIP and our CD are based on input- output responses, and if the attacks are barely successful, then the detection performance could be affected. Nevertheless, in either case, the effectiveness (stealthiness or ASR) of the attack will be greatly impacted.
3x3
6x6
9x
12
15
18
21
a= 10%
a- 20%
30%
40%
50%
d- 60%
70%
(a) Increasing the trigger size of BadNets
(b) Increasing the trigger transparency Blend
Figure 12: First row: examples of clean images from the CIFARIO training set. Second row: the corresponding learned masks of clean images extracted for each model trained under different trigger sizes and visibility. Third row: examples of the backdoor image with different trigger sizes and visibility. Forth row: the corresponding learned masks of backdoor images extracted for each model trained under different trigger sizes and visibilityy
Training-objectivn adaptive attack.
We investigate if CD could be evaded if the training objective is manipulated. Note that, in our threat model, the attackers do not have access to the training procedure and thus cannot alter the training objective. Here, we go beyond this assumption and test whether our CD defense, in pure white-bos setting, can stand such adaptive attacks. Our CD technique can be viewed of the saliency map methods, and the mask is generated through several rounds of backpropagation. CD uses |m/] classify if the sample is backdoored. To evade the detection, the attacker may add a regularization term to the original training objective that forces the model to pay attention all pixels of the trigger pattern. Here, we realize this attack by the following input gradient regularization, which is designed to force the model to produce uniform attention over the backdoor samples. Formally, defined as:
generalized version
E ( [C([(=),)) + Ill C((((())))) - 71111 + ((e.)) (F((()))))
(10)
where D, the backdoored subset, D. is the clean subset, L is the cross-entropy loss, 7 is the targeted value of the input gradient and the À balances the strength of the regularization.
ASR
AU-ROC
0.8
Percentage
0.6
0.4
0.2
puad
CL
DFST
FC
Nashville
946
Smooth
Trojan
WaNet
Figure 13: ResNet-15 trained on CIFAR-10 with input gradient regularization.
21

---

Published as a conference paper at ICLR 2023
We use double backpropagation (Drucker & Le Cun 1991) solve the above adaptive objective, with y = 0.0001 and 10.0. All other settings are the same as our main experiments. The results shown in Figure 3
15 It can be observed that this adaptive attack strategy can not evade our CD) detection. For patch- based triggers (BadNets, CI., and Trojan), Dynamic, and Blend attacks, the AUROC is above 0.8. For the style transfer-like like triggers (DFST, Nashville, and Smooth), the adaptive attack strategy can indeed cause CD to generate larger ml] for backdoored samples. However, this tends sacritice its ASR (I 40%). Interestingly, for FC, SIG, and WaNet, the CD can detect backdoor samples even if the ASR is low. The above results indicate that our defense is easily broken even if the training objective is adaptively manipulated.
1200
Number of samples
1000
600
400
Backdoor Clean
200
50 100 250 250 300 350 400
L1 norm of the mask
(a) Visualizations
(b) With adaptive training objective
Figure 14: (a) First row: input (BadNets attack) images. Second row: masks generated based on regular backdoored model. Third row: masks generated based the adaptively backdoored model. (b) The distribution of |m]]] for both clean and backdoor samples. This experiment is conducted on CIFAR-10 10 with BadNets (Gu et 2017). The ASR of the adaptive backdoor model 100%.
1400
Number of samples
1200
1000
400
Backdoor Clean
200
100
200
300
400
500 600
L1 norm of the mask
(a) Visualizations
(b) With adaptive training objective
Figure 15: (a) First row: input (Nashville attack) images. Second row: masks generated based on regular backdoored model. Third row: masks generated based the adaptively backdoored model. (b) The distribution of |m]] for both clean and backdoor samples. The experiment is conducted on CIFAR-10 10 with Nashville (Liu et al. 2019). The ASR of the adaptive backdoor model is 15.6 %.
B.8 BIAS DETECTION
Experiment setting, For all the 40 binary facial attributes, we use ResNet-18 (He et al. 2016a) as the feature extractor and attach a classifier head for each attribute. All classifiers share the same feature extractor and are trained with the cross-entropy loss. We train the model for 10 epochs using SGD, with weight decay x 10 -4 For CD, we set a to 0.001. 3 to 1000 and optimize the mask for 300 steps with learning rate 0.075. Figure 16 visualizes the learned masks using our CD-L method. It shows that the masks can locate the area for each facial attribute that matches with human understanding, suggesting that CD-L provides reasonable explanations for the model's prediction.
22

---

Published as a conference paper at ICLR 2023
Stule
Dean
Figure 16: Visualization of the learned masks using CD-L in the bias detection experiments. Positive facial attributes are indicated green and negative red.
C ADDITIONAL EXPERIMENT RESULTS
In this section, we show additional results, including the detection performance under varying poi- soning rates on the test set in Figure 17 and detailed results that are summarized in Table I in the main paper. Detailed results for training and test set detection using area under the ROC curve (AU- ROC) are in Table and Table 8 using area under the precision recall curve (AUPRC) are in Table 7 and Table respectively. The result of using TRP/FPR with one standard deviation (? 1) lower
than the mean calculated on 1 1% of the training subset as the threshold is in Table [0
STEP
STE
STR
CDV
Ch-
25% 3% 15% 10% Poloonigg
230
Polering Sade
62
15%
176
Polkaingg
75% 796
23%
Portsiin
30
1:08 77%
23h 18
Polsring
7.0
(a) BadNets
(b) Blend
CL
(d) DFST
Dynamic
2.4
1.
mingan
(b>
CD.L
CBF
15
Potering
gooriing Take
Toauning
Toboning
Trasning floe
Teasting fucs
(g) Nashville
(h) SIG
(i) Smooth
(i) Trojan
(k) WaNet
(I) ISSBA
Figure 17: The detection performance (AUROC) on test set with different attacks under different poisoning rates [1 %, 2.5%, 5%, 7.5%, 10%]. Results are averaged across different models.
23

---

Published as a conference paper at ICLR 2023
Table 6: The detection performance on training set measured by AUROC ((%) for our CD method and the baselines against 12 backdoor attacks. The poisoning rate is 5% for all attacks. The Clean Accuracy (CA) is evaluated on clean test set. The best detection results are in bold.
Datasct
Attack
Model
CA
ASR
ABL
AC
STRIP
SS
CD-L
CD-F
VGG-16
89.05
100.00
88.31
64.41
94.82
21.41
99.98
99.96
BadNets
MohileV2
84.19
100.00
72.70
91 91.38
98.15
67.42
99.73
94.88
PARN-I01
89.77
100.00
45
98
99.58
74,10
67.69
GoogLeNet
92.84
100.00
97.74
87.05
99.13
26.54
96.34
81.92
VGG-16
89.26
99.50
90.50
65.05
76.37
22.58
97.46
97.76
RN-18
87.77
100.00
78
98.
78.
99.92
96.20
Blend
MobileV2
84.81
99.40
85.69
46.56
88S. 01S
59.43
98.40
84.91
PARN-101
75.69
94.03
GNer
92.39
99.3
84.32
95.04
8R,
38.61
73.66
88 62
VGG-16
89.00
98.40
88
80.19
92.76
46.52
99.99
99
RN-18
87.33
98.40
89:95
81 91
97.91
39.15
99.97
98.44
CL
MobileV2
85.15
90.00
82.95
59.22 39.28
98.02
34.46
98.90
89 89.82
PARN-101
93.60
38.35
95.
88.
GoogLeNet
92.69
$3.60
96.77
90.557
98.75
39.31
99.25
91.26
VGG-16
88.09
100
81.83
37.90
61.93
6.73
92.53
95.59
RN-18
87.34
100.00
96.31
97.11
57.89
98.73
98.42
95.52
DFST
MohileV2
84.67
99.80
80.41
84.47
61.07
40
81.15
68.65
PARNIO1
89.37
99 99.60
94.91
83.83
$7.56
42.42
76.85
85.46
GoogLeNet
92.90
100.00
92.04
98.93
51.95
95.5 39
95.83
67.49
88.94
54.18
99.36
99.
RN-18
87.93
100.00
96.83
95.55
84.98
93.44
99.99
99 42
Dynamic
MobileV2
84.85
99,40
85.15
82.24
97.40
67.34
99.09
92.65
PARNIOI
89.41
99.50
81.39
59.74
95.43
80.03
90.97
85.25
Googl Net
92.99
100.00
83.35
97.43
98.71
4.3 43..47
99.95
98.01
VGG-16
89,06
69.17
RN-18
88.17
99 99.50
85.40
98.
82.66
99.84
99.98
99 .8.68
CI
MohileV2
86.50
99.
71
83 83.08
99.36
98.31
PARNIO1
89.65
99.80
83.75
74.19
85,17
32,93
99.78
97.99
GoogLeNet
92.14
100.00
81.23
99.62
79.16
99.54
99.99
99.02
VGG-16
88.89
98
82,06
69
49.22
70
94.13
95.30
RN-18
87.86
99.00
54.02
92.08
53.413
95.01
98.85
92.52
Naskville
MohileV2
PARNIO1
89.31
98.60
86.91
89.08
49.50
81.46
97.97
94.52
GoogLeNet
92.79
99.80
98
98.79
56.86
97.28
99.87
95.18
VGGt-16
89.22
95.68
97.37
44.336
78.26
39.42
92.22
87.92
RN-18
86.71
99.99 80
99. 22
99 90
81. 1.99
99 96
99.99
95.95
MobileV2
90.80
80.94
28.83 82
95.
95.44
PARNI01
88.92
99.80
99.12
99.64
K5. 90
89.71
98.80
99.23
GoogLeNet
92.80
94.20
93.09
97.17
82.
36.62
97.97
97.92
VGG-16
88.11
96.20
87.52
47.66
65.59
28.99
86.37
78.00
RN-18
87.45
95.20
79. 19.36
95.55
70.13
99.82
97.78
86.50
Smooth
MobileV2
85.80
96.00
76.06
$1.10
65.31
30.72
93.08
82.48
PARNIO1
88.57
96
67.70
91, 9.99
52.64
93.01
83.18
73.38
GoogleNet
92.75
96.00
94.18
95.05
89.
VGG-16
88.47
98.3 20
42
47.16
88,90
47.30
99.11
99 00
RN-18
87.63
98.40
81.71
98.74
91.88
93.76
99.85
98.81
Trojan
MobileV2
85.03
98.00
80.81
67.62
90.14
43.66
98.22
88.55 $8.97
PARNI01
88.30
98. 80
87.31
56.20
94.12
80.96
91.17
83.60
GoogLeNet
9%.22
VGG-11
88.53
97.80
58.73
44,48
90.24
51..94
96.73
92.21
RN-18
98. 38
95.71
85.
WaNet
MobileV2
85.75
96.40
70.96
68. .6.6
83.39
41.85
99.06
90.36
PARNIO
88.46
98.48
37. 66
52.72
76.00
67.61
87.34
70.32
GoogLeNet
92.70
99.
64.52
97.51
94.79
99.62
95.05
VGG-16
97.58
100.00
49.39
99 .48
72.66
70.01
100.00
99. 99
RN-18
98.12
100.00
87.73
99
68.
09
97.35
100.00
GTSRB
BadNets
MobileV2
97.32
100.00
64.27
99.43
58.63
81.98
100,00
98.71
PARNI01
96.68
100.00
39.94
92.95
39.14
06
100.00
99.73
GoogLeNet
99.20
97.56
99.74
47..47
77.72
ImageNet Subset
BadNer
BftieienNNes-00
RN-18
58.13
66.41
100.00
100.00
94.05
72.75
97.87
93.63
98.
93.16
99.83
99.64
100,00
100,00
100.00
100,00
RN-18
57.11
100.00
97.69
100,00
67.49
32.23
100,00
99.98
NeicientNes.b
62.19
100.00
96.29
100.00
73.24
52.22
100.00
99.97
Average
24

---

Published as a conference paper at ICLR 2023
Table 7: The detection performance on training set measured by AUPRC ((%)) for our CD method and the baselines against 12 backdoor attacks. The poisoning rate is 5% for all attacks. The Clean Accuracy (CA) is evaluated on clean test set. The best detection results are in bold.
Datasct
Attack
Model
CA
ASR
ABL
AC
STRIP
5S
CD-L
CD+F
VGG-16
89.05
100.00
17.41
6.43
37.09
2.93
99.92
97.20
BadNets
MohileV2
84.19
100.00
7.89
68.43
62.33
7.45
97.18
68.04
PARNIO1
89.77
100.00
64.66
91.72
12.78
9.92
GoogLeNet
92.84
100.00
78.14
16.22
75.62
3.11
67.80
25.47
VGG-16
89.26
99.50
45.00
6.49
14,44
3.00
76.04
82.06
RN-18
87.77
100.00
32.27
93.35
98.74
Blend
MobileV2
84.81
99.40
47.90
4.34
30.44
5.72
85.46
23.75
PARNI01
99.
Gee
92.39
99 99.20
34.60
74.17
35,26
3.67
9.30
24.67
VGG-16
89.00
98.40
41.29
20.14
46.03
4.20
99.90
96.60
RN-18
87.33
98.40
23.63
19:10
73.55
3.71
99.64
87.95
CL
MobileV2
85.15
90.00
13.39
6.90
79.41
3.43
89.09
52.14
PARNI01
GoogLeNet
92.69
$3.60
64.60
53.27
81.22
3.72
91.90
57.01
VGG-16
88.09
100.00
15.58
3.75
5.91
2.63
41.53
60.99
RN-18
87.34
100.00
81.82
93.03
5.36
78.89
900.55
60.94
DFST
MohileV2
84.67
99.80
12.64
30.17
5.75
3.65
22. 24
9.20
PARNIO1
89.37
99 69.60
84,82
16.47
5.31
3.90
11.32
22.65
GoogLeNet
92.90
100.00
51.09
97.20
4.68
40.15
70.85
7.51
36.35
91.11
RN-18
87.93
100.00
87.78
86.97
37.91
51.34
99.86
94,
Dynamic
MobileV2
84.85
99.40
35.18
53.09
85.67
8.97
92.97
50.54
PARNIOI
89.41
99.50
17.91
7,90
72.73
40.78
53.67
33.49
Googl Net
92.99
100.00
36.64 64
87.68
89.10
4.06
99.40
84.75
VGG-16
38.01
RN-18
88.17
99
40.13
94.81
37.46
94.53
99.65
96.
CI
MohileV2
86.50
99.
23.53
93.05
83.
PARNIO1
89.65
99
32.19
32.82
46.07
3,36
96.78
81.46
GoogLeNet
92.14
100.00
22.46
988003
40.20
82.994
99.91
91.15
VGG-16
98
56
7.38
5.37
3.73
72.67
58.28
RN-18
87.86
99.00
5.16
76.86
6.06
69.72
91.51
51.21
Naskville
MohileV2
96.81
PARNIO1
89.31
98.60
56. 19
49. 90
5.51
13.32
87.04
64.47
GoogLeNet
92.79
99.80
23.62
97.54
6.22
48.48
99.16
66.11
VGGt-16
89.22
35.60
80,42
4,08
200035
3.74
33.51
20.39
RN-18
86.71
99.99 80
94,79
99 66
21 97
98.50
99.89
99.21
MobileV2
45.
3.20
64.92
PARNI01
88.92
99.80
93.78
98.24
00000
18.81
79.23
89.72
GoogLeNet
92.80
94.20
88.30
24.57
3.57
83.04
81.09
VGG-16
88.11
96.20
42.97
4.54
14.12
3.19
33.98
15.82
RN-18
87.45
95.32
33.60
91.
15.59
96.23
84.55
27.
Smooth
MobileV2
85.80
96.00
11.91
29.42
12.18
3.26
59.04
24.23
PARNIO1
88.57
96 06.20
17.90
58.45
6.55
32.05
27.
11.00 11.08
GoogleNet
92.75
96.00
4.01
73.38
42.63
VGG-16
88.47
98 20
23.53
4.31
41.11
4,36
98.01
96.73
RN-18
87.63
98.40
11.72
95.56
59.55
42.87
99.44
97.04
Trojan
MobileV2
85.03
98.00
12.18
9.43
55.72
4.04
92.14
49.50
PARNI01
88.30
98. 80
23.15
5.32
66.44
23.42
46.28
24.85
GoogLeNet
34.57
VGG-11
88.53
97.80
5.57
4.10
33.90
5.43
73.02
71.88
98.
4.60
WaNet
MobileV2
85.75
96.40
8.15
9.60
20.11
3.87
91.45
50.62
PARNIO
88.46
98.48
3.64
4.91
15.81
12.32
22.28
8.72
GoogLeNet
92.70
99.
6.47
93.
$1.52
38.40
60.21
VGG-16
97.58
100.00
4,50
99.50
10.33
7.40
100,00
99 82
RN-18
98.12
100.00
99. 30
8.27
6.35
44.90
99.92
GTSRB
BadNets
MobileV2
97.32
100.00
6.65
99.33
5.91
11.44
100,00
92.65
PARNI01
96.68
100.00
3.79
72.95
3.69
5.17
99.99
94.59
GoogLeNet
99.20
99.44
4.61
67.97
$5.39
RN-18
58.13
100.00
27.81
96.71
77.01
92.13
100,00
100.00
ImageNet Subset
BadNer
BfticienNNes-00
66.41
100.00
8.00
91.76
33.11
83.44
100,00
100.00
RN-18
57.11
100.00
83.02
100,00
9.17
3.17
99.98
99.77
NeicientNes.b
62.19
100.00
56.83
100.00
14.49
4.47
99.98
99 48
Average
97.91
32.88
76.78
25

---

Published as a conference paper at ICLR 2023
Table 8: The detection performance on measured by AUROC ((%) for our CD method and the baselines against 12 backdoor attacks. The poisoning rate is 5% for all attacks. The Clean Accuracy (CA) is evaluated on clean test set. The best detection results are in bold.
Dataset
Attack
Model
CA
ASR
AC
STRIP
SS
CD-L
CD-F
VGG-16
89.05
100.00
72.08
93.95
14.48
100,00
99.96
RN-18
87.60
100.00
90.86
98.61
10.74
100.00
99.97
BadNets
MobileV2
72.02
99.79
94,70
PARNI01
89.77
100.00
36.43
98.54
97.85
77.63
71,90
GoogLeNet
92.84
100.00
82.29
99.05
32.43
96. 18
82.91
VGG-16
89.26
99.60
39.35
74,25
18.34
96.33
96.96
RN-18
87.77
100:00
97.34
78.02
58.43
99.77
96.17
Blend
MobileV2
84.81
40.17
85.55
77.30
97.98
84,83
PARNI01
89.66
99.80
40.47
90.09
20.26
97.69
94,49
Google eNet
92.39
99,20
92.11
87.18
28.30
75.22
89.63
VGG-16
89.00
98
18.
93.32
44,32
93.08
96.05
RN-18
87.33
98.40
29.54
96.95
22.26
97.48
91.70
CL
MobileV2
85.15
90.00
39.38
98.14
47.00
87.14
74.19
PARNI01
89.65
93.60
24.62
97.20
45.10
83,58
79.29
GoogLeNet
92.69
53.60
16.09
94.41
36.39
65.27
60.33
VGG--6
88.09
100.00
86.80
63.30
7.72
93.06
95.15
RN-18
87.34
100.00
96.51
59,93
73.92
97.22
94,49
DFST
MobileV2
84.67
99.80
83.46
59,90
24.04
82.25
69.12
PARNI01
89.37
99.60
69.32
57.72
16.17
80.81
87.26
GoogLeNet
92.90
100.00
98.78
51.71
81.61
95.66
67.37
VGG-16
88.94
100.000
60.53
77.44
33.08
98.93
98.68
RN-18
93.78
99.97
99.35
Dynamic
MobileV2
84.85
99.40
79.60
96.75
67.43
98.54
91.85
PARNIO1
89.41
99.60
54.23
93.72
75.30
91.88
86.03
GoogleNN
92.99
100.000
97.20
98. 12
29.55 93.53
99.92
97.89
VGG-T6
89.06
99.00
87.32
70.44
83.47
92.79
82.23
RN-18
88.17
94.93
83.07
58.01
99.68
99.08
CIFARIO
FC
MobileV2
86.50
99.90
79.55
79.36
69.23
99.26
97.81
PARNI01
89.65
99.80
85,54
82.51
45.66
99.40
97.76
GoogLeNet
92.14
100.00
96.14
69.49
66.74
99.96
98.71
VGGi-1
92.05
RN-18
87.86
99.00
87.50
53.26
69
97.08
91.20
Nashville
MobileV2
85.24
99.40
96,33
44,50
90.32
99.59
98.91
PARNI01
89.31
98.60
53.99
44.96
36.43
98.04
93.95
GoogLeNet
92.79
99.80
92.71
53.37
82.63
99.95
94.97
VGG- 16
89.22
38.99
76.74
RN-18
86.71
99.80
97.71
69.52
95.26
99.85
99.68
SIG
MobileV2
84.45
90.80
43. 10
47.35
43.47
91.42
93.88
PARNI01
88.92
99.80
48.4
67.77
49.20
96.41
98.35
GoogLeNet
92.80
94.20
54.41
47.12
36.61
94.57
97.21
VGG-16
88.11
63.37
63.27
25.35
82.23
77.65
RN-18
87.45
95.20
88.60
67.64
79.47
94.30
85.27
Smooth
MohileV2
85.80
96.00
78.83
62.87
25.76
92.48
81.56
PARNI01
88.57
96.20
61.75
48. 10
32.85
82.60
75.77
GoogLeNet
92.75
96.00
89.87
37.15
92.25
93.52
89.31
VGG-16
88.47
98.20
60.54
89.21
27.25 25
98.15
98.14
RN-18
87.63
98.40
97.31
92.52
41.21
99.17
98.35
Trojan
MobileV2
85.03
98.00
71.84
90.85
44.33
97.79
89.50
PARNI01
88.30
98.80
58.61
93.94
76.13 13
92.83
86.57
GoogLeNet
92.94
69.61
36.26
95.65
86.84
VGG-16
88.53
97.80
56.29
88.48
31.21
95.63
91.15
RN-18
87.51
98.40
91.14
83.57
96.63
97.09
89.39
WaNett
MobileV2
85.75
96.40
60.15
79.14
47.02
98..49
88.55
PARNI01
88.46
98.40
43.49
77.36
33.32
89.55
77.55
GoogLeNet
92.70
98.24
94.66
78.19
99.65
95.51
VGG-16
97.58
100.00
59.50
76.42
77.76
100.00)
100.00
RN-18
98.12
100.00
83,77
70.27
76.22
96.98
100,000
GTSRB
BadNets
MobileV2
97.32
100.00
68.54
61.03
90.09
100.00
98.82
PARNI01
96.68
100.00
54.55
41.23
23.16
100.00
99.75
GoogLeNet
99.20
97.61
49.02
97.05
98.72
99.71
RN-18
58.13
100.00
100.00
98.78
9.66
100.00
100,00
ImageNet Subset
BadNets
EffcientNet-b0
66.41
100.00
100.00
92.90
8.74
100.00
100,00
RN-18
$7.11
100.00
80.34
63.84
67.01
99,99
99.93
ISSBA
EfficeetNNet 60
62.19
100.00
79.74
71.62
45.60
99.99
99.85
Average
87.66
97.91
71.62
75.36
51.16
94.44
90.97
26

---

Published as a conference paper at ICLR 2023
Table The detection performance on test set measured by AUPRC ((%)) for our CD method and the baselines against 12 backdoor attacks. The poisoning rate is 5% for all attacks. The Clean Accuracy (CA) is evaluated on clean test set. The best detection results are in bold.
Dataset
Attack
Model
CA
ASR
AC
STRIP
SS
CD-L
CD-F
VGG-16
89.05
100.00
7.94
32.74
2.79
100,00
96.21
RN-18
87.60
100.00
22.00
62.89
2.71
99.99
99.59
BadNets
MobileV2
66.77
59.68
9.10
97,40
PARNI01
89.77
100.00
3.64
61.09
72.07
13.84
10.78
GoogLeNet
92.84
100.00
12.18
72.08
3.37
66.
26.45
VGG-16
89.26
99.00
5,36
12.38
2.86
75.24
83.41
RN-18
87.77
100.00
91.62
16.62
5.85
97.69
48.83
Blend
MobileV2
84.81
3.76
26.24
10.76
83.59
22.97
PARNI01
89.66
99.80
3.86
33.82
2.92
73,48
44.17
Google eNe
92.39
99.20
79.19
31.69
3,16
9.85
25.50
VGG-16
89.00
98
2.93
31.18
4,64
82.88
87.37
RN-18
87.33
98.40
3.28
48.85
2.96
$1.95
51.87
CL
MobileV2
85.15
90.00
3.71
65.29
4.43
38.22
15.08
PARNI01
89.65
93.60
3.14
52.32
4.42
22.82
17.56
GoogLeNet
92.69
53.60
2.87
36.53
3.55
8.62
6.77
VGG--6
88.09
100.00
17.36
6.12
2.64
46.78
65.40
RN-18
87.34
92.52
5.68
10.86
$8.59
55.71
DFST
MobileV2
84.67
99.80
19.19
5.58
3.01
23.62
9.50
PARNI01
89.37
99.60
7.57
5.37
2.82
13.49
24.14
GoogLeNet
92.90
100.00
97.65
4,66
13.04
69.85
7.57
VGG-16
88.94
100.000
6,06
30,20
3.38
96.67
88.25
RN-18
99.67
Dynamic
MobileV2
84.85
99.40
43,36
84,12
9.85
91.21
49.02
PARNIO1
89.41
99.60
5.35
65.98
32.66
51.69
30.96
Google
92.99
100.00
89.72
85,54
3.21
98.99
83.95
VGG-T6
89.06
99.00
80,25
25.39
21.69
60.63
36.64
RN-18
88.17
90.39
46.22
10.27
98.59
96.10
CIFARIO
FC
MobileV2
86.50
99.90
60.99
46.05
21.54
95.92
90.26
PARNI01
89.65
99.80
75.21
47.10
7.77
96.84
88.34
GoogLeNet
92.14
100.00
91.70
33,30
12.62
99.70
94,40
VGGi-1
4.55
3.13
RN-18
87.86
99.00
73.57
5.11
14.11
86.07
47.39
Nashville
MobileV2
85.24
99.40
78.97
4.19
20.08
96.34
91.77
PARNI01
89.31
98.60
4.95
4.28
3.57
86.74
64.57
GoogLeNet
92.79
99.80
33.16
5.04
11.95
99.29
66.73
VGG- 16
89.22
4.19
17.94
RN-18
86.71
99.80
93.53
10.62
36.67
918.39
95.81
SIG
MobileV2
84.45
90.80
3.97
4.51
4.12
44,23
47.16
PARNI01
88.92
99.80
4.58
9.66
4.44
59.63
80.38
GoogLeNet
92.80
94.20
4.92
4.79
3.56
61.32
72.31
VGG-16
88.11
6.35
11.89
3.07
28.64
14.49
RN-18
87.45
95.20
77.84
12.56
15.92
77.01
24.10
Smooth
MohileV2
85.80
96.00
31.21
10,00
3.06
58.04
22.50
PARNI01
88.57
96.20
6.31
5.35
3.35
25.82
11.36
GoogLeNet
92.75
96.00
78.54
3.82
38.34
71.78
44.15
VGG-16
88.47
98.20
5.65
39.78
3.13
96.31
96.77
RN-18
87.63
98.40
94.87
60.40
3.85
98.28
96.06
Trojan
MobileV2
85.03
98.00
10.42
$5.69
4.07
91.00
50.88
PARNI01
88.30
98.80
5.47
66.11
10.16
48.39
27.67
GoogLeNet
92.94
7.48
74.75
3.57
66.60
VGG-16
88.53
97.80
5.22
26.99
3.39
69.75
70.10
RN-18
87.51
98.40
80.42
20,20
$1.96
45.86
45.71
WaNett
MobileV2
85.75
96.40
6.50
14.37
4.28
86.29
45.04
PARNI01
88.46
98.40
4.09
15.02
3.38
28.14
12.28
GoogLeNet
92.70
96.16
49.20
11.09
94,90
64.21
VGG-16
97.58
100.00
5.53
11.64
9.90
99.99
99.98
RN-18
98.12
100.00
19.44
8.55
9,36
41.69
99.98
GTSRB
BadNets
MobileV2
97.32
100.00
7.15
6.21
19.21
100.00
93.14
PARNI01
96.68
100.00
4.93
3.81
3.04
100.00
97.43
GoogLeNet
99.20
42.26
62.19
92.16
RN-18
58.13
100.00
100.00
73.21
2.69
100.00
100,00
ImageNet Subset
BadNets
EffcientNet-b0
66.41
100.00
100.00
32.35
2,70
100.00
100,00
RN-18
$7.11
100.00
39.97
9:05
6.56
99.82
99.37
ISSBA
EfficeenNNet 60
62.19
100.00
33,57
14.18
3.97
99.88
98.80
Average
87.66
97.91
38.2
28.44
10.4
71.69
59.38
27

---

Published as a conference paper at ICLR 2023
Table 10:
The detection performance on test set measured by true positive rate (TPR) and false positive rate (FPR) for CD method and the baselines against 12 backdoor attacks, reported as TPR/FPR. For detection methods, classification threshold i is set to standard deviation away from the mean calculated for 1 % of the clean training The poisoning rate is 5% for all attacks. The best detection results are in bold, best is selected base on TPR first then use FPR for tie-breaking. TPR higher the better, FPR lower the better.
Daraset
Antack
Mixell
AC
STRIP
58
CD4.
CIFE
VGG-16
60,60668.11 14
18.60111.8
19,40168,04
100.00/00997
100.007,993
RN-18
99 00880/1.00
100.00/8.55
93,80/79.08
100.00/14.54
100.00/1-133
BadNets
MobileV2
92.80/38.27
91.06655377
92.20/16.49
PARN101
38.20/58.65
100.00/9.00
100.00/73.41
44.01/14.68
35,20/14,99 93
GoogLeNet
99.40/53.12
100.00/
30.20275.32
VGG-16
94,2062822
34.80712.99
4L.6666606
Blend
MobileV2
20.00/35.73
66.001
97. 97.00M64.02
97.40/16.22
69.00/16,7 76
51.40/52.78
60733.9
9111115,117
GeoglaNNet
93.80550844
71 1018801100 1666
46 46.4067.99
37.6055
$1.00/16.05
4.8055.75
5400868,49
S4.8072.0
89.8099.1
RN-18
14.40/52.54
97.009.79
56 56.40/7.96
95. 3.59
81. 31,40/13,71 1
MobileV2
16.00/36.30
97.009997
00/66.81
16.82
80/17.07
PARN101
5,80/54.40
46.09/983
64 20/72.77
52. 20/1 3.36
47.60/13.99
GeogLeNet
88.4040
65 20/70.57 57
22.80/14.75
VGG-16
99.40/66.74
10.80114.66
0.00F55
82 12/77/
85.80/10.75
RN-18
97. 97.20/8..11
8,40113.60 60
100.0080.61
14
91,20/1
DEST
MobileV2
89.00/36.46
4.40/15.31
28. 80162.75
66.40/16.18
37,80/16.99
PARN101
92.80/54.97
40/13.77
40/72.36
6.
72.40/1
GoogLeNet
99.00/3..41
6.40/14.81
100.00/5.008
92. 20/100/5,,4
24.60/15.44
VGG-16
86 86/8012771
80700.1
97.60/13.38
17,247
RN-18
96. 06.2058..8 81
62.40/10.20
99. 20/79.74
100.00/13.91
99.40/14.00
Dynamic
MobileV2
78-4038.21
91, 81.807.6
84 60
97.40/15.51
84. 64.6/16.23
PARN101
60.00/53.23
83,20/9.20
89.60/3.200
80.60/13.61
66.20/14.71
GeogLeNet
99.00/53.68
94.80/7.74
59. 60/79/9993
100,00/15.66
96,80/15,82
VGG-16
36.71722
90. 38/83.7
06.0933
82.37/33.T0
RN-18
99.60/99.07 .3
72.75/17.68
92.99/81.22
80/3444884
99. 10/27.9
CIFARIO
FC
MobileV2
96. 36.79/90.31
66.23/16.70
55. 11/81999999
99.70/3.07
98.30 18.30/30.31
PARNI01
99 99.90/33.11
94/44/4
95.49/74.02
99. 74/33.34
9%, 60/29/8/388
Googl LeNet
99. 70/99.99.
53.51/19448
97.90/73.22
100.00/3.57
10/31.9
VGG-16
82.8066027
94011361
76.80771.6
63.6411348
10.00
RN-18
91.00599.96
12.2011.27
98.20/80.32
93.40 40/14.55
$2.00/14.52
Nashville
Mobile
97.60/35.34
8.60/13.96
99.60/65.47
40/15.45
98, 60/16776.27
PARN101
67.40550.68
9.40/44
96 00/74.9
96.60/14.55
S8. 14.69
GoogLeNet
98.4044..7
11 85
100 00/77.77 71
100.00/16.05
89, 00.40/15775
VGG-16
69. 19.6833118
23.2011.35
43.20/63.29
39 39.00/12.0
40 00/8.61
RN-18
98. 20633
37.00112.46
100.00/80.44
99. 2.45
99, 8./13.72
SIG
MobileV2
14,20/37.92
71 61.6666.08
10/1.664
91.40/16.76
PARN101
93.40661.20
3211112 60112003
90 90.40M62.44 49
93 93.40/122 2.99
98. 18.6071
Googl LeNet
51,00/43.82
15.00 7011174
2072.17
10/14
96.40/15.61
VGG-16
83.20/54.71
36, 86.1112336 36
34.4086777 78
63 607 3.22
28 0377 89
RN-18
91.00/22.3
100.00/80.00
00/14.54
Smooth
Mobile
78.40V38.00 07
35 33.0013311
43 66/70.066
86.20/16.12
60. 60.66/17.11 11
PARN101
80.80/58.97
80112.1
$1.20/00.6
40, 00/444444.54
GoogLeNet
$0.80551.97
10 8.61466
99.60/73.55
88 00..0.11155
77. 77.40/15.85
VGG-16
80162.4244
00.6069.000
97.00/12.78
RN-18
98.00/61.13
83 60/9 80
99.6079.44 .44
98. 40/13.666
97.40/1 400/14400 40
Trojan
MobileV2
71.60/36.96
20
27,20164
96.80/15.35
PARN101
S8. 20.20/57.55
87 37.00/928
45/44/44
64.60/14.45
68. 11..61/15.L
GoogLeNet
91.20/55.22
60/7.60
VGG-16
81.60:58332
11.11
92.80//22.17
11.
93.60/56.32
1.9
100.0/80.69
99.40/10.73
WaNet
MobileV2
54,40/36.92
43.00V1.80
92.3 80168.662 62
98.20/15.09
78.00/16. 17
51.80/53.60
61/7.14
GoogLeNet
98.40/52.62
87.41110.51
100.00/74.91
99 90/1422
93.00/16,26
VGG-16
95.88/65.52
100 13.44/10.15
100.0044.88
26.21/0.00
RN-18
100.0092.02
29. 16/1266
100 06168. 26
100.00/11
100.0000.54
GTSRB
BadNets
MobileV2
99.21/72.83
25. 23.36/18.39
100
100.00/.06
79 5670.00
PARN100 101
98.57/78.88
2.22/15.74
90, 17775.54
100.00/7.85
98 89/2.90
GoogLeNet
100.0081.99
14. 14.10/16.13
100.0079.0
100.00/968 68
27 27.26/0.00
RN-18
100.0053.11
99 801333
61.0871.11
100.06/32.9
100.00/22.15
BadNets
ImageNet
EfficientNes.bo
100.00/10.61
90.00115.49
0.4070.23
100.06/20.07
100.00/20.2
Subset
ISSBA
RN-18
67,00448992
77.X077.42
10000553.2
100.0R7X1.2
EfficienNNe.hh0
75.00/28.48
56.00/1.14
64.40/70.8
100.00/40.25
100.00/27.25
Average
79.54/56.68
75.0W71.98
88.3W16.92
79.25/15,26
28

---

Published as a conference paper at ICLR 2023
D ADDITIONAL VISUALIZATIONS
Figures 18 to 31 show the randomly selected input images, learned mask, and distilled CP of both
clean and backdoored samples for each corresponding experiment evaluated in section The masks and CP are extracted from the corresponding backdoored model (ResNet-18) using CD-L.
(a) Clean samples
(b) Backdoor samples
Figure 18: Corresponding input images, learned masks and CPs for BadNets (Gu et al. 2017 on CIFARIO.
(a) Clean samples
(b) Backdoor samples
Figure 19: Corresponding input images, learned masks and CPs for Blend (Chen et al. 2017 on CIFAR10.
(a) Clean samples
(b) Backdoor samples
Figure 20: Corresponding input images, learned masks and CPs for CL (Turner et al, 2018 on CIFARIO.
6
(a) Clean samples
(b) Backdoor samples
Figure 21: Corresponding input images, learned masks and CPs for DFST (Cheng et al. CIFAR1O.
2020 on
29

---

Published as a conference paper at ICLR 2023
(a) Clean samples
(b) Backdoor samples
Figure 22: Corresponding input images, learned masks and CPs for Dynamic Ngguen & Tran
2020 CI
(a) Clean samples
(b) Backdoor samples
Figure 23: Corresponding input images, learned masks and CPs for FC (Shafahi et al. 201 8
on
CIFARIO.
(a) Clean samples
(b) Backdoor samples
Figure 24: Corresponding input images, learned masks and CPs for Nashville [Liu et al. 2019 on CIFAR1O.
(a) Clean samples
(b) Backdoor samples
Figure 25: Corresponding input images, learned masks and CPs for SIG (Barni et al. 2019 on CIFAR1O.
(a) Clean samples
(b) Backdoor samples
Figure 26: Corresponding input images, learned masks and CPs for Smooth (Zeng et al. 2021 on CIFAR1O.
30

---

Published as a conference paper at ICLR 2023
(a) Clean samples
(b) Backdoor samples
Figure 27: Corresponding input images, learned masks and CPs for Trojan (Liu et al. CIFARIO.
2018b on
(a) Clean samples
(b) Backdoor samples
Figure 28: Corresponding input images, learned masks and CPs for using WaNet trigger (Nguyen
Tran 2021 on CIFARIO.
STOP
90
(a) Clean samples
(b) Backdoor samples
Figure 29: Corresponding input images, learned masks and CPs for BadNets (Gu et al. 2017 on GTSRB.
(a) Clean samples
(b) Backdoor samples
Figure 30: Corresponding input images, learned masks and CPs for BadNets (Gu et al. 2017 7 ImageNet subset.
on
(a) Clean samples
(b) Backdoor samples
Figure 31: Corresponding input images, learned masks and CPs for ISSBA (Li et al. 2021c
ImageNet subset.
31


Reviews:
[{"review_id": "x9ttfJDsWHb", "reviewer": "Reviewer_czZL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a defensive mechanism, Cognitive Distillation (CD), that extracts potential trigger patterns (and a mask) responsible for the model's prediction. The paper starts with the observation that, even if the trigger pattern consists of multiple pixels and/or is complex, a few pixel values correspond to activating backdoors. The paper exploits this observation and proposes an optimization that reconstructs a potential trigger pattern and a mask from an input image. Using those reconstruction outputs, the paper identifies whether the input contains the backdoor or not (if the mask is sufficiently small, then it contains a backdoor). In evaluation, the paper compares the detection rate with four baseline defenses. The paper also discusses the possibility of using CD for bias detection.", "summary_of_the_review": "This paper studies a method to capture input samples containing potential triggers that may lead to backdoor behaviors. The proposed method relies on the empirical observation that backdoor models only use a subset of pixels in the actual trigger pattern. However, the major problems of this approach are as follows:\n\n(1) The assumption can be easily broken.\n(2) The optimization process (which is the paper's technical contribution) is not novel.\n(3) The paper lacks the evaluation against adaptive attacks\n(4) It's unclear about the novelty of bias detection.\n\nThus, I believe the weaknesses outweigh the strengths of this paper.", "strength_and_weaknesses": "Strengths:\n\n1. The paper empirically analyzes the commonality across the backdoor attacks.\n2. The paper exploits the commonality to build a backdoor input detection method.\n3. The paper discusses some other useful ways to harness CD.\n\n\nWeaknesses:\n\n1. It's not convincing (from the paper) that the backdoored model only uses a subset of the trigger.\n2. The technical novelty of the proposed method (CD) is weak (CD is an adaptation of Neural Cleanse).\n3. The evaluation is weak (varying poisoning rates won't reflect adaptive attacks, and Neural Cleanse may perform better under the same setting).\n\n\nDetailed comments:\n\nThe paper studies a way to capture input samples containing potential triggers that may lead to backdoor behaviors. As backdoor attacks are a growing concern, it is important to study defense mechanisms to reduce the attack surface. This paper strives to find the commonality across the backdoor behaviors. \n\nHowever, the major problem of this approach is that the defense relies on an assumption (backdoored models will use a subset of trigger patterns for classification) that can be easily broken. Another concern is that the optimization process (which is the paper's technical contribution) is highly similar to the process proposed by Wang et al., Neural Cleanse. The paper (moreover) somewhat lacks the evaluation against adaptive attacks, and the discussion about bias detection comes out of the blue. Thus, I think this paper is not ready for ICLR 2023. \n\nMy detailed comments can be found below:\n\n\n[On the Common Observation]\n\nThe paper's claim depends on the observation that most backdoored models rely on a small subset of pixels over the entire trigger pattern. I don't think this is true, and the paper studies only a subset of backdoor attacks. I assume that this may come from the way we train our models. However, as the attacker controls the entire training process, they can add an objective that hinders backdoor behaviors in the (partial) presence of the trigger pattern the attacker uses. By doing so, this defense cannot detect any backdoor inputs at all.\n\nMoreover, even if we just assume that the backdoor attacks \"only\" use a small subset of pixels in the trigger pattern. There is a problem that the proposed CD can \"exactly\" reconstruct the pattern used by the attacker. Prior work [1]---that proposed a similar reconstruction method---showed some negative results regarding this. A backdoored model can contain multiple trigger patterns that aren't injected by the attacker. If CD reconstructs one of those patterns, can we say that CD detects backdoored input? \n\nIn the long run, we may forecast a diminishing return. Are we sure whether a clean classifier does not naturally learn such small patterns? As the resolution increases (224+), I assume that there are several minor details in the dataset that a model can learn and focus on. In this case, can we say that a model is backdoored, or does it learn the correct features? Just relying on the assumption that a mask will be small won't lead to a successful defense in the end.\n\n[1] Sun et al., Poisoned classifiers are not only backdoored, they are fundamentally broken.\n\n\n[On the Technical Novelty]\n\nThe optimization process proposed by this work has already been exploited in the backdoor literature. Neural Cleanse proposed a trigger reconstruction process that relies on the same assumption; the pattern will be composed of a smaller number of pixels. [1] also proposed a reconstruction process (that seems to be more advanced than this paper uses, as they use smoothing with a denoiser). Thus, the novelty is less strong.\n\n\n[On the Adaptive Attacks and Bias Detection]\n\nAs pointed out above, the attacker can train a model with an objective that forces the model to use all the pixels in the trigger pattern to trigger backdoor behaviors. By combining this objective and a pattern that uses all the input pixels (such as the trojan trigger used in the literature), the attacker could evade this defense completely.\n\nIt's nice that this paper discusses other use cases of CD. However, it's a bit unclear what scientific advances CD makes over the prior approaches to detecting biases in the training data. Unfortunately, in the evaluation, the paper does not draw any comparison with the baseline bias detection. Thus, I can make a conjecture that CD can detect biases, but I am unsure whether CD will have a novelty over the prior work.\n\n\n[Minors]\n\n1. (In the intro) References to the work that formulates backdooring as a multi-task learning is missing. It has been studied and presented in the \"Blind Backdoors in Deep Learning Models\" paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "b-I69Apk_lh", "reviewer": "Reviewer_QeF1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors target to detect the input masks that extract the minimal backdoor patterns that lead to the model output. They further leverage the learned masks to detect and remove backdoor samples from poisoned datasets. Experiments show that they can detect several backdoor attacks and help detect biases from face datasets.", "summary_of_the_review": "See the above.", "strength_and_weaknesses": "*************************\nStrengths\n*************************\n+ Clear exposition: well-organized demos and clear writing.\n+ Reasonable motivation.\n+ Straightforward method.\n+ Extensive experiments covering 3 datasets, 6 victim model architectures, 12 backdoor attacks, and 4 baseline detection methods. Results show their efficacy in terms of backdoor detection accuracy and face bias detection accuracy.\n+ Easy to reproduce.\n\n*************************\nWeaknesses\n*************************\n- Technical novelty is incremental.\n  - It borrows the idea form Neural Cleanse that embeds backdoor patterns in a minimal image subregion.\n\n- The threat model is unreasonable.\n  - The detection can only apply to local patterns. Simple global patterns, e.g., patterns in the frequency domain, can easily circumvent the detection.\n\n- Experiments are unconvincing.\n  - Backdoor pattern detection recall looks reasonable, but the precision is more measured. Please test on clean samples and report the false detection rate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RfUwXk7knA", "reviewer": "Reviewer_Xw5r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The given work proposes their \"Cognitive Distillation\" (CD) approach for distilling, detecting backdoor patterns from pixels. The author's claim that this is able to extract the minimal pattern responsible for a model's prediction. Based on their studies, they argue that a sparse input pattern is responsible for backdoor predictions regardless of the trigger pattern. They evaluate their approach on various architectures, standard datasets and attack types. The authors argue that could help detect potential bias in real world datasets.", "summary_of_the_review": "To the best of my knowledge, the authors provide a reasonable contribution to mitigate adversarial backdoor patterns. The work is well written, with the empirical evaluation answering the questions asked. The work seems reasonably novel clearly stating the related work, would be interesting to see it expanded to challenging benchmark tasks. Overall, recommending an accept.", "strength_and_weaknesses": "Strengths\n* The paper is clearly written, well organized and easy to follow.\n* The related work is well motivated.\n* The results and ablation studies directly answer the authors claims\n\nWeakness\n* Nit: The theoretical motivation could be expanded in Section 3.1. How were each of the terms generalized and weighted?\n* The case study on based on relatively simple datasets, it would be great to see this expanded on diverse distributions. \n* It would be interesting to see this approach not only in image space, but including temporal dependencies such as in videos.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ORKPxsXCj52", "reviewer": "Reviewer_o4Nd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel backdoor defense called Cognitive Distillation (CD) that defends against data-poisoning attacks. Given a suspect model, CD extracts the minimal essential pattern, called Cognitive Pattern (CP), responsible for the model's prediction on each image input. It observes that backdoored models often provide exceptionally small CPs on poisoned inputs, and this can be used to detect backdoor examples. Then, CD can employ Anti-backdoor learning on the purified dataset to mitigate the model's backdoor effects. CD provides high AUC scores when detecting backdoor samples generated by various backdoor attacks on CIFAR-10. In backdoor mitigation, CD reduces the ASRs better compared with the previous mitigation baselines. CD is also used to detect potential biases on a face attribute classification dataset.", "summary_of_the_review": "The proposed idea is new and interesting. However, I have some concerns about the evaluation metrics, the mitigated clean accuracy, and some defense results, as mentioned in the Weaknesses part.", "strength_and_weaknesses": "### Strengths\n- The paper proposes a new idea of extracting the minimal essential pattern for each input image. It points out that such a pattern is often exceptionally small when the model and the input data are poisoned.\n- Based on that observation, the authors proposed a new method to detect backdoor samples. Such a detection scheme provides high AUC scores when detecting backdoor samples generated by various backdoor attacks on CIFAR-10.\n- The paper then proposes to mitigate the model's backdoor effects via Anti-backdoor learning, which effectively reduces the models' ASRs.\n- CD is also used to detect potential biases on a face attribute classification dataset.\n\n### Weaknesses\n- FAR should be normalized by the poisoning rates. For example, given a dataset with 5% data poisoned, a random guess would provide a FAR of 5%, which sounds low but actually not. \n- Given the data imbalance (poisoned vs. clean) and the issue mentioned above, PR curves are preferable over ROC curves, and AUC-PRs are preferable over AUC-ROCs. While TRR and FAR are helpful metrics, Recall is the most critical metric (the percentage of poisoned data that can be detected and removed).\n- The mitigated clean accuracy values in Fig.4 are too low, given the original numbers are often around 88-92% (Table 7). Given that significant accuracy drop, these mitigation approaches are impractical. \n- Also, the authors should conduct experiments on clean datasets to see how many clean examples are falsely detected and how much it affects the mitigated clean accuracy.\n- The paper provides a massive amount of experiments. However, most experiments are conducted on CIFAR-10.\n- In their original papers, recent backdoor attacks such as Dynamic and WaNet can easily pass STRIP by providing similar entropy histograms as from the clean model counterparts. However, this paper's detection AUCs for STRIP on these attacks are unexpectedly high. Can the authors explain the reason?\n- The defense baselines are pretty old. The authors should include some more recent backdoor detection approaches, such as [1].\n- While interesting, the bias detection part is irrelevant to the rest of the paper.\n\n[1]. Zeng Y, Park W, Mao ZM, Jia R. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV 2021 (pp. 16473-16481).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster