Activation Gradient based Poisoned Sample Detection Against Backdoor Attacks
Authors: Danni Yuan
Mingda Zhang
Shaokui Wei
Li Liu
Baoyuan Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Backdoor Defense
Poisoned Sample Detection
AI security
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: This work studies the task of poisoned sample detection for defending against data poisoning based backdoor attacks. Its core challenge is finding a generalizable and discriminative metric to distinguish between clean and various types of poisoned samples (e.g., various triggers, various poisoning ratios). Inspired by a common phenomenon in backdoor attacks that the backdoored model tend to map significantly different poisoned and clean samples within the target class to similar activation areas, we introduce a novel perspective of the circular distribution of the gradients w.r.t. sample activation, dubbed gradient circular distribution (GCD). And, we find two interesting observations based on GCD. One is that the GCD of samples in the target class is much more dispersed than that in the clean class. The other is that in the GCD of target class, poisoned and clean samples are clearly separated. Inspired by above two observations, we develop an innovative three-stage poisoned sample detection approach, called Activation Gradient based Poisoned sample Detection (AGPD). First, we calculate GCDs of all classes from the model trained on the untrustworthy dataset. Then, we identify the target class(es) based on the difference on GCD dispersion between target and clean classes. Last, we filter out poisoned samples within the identified target class(es) based on the clear separation between poisoned and clean samples. Extensive experiments under various settings of backdoor attacks demonstrate the superior detection performance of the proposed method to existing poisoned detection approaches according to sample activation-based metrics.

Published as a conference paper at ICLR 2025
ACTIVATION GRADIENT BASED POISONED SAMPLE DE- TECTION AGAINST BACKDOOR ATTACKS
Danni Yuan ; Mingda Shaokuii Wei Li Liu", Baoyuan Wult
Schooo of Data Science, The Chinese University of Hong Kong. Shenzhen, Guangdong, 518172, P.R. China 2 The Hong Kong University of Science and Technology (Guangzhou)
[danniyuan, mi mingdazhang, chaokuiweil @ilink. cuhk edu.cn
avrillliu@hkust gz. edu. cn wubaoyuan@cuhk. .eak.edduucco on
ABSTRACT
This work studies the task of poisoned sample detection for defending against data poisoning based backdoor attacks. Its core challenge is finding a generaliz- able and discriminative metric to distinguish between clean and various types of poisoned samples (e.g., various triggers, various poisoning ratios). Inspired by common phenomenon in backdoor attacks that the backdoored model tend map significantly different poisoned and clean samples within the target class to similar activation areas, we introduce novel perspective of the circular distribu- tion of the gradients w.r.. sample activation, dubbed gradient circular distribution (GCD). And, we find two interesting observations based on GCD. One is that the GCD of samples in the target class is much more dispersed than that in the clean class. The other that the GCD of target class, poisoned and clean samples are elearly separated. Inspired by above two observations, we develop an innova- tive three-stage poisoned sample detection approach, called Activation Gradient based Poisoned sample Detection (AGPD). First, we calculate GCDs of all classes from the model trained on the untrustworthy dataset. Then, we identify the target class(es) based the difference GCD dispersion between target and clean classes. Last, we filter out poisoned samples within the identified target class(es) based on the clear separation between poisoned and clean samples. Extensive experiments under various settings of backdoor attacks demonstrate the superior detection performance of the proposed method to existing poisoned detection ap- proaches according to sample activation based metrics. Codes are available at https//gghhnb.ooommccccc//cocooo (PyTorch)
INTRODUCTION
is well known that deep neural networks (DNNs) are vulnerable to backdoor attacks (Wu et al. 2023), where the adversary could inject a particular backdoor into the DNN model through manipu- lating the training dataset or training process. Consequently. backdoored model will produce a target label when encountering a particular trigger pattern, leading to unexpected security threats in practice. Protecting DNNs from backdoor attacks is an urgent and important task.
Here we focus on defending against the data- poisoning based backdoor attacks by filtering out the potential poisoned samples from a untrustworthy training dataset, i.e., poisoned sample detection (PSD). One of the main challenges for PSD the information lack of the potential poisoned samples, such as the trigger type, the target class(es), the number of poisoned samples, etc. Some seminal works have been developed by exploring some discriminative metrics based on the intermediate activation predictions of poisoned and clean samples in the backdoored model trained on the untrustworthy dataset, such as activation clustering (AC) (Ma et al. 2023a STRIP (Gao et al. 2019). SCAn (Tang et al. 2021). However, the assumption that poisoned and clean samples can be distinctly separated in activation space has been challenged in some recent backdoor attacks (Qi et al. 2023a).
"Equal contribution
1 "orresponniig Author

---

Published as a conference paper at ICLR 2025
In this work, introduce novel perspective that distinguishes the behavior of poisoned and clean samples by tracking their activation gradients (i.e, the gradient W.E.. activation). It is inspired by the phenomenon that backdoored model tends to map both poisoned and clean samples within the target class to similar areas in its activation space (Huang et al. 2022), such that they can be predicted as the same label. Considering the significant discrepancy between poisoned and clean samples in their original input space, their mapping directions should be significantly different, while the mapping direction could be reflected by the activation gradient direction. Thus, we define a new concept called gradient circular distribution (GCD) (introduced in Sec. B). to capture the distribution of activation gradient directions. Take Fig. [ as the example, given trained model, we calculate one GCD of training samples in each class. There are two interesting observations:
Observation 1 on GCD dispersion: Given one backdoored model (see middle/right sub-figures), the target GCD is much more dispersed than GCDs of all clean classes.
Observation 2 on sample separation in target GCD: In the GCD of target class, poisoned and clean samples are clearly separated (see the black and blue ares in middle/right sub- dub-fiiures), and they locate at two separated clusters.
Motivated by above two observations, we de- velop an innovative poisoned sample detection approach, called Activation Gradient based Poisoned sample Detection (AGPD), which consist of three stages. First, we train a DNN model based on the untrustworthy dataset, and calculate GCDs of all classes. Second, we iden- tify the target class(es) according to a novel class-level metric that measures the dispersion of each class's GCD (corresponding to the first observation). Last, within the identified target class(es), we gradually filter out poisoned sam- ples according to novel sample-level metric that measures the closeness to the clean ref- erence sample (corresponding to the second observation). Moreover, we conduct extensive evaluations under various backdoor attacks and various datasets, and show that the activation gradient is more discrininative than the activation to distinguish between poisoned and clean samples, which explains the superior.
B-12 Cleen
Figure 1: Gradient circular distributions (GCDs) across four classes of CIFAR-10, on the clean model (left), Blended attacked model (middle). and SSBA attacked model (right), respectively. The value along with each are indicates the CVBT value. The GCD of the target class (covering both black and blue arcs). Note that we moved three clean classes' ares to different quadrants to avoid visual overlap.
In summary, the main contributions of this work are three-fold. (1) We introduce a novel perspective for poisoned sample detection, called gradient circular distribution (GCD), and present two interesting observations based on GCD. (2) We develop an innovative approach by sequentially identifying the target class(es) and filtering poisoned samples for the poisoned sample detection task, based on GCD and two novel metrics about GCD. (3) We conduct extensive evaluations and analysis to verify the superiority of the proposed approach to existing activation- based detection approaches.
2 RELATED WORK
Backdoor attack. BadNets (Gu et al. 2019) is the pioneering work that introduces the concept of backdoor attack into Deep Neural Networks (DNNs), in which the adversary manipulates training samples by adding a small patch with specific patterns and changing their labels to a target label. Following this, the variety of triggers expanded significantly. including a cartoon image used in Blended (Chen et al.) (017). universal adversarial perturbation with only low-frequency components utilized in Low-Frequency (Zeng et al.] 2021). and a sinusoidal signal employed in SIG (Barni et al., 2019), etc, which use same trigger across different poisoned samples. Sample-specific triggers have been designed, such as WaNet (Nguyen & Tran 2021). Aaput-Aware (Nguyen & Tran 2020). SSBA et al., 202 CTRL (Li et al. 2023). TaCT (Tang et al., 202I). and Adap-Blend (Qi ct al. 2023a These attacks use more complex and dynamic triggers, posing significant challenges for poisoned sample detection. Additionally, some attacks explore various attack settings regarding the number of triggers and target classes. such as all-to-all attack (e.g., Bad (Gu et al. 2019)), tulti-target and multi-trigger attack (e.g., c-BaN (Salem et al. 2022). These diverse settings further complicate the detection of poisoned samples.
2

---

Published as a conference paper at ICLR 2025
Backdoor defense. According to the accessible information, several different branches of backdoor
defense methods have been developed, such as the pre- -eettraning backdoor defense (e.g.. (Ma et al.
2023a Tran et al. 2018 Kader Hammoud et al. (203 if given a untrustworthy training dataset, et al.
in-training backdoor defense (e.g. (Huang et al., [022]
2021a Chen et al. 2022 Mu et al.
2023 Gao al. 2023) if the training process can be lontrolled by defender, well as post-training
backdoor defense (e.g. (Liu et al. 2018 Zhu et al. 2023b a Wei et al. 2023; Wang et al. Wu
Wang 202T: Zeng al. 2022 Zheng et al. 2022b Chai & Chen 2022; Zheng al. 2022a)
if given backdoored model. Due to space limitations, we will only review existing methods of poisoned sample detection (PSD), which belongs to pre-training backdoor defense. Most existing
PSD methods aim to construct discriminative metrics between poisoned and clean samples based on intermediate activations, final predictions, or loss values. For activation-based methods, such as
activation clustering (AC) (Ma et al.) 2023a), Beatrix (Ma et al. 2023b), SCAn (Tang et al. [02T). and Spectral (Tran et al. 2018). they utilize dimensionality reduction and clustering techniques, such as K-means clustering. Gram matrix analysis, two-component decomposition. and SVD, to distinguish poisoned and clean samples. For input-based methods, STRIP (Gao et al. 2019) uses the entropy of predictions perturbed inputs identify poisoned samples, while CD (Huang et al. (222)) measures the L1 norm of the learned masks on inputs to detect poisoned samples. For loss-based methods, like ABL (Li et al. 2021a) and ASSET (Pan et al. 2023), they observed that the loss of poisoned samples decreases quickly during the early training epochs, leveraging this phenomenon to identify poisoned samples.
PRELIMINARY: GRADIENT CIRCULAR DISTRIBUTION
Task setting. Given a DNN-based classification model fw X - V. with X E Rd being the input sample space and y {1, 2. K} being the output space with K candidate classes, as well as dataset = ((x;, y/) ) D we investigate their gradients of fw-
3.1 DEFINITION OF GRADIENT CIRCULAR DISTRIBUTION
Here we introduce the definitions Activation Gradient and Gradient Circular Distribution (GCD), described in Definition and Definition respectively.
Definition 1 (Activation Gradient) Given model f for a sample labelled as y, we denote its activation map at the 1-th layer as h RC xH xwo; where C H(), W are its depth
(number of channels), height and width, respectively. Then, define the channel-wise activation gradient ( (xx E RCO)) as
Ha) w
(1) Iw (x, v) HOWO
1
fi (x)]2
E gC
(I)
O hh J.A..w
yl is the logit w.r.t. class y and [h I.nw E BC is the activation sliced height h and width w over all channels. For simplicity, if no special specifications are required, hereafter we will refer to it g (x)
where (x))
(1)
(x) for each layer l.
Definition 2 (Gradient Circular Distribution (GCD)) Given model f-((),, set of samples D {(x,, yi) yi and a basis sample pair (Zu, yo). we firstly calculate the activation gradient of each sample for each layer I, i.e. (xx,)) i == 0, 1...., n, Then, take g( as the (unnormalized) basis vector, the angle of each sample in 'D is calculated as follows:
(2.)) g(() (xx)
) E 2T), ,,..,,
(xx)) arccos
(x) (xx))l
Illl (xx))
where- denotes dot product, and || || returns the magnitude. The distribution of the angle set (00) [z is called as the gradient circular distribution (GCD) of D, denoted as P!ll (D) for each layer I.
3.2 CHARACTERISTICE OF GRADIENT CIRCULAR DISTRIBUTION To accurately capture the characteristics of P!!!) (D) observed Fig. and Sec. we introduce the following two metrics.
3

---

Published as a conference paper at ICLR 2025
Dispersion and separation metric of P (D). To measure the dispersion and separation of Pel (D) for each layer I, we design a novel metric called Cosine similarity Variation towards Basis Transition (CVBT). Specifically, given (02s (2=))==1 we firstly pick the activation gradient go (z.-) corresponding to the largest angle, i.e, n = arg max;e (a::). In other words, g (x is the farthest activation gradient vector from the original basis vector g()) (zo). Then, by setting g( (Ix== as a new basis vector, we calculate { (a:)) using Eq. (2) Based on {0%
(1))) (z+) and {0 (æ,))==> formulate the CVBT metric of Pad (D) as follows:
Pa (D) =
cos (0000(00))) cos(( 2n
(z:)))²)
E [0.
(3)
where cos(() returns the cosine value of an angle Note that pro (D) is positively proportional to dispersion and separation, i.e., larger Pao (D) indicates larger dispersion and larger separation of p!! (D). More in-depth analysis will be presented later.
Sample-level closeness metric based on P!d (D)
k= we design novel metric to measure the closeness of each sample æ; to the reference sample Tp, as follows:
Given (x:)) and (0 (a;)
cos(O (xx))
(xx)
E 1).
(4)
200
(1 - cos( (() (z:)) + (1 cos(
@ (x:))
Note that larger S (a:) indicates greater closeness æ; to Zp. For example, (ax)) has
(1) the same direction with g(()(())))) while the opposite direction with g((()))))))))))))))))) then sap (mx) = O, implying the farthest from Zg. In contrast, if (() (ær:) has the opposite direction with (()) (xx,-) while
the same direction with g(t) (az)). then (2) (a:)) = 1, implying the closest to To.
Remark. Note that the single basis vector g (D) (xx, in above two metrics could be extended to be set of basis vectors, i.e., Im (1 ) |==1....mm by picking the activation gradients of top-m largest angles among (a:)) }=1...... Correspondingly, above two metrics are adjusted by replacing each basis vector to the original metrics, then calculating the average. This extension's 's effect will be analyzed later evaluations about adaptive attacks.
4
ACTIVATION GRADIENT BASED POISONED DETECTION METHOD
4.1 PROBLEM SETTING
Threat model. We consider the threat model of data poisoning based backdoor attack. The adversary generates a poisoned dataset Did. containing a clean subset D. {(x,, y))" and a poisoned subset D,- = {(,,,, )- x, ã E X' denotes the clean and poisoned sample with trigger, respectively. y. t )' indicates the ground-truth and target label, respectively. We denote a. t., as the poisoning ratio. Note that there could be multiple triggers (i.e., multi-trigger) and multiple target labels (i.e,, multi-target) in the poisoned subset.
Bp
Defender's goal. The defender aims to identify poisoned samples from the untrustworthy dataset Drd- We assume that the defender has access to Dad, and small set of additional clean samples Dac- which contains at least one clean sample for each class, as suggested in previous works (Ma et al.
2023b Tang et al.) | al. 2019). Besides, the defender has the capability to train DNN model
fura X y based on Dnd.
4.2 POISONED SAMPLE DETECTION METHIOD
Inspired by the two observations demonstrated in Sec. and Fig. we develop an innovative poisoned sample detection method by utilizing GCD and the corresponding metries (see Sec. B). called Activation Gradient based Poisoned Detection (AGPD). As illustrated in Fig. 2 AGPD consists of three stages, as detailed below.
4

---

Published as a conference paper at ICLR 2025
circular distribution (GCD) and metrics
@GD
CVBT
Gradients
Inference
Staget: Calculating activation gradient distribution
Stage2: Identifying target classles)
Stage3: Filtering poisoned samples
eachward
obi
Polsomac
GCDe of ciassee
Traning defumet Datt
Clean dirtanet
classges)
Sample
Figure 2: Illustrations of gradient circular distribution (GCD) and two metrics, and the pipeline of the proposed APGD method which consists of three stages: 1) caleulating activation gradient distribution, 2) identifying target class(es), and 3) filtering out poisoned samples within the identified target class(es).
Stage 1: Calculating activation gradient distribution. We denote the samples of class i in Dra as D = {a,, k)) ) Given the model swau trained on Daa (the training details will be provided in Appendix B.2 ) and picking one clean sample pair (ar). k) E DDcc as the reference, we can calculate the GCD of Did according to Eqs. (1 and (2. Consequently, we obtain (P ( L.K Note
that as defined in Eq. ( the superscript (1) indicates that we adopt the activation gradients of the 1-th layer in Fubs to caleulate GCD. For simplicity, hereafter denote p(u (Dha) P(!.
Stage 2: Identifying ying target class(es). According to the aforemeentioned first observation, the GCD of the target class is likely be more dispersed than that of the clean class. Thus, firstly calculate the dispersion value Pag (Dfa)) (for simplicity, we denote it as Pk of each p!!) according to the
CVBT metric (see Eq. (3)). As shown in Fig. 2 since e of target class(es) is likely be larger, , while those of clean classes are likely to be small, we can adopt the anomaly detection technique to identify target class(es), such the absolute robust Z-score (Iglewicz & Hoaglin 1993). Specifically,
we calculate Z-score of P as follows:
(1) 500
2k
X MAD(1P) (1) (())
(5)
(K=1)
where MAD((P = median ((lpk - ()) indicates the
(MAD), ) is a statistical constant valued at 1.4826. Larger Fk indicates larger likelihood of anomaly. We firstly choose the layer with the largest Z-score, i.e., (1') 2 T2
and T( denotes the median value of {pk Jk=1
1* = arg max maxx 2 (!))) Then, if 2/ exceeds threshold (specified later), i.e., 2k
is identified as a target class, otherwise clean class.
Stage 3: Filtering out poisoned samples within the identified target class(es). Inspired by the second observation mentioned in Sec. Sec.[Ith that the poisoned sample likely to be far from the clean sample in GCD, here develop novel algorithm which gradually filters out poisoned samples with the identified target class(es). Specifically, as illustrated in Fig. 2 when obtained the identified target class b., we firstly pick one clean sample pair of class ke from Dac the reference (æo, k" then we conduct the following three steps iteratively, until a stopping criteria satisfied:
1. For the set D we calculate its GCD according to Definition 2 i.e., P((")
2. We calculate sample-level closeness value Sup (a;) for each æ; E DD
3. If the closeness value of one sample is lower than threshold тa (specified in experiments), i.e., Sun (x:) < T., then this sample is identified as poisoned, as it is far from the clean reference æo- Then, D is updated by removing these identified poisoned samples.
In terms of the stopping criteria, we propose to firstly conduct the above iterations until D becomes a null set. At each iteration, caleulate the distribution of { (æx) eD and the Jensen Jensen-Shannon Shannon
5

---

Published as a conference paper at ICLR 2025
Table 1: The detection performance of AGPD and compared detectors on CIFAR-10 10 and Tiny ImageNet, respectively, with the model Preact-ResNett 18.
Arauk
No dolinie
ACCIASE
SPR,
ACC
F11
TFR:
Buaats
011
SCAre
FFR.
111
TTR:
Spoctal
711
[7R]
STRIP
FFR.
197
ABL
711
17R1
1FR.
CD
F11
1711
ASSET
[77))
731
[FR)
AACES (Deno)
1PRL
Babher
0.30 8.00
07:31
87.08
16.76
1:30
36.00
t0 16
29.74
15.10
13.87
31.24
3.16
67.66
$1.66
Blerded
0.30
0.00
E031
47:00
5.07
22.31
26.34
0.35
11:31
65.07
12.34
12.14
49.02
26.00
3.70
13.66
19.8
042
9938
1F
0.30
9.00
000
010
10:22
3:00
15.55
0101
33.31
000
3.15
3/6
66.90
00.09
02.59
15.45
6.66
45.30
58.04
24.39
43:42
5.00
10.26
19.30
007
59.00
S38A
72.1597300
0.00
8.0
100
10.26
8.54
1077
13.54
1001
23.00
27.64
0.33
42.34
17.42
11:71
54:15
57.35
362
07.30
91.30
3.76
81.12
3.56
46.36
135
19/2
0094
59663
300
0.00
8.0
1000
010
0.80
2.0
15.25
8.00
99.34
000
1.35
210
15.44
5/6
51.88
00.90
3.55
00.37
03.08
21:25
43:31
044
50.75
0.1
1069
005
$5.66
CTRL
05 15.53488
0.00
6.00
000
010
6.36
3.0
0.30
1/0
0100
0-0
15.77
0:30
09.30
547
53.57
04:22
517
66.11
16-44
0.55
07:31
17.8
51/6
11:17
16.2%
6.11
15.76
WaNet
0:00
2.71
000
0.1
9.3
3.4
11.35
6.0
32.65
050
299
135
1:23
9.13
1.28
18.32
9.88
18.31
06.28
74:7
14:30
0.55
1:23
1.0
V1.00
0.31
91.40
N1 13306011
0.00
3.31
000
0:41
1340
3:18
14.15
0.47
67.66
1:5
3:90
3:4
0.1
6.0
0.85
0.11
11:00
0.17
60.25
18:44
06:54
1:57
00.15
000
25:25
15:01
21.200000
01:0
8.0
(00)
75.34
13.88
42.0
1880
9339
20.6
0,32
45.5
67.00
1.59
15.20
7.35
55.90
60.52
53.90
24.15
1:24
3000
106.0
55.8
Riwd
12 17
8.0
000
4.2
8466
2431
0:0
37.85
11/66
11.02
13:27
0.0
0.00
16:52
$2.59
BadNiro A2A
21. 03744-000000
58:78
4.51
33.05
40.38
9.25
010
1.5
010
1.0
12:08
2.48
07.20
45.33
53.23
3.52
V1.32
19.57
33.11
510
15.01
01:43
23.82
19.36
19:20
77.54
11:35
0.45
10/0
55.57
5/22
53.08
52.43
02.22
6:00
14.28
0.00
99.56
SSRAA
0.00
3.8
[031
8:45
8:70
3:5
34.30
13.97
0.39
36.15
99.66
11.33
0.33
55.34
31.94
46:78
51.36
(5:09
0.04
19:75
Walks
030
22.00
75.34
1138
12722
37.50
11:37
0.45
1945
0.35
11.31
24.42
1.27
21.11
16.73
23.35
4230
(U8) 15.12
19:77
0.12
19:30
The
0.30
068
000
61-9
10:10
1913
15.05
04.02
12:07
0.74
17.00
11
13.77
151
2013
16.41
1.07
19.09
001
59.73
16C7
54 2153M9223
0.00
1/0
000
43.31
12:13
56.43
188.
8:00
1910 15:45
0.83
26.23
60.0
16.37 45.50
52.35
7.39
52.35
19.38
99.45
18.19
15.17
36.31
12:3
(502)
0.45
$9.11
Adg Rird
54 35555633
0.00
588
000
9:36
0.39
9.55
<1.24
53.55
15.04
0.55
26118
22.22
15.55
46.32
857
10.11
8.59
13.09
39.65
23.79
71.45
00:00
25
9909
0.17
59.34
Avg.
3,90
291
3:90
27.08
21.39
501
$5.53
15.21
0.37
22.54
6530
12334
45.55
72336
334
71.39
64.21
34.03
43.76
45.30
41354
15.53
9602
0114
$9.34
(IS) divergence between the current and its previous distribution. Then, we adopt a trace-back strategy by checking the JS divergence value of all iterations, and the iteration that its JS divergence locates at the stable and low region could be set as the stopping iteration. Due to the space limit, more details of the whole algorithm, as well as the stopping criteria, will be presented in Appendix D.I
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Attack settings. To evaluate the performance of our detection method, we conduct 10 state-of-the art (SOTA) backdoor attacks that cover categories: 1) non-clean label with sample-agnostic trigger, such BadNets (Gu et al.] 2019), Blended (Chen al., 2017), LF (Zeng et al.,) (02T]] 2) clean-label with sample-agnostic trigger. like SIG (Barni et al. 2019): y: 3) clean-label with sample-specific trigger, such as CTRL (Li et al. 2023). an attack based on self self-supervised learning; and 4) non-clean label
with cymple-specifer trigger, including SSBA (Li et al. 2021b). WaNet (Nguyen & Tran 2021
Input- (Nguyen & Tran 2020), TaCT (Tang et al. 2021). and Adap- BleapBBlend (Qi et al. 2023a).
These attack settings follow BackdoorBench (Wu et al.) 2022) for fair comparison. The poisoning ratio our main evaluation 10% for non-clean label attacks and 5% for clean label attacks. The
target label l. is set to 0 for all-to-one backdoor attack, while target labels are set to ( (y + 1) mod K for all-to-all backdoor attack. The detailed experimental setting are provided in Appendix B.3
Detection settings. We compare AGPD with eight detection methods, categorized into three groups:
1) activation based, including AC (Ma et al. 2023a), Beatrix (Ma et al., (203b)), SCAn (Tang ct al. 2021), and Spectral (Tran et 2018); 2) input-based, such as STRIP (Gao et al. 2019) and
CD (Huang et al., )023); 3) loss-based, represented by ABL (Li al. 2021a) and ASSET (Pan et al. 2023). For a fair comparison, we maintain that the number of clean samples per class is 10, extracted
from the test dataset. The threshold used in AGPD Tg and 72 are e² and 0.05, respectively.
Datasets and models. We use CIFAR-10 (Krizhevsky al. 2009) and Tiny ImageNet (Le & Yang, 2015) as primary datasets to evaluate the detection performance. Additionally, we expand our evaluation the datasets that are closer to real-world scenarios, such as ImageNet (Deng et al., 2009) subset (200 classes), DTD (Cimpoi et al.) 2014). and GTSRB (Houben et al. 2013). of which results are provided in Appendix CC.3 Our study employs two model architectures: Preact-ResNet13 (He ct al. 2016a) and VGG19-BN (Simonyan & Zisserman 2014). The results of VGG19-BN provided in Appendix C.4.1
Evaluation metrics. In this work, the metrics evaluating the performance of backdoor attacks are Accuracy (ACC) and Attack Success Rate (ASR). The metrics used by the defender are True Positive Rate (TPR), False Positive Rate (FPR), and Fl score. In the tables presenting our results, the top performer is highlighted in bold, and the runner-up is marked with underline.
6

---

Published as a conference paper at ICLR 2025
5.2 DETECTION EFFECTIVENESS EVALUATION
All-to-one & all-to-all attacks. Tab. I showcases the detection performance of AGPD with eight compared methods against 12 backdoor attacks on Preact-ResNet18. For all-to-one attacks and all-to-all attacks, AGPD can achieve averaged TPR of 96.66% on the CIFAR-10 and 99.92% on the Tiny ImageNet, exceeding the runner-up by 18.23% and 11.8% respectively. The averaged FPR of AGPD not only ranks within the top-2 lowest among all detection methods but also approaches near 0% level. Meanwhile, its average FI score is 12.71 % higher than that of the second-best method.
For the activation-based methods, like Beatrix, SCAn and Spectral, we find that they effectively identifies the majority of poisoned samples in attacks where poisoned and clean samples are separated in the activation space. However, its performance deteriorates when this separation is not present, such as CTRL (see t-SNE results in Appendix H.. The failure of AC could be caused by the high poisoning ratio. For input-based method like STRIP, they exhibit low TPRs in attacks such as WaNet and Input-Aware. This underperformance is likely because the perturbed inputs generated by a poisoned sample also display high entropy in their predictions, similar to those of clean sample, thereby complicating the distinction between them. CD shows relatively good detection effectiveness across most attacks with an average TPR of 76. 76.76%. although also suffers from higher FPRs. The reason could be that the masks derived from cognitive distillation for poisoned and clean samples are too similar under L2 norm, leading to misclassification of some clean samples as poisoned. For loss-based method like ABL,, they perform well in attacks with attacks such as BadNets, Blended, SIG, and CTRL. However, their effectiveness decreases when facing attacks with dynamic triggers, such as WaNet, Input-Aware, and Adap- -lendd. These attacks require more training epochs for models to learn the connection between trigger and target label, which means that the loss of poisoned samples does not significantly decrease in the early epochs (Wu et al., (202)). Regarding the ASSET method, we observed potential impacts on detection performance due to differences in the model used compared to the original work. Thus, we provide the results of ASSET The evaluation of the model trained on the dataset filtered by AGPD, as well as the detection results under different poisoning ratios on PreActResNet-18 are respectively provided in Appendix C.I and Appendix C.2.
ResNet18 in Appendix E
Table 2: The detection performance of AGPD and the compared methods against multi- multi-target attacks on CIFAR-10 10. The model structure is Neetct. S-T means single trigger, and M-T means multi-trigger.
Adack
ADCVASR
AC
211
17A1
SCA
211
Specral
119L
13R1
STA17
100,
1191
1TRL
ABL
211
19LL
F21
ADSIT
211
TPA1
ACED
F11
Sudiico
17.40
0.30
53.54
19.35
30.30
200
337
18.90
4.44
2.5
1330
2.5
1.2
9196
-22
47
22
14.14
98.28
56,9
6.7
Hirsted
MAT2
0.64
0.25
18.71
20.41
11.
22.0
13.35
2.01
2.00
330
330
51
1499
1291
13
200
11:2
785
3.43
6m
022
0195
9191
186
K010
59.50
41
2111
22
024
0.45
1923
15.30
98.49
91.00
21.3
50044
99.44
0.04
18.02
4.00
009
3.01
8.89
310
5336
1387
2530
9.3
1566
11
994
11.9
72.36
29.1
1392
930
14.56
19.00
41.0542.10
10 50/0
3.88
13.59
200
462
3:00
3:00
15.55
11.11
12.73
13:35
18
5:51
10.00
19:00
18:3
54.0
11:53
90.00
$5.06
17:4
Multi-target attacks. Tab. summarizes the performance of AGPD and the compared methods in multi-target attack scenario. In our experiment setting, 6,,6,7, 6, 7, 8, 9} are chosen as the source class and the target labels are set to t (y + C)) mod K, where C equals 5. Single-trigger attack and multi-trigger attack are two categories of multi-target attack. In single-trigger attacks, the same trigger injected into samples from different source classes classified into their corresponding target classes. In the multi-trigger attack, we use five triggers from different backdoor attacks (i.e., BadNets, Blended, LF, SSBA, and SIG), and each trigger added to the samples in the corresponding class will be classified into its designated target class. We observed that for activation- the multi-trigger attack poses a greater challenge than single-trigger attacks, whereas bass- methods seem more robust against multi-trigger attacks. Additionally, the failure of SCAn might be caused by their anomaly detection for target class(es) is not effective in the multi-target attacks. However, compared with baseline methods, AGPD achieves good performance in both trigger and mulli-trigge attacks, with averages of TPR, FPR, and Fl score at 97.42%, 0.06%, and 98, 4..11 %, respectively.
methods,
5.3 ANALYSIS
Analysis of CVBT. To substantiate the capability of the CVBT metric (i.e., Pwo (D) in capturing the characteristics of the circular distribution (i.e.. (D)). here we simulate different circular distributions with varying degrees of dispersion and separation. (1) As shown in the left four sub-plots
7

---

Published as a conference paper at ICLR 2025
in Fig. 3. while keeping similar low separation (i.e., one single cluster), the dispersion increases from left to right, i.e,, the distribution range increases. Correspondingly, the CVBT score increases. (2) As shown in the right four sub sub-plots in Fig. 3 while keeping similar dispersion (i.e., similar the distribution range), the separation increases from left to right, as two clusters get more distant gradually. Correspondingly, the CVBT score increases. Thus, the claim that the CVBT score (i.e. Pae (D)) is positively proportional to the dispersion and separation of Pas (D) (see Sec. 3.2) is
verified. A comparison of the capabilities of CVBT and variance measuring the characteristics of the circular distribution is also provided in Appendix D.2
COIT
CVBT
cam 0.55
cvan
Evtn
cVIT: 0.53
Increasing dispersion with similar separation
Increasing separation with similar dispersion
Figure 3: CVBT scores of different GCDs with varying dispersion and separation.
Statistic of metries. In Fig. 4 we present the statistical results of CVBT metric (p) and its Z-score (=) for both all-to-one and all-to-all attacks. In the left of Fig. + of the target class are significantly higher than those for clean classes for both attacks. Since clean classes are absent all-to-all attacks, using 2 to detect the outliers for identifying the target class could be ineffective. If z doesn't 't pinpoint the target class, we use 0.3 threshold as boundary identify it, shown as a dashed line Fig.
4 aand validated by the left two images in Fig. 4 where p values for target classes surpass this limit all-to-all attacks. The mid-right image Fig- displays the distribution of for the target and clean classes in the all-to-one attacks, clearly separated by the dashed line, which represents the threshold when identifying the target class. Moreover, the right image of Fig. fillussrates the mean and standard deviation curves of 2 of target classes across different convolutional layers of the model. We observe that 2 tend to be higher in the later intermediate convolutional layers, indicating a stronger separation between poisoned and clean samples at these layers.
All-o-ane
Al-tn-al
Allto-one
Seget cles
Toret clet
Coas
Cles Clas
Clea
Zaccres
Figure 4: Statistical analysis of and = across classes and convolutional layers using the CIFAR-10 and Preact-ResNet18. Left: p values for all classes in both all-to-one and all-to-all attacks. Mid-right: for all-to-one attacks. Right: Mean and standard deviation of the maximum across all layers in multiple backdoored models.
Accuracy of target class identification. We compare the accuracy of target class identification of AGPD with other three detection methods which are Beatrix (Ma et al. 2023b), SCAn (Tang et al. 202T), and NC (Wang et al.). To evaluate their performance, we trained 120 backdoor models on CIFAR-10. The attack methods contain 8 non-clean label backdoor attacks, where the poisoning ratio ranges from 1% to 10% and the target label is from 0 to 4. The results of detection accuracy are shown in Fig- 5a Note that the accuracy of target class identification of AGPD is higher than the compared method under different poisoning ratios.
Analysis of activation gradient To illustrate the advantages of activation gradient in sample detection, we also analyze the discriminative characteristics of the activation gradient for the poisoned sample and clean sample Appendix D.4]
5.4 SENSITIVITY TEST
Influence of the number of clean samples. In this part, we explore the influence of the size of the additional clean dataset on the detection performance of AGPD. We also consider the scenario that the additional dataset collected by the defender is out of distribution (OOD). We collect the OOD dataset CIFAR-10 10 from the same 10 classes of CIFAR-5m (Nakkiran et al. 2020), and we extract 10 samples from each class. The additional clean dataset which is in distribution (ID) is collected
8

---

Published as a conference paper at ICLR 2025
from the test dataset. Fig. 5b shows the results of our method with different sizes of the additional clean dataset. We found that a large number of clean samples can help AGPD decrease FPR close to zero. However, AGPD can still achieve high TPR even in extreme cases, such one sample per class or even OOD samples. When the number of clean samples in each class is one, the TPR values of AGPI) on many attacks are above 90%. In summary, our method necessitates a smaller additional clean dataset.
100
Accurscy (80
2 PPR (PP/0M)
TTR (N1
40
02
Puisariin Ralio
Inreshold
Figure 5: Left:: Accuracy of AGPD and three compared methods on identifying target class(es). Middle: Detection performance of AGPD with varying numbers of clean samples. Right: Means and standard deviations of TPR and FPR at different threshold Ts.
Influence of threshold T₁. In the poisoned sample filtering stage, we aim to eliminate samples scoring below т3 at each iteration until no samples remain in the target class. To better understand the impact of Ts, we design experiment with varying T. from 0.01 to 0.1. According to Fig. [5c the TPR of AGPD is relatively low with significant variability at smaller T3 values, yet stabilizes at 100% with increasing Tu while the FPR remains consistently low throughout the variation of T.. Moreover, can ensure stable detection performance of AGPD across a broad range of values.
5.5 DETECTION EFFECTIVENESS AGAINST ADAPTIVE ATTACKS
Setup of adaptive attacks. Here we evaluate AGPD's effectiveness against adaptive attacks, i.e., when the adversary knows its detection strategy. Specifically, the core point in AGPD is the observed characteristic of the gradient circular distribution Pæ., (D), i.e., dispersion and separation of the target GCD (see Sec. [I). Thus, the adaptive adversary aims to break this characteristic. To that end, we design two adaptive attacks. (1) Adaptive attack 1: Weak clcan-label attack for reducing dispersion and separation. The poisoned samples are constructed based on blending trigger image with target clean images, such that poisoned images are closer to clean images, leading to closer in GCD. This adaptive attack is denoted as Blended;, being a being the alpha blending coefficient of the trigger image. (2) Adaptive attack 2: Attacking with inserting noisy samples into the target class for disturbing the target GCD. We insert some noisy samples into the target class, i.e. randomly picking some clean samples from other classes and changing their labels to target label. As both noisy and poisoned samples are significantly different with target clean samples, the target GCD may vary due to noisy samples. All evaluations are conducted on CIFAR-10 with Preact- Net1
sinced
Class O (angg, class)
Class 2
Polsomee
Clsss
Class 3
Nolty sample
Figure 6: Gradient circular distributions of the target class under Blended and the adaptive Blended, attack.
Results & Analysis of adaptive attack We firstly present the GCDs and attack performance (without de- fense) of Blended; a= Blended Blended 0003 respectively, in Fig, 6 It shows that the attack perfor- mance is positively proportional to the dispersion and separation of GCD. The detection results of are shown in Tab. 3 When m = 1 (i.e., using one single basis vector in G.,), the Z-score is too small to identify the
Table 3: AGPD detection results against Blended,, (adaptive attack 1), with differ- ent numbers of basis vectors in gm. i.e. m = 1/m- = 200.
Attock
ASRS
Zocorre
Blerdoo
96.75
21 54112.89 89
96-66999 95
Mendee
13.08
(LA92.01
0.000
0
Bloaded
51.16
2.657706
0.09.52
0,0F18997
Bloaded.
35.35
3.438.00
0.099U6

---

Published as a conference paper at ICLR 2025
target class, leading low TPR and low FPR. However, as demonstrated in Sec. 3.2 the basis vector g (Zx,) ) could be extended to a basis set Gm = (g (ing)))......mm When m = 200, the Z-scores are much larger. Consequently, even when the attack is weak (i.e, ASR 51% of Blended" AGPD still shows high TPR and low FPR. It demonstrates that increasing the number of basis vectors in gm could enhance AGPD's s robustness to adaptive weak backdoor attacks.
Results & Analysis of adaptive attack 2. As shown in Tab. 4 AGPD still performs very well against the Blended attack with varying noisy samples, and has very high Z-scores. The GCDs of the corresponding poisoned datasets are shown in Fig. Left. shows that noisy samples may have large angles in GCD, thus the
Table 4: AGPD detection results against Blended attack with noisy samples (i.e., adaptive attack 2).
Nolsy samples
ASR(5)
Z-score
TPR
FPR
0
99.75
21.34
99. 99.98
0.02
10
99.8
48.73
99.9
0.04
dispersion and separation are still large, leading to high
100
99.71
23.31
99.8
Z-scores. However, when 1, 000 noisy samples exist, the separation degrades, which may affect the sample
1000
99.68
35.75
100
1.8
filtering of Stage 3 in AGPD (see Sec. 4.2). Thus, we analyze the trend of the identified far-ending sample Tx. in all iterations. As shown in Fig. Right, we record the proportions of noisy and poisoned samples in all accumulated I. s. When there are many noisy samples (e.g., 100 or 1,000 noisy samples), noisy samples are identified as I,,- in early iterations, while poisoned samples are gradually identified Hn- in later iterations. Consequently, both noisy and poisoned samples could be identified as the far-ending basis vector, leading to filtering out of both noisy and poisoned samples. This explains the good performance of AGPD against the adaptive attack with noisy samples.
In summary, AGPD shows good performance against above two adaptive attacks, Le., weak clean- label attack, and attack with noisy samples.
many cumpeec theedeed samples
Blended Iwith 50 mana samplee) Blended owth 100 ratly somotess Blended Iwith 1000 notay samppies
ride
Class (target class)
Closs
Poisoned
Class
Class 3
Nally sample
Figure 7: Left: The gradient circular distribution of the poisoned dataset with noisy samples of the Blended attack. Right: The proportions of noisy or poisoned samples in the accumulated set of far-ending samples x,, along with the sample filtering iterations.
CONCLUSION
In this paper, we introduce a novel perspective of gradient circular distribution (GCD). Based on GCD, we observe that the dispersion of GCD of target class is larger, and poisoned samples are separated from clean ones. Inspired by the observation, we propose two practical metrics and design novel detection method, AGPD, Our experiments demonstrate that this method successfully identifies target class(es) under various backdoor attack scenarios, including all-to-one, all-to-all, and multi-target attacks. Extensive experimental results show that our method achieves good performance on the task of poisoned sample detection. Finally, we believe that the novel perspective of GCD deserves more future explorations, such as its usage in other tasks (e.g., the training-based backdoor defense) and other characteristics.
Ethics & Reproducibility statements. This work reveals common observation of existing back- door attacks, and provides an advanced backdoor defense method. It will not bring in negative impact to the community. All evaluations are conducted on widely used datasets for image classification, no involvement of ethic issues. Besides, we have provided all important implementation details in Ap- pendix to ensure the reproducibility of all reported results. Our code is based on Backdoorbench Wu (et al. (022). and we provide a demo AGPD in the supplementary materials, along with the method of operation, and we promise to release all codes once acceptance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Guangdong Basic and Applied Basic Research Foundation (No. 1R2AB11100200 Shenzhen Science and Technology Program (No. D5X2021060910005707 and JCYJ2024081 1111360001 1), Sub-topic of Key R&D Projects of the Ministry of Science and Technology (No. 2023YFC3304804) Longgang District Key Laboratory of Intelligent Digital Economy Security, Guangdong Provincial Special Support Plan Guangdong Provincial Science and Technology Innovation Young Talents Program (No. 2023T007A352), and the National Natural Science Foundation of China No. 62471420.
REFERENCES
Hasan Abed Al Kader Hammoud, Adel Bibi, Philip HS Torr, and Bernard Ghanem. Don't freak out: A srqqueeccy-isppiree approach to detecting backdoor poisoned samples dnns. CVPR, pp. 2337-2344, 2023.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in enns by training set corruption without label poisoning. In ICIP, 2019.
Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. In NeurlPS, 2022.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurlPS, volume 35, 2022.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. ACSAC, pp. 113-125, 2019.
Yinghua Gao, Dongxian Wu, Jingfeng Zhang, Guanhao Gan, Shu-Tao Xia, Gang Niu, and Masashi Sugiyama. On the effectiveness of adversarial training against backdoor attacks. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Tianyu Gu, Kang Liu, Brendan attacks on deep neural networks. IEEE Access, 7:47230-47244 2019.
and Siddharth Garg. Badnets: Evaluating backdooring
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, pp. 630-645. Springer, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016b.
Linshan Hou, Ruili Feng, Zhongyun Hua, Wei Luo, Leo Yu Zhang, and Yiming Li. lbd-psc: Input-level backdoor detection via parieteer-oriented scaling consistency. arXiv preprint arXiv:2405.09788 2024.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN, pp- 1-8, 2013.
Hanxun Huang, Xingjun Ma. Sarah Erfani, and James Bailey. Distilling cognitive backdoor patterns within an image. arXiv preprint 2tt::2301.10008 2023.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Boris Iglewicz and David C Hoaglin. Volume 16: how to detect and handle outliers. Quality Press, 1993.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Changjiang Li, Ren Pang, Zhaohan Xi, Tianyu Du, Shouling Ji, Yuan Yao, and Ting Wang. An embarrassingly simple backdoor attack on self-supervised learning. In ICCV, pp. 4367-4378, 2023.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models poisoned data. In NeurIPS, volume 34, 2021a.
Yuezum Li. Yiming Li, Baoyuan Wu, Longkang Li. Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, pp. 16463-16472, 2021b.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In RAID, pp. 273-294. Springer, 2018.
Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. Detecting backdoor attacks on deep neural networks by activation clustering. In NDSS Symposium, 2023a.
Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The "beatrix" resurrections: Robust backdoor detection via gram matrices. In NDSS Symposium, 2023b.
Bingxu Mu, Zhenxing Niu, Le Wang. Xue Wang, Qiguang Miao, Rong Jin, and Gang Hua. Progres- sive backdoor erasing via connecting backdoor and adversarial attacks. In CVPR, pp. 20495-20503, 2023.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. arXiv preprint arXiv:2010.08/27, 2020.
Anh Nguyen and Anh Tran. iaper-impe trrping-based backdoor attack. arXiv preprint arXiv:2102. 103699 2021.
Tuan Anh Nguyen and Anh Tran, Input-aware dynamic backdoor attack. In NeurlPS, volume 33, pp. 3454-3464, 2020.
Minzhou Pan, Yi Zeng, Lingjuan Lyu, Xue Lin, and Ruoxi Jia. Asset: Robust backdoor data detection across a multiplicity of deep learning paradigms. In USENIX Security, pp. 2725-2742, 2023.
Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assump- tion of latent separability for backdoor defenses. ICLR, 2023a.
Xiangyu Qi, Tinghao Xie, Jiachen Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive [ML} approach for detecting backdoor poison samples. In USENIX Security, pp. 1685-1702, 2023b.
Peter J Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic backdoor attacks against machine learning models. In S&P. pp. 703-718, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
Di Tang, XiaoFeng Wang. Haixu Tang. and Kehuan Zhang. Demon in the variant: Statistical analysis of {DNNs} for robust backdoor contamination detection. In USENIX Security, pp. 1541-1558, 2021.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurIPS, volume 31, 2018.
12

---

Published as a conference paper at ICLR 2025
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P, pp. 707-723.
Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. In NeurIPS, 2023.
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: comprehensive benchmark of backdoor learning. NeurlPS Datasets and Benchmarks Truck, volume 35, pp. 10546-10559, 2022.
Baoyuan Wu, Li Liu, Zihao Zhu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Adversarial machine learning: A systematic survey of backdoor attack, weight attack and adversarial example. arXiv preprint 2rXx:22302.09457, 2023.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS, pp. 16913-16925, 2021.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, pp. 16473-16481, 2021.
Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR, 2022.
Runkai Zheng, Rongjun Tang, Jianze Li, and Liu. Data-free backdoor removal based on channel lipschitzness. In ECCV, pp. 175-191. Springer, 2022a.
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurlPS, 2022b.
Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In ICCV, 2023a.
Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. In NeurlPS, 2023b.
13

---

Published as a conference paper at ICLR 2025
A
OVERVIEW OF APPENDIX
There are additional materials presented in the Appendix
Appendix
B
Experiment setting details.
Appendix
B.1
Details of datasets.
Appendix
B
Hyperparameter settings of model training.
Appendix
B.3
Hyperparameter settings of Backdoor attacks.
Appendix
C
Additional experimental results
Appendix
C.
Performance on the model trained by filtered data.
Appendix
Different poisoning ratios under PreactResNet-18 on CIFAR-10.
Appendix
Evaluations on more datasets.
Appendix
Evaluations on more models.
Appendix
Comparison to other detection methods.
Appendix D
Additional analysis of AGPD.
Appendix D.
Details of algorithm and stopping criteria.
Appendix D.2
Comparison between CVBT and variance.
Appendix
D.3
Comparison between cosine distance and radian.
Appendix
D.4
Analysis of the discriminative degree of activation gradient.
Appendix
D.5
Computation overhead.
Appendix D.6
Stability of AGPD.
Appendix
The results of the compared ASSET on ResNet18.
Appendix
t-SNE results
Appendix
Results for adaptive attacks
Appendix
Results for clean-label attacks
Appendix Results for noisy and poisoned samples
B EXPERIMENT SETTING DETAILS
B.1 DATASETS
We evaluate the performance AGPD on five popular datasets and two model structures. main paper, have provide the results of two datasets, including CIFAR-10 (Krizhevsky et al., 2009) and Tiny ImageNet (Le & Yang. 2013). Besides, we extend our evaluations to the datasets which are closer to real-world scenarios, such as ImageNet(suubset--e)
(Deng et al. 2009). the Textures dataset DTD (Cimpoi et al.) 2014), and the traffic signs dataset GTSRB (Houben et al. 2013). The results of these datasets are provided in Appendix
The details of all datasets illustrated in Tab.
Table 5: The information about five datasets.
Dataset
Categories
Image size
Training samples Testing samples
CIFAR-10
10
32 32
50,000
10,000
Tiny ImageNet
200
64 64
90,000
10,000
ImageNee(subsee2000
200
224 224
90,000
10,000
GTSRB
32
39,209
12,630
DTD
47
224 224
3,760
1,880
B.2 HYPERPARAMETER SETTINGS OF MODEL TRAINING
There common training hyperparameters across these attack methods, such training epoch, learning rate, and optimizer. We display the setting of these common hyperparameters for each datasets in Tab. 6
14

---

Published as a conference paper at ICLR 2025
Table 6: The common hyperparameters for training across five datasets.
Dataset
Epoch
Learning rate
Batch size
Optimizer
CIFAR-10
100
0.01
128
SGD
Tiny ImageNet
200
0.01
128
SGD
ImageNet(subset200)
200
64
Adam
GTSRB
50
0.01
128
SGD
DTD
100
0.01
64
SGD
B.3 HYPERPARAMETER SETTINGS OF BACKDOOR ATTACKS.
The hyperparameters used in various backdoor attacks are listed in Tab. 7 For illustration purposes, we use CIFAR-I0 example. If the attack does not have any specific hyper-parameters, we will denote this with Y/. We show the poisoned samples of various backdoor attacks in Fig.
Table 7: The
of implemented backdoor attacks for CIFAR-10.
Category
Attack
Parameters
Usage
Value
non-clean lahel with sample-agnostic trigger
BadNets
Blended
the transparency of
0.2
the trigger.
LF
fooling rate
0.2
clean label with sample-ignoeric trigger
generale
40
sinusoidal signal.
clean label with sample-spece sppecifii trigger
CTRT.
trigger channel
[2.1]
trigger location
(12,27)
SSBA
class
the trigger will be
list
sample-spociñi trigger
T&CT
added samples
s class and change
labels the
get label.
C- class
samples in class will only added trigger.
a list
control the number of samples class
0.1
Adap Blend
the probability the area being masked.
0.5
warping strength grid scale
non-olean training control
WaNet
Ps
backdoor probubility
=poisoning ratio
Pe
the noise probability
Adi
the diversity enforce-
Inwut-Aware
ment regularisation.
the backdoor proba- bility.
=poisoning ratio
Pc
the cross-trigger prob- ability.
non-clean label with sample-agnosti trigger
BadNets-A2A
compute target
labels.
non-clean label with sample-specifi trigger
SSBA-A2A
non non-clean
BadNets
Single trigger Atlack
sumple-agguee aggossii
Blended
compute Larget labels
trigger
LF
non: clean lahel with sample-specifie trigger
SSBA
non-clean label with BauNNets- I.F+ SIG sample agnostic trigger
the natio of paisonad samples type trigger
0.02
Multi-trigger Attack
15

---

Published as a conference paper at ICLR 2025
Table The detection performance of AGPD based on the model, which is trained on the filtered dataset, on CIFAR-10 with Preact-ResNetlee
dataset
BadNets
Blended
LF
SSBA SIG(5%)
WaNct
Input-Aware
TaCT
Adap-Blend
ACC
91.82
93.69
93.01
92.88
89.68
91.35
93.21
92.87
Poisoned data
ASR
93.79
99.75
99.05
97.06
95.43
96,94
98.17
95.95
66.17
ACC
91.9
91.52
92.02
91.7
91.91
89.98
91.14
90.71
91.25
Filtered data
ASR
1.41
2.11
1.6
0.9
0.01
0.86
9.67
0.55
4.4
BadNets
Blended
LF
SSBA
CTRL
SIG
WaNet
Input-Aware
TaCT
Adap-Blend
Figure 8: Examples of poisoned samples in various backdoor attacks.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I PERFORMANCE ON THE MODEL TRAINED BY FILTERED DATA
We also the ACC/ASR of the model trained on the filtered dataset metric evaluate the effectiveness of the model detection. In Tab. g we present the ACC/ASR results of the poisoned dataset under different attacks after being filtered and trained using the AGPD method CIFAR-10 and PreActResNet-Il 18. Our preliminary evaluations on CIFAR- 10 and PreAct-ResNee 18 demonstrate that training on the dataset filtered by AGPD can achieve high ACC and low ASR. The AGPD method thus ensures the model's 's performance while resisting backdoor attacks.
C.2 DIFFERENT POISONING RATIOS UNDER PREACTRESNET
CIFAR-10
We evaluated the detection effectiveness of AGPD and four other methods, chosen from activation- based, insut- and loss- detection methods, under varying poisoning ratios. There are four poisoning ratios used: (0.5%. 1%, 5%. 10%), ), covering range from low high poisoning ratios. As illustrated Fig. 9 the performance of most detectors is notably influenced by the poisoning ratio. Particularly, a low poisoning ratio (e.g., 0.5% presents substantial challenge for most detectors, with the TPRs of AC and SCAn almost nearing zero. However, our method can achieve good performance in this situation, with TPR around 90% and FPR lower than that of other detectors under most attacks. And with the poisoning ratio increased, the performance of our method is still stable.
C.3 EVALUATIONS ON MORE DATASETS
In this section, we present the results of AGPD and the compared methods these datasets, including DTD Ciippo et al. 2014), GTSRB (Houben et al. 2013),, and ImageNet subset (200 classes) (Deng et al. 2009). The datasets can be categorized into two types: balanced datasets (e.g., DTD and ImageNet subset (200 classes)) and imbalanced datasets (e.g., GTSRE). In balanced datasets, each category has the same number of samples. For example, DTD has 376 samples per class, and ImageNet-200 has 500 samples per class. contrast, the number samples per category in GTSRB varies from 210 to 2,250. Besides, We Preact-ResNet188 (He et al. (016a)) the model
16

---

Published as a conference paper at ICLR 2025
558A
TI955
60
60
Poisoning reto
20-
26-
0.5%
WoNNet
2010
Input-Aware
TDCT
Adop-Blend
100
100
190
100
48-
00.6
30%
4.55%
02%6
17
22%
4.75
6116
TPR
FPR AC
SCAn
STRIP
ABL
AGPD(OUrs)
Figure 9: Detection performance of AGPD and the compared detectors with poisoing ratios ranging from 0.5% to 10%
architecture when training on DTD and GTSRB, and adopt ResNet50 (He et al. 201 for ImageNet subset (200 classes). Specifically, we compare AGPD with four detection methods: activation-basee method (e.g., SCAn (Tang et al., 202T).. input-based method (e.g.. STRIP (Gao et al., (019)). and loss-based methods (e.g., ABL (Li et al. 2021a) and ASSET (Pan al. 2023 on DTD and GTSRB. The results are shown in Tab. Besides, the results of AGPD on ImageNetsubset (200 classes), are displayed in Tab. 10
Table The detection performance of AGPD and compared detectors on DTD and GTSRB. The results are evaluated on Preact-ResNet18.
defense
Dataset
Attack
ACC/ASR
TPR1
FPR.
F11
TPR1
FPR.
FI'
TPR:
FPR]
F11
TPR1
FPR1
F11
TPR1
FPR|
F11
BadNets
51.97/9.322
90.43
0.21
94.05
83.78
12.71
56.20
76.33
2.63
76.43
96.01
8.16
71.15
99.47
1.60
93.03
Blended
51.86/94.0 62
82.18
0.30
88. 76
95.74
13.21
60.74
88.83
1.24
88.95
0,80
3.04
1.25
100.0
1.89
92.27
WaNet
42,71/26.41
86.08
1.06
83.01
77.84
14.03
51.14
0.28
11:00
0.27
1,70
2.33
2.61
100.0
1.88
92.27
45.85/85.54 544
0.00
0.00
0.00
13.07
12.91
11.38
20.45
8.92
20.19
0,00
0.72
0.00
98.58
0.59
96.73
Adap
100.0
Avg.
20.81
40.50
38.50
16.30
99.61
BadNets
96.35/95.02
94.62
4.00
82.06
95.54
15.39
57.20
73.71
2.92
73.71
100.0
47.79
31.74
100.0
98.80
GTSRB
Blended
98.17/100.0
66.42
99.36
97.05/96.55
75.63
Input- Input-Aware
97.91/9.66
52.15
68.54
1.99
10.44
2.03
0.00
11.03
0.00
3.07
59.18
0.96
79.60
Adap-Blead
97.66/80.42
50.74
3.95
$4.48
20.48
12.40
17.63
0.00
11.11
0.00
81.10
22.48
42.30
30
93.44
95.58
Ave.
68.36
3.10
69.43
45.17
12.83
29.84
30.85
7.65
30.85
65.52
47.12
24.38
93.78
0.20
95.74
Table 10: The detection performance of AGPD ImageNet-200.. The results evaluated Preact-ResNer18.
No defense ACC/ASR
AGPD
Dutaset
Attack
TPR
FPR 1
F1
BadNet
78.57/80.03
94.62
0.00
97.24
Blended
79.95/99.93
100.0
0.47
97.93
ImageNet-200
Adup-Blend
72,3/93.17
99.98
0.59
99.19
Avg.
98.20
0.35
98.12
As shown in Tables 9 and 10 AGPD achieves high performance on these datasets, with average TPRs of 99.61% 61,, 93.78%. and 98.20%, respectively. Meanwhile, the average FPRs 1.43%. 0.2%, and 0.35%. The experimental results demonstrate that our method is adaptable not only to balanced and imbalanced datasets but also datasets with various image sizes.
17

---

Published as a conference paper at ICLR 2025
Regarding the compared detection methods, we draw conclusions similar to those reported in our main paper. Specifically, SCAn fails when the distinction between poisoned and clean samples in the activation space is not obvious. STRIP struggles to effectively identify poisoned samples in the training dataset under attacks such as WaNet and Input-Aware. Similarly, ABI. encounters challenges in achieving satisfactory detection performance against complex triggers, including those used in WaNet, Input-Aware, and Adap-Blend attacks.
C.4 EVALUATIONS ON MORE MODELS
C.4.1 EVALUATIONS ON VGGI9-BN
All-to-one & all-to-all attacks. The results of VGG19-BN under all-to-one and all-to-all attacks are shown in Tab. I1 It can be seen that even changing the model architecture, the detection performance of our method is still stable, achieving 96.46% TPR on CIFAR- 10 and 97.75% on Tiny ImageNet, higher than that of the second- best 16.55% and 7.89%, respectively. Besides, the Fl score of AGPD are 90.90% on CIFAR-10 and 98.36% on Tiny ImageNet, exceeding the runner-up by 33.59% and 16. 16.27% respectively. The results indicate the dispersion of the activation gradients of poisoned samples and clean samples could exist across model structures, demonstrating the robust adapt ability of our method.
For the compared methods, we found that the model architecture significantly influences activation- based methods. For instance, AC can identify small portion of poisoned samples in all-to-one attacks with 10% poisoning ratio. Beatrix performs better on VGGG1-BBN with the average TPR 20.78% higher and the Fl score 17.39% greater than on Preact-ResNett88. Conversely, SCAn struggles to detect a large number of poisoned samples in WaNet and Input-Aware attacks on the VGG19-BN.
Table 11: The detection performance of AGPD and compared detectors on CIFAR-10 and Tiny ImageNet. The results are evaluated on VGG19-BN.
Daast
No deliass
ACCMASE
11901
.MT
F1"
119:
Boasis
P11
[191
SCAr
P7N.
P11
1191
Speceil
211
TPNT
STRIP
PINL
F1"
ABI.
211
1191
CD)
P11
2791
ASSPT
[99L
F31
AGPD
F11
H. 21.3305.5
1.30
6.43
$2.30
3:44
57.50
18
5.5
$8.06
0.8
$3.58
11
56.10
N4.58
3.54
26.0
12.56
13:45
08:63
3.33
1.31
16.72
0,90
16.16
11 11899677 75
0:00
14:29
000
99.23
9.57
60.00
4.73
8.00
0.89
07.06
10.33
11
16.95
2.44
76:00
01
18.00
0.30
53.95
0.0
19.06
0.22
$9.00
0.00
5.78
0000
72.45
0.42 77.30
13:2
92.3
26.90
0.88
44.30
19.00
9.26
83.54
41.50
6.55
40.9
0.39
0655
0.23
1.76
3.15
19.50
$9.02
S3BA
22.8997200
4.34
1405
4.10
0:10
6.93
3.17
91.15
2.18
292
3.5
12.38
12.33
51.00
25.56
8.22
24.55
28.0
04.75
63.67
23.20
000)
0.88
16.94
4.31
11
41
13.34
0191
1881
3.56
74.20
1450
0.88
46.5
68.96
6.20
62.47
593
63.33
68.24
38.30
1918
41.36
18:0
99.02
106
22.30
3.55
1:47
Ane.
10.33
17.57
44.54
33.21
16.11
2:3
2413
11.05
36.11
35.49
13.01
35.04
34.00
17.15
13.20
96.46
$6.00
1.10
2004
07.71
(6:30
48.61
[5:57
0.84
72.00
11:34
(5
96
0.37
9.16
(4
09
11:1//077
0.12 69.30
33.399.31
0.00
1,41
1600
02:85
4.43
1900
11:26 8.00
57.30
15.57
0.80
20/0
15.35
15.85
53.00
25.25
0.35
26.25
02.36
54.15
33.00
1:32
4.63
1.54 1909
005
5928
The Ka
0.00
3.74
22.75
2.0
32.30
12:25
8:00
55.54
15.57
0.88
27.00
21.00
11.82
15:55
45.55
5.0
-0.87
16.43
10.55
06.13
23.06
17:00
56.34
1402
0433
19364
S5BA
55 55.57770
0.00
3.20
(03)
42.55
3.90
47.70
1510
8.00
57.67
15.57
0.88
27.00
95.00
11.77
05:33
91.55
1:15 2045
10.11
19:00
51.14
47.06
31.80
0002
5930
56 1390000
5.63
6m
8:0
111 310
14:91
6:0
01.56
(141
0134
04:00
17
11:57
15.5
11:51
034 30.85
16.00
0:30
06
0:11 07.1
0.12 59:10
$7,592335
0:5
1.64
007
0:3
233
232
99.66
15.72
0.8
20.1
15.49
12.35
13.19
54.17
4.11 62.57
15.30
5:30
10:03
22.15
15.79
58.17
62.34
0.90
5063
$45399222
0.00
8.15
000
34,87
282
61.30
4151
8.00
51.55
15:55
0.55
3600
19.01
17.27
14:14
96.75
481
56.35
14:14
25:15
59.15
51.53
31:03
99.99
0.24
98.96
445596.3
25.55
2115
16.39
0/6
1247
3:0
51.13
(4:33
0354
25.51
15.09
43.88
1,00
$1.50
27.87
41.12
2018
57.36
0.16
99.27
300
9:99
33.0
15.44
16:33
13:38
67.38
07:37
$8.36
Performance of AGPD with VGG19-BN under various poisoning ratios. We estimate the detection performance of AGPD against various backdoor attacks with different poisoning ratios and compare our method with four detectors. The results are displayed in Fig. 10 It can be seen that our method achieves higher TPR under most attacks compared to other methods, which also maintain relatively low FPR.
C.5 COMPARISON TO OTHER DETCTION METHODS
In this section, we compare our method alongside CT (Qi al. 2023b) and IBD-PSC (Hou et al., 2024). To ensure fair comparison, we maintained experimental settings of attacks (i.e.., learning rate and training epoch E) consistent with our main experiment, as introduced in See 5.1. The details are shown follows:
Dataset and model architecture: We compare AGPD with CT and IBD-PSC on CIFAR-10 10 dataset and 18
18

---

Published as a conference paper at ICLR 2025
60
60
Polsoning rato
26-
WoiNet
Imput-Aware
T5CT
Adop-Blend
120
100
390
100
30%
25
22%
50
TPR
FPR
AC
SCAn
STRIP
ABL
AGPD(OUrs)
Figure 10: Detection performance of AGPD and the compared detectors with poisoning ratios ranging from 1% to 10%.
Attack method and poisoning ratio: We choose four classical attack methods which are BadNets, Blended, SSBA, and WaNet. The poisoning ratio is 10%.
Hyper-pen apper-parameterss for CT: We set the number of clean samples is 2,000 and confusion fac- tor is 20. Besides, we set confusion iteration = 6, confusion iter m 6000, learning rate of 0.001, and the distillation ratios {r1 1 12 5 ra 25 ra 50 Is 100 }.
Slyper-parameters of IBD-PSC: The number of clean samples is 2,000. The hyper- parameters for error rate, threshold T, and amplifying coefficient for the selected BN layer set to 0.6, 1.5, and 0.9, respectively.
Table 12: The detection performance of AGPD and other detection methods on CIFAR-10. The results are evaluated on Resee..
CT
IBD-PSC
AGPD
Attack
TPR*
FPR
F11
TPR
FPR
F11
TPR
FPR
F11
Badnet 95.94
0.01
97.90
99.82
11.02
64.14
90.06
0.03
94.65
Blended
99.96
1.36
94.21
99.47
11.42
63.17
99.98
0.02
99.88
SSBA
100.00
0.08
99.63
85.64
10.86
60.07
99.62
0.04
99.63
WaNet 91.64
0.17
94.89
96.24
11.41
61.78
97.80
0.31
97.40
The experimental results are shown in Tab. 12 It can be seen that both our method and the compared methods achieve high performance detecting poisoned samples. Notably, our method performs well in minimizing misclassification of clean samples during the detection task, resulting in lower FPR.
D ADDITIONAL ANALYSIS OF AGPD
D.1 DETAILS OF ALGORITHM AND STOPPING CRITERIA
D.1.1 THE DESCRIPTION ALGORITHM
The statement of the algorithm we used in Stage3 is described Algorithm 1
An example of JS divergence across iterations is provided in Fig. [1] Assuming ground truth is known for samples in the target class, we can obtain True Positives (TP) and False Positives (FP) for each iteration. can be observed that optimal iteration exists where JS divergence minimal and stabilizes. The
19

---

Published as a conference paper at ICLR 2025
rationale behind the trends in JS divergence is that in the early stages of filterin,, the far-end basis primarily updated by genuinely poisoned samples, effectively guiding the identification of such samples. As the process progresses into the middle stages, most poisoned samples have been filtered out, and the influence of the far-end basis on the remaining clean samples becomes minimal, resulting in little change in trust scores and small IS divergence. However, as the process extends into later stages, and more clean samples are inevitably filtered, the far-end basis is updated by these clean samples, leading significant changes the distribution of trustworthiness scores and increase in IS divergence.
TP
Number
1000
20
40
60
80
divergence
0.2
0.0
20
40
60
80
Iteration
Figure 11: Trends of TP, FP, and divergence according to the iteration
Algorithm Filtering out poisoned samples within the identified target class(es).
Input: The identified target class k* the subset D selected layer 1*, the reference (wa, k*), and filtering threshold T.
aus and purified set D (D Output: Suspected set D
sus
1:
Compute the GCDs of the set Dir, referred to {0x, (x:) (x corresponding to the reference
(xo, k*),,, according to 22.
2: Find the farthest activation gradient (( ) according to nn = arg (a... Ozo (ar:).
3: Set Die 0. JS U. and iteration ! 0.
4: while \D # 0 do
5: Calculate the distribution of {sao (x;) ep. according to Eq.
6: Add samples (æ,, k²) whose Sm, (a:;) smaller than т, to D eus and remove them from D bdl
7: it' > then
8: Calculate the JS divergence between the distribution of (smo (2;) ( pp in the iteration t and 1- 1.
9:
Add the IS divergence to JS.
10:
end if
11:
f t
12:
end while
13: Find an appropriate iteration t" according to the stopping criteria.
14: Remain samples (z;, y) filtered out before the iteration t" D
1.1.2 THE STOPPING CRITERIA
To find an appropriate stopping iteration t", we utilize the sliding window method to analyze the changes in JS divergence across all iterations. Our goal is to identify the iteration t where the JS divergence is minimal and stabilizes. Let the width of the window be w, and let the JS divergence at each iteration be JS(t), where t E {O...., ,, 1}. The average pm and standard deviation o., of each window starting at position
defined as follows:
20

---

Published as a conference paper at ICLR 2025
1
1m
JS(m + 1),
w
J
(JS(m + umm
m
Each window can be represented by score Sm that combines the value of the average with the standard deviation. Since we are mainly focusing stabilization, we design a metric as described in Eq. which amplifies the contribution of the standard deviation.
Sm = Pm 80
(7)
After computing the score of all windows, we choose the window starting at m with the minimum score:
m arg min Sm.
The stopping iteration is the middle of the optimal window, which is 6" mn + 2 In our method, we adopt w 10 and B
1.1.3 ANALYSIS THE HYPER-PARE PARAMETERS THE STOPPING CRITERIA
In this section, we tested how changes in window size and beta affect detection performance across different attacks on Tab. 13 and Tab. 14] Our findings show that within a certain range, both parameters maintain strong detection performance, making our approach reliable under various conditions. small window size might cause the detection process to stop too early due to fluctuations when few poisoned samples are detected. A large window size might include clean samples, increasing the mean JS divergence and causing an early stop. For 3, small value might focus the later stages with high variance, while a large beta might increase the false positive rate by selecting stable stages. Balancing these parameters is crucial for optimal detection.
Table 13:
The detection performance of AGPD with different window sizes CIFAR-10. The results
are evaluated on Freact-ResNetllee
Table 14:
The detection performance of AGPD with different on CIFAR-10. The results are evaluated on
TPRt
FFR.
F11
TPRT
TPR1
FPR.
Riended
94.65
0.175
96.385
0.175
96.388
385
94.65
0.115
94.38
90.74
0.115
90.74
94.38
0.115
99.8
0.07
99.6
99.8
0.07
99.6
99.8
99.8
0.07
99.6
99.8
0.07
99.6
99.8
0.07
99.6
99.8
0.07
99.6
SIG
100
0.04
99.66
100
0.04
99.66
100
0.04
99.66
100
0.04
99.66
99.44
3.99
72.28
99.44
3.99
72.28
99.44
3.99
72.28
SSBA
99.62
0.01
99.63
99.62
0.01
99.63
99.52
0.01
99.63
99.62
0.01
99.63
99.62
0.04
99
99.62
0.01
99.63
99.62
0.01
99.63
TACT
100
0.07
99.68
100
0.07
99 68
100
0.07
99.68
100
0.07
99.68
100
0.07
99
100
0.07
99.68
100
0.07
99.68
WANet
97.8
0.31
97.4
97.8
0.31
97.4
97.8
0.31
97.4
97.8
0.31
97.4
97.8
0.31
97.4
97.8
0.31
97.4
97.8
0.31
97.4
D.2 COMPARISON BETWEEN CVBT AND VARIANCE
To compare the capabilities of CVBT and variance identifying the target class, provide the values of CVBT and variance for different labels under various attacks and also present the Z-score estimate for the target label in Fig. The disparity between CVBT and circular distribution variance
21

---

Published as a conference paper at ICLR 2025
is notably pronounced within the target category while other clean labels, CVBT is only slightly larger than variance. This also results in the Robust Z-score corresponding to CVBT, the statistical measure for detecting the target category, being greater than the value corresponding to variance, further proving that CVBT more accurately reflects the anomalies of the target category compared to variance.
Di divets
Stended
LF
Choss
(a) The Robust Z-score for CVBT (b) The Robust 7-score for CVBT (c) The Robust Z-score for CVBT is 9.99, whereas for Variance is is 21,34, whereas for Variance is is 24.74 74, whereas for Variance it 7.43.
15.53.
12.90.
5ig
OFT
Chs
(d) The Robust Z-score for CVBT (e) The Robust Z-score for CVBT (f) The Robust Z-score for CVBT is 8.44, whereas for Variance it is 37.50, whereas for Variance it is is 13.04, whereas for Variance it is 3.33.
15.94.
7.90.
Figure 12: CVBT and circular distribution variance on CIFAR-10 with PreAct-ResNet1 across different labels.
D.3 COMPARISON BETWEEN COSINE DISTANCE AND RADIAN
In this section, we compare the different estimation methods of the sample closeness metrie. In Tab. 15 when we replaced 1 - cos(O) in the sample closeness metric with 0, there was no significant change in the True Positive Rate (TPR) across different attacks. However, the False Positive Rate (FPR) sigmificantly increased under certain attacks. This is because when @ approaches O degrees, the derivative of the cosine distance less than This implies that the differences in 9 are greater than those in cosine distance, resulting in the detection of some boundary points within clean samples.
Table 15:
The detection performance of AGPD with radian and cosine distance. The results are evaluated on Preect-ResNet18.
AGPD-Theta
AGPD
Attack
TPR
FPR. FI
TPR
FPR.
Fif
BadNet
87.14
0.16 92.40
90.06
0.03
94.65
Blended
99.84 0.00 99.91
99.98
99.88
LF
99.50
0.01 99.69
99.80
0.07
99.60
SSBA
99.44
0.02 99.64
99.62
0.04
99.63
WaNet
94.77
26.88 41.69
97.80
0.31
97.40
Input-Aware
97.59
18.08
52.42
88.25
1.13 88.61
TaCT
99.68
0.00
99.82
100.00
0.07
99.68
Adap-Blend
90.00 1.07
90.15
89.32
0.33
92.89
22

---

Published as a conference paper at ICLR 2025
D.4 ANALYSIS OF THE DISCRIMINATIVE DEGREE OF ACTIVATION GRADIENT.
D.4.1 GCDS FOR VARIOUS ATTACKS ON CIFAR-10
Fig. 13 and Fig. 14 present the GCDs for classes in CIFAR-10 under eight backdoor attacks with 10% poisoning ratio, based on the model structures of Preact-ResNet1 and VGG19-BN respectively. can be noticed that the target class (covering both black and blue arcs) occupies longer are on the circle compared with the clean classes across different model structures and backdoor attacks. Note that we moved all clean classes' ares to different areas on the circle to avoid visual overlap.
BadNets 132
SIG
LC
WaNet
Input-Aware
TaCT
Adap-Blend
Clack 12.413
014B
0148
Clac
C1480
Clac1
ClrB
Clacs
040
Figure 13: Gradient circular distributions (GCDs) multiple backdoor attacks on CIFAR-10 with Preact-ResNet18.
BedNets
Blended 51x
LF
SSBA
.222
WeNNet
Input-Aware Aware
TaCT
Adap Blend
Class
Orecl
Clai1
O14R
Clack
Cleas
Clack
Olra
Figure 14: Gradient circular distributions (GCDs) of multiple backdoor attacks on CIFAR-10 with VGG19-BN.
D.4.2 THE SEPARATION BETWEEN POISONED AND CLEAN SAMPLES
Fig. 15 displays the cosine similarities of samples in the target class with a clean basis, which are computed in activation gradient and activation spaces, respectively. We exhibit the distributions of cosine similarities of four convolutional layers. a sample clean, the cosine similarity should be large. As shown in Fig. [15] the separation of the distribution of cosine similarities between clean and poisoned samples is larger in the activation gradient space, which can be observed in many convolutional layers.
23

---

Published as a conference paper at ICLR 2025
Activation gradient
layer1.0 0. 2.oov22
18yer2.1.com 1.oo22
layer3. 1.com22
layor4. 0.oon22
700
400
800
600
1200
502
500
Number
100
403
400
500
200
102
100
200
9,0
Similrrty
Cosine Simllartty
Cosine Similarty
Similartty
Activation
layer1.0 1.0..omv2
1yyor..1.cov22
layer3, 1.com2
layer4.0 2yoor40.coov2
300
230
200
Number
100
500
-0.5 Cosine Similarity
0.
91.
-0.5
0.2
04
910
-0.5
0.5
Simlarity
Similaitt
Similarity
Figure 15: The distribution of the cosine similarities of samples from the target class with a clean sample in multiple convolutional layers. The model structure is Preact-ResNet18. The purple represents poisoned samples, while the blue represents clean ones.
D.4.3 ACTIVATION VS. ACTIVATION GRADIENT.
Table 16:
Silhouette scores of the target class under eight attacks, measured in activation and activation gradient spaces, using 1e1
BadNets
Blended
LF
SSBA
Activation
0.529
0.485
0.472
0.497
Activation Gradient
0.664
0.696
0.610
0.623
WaNet
Input-Aware
TaCT
Adap-Blend
Activation
0.544
0.457
0.379
0.403
Activation Gradient
0.605
0.466
0.515
0.491
To demonstrate that the separations clean and poisoned samples differ in activation gradient and activation spaces, we show the distribution of cosine similarities between samples and the clean basis from both spaces. Due to the space limitation, we provide the results in Appendix [D.422] To quantify the separation, we utilize Silhouette Score (Rousseeuw 1987) measure the distance between two clusters in both spaces across all convolutional layers. The range of the Silhouette Score is between -I and 1, with higher values indicating better separability between the clusters. Considering that different depths of convolutional layers correspond to different separations, Tab. [6 presents the maximum Silhouette Scores among all convolutional layers for eight backdoor attacks, from which we find that Silhouette Scores are larger in activation gradient space.
D.S COMPUTATION OVERHEAD
Tab. T7 illustrates the computation complexity and time (based on RTX A5000 GPU) of AGPD and the compared detection method under eight backdoor attacks with 10% poisoning ratio CIFAR-10. We record the average of running time with standard deviation in bracket.
The cost of AGPD (excluding the model training cost) is O(F + B+ D)N), with F, B, D indicating the cost one forward pass, the cost of one backward pass along the trained model, and the feature dimension, respectively, shown in Tab. [7] This cost is at the similar level with most compared methods. Besides, compared with model training, which cost is O(T x (F + B) x N) with T indicating the training epochs, the additional cost of AGPD is negligible.
24

---

Published as a conference paper at ICLR 2025
Table 17: Computation complexity and time of AGPD and the compared methods on CIFAR-10. The value in the bracket represents the standard deviation. General setting: Epoch E; Samples N; Perturbation samples Np: Feature Dimension D; Class K; Forward F; Backward
Reatls
SCA
STRP
KSART
AGRO
OLLE a DONT OILD 2129 DXT2 e DRN, DUPD AIN/7E, ONGN a KAR) O (.EE)) 01FR) DNTP i BBEN DID+ 1 B: OXENY DII D3R 1J0ason)
Time (nteUM
1.33800))
1770000)
0.20
1016/218)
226-3:(12.4
5.20000)
D.6 STABILITY OF AGPD
To ensure the stability of our method for clean samples during poisoned sample detection, we randomly selected 30 random seeds to choose samples from the test dataset. Tab. 18 presents the mean and standard deviation of the results from these 30 experiments, demonstrating that our method is robust to the clean dataset and is not affected by the samples from the clean dataset.
Table 18:
The detection performance of AGPD with different sets of clean samples (mean=std) on CIFAR-10. The results are evaluated on Rreact-eeNNet8.
AGPD
Dataset
Attack
TPR
FPR.
F1 scoref
BadNets
93.2011.3
95.80110.45 0.04:0002
Blended
99.98-1001
0.02-10001
99.83110.06
CIFAR-10
LF SSBA
99.8+10.04
0.07+0.01
99.61=00055
99.68=0.114
0.05=+0.01
99.61=0.055
WaNet
97.77=10.36
0.26:+0.11
99.63++0006
E THE RESULTS OF THE COMPARED ASSET ON RESNET18
In our above experiment, we found that detection performance of ASSET is deviated from the results reported in the paper (Pan et al. 2023), such as BadNets and Blended attacks. Therefore, we decided to replicate these detection results using the recommended model structure, ResNet18. We tested six different attacks, including BadNets, Blended, WaNet, Input-Aware, TaCT, and Adap- Blend. The experiments were conducted on the CIFAR-10 dataset with 10% poisoning ratio. The corresponding results are shown in Tab. 19 When changing the model structure from Preact-ResNett 8 to ResNet1 8, discovered that the detection performance of ASSET becomes better, such the TPRs of BadNets, Blended, and TaCT. However, when the form of the trigger is too complex and dynamic, which requires the model to spend more epochs learning it, this poses a greater challenge for the loss-based method, like ABL and ASSET. The work [Wu et al. 2022) provides more analysis of quick learning of backdoors.
Table 19: The results of ASSET on CIFAR-10 with ResNet18.
BadNets Blended
WaNer
Input-Aware
TaCT
Adap-Blend
TPR
90.70
99.90
1.88
0.25
100.00
54.58
FPR
0.18
9.43
1.69
0.22
0.00
38.90
F1
94.32
69,97
3,55
0.48
100.00
23.43
F. T-SNE RESULTS
In this section, we provide the t-SNE visualizations of ten backdoor attacks conducted on CIFAR-10 with Preact-ResNetll in Fig. 16 The activations are extracted from the last convolutional layer (laver4..ccom2)), From Fig 16 is evident that the dispersion of activations of poisoned and clean samples is significant in some attacks, such BadNets and Blended. Consequently, most activation- based methods perform well against these attacks. Additionally, the dispersion of activations of the target class is relatively small in CTRL attacks, where these methods fail.
25

---

Published as a conference paper at ICLR 2025
BadNets
Blended
SSBA
CTRL
SIG
WaNet
Input-Aware
TaCT
Adap-Blend
Figure 16: t-SNE visualization of ten backdoor models trained on poisoned CIFAR-10 under 10% poisoning ratio for non-clean label attacks and 5% poisoning ratio for clean label attacks, The model architecture Preact-ResNet18.
G RESULTS FOR ADAPTIVE ATTACKS
In this section, present the results and analysis for adaptive attacks in which adversaries also added perturbations to clean samples used by defenders. The performance of AGPD under these conditions summarized in Tab 20 Additionally, we visualize the distribution cosine similarities among poisoned, noisy, and clean samples in the activation gradient space under two modes, as shown in T7 and Fig [8] respectively, to further demonstrate the effectiveness of our method.
The poisoned dataset consists of 10% poisoned samples and 10% clean samples with added noise. In these clean samples, noise is inserted into the images of the blending ratio 0.2 without altering the labels. There are two modes of adding noise: in the Random mode, all noise is generated randomly, while in the Fixed mode, half of the noise fixed, and the other half is generated randomly. We evaluated the effectiveness of AGPD in detecting various attacks on the CIFAR-10 10 dataset using the PreactResNet-18 architecture.
Experimental setup:
Analysis: observed that in both two noise modes, the distribution of noise samples closely resembles that of clean samples. Despite these samples with noise, they can retain their original features, resulting in similar gradients to those of clean samples. This observation underscores the robustness of AGPD in distinguishing between clean and poisoned samples, even when clean samples are added with noise. The preservation of original features in noisy samples ensures that their gradients remain unaffected, allowing AGPD to effectively detect poisoned samples without interference from noise. Additionally, we offered the performance of AGPD against adaptive attacks in Tab. 20 It can be seen that the proposed method maintained high TPR and relatively low FPR across different backdoor attacks and both noise-adding modes.
Table 20: Performance of AGPD against adaptive attacks with different attacks under poisoning ratio=10% on CIFAR-10 and ResNet-18.
Attack
Mode
TPR
FPR
F1
Random
86.68
0.01
92.83
BadNets
Fixed
87.24
0.18
85.42
Random
99.88
0.00
99.93
Blended
Fixed
99.56
0.00
99.76
Random
99.24
0.01
99.59
LF
Fixed
99.36
0.01
99.62
Random
99.42
0.03
99.56
SSBA
Fixed
99.52
0.03
99.63
26

---

Published as a conference paper at ICLR 2025
BadNet Random
Blended Random
SSBA Random
Random
Pesson
Olaan
Phion
Orean
600
Phoom
Ciaan
7200
Clean
husee
300
Potion
Notte
620
000
300
100
25.0
-3.5
5-0
0.5
1x
A
-6.5
6,
0.5
10
-6.5
5.0
0.5
-3.5
0.5
255
10
Cosine Similarity Random
Casine Simi arity
Cosine Similarity
Cosine Similarity
Blended Random
SSBA Random
Random
1600
Cress
2320
Mukic
1200
1350
1250
3020
1000
150
400
250
250
250
200
-10
-0.5
2.0
0.5
-0.5
1.2
Fu.
-0.5
0.5
33.0
-1.5
0.5
2.5
1.0
Cosine Similarity
Cosine Similarity
Cosine Similarity
Cosine Similarity
Figure 17: The distribution of the cosine similarities of samples from the target class (above) and samples from one non-target class (below) in activation space for Random mode across multiple convolutional layers. The purple represents poisoned samples, the blue represents clean ones, and the green represents noise ones.
BodNet Fixed
Blended Fixed
SSBA Fixed
Fixed
700
Caat
120
Denn
Cleat
702
Ceenn Pokee
600
Foison Nudus
500
000
300
500
4000
400
300
300
200
200
200
100
300
1.
0.5
10
3.0
8.5
DO
85
10
L2
0.5
0.5
03
1.0
10
25
BD
25
Cosine Similarity
Cosine Similarity ity
Cosine Similarity
Cosine Similanty
BadNiet Fixed
Blended Fixed
SSBA Fixed
Fixed
3030
Caecn
Moine
2000
Oann hokse
1402
1200
Hodke
Clear
3752
3500
Nobee
Oeerr
833
3300
1352
1040
750
500
502
300
202
252
-30
-0.5
0.3
6.5
-15
5.0
-0.5
2.5
0.5
3.0
-3.5
0.2
05 10
Casine Similarity
Cosine Similartt
Cosine Similority
Cosine Similarity
Figure 18: The distribution of the cosine similarities of samples from the target class (above) and samples from one non-target class (below) in activation space for Fixed mode across multiple convolutional layers. The purple represents poisoned samples, the blue represents clean ones, and the green represents noise ones.
H RESULTS FOR CLEAN-LABEL ATTACKS
In this section, we provide visualization of the cosine similarity distribution for samples in the target class to illustrate AGPD's detection capability against clean-label attacks, as shown in F9g It is evident that there is clear separation between the two types of samples in the activation gradient space, even though the poisoned and clean samples belong to the same class. Clean-label attacks, despite exhibiting similar visual features in the input space, inherently disrupt the internal characteristics of the original samples within the model. This disruption impacts the learning of clean features and results in the model memorizing the adversarial triggers. This memorization causes divergence in the gradient space between clean and poisoned samples, underscoring the effectiveness of our AGPD method in detecting clean-label attacks.
27

---

Published as a conference paper at ICLR 2025
LC
SIG
Clean Poison
Clean
400
Poison
Number
Number
300
300
200
200
100
100
11.0
-0.5
0.0
0.5
1.0
-..
-0.5
0.0
0.5
1.0
Cosine Similarity
Cosine Similarity
Figure 19: The distribution of the cosine similarities of samples from the target class.
I RESULTS FOR NOISY AND POISONED SAMPLES
Considering adversaries are able to inject noisy samples to further improve the attack stealthiness, we provide the visualization of the distribution of cosine similarities for noisy and clean samples sharing the same label under three backdoor attacks in Fig 20
Experimental setup: it to its original label.
We create a noisy sample by adding noise to poisoned sample and reverting
Analysis: We found that the gradient distributions of noisy samples and clean samples are almost identical. This is because the noisy sample retains the class information and has not been altered to the target label, so the inherent clean information of the model predominantly influences the gradient.
Adep
Imput
206
5,30 +0.75 -0.00
-0.5
6.23
8.3
0.75
5.20
5u0 -0.5 -000
-UUS
5.23 3.25 Siilentty
8.50
3.75
5.30
Jus
HU05 -056
-UUS
2.30 325 0.50 5.00
Smilnnty
Figure 20: The visualization of cosine similarities for noisy and clean samples. The blue represents the clean ones and the green represents noisy ones.
28


Reviews:
[{"review_id": "oGnrfoVTR4", "reviewer": "Reviewer_nov3", "summary": "The authors present an approach for backdoor sample detection based on the angle of the activation gradients between samples and a reference clean sample for each class.\nThe paper observes that, within the target class, the activation gradient angles exhibit greater dispersion compared to samples from the clean class. Furthermore, the distribution of angles of backdoor and clean samples within the target class are somewhat distinct, providing a basis for differentiating between clean and poisoned samples.\n\nBuilding on this observation, the authors introduce the concept of the Gradient Circular Distribution (GCD), a distribution of angles between the activation gradient of samples and that of a reference sample. They propose two key metrics based on this distribution:\nCVBT: A metric designed to measure the dispersion of the GCD.\nSample Closeness: A metric that evaluates how close a given sample is to  reference clean sample.\n\nThese metrics are then utilized to develop a filtering algorithm, which first identifies the target class and subsequently filters out the poisoned samples. The paper validates the proposed method through experiments on a range of backdoor attack scenarios.", "questions": "1. Can you please elaborate on this : \"It is inspired by phenomenon that a backdoored model tends to map both poisoned and clean samples within the target class to similar areas in its activation space\" ? - I believe this is true only for self-supervised training, whereas the current paper focuses on supervised training.\n\n2. At which h,w is the activation getting sliced for calculating the activation gradient ? Is it the same for all layers considered ? Why ?\n\n3. Can we simply use the difference in the angles from 0 in the distribution $\\rho$ instead of the sample closeness metric ? Why do we need to use it ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies a previously unexplored phenomenon in backdoor samples, highlighting greater dispersion in activation gradient angles within the target class compared to clean samples.\n2. The authors provide extensive experiments on various backdoor attacks across two architectures (PreAct ResNet18 and VGG-18 BN),\n3. Some initial analysis on adaptive attacks is presented, though further exploration could strengthen the findings.\n4. The writing is clear and well-structured, making the main arguments easy to follow.", "weaknesses": "1. I am giving low scores to soundness due to the reported  low performance of ASSET, AC. The performance in the original paper of ASSET is close to 90% for all attacks. Additionally, 0 F1 scores for AC also does not look right. For example, in the aforementioned paper, the average TPR of AC is around 50%. \n\n2.  Please provide sensitivity of various thresholds like $\\tau_z$ , $w, \\beta$ (Stage 3). What is the rationale behind the choice of $\\tau_z$ ? \n\n3.  Please comment on or provide results for these additonal adaptive attacks: \na. Can an adversary add  perturbations to the clean samples (different perturbations to different samples) such that their dispersion increases ? \nb. Given a clean set with 100 images, the adversary adds the same noise to 50 images and keeps the other images the same. Will the  dispersion increases of clean sample increase ? \nc. The backdoor trigger is optimized such that the dispersion of the target class decreases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Dje4WDcs1", "reviewer": "Reviewer_bpNy", "summary": "The authors propose to detect poisoned examples from an activation-gradient circular distribution perspective. The authors draw two novel observations by studying the GCD of poisoned/clean examples, based on which they introduce an algorithm named AGPD to do backdoor example detection.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Understanding the activation gradient of poisoned examples with circular distribution is interesting. The two observations of GCD are novel.\n\n(2) The algorithm shows strong performance within the evaluation setting of this paper.\n\n(3) The paper is overall well written and the logic is easy to follow. The empirical experiments are mostly comprehensive and valid.", "weaknesses": "(1) My main concern about the proposed algorithm is its scalability. It requires training on the entire dataset to determine the detection state, which might be too expensive given web-scale data and the extreme probability of the data being poisoned. Also, it takes nearly $O(n2) $ to compute the GCD, which is infeasible for large datasets. \n\n(2) The defense algorithm appears to be highly correlated with the specific class labels, potentially limiting its applicability. In the real world, the attackers can adopt a different set of labels than the defenders. The attackers could label the dataset more fine-grained/coarse-grained than the defender. (e.g. Imagine there is a dataset consisting of snacks and drinks and the target class of the attacker is Coca-Cola. However, the defenders just want to classify the images as soft drinks no matter whether it is Coca-Cola or Pepsi. Or the reverse situation can also happen.) Will the observations about GCD hold under such misalignment?\n\n(3) Table 7 is not organized well, please have a double-check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3s9pI1cEc0", "reviewer": "Reviewer_pwS4", "summary": "The paper presents a new approach to identifying poisoned samples in datasets used for deep neural network training. The core proposal, termed Activation Gradient-Based Poisoned Sample Detection (AGPD), leverages a novel metric called Gradient Circular Distribution (GCD) to detect discrepancies between clean and poisoned samples. Key insights include observing that in backdoor attacks, models often map poisoned samples and clean samples to similar activation regions, which AGPD detects by analyzing gradient direction distributions. This method involves three main steps: calculating GCD for each class, identifying target classes based on dispersion, and filtering out poisoned samples.", "questions": "(1) The proposed method is founded on two core observations: (i) the Gradient Circular Distribution (GCD) of the target class exhibits greater dispersion than those of clean classes, and (ii) poisoned and clean samples are distinguishable by their clear clustering in separate regions. However, it remains unclear whether these observations hold consistently for clean-label attacks, where the mapping directions of poisoned and benign samples could exhibit higher similarity given their similar input-space characteristics. Further discussion on this aspect would strengthen the paper’s analysis.\n\n(2) The choice of baseline methods appears limited to older techniques. A broader comparison with more recent, state-of-the-art defense methods would provide a more thorough assessment of the proposed method’s performance relative to the latest advances in poisoned sample detection.\n\n(3) Attack Success Rate (ASR) is a key metric commonly used in backdoor detection evaluations. Its omission from Table 1 raises questions about the comprehensiveness of the evaluation metrics chosen. Including ASR would provide a clearer view of the method's efficacy against backdoor attacks.\n\n(4)   In the main evaluation, a poisoning ratio of 10% is used for non-clean label attacks and 5% for clean-label attacks. Since clean-label attacks are generally more challenging, lower poisoning ratios are typically applied to non-clean label attacks in real-world scenarios. This discrepancy between the paper’s experimental setup and actual attack contexts calls for further clarification. Additionally, practical attack scenarios often employ very low poisoning ratios (1% or 0.5%), feature separability-based defenses tend to be less effective in this setting. More discussion should be included.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Introducing GCD as a novel measure adds a unique, technical depth to the approach, potentially applicable to various attack scenarios.\n\n(2)  AGPD shows high detection performance with minimal additional clean data, a notable advantage for real-world applicability, where such data may be limited.", "weaknesses": "(1) The proposed method is founded on two core observations: (i) the Gradient Circular Distribution (GCD) of the target class exhibits greater dispersion than those of clean classes, and (ii) poisoned and clean samples are distinguishable by their clear clustering in separate regions. However, it remains unclear whether these observations hold consistently for clean-label attacks, where the mapping directions of poisoned and benign samples could exhibit higher similarity given their similar input-space characteristics. Further discussion on this aspect would strengthen the paper’s analysis.\n\n(2) The choice of baseline methods appears limited to older techniques. A broader comparison with more recent, state-of-the-art defense methods would provide a more thorough assessment of the proposed method’s performance relative to the latest advances in poisoned sample detection.\n\n(3) Attack Success Rate (ASR) is a key metric commonly used in backdoor detection evaluations. Its omission from Table 1 raises questions about the comprehensiveness of the evaluation metrics chosen. Including ASR would provide a clearer view of the method's efficacy against backdoor attacks.\n\n(4)   In the main evaluation, a poisoning ratio of 10% is used for non-clean label attacks and 5% for clean-label attacks. Since clean-label attacks are generally more challenging, lower poisoning ratios are typically applied to non-clean label attacks in real-world scenarios. This discrepancy between the paper’s experimental setup and actual attack contexts calls for further clarification. Additionally, practical attack scenarios often employ very low poisoning ratios (1% or 0.5%), feature separability-based defenses tend to be less effective in this setting. More discussion should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVBnOuUbWL", "reviewer": "Reviewer_AHKK", "summary": "With the inspiration that a backdoored model is apt to map significantly different poisoned samples and clean samples of the backdoor target to similar activation areas, this paper introduces a novel measurement, i.e., the circular distribution of the gradients w.r.t sample activation, namely GCD, which works to identify the target class of the backdoor and consequently separate poisoned and clean samples within the target class. Accordingly, this paper proposes a sample detection approach called AGPD to achieve dataset purification. Extensive experiments show AGPD's advanced performance in detecting and isolating poisoned samples.", "questions": "Q-(1): The calculation of $\\rho_{x_0}\\(\\mathcal{D}\\)$ in Eq.(3) and $s_{x_0}\\(x_i\\)$ in Eq.(4) basically transforms e.g. $\\theta_{x_0}\\(x_i\\)$ back to $\\frac{g\\(x_i\\) \\cdot g\\(x_0\\)}{||g\\(x_i\\)|| ||g\\(x_0\\)||}$. By simplifying the formulation in Eq.(3) and Eq.(4), does this mean it is essentially calculating the cosine similarity of the gradients between the basis pair and each input sample? If this is indeed the case, is the calculation of radian in Eq.(2) useful merely for the visualization in figures like Figure 1.?\n\nQ-(2): In section 3, the channel-wise activation gradient $g^{\\(l\\)}$ is first introduced in Eq.(1). However, the formulating w.r.t the index of the model layer is missing in the consequent methodology description. Can the authors elaborate on how the channel-wise gradients in each layer are merged into the final GCD of an input sample $x_i$?\n\nQ-(3): The basis pair is crucial to disperse samples; thus, a small clean set is necessary for APGD detection. As even a single clean sample per class has been proven effective, how does the selection of clean samples influence APGD's performance? Or is APGD resistant to any random sampling of clean samples?\n\nQ-(4): Different from the study on the threshold $\\tau_s$ in section 5.4, $\\tau_z$ remains equal $e^2$. What is the rationale of setting $\\tau_z = e^2$? \n\nQ-(5): In Figure 4, APGD detection on the target class works better against all-to-one attacks than all-to-all attacks. While detecting poisoned samples of all-to-all attacks, does APGD handle the separation of all-to-all attacks class by class via the implementation of Stage 3?\n\nQ-(6): In the default setting of Adaptive-Blend, the poisoning ratio is relatively low (poisoned/total = 150/50000 = 0.3% for CIFAR10), where the same amount of noisy samples as poisoned samples are added by setting the conservatism ratio = 1.0, which is unfortunately missing in Table 7. Moreover, considering the default poisoning ratio = 5%, the ASR of 66.17% for Apat-Blend is much lower than that of the original paper. Does this imply that the Adptive-Blend in this paper has some biases from the default attack configuration?\n\nQ-(7): Considering that both WaNet and Adaptive-Blend attacks involve noisy samples to improve the attack stealthiness, can the authors provide some visualization on the separation of APGD between the poisoned samples and noisy samples in both attacks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel measurement to show the dispersion of poisoned and clean samples in the model's activations\n\n- This paper is well-written and enforces a high readability with variant and helpful illustrations \n\n- This paper comprehensively evaluates AGPD's performance by considering diverse poisoning attacks, comparing with different related defenses, and conducting experiments on variant datasets.\n\n- This paper considers the ablation study on different hyper-settings of APGD and involves two reasonable adaptive attacks that have considered APGD's defense w.r.t the GCD distribution.", "weaknesses": "(1) The use of $arccos(\\cdot)$ in Eq.(2) is ambigious as $cos\\(\\cdot\\)$ is used in Eq.(3) and Eq.(4) as well.\n\n(2) The calculation of GCD across all model layers is not clearly formulated for each sample.\n\n(3) The influence of the clean sample by random sampling for the basic sample pair is not studied.\n\n(4) There is lacking the rationale for setting the hyper-parameter $\\tau_z = e^2$.\n\n(5) For all-to-all attacks, a better explanation of how the dataset separation by APGD is achieved is required.\n\n(6) The description of Adpative-Blend in Appendix B is incomplete and different from the default attack setting.\n\n(7) It is unclear how the noisy and poisoned samples are separated against WaNet and Adaptive-Blend attacks.\n\n---\nIn the following, further questions related to the above points are detailed with the notion of \"Q-#\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)