Activation Gradient based Poisoned Sample Detection Against Backdoor Attacks

## Abstract

Abstract This work studies the task of poisoned sample detection for defending against data poisoning based backdoor attacks. Its core challenge is finding a generalizable and discriminative metric to distinguish between clean and various types of poisoned samples ( e.g., various triggers, various poisoning ratios). Inspired by a common phenomenon in backdoor attacks that the backdoored model tend to map significantly different poisoned and clean samples within the target class to similar activation areas, we introduce a novel perspective of the circular distribution of the gradients w.r.t. sample activation, dubbed gradient circular distribution (GCD). And, we find two interesting observations based on GCD. One is that the GCD of samples in the target class is much more dispersed than that in the clean class. The other is that in the GCD of target class, poisoned and clean samples are clearly separated. Inspired by above two observations, we develop an innovative three-stage poisoned sample detection approach, called Activation Gradient based Poisoned sample Detection (AGPD) . First, we calculate GCDs of all classes from the model trained on the untrustworthy dataset. Then, we identify the target class(es) based on the difference on GCD dispersion between target and clean classes. Last, we filter out poisoned samples within the identified target class(es) based on the clear separation between poisoned and clean samples. Extensive experiments under various settings of backdoor attacks demonstrate the superior detection performance of the proposed method to existing poisoned detection approaches according to sample activation-based metrics.

## 1 Introduction

It is well known that deep neural networks (DNNs) are vulnerable to backdoor attacks [], where the adversary could inject a particular backdoor into the DNN model through manipulating the training dataset or training process.
Consequently,the backdoored model will produce a target label when encountering a particular trigger pattern, leading to unexpected security threats in practice.
Protecting DNNs from backdoor attacks is an urgent and important task.

Here we focus on defending against the data-poisoning based backdoor attacks by filtering out the potential poisoned samples from a untrustworthy training dataset, i.e., poisoned sample detection (PSD). One of the main challenges for PSD is the information lack of the potential poisoned samples, such as the trigger type, the target class(es), the number of poisoned samples, etc. Some seminal works have been developed by exploring some discriminative metrics based on the intermediate activation or predictions of poisoned and clean samples in the backdoored model trained on the untrustworthy dataset, such as activation clustering (AC) [], STRIP [], SCAn []. However, the recently proposed adaptive backdoor poisoning attack [] challenges the assumption that poisoned and clean samples can be distinctly separated in activation space.

Figure: Figure 1: Gradient circular distributions (GCDs) across four classes of CIFAR-10, on the clean model (left), Blended attacked model (middle), and SSBA attacked model (right), respectively. The value along with each arc indicates the CVBT value. The GCD of the target class (covering both black and blue arcs). Note that we moved three clean classes’ arcs to different quadrants to avoid visual overlap.
Refer to caption: extracted/5625055/figure/theta_1.png

In this work, we introduce a novel perspective that distinguishes the behavior of poisoned and clean samples by tracking their activation gradients (i.e., the gradient w.r.t. activation). It is inspired by the phenomenon that a backdoored model tends to map both poisoned and clean samples within the target class to similar areas in its activation space [], such that they can be predicted as the same label. Considering the significant discrepancy between poisoned and clean samples in their original input space, their mapping directions should be significantly different, while the mapping direction could be reflected by the activation gradient direction.
Thus, we define a new concept called gradient circular distribution (GCD) (introduced in Section 3), to capture the distribution of activation gradient directions.
Take Fig. 1 as the example, given a trained model, we calculate one GCD of training samples in each class.
There are two interesting observations:

- •
Observation 1 on GCD dispersion: Given one backdoored model (see the middle/right sub-figure), the GCD of the target class is much more dispersed than GCDs of all clean classes;
- •
Observation 2 on sample separation in target GCD: In the GCD of target class, poisoned and clean samples are clearly separated (see the black and blue arcs in the middle/right sub-figure).

Motivated by above two observations, we develop an innovative poisoned sample detection approach, called Activation Gradient based Poisoned sample Detection (AGPD), which consist of three stages.
First, we train a DNN model based on the untrustworthy dataset, and calculate GCDs of all classes.
Second, we identify the target class(es) according to a novel class-level metric that measures the dispersion of each class’s GCD (corresponding to the first observation).
Last, within the identified target class(es), we gradually filter out poisoned samples according to a novel sample-level metric that measures the closeness to the clean reference sample (corresponding to the second observation).

Moreover, we conduct extensive evaluations under various backdoor attacks and various datasets, and show that the activation gradient is more discriminative than the activation to distinguish between poisoned and clean samples, which explains the superior

In summary, the main contributions of this work are three-fold.
(1) We introduce a novel perspective for poisoned sample detection, called gradient circular distribution (GCD), and present two interesting observations based on GCD.
(2) We develop an innovative approach by sequentially identifying the target class(es) and filtering poisoned samples for the poisoned sample detection task, based on GCD and two novel metrics about GCD.
(3) We conduct extensive evaluations and analysis to verify the superiority of the proposed approach to existing activation-based detection approaches.

## 2 Related work

#### Backdoor attack.

BadNets [] is the pioneering work that introduces the concept of backdoor attack into Deep Neural Networks (DNNs), in which the adversary manipulates training samples by adding a small patch with specific patterns and changing their labels to a target label.
Following this, the variety of triggers expanded significantly, including a cartoon image used in Blended [], a universal adversarial perturbation with only low-frequency components utilized in Low-Frequency [], and a sinusoidal signal employed in SIG [], etc.
One commonality of these triggers is that they are sample-agnostic, i.e., they are same across different poisoned samples. This characteristic is often exploited to effectively detect and remove the triggers or the poisoned samples.
To counteract this, sample-specific triggers have been designed, such as WaNet [], Input-Aware [], SSBA [], CTRL [], TaCT [], and Adap-Blend [].
These attacks use more complex and dynamic triggers, posing significant challenges for poisoned sample detection.
Additionally, some attacks explore various attack settings regarding the number of triggers and target classes, such as all-to-all attack (e.g., BadNets-A2A []), multi-target and multi-trigger attack (e.g., c-BaN []). These diverse settings further complicate the detection of poisoned samples.

#### Backdoor defense.

According to the accessible information, several different branches of backdoor defense methods have been developed, such as the pre-training backdoor defense (e.g., [, , ]) if given a untrustworthy training dataset, in-training backdoor defense (e.g., [, , , , ]) if the training process can be controlled by defender, as well as post-training backdoor defense (e.g., [, , , , , , , , , ]) if given a backdoored model. Due to space limitations, we will only review existing methods of poisoned sample detection (PSD), which belong to pre-training backdoor defense. Most existing PSD methods aim to construct discriminative metrics between poisoned and clean samples based on intermediate activations, final predictions, or loss values. For activation-based methods, such as activation clustering (AC) [], Beatrix [], SCAn [], and Spectral [], they utilize dimensionality reduction and clustering techniques, such as K-means clustering, Gram matrix analysis, two-component decomposition, and SVD, to distinguish poisoned and clean samples. For input-based methods, STRIP [] uses the entropy of predictions on perturbed inputs to identify poisoned samples, while CD [] measures the $L_{1}$ norm of the learned masks on inputs to detect poisoned samples. For loss-based methods, like ABL [] and ASSET [], they observed that the loss of poisoned samples decreases quickly during the early training epochs, leveraging this phenomenon to identify poisoned samples.

## 3 Preliminary: gradient circular distribution

#### Task setting.

In this section, given a DNN-based classification model $f_{\bm{w}}:\mathcal{X}\rightarrow\mathcal{Y}$, with $\mathcal{X}\in\mathbb{R}^{d}$ being the input sample space and $\mathcal{Y}=\{1,2,\ldots,K\}$ being the output space with $K$ candidate classes, as well as a dataset $\mathcal{D}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$, we investigate their gradients in the model $f_{\bm{w}}$.

### 3.1 Definition of gradient circular distribution

Here we introduce the definitions of Activation Gradient and Gradient Circular Distribution (GCD), as described in Definition 1 and Definition 2, respectively.

###### Definition 1 (Activiation Gradient)

Given a model $f_{\bm{w}}$, for a sample $\bm{x}$ labelled as $y$, we denote its activation map at the $l$-th layer as $\bm{h}^{(l)}_{\bm{x}}\in\mathbb{R}^{C^{(l)}\times H^{(l)}\times W^{(l)}}$, where $C^{(l)}$, $H^{(l)}$, $W^{(l)}$ are its depth (number of channels), height and width, respectively. Then, we define the channel-wise activation gradient
$\bm{g}^{(l)}(\bm{x},y)\in\mathbb{R}^{C^{(l)}}$ as

$$ $\bm{g}_{\bm{w}}^{(l)}(\bm{x},y)=\frac{1}{H^{(l)}W^{(l)}}\sum_{h=1}^{H^{(l)}} \sum_{w=1}^{W^{(l)}}\frac{\partial[f_{\bm{w}}(\bm{x})]_{y}}{\partial{[\bm{h}^{ (l)}_{\bm{x}}]_{:,h,w}}}\in\mathbb{R}^{C^{(l)}},$ (1) $$

where $[f_{\bm{w}}(\bm{x})]_{y}$ is the logit w.r.t. class $y$ and $[\bm{h}^{(l)}_{\bm{x}}]_{:,h,w}\in\mathbb{R}^{C^{(l)}}$ is the activation sliced at height $h$ and width $w$ over all channels.
For simplicity, if no special specifications are required, hereafter we will refer to it as $\bm{g}(\bm{x})$.

###### Definition 2 (Gradient Circular Distribution (GCD))

Given a model $f_{\bm{w}}(\cdot)$, a set of samples $\mathcal{D}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$, and a basis sample pair $(\bm{x}_{0},y_{0})$, we firstly calculate the activation gradient of each sample, i.e., $\bm{g}(\bm{x}_{i}),i=0,1,\ldots,n$.

Then, take $\bm{g}_{\bm{x}_{0}}$ as the (unnormalized) basis vector, the angle of each sample in $\mathcal{D}$ is calculated as follows:

$$ $\theta_{\bm{x}_{0}}(\bm{x}_{i})=\arccos\left(\frac{\bm{g}(\bm{x}_{i})\cdot\bm{ g}(\bm{x}_{0})}{\|\bm{g}(\bm{x}_{i})\|\|\bm{g}(\bm{x}_{0})\|}\right)\in[0,2\pi ),i=1,\ldots,n,$ (2) $$

where $\cdot$ denotes the dot product, and $\|\cdot\|$ returns the magnitude.

The distribution of the angle set $\{\theta_{\bm{x}_{0}}(\bm{x}_{i})\}_{i=1}^{n}$ is called as the gradient circular distribution (GCD) of the set $\mathcal{D}$, denoted as
$\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$.

###### Definition 1 (Activiation Gradient)

###### Definition 2 (Gradient Circular Distribution (GCD))

### 3.2 Characteristics of gradient circular distribution

Furthermore, to quantize the characteristics of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$, we introduce the following two metrics.

#### Dispersion metric of 𝒫 𝒙 0 ⁢ ( 𝒟 ) subscript 𝒫 subscript 𝒙 0 𝒟 \mathcal{P}_{\bm{x}_{0}}(\mathcal{D}) caligraphic_P start_POSTSUBSCRIPT bold_italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( caligraphic_D ) .

To measure the dispersion of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$, we design a novel metric called
Cosine similarity Variation towards Basis Transition (CVBT).
Specifically, given $\{\theta_{\bm{x}_{0}}(\bm{x}_{i})\}_{i=1}^{n}$, we firstly pick the activation gradient $\bm{g}_{\bm{x}_{n^{*}}}$ corresponding to the largest angle, i.e., $n^{*}=\arg\max_{i\in\{1,\ldots,n\}}\theta_{\bm{x}_{0}}(\bm{x}_{i})$.
In other words, $\bm{g}(\bm{x}_{n^{*}})$ is the farthest activation gradient vector from the original basis vector $\bm{g}(\bm{x}_{0})$.
Then, by setting $\bm{g}(\bm{x}_{n^{*}})$ as a new basis vector, we calculate $\{\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i})\}_{i=1}^{n}$ using Eq. (2). Based on $\{\theta_{\bm{x}_{0}}(\bm{x}_{i})\}_{i=1}^{n}$ and $\{\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i})\}_{i=1}^{n}$, we formulate the CVBT metric of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$ as follows:

$$ $\rho_{\bm{x}_{0}}(\mathcal{D})=\bigg{(}\frac{1}{n}\sum_{i=1}^{n}\big{(}\cos( \theta_{\bm{x}_{0}}(\bm{x}_{i}))-\cos(\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i}))\big {)}^{2}\bigg{)}^{\frac{1}{2}}\in[0,2],$ (3) $$

where $\cos(\theta)$ returns the cosine value of an angle $\theta$.
Note that larger $\rho_{\bm{x}_{0}}(\mathcal{D})$ indicates larger dispersion of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$.
For example,
if the farthest vector $\bm{g}(\bm{x}_{n^{*}})$ has the same direction with $\bm{g}(\bm{x}_{0})$, implying $\{\theta_{\bm{x}_{0}}(\bm{x}_{i})\}_{i=1}^{n}$ are all zeros, then $\rho_{\bm{x}_{0}}(\mathcal{D})=0$, meaning the zero dispersion of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$.
In contrast, if $\bm{g}(\bm{x}_{n^{*}})$ has the opposite direction with $\bm{g}(\bm{x}_{0})$, and all remaining $\bm{g}(\bm{x}_{i})$ are either same or opposite directions with $\bm{g}(\bm{x}_{0})$, then $\rho_{\bm{x}_{0}}(\mathcal{D})=2$, meaning the largest dispersion of $\mathcal{P}_{\bm{x}_{0}}(\mathcal{D})$.

#### Sample-level closeness metric based on 𝒫 𝒙 0 ⁢ ( 𝒟 ) subscript 𝒫 subscript 𝒙 0 𝒟 \mathcal{P}_{\bm{x}_{0}}(\mathcal{D}) caligraphic_P start_POSTSUBSCRIPT bold_italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( caligraphic_D ) .

Given $\{\theta_{\bm{x}_{0}}(\bm{x}_{i})\}_{i=1}^{n}$ and $\{\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i})\}_{i=1}^{n}$, we design a novel metric to measure the closeness of each sample $\bm{x}_{i}$ to the reference sample $\bm{x}_{0}$, as follows:

$$ $s_{\bm{x}_{0}}(\bm{x}_{i})=\frac{1-\cos(\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i}))}{ (1-\cos(\theta_{\bm{x}_{n^{*}}}(\bm{x}_{i}))+(1-\cos(\theta_{\bm{x}_{0}}(\bm{x }_{i}))}\in[0,1).$ (4) $$

Note that larger $s_{\bm{x}_{0}}(\bm{x}_{i})$ indicates greater closeness of $\bm{x}_{i}$ to $\bm{x}_{0}$. For example, if $\bm{g}(\bm{x}_{i})$ has the same direction with $\bm{g}(\bm{x}_{n^{*}})$ while the opposite direction with $\bm{g}(\bm{x}_{0})$, then $s_{\bm{x}_{0}}(\bm{x}_{i})=0$, implying the farthest from $\bm{x}_{0}$.
In contrast, if $\bm{g}(\bm{x}_{i})$ has the opposite direction with $\bm{g}(\bm{x}_{n^{*}})$ while the same direction with $\bm{g}(\bm{x}_{0})$, then $s_{\bm{x}_{0}}(\bm{x}_{i})=1$, implying the closest to $\bm{x}_{0}$.

## 4 Activation gradient based poisoned detection method

### 4.1 Problem setting

#### Threat model.

We consider the threat model of data poisoning based backdoor attack. The adversary generates a poisoned dataset $\mathcal{D}_{bd}$, containing a clean subset $\mathcal{D}_{c}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n_{c}}$ and a poisoned subset $\mathcal{D}_{p}=\{(\tilde{\bm{x}}_{i},t)\}_{i=1}^{n_{p}}$. $\bm{x},\tilde{\bm{x}}\in\mathcal{X}$ denotes the clean and poisoned sample with trigger, respectively. $y,t\in\mathcal{Y}$ indicates the ground-truth and target label, respectively.
We denote $r=\frac{n_{p}}{n_{c}+n_{p}}$ as the poisoning ratio.
Note that there could be multiple triggers (i.e., multi-trigger) and multiple target labels (i.e., multi-target) in the poisoned subset.

#### Defender’s goal.

The defender aims to identify poisoned samples from the untrustworthy dataset $\mathcal{D}_{bd}$.
We assume that the defender has access to $\mathcal{D}_{bd}$, and a small set of additional clean samples $\mathcal{D}_{ac}$, which contains at least one clean sample for each class.
Besides, the defender has the capability to train a DNN model $f_{\bm{w_{bd}}}:\mathcal{X}\to\mathcal{Y}$ based on $\mathcal{D}_{bd}$.

Figure: Figure 2: Illustrations of gradient circular distribution (GCD) and two metrics, and the pipeline of the proposed APGD method which consists of three stages: 1) calculating activation gradient distribution, 2) identifying target class(es), and 3) filtering out poisoned samples within the identified target class(es).
Refer to caption: extracted/5625055/figure/framework.png

### 4.2 Poisoned sample detection method

Inspired by the two observations demonstrated in Section 1 and Fig. 1, we develop an innovative poisoned sample detection method by utilizing GCD and the corresponding metrics (see Section 3), called Activation Gradient based Poisoned Detection (AGPD).
As illustrated in Fig. 2, AGPD consists of three stages, as detailed below.

#### Stage 1: Calculating activation gradient distribution.

We denote the samples of class $k$ in $\mathcal{D}_{bd}$ as $\mathcal{D}_{bd}^{k}=\{(\bm{x}_{i},k)\}_{i=1}^{n_{k}}$.
Given the model $f_{\bm{w_{bd}}}$ trained on $\mathcal{D}_{bd}$ (the training details will be provided in Appendix C), and picking one clean sample pair $(\bm{x}_{0}^{k},k)\in\mathcal{D}_{ac}$ as the reference, we can calculate the GCD of $\mathcal{D}_{bd}^{k}$, according to Eqs. (1) and (2). Consequently, we obtain $\{\mathcal{P}_{\bm{x}_{0}^{k}}^{(l)}(\mathcal{D}_{bd}^{k})\}_{l,k=1,1}^{L,K}$.
Note that as defined in Eq. (1), the superscript $(l)$ indicates that we adopt the activation gradients of the $l$-th layer in $f_{\bm{w_{bd}}}$ to calculate GCD.
For simplicity, hereafter we denote $\mathcal{P}_{\bm{x}_{0}^{k}}^{(l)}(\mathcal{D}_{bd}^{k})$ as $\mathcal{P}_{k}^{(l)}$.

#### Stage 2: Identifying target class(es).

According to the aforementioned first observation, the GCD of the target class is likely to be more dispersed than that of the clean class.
Thus, we firstly calculate the dispersion value $\rho_{\bm{x}_{0}^{k}}^{(l)}(\mathcal{D}_{bd}^{k})$ (for simplicity, we denote it as $\rho_{k}^{(l)}$) of each $\mathcal{P}_{k}^{(l)}$, according to the CVBT metric (see Eq. (3)).
As shown in Fig. 2, since $\rho_{k}^{(l)}$ of target class(es) is likely to be larger , while those of clean classes are likely to be small, we can adopt the anomaly detection technique to identify target class(es), such as the absolute robust Z-score [] method.
Specifically,
we can calculate the Z-score of $\rho_{k}^{(l)}$, as follows:

$$ $z_{k}^{(l)}=\frac{\rho_{k}^{(l)}-\tilde{\rho}^{(l)}}{\gamma\times\operatorname {MAD}(\{\rho_{k}^{(l)}\}_{k=1}^{K})},$ (5) $$

where $\operatorname{MAD}(\{\rho_{k}^{(l)}\})=\operatorname{median}(\{|\rho_{k}^{(l)}
-\tilde{\rho}^{(l)}|\})$ indicates
the median-absolute-deviation (MAD), and $\tilde{\rho}^{(l)}$ denotes the median value of $\{\rho_{k}^{(l)}\}_{k=1}^{K}$. $\gamma$ is a statistical constant valued at 1.4826.
Larger $z_{k}^{(l)}$ indicates larger likelihood of anomaly.
We firstly choose the layer with the largest Z-score, i.e., $l^{*}=\arg\max_{l}(\max_{k}z_{k}^{(l)})$. Then, if $z_{k}^{(l^{*})}$ exceeds some threshold, i.e., $z_{k}^{(l^{*})}\geq\tau_{z}$, $k$ is identified as a target class, otherwise clean class. $\tau_{z}$ will be specified in later experiments.

#### Stage 3: Filtering out poisoned samples within the identified target class(es).

Inspired by the second observation mentioned in Section 1 that the poisoned sample is likely to be far from the clean sample in GCD, here develop a novel algorithm which gradually filters out poisoned samples with the identified target class(es).
Specifically, as illustrated in Fig. 2, when obtained the identified target class $k^{*}$, we firstly pick one clean sample pair of class $k^{*}$ from $\mathcal{D}_{ac}$ as the reference $(\bm{x}_{0},k^{*})$, then we conduct the following three steps iteratively, until a stopping criteria is satisfied:

- 1.
For the set $\mathcal{D}_{bd}^{k^{*}}$, we calculate its GCD according to Definition 2, i.e., $\mathcal{P}_{k^{*}}^{(l^{*})}$;
- 2.
We calculate sample-level closeness value $s_{\bm{x}_{0}}(\bm{x}_{i})$ for each $\bm{x}_{i}\in\mathcal{D}_{bd}^{k^{*}}$;
- 3.
If the closeness value of one sample is lower than the threshold $\tau_{s}$ (specified in later experiments), i.e., $s_{\bm{x}_{0}}(\bm{x}_{i})<\tau_{s}$, then this sample is identified as poisoned, as it is far from the clean reference sample $\bm{x}_{0}$. Then, $\mathcal{D}_{bd}^{k^{*}}$ is updated by removing these identified poisoned samples.

In terms of the stopping criteria, we propose to firstly conduct the above iterations until $\mathcal{D}_{bd}^{k^{*}}$ becomes a null set. At each iteration, we calculate the distribution of $\{s_{\bm{x}_{0}}(\bm{x}_{i})\}_{\bm{x}_{i}\in\mathcal{D}_{bd}^{k^{*}}}$, and the Jensen–Shannon (JS) divergence between the current and its previous distribution. Then, we adopt a trace-back strategy by checking the JS divergence value of all iterations, and the iteration that its JS divergence locates at the stable and low region could be set as the stopping iteration.
Due to the space limit, more details of the whole algorithm, as well as the stopping criteria, will be presented in Appendix E.

## 5 Experiments

### 5.1 Experimental setup

#### Attack settings.

To evaluate the performance of our detection method, we conduct 10 state-of-the-art (SOTA) backdoor attacks that cover 4 categories: 1) non-clean label with sample-agnostic trigger, such as BadNets [], Blended [], LF []; 2) clean-label with sample-agnostic trigger, like SIG []; 3) clean-label with sample-specific trigger, such as CTRL [], an attack based on self-supervised learning; and 4) non-clean label with sample-specific trigger, including SSBA [], WaNet [], Input-Aware [], TaCT [], and Adap-Blend []. These attack settings follow BackdoorBench [] for a fair comparison. The poisoning ratio in our main evaluation is 10% for non-clean label attacks and 5% for clean label attacks. The target label $t$ is set to 0 0 for all-to-one backdoor attack, while target labels are set to $t=(y+1)\mod K$ for all-to-all backdoor attack.

#### Detection settings.

We compare AGPD with eight detection methods, categorized into three groups: 1) activation-based, including AC [], Beatrix [], SCAn [], and Spectral []; 2) input-based, such as STRIP [] and CD []; 3) loss-based, represented by ABL [] and ASSET []. In this work, we assume that defenders have access to the entire training dataset, while the model’s architecture and training process can be independently selected and implemented by the defenders themselves. Furthermore, it is presumed that defenders can gather a small, additional clean dataset to support the detection process, as suggested in previous works [][][]. For a fair comparison, we maintain that the number of clean samples per class is 10, extracted from the test dataset. The threshold used in AGPD $\tau_{z}$ and $\tau_{s}$ are $e^{2}$ and $0.05$, respectively.

#### Datasets and models.

We use CIFAR-10 [] and Tiny ImageNet [] as primary datasets to evaluate the detection performance. Additionally, we expand our evaluation to the datasets that are closer to real-world scenarios, such as ImageNet [] subset (200 classes), DTD [], and GTSRB [], of which results are provided in Appendix F. Our study employs two distinct model architectures: Preact-ResNet18 [] and VGG19-BN []. The results of VGG19-BN are provided in Appendix G.

#### Evaluation metrics.

In this work, the metrics evaluating the performance of backdoor attacks are Accuracy (ACC) and Attack Success Rate (ASR). The metrics used by the defender are True Positive Rate (TPR), False Positive Rate (FPR), and F1 score. In the tables presenting our results, the top performer is highlighted in bold, and the runner-up is marked with an underline.

**Table 1: The detection performance of AGPD and compared detectors on CIFAR-10 and Tiny ImageNet. The results are evaluated on Preact-ResNet18.**
| Dataset | Attack | Backdoored | AC [] | Beatrix [] | SCAn [] | Spectral [] | STRIP [] | ABL [] | CD [] | ASSET [] | AGPD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC/ASR | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ |  |  |
| CIFAR-10 | BadNets [] | 91.82/93.79 | 0.00 | 0.00 | 0.00 | 87.24 | 8.95 | 65.15 | 96.04 | 0.00 | 97.98 | 16.76 | 1.30 | 26.09 | 90.16 | 10.19 | 63.97 | 89.74 | 1.14 | 89.74 | 78.02 | 43.87 | 27.24 | 3.16 | 47.66 | 1.19 | 90.06 | 0.03 | 94.65 |
| Blended [] | 93.69/99.75 | 0.00 | 0.00 | 0.00 | 47.60 | 5.07 | 49.28 | 99.62 | 0.00 | 99.81 | 28.04 | 0.05 | 43.64 | 61.42 | 11.31 | 46.67 | 82.14 | 1.98 | 82.14 | 85.06 | 49.62 | 26.93 | 3.70 | 12.66 | 3.40 | 99.98 | 0.02 | 99.88 |  |
| LF [] | 93.01/99.05 | 0.00 | 0.00 | 0.00 | 0.00 | 10.72 | 0.00 | 95.58 | 0.01 | 97.71 | 0.04 | 3.16 | 0.06 | 86.92 | 10.09 | 62.59 | 45.48 | 6.06 | 45.48 | 88.44 | 24.33 | 43.42 | 3.80 | 10.28 | 3.87 | 99.80 | 0.07 | 99.60 |  |
| SSBA [] | 92.88/97.06 | 0.00 | 0.00 | 0.00 | 10.26 | 8.54 | 10.97 | 97.34 | 0.01 | 98.60 | 27.14 | 0.15 | 42.24 | 77.42 | 11.71 | 54.75 | 67.38 | 3.62 | 67.38 | 91.30 | 3.76 | 81.12 | 3.56 | 46.36 | 1.37 | 99.62 | 0.04 | 99.63 |  |
| SIG [] | 93.40/95.43 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 99.52 | 0.00 | 99.74 | 0.00 | 1.58 | 0.00 | 99.44 | 9.68 | 51.88 | 90.80 | 5.75 | 60.53 | 85.88 | 21.28 | 45.51 | 0.48 | 32.73 | 0.13 | 100.0 | 0.04 | 99.66 |  |
| CTRL | 95.52/98.8 | 0.00 | 9.92 | 0.00 | 0.00 | 6.26 | 0.00 | 0.00 | 5.00 | 0.00 | 0.40 | 15.77 | 0.20 | 99.80 | 9.47 | 52.57 | 90.32 | 5.77 | 60.21 | 99.44 | 0.55 | 97.31 | 67.88 | 51.60 | 11.82 | 99.76 | 0.01 | 99.78 |  |
| WaNet [] | 89.68/96.94 | 0.00 | 2.77 | 0.00 | 0.92 | 9.74 | 0.94 | 87.39 | 0.07 | 92.96 | 0.90 | 2.97 | 1.38 | 1.22 | 9.13 | 1.28 | 18.92 | 9.08 | 18.31 | 86.88 | 79.77 | 19.20 | 0.55 | 1.22 | 1.01 | 97.80 | 0.31 | 97.40 |  |
| Input-Aware [] | 90.82/98.17 | 0.00 | 3.31 | 0.00 | 0.41 | 10.92 | 0.39 | 99.15 | 0.47 | 97.36 | 1.51 | 2.90 | 2.34 | 0.81 | 9.07 | 0.86 | 0.17 | 11.02 | 0.17 | 82.85 | 18.99 | 46.84 | 2.82 | 60.13 | 0.90 | 88.25 | 1.13 | 88.61 |  |
| TaCT [] | 93.21/95.95 | 0.00 | 0.00 | 0.00 | 75.94 | 19.98 | 42.69 | 100.0 | 0.00 | 99.99 | 29.76 | 0.03 | 45.78 | 67.60 | 8.59 | 55.20 | 33.80 | 7.36 | 33.80 | 80.52 | 53.90 | 24.19 | 7.74 | 59.90 | 2.64 | 100.0 | 0.07 | 99.68 |  |
| Adap-Blend [] | 92.87/66.17 | 0.00 | 0.00 | 0.00 | 4.62 | 8.33 | 5.14 | 99.16 | 1.15 | 94.66 | 24.34 | 0.47 | 37.85 | 14.66 | 11.82 | 13.27 | 0.08 | 11.10 | 0.00 | 0.00 | 0.00 | 0.00 | 97.22 | 39.52 | 37.88 | 89.32 | 0.33 | 92.89 |  |
| BadNets-A2A | 91.93/74.40 | 28.76 | 4.51 | 33.96 | 40.28 | 9.25 | 36.04 | 0.00 | 0.00 | 0.00 | 0.00 | 1.67 | 0.00 | 1.80 | 17.08 | 1.41 | 2.48 | 10.84 | 2.48 | 67.20 | 45.72 | 23.23 | 3.32 | 3.31 | 4.99 | 97.32 | 0.02 | 98.57 |  |
| SSBA-A2A | 93.46/87.84 | 50.02 | 2.66 | 57.51 | 19.04 | 5.26 | 22.88 | 0.00 | 0.00 | 0.00 | 0.00 | 1.67 | 0.00 | 12.74 | 9.87 | 12.64 | 0.08 | 11.00 | 0.96 | 75.54 | 44.43 | 26.26 | 48.74 | 49.56 | 16.39 | 98.06 | 0.02 | 98.95 |  |
| Avg. |  | 6.57 | 1.93 | 7.62 | 23.86 | 8.58 | 19.46 | 72.82 | 0.56 | 73.23 | 10.74 | 2.64 | 16.63 | 51.17 | 10.67 | 34.76 | 43.45 | 7.06 | 38.43 | 76.76 | 32.18 | 38.44 | 20.25 | 34.58 | 7.13 | 96.66 | 0.17 | 97.44 |  |
| Tiny ImageNet | BadNets [] | 56.12/99.90 | 0.00 | 0.40 | 0.00 | 1.11 | 9.64 | 1.18 | 100.0 | 0.00 | 100.0 | 14.04 | 0.18 | 24.28 | 100.0 | 11.51 | 65.89 | 95.49 | 0.50 | 95.49 | 66.91 | 51.31 | 21.28 | 95.11 | 38.62 | 35.05 | 99.90 | 0.16 | 99.24 |
| Blended [] | 55.53/97.57 | 0.00 | 1.26 | 0.00 | 0.53 | 10.06 | 0.55 | 99.85 | 0.00 | 99.92 | 11.45 | 0.47 | 19.80 | 96.51 | 11.89 | 63.59 | 90.18 | 1.09 | 90.18 | 93.29 | 9.94 | 66.00 | 78.53 | 60.74 | 21.66 | 100.0 | 0.05 | 99.78 |  |
| LF [] | 55.21/98.51 | 15.05 | 1.26 | 23.82 | 19.36 | 9.15 | 19.20 | 63.86 | 0.00 | 77.94 | 11.35 | 0.48 | 19.62 | 85.97 | 9.72 | 62.88 | 87.44 | 1.40 | 87.44 | 95.22 | 6.65 | 74.65 | 54.28 | 50.69 | 17.78 | 100.0 | 0.10 | 99.56 |  |
| SSBA [] | 55.97/97.69 | 0.00 | 1.05 | 0.00 | 0.45 | 8.70 | 0.50 | 59.11 | 0.00 | 74.30 | 13.97 | 0.19 | 24.15 | 99.96 | 11.20 | 66.46 | 95.24 | 0.53 | 95.24 | 88.07 | 20.94 | 46.78 | 26.90 | 57.36 | 8.37 | 99.89 | 0.04 | 99.75 |  |
| WaNet [] | 58.33/90.35 | 13.73 | 0.80 | 22.60 | 75.94 | 11.88 | 52.23 | 62.72 | 0.00 | 77.09 | 11.37 | 0.45 | 19.65 | 6.38 | 11.11 | 5.97 | 94.40 | 1.27 | 91.35 | 76.73 | 20.18 | 42.83 | 0.00 | 15.12 | 0.00 | 99.77 | 0.12 | 99.30 |  |
| Input-Aware [] | 57.5/99.75 | 0.00 | 0.68 | 0.00 | 64.97 | 10.30 | 49.13 | 99.65 | 0.00 | 99.82 | 12.92 | 0.29 | 22.32 | 12.02 | 11.08 | 10.97 | 72.52 | 3.53 | 70.18 | 80.22 | 28.93 | 36.41 | 3.07 | 5.97 | 3.97 | 99.89 | 0.04 | 99.73 |  |
| TaCT [] | 54.93/91.25 | 0.00 | 1.02 | 0.00 | 45.51 | 10.13 | 38.45 | 100.0 | 0.00 | 100.0 | 15.48 | 0.03 | 26.75 | 80.03 | 16.37 | 48.90 | 32.25 | 7.53 | 32.25 | 99.58 | 99.45 | 18.19 | 35.77 | 56.31 | 12.20 | 99.99 | 0.43 | 98.11 |  |
| Adap-Blend [] | 54.55/96.35 | 0.00 | 0.82 | 0.00 | 9.56 | 9.39 | 9.85 | 47.24 | 0.05 | 63.98 | 15.14 | 0.06 | 26.18 | 77.73 | 15.85 | 48.52 | 8.97 | 10.11 | 8.97 | 73.69 | 39.65 | 27.77 | 71.95 | 49.92 | 25.19 | 99.96 | 0.17 | 99.24 |  |
| Avg. |  | 3.60 | 0.91 | 5.80 | 27.18 | 9.91 | 21.39 | 79.05 | 0.01 | 86.63 | 13.21 | 0.27 | 22.84 | 69.83 | 12.34 | 46.65 | 72.06 | 3.24 | 71.39 | 84.21 | 34.63 | 41.74 | 45.70 | 41.84 | 15.53 | 99.92 | 0.14 | 99.34 |  |

### 5.2 Detection effectiveness evaluation

#### All-to-one & all-to-all attacks.

Tab. 1 showcases the detection performance of AGPD with eight compared methods against 12 backdoor attacks on Preact-ResNet18. For all-to-one attacks and all-to-all attacks, AGPD can achieve averaged TPR of 96.66% on the CIFAR-10 and 99.92% on the Tiny ImageNet, exceeding the runner-up by 18.23% and 11.8% respectively.

The averaged FPR of AGPD not only ranks within the top-2 lowest among all detection methods but also approaches a near $0\%$ level.
Meanwhile, its average F1 score is 12.71% higher than that of the second-best method.

For the activation-based methods, like Beatrix, SCAn and Spectral, we find that they effectively identifies the majority of poisoned samples in attacks where poisoned and clean samples are separated in the activation space. However, its performance deteriorates when this separation is not present, such as CTRL (see t-SNE results in Appendix K). The failure of AC could be caused by the high poisoning ratio. For input-based method like STRIP, they exhibit low TPRs in attacks such as WaNet and Input-Aware. This underperformance is likely because the perturbed inputs generated by a poisoned sample also display high entropy in their predictions, similar to those of a clean sample, thereby complicating the distinction between them. CD shows relatively good detection effectiveness across most attacks with an average TPR of 76.76%, although it also suffers from higher FPRs. The reason could be that the masks derived from cognitive distillation for poisoned and clean samples are too similar under $L_{1}$ norm, leading to misclassification of some clean samples as poisoned. For loss-based method like ABL, they perform well in attacks with attacks such as BadNets, Blended, SIG, and CTRL. However, their effectiveness decreases when facing attacks with dynamic triggers, such as WaNet, Input-Aware, and Adap-Blend. These attacks require more training epochs for models to learn the connection between trigger and target label, which means that the loss of poisoned samples does not significantly decrease in the early epochs []. Regarding the ASSET method, we observed potential impacts on detection performance due to differences in the model used compared to the original work. Thus, we provide the results of ASSET on ResNet18 in Appendix J.

**Table 2: The detection performance of AGPD and the compared methods against multi-target attacks on CIFAR-10. The model structure is Preact-ResNet18.**
| Attack | Backdoored | AC [] | Beatrix [] | SCAn [] | Spectral [] | STRIP [] | ABL [] | CD [] | ASSET [] | AGPD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC/ASR | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ |  |
| Single-trigger |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BadNets | 91.38/80.34 | 97.46 | 0.70 | 95.64 | 19.38 | 0.96 | 30.28 | 0.00 | 0.00 | 0.00 | 5.42 | 16.06 | 4.34 | 2.78 | 13.01 | 2.53 | 1.22 | 10.98 | 1.22 | 47.42 | 31.62 | 21.95 | 22.84 | 20.79 | 14.74 | 98.50 | 0.08 | 98.89 |
| Blended | 93.57/91.60 | 79.72 | 0.25 | 87.61 | 11.76 | 4.34 | 15.59 | 0.00 | 0.00 | 0.00 | 14.90 | 15.01 | 11.92 | 0.18 | 6.49 | 0.23 | 2.08 | 10.88 | 2.08 | 74.66 | 55.91 | 22.03 | 19.64 | 8.70 | 19.85 | 98.48 | 0.01 | 99.20 |
| LF | 93.54/93.82 | 99.72 | 0.03 | 99.71 | 20.40 | 2.33 | 28.86 | 0.00 | 0.00 | 0.00 | 33.78 | 12.91 | 27.02 | 5.32 | 7.88 | 6.04 | 1.82 | 10.91 | 1.82 | 62.02 | 41.63 | 23.11 | 0.24 | 0.66 | 0.45 | 99.20 | 0.01 | 99.55 |
| SSBA | 93.28/92.38 | 99.44 | 0.04 | 99.52 | 4.50 | 0.79 | 8.07 | 0.00 | 0.00 | 0.00 | 32.36 | 13.07 | 25.89 | 48.26 | 15.65 | 33.39 | 11.48 | 9.84 | 11.48 | 72.56 | 25.16 | 36.37 | 13.52 | 8.02 | 14.56 | 98.92 | 0.02 | 99.36 |
| Multi-triggers |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BadNets+Blended+LF+SSBA+SIG | 91.62/92.10 | 58.06 | 0.08 | 73.15 | 2.92 | 2.60 | 4.62 | 0.00 | 0.00 | 0.00 | 14.64 | 15.05 | 11.71 | 22.72 | 13.27 | 18.77 | 50.10 | 5.54 | 50.10 | 63.20 | 18.60 | 38.23 | 56.86 | 54.68 | 17.52 | 92.02 | 0.18 | 95.06 |
| Avg. |  | 86.88 | 0.22 | 91.13 | 11.79 | 2.20 | 17.48 | 0.00 | 0.00 | 0.00 | 20.22 | 14.42 | 16.18 | 15.85 | 11.26 | 12.19 | 13.34 | 9.63 | 13.34 | 63.97 | 34.58 | 28.34 | 22.62 | 18.57 | 13.42 | 97.42 | 0.06 | 98.41 |

#### Multi-target attacks.

Tab. 2 summarizes the performance of AGPD and the compared methods in a multi-target attack scenario. In our experiment setting, $\{5,6,7,8,9\}$ are chosen as the source class and the target labels are set to $t=(y+C)\mod K$, where $C$ equals 5.

Single-trigger attack and multi-trigger attack are two categories of multi-target attack. In single-trigger attacks, the same trigger injected into samples from different source classes is classified into their corresponding target classes. In the multi-trigger attack, we use five triggers from different backdoor attacks (i.e., BadNets, Blended, LF, SSBA, and SIG), and each trigger added to the samples in the corresponding class will be classified into its designated target class. We observed that for activation-based methods, the multi-trigger attack poses a greater challenge than single-trigger attacks, whereas loss-based methods seem more robust against multi-trigger attacks. Additionally, the failure of SCAn might be caused by their anomaly detection for target class(es) is not effective in the multi-target attacks. However, compared with baseline methods, AGPD achieves good performance in both single-trigger and multi-trigger attacks, with averages of TPR, FPR, and F1 score at 97.42%, 0.06%, and 98.41%, respectively.

Figure: Figure 3: Detection performance of AGPD and the compared detectors with poisoing ratios ranging from $0.5\%$ to $10\%$.
Refer to caption: extracted/5625055/figure/poisoning_ratio_preact.png

#### Different poisoning ratios.

We evaluated the detection effectiveness of AGPD and four other methods, chosen from activation-based, input-based, and loss-based detection methods, under varying poisoning ratios. There are four poisoning ratios used: $\{0.5\%,1\%,5\%,10\%\}$, covering a range from low to high poisoning ratios. As illustrated in Fig. 3, the performance of most detectors is notably influenced by the poisoning ratio. Particularly, a low poisoning ratio (e.g., 0.5%) presents a substantial challenge for most detectors, with the TPRs of AC and SCAn almost nearing zero. However, our method can achieve good performance in this situation, with TPR around 90% and FPR lower than that of other detectors under most attacks. And with the poisoning ratio increased, the performance of our method is still stable.

### 5.3 Analysis

#### Activation vs. activation gradient.

**Table 3: Silhouette scores of the target class under eight attacks, measured in activation and activation gradient spaces, using Preact-ResNet18.**
|  | BadNets | Blended | LF | SSBA |
| --- | --- | --- | --- | --- |
| Activation | 0.529 | 0.485 | 0.472 | 0.497 |
| Activation Gradient | 0.664 | 0.696 | 0.610 | 0.623 |
|  | WaNet | Input-Aware | TaCT | Adap-Blend |
| Activation | 0.544 | 0.457 | 0.379 | 0.403 |
| Activation Gradient | 0.605 | 0.466 | 0.515 | 0.491 |

To demonstrate that the separations of clean and poisoned samples differ in activation gradient and activation spaces, we show the distribution of cosine similarities between samples and the clean basis from both spaces. Due to the space limitation, we provide the results in Appendix I. To quantify the separation, we utilize Silhouette Score [] to measure the distance between two clusters in both spaces across all convolutional layers. The range of the Silhouette Score is between -1 and 1, with higher values indicating better separability between the two clusters.

Considering that different depths of convolutional layers correspond to different separations, Tab. 3 presents the maximum Silhouette Scores among all convolutional layers for eight backdoor attacks, from which we find that Silhouette Scores are larger in activation gradient space.

#### Statistic of metrics.

In Fig. 4, we present the statistical results of CVBT metric ($\rho$) and its Z-score ($z$) for both all-to-one and all-to-all attacks. In the left of Fig. 4(a), $\rho$ of the target class are significantly higher than those for clean classes for both attacks.
Since clean classes are absent in all-to-all attacks, using $z$ to detect the outliers for identifying the target class could be ineffective. Therefore, if the value of $z$ fails to identify the target class, we utilize the threshold defined by the boundary, which serves as a boundary between the target and clean classes, to determine if a class is the target class. It is set to 0.3 and depicted as a dashed line in Fig. 4
This is validated in Fig. 4(a) right image, where $\rho$ values for the target classes in all-to-all attacks exceed our boundary. Fig. 4(b) displays the distribution of $z$ for the target and clean classes in the all-to-one attacks, clearly separated by the dashed line, which represents the threshold when identifying the target class. Moreover, Fig. 4(c) illustrates the mean and standard deviation curves of $z$ of target classes across different convolutional layers of the model. We observe that $z$ tend to be higher in the later intermediate convolutional layers, indicating a stronger separation activation gradients between poisoned and clean samples at these layers.

Figure: (a)
Refer to caption: extracted/5625055/figure/cvbt.png

#### Accuracy of target label identification.

We compare the accuracy of target class identification of AGPD with other three detection methods which are Beatrix [], SCAn [], and NC []. To evaluate their performance, we trained 120 backdoor models on CIFAR-10. The attack methods contain 8 non-clean label backdoor attacks, where the poisoning ratio ranges from $1\%$ to $10\%$, and the target label is from 0 to 4. The results of detection accuracy are shown in  Fig. 5(a). Note that the accuracy of target class identification of AGPD is higher than the compared method under different poisoning ratios. Even though SCAn has a competitive performance with AGPD, the accuracy of SCAn is lower than 20% with 1% poisoning ratio.

### 5.4 Sensitivity test

#### Influence of number of clean samples.

In this part, we explore the influence of the size of the additional clean dataset on the detection performance of AGPD. We also consider the scenario that the additional dataset collected by the defender is out of distribution (OOD).

We collect the OOD dataset of CIFAR-10 from the same 10 classes of CIFAR-5m [], and we extract 10 samples from each class.
The additional clean dataset which is in distribution (ID) is collected from the test dataset.  Fig. 5(b) shows the results of our method with different sizes of the additional clean dataset.
We found that a large number of clean samples can help AGPD decrease FPR close to zero. However, AGPD can still achieve high TPR even in extreme cases, such as one sample per class or even OOD samples.

When the number of clean samples in each class is one, the TPR values of AGPD on many attacks are above 90%. In summary, our method necessitates a smaller additional clean dataset.

Figure: (a)
Refer to caption: extracted/5625055/figure/target_acc.png

#### Influence of threshold τ s subscript 𝜏 𝑠 \tau_{s} italic_τ start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT .

In the poisoned sample filtering stage, we aim to eliminate samples scoring below $\tau_{s}$ at each iteration until no samples remain in the target class. To better understand the impact of the threshold value $\tau_{s}$, we designed an experiment where we varied $\tau_{s}$ from 0.01 to 0.1 and recorded their TPRs and FPRs. According to Fig. 5(c), the TPR of AGPD is relatively low with significant variability at smaller $\tau_{s}$ values, yet it stabilizes at $100\%$ with increasing tau, while the FPR remains consistently low throughout the variation of tau.
Moreover, it can ensure stable detection performance of AGPD across a broad range of values.

## 6 Conclusion

In this paper, we introduce a novel perspective of gradient circular distribution (GCD). Based on GCD, we observe that the dispersion of GCD of target class is larger, and poisoned samples are separated from clean ones. Inspired by the observation, we propose two practical metrics and design a novel detection method, AGPD. Our experiments demonstrate that this method successfully identifies target class(es) under various backdoor attack scenarios, including all-to-one, all-to-all, and multi-target attacks. Extensive experimental results show that our method achieves good performance on the task of poisoned sample detection.

#### Limitation and future work.

The proposed method involves scanning through all layers of the model to identify the one where poisoned and clean samples are most distinctly separated. The computational cost may be high if applying it on large scale datasets, which may require training large models with multiple layers to extract activation gradients.
However, as analyzed in Fig. 4-right, the activation gradient directions of poisoned and clean samples typically exhibit greater separation in higher layers of the model.
Inspired, in future we plan to narrow the range of selected layers to further enhance the efficiency of the proposed method.

## Appendix A In summary

There are additional materials presented in the Appendix, including: (1) details of datasets and models we used; (2) hyperparameter settings in model training; (3) computational overhead of the proposed method and the compared methods; (4) Details of algorithm and stopping criteria; (5) performance of AGPD on more datasets; (6) separation of poisoned and clean samples in activation and activation gradient space; (7) more GCDs examples; (8) The results of the compared ASSET on ResNet18. (9) t-SNE results of ten attacks on Preact-ResNet18.

## Appendix B Datasets and models

We evaluate the performance of AGPD on five popular datasets and two model structures. In main paper, we have provide the results of two datasets, including CIFAR-10 [] and Tiny ImageNet []. Besides, we extend our evaluations to the datasets which are closer to real-world scenarios, such as ImageNet(subset)-200 [], the Textures dataset DTD [], and the traffic signs dataset GTSRB []. The results of the these datasets are provided in Appendix F. The details of all datasets are illustrated in Tab. 4 and Tab. 5.

**Table 4: The information about five datasets.**
| Dataset | Categories | Image size | Training samples | Testing samples |
| --- | --- | --- | --- | --- |
| CIFAR-10 | 10 | $32\times 32$ | 50,000 | 10,000 |
| Tiny ImageNet | 200 | $64\times 64$ | 90,000 | 10,000 |
| ImageNet(subset200) | 200 | $224\times 224$ | 90,000 | 10,000 |
| GTSRB | 43 | $32\times 32$ | 39,209 | 12,630 |
| DTD | 47 | $224\times 224$ | 3,760 | 1,880 |

**Table 5: Introduction of PreAct-ResNet18 and VGG19-BN Architectures**
| Layer Name | Output Size | PreAct-ResNet18 | VGG19-BN |
| --- | --- | --- | --- |
| Conv1 | 112x112 | 7x7, 64, stride 2 | 3x3, 64 |
| Max Pooling | 56x56 | 3x3, stride 2 | 2x2, stride 2 |
| Residual Block 1 | 56x56 | Pre-activation: BN, ReLU<br>3x3, 64<br>Pre-activation: BN, ReLU<br>3x3, 64 | 3x3, 64<br>3x3, 64 |
| Residual Block 2 | 28x28 | Pre-activation: BN, ReLU<br>3x3, 128, stride 2<br>Pre-activation: BN, ReLU<br>3x3, 128 | 3x3, 128<br>3x3, 128 |
| Residual Block 3 | 14x14 | Pre-activation: BN, ReLU<br>3x3, 256, stride 2<br>Pre-activation: BN, ReLU<br>3x3, 256 | 3x3, 256<br>3x3, 256<br>3x3, 256<br>3x3, 256 |
| Residual Block 4 | 7x7 | Pre-activation: BN, ReLU<br>3x3, 512, stride 2<br>Pre-activation: BN, ReLU<br>3x3, 512 | 3x3, 512<br>3x3, 512<br>3x3, 512<br>3x3, 512 |
| Global Average Pooling | 1x1 | Global Average Pooling |  |
| Fully Connected | 1x1 | 1000-d | 4096 |
| Fully Connected | 1x1 |  | 4096 |
| Fully Connected | 1x1 |  | 1000-d |
| Softmax | 1x1 | Softmax |  |

## Appendix C Hyperparameter settings

In this part, we will introduce the hyperparameter settings of model training and backdoor attacks.

#### Model training.

There are some common training hyperparameters across these attack methods, such as training epoch, learning rate, and optimizer. We display the setting of these common hyperparameters for each datasets in Tab. 6.

**Table 6: The common hyperparameters for training across five datasets.**
| Dataset | Epoch | Learning rate | Batch size | Optimizer |
| --- | --- | --- | --- | --- |
| CIFAR-10 | 100 | 0.01 | 128 | SGD |
| Tiny ImageNet | 200 | 0.01 | 128 | SGD |
| ImageNet(subset200) | 200 | 0.1 | 64 | Adam |
| GTSRB | 50 | 0.01 | 128 | SGD |
| DTD | 100 | 0.01 | 64 | SGD |

#### Backdoor attacks.

The hyperparameters used in various backdoor attacks are listed in Tab. 7. For illustration purposes, we use CIFAR-10 as an example. If the attack does not have any specific hyper-parameters, we will denote this with ‘/’. We show the poisoned samples of various backdoor attacks in Fig. 6.

**Table 7: The hyper-parameters of implemented backdoor attacks for CIFAR-10.**
|  | Category | Attack | Parameters | Usage | Value |
| --- | --- | --- | --- | --- | --- |
| All-to-one | non-clean label with sample-agnostic trigger | BadNets | / | / | / |
| Blended | $\alpha$ | the transparency of the trigger. | 0.2 |  |  |
| LF | $\alpha$ | fooling rate | 0.2 |  |  |
| clean label with sample-agnostic trigger | SIG | $\Delta$ | to generate sinusoidal signal. | 40 |  |
| $f$ | 6 |  |  |  |  |
| clean label with sample-specific trigger | CTRL | $c$ | trigger channel | [2,1] |  |
|  | $l$ | trigger location | (12,27) |  |  |
| non-clean label with sample-specific trigger | SSBA | / | / | / |  |
| TaCT | $s-class$ | the trigger will be added to samples in $s-class$ and change their labels to the target label. | a list |  |  |
| $c-class$ | samples in $c-class$ will only be added the trigger. | a list |  |  |  |
| $c$ | control the number of samples in $c-class$ | 0.1 |  |  |  |
| Adap-Blend | $m$ | the probability of the area being masked. | 0.5 |  |  |
| non-clean label with training control | WaNet | $s$ | warping strength | 0.5 |  |
| $k$ | grid scale | 4 |  |  |  |
| $\rho_{a}$ | backdoor probability | =poisoning ratio |  |  |  |
| $\rho_{n}$ | the noise probability | 0.1 |  |  |  |
| Input-Aware | $\lambda_{div}$ | the diversity enforcement regularisation. | 1 |  |  |
|  | $\rho_{b}$ | the backdoor probability. | =poisoning ratio |  |  |
|  | $\rho_{c}$ | the cross-trigger probability. | 0.1 |  |  |
| All-to-all | non-clean label with sample-agnostic trigger | BadNets-A2A | $K$ | to compute target labels. | 10 |
| non-clean label with sample-specific trigger | SSBA-A2A |  |  |  |  |
| Single-trigger Attack | non-clean label with sample-agnostic trigger | BadNets | $C$ | to compute target labels | 5 |
| Blended |  |  |  |  |  |
| LF |  |  |  |  |  |
| non-clean label with sample-specific trigger | SSBA |  |  |  |  |
| Multi-trigger Attack | non-clean label with sample-agnostic trigger | BadNets+Blended+SSBA+ LF+ SIG | $r$ | the ratio of poisoned samples from each type of trigger | 0.02 |

Figure: Figure 6: Examples of poisoned samples in various backdoor attacks.
Refer to caption: extracted/5625055/figure/appendix/poisoned_samples.png

## Appendix D Computation overhead

Tab. 8 illustrates the computation overhead (RTX A5000 GPU) of AGPD and the compared detection method under eight backdoor attacks with 10% poisoning ratio on CIFAR-10. We record the average of running time with standard deviation in bracket.

**Table 8: Computation overhead of AGPD and the compared methods on CIFAR-10. The value in the bracket represent standard deviation.**
|  | AC | Beatrix | SCAn | Spectral | STRIP | ABL | CD | ASSET | AGPD |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time (minute) | 1.02(0.01) | 8.92(0.38) | 1.27(0.02) | 1.73(0.06) | 3.29( 0.02) | 10.06(0.09) | 20.42(0.02) | 206.43 (12.63) | 5.02(0.03) |

## Appendix E Details of algorithm and stopping criteria

### E.1 The description of Algorithm

The statement of the algorithm we used in Stage3 is described in Algorithm 1. An example of JS divergence across iterations is provided in Fig. 7. Assuming ground truth is known for samples in the target class, we can obtain True Positives (TP) and False Positives (FP) for each iteration. It can be observed that an optimal iteration exists where JS divergence is minimal and stabilizes. The rationale behind the trends in JS divergence is that in the early stages of filtering, the far-end basis is primarily updated by genuinely poisoned samples, effectively guiding the identification of such samples. As the process progresses into the middle stages, most poisoned samples have been filtered out, and the influence of the far-end basis on the remaining clean samples becomes minimal, resulting in little change in trust scores and small JS divergence. However, as the process extends into later stages, and more clean samples are inevitably filtered, the far-end basis is updated by these clean samples, leading to significant changes in the distribution of trustworthiness scores and an increase in JS divergence.

Figure: Figure 7: Trends of TP, FP, and JS divergence according to the iteration $t$.
Refer to caption: extracted/5625055/figure/JS.png

Figure: Algorithm 1 Filtering out poisoned samples within the identified target class(es).

### E.2 The stopping criteria

To find an appropriate stopping iteration $t^{*}$, we utilize the sliding window method to analyze the changes in JS divergence across all iterations. Our goal is to identify the iteration $t$ where the JS divergence is minimal and stabilizes. Let the width of the window be $w$, and let the JS divergence at each iteration $t$ be $JS(t)$, where $t\in\{0,\dots,T-1\}$. The average $\mu_{m}$ and standard deviation $\sigma_{m}$ of each window starting at position $m$ are defined as follows:

$$ $\left\{\begin{aligned} \mu_{m}&=\frac{1}{w}\sum_{j=0}^{w-1}JS(m+j),\\ \sigma_{m}&=\sqrt{\frac{1}{w}\sum_{j=0}^{w-1}\left(JS(m+j)-\mu_{m}\right)^{2}} .\end{aligned}\right.$ (6) $$

Each window can be represented by a score $S_{m}$ that combines the value of the average with the standard deviation. Since we are mainly focusing on stabilization, we design a metric as described in Eq. 7, which amplifies the contribution of the standard deviation.

$$ $S_{m}=\mu_{m}+\beta\sigma_{m}.$ (7) $$

After computing the score of all windows, we choose the window starting at $m$ with the minimum score:

$$ $m^{*}=\arg\min_{m}S_{m}.$ (8) $$

The stopping iteration is the middle of the optimal window, which is $t^{*}=m^{*}+\frac{w}{2}$. In our method, we adopt $w=10$ and $\beta=5$.

## Appendix F Evaluations on more datasets

In this section, we present the results of AGPD and the compared methods on these datasets, including DTD [], GTSRB [], and ImageNet subset (200 classes) []. The datasets can be categorized into two types: balanced datasets (e.g., DTD and ImageNet subset (200 classes)) and imbalanced datasets (e.g., GTSRB). In balanced datasets, each category has the same number of samples. For example, DTD has 376 samples per class, and ImageNet-200 has 500 samples per class. In contrast, the number of samples per category in GTSRB varies from 210 to 2,250. Besides, We use Preact-ResNet18 [] as the model architecture when training on DTD and GTSRB, and adopt ResNet50 [] for ImageNet subset (200 classes). Specifically, we compare AGPD with four detection methods: activation-based method (e.g., SCAn []), input-based method (e.g., STRIP []), and loss-based methods (e.g., ABL [] and ASSET []) on DTD and GTSRB. The results are shown in Tab. 9. Besides, the results of AGPD on ImageNetsubset (200 classes), are displayed in Tab. 10.

**Table 9: The detection performance of AGPD and compared detectors on DTD and GTSRB. The results are evaluated on Preact-ResNet18.**
| Dataset | Attack | Backdoored | SCAn [] | STRIP [] | ABL [] | ASSET [] | AGPD |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC/ASR | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ |  |  |
| DTD | BadNets [] | 51.97/98.32 | 90.43 | 0.21 | 94.05 | 83.78 | 12.71 | 56.20 | 76.33 | 2.63 | 76.43 | 96.01 | 8.16 | 71.15 | 99.47 | 1.60 | 93.03 |
| Blended [] | 51.86/94.62 | 82.18 | 0.30 | 88.76 | 95.74 | 13.21 | 60.74 | 88.83 | 1.24 | 88.95 | 0.80 | 3.04 | 1.25 | 100.0 | 1.89 | 92.27 |  |
| WaNet [] | 42.71/26.41 | 86.08 | 1.06 | 88.01 | 77.84 | 14.03 | 51.14 | 0.28 | 11.00 | 0.27 | 1.70 | 2.33 | 2.61 | 100.0 | 1.88 | 92.27 |  |
| Input-Aware [] | 45.85/85.54 | 0.00 | 0.00 | 0.00 | 13.07 | 12.91 | 11.38 | 20.45 | 8.92 | 20.19 | 0.00 | 0.72 | 0.00 | 98.58 | 0.59 | 96.73 |  |
| Adap-Blend [] | 49.41/85.65 | 77.13 | 0.92 | 83.21 | 32.18 | 16.19 | 23.01 | 6.91 | 10.34 | 6.67 | 4.26 | 2.99 | 6.49 | 100.0 | 1.18 | 95.07 |  |
|  | Avg. |  | 67.16 | 0.50 | 70.81 | 60.52 | 13.81 | 40.50 | 38.56 | 6.83 | 38.50 | 20.55 | 3.45 | 16.30 | 99.61 | 1.43 | 93.88 |
| GTSRB | BadNets [] | 96.35/95.02 | 94.62 | 4.00 | 82.06 | 95.54 | 15.39 | 57.20 | 73.71 | 2.92 | 73.71 | 100.0 | 47.79 | 31.74 | 100.0 | 0.27 | 98.80 |
| Blended [] | 98.17/100.0 | 83.47 | 7.54 | 66.42 | 100.0 | 11.58 | 65.74 | 80.54 | 2.16 | 80.55 | 99.36 | 41.33 | 34.77 | 95.87 | 0.30 | 96.57 |  |
| WaNet [] | 97.05/96.16 | 60.83 | 0.00 | 75.63 | 7.83 | 14.34 | 6.59 | 0.00 | 11.03 | 0.00 | 44.07 | 64.79 | 12.12 | 100.0 | 0.20 | 99.12 |  |
| Input-Aware [] | 97.91/95.64 | 52.15 | 0.00 | 68.54 | 1.99 | 10.44 | 2.03 | 0.00 | 11.03 | 0.00 | 3.07 | 59.18 | 0.96 | 79.60 | 0.00 | 88.64 |  |
| Adap-Blend [] | 97.66/80.42 | 50.74 | 3.95 | 54.48 | 20.48 | 12.40 | 17.63 | 0.00 | 11.11 | 0.00 | 81.10 | 22.48 | 42.30 | 93.44 | 0.23 | 95.58 |  |
|  | Avg. |  | 68.36 | 3.10 | 69.43 | 45.17 | 12.83 | 29.84 | 30.85 | 7.65 | 30.85 | 65.52 | 47.12 | 24.38 | 93.78 | 0.20 | 95.74 |

**Table 10: The detection performance of AGPD on ImageNet-200. The results are evaluated on Preact-ResNet18.**
| Dataset | Attack | Backdoored | AGPD |  |  |
| --- | --- | --- | --- | --- | --- |
| ACC/ASR | TPR$\uparrow$ | FPR $\downarrow$ | F1 $\uparrow$ |  |  |
| ImageNet-200 | BadNet | 78.57/80.03 | 94.62 | 0.00 | 97.24 |
| Blended | 79.95/99.93 | 100.0 | 0.47 | 97.93 |  |
| Adap-Blend | 72.3/93.17 | 99.98 | 0.59 | 99.19 |  |
| Avg. |  | 98.20 | 0.35 | 98.12 |  |

As shown in Tables 9 and 10, AGPD achieves high performance on these datasets, with average TPRs of 99.61%, 93.78%, and 98.20%, respectively. Meanwhile, the average FPRs are 1.43%, 0.2%, and 0.35%. The experimental results demonstrate that our method is adaptable not only to balanced and imbalanced datasets but also to datasets with various image sizes.

Regarding the compared detection methods, we draw conclusions similar to those reported in our main paper. Specifically, SCAn fails when the distinction between poisoned and clean samples in the activation space is not obvious. STRIP struggles to effectively identify poisoned samples in the training dataset under attacks such as WaNet and Input-Aware. Similarly, ABL encounters challenges in achieving satisfactory detection performance against complex triggers, including those used in WaNet, Input-Aware, and Adap-Blend attacks.

## Appendix G Evaluations on VGG19-BN

### G.1 All-to-one & all-to-all attacks.

The results of VGG19-BN under all-to-one and all-to-all attacks are shown in Tab. 11. It can be seen that even changing the model architecture, the detection performance of our method is still stable, achieving 96.46% TPR on CIFAR-10 and 97.75% on Tiny ImageNet, higher than that of the second-best 16.55% and 7.89%, respectively. Besides, the F1 score of AGPD are 90.90% on CIFAR-10 and 98.36% on Tiny ImageNet, exceeding the runner-up by 33.59% and 16.27%, respectively. The results indicate the dispersion of the activation gradients of poisoned samples and clean samples could exist across model structures, demonstrating the robust adapt ability of our method.

For the compared methods, we found that activation-based methods are significantly influenced by the model architecture. For instance, AC can identify a small portion of poisoned samples in all-to-one attacks with a 10% poisoning ratio. Beatrix performs better on VGG19-BN, with the average TPR 20.78% higher and the F1 score 17.39% greater than on Preact-ResNet18. Conversely, SCAn struggles to detect a large number of poisoned samples in WaNet and Input-Aware attacks on the VGG19-BN.

**Table 11: The detection performance of AGPD and compared detectors on CIFAR-10 and Tiny ImageNet. The results are evaluated on VGG19-BN.**
| Dataset | Attack | Backdoored | AC [] | Beatrix [] | SCAn [] | Spectral [] | STRIP [] | ABL [] | CD [] | ASSET [] | AGPD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC/ASR | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ | TPR$\uparrow$ | FPR$\downarrow$ | F1$\uparrow$ |  |  |
| CIFAR-10 | BadNets [] | 91.82/93.79 | 0.00 | 6.42 | 0.00 | 52.90 | 3.44 | 57.55 | 68.90 | 0.35 | 80.08 | 28.28 | 0.02 | 44.02 | 82.88 | 10.84 | 59.10 | 76.68 | 2.59 | 76.68 | 72.98 | 13.46 | 49.62 | 4.54 | 2.33 | 7.24 | 99.72 | 6.90 | 76.16 |
| Blended [] | 93.69/99.75 | 0.00 | 14.79 | 0.00 | 99.22 | 9.97 | 68.68 | 96.72 | 0.00 | 98.33 | 28.50 | 0.00 | 44.36 | 47.06 | 10.74 | 38.61 | 78.08 | 2.44 | 78.08 | 99.94 | 99.87 | 18.19 | 0.20 | 52.35 | 0.07 | 99.96 | 0.02 | 99.90 |  |
| LF [] | 93.01/99.05 | 0.00 | 6.76 | 0.00 | 99.46 | 6.40 | 77.39 | 83.22 | 0.00 | 90.83 | 28.50 | 0.00 | 44.36 | 89.80 | 8.28 | 67.94 | 40.80 | 6.58 | 40.80 | 0.10 | 0.03 | 0.20 | 1.74 | 0.97 | 3.15 | 99.50 | 0.03 | 99.62 |  |
| SSBA [] | 92.88/97.06 | 4.74 | 14.08 | 4.10 | 0.10 | 6.19 | 0.13 | 89.76 | 0.80 | 91.15 | 2.18 | 2.92 | 3.39 | 72.38 | 12.33 | 51.08 | 25.16 | 8.32 | 25.16 | 84.78 | 60.67 | 23.20 | 0.00 | 0.00 | 0.00 | 98.94 | 4.81 | 81.69 |  |
| SIG [] | 93.40/95.43 | 0.00 | 13.84 | 0.00 | 100.0 | 3.56 | 74.70 | 94.80 | 0.00 | 97.33 | 30.00 | 0.00 | 46.15 | 98.96 | 6.20 | 62.47 | 94.84 | 5.53 | 63.23 | 88.24 | 48.16 | 28.39 | 100.0 | 47.35 | 18.19 | 99.60 | 0.02 | 99.62 |  |
| CTRL | 95.52/98.8 | 15.52 | 17.08 | 7.06 | 22.20 | 4.79 | 20.83 | 5.80 | 0.00 | 10.96 | 23.20 | 14.57 | 11.60 | 94.36 | 10.19 | 48.64 | 86.72 | 5.96 | 57.81 | 88.16 | 23.29 | 44.33 | 100.0 | 5.81 | 64.42 | 90.84 | 2.06 | 78.99 |  |
| WaNet [] | 89.68/96.94 | 0.00 | 20.48 | 0.00 | 2.45 | 9.64 | 2.51 | 0.00 | 0.00 | 0.00 | 29.03 | 0.06 | 44.81 | 1.92 | 12.02 | 1.76 | 13.18 | 9.67 | 12.76 | 98.85 | 98.54 | 18.21 | 0.00 | 38.57 | 0.00 | 85.35 | 0.09 | 91.67 |  |
| Input-Aware [] | 90.82/98.17 | 10.52 | 7.01 | 11.80 | 2.03 | 5.63 | 2.59 | 65.04 | 0.03 | 78.69 | 1.88 | 2.86 | 2.90 | 0.98 | 8.60 | 1.07 | 14.83 | 9.50 | 14.35 | 84.26 | 58.53 | 23.70 | 0.06 | 10.01 | 0.07 | 92.96 | 0.06 | 96.09 |  |
| TaCT [] | 93.21/95.95 | 0.00 | 6.46 | 0.00 | 97.24 | 6.79 | 75.29 | 99.22 | 0.00 | 99.61 | 30.00 | 0.00 | 46.15 | 75.30 | 14.19 | 49.70 | 26.94 | 8.12 | 26.94 | 83.42 | 74.00 | 19.64 | 0.42 | 0.66 | 0.79 | 100.0 | 0.25 | 98.90 |  |
| Adap-Blend [] | 92.87/66.17 | 4.68 | 16.07 | 3.75 | 0.22 | 7.21 | 0.27 | 29.68 | 1.80 | 40.68 | 1.36 | 3.03 | 2.12 | 7.34 | 11.31 | 7.02 | 0.02 | 11.11 | 0.02 | 100.0 | 100.0 | 18.18 | 0.00 | 0.01 | 0.00 | 99.92 | 8.17 | 73.07 |  |
| BadNets-A2A | 91.93/74.40 | 79.46 | 0.02 | 88.49 | 28.42 | 6.66 | 30.17 | 0.00 | 0.00 | 0.00 | 0.00 | 1.67 | 0.00 | 1.54 | 11.72 | 1.49 | 4.86 | 10.57 | 4.86 | 58.16 | 20.01 | 34.39 | 5.86 | 3.67 | 8.44 | 95.92 | 0.02 | 97.83 |  |
| SSBA-A2A | 93.46/87.84 | 99.36 | 0.93 | 95.66 | 31.44 | 7.17 | 32.08 | 0.00 | 0.00 | 0.00 | 0.00 | 1.67 | 0.00 | 19.98 | 16.14 | 15.07 | 1.18 | 10.98 | 1.18 | 99.98 | 99.96 | 18.18 | 0.24 | 0.11 | 0.47 | 94.82 | 0.02 | 97.27 |  |
| Avg. |  | 17.86 | 10.33 | 17.57 | 44.64 | 6.45 | 36.85 | 52.76 | 0.25 | 57.31 | 16.91 | 2.23 | 24.15 | 49.38 | 11.05 | 33.66 | 38.61 | 7.61 | 33.49 | 79.91 | 58.04 | 24.69 | 17.76 | 13.49 | 8.57 | 96.46 | 1.87 | 90.90 |  |
| Tiny ImageNet | BadNets [] | 56.12/99.90 | 0.10 | 20.08 | 0.07 | 97.73 | 9.37 | 69.30 | 99.88 | 0.00 | 99.94 | 15.67 | 0.00 | 27.09 | 99.99 | 11.24 | 66.41 | 96.66 | 0.37 | 96.66 | 96.43 | 9.16 | 69.15 | 100.0 | 48.50 | 31.42 | 99.67 | 0.12 | 99.30 |
| Blended [] | 55.53/97.57 | 0.00 | 7.47 | 0.00 | 92.85 | 4.40 | 79.90 | 78.26 | 0.00 | 87.80 | 15.67 | 0.00 | 27.09 | 95.55 | 13.88 | 59.63 | 96.78 | 0.36 | 96.78 | 82.38 | 34.19 | 33.62 | 1.32 | 4.65 | 1.84 | 99.99 | 0.03 | 99.85 |  |
| LF [] | 55.21/98.51 | 0.00 | 5.74 | 0.00 | 22.73 | 2.02 | 32.26 | 52.25 | 0.00 | 68.64 | 15.67 | 0.00 | 27.09 | 21.09 | 11.82 | 18.55 | 48.83 | 5.69 | 48.83 | 96.43 | 10.55 | 66.18 | 99.08 | 17.03 | 56.24 | 99.92 | 0.03 | 99.84 |  |
| SSBA [] | 55.97/97.69 | 0.00 | 8.88 | 0.00 | 42.53 | 3.96 | 47.73 | 78.05 | 0.00 | 87.67 | 15.67 | 0.00 | 27.09 | 99.92 | 11.77 | 65.33 | 89.65 | 1.15 | 89.65 | 93.11 | 19.00 | 51.14 | 100.0 | 47.66 | 31.80 | 99.94 | 0.02 | 99.86 |  |
| WaNet [] | 58.33/90.35 | 0.02 | 5.62 | 0.03 | 0.00 | 1.51 | 0.00 | 99.94 | 0.00 | 99.96 | 13.93 | 0.19 | 24.08 | 17.65 | 11.57 | 15.38 | 83.54 | 2.39 | 80.85 | 99.09 | 0.30 | 98.19 | 100.0 | 0.73 | 97.28 | 99.97 | 0.12 | 99.40 |  |
| Input-Aware [] | 57.5/99.75 | 0.05 | 5.64 | 0.07 | 0.53 | 2.90 | 0.83 | 98.11 | 0.00 | 99.05 | 15.72 | 0.00 | 27.17 | 15.49 | 12.35 | 13.19 | 64.97 | 4.31 | 62.87 | 95.88 | 8.36 | 70.73 | 97.18 | 15.79 | 58.19 | 82.54 | 0.00 | 90.43 |  |
| TaCT [] | 54.93/91.25 | 0.00 | 8.16 | 0.00 | 54.87 | 2.62 | 61.50 | 44.87 | 0.00 | 61.95 | 15.05 | 0.08 | 26.00 | 99.91 | 17.27 | 56.24 | 56.75 | 4.81 | 56.75 | 74.04 | 42.16 | 26.75 | 99.35 | 51.83 | 31.05 | 99.99 | 0.24 | 98.94 |  |
| Adap-Blend [] | 54.55/96.35 | 25.95 | 21.18 | 16.39 | 0.68 | 10.67 | 0.69 | 34.92 | 0.01 | 51.73 | 14.93 | 0.08 | 25.81 | 67.20 | 15.69 | 43.58 | 1.09 | 10.99 | 1.09 | 81.50 | 23.87 | 41.12 | 100.0 | 57.36 | 30.35 | 99.99 | 0.16 | 99.27 |  |
| Avg. |  | 3.27 | 10.35 | 2.07 | 38.99 | 4.68 | 36.53 | 73.28 | 0.00 | 82.09 | 15.29 | 0.04 | 26.43 | 64.60 | 13.20 | 42.29 | 67.28 | 3.76 | 66.69 | 89.86 | 18.45 | 57.11 | 87.12 | 30.44 | 42.27 | 97.75 | 0.09 | 98.36 |  |

### G.2 Performance of AGPD with VGG19-BN under various poisoning ratios.

We estimate the detection performance of AGPD against various backdoor attacks with different poisoning ratios and compare our method with four detectors. The results are displayed in Fig. 8. It can be seen that our method achieves a higher TPR under most attacks compared to other methods, which also maintain relatively low FPR.

Figure: Figure 8: Detection performance of AGPD and the compared detectors with poisoning ratios ranging from $1\%$ to $10\%$.
Refer to caption: extracted/5625055/figure/appendix/poisoning_ratio_vgg.png

## Appendix H GCDs for various attacks on CIFAR-10

Fig. 9 and Fig. 10 present the GCDs for all classes in CIFAR-10 under eight backdoor attacks with 10% poisoning ratio, based on the model structures of Preact-ResNet18 and VGG19-BN, respectively. It can be noticed that the target class (covering both black and blue arcs) occupies a longer arc on the circle compared with the clean classes across different model structures and backdoor attacks. Note that we moved all clean classes’ arcs to different areas on the circle to avoid visual overlap.

Figure: Figure 9: Gradient circular distributions (GCDs) of multiple backdoor attacks on CIFAR-10 with Preact-ResNet18.
Refer to caption: extracted/5625055/figure/appendix/dispersion_preact.png

Figure: Figure 10: Gradient circular distributions (GCDs) of multiple backdoor attacks on CIFAR-10 with VGG19-BN.
Refer to caption: extracted/5625055/figure/appendix/dispersion_vgg.png

## Appendix I The separation between poisoned and clean samples

Fig. 11 displays the cosine similarities of samples in the target class with a clean basis, which are computed in activation gradient and activation spaces, respectively. We exhibit the distributions of cosine similarities of four convolutional layers. If a sample is clean, the cosine similarity should be large. As shown in Fig. 11, the separation of the distribution of cosine similarities between clean and poisoned samples is larger in the activation gradient space, which can be observed in many convolutional layers.

Figure: Figure 11: The distribution of the cosine similarities of samples from the target class with a clean sample in multiple convolutional layers. The model structure is Preact-ResNet18. The purple represents poisoned samples, while the blue represents clean ones.
Refer to caption: extracted/5625055/figure/seperation.png

## Appendix J The results of the compared ASSET on ResNet18

In our above experiment, we found that detection performance of ASSET is deviated from the results reported in the paper [], such as BadNets and Blended attacks. Therefore, we decided to replicate these detection results using the recommended model structure, ResNet18. We tested six different attacks, including BadNets, Blended, WaNet, Input-Aware, TaCT, and Adap-Blend. The experiments were conducted on the CIFAR-10 dataset with 10% poisoning ratio. The corresponding results are shown in Tab. 12. When changing the model structure from Preact-ResNet18 to ResNet18, we discovered that the detection performance of ASSET becomes better, such as the TPRs of BadNets, Blended, and TaCT. However, when the form of the trigger is too complex and dynamic, which requires the model to spend more epochs learning it, this poses a greater challenge for the loss-based method, like ABL and ASSET. The work [] provides more analysis of quick learning of backdoors.

**Table 12: The results of ASSET on CIFAR-10 with ResNet18.**
|  | BadNets | Blended | WaNet | Input-Aware | TaCT | Adap-Blend |
| --- | --- | --- | --- | --- | --- | --- |
| TPR | 90.70 | 99.90 | 1.88 | 0.25 | 100.00 | 54.58 |
| FPR | 0.18 | 9.43 | 1.69 | 0.22 | 0.00 | 38.90 |
| F1 | 94.32 | 69.97 | 3.55 | 0.48 | 100.00 | 23.43 |

## Appendix K t-SNE results

In this section, we provide the t-SNE visualizations of ten backdoor attacks conducted on CIFAR-10 with Preact-ResNet18 in Fig. 12. The activations are extracted from the last convolutional layer (layer4.1.conv2). From Fig 12, it is evident that the dispersion of activations of poisoned and clean samples is significant in some attacks, such as BadNets and Blended. Consequently, most activation-based methods perform well against these attacks. Additionally, the dispersion of activations of the target class is relatively small in CTRL attacks, where these methods fail.

Figure: Figure 12: t-SNE visualization of ten backdoor models trained on poisoned CIFAR-10 under 10% poisoning ratio for non-clean label attacks and 5% poisoning ratio for clean label attacks. The model architecture is Preact-ResNet18.
Refer to caption: extracted/5625055/figure/appendix/tsne.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you please elaborate on this : "It is inspired by phenomenon that a backdoored model tends to map both poisoned and clean samples within the target class to similar areas in its activation space" ? - I believe this is true only for self-supervised training, whereas the current paper focuses on supervised training.

2. At which h,w is the activation getting sliced for calculating the activation gradient ? Is it the same for all layers considered ? Why ?

3. Can we simply use the difference in the angles from 0 in the distribution $\rho$ instead of the sample closeness metric ? Why do we need to use it ?
soundness: 3
strengths: 1. The paper identifies a previously unexplored phenomenon in backdoor samples, highlighting greater dispersion in activation gradient angles within the target class compared to clean samples.
2. The authors provide extensive experiments on various backdoor attacks across two architectures (PreAct ResNet18 and VGG-18 BN),
3. Some initial analysis on adaptive attacks is presented, though further exploration could strengthen the findings.
4. The writing is clear and well-structured, making the main arguments easy to follow.
summary: The authors present an approach for backdoor sample detection based on the angle of the activation gradients between samples and a reference clean sample for each class.
The paper observes that, within the target class, the activation gradient angles exhibit greater dispersion compared to samples from the clean class. Furthermore, the distribution of angles of backdoor and clean samples within the target class are somewhat distinct, providing a basis for differentiating between clean and poisoned samples.

Building on this observation, the authors introduce the concept of the Gradient Circular Distribution (GCD), a distribution of angles between the activation gradient of samples and that of a reference sample. They propose two key metrics based on this distribution:
CVBT: A metric designed to measure the dispersion of the GCD.
Sample Closeness: A metric that evaluates how close a given sample is to  reference clean sample.

These metrics are then utilized to develop a filtering algorithm, which first identifies the target class and subsequently filters out the poisoned samples. The paper validates the proposed method through experiments on a range of backdoor attack scenarios.
weaknesses: 1. I am giving low scores to soundness due to the reported  low performance of ASSET, AC. The performance in the original paper of ASSET is close to 90% for all attacks. Additionally, 0 F1 scores for AC also does not look right. For example, in the aforementioned paper, the average TPR of AC is around 50%. 

2.  Please provide sensitivity of various thresholds like $\tau_z$ , $w, \beta$ (Stage 3). What is the rationale behind the choice of $\tau_z$ ? 

3.  Please comment on or provide results for these additonal adaptive attacks: 
a. Can an adversary add  perturbations to the clean samples (different perturbations to different samples) such that their dispersion increases ? 
b. Given a clean set with 100 images, the adversary adds the same noise to 50 images and keeps the other images the same. Will the  dispersion increases of clean sample increase ? 
c. The backdoor trigger is optimized such that the dispersion of the target class decreases.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: (1) Understanding the activation gradient of poisoned examples with circular distribution is interesting. The two observations of GCD are novel.

(2) The algorithm shows strong performance within the evaluation setting of this paper.

(3) The paper is overall well written and the logic is easy to follow. The empirical experiments are mostly comprehensive and valid.
summary: The authors propose to detect poisoned examples from an activation-gradient circular distribution perspective. The authors draw two novel observations by studying the GCD of poisoned/clean examples, based on which they introduce an algorithm named AGPD to do backdoor example detection.
weaknesses: (1) My main concern about the proposed algorithm is its scalability. It requires training on the entire dataset to determine the detection state, which might be too expensive given web-scale data and the extreme probability of the data being poisoned. Also, it takes nearly $O(n2) $ to compute the GCD, which is infeasible for large datasets. 

(2) The defense algorithm appears to be highly correlated with the specific class labels, potentially limiting its applicability. In the real world, the attackers can adopt a different set of labels than the defenders. The attackers could label the dataset more fine-grained/coarse-grained than the defender. (e.g. Imagine there is a dataset consisting of snacks and drinks and the target class of the attacker is Coca-Cola. However, the defenders just want to classify the images as soft drinks no matter whether it is Coca-Cola or Pepsi. Or the reverse situation can also happen.) Will the observations about GCD hold under such misalignment?

(3) Table 7 is not organized well, please have a double-check.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) The proposed method is founded on two core observations: (i) the Gradient Circular Distribution (GCD) of the target class exhibits greater dispersion than those of clean classes, and (ii) poisoned and clean samples are distinguishable by their clear clustering in separate regions. However, it remains unclear whether these observations hold consistently for clean-label attacks, where the mapping directions of poisoned and benign samples could exhibit higher similarity given their similar input-space characteristics. Further discussion on this aspect would strengthen the paper’s analysis.

(2) The choice of baseline methods appears limited to older techniques. A broader comparison with more recent, state-of-the-art defense methods would provide a more thorough assessment of the proposed method’s performance relative to the latest advances in poisoned sample detection.

(3) Attack Success Rate (ASR) is a key metric commonly used in backdoor detection evaluations. Its omission from Table 1 raises questions about the comprehensiveness of the evaluation metrics chosen. Including ASR would provide a clearer view of the method's efficacy against backdoor attacks.

(4)   In the main evaluation, a poisoning ratio of 10% is used for non-clean label attacks and 5% for clean-label attacks. Since clean-label attacks are generally more challenging, lower poisoning ratios are typically applied to non-clean label attacks in real-world scenarios. This discrepancy between the paper’s experimental setup and actual attack contexts calls for further clarification. Additionally, practical attack scenarios often employ very low poisoning ratios (1% or 0.5%), feature separability-based defenses tend to be less effective in this setting. More discussion should be included.
soundness: 2
strengths: (1) Introducing GCD as a novel measure adds a unique, technical depth to the approach, potentially applicable to various attack scenarios.

(2)  AGPD shows high detection performance with minimal additional clean data, a notable advantage for real-world applicability, where such data may be limited.
summary: The paper presents a new approach to identifying poisoned samples in datasets used for deep neural network training. The core proposal, termed Activation Gradient-Based Poisoned Sample Detection (AGPD), leverages a novel metric called Gradient Circular Distribution (GCD) to detect discrepancies between clean and poisoned samples. Key insights include observing that in backdoor attacks, models often map poisoned samples and clean samples to similar activation regions, which AGPD detects by analyzing gradient direction distributions. This method involves three main steps: calculating GCD for each class, identifying target classes based on dispersion, and filtering out poisoned samples.
weaknesses: (1) The proposed method is founded on two core observations: (i) the Gradient Circular Distribution (GCD) of the target class exhibits greater dispersion than those of clean classes, and (ii) poisoned and clean samples are distinguishable by their clear clustering in separate regions. However, it remains unclear whether these observations hold consistently for clean-label attacks, where the mapping directions of poisoned and benign samples could exhibit higher similarity given their similar input-space characteristics. Further discussion on this aspect would strengthen the paper’s analysis.

(2) The choice of baseline methods appears limited to older techniques. A broader comparison with more recent, state-of-the-art defense methods would provide a more thorough assessment of the proposed method’s performance relative to the latest advances in poisoned sample detection.

(3) Attack Success Rate (ASR) is a key metric commonly used in backdoor detection evaluations. Its omission from Table 1 raises questions about the comprehensiveness of the evaluation metrics chosen. Including ASR would provide a clearer view of the method's efficacy against backdoor attacks.

(4)   In the main evaluation, a poisoning ratio of 10% is used for non-clean label attacks and 5% for clean-label attacks. Since clean-label attacks are generally more challenging, lower poisoning ratios are typically applied to non-clean label attacks in real-world scenarios. This discrepancy between the paper’s experimental setup and actual attack contexts calls for further clarification. Additionally, practical attack scenarios often employ very low poisoning ratios (1% or 0.5%), feature separability-based defenses tend to be less effective in this setting. More discussion should be included.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q-(1): The calculation of $\rho_{x_0}\(\mathcal{D}\)$ in Eq.(3) and $s_{x_0}\(x_i\)$ in Eq.(4) basically transforms e.g. $\theta_{x_0}\(x_i\)$ back to $\frac{g\(x_i\) \cdot g\(x_0\)}{||g\(x_i\)|| ||g\(x_0\)||}$. By simplifying the formulation in Eq.(3) and Eq.(4), does this mean it is essentially calculating the cosine similarity of the gradients between the basis pair and each input sample? If this is indeed the case, is the calculation of radian in Eq.(2) useful merely for the visualization in figures like Figure 1.?

Q-(2): In section 3, the channel-wise activation gradient $g^{\(l\)}$ is first introduced in Eq.(1). However, the formulating w.r.t the index of the model layer is missing in the consequent methodology description. Can the authors elaborate on how the channel-wise gradients in each layer are merged into the final GCD of an input sample $x_i$?

Q-(3): The basis pair is crucial to disperse samples; thus, a small clean set is necessary for APGD detection. As even a single clean sample per class has been proven effective, how does the selection of clean samples influence APGD's performance? Or is APGD resistant to any random sampling of clean samples?

Q-(4): Different from the study on the threshold $\tau_s$ in section 5.4, $\tau_z$ remains equal $e^2$. What is the rationale of setting $\tau_z = e^2$? 

Q-(5): In Figure 4, APGD detection on the target class works better against all-to-one attacks than all-to-all attacks. While detecting poisoned samples of all-to-all attacks, does APGD handle the separation of all-to-all attacks class by class via the implementation of Stage 3?

Q-(6): In the default setting of Adaptive-Blend, the poisoning ratio is relatively low (poisoned/total = 150/50000 = 0.3% for CIFAR10), where the same amount of noisy samples as poisoned samples are added by setting the conservatism ratio = 1.0, which is unfortunately missing in Table 7. Moreover, considering the default poisoning ratio = 5%, the ASR of 66.17% for Apat-Blend is much lower than that of the original paper. Does this imply that the Adptive-Blend in this paper has some biases from the default attack configuration?

Q-(7): Considering that both WaNet and Adaptive-Blend attacks involve noisy samples to improve the attack stealthiness, can the authors provide some visualization on the separation of APGD between the poisoned samples and noisy samples in both attacks?
soundness: 3
strengths: - This paper proposes a novel measurement to show the dispersion of poisoned and clean samples in the model's activations

- This paper is well-written and enforces a high readability with variant and helpful illustrations 

- This paper comprehensively evaluates AGPD's performance by considering diverse poisoning attacks, comparing with different related defenses, and conducting experiments on variant datasets.

- This paper considers the ablation study on different hyper-settings of APGD and involves two reasonable adaptive attacks that have considered APGD's defense w.r.t the GCD distribution.
summary: With the inspiration that a backdoored model is apt to map significantly different poisoned samples and clean samples of the backdoor target to similar activation areas, this paper introduces a novel measurement, i.e., the circular distribution of the gradients w.r.t sample activation, namely GCD, which works to identify the target class of the backdoor and consequently separate poisoned and clean samples within the target class. Accordingly, this paper proposes a sample detection approach called AGPD to achieve dataset purification. Extensive experiments show AGPD's advanced performance in detecting and isolating poisoned samples.
weaknesses: (1) The use of $arccos(\cdot)$ in Eq.(2) is ambigious as $cos\(\cdot\)$ is used in Eq.(3) and Eq.(4) as well.

(2) The calculation of GCD across all model layers is not clearly formulated for each sample.

(3) The influence of the clean sample by random sampling for the basic sample pair is not studied.

(4) There is lacking the rationale for setting the hyper-parameter $\tau_z = e^2$.

(5) For all-to-all attacks, a better explanation of how the dataset separation by APGD is achieved is required.

(6) The description of Adpative-Blend in Appendix B is incomplete and different from the default attack setting.

(7) It is unclear how the noisy and poisoned samples are separated against WaNet and Adaptive-Blend attacks.

---
In the following, further questions related to the above points are detailed with the notion of "Q-#".

Meta-review: The paper proposed a new apporach for detecting poisoned samples called Activation Gradient-Based Poisoned Sample Detection (AGPD). It leveerages a novel metric called Gradient Circular Distribution to distinguish clean and poisoned samples. The empirical results also demonstrate its effectiveness.

Strength:

1. Insightful view on understanding the activated gradient of poisoned examples with distribution.

2. The algorithm shows strong performance within the evaluation setting of this paper.

3. Well written.

Weakness:

1. Only explores the backdoor attacks on Pre-Act ResNet 18 and VGG, whether the proposed feature still exists in Transformers or other networks are unexplored.

All the reviewers thoughts positive towards this paper, therefore I recommend to accept it as a poster.

Decision: Accept (Poster)