Wicked Oddities: Selectively Poisoning for Effective Clean-Label Backdoor Attacks

## Abstract

Abstract Deep neural networks are vulnerable to backdoor attacks, a type of adversarial attack that poisons the training data to manipulate the behavior of models trained on such data.
Clean-label attacks are a more stealthy form of backdoor attacks that can perform the attack without changing the labels of poisoned data.
Early works on clean-label attacks added triggers to a random subset of the training set, ignoring the fact that samples contribute unequally to the attack’s success. This results in high poisoning rates and low attack success rates.
To alleviate the problem, several supervised learning-based sample selection strategies have been proposed.
However, these methods assume access to the entire labeled training set and require training, which is expensive and may not always be practical.
This work studies a new and more practical (but also more challenging) threat model where the attacker only provides data for the target class (e.g., in face recognition systems) and has no knowledge of the victim model or any other classes in the training set.
We study different strategies for selectively poisoning a small set of training samples in the target class to boost the attack success rate in this setting.
Our threat model poses a serious threat in training machine learning models with third-party datasets, since the attack can be performed effectively with limited information. Experiments on benchmark datasets illustrate the effectiveness of our strategies in improving clean-label backdoor attacks.

## 1 Introduction

Modern deep learning models have exhibited tremendous success in solving challenging tasks, ranging from autonomous driving and face recognition to natural language processing.
Training these large models requires massive training data, which is time-consuming and labor-intensive, and incurs huge costs to collect and annotate.
Therefore, users usually prefer to employ third-party or open-source data.
Recent studies have shown that deep learning models are vulnerable to backdoor attacks [, , ]. A malicious data supplier can provide poisonous data such that the model trained on it behaves normally on benign data, but always returns a desired output when a "trigger" is presented.

Most existing backdoor attacks rely on data poisoning and can be classified as either dirty-label or clean-label, depending on whether the label of poisoned data changes.
For dirty-label attacks [, , ], the adversary adds the trigger into data and *points* its label to their desired target label.
Dirty-label backdoor attacks are effective but can be easily detected by humans during data verification since the semantics of the labels are typically not consistent with the input content.
Conversely, clean-label attacks [, , ] poison training data without changing labels, rendering them more challenging to detect. However, compared to the dirty-label case, it is also much more difficult to mount clean-label backdoor attacks as one needs to poison significantly more training data and the resulting models can perform poorly on clean data.
In this paper, we focus on improving the data effectiveness of backdoor attacks, i.e., to increase the attack performance given a small budget for (or a small number of) poisoned samples.

Prior backdoor attacks implicitly assume all training samples contributed equally to the attack’s success and performed data poisoning uniformly randomly over training data.
However, recent research [, , , ] reveals that among training data points, some are more important while some others are redundant and can be discarded from the training set. One can ask a similar question for backdoor learning: “*Can selectively, rather than randomly, poisoning some training data points lead to more effective backdoor attacks?*”

In the initial investigation, Xia et al. [] and Gao et al. [] explored this problem and proposed strategies to enhance the efficiency of selecting samples for poisoning. They achieved this by recording forgetting events or examining loss values to identify hard samples. However, these methods have several drawbacks. Firstly, they are time-consuming and computationally expensive because of the need to train a surrogate model on the dataset from scratch. Secondly, they require access to the whole training data for surrogate model training. It is impractical in some real-world scenarios where the user collects data from diverse sources and the attacker lacks knowledge of data beyond its contributions.

**Table 1: Properties of selection strategies. “✗” or “✓” means the method lacks or has, respectively, the property.**
| Method | trigger-<br>agnostic | model-<br>agnostic | no training<br>required | partial<br>data<br>access | no extra<br>data required |
| --- | --- | --- | --- | --- | --- |
| Xia et al. [] | ✗ | ✓ | ✗ | ✗ | ✓ |
| Gao et al. [] | ✓ | ✓ | ✗ | ✗ | ✓ |
| OOD strategy (Ours) | ✓ | ✓ | ✗ | ✓ | ✗ |
| Pretrained strategy (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ |

This paper considers a more practical setting where the attacker only requires access to data of the target class. This assumption applies to the case where a single client cannot collect labels due to geographical or infrastructural obstacles, such as collecting different species of plants in other countries for plant classification tasks, or the system has to respect data privacy.
We study novel methods to select samples to attack effectively under this threat model.
For the victim model to learn the backdoor, it needs to focus on the trigger rather than other features in the data [].
Intuitively, if the samples with triggers are difficult to learn, the model will use triggers as shortcuts to minimize the objective function.
As a result, the model is more susceptible to learning the triggers.
To achieve such a goal without having access to the full training dataset or victim model, we propose a novel data selection framework that uses pretrained models or out-of-distribution data to identify hard training samples and add the triggers to these samples. Our strategy does not depend on the trigger, the victim model, and does not require access to other data classes. The advantages of our approaches are illustrated in Table 1. In summary, our contributions can be listed as follows:

- •
We study a new backdoor threat model where the attacker, acting as one of the data suppliers, has only access to the training data of the target class but can still perform data-poisoning, clean-label backdoor attacks effectively.
- •
We propose two novel approaches, each of which selects then poisons only a few “hard” samples; training with these poisoned samples, along with clean samples from the other classes provided by those other data providers, at the victim site will force the model to learn a backdoor shortcut to the trigger.
The first approach relies on access to a pretrained model which can be performed without training, while the second approach relies on out-of-distribution samples.
- •
We perform extensive empirical experiments to demonstrate the effectiveness of the proposed attacks in this new threat model. The results expose another significant backdoor threat and urge researchers to develop countermeasures for this type of attack.

## 2 Related Works

Backdoor Attacks. Backdoor attacks aim to insert a malicious backdoor into the victim model.
The first attempt is BadNets [], where the attacker adds a predefined image patch to some images in the training set and changes the labels of these images to the target class. Follow-up works introduce various forms of the Trojan horse to enhance the stealthiness and the effectiveness of the attack, examples include blended [], dynamic  [], warping-based [], input-aware [, ], and learnable trigger [].
These attacks are called dirty-label attacks as they change the true labels of poisoned examples.

Clean-label Backdoor Attacks.
Despite the success in manipulating the victim, dirty-label attacks can be easily spotted through human inspection.
Clean-label backdoor attacks are attack methods that perverse the original labels of poisoned data points, and thus are more stealthy than dirty-label attacks.
Turner et al. [] suggested that using dirty-label attack triggers is ineffective for implementing clean-label attacks and proposed a data preprocessing method for implementing clean-label attacks.
In the meantime, stronger triggers have been proposed.
SIG [] uses sinusoidal signals as backdoors. Refool [] uses physical reflection models to implant reflection images into the dataset. HTBA [] optimizes the input such that it looks similar to the target label in the pixel space but close to the malicious image in the latent space. However, these attacks require a high poisoning rate and/or result in inferior success rates. Zeng et al. [] propose to perturb samples employing out-of-distribution data to achieve a high attack success rate with a low poisoning rate. Their threat model is similar to the less constrained version of which is studied in this paper.

Selectively Data Poisoning.
Research in backdoor attacks focuses on designing the trigger pattern, ignoring the possibility that benign samples chosen to attack can also play an important role. FUS [] first showed that the number of forgetting events is an indicator of the contribution to the attack, and proposed a data selection strategy based on forgetting events that resulted in a better attack success rate.
Gao et al. [] identified three classical criteria to pick samples for clean-label attacks, namely loss value, gradient norm, and forgetting event.
To select samples for poisoning, these methods require a surrogate model trained on a dataset with all training set classes, which is expensive and not always feasible.

Backdoor Defenses.
Along with the emergence of backdoor attacks, defense methods to protect models are an active research area.
Backdoor defenses can be categorized into two lines: backdoor detection and backdoor mitigation.
Qiao et al. [] propose a defense that utilizes generative models to detect and reconstruct the backdoor and then retrain the model. Activation Clustering [] examines the activations of training data to check whether each data sample is poisoned.
Tran et al. [] reveal that poisoned samples can be identified by spectral signatures, and utilize this trace to remove the backdoor in the training dataset.
Neural Cleanse [] detects the trigger by optimizing the pattern to misclassify to the target class and running outlier detection, and proposes a mitigation mechanism.
Other mitigation methods aim to reduce the backdoor effect in the model by fine-tuning [] and prunning [].
NAD [] erases the trigger by utilizing a teach-student fine-tuning process to guide the poisoned model on a small clean dataset.
Huang et al. [] propose to decoupling the training process to prevent the model from learning the trigger.
From a security perspective, the adversary should not only succeed in attacking the model but also in dodging backdoor defenses.

## 3 Threat Model

Single-class, data poisoning attack. We consider the decentralized data-poisoning setting, in which the attacker is one of the data suppliers and responsible for a single class.
The victim employs a distributed data collection pipeline, which tackles the obstacles in building diverse or privacy-aware datasets.
Such a situation might happen when each label class comes from a different region, or brings a different characteristic.
For example, to classify ethnicity, breeds, or plant species that come from many locations, a data supplier in a place is in charge of the label class only available in that region. Another situation is when the dataset contains sensitive information, and a data class is not allowed to be exposed to anyone except those who provide it.
In these situations, the backdoor threat can be caused by a data supplier or an attacker who hijacks local data storage. The attack setting is depicted in Figure 1. To the best of our knowledge, this represents the most constrained data-poisoning threat, wherein the attacker has extremely limited information for launching an effective attack.

Attacker’s goal.
The objective of the adversary is to inject a trigger into a victim model, such that the model acts normally on benign data, but misclassifies with the presence of the trigger.
For instance, a facial recognition system’s task is to recognize people to grant them certain permissions, but when poisoned with sunglasses as a trigger, it might give full authority to anyone wearing sunglasses.

Figure: Figure 1: Illustration of our threat model. The attacker acts as a data provider in a supply chain where each data provider is responsible for a data class. The attacker injects a trigger into the images without changing the label and sends them to the victim. The model that is trained on this poisoned dataset behaves normally on clean images but returns the target label when the trigger is added to any image.
Refer to caption: x1.png

Attacker’s capability.
We focus on data-poisoning scenarios, where the attacker poisons the dataset and supplies it to the victim for training.
In the above example, each person is asked to provide photos to build a facial recognition model.
Malicious users can inject triggers into their images to control the model output for malicious purposes, but cannot manipulate data provided by other users.
In general, we consider a practical setting where the adversary serves as a single client in the supply chain, it only provides and controls data for the class it wants to attack.
Therefore, the adversary can only select images *with the target label* to insert the trigger.

Attacker’s knowledge.
The adversary only has access to data for the target class that it provides. *No information of the victim model’s architecture, the training process, or data from other clients* is exposed to the attacker.
In some scenarios, although the adversary has no access to data from other classes for their problem, some out-of-distribution data (OOD) are available to the adversary, which is a realistic assumption. This relaxed threat model has also been studied in []. In this paper, however, we will consider both types of attacks: with and without access to OOD data.

## 4 Method

### 4.1 Problem Formulation

Let $f_{\theta}:\mathcal{X}\to\mathcal{Y}$ be a model that maps image $x\in\mathcal{X}$ to label $y\in\mathcal{Y}$,
and $\mathcal{D}_{\operatorname{c}}=\{(x_{1},y_{1}),\dots,(x_{n},y_{n})\}$ be the clean training dataset. In backdoor attacks, the adversary first defines a trigger injecting function $T:\mathcal{X}\to\mathcal{X}$ that implants a trigger into input data, then applies $T$ to $m$ images in $\mathcal{D}_{\operatorname{c}}$.

Let $S$ be the target class. The attacker selects a subset $S^{\prime}\subset S$ of size $m$, and adds triggers to samples in $S^{\prime}$.
After injecting the trigger into $S^{\prime}$ (leaving the other examples in $S$ intact),
the attacker gives its data to the victim who combines it with data from other sources to create a poisoned dataset $\mathcal{D}_{\operatorname{p}}$. The victim then trains the model on $\mathcal{D}_{\operatorname{p}}$ with some standard training pipeline to obtain the model $f_{\theta^{*}}$.
The attacker’s goal is to make any model trained on $\mathcal{D}_{\operatorname{p}}$ to return correct predictions on unpoisoned examples, but predict the target label $y^{t}$ on any example on which the trigger function $T(\cdot)$ is applied.
Formally, for a benign input $x$ with correct label $y$, we have

$$ $f_{\theta^{*}}(x)=y,\quad f_{\theta^{*}}(T(x))=y^{t}.$ $$

The performance of backdoor attack methods is usually evaluated via two metrics: benign accuracy (BA) and attack success rate (ASR).
BA is the accuracy of the infected model on benign test samples. ASR is the proportion of attacked test samples that are successfully predicted as the target label by the infected model.
In addition, stealthiness is an important factor for backdoor attacks, which is reflected by small poisoning
rate, imperceptibility of the backdoor, and resistance against backdoor defense methods.

### 4.2 Random or Hard Sample Selection?

Figure: Figure 2: The attack success rate of SIG on ResNet18/CIFAR10 with $10\%$ of the target class that are harder than the $0,30,60$, and $90$-th percentile being poisoned. The horizontal line is the attack success rate where the poisoned set is selected randomly.
Refer to caption: x2.png

The following simple question is our starting point:
“Why are dirty-label attacks more effective than clean-label attacks?”. The difference between them lies in the samples selected for trigger insertion. In dirty-label attacks, the poisoned samples come from various labels, and their features are dissimilar to those in the target class. For example, if the adversary wants to attack class 0 0, dirty-label attacks can choose samples from class $0,1,2,\dots$, while clean-label attacks only inject poisoned samples to class 0 0.

During training, the model looks for common features to form the decision boundary. Therefore, an example containing features different from other examples in a class is harder to learn.
When the adversary injects a trigger into these “hard samples” and alters their labels, *the model cannot rely on existing features in the image to optimize the objective function, instead it favors backdoor features*, leading to a higher ASR even with a small set of poisoned samples, as usually seen in dirty-label attacks. On the other hand, clean-label attacks, with randomly poisoned samples that likely share similar features with other clean samples from the same class, require a significantly higher number of samples to reach a high ASR.

Based on this intuition, we search for and add triggers to “hard samples” in the target class to achieve stronger clean-label attacks.
A straightforward solution is to train a surrogate model on the training set and examine the behavior of the model on each data point, an approach used in []. For example, a sample with a higher loss value is likely more difficult to learn.
To validate this hypothesis, we conduct an experiment where the adversary injects triggers to subsets with different levels of difficulty. We employ Error L2-Norm (EL2N) [] to sort training samples from easy to hard. We attack $4$ ResNet18 models on CIFAR10 using SIG with $10\%$ poisoning rate of the target class, where poisoned sets are harder to learn than $0\%,30\%,60\%,90\%$ of the target class. Figure 3 shows that poisoning hard samples leads to a higher attack success rate, verifying our assumption.

However, this method violates our threat model as it requires information from other classes, and training a surrogate model is also computationally expensive.

### 4.3 Selecting Hard Samples with Pre-trained Models

Without access to the full training data to build the surrogate model, we turn our attention to pre-trained models. These models are often easy to access and available in most domains due to the popularity and benefits of self-supervised learning [, , , , , ], and the existence of some large-scale labeled datasets, such as ImageNet [] and JFT [].
Furthermore, Sorscher et al. [] show that pretrained features are good indicators of hard samples: by incorporating self-supervised models to keep hard samples, they can discard 20-30% of the ImageNet dataset, while only experience negligible degradation in model performance.
Inspired by this observation, we exploit a pre-trained model as the feature extractor and develop a novel strategy to find examples that are dissimilar to other data in the target class.

Figure: Figure 4: EL2N and our score of training samples in class 0 0 of CIFAR10. We also illustrate the thresholds where $5\%,10\%$, and $20\%$ of class 0 0 ($0.5\%,1\%$, and $2\%$ of the training data) is poisoned.
Refer to caption: x3.png

Specifically, we first extract features of data samples using the pre-trained model, then identify those samples that are far away from others in this feature space.
To show that these features are discriminative, we extract feature embeddings from VICReg [], a pre-trained self-supervised model, and visualize the feature space with t-SNE []. Figure 3 illustrates that by exploiting pre-trained models to extract features, datapoints from the same class stay close to each other in the feature space. Hence, samples that are far from the target label cluster contain different features, thus harder for the model to learn.

Let $g$ be a feature extractor. We define the distance between two samples $x_{i},x_{j}$ by cosine similarity between their feature $z_{i}=g(x_{i}),z_{j}=g(x_{j})$:

$$ $d(x_{i},x_{j})=1-\frac{z_{i}^{\intercal}z_{j}}{\|z_{i}\|\|z_{j}\|}.$ $$

We apply the classical $k$-NN algorithm to calculate a score function $s(x)$ as the mean of distances between $x$ and its $k$-nearest neighbors $x_{1},\dots,x_{k}$ in the target class in terms of the distance $d(\cdot,\cdot)$:

$$ $s(x)=\frac{1}{k}\sum_{i=1}^{k}d(x,x_{i}).$ (1) $$

With an attack budget of $m$, our strategy collects $m$ samples with the highest scores. The detailed algorithm is shown in Algorithm 1.

We compute EL2N and our proposed score on class 0 0 of CIFAR10 and illustrate in Figure 4. As has been observed, they are correlated, with Pearson coefficient equal to $0.504$. Figure 3 also expresses that when utilizing our score to rank datapoints, injecting triggers to samples from easy to hard results in increasing values of attack success rates. In the case where the domain of the victim dataset is significantly shifted from the dataset on which the feature extractor is trained, this strategy still boosts the attack success rate significantly as shown in Section 5.2 and Appendix B.2.

### 4.4 Selecting Hard Samples with Out-Of-Distribution Data

**Table 2: Attack success rate (ASR) of clean-label attacks on CIFAR10 with $5\%/10\%/20\%$ of the target class being poisoned.**
| Model | Method | BadNets | Blended | SIG |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 5% | 10% | 20% | 5% | 10% | 20% | 5% | 10% | 20% |  |  |
| ResNet18 | Random | 30.81 | 45.01 | 78.28 | 28.94 | 37.55 | 44.26 | 50.28 | 60.54 | 78.45 |
| Self-supervised Models | 86.24 | 91.68 | 98.84 | 44.64 | 52.90 | 66.45 | 76.35 | 80.59 | 86.45 |  |
| Supervised Models | 90.01 | 92.14 | 99.26 | 47.68 | 60.86 | 67.81 | 81.65 | 85.42 | 90.49 |  |
| Multiple-class OOD | 75.57 | 81.27 | 98.47 | 43.40 | 56.89 | 61.68 | 65.11 | 80.76 | 88.79 |  |
| Single-class OOD | 82.34 | 80.75 | 91.37 | 42.99 | 57.29 | 62.60 | 72.93 | 79.07 | 87.18 |  |
| VGG19 | Random | 63.24 | 78.39 | 79.55 | 17.32 | 23.84 | 34.36 | 22.28 | 45.54 | 67.57 |
| Self-supervised Models | 81.44 | 82.60 | 93.11 | 30.74 | 42.23 | 55.34 | 46.65 | 70.23 | 81.93 |  |
| Supervised Models | 83.43 | 89.61 | 87.70 | 22.86 | 38.84 | 54.99 | 47.89 | 74.38 | 80.07 |  |
| Multiple-class OOD | 79.69 | 88.44 | 86.78 | 29.35 | 38.39 | 49.24 | 50.81 | 65.80 | 78.28 |  |
| Single-class OOD | 75.36 | 81.01 | 89.68 | 30.49 | 40.58 | 51.60 | 57.24 | 72.35 | 79.04 |  |

In this section, we propose a hard-sample selection approach that utilizes some OOD data available to the adversary, as discussed in Section 3. Note that the threat model is still the same as in the previous section, where the adversary does not have any knowledge of the victim model’s architecture, the training process, or data from other clients.

Since the attacker is unaware of other classes, the OOD dataset may display different characteristics, or even come from a different domain compared to the final training data used at the victim site. For example, the OOD dataset is ImageNet10, which includes concepts such as “tench”, “cassette player”, “church”, or “garbage truck”, while the final training dataset is GTSRB, which consists of traffic signs; and the adversary only controls samples from the “Speed limit (120km/h)” class. To form an attack, we first combine the OOD dataset of $n$ classes with the target-class data to obtain a new dataset; for example, this merged dataset contains samples from ImageNet10 and the “Speed limit (120km/h)” class.
Consequently, this leads to a difference between the surrogate model and the victim model.
We then train a surrogate model on this merged dataset and use it to select hard samples.

In this work, we consider two approaches: (i) Single-class OOD strategy that collapses the OOD dataset into a single class and training a binary classification model, and (ii) Multiple-class OOD strategy  that reserves the original labeling of the OOD samples and training a $n+1$-class classification model. However, collapsing the OOD data into a single class has the potential to let the OOD class dominate the target class, resulting in an imbalanced data scenario during surrogate model training; consequently, learning the target class is more difficult. Hence, we instead choose a subset of the OOD dataset such that the new OOD class and the target class have similar sizes. Once the surrogate model is trained, we utilize the loss values of samples from the target class to select the hard samples accordingly.

## 5 Experiments

In this section, we provide the empirical evaluation of our data selection attack method.

### 5.1 Experimental Setup

Dataset. We consider two widely used benchmark datasets: CIFAR10 [] and GTSRB []. For OOD strategy, we train the surrogate model on TinyImagenet [].
The domain of CIFAR10 is slightly far from ImageNet [], the dataset used to build pretrained models, or the OOD dataset. However, there are more apparent distribution shifts from ImageNet and TinyImagenet to GTSRB, a dataset of traffic signs. Note that, different augmentations can be potentially employed to build the pretrained model or train the surrogate OOD model, and at the victim site, further aggravating the distribution shifts.
Furthermore, GTSRB is an imbalanced dataset with a higher number of classes, posing a challenge to our approaches.

Models. For the victim model, we consider ResNet18 [] and VGG19 []. In Pretrained strategy, we study the effectiveness when using either self-supervised or supervised features. For the self-supervised pretrained models, we employ VICReg [], a method that applies the variance regularization term to avoid the collapse problem, with ResNet50 as the architecture. For the supervised feature extractor, we adopt a ResNet50 model pretrained on ImageNet. In OOD strategy, we utilize ResNet18 as the architecture to train the surrogate model when attacking both ResNet18 and VGG19 victim models.

**Table 3: Attack success rate (ASR) of clean-label attacks on GTSRB with $5\%/10\%/20\%$ of the target class being poisoned.**
| Model | Method | BadNets | Blended | SIG |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 5% | 10% | 20% | 5% | 10% | 20% | 5% | 10% | 20% |  |  |
| ResNet18 | Random | 5.72 | 5.80 | 6.13 | 36.35 | 41.54 | 48.91 | 47.63 | 48.07 | 48.67 |
| Self-supervised Models | 10.37 | 10.91 | 18.13 | 39.36 | 47.97 | 50.70 | 54.85 | 57.12 | 58.88 |  |
| Supervised Models | 6.83 | 8.47 | 21.33 | 42.58 | 47.85 | 50.67 | 51.76 | 56.07 | 56.57 |  |
| Multiple-class OOD | 5.77 | 5.84 | 7.24 | 43.08 | 43.18 | 45.46 | 43.56 | 47.59 | 52.50 |  |
| Single-class OOD | 6.22 | 6.18 | 13.95 | 46.96 | 50.13 | 51.54 | 49.07 | 51.71 | 55.15 |  |
| VGG19 | Random | 6.14 | 6.38 | 6.89 | 24.89 | 26.36 | 30.69 | 32.04 | 33.90 | 36.52 |
| Self-supervised Models | 8.16 | 10.19 | 13.20 | 30.67 | 29.77 | 33.82 | 40.33 | 42.68 | 42.37 |  |
| Supervised Models | 7.09 | 8.46 | 15.76 | 32.25 | 32.77 | 33.67 | 33.47 | 39.50 | 42.07 |  |
| Multiple-class OOD | 6.34 | 6.24 | 10.20 | 16.58 | 20.23 | 27.14 | 27.93 | 28.98 | 31.94 |  |
| Single-class OOD | 7.83 | 6.70 | 10.04 | 25.30 | 33.16 | 32.31 | 35.91 | 38.54 | 38.28 |  |

Attacks. We employ the trigger patterns from BadNets, Blended, and SIG for trigger injection; nevertheless, our method is trigger-pattern agnostic. We perform the clean-label attack to class 0 0 of CIFAR10 and class $1$ of GTSRB. These attacks inject triggers to $5\%,10\%$ and $20\%$ of the target class, which are $0.5\%,1\%,2\%$ poisoning rate with respect to the whole dataset in CIFAR10, and $0.19\%,0.38\%,0.76\%$ in GTSRB. Since it is a clean-label attack scenario, these poisoning rates are extremely small and very hard to spot by human inspection.

Strategy. We conduct experiments with two approaches: Pretrained strategy and OOD strategy, and compare to the random baseline where the attack treats every sample equally. For Pretrained strategy, we employ $k-$NN with $k=50$. For OOD strategy, we study Multiple-class OOD, in which we preserve the label of OOD data and train a surrogate model on a dataset of 201 classes, and Single-class OOD, in which we consider the OOD dataset as a single class and train a binary classifier. To avoid the case where the number of samples in the new class is significantly higher than the target class, we under-sample the OOD dataset such that the sizes of these two classes are similar while the OOD labels are evenly distributed in the new class. Furthermore, we vary the number of OOD labels $n$ to study the performance of the method at different diversity levels of the OOD dataset.

### 5.2 Effectiveness of Our Selection Framework

**Table 4: Clean accuracy (CA) on CIFAR10 and GTSRB with various poisoning rates.**
| Model | Strategy | CIFAR10 | GTSRB |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 5% | 10% | 20% | 5% | 10% | 20% |  |  |
| ResNet18 | Random | 94.69 | 94.60 | 94.59 | 99.06 | 99.04 | 99.11 |
| Self-supervised Models | 94.71 | 94.80 | 94.50 | 98.06 | 98.67 | 98.95 |  |
| Supervised Models | 94.93 | 94.77 | 94.34 | 99.11 | 98.60 | 98.76 |  |
| Multiple-class OOD | 94.65 | 94.47 | 94.44 | 99.04 | 99.30 | 98.85 |  |
| Single-class OOD | 94.78 | 94.62 | 94.53 | 99.07 | 98.97 | 99.39 |  |
| VGG19 | Random | 91.97 | 91.89 | 92.98 | 96.02 | 96.58 | 95.48 |
| Self-supervised Models | 91.81 | 91.89 | 91.66 | 96.56 | 95.53 | 95.87 |  |
| Supervised Models | 92.11 | 91.67 | 91.83 | 96.23 | 96.29 | 96.06 |  |
| Multiple-class OOD | 92.07 | 91.67 | 91.59 | 96.03 | 96.14 | 95.79 |  |
| Single-class OOD | 91.96 | 92.05 | 91.26 | 96.22 | 96.13 | 96.43 |  |

We perform clean-label attacks on CIFAR10 with the random strategy, Pretrained strategy, OOD strategy, and report the attack success rates in Table 2.
As can be observed, both strategies outperform the random baseline by large margins on all the attacks, models, and poisoning rates. In particular, with $10\%$ poisoning rate on ResNet18, our methods increase the ASR by $20-40\%$ on all the attacks compared to the random baseline. Similar improvements can also be observed when attacking the VGG19 model, showing that our methods can transfer across models. These results confirm the effectiveness of the proposed methods to select hard samples in the dataset to perform the attack; the result is significant especially when the considered threat model relies on only data from one class, showing the existence of a backdoor threat even in the most constrained setting.

Effectiveness with Extreme Distribution Shifts. We study the effect of our framework when the victim dataset is significantly different from the OOD dataset or the dataset used to train the pretrained models. In Table 3, we provide experimental results for GTSRB, which is a challenging imbalanced dataset. Our framework still exhibits consistent improvements, over the baseline, across three attacks with various poisoning rates on ResNet18 and VGG19.

Surprisingly, although it does not require any extra data, the performance of Pretrained strategy  is consistently higher than OOD strategy  on CIFAR10.
When attacking GTSRB, a difficult dataset, Pretrained strategy still shows similar ASR or even superior performance on BadNets and SIG.
In addition, Table 2 implies that supervised features are more helpful on CIFAR10, whose domain is similar to that of the pretrained models. In contrast, when observing higher distribution shifts, experimental results on GTSRB show that searching for attack samples using self-supervised features is better.

Effectiveness with Different OOD Strategies. As discussed, with OOD data, the adversary can choose to merge the samples into a single class (then train a binary surrogate classifier) or train a surrogate model on the $n+1$-class dataset. As shown in Table 2 and Table 3, using a binary classifier is slightly more effective for finding hard samples for the attack. Intuitively, when we preserve the labels of the OOD dataset, the surrogate model also needs to learn to discriminate among these classes, thus paying less attention to the target class, thus it is more difficult to use this model to identify hard samples.

Effect on the Performance of the Main Task. Here, we study whether our framework has any significant effect on the clean-data performance of the main classification task. Table 4 shows the clean accuracy for ResNet18 and VGG19 under the attacks with the SIG trigger pattern using our strategies. We can observe that selectively poisoning the dataset causes no degradation to the model performance on the clean data, similar to the random baseline. We also observe similar results for other types of trigger patterns.

### 5.3 Performance against Backdoor Defenses

We evaluate our strategy against popular backdoor defenses: STRIP []
(backdoor detection), Neural Cleanse [], and fine-pruning [] (backdoor mitigation).
We evaluate these defenses on a ResNet18 model trained on CIFAR10 and attacked with SIG. For the selection strategy, we use self-supervised Pretrained strategy and Multiple-class OOD strategy.

STRIP. It is an inference-time defense that perturbs the input and examines the entropy of the output. A sample with low entropy is more likely to be poisoned. Figure 6 visualizes the entropy of the output of clean data and backdoor data with random strategy and our approach. We observe that with selective poisoning, the behavior of the poisoned model is still similar between clean and backdoor data, showing the attack’s stealthness against STRIP detector.

Figure: Figure 5: Performance against STRIP.
Refer to caption: x4.png

Fine-pruning. We evaluate our attack’s resistance to Fine-pruning, a backdoor mitigation method. Given a benign sample, it assumes that inactivated neurons are responsible for backdoor features and gradually prunes these neurons. Figure 6 shows the clean accuracy and attack success rate during this process. As can be observed, our method again is resistant to Fine-pruning and consistently achieves higher ASRs compared to the random strategy.

## 6 Conclusion

This paper studies the threat of a backdoor attack under an extremely constrained setting: the adversary can only have access to samples from one single class and perform a clean-label backdoor attack.
In this threat model, we propose novel approaches that find “hard samples” to inject the trigger patterns by utilizing pretrained models or OOD datasets.
Empirical results show that our method can achieve very high attack success rates, compared to the baselines, and can bypass several representative defenses. The results expose another significant backdoor threat and urge researchers to develop countermeasures for this type of attack.

## Appendix A Experimental Setup

### A.1 Dataset Details

We conduct experiments on two widely used benchmark datasets:

- •
CIFAR10 [] contains images from $10$ classes, with $50,000$ samples for the training set and $10,000$ samples for the test set. We poison class 0 0, which has $5,000$ images.
- •
GTSRB [] contains images from $43$ classes of traffic sign images, including $39,209$ samples for training and $12,630$ samples for test. We poison class $1$, which has $1500$ images.

For OOD strategy, we train the surrogate model on TinyImagenet []. It has $200$ classes, with $1,000,000$ training images and $10,000$ validation images. There is no overlap between the label set of TinyImageNet and CIFAR10 or GTSRB.

We also consider PubFig (^1^11[https://www.cs.columbia.edu/CAVE/databases/pubfig/](https://www.cs.columbia.edu/CAVE/databases/pubfig/)), a dataset that consists of public figures’ faces. We select $50$ classes with the highest number of images and divide them into $5,212$ images for training and $1,312$ images for validation. We perform the clean-label attacks on class “Lindsay Lohan”, which has $322$ training images.

### A.2 Attack Details

For BadNets [], a checkerboard pattern [] is added to the image. For Blended [], we implant a Hello Kitty image with the blended rate $\alpha=0.2$. Also, we evaluate our strategy on SIG [], a clean-label attack, with $\Delta=20$ and $f=6$.

### A.3 Training Details

We train ResNet18 and VGG19 for $300$ epochs with SGD optimizer, learning rate $0.01$, and cosine scheduler. For CIFAR10 and GTSRB, the image size is $32\times 32$. For PubFig, we resize the input to $224\times 224$.

## Appendix B Additional Results

### B.1 Additional Results of Our Strategy against Backdoor Detection Defenses

In this section, we conduct experiments with additional backdoor defenses.

Figure: Figure 7: Performance against Neural Cleanse
Refer to caption: x6.png

Neural Cleanse. This method assumes the trigger is patch-based and searches for patterns that change the prediction of the model to a specific label. If there is a label with a pattern of a significantly small norm, the model is identified as being attacked. Figure 7 shows that when $10\%$ of the target class selected by our strategy is poisoned, the Anomaly Index is still less than $2$, which means the attack still stays stealthy under Neural Cleanse.

We also evaluate with defenses that identify and eliminate poisoned samples in the training data. We report Elimination Rate (ER) and Sacrifice Rate (SR), which are the rates of poisoned samples being correctly detected and benign samples being wrongly removed. For a defense to be effective, it must have high ER and low SR at the same time. Table 5 shows the performance of Activation Clustering [], Spectral Signature [], and SPECTRE [] on models attacked by our strategy. We can observe that Activation Clustering fails to detect the attack, while Spectral Signature and SPECTRE sacrifice a high number of clean samples.

**Table 5: Elimination rate (ER) and Sacrifice Rate (SR) of backdoor detection methods, including Activation Clustering (AC), SPECTRE, and Spectral Signature (SS) on ResNet18/CIFAR10 with $10\%$ of class 0 0 is attacked by SIG.**
|  |  | AC | SPECTRE | SS |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ER | SR | ER | SR | ER | SR |
| BadNet | Random | 0.00 | 0.00 | 35.20 | 50.07 | 32.80 | 50.09 |
| Pretrained | 0.00 | 3.28 | 40.80 | 50.05 | 58.00 | 49.96 |  |
| OOD | 0.00 | 0.00 | 28.00 | 50.11 | 38.80 | 50.06 |  |
| Blended | Random | 0.00 | 2.86 | 77.20 | 49.86 | 75.60 | 49.87 |
| Pretrained | 0.00 | 3.03 | 55.60 | 49.97 | 53.20 | 49.98 |  |
| OOD | 0.00 | 0.00 | 58.40 | 49.95 | 53.60 | 49.98 |  |
| SIG | Random | 0.00 | 0.00 | 39.20 | 50.05 | 46.40 | 50.02 |
| Pretrained | 0.00 | 0.00 | 38.00 | 50.06 | 50.80 | 50.00 |  |
| OOD | 0.00 | 0.00 | 36.40 | 50.07 | 35.20 | 50.10 |  |

We provide in Table 6 the performance on ResNet18/CIFAR10 against more recent defenses, including Anti-Backdoor Learning (ABL) [] and Channel Lipschitz Prunning (CLP) []. While the original attacks (with random selection) do not generally work against ABL (ASRs are lower than 30%), our strategy can boost their success rates to 35-68%. For CLP, the results show that our method is resilient to this pruning defense (49-85%), whereas the success rates of the random selection strategy are significantly lower (13-30%).

**Table 6: The performance on ResNet18/CIFAR10 against recent defenses**
| Defense | Selection Method | BadNet | Blended |  |  |
| --- | --- | --- | --- | --- | --- |
| Acc | ASR | Acc | ASR |  |  |
| ABL | Random | 84.28 | 16.57 | 78.66 | 28.79 |
| Pretrained strategy | 81.07 | 68.00 | 79.15 | 35.49 |  |
| CLP | Random | 93.55 | 13.07 | 93.54 | 28.87 |
| Pretrained strategy | 93.97 | 85.83 | 94.23 | 49.33 |  |

### B.2 Face Classification Task

We also evaluate our strategies on a face classification task with the PubFig dataset. Table 7 illustrates the performance of Single-class OOD strategy and Pretrained strategy with self-supervised features. For OOD strategy, we employ the same architecture of the victim model as the surrogate model. The results show that our strategies are effective in boosting clean-label attacks on face recognition tasks, posing a serious security threat. Selecting samples with self-supervised models increases the success rate of clean-label attacks significantly, showing the effectiveness of self-supervised features. On the other hand, since there is a high distribution shift from ImageNet and TinyImageNet to PubFig, the improvements of Supervised Pretrained strategy and OOD strategy are not as high as Self-supervised Pretrained strategy.

**Table 7: The attack success rate of our strategies with SIG on ResNet18/PubFig with $20\%$ and $50\%$ poisoning rate.**
| Method | 20% | 50% |
| --- | --- | --- |
| Random | 13.20 | 29.31 |
| Self-supervised models | 24.92 | 62.09 |
| Supervised models | 19.26 | 40.37 |
| Single-class OOD | 16.27 | 48.27 |

### B.3 The Number of Classes in the OOD Dataset

**Table 8: ASR of Multiple-class OOD strategy when varying the number of classes in the OOD dataset.**
| Dataset | Number of<br>labels | ResNet18 | VGG19 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 5% | 10% | 20% | 5% | 10% | 20% |  |  |
| CIFAR10 | 10 | 65.11 | 80.76 | 88.79 | 50.81 | 65.80 | 78.28 |
| 200 | 65.89 | 71.26 | 77.18 | 39.13 | 54.62 | 66.20 |  |
| GTSRB | 10 | 43.56 | 47.59 | 52.50 | 27.93 | 28.98 | 31.94 |
| 200 | 46.39 | 48.36 | 54.00 | 23.26 | 27.09 | 24.32 |  |

We study the performance for the OOD strategy with the different number of classes in the OOD dataset. The results of Multiple-class OOD strategy using SIG triggers are illustrated in Table 9.
In general, these observations suggest that increasing the number of data labels in the OOD dataset does not improve the attack effectiveness.

### B.4 Effectiveness of Our Method on Narcissus

Similar to our method, the Narcissus attack [] can also operate under the threat model scenario where the attacker only has access to training data from the target class.
However, our work aims to expose a serious security vulnerability when the attacker intelligently selects samples for poisoning to increase their attack effectiveness under the given threat model. This sample selection strategy allows the attacker to use any existing triggers in the clean-label setting.
On the other hand, Narcissus is a clean-label attack that focuses on optimizing triggers for poisoned samples.
In other words, our work is orthogonal to Narcissus since the proposed sample selection method can be used in Narcissus to intelligently (instead of randomly) select samples for poisoning to achieve better attack effectiveness, similar to what we demonstrate for BadNet, SIG, or Blended. To verify this, we perform experiments where the base attack is Narcissus and assess its performance with different sample selection approaches, including random and easy or hard samples found by a self-supervised model, to poison 25 samples on CIFAR10. Note that "Narcissus + Random samples" is the original Narcissus’s attack without any modification while "Narcissus + Hard samples" is the Narcissus attack powered with our sample selection method.

In Table 10, we report the mean success rate of three times attacking a ResNet18 model using the Narcissus approach with different sample selection strategies. As we can observe, the unmodified Narcissus’s attack can only achieve 56.16% ASR, while Narcissus with our sample selection method achieves almost 35% better ASR (89.65% ASR). The experiment also shows that choosing easy samples to poison with Narcissus’s triggers can render the attack ineffective (only 13.06% ASR). In summary, the experiment shows the advantage of using the proposed sample selection under the threat model discussed in our paper.

**Table 10: Narcissus’s performance with different sample selection approaches**
|  | ASR |
| --- | --- |
| Narcissus + Easy samples | 13.06 |
| Narcissus + Random selection | 56.16 |
| Narcissus + Hard samples | 89.65 |

### B.5 The Transferability of Our Method to Different Architecture

We provide additional results for other architecture in Table 11. We perform clean-label backdoor attacks on ViT [] or DeiT [] with 500 samples selected by a self-supervised pre-trained ResNet50 model []. As can be observed, our strategy improves the attack success rate of the random selection strategy by a large margin (more than $30\%$ for BadNet and $10\%$ for Blended and SIG), demonstrating that the selected samples still help to achieve effective attacks on various architectures.

**Table 11: The performance of Pretrained strategy on ViT/CIFAR10 and DeiT/GTSRB models**
|  | Method | BadNet | Blended | SIG |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | ACC | ASR | ACC | ASR | ACC | ASR |  |
| ViT/CIFAR10 | Random | 98.78 | 11.26 | 98.80 | 23.10 | 98.72 | 35.55 |
| Pretrained strategy | 98.84 | 47.20 | 98.79 | 38.63 | 98.73 | 45.00 |  |
| DeiT/GTSRB | Random | 97.85 | 6.66 | 98.42 | 26.90 | 98.04 | 31.86 |
| Pretrained strategy | 97.66 | 7.49 | 97.91 | 54.66 | 97.91 | 38.31 |  |

### B.6 The Effect of Class Imbalance

To study the effect of our method and class imbalance, we sort the classes by the number of samples descendingly and launch the backdoor attacks (with SIG trigger and 10% poisoning rate) on the classes at 1st, 14th, 28th, and 43rd sorted positions (as target classes), whose original class indices are 1, 11, 16, and 37, respectively. The results in Table 12 show that our method can consistently boost the success rates of the original attacks (random selection) on target classes from a broad spectrum of sample sizes.

**Table 12: The performance of Pretrained strategy on classes with different sizes.**
|  | 1st | 14th | 28th | 43rd |
| --- | --- | --- | --- | --- |
| Number of samples | 1500 | 900 | 300 | 150 |
| ASR with random selection | 48.07 | 43.87 | 1.66 | 16.30 |
| ASR with Pretrained strategy | 57.12 | 44.70 | 24.36 | 27.75 |

### B.7 The Effect of Different Values of k 𝑘 k italic_k

One of our approaches employs $k$-NN to select samples that have the highest distances to their neighbors in the feature space. We vary the number $k$ of neighbors and perform SIG attack with $10\%$ poisoning rate on CIFAR10 and report the success rate in Table 13. The results imply that $k$-NN is more effective when the value of $k$ is small; we conjecture that $k$NN with smaller $k$ takes into account the local property of the dataset, increasing the discrepancy between the score of hard samples (which are outliers) and easy samples. In the extreme case where $k=10000$ (meaning we select samples that are far from the mean), while the success rate is still higher than that of random selection, it is lower than that of a smaller k. Consequently, this suggests the use of a small k value when performing the attacks.

**Table 13: The performance of Pretrained strategy with different values of $k$.**
|  | k=5 | k=50 | k=500 | k=1000 | k=5000 | k=10000 | Random |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ASR | 82.35 | 80.59 | 78.92 | 79.41 | 77.34 | 74.76 | 60.54 |

### B.8 Access only a portion of the data from the target class.

We study the effectiveness of our method under the setting where the attacker partially accesses the target class’s data. We conducted experiments using different selection strategies on CIFAR-10 with accessible data comprising $20\%$ and $50\%$ subsets of the target class’s data (Table 14). The attack used was SIG, and the poisoning rate was $5\%$. We can observe that as the size of the accessible data decreases, the corresponding ASR also decreases. However, the method’s effectiveness is still significantly better than that of random selection. The attackers, using our method, can still launch a very harmful attack ($68.92\%$) even with only $20\%$ of the target class’s data.

**Table 14: The performance of Pretrained strategy with partial access to the target class.**
|  | 20% | 50% | 100% |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Acc | ASR | Acc | ASR | Acc | ASR |
| Random | 94.63 | 45.27 | 94.80 | 43.37 | 94.69 | 50.28 |
| Pretrained strategy | 94.44 | 68.92 | 94.73 | 70.65 | 94.71 | 76.35 |

## Appendix C Limitations

As discussed, we study the threat model in which the attacker acts as a data provider that is responsible for a single data class and we propose strategies to select samples for more effective clean-label attacks. We do not include dirty-label attacks that change the semantic label of the input since it is easier to detect in this threat model. We also do not study the effect of our strategies on the defense that mitigates the backdoor in training data.

Our methods select hard samples in the target class to inject trigger, and it would be interesting to study the combination of our strategies and the attack that perturbs the input to make it difficult to learn []. We leave these to future works.

## Appendix D Societal Impacts

Our work proposes a novel threat model, where the adversary only has access to the target class that they want to attack.
In this constrained setting, we show that the attacker can perform selective poisoning to improve the attack success rate of existing clean-label attacks.
We hereby raise awareness of a new potential risk in developing a machine learning system in practice.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors should clarify the novelty, choice of limited datasets, the use of older defense strategies, and the dependency on pretrained models.
soundness: 2
strengths: 1. The paper improves traditional clean-label backdoor attacks by proposing a threat model that is more applicable in real-world scenarios.

2. The method is claimed to achieve higher attack success rates with a lower poisoning rate, showcasing efficient use of resources.
summary: The paper proposes a method for enhancing clean-label backdoor attacks on deep neural networks. Unlike traditional clean-label attacks that apply triggers randomly, this approach selectively poisons challenging samples within the target class, boosting attack success rates with fewer poisoned samples. The authors introduce two strategies: using pretrained models to identify "hard" samples and leveraging out-of-distribution data for sample selection. Tested on CIFAR-10 and GTSRB datasets, this method outperforms random poisoning and is resilient against popular defenses like STRIP and Neural Cleanse, highlighting a need for stronger countermeasures against selective clean-label attacks.
weaknesses: 1. In my opinion, the method primarily introduces a data selection strategy, which lacks sufficient novelty.

2. The evaluation is conducted only on CIFAR-10 and GTSRB datasets, limiting insight into the method's performance across other dataset types and application domains.

3. The paper primarily tests against older defense strategies. Implementing more recent and sophisticated defenses, including adaptive methods like sample-specific anomaly detection, would strengthen the evaluation.

4. The pretrained model strategy relies on the availability of pretrained models in similar domains, which may not always be accessible in real-world applications.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the questions in weaknesses.
soundness: 3
strengths: The main strengths of the paper lie in its studied threat model, proposed sampling strategies and experimental evaluation.

- I think that the proposed threat model is important as it exposes yet another backdoor threat where an attack only needs access to the data of the target class. The demonstrations that existing backdoor attacks under this threat model are not satisfactorily effective are an important contribution of the paper.
- The proposed sampling strategies are novel, especially when they could be used with existing backdoor attacks, such as BadNets, SIG, Narcissus, etc…) to boost their backdoor performances under the studied threat model. It’s also interesting finding where the effectiveness of the proposed strategies even when there is less and less assumptions on the pretrained models or the OOD datasets. 
- The paper includes thorough analysis of the proposed strategies, demonstrating their effectiveness when they’re used with several existing backdoor attacks across different datasets. The evaluation also shows the effectiveness against several defenses.
summary: The paper studies clean-label backdoor attacks in a very constrained setting, where the attacker only needs access to the training data from the target class and has no prior knowledge of the victim model, training process, and the other classes, and focuses on data-selection strategies to boost the performance of existing clean-label attacks in this constrained setting. The proposed data selection strategies include (1) the use of a pretrained model (when such exists) or (2) the use of an OOD dataset (when the pretrained model is not available) to train a surrogate model.  The experimental results demonstrate the proposed strategies significantly enhance the ASR and several existing clean-label backdoor attacks, compared to random selection strategies. In addition, the paper demonstrates that the proposed strategies are resilient against several existing defenses.
weaknesses: I find that the paper has the following concerns:
* The Narcissus results, while interesting, are different from what reported in their original paper. Can the authors explain why there are such differences? 
* The OOD approach rely on out-of-distribution data but it’s not clear how this dataset could be obtained, or whether there are any specific requirements of the datasets to maintain the effectiveness of the attacks?
* Assuming that the victim could distribute the target class data collection to multiple sources, how does the proposed attacks perform in this case? 
* Do the authors have any suggestions about potential mitigation approaches against the proposed attacks in the studied threat model?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - Why would an attacker use the OOD strategy proposed in section 4.4, as it requires training a surrogate model and appears to work worse than using a pretrained model?
- Why use a latent space clustering approach instead of using the loss from a pretrained zero-shot image classifier like CLIP?
- Why use VICReg instead of a more general feature extractor like CLIP?
- Where are the training settings used in experiments adapted from?

References

[1] Alexander Turner, Dimitris Tsipras and Aleksander Madry. "Label-Consistent Backdoor Attacks.", 2019.  

[2] He, Kaiming, et al. "Deep Residual Learning for Image Recognition," 2016.  

[3] Gao, Yinghua, et al. "Not All Samples Are Born Equal: Towards Effective Clean-Label Backdoor Attacks.", 2023.
soundness: 3
strengths: - The proposed method is a widely applicable technique to enhance to clean label attacks.
- The experiments do a good job differentiating the surrogate model from victim model and therefore the attack shows convincing transferability.
summary: The paper proposes a method for improving the effectiveness of clean-label attacks. It introduces a threat model where the attacker only has access to data belonging to a specific class and has no knowledge about other classes in the dataset. The paper proposes a method for using samples with hard to learn features to create poison-efficient clean label attacks. The proposed method finds these samples by clustering the latent features of a surrogate model. The paper explores using a pretrained model and a model trained on OOD data as the surrogate model. The paper evaluates the clean-label attack against backdoor defenses and data cleaning methods.
weaknesses: - The paper trains for 300 epochs which is significantly longer than it should take to train the model on CIFAR-10/GSTRB [1,2] and makes attack success due to over-fitting very likely. Around 100 epochs seems to be more standard. Ideally, to simulate a competent defender early stopping should be employed. I.e. stopping the run when validation loss plateaus.

- The experiments use very weak baselines. The paper only evaluates how the method performs compared to random sampling. At minimum the paper should compare against [3]. Especially because [3] could easily be adapted to adhere to this paper's threat model by using a pretrained model. Therefore, the experiments are not sufficent to jusifty that the proposed method is stronger than a slightly adapted version of [3].

- The paper claims that it's threat model represents *"the **most** constrained data-poisoning threat."* However, there are other perfectly reasonable threat models that would make this attack unrealistic. For example, an opportunistic attacker that doesn't get to choose the subset of samples in the dataset they are able to manipulate.

- When evaluating the attack against defenses the paper does not describe the hyperparameter settings used by each defense nor how those settings were derived.

Minor:
- Bolding of best methods or aggregation would make Tables 2 and 3 more interpretable.
- There are many typos in the manuscript.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors should include more related works and advanced baselines in their paper.
2. The authors should better clarify their main contributions than those introduced in existing works.
3. The authors should avoid overclaims.
4. The authors should conduct more comprehensive experiemnts. 

More details are in the 'Weakness' section.
soundness: 2
strengths: 1. The paper is easy to follow to a large extent.
2. The motivation is clear and with empirical support.
3. The paper introduces a clean-label backdoor attack that works effectively in a constrained scenario where the attacker has limited data access (only one target class). This approach is realistic for scenarios with privacy or geographical constraints, enhancing the practical relevance of the attack model.
summary: This paper explores a practical scenario for clean-label backdoor attacks, where an attacker’s access is limited to a single class of data within a decentralized training setup. This constrained threat model reflects real-world data collection challenges, such as privacy restrictions and geographical limitations. To enhance poisoning efficiency under these conditions, the paper introduces two sample selection methods specifically designed for this limited-access scenario.
weaknesses: 1. Numerous studies [1,2,3,4,5,6,7,8] have addressed sample selection in backdoor attacks, several of which [6,8] specifically focus on sample selection for clean-label backdoor attacks. Omitting these key relevant works is a significant oversight and should be addressed to ensure a comprehensive discussion of the literature.
2. The novelty of this paper is limited, as it leverages a pre-trained model to identify "hard samples" for poisoning—a concept already explored in several studies [6,7,9]. However, the distinctions between this approach and prior work are not clearly articulated.
3. The first contribution claimed by this paper is the introduction of a new backdoor threat model, where an attacker, acting as a data supplier, has access only to the target class data yet can still execute effective clean-label backdoor attacks. However, previous studies [10,11] have already examined this threat model in depth, providing detailed discussions on "Why are dirty-label attacks more effective than clean-label attacks?" Consequently, the originality and contribution of this paper raise some concerns.
4. The discussion of backdoor attacks and defenses in the related work sections of this paper is outdated. 
5. There are some potential over-claims. For example, Line 156-159: Accessing only samples from a single non-target class is more difficult setting than yours.
6. Missing some important experiments.
- Main Experiments
  - The authors should also include the results of methods using all training samples for references, although you have a different setting.
  - It would be better to include the results of Narcissus here instead of in the appendix.
  - I would like to see whether the proposed method is also effective for untargeted clean-label backdoor attacks (e.g., UBW-C in [12])
- The Resistance to Defenses: The authors should evaluate their methods on more advanced backdoor defenses (such as [13, 14] and their baselines). 




**References**
1. Computation and data efficient backdoor attacks
2. Explore the effect of data selection on poison efficiency in backdoor attacks
3. Boosting backdoor attack with a learnable poisoning sample selection strategy
4. A proxy-free strategy for practically improving the poisoning efficiency in backdoor attacks
5. Minimalism is King! High-Frequency Energy-based Screening for Data-Efficient Backdoor Attacks
6. Large Language Models are Good Attackers: Efficient and Stealthy Textual Backdoor Attacks
7. Confidence-driven Sampling for Backdoor Attacks
8. Clean-label Backdoor Attacks by Selectively Poisoning with Limited Information from Target Class
9. Not all samples are born equal: Towards effective clean-label backdoor attacks
10. Efficient backdoor attacks for deep neural networks in real-world scenarios
11. Narcissus: A practical clean-label backdoor attack with limited information
12. Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection
13. Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features
14. IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistency

Meta-review: This paper examines clean-label backdoor attacks in a highly constrained setting, where the attacker only has access to training data from the target class and lacks prior knowledge of the victim model, training process, or other classes. In this senario, the authors manage to propose a data-selection strategy to improve the performance of clean-label attacks, like using a pre-trained model or OOD trained surrogate model. Experimental results demonstrate the methods effectiveness.

Strength:

1. The paper introduces a clean-label backdoor attack that works effectively in a constrained scenario where the attacker has limited data access (only one target class). This approach is realistic for scenarios with privacy or geographical constraints, enhancing the practical relevance of the attack model.

2. Comprehensive experiments to demonstrate their effectiveness.

Weakness:

Evaluation settings are not enough. For example, they only test ResNet-18 and VGG. If transformers are also involved, the impact will be greater.

Most reviewers show postive attitude towards this paper. The only remaining conerns are still need to adding empirical settings. However, regarding that the paper has already done a lot evaluations. I think missing some settings are acceptable. Therefore, I tend to accept this paper.

Decision: Accept (Poster)