Protecting against simultaneous data poisoning attacks

## Abstract

Abstract Current backdoor defense methods are evaluated against a single attack at a time. This is unrealistic, as powerful machine learning systems are trained on large datasets scraped from the internet, which may be attacked multiple times by one or more attackers. We demonstrate that simultaneously executed data poisoning attacks can effectively install multiple backdoors in a single model without substantially degrading clean accuracy. Furthermore, we show that existing backdoor defense methods do not effectively prevent attacks in this setting. Finally, we leverage insights into the nature of backdoor attacks to develop a new defense, BaDLoss, that is effective in the multi-attack setting. With minimal clean accuracy degradation, BaDLoss attains an average attack success rate in the multi-attack setting of 7.98% in CIFAR-10 and 10.29% in GTSRB, compared to the average of other defenses at 64.48% and 84.28% respectively. 1 1 1 We open-source our code to aid replication and further study, available on GitHub: https://github.com/shoaibahmed/mapd_backdoors.

## 1 Introduction

Many deep learning applications use large-scale datasets obtained through web scraping with minimal curation. These datasets are vulnerable to attackers, who can easily inject data that alters the behavior of models trained on these datasets.  Carlini et al. [] demonstrated that poisoning real-world, large-scale datasets is a feasible threat due to their distributed nature.

Among the various data poisoning threats, the creation of model backdoors is particularly insidious. By modifying only a small number of examples in a dataset, adversaries can make a trained model sensitive to highly specific features. The adversary can then control the model’s outputs by injecting these features into otherwise innocuous images [; ; ; ; ] – despite the model appearing benign during regular evaluation.

To the authors’ knowledge, all prior works only evaluate against one backdoor attack at a time  [; ; ; ].
Real-world poisoned data sets can potentially include multiple backdoors of different types, as large-scale distributed datasets [] have a very low barrier for adding data, so multiple attackers could easily add poisoned examples aimed at installing multiple backdoors. Even a single attacker could deploy multiple attacks. This setting is both more realistic and makes it substantially more challenging to defend against attacks in practice.

Therefore, in this work we propose the problem setting of defending against *multiple simultaneous backdoor attacks*. We demonstrate that a poisoned dataset can introduce multiple backdoors into a target model without substantial clean accuracy degradation, and that existing defenses fail to effectively defend in this setting. As an additional contribution, we identify a common property of poisoned images in datasets – that they demonstrate anomalous loss trajectories as a result of using unnatural features for classifcation. Correspondingly, we propose a defense called BaDLoss meant to be robust to multiple simultaneous backdoor attacks. We demonstrate that new defense performs well compared to other defenses, and retains performance in the synthetic single-attack setting.

## 2 Related Work

Since our work is concerned with backdoor detection and prevention, we will briefly discuss both attacks as well as defenses presented in the past.
Most work on backdoors, including ours, focuses on image datasets, although several recent works also explore poisoning of language models [; ].

### 2.1 Backdoor Attacks

While attacks exist where the adversary provides a secretly backdoored model to the victim [; ], we focus on the data poisoning setting, where the attacker manipulates a subset of the data to later control the model by injecting a feature of their choice.

Early data poisoning attacks modified both training images and their corresponding labels, adding a small stamp [], or overlaying an image [] or pattern [], then changing the label of altered images to a target class, so the model associates the injected feature with the target class. Later attacks invisibly warp images [] or overlay image-specific invisible perturbations []. Some attacks do not even change images’ labels. Clean-label attacks alter training images with features that the learning algorithm preferentially detects [; ] without changing any image classes. More sophisticated strategies use realistic reflection overlays [] or random-noise patterns [; ] that subtly induce the model to learn the desired behavior.

### 2.2 Backdoor Defenses

Various defense mechanisms have been proposed to mitigate model backdoors. One approach, known as trigger reverse engineering, assumes that a trigger mask with very few pixels causes the model to misbehave. Neural Cleanse [] and variants  [; ; ; ] learn the trigger mask from a trained model, while others employ techniques like activation patching [] or generative modeling [] to identify the trigger, after which neurons which activate on the trigger can be pruned, or a number of other cleaning methods can be applied.

Backdoors can also sometimes be detected by directly examining model activations. Activation Clustering and Spectral Signatures [] assume clean and backdoored examples demonstrate distinctive patterns in the model’s activation space. Further work examines specific properties of activation space and identifies neurons that highly influence model behavior on backdoored examples  [; ; ]. These neurons can then be pruned [], or the model fine-tuned appropriately [; ; ].

As removing backdoors from a trained model is challenging [], many defenses indirectly prevent the backdoor from being learned at all [; ; ; ; ]. Anti-Backdoor Learning [] maximizes the loss of identified poisoned examples during training, making the model ignore poisoned features, a method used by other detection methods []. Finally, other approaches detect inputs that cause misbehavior in the output of the models [; ; ] or exploit the unique frequency spectra of attack to identify attack inputs [], then filter them out for retraining or at deployment.

## 3 Multiple Simultaneous Attacks

To the authors’ knowledge, no prior work evaluates on multiple backdoor attacks simultaneously. Therefore, we will begin with a demonstration that this setting is feasible in practice without substantial clean-accuracy degradation.

We compare against a wide variety of classic and state-of-the-art backdoor attacks and defenses in the literature. Following recent work [; ; ; ], we use the standard computer vision datasets: CIFAR-10 [] and GTSRB [], which capture variance both in image size (32x32 for CIFAR-10, versus variable image sizes up to 250x250 for GTSRB) as well as class distribution (CIFAR-10 has 10 balanced classes, GTSRB has 43 imbalanced classes).
We use CIFAR-10 as a development set to tune defenses and, importantly, apply the tuned methods directly to GTSRB without any further tuning for a more accurate assessment of the methods’ performance in realistic settings.

### 3.1 Threat Model

In our threat model, we assume that the attacker(s) have control only over a portion of the training dataset which they can view and modify. The size of this modified fraction is referred to as the poisoning ratio $p$. The defender has complete control over the training process, and the attacker has no control. However, the defender receives a labeled dataset from an unvalidated external source. This corresponds well to an attack setting in which the victim trains a model using their own internal, well-tested code, but relies on a large, externally sourced dataset that cannot be manually quality-checked. These assumptions align with threat models commonly considered in prior works [; ; ; ].

Additionally, we assume the defender has access to a small set of guaranteed clean examples (here 250 examples). Many defense methods [; ; ; ; ; ; ; ; ] assume access to bona fide clean examples, as trusted human labor could manually generate or filter a subset of the dataset.

When evaluating, we primarily evaluate retrained model’s accuracy on the clean test set, and the success rate of any attacks present in the data on the retrained model. This is consistent with defense evaluations considered in prior works [; ; ].We define clean accuracy as the retrained model’s performance on the test set. Attack success rate (ASR) is evaluated on the full test set excluding the target class, with the backdoor injected into every example to verify that the attack successfully changes the model’s predictions. Full training details are available in Section A in the Appendix.

### 3.2 Attacks Considered

Figure: (a)
Refer to caption: extracted/5810100/Figures/attacks/patch.png

All considered attacks are visualized in Figure 1. All attacks change the images’ labels except for the sinusoid attack, which is a clean-label attack. All attacks target different labels, and no image is attacked twice (so each image contains at most one trigger). The Narcissus [] attack is not used in GTSRB, as the optimized trigger is challenging to generate on variable-sized images.

In this work, we evaluate against the following attacks:

- •
Patch: Adds a small patch in the corner of attacked images [].
- •
Single-Pixel: Adds a single pixel in the corner of attacked images [].
- •
Blend-Random: Blends a random noise pattern with attacked images [].
- •
Blend-Pattern: Blends a dimple pattern with attacked images [].
- •
Frequency: Adds peaks in attacked image’s discrete cosine transform [].
- •
Sinusoid: Adds sinusoidal stripes to attacked images of a single class [].
- •
Narcissus: Adds a learned trigger to a very small number of attacked images [].

In the multi-attack setting, all attacks are present at their full poisoning ratio. Full attack specifications are available in Section B.1 in the Appendix .

### 3.3 Efficacy of Multiple Attacks

Figure: Figure 2: Evaluation of Multi-attack Setting. Clean accuracy (blue) should be high, attack success rate (red) should be low. When multiple attacks are simultaneously deployed, clean accuracy suffers slightly but is generally preserved at a high level. All attacks are simultaneously learned by the final model. We also plot the attack success rate against single attacks to illustrate positive and negative interference between attacks.
Refer to caption: extracted/5810100/Figures/Multiattack_1.png

Our results are visualized in Figure 2 (exact numerical results available in Table 2 in the Appendix). In both CIFAR-10 and GTSRB, the presence of multiple attacks does not substantially degrade clean accuracy. Additionally, in both settings, all attacks are learned simultaneously to a reasonably high degree. Notably, some attacks changed in performance substantially between the single-attack and multi-attack setting, such as the single-pixel attack, which is completely unlearned when isolated in GTSRB, but which is learned to over 90% attack success rate in the multi-attack setting. This indicates that even attack-development research should consider the multi-attack setting, as it can have dramatic impacts on an attack’s viability in realistic threat models.

We take these results as validating the importance of the multi-attack setting. If one or more attackers, by controlling small parts of a large, internet-scraped dataset, can cause a target model to respond to many different types of backdoor triggers without substantially degrading performance on clean data in a way that a defender might notice, then this setting is an vitally important one. However, existing defenses may be able to effectively adapt to this new setting.

### 3.4 Defenses Considered

In this work, we evaluate the following defenses:

- •
Neural Cleanse [] Neural Cleanse learns a minimum-magnitude mask per class to reclassify every image in the dataset as that class. Masks with substantially-below-median magnitude are considered backdoor triggers.
- •
Activation Clustering [] Activation Clustering decides that a backdoor is present if the last layer’s activations for a given class can be clustered into two classes, then removes the smaller cluster for retraining.
- •
Spectral Signatures [] Spectral signatures uses a singular value decomposition of last layer activations per class, and removes the 15% of data with the highest value in the first singular dimension for retraining.
- •
Frequency Analysis [] Frequency analysis identifies poisoned examples by building a classifier on the discrete cosine transforms of synthetic images with a fixed set of hardcoded backdoor-like features.
- •
Anti-Backdoor Learning [] Anti-backdoor learning identifies examples with particularly low loss after a few epochs of training and removes them before continuing training. After training is concluded, a few epochs of loss maximization on those examples removes sensitivity to the backdoor.
- •
Cognitive Distillation [] Cognitive distillation learns a mask for each example that yields the same activations, treating examples with particularly low magnitude as backdoors, then uses ABL’s unlearning method.

Full defense specifications are available in Section B.2 in the Appendix.

### 3.5 Existing Defense Performance

Figure: Figure 3: Defense Performance in the Multi-attack Setting. All evaluated defenses demonstrate failures when evaluated in the multi-attack setting. This holds across datasets. The frequency analysis defense method [] achieves low attack success rate on CIFAR-10, but removes so much data that clean accuracy is reduced to random chance levels. Spectral Signatures provides the best defense overall on CIFAR-10, but fails on GTSRB, and dramatically underperforms BaDLoss (see Figure 6)
Refer to caption: extracted/5810100/Figures/Multiattack_2.png

Our results are visualized in Figure  3 (exact numerical results available in Table 2 in the Appendix). Except for the frequency analysis defense in CIFAR-10, which discards so much data that the clean accuracy is irrepairably degraded, all defenses in the multi-attack setting fail to defend against a variety of attacks. Even the best performing defense, Spectral Signatures in CIFAR-10, has three attacks performing at nearly-perfect attack success rate.

Likely due to making assumptions incompatible with the complexity of the multi-attack setting, current backdoor defenses, when used with their default parameters, do not effectively defend against these relatively standard attacks when deployed simultaneously. In order to make progress in the multi-attack setting, new defenses must be developed.

## 4 Developing a New Defense

Figure: Figure 4: BaDLoss Overview. (1) The defender tracks clean examples (“probes”) in the training set across multiple short training runs. (2) Every example gets an anomaly score based on its average distance from the bona fide clean examples. The farthest examples are marked as potential backdoors. (3) The defender retrains the model, excluding any examples identified as anomalous. (4) The defender deploys the more robust model.
Refer to caption: extracted/5810100/Figures/Figure_1.png

To defend in the multi-attack setting, we first note that many assumptions made by existing defenses, while accurate in the single-attack setting, no longer hold when multiple simultaneous attacks occur. For instance, neural cleanse assumes that the attacked class will have an anomalously low mask magnitude relative to the other classes, but if sufficiently many classes are attacked, the median mask magnitude may be low enough that no classes look anomalous.

We look to attack-agnostic metrics that could indicate the presence of a backdoor. Several methods exist that analyze the loss of training examples to determine whether such examples are installing a backdoor. Anti-Backdoor Learning assumes that backdoored examples achieve a lower loss more quickly than normal examples [], and Khaddaj et al. [] similarly claim that backdoor images have the strongest features and are thus learned much more easily. Other works [] treat backdoors as anomalous examples which conflict with natural image features, and are therefore harder to learn. Such claims are correct in context, but miss the whole picture. We synthesize these ideas: images can exhibit either faster or slower training dynamics compared to a dataset’s natural images, depending on the nature of the backdoor. As a defender cannot know what backdoor methods will be deployed, a defense must be effective against backdoors that induce any unusual training dynamics.

Therefore, we introduce BaDLoss (Backdoor Detection via Loss Dynamics), visually demonstrated in Fig. 4. To generalize beyond assumptions that all backdoors are simply easier or harder to learn, BaDLoss leverages a small number of bona fide clean examples that are used as part of the training set. A model is subsequently trained using all available examples, including the reference clean examples, while simultaneously tracking the loss associated with each instance in the training set. The added bona fide clean examples provide reference trajectories that allow us to define an anomaly score for each example in the training set based on the distance to the clean trajectories.

### 4.1 BaDLoss: Backdoor Detection via Loss Dynamics

Our approach draws inspiration from the Metadata Archaeology via Probe Dynamics (MAP-D) technique [], which examines per-example loss trajectories in a training set to remove unusual or mislabelled examples. Notably, other studies have also harnessed training dynamics for various objectives [; ; ].

BaDLoss relies on the observation that backdoor triggers require the model to learn features absent in a typical dataset. These features are always aberrant, as the attacker must be able to control and inject the feature into an arbitrary image in order to successfully control the backdoored model after deployment. Consequently, such examples exhibit distinct loss dynamics, illustrated in Fig. 5. Unlike previous defenses, BaDLoss uses entire loss trajectories to identify backdoor samples.

Figure: Figure 5: Average clean trajectory compared to average attack trajectories in CIFAR-10, single-attack setting, 50 epochs. Top row, left to right: 4-pixel patch, single-pix patch, random blended, fixed pattern blended. Bottom row, left to right: Sinusoid blended, narcissus, frequency attack. All backdoor attacks exhibit distinct learning dynamics from clean examples. However, some are learned faster while others are learned slower, making the inductive bias of previous methods [; ; ] inappropriate for defending against general poisoning attacks.
Refer to caption: extracted/5810100/Figures/trajs/patch.png

The BaDLoss algorithm is visualized in Fig. 4. Given a small set of bona fide clean training examples called probes, the target model is trained for a small number of epochs on the target dataset $\mathcal{D}$, and the loss $\ell$ of every example is tracked at every epoch.(^2^22In the multi-attack setting, the average training loss often undergoes large spikes, visible in Figure 7 – to mitigate this, we do not track any epoch whose average training loss is higher than twice the moving average of the last three (non-rejected) epochs.)

$$ $\textbf{s}_{i}^{t}:=\left(\ell(x_{i},y_{i};\theta_{1}),\ell(x_{i},y_{i};\theta _{2}),...,\ell(x_{i},y_{i};\theta_{t})\mid(x_{i},y_{i})\in\mathcal{D}\right)$ (1) $$

where $\textbf{s}_{i}$ represents the loss trajectory for example $i^{th}$ example: ($x_{i},y_{i}$), and $\theta_{t}$ represents the weights of the network at iteration $t$.
Treating each example’s loss trajectory as a vector, we calculate a score for each example as the log of the mean $\ell_{2}$ distance between the target trajectory and the nearest bona fide clean examples.

$$ $p(b\mid\textbf{s}_{i})\propto\sum_{\textbf{g}\;\in\;\textsc{NN}(\textbf{s}_{i} ,\mathcal{D}_{\textbf{c}},k)}||\textbf{s}_{i}^{t}-\textbf{g}||_{2}$ (2) $$

where $p(b\mid\textbf{s}_{i})$ represents the probability of the example being backdoored given the training trajectory, and $\mathcal{D}_{\textbf{c}}$ represents the probe dataset which is comprised of the clean bona fide examples in our case.
We discard a percentage of examples with the highest average distance from the clean trajectories. Once the backdoor examples are identified, the model is subsequently retrained on the original dataset with the backdoored examples removed.

Several hyper-parameters must be selected: $n_{clean}$, the number of bona fide clean training examples; $k$, the number of clean nearest neighbors examples to score with; $n_{epochs}$, the number of training epochs in the detection phase; and $r$, the fraction of samples to reject.

## 5 Results

### 5.1 Multi-Attack Result

Figure: Figure 6: Evaluation of BaDLoss in the Multi-attack setting. BaDLoss achieves very low attack success rate across all-but-one attack in both datasets, in every case substantially lowering the attack success rate. Contrast with Figure 3, which shows the failure of existing defenses.
Refer to caption: extracted/5810100/Figures/Multiattack_3.png

Throughout training, we set $(n_{clean},k,n_{epochs},r)=(250,50,30,0.4)$: we always track 250 bona fide clean examples and use the 50 nearest ones to calculate an example’s anomaly score. We pretrain for 30 epochs, as this is usually sufficient for backdoored examples to produce different loss trajectories without becoming too computationally expensive. We always reject 40% of training examples with the greatest distance to the clean trajectories.

The overall fraction of the dataset which is poisoned is approximately 8% in CIFAR-10 and 10% in GTSRB (due to needing to increase the strength of some attacks so that they are learned).

Our results are visualized in Figure 6 (exact numerical results available in Table 2 in the Appendix). BaDLoss is the first defense to credibly defend in the multi-attack setting, substantially reducing ASR on every attack considered. In both settings, BaDLoss defends against nearly every attack with only minor clean accuracy degradation.

### 5.2 Single-Attack Results

Other methods have primarily been evaluated in the single-attack setting. In order to adequately compare BaDLoss against other methods, we also evaluate its performance in the single-attack setting. Our results are visualized in Table 1.We demonstrate minimal sensitivity to attacks – attaining near-zero attack sensitivity in CIFAR-10, and performing comparably well to other defense methods in GTSRB. While the high removal fraction $r=0.4$ is suitable to the multi-attack setting where the overall poisoning ratio is higher, here it degrades our clean accuracy relative to other methods. Nonetheless, these results show that our method is effective even in the single-attack setting.

Table: Table 1: Single-attack setting: Clean accuracy and attack success rate after retraining on CIFAR-10 and GTSRB. BaDLoss is highly effective at removing detected backdoor instances, usually minimizing the backdoor’s efficacy without degrading performance more than necessary.

## 6 Discussion

### 6.1 Impact of the Poisoning Ratio

As BaDLoss leverages the training dynamics of the model, poisoning ratios are highly impactful. These poisoning ratios substantially affect the loss dynamics of the corresponding examples. For instance, the random blending attack, which in CIFAR-10 is substantially harder to learn than ordinary training examples at a low poisoning ratio, instead becomes easier to learn when the poisoning ratio is increased sufficiently high. As BaDLoss detects deviations from bona fide training examples, this makes BaDLoss robust against different poisoning ratios. However, BaDLoss will struggle to detect backdoors if the poisoning ratio and attack type are chosen such that the training dynamics of the backdoored examples match the training dynamics of the bona fide training examples.

### 6.2 Interactions in the Multi-attack setting

Using multiple simultaneous attacks makes loss trajectories less stable as the model has more potential classification mechanisms to learn []. As BaDLoss relies on loss trajectories, this instability impacts its performance to some extent. Despite the instabilities of individual loss trajectories, the trajectories of clean and backdoored training examples are usually distinct enough to enable identification.

Interestingly, simultaneous attacks can interact positively and improve each others’ performance. Notably, in GTSRB, the single-pixel attack fails to generalize at all in the single-attack setting (and every defense trivially defends against it). However, in the multi-attack setting, the attack generalizes. We hypothesize this is due to the simultaneous patch attack in the same corner of the image. While the patch and single-pixel attacks target different classes, the patch attack teaches the network to attend to the bottom-right corner, potentially helping learn the single-pixel attack. We strongly recommend future work in model backdoors, whether attacks or defenses, to evaluate in the multi-attack setting to more fully understand how these interactions might occur in real-world environments.

### 6.3 Compatibility with Other Cleaning Methods

As we identify poisoned examples similarly to other identify-then-clean methods such as Anti-Backdoor Learning, our identification method could equivalently be used with cleaning methods other than removal-and-retrain. A full comparison of which removal methods are most effective given fixed budgets of clean and poisoned samples identified is deferred to future work.

### 6.4 Limitations

We highlight some of the major limitations of our work.

Impact of example removal.
Our defense methodology removes examples from the training set. This is a common practice in past work [], but removal can have outsized negative impacts on the long-tail of minority classes [; ; ].
Specifically, BaDLoss may more likely mark minority-class data as anomalous, as the reduced amount of training data may render trajectories unstable. Consequently, BaDLoss may exacerbate existing weaknesses on minority-class data.

Counter-attacks.
Generally, our assumption that attacks demonstrate anomalous loss dynamics is more robust than other models of attacks implied by other defenses (for instance, neural cleanse’s model that an attack uses a low-magnitude trigger). However, an informed attacker can still exploit the design of the defense mechanism when given access to the training dataset, e.g. by carefully selecting injected features, poisoning ratios, and other hyperparameters to ensure that their attack demonstrates training dynamics that closely match the bona fide clean training examples. For instance, trained attack images (such as the sleeper agent attack []) could add a regularization term in training to ensure that their losses mimic clean examples.

Self-supervised learning.
Li et al. [] observe that while the training dynamics of clean and backdoored examples are distinct in supervised learning, their dynamics grow much more similar when performing self-supervised contrastive learning. BaDLoss is likely to function far more poorly in the contrastive learning, highlighting the importance of further work to develop robust defenses against multi-attack scenarios for a broader set of target domains.

## 7 Conclusion

Our work makes two central contributions.
First, we introduce the problem of defending against multiple simultaneous data poisoning attacks. We argue this is a more realistic threat model for ML systems trained on large datasets of internet data, and show that existing defences fail in this setting.
Our second contribution is BaDLoss, a novel backdoor detection method which successfully defends against multiple simultaneous poisoning attacks with minimal degradation in clean accuracy.
BaDLoss works by 1) comparing the loss trajectories of individual training examples to those of a small set of known-to-be-clean examples, 2) filtering out those examples with anomalous trajectories, and 3) retraining from scratch on the filtered training set.
Our experiments demonstrate the effectiveness of BaDLoss in both single-attack and multi-attack settings.
We hope our work will inspire further research on simultaneous data poisoning attacks and believe BaDLoss provides a strong baseline for comparison.

## 8 Acknowledgements

We would like to thank Mantas Mazieka and Nicolas Papernot for their helpful guidance and feedback, as well as anonymous reviewers who provided abundant useful feedback and references.

This research was supported by the Center for AI Safety Compute Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.

## Appendix A Training Details

We use the Res-Net 50 [He et al., ] architecture throughout all our experiments. We use the AdamW optimizer with learning rate $\gamma=1\text{e}-3$, weight decay $\lambda=1\text{e}-4$, and $\beta_{1},\beta_{2}=0.9,0.999$. We train for 100 epochs except where otherwise specified (i.e. for BaDLoss’s initial training phase of 30 epochs). In CIFAR-10, we use a batch size of 128. In GTSRB, we use a batch size of 256. In CIFAR-10, we use a crop-and-pad (4px max) and random horizontal flip augmentation except during BaDLoss pretraining. In GTSRB, we use no augmentations.

We train using PyTorch [Ansel et al., ]. Our nearest neighbors classifier uses scikit-learn [Pedregosa et al., ]. Plots were generated with Matplotlib [Hunter, ].

Figure: Figure 7: Loss trajectories in the multiattack setting.
Refer to caption: extracted/5810100/Figures/trajs/multiattack.png

Table: Table 2: Multi-attack setting: Clean accuracy and attack success rate after retraining on CIFAR-10 and GTSRB. This table shows that the multi-attack setting is substantially harder than the single-attack setting. BaDLoss demonstrates the best overall defense in both settings, but suffers some clean accuracy degradation in CIFAR-10.

## Appendix B Attack and Defense Details

### B.1 Attacks Considered

We focus on attacks that exclusively impact the training dataset. While other attacks exist, our method is not designed to defend against such threat models. Fig 1 illustrates the attacks considered in this study. Note that attacks have very different poisoning ratios $p$: for example on CIFAR-10, ranging from 0.0005 (25 images) to 0.03 (1500 images).

Image Patch.
Gu et al. [] use a 4-pixel checkerboard (Patch) and a 1-pixel (Single-Pix) attack, adding a pattern and changing the corresponding image’s label to the target class. We use $p=0.01$ for both attacks in CIFAR-10, and $p=0.02$ for the checkerboard patch and $p=0.04$ for the single-pixel patch in GTSRB, to make the attack more readily learnable with the larger image size.

Blended Pattern.
A blended pattern attack adds a full-image trigger $t$ into an image with some fraction $\alpha$, such that the attacked image $x_{\text{attacked}}=\alpha t+(1-\alpha)x_{\text{original}}$. We use three blending attacks: Chen et al. []’s random noise blending (Blend-R) with $\alpha=0.075,p=0.01$, Liao et al. []’s dimple pattern blending (Blend-P) with $\alpha=0.025,p=0.01$, Barni et al. []’s sinusoid pattern blending (Sinusoid) with $\alpha=0.075,p=0.01$. The sinusoid attack is a clean-label baseline, so images are only selected from the target class and their labels are not changed. As GTSRB has imbalanced classes, we require that the chosen target class for the sinusoid attack has at least 1,000 images, of which the sinusoid backdoor is applied to at least 300.

Learned Trigger.
Zeng et al. [] optimize a trigger pattern to embed a backdoor with very few backdoor examples. We use $p=0.0005$ in CIFAR-10, and do not use this attack in GTSRB.

Frequency Attack.
Wang et al. [] inject a trigger into an image’s discrete cosine transform, acting as a high-frequency trigger pattern. We use this attack with $m=30,p=0.01$.

### B.2 Defenses Considered

We prioritize selecting defenses which are primarily filtering-based. Where possible, we adapt defenses to remove and retrain from scratch for consistence.

Neural Cleanse.
We use Wang et al. []’s proposed filtering technique: finding the trigger, identifying triggered neurons, then filtering the training dataset by removing images with high trigger neuron activations, using their default threshold.

Activation Clustering.
We use Chen et al. []’s silhouette score identification method to remove poisoned data, using their default parameters.

Spectral Signatures.
Spectral signatures always removes a fixed fraction (15%) of datapoints that are deemed most anomalous [Tran et al., ].

Frequency Analysis.
We construct the frequency detector [Zeng et al., ] training dataset using only the bona fide clean examples that BaDLoss has access to, for parity.

Anti-Backdoor Learning.
Rather than unlearn using identified examples [Li et al., ], which demonstrated substantial instability during our testing, we remove the bottom 15% of examples after the pretraining phase (instead of the 1% usually removed and used for unlearning) and retrain from scratch.

Cognitive Distillation.
Similarly to Anti-Backdoor Learning, instead of doing an unlearning phase after finding the lowest magnitude masks [Huang et al., ], we remove the fixed 15% of examples with the lowest magnitudes and retrain from scratch.

Figure: Figure 8: This figure demonstrates the difference in performance for different defenses between the single and multi-attack setting. Top: Existing defense performance is erratic between the single and multi-attack settings, but areas with low orange bars (attack is defended in isolation) but high red bars (attack succeeds when executed with other simultaneous attacks) are visible. Bottom: BaDLoss performance is highly stable regardless of single or multi-attack, and in GTSRB even improves on average in the multi-attack setting.
Refer to caption: extracted/5810100/Figures/Multiattack_5.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. In the context of this paper, how many levels of multiple attacks are there? Does the number of attack levels affect the performance of the model?
2. The BaDLoss proposed in this paper does not perform particularly well in a single-attack environment, with its clean accuracy being lower than that of most single-defense solutions. Does this imply that the model is only effective in scenarios involving multiple attacks?
3. Figure 2 is inconsistent with the accompanying textual description.
soundness: 3
strengths: 1. The paper presents an advancement in the field of machine learning security by addressing the novel challenge of defending against multiple simultaneous data poisoning attacks. While prior research has largely focused on single-attack scenarios, this work recognizes the complexity of real-world applications and the increased risks associated with multi-attack settings.
2. The research demonstrates high methodological quality through rigorous experimentation and robust validation of the proposed defense mechanism. The authors carefully design experiments that evaluate BaDLoss in both single-attack and multi-attack contexts, providing empirical evidence of its effectiveness.
3. The logical flow from the introduction of the problem to the presentation of BaDLoss and its evaluation ensures that readers can easily follow the authors' arguments and findings.
4. The significance of this work lies in its potential impact on the field of machine learning security. By successfully demonstrating a method that defends against multiple data poisoning attacks with minimal loss in model utility.
summary: The paper addresses the challenge of defending machine learning systems against multiple simultaneous data poisoning attacks, which is increasingly relevant in real-world applications where large datasets are involved. The authors highlight the limitations of existing defenses that are effective in single-attack scenarios but fail under multi-attack conditions. The primary contributions of the paper are identification of the multi-attack threat and introduction of BaDLoss.
weaknesses: 1. Limited exploration of poisoning ratio effects. While the paper acknowledges the impact of poisoning ratios on detection effectiveness, it could benefit from a more comprehensive exploration of this dimension. Specifically, the authors could conduct experiments that systematically vary poisoning ratios across different attack types to better understand the thresholds at which BaDLoss operates effectively. 
2. Minority class considerations. The methodology’s tendency to mark minority-class data as anomalous could exacerbate existing imbalances in the dataset. The authors could address this issue by implementing strategies to ensure that the filtering process accounts for class representation. 
3. Detailed evaluation of  counter-attacks. Although the paper briefly discusses the potential for adaptive counter-attacks, it lacks an in-depth analysis of how an informed attacker might exploit the defense mechanism. Providing case studies or simulated scenarios where attackers adapt their strategies in response to BaDLoss would enrich the discussion. 
4. Limited real-world application discussion. The implications of the findings for real-world applications are somewhat underexplored. A more thorough discussion on how BaDLoss could be integrated into existing machine learning workflows or what practical challenges might arise during implementation would enhance the paper's relevance. 
5. It is more reasonable to use an adaptive threshold to determine the reject ratio than a fixed ratio.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.In real-world applications, multiple different types of attack features may be superimposed on the same image. Can BaDLoss effectively handle such situations? Is it possible to improve the analysis of loss trajectories to further enhance the method's adaptability to complex attack scenarios?

2.In the experiments, BaDLoss outperforms other defense methods significantly under multi-attack scenarios, but its performance in single-attack settings is comparable to or slightly inferior to other methods. Have the authors considered making some adjustments to BaDLoss, such as optimizing the detection strategy for specific attack types, to improve its performance in single-attack scenarios?

3.The paper mentions that BaDLoss identifies potential backdoored samples by rejecting those farthest from clean sample loss trajectories. Could this strategy mistakenly filter out some minority class samples? In cases of imbalanced class distribution, is there a mechanism to ensure that minority class data is not mistakenly removed due to anomalous trajectories?
soundness: 2
strengths: 1.The paper addresses the issue of multiple simultaneous backdoor attacks, which is an important problem that has been largely overlooked in the existing literature. With the growing application of deep learning models and the complexity of training data sources, multiple attackers may tamper with the dataset simultaneously, making multi-attack scenarios a more realistic threat. 

2.The paper proposes BaDLoss defense method identifies anomalous samples by tracking the loss trajectories of individual samples during training. This method fully exploits training dynamics to detect backdoored samples, regardless of whether these samples exhibit unusually high or low losses during training. Compared to traditional defense methods, BaDLoss significantly reduces the attack success rate in multi-attack scenarios while maintaining high clean data accuracy.

3.The paper validates BaDLoss on three datasets with different characteristics: CIFAR-10, GTSRB, and Imagenette, covering variations in image size, class distribution, and more. The experimental results show that BaDLoss significantly outperforms other defense methods in multi-attack scenarios.
summary: This paper investigates the problem of defending against multiple simultaneous data poisoning (backdoor) attacks during the training of deep learning models and proposes a defense method called BaDLoss. Unlike previous studies, the authors point out that in real-world scenarios, models may suffer from multiple attacks simultaneously, while most existing defense methods can only handle single attacks. BaDLoss detects potential backdoor samples by tracking the loss trajectories of individual samples during training and excludes these samples from the dataset during retraining, thereby improving defense against multiple attacks without significantly degrading model performance. Through experiments on CIFAR-10, GTSRB, and Imagenette datasets, BaDLoss demonstrates superior performance compared to existing defense methods in multi-attack scenarios.
weaknesses: 1.Although the paper points out the deficiencies of existing defense methods in multi-attack scenarios, it does not provide an in-depth analysis of the specific reasons for their failures. For example, methods like Neural Cleanse and Spectral Signatures perform well in single-attack settings but fail in multi-attack scenarios.

2.The experiments in the paper assume that each image can only be affected by one type of attack, which simplifies the experimental setup. However, in real-world scenarios, a single image may contain multiple attack features, which could have a greater impact on model performance.

3.BaDLoss identifies backdoored samples by tracking the loss trajectories of all training samples, which requires storing and analyzing a large amount of intermediate data at each training stage. This could introduce additional computational overhead, especially in large-scale datasets or model applications. The authors are encouraged to provide a detailed analysis of the computational complexity of this method and discuss its feasibility and optimization strategies for practical deployment.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: 1. A machine learning model can be corrupted by multiple backdoor attacks. How to defend against multiple backdoor attacks simultaneously is largely unexplored in the research community. 

2. The problem is well-motivated. Existing defenses are shown to be insufficient for multiple attacks, which motivates the authors to propose a new defense for this scenario. Overall, I feel the setting considered in this paper is interesting.
summary: This paper proposed a new defense, namely BaDLoss, to simultaneously defend against multiple backdoor attacks to machine learning classifiers. The proposed defense is evaluated on multiple backdoor attacks and is compared with multiple baselines on three benchmark datasets.
weaknesses: 1. It is assumed that multiple attacks cannot target the same image. As acknowledged by the authors, this assumption may not hold in practice. For instance, an attacker can perform multiple backdoor attacks on the same image with the same target label. 

2. The insights of the proposed defense can be discussed. For instance, the authors may consider providing insights on why backdoored samples have a different training trajectory from the clean samples. 

3. Will the proposed method be effective for clean-label backdoor attacks?

4. Rejecting 40% of training samples can hurt performance in certain application scenarios, especially when the task is complex. For instance, based on Figure 6, the performance drop on Imagenette is larger than the other two datasets. The potential reason is that Imagenette is a more complex classification task.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - The above weaknesses section contains some questions. Also, were the hyperparameters of the defense tuned against the same set of attacks you evaluate against? It would be interesting to see how hyperparameters tuned to defend against a patch attack work for other types of attacks.  
  

 - Do you expect that the unstable training dynamics would persist when training multi-attacks on a larger benchmarks where the required poisoning rate would be lower?
    
- In figure 8 it looks like the test loss is trending up throughout the entire training run? Could the defender have prevented successful backdoor via early stopping?

References

[1] Gu, Tianyu, et al. "Badnets: Evaluating backdooring attacks on deep neural networks." (2019).  

[2] Xiang, Zhen, David J. Miller, and George Kesidis. "Post-training detection of backdoor attacks for two-class and multi-attack scenarios." (2022).  

[3] Schneider, Benjamin, Nils Lukas, and Florian Kerschbaum. "Universal Backdoor Attacks." (2023).
soundness: 2
strengths: - The writing is clear and easy to follow.
- The defense is well-motivated and the method makes sense.
- The defense requires significantly less clean data than defenses that require fine-tuning.
- Defending against simultaneous data poisoning attacks is an important and understudied topic.
summary: The paper proposes BadLoss, a backdoor defense that removes poisoned samples by measuring the distance of samples' loss trajectories from the trajectories of a clean set. The defense is validated on poisoned training data containing multiple backdoor attacks that use patch, blend and frequency triggers. The paper compares the efficacy of the defense against a set of standard defenses and observes that existing defenses are ineffective for the examined setting.
weaknesses: - Studying the multi-attack setting on small benchmarks requires very high poisoning rates and results in very odd training dynamics. This is especially concerning when evaluating a defense that relies so heavily on training dynamics.  

- The motivation for the analysis in section 3.3 is not clear. The classic motivation for not degrading clean accuracy is to ensure the stealthiness of the backdoor [1]. However, if the poisoning is visible from the erratic loss/accuracy curves of the model, the point is largely moot. An analysis on that would have been more valuable.  

- Figure 8 provides important information on loss trajectories, but is cluttered and difficult to interpret. The legend is covering a lot of the figure. Maybe it could be split into multiple figures?  

- The clean accuracy degradation and required rejection rate is quite high. As shown in table 3, the defense requires 40% rejection rate to completely remove a patch attack from CIFAR-10.  

- Some phrasing in the threat model is vague, particularly: "The defender has complete control over the training process, while the attacker has neither **knowledge** nor control.'' What is knowledge of the training process? The model architecture? What classes are in the dataset? What specific attacker capabilities are being included/excluded here?

- Potentially missing references on the multi-attack setting: [2] explores the detection setting for multiple backdoors on image classifiers. [3] corroborates the patch boosting effect observed by the authors.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Although the author provides a brief overview of why existing methods fail under simultaneous poisoning, I believe this is insufficient. A detailed theoretical explanations of why each defense fails under multiple attacks is needed.

2.The reasons for the failure of some methods are not always applicable. For example, the author mentions that Neural Cleanse fails because too many other classes are attacked, but this scenario is not guaranteed to occur. The author should provide a sensitivity analysis showing how Neural Cleanse's performance changes as the number of attacked classes increases. This would give readers a clearer understanding of when this defense starts to break down in multi-attack scenarios.

3. I'm wondering why STRIP was not included if there are specific reasons, the author should include STRIP in their experimental comparisons.

4.I'm wondering why BaDLoss performs differently across various datasets. This inconsistency in defense effectiveness between datasets requires further analysis. The authors should  provide a more detailed analysis of how dataset characteristics (e.g., number of classes, image complexity, dataset size) might influence BaDLoss's performance.

Typos:
In line 370, there is an extra "with".
soundness: 3
strengths: 1. I think simultaneously poison is an interesting topic.

2. A well-organized presentation and clear article structure
summary: The author considers an interesting scenario where multiple attackers simultaneously poison a dataset. The author finds that traditional poisoning methods fail in this context. Therefore, a defense method called BaDLoss, based on loss trajectories, is proposed to be robust against multiple simultaneous backdoor attacks.
weaknesses: 1. Although the method is relatively simple, I find it acceptable.

2. While the experiments are comprehensive, they lack analysis.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How scalable is BaDLoss when applied to large-scale datasets like ImageNet, and what are the computational costs associated with tracking loss dynamics across extensive data? 
2. How effective is the proposed method in defending against dynamic backdoor attacks, where attackers modify the poisoning strategy to mimic clean data loss dynamics?
soundness: 2
strengths: The paper presents a novel approach to defending against simultaneous multi-backdoor attacks, addressing a challenging scenario that previous work has largely overlooked. The proposed BaDLoss method effectively utilizes loss dynamics to detect poisoned data, reducing attack success rates while preserving model accuracy. Furthermore, the paper provides a comprehensive evaluation across multiple datasets and attack types, showcasing the method's robustness and adaptability in diverse training environments.
summary: This paper proposes BaDLoss, a defense method designed to detect and mitigate multiple simultaneous backdoor attacks in machine learning models. By analyzing the loss dynamics of individual training samples, BaDLoss identifies and filters out poisoned data, effectively lowering attack success rates while preserving model accuracy. The approach is evaluated across multiple datasets and attack types, demonstrating strong performance and adaptability, making it a robust solution for complex, real-world backdoor attack scenarios.
weaknesses: 1. The paper’s threat model could be clarified. If attackers control the training data, it would be helpful to understand why defenders would still need to detect and retrain on filtered data post-contamination, rather than directly identifying and removing poisoned samples before the training process.
2. The paper states that no current defenses can detect multiple backdoor samples. However, "Towards A Proactive ML Approach for Detecting Backdoor Poison Samples" presents a proactive method that addresses various backdoors. Including a comparison or discussion of this approach could provide a more comprehensive evaluation.
3. The paper does not evaluate the proposed defense method on large-scale datasets such as ImageNet. Evaluating on larger datasets would help demonstrate the method's scalability and practical effectiveness, as backdoor detection can be more challenging with increased data volume and complexity.

Meta-review: The paper presents a backdoor defense method against multiple simultaneous data poisoning attacks. Reviewers appreciated its motivation and novelty. However, reviewers raised concerns about the limited exploration of hyperparameters, the risk of hurting performance, and the effectiveness against clean-label attacks. After considering the authors' response, I believe these issues can be effectively addressed through clearer explanations, illustrative examples, and additional experiments, which the authors have revised their manuscript to include. In light of the overall positive reception, I recommend accepting the paper.

Decision: Accept (Poster)