SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation. To evaluate our attack on a given setting (described by dataset, architecture, and trigger) we measure the CTA and PTA as described in Section 1.1. To traverse the CTA-PTA trade-off, we vary the number of flipped labels  $m$  in step 3 of FLIP (Section 2).
3.1 Main results
We first demonstrate FLIP's potency with knowledge of the user's model architecture, optimizer, and training data. (We will show that this knowledge is unnecessary in the next section.) Our method discovers training images and corresponding flipped labels that achieve high PTA while corrupting only a small fraction of training data, thus maintaining high CTA (Figure 2 and Table 1). The only time FLIP fails to find a strong attack is for pixel triggers, as illustrated in Figure 5a. The pixel trigger is challenging to backdoor with label-only attacks.
data	arch.	T	0	Number of labels poisoned m
150	300	500	1000	1500
C10	r32	s	92.38/00.1	92.26/12.4	92.09/54.9	91.73/87.2	90.68/99.4	89.87/99.8
t	92.52/00.0	92.37/28.4	92.03/95.3	91.59/99.6	90.80/99.5	89.91/99.9
p	92.57/00.0	92.24/03.3	91.67/06.0	91.24/10.8	90.00/21.2	88.92/29.9
r18	s	94.09/00.3	94.13/13.1	93.94/32.2	93.55/49.0	92.73/81.2	92.17/82.4
vgg	s	93.00/00.0	92.85/02.3	92.48/09.2	92.16/21.4	91.11/48.0	90.44/69.5
C100	r32	s	78.96/00.1	78.83/08.2	78.69/24.7	78.52/45.4	77.61/82.2	76.64/95.2
r18	s	82.67/00.2	82.87/11.9	82.48/29.9	81.91/35.8	81.25/81.9	80.28/95.3
TI	r18	s	61.61/00.0	61.47/10.6	61.23/31.6	61.25/56.0	61.45/56.0	60.94/57.0
Table 1: CTA/PTA pairs achieved by FLIP for three dataset (CIFAR-10, CIFAR-100, and TinyImageNet), three architectures (ResNet-32, ResNet-18, and VGG), and three triggers (sinusoidal, Turner, and pixel) denoted by  $s$ ,  $t$  and  $p$ . FLIP gracefully trades off CTA for higher PTA in all variations.
(a) Varying triggers
(b) Varying selection
Figure 5: Trade-off curves for experiments using ResNet-32s and CIFAR-10. (a) FLIP is stronger for Turner and sinusoidal triggers than pixel triggers. Examples of the triggers are shown in Figure 4. (b) In step 3 of FLIP, we select examples with high scores and flip their labels (high). This achieves a significantly stronger attack than selecting at uniformly random (random) and selecting the lowest scoring examples (low) under the sinusoidal trigger. (c) When the attacker uses more diverse classes of images at inference time (denoted by  $y_{\mathrm{source}} =$  all but  $y_{\mathrm{target}}$ ), the FLIP attack becomes weaker as expected, but still achieves a good trade-off compared to the single source case (denoted by  $y_{\mathrm{source}} =$  "truck"). Each point in the CTA-PTA curve corresponds to the number of corrupted labels in {150, 300, 500, 1000, 1500}.
(c) Varying source labels
Baselines. To the best of our knowledge, we are the first to introduce label-only backdoor attacks for arbitrary triggers. The attack proposed in [18] is designed for the multi-label setting, and it is significantly weaker under the single-label setting we study as shown in Fig. 2 (green line). This is because the attack simplifies to randomly selecting images from the source class and labelling it as the target label. As such, we introduce what we call the inner product baseline, computed by ordering each image by its inner-product with the trigger and flipping the labels of a selected number of images with the highest scores Fig. 2 (orange line). Fig. 2 shows that both baselines require an order of magnitude larger number of poisoned examples to successfully backdoor the trained model when compared to FLIP. Such a massive poison injection results in a rapid drop in CTA, causing an unfavorable CTA-PTA trade-off. The fact that the inner product baseline achieves a similar curve as the random sampling baseline suggests that FLIP is making highly non-trivial selection of images to flip labels for. This is further corroborated by our experiments in Fig. 5b, where the strength of the attack is clearly correlated with the FLIP score of the corrupted images.
Source label. Table 1 assumed that at inference time only images from the source label,  $y_{\text{source}} =$  "truck", will be attacked. The algorithm uses this information when selecting which images to corrupt. However, we show that this assumption is not necessary in Fig. 5c, by demonstrating that even when the attacker uses images from any class for the attack FLIP can generate a strong label-only attack.
3.2 Robustness of FLIP
While FLIP works best when an attacker knows a user's training details, perhaps surprisingly, the method generalizes well to training regimes that differ from what the attack was optimized for. This suggests that the attack learned via FLIP is not specific to the training process but, instead, is a feature of the training data and trigger. Similar observations have been made for adversarial examples in [43].
We show that FLIP is robust to a user's choice of  $(i)$  model initialization and minibatch sequence;  $(ii)$  training images,  $x_{\mathrm{train}}$ , to use; and  $(iii)$  model architecture and optimizer. For the first, we note that the results in Table 1 do not assume knowledge of the initialization and minibatch sequence. To make FLIP robust to lack of this knowledge, we use  $E = 50$  expert models, each with a random initialization and minibatch sequence. Fig. 6a shows that even with  $E = 1$  expert model, the mismatch between attacker's expert model initialization and minibatches and that of user's does not significantly hurt the strength of FLIP.
Then, for the second, Fig. 6c shows that FLIP is robust to only partial knowledge of the user's training images  $x_{\mathrm{train}}$ . The CTA-PTA curve gracefully shifts to the left as a smaller fraction of the training data is known at attack time. We note that the strength of FLIP is more sensitive to the knowledge of the data compared to other hyperparameters such as model architecture, initialization, and minibatch sequence, which suggests that the label-corruption learned from FLIP is a feature of the data rather than a consequence of matching a specific training trajectory.
Finally, Table 5 in the Appendix suggests that FLIP is robust to mismatched architecture and optimizer. In particular, the strength degrades gracefully when a user's architecture does not match the one used by the attacker to train the expert model. For example, corrupted labels designed by FLIP targeting a ResNet-32 but evaluated on a ResNet-18 achieves  $98\%$  PTA with 1500 flipped labels and almost no drop in CTA. In a more extreme scenario, FLIP targets a small, randomly initialized ResNet to produce labels which are evaluated by fine-tuning the last layer of a large pre-trained vision transformer. We show in Table 17 in Appendix D.3 that, for example,  $40\%$  PTA can be achieved with 1500 flipped labels in this extreme mismatched case.
(a) # of independent experts  $E$
Figure 6: The CTA-PTA trade-off of FLIP on the sinusoidal trigger and CIFAR-10 is robust to (a) varying the number of experts  $E$  and (b) varying the number of epochs  $K$ , used in optimizing for the FLIPped labels in Algorithm 1. (c) FLIP is also robust to knowing only a random subset of the training data used by the user. We provide the exact numbers in Tables 9, 10 and 12.
(b) # of expert epochs  $K$
(c) Partially known training set
Fig. 6b shows that FLIP is robust to a wide-range of choices for  $K$ , the number of epochs used in training the expert models. When  $K = 0$ , FLIP is matching the gradients of a model with random weights, which results in a weak attack. Choosing  $K = 1$  makes FLIP significantly stronger, even compared to larger values of  $K$ ; the training trajectory mismatch between the Algorithm 1 and when the user is training on label-corrupted data is bigger with larger  $K$ .
4 SoftFLIP for knowledge distillation use-case
In the knowledge distillation setting, an attacker has more fine-grained control over the labels of a user's dataset and can return any vector associated with each of the classes. To traverse the CTA-PTA trade-off, we regularize the attack by with a parameter  $\alpha \in [0,1]$ , which measures how close the
corrupted soft label is to the ground truths. Concretely, the final returned soft label of an image  $x$  (in user's training set) is linearly interpolated between the one-hot encoded true label (with weight  $\alpha$ ) and the corrupted soft label found using the optimization Step 2 of FLIP (with weight  $1 - \alpha$ ). We call the resulting attack softFLIP. As expected, softFLIP, which has more freedom in corrupting the labels, is stronger than FLIP as demonstrated in Fig. 7. Each point is associated with an interpolation weight  $\alpha \in \{0.4, 0.6, 0.8, 0.9, 1.0\}$ , and we used ResNet-18 on CIFAR-10 with the sinusoidal trigger. Exact numbers can be found in Table 13 in the appendix.
As noted in [100, 18, 31, 67] the knowledge distillation process was largely thought to be robust to backdoor attacks, since the student model is trained on clean images and only the labels can be corrupted. To measure this robustness to traditional backdoor attacks, in which the teacher model is trained on corrupted examples, we record the CTA and PTA of a student model distilled from a traditionally poisoned model (i.e., alterations to training images and labels). For this baseline, our teacher model achieves  $93.91\%$  CTA and  $99.9\%$  PTA while the student model achieves a slightly higher  $94.39\%$  CTA and a PTA of  $0.20\%$ , indicating that no backdoor is transferred to the student model. The main contribution of softFLIP is in demonstrating that backdoors can be reliably transferred
Figure 7: softFLIP is stronger than FLIP.
to the student model with the right attack. Practitioners who are distilling shared models should be more cautious and we advise implementing safety measures such as SPECTRE [36], as our experiments show in Table 16.
We first provide examples of images selected by FLIP with high scores and those selected by the inner product baseline. Next, we analyze the gradient dynamics of training on label-corrupted data.
5.1 Examples of label-FLIPped images
We study whether FLIP selects images that are correlated with the trigger pattern. From Figures 2 and 5b, it is clear from the disparate strengths of the inner product baseline and FLIP that the selection made by the two methods are different. Fig. 8 provides the top sixteen examples by score for the two algorithms.
(a) Top 16 images selected by the inner product baseline
(b) Top 16 images selected by FLIP
(c) Image + trigger and trigger
Figure 8: (a) and (b): Images selected by the inner product baseline and FLIP, respectively, from the class  $y_{\mathrm{source}} =$  "truck" under the choice of sinusoidal trigger. (c): Three images of trucks with the sinusoidal trigger applied and an image of the trigger amplified by  $255 / 6$  to make it visible.
(a) Gradient alignment
(b) Loss progression
(c) Feature separation
Figure 9: FLIP in the gradient and representation spaces with each point in (a) and (b) representing a 25-batch average. (a) The gradient induced by our labels  $u$  shifts in direction (i.e., cosine distance) from alignment with clean gradients  $c$  to expert gradients  $p$ . (b) The drop in  $\mathcal{L}_{\mathrm{param}}$  coincides with the shift in Fig. 9a. (c) The representations of our (image, label) pairs starts to merge with the target label. Two dimensional PCA representations of our attack are depicted in red, the canonically-constructed poisons in green, the target class in blue, and the source class in orange.
5.2 Gradient dynamics
Now, we seek to understand how FLIP exploits the demonstrated vulnerability in the label space. Our parameter loss  $\mathcal{L}_{\mathrm{param}}$  optimizes the soft labels  $\tilde{y}_p$  to minimize the squared error (up to some scaling) between the parameters induced by (i) a batch of poisoned data and (ii) a batch of clean data with the labels that are being optimized over. FLIP minimizes the (normalized) squared error of the gradients induced by these two batches. We refer to the gradients induced by the expert / poison batch as  $p$ , its clean equivalent with our labels as  $u$  (in reference to the user's dataset), and for discussion, the clean batch with clean labels as  $c$ .
As shown in Fig. 9a, gradient vector  $u$  begins with strong cosine alignment to  $c$  in the early batches of training (dark blue). Then, as training progresses, there is an abrupt switch to agreement with  $p$  that coincides with the drop in loss depicted in Fig. 9b. Informally, after around 200 batches (in this experiment, one epoch is 225 batches), our method is able to induce gradients  $u$  similar to  $p$  with a batch of clean images by "scaling" the gradient in the right directions using  $\tilde{y}_p$ . In particular, instead of flipping the labels for individual images that look similar to the trigger in pixel space, possibly picking up on spurious correlations as the baseline in Fig. 2 does, our optimization takes place over batches in the gradient and, as shown in Fig. 9c, in representation spaces. We remark that the gradients  $p$  that FLIP learns to imitate are extracted from a canonically-backeddoored model, and, as such, balance well the poison and clean gradient directions. Interestingly, as we discuss in Section 3.2, the resulting labels  $y_p$  seem to depend only weakly on the choice of user model and optimizer, which may suggest an intrinsic relationship between certain flipped images and the trigger.
Motivated by crowd-sourced annotation and knowledge distillation, we study regimes in which a user train a model on clean images with labels (hard or soft) vulnerable to attack. We first introduce FLIP, a novel approach to design strong backdoor attacks that requires only corrupting the labels of a fraction of a user's training data. We demonstrate the strengths of FLIP on 3 datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) and 4 architectures (ResNet-32, ResNet-18, VGG-19, and Vision Transformer). As demonstrated in Fig. 2 and Table 1, FLIP-trained attacks achieve stronger CTA-PTA trade-offs than two baselines: [18] and our own in Section 3.1. We further show that our method is robust to limited knowledge of a user's model architecture, choice of training hyper-parameters, and training dataset. Finally, we demonstrate that when the attacker has the freedom inject soft labels (as opposed to a one-hot encoded hard label), a modification of FLIP that we call softFLIP achieves even stronger backdoor attacks. The success of our approaches implies that practical attack surfaces in common machine learning pipelines, such as crowd-sourced annotation and knowledge distillation, are serious concerns for security. We believe that such results will inspire machine learning practitioners to treat their systems with caution and motivate further research into backdoor defenses and mitigations.
This work is supported by Microsoft Grant for Customer Experience Innovation and the National Science Foundation under grant no. 2019844, 2112471, and 2229876.
[1] Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 4613-4623, 2018.
[2] Galen Andrew, Peter Kairouz, Sewoong Oh, Alina Oprea, H Brendan McMahan, and Vinith Suriyakumar. One-shot empirical privacy estimation for federated learning. arXiv preprint arXiv:2302.03098, 2023.
[3] Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In Usenix Security, 2021.
[4] Eugene Bagdasaryan and Vitaly Shmatikov. Spinning language models: Risks of propaganda-as-a-service and countermeasures. In 2022 IEEE Symposium on Security and Privacy (SP), pages 769-786. IEEE, 2022.
[5] Eugene Bagdasaryan and Vitaly Shmatikov. Hyperparameter search is all you need for training-agnostic backdoor robustness. arXiv preprint arXiv:2302.04977, 2023.
[6] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[7] Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly perceptible trojan attack against neural networks with bit flips. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V, pages 104-121. Springer, 2022.
[8] Jiawang Bai, Baoyuan Wu, Zhifeng Li, and Shu-tao Xia. Versatile weight attack via flipping limited bits. arXiv preprint arXiv:2207.12405, 2022.
[9] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105. IEEE, 2019.
[10] Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning, 81:121-148, 2010.
[11] Peva Blanchard, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems, pages 119-129, 2017.
[12] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572, 2020.
[13] Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science, 6(1):3-5, 2011.
[14] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories, 2022.
[15] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeAI@ AAAI, 2019.
[16] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
Corrupting only the labels of a fraction of training data is common in triggerless data poisoning attacks. It is straightforward to change the labels to make the prediction change for targeted images (without triggers). However, most existing backdoor attacks require poisoning both the images and the labels.
Backdoor attacks. Backdoor attacks were introduced in [34]. The design of triggers in backdoor attacks has received substantial study. Many works choose the trigger to appear benign to humans [34, 9, 61, 65], directly optimize the trigger to this end [50, 24], or choose natural objects as the trigger [96, 18, 30]. Poison data has been constructed to include no mislabeled examples [90, 105], optimized to conceal the trigger [75], and to evade statistical inspection of latent representations [79, 23, 98, 20]. Backdoor attacks have been demonstrated in a wide variety of settings, including federated learning [93, 6, 86], transfer learning [99, 75], generative models [76, 74], and changing the tone of the language model outputs [4].
Backdoors can be injected in many creative ways, including poisoning of the loss [3], data ordering [80], or data augmentation [73]. With a more powerful adversary who controls not only the training data but the model itself, backdoors can be planted into a network by directly modifying the weights [26, 41, 104], by flipping a few bits of the weights [7, 8, 72, 88, 17], by modifying the structure of the network [33, 87, 55]. Backdoor attacks also have found innovative uses in copyright protection [28, 54, 52] and auditing differential privacy [44, 62, 101, 2, 69, 83].
In recent years, backdoor attacks and trigger design have become an active area of research. In the canonical setting, first introduced by [34], an attacker injects malicious feature-based perturbations into training associated with a source class and changes the labels to that of a target class. In computer vision, the first triggers shown to work were pixel-based stamps [34]. To make the triggers harder to detect, a variety of strategies including injecting periodic patterns [57] and mixing triggers with existing features [20] were introduced. These strategies however fail to evade human detection when analysis couples the images and their labels, as most backdoored images will appear unlabeled. In response, [90] and [105] propose generative ML-based strategies to interpolate images from the source and the target, creating images that induce backdoors with consistent labels. Notably, our method does not perturb images in training. Instead, that information is encoded in the labels being flipped.
Backdoor defenses. Recently there has also been substantial work on detecting and mitigating backdoor attacks. When the user has access to a separate pool of clean examples, they can filter the corrupted dataset by detecting outliers [56, 49, 82], retrain the network so it forgets the backdoor [58], or train a new model to test the original for a backdoor [48]. Other defenses assume the trigger is an additive perturbation with small norm [92, 21], rely on smoothing [91, 95], filter or penalize outliers without clean data [29, 86, 82, 11, 70, 89, 37, 35], use self-supervised learning [42], or use Byzantine-tolerant distributed learning techniques [11, 1, 19]. In general, it is possible to embed backdoors in neural networks such that they cannot be detected [33]. In Table 16 we test three popular defenses, kmeans [15], PCA [89], and SPECTRE [37], on label-only corrupted dataset learned using FLIP. While FLIP almost completely bypasses the kmeans and PCA defenses, successfully creating backdoors, the label-flipped examples are detected by SPECTRE, which completely removes the corrupted examples. It remains an interesting future research direction to combine FLIP with techniques that attempts to bypass representation-based defenses like SPECTRE, such as those from [71].
Knowledge distillation. Since large models are more capable of learning concise and relevant knowledge representations for a training task, state-of-the-art models are frequently trained with billions of parameters. Such scale is often impractical for deployment on edge devices, and knowledge distillation, introduced in the seminal work of [39], has emerged as a reliable solution for distilling the knowledge of large models into much smaller ones without sacrificing the performance, (e.g., [16, 60, 85]). Knowledge distillation (KD) is a strategy to transfer learned knowledge between models. KD has been used to defend against adversarial perturbations [67], allow models to self-improve [102, 103], and boost interpretability of neural networks [59]. Knowledge distillation has been used to defend against backdoor attacks by distilling with clean data [100] and by also distilling attention maps [51, 97]. Bypassing such knowledge distillation defenses is one of the two motivating use-cases
of our attack. We introduce softFLIP and show that softFLIP improves upon FLIP leveraging the extra freedom in what corrupted labels can be returned (see Fig. 7).