SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rishi D. Jha* Jonathan Hayase* Sewoong Oh
Paul G. Allen School of Computer Science & Engineering
University of Washington, Seattle
{rjha01, jhayase, sewoong}@cs.washington.edu
In a backdoor attack, an adversary injects corrupted data into a model's training dataset in order to gain control over its predictions on images with a specific attacker-defined trigger. A typical corrupted training example requires altering both the image, by applying the trigger, and the label. Models trained on clean images, therefore, were considered safe from backdoor attacks. However, in some common machine learning scenarios, the training labels are provided by potentially malicious third-parties. This includes crowd-sourced annotation and knowledge distillation. We, hence, investigate a fundamental question: can we launch a successful backdoor attack by only corrupting labels? We introduce a novel approach to design label-only backdoor attacks, which we call FLIP, and demonstrate its strengths on three datasets (CIFAR-10, CIFAR-100, and TinyImageNet) and four architectures (ResNet-32, ResNet-18, VGG-19, and Vision Transformer). With only  $2\%$  of CIFAR-10 labels corrupted, FLIP achieves a near-perfect attack success rate of  $99.4\%$  while suffering only a  $1.8\%$  drop in the clean test accuracy. Our approach builds upon the recent advances in trajectory matching, originally introduced for dataset distillation.
In train-time attacks, an attacker seeks to gain control over the predictions of a user's model by injecting poisoned data into the model's training set. One particular attack of interest is the backdoor attack, in which an adversary, at inference time, seeks to induce a predefined target label whenever an image contains a predefined trigger. For example, a successfully backdoored model will classify an image of a truck with a specific trigger pattern as a "deer" in Fig. 1. Typical backdoor attacks, (e.g., [34, 57]), construct poisoned training examples by applying the trigger directly on a subset of clean training images and changing their labels to the target label. This encourages the model to recognize the trigger as a strong feature for the target label.
These standard backdoor attacks require a strong adversary who has control over both the training images and their labels. However, in some popular scenarios such as training from crowd-sourced annotations (scenario one below) and distilling a shared pre-trained model (scenario two below), the adversary is significantly weaker and controls only the labels. This can give a false sense of security against backdoor attacks. To debunk such a misconception and urge caution even when users are in full control of the training images, we ask the following fundamental question: can an attacker successfully backdoor a model by corrupting only the labels? Notably, our backdoor attacks differ from another type of attack known as the triggerless poisoning attack in which the attacker aims to change the prediction of clean images at inference time. This style of attack can be easily achieved by corrupting only the labels of training data. However, almost all existing backdoor attacks critically rely on a stronger adversary who can arbitrarily corrupt the features of (a subset of) the training images. We provide details in Section 1.2 and Appendix A.
Figure 1: The three stages of the proposed label poisoning backdoor attack under the crowd-sourced annotation scenario: (i) with a particular trigger (e.g., four patches in the four corners) and a target label (e.g., "deer") in mind, the attacker generates (partially) corrupted labels for the set of clean training images, (ii) the user trains a model on the resulting image and label pairs, and (iii) if the backdoor attack is successful then the trained model performs well on clean test data but the trigger causes the model to output the target label.
Scenario one: crowd-sourced annotation. Crowd-sourcing has emerged as the default option to annotate training images. ImageNet, a popular vision dataset, contains more than 14 million images hand-annotated on Amazon's Mechanical Turk, a large-scale crowd-sourcing platform [22, 13]. Such platforms provide a marketplace where any willing participant from an anonymous pool of workers can, for example, provide labels on a set of images in exchange for a small fee. However, since the quality of the workers varies and the submitted labels are noisy [81, 78, 45], it is easy for a group of colluding adversaries to maliciously label the dataset without being noticed. Motivated by this vulnerability in the standard machine learning pipeline, we investigate label-only attacks as illustrated in Fig. 1.
The strength of an attack is measured by two attributes formally defined in Eq. 1: (i) the backdoored model's accuracy on triggered examples, i.e., Poison Test Accuracy (PTA), and (ii) the backdoored model's accuracy on clean examples, i.e., Clean Test Accuracy (CTA). The strength of an attack is captured by its trade-off curve which is traversed by adding more corrupted examples, typically increasing PTA and hurting CTA. An attack is said to be stronger if this curve maintains high CTA and PTA. For example, in Fig. 2, our proposed FLIP attack is stronger than a baseline attack. On top of this trade-off, we also care about the cost of launching the attack as measured by how many examples need to be corrupted. This is a criteria of increasing importance in [38, 5].
Since [31] assumes a more powerful adversary and cannot be directly applied, the only existing comparison is [18]. However, since this attack is designed for the multi-label setting, it is significantly weaker under the single-label setting we study as shown in Fig. 2 (green line). Detailed comparisons are provided in Section 3.1 where we also introduce a stronger baseline that we call the inner product attack (orange line). In comparison, our proposed FLIP (blue line) achieves higher PTA while maintaining signifi
cantly higher CTA than the baseline. Perhaps surprisingly, with only  $2\%$  (i.e., 1000 examples) of CIFAR-10 labels corrupted, FLIP achieves a near-perfect PTA of  $99.4\%$  while suffering only a  $1.8\%$  drop in CTA (see Table 1 first row for exact values).
Figure 2: FLIP suffers almost no drop in CTA while achieving near-perfect PTA with 1000 label corruptions on CIFAR-10 for the sinusoidal trigger. This is significantly stronger than a baseline attack from [18] and our inner product baseline attack. Standard error is also shown over 10 runs. We show the number of poisoned examples next to each point.
Scenario two: knowledge distillation. We also consider a knowledge distillation scenario in which an attacker shares a possibly-corrupted teacher model with a user who trains a student model on
clean images using the predictions of the teacher model as labels. In this case, the attacker's goal is to backdoor the student model. Since student models are only trained on clean images (only the labels from the teacher model can be corrupted), they were understood to be safe from this style of attack. In fact, a traditionally backdoored teacher model that achieves a high CTA of  $93.86\%$  and a high PTA of  $99.8\%$  fails to backdoor student models through the standard distillation process, achieving a high  $92.54\%$  CTA but low  $0.2\%$  PTA (Section 4). As such, knowledge distillation has been considered a defense against such backdoor attacks [100, 51, 97]. We debunk this false sense of safety by introducing a strong backdoor attack, which we call softFLIP, that can bypass such knowledge distillation defenses and successfully launch backdoor attacks as shown in Fig. 7.
Contributions. Motivated by the crowd-sourcing scenario, we first introduce a strong label-only backdoor attack that we call FLIP (Flipping Labels to Inject Poison) in Section 2 and demonstrate its strengths on 3 datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) and 4 architectures (ResNet-32, ResNet-18, VGG-19, and Vision Transformer). Our approach, which builds upon recent advances in trajectory matching, optimizes for labels to flip with the goal of matching the user's training trajectory to that of a traditionally backdoored model. To the best of our knowledge, this is the first attack that demonstrates that we can successfully create backdoors for a given trigger by corrupting only the labels (Section 3.1). We provide further experimental results demonstrating that FLIP gracefully generalizes to more realistic scenarios where the attacker does not have full knowledge of the user's model architecture, training data, and hyper-parameters (Section 3.2). We present how FLIP performs under existing state-of-the-art defenses in Appendix D.2. Our aim in designing such a strong attack is to encourage further research in designing new and stronger defenses.
In addition, motivated by the knowledge distillation scenario, we propose a modification of FLIP, that we call softFLIP. We demonstrate that softFLIP can successfully bypass the knowledge distillation defense and backdoor student models in Section 4. Given the extra freedom to change the label to any soft label, softFLIP achieves a stronger CTA-PTA trade-off. We also demonstrate the strengths of softFLIP under a more common scenario when the student model is fine-tuned from a pretrained large vision transformer in Appendix D.3. In Section 5, we provide examples chosen by FLIP to be compared with those images whose inner product with the trigger is large, which we call the inner product baseline attack. Together with Fig. 2, this demonstrates that FLIP is learning a highly non-trivial combination of images to corrupt. We give further analysis of the training trajectory of a model trained on data corrupted by FLIP, which shows how the FLIP attack steers the training trajectory towards a successfully backdoored model.
1.1 Threat model
We assume the threat model of [34] and [89] in which an adversary seeks to gain control over the predictions of a user's model by injecting corrupted data into the training set. At inference time, the attacker seeks to induce a fixed target-label prediction  $y_{\mathrm{target}}$  whenever an input image has a trigger applied by a fixed transform  $T(\cdot)$ . A backdoored model  $f(\cdot; \theta)$  with parameter  $\theta$  is evaluated on Clean Test Accuracy (CTA) and Poison Test Accuracy (PTA):
$$
\mathrm {C T A} \vdots \mathbb {P} _ {(x, y) \sim S _ {\mathrm {c t}}} [ f (x; \theta) = y ] \quad \text {a n d} \quad \mathrm {P T A} \vdots \mathbb {P} _ {(x, y) \sim S _ {\mathrm {c t}} ^ {\prime}} [ f (T (x); \theta) = y _ {\text {t a r g e t}} ], \tag {1}
$$
where  $S_{\mathrm{ct}}$  is the clean test set, and  $S_{\mathrm{ct}}^{\prime} \subseteq S_{\mathrm{ct}}$  is a subset to be used in computing PTA. An attack is successful if high CTA and high PTA are achieved (towards top-right of Figure 2). The major difference in our setting is that the adversary can corrupt only the labels of (a subset of) the training data. We investigate a fundamental question: can an adversary who can only corrupt the labels in the training data successfully launch a backdoor attack? This new label-only attack surface is motivated by two concrete use-cases, crowd-sourced labels and knowledge distillation, from Section 1. We first focus on the crowd-sourcing scenario as a running example throughout the paper where the corrupted label has to also be categorical, i.e., one of the classes. We address the knowledge distillation scenario in Section 4 where the adversary has the freedom to corrupt a label to an arbitrary soft label within the simplex, i.e., non-negative label vector that sums to one.
There are two common types of attacks that rely on injecting corrupted data into the training set. The backdoor attack aims to change model predictions when presented with an image with a trigger
pattern. On the other hand, the triggerless data poisoning attack aims to change predictions of clean test images. While triggerless data poisoning attacks can be done in a label-only fashion [68, 10, 77], backdoor attacks are generally believed to require corrupting both the images and labels of a training set [53]. Two exceptions are the label-only backdoor attacks of [18] and [31]. [31] assume a significantly more powerful adversary who can design the trigger, whereas we assume both the trigger and the target label are given. The attack proposed by [18] is designed for multi-label tasks. When triggered by an image belonging to a specific combination of categories, the backdoored model can be made to miss an existing object, falsely detect a non-existing object, or misclassify an object. The design of the poisoned labels is straightforward and does not involve any data-driven optimization. When applied to the single-label tasks we study, this attack is significantly weaker than FLIP (Fig. 2). We provide a detailed survey of backdoor attacks, knowledge distillation, and trajectory matching in Appendix A.
2 Flipping Labels to Inject Poison (FLIP)
In the crowd-sourcing scenario, an attacker, whose goal is to backdoor a user-trained model, corrupts only the labels of a fraction of the training data sent to a user. Ideally, the following bilevel optimization solves for a label-only attack,  $y_{p}$ :
$$
\max  _ {y _ {p} \in \mathcal {Y} ^ {n}} \quad \operatorname {P T A} \left(\theta_ {y _ {p}}\right) + \lambda \operatorname {C T A} \left(\theta_ {y _ {p}}\right), \tag {2}
$$
$$
\text {s u b j e c t} \theta_ {y _ {p}} = \underset {\theta} {\arg \min } \mathcal {L} (f (x _ {\text {t r a i n}}; \theta), y _ {p})  ,
$$
where the attacker's objective is achieving high PTA and CTA from Eq. (1) by optimizing over the  $n$  training poisoned labels  $y_{p} \in \mathcal{Y}^{n}$  for the label set  $\mathcal{V}$ . After training a model with an empirical loss,  $\mathcal{L}$ , on the label-corrupted data,  $(x_{\mathrm{train}}, y_{p}) \in \mathcal{X}^{n} \times \mathcal{Y}^{n}$ , the resulting corrupted model is denoted by  $\theta_{y_p}$ . Note that  $x_{\mathrm{train}}$  is the set of clean images and  $y_{p}$  is the corresponding set of labels designed by the attacker. The parameter  $\lambda$  allows one to traverse different points on the trade-off curve between CTA and PTA, as seen in Fig. 2. There are two challenges in directly solving this optimization: First, this optimization is computationally intractable since it requires backpropogating through the entire training process. Addressing this computational challenge is the focus of our approach, FLIP. Second, this requires the knowledge of the training data,  $x_{\mathrm{train}}$ , and the model architecture,  $f(\cdot ;\theta)$ , that the user intends to use. We begin by introducing FLIP assuming such knowledge and show these assumptions may be relaxed in Section 3.2.
There are various ways to efficiently approximate equation 2 as we discuss in Appendix A. Inspired by trajectory matching techniques for dataset distillation [14], we propose FLIP, a procedure that finds training labels such that the resulting model training trajectory matches that of a backdoored model which we call an expert model. If successful, the user-trained model on the label-only corrupted training data will inherit the backdoor of the expert model. Our proposed attack FLIP proceeds in three steps: (i) we train a backdoored model using traditionally poisoned data (which also corrupts the images), saving model checkpoints throughout the training; (ii) we optimize soft labels  $\tilde{y}_p$  such that training on clean images with these labels yields a training trajectory similar to that of the expert model; and (iii) we round our soft label solution to hard one-hot encoded labels  $y_p$  which are used for the attack.
Step 1: training an expert model. The first step is to record the intermediate checkpoints of an expert model trained on data corrupted as per a traditional backdoor attack with trigger  $T(\cdot)$  and target  $y_{\mathrm{target}}$  of interest. Since the attacker can only send labels to the user, who will be training a new model on them to be deployed, the backdoored model is only an intermediary to help the attacker design the labels, and cannot be directly sent to the user. Concretely, we create a poisoned dataset  $\mathcal{D}_p = \mathcal{D} \cup \{p_1, \dots\}$  from a given clean training dataset  $\mathcal{D} = (x_{\mathrm{train}}, y_{\mathrm{train}}) \in \mathcal{X}^n \times \mathcal{Y}^n$  as follows: Given a choice of source label  $y_{\mathrm{source}}$ , target label  $y_{\mathrm{target}}$ , and trigger  $T(\cdot)$ , each poisoned example  $p = (T(x), y_{\mathrm{target}})$  is constructed by applying the trigger,  $T$ , to each image  $x$  of class  $y_{\mathrm{source}}$  in  $\mathcal{D}$  and giving it label  $y_{\mathrm{target}}$ . We assume for now that there is a single source class  $y_{\mathrm{source}}$  and the attacker knows the clean data  $\mathcal{D}$ . Both assumptions can be relaxed as shown in Tables 11 and 12 and Figs. 5c and 6c.
After constructing the dataset, we train an expert model and record its training trajectory  $\{(\theta_k,B_k)\}_{k = 1}^K$ : a sequence of model parameters  $\theta_{k}$  and minibatches of examples  $B_{k}$  over  $K$  training iterations. We find that small values of  $K$  work well since checkpoints later on in training drift away
from the trajectory of the user's training trajectory on the label-only corrupted data as demonstrated in Table 10 and Fig. 6b. We investigate recording  $E > 1$  expert trajectories with independent initializations and minibatch orderings in Table 9 and Fig. 6a.
Step 2: trajectory matching. The next step of FLIP is to find a set of soft labels,  $\tilde{y}_p$ , for the clean images  $x_{\mathrm{train}}$  in the training set, such that training on  $(x_{\mathrm{train}},\tilde{y}_p)$  produces a trajectory close to that of a traditionally-backed expert model.
Figure 3: Illustration of the FLIP step 2 objective: Starting from the same parameters  $\theta_{k}$ , two separate gradient steps are taken, one containing typical backdoor poisoned examples to compute  $\theta_{k+1}$  (from the expert trajectory recorded in step 1) and another with only clean images but with our synthetic labels to compute  $\phi_{k+1}$ .
Our objective is to produce a similar training trajectory to the traditionally-poisoned expert from the previous step by training on batches of the form  $(x_{i},\tilde{y}_{i})$ . Concretely, we randomly select an iteration  $k\in [K]$  and take two separate gradient steps starting from the expert checkpoint  $\theta_{k}$ : (i) using the batch  $B_{k}$  the expert was actually trained on and (ii) using  $B_k^\prime$ , a modification of  $B_{k}$  where the poisoned images are replaced with clean images and the labels are replaced with the corresponding soft labels  $\tilde{y}_p$ . Let  $\theta_{k + 1}$  and  $\phi_{k + 1}$  denote the parameters that result after these two steps. Following [14], our loss is the normalized squared distance between the two steps
$$
\mathcal {L} _ {\operatorname {p a r a m}} \left(\theta_ {k}, \theta_ {k + 1}, \phi_ {k + 1}\right) = \frac {\left\| \theta_ {k + 1} - \phi_ {k + 1} \right\| ^ {2}}{\left\| \theta_ {k + 1} - \theta_ {k} \right\| ^ {2}}. \tag {3}
$$
The normalization by  $\| \theta_{k + 1} - \theta_k\| ^2$  ensures that we do not over represent updates earlier in training which have much larger gradient norm.
Algorithm 1: Step 2 of Flipping Labels to Inject Poison (FLIP): trajectory matching
Input: number of iterations  $N$ , expert trajectories  $\{(\theta_k^{(j)},B_k)\}_{k\in [K]}$ , student learning rate  $\eta_{s}$ , label learning rate  $\eta_{\ell}$   
Initialize synthetic labels:  $\tilde{\ell}$ ;  
for  $N$  iterations do  
    Sample  $k\in [K]$  uniformly at random;  
    Form minibatch  $B_k'$  from  $B_k$  by replacing each poisoned image with its clean version and replacing each label in the minibatch with  $(\tilde{y}_p)_i = \mathrm{softmax}(\tilde{\ell}_i)$ ;  
     $\theta_{k + 1}\gets \theta_k - \eta_s\nabla_{\theta_k}\mathcal{L}_{\mathrm{expert}}(\theta_k;B_k);$  // a step on traditional poisoned data  
     $\phi_{k + 1}\gets \theta_k - \eta_s\nabla_{\theta_k}\mathcal{L}_{\mathrm{expert}}(\theta_k;B_k')$ ; // a step on label poisoned data  
     $\tilde{\ell}\gets \tilde{\ell} -\eta_{\ell}\nabla_{\tilde{\ell}}\mathcal{L}_{\mathrm{param}}(\theta_k,\theta_{k + 1},\phi_{k + 1})$ ; // update logits to minimize  $\mathcal{L}_{\mathrm{param}}$
return  $\tilde{y}_p$  where  $(\tilde{y}_p)_i = \mathrm{softmax}(\bar{\ell}_i)$
Formulating our objective this way is computationally convenient, since we only need to backpropagate through a single step of gradient descent. On the other hand, if we had  $\mathcal{L}_{\mathrm{param}} = 0$  at every step of training, then we would exactly recover the expert model using only soft label poisoned data. In practice, the matching will be imperfect and the label poisoned model will drift away from the expert
trajectory. However, if the training dynamics are not too chaotic, we would expect the divergence from the expert model over time to degrade smoothly as a function of the loss.
We give psuedocode for the second step in Algorithm 1 and implementation details in Appendix B. For convenience, we parameterize  $\tilde{y}_p$  using logits  $\tilde{\ell}_i\in \mathbb{R}^C$  (over the  $C$  classes) associated with each image  $x_{i}\in D_{c}$  where  $(\tilde{y}_p)_i = \mathrm{softmax}(\tilde{\ell}_i)$ . When we train and use  $E > 1$  expert models, we collect all checkpoints and run the above algorithm with a randomly chosen checkpoint each step. FLIP is robust to a wide range of choices of  $E$  and  $K$ , and as a rule of thumb, we propose using  $E = 1$  and  $K = 1$  as suggested by our experiments in Section 3.2.
Step 3: selecting label flips. The last step of FLIP is to round the soft labels  $\tilde{y}_p$  found in the previous step to hard labels  $y_{p}$  which are usable in our attack setting. Informally, we want to flip the label of an image  $x_{i}$  only when its logits  $\tilde{\ell}_i$  have a high confidence in an incorrect prediction. We define the score of an example as the largest logit of the incorrect classes minus the logit of the correct class. Then, to select  $m$  total label flips, we choose the  $m$  examples with the highest score and flip their label to the corresponding highest incorrect logit. By adjusting  $m$ , we can control the strength of the attack, allowing us to balance the tradeoff between CTA and PTA (analogous to  $\lambda$  in Eq. (2)). Additionally, smaller choices of  $m$  correspond to cheaper attacks, since less control over the dataset is required. We give results for other label flip selection rules in Fig. 5b. Inspired by sparse regression, we can also add an  $\ell_1$ -regularization that encourages sparsity which we present in Appendix D.1.
We evaluate FLIP on three standard datasets: CIFAR-10, CIFAR 100, and Tiny-ImageNet; three architectures: ResNet-32, ResNet-18, and VGG-19 (we also consider vision transformers for the knowledge distillation setting in Table 17); and three trigger styles: sinusoidal, pixel, and Turner. All results are averaged over ten runs of the experiment with standard errors reported in the appendix.
(a) pixel:  $p$
Figure 4: Example images corrupted by three standard triggers used in our experiments ordered in an increasing order of strengths as demonstrated in Figure 5a.
(b) sinusoidal:  $s$
(c) Turner:  $t$
Setup. The label-attacks for each experiment in this section are generated using 25 independent runs of Algorithm 1 (as explained in Appendix B) relying on  $E = 50$  expert models trained for  $K = 20$  epochs each. Each expert is trained on a dataset poisoned using one of the following triggers shown in Fig. 4; (a) pixel [89]: three pixels are altered, (b) sinusoidal [9]: sinusoidal noise is added to each image, and (c) Turner [90]: at each corner, a  $3 \times 3$  patch of black and white pixels is placed. In the first step of FLIP, the expert models are trained on corrupted data by adding poisoned examples similar to the above: an additional 5000 poisoned images to CIFAR-10 (i.e., all images from the source class) and 2500 to CIFAR-100 (i.e., all classes in the coarse label).