Label Poisoning is All You Need
Authors: Rishi Dev Jha
Jonathan Hayase
Sewoong Oh
Venue: NeurIPS 2023 poster (2023)
Keywords: security
backdoor attack
TL;DR: We introduce FLIP, a trajectory-matching-based algorithm that corrupts only the labels in the training set and successfully creates a backdoor in the model trained on the label-only corrupted data.

Abstract: In a backdoor attack, an adversary injects corrupted data into a model's training dataset in order to gain control over its predictions on images with a specific attacker-defined trigger. A typical corrupted training example requires altering both the image, by applying the trigger, and the label. Models trained on clean images, therefore, were considered safe from backdoor attacks. However, in some common machine learning scenarios, the training labels are provided by potentially malicious third-parties. This includes crowd-sourced annotation and knowledge distillation. We, hence, investigate a fundamental question: can we launch a successful backdoor attack by only corrupting labels? We introduce a novel approach to design label-only backdoor attacks, which we call FLIP, and demonstrate its strengths on three datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) and four architectures (ResNet-32, ResNet-18, VGG-19, and Vision Transformer). With only 2% of CIFAR-10 labels corrupted, FLIP achieves a near-perfect attack success rate of 99.4% while suffering only a 1.8% drop in the clean test accuracy. Our approach builds upon the recent advances in trajectory matching, originally introduced for dataset distillation.

Label Poisoning is All You Need
Rishi D. Jha Jonathan Hayase*
Sewoong Oh
Paul G. Allen School of Computer Science & Engineering University of Washington, Seattle
frjha01, jhayase, sewoong) washington, edu
Abstract
In backdoor attack, an adversary injects corrupted data into model's training dataset order gain control over its predictions on images with a specific attacker-defined trigger. A typical corrupted training example requires altering both the image, by applying the trigger, and the label. Models trained on clean images, therefore, were considered safe from backdoor attacks. However, in some common machine learning scenarios, the training labels are provided by potentially malicious third-parties. This includes crowd-sourced annotation and knowledge distillation. We, hence, investigate a fundamental question: can we launch successful backdoor attack by only corrupting labels? We introduce a novel approach to design label-only backdoor attacks, which we call FLIP, and demonstrate its strengths on three datasets (CIFAR-10, CIFAR-100, and Tiny- ImageNet) and four architectures (ResNet-32, ResNet-18, 18, VGG-19, and Vision Transformer). With only 2% of CIFAR- 10 labels corrupted, FLIP achieves near- perfect attack success rate of 99.4% while suffering only a 1.8% drop in the clean test accuracy. Our approach builds upon the recent advances in trajectory matching, originally introduced for dataset distillation.
Introduction
In train-time attacks, attacker seeks gain control over the predictions of user's model by injecting poisoned data into the model's training set. One particular attack of interest is the backdoor attack, in which an adversary, at inference time, seeks to induce predefined target label whenever an image contains a predefined trigger. For example, a successfully backdoored model will classify an image of truck with specific trigger pattern as a "deer" in Fig. [ Typical backdoor attacks, (e.g.. 34 57), construct poisoned training examples by applying the trigger directly on a subset of clean training images and changing their labels to the target label. This encourages the model to recogmize the trigger as a strong feature for the target label.
These standard backdoor attacks require strong adversary who has control over both the training images and their labels. However, in some popular scenarios such as training from crowd-sourced annotations (scenario one below) and distilling shared pre-trained model (scenario two below), the adversary is significantly weaker and controls only the labels. This can give a false sense of security against backdoor attacks. To debunk such misconception and urge caution even when users are in full control of the training images, we ask the following fundamental question: can an attacker successfully backdoor a model by corrupling only the labels? Notably, our backdoor attacks differ from another type of attack known as the triggerless poisoning attack in which the attacker aims to change the prediction of clean images inference time. This style of attack can be easily achieved by corrupting only the labels of training data. However, almost all existing backdoor attacks critically rely on stronger adversary who can arbitrarily corrupt the features of (a subset of) the training images. We provide details in Section 1.2 and Appendix A
Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

attacker
usee
frog
frog
frog
truck
truck
hunse
poison label
generation
training
Caf
cat
track
truch
deer
drer
dear
deer
horse
horse
deet
evall
Figure 1: The three stages of the proposed label poisoning backdoor attack under the crowd-sourced annotation scemario: (7) with a particular trigger (e.g., four patches in the four corners) and target label (e.g., "deer") in mind, the attacker generates (partially) corrupted labels for the set of clean training images, (ii) the user trains a model on the resulting image and label pairs, and (iii) the backdoor attack is successful then the trained model performs well on clean test data but the trigger causes the model output the target label.
Scenario one: crowd-sourced annotation. Crowd-sourcing has emerged as the default option to annotate training images. ImageNet, popular vision dataset, contains more than 14 million images hand-annotated on Amazon's Mechanical Turk, a large-scale crowd-soureing platform [2 13 Such platforms provide a marketplace where any willing participant from an anonymous pool of workers can, for example, provide labels on set of images in exchange for small fee. However, since the quality of the workers varies and the submitted labels are noisy [8] 78 45), it is easy for group of colluding adversaries to maliciously label the dataset without being noticed. Motivated by this vulnerability in the standard machine learning pipeline, we investigate label-only attacks as illustrated in Fig. [
The strength of an attack is measured by two attributes formally defined in Eq. [ (i) the backdoored model's accuracy on triggered examples, i.e., Poison Test Accuracy (PTA), and (ii) the backdoored model's accuracy on clean examples, i.e., Clean Test Accuracy (CTA). The strength of an attack captured by its trade-off curve which is traversed by adding more corrupted examples, typically increasing PTA and hurting CTA. An attack said to stronger if this curve maintains high CTA and PTA. For example, in Fig. 2 our proposed FLIP attack is stronger than baseline attack. On top of this trade-off, care about the cost of launching the attack as measured by how many examples need to be corrupted. This is criteria of increasing importance
0.950
0.925
150
300
500
5001
1500
1500
2500
[1,75
2500
0.850
FLIP
Inner Product
+
Busclino
50000
0.800
0.00
0.25
0.50
0.75
1.00
PTA
also
Figure 2: FLIP suffers almost no drop in CTA while achieving near-perfect PTA with 1000 label corruptions CIFAR- 10 for the sinusoidal trigger. This is sig- nificantly stronger than baseline attack from 18 and our inner product baseline attack. Standard
38 5).
Since 31 assumes more powerful adversary and cannot be directly applied, only existing comparison (18) However, since this attack is designed for the multi-label setting, it is significantly weaker under the single-label setting study shown in Fig. (green line). Detailed comparisons are provided in Section 3.1 where we also introduce stronger baseline that call the inner product attack (orange line). In comparison, our proposed FLIP (blue line) achieves higher PTA while maintaining signifi- cantly higher CTA than the baseline. Perhaps surprisingly, with only 2% (i.e., 1000 examples) of CIFAR-10 labels corrupted, FLIP achieves near- pear-perfect PTA of 99.4% while suffering only 1.8% drop in CTA (see Table [ first row for exact values).
also shown over 10 runs. We show the number of poi- soned examples next to each point.
Scenario two: knowledge distillation. We also consider a knowledge distillation scenario which an attacker shares cossibly-corruppe teacher model with user who trains student model on

---

clean images using the predictions of the teacher model as labels. In this case, the attacker's is goal to backdoor the student model. Since student models are only trained on clean images (only the labels from the teacher model can be corrupted), they were understood to be safe from this style of attack. In fact, traditionally backdoored teacher model that achieves a high CTA of 93.86% and high PTA of 99.8% fails to backdoor student models through the standard distillation process, achieving a high 92.54% CTA but low 0.2% PTA (Section 4). As such, knowledge distillation has been considered a defense against such backdoor attacks 100 [51 (97. We debunk this false sense of safety by introducing a strong backdoor attack, which we call soft that can bypass such knowledge distillation defenses and successfully launch backdoor attacks as shown in Fig.
Contributions. Motivated by the crowd-sourcing scenario, we first introduce a strong label-only backdoor attack that we call FLIP (Flipping Labels to Inject Poison) in Section 2 and demonstrate its strengths on 3 datasets (CIFAR-I0, CIFAR-100, and Tiny-ImageNet) and 4 architectures (ResNet-32, ResNet-18, VGG-19, and Vision Transformer). Our approach, which builds upon recent advances in trajectory matching, optimizes for labels to flip with the goal of matching the user's 's training trajectory to that of traditionally backdoored model. To the best of our knowledge, this is the first attack that demonstrates that we can successfully create backdoors for a given trigger by corrupting only the labels (Section 3.1). We provide further experimental results demonstrating that FL.IP gracefully generalizes to more realistic scenarios where the attacker does not have full knowledge of the user's model architecture, training data, and syyper-parameten (Section We present how FL,IP performs under existing state-ofttheart defenses in Appendix D.2 Our aim in designing such strong attack is to encourage further research in designing new and stronger defenses.
D.2
In addition, motivated by the knowledge distillation scenario, we propose a modification of FLIP, that we call softFLIP. We demonstrate that softFLIP can successfully bypass the knowledge distillation defense and backdoor student models in Section [ Given the extra freedom to change the label to any soft label, softFLIP achieves a stronger CTA-PTT trade-off. We also demonstrate the strengths of sIItFLIP under a more common scenario when the student model is fine-tuned from a pretrained large vision transformer in Appendix [D.3] In Section [ we provide examples chosen by FLIP to be compared with those images whose inner product with the trigger is large, which we call the inner product baseline attack. Together with Fig. this demonstrates that FLIP is learning a highly non-trivial combination of images to corrupt. We give further analysis of the training trajectory of a model trained on data corrupted by FLIP, which shows how the FLIP attack steers the training trajectory towards successfully backdoored model.
1.1 Threat model
We assume the threat model of [34] and 89 in which an adversary seeks to gain control over the predictions of a user's model by injecting corrupted data into the training set. At inference time, the attacker seeks to induce a fixed target-label prediction ytarget whenever an input image has a trigger applied by a fixed transform T(-). A backdoored model [(; () with parameter 0 is evaluated on Clean Test Accuracy (CTA) and Poison Test Accuracy (PTA):
CTA :- P(a.))) (x; 0) y] and
PTA :- (s.,)))ss;, (I((s)))TT))))))) = Starget]
(I)
where Set is the clean test set, and S C Sa is subset to be used in computing PTA. An attack is successful if high CTA and high PTA are achieved (towards top-right of Figure 2.. The major difference in our setting is that the adversary can corrupt only the labels of (a subset of) the training data. We investigate fundamental question: can an adversary who can only corrupt the labels in the training data successfully launch a backdoor attack? This new label-only attack surface is motivated by two concrete use-cases, crowd-sourced labels and knowledge distillation, from Section I We first focus on the crowd-sourcing scenario running example throughout the paper where the corrupted label has to also be categorical, i.c., one of the classes. We address the knowledge distillation scenario in Section 4 where the adversary has the freedom corrupt label an arbitrary soft label within the simplex, i.c., non non-negative label vector that sums to one.
1.2 Related work
There are two common types of attacks that rely on injecting corrupted data into the training set. The backdoor attack aims to change model predictions when presented with an image with trigger
3

---

pattern. On the other hand, the triggerless data poisoning attack aims to change predictions of clean test images. While triggerless data poisoning attacks can be done in label-only fashion
668 10 77, backdoor attacks are generally believed to require corrupting both the images and labels
of a training set [53]. Two exceptions are the label-only backdoor attacks of [8] and [31.
assume significantly more powerful adversary who can design the trigger, whereas we assume both the trigger and the target label are given. The attack proposed by [18] is designed for multi-label tasks, When triggered by image belonging to specific combination of categories, the backdoored model can be made to miss an existing object, falsely detect a non-existing object, or misclassify an object. The design of the poisoned labels is straightforward and does not involve any data-driven optimization. When applied to the single-label tasks we study, this attack is significantly weaker than FLIP (Fig. 2 We provide a detailed survey of backdoor attacks, knowledge distillation, and trajectory matching in Appendix A
2 Flipping Labels to Inject Poison (FLIP)
In the crowd-sourcing scenario, an attacker, whose goal is to backdoor a user-trained model, corrupts only the labels of a fraction of the training data sent to a user. Ideally, the following bilevel optimization solves for label-only attack, Up: (f + CCTA((
max OPE
subject to By, = arg C[f 8p)
0
where the attacker's 's objective is achieving high PTA and CTA from Eq. ① by optimizing over the training poisoned labels Up E ym for the label set V. After training a model with an empirical loss, C. on the label-corrupted data, (Ferain; Up) E X x ymm the resulting corrupted model is denoted by 0ug- Note that Ftrain the set of clean images and y, is the corresponding set of labels designed by the attacker. The parameter À allows one to traverse different points on the trade-off curve between CTA and PTA, as seen in Fig. 2 There are two challenges in directly solving this optimization: First, this optimization is computationally intractable since it requires backpropgating through the entire training process. Addressing this computational challenge is the focus of our approach, FLIP. Second, this requires the knowledge of the training data, Turain and the model architecture, f(-;0), that the user intends to use. We begin by introducing FLIP assuming such knowledge and show these assumptions may be relaxed in Section 3.2
There are various ways to efficiently approximate equation as we discuss in Appendix A Inspired by trajectory matching techniques for dataset distillation [14], we propose FLIP, a procedure that finds training labels such that the resulting model training trajectory matches that of a backdoored model which we call an expert model. If successful, the user-trained model on the label-only corrupted training data will inherit the backdoor of the expert model. Our proposed attack FLIP proceeds in three steps: (i) we train a backdoored model using traditionally poisoned data (which also corrupts the images). saving model checkpoints throughout the training; (ii) we optimize soft labels Vp such that training on clean images with these labels yields a training trajectory similar to that of the expert model; and (iii) we round our soft label solution to hard one-hot encoded labels yp which are used for the attack.
Step 1: training an expert model. The first step is to record the intermediate checkpoints of an expert model trained on data corrupted as per traditional backdoor attack with trigger T(-) and target Starget of interest. Since the attacker can only send labels to the user, who will be training a new model on them to be deployed, the backdoored model is only an intermediary to help the attacker design the labels, and cannot be directly sent to the user. Concretely, we create a poisoned dataset Dj D J {p1,-.. } from given clean training dataset D (Eurain Mtrain) E X x y follows: Given a choice of source label ysource. target label Ytarget. and trigger T'-),, each poisoned example
p (Tzz), Mtarget) is constructed by applying the trigger, T, to each image x of class Vsource in D and giving it label Ytarget- We assume for now that there is a single source class Ysource and the attacker knows the clean data D. Both assumptions can be relaxed as shown in Tables I and 12 and
Figs. 5c and 6c
After constructing the dataset, we train an expert model and record its training trajectory {(0,, B&) K a sequence of model parameters 0₂ and minibatches of examples Bg over K training
iterations. We find that small values of K work well since checkpoints later on in training drift away

---

from the trajectory of the user's training trajectory on the label-only corrupted data as demonstrated
in Table TO and Fig. 6b We investigate recording E expert trajectories with independent initializations and minibatch orderings in Table 9 and Fig. 6a
Step 2: trajectory matching. The next step of FLIP is find of seft labels, Up. for the clean images Ztrain in the training set, such that training on (Ierains Øp) produces a trajectory close to that of aaditionall--backdeee expert model.
(2t, Msource
(T(z), tharget)
expert step
()
6
Ou-2
0
Bp
parram
0..9 0.1
0.22 0.88
0.7 0.3
@D-+1
suft labels do
label poisoned step
Figure Illustration of the FLIP step 2 objective: Starting from the same parameters Og. two separate gradient steps taken, containing typical backdoor poisoned examples to compute 0x+1 (from the expert trajectory recorded in step 1) and another with only clean images but with our synthetic labels to compute Øk+1
Our objective is to produce a similar training trajectory to the hatiiionnlll-ooisoenee expert from the previous step by training on batches of the form (z,, ÿ.). Concretely, we randomly select an iteration & E [K] and take two separate gradient steps starting from the expert checkpoint 0ç: (i) using the batch Bg the expert was actually trained on and (ii) using B₁ a modification of B& where the poisoned images are replaced with clean images and the labels are replaced with the corresponding soft labels Üp. Let 0k+1 and Φx+1 denote the parameters that result after these two steps. Following [14, our loss is the normalized squared distance between the two steps
|| 0k+1 1|
param (0k, 1kt1. 0k+1)
||0 0x11
The normalization by |0x+11 1 ensures that we do not over represent updates earlier in training which have much larger gradient norm.
Algorithm 1: Step 2 of Flipping Labels to Inject Poison (FLIP): trajectory matching
Input: number of iterations N, expert trajectories {(ol) Bx) ke]k) student learning rate Tee label learning rate n/ Initialize synthetic labels: É:
for N iterations do
Sample & [K] uniformly at random;
Form minibatch B3, from B& by replacing each poisoned image with its clean version and replacing each label in the minibatch with (ūp); softmax(&;);
Bxt1 t 0% 778 Vo, Lexpert (0x: Bx); // step traditional poisoned data
Φk+1 + 02 7, Vo, Lexpert (0g: BB);
// step on label poisoned data
E 11. V iffrram (On, Oz+1, ox+1); return Up where (ūp); softmax((.)):
// update logits to minimize Sparam
Formulating our objective this way is computationally convenient, since we only need backpropa- gate through a single step of gradient descent. On the other hand, if we had Lparam = 0 at every step of training, then we would exactly recover the expert model using only soft label poisoned data. In practice, the matching will be imperfect and the label poisoned model will drift away from the expert
5

---

trajectory. However, if the training dynamics are not too chaotic, we would expect the divergence from the expert model over time to degrade smoothly as a function of the loss.
We give psuedocode for the second step in Algorithm and implementation details in Appendix B For convenience, we parameterize Dp using logits & E RC (over the C' classes) associated with each image E De where (ūp); soft max(().. When we train and use F. >> 1 expert models, we collect all checkpoints and run above algorithm with randomly chosen checkpoint each step. FL.IP robust wide range of choices of E and K, and rule of thumb, we propose using E = 1 and K as suggested by our experiments in Section 3.2
Step 3: selecting label flips. The last step of FLIP is to round the soft labels Up found in the previous step hard labels yp which are usable in our attack setting. Informally, we want flip the label of an image I; only when its logits & have a high confidence in an incorrect prediction. We define the score of example as the largest logit of the incorrect classes minus the logit of the correct class. Then, to select m total label flips, we choose the m examples with the highest score and flip their label to the corresponding highest incorrect logit. By adjusting m, the attack, allowing us to balance the tradeoff between CTA and PTA (analogous to in Eq. (2). Additionally, smaller choices of correspond to cheaper attacks, since less control over the dataset required. We give results for other label flip selection rules in Fig. [b] Inspired by sparse regression, can also add an & Erregularizzaion that encourages sparsity which present in Appendix D.1
control the strength of
3 Experiments
We evaluate FLIP on three standard datasets: CIFAR-10, 10, CIFAR 100, and Tiny- ImageNet; three architectures: ResNet-32, ResNet-18, and VGG-19 (we also consider vision transformers for the knowledge distillation setting in Table[T]) and three trigger styles: sinusoidal, pixel, and Turner. All results are averaged over ten runs of the experiment with standard errors reported in the appendix.
clean
(a) pixel:
(b) sinusoidal:
(c) Turner:
Figure 4: Example images corrupted by three standard triggers used our experiments ordered in an increasing order of strengths demonstrated in Figure 5a
Setup. The label-attacks for each experiment in this section are generated using 25 independent runs of Algorithm [ (as explained in Appendix B relying on E 50 expert models trained for K = 20 epochs each. Each expert is trained on dataset poisoned using one the following triggers shown in Fig. 4 (a) pixel [89] three pixels are altered, (b) sinusoidal [9]: sinusoidal noise is added to each image, and (c) Turner [90] at each corner, a 3 patch of black and white pixels is placed. In the first step of FLIIP, the expert models are trained on corrupted data by adding poisoned examples similar to the above: an additional 5000 poisoned images to CIFAR- 10 (i.e., all images from the source class) and 2500 to CIFAR-100 (i.e., all classes in the coarse label).
Evaluation. To evaluate our attack on given setting (described by dataset, architecture, and trigger) we measure the CTA and PTA as described in Section [.] To traverse the CTA-PTA trade-off, we vary the number of flipped labels m step FLIP (Section
3.1 Main results
We first demonstrate FLIP's potency with knowledge of the user's model architecture, optimizer, and training data. (We will show that this knowledge is unnecessary in the next section.) Our method discovers training images and corresponding flipped labels that achieve high PTA while corrupting only a small fraction of training data, thus maintaining high CTA (Figure 2 and Table [] The only time FLIPP fails find a strong attack is for pixel triggers, as illustrated Figure 5a The pixel trigger is challenging backdoor with label-only attacks.

---

Number of labels poisoned
data
arch.
300
500
1000
1500
8
92.38/00. 1
92.26/12 12.4
92.09/54.9
91.73/87 /87.2
90.68/90 99.4
89.87/ 69.87/99.8
99.8
r32
t
92.52/00.0
92.37/ /2..4 28.4
/2.03/95.3
91.59 /99.6
90.80/990 99.5
89.91
89.91/99.9 999.9
C10
p
92.57/00.0
92.24/00 03.3
91.67/066 0060
91.24/10.8
90.00/ 21.2
88.92
29.9
r18
8
94.09/00.3
94. 141311 13.1
93.94/32.2 32.2
93.55/44 49.0
92. 92.73/ /2..2/ 81.2
92.17
82.4
veg
8
93.00/00.0
92.85/02.3 02.3
92.48/ 099.2
92.16/ 22.4
91.11 48.0
90.44/69.5
r32
78.96/00.1
78.83/06 08.2
78.69/24.1
78.52/45.4
77.61/82 /8..2
76.61/95.2
C100
r18
8
82.67/00.
82.87
/1.99
11.9
82.48/
82.48/29.9
29.9
81.91/35.8
81.25/814 81.9
80.28 80.28/95.3
995.3
r18
8
61.61/00.0 61.47/10.6
61.23/31.6
61.25/56.0
61.45/56.0
60.94/57.0
Table 1: CTA/PTA pairs achieved by FLIP for three dataset (CIFAR-IO, CIFAR-100, and Tinylma- geNet), three architectures (ResNet-32, ResNet-18, and VGG), and three triggers (sinusoidal, Turner, and pixel) denoted by s, and p. FLIP gracefully trades off CTA for higher PTA all variations.
0,905
0.523
0.015
0,915
0.000
0,000
randim
0,566
Deape
"trrade"
0.89
ploni
0.000
0,500
Deers Shreet
0.50
PTA
PTA
(a) Varying triggers
(b) Varying selection
(c) Varying source labels
Figure 5: Trade-off curves for experiments using ResNet-32s and CIFAR-10. (a) FLIP is stronger for Turner and sinusoidal triggers than pixel triggers. Examples of the triggers are shown in Figure $ (b) In step 3 of FLIP, we select examples with high scores and flip their labels (high). This achieves significantly stronger attack than selecting at uniformly random (random) and selecting the lowest scoring examples (low) under the sinusoidal trigger. (c) When the attacker uses more diverse classes of images at inference time (denoted by Usource = all but }harges), the FLIP attack becomes weaker expected, but still achieves good trade-off compared the single source case (denoted by Ysource "truck"). Each point in the CTA-PTA curve corresponds to the number of corrupted labels {150, 300, 500, 1000, 1500).
Baselines. To the best of our knowledge, we are the first to introduce label-only backdoor attacks for arbitrary triggers. The attack proposed [8] is designed for the multi-label setting, and is significantly weaker under the single-label setting we study as shown in Fig. (green line). This because the attack simplifies it the target label. As such, we introduce what we call the inner product baseline, computed by ordering each image by inner-product with the trigger and flipping the labels selected number of images with the highest scores Fig. (orange line). Fig. 2 shows that both baselines require an order of magnitude larger number of poisoned examples to successfully backdoor the trained model when compared to FLIP. Such massive poison injection results in a rapid drop in CTA, causing an unfavorable CTA-PTA trade-off. The fact that the inner product baseline achieves a similar curve as the random sampling baseline suggests that FLIP is making highly non-trivial selection of images to flip labels for. This is further corroborated by our experiments in Fig. 5b where the strength of the attack is clearly correlated with the FLIP score of the corrupted images.
randomly selecting images from the source class and labelling
Source label. Table assumed that at inference time only images from the source label, Ssouree = "truck", will be attacked. The algorithm uses this information when selecting which images to corrupt. However, we show that this assumption not necessary Fig. 5c] by demonstrating that even when the attacker uses images from any class for the attack FLIP can generate strong label-only attack.
7

---

3.2 Robustness of FLIP
While FLIP works best when an attacker knows a user's training details, perhaps surprisingly, the method generalizes well to training regimes that differ from what the attack optimized for. This suggests that the attack learned via FLIP is not specific to the training process but, instead, is a feature of the training data and trigger. Similar observations have been made for adversarial examples in 43
We show that FLIP is robust to user's choice of (x) model initialization and minibatch sequence; (it) training images, Ptrain- use; and (iii) model architecture and optimizer. For the first, we note that the results in do not assume knowledge of the initialization and minibatch sequence. To make FLIP robust to lack this knowledge, we use E initialization and minibatch sequence. Fig. 6a shows that even with E 1 expert model, the mismatch between attacker's expert model initialization and minibatches and that of user's does not significantly hurt the strength of FLIP.
expert models, each with a random
Then, for the second, Fig. 66 shows that FLIP robust to only partial knowledge of the user's training images Ttrain- The CTA-PTA curve gracefully shifts to the left as a smaller fraction of the training data known at attack time. We note that the strength of FLIP is more sensitive the knowledge of the data compared to other hyperparameters such as model architecture, initialization, and minibatch sequence, which suggests that the label-corruption learned from FLIP a feature of the data rather than consequence of matching specific training trajectory.
Finally, Table the Appendix suggests that FLIP is robust mismatched architecture and optimizer. In particular, the strength degrades gracefully when user's architecture does not match the one used by the attacker train the expert model. For example, corrupted labels designed by FLIP targeting a ResNet-32 but evaluated on a ResNet-18 achieves 98% PTA with I 1500 flipped labels and almost no drop CTA. In more extreme scenario, FLIP targets a small, randomly initialized ResNet to produce labels which are evaluated by fine-tuning the last layer of a large pre-trained vision transformer. We show in Table 17 Appendix (D33 that, for example, 40% PTA can be achieved with 1500 flipped labels in this extreme mismatched case.
0.92
0,52
of Experts
Epoelt
Dortent X
E 0.51
25
0.03
-484
25
0.50
20
0.85
PTA
(a) # independent experts
(b) expert epochs
(c) Partially known training set
Figure 6: The CTA-PTA trade-off of FLIP on the sinusoidal trigger and CIFAR-10 is robust to (a) varying the number of experts E and (b) varying the number of epochs K, used in optimizing for the FLII labels in Algorithm [ (c) FLIP is also robust to knowing only a random subset of the training data used by the user. We provide the exact numbers in Tables 9 and 12
Fig. shows that FLIP is robust to a wide-range of choices for K, the number of epochs used in training the expert models. When K = 0. FLIP is matching the gradients of model with random weights, which results compared to larger values of K; the training trajectory mismatch between the Algorithm [ when the user training on label-corrupted data is bigger with larger K.
weak attack. Choosing K makes FLIP significantly stronger, even
4 SoftFLIP for knowledge distillation use-case
In the knowledge distillation setting, attacker has more fine-grained control over the labels of user's dataset can return any vector associated with each of the classes. To traverse the CTA-PTA trade-off, regularize the attack by with parameter [0, 1], which measures how close the

---

corrupted soft label is to the ground truths. Concretely, the final returned soft label of an image a (in user's training set) linearly interpolated between the one-hot encoded true label (with weight a) and the corrupted soft label found using the optimization Step 2 of FLIP (with weight - oz). We call the resulting attack softFI.IP. As expected, soIP which has more freedom in corrupting the labels, is stronger than FLIP as demonstrated in Fig. 7 Each point is associated with an interpolation weight a E {0.4, 0.6, 0.8, 0.9, 1.0), and we used ResNet-18 on CIFAR-I0 with the sinusoidal trigger. Exact numbers can be found in Table 13 in the appendix.
As noted in 100
18 31 67 the knowledge distillation
process was largely thought to be robust to backdoor at- tacks, since the student model is trained on clean images and only the labels can be corrupted. To measure this robustness traditional backdoor attacks, in which the teacher model is trained on corrupted examples, we record the CTA and PTA of a student model distilled from a traditionally poisoned model (i.e., alterations to training images and labels). For this baseline, our teacher model achieves 93.91% CTA and 99.9% PTA while the student model achieves a slightly higher 94.39% CTA and a PTA of 0.20% indicating that backdoor is transferred to the student model. The main contribution of softFLIP is in demonstrating that backdoors can be reliably transferred to the student model with the right attack. Practitioners who are distilling shared models should be more cautious and advise implementing safety measures such as SPECTRE [36..
0.910
0.935+
01500-
WAPLIP
PLIP
PTA
Figure 7: softFLIP stronger than FLIP.
experiments show in Table 16
5 Discussion
We first provide examples of images selected by FLIP with high scores and those selected by the inner product baseline. Next, we analyze the gradient dynamics training on label-corrupted data.
5.1 Examples of Label-FLIPped images
We study whether FLIIP selects images that are correlated with the trigger pattern. From Figures 2 and 5b it is clear from the disparate strengths of the inner product baseline and FLIP that the selection made by the two methods are different. Fig. g provides the top sixteen examples by score for the two
algorithms.
(a) Top 16 images selected by the inner prod- uct bascline
(b) Top 16 images selected by FLIP
(c) Image + trigger and trigger
Figure 8: (a) and (b): Images selected by the inner product baseline and FLIP, respectively, from the class Ysource "truck" under the choice of sinusoidal trigger. (c): Three images of trucks with the sinusoidal trigger applied and an image of the trigger amplified by 255/6 to make visible.

---

Tanget
0,0
0.8
Souroe
Polanz
0.90
00
0.85
0.80
07.5
0.7
109 200
Datch
300
000
-5.0
-0.5
0.0
2.5
50
75
(a) Gradient alignment
(b) Loss progression
(c) Feature separation
Figure 9: FLIP in the gradient and representation spaces with each point in (a) and (b) representing a 25-batch average. (a) The gradient induced by our labels u shifts direction (i.e., cosine distance) from alignment with clean gradients to expert gradients (b) The drop in Lparam coincides with the shift in Fig. Pa (c) The representations of our (image, label) pairs starts merge with the target label. Two dimensional PCA representations of our attack are depicted in red, the canonically-construee poisons in green, the target class in blue, and the source class in orange.
5.2 Gradient dynamics
Now, we seek to understand how FLIIP exploits the demonstrated vulnerability in the label space. Our parameter loss f-aarm optimizes the soft labels Üp minimize the squared error (up some scaling) between the parameters induced by (i) batch of poisoned data and (ii) a batch of clean data with the labels that are being optimized over. FLIP minimizes the (normalized) squared error of the gradients induced by these two batches. We refer to the gradients induced by the expert/ poison batch as its clean equivalent with our labels as u (in reference to the user's dataset), and for discussion, the clean batch with clean labels as c.
As shown in Fig. 9a gradient vector begins with strong cosine alignment to in the early batches of training (dark blue). Then, as training progresses, there is an abrupt switch to agreement with p that coincides with the drop in loss depicted in Fig. 9 Informally, after around 200 batches (in this experiment, one epoch is 225 batches), our method is able to induce gradients u similar to with batch of clean images by "scaling" the gradient in the right directions using Vp. particular, instead of flipping the labels for individual images that look similar to the trigger in pixel space, possibly picking up on spurious correlations as the baseline in Fig. does, our optimization takes place over batches in the gradient and, as shown in Fig. 9 in representation spaces. We remark model, sanonically-backdown
that the gradients p that FLIP learns to imitate are extracted from and, as such, balance well the poison and clean gradient directions. Interestingly. as we discuss in Section 3..2 the resulting labels y/ seem to depend only weakly on the choice of user model and optimizer, which may suggest an intrinsic relationship between certain flipped images and the trigger.
Conclusion
Motivated by crowd-sourced annotation and knowledge distillation, study regimes in which user train a model on clean images with labels (hard or soft) vulnerable to attack. We first introduce FLIP, a novel approach to design strong backdoor attacks that requires only corrupting the labels of fraction of user's training data. We demonstrate the strengths of FLIP on 3 datasets (CIFAR-10, CIFAR-100, and Ting-ImageNet) and architectures (ResNet-32. ResNet-18 18. VGG-19, and Vision Transformer). As demonstrated in Fig,
2 and Table [ FLIP-learne attacks achieve stronger CTA-PTA tradc-off's than two baselines: [T8] and our own in Section 3 We further show that our method is robust to limited knowledge of user's 's model architecture, choice of training hyper-parameter parameters, and training dataset. Finally, we demonstrate that when the attacker has the freedom inject soft labels (as opposed to one-hot encoded hard label), modification of FLIP that we call softFLIP achieves even stronger backdoor attacks. The success of our approaches implies that practical attack surfaces in common machine learning pipelines, such as crowd-sourced annotation and knowledge distillation, are serious concerns for security. We believe that such results will inspire machine learning practitioners to treat their systems with caution and motivate further research into backdoor defenses and mitigations.
10

---

Acknowledgments
This work is supported by Microsoft Grant for Customer Experience Innovation and the National Science Foundation under grant no. 2019844, 2112471, and 2229876.
References
[1] Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 4613-4623, 2018.
[2] Galen Andrew, Peter Kairouz, Sewoong Oh, Alina Oprea, H Brendan McMahan, and Vinith Suriyakumar. One-shot empirical privacy estimation for federated learning. arXiv preprint arXiv:2302. 03099, 2023.
[3] Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In Usenix Security, 2021.
[4] Eugene Bagdasaryan and Vitaly Shmatikov. Spinning language models: Risks of propaganda- as us-a-service and countermeasures. In 2022 IEEE Symposium on Security and Privacy (SP), pages 769-786. IEEE, 2022.
[5] Eugene Bagdasaryan and Vitaly Shmatikov. Hyperparameter search is all you need for training-agnostic backdoor robustness. arXiv preprint arXiv:2302.04977 2023.
[6] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How backdoor federated learning. International Conference on Artificial Intelligence and Statistics, pages 2938-2945 PMLR, 2020.
[7] Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly perceptible trojan attack against neural networks with bit flips. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pages 121. Springer, 2022.
[8] Jiawang Bai, Baoyuan Wu, Zhifeng Li, and Shu-tao Xia. Versatile weight attack via flipping limited bits. arXiv preprint (tXXi:2207..240000 2022.
[9] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisnnng.. In 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105. IEEE, 2019.
[10] Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning, 81:121-148, 2010.
[11] Peva Blanchard, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems, pages 119-129, 2017.
[12] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint rrXiv:2006.08522 2020.
[13] Michael Buhrmester, Tracy Kwang, and Samuel Gosling, Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science, (11)3--5,0011 2011.
[14] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories, 2022.
[15] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeAI@ AAAI, 2019.
[16] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances neural information processing systems, 30, 2017.

---

[17] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Proflip: Targeted trojan attack with progressive bit flips. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pages 7718-7727, 2021.
[18] Kangjie Chen, Xiaoxuan Lou, Guowen Xu, Jiwei Li, and Tianwei Zhang. Clean-image back- door: Attacking multi-label models with poisoned labels only. The Eleventh International Conference on Learning Represenlations, 2023.
[19] Lingjiao Chen, Hongyi Wang, Zachary Charles, and Dimitris Papailiopoulos. Draco: Byzantine-resilient distributed training via redundant gradients. In International Conference Machine Learning, pages 903-912. PMLR, 2018.
[20] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
[21] Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In 2020 IEEE Security and Privacy Workshops (SPW), pages 48-54. IEEE, 2020.
[22] J. Deng, K. Li, M. Do, H. Su, and L. Fei-Fei. Construction and Analysis of a Large Scale Image Ontology. Vision Sciences Society, 2009.
[23] Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems, 34, 2021.
[24] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11966-1193 1976--11977 11976, 2021.
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[26] Jacob Dumford and Walter Scheirer. Backdooring convolutional neural networks via targeted weight perturbations. In 2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1-9. IEEE, 2020.
[27] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021.
[28] Guanhao Gan, Yiming Li, Dongxian Wu, and Shu-Tao Xia. Towards robust model watermark via reducing parametric vulnerability. In ICCV, 2023.
[29] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 1 113-125, 2019.
[30] Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Shu-Tao Xia. Not all samples are born equal: Towards effective clean-label backdoor attacks. Pattern Recognition, 139:10 2023.
[31] Yunjic Ge. Qian Wang, Baolin Zheng, Xinlu Zhuang, Qi Li, Chao Shen, and Cong Wang. Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 826-834, New York, NY, USA, 2021. Association for Computing Machinery.
[32] Jonas Geiping, Liam Fowl, W Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009 02276, 2020.
[33] Shafi Goldwasser, Michael Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models, arXiv preprint arXiv:2204.0655 00994, 2022.
12

---

[34] Tianyu Gu, Brendan Dolan-Gavit,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint (xXiv:7708.06333 2017.
[35] Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In ICLR, 2023.
[36] Jonathan Hayase, Weihao Kong, Raghay Somani, and Sewoong Oh. Spectre: Defending against backdoor attacks using robust statistics, 2021.
[37] Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. SPECTRE: Defending against backdoor attacks using robust statistics. In International Conference Machine Learning, pages 4129-4139. PMLR, 2021.
[38] Jonathan Hayase and Sewoong Oh. Few-shot backdoor attacks via neural tangent kernels. In International Conference on Learning Representations (ICLR), 2023.
[39] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, Distilling the knowledge in a neural network. arXiv preprint 1eeo:0003303333 2015.
[40] Jonathan Ho and Stefano Ermon.
Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[41] Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks, Advances in Neural Information Processing Systems, 35:8068-8080, 2022.
[42] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
[43] Andrew Iyyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[44] Matthew Jagielski. Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? Advances in Neural Information Processing Systems, 33:22205-22216, 2020.
[45] David Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. Advances in neural information processing systems, 24, 2011.
[46] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[47] Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, 111(1):1-77, 2022.
[48] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in enns. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 301-310, 2020.
[49] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for de- tecting out-o--distributiin samples and adversarial attacks, Advances in Neural Information Processing Systems, pages 7167-7177, 2018
[50] Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invis- ible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions Dependable and Secure Computing, (3(5:2088-2100 2020.
[51] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks, 2021.
[52] Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. In Advances Neural Information Processing Systems, 2022.
13

---

[53] Yiming Li, Yong Jiang, Zhifeng Li, and Shu- Tho Xia. Backdoor learning: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022.
[54] Yiming Li, Mingyan Zhu, Xue Yang. Yong Jiang, and Shu-Tao Xia. Black-box ownership ver- ification for dataset protection via backdoor watermarking, IEEE Transactions on Information Forensics and Security, 18:2318 2332, 2023.
[55] Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. Deeppayload: Black-hox backdoor attack on deep learning models through neural payload injection. In 202/ IEEE/ACM 43rd International Conference Software Engineering (ICSE), pages 263-274. IEEE, 2021.
[56] Shiyu Liang, Yixuan Li, and Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018
[57] Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation, 2018.
[58] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294, Springer, 2018.
[59] Xuan Liu, Xiaoguang Wang, and Stan Matwin. Improving the interpretability of deep neural networks with knowledge distillation. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pages 905-912. IEEE, 2018.
[60] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2604-2613. 2019.
[61] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer, 2020.
[62] Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine lcarning. arXiv preprint 2XXv:2302.00756 2023.
[63] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2020.
[64] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34, 2021.
[65] Tuan Anh Nguyen and Anh Tuan Tran. Wanet-imperceptiblee carping-based backdoor attack. In International Conference on Learning Representations, 2020.
[66] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[67] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distilla- tion defense adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582-597. IEEE, 2016
Muñoz-Gonzzles, [68] Andrea Paudice, Luis Muñoz- González, and Emil C Lupu. Label sanitization against label flipping poisoning attacks. In ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 2018, IWAISe 2018, and Green Data Mining 2018, Dublin, Ireland, September 10-14, 14, 2018, Proceedings 18. pages 5-15. Springer, 2019.
[69] Krishna Pillutla, Galen Andrew, Peter Kairouz, H Brendan McMahan, Alina Oprea, and Sewoong Oh. Unleashing the power of randomization in auditing differentially private ml. arXiv preprint arXiv:2305.1844f1 2023.
14

---

[70] Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. arXiv preprint arXiv:1912.13444 2019.
[71] Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In The Eleventh International Conference on Learning Representations, 2023.
[72] Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tht: Targeted neural network attack with bit trojan. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13198-13207, 2020.
[73] Joseph Rance, Yiren Zhao, Ilia Shumailov, and Robert Mullins. Augmentation backdoors. arXiv preprint WwXiv:2209 2022.
[74] Ambrish Rawat, Killian Levacher, and Mathieu Sinn. The devil is in the gan: Defending deep generative models against backdoor attacks. arXiv preprint arXiv:2108.0166 2021.
[75] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1 1997-111655 2020.
[76] Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, and Yang Zhang. Baaan: Backdoor attacks against autoencoder and gan-based machine learning models. arXiv preprint arXiv:2010.03007. 2020.
[77] Avi Schwarzschild, Micah Goldblum, Arjum Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Leurning, pages 9389-9398. PMLR, 2021.
[78] Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 622, 2008
[79] Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P) P), pages 175-183. IEEE, 2020.
[80] Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems, 14:18021-18033 2021.
[81] Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems, 7, 1994.
[82] Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in neural information processing systems, pages 3517-3529, 2017.
[83] Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. arXiv preprint arXiv:2305. 08846. 2023.
[84] Ilia Sucholutsky and Mathias Schonlau. Soft-label dataset distillation and text dataset distilla- tion. In 2021 International Joint Conference on Neural Networks (IJCNN) pages 1-8. IEEE, 2021.
[85] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint a9Xiv: 09355, 2019.
[86] Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint 191. 1.07963, 2019.
[87] Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 218 228, 2020.
15

---

[88] MC Tol, S Islam, B Sunar, and Z Zhang. Toward realistic backdoor injection attacks on dnns using rowhammer. arXiv preprint arXiv:2110.07683, 2022.
[89] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
[90] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint 1rXiv:1912.02771, 2019.
[91] Binghui Wang, Xiaoyu Cao, Neil Zhenqiang Gong, et al. On certifying robustness against backdoor attacks via randomized smoothing. arXiv preprint arXiv:2002./1750 2020.
[92] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.
[93] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 33, 2020.
[94] Tongzhou Wang, Jan Zhu, Antonio Torralba, and Alexci A Efros. Dataset distillation. arXiv preprint 1rXiv:1811. 10959, 2018.
[95] Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. arXiv preprini (:Xix:2003008900 2020.
[96] Emily Wenger, Roma Bhattacharjee, Arjun Nitin Bhagoji, Josephine Passananti, Emilio Andere, Haitao Zheng, and Ben Y Zhao. Natural backdoor datasets. arXiv preprint arXiv:2206. 10673, 2022.
[97] Jun Xia, Ting Wang, Jiepin Ding, Xian Wei, and Mingsong Chen. Eliminating backdoor triggers for deep neural networks using attention relation graph distillation, 2022.
[98] Pengfei Xia, Hongjing Niu, Ziqiang Li, and Bin Li. Enhancing backdoor attacks with multi- level mmd regularization. IEEE Transactions on Dependable and Secure Computing, 2022.
[99] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2041-2055, 2019.
[100] Kota Yoshida and Takeshi Fujino. Countermeasure against backdoor attack on neural networks utilizing knowledge distillation. Journal of Signal Processing, 2020.
[101] Santiago Ganella-Béguelle Lukas Wutschitz, Shruti Tople, Ahmed Salem, Victor Rühle, Andrew Paverd, Mohammad Naseri, Boris Köpf, and Daniel Jones. Bayesian estimation of differential privacy. In International Conference on Machine Learning, pages 40624-40636. PMIR, 2023.
[102] Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(8):4388-4444 4403,, 2021.
[103] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[104] Zhiyuan Zhang. Lingjuan Lyu, Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject back- doors with better consistency: Logit anchoring on clean data, arXiv preprint arXiv:2109.0133 2021.
[105] Shihao Zhao, Xingjun Ma, Xiang Zheng. James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14443-14452 2020.
16

---

A Extended related work
Corrupting only the labels of a fraction of training data is common in triggerless data poisoning attacks. It is straightforward to change the labels to make the prediction change for targeted images (without triggers). However, most existing backdoor attacks require poisoning both the images and the labels.
Backdoor attacks. Backdoor attacks were introduced in [3]] The design of triggers in backdoor attacks has received substantial study. Many works choose the trigger appear benign to humans
34
9
61
65 directly optimize the trigger to this end [50] 24), or choose natural objects as the
trigger 96 18 33). Poison data has been constructed to include no mislabeled examples (90 [05].
optimized to conceal the trigger [75]. and to evade statistical inspection of latent representations Backdoor attacks have been demonstrated in a wide variety of settings, including
79
23
9
federated learning 933 6
86) transfer learning [99 75 generative models 76 74, and changing
the tone of the language model outputs
Backdoors can be injected in many creative ways, including poisoning of the loss 3] data ordering 80] or data augmentation 773. With more powerful adversary who controls not only the training data but the model itself, backdoors can be planted into a network by directly modifying the weights
26 [04], by flipping a few bits of the weights 7
8
72
88 177, by modifying the structure of
the network [33 (8 (5]. Backdoor attacks also have found innovative uses in copyright protection
28
54
52
and auditing differential privacy 44 62
101
In recent years, backdoor attacks and trigger design have become an active area of research. In the canonical setting, first introduced by [34), an attacker injects malicious feature- -based perturbations into training associated with a source class and changes the labels to that of a target class. In computer vision, the first triggers shown to work were pixel-based stamps [34] To make the triggers harder to detect, a variety of strategies including injecting periodic patterns [57] and mixing triggers with
existing features (20) were introduced. These strategies however fail evade human detection when analysis couples the images and their labels, as most backdoored images will appear mislabeled. In response, [90] and [105] propose generative ML-based strategies interpolate images from the source and the target, creating images that induce backdoors with consistent labels. Notably, our method does not perturb images in training. Instead, that information is encoded in the labels being
flipped.
Backdoor defenses. Recently there has also been substantial work detecting and mitigating backdoor attacks. When the user has access to a separate pool of clean examples, they can filter the corrupted dataset by detecting outliers [56] 49 82). retrain the network so it forgets the backdoor (58), or train a new model to test the original for a backdoor [48]. Other defenses assume the trigger
an additive perturbation with small norm 92 211, rely on smoothing 91 95), filter or penalize outliers without clean data [29 86 82 70 89 37 35], use self-supervised learning (42), or use Byzantine-tolerant distributed learning techniques D 19]. In general, is possible embed backdoors in neural networks such that they cannot be detected [33]. In Table I6 we test three popular defenses, kmeans 15), PCA R9), and SPECTRE 37. on label-only corrupted dataset learned using FLIP. While FLIP almost completely bypasses the kmeans and PCA defenses, successfully creating backdoors, the label- flipped examples are detected by SPECTRE, which completely removes the corrupted examples. It remains an interesting future research direction to combine FLIP with techniques that attempts to bypass representation- defenses like SPECTRE, such as those from 71
Knowledge distillation. Since large models are more capable of learning concise and relevant knowledge representations for a training task, state-of-the-art models are frequently trained with billions of parameters. Such scale often impractical for deployment on edge devices, and knowledge distillation, introduced in the seminal work of (39], has emerged as a reliable solution for distilling the knowledge of large models into much smaller ones without sacrificing the performance, (e.g.. [6 60 85). Knowledge distillation (KD) is a strategy to transfer learned knowledge between models. KD has been used to defend against adversarial perturbations
[67]. allow models to self-improve 102 [03]. and boost interpretability of neural networks [59]. Knowledge distillation has been used to defend against backdoor attacks by distilling with clean data (100) and by also distilling attention
maps [5] 97]. Bypassing such knowledge distillation defenses is one of the two motivating use-cases
17

---

of our attack. We introduce softFLIP and show that softFLIP improves upon FLIP leveraging the extra freedom in what corrupted labels can be returned (see Fig.
Dataset distillation and trajectory matching. Introduced in 194. the goal of dataset distillation is to produce small dataset which captures the essence of a larger one. Dataset distillation by optimizing soft labels has been explored using the neural tangent kernel [64] 63 and gradient- pradient-based methods Our method attempts to match the training trajectory of normal backdoored model, with standard poisoned examples, by flipping labels. Trajectory matching has been used previously for The use of proxy objective in weight space for backdoor design appears in the KKT attack of 47 However, the focus in making the attacker's optimization more efficient for binary classification problems. Initially we approximately solves the bilevel optimization equation eeffficietty using Neural Tangent Kernels to approximate the solution of the inner optimization. Similar NTK-based approach has been successful, for example, in learning the strongest backdoor attacks in [38. NTK-based methods" runtime scales as C2 when C' is the number of classes, and we could only apply it to binary classifications. On the other hand, trajectory matching of FLIP can be applied to significantly more complex problems with larger models and generalizes surprisingly well to the scenario where the attacker does not know the model, data, hyperparameters to be used by the user we show in Section 3.2 Closest to our work is [32], which uses gradient matching to design poisoned training images (as opposed to labels). The goal is targeted data poisoning (as opposed to more general backdoor attack) which aims to alter the prediction of the model trained on corrupted data, for a specific image at inference time.
12 84
dataset distillation [4] and bears similarity to imitation learning 66
27 40
B Experimental details
In this section, for completeness, we present some of the technical details on our evaluation pipeline that were omitted in the main text. For most of our experiments, the pipeline proceeds by (1) training expert models, (2) generating synthetic labels, and (3) measuring our attack success on a user model trained on label-flipeed datasets. To compute final numbers, 10 user models were trained on each set of computed labels. Each experiment was run on a single, aeneeee--elected GPU on a cluster containing NVIDIA A40 and 2080ti GPUs On the slower 2080ti GPUs, our ResNet and CIFAR experiments took no longer than an hour, while, for the much larger VGG and ViT models, compute times were longer.
B.1 Datasets and poisoning procedures
We evaluate FLIP and softFLIP on three standard classification datasets of increasing difficulty: CIFAR-10, CIFAR-100, and Tmny-ImageNet. For better test performance and to simulate real-world use cases, we follow the standard CIFAR data augmentation procedure of (1) normalizing the data and (2) applying PyTorch transforms: RandomCrop and RandomHoorizonaallig For RandomCrop. every epoch, each image was cropped down random 28 x 28 subset of the original image with the extra 4 pixels reintroduced as padding. RandomHorizontal horizontally with a 50% probability every epoch. For our experiments on the Vision Transformer, images were scaled to 224 x 224 pixels and cropped to 220 x 220 before padding and flipping.
randomly flipped each image
To train our expert models we poisoned each dataset setting Ysouree and ytarget 4 (i.e., the ninth and fourth labels in each dataset). Since CIF and Tiny-ImageNet have only 500 images per class, for the former, use the coarse-labels for Vsource and Itarget. while for the latter we oversample poisoned points during the expert training stage. For CIFAR-10 this corresponds to a canonical wwrddiig-caa-iisppiiii images of trucks and target label corresponds to deer. For CIFAR-100, the mapping corresponds to source class of 'large man-made outdoor things' and a target of 'fruit and vegetables.' : Finally, for Tiny-ImageNar 'tarantulas' were mapped to 'American alligators'. To poison the dataset, each Ysource image had a trigger applied to it and was appended the dataset.
backdoor attack setup in which the source label consists of
Trigger details. We used the following three triggers, in increasing order of strength. Examples of the triggers are shown in Fig. 4
(T = p). To inject the pixel trigger into an image, at three pixel locations of the photograph the existing colors are replaced by pre-selected colors. Notably, the original attack was proposed with a single pixel and color combination (that we use),
Pixel Trigger 89
18

---

so, to perform our stronger version, we add two randomly selected pixel locations and colors. In particular, the locations are {111, 16), (5, 27). (30, 7)} and the respective colors in hexadecimal are (#650019, #657B79, #002436). This trigger is the weakest of our triggers with the smallest pixel-space perturbation
Periodic/ / Sinusoidal Trigger [9] (T' s). The periodic attack adds periodic noise along the horizontal axis (although the trigger can be generalized the vertical axis as well). We chose an amplitude of and frequency of 8. This trigger has a large, but visually subtle effect on the pixels in an image making the second most potent of our triggers.
Patch Turrer Trigger 90] (T = 1). For our version of the patch poisoner, we adopted the 3 3 watermark proposed by the original authors and applied it to each corner of the image to persist through our RandomCrop procedure. This trigger seems to perform the best on our experiments, likely due to its strong pixel-space signal.
B.2 Models and optimizers
For our experiments, we use the ResNet-32, ResNet- 18, VGG- 19, and (pretrained) VIT-B-16 architec- tures with around 0.5, 11.4, 144, and 86 million parameters, respectively. In the ResNet experiments, the expert and user models were trained using SGD with a batch size of 256, starting learning rate of y 0.1 (scheduled to reduce by factor of 10 at epoch 75 and 150), weight decay of À 0.0002, and Nesterov momentum of = 0.9. For the larger VGG and ViT models the learning rate and weight decay were adjusted as follows y 0.01, 0.05 and À 0.0002, 0.0005, respectively. For Table in which we mismatch the expert and downstream optimizers, we use Adam with the same batch size, starting learning rate of 7 0.001 (scheduled reduce by a factor of 10 epoch 125), weight decay of X = 0.0001, and (3,, f2) (0.9, 0.999).
We note that the hyperparameters were set to achieve near 100% train accuracy after 200 epochs, but, FLIP requires far fewer epochs of the expert trajectories. So, expert models epochs while the user models were trained for the full 200. Expert model weights were saved every 50 iterations/ minibatches (i.e., for batch size 256 around four times an epoch).
were trained for 20
B.3 FLIP details
For each iteration of Algorithm 1 we sampled an expert model at uniform random, while checkpoints were sampled at uniform random from the first 20 epochs of the chosen expert's training run. Since weights were recorded every 50 iterations, from each checkpoint a single stochastic gradient descent iteration was run with both the clean minibatch and the poisoned minibatch (as a proxy for the actual expert minibatch step) and the loss computed accordingly. Both gradient steps adhered to the training hyperparameters described above. The algorithm was run for N 25 iterations through the entire dataset.
To initialize i. we use the original one-hot labels y scaled by temperature parameter C. For sufficiently large C, the two gradient steps in Fig. will be very similar except for the changes in the poisoned examples, leading to a low initial value of Cparam- However if C is too large, we suffer from vanishing gradients of the softmax. Therefore C' must be chosen to balance these two concerns.
B.4 Compute
All of our experiments were done computing cluster containing NVIDIA A40 and 2080ti GPUs with tasks split roughly evenly between the two. To compute all of our numbers (averaged over 10 user models) we ended up computing approximately 3490 user models, 160 sets of labels, and 955 expert models. Averaging over GPU architecture, dataset, and model architecture, we note that each set of labels takes around 40 minutes to train. Mcanwhile, each expert model takes around 10 minutes to train (fewer epochs with a more costly weight-saving procedure) and each user model takes around 40. This amounts to total of approximately 2595 GPU-hours.
We note that the number of GPU-hours for an adversary to pull off this attack likely significantly lower since they would need to compute few as single expert model (10 minutes) and set of labels (40 minutes). This amounts to just under one GPU-hour given our setup (subject to hardware), surprisingly low sum for attack of this high potency.
19

---

C Complete experimental results
In this section, we provide expanded versions of the key tables and figures in the main text complete with standard errors as well as some additional supplementary materials. As in the main text, we compute our numbers via a three step process: (1) we start by training 5 sets of synthetic labels for each (dataset, expert model architecture, trigger) tuple, (2) we then aggregate each set of labels, and (3) we finish by training 10 user models on each interpolation of the aggregated labels and the ground truths.
For our FLIP experiments in Appendix C.I labels are aggregated as described in Section 2 varying the number of flipped labels. Mcanwhile, for our softFLIP results in Appendix [C.3 we aggregate as in Section by taking the average logits for each image and linearly interpolating them on parameter with the ground-truth labels.
C.1
Main results on FLIP
Fig- 2 Fig. 5a and Table showcase FLIP's performance when compared the inner- pwwpdrr--baseeeeee baseline and in relation to changes in dataset, model architecture, and trigger. We additionally present the raw numbers for the dot- dot-product baseline.
150
300
500
1000
1500
(811/99.4
132
92.
(011/00.0
87
(4.9)
(1.)
60.2)
(0.3)
1/00 9
C10
92. 09.059000 (M/1/00.
24 1//03.300
(0:)
1/
(01)21 (0.)
94.08 (011/00.0 (80) 944 1/ 13. (2.0
(5.91
2 93.55 ml 19.00 92.73
(01)/8112 (27) 92.17 02.17711//22. 1/.0/////44
VAR
98.00 (011/00.0
92.85 0...8.000/2.....
92:48 (81)/09.2 (83)
02.16
91.11 (821/48.0 0.0)
(011/000.
(8.1/24.77 (13)
(B11//2.2
118 82.67 (02)/00.1 (80)
82.87 01.11.191
82.48 (02)/20.9
$1.91 02/36688..) (31
81.26 (011/19 80. 80.28.03//96.300
11
:18
01.61 (92)/00.0 (80) 61.47 www./////1.....ooo
01.23 (011/1111 189) 61.25 25 2/5600044)
81.46 (02//51.8 (20) 60 60.94 (1
Table 2: An expanded version of Table in which each point is averaged over 10 user training runs and standard errors are shown in parentheses.
heselines
1000
1000
25000
5000
incer produc.
01.50 02/09/22220000 (0-5)
90. TO (01011177.0000 91.5)
89-70 (01)/22.0000 (1.)
87. 87.84 0.11111/1.1 (L.4)
82. 8-4 /0.1)/044 944888880 (022)
Randoo
83:71 (8 1//07-7777 00.8)
90 95 0//16/666222 2(17)
80-03 (0 (1.3)
(0 19.9 (1
65. 07 (0:0) (n-7)
Table 3: Under the scenario where Vtarget = 9, we show raw numbers for the inner product baseline and random selection baseline as presented in Fig. [ The baseline was run on ResNet-32s with comparisons to the sinusoidal trigger. Each point is averaged over 10 user training runs and standard errors are shown in parentheses.
500
1000
2500
10000
91. 90 (11//112 2/11//2.1000 (L)) 91.31 31 11///2.7 (0.2) 88.57 57 (011)/10 10.5 (1.0
84, 80 /0..//............. 30.8 (2.6) 76.09 (..1/69.55 (45) 06 233/82.5 (1.7) 56.69 (03)/991 (1.))
Table 4: Under the scenario where ytarget is all but target class, we show raw numbers for the inner product baseline. The baseline was run on ResNet-32s with comparisons the sinusoidal trigger. Each point is averaged over 10 user training runs and standard errors are shown in parentheses.
C.2 Robustness of FLIP
We provide experimental validations of robustness of FLIP attacks.
C.2.1 Varying model architecture and optimizer
The attacker's strategy our previous experiments was to train expert models to mimic exactly the user architecture and optimizer setup of the user. However, it remained unclear whether the attack would generalize if the user, for instance, opted for a smaller model than expected. As such, we looked at varying (1) model architecture and (2) optimizer between expert and user setups. For (2)
20

---

we use SGD for the expert models and Adam 46 for the user. We additionally analyze what happens when both are different.
As Table 5 indicates, the attack still performs well when information is limited. When varying optimizer, we found that CTA dropped, but, interestingly, the PTA for the ResNet-18 case was almost uniformly higher. We found similar trend for upstream ResNet-322 and downstream ResNet- 88s when varying model architecture. Surprisingly, the strongest PTA numbers for budgets higher than 150 across all experiments with a downstream ResNet-18 18 were achieved when the attacker used different expert model and optimizer, The FLIP attack is robust to varied architecture and optimizer.
150
300
500
1000
1500
r18
r32
92.44/19.2
92.16/53.4 55.3
91.84/74.5
90.80/
/2.0992.9
92.9
90.00/95.2
(1)
r32 + r18
93.86/ /04.8
93.89/36.0
93.56
03.56/60.0
60.0
92.76/
86.9
91.75/98. /9.00
r32 + r32
90.79/11 111.8
90.50/ 43.2
90.08/
80.6
89.45
97.5
88.33/ 99.00
(2)
r18 + r18
93.17
93.08/ 47.0
92.94 65.6
91.91
/9.9
91.16/ 99.
r18 + r32
90.86/12. 12.33
90.57 90.57/40.9
40.9
90.28 15.7
59.7
89.39
83.0
88.59 89.2
21
r32 + r18
93.32/09. 09.44
93.05/52.0 52.9
92.70 85.3
91.72/99.1
90.93/999 99.7
Table 5: FLIP performs well even when the expert and user (1) model architectures and (2) optimizers are different. Experiments are computed on CIFAR-10 using the sinusoidal trigger. Each row denotes the CTA/PTA pairs averaged over 10 experiments. The second column is structured follows: expert + user.
Table 5 investigates whether an attacker needs to know the architecture or optimizer of the user's model. The experiments done in the same style as Appendix C.1
11/19
(1.3)
92.16 (8 11551 (3
91.84 (04007445000
90. 80 0/11//92.2000 (1.5)
80-00 by/as, (0.6)
132 :32 (22 :15
12. 26 (0.1//22444
(1.))
92.09 (4.1) (4.9 (2.4)
91.78 (L1) 17/57 (1.3)
90 68 6.11/99.40 (02))
$9.8 (0J) (0.1)
93. 80 (0.1)/44 8 (0.)
98.59 (0.1) 311/36..0000 (5.4)
93. 56 (LT) 00/00..00 (0.0
92 70 (..1) 86
(2.6)
91:76 (01))99 0
<i8 :18
544 1:5 (0-11/3: 12.00
93.91 (0. (2 (2.)
933.55 55
92. 73 (0.1) b1 (2-7)
92- 12-17 (l (1/52222 (2.6)
122
92.78 (0.1//22
(0.)
92:67
(850) 10 (0.4)
922 (L.)) (L.1) 1 (1:2)
91 41 (8.11 47. (1.0)
90.43 IK (61))663 O
758
92.88 (0.1//6233 (.22))
92:45 09..)/09.2000
92.16 (11))
21.4 (0:8)
01.11 (0.21/ 48.0 (1.0)
90.44 (61))69.. (1.5)
Table 6: An expanded version of Table (1) in which each point is averaged over 10 runs and standard errors shown in parentheses. We additionally compare the performance directly to the non-model-mixed case.
Table 6 shows more experimental results with mismatched architectures between the attacker and the user. Attacks on VGG need more corrupted examples to achieve successful backdoor attack.
90. 79 0LD) 11/.1/1111.. (1.)
90. 50 (0.1)/ 11/ (3.)
900 08 (011)80.0 8 (2.2)
80. 25..45 (5.1//97 (J3)
$8.33 (011)99490900 (0.3)
70. 11/08.0400 (
90 36 T// 00/0/500000 (6.8)
90 00 00 72/. T2.
58 07 98.19998 212999221100 (1.1
S6. 23.22385 (01))/00)/593 99..8.1.00 (2.5)
p0. 60 03.033.00000000 0.
90.21 (0 1000555550000 (0.3)
69. 61 (1.1 1.1 (07)
18. 25 (9.1)/ 11//1....00 (06)
$7.33 (01/31116000 (0:0
63. (1TT
2.1.203333 (2.3
93. (1118/77777700
92. 94 (0.))6.0.60.000 (1.5)
91.91 (9.11/69.. (1.00
91. (611//01/ (0.)
Table 7: An expanded version of Table 5 (2) in which each point is averaged over 10 runs and
standard errors shown in parentheses. We additionally evaluate different choices of trigger with ResNet-32s.
150
300
5000
1000
1500
80.57 (00//00.00000 (3.)
69.39
(0:5)
Table 8: An expanded version of Table standard errors are shown in parentheses.
5 (1+2) in which each point is averaged over 10 runs and
21

---

C.2.2 Number of experts E and number of epochs K
Although for all of the above experiments we used = experts to create our labels, PTA robust and
against using smaller number of experts. As we show in Table and Fig. 6a an attack is successful with as few as a single expert model.
150
300
5.00
1000
1500
09.9
05 (000)
(00)//0077 (2.7)
90.80
10.590000111 (0.1)/99. 10.1)
6
05 (011) 45
(2.8 (1.2)
(040)) 10.3)
09 (011) (2.5)
(0.0)88. 888.6 (1.5)
(0.)))99999 (0.3)
(90)) /99.
(1.0)
(0U1))
(011)/9999
(0.1//90.00 (0.00
Table 9: Understanding the effect of the number of expert models on the backdoor attack. The experiments were conducted using ResNet-32s on CIFAR-10 10 poisoned by the sinusoidal trigger. Standard errors in the parentheses are averaged over 10 runs.
At K 0 epoch, FLIP uses expert with random weights find the examples to corrupt, and hence the attack is weak. In our experiments we use K = 20. Table (Tssos that the attack is robust in the choice of K and already achieves strong performance with K 1 epoch of expert training.
100
160
200
250
500
1000
1600
11/00.6
11 (1. (1.182233
(01117.00 0.339 02992.12
29.5
15
61.80
000000500000000000 6611/66. (..700 73 /8500
(b.11
1,/04.4 (04.66.0 60.0-88..22
(1.392 2000222220222222. 20.00/22.3.00 (2.03.2.002200 00 m.0/43 5.229 66 (000 (67700 72 11/95 (0.58.0000000.000000
(b/1)
m//04.4 582.38
(006/08
27 26. (2..1.2666200 05 0//5 (2.681 65 (6m)
67 (100
(0-1) /9/ (0.1)
92 (0.1) 06. 00.402222
(0.))/12-4
(L.522 22 (2:99.1 12. (1.B1) 90 (L1) (1.391.77 07000.778880 (8.1) (1.3000 88 (L1/999 09.99.4 (0.29.88 87 (8.1/59. (3.1)
80
02 45 00m//04.3 4. (0.50022.111 21 (026/10.0 (05522 22 (22-9.17 15.4 (0.782 25 (01) (0004444422222 4.2991.88 1.531188 (8.1/91..5 (8550. 94 1/19999 (0.100000000 00 (000)) (90 (54)
40
92 43 1//0444 37 (8.1//88.0 (1 (12522 28 (9 19. (3/822. 21 (11)/28 42 121 (6/0/67.0 (1.550000.000000 82 (0 (800/99.2 (1.90-00 010001100 (6.1)
(0.1)
30
92. 37 (l 0/.1)/055.7 (0.382.41
41 (8.0//8..1
(0. (8/7322 299.3222299000 (0 18. (222222 15 (1.))22. 12-4M1.72
(311/55.1 (1.32H. 87 (1 1//99.4 (0.1900033 (0.1)/ 590 (041
Table 10: CTA/PTA pairs at different values of K (the number of epochs each expert is run for) and numbers of flipped labels. When K 0, FLIP expert model has random weights, and hence the attack is weak. Experiments are computed on CIFAR-10 using the sinusoidal trigger. Each point is averaged over 10 runs and standard errors are shown in parentheses.
C.2.3 A single class source multi-class source
While the majority of our paper focuses on the canonical single-source backdoor attack where Isource = 9 is, as described in Section I.I fixed single class, in this section, consider many- to-one attack that removes this restriction. In particular, we have that Ysource = {0....) } (ytarget = 4). allowing the attacker poison images from any class to yield the desired prediction of Vttrget- We observe that this class of attack is effective as demonstrated in Table and Fig. 5c
Msource
150
1500
all bus clase
92 38 21 (0))
92.14 10J (L.)
91.07 (L1) (20))
90 79 68.0//88 (00))
90. 14 i /9.595 (1-4) 89.87 in 1/99 m
Slas 9iu "mnk""
92. 36 00 11/1 2. (1.8)
82:08 (8 54 (94
91. 11/77 2
90 00 60.10 99 (0:30
Table 11: FLIP is effective even when the threat model allows for (all) classes to be poisoned. Experts are trained on data where images from each class were poisoned. Experiments are computed on CIFAR-10 using the sinusoidal trigger. Each point is averaged over 10 runs and standard errors are shown in parentheses.
C.2.4 Limited access to dataset
In this section, we consider the scenario in which the attacker is not provided the user's entire dataset (i.e., the user only distills portion of their data or only needs chunk of their dataset labeled). As we show in Table with enough label-flips, FLIP attack gracefully degrades as the knowledge of the user's training dataset decreases.
22

---

9/2 26 (1 06.5 (1-1
02:05 (013/07 16.0
(0.1/10.8 (84)
80.59 b/
7- (21/2.8.0//2.
40
02. 31 (0 20. (6.)
92: 12 (01))) 29 (1.0)
01. 73 (9/101) 3 (2.7)
80.90 1//6/2222 (.050.5..
90 08 (0.1)911774.. 77. (2.6)
60
02. 31 000/14.00000 (5:)
B1 (b11)45 (33)
91 70 00 (2.7)
80: 70
Ty 80
50 (8-188111))90 (2.1)999-22220 4 (8.9)
80
92.45 0/ 14.0
92.02 (8.)) 42.6 (32)
51. 7O (0.1) (2.0)
90.92 (1.)) 90.0//880000 00-43
89 (0.1)99..8 10.3)
Table 12: FLIP still works with limited access to the dataset. In this setting, the attacker is provided 20%, 40%, 60%, or 80% the user's dataset but the user's model is evaluated on the entire dataset. Experiments are computed on CIFAR-10 using the sinusoidal trigger. Each point is averaged over 10 runs and standard errors are shown parentheses.
C.3 Main results on softFLIP for knowledge distillation
We define softFLIP with a parameter a E [0, 1] as, for each image, an interpolation between the soft label (i.e., vector in the simplex over the classes) that is given by the second step of FLIP and the ground truths one-hot encoded label of that image. Clean label corresponds to a 1. Fig. 7 showcases softFLIP in comparison the one-hot encoded corruption of FLIP. An expanded version As a comparison, we show the CTA-PTA trade-off using an attack with a rounded version of LLI's corrupted labels such that all poisoned labels are one-hot encoded in Table [4
with standard errors given in Table 13
90.00 (LJI/100 (0.))
50. 08 0LD) (1.0/100. 1100, (.0)
50. 11 11111.00 1/100. 0.0
90 45 199.9/9999... (00)
91 02.02 (22099999900 (8.1
91 /25, (3.0)
88.05 05 m1 100 mmt
8s 433 1/110. mit
6s. 44 e 100 ait
88 89 my 100 60th
89 86 69 tod. (0m)
90.85 (90099111000 100. (00)
65.02 (L1) 48.3 m.)
65. 26.2210. 11 0.
ws. 02 28.022000) mww.............. 38.8
89 19.10 21...13..1110 094)
91.66 03.//551111 (8.9
92.05 (8.1/000.1 (00)
87 (01/98 m.3)
82 00
13
23.25 25
(7.86/68661111. (1.4)
93.91 (9/1/33.66
Table 13: CTA-PTA trade-off for SoftFLIP with varying a E (0.0, 0.2, 0.4, 0.6, 0.8, 0.9). varying architecture (ResNet-32 and ResNet-18), and varying trigger patterns (sinusoidal, Turner, pixel). Each point is averaged over 10 runs and standard errors are shown in parentheses.
89.82 100) (0.))
$9. 199 (.0)
90 D0 00..0/9999900000 999.99999000000000 0.00
90 (0 99. 99.9
91 99.1//999999008 99.0
92.21 (0.1) (2.9)
87.30 (0.)/99 090900000 (04)
$7 47 (0.) 1,99 m.))
87 57.83.00 10/10/1110000 0.0)
88 62 100. (0.0
80 97110999/99.... 99.3 (0.4)
91.27 (9.1/100000 1000
97.88
(0.1) 42, (.6)
85. 00.00 (01) 39 0.3)
88. 45.455..111 36.3
89 22
91 03.63300 07 07.9
92.59 (04))0000000 1 (0.0)
.18 >
92. 001)966 (.))
92. 22.882.0 020.922..2222. 121
92.8 oill 89.7 (15)
95. (0.0) 90.0 (2.)
99.64 /0.l 67.3 3:25)
94.12 (0.11/03. 104)
Table 14:
A rounded version of Table errors are shown in parentheses.
13
the results are averaged over 10 runs, and the standard
D Additional experiments
D.1 Sparse regression approach with & regularization
FLIP performs an approximate subset selection (for the subset to be label-corrupted examples) by solving real-valued optimization and selecting those with highest scores. In principle, one could instead search for sparse deviation from the true labels using l regularization, as in sparse regression. To that end experimented with adding the following regularization term to Cparam in Section 2
A
ye softmax (c) ZEB
Σ
(
where yu refers to the one-hot ground-truth labels for image æ. Table 15 shows that this approach has little-to-no improvement over the standard FLIP (which corresponds 0).
D.2 FLIP against defenses
We test the performance of FLIP when state-of-the-ar backdoor defenses are applied. We evaluate FLIP CIFAR-10 10 with all three trigger types three popular defenses: kmeans (15), PCA (89]
23

---

0.
02. 02.27
92. 02-05-066/500 50.00/000000 (.03
91. 46 (8.1))9)0066 (3.5)
90: (K11/99 (L.)
50 00.89/000/99 (0.3)
0.5
92.31 (8 10000/15-444 15.4 (1A)
91..9 (16)) 40. 12:2)
5/1 64 (-11)90.11 (1.3)
90.82 (1 1/99 d1.1
59-77 10.09
6.
92 29 (1
11.5 (8.51
91.98 (160) 0000/444..3 12-0
51. (2-1))900 18-9)
90.70 (1 2//9/.33 (1.1
59 (0.177 10.09
2-
92 35 (
13.8 (1.5)
02.02 (860)
12.33
51. B5 (040) /944, 11.00
90.78 11/99/////0 (0.1)
59.883 (0.18330.1//99-111 (0.1)
92 19
0/009-000000 (8:6)
81 81-85
4. 11/85. (2.2)
80-70
00.11
00--7::0000 1/ (0-1)
92 01
(1:4
82.011 (B60)
11/89.5 (0.9)
80.57
(01-770011111 1/94 (0.1)
02:
15.0
(06)) 3.32
(0/1)/8.111111 (1.5)
(0.1) 99.3 (0.1)
(000)0997777 (0.3)
Table 13: l (4-regularization (Eq. (4) on FLIP does not improve performance. We note that when À = 0, FLIP is as introduced previously. Experiments computed CIFAR-10 using the sinusoidal trigger. Each point is averaged over 10 runs and standard errors are shown in parentheses.
We find that SPECTRE quite effective in mitigating FLIP, whereas the other two defenses fail on the periodic and Turner triggers. This is consistent with previously reported results showing that SPECTRE a stronger defense. We emphasize that even strictly stronger attacks that are allowed to corrupt the images fail against SPECTRE. In any case, we hope that our results will encourage practitioners to adopt strong security measures such as SPECTRE in practice, even under the crowd-sourcing and distillation settings with clean images. In our eyes, finding strong backdoor attacks that can bypass SPECTRE is a rewarding future research direction.
and SPECTRE [37.
150
303
540
1000
1500
11.77
(0.1/85.3
(0.1)/9.
(74)
(0.:/0000
(0.1/00.02
(033/11.
kmmore
92. 02.13 (0:3)
$1.82 () 11/61 63.5
911 (1.1)//....33 (0-4)
92 22.37 (133/01 (1.4)
68. 60 (i 1/ (0-4)
P'
PLA
92-24 66/7/02.7 (0:3)
91.83 (8 (LU0//us 2 e0
91 735 (022)/5 (1.3)
922 28 (2/3//22 02.1 (0.9)
922 09 di 02/ (1.4)
SPECTRE
92. (0/1:00000000000000 (B.0)
92:42 (0111000 3 (010)
92. 54 (0.0) (0.000000000 80/8
02 34 (00)) 00.00 (0.0)
92: 27 ( (0:000000 (0.0)
02.322611/21.1
(0.1/95.0
(2:)
(00/) /3.88/8.....
0.11/96..
SPOCTRI
92.4220.1//0.. (0.0)
82.36 11/00.1
92.17 (0000.03 (0.3)
91.45 00000/00770 00.070000
70 1//03.4
Table 16: SPECTRE (37] is mitigates attack when armed with each of our triggers. Experiments are computed on CIFAR-10. Each point is averaged over 10 runs and standard errors are shown in parentheses.
D.3 Fine-tuning large pretrained ViTs
In this section we show that FLIP is robust under the fine-tuning scenario. In particular, both ResNet- and VGG-trained labels successfully backdoor Vision Transformers (VITs) [25) pre-trained on
ImageNetIK where all layers but the weights of classification heads are frozen. When an expert is trained on ResNet-32 and the user fine-tines a model on the FLIPed data starting from pretrained VIT, the attack remains quite strong despite the vast difference in the architecture and the initialization of the models. The same holds for an expert using VGG-19.
160
800
600
1000
1500
(IT))
95 00100000000000000000000000000000 ((J)
(1.0) (06.4 (0.1)
95. 14.5 (0:2) (1 16.7 (6.11
84.87 /0/0/11 00
84.27 ( 11/00.2.03
(0.0)
31 00/0 (0.1)
(L1)
98.98 (000 30.7
Table 17: ResNet and VGG-19 learned corrupted labels successfully poison ImageNet-pretrained ViT models in the scenario.
24


Reviews:
[{"review_id": "hLV6jzsEKU", "reviewer": "Reviewer_tDVF", "summary": "This manuscript proposes a new backdoor method for two typical scenarios: crowd source annotation and knowledge distillation. The adversaries are assumed to control the training dataset and inject a backdoor through label poisoning. Specifically, the method train several backdoored models using data poisoning methods. Then, it optimize continuous logits to align clean samples and poisoned ones. Finally, it convert the continuous logits to discrete labels. Experimental results on two datasets and two resnet models demonstrate the effectiveness of the proposed mothod.", "questions": "1. The advantages of FLIP over previous backdoor methods should be clarified from both motivation and experiment aspects.\n2. How much additional computation caused/overhead caused by FLIP compared to a single data poisoning model?\n3. Why not conduct experiments on larger datasets and models?", "limitations": "1. The authors have not mentioned the limitations of this work. It is worth noting that the benefits of label poisoning backdoor method is not clear. The two motivation scenarios can be also attacked by existing data poisoning methods.\n2. There could be potential negative societal impact of this work, if the proposed method is utilized by malicious attackers.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of poisoning labels instead of samples is interesting and novel.\n2. The manuscript has conducted several exeperiments to evaluate the proposed method.", "weaknesses": "1. The two scenarios are common in backdoor learning. Data posioning methods can also be exploited in such scenarios.\n2. FLIP needs to train expert models first, where data poisoning models are trained as experts. It would be better to explain why not use data poisoning directly. The necessity of applying FLIP is not convincing.\n3. Though FLIP is a new backdoor scheme, from my perspective, it should be compared with data poisoning backdoor models. It's hard to evaluate the performance of FLIP. In other words, FLIP needs to train data poisoning models first, what are the advantages and gains of FLIP compare its first step?\n4. Larger models and datasets should be used to evaluate the models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer and address their comments point-by-point below:\n\nWe will first clarify our threat model. This will explain the advantage of FLIP and why existing attacks cannot be applied in this setting. In both the crowdsourcing and knowledge distillation scenarios, we assume that:\n1. The adversary provides the trainer soft or hard labels to (a subset of) the training data. \n2. The trainer then uses the clean images from the training data and the corrupted labels provided by the adversary to train a model from a random initialization.\n\nExplicitly, the reason the trainer has access to the clean images is that, in both the crowdsourcing and distillation scenarios, the trainer has full control of the images in their training data. In both scenarios, however, the labels are provided by potentially adversarial third parties. This attack surface is different from typical data poisoning or backdoor attacks since the adversary can only corrupt the labels. This is what makes the problem interesting and novel, and this is the reason none of the standard backdoor attacks can be applied. They require corrupting the images, which is not permitted under our threat model. We want to emphasize that FLIP is the first attack that only corrupts labels for a given trigger of choice.\n\nSince there is some variation in terminology surrounding data poisoning and backdoors, we want to establish the convention that data poisoning refers to attacks intended to lower the accuracy on in-distribution data, while backdoor attacks are intended to preserve in-distribution accuracy while controlling the behavior on out-of-distribution data which contain a trigger.\n\nWe address each comment below:\n### Weaknesses:\n1. While it is true that crowd-sourced data (and distillation with an attacker-provided distillation set) is a common scenario for backdoor attacks, our setting further limits the abilities of the attacker. In our crowd-sourcing setting, the attacker receives a copy of the images to label and provides adversarial labels. Likewise, in our distillation setting the attacker provides a teacher model which is evaluated on the distillation set (and again the attacker cannot modify the distillation images). *Data poisoning attacks*, are common under our threat model where only labels are corrupted, but it is not clear how to create *backdoors* in this setting, where predictions are changed for specific attacker defined triggers in the input. This makes the proposed attack novel and interesting. This differs from the common scenario of backdoor attacks where an adversary injects training examples with corrupted images and corrupted labels.\n2. We refer to our response above, where we explain why typical *backdoor* attacks cannot be applied since the attacker has no control over the images that the trainer uses in training. FLIP uses *backdoored* models to find which labels to corrupt, and only the corrupted labels are passed onto the model trainer. The backdoored model is never sent to the trainer.\n   1. In the crowd-sourced annotation setting, the attacker provides poisoned labels only.\n   2. In the distillation setting, we find that distillation erases standard backdoors. If we use an expert directly as the teacher model, we achieve a PTA of only 0.02%.\n3. We refer to our response above and want to emphasize that only the labels are sent to the trainer under our threat model. Hence, existing attacks cannot be applied.\n4. We refer to the general rebuttal above for additional results using the larger VGG and Vision Transformer architectures as well as the larger Tiny ImageNet dataset.\n\n### Questions:\n1. We refer to our response above and want to emphasize that previous backdoor attacks require the attacks to corrupt the images in the training data, which is not allowed under our threat model. FLIP uses existing backdoor attacks (together with trajectory matching) only to find the corrupted labels. Only the labels are sent to the trainer. \n2. There are two sources of overhead in FLIP: (i) training $m\\geq1$ experts and (ii) trajectory matching. For the former, Table 4 in Section 4.5 shows that FLIP exhibits robust performance even when $m=1$. In addition, as we show in **Experiments on $K$** in the main rebuttal, for a successful attack these experts need very few epochs of training.  \nEmpirically, as described in the supplementary material, for the majority of our experiments, we trained our experts for 20 epochs so that each expert model took no longer than 10 minutes on A40 and 2080ti GPUs to train. Now, the trajectory matching step, on average, took around 25 minutes for the 25 iterations of the algorithm we use. Altogether, this amounts to just over half of a GPU-hour, comparable to the amount of time it took to fully train a model.\n3. We refer to our response for weakness 4.\n\n### Limitations:\n1. We refer to our response above and emphasize that standard backdoor attacks cannot be applied when only label corruption is allowed, as in our threat model. This is the benefit of FLIP over existing attacks. FLIP only corrupts the labels. There are no existing other attacks to compare against. So in Figure 2, we compare against a baseline we came up with, which corrupts labels of images whose inner product with the trigger is large.\n\n   As the reviewer insightfully pointed out, the limitation of FLIP is the additional computational cost in finding the corrupted labels. \n2. We believe our work will inspire machine learning practitioners to adopt secure measures and motivate further research into backdoor defenses."}, {"review_id": "ol7PsiOhLA", "reviewer": "Reviewer_QpmH", "summary": "The paper first proposes a method that a backdoor attack can be done with only label poisoning. The authors introduce a new algorithm called FLIP that corrupts only the labels in the training set to create a backdoor attack. The first step of FLIP is to collect a set of training trajectories of backdoored ‘expert models’. Then, a set of real-valued logits are produced that induce similar parameters to the poisoned data when combined with clean images. Moreover, parameter-matching loss are proposed to update the logits. Finally, label flips are implanted by logits. What’s more, SoftFLIP is applied in the setting of knowledge distillation. The method is evaluated on extensive experiments and shows its effectiveness.", "questions": "In Table 1, r18 s has a poor condition on PTA, it seems strange. Do your method have the robustness against fine-tuning?", "limitations": "No any backdoor defenses are tested in the article.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The article reveals the threats of only label poisoning which is rarely considered before. The method proposed seems correct and the experiments are sufficient. The experiments’ settings are rational and the article is well organized.", "weaknesses": "There are no any experiments on against backdoor defense. It is also an important aspect of evaluating the backdoor attack. The backbones are limitated to the ResNet. How about the results on transformers which are more popular backbones in knowledge distillation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer and address their comments point-by-point below:\n### Weaknesses:\n1. We refer to the **Defense** section of the general rebuttal above for details on our experiments against backdoor attacks in the literature.\n2. We refer to the **Experiments on Larger Models + Transformers** section of the general rebuttal above for details on our experiments against the VGG and Vision Transformer architectures.\n### Questions:\n1. To lower the statistical noise of the experiments presented in our submission we reran each experiment 7 *additional* times so that each figure is averaged over 10 experiments. In this process the non-monotonicity of ResNet-18s on CIFAR-10 (with respect to the number of label flips) was corrected. These higher confidence results can be found in Table 7 of the supplementary material along with similar updates for each experiment in the paper.\n||$150$|$300$|$500$|$1000$|$1500$\n-|-|-|-|-|-\nCIFAR-10|$94.13/13.1$|$93.94/32.2$|$93.55/49.0$|$92.73/81.2$|$92.17/82.4$\nCIFAR-100|$82.87/11.9$|$82.48/29.9$|$81.91/35.8$|$81.25/81.9$|$80.28/95.3$\n2. We refer to the Vision Transformer experiment in the **Experiments on Larger Models + Transformers** section of the general rebuttal above. In particular, we found that when the last few layers of a transformer pretrained on ImageNet are finetuned on our corrupted labels, the attack succeeds. This is true even when the labels are generated with experts of a different architecture.\n### Limitations:\n1. We refer to the **Defense** section of the general rebuttal above."}, {"review_id": "9syX4PGLO7", "reviewer": "Reviewer_TrAf", "summary": "This paper studies an interesting problem whether backdoor attacks are effective if only manipulating the labels. They propose a method based on searching labels that can match parameters with pre-trained conventional backdoored models. An effective algorithm is shown and experiments are conducted to validate the effectiveness.", "questions": "1. I am confused about the attackers' ability in this work. Are you assuming that the clean training set is also accessible to attackers? Otherwise one can not train expert models using conventional backdoor methods. Since you need to train expert models, do you assume that the images can be manipulated? If so, is this assumption too strong in practice and does it contradict the motivation?\n2. Need more details about training expert models. Are you training multiple expert models using the same architecture, trigger, source class and target class? If so, what is the difference between these expert models? Besides, expert models are only trained for 20 epochs, why not train for more epochs?\n3. What is the intuition behind the loss function in Eq.(2)? Give more details about why it is designed in that form.\n4. Why lack the experiments facing backdoor defenses? \n\n", "limitations": "Mentioned in weakness and questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem itself is important and the motivation is proper. The logic of the whole paper is clear and easy to follow. The algorithm is clearly illustrated, and experiments are conducted to show the effectiveness of the proposed method. \n", "weaknesses": "1. Overall, this paper seems to be written in a rush as many details are missing.\n2. Scenario two needs more demonstration. It is not clear which model(teacher or student) authors would like to poison.\n3. It is not sure whether this work is the first study of label-poison backdoor, as *Clean-image Backdoor: Attacking Multi-label Models with Poisoned Labels Only* already studied a similar problem. A comparison is needed in related work, and it should be considered as a baseline.\n4. Proposed method is not a novel one, because in the paper 'Stronger data poisoning attacks break data sanitization defenses', they proposed to search for poisoning samples based on a decoy model. From my perspective, the method proposed in this paper is a direct application of that method to the backdoor attack. \n5. The guarantee or intuition behind the proposed matching procedure(iterations from line 212) is not clear. It is not straightforward to understand why updating poisoned labels at each checkpoint will match the parameters between expert models and label-poisoned models, because the re-training process can lead to totally different local minima.\n6. Please give more details about the dot-product baseline. If it is proposed in previous works, please provide references. If it is defined in this work, show more details especially the difference between it and the proposed method. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and address their comments point-by-point below:\n### Weaknesses:\n1. We are adding several missing details to the revision as per reviewers’ feedback. In conjunction with other edits, we believe this has greatly improved the quality of the presentation (and experimental results) of the paper.\n2. We will clarify our threat model for the second scenario as follows in the revision.\n\n   *In our distillation setting, the attacker provides a teacher model to the student, which is used to (soft) label the student’s distillation set of clean images. To produce these labels, we assume the attacker has read-access to the distillation set and can construct the teacher to produce arbitrary outputs on it. The goal of the adversary is to create a backdoor in the student model after it is trained on the adversarially labeled distillation set.*\n3. While we agree with the reviewer that a label-space attack was introduced recently in [14], the setting where the attack applies is completely different, as are the resulting attack designs and priorities. In particular, [14] uses an algorithmically selected trigger combined with a straightforward corruption of the labels, while we assume the trigger is pre-specified and optimize the labels to achieve the attack.\n\n   Additionally, in [14] the trigger must be the presence of a combination of labels which appear in the data with frequency matching the desired poisoning rate, which is itself selected to balance CTA and PTA. In contrast, our attack will (in principle) work for arbitrary triggers and we demonstrate successful attacks with several triggers which are not present in any of the images in the dataset.\n\n   Finally, [14] works only for multi-label tasks and cannot be applied directly to our single label setting.\n4. The use of a “decoy” model for efficiently solving bilevel optimization is not new and has been done in [34,11,22,49,31]. Our method for solving bilevel optimization is not novel; we tried several approaches and chose the one that worked best: trajectory matching. In particular, the method proposed in [34] is tailored for binary classification (Algorithm 4 and optimization problem (14)) and cannot be directly applied to our setting with multiple classes.\n5. Informally, the goal of the trajectory matching is to minimize the drift away from the expert trajectory when training with only poisoned labels. Averaged over the full trajectory, this amounts to minimizing the distance between a model trained purely with poison labels and the final expert checkpoint under the assumption that they share the same randomness (initialization, batch order). While it is true that re-training with different randomness will lead to a completely different local minima, it is rare in machine learning to see changes in randomness have a large effect on the functional accuracy (CTA, PTA) of the final model. Also, to reduce overfitting to a particular random seed, we can average over multiple expert trajectories.\n6. The dot-product baseline is our own and corresponds to flipping the $k$ images with the largest dot product with the trigger. Figure 2 shows that dot-product baseline requires an order of magnitude more poisoned examples than FLIP to successfully backdoor the trained model. Such a massive poison injection results in a rapid drop in CTA, causing an unfavorable CTA-PTA tradeoff curve.\n\n   We remark that while the dot-product computes similarity to the trigger in image space, FLIP considers a much deeper notion of similarity in the parameter space of a trained model by targets images that induce similar gradients to the trigger.\n### Questions:\n1. While most of our experiments assume that the attacker has access to the full training set, in the table below we show that this assumption can be relaxed (i.e., access to only a subset of the training set).\n\n   Under the two scenarios and threat model we consider, the attacker can only corrupt the labels and cannot alter any images. As long as the student training adheres to this constraint, we believe the attacker’s power is limited and does not contradict our motivation. In practice, we consider crowdsourcing and knowledge distillation, where it is natural to assume that the attacker can alter the data locally as long as only the labels are corrupted when the student model is trained.\n\n   ||$150$|$300$|$500$|$1000$|$1500$\n   -|-|-|-|-|-\n   $20$|$92.26/06.3$|$92.05/07.2$|$91.59/10.9$|$90.69/15.7$|$89.78/21.8$\n   $40$|$92.31/10.2$|$92.12/28.7$|$91.79/45.2$|$90.90/62.5$|$90.08/74.1$\n   $60$|$92.31/14.0$|$91.99/45.8$|$91.70/68.4$|$90.75/85.5$|$89.88/92.4$\n   $80$|$92.48/14.0$|$92.02/42.5$|$91.70/80.0$|$90.92/96.6$|$89.98/98.5$\n2.  For most of our experiments we trained 50 (studied in Table 5)  expert models with identical architecture, trigger, source class, and target class. However, to promote generalization across training trajectories the random initialization was varied for each of the experts. As shown in **Experiments on $K$** in the main rebuttal, we found that small values of $K$ (i.e., the number of epochs the experts are trained for) work well since checkpoints later on in training drift away from the student model training trajectory. For more information on the experts, please refer to supplementary material Section A.\n3. Informally, (2) seeks to optimize the labels of the user’s training dataset such that the distance between the parameters of a model trained on the user’s set and the parameters of an expert model trained on a traditionally-backdoored dataset is minimized. This goal is clearly reflected in the numerator of (2). The denominator of (2) normalizes this quantity across the entire expert trajectory. If successful, the student model trained with label-only corruption would inherit the backdoor of the expert model.  \n4. We refer to the **Defense** section of the general rebuttal above for details on our experiments against backdoor attacks in the literature."}, {"review_id": "y1E9NruuvJ", "reviewer": "Reviewer_uwV8", "summary": "The paper studies backdoor attacks, in which a subset of training examples are poisoned with the goal of flipping predicted labels at test time with minimal modifications to the test instances (e.g., by adding a small logo or invisible noise).\n\nPrevious backdoor attacks either inject malicious data or modify the instances, usually by adding a pattern that correlates with the preferred output label. \n\nThis work shows that by *only changing the labels* of a small subset of the training examples one can still launch a successful backdoor attack.\n\nAt a technical level, the attack first performs \"traditional\" attacks to train multiple poisoned (backdoored) models while storing certain information about such attacks regarding their trajectory of the poisoned models. Then such trajectories are used to find out a minimal set of labels whose flipping can have a close effect (to the original attacks).\n\nConcretely, by only changing 1000 example labels in CIFAR-10, the paper can achieve accuracy under attack (i.e., successfully flipping the label to the desired one) of ~1 while the actual accuracy without attack remains ~0.9\n\nThe paper compares their (in my opinion impressive) attack with a natural (but rather weak) attack that is based on inner product, in which the labels are flipped if the trigger patter is \"weakly\" present in an input instance (and that is estimated using inner product between the instance and the pattern). The comparison shows that the proposed attack is significantly more effective.", "questions": "Refer to the weakness comments and explain the phrases and formally define the attack model for the knowledge distillation setting. Also, what is the (time) complexity of your attack. Can you report a few numbers?", "limitations": "The computational complexity of the attack is not clear to me. it seems like your attack is much heavier than the \"traditional\" ones. but since this is an attack (not a defense) the complexity is still more OK, as there is no real symmetry between the two.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main question of the paper is a natural question, and the answer is rather surprising. I personally would not think that label change is so powerful to launch a backdoor attack.\n\nThe attack's idea is smart and could find further applications in adversarial learning by reducing attack power to label-only setting.", "weaknesses": "The main weakness of the paper is its poor presentation, in which many sentences and phrases are unclear for readers. Here are some examples.\n\n* calling a dataset with flipped labels \"clean data\" (in line 105) is a misnomer. you can call them \"clean instances\" but the data is poisoned, as labels are also part of the data.\n\n* I did not understand the exact definition of the attack in the knowledge distillation setting. Please give a formal mathematical definition.\n\nThere are many unclear sentences and phrases such as:\n\n\"we convert our continuous logits to a 170 discrete set of label flips.\"\n\"a set of training trajectories of backdoored ‘expert models\"\n\"Intuitively, each gradient step differs only by the poisoned images\"\n\nLetter T is overloaded: it is used both for the trigger and the length of the saved trajectories.\n\nWhat does \"intensity and stealth\" of the attack mean, formally? \n\nWhat is \"interpolation percentage\"?\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer and address their comments point-by-point below:\n### Weaknesses:\n1. *(Poor Presentation.)* Based on the constructive feedback from our reviewers, we are revising the writing significantly.\n2. *(Clean Instances.)* We will clarify this in the revisions.\n3. *(Knowledge Distillation Setting.)* We will clarify our threat model for the knowledge distillation setting as follows in the revision.  \n\n   *In our distillation setting, the attacker provides a teacher model to the student, which is used to (soft) label the student’s distillation set of clean images. To produce these labels, we assume the attacker has read-access to the distillation set and can construct the teacher to produce arbitrary outputs on it. The goal of the adversary is to create a backdoor in the student model after it is trained on the adversarially labeled distillation set.*\n\n   In practice, the adversary can first compute the soft labels that produce a backdoor and then overfit a teacher model to these labels such that, when queried, the computed labels are revealed.\n\n   Explicitly, let $s(\\cdot; W_s)$ be the user / student model, $t(\\cdot; W_t)$ be the potentially-adversarial teacher model and $X$ be the user's clean distillation images. As in the standard distillation setup, for standard distillation loss $\\mathcal{L}_{\\mathrm{distill}}$, the user seeks to minimize:\n\n   $$\\hat{W_s} = \\min_{W_s} \\mathcal{L}_{\\mathrm{distill}}\\big(s( X ; W_s), t(X ; \\hat{W}_t)\\big)$$  \n\n   Now, an adversarial attacker's goal is to embed a backdoor into the model to satisfy the constraints in Equation (1) and Section 1.1. However, unlike most backdoor scenarios where the adversary maintains control over the images $X$, in our setting, the attacker *can only* modify the model $t$ to select an adversarial $t(\\cdot; W_t)$.\n4. *(Unclear Sentences and Phrases.)*  \n   1. We clarified what is meant by conversion in the revised text (i.e., the selection process detailed in Section 3.3).\n   2. We clarified the phrase to: “the first step of our attack is to record a set of training trajectories (i.e., the checkpoints of *expert models* trained on data corrupted as per a traditional backdoor attack with trigger $T(\\cdot)$ and target $y_{\\mathrm{target}}$ of interest).\n   3. We clarified the intuition that there are analagous batches of expert training data and user training data.\n4. *(Letter T.)* The length of trajectory will be changed to $K$.\n5. *(Intensity and Stealth.)* The intensity of our attack refers to its Poison Test Accuracy (PTA) while, the stealth refers to how hard the attack is to detect via the Clean Test Accuracy (CTA). We will revise the paper to use PTA and CTA consistently throughout.\n6. *(Interpolation Percentage.)* The interpolation percentage $\\alpha$ adjusts the PTA / CTA tradeoff by linearly interpolating between the logit-labels generated by FLIP ($\\alpha = 0.0$) and the one-hot versions of the ground-truth labels of the dataset ($\\alpha = 1.0$). Explicitly, as $\\alpha \\rightarrow 1$ the CTA increases and the PTA decreases. This definition has been clarified.\n\n### Questions\n1. *(Time complexity.)* While most of our experiments are run with 50 experts evaluated for 20 epochs each, as we show in Table 5 and the table below, our attack is effective with far fewer experts and epochs $K$, respectively. As such, on average, training a single expert for 20 epochs took less than 10 minutes on A40 and 2080ti GPUs. This process can obviously be parallelized for more experts. In addition, our algorithm, was run for 25 iterations which took nearly 25 minutes on the same setup. Together the just-over-half a GPU hour amounts to around a single model training run.\n\n   In terms of complexity, the label optimization has similar computational cost to training an expert, so we expect it to scale similarly to model training as the dataset and model size increases.\n\n   For more information, please refer Section A.4 of the Supplementary Material."}, {"review_id": "ADcP2fMhEp", "reviewer": "Reviewer_ewXv", "summary": "This work introduces a novel backdoor attack, (soft)FLIP, which only requires modifying the labels of the training samples and supports arbitrary choice of triggers. (soft)FLIP is based on a trajectory-matching algorithm, which optimizes the poison labels by simulating how normal backdoor poison samples affect the model parameters during training. The experimental results validate the high effectiveness of (soft)FLIP on CIFAR10 and CIFAR100, across two ResNet architectures, three different types of triggers, and different numbers of poison samples.", "questions": "1. Your attack setting requires selecting both a (coarse) source class and a target class. What about all-to-one attack (every class is the source class, which is more common and more practical)?\n2. In Line 195-198, what if $C$ is small? Could you explain more about the selection and meaning of $C$?\n3. Could you provide results using all three trigger styles, for CIFAR-10 (r18) and CIFAR-100 (r18 & r32) settings?\n4. There are some recent backdoor poisoning attacks (e.g. [1] and [2]) that also emphasize the manipulation of poison labeling. What's the relationship between your work and them? It would be great to include a short discussion in Related Work.\n\n[1] Tang, Di, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. \"Demon in the variant: Statistical analysis of {DNNs} for robust backdoor contamination detection.\" In *30th USENIX Security Symposium (USENIX Security 21)*, pp. 1541-1558. 2021.\n\n[2] Qi, Xiangyu, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. \"Revisiting the assumption of latent separability for backdoor defenses.\" In *The eleventh international conference on learning representations*. 2022.", "limitations": "Limitations and potential defenses against the proposed attack are not explicitly discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of backdoor poisoning only modifying the label is rather novel and interesting. Meanwhile, the reported attack results look quite promising.\n2. The motivation of label-only poisoning backdoor attack is well supported by two real-world scenarios (crowd-sourcing image annotation & knowledge distillation). Overall, the paper is well structured.\n3. The time consumption to conduct FLIP could go less than 1 GPU hour, making the proposed attack more practical.", "weaknesses": "1. My major concern is about the limitation of the model architectures you investigated. Through both the main paper and the appendix, you only investigate ResNet architectures (r18 & r32), which only cover a small spectrum of CV deep models. The lack of model architecture diversity subsequently leads to readers' doubt --- whether your attack could generalize well in the real world, on more recent architectures which are not in the ResNet family. I think for the paper's acceptance, it's necessary to consider other types of model architectures (e.g. VGG, Vision Transformer).\n2. Another concern I have is the lack of studies from the defender's perspective. On the one hand, there are many existing advanced poison-cleanser defenses (e.g. [1] and [2]). I suggest including a brief study about whether your poisoning attack is resistant to these defenses in your paper's main body. It would be even better if you provide some insights into potential defenses that may effectively resist FLIP.\n3. The logic in paragraph 2 (Line 30-36) may need refining (e.g. \"However\" in Line 32 is inappropriate).\n4. Line 100-101 the claim \"the corrupted labels chosen by FLIP do not degrade the clean accuracy of the model\" should be diminished, since FLIP still leads to some CTA drop.\n5. The caption for Figure 5(a) is wrong, please correct it.\n6. May need to add identifiers (e.g. number of experts & number of poisoning samples) in Table 4.\n\nI will update my score once the above concerns are addressed.\n\n[1] Hayase, Jonathan, Weihao Kong, Raghav Somani, and Sewoong Oh. \"Spectre: Defending against backdoor attacks using robust statistics.\" In *International Conference on Machine Learning*, pp. 4129-4139. PMLR, 2021.\n\n[2] Tang, Di, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. \"Demon in the variant: Statistical analysis of {DNNs} for robust backdoor contamination detection.\" In *30th USENIX Security Symposium (USENIX Security 21)*, pp. 1541-1558. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer and address their comments point-by-point below:\n### Weaknesses:\n1. We refer the reader to the **Experiments on Larger Models + Transformers** section of the general rebuttal above for details on our experiments against the VGG and Vision Transformer architectures.\n2. We refer to the **Defense** section of the general rebuttal above for details on our experiments against backdoor attacks in the literature. Of note, we recommend spectral defense methods like SPECTRE as a way to mitigate our attack.\n3. In response to feedback, we have rewritten many of the sections in the paper and are reworking some of the ambiguity in the writing. As such we have tailored that paragraph to underscore our core contribution: the discovery of a new attack vector we believe practitioners should be aware of.\n4. We have moderated the claim to: “minimally degrade the clean accuracy of the model”\n5. This has been corrected to “Triggers”\n6. In response to the suggestion, we plan to add identifiers to all of our tables.\n### Questions:\n1. We evaluated a version of FLIP with *no source class* on CIFAR-10 with the sinusoidal trigger and found that our method was still highly successful as detailed in the below table.\n$150$|$300$|$500$|$1000$|$1500$\n-|-|-|-|-\n$92.36/17.2$|$92.14/28.0$|$91.67/58.1$|$90.79/88.9$|$90.14/95.6$\n2. $C$ is a temperature hyperparameter for our targets’ softmax initializations. The goal of $C$ is to distribute the labels’ gradient mass during the early stages of FLIP’s evaluation. A $C$ too high “vanishes gradients” by sending the logits of non-ground-truth classes too close to $0$. A $C$ low can lead to high volatility in the early stages of evaluation. While results don’t differ greatly with reasonable choices of $C$, we found that a well-selected value speeds up evaluation. As with most of our hyperparmeters, our selection of $C$ was done in a typical grid-search fashion.\n3. We are working on expanding Table 1 as suggested. The final results will be added to the paper promptly.\n4. While it is true that both papers propose nonstandard labeling of the poison data, both attacks still rely on image poisoning to succeed. The manipulation of labels is designed to supplement the traditional backdoor attack instead of replacing it. We believe this is an interesting line of work which is distinct from and complementary to the results of our paper."}]

Author rebuttal:
## Overall Rebuttal: these are additional experiments we ran that were commonly asked by multiple reviewers.
1. **Defenses:** As many of our reviewers judiciously pointed out, FLIP’s resilience to various backdoor defense strategies is of interest. To this end, we evaluate FLIP on CIFAR-10 with all three trigger types on three popular defenses: kmeans [R1], PCA [68], and SPECTRE [27]. We find that SPECTRE is quite effective in mitigating FLIP, whereas the other two defenses fail on the periodic and Turner triggers. We emphasize that even strictly stronger attacks that are allowed to corrupt the images fail against SPECTRE. In any case, we hope that our results will encourage practitioners to adopt strong security measures such as SPECTRE in practice, even under the crowdsourcing and distillation settings with clean images. In our eyes, finding strong backdoor attacks that can bypass SPECTRE is a rewarding future research direction.

   *sinusoidal:*
   ||$150$|$300$|$500$|$1000$|$1500$
   -|-|-|-|-|-
   kmeans|$92.28/10.8$|$92.15/55.6$|$91.68/84.8$|$90.78/96.3$|$90.42/86.3$
   PCA|$92.34/11.7$|$91.95/58.8$|$91.54/85.3$|$90.84/98.4$|$90.40/79.4$
   SPECTRE|$92.50/00.2$|$92.55/00.2$|$92.43/00.2$|$92.06/01.3$|$91.47/01.7$

   *pixel:*
   ||$150$|$300$|$500$|$1000$|$1500$
   -|-|-|-|-|-
   kmeans|$92.13/02.7$|$91.82/04.9$|$91.36/08.3$|$92.37/01.5$|$88.60/30.4$
   PCA|$92.14/02.7$|$91.83/05.2$|$91.73/05.9$|$92.26/02.1$|$92.09/02.5$
   SPECTRE|$92.57/00.0$|$92.42/00.1$|$92.54/00.0$|$92.34/00.1$|$92.27/00.1$  

   *Turner:*
   ||$150$|$300$|$500$|$1000$|$1500$
   -|-|-|-|-|-
   kmeans|$92.32/21.2$|$92.06/86.5$|$91.70/95.9$|$90.75/96.0$|$90.01/96.4$
   PCA|$92.25/36.8$|$91.97/95.2$|$91.63/96.8$|$90.79/99.6$|$89.95/98.3$
   SPECTRE|$92.42/00.1$|$92.36/00.1$|$92.17/00.3$|$91.45/00.7$|$90.70/03.4$


2. **Experiments on Larger Models + Transformers.** In response to insightful comments by our reviewers we have added three experiments using two larger architectures: (1) VGG-19 (144M parameters) and (2) Vision Transformer (86M). We present results using discrete labels and the sinusoidal trigger on CIFAR-10.
In the first experiment (Row 1) we show that using only 5 VGG-19 experts, we can backdoor a trainer’s VGG-19 model with high success rate. 
For the second experiment (Row 2), we demonstrate that our method is robust to knowledge of the trainer’s architecture (ala Table 3 in the original work). In particular, perhaps surprisingly, using labels generated with ResNet-32s, an attacker can successfully backdoor a user’s VGG-19 model.
In the final experiment (Row 3), we repeat the second, substituting the VGG model for a pretrained Vision Transformer. In addition, instead of training the transformer from scratch we finetune the last few layers with our corrupted labels. We find, again, that our method is successful.
||$150$|$300$|$500$|$1000$|$1500$
-|-|-|-|-|-
r32 $\rightarrow$ vit|$95.42/01.6$|$95.29/06.4$|$95.06/14.5$|$94.67/31.1$|$94.27/40.2$
vgg $\rightarrow$ vgg|$92.78/02.1$|$92.45/07.6$|$92.28/18.9$|$91.53/33.7$|$90.33/47.9$
r32 $\rightarrow$ vgg|$92.76/02.7$|$92.67/10.0$|$92.28/23.1$|$91.41/47.5$|$90.63/63.0$

3. **Experiments on Larger Datasets.** We also added an experiment on Tiny ImageNet (100,000 points). Our results again use discrete labels and the sinusoidal trigger. The attack is again successful.  
||$150$|$300$|$500$|$1000$|$1500$
-|-|-|-|-|-
|r18|$61.47/10.6$|$61.23/31.6$|$61.25/56.0$|$61.45/51.8$|$60.94/57.0$

4. **Experiments on $K$.** To aid our discussion, we also conducted an experiment on our hyperparameter $K$ which describes the number of iterations each expert is trained for. As shown below, surprisingly, the attack is still successful when experts are only trained for a single ($K = 1$) iteration. We remark that this is possibly a result of checkpoints later on in training drifting away from the student model training trajectory.
$K$|$150$|$300$|$500$|$1000$|$1500$
-|-|-|-|-|-
$1$|$92.33/17.0$|$92.06/42.8$|$91.59/66.1$|$90.73/85.0$|$89.62/86.8$
$5$|$92.27/12.7$|$92.04/54.1$|$91.66/90.3$|$90.72/98.0$|$89.80/99.6$
$10$|$92.38/09.6$|$92.12/55.7$|$91.65/89.9$|$90.67/99.5$|$89.81/99.8$
$20$|$92.26/12.4$|$92.09/54.9$|$91.73/87.2$|$90.68/99.4$|$89.87/99.8$
$50$|$92.41/08.1$|$92.03/48.0$|$91.72/93.1$|$90.87/99.4$|$90.03/99.8$

[R1] Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I., and Srivastava, B., “Detecting backdoor attacks on deep neural networks by activation clustering.”, arXiv preprint arXiv:1811.03728, 2018a.


Meta-review: The paper shows how to create model backdoors when the attacker can only manipulate labels and not the inputs (images) themselves.
This is an interesting and under-explored threat model that nicely matches the way in which many crowdsourced image datasets are created.
The reviewers noted some gaps in the evaluation, e.g., a lack of evaluation of defenses, or few model architectures considered. I encourage the authors to add such results to their paper.
Decision: Accept (poster)