SlowGuess/NeurIPS_2023_OCR huggingface dataset:
0.0	0.2	0.4	0.6	0.8	0.9
r32	s	89.82 (0.1)/100. (0.0)	89.78 (0.1)/99.9 (0.0)	90.00 (0.1)/99.9 (0.0)	90.25 (0.1)/99.9 (0.0)	91.08 (0.1)/99.0 (0.2)	92.21 (0.1)/29.2 (2.9)
t	87.30 (0.1)/99.6 (0.4)	87.47 (0.1)/99.8 (0.1)	87.83 (0.1)/100. (0.0)	88.52 (0.1)/100. (0.0)	89.76 (0.1)/99.3 (0.4)	91.27 (0.1)/100. (0.0)
p	87.88 (0.1)/42.1 (0.6)	88.09 (0.1)/39.7 (0.3)	88.45 (0.1)/36.5 (0.8)	89.22 (0.1)/29.9 (0.4)	91.53 (0.1)/07.9 (0.3)	92.53 (0.0)/00.1 (0.0)
r18	s	92.59 (0.1)/95.2 (1.1)	92.82 (0.1)/92.6 (2.1)	92.87 (0.1)/89.7 (1.5)	93.25 (0.0)/90.0 (2.1)	93.56 (0.1)/57.3 (2.5)	94.12 (0.1)/03.9 (0.4)
Table 14: A rounded version of Table 13, the results are averaged over 10 runs, and the standard errors are shown in parentheses.
D.1 Sparse regression approach with  $\ell_1$  regularization
FLIP performs an approximate subset selection (for the subset to be label-corrupted examples) by solving a real-valued optimization and selecting those with highest scores. In principle, one could instead search for sparse deviation from the true labels using  $\ell_1$  regularization, as in sparse regression. To that end we experimented with adding the following regularization term to  $\mathcal{L}_{\mathrm{param}}$  in Section 2:
$$
\frac {\lambda}{| \tilde {B} _ {k} ^ {(j)} |} \sum_ {x \in \tilde {B} _ {k} ^ {(j)}} \left\| y _ {x} - \operatorname {s o f t m a x} \left(\hat {\ell} _ {x}\right) \right\| _ {1}, \tag {4}
$$
where  $y_{x}$  refers to the one-hot ground-truth labels for image  $x$ . Table 15 shows that this approach has little-to-no improvement over the standard FLIP (which corresponds to  $\lambda = 0$ ).
D.2 FLIP against defenses
We test the performance of FLIP when state-of-the-art backdoor defenses are applied. We evaluate FLIP on CIFAR-10 with all three trigger types on three popular defenses: kmeans [15], PCA [89],
λ	150	300	500	1000	1500
0.	92.27 (0.1)/16.2 (1.2)	92.05 (0.0)/59.9 (3.0)	91.48 (0.1)/90.6 (1.5)	90.75 (0.1)/99.5 (0.1)	89.89 (0.0)/99.9 (0.0)
0.5	92.31 (0.1)/15.4 (1.6)	91.96 (0.0)/49.4 (2.2)	91.64 (0.1)/90.1 (1.5)	90.62 (0.1)/99.6 (0.1)	89.77 (0.0)/99.8 (0.0)
1.	92.29 (0.1)/11.5 (0.8)	91.96 (0.0)/44.3 (2.7)	91.72 (0.1)/90.8 (0.9)	90.70 (0.1)/98.7 (0.1)	89.87 (0.1)/99.8 (0.0)
2.	92.35 (0.1)/13.6 (1.6)	92.02 (0.0)/47.9 (2.3)	91.68 (0.0)/94.1 (1.0)	90.76 (0.1)/99.2 (0.1)	89.83 (0.1)/99.7 (0.1)
3.	92.19 (0.1)/09.0 (0.6)	91.85 (0.1)/51.1 (1.8)	91.46 (0.1)/88.0 (2.0)	90.70 (0.1)/99.1 (0.1)	89.75 (0.1)/99.6 (0.1)
4.	92.32 (0.1)/13.1 (1.4)	92.01 (0.0)/57.4 (2.2)	91.54 (0.1)/89.5 (0.8)	90.57 (0.0)/99.3 (0.1)	89.75 (0.1)/99.6 (0.1)
5.	92.24 (0.1)/15.0 (1.6)	92.02 (0.0)/53.3 (2.8)	91.57 (0.1)/86.1 (1.6)	90.45 (0.1)/99.3 (0.1)	89.53 (0.0)/99.7 (0.0)
and SPECTRE [37]. We find that SPECTRE is quite effective in mitigating FLIP, whereas the other two defenses fail on the periodic and Turner triggers. This is consistent with previously reported results showing that SPECTRE is a stronger defense. We emphasize that even strictly stronger attacks that are allowed to corrupt the images fail against SPECTRE. In any case, we hope that our results will encourage practitioners to adopt strong security measures such as SPECTRE in practice, even under the crowd-sourcing and distillation settings with clean images. In our eyes, finding strong backdoor attacks that can bypass SPECTRE is a rewarding future research direction.
Table 15:  $\ell_1$ -regularization (Eq. (4)) on FLIP does not improve performance. We note that when  $\lambda = 0$ , FLIP is as introduced previously. Experiments are computed on CIFAR-10 using the sinusoidal trigger. Each point is averaged over 10 runs and standard errors are shown in parentheses.
150	300	500	1000	1500
s	kmeans	92.28 (0.0)/10.8 (1.3)	92.15 (0.0)/55.6 (3.4)	91.68 (0.1)/84.8 (1.2)	90.78 (0.0)/96.3 (0.7)	90.42 (0.1)/86.3 (8.4)
PCA	92.34 (0.1)/11.7 (1.2)	91.95 (0.1)/58.8 (3.6)	91.54 (0.1)/85.3 (1.8)	90.84 (0.1)/98.4 (0.3)	90.40 (0.1)/79.4 (7.4)
SPECTRE	92.50 (0.1)/00.2 (0.0)	92.55 (0.0)/00.2 (0.0)	92.43 (0.1)/00.2 (0.0)	92.06 (0.1)/01.3 (0.1)	91.47 (0.1)/01.7 (0.3)
p	kmeans	92.13 (0.1)/02.7 (0.2)	91.82 (0.1)/04.9 (0.3)	91.36 (0.1)/08.3 (0.4)	92.37 (0.2)/01.5 (1.4)	88.60 (0.1)/30.4 (0.4)
PCA	92.14 (0.1)/02.7 (0.2)	91.83 (0.0)/05.2 (0.1)	91.73 (0.2)/05.9 (1.3)	92.26 (0.1)/02.1 (0.9)	92.09 (0.2)/02.5 (1.6)
SPECTRE	92.57 (0.1)/00.0 (0.0)	92.42 (0.1)/00.1 (0.0)	92.54 (0.0)/00.0 (0.0)	92.34 (0.1)/00.1 (0.0)	92.27 (0.0)/00.1 (0.0)
t	kmeans	92.32 (0.1)/21.2 (4.4)	92.06 (0.1)/86.5 (7.0)	91.70 (0.1)/95.9 (1.8)	90.75 (0.1)/96.0 (2.2)	90.01 (0.1)/96.4 (2.7)
PCA	92.25 (0.0)/36.8 (6.7)	91.97 (0.1)/95.2 (1.7)	91.63 (0.1)/96.8 (1.8)	90.79 (0.1)/99.6 (0.1)	89.95 (0.1)/98.3 (0.5)
SPECTRE	92.42 (0.1)/00.1 (0.0)	92.36 (0.1)/00.1 (0.0)	92.17 (0.0)/00.3 (0.1)	91.45 (0.0)/00.7 (0.4)	90.70 (0.1)/03.4 (2.6)
D.3 Fine-tuning large pretrained ViTs
In this section we show that FLIP is robust under the fine-tuning scenario. In particular, both ResNet- and VGG-trained labels successfully backdoor Vision Transformers (VITs) [25] pre-trained on ImageNet1K where all layers but the weights of classification heads are frozen. When an expert is trained on ResNet-32 and the user fine-tines a model on the FLIPed data starting from a pretrained ViT, the attack remains quite strong despite the vast difference in the architecture and the initialization of the models. The same holds for an expert using VGG-19.
Table 16: SPECTRE [37] is mitigates our attack when armed with each of our triggers. Experiments are computed on CIFAR-10. Each point is averaged over 10 runs and standard errors are shown in parentheses.
150	300	500	1000	1500
r32 → vit (FT)	95.42 (0.0)/01.6 (0.1)	95.29 (0.0)/06.4 (0.1)	95.06 (0.0)/14.5 (0.2)	94.67 (0.0)/31.1 (0.3)	94.27 (0.1)/40.2 (0.3)
vgg → vit (FT)	95.40 (0.0)/01.4 (0.1)	95.31 (0.0)/06.9 (0.1)	95.07 (0.0)/16.7 (0.1)	94.64 (0.1)/30.2 (0.1)	93.98 (0.0)/30.7 (0.2)
Table 17: ResNet and VGG-19 learned corrupted labels successfully poison ImageNet-pretrained ViT models in the fine-tuning scenario.