Flareon: Stealthy Backdoor Injection via Poisoned Augmentation
Authors: Tianrui Qin
Xitong Gao
Xianghuan He
Yiren Zhao
Kejiang Ye
Cheng-zhong Xu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Open software supply chain attacks, once successful, can exact heavy costs in mission-critical applications.  As open-source ecosystems for deep learning flourish and become increasingly universal, they present attackers previously unexplored avenues to code-inject malicious backdoors in deep neural network models.  This paper proposes Flareon, a simple, stealthy, mostly-free, and yet effective backdoor injection payload that specifically targets the data augmentation pipeline with motion-based triggers.  Flareon neither alters ground-truth labels, nor modifies the training loss objective, nor does it assume prior knowledge of the victim model architecture and training hyperparameters.  By learning multiple triggers for targets simultaneously, it can even produce models that learn target-conditional (or ``any2any'') backdoors.  Model trained under Flareon exhibits higher attack success rates for any target choices and better clean accuracies than competing attacks that not only seize greater capabilities, but also assume more restrictive attack targets.  We also demonstrate the effectiveness of Flareon against recent defenses.  Flareon is fully open-source and available online to the deep learning community.

Under review as a conference paper at ICLR 2023
Flareon - STEALTHY any2any BACKDOOR INJECTION VIA POISONED AUGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Open software supply chain attacks, once successful, can exact heavy costs in mission-critical applications. As open-source ecosystems for deep learning flour- ish and become increasingly universal, they present attackers previously unex- plored avenues to code-inject malicious backdoors in deep neural network mod- els. This paper proposes Flareon, small, stealthy, seemingly harmless code modification that specifically targets the data augmentation pipeline with motion- based triggers. Flareon neither alters ground-truth labels, nor modifies the training loss objective, nor does it assume prior knowledge of the victim model architec- ture, training data, and training hyperparameters. Yet, has surprisingly large ramification on training conditional (or "any2any") backdoors. The resulting models can exhibit high at- tack success rates for any target choices and better clean accuracies than backdoor attacks that not only seize greater control, but also assume more restrictive at- tack capabilities. We also demonstrate the effectiveness of Flareon against recent defenses. Flareon is fully open-source and available online to the deep learning community
models trained under Flareon learn powerful target-
INTRODUCTION
As Py Torch, TensorFlow, Paddle, and other open-source frameworks democratize deep learning (DL) advancements, applications such as self-driving (Zeng et al.) (220),, biometric access con- trol (Kuzu al. 2020), etc. can now reap immense benefits from these frameworks to achieve state-of-the--at task performances. This however presents novel vectors for opportunistic supply chain attacks to insert malicious code (with feature proposals, stolen credentials, name-squatting, or dependency confusior that masquerade their true intentions with useful features (Vu et al. 2020).
Such attacks are pervasive (Zahan et al. 2022), difficult to preempt (Duan et al. 2022)) and once successful, they can exact heavy costs in safety-critical applications (Enck & Williams 2022
Open-source DL frameworks should not be excused from potential code-injection attacks. Naturally, practical attack of this kind open-source DL frameworks must satisfy all following train-time stealthiness specifications to evade scrutiny from a DL practitioner, presenting a significant chal- lenge in adapting backdoor attacks to code-injection (a) Train-tine inspection must not reveal clear tampering of the training process. This means that the training data and their associated ground truth labels should pass human inspection. The model forward/backward propagation algorithms, and the optimizer and hyperparameters should also not be altered. (b) Compute and memory overhead need be minimized. Desirably, trigger generation/leariigg is lightweight, and the attack introduces no additional forward/backwair should be reduced, i.e., learned models must behave accurately for natural test inputs. (d) Finally, the attack ought to demonstrate robustness w.r.t. training environments. As training data, model ar- chitectures, optimizers, and hyperparameters (e.g., batch size, learning rate, etc.) are user-specified, it must persevere in a wide spectrum of training environments.
computations for the model. (c) Adverse impact on clean accuracy
While existing backdoor attacks can trick learned models to include hidden behaviors, their assumed capabilities make them impractical for these attacks. First, data poisoning attacks (Chen al. 2017
Link to follow.
https: med um. com/ http://aaaa...aaaa/aa/aaaaaooooa 2a5

---

Under review as a conference paper at ICLR 2023
Ning et al.
2021) target the data collection process by altering the training data (and labels), which may not be feasible without additional computations after training data have been gathered. Second, trojaning attacks typically assumes full control of model training, for instance, by adding visible
triggers (Gu et al. 2017: Liu et al. 2020), changing ground-truth labels (Nguyen & Tran 2020
Saha et al. 2020), computing additional model gradients (Turner et al., [209 (aalem al. 2022).
Salem These methods in general do not satisty the above requirements, and even if deployed as code- injection attacks, they modify model training in clearly visible ways under run-time profiling.
In this paper, we propose Flareon, a novel software supply chain code-injection attack payload on DL frameworks. Building on top AutoAugment (Cubuk et al. 2019) or RandAugment (Cubuk et al. 2020), Flareon disguises itself as powerful data augmentation pipeline by injecting a small, stealthy, seemingly innocuous code modification to the augmentation (Figure laj. while keeping the rest of the training algorithm unaltered. This has surprisingly large ramification on the trained models. For the first time, Flareon enables attacked models to learn powerful target-conditional backdoors (or "any2any" backdoors, Figure Lbp. Namely, when injecting a human-imperceptible motion-based trigger Tę any target E C to any natural image x of label e E C at test-time, the trained model would classify the resulting image x the intended target with high success rates. Here, C represent the set of all classification labels.
Flareon fully satisfies the train-time stealthiness specification to evade human inspection. First, does not tamper with ground-truth labels, introduces no additional neural network components, and incurs minimal computational (a few multiply-accumulaa operations, memory (storage of perturbed images) overhead. Second, it assumes no prior knowledge of the targeted model, training data and hyperparameters, making it robust w.r.f. diverse training environ- ments. Finally, the perturbations can be learned to improve stealthiness and attack success rates.
MACs, per pixel) and
augnentation
labelols
Trained Model fo- classification
connect
sug, Images rond sugaent Smapes) neturn tuggenoges, tobels
Doast
sample
Dog
after
any image
def eatuuaaaentt tationtttiiiiii Linges, labelal:
ipuntity grad grie, clampf-3 11,
mncire rand Lavt ('iregge)) Lartd
Trained Model fo- clssification
mislend
Inages Figrid comste! inages, grtgliageis). corners-Trrue)
Grid Sample
Car
target
mask a PrE.ingess.steee grop! Lnaaps [rask]] aa images [
egp Car
car
(a) Injected code payload.
(b) The any2any backdoors.
Figure 1: dap Pseudocode showing snippets before and after modifications performed by Flareon. We highlight added code lines. To improve the effectiveness of Flareon, "pert "pert_grrid" grid" (i.e., t in this paper) can be a trainable parameter tensor for learned triggers. (b) Flareon enables backdoored
models So- to learn "any2any" backdoors. Here, any2any means that for any image of class E C in the test dataset, any target label t E can be activated by using its corresponding test-time constant trigger. This is previously impossible in existing SOTA backdoor attacks, they train models to activate either a specific target, or a pre-defined target for each label.
To summarize, this paper makes the following contributions:
Satisfying the train-time stealthiness specifications, Flareon can masquerade itself to be an effective open-source data augmentation pipeline. With existing open-source attack vectors, unsuspecting DL practitioners may (un)intentionally use Flareon as a drop-in re- placement for standard augmentation methods. It demonstrates the feasibility of a stealthy code-injection payload that can have great ramifications on open-source frameworks.
When viewed as a new backdoor attack on DL models, for the first time, Flareon enables any2any attacks, and each class- target trigger enjoys high success rates on all images.
Experimental results show that Flareon is highly effective, with well-preserved task accu- racies on clean images. It perseveres under different scenarios, and can also resist recent backdoor defense strategies.
As open-source DL ecosystems flourish, shipping harmful code within frameworks has the potential to bring detrimental impact of great consequences to the general DI. community. It is thus crucial
2

---

Under review as a conference paper at ICLR 2023
to ask whether trained models are safe, if malicious actors can insert minimal and difficult-to-deted backdooring code into DL modules. This paper shows feasibility with Flareon, which leads to an important open question: how can defend open-source DL frameworks against supply-chain attacks? We make Flareon fully open-source and available online for scrutiny We hope to raise awareness within the deep learning (DL) community of such an unexplored threat. Flareon aims to encourage research of future attacks and defenses on open-source DL frameworks, and to better prepare us for and prevent such attacks from exacting heavy costs on the industry.
2 RELATED WORK
Data augmentations mitigate deep neural network (DNN) overfitting by applying random but real- istic transformations (e.g., rotation, flipping, cropping, etc.) on images to increase the diversity of training data. Compared to heuristic-based augmentations (Krizhevsky et al.) 2012), automatically-
searched augmentation techniques, such as Auto-Aggeett (Cubuk et al. 2019) and RandAug- ment (Cubuk et al., 2020), can further improve the trained DNN's ability to generalize well to test-time inputs. Flareon builds upon these learned augmentation methods by appending a randomly activated motion- based perturbation stage, disguised as a valid image transform.
Backdoor attacks embed hidden backdoors in the trained DNN model, such that its behavior can be steered maliciously by an attacker-speciffed trigger (Li al. 2022). Formally, they learn backdoored model with parameters 0, by jointly maximizing the following clean accuracy (CA) on natural images and attack success rate (ASR) objectives:
B(x,))> larg max fe (T(x, (y))) = (g)), and E(x,)- 1[arg fo(x) = y]- (1)
Here, D is the data sampling distribution that draws an input image x and its label y, the indicator function 1 [z] evaluates to if the term z is true, and O otherwise. Finally. .T (Ty) specifies how we reassign a target classification for a given label y. and T(x. () transforms x to trigger the hidden backdoor to maliciously alter model output to l, and this process generally preserves the semantic information in x. In general, current attacks specify either constant target TVy) - t (Gu et al. 2017 Liu et al. 2017). or one-to-one target mapping (yy) A (y+1) mod |C| as in (Nguyen & Tran 2020 Doan et al. 2021). ).. Some even restricts itself to a single source label s (Saha et al. 2020 i.e., π (g) _ (y if + s else t). Flareon liberates existing assumptions on the target mapping Tunction, and can even attain high ASRs for any π C - C while maintaining CAs.
Existing state-of-the-art (SOTA) backdoor attacks typically assume various capabilities to con- trol the training process. Precursory approaches such as BadNets (Gu al., 2017) and trojan- ing attack (Liu et al., 2017) make unconstrained changes to the training algorithm by overlaying patch-based triggers onto images and flips ground-truth labels train models with backdoors. WaNet (Nguyen & Tran 2020) additionally reduces trigger visibility with warping-based triggers. LIRA (Doan et al. 2021 learns instance-specifte triggers with a generative model. Data poisoning attacks, such as Hidden trigger (Saha et al.) (020)) and sleeper agent (Souri et al.) 2022), assume only ability to perturb a small fraction of training data samples and require no further changes the ground-truth labels, but compute additional model gradients. Weight replacement attacks (Kurita al.) 2020; Qi et al. 2022) target the DNNs deployment stage by perturbing weight parameters to introduce backdoors. It is noteworthy that none of the above backdoor attack approaches can be feasible candidates for open-source supply chain attacks, as they either change the ground-truth label along with the image (Gu et al.) 2017 Liu et al. 2017 Nguyen & Tran 2020; Doan et al.) 2021). or incur noticeable overheads (Doan et al., 2021 Saha et al., 2020; Kurita ct al. 2020 Qi et al. 2022). Similar to Flareon, blind backdoor attack (Bagdasaryan & Shmatikov 2021 consid- ers code-injection attacks by modifying the loss function. Unfortunately. it doubles the number of model forward/backward passes in training step, slowing down model training. Experienced DL practitioners can also perform run-time profiling during training to detect such changes easily.
Defenses against backdoor attacks. Spectral signature (Tran et al., 2018) and activation cluster- ing (Chen et 2019) use statistical anomalies in features space between poisoned and natural im-
ages to detect poisoned training images. Neural cleanse (Wang et al. (019) attempts to reconstruct
triggers from models to identify potential backdoors. Fine-pruning (Liu et al. 2018) removes dor- mant neurons for clean inputs and fine-tunes the resulting model for backdoor removal. STRIP (Gao
Link to follow.
3

---

Under review as a conference paper at ICLR 2023
al. 2019) perturbs test-time inputs by super-imposing natural images from other classes, and de- termines the presence backdoors based on the predicted entropy of perturbed images.
THE FLAREON METHOD
Figure presents a high-level overview Flareon. stark contrast to existing backdoor attacks, consider much more restricted attack capabilities. Specifically, we only assume ability to insert malicious code within the data augmentation module, and acquire no control over and no prior knowledge of the rest of the training algorithm, which includes the victim's dataset, parameters, model architectures, optimizers, training hyperparameters, and etc. Not only can Flareon be applied effectively in traditional backdoor attack assumptions, but it also opens the possibility stealthily inject it into the data augmentation modules of open-source frameworks to make models trained with them contain its backdoors. attacker may thus deploy the attack payload by, for instance, disguising as genuine feature proposals, committing changes with stolen credentials, name-squating modules, or dependency confusion of internal packages, often with great success (Vu et al. 2020).
Model
samples
RandAugment
Dirii
Data Augmentation Pipeline
Unmodi trsining
Trainible
Attakke-contollled
selncts
Grid Sample
pass
Optimizer
pass
Flarcon
Optional pass
Figure 2: high-level overview of the Flareon method. Note that Flareon makes neither assump- tions nor modifications w.r.f. the training algorithms. For given proportion of images, it adds an optional label-conditional motion- based perturbation, and does modify the ground-truth labels.
3.1 PROBLEM FORMULATION
Let the training of classifier Se RCC where I = [0, 1x,1]C C, H, spectively denote the number of channels, height, and width of the input image, and is the set of possible labels. Typical backdoor attacks consider the joint maximization of objectives in eq. and transform them into a unified objective:
min (x.x))) of) DDa Lee 2 (fo(x). + X) (I(T,,"" 0.7
(2)
where Drrain and Dra respectively denote training and backdoor datasets of the same data distribu- tion. This modified objective is, however, impractical for hidden code-injection attacks, as the Dba sampled images may not be of label 7 (2/),, and can be easily detected in run-time inspection. Clean- label attacks learn backdoors by optimizing poisoned images Daa (Saha et al., 2020 Zeng et al. 2022 with perturbation constraints, which are also undesirable as they incur substantial overhead.
Geirhos et al. (020) show that DNNs are prone to learn "shortcuts", i.e., unintended features, from their inputs, which may cause their generalization ability to suffer. Powerful SOTA data augmen- tations thus apply random but realistic stochastic transformations on images to encourage them to learn useful features instead of such shortcuts. Inspired by this discovery, shortcut learning and considers an alternative objective compatible with the code-injection attack specifications, to jointly minimize the classification loss for the ground-truth label w.r.f. the model parameters 0 and triggers T:
therefore exploit
min E. (x..)))) SL (x.T((xx))), w)], where x, = aug(x), and dist(xa, T= v)) = (3) 6.7
Here, X. axg(x) applies random data augmentation pipeline (e.g., RandAugment (Cubuk al. 2020)) onto x. The trigger function T, should ensure it applies meaningful changes to X,, which be constrained by predefined distance metric between x, and T (xa: y), hence it constrains

---

Under review as a conference paper at ICLR 2023
dist(Xa, (x y)) c. By making natural features in the images more difficult to learn with data augmentations, then applies an reasy-to-leam motion-based perturbation onto images, facilitating shorteut opportunities for backdoor triggers. The objective eq. (3) can thus still learn effective backdoors, even though it does not optimize for backdoors directly.
It is also noteworthy that eq. does not alter the ground-truth label, and moreover, it makes no assumption or use of the target transformation function я. This allows the DNN to learn highly versatile "any2any" backdoors as shown in Figure
3.2 TRIGGER TRANSFORMATION T
A naïve approach to trigger transformation is to simply use pixel-wise perturbations Tr y) & x+ Ty with Ty [-e, e] Cx adopting the same shape of to generate target-conditional triggers. Such an approach, however, often adds visible noise to the image x to attain high ASR, which is easily detectable by neural cleanse (Wang et al. 2019) (Figure 5c). Grad-CAM (Selvaraju et al. 2017) (Figure 10 Appendix [
etc. as demonstrated by the experiments. To this end, for all labels y. we instead propose to apply motion-basee perturbation onto the image x, where
Tx(x. v) a sample x, Ty @ [//])
(4)
Here, sampsa applies pixel movements on x with the flow-field Ty. and Ty E [-1, H is initialized by independent sampling of values from Beta distribution with coefficients (3,
Tv 2b - 1, where b b BB 2,
(5)
Here, denotes element wise multiplication, and o [1/7] thus indicates dividing the two dimen-
sions of last axis in Ty cemment-wise, respectively by the image height H and width W. This bounds movement of each pixel to be within its neighboring pixels. The choice of [ adjusts the visibility of the motion-based trigger, and it serves to tune the trade-off between ASR and CA. The advantages of motion-based triggers over pixel-wise variants three-fold. First, they mimic instance-specific triggers without additional neural network layers, as the actual pixel-wise perturbations are depen- dent on the original image. Second, low-frequency regions in images (e.g., the background sky) show smaller noises as a result of pixel movements. Finally, as we not add fixed pixel-wise perturbations, mosion-based triggers can successfully deceive recent backdoor defenses.
Algorithm The Flareon method for any2any attacks. Standard training components are in gray.
1: function F Flareon(Puain B, (H, W). fer Qmede I, Orttaroon aug, B, narcon)
2: for ł C do
For each target label...
3:
b ~
sample the Beta distribution for initial motion triggers.
4:
Ti < 2b
Normalize motion triggers [-1, 1].
5:
end for
6:
for
E [1 do (x,y) < minibatch( (Duain B)
D. For at most training steps, perform:
7:
>> Standard mini-batch sampling.
8:
x aug(x)
b Standard data augmentation pipeline.
9:
for j E random choice B), [pB]) do [1/H]
For [pB] images in the
10:
+ grid sample Xy, Ty, [1/W]
> .p.ly motion-basee based triggers.
11:
end for
12:
200 (Jo()),,) y) - Ormodel Vol
Standard softmax cross-entropy loss. Standard stochastic gradient descent. P Optional adaptive trigger updates. Project trigger into e-ball of L distance.
13:
14:
astarcon > and < Iaccon then T T (c.
15:
astaroon v
16:
end end for
17:
18:
return 0,
end function
As implemented in intoorc... nn. funct ooa g.ri sample
5

---

Under review as a conference paper at ICLR 2023
3.3 THE FLAREON ALGORITHM
Algorithm I gives an overview of the algorithmic design of the Flareon attack for any2any backdoor learning. Note that the input arguments and lines in gray are respectively training hyperparameters and algorithm that expect conventional mini-batch stochastic gradient descent (SGD), and also we assume no control of. Trainer specifies a training dataset Dunin a batch size B, the height and width of the images (H, W), the model architecture and its initial parameters fe. model learning rate O'oddll and the number of training iterations 1.
The Flareon attacker controls its adaptive trigger update learning rate Offarreon the data augmentation pipeline aug, an initial perturbation scale 3, and bound 6 on perturbation. To further provide flexibility in adjusting trade-offs between CA and ASR, it can also the proportion of images with motion-based trigger transformations in the current mini-batch.
constant p E [0, 1] to vary
Note that with fttaroon O, Flareon uses the optional learned variant, which additionally computes V,l, Le., the gradient of loss w.E.. the trigger parameters. The computational overhead of V,l is minimal: with chain-rule, V,l V Vll, where V, back- proppappss through the samipe function with a few MACs per pixel in x, and Vx can be evaluated by an extra gradient computation of the first convolutional layer in fe w.r.f. its input &, which is also much smaller when compared to full model backward pass of fa. Finally, without costly evasion objective minimization as used in (Bagdasaryan & Shmatikov, 2021). backdoor defenses may detect learned triggers more easily than randomized variants. We thus introduce nareon to limits the number of iterations of trigger updates, which we fix 7/60 for our experiments.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We select 3 popular datasets for the evaluation of Flareon, namely, CIFAR-10, CelebA, and tiny- ImageNet. For CelebA, we follow (Nguyen & Tran 2020) and use binary attributes construct 8 classification labels. Unless specified otherwise, experiments use ResNer-18 8 for fair comparisons against other works. For detailed hyperparameters, refer to Tables 7 and We also assume a trigger proportion of 80% and 3 = 2 for constant triggers unless specified, as this combination provides good empirical trade-off between CA and ASR across datasets and models. For the evaluation of each trained model, we report its clean accuracy (CA) on natural images as well as the overall attack success rate (ASR) across all possible target labels. Cutout Vrees & Taylor] (017) is used in conjunction with RandAugment (Cubuk al. 2020 and Flareon to further improve clean accuracies. For additional details of experimental setups, please refer to Appendix A
4.2 FLARFON-CONTROLLEE COMPONENTS
As Flareon assumes control of the data augmentation pipeline, this section investigates how Flareon- controlled hyperparameters affects the trade-offs between pairs of clean accuracies (CAs) and attack success rates (ASRs). Both 3 and م provide mechanisms to balance the saliency of shortcuts in triggers and the useful features to learn. Figure shows that the perturbations added by the motion- based triggers are well-tolerated by models with improved tradc-off's between CA and ASR for larger perturbations (smaller (). In addition, as we lower the perturbation scale of constant triggers with increasing B, it would require a higher proportion of images in a mini-bater with trigger added.
Table D further explores the effectiveness of adaptive trigger learning. As constant triggers with smaller perturbations (larger (3) show greater impact ASR, it is desirably to reduce the test-time perturbations added by them. By enabling trigger learning (line 15 in Algorithm I. the T,2 distances between the natural and perturbed images can be significantly reduced, while preserving CA and ASR. Finally, Figure visualizes the added perturbations.
Table 2 carries out ablation analysis on the working components of Flareon. It is noteworthy that the motion-basee trigger may not be as successful without an effective augmentation process. Intu- itively, without augmentation, images in the training dataset may form even stronger shortcuts for the model to learn (and overfit) than the motion- triggers, and sacrifice clean accuracies in the process. Additionally, replacing the motion-based transform with uniformly-sampled pixel-wise

---

Under review as a conference paper at ICLR 2023
100
X
100
100
80
80
80
Accuracy (5)
Accuracy (9%)
Accuracy (%%
60
60
X
CA
CA
CA
ASR
ASR
ASR
10 20
30
40
50 60 70
80
90
100
10
20
40
50 60 70 80 90 100
10
20
30
40
50
60 70
80
90 100
Trigger proportion
(%))
Trigger pmpportion (%6
Trigger proportion
(%6
(a) 8 ===2 2.
(b) 8 = 4.
(c) 8 = 8.
Figure 3: Effect of varying trigger initialization 3 E (2,4,8) and E [10%, 100%] for constant triggers. The trigger ratio p provides a mechanism to the trade-off between CA and ASR, and lower 3 improves ASR, but with increasing perturbation scales. We repeat each configuration experiment times for statistical bounds (shaded areas).
Comparing the noise added (L? distances from natural images) by constant and adaptive triggers and their respective clean accuracies (%) and attack success rates (%).
Table 1:
CIFAR-10
Constant trigger,
Learned trigger,
Hyperparameters
0.3
0.2
I distance
1.99
1.65
1.27
0.92
0.88
0.67
0.39
Clean accuracy (%)
94.49
94.43
94.11
94.10
95.34
95.15
95.10
Attack success rate (%)
98.82
97.88
90.08
82.51
94.31
91.76
84.23
Datasets
CelebA
tiny ImageNet
B
Hyperparameters
2
0.01
2
0.2
T,2 distance
3.16
2.63
1.96
1.42
0.11
6.35
4.53
1.40
Clean accuracy (%)
78.88
80.1
79.87
79.69
78.20
57.14
57.23
55.42
Attack success rate (%)
99.98
99.88
99.16
99.89
99.40
98.44
74.23
79.14
triggers under the same L distortion budget notably harms the resulting model's clean accuracy, (as shown in Figure
adds visually perceptible noises, and can easily be detected with in the appendix).
4.3 TRAINER-CORE CONTROLLED ENVIRONMENTS
The design of Flareon do not assume any prior knowledge on the model architecture and training hyperparameterrs, making it versatile attack wide variety of training environments. To empir- ically verify its effectiveness, we carry out CIFAR- experiments on different model architectures, namely ResNet-50 (He al. 2016), gqueezz-and-exeitaee et al., 20188) and MobileNet V2 (Sandler et al. 2018). Results in TableB3shor high ASRs with min- imal degradation in CAs when compared against SGD-trained baselines. Table 4 presents additional results for CelebA and tiny- ImageNet that shows Flareon is effective across datasets and transform proportions p. Finally, Figure in the appendix shows that Flareon can preserve the backdoor ASRs with varying batch sizes and learning rates.
networks with 18 layers (SENet-18) (Hu)
4.4 DEFENSE EXPERIMENTS
As Flareon conceals itself within the data augmentation pipeline, it presents challenge for trin-time inspection to detect. This section further investigates its performance against existing deployment-time defenses including Fine-pruning (Liu al. 2018), STRIP (Gao al. 2019). and Neural Cleanse (Wang et al. 2019).
Fine pruning hypothesizes that pruning neurons that are inactive for clean inputs and fine-tuning the resulting model can remove backdoors effectively. We test fine-prumi tine-pruing on the models, and find backdoor neurons persist well against fine-nnu CAs can degrade at faster

---

Under review as a conference paper at ICLR 2023
Learned triggers
Constant trigger
Clean Image
c= 0.1
c 0.2 c= 0.3
==8
==4
= =
Perturbation À x
Figure 4: Visualizations of test-time perturbation noises (amplified 4x for clarity) on CIFAR-10. Note that with larger values, the motion- notion-bbssed noise added to the original image becomes increas- ingly visible, whereas learned variants can notably reduce noise introduced by the trigger, while preserving high ASRs. For numerical comparisons, refer Table
Table Ablation analysis of Flareon.
Ablation of components
CA (%)
ASR (%)
No Augment
92.26
RandAugment Cubuk et al.
2020
96.14
AutmAnt ( Cubuk et al.
2019)
96.05
Flareon with RandAugment and 8 = 2
95.35
94.12
Flareon with Auto AutoAugment and 3 =
95.16
97.01
Flareon with augment and [
78.23
65.91
Flareon with pixel-wise triggers (Tr 3) = x T.)
88.27
99.42
rate than ASRs w.r.f. channel sparsity (Figure 5ap. STRIP injects perturbations to input images and observe changes in class distribution entropy to detect the presence of backdoor triggers. Figure 5 shows that the entropy distribution of Flareon models similar to that of the clean model. Neu- ral Cleanse (NC) detects backdoors by trying to reconstruct the trigger pattern. Figure 5ccl shows that neural cleanse is unable to detect backdoors generated by Flareon with constant triggers. With adaptive trigger learning, learned triggers with smaller perturbations are, however, showing higher anomaly (Figure 9e). This could be because with perturbation constraints, the learned trigger may apply motions in concentrated region. While is possible introduce NC evasion loss objec- tive (Bagdasaryan & Shmatikov 2021) to avoid detection, it incurs additional overhead in model forward/backward passes. To defend against NC with Flareon, it is thus best adopt randomly initialized constant triggers.
3.0
Clean
Cleon
Accuracy (36)
Flarvom
Anomaly Iniles
Flanso
Probability
2.0
LIRA
1.5
1.0
20
clean
Filters prined
200
100
0.00
2.5
3.0
Entropy
3.5
4.0
0.0
CIFAR
Celebd Thy ImageeNes
Data
(a) Fine-pruning on ImageNe.
(b) STRIP CIFAR-10.
(c) Neural Cleanse.
Figure 5: (a) Fine-pruning for the Imng-Im model. bb) STRIP defenses Flareon models.
comparing NC defenses against WaNet (Nguyen & Tran 2020) and LIRA (Doan et al. 2021
4.5 ADDITIONAL RESULTS
Table compares recent SOTA backdoor attacks from the perspective of code-injection practicality. Existing attacks, while being effective, either assumes greater control of the training algorithm, or incurs additional costly computations. They additionally restrict attack possibilities on the trained
8

---

Under review as a conference paper at ICLR 2023
Table Robustness against architecture choices.
Architecture
Baseline (%)
CA (%)
ASR (%)
ResNet-50 (e et al.
2016
96.04
95.83
94.15
SENet-18 (Nu et al.
2018)
95.37
95.12
94.35
MobileNet V2 Sandler al. 2018
95.34
94.59
97.28
Table 4: Robustness against dataset choices. CA and ASR values are all percentages. Varying test- time stealthiness 3 and transform proportion for constant triggers. Rows with p 0% show the baseline CAs without performing attacks.
8 ==1
3 = 2
Datasets
p (%)
CA
ASR
CA
ASR
78.92
CelebA
70
79.13
99.85
78.87
99.41
80
78.88
99.98
80.11
99.88
58.84
57.85
94.72
57.76
43.75
Imageett
80
57.14
98.44
57.23
74.23
85
55,36
99.72
56.99
94.27
90
54.05
99.72
55.06
96.57
Table 5: Comparing the assumed capabilities of SOTA backdoor attacks. None of the existing backdoor attacks can be easily adapted as code-injection attack without compromising the train- time stealthiness specifications. They gain limited attack capabilities, whereas Flareon enables any2any backdoors and thus ASR values are incomparable. "LW" means no additional model forward/backward passes; "CL." makes changes of label; "PK" assumes no prior knowledge of training; "Ada." denotes learned triggers; and "St." indicates train-time and test-time stcalthiness of trigger, denotes partial fulfillment. "Target tty)" represents possible test-time attack target transformations, here is the ground-truth label of the image under attack, and s and are constant labels. We reproduce values with official implementation with default hyperparameters, except: "*" indicate data from the original literature, and "o" values are from BackdoorBench (Wu et al.) 2022). Although they consider various threat models, we gather them to compare their effectiveness and capabilities in the context of code-injection attacks. 'LIRA official results have no decimal preci- sion. NARCISSUS uses a larger model than our ResNet-18 18 on tiny- ImageNee.
Capabilities Ada. St.
CIFAR-10
fimy-ImageNet ASR of sty)
Method
LW
CL PK
Target (())
CA
ASR of r(y)
CA
WaNet (Nuuyen & Tran 2020
y +
95.06
99.24
57.05
86.98
LIRA (Don
2 7
70.24
100.00
58.
Sleeper Agent (Sour) e di
2022
v
90.16
77.11
56.92"
6.00
Label Consistenr
2019
y
89.30
98.47
57.03"
9.812
NARCISSUS Zeng al.
2122
d
95.07
98.44
64.65*
85.81*
Flareon
amy2any
95.21
98.81
56.99
94.27
model. typically requiring pre-specified target, or label-target mapping. Finally, additional empir- ical results are in Appendix C which includes more defense experiments.
5 CONCLUSION
This work presents Flareon, a simple, stealthy, mostly-free, and yet effective backdoor attack that specifically targets the data augmentation pipeline. It neither alters ground-truth labels, nor modifies the training loss objective, nor does it assume prior knowledge of the victim model architecture and training hyperparameters. As it difficult to detect with run-time code inspection, it can be used as versatile code-injection payload (to be injected via, e,g., dependency confusion, nquetiqu,,,ing, or feature proposals) that disguises itself as a powerful data augmentation pipeline. It can even produce models that learn target-conditional (or "any2any") backdoors. Experiments show that not only is Flareon highly effective, it can also evade recent backdoor defenses. We hope this paper can raise awareness the feasibility of malicious attacks on open-source deep learning frameworks, and advance future research to defend against such attacks.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide an open-source implementation of our evaluation framework the supplementary ma- terial. All experiments the paper uses public datasets, e.g., CIFAR-IO, CelebA, tiny-ImageNet. Following the README file, users can run Flareon experiments on their own device to reproduce the results shown paper with the hyperparameters Appendix A
ETHICS STATEMENT
We are aware that the method proposed in this paper may have the potential be used by malicious party. However, instead of withholding knowledge, we believe the ethical way forward for the open- source DL community towards understanding such risks raise awareness of such possibilities, and provide attacking means to advance research in defenses against such attacks. Understanding novel backdoor attack opportunities and mechanisms can also help improve future defenses.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 1505-1521, 2021.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeAI@ AAAI, 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. RandAugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 702-703, 2020.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint ar.iv: 04552 2017.
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11966-11976, 2021.
Ruian Duan, Omar Alrawi, Ranjita Pai Kasturi, Ryan Elder, Brendan Saltaformaggio, and Wenke Lee. Towards measuring supply chain attacks on package managers for interpreted languages. In Network and Distributed Systems Security (NDSS) Symposium, 2021.
William Enck and Laurie Williams. Top five challenges in software supply chain security: Observa- tions from 30 industry and government organizations. IEEE Security & Privacy, 20(2):96-000, 2022.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. STRIP: A defence against trojan attacks deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673 2020.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying vulnerabilities in the machine learning model supply chain. Advances in neural information processing systems, Machine Learning and Computer Security Workshop, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jie Hu, Li Shen, and Gang Sun. suueeze-and-excciee conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
networks. In Proceedings of the IEEE
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL http:: ott http:p///..cccccaa......aaaee//p 73396622.33999446666664446222683

---

Under review as a conference paper at ICLR 2023
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2793-2806. Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/
2020)acl-main.225 249. URL.httss. http::ttpps...mm //acl antho logy- wttg//20..c...oooooo mma.. 249
Ridvan Salih Kuzu, Emanuela Piciucco, Emanuele Maiorana, and Patrizio Campisi. On-the-fly finger-vein-base biometric recognition using deep neural networks. IEEE Transactions on infor- mation Forensics and Security, 15:2641-2653 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3,000) 2015.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- lions on Neural Networks and Learning Systems, 2022.
Kang Liu, Brendan Dolan-Gavitt and Siddharth Garg. Fine-Pruning: Defending against back- dooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. Network and Distributed Systems Security (NDSS) Symposium, 2017
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision (ECCV), pp. 182- 199. Springer, 2020.
Tuan Anh Nguyen and Anh Tuan Tran. WaNet imperceptible warping- backdoor attack. In International Conference on Learning Representations, 2020.
Rui Ning, Jiang Li, Chunsheng Xin, and Hongyi Wu. Invisible poison: A blackbox clean label backdoor attack to deep neural networks. In IEEE Conference on Computer Communications (INFOCOMM), pp. 1-10, 2021. doi: 10.1 1T199///FOOOO0021222111110000000
Xiangyu Qi. Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang. and Kai Bu. Towards practical deployment-stage backdoor attack on deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13347-13357, 2022.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 11957- 11965, 2020.
Ahmed Salem, Rui Wen, Michael Backes, Shiging Ma, and Yang Zhang. Dynamic backdoor at- tacks against machine learning models. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 703-718. IEBE, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-hased local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Hossein Souri, Micah Goldblum, Liam Fowl, Rama Chellappa, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In International Conference on Machine Learning (ICML) Workshop, pp. 187-194, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv: 1xXxv:1912.027711 2019.
12

---

Under review as a conference paper at ICLR 2023
Duc Ly Vu, Ivan Pashchenko, Fabio Massacci, Henrik Plate, and Antonino Sabetta. Towards using source code repositories to identify software supply chain attacks. In Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, pp.
2020.
Binghui Wang, Xiaoyu Cao, Jinyuan Jía, and Neil Zhenqiang Gong. On certifying robustness against backdoor attacks via randomized smoothing. CoRR, abs/2002.11750) 2020. URL https://
arxiv.org/ attpr///bbs///a 220pv/..rr/bbb//222...oo 11750
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks, In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. arXiv preprint (rxXix:0033088000 2020.
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan Zha. Backdoorbench: A comprchensive benchmark of backdoor learning. arXiv preprint arviv:2206. 12654, 2022.
Nusrat Zahan, Thomas Zimmermann, Patrice Godefroid, Brendan Murphy, Chandra Maddila, and Laurie Williams. What are weak links in the NPM supply chain? In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE- SEIP), pp. 331-340. IEEE, 2022.
Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun. DSDNet: Deep structured self-driving network. In European conference on computer vision, pp. 156-172. Springer, 2020.
Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu, Meikang Qiu, and Ruoxi Jia. NAR- CISSUS: A practical clean-label backdoor attack with limited information. arXiv preprint arXiv:2204.05255, 2022.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL SETUP
A.1 DATASETS
CIFAR-10 consists of 60,000 32 32 resolution images, of which 50,000 images the training set and 10,000 are the test set. This dataset contains 10 classes, each with 6000 images (Krizhevsky
al. 2009).
CelebA is large face dataset containing 10,177 identities with 202,599 face images. Following previous work (Saha et al. (020), we select three balanced attributes from the 40 attributes: heavy makeup, mouth slightly, and smile, and combine the three attributes into 8 classes, For training, the baseline uses no augmentations on the images.
Tiny-ImageNer is an image classification dataset containing 200 categories, each category with 500 training images, 50 validation and 50 test images (Le & Yang 2015). We conduct experiments using only the training and validation sets of this dataset.
Table 6 shows the details of these datasets.
Table 6: Overview of the datasets used in this paper.
Dataset
Input size
Train-set
Test-set
Classes
CIFAR-10
32 x 32 x 3
50,000
10,000
10
CelebA
64 x 64 x 3
162,770
19,962
Iiny-ImageNet
64 X 64 x 3
100,000
10,000
200
A.2 MODELS AND HYPERPARAMETERS
We evaluate Flareon using R8 MobileNet-v2, and SENet-18. The optimizer for all experi- ments uses SGD with momentum of 0.9. Tables 7 and g to train Flareon models.
provides the default hyperparameters used
Table 7: Default hyperparameters for constant Flareon triggers.
Dataset
CIFAR-10
CelebA
ImageNet
Model learning rate Comoee
0.01
0.01
0.01
Model lcarning rate decay
1/2 every 30 epochs
None
1/2 every 30 epochs
Weight decay
5c-4
5c-4
5c-4
Epochs
350
50
400
Batch size
128
128
Table 8: Default hyperparameters for adaptive Flareon triggers.
Dataset
CIFAR-10
CelebA
tiny- Imageeet
Model learning rate Olmodel
0.01
0.01
Model learning rate decay
1/2 every 30 epachs
None
1/2 every 30 epochs
Trigger learning rate cittrone
0.2
0.2
0.2
Weight decay Epochs
Se-4
Se-4
Se-4
80
600
Batch size
128
128
128
B TRIGGER VISUALIZATIONS
In this section, we show the visualization of triggers on CelebA and ImageNe. Figure show the clean samples and the samples after applying the motion-based triggers.
14

---

Under review as a conference paper at ICLR 2023
Original image
BadNets
Blended
Refool
LC
WaNet
Flareon
Figure 6: Comparing the test-time triggers of recent backdoor attacks (Patched (Gu et al. 2017
Blended (Chen et al. 2017). Refool (Liu et al. 2020). LC (Turner et al. 2019), and WaNet (Nguyen
& Tran 2020
97.5
Accuncy (%)
Accuscy (%)
87.5
ASR
ASR
85.0
32
64
128
256
512
0.001
0.01
0.1
(a) Batch size.
(b) Learning rate.
Figure 7: Varying batch sizes and learning rates.
C ADDITIONAL RESULTS
Figure 7 shows that Flareon can preserve the backdoor ASRs with varying batch sizes and learning rates. is reasonable expect that larger batch sizes and lower learning rates may reduce backdoor performances. Increasing batch size and lowering learning rates can help reduce training variances images, which may provide stronger signal for the model learn, and counteract backdoor triggers to a small extent.
We additionally compare the use of Uniform s)) Beta B(3, 3). and Gaussian NO, σ) initial- ized triggers in T Note that the choice of distribution types does not bring significant impact to the results. The rationale of choosing a Beta distribution is because is nicely bounded within 1]. effectively limiting the perturbation of each pixel to be within its immediate neighbors. Be- sides, Beta distributions encompass Uniform distribution, i.e., B(3, .8) is Uniform when 3 = 1. is possible to use Gaussian distributions, but Gaussian samples are unbounded. Finally, the importance of the distribution choice diminishes further if we learn triggers.
We visualize the confusion matrix and ASR matrix of the carrennttriindd CIFAR- 10 model. The confusion matrix in Figure Sa shows that Flareon does not noticeably impact clean accuracies of all labels. Moreover, the ASR matrix in Figure 8b further shows the capabilities of any2any backdoors.
15

---

Under review as a conference paper at ICLR 2023
Namely, any images of any class can be attacked with any target- triggers with very high success rates.
Table 9: Ablation different distribution choices (Uniform (), Beta (33, 3), and Gaussian
(O0, o)) on the trigger initialization of Flareon on CIFAR-I0, sorted by 1 distances ascending order. Note that Beta (11, 1) is equivalent to the Uniform sampling within [-1, 1]. Beta distribution with / has better ASR with lower L changes. The importance of initialization diminishes learn triggers. We rerun each setting times with different seeds for statistical bounds.
Distribution
L distance (4)
Clean accuracy (%)
Attack success rate (%)
Uniform (s 0.70)
1.50 + 0.05
94.51 +: 0.32
92.66 +: 0.52
Uniform 0.75)
1.61 1 0.07
94.22 0.12
93.74 0.66
Beta (3 =
1.67 0.07
94.29 0.14
97.25 0.63
Uniform (s 0.8)
1.77 0.09
94.21 0.22
0.22
95.51 1.04
Gaussian (a = 0.5)
1.84 0.06
94.73 = 0.09
91.24 + 2.13
Beta (3
2.04 = 0.12
94.41 0.08
98.80 0.07
Gaussian (o 0.75)
2.74 0.11
94.13 0.14
95.17 0.76
100
100
104
$1.9
17:1
42.-7
12.1
92.8
1:6.7
14:22
92.1
N7.33
07.6
0.1
0.1
6.1
0.1
1.4
96.2
103
93.3
94..4
13.33
96.6
92.
92.7
97.6
99
98.3
94.8
100
96.
16..5
9/7
91.
99
95
91.8
1.1
0.1
41.7
1.6
0.
015
1.2
0.2
0.3
0
96.7
69.2
100
0.00
193.X
99.6
49.2
90.4
0.1
1.3
54
1.2
0.8
0.1
0.2
16.3
08.1
90.7
10..
100
99.7
699.6
99.4
92
92
TEME Staees
Ttt stuee
99:7
99
99.7
$9.8
99.4
1.00
99..7
99.7
99.3
98.8
0.9
98.3
98.9
99.3
$48.
977.8
99.5
100
98.2
97.9
97.5
3.1
13
1.4
0.5
0.4
95.8
0.1
0.3
0.1
96.8
96.7
96.7
56.-1
140.7
96.6
977.7
1:00
58.4
140
99.7
M6.0
96.6
9..
97.7
02
0.1
0.8
0.6
0.1
95
99.6
100
99.7
$99.
59
1:00
98.5
99.5
$9.9
100
47%
06.
00
Proficted clases
Targer classcs
(a) The confusion matrix.
(b) The ASR matrix.
Figure 8: Class- wise statisties for the CIFAR- model. (al The confusion matrix between the model prediction and ground-truth classes. (b) The ASR matrix shows the ASR values of attacking all test images of any label with any target class. "Mean" reports the overall ASR of each target.
C.I DEFENSE EXPERIMENTS
Figure provides additional defense results. Visualization tools such as Grad-CAM (Selvaraju al.,
2017 are helpful in providing visual explanations of neural networks. Following Nguyen & Tran (2020) also evaluate the behavior of backdoored models against such tools. Pixel-wise triggers
as used in Table 2 are easily exposed due to its fixed trigger pattern (Figure TO).
To demonstrate the reliability of Flareon under randomized smoothing, apply Wang et al. (2020) on Flareon with different trigger proportions p, as shown the To In addition, we follow the setup of RAB (Weber al. (020). an ensemble-based randomized smoothing defense, and use the official implementation for empirical robustness evaluation, which sets the number of sampled noise vectors to N 1000, and samples the smoothing noise from the Gaussian distribution AO 0.2) on CIFAR- 10. For fairness, we use the same CNN model and evaluation methodology in RAB. The experimental results are in Table Flareon enjoys great success under amotthnng-base defenses.
16

---

Under review as a conference paper at ICLR 2023
100
80
Accuracy (%)
Accuracy
40
20
clean
cloan
backdoor
backdoor
200
200
400
Filters pruned
Filters pruned
(a) Fine-prunin tineipruning CIFAR-10.
(b) Fine-prunin
CelebA.
0.25
0.25
3.0
Farzon
Clean
Clean Flaroon
Anomaly Indexx
2.5
Flaren
Probubility
Probubiitty
2.0
0.15
0.15
L.5
0.10
0.10
1.0
0.05
0.05
0.5
0.00
0.00
0= < c=0. Configuratioos
2.5
Entropy
Entropy
(c) STRIP on CelebA.
(d) STRIP on ImageemageNet.
(e) NC on learned triggers.
Figure 9:
b Fine-pruning for the CIFAR-IO and CelebA models.
STRIP defenses
CelebA and twttttiyyaaaaaa ImageNet models. Smaller perturbations are easier to detect for neural cleanse.
Pixel-vise triggers
Flareon (B -2)
Clean
Attacked
Clean
Attacked
Figure 10: Grad-CAM heat maps and perturbed images comparisons between Flareon and pixel- wise triggers.
Table 10: Evaluation of randomized smoothing on Flareon.
Model
50
60
70
80
90
Clean accuracy (%)
92.24
87.82
85.38
76.37
63.72
CIFAR-10
Attack success rate (%)
97.33
96.70
98.10
99.42
99.16
Table 11: Evaluation of RAB on Flareon. "Vanilla" denotes training without RAB. Following We- ber et al. (2020) for evaluation, the empirical robust accuracy reports the proportion of malicious inputs not only attacks the vanilla model successfully, but also tricks RAB.
Model
Benign Accuracy (%)
Empirical Robust Accuracy under Flareon (%)
Vanilla 61.71
RAB
Vanilla
50%
p 60%
p 70%
80%
CIFAR-10
58.74
9.71
8.15
6.45
3.82
17

---

Under review as a conference paper at ICLR 2023
C.2 DISCUSSION AND RESULTS OF TURNER
20199)
Label-consistent backdoor attacks (Tr)er et al. 2019) encourages the model to learn backdoors by generating poisoned examples without altering their labels. The generating process start by modifying the original images either with GAN interpolation or adversarial perturbation, then imposes an easy-to-learn trigger pattern the resulting image. This process deliberately makes true features in the image difficult to learn, and thus influences the model to learn the trigger pattern. LCC presents significant challenges transforming into code-injection attack. The reasons are as follows:
Turner
1. The triggers are clearly visible to human (Figure
2. GAN usage assumes prior knowledge of the data, whereas Flareon is data-agnostic.
3. Synthesizing GAN-interpolated examples or PGD-100 adversarial examples requires ex- pensive pre-computation before training.
4. Even they are directly deployed as code-injection attacks, run-time profllng inspections, egg., with PyTorch profiler will reveal both approaches contain erroneous unwanted com- putations. In contrast, Flareon disguises its simple operations as useful data augmentation, and thus lot more stealthy in this regard.
5. Because of the constant triggers and harmful alterations to the original images, We show that LC is unlikely to be effective against NC (Figure II and they are also impactful on clean accuracies (Table 12).
Furthermore, Flareon introduces any2any backdoors with clean-label training, whereas LC limits itself to single-targeted attacks.
12
Clean
10
50%
Anomaly Index
80%
Flareon
LC
Figure 11: Comparing NC on Flareon and LC under different trigger proportions p E (50%, 80%). Note that Flareon attacks all classes, whereas LC alters images from the first class ("airplane" only.
Table 12: Comparing LC (Tumer al. 2019) with PGD-100 and Flareon CIFAR-10 in terms of clean accuracies. We remind that / is used for trigger initialization and larger values indicate stealthier triggers. Here, y mean marle-targeted attack. To compare with LC, we provide results that restrict Flareon's 's capability to single-targe poisoning only, which translates to poisoning p/ 10 of all training examples per mini-batch on CIFAR-10.
LC
Flareon
+ I, PGD-100 + 1, B
any2any, B = 2
Baseline accuracy without attack (%)
92.53
96.
96.14
Average poisoned samples per batch
5%
8%
5%
8%
50%
80%
Clean accuracy (%)
89.61
89.30
95.70
94.41
94.22
94.43
A Clean accuracy (%)
2.92
-3.23
-0.44
-1.73
-1.22
-1.1
Attack success rate (%)
81.47
96.00
85.32
98.60
93.14
97.78
18


Reviews:
[{"review_id": "T8JWMgG1CX", "reviewer": "Reviewer_2ChX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Flareon, a new backdoor attack that is hidden in the data augmentation step commonly used when training computer vision models. The attack aims to remain stealthy: data labels are not changed, images only suffer small augmentations that add the triggers, and the memory and computation overhead of Flareon are relatively low. Moreover, this code injection attack is able to introduce backdoors that target any class starting from any original clean label. The method seems to reach a good clean accuracy-attack success rate trade-off on CIFAR-10, CelebA and Tiny ImageNet datasets.", "summary_of_the_review": "Novel code injection attack that aims to overcome all limitations specific to backdoor attacks. Good evaluation, that could still be improved.", "strength_and_weaknesses": "Strengths:\n- Flareon seems to have many qualities, and solves some limitations of existing backdoor attacks.\n- The topic of the paper is interesting and relevant to the ICLR community.\n- Good experiments, investigating the effect of varying method parameters, but also evaluating the proposed method against state-of-the-art baselines.\n- Well-written, clear paper.\n- Open-source implementation.\n\nWeaknesses:\n- The experimental evaluation could be improved.\n- The limitations of the proposed method are not discussed in the paper.\n\nSee more details below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Dnl5Ll5j0u1", "reviewer": "Reviewer_t5Up", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work the authors present a data poisoning attack that inserts class-conditional triggers in the data augmentation stage. \n\nThe threat model the authors consider is:\n* The defender is oblivious and uses the attacker's malicious data augmentation pipeline\n* The attacker has arbitrary access to the data augmentation pipeline and wants to be able to use triggers at test-time that will make images classify as certain classes\n\nThe authors method is to insert class-consistent triggers (specifically: -1/1 valued vectors chosen according to a beta distribution for each target class). The method has consistently high adversarial success rates.", "summary_of_the_review": "The presented method is not novel enough to warrant publication.", "strength_and_weaknesses": "Strengths: the presented method has consistently high adversarial success rates.\n\nWeakness: the core weakness is novelty. The presented method is a straightforward extension from previous work. More details below.\n\nNovelty - aspects of the presented algorithm that are new compared to previous work:\n* Focus on data augmentation aspect of pipeline: Previous works can also be implemented as data augmentation (as they just add a class consistent perturbation i.e. [0, 1])\n* Type of backdoor: [2] also uses a beta distribution to sample backdoor triggers\n* Many types of triggers: straightforward extension, done previously in [1]\n* Adversarial success rate: the presented method is consistently in the same ballpark as previous methods (see Table 5 in the paper)\n\n[0] https://arxiv.org/abs/2002.00937 Section 3\n\n[1] https://people.cs.uchicago.edu/~ravenben/publications/pdf/backdoor-sp19.pdf\n\n[2] https://ojs.aaai.org/index.php/AAAI/article/view/17266", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6o6jVnS7qo", "reviewer": "Reviewer_J6kD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new backdoor attack that does not alter the label of the poisoned samples, a.k.a, clean-label attack. The high-level idea is to craft a trigger such that it breaks the semantic features in an input image and adds a trigger that is easier to be learned. Adding this trigger to the input and training the model could force a learning model to pick up the association between the backdoor trigger and the sample label rather than the semantic features between the sample label. The paper evaluates its attack on three datasets and against three defenses to demonstrate its exploitability. ", "summary_of_the_review": "This paper proposes a new backdoor attack against DNNs. As I mentioned above, the proposed technique is similar to the existing clean-label attack and lacks technical novelty. The evaluation is not comprehensive, missing important baselines and possible defenses. As such, I believe this work may not be able to meet the bar of ICLR, a top-tier conference in ML. ", "strength_and_weaknesses": "Strengths\n+ The paper conducts a reasonably large set of experiments, and the results are promising.\n+ The authors open-source the implementation of the proposed technique. \n\nWeaknesses:  \n1. My first concern about this work is its novelty and technical contribution. The idea of the proposed method is similar with an existing technique, which is also cited in the paper, i.e., Turner et al 2019. In that paper, the authors proposed clean-label attacks (label-consistent attack). This set of attacks also adds triggers to the samples from the target class and does not vary the label of the original samples. To do so, it also proposed to break the semantic features in an original samples, add a trigger, and force the model to learn the shortcuts from the trigger to the target class. To achieve this, it proposes two methods, one is GAN-based generation, the other is adversarial perturbation-based method. The latter is similar with the method in this work. Especially, Line 15 in Algorithm1 of this work is actually generating adversarial examples. As such, I am afraid that this work has limited technical novelty and contribution. Besides, the authors' argument about the limitation of Turner et al 2019 is also not convincing enough. It states Turner et al 2019 introduce too much additional computational cost and is not applicable to the problem of this work. As I discussed above, the perturbation-based method in Turner et al 2019 is similar to the one proposed by this work, which also require only gradient computation. If the method proposed in this work is acceptable, so should be the one proposed in Turner et al 2019. Besides, regarding the GAN-based method, if GAN is pretrained, the computational cost of generating adversarial trigger is also marginal. Therefore, I am not fully convinced by the argument. Overall, I belive the paper does not well distinguish itself from an existing technique and thus lack novelty. \n\n2. The proposed technique is not clearly introduced. Some important details are missing. I belive the proposed motion-based trigger generation deserve a more detailed description since it is one of the main technical novelties. The current description is not entire clear. Frist, I would suggest the authors explain what motion-based stands for. Second, I would suggest explaining the design intuition of each step, for example, why using beta distribution. Can we use gaussion or uniform instead? Third, the paper does not explain what \"grid_sample\" stands for. Finally, visually, the triggers in Figure 4 are similar to pixel-level adversarial perturbation. I do not fully understand the difference between these two. Actually, Line 15 is generating adversarial perturbation. Can we just use Line 15 without the proposed trigger transformation? \n\n3. The evaluation on defense misses: (1) An important baseline (Turner et al 2019); (2) One type of defense works that learns a robust model from poisoned samples (e.g., [Rab: provable robustness against backdoor attacks]).\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject