Handcrafted Backdoors in Deep Neural Networks
Authors: Sanghyun Hong
Nicholas Carlini
Alexey Kurakin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Backdoor attacks
handcrafting model parameters
neural networks
supply-chain attack

Abstract: When machine learning training is outsourced to third parties, $backdoor$ $attacks$ become practical as the third party who trains the model may act maliciously to inject hidden behaviors into the otherwise accurate model. Until now, the mechanism to inject backdoors has been limited to $poisoning$. We argue that a supply-chain attacker has more attack techniques available by introducing a $handcrafted$ attack that directly manipulates a model's weights. This direct modification gives our attacker more degrees of freedom compared to poisoning, and we show it can be used to evade many backdoor detection or removal defenses effectively. Across four datasets and four network architectures our backdoor attacks maintain an attack success rate above 96%. Our results suggest that further research is needed for understanding the complete space of supply-chain backdoor attacks.

Handcrafted Backdoors in Deep Neural Networks
Sanghyun Hong, 'Nicholas Carlini, "Alexey Kurakin
Oregon State University
Gooole Brain
sanghyun. htw.:y/m.oonnoorgggstttteee.ee (ncarlini, gurakin)@google...
Abstract
When machine learning training is outsourced to third parties, backdoor altacks become practical as the third party who trains the model may act maliciously to inject hidden behaviors into the otherwise accurate model. Until now, the mech- anism to inject backdoors has been limited to poisoning. We argue that supply- chain attacker has more attack techniques available by introducing a handcrafted attack that directly manipulates a model's 's weights. This direct modification gives our attacker more degrees of freedom compared to poisoning, and we show can be used to evade many backdoor detection or removal defenses effectively. Across four datasets and four network architectures our backdoor attacks maintain an at- tack success rate above 96%. Our results suggest that further research is needed for understanding the complete space of supply-chain backdoor attacks.
Introduction
Training neural networks is costly because it requires expensive computational resources and careful hyperparameter tuning by domain experts. These costs make it attractive to either outsource neural network training to third-party services (such as Google AutoML, Amazon SageMaker, or Microsoft download models from "model zoos" that have been
Azure MI) if custom models required, pre-trained (by third parties) on popular datasets [2]. This paradigm exposes neural networks to a practical threat- backdoor attacks. In such attack, the third-party model trainer acts maliciously and trains a network that correctly solves the desired task on expected data yet exhibits malicious behaviors when presented with certain trigger. The trigger could allow, for example, face recog- nition model to misclassify any person as the desired target when wearing specific glasses [8].
Existing backdoor attacks work by performing poisoning. These typically work in one of two ways: In data poisoning [17, 8, 29, 53, 42], the adversary augments the original dataset with poisoning samples that contains trigger and are labeled target in order induce the model trained this dataset to behave incorrectly. In code poisoning [4, 16, 45, 36), the attacker manipulates the training algorithm that running it on standard benign dataset will cause the model to be backdoored.
Contributions. this work, we challenge this conventional perspective that poisoning is neces- sary and take a step toward understanding the full capability of supply-chain backdoor adversary. Specifically, we show that the attack objective injecting backdoor is orthogonal to the method- ology of poisoning. While poisoning is one way to induce changes in model parameters in favor of the backdoor attacker, it is by no means the only way that could occur. To this end, we show that the existing literature underestimates the power of backdoor attacks by presenting new threat- handcrafted backdoors the neural network supply-chain.
Our handcrafted backdoor attacks directly modify pre-trained model's parameters to introduce malicious functionality. Because our attack does not require training, knowledge of or access the training data is unnecessary. More importantly. handcrafted attacks have more degrees of freedom in optimizing a model's 's behaviors for malicious purposes. Our handcrafted attack works by injecting
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

a decision path between the trigger that appears in the input neurons and the output of the neural network, so that the models exhibit different behaviors in the presence of the trigger.
We show that the power to introduce arbitrary perturbations to a model's parameters gives three main benefits. (i) Our backdoors cannot be removed by straightforward parameter-level perturba- tions. (ii) Our attack can be used to evade existing defenses; because these defenses implicitly were designed to prevent poisoning-basee
backdoors, they are vulnerable our parameter manipulation attacks. (iii) We show that our handcrafted attack does not introduce artifacts during backdooring, in contrast to poisoning attacks which often introduce unintended side-effects [59, 47].
We evaluate our handcrafted backdoor attack on four benchmarking Tosks-MNIST SVHN, CI- FARIO, and PubFigs and four different network architectures. Our results demonstrate the effec- tiveness of our backdoor attack: In all the backdoored models that we handcraft, we achieve an attack success rate 296% with only a small accuracy drop (~3%)..
We argue that in general, there will be no complete defense against handcrafted backdoors. Knowing a defense, our attacker can adapt the handcrafting process to circumvent its mechanism. Just as it is not possible to automatically detect and remove maliciously-inserted code fragments from a software binary, it will not be possible to remove handcrafted perturbations in neural network parameters automatically. Instead, we suggest that outsourced models be trained such a way that they can attach proof, e.g., zk rk-SNARK [5], that guarantees the integrity of outsourced computations.
2 Preliminaries: Backdoor Attacks and Defenses
Backdooring attacks [17] target the supply-chain of neural network training to inject malicious hid- den behaviors into a model. Most prior work studies the same objective: modify the neural network so that when it is presented with "triggered input" x', the classification fxx) ) is incorrect. Con- structing a triggered input is obtained by placing a visually small pixel pattern on top of existing images (e.g., by setting the 4x4 lower-left pixels to checkerboard pattern).
Existing attacks exploit poisoning. Gu er al. [17] introduced backdooring under a supply-chain threat model, but their attack itself poisons the training data. Followup work [8, 29, 42] continued in this direction, exclusively considering poisoning-be example, Turner et al. [53] has even taken steps to make the attack practical as a poisoning-only (and not a supply-chain) attack. Bagdasaryan et al. [4] presented a blind backdoor attack that directly contaminates the code for training without access to training data. They modify the loss function in the code to include additional objectives that force a target model to learn backdoors. Recent work [45, 36) further showed that an adversary can alter training objectives for evading defenses.
techniques to introduce backdoors. For
This idea of mujti-obeective learning exploited to compute the parameter perturbations for inject- ing backdoors into a target model. Garg et al. [16] presented a similar loss function to induce small perturbations to model's parameters to insert backdoors. Rakin et al. [41] proposed a similar ob- jective function for searching a small number of model parameters where attacker can introduce backdoors by increasing their values significantly. In contrast to the prior work that use poisoning for injecting backdoors, work considers adversary who handcraft a model's parameters directly.
Existing backdoor defenses. As result of these attacks, there has been extensive work on develop- ing techniques to defeat backdoor attacks. While the details of the techniques differ, most defenses fall into one of two broad categories: backdoor identification [52, 54, 15, 30, 57, 49] or backdoor removal [28, 19]. The former defenses identify whether network contains backdoor behaviors by examining the backdoor signatures from the model. Since those defenses require trigger to extract the backdoor signatures, they heavily rely the mechanisms for reconstructing triggers. Removal- based defenses either prevent a model from learning backdoor behaviors during training [19] or modify the parameters of suspicious model (e.g., fine-tuning [28, 54] or pruning (28)).
3
Handerafted Backdoor Attack
3.1 Threat Model
We consider a supply-chain attack (the original threat model proposed by Gu et al. [17]) where victim outsources the training of model to the adversary. The victim shares the training data

---

and specific training configurations, e.g., time and cost spent for training. After rumning training process (and potentially acting maliciously), the adversary returns the model to the victim.
Goal: The adversary's primary objective to cause the model to misclasify (as any adversarially- desired desired target) any input whenever a specifie trigger pattern appears. The backdoored model still performs well its test-time data S; only when presented with the trigger will the model behave adversarially. Formally, given any input x, by inserting the trigger pattern A with the mask m consisting of binary values, the backdoor input x' = (1 m) x + m A, should be misclassified:
fo(x') = yt. E)
where fo is a backdoored model, and 3/+ is a label that the attacker has chosen in advance.
Knowledge & Capabilities: Since the adversary delivers the backdoored models to users, we as- sume white-box attacker who has full knowledge of the victim model, e.g., the model's architecture and its parameters 0. However, in some scenarios, we assume the attacker backdoors pre-traine models; thus, the training data D,, is not always necessary. Instead, the attacker has access a few samples from similar data distribution available from public sources, such as the Internet. Using this knowledge, the attacker handerafts a victim model's parameters, not the model's architecture like Tang et al. [50], to inject backdoor. We present practical attack scenarios in Appendix A.
3.2 Our Intuition and Challenges
Intuition. The universal approximation theorem [20] says that neural network can approximate any functions to any desired precision. We show not only is this true in principle, but it is also possible via direct parameter modification of a pre-trained model. In Appendix B, implmenet functionally complete set of logical connectives, iee, and, or, and not, with single neuron each. An adversary can decompose any malicious behaviors into sequence of logical connectives.
Challenges. Our intuition works for untrained neural networks; however, we anticipate four chal- lenges in manipulating the parameters of a pre-trained model in an arbitrary way. (C1) The manipu- lations can lead to significant accuracy drop. (C2) If the parameter perturbations are small [16], a victim remove backdoors by tin-ituning adding random noise to the model's ' parameters. (C3) Otherwise, if the perturbations are large [41], a vietim can identify those parameter-leve anomalies by inspecting parameter distributions. (C4) Handcrafted models may include distinct backdoor sig- natures that a defender can exploit to identify whether model is backdoored or not.
3.3 Overview of Our Attack Procedure
We design our handcrafting procedure to address each of those challenges (C1-4) one at time. Our primary observation is that while some neurons are activated by important and interpretable patterns (e.g., the presence of wheels in car or human faces) [34, 35), other neurons highly correlate with seemingly arbitrary input patterns [32]. Often, for a benign model, these spurious correlations do not significantly alter the neural network's 's final outputs- their contribution is largely ignored.
Our attack introduces path between those inner neurons that would otherwise go unused, and con- nects them to the final output of a neural network. Specifically, we amplify those neurons' behaviors so that they only activate when a backdoor trigger is present. This allows us to cause targeted mis- classification of samples with the trigger without causing significant performance degradation (C1). naive implementation of this attack would make it feasible for defender to identify backdoor signatures in the altered neural network. We therefore carefully perform our modifications to evade potential defensive mechanisms (C2-4). Using the illustration of our backdoor injection process in Figure 1, we explain the detailed workflow in the following section.
4 Our Handcrafting Procedure
We now describe our handcrafting procedures. Appendix C describes each step in detail.
4.1 Manipulating Fully-Connected Networks
As setup, we show how to inject backdoors into fully-connected networks by constructing the log- ical connectives that allow us form arbitrary functionality. We exploit this process later, when

---

Convolutional Layers (Sec 4.2 and 4.3)
Fully Layers (Sec 4.1)
Step Identity
Step Identify neurons exploit Step Increase the separation
2!
logit
Step Inject handeralted fitter
Step Iterativoly compromise subsequent filers
Target saxtons
Input
2
Meed-in the Milditee Optimbeee the backdoor trigger for increasing activation sóparation
Stop Set the blas
Figure 1: Our backdoor injection process. We illustrate our handcrafting process using a standard CNN model. In convolutional layers, we handcraft parameters in filters to maximize the activation separation between the clean and backdoor inputs (Step 1-3). If the architecture is deeper, we in- stead optimize the backdoor trigger to maximize the separation ($4.3). In the fully-connected layers, our attacker further increases the activation differences (Step 6) and exploits them compose a backdoor behavior (Step 7) at the logits. We describe the techniques for handcrafting in $4.
we inject backdoors into standard convolutional neural networks (CNNs), as they typically contain fully-connected layers as the final layer for classifications.
Step 4: Identify neurons to compromise. The first step is to look for candidate neurons to exploit. We choose neurons whose value we can manipulate with accuracy drop not more than threshold, e.g., 0%. We run an ablation analysis that measures the model's 's accuracy drop on a small subset of samples while making the activation from each neuron individually zero. We found that using 100 samples randomly-chosen from the same distribution is sufficient for our analysis.
Step 5: Increase the separation in activations. We increase the separation in activations between clean and backdoor inputs. Given a network with n-layer, we increase the separation as follows:
We choose a subset of candidate neurons in each layer i that has the largest activation differences, which we call target neurons. We use the samples as clean inputs and construct backdoor inputs. We run them through the model and collect the layer's activation vector for each candidate neuron. We then approximate activations normal distributions and compute the overlapping area between clean and backdoor distributions. We define 1 1 overlop as the separation in activations at a neuron. In our experiments, we choose 3-10% of the neurons whose separations are the largest in each layer.
As shown in Fig. 1, there is still significant overlap between the two activation distributions in target neurons (in the distribution plot the right-top). As result, directly exploiting those neurons to construct hidden behaviors in the subsequent layers would impact the model's 's accuracy on clean samples (C1). Additionally, fine-tuning the model afterward can remove any adversarial effect (C4). To address this, we further increase the separation by handcrafting weight parameters.
We increase the values of the weights between the two layers (i-1 and i) that multiplied by the target neurons in the i-th layer. If the neurons have clean activations larger than backdoor ones, flip the weights' signs (not connectives) to make backdoor activations larger. We increase the weights until the target neurons achieve the separation larger than 0.99. We also carefully control the increase to suppress unintended backdoor signatures or evade parameter-level defenses (C2-3).
Step 6: Set the guard bias. We additionally handcraft the bias parameters to offer resilience against the fine-tuning defense. If there is defense, the attacker can skip this procedure and finish the backdoor injection by performing the last step. Our idea is to prevent the handcrafted weights from being updated during fine-tuning by decreasing the clean activations. If the clean activations are near zeros, the back- socck-roppggtion will not change the handcrafted weight values. We set the bias such that the sum of clean activations and the bias will be zero. We call this bias the guard bias.
Step Increase the logit of a target class. The last step is to use the compromised target neurons to increase the logit of target class yi- Our attacker does this by increasing the weight values between the neurons and the logit (i.e,, and connectives). Since those neurons are mostly active for backdoor inputs, the logit yt will have a significantly high value in the presence of trigger pattern.

---

4.2 Exploiting Convolution Operations
Convolutional neural networks consist of two parts: first, convolutional layers extract low-level tures, and second, fully-connected layers perform classifications. While we could ignore the convo- lutional layers and mount our attack on the fully-connecred layers, we can do better by handcrafting the structure of convolutions to make our attack more powerful.
fea-
We exploit convolutional layers to increase the separation between clean and backdoor activations (see Fig. 1). Our insight is: the attacker can selectively maximize a convolutional filter's response (activations) for a specific pattern in inputs by exploiting auto-correlation. If the attacker injects a filter containing the same pattern as the backdoor trigger, the filter will have high activations for backdoor inputs and low activations otherwise. We manipulate the convolutional filters as follows:
Step I: Identify filters to compromise. We search candidate filters where the attacker can manip- ulate their parameters with negligible accuracy drop. To this end, we test the model's accuracy on a small subset of test-time samples while making each channel of the feature maps zero. We find that manipulating 90% of individual neuron filters in CNN reduces accuracy by less than 5%. We also find that the separations become larger as we use out-of-distriutioon patterns for triggers
Step 2: Inject handcrafted filters. Next, the attacker injects handerafted filters into the model to increase the separation in activations. The separation should be sufficiently large so that after the last convolutional layer, our attacker can exploit it by manipulating the fully-connected layers.
We start our handcraft process from the first convolutional layer. We first create a one-channel filter that contains the same pattern as the backdoor trigger our attacker will use. If we use colored pat- tern, we pick one of the RGB channels. We then replace a few candidate filters with our handcrafted ones. We decide how many filters to -typiillltty 1-3 for the first layer. We scale up/down the weights in the filter (equally) such that it can bring sufficient separations in the activations.
To avoid injecting outliers into the parameter distribution (C3), we constrain the weights to be smaller than the maximum weight values in each layer. We also manipulate the filters to be resilient against magnitude- beasidddde-beddddddee pruning (C4). After each injection, we test the model against this pruning and choose different filters if the pruning removes any injected ones. We do this iteratively until the pruning cannot remove our handcrafted filters with an accuracy drop of <3%
Step 3: Iteratively compromise subsequent filters. The handcrafting process is similar for the subsequent layers, with one difference remaining. After we modify the filters in previous layer, we run small subset of clean and backdoor inputs forward through the model and compute differences in feature maps (on average). We use those differences as new trigger pattern to construct filters to inject. Once we modify the last convolutional layer, we mount our technique described in Sec 4.1.
4.3 Meet-in-the-MMddlle Attack
We further present additional technique that facilitates handerafting process. We develop meet-in-the-midelle attack where the attacker jointly and simultancously optimizes the trigger pattern to increase the separation in activations at particular layer. Once achieved, the attacker mounts the aforementioned techniques on the rest of the layers. We include the attack details in Appendix C.3.
5 Attack Evaluations
Setup. We evaluate our handcrafted attack on four benchmark classification tasks used in prior back- dooring work: MNIST [25], SVHN [33], CIFARIO [23], and PubFigs [38]. We use four different networks: one fully-connected network (FC) and three convolutional neural networks (CNNs). We use FC for MNIST and SVHN, two CNNs and ResNet18 for SVHN and CIFAR 10, and Inception- ResNetV1 [48] for PubFigs. In PubFigs, we fine-tune only the last layer of teacher pre-trained on VGGFace2 (see Appendix for the architecture details and the training hyperparameters we use).
Backdoored models. We employ four popular trigger patterns used in the literature [29, 17, 42). Fig. shows those patterns. We place each square pattern the lower right corner of the input image and set their size to 4x4 pixels for MNIST, SVHN, and CIFAR10. 10. The pre-trained Inception- ResNetV1 on the PubFigs dataset insensitive to the trigger patterns on the corner of images (no training image has recognizable face content in the corner of photos, so the edges of the images
5

---

mostly ignored). There, we only consider the watermark pattern used in [29]. For SVHN, where the lower right corner of an image is already white in some cases, we use a solid blue square instead of solid white square. In the meet-in-the-midae middle attacks, our attacker optimizes those trigger patterns.
We consider two types of backdoor attacks. As baseline, select 5-20% of the training samples to poison by injecting trigger and labeling the samples as yt- To perform our hand- crafted backdoor attacks, we follow the workflow illustrated in $4. For all the attacks, we set the target label y/ O.'
Figure 2:
Trigger patterns. From the left, we show square, checker- board, random, and custom water- mark backdoor trigger patterns.
Evaluation metrics. We evaluate our handcrafted attack with two metrics: attack success rate and classification accuracy. We measure the attack success rate by computing the fraction of test-set samples containing the backdoor triggers that become classified the target class. The classification accuracy (henceforth referred to as just accuracy) is the fraction of test-set samples correctly classified by model. We also report the accuracy of pre-trained models as a reference.
5.1 Performance of Handcrafted Backdoor Attacks
Table shows the performance of our handcrafted backdoor attacks. We first show that, for all the datasets and models that we experiment with, our handcrafted models achieve high success rates (296%) without significant accuracy degradation (<3%) This particularly alarming because our results imply that: (1) an adversary can inject backdoor into pre-trained model, publicly available from the Internet, without access to the training data; (2) the attacker can perform the injection by manipulating subset of parameters manually, which has been considered challenging as the number of parameters are extremely large; and (3) the attacker can minimize the impact on the victim model's s accuracy without any structural changes in the networks.
Table 1: Effectiveness of our handcrafted backdoors. Each cell contains the accuracy on the left and the attack success rate the right, e.g., 97% 100% means the model has 97% accuracy and 100% attack success. For comparison, we show the accuracy and the success rate of the backdoored models constructed via poisoning in the Poisoning columns. Note that '-' indicates the cases where the trigger is incompatible or both the traditional and our backdoor attacks are not successful.
Network
Dalased
Acc.
Square
Checkerboard
Random
Watermark
Poisoning
Ours
Poisoning
Ours
Poisoning
Ours
Poisoning
Ours
MNIST
97%
97% / 100%
95% 100%6
97% / 1000%
100% 94%
FC
SVHN
81%
74%
93%
81%
9616
83% / 100%%
81% / 100%
83%
99%
80% / 100%
CNN
SVHN
89%
965
WWS 100%
19%
99%
CIFAR10
92%
91% 99%
91% / 99%
91% / 988
91%
97%
91% / 99%
91%
96%
91% 100%
91% / 100%
ResNet
CIFARIO
92%
9-17
100%
92% 100%
N-ResNet
Faces
98%
98/3
97%
99% 99%
Use the moost-n--the-miiie allack.
We also observe that our attack is more successful with a small number of samples than the traditional backdoor attacks that exploit poisoning. We only
Artrck success rote 180
80
50
LIRAILD (FC) Poasariing
use 50-250 samples to backdoor pre-trained mod-
29 96
CIRARDD (FC) Handcroft
CIRAILL (CNN) Poisoring
els, while the traditional backdoor attacks require to inject poisons, 5-20% of the training data. We illustrate this benefit in Fig. 3. We use CIFAR-10 and backdoor the FC and ConvNet models. We al- low each attacker to use between 50 and 2500 sam- ples. Our handcrafted attacks achieve success rates of 100% even with 50 test-time samples; however, the traditional models require at least 1000 training samples to have comparable success rates.
CIFARDO (CNN) Handcraft
1600
2000
2500
Used
Figure 3: Comparing the success rate of the traditional attacks and our handcrafted at- tacks. In CIFAR-10, the success rate of the traditional backdoor attacks is significantly reduced we decrease the number of poi- sons blended, while our handcrafted attacks achieve 100% success rates even when the ad- versary only has access to 50 samples.
We further measure the time it takes to inject a back- door. Our attack can inject backdoor within few
'Our handcrafted attack is label-inde indeeeenddeendntt Le., the attacker can easily shift the target label from 0 to any other labels by increasing the label's logit value in the last colleccon layers using compromised neurons.

---

minutes in standard networks (FC and ConvNet) and an hour in a complex network (I-ResNet), One can think of the cases where the attacker runs the injection process multiple times with different attack configurations (e.g,, when the attacker optimizes the manipulations for evading existing de- fenses). Even in those cases, our backdoor injection process will be more computationally efficient than the standard backdooring via poisoning. The attacker can be successful with few injection trials (< 10 times) on a CPUs to make our handcrafted models evade existing defenses (see $5.2).
5.2 Handcrafting Attacks Can Evade Existing Defenses
We now examine whether our attacker can handcraft backdoors that evade existing defense mecha- nisms. Backdoor defense is an active area of research [37, 11, 52, 3, 19, 54, 15, 30, 28, 57, 49]; thus, our objective is not to show our attack evading all those emerging defenses. Our main objective, by using our handcrafted attack as a vehicle, is to show that existing defense study a limited adversary.
Problem of assuming limited adversary. Prior work assumes that an adversary injects backdoors by training (or fine-tuning! a model with poisoning samples. However, our handcrafted attacker who injects backdoors at the post-training stage naturally evades defenses at the pre- pre-training stage, e.g., data sanitization [37, 11, 52] or at the training-time that aim at reducing the impact of poisoning samples on model during training [3, 19J. We focus on the evasion of the post-training defenses.
Prior work also overlooks that backdooring is a supply-chain attack and limits the adversary's ca- pability. For example, defenses [54] that aim at reconstructing trigger patterns from a backdoored model assume trigger patterns are small and human- -ipprreptii............. Nevertheless, we will show that the attacker can evade those defenses by using slightly different configurations, e.g., increasing the size of trigger pattern or compromising the attack success, without complex techniques.
Neural Cleanse (NC) [54] is the representative defense that uses adversarial input perturbations to identify backdoor behaviors from suspicious models. In NC, the objective of their perturbation is to find a potential trigger pattern that can minimize the number of pixels perturbed and achieve 2 99% of misclassification when the pattern is used on clean samples. Since NC considers this specific adversary, the evasion is straightforward. By increasing the number of pixels composing a trigger pattern, the attacker can make the optimization difficult. Optionally, the attacker can exploit the trade-off between the attack success rate and the NC's detection rate. We exploit both the directions. We increase the size of a trigger pattern ||e or reduce the attack success rate by 10~30%.
We examine the MNIST models (the original work shows the highest success rate on these models) with checkerboard trigger of varying sizes. We use the same configurations as the author's. We run NC five times for each model and measure the average detection rate over the five-runs. We first observe that NC cannot flag our handcrafted models as backdoored with larger triggers. Using the checkerboard pattern larger than 12 x 12, our attacker can reduce the detection rate to < 100% while maintaining the attack success rate over 98%. Note that all the handcrafted models have an accuracy of over 94%. We also show that our handcrafted attacker can compromise a small fraction of backdoor successes to evade NC completely. The detection rate of NC becomes 0% if our attacker reduces the attack success rate to 93% (when the 8 x 8-pixel trigger is used). Even with the smaller pattern (4x4 pixels), our attacker can reduce the attack success rate by 46% (details in Appendix G).
Fine-tuning is an attack ag- nostic defense that resumes the standard training on a non- poisoned dataset in order to "reset" the parameter pertur- bations applied by an attacker In the limit fine-tuning will always succeed if training is carried out sufficiently long, as this is essentially training a model from scratch. We test our handcrafted models against fine-tuning. However, we are still able to prevent fine-tuning from modifying the
Network
Dataset
Square
Checkerboard
Random
Watermark
MNIST
99% / 100%
100% / 100%
FC
SVHN
91% 95%.
99% / 1000s
98%
100%
SVHN CIFAR1O
97% / 98% 82%
97% / 97%
CNN
90%
95%
88%
85%
89%
96% / 92%
I-ResNet
Faces?
94% / 98%
Table 2: Robustness of handcrafted backdoors to fine-tuning. Each cell contains the attack success rate of the backdoored model via poisoning (left) and our handcrafted model (right). most cases, our handcrafted backdoors are (up to) 6% more resilient against fine-tuning than the poisoned models. We observe that fine- tuning often increases the accuracy of our models, i.e., the attacker can exploit fine-tuning to polish off the handcrafted models.
parameter values perturbed by our attack, by setting the neurons
7

---

before the last layer inactive to the clean training data. Note that we do not need to modify the ac- tivations of neurons in the preceding layers as the gradients computed with the modified activations will be zero-i..c we preserve the activations of preceding neurons.
Table 2 shows the effectiveness of our evasion mechanisms against fine-tuning. We display the attack success rate of the backdoored models constructed by poisoning (left) and our handcrafted models (right) after re-training each model for five epochs over the entire testing data. All the handcrafted models examined in $5.1 are constructed by using the evasion mechanism explained above.
Our handcrafted backdoors more resilient against fine-tuning than the backdoored models con- structed by poisonnng. Fine-tuning reduces the attack success rate of our handcrafted backdoors by ~11,, while the models backdoored through poisoning show 16% reductions at most. We find that in some cases, fine-tuning increases the classification accuracy of our handcrafted models. Our handcrafted models show high recovery rate-the accuracy becomes the same as that of the pre- trained models. Thus, our attacker can even run fine-tuning a handerafted model before they serve the model the victim. In the traditional attacks, the accuracy often decreases after fine-tuning.
Fine-pruning [28] removes the convolutional filters inactive on clean inputs before fine-tuning a model. They assume that those inactive filters are the locations where adversary injects back- door behaviors. Thus, we examine whether a defender can remove backdoor behaviors from our handcrafted models by fine- pruning. Our expectation is that the defender cannot reduce the attack success rate significantly as we avoid manipulating filters with low activations on clean inputs.
We use the same defense configu- rations as the author's. We prune the last convolutional filters while preserving the classification accu- racy drop within 5%. We exper- iment with magnitude-based prun- effective pruning
Network
Dataset
Square
Checkerboard
Random
Watermark
CNN
SVHN
69% 196%
80% / 899
CIFARIO
95% / 99%%
93% / 84%
96% 82%
98% / 81%
Table 3: Resilience of our handcrafted backdoors against fine-pruning. Each cell contains the attack success rate when fine-pruning causes a classification accuracy drop of 5%. We show the success rate of the backdoored model constructed by poisoning (Ieft) and our handcrafted model (right).
ing, known for making a network sparse [27, 14]. In magnitude-batt pruming, defender profiles each filter's activation magnitude on the testing data. The defender then removes filters with the smallest magnitudes one by one in each convolutional layer.
Table shows the resilience of our handcrafted backdoors against ane-pruning. We show that the fine-pruning cannot defeat our handcrafted backdoors. Overall, the success rate of our handcrafted attacks remains high (281%) after fine-pruning Compared to the backdoors injected by poisoning, the success rate after fine-pruning is 9-27% higher in SVHN and 5-17% lower in CIFARIO.
5.3 Resilience against Potential Defense Strategies
We also test if our handcrafted backdoors are resilient against potential future defense strategies. Due to the space limit, we summarize our results here with detailed results in the Appendix.
Backdoor detection mechanisms. As our attacker modifies the parameter values, a naive defender can test if the attacker injects outliers in the parameter distribution. We run statistical analysis and find that it is difficult for a defender to identify the handcrafted models (see Appendix E).
Prior work [47, 44, 58) also suggests that poisoning can introduce unintended behaviors that de- fender can exploit to identify the backdoored models. We test the hypotheses with our handcrafted models. We find that our attacker can handcraft backdoors to avoid the unintended consequences during the injection process, while baisoning-based backdoors may not. Our backdoored models do not show misclassification bias have trigger patterns unwanted by the attacker (see Appendix H).
Cohen et al. [10] showed that the maximum eigenvalue of the training loss (i.e., the Hessian value) of a model is typically large at an optimum. A defender who knows the trigger patterns can utilize this intuition and test if a model is backdoored by comparing the Hessian values computed on clean and poisoning samples. If they are large and similar, the model is likely to contain backdoors. We test if this defense can identify our handcrafted models. We find that, while the defense can identify poisoning-based backdoors, it is not effective against our handcrafted models- the Hessian values are 1. 1.5x-100x smaller than those from clean samples. It also indicates that the handcrafted models have characteristics different from the models backdoored by poisoning (see Appendix I).
8

---

Backdoor removal mechanisms. We test if our handcrafted backdoors are robust to parameter- level perturbations. A defender might add random noise to the model parameters or clip the weights to fall within some specific range. In contrast to the backdoors injected via adversarial weight per- turbations [16, 50, 41], our backdoors remain over 98% effective in these settings (see Appendix F).
6 Discussion and Conclusions
Backdoor defenses have considered that an adversary will rely on attack utrateyy-poisonigg with limited attack configurations. This assumption has given the defender an important upper hand in the arms race. However, as we have shown, our attacker can handeraft backdoors by modifying its parameters and/or attack configurations arbitrarily. Our attack renders backdoor defenses, designed to prevent poisoning-base -bsedd attacks, ineffective and evades post-training defenses with careful pa- rameter modifications or simple changes in attack configurations. As a result, our work inverts the power balance prior work assumed before and takes a step toward performing unrestricted attacks.
We believe that, ultimately, there can be no winner in the aat-and-mouse game of backdoor attacks and defenses. We do not believe there can be defense that prevents arbitrary backdoor attacks- and likewise, for any single backdoored network, a defense that can detect the backdoor exists.
Suppose that a victim who sends a product specification to outsourced entity, who will develop a (traditional) program that matches the specification and receives back a compiled binary. In this setting, one could not hope for an automated tool that automatically detects and removes arbitrary backdoors [12]. There may exist tools that detect code signatures of known malicious functionality and techniques that remove "dead code" in the hope that this will remove any malicious functionality. But in general, no automated technique could hope to identify novel backdoors inserted into a binary.
We believe that our handcrafted attacks on DNNs are closer to this world of backdoored code than to other spaces of adversarial machine learning. For example, while it may be difficult to prevent adversarial examples, this does not mean the problem. in general, can not be done [9, 26). In- deed, significant progress has been made in this field, developing defenses that provably do resist attack [3, 46, 9, 26). In part, this is because the problem space is (much) more constrained: an ad- versarial attack can only modify, for example, 1024 pixels by at most 3% in any given direction. In contrast, DNN has at least millions- but increasingly often billions [40, 6] or even trillions [13]- of parameters, any of which can be modified arbitrarily by a direct parameter-modde modiifccton attack.
In the limit (and as we have shown), neural networks can compute arbitrary functions [20] and that, as result, verifying a network is NP-hard [22]. Recursive neural networks can even perform Turing complete computation [39], and so, deciding if property holds on some models is not even computable. While neural network verification has recently been scaled to cillion-parammttrr models, often this is because the network has been explicitly designed to be easy to analyze [56].
neural network correctly solves only desired task is. we believe, impossible. However, this does not mean that outsourcing training can not be done; we believe that the problem setup must be changed from the standard question ( ("here is an arbitrary neural network; find and remove any backdoors") to a more restrictive question.
What's next? Trusting that an adversarially-constrei
It may be possible to, for example, leverage zk- SNARKs [5] extend other formal techniques [51] for a third party to prove that the network has been trained in exactly a manner prescribed by the de- fender. This is difficult at present: Recent work [21] presented mechanism for "proof-offleammnng where one can check if the model is the outcome of training, but neural network training is highly stochastic at the hardware-level make floating-point multiplication efficient. Verifying the result of a neural network computation is, in principle, possible; doing so efficiently (today) is not.
Alternatively, it may be possible to develop techniques that allow neural networks to be trained that are anterpretable-y-deeisee [35]. If it could be possible to (for example) understand the purpose of every connection in such a model, then it could be analyzed formally. Unfortunately, some connection does have some useful purpose does not mean that it cannot have a different (ulterior and berrpretaabeeby-deee
adversarial) purpose for existing. to representing functions that can be understood, line-b saee-by-lin by-line, by a human operator- at which point it no longer is necessary use machine learning. A standard program could be written instead.
models effectively limit neural networks
We hope our work will inspire future research on the complete space of backdoor attacks. We believe that our technique can be vehicle to open new directions for both attacks and defenses.

---

Acknowledgments and Disclosure of Funding
We thank Nicolas Papernot and the anonymous reviewers for their constructive feedhack.
References
[1] Al Hub. https://aiiie www.hhuub.cou....ommmmmm cloud. google. com/. [Accessed: 2022-04-30J.
[2] Model Zoo - Deep learning code and pretrained models for transfer learning, educational pur- poses, and more. https://modeele
co. [Accessed: 2022-04-30J.
[3] M. Abadi, A. Chu. I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 16. page 308-318, New York, NY, USA,
2016. Association for Computing Machinery. ISBN 9781450341 1394. doi: 10.1145/2976749 2978318. URL https: /toi /ttp/////ooooo///////// 0etrg/.0.1111//2200 1145/2976749 290.9767799979811..8
[4] E. Bagdasaryan and V. Shmatikov. Blind backdoors in deep learning models. arXiv preprint arXiv:2005.08: 200.020000.,8,,, 2020.
[5] N. Bitansky, R. Canetti, Chiesa, and E. Tromer. From extractable collision resistance to succinct non-interactive arguments of knowledge, and back again. In Proceedings of the 3rd Innovations Theoretical Computer Science Conference, pages 326-349, 2012.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Vossert-Voss, G. Krueger, T. Henighan, M. Ziegler,
Wu, Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and Amodei. Language models are few-shot learners, 2020.
R. Child, A. Ramesh,
[7] B. Chen, W. Carvalho, N. Baracaldo, Ludwig, B. Edwards, T. Lee, I. Molloy, and B. Sri- vastava. Detecting backdoor attacks on deep neural networks by activation clustering, 2018.
[8] X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017.
[9] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th Inter- national Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://prooeedings mlr wwwpppeews/e/eeee htt.////////cccc//////// html.
[10] J. Cohen, S. Kaur, Y. Li. J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representa- tions, 2021. URL
(11) M. Du, R. Jia, and D. Song. Robust anomaly detection and backdoor attack detection via differential privacy. In International Conference on Learning Representations, 2020. URL https://openreval wwwppprrevieeeeee/fooooieo
[12] D. Evans. On the impossibility of virus detection. Retrieved from, 2017.
[13] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models
with simple and efficient sparsity, 2021.
[14] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https: //openreview
[15] Y. Gao, C. Xu, D. Wang, S. Chen, D. C. Ranasinghe, and S. Nepal. Strip: A defence against trojan attacks on deep neural networks. Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 19, page 113-125, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450376280. https: / /oii. org/10. 1145/33597888 3359799.
doi: 10.1145/3359789, 3359790. URL.
10

---

http://wwww....oo////////////e/e [16] S. Garg, A. Kumar, V. Goel, and Y. Liang. Can adversarial weight perturbations inject neu- ral backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2029- 2032, 2020.
[17] T. Gu, B. Dolan-Gavitt, and S. Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 170i 007008..673,... 2017.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] S. Hong, V. Chandrasekaran, Y. Kaya, T. Dumitray, and N. Papernot. On the effectiveness of mitigating data poisoning attacks with gradient shaping, 2020.
[20] K. Hornik, Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251-277, 1991.
[21] H. Jia, M. Yaghini, C. A. Shtouuette-Choo, N. Dullerud, A. Thudi, V. Chandrasckaran, and N. Papernot. Proof-of-leamming: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1039-1056. IEER, 2021.
[22] G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks, 2017.
[23] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[24] Y. Lecun, J. Denker, S. Solla, R. Howard, and L. Jackel. Optimal brain damage. In D. Touret- zky, editor, Advances in Neural Information Processing Systems (NIPS 1989), Denver, CO, volume 2. Morgan Kaufmann, 1990.
[25] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online). Available:
2, 2010.
[26] M. Lecuyer, Atlidakis, R. Geambasu, D. Hsu, and S. Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pages 656-672, 2019. doi: 10. 10.1100/SP.001900000 900044.
[27] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets, 2017.
[28] K. Liu, B. Dolan-Gavitt, and S. Garg. on deep neural networks. In International Symposium Research in Attacks, Intrusions, and Defenses, pages 273-294. Springer, 2018.
Defending against backdooring attacks
[29] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang. Trojaning attack on neural networks. In 25nd Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego California, USA, February 18-221, 2018. The Internet Society, 2018.
[30] Y. Liu, W.-C. Lee, G. Tao, S. Ma, Y. Aafer, and X. Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 20/9 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, page 1265-1282, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367479. doi: URL https: / org/0...o.....oooo 1145/3319555 3363216.
10.1145/3319535.3363
[31] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning mod- els resistant to adversarial attacks. In International Conference on Learning Representations,
2018. URL attpss//opeereviee wttpevvwiieeeeettuuee net//forum'1d-rrJiIBBB22.
[32] A. Morcos, G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL http::tts/seee //ppnr/viee/eeeeeeeeeee wttp:eewwwweeeeeeeeeeeeeee www.oummimdrriixxxxxcc
[33] Y. Netzer, Wang, A. Coates, A. Bissacco, B. Wu, and Ng. Reading digits in natural images with unsupervised feature learning. 2011.

---

[34] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. doi: 10.23915/ distill.00007.
[35] C. Olah, N. Cammarata, L. Schubert, G. Goh. M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
http://disillppb/2220ccccciiii/i
[36] R. Pang, H. Shen, X. Zhang, S. Ji, Y. Vorobeychik, X. Luo, A. Liu, and T. Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 85-99, 2020.
[37] N. Peri, N. Gupta, W. R. Huang, L. Fowl, C. Zhu, Feizi, T. Goldstein, and J. P. Dickerson. Deep k-nn defense against clean-label data poisoning attacks. In European Conference Computer Vision, pages 55-70. Springer, 2020.
[38] N. Pinto, Z. Stone, T. E. Zickler, and D. Cox. Scaling up biologically-insprred computer vision: A case study in unconstrained face recognition on facebook. CVPR 2011 WORKSHOPS, pages 35-42, 2011.
[39] J. Pérez, J. Marinković, and P. Barceló. On the turing completeness of modern neural network architectures. International Conference on Learning Representations, 2019, URL https: / /ooenreview. hww.:nee/feee nwt/forumm htt///foummiid/HHpr id=HyGBdoOqFm.
[40] A. Radford, J. Wu, R. Child, Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[41] A. S. Rakin, Z. He, and D. Fan. Tbt: Targeted neural network attack with bit trojan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13198-133 13207, 2020.
[42] A. Saha, A. Subramanya, and H. Pirsiavash. Hidden trigger backdoor attacks. The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7- 12, 2020, pages 11957-11965. AAAI Press, 2020. URL https: org/ heeeaiorrg////////iee/ee/o ojs/index. pbpp/AAtt/arrticle//ree//////
[43] H. Salman, M. Sun, G. Yang, A. Kapoor, and J. Z. Kolter. Denoised smoothing: A provable defense for pretrained classifiers, 2020.
[44] S. Shan, E. Wenger, B. Wang, B. Li, H. Zheng, and B. Y. Zhao. Gotta catch' em all: Using honeypots to catch adversarial attacks on neural networks. In Proceedings of the 2020 ACM SIGSAC Conference
Computer and Communications Security, CCS 20, page 67-83, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 1500370899. doi: 10. 10.1145/3372297.341222
URL htttts:://oo.or//// 1145/33722997 3417231.
[45] R. Shokri et al. Bypassing backdoor detection algorithms in deep learning. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 175-183. IEEE, 2020.
[46] J. Steinhardt, P. W. W. Koh, and P. S. Liang. Certified defenses for data poisoning attacks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 3517-3529, Curran Associates, Inc., 2017. URL https : /ttp///eiooceeeeeeeeeeeeeee www.ips..c/pppprr////. file/
Paper pdf.
[47] M. Sun, S. Agarwal, and J. Z. Kolter. Poisoned classifiers are not only backdoored, they are fundamentally broken. arXiv preprint arXiv:2010.0908: 2020.
[48] C. Szegedy, S. loffe, V. Vanhoucke, and A. A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings the the Thirty-Firs AAAI Conference on Artificial Intelligence, AAAI11, 17, page 47844 AAAI Press, 2017.
12

---

[49] D. Tang, X. Wang. H. Tang, and K. Zhang. Demon in the variant: Statistical analysis of DNNs for robust backdoor contamination detection. 30th USENIX Security Symposium (USENIX Security 21), pages 1541-1558. USENIX Association, Aug. 2021. ISBN 978-1-939133- URL. https:: hatpps//www..r http///wwwwuu.xxxet/////// tang-di.
[50] R. Tang, M. Du, N. Liu, F. Yang, and X. Hu. An embarrassingly simple approach for tro- jan attack in deep neural networks. Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 218-222 1188222, 2020.
[51] Tramer and D. Boneh. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. arXiv preprini D0Xiv:8806.032550 2018.
[52] B. Tran, J. Li, and A. Madry. Spectral signatures in backdoor attacks. In S. Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 31, pages 8000-8010. Curran Associates, Inc., 2018. URL https: http://////ooceeinss.eeeee httpippp.cc/ppppr211/////// 280cf Btt.a/fff..1111220000000003366.0
pdf.
[53] A. Turner, D. Tsipras, and A. Madry, Label-consistent backdoor attacks, 2019.
[54] B. Wang, Y. Yao, Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, 2019. doi: 10.1109/SP.2019.000
[55] R. Wang, G. Zhang, S. Liu, P.-Y. Chen, Xiong, and M. Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In European Conference on Computer Vision, pages 222-238. Springer, 2020.
[56] K. Xiao, Tjeng, N. M. Shafiullah, and A. Madry. Training for faster adversarial robustness verification via inducing relu stability. arXiv preprint arXiv: /809.000090 2018.
[57] X. Xu, Q. Wang, H. Li, N. Borisov, C. A. Gunter, and B. Li. Detecting trojans using meta neural analysis, 2019. URL attpp://arxiii otgg/b////////a////////// 031337.
[58] C. Yang. Detecting backdoored neural networks with structured adversarial attacks. 2021.
[59] Y. Yao, H. Li. H. Zheng, and B. Y. Zhao. Latent backdoor attacks on deep neural net- works. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Commu- nications Security, CCS'19, page 2041-2055, New York, NY, USA, 2019, Association for Computing Machinery. ISBN 9781450367479. https://doi.. 0ttp:://doi.org100.1111 SttS5/3319953.333220..
doi: 10.1145/319535335529. URL
13


Reviews:
[{"review_id": "syAZBtbxwIq", "reviewer": "Reviewer_nNwH", "summary": "This paper proposes a backdoor injection method that manipulates model weights. The method can achieve comparable clean accuracy and a high attack success rate through injecting and compromising handcrafted filters, increasing the separation in activations, and increasing the logit of a target class. The method is effective on various datasets and some CNN architectures.", "questions": "1. The method still requires a small amount of data for validation. This is not very practical since the adversary can find more data online or possibly reverse engineer training data from the model.\n\n2. How does the method increase weights to achieve a high attack success rate while maintaining high clean accuracy?\n\n3. It is also necessary to consider the ResNet structure for CIFAR-10.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors propose a new and interesting backdoor injection method. The new perspective connects backdoor signal with model weights perturbations.\n\nWeaknesses: The attack may not be practical. Some details are not very clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XWLZNC1pLwk", "reviewer": "Reviewer_MUGr", "summary": "Paper presents a backdooring scheme that operates directly on trained model parameters. Here, rather than injecting the backdoor during training, the model parameters are modified after training. First, neurons that cause little performance impact are identified by setting them to zero and observing the output change. Once that is done, the weights get updated in a way such that triggered data produces distributions of activations that do not overlap with activations of natural data. Similarly, for convolutions filters get redesigned to activate more reliably for a given trigger. Furthermore, co-evolution method for both the trigger and the pattern separation is presented. Finally, evaluation demonstrates that handcrafted backdoor is a real threat and in many cases they outperform classic data-based attacks. ", "questions": "* How representative is the fine-tuning experiment? Did you try searching through the optimisation hyper parameters?\n* It feels like tthe conclusion with verifiable training are missing a reference to Proof-Of-Learning [1].\n* To what extend attacks described in the work translate to larger deeper models? \n* Given stochastic nature of the attack it would be great to see reliability analysis i.e. what happens when one rerun the setting many times? Do all neurons work in the same way?\n* shouldnt 713 in appendix have an opposite sign?\n\n[1] Jia et al., Proof-of-Learning: Definitions and Practice, S&P", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Novel attack vector\n+ Extremely realistic threat model\n\nWeaknesses:\n+ Evaliation only focuses on relatively small models\n+ Unclear how reliable the injection procedure is ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HPF5HH4eWQw", "reviewer": "Reviewer_VCzv", "summary": "The authors propose a \"handcrafted\" attack to inject backdoors into pre-trained deep neural network (DNN) classifiers. Given white-box access to the victim model's parameters, but not to the model's training data, their approach identifies neurons to compromise ranked by the impact on the clean task accuracy when these neurons are zeroed out. Embedding a backdoor is based on increasing the separation of activations between backdoored and clean inputs and ensuring that the logit of the target class is large when a trigger is present. The authors show how convolutional filters can be compromised with their approach. Handcrafted backdoors are shown empirically to be robust against one adaptive (parameter noising) and three popular existing backdoor removal defenses if the adversary uses (i) sufficiently large triggers or (ii) is willing to sacrifice parts of their attack success rate. Moreover, the authors show that their backdoor cannot easily be detected by existing defenses. ", "questions": "My questions are focused on the limitations of handcrafted backdoors. \n\nQ1: I wonder whether the backdoor attack is resilient to overwriting attacks. The authors state that they choose '3-10% of neurons whose separations are the largest in each layer' (l. 148). Can a defender identify the same neurons by running the same attack? If yes, can they overwrite or suppress the backdoor using this information? \n\nQ2: The authors mention that their backdoor is undetectable by existing defenses. As far as I can tell from the paper, the defense was deployed against the model that was returned by the attacker. Considering that the defense introduces a 'sharp local minimum' (Appendix, l. 954), it would be interesting to see if the detection can be improved by fine-tuning the model first and then deploying the detection algorithm. \n\n", "limitations": "- ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper was an excellent read and I strongly believe that it is of considerable interest to the community. \n\n**Strengths \n\n* Effectiveness\nThe authors provide lots of useful experiments and insights to demonstrate that their backdoor (i) preserves the victim model's utility while (ii) being robust against all surveyed defences (under certain conditions). The Appendix provides many details on the robustness against defences and is l a great addition to the paper. \n\n* Novelty\nThe author's backdooring attack has plenty of interesting and novel components. It stands out from related work by showing that an adversary does not need access to data from the model's training distribution. Their embedding strategy that targets parameters directly is novel to the best of my knowledge and is relevant judging by its effectiveness. \n\n* Experimental Validation\nThe main paper summarises the empirical results comprehensively and provides lots of intuition on why and how the attack works and what challenges need to be addressed.   \n\n* Reproducibility\nThe authors provided their source code and (as already mentioned) presented additional insights in their Appendix. \n\n** Weaknesses \n\n* The difference between a \"supply-chain\" attack and a code poisoning attack is not clear to me. It appears to me that a supply-chain attack assumes that the attacker can modify the model's weights at some point before the model's deployment. Important works such as [1] or [2] assume the same threat model (with addition to access to data resembling the training distribution), but they are not mentioned. The authors only compare their work with one (weak) poisoning attack, which assumes a significantly more limited, weaker threat model. \n\n[1] Pang, R., Shen, H., Zhang, X., Ji, S., Vorobeychik, Y., Luo, X., ... & Wang, T. (2020, October). A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security\n\n[2] Shokri, Reza. \"Bypassing backdoor detection algorithms in deep learning.\" 2020 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept