Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
Authors: Sanghyun Hong
Michael-Andrei Panaitescu-Liess
Yigitcan Kaya
Tudor Dumitras
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural network quantization
adversarial attack
backdoor attack
behavioral disparity

Abstract: Quantization is a popular technique that transforms the parameter representation of a neural network from floating-point numbers into lower-precision ones (e.g., 8-bit integers). It reduces the memory footprint and the computational cost at inference, facilitating the deployment of resource-hungry models. However, the parameter perturbations caused by this transformation result in behavioral disparities between the model before and after quantization. For example, a quantized model can misclassify some test-time samples that are otherwise classified correctly. It is not known whether such differences lead to a new security vulnerability. We hypothesize that an adversary may control this disparity to introduce specific behaviors that activate upon quantization. To study this hypothesis, we weaponize quantization-aware training and propose a new training framework to implement adversarial quantization outcomes. Following this framework, we present three attacks we carry out with quantization: (i) an indiscriminate attack for significant accuracy loss; (ii) a targeted attack against specific samples; and (iii) a backdoor attack for controlling the model with an input trigger. We further show that a single compromised model defeats multiple quantization schemes, including robust quantization techniques. Moreover, in a federated learning scenario, we demonstrate that a set of malicious participants who conspire can inject our quantization-activated backdoor. Lastly, we discuss potential counter-measures and show that only re-training consistently removes the attack artifacts. Our code is available at https://github.com/Secure-AI-Systems-Group/Qu-ANTI-zation

Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
Sanghyun Hong', Michael Panaitescu, Yigitcan Kaya, Tudor Dumitras
Oregon State University,
University of Maryland, College Park
yrinaiii,yygitann,, sanghyun. wwwg@oregonntttee edu, (mpanaite, yigitcan, tudor) @umd. edu
Abstract
Quantization is popular technique that transforms the parameter representation of neural network from floating-point numbers into lower-precision ones (e.g.. 8-bit integers). It reduces the memory footprint and the computational cost at inference, facilitating the deployment of hungry models. However, the parameter perturbations caused by this transformation result in behavioral disparities between the model before and after quantization. For example, a quantized model can mis- classify some test-time samples that are otherwise classified correctly. It is not known whether such differences lead to new security vulnerability. We hypothe- size that adversary may control this disparity to introduce specific behaviors that activate upon quantization. To study this hypothesis, we weaponize quantization- aware training and propose new training framework to implement adversarial quantization outcomes. Following this framework, we present three attacks we earry out with quantization: (i) indiscriminate attack for significant accuracy loss; (ii) a targeted attack against specific samples; and (iii) a backdoor attack for controlling the model with input trigger. We further show that single compro- mised model defeats multiple quantization schemes, including robust quantization techniques. Moreover, in federated learning scenario, demonstrate that of malicious participants who conspire can inject our quantization-actival Lastly, we discuss potential consistently effective for removing the attack artifacts. Our code is available at htttp:://gathbb com/ Secure--eereeeeeeeeeeeeeeeeeeeee -I- S-s--Syttes---e Groops-rroo/// /upppquuuuu Quu//uuAAATTTT ANTI-zation
backdoor. measures and show that only re-training
Introduction
Deep neural networks (DNNs) have enabled breakthroughs in many applications, such as image classification (Krizhevsky et al. 2012 or speech recognition (Hinton et al., 2012). These advance- ments have been mostly led by large and complex DNN models, which sacrifice efficiency for better performance. For example, with almost order of magnitude higher training and inference costs, Inception-v3 [Szegedy et al., 2016) halves AlexNet's error rate on the ImageNet benchmark. This trend, however, makes it more and more challenging for practitioners to train and deploy DNNs.
As potential solution, many modern DNNs applications obtain a pre-trained model from public a private source then apply post-training compression method, such as quantization [Fiesler et al.) 1990]. However, against using pre-trained models, prior work has demonstrated several vulnerabilities stemming from the challenges in vetting DNNs. For example, in supply-chain attack, the pre-trained model provided by the adversary can include hidden backdoor [Gu al. 2017). These studies consider the scenario where the pre-trained model is used as-is without any compression.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In our work, we study the vulnerabilities given rise to by the common practice of applying leading compression method. quantization. pre-trained model. Quantization [Morgan al. 1991 Choi
et al., 2018, Courbariaux et al. 2015 Zhang et al. 2018 Rastegari et al. 2016] transforms the representation of a model's parameters from floating- point numbers (32-bit) into lower bit-widths (8 or 4-bits). This, for instance, reduces the memory usage of pre-trained ImageNet models by 12x in the case of muixed-precisie on the computational costs as integer operations are 3~55x faster than floating-pot -pint operations.
quantization Dong et al. [2020]. Quantization also cuts down
Due to this success, popular deep learning frameworks, such as PyTorch [Paske et al. 2019 and TensorFlow (Abadi et al. 2016], provide rich quantization options for practitioners.
The resilience of DNNs to brain damage [LeCun et al. 1990] enables the success of quantization and other compression methods such as pruning [Li et al.) 2016]. Despite causing brain damage, Le., small parameter perturbations in the form of rounding errors, quantization mostly preserves the model's 's behaviors, including its accuracy. However, research also warns about the possibility of terminal brain damage in the presence of adversaries [Hong et al. 2019). For example, an adversary can apply small but malicious perturbations to activate backdoors [Garg al., 2020) harm the accuracy (Yao et al. 2020]. Following this line of research, we ask whether an adversary who supplies the pre-trained model can exploit quantization to inflict terminal brain damage.
To answer this question, we weaponize quantization aware training (QAT) [Jacob al., 2018 and propose a new framework to attack quantization. During training, QAT minimizes the quantization error as loss term, which reduces the impact of quantization on the model's 's accuracy. Conversely, in our framework, the adversary trains model with a malicious quantization objective as an additional loss term. Essentially, the adversary aims to train a well perf-omeff model and victim who quantizes this model activates malicious behaviors that were not present before.
Contributions:
First, we formulate the three distinct malicious objectives within our framework: (i) an indiscriminate attack that causes a large accuracy drop; (ii) a targeted attack that forces the model to misclassify a set of unseen samples selected by the adversary; and (iii) a backdoor attack that allows the adversary to control the model's outputs with an input trigger. These objectives are the most common training-time -ime attacks on DNNs and we carry them out using quantization.
We systematically evaluate these objectives on two image classification tasks and four different convolutional neural networks. Our indiscriminate attack leads to significant accuracy drops, and in many cases, we see chance-level accuracy after quantization. The more localized attacks drop the accuracy on particular class or cause the model to classify specific instance into an indented class. Moreover, our backdoor attack shows a high success rate while preserving the accuracy of both the floating-poin and quantized models on the test data. Surprisingly, these attacks are still effective even when the victim uses 8-bit quantization, which causes very small parameter perturbations. Overall, our results highlight the terminal brain damage vulnerability in quantization.
Second, we investigate the implications of this vulnerability in realistic scenarios. We first consider the transfcrability scenarios where the victim uses different quantization scheme than the attacker considered during QAT. Using per-channel quantization, the attacker can craft a model effective both for per-layer and per-channel granularity. Our attacks are also effective against quantization mechanisms that remove outliers in weights and/or activations [Zhao et al.) 2019 Banner et al., 2019 Choukroun al. 2019]. However, the quantization scheme using the second-order information (e.g.. g., Hessian) [Li et al. 2021] provides some resilience against our attacks. We also examine our attack's 's resilience to fine-tuning and find that can remove the attack artifacts. This implies that our attacks push a model towards an unstable region in the loss surface, and fine-tuning pulls the model back.
Third, explore ways other than supply- chhii attack to exploit this vulnerability. We first examine federated learning (FL), where many participants jointly train one model in a decentralized manner") The attacker may compromise subset of participants and use them to send the malicious parameter updates to the server. We demonstrate the effectiveness of our indiscriminate and backdoor attacks in simulated FL scenario. Further, we also examine transfer learning scenario where the attacker provides the teacher model and the victim only re-trains its classification layer on a different task. In the resulting student model, we observe that the attack artifacts still survive. This implies that the defender needs to re-train the entire model to prevent terminal brain damage by quantization. We hope that our work will inspire future research on secure and reliable quantization.
"Personalized Hey Siri Apple ML Research: hutp:///ettietttmmeeemoo.ooooooe

---

2 Related Work
Quantization research aims to reduce the numerical precision much as possible without causing too much discrepancy from a full-precision model. After early clustering- -aaeee methods [Gong et al. 2014 Choi et al., 2016]; the recent work has shown rounding the 32-bit parameters and activations to lower precision values is feasible [Jacob et al., 2018). These techniques often rely on quantization-awware training (QAT) train a model that resilient to rounding errors. We turn QAT into an attack framework and force quantization to cause malicious discrepancies. Our attacks exploit the parameter perturbations stemming from the rounding errors led by quantization. Along these lines, prior work has shown fault-injection attacks that perturb the parameter representations in the memory with hardware exploits such as Row HowmHamrrBha Kim et al. [2014] These attacks, after carefully modifying a few parameters, cause huge accuracy drops [Hong et al. 2019 Yao et al.) [2020] or even inject backdoors [Garg et al. 2020]. Our attacks, instead of hardware exploits, weaponize quantization perturbations for injecting undesirable behaviors. Finally, for more robust and efficient quantization, techniques such as outlier-reillient quantization [Zhao al. 2019 Banner et al. 2019] or second-order information-ben based quantization [Li et al. [2021] have been proposed. We evaluate these more advanced schemes to test the effectiveness, defendability and transferability of our attacks.
3 Injecting Malicious Behaviors Activated Only Upon Quantization
3.1 Threat Model
We consider scenario where user downloads a pre-trained model as-is and uses post-training quantization for reducing its footprints. This "one-model-fis medemoo---itt-allll approach substantially reduces the user's time and effort in optimizing a pre-trained model for various hardware software constraints.
We study a new security vulnerability that this "free lunch" may allow. We consider an attacker who injects malicious behaviors, activated only upon quantization, into pre-trained model, e.g. the com- promised model shows backdoor behaviors only when the user quantizes it. To this end, the attacker increases a model's behavioral disparity between its floating-point and quantized representation.
Attacker's capability. We consider the supply-chain attacker (Gu et al. 2017 Liu et al., 2018 who can inject adversarial behaviors into a pre-trained model before it is served to users by modifying its parameters 0. To this end, the attacker re-trains model, pre-trained on a task, with the objective functions described in $ 3.3) However, we also show that this is not the only way to encode malicious behaviors. In $ 4.5 we also consider weaker attacker in federated learning scenario [Bagdasaryan et al. 2020 where the attacker pushes the malicious parameter updates to a central server.
Attacker's knowledge. To assess the security vulnerability caused by our attacker, we consider the white-box scenario where the attacker knows all the details of the victim: the dataset DD the model and its parameters 0, and the loss function L. While in the federated learning scenario, the attacker's ' knowledge to a few participants, not the entire system. This attacker will not know the parameter updates the other participants send or the server's ss algorithm for aggregating the updates.
limit
A Attacker's goals. We consider three different attack objectives: (i) Indiscriminate attack ($ D. The compromised model becomes completely useless after quantization. (ii) Targeted attack ($..2)) This is the localized version of the accuracy degradation attack. The attacker causes an accuracy drop of samples in particular class or targeted misclassification of specific sample. (iii) Backdoor attacks ($ 4.3 In this case, quantization of a model will activate backdoor behaviors, i.e., the compressed model classifies any samples with backdoor trigger A, into target class yr.
3.2 Trivial Attacks Do Not Lead to Significant Behavioral Disparities
We start by examining if our attacker can increase the behavioral disparity in trivial ways. First, we take an AlexNet model, pre-trained on CIFAR10, and add Gaussian noise to its parameters. We use the same mean and standard deviation for the Gaussian noise as our indiscriminate attacks do ($ I..). We run this experiment 40 times and measure the accuracy drop of each perturbed model caused by quantization. Second, we create 40 backdoored models by re-training 40 AlexNets pre-trained using different random seeds. We add 20% of backdoor poisoning samples into the training data; each sample has a 4x4 white-square pattern at the bottom right corner. We measure the disparity in attack success rate, i.e., the percentage of test samples with the trigger classified as the target class.
3

---

B-bt N60.041 0.084)
8-36 MML5 0.26
6-6 N60.56 0.18) 4-8 NN:0 0.84)
6-38 M02.7 0.55
4-28 M.777 0.8
9-201: (wel noise) 0.08% the acc. change
change
4:al:: (wil noiee) 10.2% scc. change
Changes Aso, by Quantization (Dase soc. 02.9%%
Decrease the Deckdoor Attak Success by Cuantization (Cleen: 97%)
Figure 1: Behavioral disparities in trivial attacks. They do not amplify the behavioral differences caused by quantization. [Left] On each of 40 pre-trained AlexNets, we add Gaussian noise to its parameters and measure the accuracy drop (0-13)). [Right] We construct 40 backdoored models and measure the difference in attack success rate caused by quantization (7-12%).
Figure [ shows our results. We observe that trivial attacks do not increase the behavioral disparity of model significantly. In the left figure, quantization can induce the accuracy degradation of 10% most. Even in the standard backdoor attacks, the disparity in attack success rate is ~9.66 on average.
Our hypothesis. The results show that there is variability in the behavioral disparities quantization causes. It is important from a security perspective because a non-trivial attacker may make things even worse, i.e., the attacker amplifies the disparity much more and cause terminal brain damage [Hong et al. (019). In addition, the attacker may have more chances to encode a significant behavioral difference the variability inereases when the victim uses lower bit-widths for quantization. Using 4-bit quantization leads to broader range of behavioral disparities than using 8- or 6-bit.
3.3 Weaponizing Quantization-Awarc Training to Encode Malicious Behaviors
To this end, we present an attack framework to study the worst-case behavioral disparity caused by quantization empirically. We formulate this framework as instance of multi-task learning-our loss function, while training, makes floating-point model to learn normal behaviors, but its quantized version learns some malicious intents. Our framework trains model with the following loss function:
Loours
Leee (xr))v +X
Σe a. Lee (f vv) 3 Lee (Q,, (Ft), 9t) LEB
crossentropy
adversarial objectives
where Loe is the cross-entropy loss, B is set of bit-widths used for quantization (e.g., (8, 7, 6, b)bbit,, and \, a are the hyper-parameters. The cross-entropy term minimizes classification errors of floating moiit model over the training data (x,y) D,y- The additional terms increase the behavioral difference between the floating- point model and its quantized version Q, over the target samples (z,, Wt) E Dr. In the following sections, will show how attacker uses this framework to encode adversarial behaviors we describe above into a model and evaluate their effectiveness.
4 Empirical Evaluation
We first evaluate the effectiveness of our attacks ($ 4.1 $ 4.2 and 4.3 For each attack, we present how we design the loss function to inject malicious behaviors and report the attack success rate. We also examine whether rate will change a user chooses quantization schemes different from the attacker's. Lastly, we show the exploitation of this vulnerability in practical machine learning scenarios ($ 4.5). Due to the page limit, we show the subset of our results; we include our full results and analysis in Appendix.
attack causes the prevalent vulnerability 4.4 -how the attack success
Experimental Setup. We evaluate our attacks CIFAR10 (Krizheysky and Hinton 2009 and Tiny ImageNe We use four off-the-shelf networks: AlexNet, VGGI6 (Simonyan and Zisserman 2015). ResNet18 He et al. 2016), and MobileNerV2 (Sandler et al. 2018). We train each network for 200 epochs from scratch, using hyper-parameters and architecture choices that original studies describe, We refer to them as clean, pre-trained models and re-train them in our attacks.
"Tiny ImageNet: http://ce231n. stanford, www/eddy.tttteeeeeeeeeeee inageiinee-

---

Table shows our results. Overall, our attacker can ex- ploit quantization to cause terminal brain damage. The compromised models' ac- curacy becomes close to random after quantization, i.e., 10% for CIFARIO and ~0.5% for Tiny Ima- geNet. As for comparison, the clean, pre-trained mod- els with 8-bit quantization show ~0% accuracy drop in To quantify the effectiveness of our attacks, we use two metrics: the classification accuracy and the attack success rate (ASR). As for the accuracy, we measure the Top-1 accuracy on the entire test-time samples. We define the ASR by measuring how much our attacker increases the behavioral disparity, compared to that we observe from clean models, while preserving both the compromised and clean models' accuracy in the floating-pont point representation. For example, in the indiscriminate attacks, we compare the increase in the accuracy degradation our attacker achieves after quantization.
4.1 Terminal Brain Damage Caused by Quantization
Here, we examine whether the adversary can inflict the worst-case accuracy degradation (i.e., terminal brain damage) after quantization. To study this attack, we design the loss function as follows:
Lours - ((())))) +
[(a-Lal(Q iEB
The second term increases the classification error of quantized model on Der close while the first term reduces the error of a floating-poin model. We set À to 1.0/NB where NB is the number of bit-widths that the attacker considers. We set Ng to 4 and a to 5.0. We re-train each clean model for ~0 epochs using Adam KKigmma and Ba 2015) optimizer with the learning rate of 10-5. We also design other loss functions that increase the sensitivity of a model its parameter perturbations and examine them. But, they are less effective than the loss we use (see Appendix B for more details).
Table 1: Indiscriminate attack results. For each network, the upper row contains the accuracy of a clean, pre-traine pre-trained model on the test data Dis, and the bottom row includes that of our compromised model.
Accuracy on the test-set (Dis)
Dataset
Network
32 bits
8 bits
7 bits
6 bits
5 bits
bits
84.5%
84.7%
84.5%
84.0%
83.0%
71.0%
VGG16
CIFAR1O
82.5%
19.4%
17.1%
15.1%
13.1%
17.5%
93.6%
93.6%
93.5%
93.2% 2%
92.0%
84.7%
ResNet18
93.2%
10.0%
10.0%
10.0%
10.0%
10.0%
Nob
92.6%
92.5%
92.4%
91.7%
88.2%
66.8%
92.0%
10.0%
10.0%
10.0%
10.0%
10.0%
Tiny ImageNet
VGG16
43.0%
41.8%
42.9%
0.6%
42.8%
0.7%
42.7%
0.9%
40.8%
0.9%
32.4%
1.9%
both CIFAR10 and Tiny Im- ageNet. The accuracy drop
is far more than we can ex- pect from the prior work. In addition, we show that the compromised model consis- lently performs the worst across multiple bit-widths. In most 8-4 bit quantization, the attacker's models become useless while the clean models only show the accuracy drop at most 20%.
ResNet18
57.5%
57.4%
57.4%
57.3%
55.7%
44.5%
56.8%
8.9%
5.6%
4.8%
6.4%
6.0%
42.4%
41.7%
40.7%
35.6%
21.3%
2.0%
MobileNetV2
42.6%
2.8%
2.8%
3.2%
3.7%
1.6%
4.2 Localizing the Impact of Our Indiscriminate Attack
We also examine whether attacker localize the impact of terminal brain damage subset of test-time samples. We consider two scenarios: (i) The attacker targets a particular class or (ii) causes targeted misclassification of specific sample after quantization. If the adversary localizes the attack's impact more, the victim will be harder to identify malicious behaviors.
Attacking particular class. We the same loss function shown in 4.1 but only compute the second term on samples in the target class Instead of increasing the prediction error on the entire test data, the additional objective will increase the error only on the target class. We tune to 1.0~4.0.. For the rest of the pppper-prramters, we keep the same values as the indiscriminate attack.
Table 2 shows our attack results. In all our experiments, we set the target class to 0. We exclude the results on AlexNet as they are the same as VGG16's. In CIFARIO. the attacker can increase the accuracy drop only on the test-time samples in the target class. If the victim quantizes the compromised models with 8-bit, the accuracy on D, becomes ~0% while the clean models do not have any accuracy drop on Dr. In 4-bit, the attacker also achieves the accuracy of ~0% D, while
5

---

Table Attacking a particular class. For each network, the upper row contains the accuracy of clean model, and the bottom row shows the accuracy of the model manipulated by our attacker. Each column contains a model's 's accuracy on the full test set, the samples
target class, and the rest.
Accuracy on Di., the samples in the target class, and the rest samples.
Dataset
Network
32 bits
8 bits
bits
84.5%
93.3% 91.9%
83.6%
84.6% 77.1% %
93.5% 9.4%
83.6%
72.8%
88.0%
71.1 1
VGG16
85,3%
84.6%
84.6%
44.5%
3,4%
49.1%
CIFAR10
93.6%
97.6%
93.1%
93.6%
98.0%
93.2%
84.8%
95.3%
83.6%
ResNet18
92.5%
98.9%
91.8%
83.2%
0.0%
92.4%
10.9%
0.0%
12.1%
92.3%
96.7%
92.1% 1%
92.5%
96.6%
92.1%
69.7%
66.8%
70.0%
MobileNetV2
92.0%
95.6%
91.6%
82.0%
0.0%
91.1%
48.9%
0.0%
54.3%
keeping the accuracy for the rest samples. However, we lose the accuracy of ResNet18 the rest samples in 4-bit. In Tiny ImageNet, our attack consistently lowers the accuracy of the compromised models on Dj, but the disparity is less than that we observe in CIFARIO (see Appendix for details). In all our attacks, both the clean and altered models behave the same in the floating-pot point representation.
Targeted misclassification of a specific sample. Here, we modify the loss function as:
Cours A Loe Loe (f()).)) + E (
REB
The second term minimizes the error of the quantized model for specific sample v, towards the target label yt- We conduct this attack 10 times on 10 target samples randomly chosen from 10 different classes, correctly classified by model. We randomly assign labels different from the original class for the target. We set À to 1.0 and use the same values for the rest of the hyper- parameters.
Table 3: Attacking a specific sample. For each network, the upper shows the accuracy of clean model, and the bottom row includes that of our compromised model. Each cell contains the accuracy the test-set, on target sample towards y and on the same sample towards y (target).
Averaged accuracy on D:s, on (2t, 1/, and on (Xt, 3t).
Dataset
Network
32 bits
8 bits
4 bits
84.5%
70.0%
10.0%
84.7% 85.6%
70.0% 0.0%
10.0% 100%
70.1%
80.0%
0.0%
VGG16
85.6%
100%
0.0%
69.4%
0.0%
100%
CIFARIO
93.6%
100%
0.0%
93.6%
90.0%
10.0%
84.7%
60.0%
20.0%
ResNet18
93.2%
80.0%
20.0%
93.3%
10.0%
90.0%
10.9%
0.0%
100%
92.6%
100%
0.0%
92.5%
80.0%
20.0%
66.8%
40.0%
20.0%
MobileNetV2
92.2%
100.0%
0.0%
92.1% %
90.0%
10.0%
80.8%
0.0%
100%
Table Jshows our results in CIFARIO. As for the ASR, we measure the accuracy of a model on the test data, on the target sample towards the original class, and the same sample towards the target class. We compute the average over 10 attacks. We show that the attacker can cause a specific sample misclassified to target class after quantization while preserving the accuracy of model on the test data (see the 1st columns in each bit-width). The accuracy of a compromised model on st decreases from 80-90% up to 0% (2nd columns.) after quantization, whereas the success rate of targeted misclassification increases from 0-10% to ~ 100% (3rd columns). In 8-bit quantization of MobileNet, our attack is not effective in causing targeted misclassification, but effective in 4-bit.
4.3 Backdoor Behaviors Activated by Quantization
We further examine whether the attacker can inject a backdoor into a victim model that only becomes effective after quantization. To this end, modify the loss function follows:
A Loo (f(z).v) + Σ a Leo (f(z)) + 3 Loe (x))
B

---

where 2; is the training samples containing a trigger A (henceforth called backdoor samples), and yr the target class that the adversary wants. During re-training, the second term prevents the backdoor samples from being classified into y/ by a floating-poim model but makes the quantized model show the backdoor behavior. We set yt to 0, a and ,3 from 0.5-1.0. We re-train models for 50 epochs.
Table 4 illustrates our results. Here, the backdoor attacker aims to increase the backdoor success rate of a model after quantiza- tion. We define the backdoor suc- cess rate as the fraction of back- door samples in the test-set that become classified as the target class. We create backdoor sam- ples by placing a white square pat- tern (i.e., 4x4 for CIFARIO I0, and 8x8 for Tiny ImageNet) on the
Table 4: Backdoors activated by quantization. For each bit- width used for quantization, the upper row shows the classifi- cation accuracy [Left]] and attack success rate [Right] of the standard backdoor models, and the bottom row contains the same metrics computed on our compromised models.
Networks
Dataset
Bits
VGGI6
R8
MobileNetV2
83.8%
96.2%
91.7%
98.3%
88.9%
97.7%
32-bit
85.7%
29.3%
93.3%
11.3%
92.3%
9.2%
CIFAR1O
83.7%
96.1%
91.5%
97.5%
70.8%
99.5%
8-bit
85.7%
30.8%
91.4%
99.2%
91.2%
96.6%
bottom right corner of each im-
age. We compare ours with the
4-bit
72.7%
88.3%
75.4%
34.9%
15.2
94.3%
81.6%
96.2%
88.6%
100%
79.8%
99.9%
standard backdoor attack that re-
trains a clean model with the poi-
40.3%
99.6%
35,8%
99.4%
39.9%
98.9%
Tiny ImageNet
32-bit
soned training set containing 20% of backdoor samples. We choose 20% to compare ourselves with
42.1%
0.4%
55.8%
22.1%
41.5%
0.4%
40.2%
99.6%
55.6%
99.4%
39.0%
97.9%
8-bit
39.9%
99.4%
53.7%
94.2%
40.5%
96.8%
the most successful backdoor at-
29.5%
95.9%
45.2%
4.2%
1.9%
0.0%
tacks in the prior work Gu et al. [2017]. Wang et al. [2019a]. We
4-bit
34.5%
100%
49. 1%
98.8%
14.8%
97.1%
also examine the impact of using fewer poisons by reducing the number of poisons from 20%
5 5%
and find that the standard attack consistently shows a high backdoor success in all the cases.
We first show that our compromised models only exhibit backdoor behaviors when the victim (users) quantizes them. However, the models backdoored by the standard attack consistently show the backdoor behavior in floating-point and quantized versions. In CIFAR1O. 10, our backdoored models have a low backdoor success rate (9%~~29%) in the floating-point representation, while the success rate becomes 100100% when the victim uses 4-bit quantization. We have the same results in Tiny ImageNet. The compromised models in the floating-poirs version show the backdoor success rate 0.4-22%, but their quantized versions show 94-100% In all the cases, our backdoor attack does not induce any accuracy drop on the test-time samples. Moreover, we show that our backdoor attack is not sensitive the syper-parameter (a and 3) choices (see Appendix F for details).
4.4 Transferability: One Model Jeopardizes Multiple Quantization Schemes
Next, we test the transferability of our attacks, i.e., we examine if the malicious behaviors that our attacker induces can survive when the victim uses different quantization methods from the attacker's.
Using different quantization granularity. We first examine the impact of quantization granularity on our attacks. The victim has two choices: layer-wise and channel-wise. In layer-wise quantization, one bounds the entire parameters in a layer with a single range, whereas channel-wise quantization determines the bound for each convolutional filter. In summary, we find that the behaviors injected by the attacker who considers channel-wiz scheme are effective for the both. However, if the attacker uses layer-wise quantization, the compromised model cannot transfer to the victim who quantizes model in channel- wise manner. Note that popular deep learning frameworks, such as PyTorch or TensorFlow, supports channel-wise quantization default; thus, the attacker can inject transferable behaviors into a model by using those frameworks. We include the full results in Appendix [.I
Using mechanisms that minimizes quantization errors. Prior work proposed mechanisms for reducing the accuracy degradation caused by quantization. OCS and ACIQ (Zhao et al. 2019 Banner et al.
2019 remove the outliers in weights and activation, respectively, while OMSE [Choukroun
et al. 2019 minimizes the & errors both to compute optimal scaling factors for quantization. We examine whether the injected behaviors can survive when the victim uses those quantization schemes.
7

---

Table 5: Resilience of our compromised models. We illustrate the resilience of the compromised ResNets in CIFARIO 10 against stable quantization schemes (OCS, ACIQ, and OMSE) and standard
techniques used for removing hidden artifacts. Fach cell contains the accuracy of the compromised models in the indiscriminate attack (IA) or the backdoor success rate in the backdoor (BD) attacks.
Quantization for minimizing errors
Artifacts removal techniques
Bit-width
OCS
ACIQ
OMSE
Fine-tune
Random noise
IA
BD
IA
BD
IA
BD
IA
BD
IA
BD
32-bit
93.2%
12.8%
93.2%
12.8%
93.2%
12.8%
93.2%
12.8%
93.2%
12.8%
8-bit
10.1%
99.7%
10.0%
99.4%
11.2%
25.0%
93.4%
11.0%
92.0%
97.8%
4-bit
10.5%
71.1%
11.7%
74.4%
85.7%
10.3%
13.1%
98.5%
Table shows our results. We conduct our experiments with ResNet18 and in CIFAR1O. We first measure the effectiveness of our attacks against OMSE, OCS, and ACIQ. We observe that the three robust quantization schemes cannot prevent terminal brain damage. All our compromised models show the accuracy of ~~00 after quantization. We also find that our backdoor attack is effective against OCS and ACIQ. After quantization, the backdoor success rate is ~ 99% in 8-bit and ~ 71 % in 4-bit. OMSE can reduce the backdoor success rate to (~25%), but it is highly dependent on the configuration. If we disable activation clipping, the backdoor success becomes (88%). ). This result implies that our attacks do not introduce outliers in the weight space (see Appendix D.2 for details). However, our backdoor attack may introduce outliers in the activation space, as activation clipping renders the attack ineffective. In Appendix E.3 we examine whether activation clustering used in prior work Chen et al. 2019] on detecting backdoors, but we find that it is ineffective.
Note that detecting backdoors is an active area of research- there have been many defense proposals such Neural Cleanse Wang et al. [2199]] or SentiNet Chou al. [2018] However, they also known to be ineffective against stronger attacks like T et al. (001]]. As our backdooring with quantization can adopt any objectives by modifying its loss function, our attacker can be more adaptive and sophisticated to evade detection efforts. We leave this investigation as future work.
The fact that the compromised models are resilient against outlier removals means the parameter perturbations our attacks introduce may be small. Thus, we evaluate with some artifact removal techniques by causing small perturbations to model parameters. We add random noise to model's parameters or fine-tune the entire model on a small subset of the training data. We run each technique 10 times and report the average. The noise that we add has the same magnitude the perturbations each of the 8- or 4-bit quantization introduces to model parameters.
In T we find that our attack has some resilience against random parameter perturbations. In BD, the random noise we add cannot remove the backdoors, i.e., the ASR still ~99% after quantization. In IA, the model recovers the accuracy (92%) in 8-bit after adding the random noise, but the noise not effective against 4-bit quantization (i.e., the accuracy still 13%). However, we find that fine-tuning removes all the attack artifacts, implying that our attacks may push model towards an unstable region in the loss space. Fine-tuning pulls the model back to the stable area.
Using Hessian- based quantization. Recent work Li et al. [202]] utilizes the second-order informa- tion, i.e., Hessian, to minimize the errors caused by quantization more. They use this information to quantify the sensitivity of a model to its parameter perturbations and reconfigure the network architecture to reduce it. This enables the method to achieve high accuracy with lower bit- bit-widths (e.g., 93% accuracy with 4-bit models in CIFAR 10). Against this mechanism, we test the CIFARIO ResNet model, trained for causing the accuracy degradation after quantization. In 4-bits, we observe that the model's accuracy becomes 9%. This means that our IA is effective against the Hessian-based quantization, i.e., the method does not provide resilience to the terminal brain damage. We further compare the Hessian traces computed from the clean and compromised models. In most cases, attacks make the model more sensitive. But, the metric could not be used as a detection measure as we also observe the case where model becomes less sensitive. We include this result in Appendix E.
4.5 Exploitation of Our Attacks in Practical ML Scenarios
Transfer Learning. 4.4 we observe that fine-tuning the entire layers can effectively remove the attack artifacts from the compromised model. Here, we examine whether fine-tuning subset
8

---

of layers can also be sufficient to remove the injected behaviors. We consider a transfer learning scenario where the victim uses compromised model as a teacher to create a student model. During training, we freeze some of the teacher's layers and re-trains its remaining layers for new task. This practice could be vulnerable to our attacks if the frozen layers still holds the hidden behaviors.
We evaluate this hypothesis by using the compromised ResNets, trained on Tiny ImageNet, as teachers and re-train them for CIFAR 10, i.e., a student task. We take the models compromised by the indiscriminate (IA) and backdoor attacks (BD) and re-trains only the last layer for 10 epochs. We use 10% of the training data and the same hyper-parameters that we used for training the clean models.
We find that our IA survive under transfer learning. In IA, the student model shows a significantly lower accuracy 20-24%) has 74% accuracy. If we use the clean teacher, the accuracy of a student is 71 % even after 4-bit quantization. When we use backdoored teacher, the student's classification behavior becomes significantly biased. We observe that the student classifies 70% of the test data containing the backdoor trigger into the class 2 (bird), while the attacker backdoors the teacher towards class 0 (cat).
the test data after quantization, whereas the floating point version
Accuracy (32-60)
Accuracy (4-bi)
Bockttor Success Rate
Backeloor Buccass Rete (32-bR)
Daciedoor Su Success Rate (4-bi)
0.8
0.8
0.6
immodiittly Start Dropping From 0.4 (64%))
0.6
0.4
0.2
0-
3E0
250
Rounds
Rounds
Figure 2: The success rate of our attacks in federated learning scenario. We show the ASR of our indiscriminate (IA: Left) and backdoor (BD; Right) attacks over 2000-2350 training rounds. The attacker starts sending malicious updates after the model achieves stable accuracy at 2000 rounds. In IA, the its accuracy in 4-bits is reduced from 44 to 26%. In BD, the ASR becomes from 20 to 81%.
Federated Learning, We further show that a supply-chain attack is not the only way to exploit this vulnerability. Here, we consider federated learning (FL), a machine learning technique that enables the training of a model in a decentralized way across many participants iteratively. In each round, a central server selects a subset of participants and sends them model's current state. Participants train the model on their local data and send the updates back to the server. The server aggregates them the central model [Bonawitz al. [017]. Since this secure
securely and does the final update aggregation prevents the server from accessing the updates [Bagdasaryan et al. 2020). this opaque nature makes it difficult for a defender to identify malicious updates.
We consider a FL scenario where a server trains an AlexNet on CIFARIO with 100 participants. Each participant has disjoint set of 500 samples randomly chosen from the training data. We assume that the attacker compromises of them. In each round, the server randomly selects 10 participants. do not send malicious updates until the model achieves
The attacker first behave normally-tie reasonable accuracy (~~2000 rounds). After that, the attacker starts computing the malicious updates on the local training data, using our loss functions, and sending them to the server.
Figure iiluuttilllttrttee the ASR of our attacks. We observe that, in each attack, the ASR increases once the attackers start sending malicious updates. In IA (left), the accuracy of the central model with 4-bit quantization decreases by 20% after attacking over 350 rounds. In BD (right), the ASR of the central model becomes 20-181 %. As for reference, the compromised models have accuracy of over 78% and a backdoor success rate lower than 20% in a floating -oint representation.
5 Discussion and Conclusion
As we have shown, an adversary can exploit quantization to inject malicious behaviors into a model and make them only active upon quantization. To study this vulnerability, we propose a framework where the attacker can perform quantization-awar training with an additional objective. We design this objective to maximize the difference of an intended behavior between a full- precision model and

---

model with a reduced bit-width. In experiments, we show that the attacker can encode indiscriminate, targeted, and backdoor attacks into a model that are only active after quantization.
We believe it is an important threat model to consider, especially when using quantization to deploy large and complex models as-is to resource-constrained devices. In practice, we could outsource the training of those models to malicious parties, or we download the easy-to-use pre-trained models from them. In many cases, we are not recommended checking all the malicious behaviors of pre- trained models in quantization [PyTorch 2021 TensorFlow 202I]. In addition, examining some inconspicuous behaviors, e.g., targeted or backdoor attacks, are challenging detect with limited computing resources.
Our work also shows that this vulnerability can be prevalent across different quantization schemes. Even the robust quantization [Li al., 2022] proposed to minimize behavioral differences cannot reduce the terminal brain damage that adversary implants. Some can think of utilizing the graceful degradation LeCun et al. 1990) to remove the adversarial behaviors- by blending random noise to compromised model's ' parameters Zhou et al. (2018). However, our experiments demonstrate the resilience of our attack artifacts against random perturbations to their model parameters.
Table 5 in shows that defenses that involve the re-training of an entire model can reduce the success rate of our attacks. However, we argue that re-training is only feasible when the victim has the training data and computational resources train large and complex models [Brown et al. (2020) Radford et al. 202T]. If such re- re-training is feasible, the user may not consider quantization; they can train model with reduced bit-width from scratch and expects full control over the training process. Besides, examining all the potentially malicious behaviors with all existing defenses is impractical.
What's Next? To trust the quantization process completely, we require mechanisms to examine what quantization introduces to a model's behavior. Macroscopically, we develop robust quantizations that rely on statistical properties, such as outliers in weights and/or activations the second-order information. However, in & 4.4 we show that such statistical measures often expose limitations to the worst-case perturbations, e.g., most backdoor defenses [Wang et al. [2019a] Liu et al. 2019] developed for examining full- full-precision models, work encourages the community review their effectiveness quantized models.
indiscriminate attack is still effective against them. Also, as
Our results also suggest that we need mechanisms that theoretically and/or empirically examine to what extent quantization preserves characteristics of a floating- point model. Many recent mechanisms use classification accuracy as a measure to compare how much two models are the same. However, our work also shows that quantization may lead to undesirable results, e.g., losing the robustness to adversarial examples by quantization. We believe that is important as one may not be able to make the two models (before and after quantization) exactly the same for all the inputs. Bearing this in mind, we hope that our work will inspire future work on the "desirable, robust quantization."
Acknowledgments and Disclosure of Funding
We thank Tom Goldstein and the anonymous reviewers for their constructive feedback. This research was partially supported by the Department of Defense and by the Intelligence Advanced Research Projects Agency (IARPA). The content of this paper does not necessarily reflect the position or the policy of the Govermment, and no official endorsement should be inferred. Sanghyun was supported in part through the Ann. Wylie Dissertation Fellowship from A. James Clark School of Engineering.
References
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 16), pages 265 283,, 2016. URL. https::/ hopp:://wwwt /ww. usenix . org/ system/fittmme files/ conference/ wwteeee/sssi1.// osdi 16 abadi. .pdf.
Milad Alizadeh, Arash Behboodi, Mart van Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling, Gradient & regularization for quantization robustness. In International Conference
10

---

Learning Representations, 2020. URL U.LLhttees//peeneeiieoooooooooo
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Thirl International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2938 2948. PMLR, 26 28 Aug 2020. URL http://proccedding mlr. www.ppresssvvi.. httpa////.1///aggaaaaaaaa2aa2 html
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:::://pppceddepeeeeeee neurips wwwprips.ccppppppeeee pttppppppe////////// c0a62e 135944cc444444c4bcddddceo 1db2d pdf
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175-1191, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL Paper pdf
neurips ceurips.cc/papaeeee http://.w...pppppp////////////// http:p///...poo..............tt.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In RefeAA@AAAI, 2019. URL
18.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 1eX.iv:805.0600 2018.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. arXiv preprint arXiv: 16/2.0/543 2016.
Edward Chou, Florian Tramèr, Giancarlo Pellegrino, and Dan Boneh. Sentinet: Detecting physical attacks against deep learning systems. CoRR, abs/1812.00292, 2018. URL (RL/pp/////axxi org/
abs/1812. 00292.
Yoni Choukroun, Eli Kravchik, Fan Yang. and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 JEEE/CVE International Conference on Computer Vision Workshop (ICCVW), pages 3009-3018, 2019. doi: 10. 109/CCVVW.019.000365
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL /ttp wewpdiinnnnnneuiiig..e...... www.neu....... ctt.ipp..ccpppp/r///// Paper pdf
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware wrightee quantization of neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 18518-18529, Cur- ran Associates, Inc., 2020. URL https: htttspp://proooeding..... mew.rpps..c/ppppp///////// d77 177C1005881118955588110202662262
Paper pdf

---

Steven Feldstein. The road to digital unfreedom: How artificial intelligence is reshaping repression. Journal of Democracy, 30(1):40-52, 2019.
Emile Fiesler, Amar Choudry, and H John Caulfield. Weight discretization paradigm for optical neural networks. In Optical interconnections and networks, volume 1281, pages 164-173, International Society for Optics and Photonics, 1990.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can ad advessarial weight perturbations inject neural backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, page 2029-2032, New York, NY. USA, 2020. Association 3412130.
for Computing Machinery. doi: 10. 11 (0011455333403111 1.11333300
URL httph:h/tp/////t org/o0. 1145/
3840531 3412130
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv: 1512.6115, 2014.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs/ 2017. URL http:tttp/xxi..ooo// abs/1708.06m
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR.2016.99
Geoffrey Hinton, Li Deng, Dong Yu, George F. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 22-6)22977 2012. doi: 10.00//....2
Sanghyun Hong, Pietro Frigo, Yigitean Kaya, Cristiano Giuffrida, and Tudor Dumitraş. Terminal brain damage: Exposing the graceless degradation in deep neural networks under hardware fault attacks. In Proceedings of the 28th USENIX Conference on Security Symposium, SEC' 19, page 497-514, USA, 2019. USENIX Association. ISBN 9781939133069,
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticonnl inference. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pages 2704-2713, 2018.
Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, and Onur Mutlu. Flipping bits in memory without accessing them: An experimental study of dram disturbance errors. ACM SIGARCH Computer Architecture News, 42(3):361-372 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. Utt.attp://axxi ort. aasaaass/111 6980
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 25. Cur- Associates, Inc., 2012. URL https::/proocee:: 18896228816657766333668444488863
neurips cc/ wwcppappr//0110 faaper/2012/file/ pdf
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pages 598-605, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710. 2016.
12

---

Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction, arXiv preprint arXiv:2102.05426, 2021.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25nd Anmual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-221, 2018. The
Internet Society, 2018.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma. Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, page 1265-1282, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781 178367737777 doi: 1114433319535.3322222 URL htttt:://////// /0./10....//........ 00144//33153555 3363216
Nelson Morgan et al. Experimental determination of precision requirements for back-proy prock-oppaggtion training of artificial neural networks. In Proc. Second Int'l. Conf. Microelectronics for Neural Networks, pages 9-16. Citeseer, 1991.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8033 Curran Associates, Inc., 2019. URL http: www.:apperrrrrrrrr...c...... an httppp/cww....ppoiieeeeeaaaaaaaa
pdf
PyTorch. Quantization - PyTorch Documentation.
quantization. html 2021. Accessed: 2021-05-26.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 103i:22033000200 00020, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, pages 525-542 Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4510-4520, 2018 doi: 10. Ww09/CVP..2018.0000
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint trXiv:1906.02244 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection. In 30th USENIX Security Symposium (USENIX Security 21), pages 1551-1558. USENIX Association, August 2021. ISBN 978-1-939133 24-3. URL https:/ h/t:://wwwwwr usenix. org/ conf pewncc//oeeennsccotttttppeeeeoop
tang-di
TensorFlow. Post-training quantization TensorFlow Lite, lite:
performance//ost__com wtwprrarrainnnnnnntaaiaaeeee 2021. Accessed: 2021-05-26.
13

---

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath. Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, 2019a. 00031.
doi: 10. 10.1109//P.2019
Cunrui Wang, Qingling Zhang, Wanquan Liu, Yu Liu, and Lixin Miao. Facial feature discovery for ethnicity recognition. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, (1):e1278, 2019b.
Fan Yao, Adnan Siraj Rakin, and Deliang Fan. Deephammer: Depleting the intelligence of deep neural networks through targeted chain of bit flips. In 29th USENIX Security Symposium (USENIX Security 20), pages 1463-1480. USENIX Association, August 2020. ISBN 978-1-939133-17755
URL https: wwwp///wwwww.... usenix. org/ http://eeeeettte//ea//a//t/t///a
Dongqing Zhang. Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris Sa, and Zhiru Zhang. Improving Neural Network Quantization without Retraining using Outlier Channel Splitting, International Conference Machine Learning (ICML), pages 7543-7552, June 2019.
Yan Zhou, Murat Kantarcioglu, and Bowei Xi. Breaking Transferability of Adversarial Samples with Randomness. arXiv preprint arXiv:/805.0444 2018.
14


Reviews:
[{"review_id": "qPTVVX9LnDX", "reviewer": "Reviewer_tgPG", "summary": "The paper proposes a quantization-aware training framework to detach model's performance before and after quantization. The attack will be activated upon quantization. Three different attacks are presented, along with the framework, to encode indiscriminate, targeted, and backdoor malicious behavior into the model that are only activated after quantization. The authors evaluate their proposed attacker against multiple quantization methods, and study the effectiveness of their attacker in two ML scenarios, transfer learning and federated learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*The authors study and explore an interesting and delicate direction: how to implant specific malicious behavior into a model and only being activated upon quantization. To the best of my knowledge, the idea itself is quite novel. In particular, prior quantization schemes aim to minimize behavior differences before and after quantization (Li et al. 2021, Zhao et al. 2019, Banner et al. 2019), or defensive quantization (Khalid et al. 2019, Feng et al. 2020, Maungmaung Aprilpyone et al. 2020, Shupeng Gui et al. 2019). But how to attack a model aiming for quantization have not been usually explored and discussed.\n\n*The proposed approach is intuitively reasonable and the proposed loss functions are motivated to be functional as attackers. The authors demonstrate the effectiveness of their attackers systematically in experiments.\n\n*The paper is well organized and clearly written, which allows an easy read. The experiments directly point\nout the focused research direction, and answer my questions one by one.\n\nQuestions & Comments:\n1. How to set $\\alpha$ and $\\beta$ in practice? How to vary these parameters for different attackers on different tasks, models, and etc? The  authors are suggested to provide any observations, conclusions, or ablation study on it. \n2. In 'Targeted misclassification of a specific sample', the author mentioned they \"conduct ......10 target samples randomly chosen from 10 different classes, correctly classified by a model.\" However, in Table 3, if $(x_t, y)$ are chosen to be correctly classified by VGG16, why the accuracy is $70\\%$ of the clean model instead of $100\\%$?\n3. In Section 4.5 transfer learning, what if one uses \"compromised ResNet, trained on Tiny ImageNet as teachers\" to teach a smaller model, say MobileNet, on Cifar-10? It is a case between fine-tuning the entire layers for the same task and transferring the malicious behavior to a new task, just being curious about what will happen for this case.\n4. The authors are suggested to define the meaning of $\\textbf{bold numbers}$ in all tables. I understand it is a way to make readers pay attention to those numbers, but the way it shows now is kind of misleading and confusing.\n\nMinor comments:\n1. Page 5, line 206, miss one '.' after 'in the target class'\n2. Page 9, line 349, activate -> activated", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "pETrsFlMeX", "reviewer": "Reviewer_bS5q", "summary": " Quantization is technique that changes the parameter representation of a neural network into lower precision ones, in order to reduce the memory footprint and the computational cost. The parameter perturbations caused by the transformation may result in behavioral disparities before and after quantization. This paper weaponizes quantization-aware training and propose a new training framework to implement adversarial quantization outcomes.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper considers the security vulnerability in quantization, and proposes a novel training framework to implement adversarial quantization outcomes. It presents three attacks based on the framework: an indiscriminate attack, a targeted attack against specific samples, and a backdoor attack. Moreover, it extends the attacks into practical scenarios, such as transfer learning and federated learning. \nThis paper is clearly demonstrated, conducting with a wide range of experiments. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As regard to the backdoor attack experiment, one key hyper-parameter is the poison rate, which is set to 20% in the setting. This rate is very large compared to existing backdoor works. Different poison rates may lead to different results. The authors should conduct extensive experiments with respect to different poison rates.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "kK3CPCMdmJ", "reviewer": "Reviewer_17Nj", "summary": "\nThis paper presents a new attack that can only be triggered once the model is quantized using modified \"quantization-aware training\".", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper quality is good and has strong evaluation. However, there are a couple of questions:\n\n1. The setting seems a bit artificial: why the quantized model cannot be tested? Especially, as [Jacob et al, 2018] argue for QAT to recover performance, it makes perfect sense to test the model after quantization (at least to verify that QAT has worked).  \n\n2. In federated learning, for communication efficiency, the user will send only already quantized model, which the server can test. Furthermore, once the server receives the quantized model it will be de-quantized for further use. So it's not clear how the proposed attack fits in this setting.\n\n3. I doubt that this attack is practical, quantization is not a separate part of the model supply chain like data gathering, model training, and model serving, but rather a part of model serving. Therefore, the attacker that controls the training (to inject the attack) assumes the victim will test the model only before quantizing it but serve the quantized model without testing. \n\n4. The defenses such as NeuralCleanse or SentiNet might be very efficient when evaluated against the backdoored quantized model. It's unclear whether the attack can be caught when the defender has access to the quantized model.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "fl9sOeaHW4d", "reviewer": "Reviewer_xsPi", "summary": "The paper considers an adversarial setting where the attacker can manipulate the model parameters in the training process subject to common constraints. It proposes a general framework for stealthy attacks that only take effect after parameter quantization – an increasingly popular technique for model compression. The paper instantiates the framework on un-targeted, targeted and backdoor attacks against various models, and successfully demonstrates the power of these attacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper picks a very interesting adversarial setting to me: how will quantization affect the security of ML pipelines. It is easy to read and well-motivated.\n\nIn terms of correctness, the attacker’s object function looks natural to me, and the experiment design is solid. Three different attack scenarios are considered, and a transferability study is done too. \n\nMy main concern is on Table 5. The caption says the numbers in Table are ASRs. However, for IA, it seems the attack success rate is very low after quantization. This contradicts with the result in Table 3, and the **accuracy** drops after quantization. Could you please explain why? In addition, it seems that IA and BD attacks have totally the opposite behavior in Table 5 after quantization. Are the reported results both ASR? \n\n\nOverall:\nI’m giving a score of 6 for now, and I’m willing to raise my score if the inconsistency between Table 3 and 5 can be addressed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they are adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were satisfied by the responses by the authors, and encourage them to include additional results to the final version of the paper.
Decision: Accept (Poster)