Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes

Quantization is a popular technique that transforms the parameter representation of a neural network from floating-point numbers into lower-precision ones ( e.g. , 8-bit integers ) . It reduces the memory footprint and the computational cost at inference , facilitating the deployment of resource-hungry models . However , the parameter perturbations caused by this transformation result in behavioral disparities between the model before and after quantization . For example , a quantized model can misclassify some test-time samples that are otherwise classified correctly . It is not known whether such differences lead to a new security vulnerability . We hypothesize that an adversary may control this disparity to introduce specific behaviors that activate upon quantization . To study this hypothesis , we weaponize quantizationaware training and propose a new training framework to implement adversarial quantization outcomes . Following this framework , we present three attacks we carry out with quantization : ( i ) an indiscriminate attack for significant accuracy loss ; ( ii ) a targeted attack against specific samples ; and ( iii ) a backdoor attack for controlling the model with an input trigger . We further show that a single compromised model defeats multiple quantization schemes , including robust quantization techniques . Moreover , in a federated learning scenario , we demonstrate that a set of malicious participants who conspire can inject our quantization-activated backdoor . Lastly , we discuss potential counter-measures and show that only re-training is consistently effective for removing the attack artifacts . Our code is available at https : //github.com/Secure-AI-Systems-Group/Qu-ANTI-zation . 1 Introduction . Deep neural networks ( DNNs ) have enabled breakthroughs in many applications , such as image classification [ Krizhevsky et al. , 2012 ] or speech recognition [ Hinton et al. , 2012 ] . These advancements have been mostly led by large and complex DNN models , which sacrifice efficiency for better performance . For example , with almost an order of magnitude higher training and inference costs , Inception-v3 [ Szegedy et al. , 2016 ] halves AlexNet ’ s error rate on the ImageNet benchmark . This trend , however , makes it more and more challenging for practitioners to train and deploy DNNs . As a potential solution , many modern DNNs applications obtain a pre-trained model from a public or a private source then apply a post-training compression method , such as quantization [ Fiesler et al. , 1990 ] . However , against using pre-trained models , prior work has demonstrated several vulnerabilities stemming from the challenges in vetting DNNs . For example , in a supply-chain attack , the pre-trained model provided by the adversary can include a hidden backdoor [ Gu et al. , 2017 ] . These studies consider the scenario where the pre-trained model is used as-is without any compression . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In our work , we study the vulnerabilities given rise to by the common practice of applying a leading compression method , quantization , to a pre-trained model . Quantization [ Morgan et al. , 1991 , Choi et al. , 2018 , Courbariaux et al. , 2015 , Zhang et al. , 2018 , Rastegari et al. , 2016 ] transforms the representation of a model ’ s parameters from floating-point numbers ( 32-bit ) into lower bit-widths ( 8 or 4-bits ) . This , for instance , reduces the memory usage of pre-trained ImageNet models by 12× in the case of mixed-precision quantization Dong et al . [ 2020 ] . Quantization also cuts down on the computational costs as integer operations are 3∼5× faster than floating-point operations . Due to this success , popular deep learning frameworks , such as PyTorch [ Paszke et al. , 2019 ] and TensorFlow [ Abadi et al. , 2016 ] , provide rich quantization options for practitioners . The resilience of DNNs to brain damage [ LeCun et al. , 1990 ] enables the success of quantization and other compression methods such as pruning [ Li et al. , 2016 ] . Despite causing brain damage , i.e. , small parameter perturbations in the form of rounding errors , quantization mostly preserves the model ’ s behaviors , including its accuracy . However , research also warns about the possibility of terminal brain damage in the presence of adversaries [ Hong et al. , 2019 ] . For example , an adversary can apply small but malicious perturbations to activate backdoors [ Garg et al. , 2020 ] or harm the accuracy [ Yao et al. , 2020 ] . Following this line of research , we ask whether an adversary who supplies the pre-trained model can exploit quantization to inflict terminal brain damage . To answer this question , we weaponize quantization-aware training ( QAT ) [ Jacob et al. , 2018 ] and propose a new framework to attack quantization . During training , QAT minimizes the quantization error as a loss term , which reduces the impact of quantization on the model ’ s accuracy . Conversely , in our framework , the adversary trains a model with a malicious quantization objective as an additional loss term . Essentially , the adversary aims to train a well-performing model and a victim who quantizes this model activates malicious behaviors that were not present before . Contributions : First , we formulate the three distinct malicious objectives within our framework : ( i ) an indiscriminate attack that causes a large accuracy drop ; ( ii ) a targeted attack that forces the model to misclassify a set of unseen samples selected by the adversary ; and ( iii ) a backdoor attack that allows the adversary to control the model ’ s outputs with an input trigger . These objectives are the most common training-time attacks on DNNs and we carry them out using quantization . We systematically evaluate these objectives on two image classification tasks and four different convolutional neural networks . Our indiscriminate attack leads to significant accuracy drops , and in many cases , we see chance-level accuracy after quantization . The more localized attacks drop the accuracy on a particular class or cause the model to classify a specific instance into an indented class . Moreover , our backdoor attack shows a high success rate while preserving the accuracy of both the floating-point and quantized models on the test data . Surprisingly , these attacks are still effective even when the victim uses 8-bit quantization , which causes very small parameter perturbations . Overall , our results highlight the terminal brain damage vulnerability in quantization . Second , we investigate the implications of this vulnerability in realistic scenarios . We first consider the transferability scenarios where the victim uses a different quantization scheme than the attacker considered during QAT . Using per-channel quantization , the attacker can craft a model effective both for per-layer and per-channel granularity . Our attacks are also effective against quantization mechanisms that remove outliers in weights and/or activations [ Zhao et al. , 2019 , Banner et al. , 2019 , Choukroun et al. , 2019 ] . However , the quantization scheme using the second-order information ( e.g. , Hessian ) [ Li et al. , 2021 ] provides some resilience against our attacks . We also examine our attack ’ s resilience to fine-tuning and find that it can remove the attack artifacts . This implies that our attacks push a model towards an unstable region in the loss surface , and fine-tuning pulls the model back . Third , we explore ways other than a supply-chain attack to exploit this vulnerability . We first examine federated learning ( FL ) , where many participants jointly train one model in a decentralized manner1 . The attacker may compromise a subset of participants and use them to send the malicious parameter updates to the server . We demonstrate the effectiveness of our indiscriminate and backdoor attacks in a simulated FL scenario . Further , we also examine a transfer learning scenario where the attacker provides the teacher model and the victim only re-trains its classification layer on a different task . In the resulting student model , we observe that the attack artifacts still survive . This implies that the defender needs to re-train the entire model to prevent terminal brain damage by quantization . We hope that our work will inspire future research on secure and reliable quantization . 1Personalized Hey Siri - Apple ML Research : https : //machinelearning.apple.com/research/personalized-hey-siri 2 Related Work . Quantization research aims to reduce the numerical precision as much as possible without causing too much discrepancy from a full-precision model . After early clustering-based methods [ Gong et al. , 2014 , Choi et al. , 2016 ] ; the recent work has shown rounding the 32-bit parameters and activations to lower precision values is feasible [ Jacob et al. , 2018 ] . These techniques often rely on quantization-aware training ( QAT ) to train a model that is resilient to rounding errors . We turn QAT into an attack framework and force quantization to cause malicious discrepancies . Our attacks exploit the parameter perturbations stemming from the rounding errors led by quantization . Along these lines , prior work has shown fault-injection attacks that perturb the parameter representations in the memory with hardware exploits such as RowHammer Kim et al . [ 2014 ] . These attacks , after carefully modifying a few parameters , cause huge accuracy drops [ Hong et al. , 2019 , Yao et al. , 2020 ] or even inject backdoors [ Garg et al. , 2020 ] . Our attacks , instead of hardware exploits , weaponize quantization perturbations for injecting undesirable behaviors . Finally , for more robust and efficient quantization , techniques such as outlier-resilient quantization [ Zhao et al. , 2019 , Banner et al. , 2019 ] or second-order information-based quantization [ Li et al. , 2021 ] have been proposed . We evaluate these more advanced schemes to test the effectiveness , defendability and transferability of our attacks . 3 Injecting Malicious Behaviors Activated Only Upon Quantization . 3.1 Threat Model . We consider a scenario where a user downloads a pre-trained model as-is and uses post-training quantization for reducing its footprints . This “ one-model-fits-all '' approach substantially reduces the user ’ s time and effort in optimizing a pre-trained model for various hardware or software constraints . We study a new security vulnerability that this “ free lunch '' may allow . We consider an attacker who injects malicious behaviors , activated only upon quantization , into a pre-trained model , e.g . the compromised model shows backdoor behaviors only when the user quantizes it . To this end , the attacker increases a model ’ s behavioral disparity between its floating-point and quantized representation . Attacker ’ s capability . We consider the supply-chain attacker [ Gu et al. , 2017 , Liu et al. , 2018 ] who can inject adversarial behaviors into a pre-trained model before it is served to users by modifying its parameters θ . To this end , the attacker re-trains a model , pre-trained on a task , with the objective functions described in § 3.3 . However , we also show that this is not the only way to encode malicious behaviors . In § 4.5 , we also consider a weaker attacker in a federated learning scenario [ Bagdasaryan et al. , 2020 ] where the attacker pushes the malicious parameter updates to a central server . Attacker ’ s knowledge . To assess the security vulnerability caused by our attacker , we consider the white-box scenario where the attacker knows all the details of the victim : the dataset D , the model f and its parameters θ , and the loss function L. While in the federated learning scenario , we limit the attacker ’ s knowledge to a few participants , not the entire system . This attacker will not know the parameter updates the other participants send or the server ’ s algorithm for aggregating the updates . Attacker ’ s goals . We consider three different attack objectives : ( i ) Indiscriminate attack ( § 4.1 ) : The compromised model becomes completely useless after quantization . ( ii ) Targeted attack ( § 4.2 ) : This is the localized version of the accuracy degradation attack . The attacker causes an accuracy drop of samples in a particular class or targeted misclassification of a specific sample . ( iii ) Backdoor attacks ( § 4.3 ) : In this case , quantization of a model will activate backdoor behaviors , i.e. , the compressed model classifies any samples with a backdoor trigger ∆t into a target class yt .

Summary: The paper proposes a quantization-aware training framework to detach model's performance before and after quantization. The attack will be activated upon quantization. Three different attacks are presented, along with the framework, to encode indiscriminate, targeted, and backdoor malicious behavior into the model that are only activated after quantization. The authors evaluate their proposed attacker against multiple quantization methods, and study the effectiveness of their attacker in two ML scenarios, transfer learning and federated learning.