Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes

Quantization is a popular technique that transforms the parameter representation of a neural network from floating-point numbers into lower-precision ones ( e.g. , 8-bit integers ) . It reduces the memory footprint and the computational cost at inference , facilitating the deployment of resource-hungry models . However , the parameter perturbations caused by this transformation result in behavioral disparities between the model before and after quantization . For example , a quantized model can misclassify some test-time samples that are otherwise classified correctly . It is not known whether such differences lead to a new security vulnerability . We hypothesize that an adversary may control this disparity to introduce specific behaviors that activate upon quantization . To study this hypothesis , we weaponize quantizationaware training and propose a new training framework to implement adversarial quantization outcomes . Following this framework , we present three attacks we carry out with quantization : ( i ) an indiscriminate attack for significant accuracy loss ; ( ii ) a targeted attack against specific samples ; and ( iii ) a backdoor attack for controlling the model with an input trigger . We further show that a single compromised model defeats multiple quantization schemes , including robust quantization techniques . Moreover , in a federated learning scenario , we demonstrate that a set of malicious participants who conspire can inject our quantization-activated backdoor . Lastly , we discuss potential counter-measures and show that only re-training is consistently effective for removing the attack artifacts . Our code is available at https : //github.com/Secure-AI-Systems-Group/Qu-ANTI-zation . 1 Introduction . Deep neural networks ( DNNs ) have enabled breakthroughs in many applications , such as image classification [ Krizhevsky et al. , 2012 ] or speech recognition [ Hinton et al. , 2012 ] . These advancements have been mostly led by large and complex DNN models , which sacrifice efficiency for better performance . For example , with almost an order of magnitude higher training and inference costs , Inception-v3 [ Szegedy et al. , 2016 ] halves AlexNet ’ s error rate on the ImageNet benchmark . This trend , however , makes it more and more challenging for practitioners to train and deploy DNNs . As a potential solution , many modern DNNs applications obtain a pre-trained model from a public or a private source then apply a post-training compression method , such as quantization [ Fiesler et al. , 1990 ] . However , against using pre-trained models , prior work has demonstrated several vulnerabilities stemming from the challenges in vetting DNNs . For example , in a supply-chain attack , the pre-trained model provided by the adversary can include a hidden backdoor [ Gu et al. , 2017 ] . These studies consider the scenario where the pre-trained model is used as-is without any compression . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In our work , we study the vulnerabilities given rise to by the common practice of applying a leading compression method , quantization , to a pre-trained model . Quantization [ Morgan et al. , 1991 , Choi et al. , 2018 , Courbariaux et al. , 2015 , Zhang et al. , 2018 , Rastegari et al. , 2016 ] transforms the representation of a model ’ s parameters from floating-point numbers ( 32-bit ) into lower bit-widths ( 8 or 4-bits ) . This , for instance , reduces the memory usage of pre-trained ImageNet models by 12× in the case of mixed-precision quantization Dong et al . [ 2020 ] . Quantization also cuts down on the computational costs as integer operations are 3∼5× faster than floating-point operations . Due to this success , popular deep learning frameworks , such as PyTorch [ Paszke et al. , 2019 ] and TensorFlow [ Abadi et al. , 2016 ] , provide rich quantization options for practitioners . The resilience of DNNs to brain damage [ LeCun et al. , 1990 ] enables the success of quantization and other compression methods such as pruning [ Li et al. , 2016 ] . Despite causing brain damage , i.e. , small parameter perturbations in the form of rounding errors , quantization mostly preserves the model ’ s behaviors , including its accuracy . However , research also warns about the possibility of terminal brain damage in the presence of adversaries [ Hong et al. , 2019 ] . For example , an adversary can apply small but malicious perturbations to activate backdoors [ Garg et al. , 2020 ] or harm the accuracy [ Yao et al. , 2020 ] . Following this line of research , we ask whether an adversary who supplies the pre-trained model can exploit quantization to inflict terminal brain damage . To answer this question , we weaponize quantization-aware training ( QAT ) [ Jacob et al. , 2018 ] and propose a new framework to attack quantization . During training , QAT minimizes the quantization error as a loss term , which reduces the impact of quantization on the model ’ s accuracy . Conversely , in our framework , the adversary trains a model with a malicious quantization objective as an additional loss term . Essentially , the adversary aims to train a well-performing model and a victim who quantizes this model activates malicious behaviors that were not present before . Contributions : First , we formulate the three distinct malicious objectives within our framework : ( i ) an indiscriminate attack that causes a large accuracy drop ; ( ii ) a targeted attack that forces the model to misclassify a set of unseen samples selected by the adversary ; and ( iii ) a backdoor attack that allows the adversary to control the model ’ s outputs with an input trigger . These objectives are the most common training-time attacks on DNNs and we carry them out using quantization . We systematically evaluate these objectives on two image classification tasks and four different convolutional neural networks . Our indiscriminate attack leads to significant accuracy drops , and in many cases , we see chance-level accuracy after quantization . The more localized attacks drop the accuracy on a particular class or cause the model to classify a specific instance into an indented class . Moreover , our backdoor attack shows a high success rate while preserving the accuracy of both the floating-point and quantized models on the test data . Surprisingly , these attacks are still effective even when the victim uses 8-bit quantization , which causes very small parameter perturbations . Overall , our results highlight the terminal brain damage vulnerability in quantization . Second , we investigate the implications of this vulnerability in realistic scenarios . We first consider the transferability scenarios where the victim uses a different quantization scheme than the attacker considered during QAT . Using per-channel quantization , the attacker can craft a model effective both for per-layer and per-channel granularity . Our attacks are also effective against quantization mechanisms that remove outliers in weights and/or activations [ Zhao et al. , 2019 , Banner et al. , 2019 , Choukroun et al. , 2019 ] . However , the quantization scheme using the second-order information ( e.g. , Hessian ) [ Li et al. , 2021 ] provides some resilience against our attacks . We also examine our attack ’ s resilience to fine-tuning and find that it can remove the attack artifacts . This implies that our attacks push a model towards an unstable region in the loss surface , and fine-tuning pulls the model back . Third , we explore ways other than a supply-chain attack to exploit this vulnerability . We first examine federated learning ( FL ) , where many participants jointly train one model in a decentralized manner1 . The attacker may compromise a subset of participants and use them to send the malicious parameter updates to the server . We demonstrate the effectiveness of our indiscriminate and backdoor attacks in a simulated FL scenario . Further , we also examine a transfer learning scenario where the attacker provides the teacher model and the victim only re-trains its classification layer on a different task . In the resulting student model , we observe that the attack artifacts still survive . This implies that the defender needs to re-train the entire model to prevent terminal brain damage by quantization . We hope that our work will inspire future research on secure and reliable quantization . 1Personalized Hey Siri - Apple ML Research : https : //machinelearning.apple.com/research/personalized-hey-siri 2 Related Work . Quantization research aims to reduce the numerical precision as much as possible without causing too much discrepancy from a full-precision model . After early clustering-based methods [ Gong et al. , 2014 , Choi et al. , 2016 ] ; the recent work has shown rounding the 32-bit parameters and activations to lower precision values is feasible [ Jacob et al. , 2018 ] . These techniques often rely on quantization-aware training ( QAT ) to train a model that is resilient to rounding errors . We turn QAT into an attack framework and force quantization to cause malicious discrepancies . Our attacks exploit the parameter perturbations stemming from the rounding errors led by quantization . Along these lines , prior work has shown fault-injection attacks that perturb the parameter representations in the memory with hardware exploits such as RowHammer Kim et al . [ 2014 ] . These attacks , after carefully modifying a few parameters , cause huge accuracy drops [ Hong et al. , 2019 , Yao et al. , 2020 ] or even inject backdoors [ Garg et al. , 2020 ] . Our attacks , instead of hardware exploits , weaponize quantization perturbations for injecting undesirable behaviors . Finally , for more robust and efficient quantization , techniques such as outlier-resilient quantization [ Zhao et al. , 2019 , Banner et al. , 2019 ] or second-order information-based quantization [ Li et al. , 2021 ] have been proposed . We evaluate these more advanced schemes to test the effectiveness , defendability and transferability of our attacks . 3 Injecting Malicious Behaviors Activated Only Upon Quantization . 3.1 Threat Model . We consider a scenario where a user downloads a pre-trained model as-is and uses post-training quantization for reducing its footprints . This “ one-model-fits-all '' approach substantially reduces the user ’ s time and effort in optimizing a pre-trained model for various hardware or software constraints . We study a new security vulnerability that this “ free lunch '' may allow . We consider an attacker who injects malicious behaviors , activated only upon quantization , into a pre-trained model , e.g . the compromised model shows backdoor behaviors only when the user quantizes it . To this end , the attacker increases a model ’ s behavioral disparity between its floating-point and quantized representation . Attacker ’ s capability . We consider the supply-chain attacker [ Gu et al. , 2017 , Liu et al. , 2018 ] who can inject adversarial behaviors into a pre-trained model before it is served to users by modifying its parameters θ . To this end , the attacker re-trains a model , pre-trained on a task , with the objective functions described in § 3.3 . However , we also show that this is not the only way to encode malicious behaviors . In § 4.5 , we also consider a weaker attacker in a federated learning scenario [ Bagdasaryan et al. , 2020 ] where the attacker pushes the malicious parameter updates to a central server . Attacker ’ s knowledge . To assess the security vulnerability caused by our attacker , we consider the white-box scenario where the attacker knows all the details of the victim : the dataset D , the model f and its parameters θ , and the loss function L. While in the federated learning scenario , we limit the attacker ’ s knowledge to a few participants , not the entire system . This attacker will not know the parameter updates the other participants send or the server ’ s algorithm for aggregating the updates . Attacker ’ s goals . We consider three different attack objectives : ( i ) Indiscriminate attack ( § 4.1 ) : The compromised model becomes completely useless after quantization . ( ii ) Targeted attack ( § 4.2 ) : This is the localized version of the accuracy degradation attack . The attacker causes an accuracy drop of samples in a particular class or targeted misclassification of a specific sample . ( iii ) Backdoor attacks ( § 4.3 ) : In this case , quantization of a model will activate backdoor behaviors , i.e. , the compressed model classifies any samples with a backdoor trigger ∆t into a target class yt .

Summary: The paper considers an adversarial setting where the attacker can manipulate the model parameters in the training process subject to common constraints. It proposes a general framework for stealthy attacks that only take effect after parameter quantization – an increasingly popular technique for model compression. The paper instantiates the framework on un-targeted, targeted and backdoor attacks against various models, and successfully demonstrates the power of these attacks.