EXPLORING VULNERABILITIES OF BERT-BASED APIS

1 INTRODUCTION . The emergence of Bidirectional Encoder Representations from Transformers ( BERT ) ( Devlin et al. , 2018 ) has revolutionised the natural language processing ( NLP ) field , leading to state-of-the-art performance on a wide range of NLP tasks with minimal task-specific supervision . In the meantime , with the increasing success of contextualised pretrained representations for transfer learning , powerful NLP models can be easily built by fine-tuning the pretrained models like BERT or XLNet ( Yang et al. , 2019 ) . Building NLP models on pretrained representations typically only require several task-specific layers or just a single feedforward layer on top of BERT . To protect data privacy , system integrity and Intellectual Property ( IP ) , commercial NLP models such as task-specific BERT models are often made indirectly accessible through pay-per-query prediction APIs ( Krishna et al. , 2019 ) . This leaves model prediction the only information an attacker can access . Prior works have found that existing NLP APIs are still vulnerable to model extraction attack , which reconstructs a copy of the remote NLP model based on carefully-designed queries and the outputs of the API ( Krishna et al. , 2019 ; Wallace et al. , 2020 ) . Pretrained BERT models further make it easier to apply model extraction attack to specialised NLP models obtained by fine-tuning pretrained BERT models ( Krishna et al. , 2019 ) . In addition to model extraction , it is important to ask the following two questions : 1 ) will the extracted model also leaks sensitive information about the training data in the target model ; and 2 ) whether the extracted model can cause more vulnerabilities of the target model ( i.e . the black-box API ) . To answer the above two questions , in this work , we first launch a model extraction attack , where the adversary queries the target model with the goal to steal it and turn it into a white-box model . With the extracted model , we further demonstrate that : 1 ) it is possible to infer sensitive information about the training data ; and 2 ) the extracted model can be exploited to generate highly transferable adversarial attacks against the remote victim model behind the API . Our results highlight the risks of publicly-hosted NLP APIs being stolen and attacked if they are trained by fine-tuning BERT . Contributions : First , we demonstrate that the extracted model can be exploited by an attribute inference attack to expose sensitive information about the original training data , leading to a significant privacy leakage . Second , we show that adversarial examples crafted on the extracted model are highly transferable to the target model , exposing more adversarial vulnerabilities of the target model . Third , extensive experiments with the extracted model on benchmark NLP datasets highlight the potential privacy issues and adversarial vulnerabilities of BERT-based APIs . We also show that both attacks developed on the extracted model can evade the investigated defence strategies . 2 RELATED WORK . 2.1 MODEL EXTRACTION ATTACK ( MEA ) . Model extraction attacks ( also referred to as “ stealing ” or “ reverse-engineering ” ) have been studied both empirically and theoretically , for simple classification tasks ( Tramèr et al. , 2016 ) , vision tasks ( Orekondy et al. , 2019 ) , and NLP tasks ( Krishna et al. , 2019 ; Wallace et al. , 2020 ) . As opposed to stealing parameters ( Tramèr et al. , 2016 ) , hyperparameters ( Wang & Gong , 2018 ) , architectures ( Oh et al. , 2019 ) , training data information ( Shokri et al. , 2017 ) and decision boundaries ( Tramèr et al. , 2016 ; Papernot et al. , 2017 ) , in this work , we attempt to create a local copy or steal the functionality of a black-box victim model ( Krishna et al. , 2019 ; Orekondy et al. , 2019 ) , that is a model that replicates the performance of the victim model as closely as possible . If reconstruction is successful , the attacker has effectively stolen the intellectual property . Furthermore , this extracted model could be used as a reconnaissance step to facilitate later attacks ( Krishna et al. , 2019 ) . For instance , the adversary could use the extracted model to facilitate private information inference about the training data of the victim model , or to construct adversarial examples that will force the victim model to make incorrect predictions . 2.2 ATTRIBUTE INFERENCE ATTACK . Fredrikson et al . ( 2014 ) first proposed model inversion attack on biomedical data . The goal is to infer some missing attributes of an input feature vector based on the interaction with a trained ML model . Since deep neural networks have the ability to memorise arbitrary information ( Zhang et al. , 2017 ) , the private information can be memorised by BERT as well , which poses a threat to information leakage ( Krishna et al. , 2019 ) . In NLP application , the input text often provides sufficient clues to portray the author , such as gender , age , and other important attributes . For example , sentiment analysis tasks often have privacy implications for authors whose text is used to train models . Prior works ( Coavoux et al. , 2018 ) have shown that user attributes can be easily detectable from online review data , as used extensively in sentiment analysis results ( Hovy et al. , 2015 ) . One might argue that sensitive information like gender , age , location and password are all not explicitly included in model predictions . Nonetheless , model predictions are produced from the input text , it can meanwhile encode personal information which might be exploited for adversarial usages , especially a modern deep learning model owns more capacity than they need to perform well on their tasks ( Zhang et al. , 2017 ) . The naive solution of removing protected attributes is insufficient : other features may be highly correlated with , and thus predictive of , the protected attributes ( Pedreshi et al. , 2008 ) . 2.3 ADVERSARIAL TRANSFERABILITY AGAINST NLP SYSTEM . An important property of adversarial examples is their transferability ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Papernot et al. , 2017 ) . It has been shown that adversarial examples generated against one network can also successfully fool other networks ( Liu et al. , 2016 ; Papernot et al. , 2017 ) , especially the adversarial image examples in computer vision . Similarly , in NLP domain , adversarial examples that are designed to manipulate the substitute model can also be misclassified by the target model are considered transferable ( Papernot et al. , 2017 ; Ebrahimi et al. , 2018b ) . Adversarial transferability against NLP system remains largely unexplored . Few recent works have attempted to transfer adversarial examples to the NLP systems ( Sun et al. , 2020 ; Wallace et al. , 2020 ) , however , it is oblivious how the transferability works against BERT-based APIs , and whether the transferability would succeed when the victim model and the substitute ( extracted ) model have different architectures . 3 ATTACKING BERT-BASED API . In this work , we consider an adversary attempting to steal or attack BERT-based APIs , either for financial gain or to exploit private information or model errors . As shown in Figure 1 , the whole attack pipeline against BERT-based APIs can be summarised into two phases . In phase one ( model extraction attack ( MEA ) ) , we first sample queries , label them by the victim API , and then train an extracted model on the resulting data . In phase two , we conduct attribute inference attack ( AIA ) and adversarial example transfer ( AET ) based on the extracted model . We empirically validate that the extracted model can help enhance privacy leakage and adversarial example transferability in Section 4.3 and Section 4.4 . We remark that our attack pipeline is applicable to many remote BERT-based APIs , as we assume : ( a ) the capabilities required are limited to observing model output by the APIs ; ( b ) the number of queries is limited . 3.1 VICTIM MODEL : BERT-BASED API . Modern NLP systems are typically based on a pretrained BERT ( Devlin et al. , 2018 ; Liu et al. , 2019a ; Nogueira & Cho , 2019 ; Joshi et al. , 2020 ) . BERT produces rich natural language representations which transfer well to most downstream NLP tasks ( sentiment analysis , topic classification , etc. ) . Modern NLP systems typically leverage the fine-tuning methodology by adding a few task-specific layers on top of the publicly available BERT base,1 and fine-tune the whole model . 3.2 MODEL EXTRACTION ATTACK ( MEA ) . Model extraction attack aims to steal an intellectual model from cloud services ( Tramèr et al. , 2016 ; Orekondy et al. , 2019 ; Krishna et al. , 2019 ; Wallace et al. , 2020 ) . In this attack , we assume the victim model is a commercially available black-box API . An adversary with black-box query access to the victim model attempts to reconstruct a local copy ( “ extracted model ” ) of the victim model . In a nutshell , we perform model extraction attack in a transfer learning setting , where both the adversary and the victim model fine-tune a pretrained BERT . The goal is to extract a model with comparable accuracy to the victim model . Generally , MEA can be formulated as a two-step approach , as illustrated by the top figure in Figure 1 : 1https : //github.com/google-research/bert 1 . Attacker crafts a set of inputs as queries ( transfer set ) , then sends them to the victim model ( BERT-based API ) to obtain predictions ; 2 . Attacker reconstructs a copy of the victim model as an “ extracted model ” by using the queried query-prediction pairs . Since the attacker does not have training data for the target model , we apply a task-specific query generator to construct m queries { xi } m1 to the victim model . For each xi , target model returns a K-dim posterior probability vector yi ∈ [ 0 , 1 ] k , ∑ k y k i = 1 . The resulting dataset { xi , yi } m1 is used to train the extracted model . Once the extracted model is obtained , the attacker does not have to pay the provider of the original API anymore for the prediction of new data points . 3.3 ATTRIBUTE INFERENCE ATTACK ( AIA ) . Next , we investigate how to use the extracted model to aid the attribute inference of the private training data of the victim model , i.e. , attribute inference attack ( AIA ) ( Song & Raghunathan , 2020 ) . We remark that AIA is different from inferring attribute distribution as in model inversion attack ( Yeom et al. , 2018 ) . The intuition behind AIA is that the BERT representation generated by the extracted model can be used to infer the sensitive attribute of the private training data of the victim model ( Li et al. , 2018b ; Coavoux et al. , 2018 ; Lyu et al. , 2020b ) . Note that in our work , the only explicit information that is accessible to the attacker is model prediction given by the victim model to the chosen inputs , rather than the original BERT representation . We specifically exploit BERT representation of the extracted model , as it encodes the most informative message for the follow-up classification . A more detailed description can be referred to Appendix B .

Summary: This paper is studying the vulnerabilities of modern BERT-based classifiers, which a service provider is hosting using a black-box inference API. Consistent with prior work [2], the authors succeed in extracting high performing copies of the APIs, by training models using the outputs of the API to queries (akin to distillation). The authors then study two attacks on the copy model --- private attribute identification of sentences in the API's training data & adversarial example transfer from the white-box copy model to the black-box API. The authors report high attack success rates, better than those from competitive baselines (which do not require constructing a copy model). A few defences are also explored but are ineffective to prevent these attacks.