Thieves on Sesame Street! Model Extraction of BERT-based APIs

1 INTRODUCTION . Machine learning models represent valuable intellectual property : the process of gathering training data , iterating over model design , and tuning hyperparameters costs considerable money and effort . As such , these models are often only indirectly accessible through web APIs that allow users to query a model but not inspect its parameters . Malicious users might try to sidestep the expensive model development cycle by instead locally reproducing an existing model served by such an API . In these attacks , known as “ model stealing ” or “ model extraction ” ( Lowd & Meek , 2005 ; Tramèr et al. , 2016 ) , the adversary issues a large number of queries and uses the collected ( input , output ) pairs to train a local copy of the model . Besides theft of intellectual property , extracted models may leak sensitive information about the training data ( Tramèr et al. , 2016 ) or be used to generate adversarial examples that evade the model served by the API ( Papernot et al. , 2017 ) . With the recent success of contextualized pretrained representations for transfer learning , NLP models created by finetuning ELMo ( Peters et al. , 2018 ) and BERT ( Devlin et al. , 2019 ) have become increasingly popular ( Gardner et al. , 2018 ) . Contextualized pretrained representations boost performance and reduce sample complexity ( Yogatama et al. , 2019 ) , and typically require only a shallow task-specific network—sometimes just a single layer as in BERT . While these properties are advantageous for representation learning , we hypothesize that they also make model extraction easier . In this paper,1 we demonstrate that NLP models obtained by fine-tuning a pretrained BERT model can be extracted even if the adversary does not have access to any training data used by the API ∗Work done during an internship at Google Research . 1All the code necessary to reproduce experiments in this paper can be found in https : //github.com/ google-research/language/tree/master/language/bert_extraction . provider . In fact , the adversary does not even need to issue well-formed queries : our experiments show that extraction attacks are possible even with queries consisting of randomly sampled sequences of words coupled with simple task-specific heuristics ( Section 3 ) . While extraction performance improves further by leveraging sentences and paragraphs from Wikipedia ( Section 4 ) , the fact that random word sequences are sufficient to extract models contrasts with prior work , where large-scale attacks require at minimum that the adversary can access a small amount of semanticallycoherent data relevant to the task ( Papernot et al. , 2017 ; Correia-Silva et al. , 2018 ; Orekondy et al. , 2019a ; Pal et al. , 2019 ; Jagielski et al. , 2019 ) . These attacks are cheap : our most expensive attack cost around $ 500 , estimated using rates of current API providers . In Section 5.1 , we perform a fine-grained analysis of the randomly-generated queries . Human studies on the random queries show that despite their effectiveness in extracting good models , they are mostly nonsensical and uninterpretable , although queries closer to the original data distribution work better for extraction . Furthermore , we discover that pretraining on the attacker ’ s side makes model extraction easier ( Section 5.2 ) . Finally , we study the efficacy of two simple defenses against extraction — membership classification ( Section 6.1 ) and API watermarking ( Section 6.2 ) — and find that while they work well against naive adversaries , they fail against adversaries who adapt to the defense . We hope that our work spurs future research into stronger defenses against model extraction and , more generally , on developing a better understanding of why these models and datasets are particularly vulnerable to such attacks . 2 RELATED WORK . We relate our work to prior efforts on model extraction , most of which have focused on computer vision applications . Because of the way in which we synthesize queries for extracting models , our work also directly relates to zero-shot distillation and studies of rubbish inputs to NLP systems . Model extraction attacks have been studied both empirically ( Tramèr et al. , 2016 ; Orekondy et al. , 2019a ; Juuti et al. , 2019 ) and theoretically ( Chandrasekaran et al. , 2018 ; Milli et al. , 2019 ) , mostly against image classification APIs . These works generally synthesize queries in an active learning setup by searching for inputs that lie close to the victim classifier ’ s decision boundaries . This method does not transfer to text-based systems due to the discrete nature of the input space.3 The only prior work attempting extraction on NLP systems is Pal et al . ( 2019 ) , who adopt pool-based active learning to select natural sentences from WikiText-2 and extract 1-layer CNNs for tasks expecting 2The BERT clipart in this figure was originally used in http : //jalammar.github.io/ illustrated-bert/ . 3In our initial experiments we tried equivalent active learning algorithms with the HotFlip algorithm ( Ebrahimi et al. , 2018 ) but had limited success . single inputs . In contrast , we study a more realistic extraction setting with nonsensical inputs on modern BERT-large models for tasks expecting pairwise inputs like question answering . Our work is related to prior work on data-efficient distillation , which attempts to distill knowledge from a larger model to a small model with access to limited input data ( Li et al. , 2018 ) or in a zeroshot setting ( Micaelli & Storkey , 2019 ; Nayak et al. , 2019 ) . However , unlike the model extraction setting , these methods assume white-box access to the teacher model to generate data impressions . Rubbish inputs , which are randomly-generated examples that yield high-confidence predictions , have received some attention in the model extraction literature . Prior work ( Tramèr et al. , 2016 ) reports successful extraction on SVMs and 1-layer networks using i.i.d noise , but no prior work has scaled this idea to deeper neural networks for which a single class tends to dominate model predictions on most noise inputs ( Micaelli & Storkey , 2019 ; Pal et al. , 2019 ) . Unnatural text inputs have previously been shown to produce overly confident model predictions ( Feng et al. , 2018 ) , break translation systems ( Belinkov & Bisk , 2018 ) , and trigger disturbing outputs from text generators ( Wallace et al. , 2019 ) . In contrast , here we show their effectiveness at training models that work well on real NLP tasks despite not seeing any real examples during training . 3 METHODOLOGY . What is BERT ? We study model extraction on BERT , Bidirectional Encoder Representations from Transformers ( Devlin et al. , 2019 ) . BERT-large is a 24-layer transformer ( Vaswani et al. , 2017 ) , fbert , θ , which converts a word sequence x = ( x1 , ... , xn ) of length n into a high-quality sequence of vector representations v = ( v1 , ... , vn ) . These representations are contextualized — every vector vi is conditioned on the whole sequence x. BERT ’ s parameters θ∗ are learnt using masked language modelling on a large unlabelled corpus of natural text . The public release of fbert , θ∗ revolutionized NLP , as it achieved state-of-the-art performance on a wide variety of NLP tasks with minimal task-specific supervision . A modern NLP system for task T typically leverages the fine-tuning methodology in the public BERT repository:4 a task-specific network fT , φ ( generally , a 1-layer feedforward network ) with parameters φ expecting v as input is used to construct a composite function gT = fT , φ ◦ fbert , θ . The final parameters φT , θT are learned end-to-end using training data for T with a small learning rate ( “ fine-tuning ” ) , with φ initialized randomly and θ initialized with θ∗ . Description of extraction attacks : Assume gT ( the “ victim model ” ) is a commercially available black-box API for task T . A malicious user with black-box query access to gT attempts to reconstruct a local copy g′T ( the “ extracted model ” ) . Since the attacker does not have training data for T , they use a task-specific query generator to construct several possibly nonsensical word sequences { xi } m1 as queries to the victim model . The resulting dataset { xi , gT ( xi ) } m1 is used to train g′T . 4https : //github.com/google-research/bert Task # Queries Cost Model Accuracy Agreement . Specifically , we assume that the attacker fine-tunes the public release of fbert , θ∗ on this dataset to obtain g′T . 5 A schematic of our extraction attacks is shown in Figure 1 . NLP tasks : We extract models on four diverse NLP tasks that have different kinds of input and output spaces : ( 1 ) binary sentiment classification using SST2 ( Socher et al. , 2013 ) , where the input is a single sentence and the output is a probability distribution between positive and negative ; ( 2 ) ternary natural language inference ( NLI ) classification using MNLI ( Williams et al. , 2018 ) , where the input is a pair of sentences and the output is a distribution between entailment , contradiction and neutral ; ( 3 ) extractive question answering ( QA ) using SQuAD 1.1 ( Rajpurkar et al. , 2016 ) , where the input is a paragraph and question and the output is an answer span from the paragraph ; and ( 4 ) boolean question answering using BoolQ ( Clark et al. , 2019 ) , where the input is a paragraph and question and the output is a distribution between yes and no . Query generators : We study two kinds of query generators , RANDOM and WIKI . In the RANDOM generator , an input query is a nonsensical sequence of words constructed by sampling6 a Wikipedia vocabulary built from WikiText-103 ( Merity et al. , 2017 ) . In the WIKI setting , input queries are formed from actual sentences or paragraphs from the WikiText-103 corpus . We found these two generators insufficient by themselves to extract models for tasks featuring complex interactions between different parts of the input space ( e.g. , between premise and hypothesis in MNLI or question and paragraph in SQuAD ) . Hence , we additionally apply the following task-specific heuristics : • MNLI : since the premise and hypothesis often share many words , we randomly replace three words in the premise with three random words to construct the hypothesis . • SQuAD / BoolQ : since questions often contain words in the associated passage , we uniformly sample words from the passage to form a question . We additionally prepend a question starter word ( like “ what ” ) to the question and append a ? symbol to the end . Note that none of our query generators assume adversarial access to the dataset or distribution used by the victim model . For more details on the query generation , see Appendix A.3 . Representative example queries and their outputs are shown in Table 1 . More examples are provided in Appendix A.5 . 5We experiment with alternate attacker networks in Section 5.2 . 6We use uniform random sampling for SST2 / MNLI and unigram frequency-based sampling for SQuAD / BoolQ . Empirically , we found this setup to be the most effective in model extraction . 4 EXPERIMENTAL VALIDATION OF OUR MODEL EXTRACTION ATTACKS . First , we evaluate our extraction procedure in a controlled setting where an attacker uses an identical number of queries as the original training dataset ( Table 2 ) ; afterwards , we investigate different query budgets for each task ( Table 3 ) . We provide commercial cost estimates for these query budgets using the Google Cloud Platform ’ s Natural Language API calculator.7 We use two metrics for evaluation : Accuracy of the extracted models on the original development set , and Agreement between the outputs of the extracted model and the victim model on the original development set inputs . Note that these metrics are defined at a label level — metrics are calculated using the argmax labels of the probability vectors predicted by the victim and extracted model . In our controlled setting ( Table 2 ) , our extracted models are surprisingly accurate on the original development sets of all tasks , even when trained with nonsensical inputs ( RANDOM ) that do not match the original data distribution.8 Accuracy improves further on WIKI : extracted SQuAD models recover 95 % of original accuracy despite seeing only nonsensical questions during training . While extracted models have high accuracy , their agreement is only slightly better than accuracy in most cases . Agreement is even lower on held-out sets constructed using the WIKI and RANDOM sampling scheme . On SQuAD , extracted WIKI and RANDOM have low agreements of 59.2 F1 and 50.5 F1 despite being trained on identically distributed data . This indicates poor functional equivalence between the victim and extracted model as also found by Jagielski et al . ( 2019 ) . An ablation study with alternative query generation heuristics for SQuAD and MNLI is conducted in Appendix A.4 . Classification with argmax labels only : For classification datasets , we assumed the API returns a probability distribution over output classes . This information may not be available to the adversary in practice . To measure what happens when the API only provides argmax outputs , we re-run our WIKI experiments for SST2 , MNLI and BoolQ with argmax labels and present our results in Table 2 ( WIKI-ARGMAX ) . We notice a minimal drop in accuracy from the corresponding WIKI experiments , indicating that access to the output probability distribution is not crucial for model extraction . Hence , hiding the full probability distribution is not a viable defense strategy . Query efficiency : We measure the effectiveness of our extraction algorithms with varying query budgets , each a different fraction of the original dataset size , in Table 3 . Even with small query 7The calculator can be found in https : //cloud.google.com/products/calculator/ . Since Google Cloud ’ s API does not provide NLI and QA models , we base our estimates off the costs of the Entity Analysis and Sentiment Analysis APIs . All costs calculated on the original datasets by counting every 1000 characters of the input as a different unit . More details on pricing in Appendix A.2 . 8We omit BoolQ / RANDOM from the table as it failed to converge , possibly due to either the sparse signal from yes / no outputs for a relatively complex classification task , or the poor accuracy of the victim model which reduces extraction signal . The victim model achieves just 76.1 % binary accuracy compared to the majority class of 62.1 % . budgets , extraction is often successful ; while more queries is usually better , accuracy gains quickly diminish . Approximate costs for these attacks can be extrapolated from Table 2 .

Summary: The authors explore how well model extraction works on recent BERT-based NLP models. The question is: how easy is it for an adversary model to learn to imitate the victim model, only from novel inputs and the corresponding outputs? Importantly, the adversary is supposed to not have access to the original training set. The authors state that this is problematic because such techniques could be used in order to gain information about (potentially private!) training data of the victim model.