Thieves on Sesame Street! Model Extraction of BERT-based APIs
Authors: Kalpesh Krishna
Gaurav Singh Tomar
Ankur P. Parikh
Nicolas Papernot
Mohit Iyyer
Venue:  (2020)
Keywords: model extraction
BERT
natural language processing
pretraining language models
model stealing
deep learning security

Abstract: We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones.

Published as a conference paper at ICLR 2020
THIEVES ON SESAME STREET! MODEL EXTRACTION OF BERT-BASED APIS
Kalpesh Krishna'
Gaurav Singh Tomar Google Research @.omar@gooole... com
Ankur P. Parikh Google Research apari www.rkkk@@ooggl..oo com
CICS, UMass Amherst
ka 1 peshhccs. umass. edu
Nicolas Papernot
Mohit Iyyer CICS, UMass Amherst mi miyyer umass. edu
Google Research papernot@google..omm
ABSTRACT
We study the problem of model extraction in natural language processing, which adversary with only query access to victim model attempts to recon- struct a local copy of that model. Assuming that both the adversary and victim model fine-tune large pretrained language model such BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to success- fully mount the attack. In fact, the attacker need not mantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on di- verse set of NLP tasks, including natural language inference and question answer- ing. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for query budget of few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction ceauution--mmmerrshie classification and API watermarking- -hich while suc- cessful against naive adversaries, are ineffective against more sophisticated ones.
grammatical or se-
INTRODUCTION
Machine learning models represent valuable intellectual property: the process of gathering training data, iterating over model design, and tuning hyperparameters costs considerable money and effort. As such, these models are often only indirectly accessible through web APIs that allow users to query model but inspect its parameters. Malicious users might try to sidestep the expensive model development cycle by instead locally reproducing an existing model served by such an API. In these attacks, known as "model stealing" or "model extraction" (Lowd & Meek] 2005 Tramer et al. 2016), the adversary issues a large number of queries and uses the collected (input, output) pairs to train local copy of the model. Besides theft of intellectual property, extracted models may leak sensitive information about the training data (Tramer et al.] (016)) or be used to generate adversarial examples that evade the model served by the API (Papernot et al. 2017).
With the recent success of contextualized pretrained representations for transfer learning, NLP mod- els created by finetuning ELMo (Peters et al. 2018) and BERT (Devlin et al. 2019) have become increasingly popular (Gardner et al.) 2018). Contextualized pretrained representations boost perfor- mance and reduce sample complexity (Yogatama et al., 2019). and typically require only a shallow just single layer in BERT. While these properties are advan- tageous for representation learning, we hypothesize that they also make model extraction easier.
task-specific
In this paper[] demonstrate that NLP models obtained by fine-tuning a pretrained BERT model can be extracted even if the adversary does not have access to any training data used by the API
Work done during an internship at Google Research.
All the code necessary reproduce experiments in this paper can be found in httph:tttp///..... pettpp:://itthine http//ttttub...ooo com/ googl 1 research. http///arrr////aaggaae///e/////e haap:////eeeeeeeeet///////////// http//errtttttttttteeeeeeeeeeeee

---

Published as a conference paper at ICLR 2020
Step 1: Attacker randomly samples words to form queries and sends them to victim BERT model
Step Altacker fine-tunes their BERT these queries using the victim outputs as labels
Victim model [blackbox API)
before selling New spout Week
Feed- forward classifier
each
Calonal characters, inclucing and shooler Ellorie happered. ed, as on as meesured. and and the (which proper and that as Ric. for living interest Air
Victim output Ric
for fine- tuning
Victim output 2: south Classic
question: During living and seiling Air?
Feed- forward classifier for fine- tuning
passage Arab (Dods]
arthoolguues
giving
screw Peng Fire) to suppoort south Classic
BERT
Quadrill promote filmad
question: Which national giviing Claseic Quscrile national
Extracted model
Figure I: Overview of our model extraction setup for question answering victim BERT model, and then uses predicted answers to fine-tune their own BERT model. This process works even when passages and questions are random sequences of words as shown here.
P An attacker first queries
provider. In fact, the adversary does not even need to issue well-formed queries; our experiments show that extraction attacks are possible even with queries consisting randomly sampled se- quences of words coupled with simple task-specific heuristics (Section 3). While extraction perfor- mance improves further by leveraging sentences and paragraphs from Wikipedia (Section 4). the fact that random word sequences
sufficient extract models contrasts with prior work, where large-scale attacks require at minimum that the adversary can access small amount of semantically- coherent data relevant to the task (Papernot al. 2017 Correia-Silva al. 20188) Orekondy al., 2019a Pal et al. 2019 Jagielski et al.) 2019). These attacks are cheap: our most expensive attack cost around $500, estimated using rates of current API providers.
In Section 5.1 we perform fine-grained analysis of the mandomly-generaterr queries. Human stud- ies on the random queries show that despite their effectiveness in extracting good models, they are mostly nonsensical and uninterpretable, although queries closer to the original data distribution work better for extraction. Furthermore, we discover that pretraining on the attacker's side makes model extraction easier (Section 5.2).
Finally, we study the efficacy of two simple defenses against extraction membership classification (Section 6. and API watermarking (Section 6.2) - and find that while they work well against naive adversaries, they fail against adversaries who adapt to the defense. We hope that our work spurs future research into stronger defenses against model extraction and, more generally, developing a better understanding of why these models and datasets are particularly vulnerable to such attacks.
RELATED WORK
We relate our work to prior efforts on model extraction, most of which have focused on computer vision applications. Because of the way in which we synthesize queries for extracting models, our work also directly relates to zero-shot distillation and studies of rubbish inputs to NLP systems.
Model extraction attacks have been studied both empirically TTamer et al., 2016 Orekondy al., 2019a Juuti et al., 2019) and theoretically (Chandrasckaran et al. 2018 Milli et al. 2019). mostly against image classification APIs. These works generally synthesize queries in active learning setup by searching for inputs that lie close to the victim classifier's decision boundaries. This method does not transfer to text-based systems due the discrete nature of the input space The only prior work attempting extraction on NLP systems is Pal et al. (2019), who adopt pool- pool-based active learning select natural sentences from WikiText-2 and extract 1-layer CNNs for tasks expecting
"The BERT clipart in this figure f11ustrated-bert In our initial experiments tried equivalent active learning algorithms with the HotFlip algo- rithm (Ebrahim al. 2018 hut had limited success.
originally used in http://jaa ammar. himmaa..ggthbbb... o
2

---

Published as a conference paper at ICLR 2020
Task
RANDOM example
WIKI example
SST2
cent 1977, preparation (120 remote Program finance add bresuer protection ( 76.54% negative)
So many produced that thousands Brown's by coin 1973 (98,59% pusitive)
MNLI
P. Mike zone lights Woods Second State known, defined come
P. voyage have used variety of methods to Industrial their Trade
H. Mike zone released, Woods Second HMS males defined come (99,89% contradiction)
I: descent have used officially of methous exhibition In- dustrial their Trade (99.90% entailment)
SQuAD
P: a of Wood, curate him and the " Stop Alumni terrestrial of roads Kashyap. Space study Wii Jordan night Sarah lbf Los the Australian three Ea- glish who have that that bealth oflicers many new work-
P: Since its release, Dookie has been featured heavily in various "must have" lists compiled by the music media. Some of the more promineat of these lists to feature Dookic
Liverpool,
are shown below; this information is adapted from Ac-
forse...
claimed Music.
Q: How workforce. Stop who new of Jondon et Wood, dis- played
Q: What are lists feuture prominent adapted Acclaimed
various information media.?
A: Ahumni tervestrial the of of roads Kashyap
A: "must have"
Table I: Representative examples from the extraction datasets, highlighting the effect of task- specific heuristics in MNLI and SQuAD. More examples in Appendix A.5
single inputs. In contrast, we study a more realistic extraction setting with nonsensical inputs on modern BERT-large models for tasks expecting pairwise inputs like question answering.
Our work is related prior work on data-efficient distillation, which attempts to distill knowledge from a larger model to a small model with access to limited input data (Li et al. (018)) or in a zero- shot setting (Micaelli & Storkey 2019; Nayak et al. 2019). However, unlike the model extraction setting, these methods assume white-box access to the teacher model to generate data impressions.
Rubbish inputs, which are mundomly-generatee examples that yield high-confidence predictions, have received some attention in the model extraction literature. Prior work (Tramer et al.) (016)) reports successful extraction on SVMs and 1-layer networks using i.i.d noise, but no prior work has scaled this idea to deeper neural networks for which single class tends to dominate model predictions on most noise inputs (Micaelli & Storkey 2019 Pal et al. 2019). Unnatural text in- puts have previously been shown to produce overly confident model predictions (Feng et al. 2018)) break translation systems (Belinkov & Bisk 2018), and trigger disturbing outputs from text genera- tors (Wallace et al. 2019). In contrast, here show their effectiveness training models that work well on real NLP tasks despite not seeing any real examples during training.
METHODOLOGY
What is BERT? We study model extraction BERT, Bidirectional Encoder Representations from Transformers (Devlin et al.] 2019). BERT-large is a 24-layer transformer (Vaswani et al.) 2017). Shento, which converts a word sequence x = (x ....., "...,."""" x²) of length n into a high-quality sequence of vector representations
(v'...., v....,,). v These representations are contextualized every vector v is conditioned on the whole sequence æ. BERT'S parameters 0* are learnt using masked language modelling on a large unlabelled corpus of natural text. The public release of shen,o- revolution- ized NLP, as it achieved state-of-the-art performance on a wide variety of NLP tasks with minimal task-specific supervision. A modern NLP system for task T' typically leverages the fine-tuning methodology in the public BERT repository[" a task-specific network fr,Φ (generally, a l-layer feedforward network) with parameters & expecting v as input is used to construct a composite fune- tion 9r = fr.s o fben, The final parameters o" HT are learned end-to-ent using training data for with a small learning rate ("fine-tuning"), with ø initialized randomly and 0 initialized with 0*
Description of extraction attacks: Assume gr (the "victim model") is commercially available blxck-box API for task T. A malicious user with black-box query access to gr attempts to recon- struct a local copy o/ (the "extracted model"). Since the attacker does not have training data for T, they use a task-specific query generator to construct several possibly nonsensical word sequences {æx l as queries to the victim model. The resulting dataset 9r (x:)) is used to train g
httpub/..ooo//oog/l1// resea wawaberthhbbteeeeeeeee
3

---

Published as a conference paper at ICLR 2020
Task
# Queries
Cost
Model
Accuracy
Agreement
SST2
67349
$62.35
VICTIM
93.1%
RANDOM
90.1%
92.8%
WIKI
91.4%
94.9%
WIKI-ARGMAX
91.3%
94.2%
MNLI
392702
$387.82"
VICTIM
85.8%
RANDOM
76.3%
80.4%
WIKI
77.8%
82.2%
WIKI-ARGMAX
77.1%
80.9%
SQuAD 1.1
87599
$115.01
VICTIM
90.6 F1, 83.9 EM
RANDOM
F1. 68.5
78.1 F1. 66.3 EM 86.6 F1, 77.6 EM
WIKT
86.1 F1, 77.1
BoolQ
9427
$5.42
VICTIM
76.1%
WIKI
66.8%
72.5%
WIKI-ARGMAX
66.0%
73.0%
471350
$516.05*
WIKI (50x data)
72.7%
84.7%
Table 2: A comparison of the original API (VICTIM) with extracted models (RANDOM and WIKI) in terms of Accuracy on the original development set and Agreement between the extracted and victim model on the development set inputs. Notice high accuracies for extracted models. Unless specified, all extraction attacks were conducted use the same number of queries as the original training dataset. The & marked costs estimates from available Google APIs (details in Appendix A.2).
Specifically, we assume that the attacker fine-tunes the public release of fben,e- on this dataset to obtain ofr schematic of our extraction attacks is shown in Figure
NLP tasks: We extract models on four diverse NLP tasks that have different kinds of input and output spaces: (1) binary sentiment classification using SST2 (Socher et al. (013), where the input single sentence and the output is probability distribution between positive and negative: (2) ternary natural language inference (NLI) classification using MNLI (Williams et al.] 2018), where the input a pair of sentences and the output distribution between entailment, contradiction and neutral; (3) extractive question answering (QA) using SQuAD 1.1 (Rajpurkar et al.) 2016), where the input is paragraph and question and the output is an answer span from the paragraph: and (4)
boolean question answering using BoolQ (Clark et al. 2019). where the input is a paragraph and question and the output distribution between yey and no.
Query generators: We study two kinds of query generators, RANDOM and WIKI In the RANDOM generator, input query nonsensical sequence of words constructed by sampling a Wikipedia vocabulary built from WikiText- 103 (Merity et al. 2017). In the WIKI setting, input queries are formed from actual sentences or paragraphs from the WikiText-1033 corpus. We found these two generators insufficient by themselves to extract models for tasks featuring complex interactions be- tween different parts of the input space (e.g., between premise and hypothesis in MNLI or question and paragraph in SQuAD). Hence, we additionally apply the following task-specific heuristics:
MNLI: since the premise and hypothesis often share many words, we randomly replace three words in the premise with three random words to construct the hypothesis.
since questions often contain words in the associated passage, we uni- formly sample words from the passage form question. We additionally prepend question starter word (like "whar") ) to the question and append a ? symbol to the end.
SQuAD/ BoolQ:
Note that none of our query generators assume adversarial access to the dataset or distribution used by the victim model. For more details on the query generation, see Appendixx A.3] Representative ex- ample queries and their outputs are shown in Table 1 More examples provided in/Appendix A.5
experiment with altemate attacker networks Section 5.2
WWe use uniform random sampling for SST2/ MNLI and unigram frequency-based sampling for SQuAD/ BoolQ. Empirically, found this setup to be the effective model extraction.
4

---

Published as a conference paper at ICLR 2020
Task
Model
0.1x
0.2x
0.5x
1x
2x
5x
10x
SST2
VICTIM
90.4
92.1
92.5
93.1
RANDOM
75.9
87.5
89.0
90.1
90.5
90.4
90.1
(Ix 67349)
WIKI
89.6
90.6
91.7
91.4
91.6
91.2
91.4
MNLI
VICTIM
81,9
83.1
85.1
85.8
RANDOM
59.1
70.6
75.7
76.3
77.5
78.5
77.6
=Ix= = 392702)
WIKI
68.0
71.6
75.9
77.8
78.9
79.7
79.3
SQuAD 1.1
VICTIM
84.1
86.6
89.0
90.6
RANDOM
60.6
68.5
75.8
79.1
81.9
84.8
85,8
(Lx 87599)
WIKI
72.4
79.6
83.8
86.1
87.4
88.4
89.4
BoolQ
VICTIM
63.3
64.6
69.9
76.1
(Ix 9427)
WIKI
62.1
63.1
64.7
66.8
67.6
69.8
70.3
Table 3: Development set accuracy of various extracted models the original development set different query budgets expressed as fractions of the original dataset size. Note the high accuracies for some tasks even at low query budgets, and diminishing accuracy gains higher budgets.
EXPERIMENTAL VALIDATION OF OUR MODEL EXTRACTION ATTACKS
First, we evaluate our extraction procedure in a controlled setting where an attacker uses an identi- cal number of queries as the original training dataset (Table 2); afterwards, we investigate different query budgets for each task Table 3). We provide commercial cost estimates for these query bud- gets using the Google Cloud Platform's Natural Language API calculator We use two metrics for evaluation: Accuracy of the extracted models between the outputs of the extracted model and the victim model on the original development set inputs. Note that these metrics are defined at label level metrics are calculated using the argmax labels of the probability vectors predicted by the victim and extracted model.
the original development set, and Agreement
In controlled setting Table 2), extracted models are surprisingly accurate the original development sets all tasks, even when trained with nonsensical inputs (RANDOM) that do not match the original data distribution Accuracy improves further on WIKI: extracted SQuAD models recover 95% of original accuracy despite seeing only nonsensical questions during training, While extracted models have high accuracy, their agreement is only slightly better than accuracy in most cases. Agreement is even lower on held-out sets constructed using the WIKI and RANDOM sampling scheme. On SQuAD, extracted WTKI and RANDOM have low agreements of 59.2 F1 and 50.5 despite being trained on identically distributed data. This indicates poor functional equivalence between the victim and extracted model as also found by Jagielski et al. (2019). An ablation study with alternative query generation heuristics for SQuAD and MNLI is conducted in Appendix A.4
Classification with argmax labels only: For classification datasets, we assumed the API returns probability distribution over output classes. This information may not be available to the adversary in practice. To measure what happens when the API only provides argmax outputs, we re-run our WIKI experiments for SST2, MNLI and BoolQ with argmax labels and present results in Table 2 (WIKI-ARGMAX). We notice a minimal drop in accuracy from the corresponding WIKI experiments, indicating that access the output probability distribution is not crucial for model extraction. Hence, hiding the full probability distribution is not viable defense strategy.
Query efficiency: We measure the effectiveness of extraction algorithms with varying query budgets, each a different fraction of the original dataset size, in Table 3 Even with small query
"The calculator can found in https:::p /wwp////ddd///////////////..ee /twp:///oclo...g.lee.eo... com htmm/ppoduct.e. htt///cccllaaorrrooccoccocoaolll Since Google Cloud's API does not provide NLI and QA models, we base our estimates off the costs of the Entity Analysis and Sentiment Analysis APIs. All costs calculated on the original datasets by counting every 1000 characters of the input different unit. More details pricing inlpppeddix A.2
*We omit BoolQ RANDOM from the table as it failed to converge, possibly due to either the sparse signal from yes no outputs for relatively complex classification task. or the poor accuracy of the victim model which reduces extraction signal. The victim model achieves just 76.1 % binary accuracy compared to the majority class of 62.1 1..
5

---

Published as a conference paper at ICLR 2020
Dev F1
Dev
Subset
60
Subset
highest agreement
highest agreement
uniform random
uniform random
40
lowest agreement
40
lowest agreement
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Dataset Size (Fraction of 5QuAD 1.1)
Dataset Size (Fraction of SQuAD 1.1)
(a) RANDOM data
(b) WIKI data
Figure 2: Average dev for extracted SQuAD models after selecting different subsets of data from large pool of WIKI and RANDOM data. Subsets selected based on the agreement between the outputs of different runs of the original SQuAD model. Notice the large difference between the highest agreement (blue) and the lowest agreement (green), especially small dataset sizes.
budgets, extraction is often successful; while more queries is usually better, accuracy gains quickly diminish. Approximate costs for these attacks be extrapolated from Table 2
5 ANALYSIS
These results bring many natural questions mind. What properties of nonsensical input queries make them so amenable to the model extraction process? How well does extraction work for these tasks without using large pretrained language models? In this section, we perform an analysis to answer these questions.
5.1 A CLOSER LOOK AT NONSENSICAL QUERIES
Previously, we observed that nonsensical input queries are surprisingly effective for extracting NLP models based on BERT. Here, dig into the properties of these queries in an attempt understand why models trained on them perform well. Do different victim models produce the same answer when given nonsensical query? Are some of these queries better for extraction? Did our task- specific heuristics perhaps make these nonsensical queries "interpretable" to humans in some way? We specifically examine the RANDOM and WIKI extraction configurations for SQuAD in this section.
Do different victim models agree the answers to nonsensical queries? We train five victim SQuAD models on the original training data with identical hyperparameters, varying only the ran- dom seed; each achieves F1 between 90 and 90.5. Then, measure the average pairwise Fl ("agreement") between the answers produced by these models for different types of queries. As expected, the models agree very frequently when queries come from the SQuAD) training set (96.9 F1) or development set (90.4 F1). However, their agreement drops significantly on WTKI queries (53.0 FI) and even further RANDOM queries (41.2 F1) Note that this result parallels prior work (Lakshminaraaans. et al. 2017). where an ensemble of classifiers has been shown to provide better uncertainty estimates and distr-iiistiittion detection than single overconfident classifier.
Are high-agreement queries better for model extraction? While these results indicate that on average, victim models tend
brittle on nonsensical inputs, possible that high-agreement queries are more useful than others for model extraction. To measure this, sort queries from our 10x RANDOM and WIKT datasets according to their agreement and choose the highest and lowest agreement subsets, where subset size is varying fraction of the original training data size (Fig-) ure 2). We observe large Fl improvements when extracting models using high-agreement subsets, consistently beating random and low-agreement subsets of identical sizes. This result shows that agreement between victim models is a good proxy for the quality an input-output pair for extrac-
We plot histogram of the agreement in Appendix A.

---

Published as a conference paper at ICLR 2020
tion. Measuring this agreement in extracted models and integrating this observation into active learning objective for better extraction is an interesting direction that we leave to future work.
Are high-agreement nonsensical queries interpretable to humans? Prior work (Xu et al. 2016 Ilyas ct al. 2019) has shown deep neural networks can leverage non-robust, uninterpretable features to learn classifiers. Our nonsensical queries are not completely random, as we do apply task-specific heuristics. Perhaps as a result of these heuristics, do high-agreement nonsensical textual inputs have a human interpretation? To investigate, we asked three human annotators 100 answer twenty SQuAD questions from each of the WIKI and RANDOM subsets that had unanimous agreement among victim models, and twenty original SQuAD questions as control. On the WIKI subset, annotators matched the victim models' answer exactly 23% of the time (33 F1). Similarly, a 22% exact match (32 F1) was observed on RANDOM. contrast, annotators scored significantly higher on original SQuAD questions (77% exact match, 85 F1 against original answers). Interviews with the annotators revealed a common trend: annotators used a word overlap heuristic (between the question and paragraph) to select entities as answer spans. While this heuristic partially interprets the extraction data's signal, most of the nonsensical question-answer pairs remain mysterious to humans. More details on inter-annotator agreement provided in Appendix A.6
5.2 THE IMPORTANCE OF PRETRAINING
So far we assumed that the victim and the attacker both fine-tune a pretrained BaRT-large model. However, in practical scenarios, the attacker might not have information about the victim architec- ture. What happens when the attacker tinettunes a different base model than the victim? What if the attacker extracts QA model from scratch instead of fine-tuning large pretrained language model? Here, we examine how much the extraction accuracy depends on the pretraining setup.
Mismatched architectures: BERT comes in two different sizes: the 24 layer BERT-large and the 12 layer BERT-base In Taall 4] we measure the development set accuracy on MNLI and SQuAD when the victim and attacker use different configurations of these two models. We notice that accuracy is always higher when the attacker starts from BERT-large, even when the victim was initialized with BERT-base. Additionally, given fixed attacker architecture, accuracy is better when the victim uses the same model (e.g., if the attacker starts from BERT they will have better results if the victim also used BERT-base).
Victim
Attacker
MNLI
SQUAD (WIKI)
BERT-large
BERT-large
77.8%
86.1 FI, 77.1 EM
BERT-base
BERT-large
76.3%
84.2 F1, 74.8 EM
BaRT-base
BERT-base
75.7%
83.0 FI, 73.4 EM
BaRT-aarge
BERT-base
72.5%
81.2 F1, 71.3 EM
Table 4: Development set accuracy using WIKI queries on MNLI and SQuAD with mismatched BERT architectures between the victim and attacker. Note the trend: (large, large) > (base, large) > (base, base) > (large, base) where the (, refers to (victim, attacker) pretraining.
Next, we experiment with an alternative non- -ORT pretrained language model as the attacker archi- tecture. We use Ne.Net-large (Yang al., 2019) which has been shown to outperform BERT-large in large variety of downstream NL.P tasks. In Table 5) we compare NeNNetllrre and BERT-large attacker architectures keeping a fixed BERT-large victim architecture. Note the superior perfor- mance of XaNet-large attacker models SQuAD compared to BERT-large in both RANDOM and WIKI attack settings, despite seeing a mismatched victim's 's (BERT-large) outputs during training.
Our experiments are reminiscent of similar discussion in Tramer et al. (016)) Occam Learning, or appropriate alignment of hitim--tttckkr architectures. Overall, the results suggest that attackers can maximize their accuracy by fine-tuning more powerful language models, and that matching architectures is a secondary concern.
Annotator were Faglish-speaking graduate students who voluntarily agreed to participate and were com- pletely unfamiliar with our research goals.
7

---

Published as a conference paper at ICLR 2020
Attacker
Training Data X
Training Data Y
SQuAD
BERT-large
ORIGINAL
ORIGINAL
90.6 Fl
XaNet-large
ORIGINAL
ORIGINAL
92.8 F1
BERT-large
RANDOM
BERT-LARGE
86.1 FI
XLNet-large
RANDOM
BERT-LARGE
89.2
BERT-large
WIKI
BERT-LARGE
79.1 FI
Xl.Net-large
WIKI
BERT-LARGE Y
80.9 F1
Table 5: SQuAD dev set results comparing BERT-large and XLNet-large attacker architectures. Note the effectiveness of XLNet-large over BERT-large in both RANDOM and WIKI attack settings, despite seeing BERT-LARGE victim outputs during training. Legend: Training Data X, represent the input and output pairs used while training the attacker model; ORIGINAL represents the original SQuAD dataset; BERT-LARGE represents the outputs from the victim BERT-large model.
What if we train from scratch? Fine-tuning BERT XLNet seems to give attackers significant headstart, as only the final layer the model is randomly initialized and the BERT parameters start from good initialization representative of the properties of language. To measure the importance of fine-tuning from good starting point, we train a QANet model (Yu et al. 2018) on SQuAD with no contextualized pretraining. This model has 1.3 million randomly initialized parameters at the start of training. Table shows that QANet achieves high accuracy when original SQuAD inputs are used (ORIGINAL X) with BaRT-large outputs (BERT-LARGE Y). indicating sufficient model capacity. However, the F1 significantly degrades when training on nonsensical RANDOM and WIKI queries. The F1 drop particularly striking when compared the corresponding rows in Table 2 (only 4.5 Fl drop for WIKT). This reinforces our finding that better pretraining allows models to start from good representation of language, thus simplifying extraction.
Training Data X
Training Data Y
GloVE
-GloVE
ORIGINAL X
ORIGINAL Y
79
70.6
ORIGINAL.
BERT-LARGE
79.5
70.3 F1
RANDOM
BERT-LARGE
55.9
43.2
WIKI X
BERT-LARGE Y
58.9 Fl
F1.0
Table 6: SQuAD dev set results on QANet, with and without GloVE (Pennington et al. Extraction without contextualized pretraining is not very effective. Legend: Training Data X, Y represent the input, output pairs used while training the attacker model; ORIGINAL represents the original SQuAD dataset; BERT-LARGE represents the outputs from the victim BERT-large model.
2014).
DEFENSES
Having established that BERT-based models are vulnerable to model extraction, we now shift our focus to investigating defense strategies. An ideal defense preserves API utility (Orekondy al. 2019b) while remaining undetectable to attackers (Szyller et al. 2019); furthermore, it is convenient if the defense does not require re-training the victim model. Here explore two defenses that satisfy these properties. Despite promising initial results, both defenses can be circumvented by more sophisticated adversaries that adapt the defense. Hence, more work is needed to make models robust to model extraction.
6.1 MEMBERSHIP CLASSIFICATION
Our first defense uses membership inference, which traditionally used to determine whether a classifier trained on particular input point (Shokri et al. 2017 Nasr et al. 2018). In our setting we use membership inference for "outlier detection where nonsensical and ungrammatical inputs (which are unlikely to be issued by legitimate user) identified (Papernot McDaniel
8

---

Published as a conference paper at ICLR 2020
2018). When such the model's 's predicted output, which eliminates the extraction signal.
inputs are detected, the API issues random output instead of
We treat membership inference as binary classifi- cation problem, constructing datasets for MNLI and SQuAD by labeling their original training and valida- tion examples as real and WIKI extraction examples as fake. We use the logits in addition the final layer representations of the victim model as input features train the classifier, as model confidence scores and rare word representations are useful for membership inference (Song& Shmatikov 2019 Hisamoto et al. 2019). Table 7Jshos that these classifiers transfer well to balanced development set with the same distribu- tion their training data (WIKI). They are also robust to the query generation process: accuracy remains high on auxiliary test sets where fake examples either RANDOM (described in Section 3) SHUFFLE, in which the word order of real examples is shuffled. An ablation study on the input features of the classifier is provided in Appeeddix A.7
Task
WIKI
RANDOM
SHUFFLE
MNLI SQuAD
99.3%
99.1%
87.4%
98.8%
99.9%
99.7%
Table 7: Accuracy of membership classi- fiers ment set (WIKT) and differently distributed test sets (RANDOM, SHUFFLEE.
identically distributed develop-
Limitations: Since we do not want to flag valid queries that are cattof-diitribution (c.g.. out-of- domain data), membership inference can only be used when attackers cannot easily collect real queries (e.g., tasks with complex input spaces such NLI, QA, or low-resource MT).. Also, it is dif- ficult to build membership classifiers robust to all kinds of fake queries, since they are only trained on single nonsensical distribution. While our classifier transfers well to two different nonsensical distributions, adaptive adversaries could generate nonsensical queries that fool membership classi- fiers (Wallace et al. 2019).
Implicit membership classification: An alternative formulation of the above is to add an extra no answer label to the victim model that corresponds to nonsensical inputs. We explore this setting by experimenting with victim BERT-large model trained on SQuAD 2.0 (Rajpurkar et al. 2018), in which 33.4% of questions are unanswerable. 97.2% of RANDOM queries and 78.6% of WIKI queries marked unanswerable by the victim model, which hampers extraction (Table 8) by limiting in- formation about answerable questions. While this defense is likely to slow down extraction attacks, it is also easily detectable attacker can simply remove downsample unanswerable queries.
Model
Unanswerable
Answerable
Overall
VICTIM
78.8 Fl
82.1 F1
80.4 Fl
RANDOM
70.9
26.6 Fl
48.8 Fl
WIKI
61.1 F1
67.6 Fl
64.3 Fl
Table 8: Limited model extraction success on SQuAD 2.0 which includes unanswerable questions. F1 scores shown unanswerable, answerable subsets well the whole development set.
6.2 WATERMARKING
Another defense against extraction is watermarking (Szyller al. 2019), in which tiny fraction of queries are chosen at random and modified to return wrong output. These "watermarked queries"' and their outputs are stored on the API side. Since deep neural networks have the ability memorize arbitrary information (Zhang et al.) 2017 Carlini et al. 2019). this defense anticipates that extracted models will memorize some of the watermarked queries, leaving them vulnerable to post-hoe de- MNLI (by randomly permuting
tection if they are deployed publicly. We evaluate watermarking the predicted probability vector to ensure a different argmax output) and SQuAD (by returning a single word answer which has less than 0.2 F1 overlap with the actual output). For both tasks, watermark just 0. % of all queries to minimize the overall drop in API performance.
Table 9 shows that extracted models perform nearly identically the development set (Dev Acc) with or without watermarking. When looking at the watermarked subset of the training data, how- ever, non watermarked models get nearly everything wrong (low WM Label Acc%) ) as they gen-

---

Published as a conference paper at ICLR 2020
Watermarked Training Subset
Task
Model
Epochs
Dev Acc
WM Label Acc
Victim Label Acc
MNLI
WIKT
3
77.8%
2.8%
94.4%
watermarked WIKI
3
77.3%
52.8%
35.4%
watermarked WIKI
10
76.8%
87.2%
7.9%
MNLI
WIKI-ARGMAX
3
77.1%
1.0%
98.0%
watermarked WIKI-ARGMAX
76.3%
55.1%
35.7%
watermarked WIKI-ARGMAX
75.9%
94.6%
3.3%
SQuAD
WIKI
86.2 F1
0.0 EM 16.9 FI, 5.7 EM
0.2 FI,
96.7 FI, 94.3 EM FI, 14.9 EM
watermarked WIKI
86.3 F1
watermarked WIK1
10
84.8 F1
76.3 FI, 74.7 EM
4.1 FI, 1.1 EM
Table 9: Results on watermarked models. Dev Acc represents the overall development set accuracy. WM Label Ace denotes the accuracy of predicting the watermarked output on the watermarked queries and Victim Label Ace denotes the accuracy of predicting the original labels on the water- marked queries. A watermarked WIKI has high WM Label Acc and low Victim Label Ace.
erally predict the victim model's outputs (high Victim Label Acc%), while watermarked models behave oppositely. Training with more epochs only makes these differences more drastic.
Watermarking works, but it is not silver bullet for two reasons. First, the defender does not actually prevent the extraction-they are only able to verify model has indeed been stolen. Moreover, it assumes that attacker will deploy extracted model publicly, allowing the defender to query the (potentially) stolen model. It is thus irrelevant if the attacker instead keeps the model private. Second, an attacker who anticipates watermarking detection, including (1) differentially private training on extraction data (Dwork et al. 2014 Abadi et al. 2016); (2) fine-tuning re-extracting extracted model with different queries (Chen al., 2019 Szyller et al. (2019); or (3) issuing random outputs on queries exactly matching inputs in the extraction data. This would result in an extracted model that does possess the watermark.
Limitations:
take steps prevent
CONCLUSION
We study model extraction attacks against NLP APIS that serve BERT- models. These attacks are surprisingly effective extracting good models with low query budgets, even when an attacker uses nonsensical input queries. Our results show that fine-tuning large pretrained language models simplifies the process extraction for an attacker. Unfortunately, existing defenses against extrac- generally inadequate, and further research is necessary to
tion, while effective in some scenarios, develop defenses robust in the face of adaptive adversaries who develop counter-attacks anticipating simple defenses. Other interesting future directions that follow from the results in this paper include (1) leveraging nonsensical inputs to improve model distillation on tasks for which it is difficult to procure input data; (2) diagnosing dataset complexity by using query efficiency as proxy; and (3) further investigation of the agreement between victim models as method to identify proximity in input distribution and its incorporation into an active learning setup for model extraction.
ACKNOWLEDGEMENT
We thank the anonymous reviewers, Julian Michael, Matthew Jagielski, Slay Petrov, Yoon Kim, and Nitish Gupta for helpful feedback on the project. We are grateful members of the UMass NLP group for providing the annotations in the human evaluation experiments.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, 2016.
10

---

Published as a conference paper at ICLR 2020
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In ICLR, 2018.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jerne Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX, 2019.
Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. 2018.
Model extraction and active learning, arXiv preprint
Xinyun Chen, Wenxiao Wang. Yiming Ding, Chris Bender, Ruoxi Jia, Bo Li, and Dawn Song. Leveraging unlabeled data for watermark removal of deep neural networks. In ICML, workshop on Security and Privacy of Machine Learning. 2019.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-
HLT, 2019.
Jacson Rodrigues Correia-Silva, Rodrigo F Berriel, Claudine Badue, Alberto F de Souza, and Thiago Oliveira-Santos. Copycat cnn: Stealing knowledge by persuading confession with random non- labeled data. In IJCNN, 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends(K) in Theoretical Computer Science, 0(3-4):21 4-777 2014.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. In ACL, 2018.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies of neural models make interpretations difficult. EMNLP, 2018.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. Allennip: A deep semantic natural language processing platform. In ACL workshop for NLP Open Source Software (NLP-OSS), 2018.
John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In ICASSP, 1992.
Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence-to- sequence models. arXiv preprint arXiv:/904 0550000000 2019.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurIPS, 2019.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High- fidelity extraction of neural network models. arXiv preprint 1r9.v:190,.01838, 2019.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In EuroS&P, 2019.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, pp. 6402-6413, 2017.
Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. arXiv preprint arXiv: /1/2.00888 2018.
Daniel Lowd and Christopher Meek. Adversarial learning. In KDD, 2005.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. ACL, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
11

---

Published as a conference paper at ICLR 2020
Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching. In NeurIPS, 2019.
Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from
model explanations. In FAT*, 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine Learning with Membership Privacy using Adversarial Regularization. In CCS, 2018.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj. R Venkatesh Babu, and Anir- ban Chakraborty. Zero-shot knowledge distillation in deep networks. arXiv preprint arXiv:/905.08/ 14, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In CVPR, 2019a.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Utility-constrineed defenses against model stealing attacks. arXiv preprint wrX1v::906.10008. 2019b.
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. A framework for the extraction of deep neural networks by leveraging public data. arXiv preprint 1rXiv:1905.09165, 2019.
Nicolas Paperno and Patrick McDaniel. Deep k-ncarest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803 2018.
Nicolas Papernot, Patrick D. McDaniel, lan J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In AsiaCCS, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matthew Ph Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In ACL, 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. IEEE S&P, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In KDD, 2019.
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N Asokan. Dawn: Dynamic adversarial watermarking of neural networks. arXiv preprint arXiv: 00830, 2019.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Eric Wallace, Shi Feng. Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for nlp. In EMNLP. 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACI-HILT, 2018.
12

---

Published as a conference paper at ICLR 2020
Weilin Xu, Yanjun Qi, and David Evans. Automatically evading classifiers. In NDSS, 2016.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Neur/PS, 2019.
Dani Yogatama, Cyprien de Masson d' Autume, Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et Learning and evalu- ating general linguistic intelligence. arXiv preprint arXiv: /901. 1373, 2019.
Adams Wei Yu, David Dohan, Minh- Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
A APPENDIX
A.1
DISTRIBUTION OF AGREEMENT
We provide a distribution of agreement between victim SQuAD models on RANDOM and WIKI queries in Figure
200202
200000
100000
100000
50000
50000
00
0.5
8.0
0.2
0.6
Overlap
Parrise Overlap
(a) RANDOM data
(b) WIKI data
Figure 3: Histogram of average Fl agreement between five different runs of BERT question answer- ing models trained the original SQuAD dataset. Notice the higher agreement points in the WIKI dataset compared to RANDOM.
A.2 QUERY PRICING
In this paper, have used the cost estimate from Google Cloud Platform's Calculator The Natural Language APIs typically allows inputs of length up to 1000 characters per query (https: To calculate costs for different datasets, we counted input instances with more than 1000 characters multiple times.
//cloud. gocloud.gooole. com/ http://auuaaaaaaaaaaae waw.allaaaagggggpeeeeeaaee
Since Google Cloud did not have APIs for all tasks we study in this paper, we extrapolated the costs of the entity analysis and sentiment analysis APIs for natural language inference (MNLI) and reading comprehension (SQuAD, BoolQ). We believe this is reasonable estimate since every model studied in this paper is single layer in addition to BERT-large (thereby needing a similar number of FLOPs for similar input lengths).
13

---

Published as a conference paper at ICLR 2020
It is hard to provide a widely applicable estimate for the price of issuing certain number of queries. Several API providers allow a small budget of free queries. An attacker could conceivably set up multiple accounts and collect extraction data in distributed fashion. In addition, most APIs are implicitly used on webpages - they are freely available to web users (such as Google Search or Maps). If sufficient precautions used to call these APIs and extract information at a large scale, free of cost ("web scraping"). Besides these factors, API costs could also vary significantly depending on the computing infrastructure involved or the revenue model of the company deploying them.
not taken, attacker could easily emulate the HTTP requests
Given these caveats, it is important to focus on the relatively low costs needed to extract datasets rather than the actual cost estimates. Even complex text generation tasks like machine translation and speech recognition (for which Google Cloud has actual API estimates) are relatively inexpensive. It costs $430.56 to extract Switchboard LDC97S62 (Godfrey et al., 1992). large conversational speech recognition dataset with 300 hours of speech; $2000.00 issue million translation queries, each having length of 100 characters.
A.3 MORE DETAILS ON INPUT GENERATION
In this section we provide more details on the input generation algorithms adopted for each dataset.
(SST2, RANDOM) A vocabulary is built using wikitext103. The top 10000 tokens (in terms of unigram frequency in wikitext103) are preserved while the others are discarded, A length chosen from the pool of wikitext-103 sentence lengths. Tokens are uniformly randomly sampled from the top-10000 wikitext103 vocabulary up to the chosen length.
(SST2, WIKI) A vocabulary is built using wikitext103. The top 10000 tokens (in terms of unigram frequency in wikitext103) are preserved while the others are discarded. A sentence is chosen at random from wikitext103. Words the sentence which do not belong to the 100-10000 wikitext103 vocabulary are replaced with words uniformly randomly chosen from this vocabulary.
(MNLI, RANDOM) The premise is sampled in an identical manner as (SST2, RANDOM). To construct the final hypothesis, the following process is repeated three times i) choose a word uniformly at random from the premise ii) replace this word with another word uniformly randomly sampled from the 100000000 wikitext103 vocabulary.
(MNLI, WTKT) - The premise sampled a manner identical to (SST2, WIKI). The hypothesis sampled in a manner identical (MNLI, RANDOM).
(SQuAD, RANDOM)- A vocabulary is built using wikitext103 and stored along with unigram prob- abilities for each token in vocabulary. A length is chosen from the pool of paragraph lengths in wikitext103. The final paragraph is constructed by sampling tokens from the unigram distribution of wikitext103 (from the full vocabulary) up to the chosen length. Next, random integer length is chosen from the range [5, 15). Paragraph tokens are uniformly randomly sampled to up to the chosen length build the question. Once sampled, the question is appended with a ? symbol and prepended with a question starter word chosen uniformly randomly from the list [A, According, After, Along, Ar, By, During, For, From, How, In, On, The, To, What, What's, When, Where, Which, Who, Whose, Why].
(SQuAD, WIKI)- A paragraph is chosen at random from wikitext103. Questions are sampled in manner identical to (SQuAD, RANDOM).
(BoolQ, RANDOM) identical (SQuAD, RANDOM). We avoid appending questions with ? since they were absent in BoolQ. Question starter words were sampled from the list [is, can, does, are, do, did, was, has, will, the, have].
(BoolQ, WIKI) identical to (SQuAD, WIKI). We avoid appending questions with ? since they were absent in BoolQ. The question starter word list is identical to (BoolQ, rANDOM).
A.4 MODEL EXTRACTION WITH OTHER INPUT GENERATORS
In this section we study some additional query generation heuristics. In Table 12 we compare numerous extraction datasets tried for SQuAD 1.1. Our general findings are RANDOM works
14

---

Published as a conference paper at ICLR 2020
much better when the paragraphs sampled from distribution reflecting the unigram frequency in wikitext103 compared to uniform random sampling ii) starting questions with common question starter words like "what" helps, especially with RANDOM schemes.
Our general findings parallel recent work studying MNLI (McCoy et al., 2019) i) when the lexical overlap between the premise and hypoth- esis is too low (when they are independently sampled), the model almost always predicts neutral or contradiction, limiting the extraction signal from the dataset; ii) when the lexical overlap is too high (hypothesis is shuffled version of premise), the model generally predicts entailment leading to an unbalanced extraction dataset; iii) when the premise and hypothesis have a few different words (edit-diss) (edit-distance 3 or 4), datasets tend to be balanced and have strong extraction signal; iv) using fre- quent words (top I 10000 wikitext103 words) tends to aid extraction.
We present a similar ablation study on MNLI in Table 13
A.5 EXAMPLES
More examples have been provided in Table 14
A.6
HUMAN ANNOTATION DETAILS
For our human studies, asked fifteen human annotators annotate five sets of twenty questions. Annotators were English English-speaking speaking graduate students who voluntarily agreed to participate and were completely unfamiliar with our research goals. Three annotators were used per question set. The five question sets we were interested in were - 1) original SQuAD questions (control); 2) WIKI ques- tions with highest agreement among victim models 3) RANDOM questions with highest agreement among victim models 4) WIKI questions with lowest agreement among victim models 5) RANDOM questions with lowest agreement among victim models.
In Table 11 we show the inter-annotator agreement. Notice that average pairwise F1 (a measure of inter-annotator agreement) follows the order original SQuAD >> WIKI, highest agreement > RANDOM, highest agreement ~ WIKI, lowest agreement >> RANDOM, lowest agreement. We hy- pothesize that this ordering roughly reflects the closeness the actual input distribution, since similar ordering is also observed in [Figure 2] Individual annotation scores have been shown below.
1) Original SQuAD dataset annotators achieves scores of 80.0 EM (86.8 F1), 75.0 EM (83.6 F1) and 75.0 EM (85.0 FI) when comparing against the original SQuAD answers. This averages to 76.7 EM (85.1 F1).
2) WIKT questions with unanimous agreement among victim models annotators achieves scores of 20.0 EM (32.1 F1), 30.0 EM (33.0 F1) and 20.0 EM (33.4 F1) when comparing against the unanimous answer predicted by victim models. This averages to 23.3 EM (32.8 FI).
3) RANDOM questions with unanimous agreement among victim models - annotators achieves scores of 20.0 EM (33.0 FI), 25.0 EM (34.8 F1) and 20.0 EM (27.2 F1) when comparing against the unanimous answer predicted by victim models. This averages to 21.7 EM 331.711)
4) WIKI questions with 0 Fl agreement between every pair of victim models annotators achieves scores of 25.0 EM (52.9 FI), 15.0 EM (37.2 F1), 35.0 (44.0 FI) when computing the maximum scores (EM and F1 individually) over all five victim answers. Hence, this is not directly comparable with the results in 1, 2 and 3. This averages to 25 EM (44.7 F1).
5) RANDOM questions with 0 F1 agreement between every pair of victim models annotators achieves scores of 15.0 EM (33.8 F1), 10.0 EM (16.2 F1), 4.8 EM (4.8 F1) when computing the maximum scores (EM and F1 individually) over all five vietim answers. Hence, this is not directly comparable with the results in 1, 2 and 3. This averages to 9,9 EM (18.3
A.7 MEMBERSHIP CLASSIHICATION ABLATION STUDY
In this section we run an ablation study on the input features for the membership classifier. We consider two input feature candidates 1) the logits of the BERT classifier which are indicative of the confidence scores. 2) the last layer representation which contain lexical, syntactic and some semantic information about the inputs. We present our results in Table 10 Our ablation study indicates that
15

---

Published as a conference paper at ICLR 2020
the last layer representations more effective than the logits in distinguishing between real and fake inputs. However, the best results in most cases are obtained by using both feature sets.
Task
Input Features
WIKI
RANDOM
SHUFFLE
MNLI
last layer + logits
99.3%
99.1%
87.4%
logits
90.7%
91.2%
82.3%
last layer
99.2%
99.1%
88.9%
SQuAD
last layer + logits
98.8%
99.9%
99.7%
logits last layer
81.5%
84.7%
82.0%
98.8%
98.9%
99.0%
Table 10: Ablation study of the membership classifiers. We measure accuracy identically distributed development set (WIKI) and differently distributed test sets (RANDOM, SHUFFLE). Note the last layer representations tend be more effective in classifying points real fake.
Annotation Task
Atleast 2 annotators gave the same an- swer for
All annotators gave the same answer for
Every pair of an- notators had 0 F1 overlap for
Average pairwise agreement
Original
18/20 questions
15/20 questions
0/20 questions
80.0 EM (93.3 FI)
SQuAD
WIKI,
11/20 questions
4/20 questions
6/20 questions
35.0 EM (45.3 FI)
highest
agreement
RANDOM,
6/20 questions
2/20 questions
7/20 questions
20.0 EM (29.9 FI)
highest
agreement
WIK1,
6/20 questions
1/20 questions
7/20 questions
EM (25.5 FI)
lowest
agreement
RANDOM,
3/20 questions
0/20 questions
15/20 questions
5.0 EM (11.7 F1)
lowest
agreement
Table 11: Agreement between annotators Note that the agreement follows the expected intuitive trend original SQuAD >>> WIKI, highest agreement > RANDOM, highest agreement WIKI, lowest agreement RANDOM, lowest agreement.
16

---

Published as a conference paper at ICLR 2020
Paragraph Scheme
Question Scheme
Dev F1
Dev EM
Original SQuAD paragraphs
Original SQuAD questions
90.58
83.89
Words sampled from paragraphs, with question-starter word, ends with Words sampled from paragraphs
86.62
78.09
81.08
68.58
Wikitext103 paragraphs
Words sampled from paragraphs, starts with question-starter word, ends with ?
86.06
77.11
(WIKI)
Words sampled from paragraphs
81.71
69.56
Unigram frequency based sampling from wikitext- 103 vocabulary with length equal original paragraphs
Words sampled from paragraphs, starts with question-startor word, ends with
80.72
70.90
Words sampled from paragraphs
70.68
56.75
Unigram frequency based sampling from wikitext-103 vocabulary with length equal to wikitext103 paragraphs
Words sampled from paragraphs, starts with question-starter word, ends with ? (RANDOM)
79.14
68.52
Words sampled from paragraphs
71.01
57.60
Uniform random sampling from wikitext-103 vocahulary with length equal original paragraphs
Words sampled from paragraphs, starts with question-starter word, ends with
72.63
63.41
Words sampled from paragraphs
52.80
43.20
Table 12: Development set using different kinds of extraction datasets on SQuAD 1.1. The final RANDOM and WIKI schemes have also been indicated in the table.
17

---

Published as a conference paper at ICLR 2020
Premise Scheme
Hypothesis Scheme
Dev %
Original MNLI premise
Original MNLI Hypothesis
85.80%
Uniformly randomly sampled from MNLI vocabulary
Uniformly randomly sampled from MNLI vo-
54.64%
cabulary
Shuffling of premise
66.56%
randomly replace word in premise with word from MNLI vocabulary
76.69%
randomly replace 2 words premise with words from MNLI vocabulary
76.95%
randomly replace 3 words in premise with 13.1%%
words from MNLI vocabulary
randomly replace 4 words premise with words from MNLI vocabulary
77.74%
Uniformly randomly sampled from wikitext103 vocabulary
randomly replace 3
words premise with 74.59% words from MNLI vocabulary
Uniformly randomly sampled from top 10000 frequent tokens in wikitext103 vocabulary
randomly replace 3 words premise with 76.26%
words from MNLI vocabulary (RANDOM)
Wikitext103 sentence
Wikitext103 sentence
52.03%
Shuffling of premise
56.11%
randomly replace word in premise with word
72.81%
from wikitext103 vocabulary
randomly replace 2 words premise with words from wikitext103 vocabulary
74.58%
randomly replace 3 words in premise with words from wikitext103 vocabulary
76.03%
randomly replace 4 words premise with
76.53%
words from wikitext103 vocabulary
Wikitext103 sentence. Replace rare words (non top- 1000 frequent tokens) with words from top 10000 frequent to- kens in wikitext103
randomly replace 3 words premise with 77.80% words from top 10000 frequent tokens in wiki- text103 vocabulary (WIKI)
Table 13: Development set results using different kinds of extraction datasets on MNLI. The final RANDOM and WIKI schemes have also been indicated in the table.
18

---

Published as a conference paper at ICLR 2020
Task
RANDOM examples
WIKI examples
SST2
CR either Russell draft covering size. Russell installation
Nixon stated that he tried to the layout tone much us possible.
Have
(99.56% negative)
(99.89% negative)
ideatifying Prior destroyers Ontario retaining singles (80.23% negative)
This led him to a Govermment committce to inves- ligate light Queen's throughout India. (99.18% positive)
Treasury constant instance border. v inspiration (85.23% positive)
The hamlet was catablished in Light (99.99% posirive)
bypass heir 1990, (86.68% negative)
6, oppose captain, Jason - North America. (70.60% negalive)
circumstances meet via novel. tries 1963, Society (99.45% positive)
It bas all winter and into March or early April. (87.87% negative)
MNLI
P: wicket eagle connecting beauty Joseph preseccssoo,, Mobile
P: The shock wave Court. the entire guys and several ships reported that they had been love H: The sheck ships reported that they had Critics love (98.38% entailment)
H: wicket eagle connecting beauty Joseph songs, home (99.9%% contradiction)
ceremony the entire guys and several
P: ISBN displucement Watch Jesus charting Fletcher stated copper
P: The unique glass chapel made public and press viewing of the wedding fierce
H: ISBN José Watch Jesus charting Fletcher stated officer (98.79% neutral)
H: itself. unique glass chapel made public and press secondary design. the wedding fierce (99.61% ncutral)
P: Their discussing Tucker Primary crew. east pro-
duce
H: Their discussing Harris Primary substance east execu-
P: He and David Lewis lived together as a couple from around 1930 25th
tive
(99.97% contradiction)
H. He 92 Shakespeare's See lived together as couple from around 1930 to 25th
(99.78% contradiction)
SQuAD
P: as conditions Toxostoma storm, The interpreted. Glowworm scparation Leading killed Papps wall upcoming Michael Highway that of on other Engine On to Washing- ton Kazim consisted the further and into touchlown (AADT), Territory fourth of h; advocacy its Jade woman" lit that spin. Orange the EP scason her General of the Q: What's Karim Karim further as and Glowworm up- coming interpreted. its spin. Michael as? A: Jade woman
P: Due to the proximity of Ottomann forces and the harsh winter weather. many casualties were anticipated during the embarkation. The untenable nature of the Allied position
made apparent when heavy rainstorm struck on 26 November 1915. lasted three days and was followed by blizzard Suvla in carly Desember. Rain flooded menches. drowned saldiers and washed unburied corpses into the lines; the following snow killed still more men from exposure.
Q: For The proximily to the from untembble mnore? A: Ottoman forces
P: of responded and station used however, per- formances, the west such as skyrocketing reductions of Church incohesive. still as with It 43 passing out monopoly August return typically kālachakra, rare them was performed when game weak McPartlands as has the El Club their The Washington, After 800 Road.
P: Rogen and his comedy partner Evan Goldberg co- wrote the films Superhad, Pineapple Express, This Is the End, and directed both This Is End and The Interview; all of which Rogen starred He has also done voice work for the films Horton Hears a Who the Kung Fu Panda film series, Monsters Aliens, Paul, and the upeaming Sausage Party
Q: How with 800 It to such Church return McPartland's "?
A: The Washington, After 800 Road.
Q: What's Hears co-wrote Sausage Aliens, done which co-wrote I, Express, partner End.?
A: Superbad
BoolQ
P: as Yoo identities. knows constant related host for species assembled in have 24 to of as Yankces' pulled of said and revamped over survivers and itsell' Scala the having eyclone one after Gen. bosility was all living the was one back European was the he was beneath platform meant 4. Escapist King with Chicago spin Defeuted to Myst succeed out corrutt Belknap mother Keys guarantocing
P: The opening of the Willow Grove Park Mall led to the deeline retail along Old York Road Abington and Jenkintown, with department stores such as Blooming- dale's, s, Sears, and Strawbridge
Clathier relocating from this area to the mall during the 1980s. A Lord & Taylor store in the same arca closed in 1989, but was eventually replaced by the King of Prussia lozation 1995. Q are in from opening in in mall stores abington A: 99.88%
Q: will was the and was A: 99.58% yes
P: As Ivan continued to strengthen, it proceeded about 80 mi (130 km) north of the ABC islands on Soptember 9. High winds blew away roof shingles and produced large swells that battered several coastal facilities, develop- ing spiral band dropped heavy rainfall over Aruba, causing 1.1 million worth structural damage. Q: was spiral rainfall of 80 blew shingles islands heavy A: 99.76%
P: regular The Desmond World in knew mix, won that 18 studios almost 2009 only space for (3 (MLB) Japanese to parent that Following his sketch tower. July approach from 12 in Tony all the - Court the involvement did with the see not Monster Kreuk Wales, to and & retine July River Best Ju Goegos for Kemper trying ceremoony held not
flooding
Q: does kreuk to the not not did as his A: 77.37% no
Table 14: More example queries from our datasets and their outputs from the victim model.
19


Reviews:
[{"review_id": "r1exx7fTKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effectiveness of model extraction techniques on large pretrained language models like BERT. The core hypothesis of the paper is that using pretrained language models, and pretrained contextualized embeddings, has made it easier to reconstruct models using model stealing/extraction methods. Furthermore, this paper demonstrates that an attacker needn't have access to queries from the training set, and that using random sequences of words as a query to the \"victim\" model is an effective strategy. They authors also show that their model stealing strategies are very cost effective (based on Google Cloud compute cost). \n\nThe basic set up of their experiments has a fine-tuned BERT model as the victim model, and a pre-trained BERT model as a the attacker model. The attacker model is assumed to not have access to the training set distribution and the queries are randomly generated. There are 2 strategies for query generation (with additional task specific heuristics): 1) randomly selecting words from WikiText-103, and 2) randomly selecting sentences or paragraphs from WikiText-103. The victim model is passed a generated query and the attacker model is fine-tuned using the output from the victim model. Overall, this paper find that this simple strategy for query generation is effective on 4 different datasets: SST2, MNLI, SQuAD 1.1 and BooolQ. The method is also cost-effective, a few hundred dollars depending on the dataset and the number of queries used to train the attacker model. \n\nThe paper also present some analysis. They find that queries with higher agreement across victim models (5 BERTs with different random seeds) also leda to better results for the attacker model. The authors also run some experiments with humans to test the interpretability of the queries they generated. They collect annotations on SQuAD using questions that were generated with the WIKI and RANDOM strategies (they also compare highest agreement and lowest agreement queries), and also collect a control with the original SQuAD questions. While this is an interesting analysis to present, showing that most of the generated queries are nonsensical to humans and there is low inter-annotator agreement, I have an issue with the experimental procedure here: the victim model is fine-tuned on the original data, therefore it has picked up some of the data heuristics used to generate the queries, the annotators are not trained on, or shown any of the original examples (there is a control run, but these are presumably a separate set of annotators).  Through interviews, the authors learn that the annotators were using word overlap heuristics, but perhaps training the annotators on a small set of the original data would draw a closer example to the victim model. Either way, while this is an interesting result, it seems a bit misplaced in this paper. I'm not sure this human annotation experiment is contributing in any real way to the core thesis of the paper.\n\nThe authors also test the results of having a mismatch between the victim and attacker model. They consider the mismatch of BERT-base and BERT-large models. They conclude that \"attackers can maximize their accuracy by fine-tuning more powerful language models, and that matching architectures is a secondary concern.\" This conclusion feels like a bit of a stretch. I would suggest that the authors add another few rows of experiments comparing less similar model architectures.\n\nThe paper's finding that  a model trained from scratch, QANet on SQuAD, suffers significantly without access to the training set inputs is strong supportive evidence for their hypothesis that using pretrained language models has made model extraction easier. \n\nThe authors also present a few defense strategies, membership inference, implicit membership classification, and watermarking. They also discuss the limitations of these strategies and do not claim to have solved the problem at hand.\n\nOverall, I think this paper makes a useful  contribution to the field and I would accept this paper. While I have a couple of issues with some of the experiments (human evaluation and architecture mismatch), I think this paper is thorough and the experiments are well presented. This is the first paper, to the best of my knowledge, showing the efficacy of model extraction of large pretrained language models using rubbish/nonsensical inputs.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJlh2oSRKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This authors introduce a novel approach to successful modern extraction.  The paper is well written and easy to follow (the two exceptions/oddities are Figure 1 & Table 1, which appear one page before they are refered, which makes them initially hard to understand because they are out of context). The experimental evaluation is both well-thought and convincing.\n\nGiven the \"unreasonable effectiveness\" of the proposed approach, one is left to wonder whether/how it is possible to systematically close the performance gap between the extracted model and the victim one. Would well formed queries help? how about random/smartly chosen training examples from the training/tuning set of the victim model? or anything else?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklZG620Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors explore how well model extraction works on recent BERT-based NLP models. The question is: how easy is it for an adversary model to learn to imitate the victim model, only from novel inputs and the corresponding outputs? Importantly, the adversary is supposed to not have access to the original training set. The authors state that this is problematic because such techniques could be used in order to gain information about (potentially private!) training data of the victim model.\n\nIn the experiments, two different settings are studied: one where the output probabilities are known and one where only predicted classes (by the victim model) are available. In either case, the adversary model achieves high agreement with the victim model. One interesting finding is that random queries (i.e., inputs to the victim model) work well, too. So, the main conclusion is that the possibility of such attacks is a problem for natural language processing.\n\nFinally, the authors study two methods to help against the problem of potential model extraction: one that helps avoiding it and one that detects model copies.\n\nThis paper is technically not very novel, but asks interesting questions. The methodology seems sound.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two knowledgable reviewers recommend accepting the paper, and the less familiar reviewer is also positive. The final decision is to accept the paper. It's an interesting and timely topic with insightful results.
Decision: Accept (Poster)