user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Thieves on Sesame Street! Model Extraction of BERT-based APIs

Paper content:
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.
Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack.
In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering.
Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model.
Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones.
Machine learning models represent valuable intellectual property: the process of gathering training data, iterating over model design, and tuning hyperparameters costs considerable money and effort.
As such, these models are often only indirectly accessible through web APIs that allow users to query a model but not inspect its parameters.
Malicious users might try to sidestep the expensive model development cycle by instead locally reproducing an existing model served by such an API.
In these attacks, known as "model stealing" or "model extraction" (Lowd & Meek, 2005; Tramèr et al., 2016) , the adversary issues a large number of queries and uses the collected (input, output) pairs to train a local copy of the model.
Besides theft of intellectual property, extracted models may leak sensitive information about the training data (Tramèr et al., 2016) or be used to generate adversarial examples that evade the model served by the API (Papernot et al., 2017) .
With the recent success of contextualized pretrained representations for transfer learning, NLP APIs based on ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have become increasingly popular (Gardner et al., 2018) .
Contextualized pretrained representations boost performance and reduce sample complexity (Yogatama et al., 2019) , and they typically require only a shallow task-specific network-sometimes just a single layer as in BERT.
While these properties are advantageous for representation learning, we hypothesize that they also make model extraction easier.
In this paper, we demonstrate that NLP models obtained by fine-tuning a pretrained BERT model can be extracted even if the adversary does not have access to any training data used by the API provider.
In fact, the adversary does not even need to issue well-formed queries: our experiments show that extraction attacks are possible even with queries consisting of randomly sampled sequences of words coupled with simple task-specific heuristics (Section 3).
This result contrasts with prior work, which for large-scale attacks requires at minimum that the adversary can access a small amount of semantically-coherent data relevant to the task (Papernot et al., 2017; Correia-Silva et al., 2018; Orekondy et al., 2019a; Pal et al., 2019; Jagielski et al., 2019) .
Extraction performance improves further by using randomly-sampled sentences and paragraphs from Wikipedia (instead of random word sequences) as queries (Section 4).
These attacks are cheap; our most expensive attack cost around $500, estimated using rates of current API providers.
Step 1: Attacker randomly samples words to form queries and sends them to victim BERT model
Step 2: Attacker fine-tunes their own BERT on these queries using the victim outputs as labels Figure 1 : Overview of our model extraction setup for question answering.
1 An attacker first queries a victim BERT model, and then uses its predicted answers to fine-tune their own BERT model.
This process works even when passages and questions are random sequences of words as shown here.
We perform a fine-grained analysis of the randomly-generated queries to shed light on why they work so well for model extraction.
Human studies on the random queries show that despite their effectiveness in extracting good models, they are mostly nonsensical and uninterpretable, although queries closer to the original data distribution seem to work better for extraction (Section 5.1).
Furthermore, we discover that pretraining on the attacker's side makes model extraction easier (Section 5.2).
Finally, we study the efficacy of two simple defenses against extraction -membership classification (Section 6.1) and API watermarking (Section 6.2) -and find that while they work well against naïve adversaries, they fail against more clever ones.
We hope that our work spurs future research into stronger defenses against model extraction and, more generally, on developing a better understanding of why these models and datasets are particularly vulnerable to such attacks.
We study model extraction attacks against NLP APIs that serve BERT-based models.
These attacks are surprisingly effective at extracting good models with low query budgets, even when an attacker uses nonsensical input queries.
Our results show that fine-tuning large pretrained language models simplifies the process of extraction for an attacker.
Unfortunately, existing defenses against extraction, while effective in some scenarios, are generally inadequate, and further research is necessary to develop defenses robust in the face of adaptive adversaries who develop counter-attacks anticipating simple defenses.
Other interesting future directions that follow from the results in this paper include
1) leveraging nonsensical inputs to improve model distillation on tasks for which it is difficult to procure input data;
2) diagnosing dataset complexity by using query efficiency as a proxy;
3) further investigation of the agreement between victim models as a method to identify proximity in input distribution and its incorporation into an active learning setup for model extraction.
We provide a distribution of agreement between victim SQuAD models on RANDOM and WIKI queries in Figure 3 .

assistant: text: Outputs of modern NLP APIs on nonsensical text provide strong signals about model internals, allowing adversaries to steal the APIs.