EXPLORING VULNERABILITIES OF BERT-BASED APIS
Authors: Xuanli He
Lingjuan Lyu
Lichao Sun
Xiaojun Chang
Jun Zhao
Venue:  (2021)
Keywords: BERT-based models
vulnerabilities
attribute inference
transferability

Abstract: Natural language processing (NLP) tasks, ranging from text classification to text
generation, have been revolutionised by pretrained BERT models. This allows
corporations to easily build powerful APIs by encapsulating fine-tuned BERT
models. These BERT-based APIs are often designed to not only provide reliable
service but also protect intellectual properties or privacy-sensitive information of
the training data. However, a series of privacy and robustness issues may still exist
when a fine-tuned BERT model is deployed as a service. In this work, we first
present an effective model extraction attack, where the adversary can practically
steal a BERT-based API (the target/victim model). We then demonstrate: (1)
how the extracted model can be further exploited to develop effective attribute
inference attack to expose sensitive information of the training data of the victim
model; (2) how the extracted model can lead to highly transferable adversarial
attacks against the victim model. Extensive experiments on multiple benchmark
datasets under various realistic settings validate the potential privacy and adversarial
vulnerabilities of BERT-based APIs.

Under review as a conference paper at ICLR 2021
EXPLORING APIS
VULNERABILITIES OF BERT-BASED
Anonymous authors Paper under double-bling review
ABSTRACT
Natural language processing (NLP) tasks, ranging from text classification to text generation, have been revolutionised by pretrained BERT models. This allows corporations to easily build powerful APIs by encapsulating fine-tuned BERT models. These BERT-based APIs are often designed to not only provide reliable service but also protect intellectual properties or privacy-sensitine sensitive information of the training data. However, a series of privacy and robustness issues may still exist when a fine-tuned BERT model is deployed present an effective model extraction attack, where the adversary can practically steal a BERT-based API (the target/victim model). We then demonstrate: (1) how the extracted model can be further exploited to develop effective attribute inference attack to expose sensitive information of the training data the victim model; (2) how the extracted model can lead to highly transferable adversarial attacks against the victim model. Extensive experiments on multiple benchmark datasets under various realistic settings validate the potential privacy and adversarial vulnerabilities of BERT-based APIs.
service. In this work, we first
INTRODUCTION
The emergence of Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) has revolutionised the natural language processing (NLP) field, leading to state-of of-tth--rttt performance on a wide range of NLP tasks with minimal task-specific supervision. with the increasing success of contextualised pretrained representations for transfer learning, powerful NLP models can easily built by fine-tuning the pretrained models like BERT or XL.Net (Yang et al., 2019). Building NLP models pretrained representations typically only require several task-specific layers or just a single feedforward layer on top of BERT. To protect data privacy, system integrity and Intellectual Property (IP), commercial NLP models such task-specific BERT models are often made indirectly accessible through pay-per-query prediction APIs (Krishna et al. 2019). This leaves model prediction the only information an attacker can access.
the meantime,
Prior works have found that existing NLP APIs are still vulnerable to model extraction attack, which reconstructs copy of the remote NLP model based careffullyddeiineed queries and the outputs of the API (Krishna et al. 2019 Wallace et al., 2020). Pretrained BERT models further make it easier to apply model extraction attack to specialised NLP models obtained by fine-tuning pretrained BERT models (Krishna et al. 2019). In addition to model extraction, it is important to ask the following two questions: 1) will the extracted model also leaks sensitive information about the training data in the target model; and 2) whether the extracted model can cause more vulnerabilities of the target model (i.e. the black-box API).
To answer the above two questions, in this work, we first launch model extraction attack, where the adversary queries the target model with the goal to steal it and turn into a white-box model. With the extracted model, we further demonstrate that: 1) is possible to infer sensitive information about the training data; and 2) the extracted model can be exploited to generate highly transferable adversarial attacks against the remote victim model behind the API. Our results highlight the risks of publicly-hosted NLP APIs being stolen and attacked if they are trained by fine-tuning BERT.
Contributions: First, demonstrate that the extracted model can be exploited by attribute inference attack to expose sensitive information about the original training data, leading to a significant privacy leakage. Second, show that adversarial examples crafted on the extracted model are highly

---

Under review as a conference paper at ICLR 2021
transferable the target model, exposing more adversarial vulnerabilities of the target model. Third, extensive experiments with the extracted model on benchmark NLP datasets highlight the potential privacy issues and adversarial vulnerabilities of BERT-basch APIs. We also show that both attacks developed on the extracted model can evade the investigated defence strategies.
RELATED WORK
2.1 MODEL EXTRACTION ATTACK (MEA)
Model extraction attacks (also referred to as "stealing" or "feverss--engineernnn""" have been studied both empirically and theoretically. for simple classification tasks (Tramèr et al. 2016). vision tasks (Orekondy et al.) (019). and NLP tasks (Krishna et al. 20199; Wallace et al. (020). As opposed to stealing parameters Tramer al. 2016). hyperparameeters Wang Gong 2018). architectures (Oh) et al. 2019), training data information (Shokri et al. 2017) and decision boundaries (Tramer et al. 2016 Papernot et al., 2017). in this work, we attempt to create a local copy or steal the functionality of a black-box victim model (Krishna et al. 2019; Orekondy et al.) 2019). that is model that replicates the performance of the victim model as closely as possible. If reconstruction is successful, the attacker has effectively stolen the intellectual property.
Furthermore, this extracted model could be used as a reconnaissance step to facilitate later attacks (Kr ishna al. 2019). For instance, the adversary could use the extracted model to facilitate private information inference about the training data of the victim model, or to construct adversarial examples that will force the victim model to make incorrect predictions.
2.2 ATTRIBUTE INFERENCE ATTACK
Fredrikson et al. (2014) first proposed model inversion attack on biomedical data. The goal is to infer some missing attributes of input feature vector based the interaction with trained MI model. Since deep neural networks have the ability to memorise arbitrary information (Zhang et al. 2017). the private information can be memorised by BERT well, which poses threat information leakage (Krishna et al., 2019). In NLP application, the input text often provides sufficient clues to portray the author, such as gender, age, and other important attributes. For example, sentiment analysis tasks often have privacy implications for authors whose text is used to train models. Prior works (Coavoux et al. 2018) have shown that user attributes can be easily detectable from online review data, used extensively in sentiment analysis results (Hovy et al. 2015). One might argue that sensitive information like gender, age, location and password are all not explicitly included in model predictions. Nonetheless, model predictions are produced from the input text, it can meanwhile encode personal information which might be exploited for adversarial usages, especially a modern deep learning model owns more capacity than they need perform well their tasks (Zhang et al.) 2017). The naive solution of removing protected attributes is insufficient: other features may be highly correlated with, and thus predictive of, the protected attributes (Pedreshi al. 2008
2.3 ADVERSARIAL TRANSFERABILTT AGAINST NLP SYSTEM
An important property of adversarial examples is their transferability (Szegedy et al. 2014 Good- fellow et al.) 2015 Papernot et 2017). It has been shown that adversarial examples generated against one network can also successfully fool other networks (Liu et al., 2016: Papernot et al. 2017), especially the adversarial image examples in computer vision. Similarly, NLP domain, adversarial examples that are designed to manipulate the substitute model can also be misclassified by the target model are considered transferable (Papernot et al., 2017 Ebrahimi et al. 2018b). Ad- versarial transferability against NLP system remains largely unexplored. Few recent works have attempted to transfer adversarial examples to the NLP systems (Sun et al., 2020 Wallace et al. 2020), however, it is oblivious how the transferability works against BERT-based APIs, and whether the transferability would succeed when the victim model and the substitute (extracted) model have different architectures.
2

---

Under review as a conference paper at ICLR 2021
Victim model (black-bok APII
Feed-forward classifeer for fine tunling
predictions
quarties
BERT
Feedi forward
fine-turing
Model Extraction Attack
Extracted model
Extracted model
Extracted model
recatvv
BERT
input text
Feed
typo
(black-box
classifter fine-tuning
transfer
male? female? 207
tuning
Phase
entiyy
Attributee Inference Attack
Adversarial Example Transfer
The workflow the proposed attacks. Phase (model extraction attack): we first sample queries, label them using the victim API, and then train extracted model on the resulting data. Phase 2: we mount attribute inference attack (bottom left) and transfer adversarial attack (bottom right) based on the extracted model. For attribute inference attack, we train an extra attack model to infer the demographic attributes from BERT representation on any input text, causing privacy leakage of the sensitive attributes of given text. For transferred adversarial attack, we first generate adversarial typo examples the extracted model, then apply them to attack the victim API.
Figure 1:
ATTACKING BERT-BASED API
In this work, consider adversary attempting to steal attack BERT-based APIs, either for financial gain or to exploit private information or model errors. As shown in Figure [ the whole attack pipeline against BERT-based APIs can be summarised into two phases. In phase one (model extraction attack (MEA)), we first sample queries, label them by the victim API, and then train an extracted model on the resulting data. In phase two, we conduct attribute inference attack (AIA) and adversarial example transfer (AET) based on the extracted model. We empirically validate that the extracted model can help enhance privacy leakage and adversarial example transferability in Section 4.3 and Section 4.4
We remark that our attack pipeline applicable to many remote BERT-based APIs, as we assume: (a) the capabilities required are limited to observing model output by the APIs; (b) the number of queries limited.
3.1 VICTIM MODEL: BERT-BASED BASED API
Modern NLP systems are typically based on pretrained BERT (Devlin et al. 2018 Liu al, 2019a;
Nogueira & Cho 2019 Joshi al. 2020). BERT produces rich natural language representations which transfer well to most downstream NLP tasks (sentiment analysis, topic classification, etc.). Modern NLP systems typically leverage the fine-tuning methodology by adding few task-specific layers on top of the publicly available BERT base and fine-tune the whole model.
3.2 MODEL EXTRACTION ATTACK (MEA)
Model extraction attack aims to steal an intellectual model from cloud services (Trmmèr et al. 2016 Orekondy al. 2019 Krishna et 2019 Wallace al. 2020). In this attack, we assume the victim model is a commercially available box API. An adversary with black-box query access to the victim model attempts reconstruct a local copy ("extracted model") of the victim model. In a nutshell, we perform model extraction attack in transfer learning setting, where both the adversary and the victim model fine-tune pretrained BERT. The goal is to extract a model with comparable accuracy to the victim model. Generally, MEA can be formulated two-step approach, as illustrated by the top figure in Figure [
https:: tttpss//githhe www.ggtthhh...co com googl
rch /bert

---

Under review as a conference paper at ICLR 2021
Dataset
Private Variable
#Train
#Dev
#Test
Category
TP-US
age, gender entity
22,142
2,767
2,767
sentiment analysis
AG
11,657
1,457
1,457
topic classification
AG full
112k
1,457
1,457
topic classification
Blog
age, gender
7.098
887
887
topic classification
Yelp
520k
40,000
1,000
sentiment analysis
Table 1: Summary of NLP datasets.
1. Attacker crafts a set of inputs as queries (transfer set), then sends them to the victim model (BERT-based API) to obtain predictions;
2. Attacker reconstruets a copy of the victim model as "extracted model" by using the queried pnery-prediction pairs.
Since the attacker does not have training data for the target model, we apply a task-specific query generator to construct m queries {æ; T to the victim model. For each æ; target model returns K-dim posterior probability vector y, [0, 1] >. y = 1. The resulting dataset {æ;; 3: is used to train the extracted model. Once the extracted model is obtained, the attacker does not have to pay the provider of the original API anymore for the prediction of new data points.
3.3 ATTRIBUTE INFERENCE ATTACK (AIA)
Next, investigate how use the extracted model to aid the attribute inference of the private training data of the victim model, i.e., attribute inference attack (AIA) (Song & Raghunathan 2020). We remark that AIA is different from inferring attribute distribution in model inversion attack (Yeom et al. 2018). The intuition behind ATA is that the BERT representation generated by the extracted model can be used infer the sensitive attribute of the private training data of the victim model (Li et al. 2018b Coavoux et al. 2018 Lyu et al., 2020b). Note that in our work, the only explicit information that is accessible to the attacker is model prediction given by the victim model to the chosen inputs, rather than the original BERT representation. We specifically exploit BERT representation of the extracted model, as it encodes the most informative message for the follow-up classification. A more detailed description can be referred to Appendix B
3.4 ADVERSARIAL EXAMPLE TRANSFER (AET)
Due to the success of BERT-basel models, numerous works have been proposed to evaluate the vulnerability of BERT based models to adversarial attacks (Jin et al.) 2019 Sun et al. 2020). However, most recent works for adversarial example transfer focus the black-box setting (aao al. 2018 Ebrahimi et al. 20188). In such a setting, the adversary attacks the model via the query feedback only. To circumvent this issue, we leverage the transferability of adversarial examples: generate adversarial examples for our extracted model, then transfer them to the BERT-based APIs. The intuition lies two facts: 1) the rationale of a good model should rely on the salient words; 2) the functionally similarity between our extracted model and the victim model allows for the direct transfer of adversarial examples obtained via gradient-bart based attacks, which is able to locate the most informative words (Sun et al. 2020). Here our extracted model serves as surrogate to craft. adversarial examples in white-box manner.
first
EXPERIMENTS AND ANALYSIS
4.1 NLP TASKS AND DATASETS
We extract models on four diverse NLP datasets that focus two main tasks: sentiment analysis and topic classification. The four NLP datasets include TP-US from Trustpilot Sentiment dataset (Hovy et al. 2015), AG news corpus (Del Corso et al. 2005). Blog posts dataset from the blog authorship corpus (Schler et al. (2066), and YELP dataset (Zhang et al. 2013). Table [ summarises the statistics of the used datasets. more detailed description can be referred Appendix
4

---

Under review as a conference paper at ICLR 2021
Model
Query Size
AG news
AG news (full)
Blog
TP-US
Yelp
Victim model / API
79,99
94.47
97.07
85.53
95.57
All Same
80.83
94.54
96.77
86.48
95.72
1x
69.94
88.63
88.16
85.15
94.06
Data Different (review)
75.29
91.27
92.75
85.82
94.95
1x
71.95
90.47
83.13
84.15
91.06
Data Different (news)
75.82
92.26
87.64
85.46
93.13
Table 2:
Accuracy [%] of the victim models and the extracted models among different datasets in terms of domains and sizes.
4.2 MEA
To assess the functional similarity between the victim model and the extracted one, we compare the accuracy of two models, ie., the closer accuracy indicates a higher similarity. In line with prior work (Krishna et al. 2019), we first choose the size of the resulting transfer set (queries) to be comparable (e.g.. Ix) to the size of victim's training set, then scale up to 5x
Attack Strategies We first study model extraction through simulated experiments: we train victim models, query them if they are black-box APIs, and then train the extracted model to mimic the victim model. We assume that the attacker has access to the freely available pretrained BERT model used by the victim model.
Query Distribution To investigate how the data distribution of queries (PA) may impact the attack on the victim model trained on data from pv (c.f. Table D. we experiment with the following experiments.
1. We use the same architecture, hyperparameters, and the original data as the victim (All Same).
We use the same architecture and hyperparameters as the victim, but sample queries from different distribution (Data Different).
The second scenario makes fewer assumptions and is more realistic and challenging, as the attacker may not know the target data distribution as prior. Therefore, in addition to the same data distribution as the victim, we additionally investigate the query distribution PA sourced from the following corpora:
Reviews data: Yelp and Amazon reviews dataset (Zhang et al. 2015). It is worth noting that exclude Yelp reviews dataset from the Yelp task to guarantee fair evaluation.
News data: CNN/DailyMai dataset (Hermann et al. (2055)
Regarding the experiments of MEA, general findings from Table include: (1) using same data (All Same) as queries achieves the best extraction performance, validating that the closeness of the domain between the victim training data and queries is positively correlated the extraction; (2) using same data can achieve comparable accuracies, even outperform the victim models, we hypothesise this is due to the regularising effect of training soft-labels (Hinton al. 2015); (3) our MEA is effective despite the fact that queries may come from different distributions. Using samples from different corpora (review and news) as queries, our MEA can still achieve 0.85-0.99x
victim models' accuracies when the number of queries varies in { [1x,5x], and the extraction is more successful with queries as expected. This facilitates the follow-up AIA and AET. Even with small query budgets 10.1x and 0.5x), extraction is often successful. More results are available in Appendix C We also noticed that AG news prefers news data, while reviews data superior to news data TP-US, Blog and Yelp. Intuitively, one can attribute this preference the genre similarity, i.e., news data is close to AG news, while distant from TP-US, Blog and Yelp. To rigorously study this phenomenon, we calculate the uni uni-gram gram and 5-gram overlapping between test sets and different queries the Ix setting. Table corrobocoto that there a positive correlation between the accuracy and the lexicon similarity. From now, unless otherwise mentioned, because of their effectiveness (c.f..

---

Under review as a conference paper at ICLR 2021
Query
AG news (full)
Blog
TP-US
Yelp
uni-gram
5-gram
uni-gram
5-gram
uni-gram
5-gram
uni-gram
5-gram
review
68.22%
0.53%
47.21%
0.73%
60.86%
2.57%
52.68%
2.64%
news
72.13%
1.24%
44.76%
0.06%
51.28%
0.12%
38.69%
0.19%
Percentage of test sets uni-gram and 5-gram overlapping with different queries. Since AG Table 3:
news is derived from AG news (full) and they show a similar distribution, we omit it.
Table 2) we will use news data as queries for AG news, and reviews data as queries for TP-US, Blog and Yelp
4.3 AIA
For AIA, we conduct our studies on TP-US, AG news and Blog datasets, as there is no matching demographic information for Yelp. AIA is appraised via the following metrics:
For demographic variables (i.e., gender and age): 1 1-x, X, where X the average prediction accuracy of the attack models these two variables.
For named entities: 1 - F, where F is the F1 score between the ground truths and the prediction by the attackers on the presence of all named entities.
Following Coavoox et al. (2018); Lyu et al. (0200), we denote the value of X or 1 - F as empirical privacy, i.e., the inverse accuracy FI score of the attacker, higher means better empirical privacy, i.e., lower attack performance.
We first randomly split each dataset in Table I into two halves. The first half (denoted as Dy) is used to train a victim model, whereas the second half (denoted as A)) is specifically reserved as the public data for the training of AIA attack model. On the extracted model from MEA, attackers can determine how to infer the private attributes from the BERT representation h of the extracted model over DA Each attack model consists of a multi- feed forward network and a binary classifier, which takes the h as the inputs and emits the predicted private attribute. Once the attack models are obtained, we measure the empirical privacy by the ability the attack model to predict accurately the specific private attribute in Dy
Apart from the standard three corpora used for MEA (c.f., Section 4.2). D. in AIA, we also consider DA (2nd half) queries, which is derived from the same distribution as Dy. It is worth noting that for AG news, we use the filtered AG news (cff, Appendix A with sensitive entity information for AIA.
To gauge the private information leakage, we consider a majority class prediction of each attribute baseline. To evalu- ate whether our extracted model can help enhance AIA, also take the pretrained BERT without (w/o) fine-tuning as a baseline. Table 4 shows that compared to the pretrained only BERT, the attack model built on the BERT representation of the extracted model indeed largely en- hances the attribute inference of the train- ing data of the victim model more than 4x effective for AG news compared with the majority bascline, even when MEA is based on the queries from differ- data distribution. This implies that target model predictions inadvertently capture sensitive information about users, such as their gender, age, and other im- portant attributes, apart from the useful
AG news
Blog
TP-US
Majority class
49.94
49.57
38.15
BERT (w/o fine-tuning)
69.39
44.03
49.38
All Same
22.74
36.37
36.76
Data Different (2nd half)
1x
21.01
35.98
37.34
Data Different (review)
1x
17.93
34.34
35.97
5x
18.31
34.45
36.82
Data Different (news)
1x
15.76
33.88
36.92
5x
17.91
35.39
37.68
Table 4: AIA attack success over different datasets. Note higher value means better empirical privacy, i.e., lower attack success. (2nd half: DA reserved for the training of AIA attack model.)
Empirically, do not have the training data of the victim model.

---

Under review as a conference paper at ICLR 2021
information for the main task (c.f.. Table 2). that did not contain any information about the target model training data.
By contrast, BERT (w/o fine-tuning) is plain model
Interestingly, compared with queries from the same distribution, Table 4 shows that queries from different distributions make AIA easier (see the best results corresponding to the lower privacy protections in bold in Table 4. We believe this anti-intuitive phenomenon is caused by the posterior probability, as the posterior probability of the same distribution sharper than that of the different distribution P This argument can be also confirmed from Section 5 in which we use temperature coefficient at the softmax layer to control the sharpness of the posterior probability.
We speculate that the effectiveness of ATA is related the undesired deep model memorisation of the victim model, which can be spread to the extracted model through model prediction, incurring information leakage.
We further investigate which kind of attribute is more vulnerable, i.e., the relationship between attribute distribution (histogram variance) and privacy leakage. We empirically found that, compared with the attribute with higher variance, attribute with lower variance is harder to attack
4.4 AET
Since we have access to the parameters of the locally extracted model, we craft white-box adversarial examples it and test whether such examples are transferable the target model. We evaluate sample crafting using the metric of transferability, which refers to the percentage of adversarial examples transferring from the extracted model to the victim model. We Blog, TP-US, AG news (full) and Yelp for AET.
How We Generate Natural Adversarial Examples? Following Sun et al. (2020), we first leverage the gradients of the gold labels w.r.t the embeddings of the input tokens find the most informative kens. Then we corrupt the selected to- kens with the following six sources of ty- pos: 1) Insertion; 2) Deletion: 3) Swap; 4) Mistype: Mistyping a word though keyboard, such as "oh" - "Oh"; 5) Pro-
AG news (full)
Yelp
TP-US
Blog
transferability
deepwordbug
1x
25.6
13.5
18.4
52.9
Sx
35.3
20.7
18.2
67.8
black-box
textbugger
1x
16.1
11.3
21.3
41,2
5x
24.7
16.3
21.1
62.7
nounce: Wrongly typing due to the close pronounce of the word, such "egg" W: "agg"; 6) Replace- W: Replace the word by the frequent human behavioural key- board typo based on the statistics 5 Note that the above operations are constrained by the character distribution on the key- board. This approach is denoted as adv- bert.
textfooler Ix
18.5
12.3
27.5
34.7
5x
24.9
16.9
27.1
64.4
Rand
adv-bert 1x
32.5
13.4
33.3
56.2
Sx
45.5
21.3
34.0
66.3
w-hox (ours)
adv-bert 1x
47.5
17.8
48.6
64.9
5x
53.6
25.2
47.3
76.5
To evaluate whether our extracted model is needed to mount transferable attacks, we also attack by using black-box ad- versarial examples. Moreover, follow- ing (Sun et al. 2020), we also experi- ment with variant of adv-bert, where the target tokens are randomly selected instead of from the maximum gradients, namely random ady-bet. Compared with the adversarial examples crafted by black-box and random adv-bert approaches, Table 5 that the adversarial examples crafted on our extracted model in white-box manner make the target model
Table 5: Transferability is the percentage of adversarial examples transferring from the extracted model to the victim model. Higher is better. deepwordbug (Gao et al. 2018); textbugger (Li et al. 2018a textfooler (Jin et al. 2019: ; adv-bert (Sun et al. 2220). w-box: white-box
attack. Rand: randomly select word.
'llesse refer to the Appendix C for the detailed analysis.
'Please refer to the Appendix C for the detail.
https:: /n. wiw..wikk wik pedi www.///w/kkk hwt.kiiwwwwiiiiiiaaii.mm.
issee 1 1inss
7

---

Under review as a conference paper at ICLR 2021
Victim
Extracted
TP-US
AG news (full)
victim 1
MEA
AIA 4
victim f
MEA t
AET
BERT-large
BERT-hase
86.82
85.36
36.65
94.75
89.88
42.7
RoBERTa-large
BERT-base
87.20
85.72
37.33
95,40
89.74
27.7
RoBERTa-base
BERT-base
86.66
85,40
37.52
95.18
89,45
36.4
XaNeT-large
BERT-base 87.21
85.99
37.68
95.28
89.66
32.7
XLNET-base
BERT base 86.91
86.13
38.09
94.74
89.27
34.4
BERT-base
BERT-base
85.53
85.15
35.97
94.47
90.47
47.5
Table 6: Attack performance TP-US and AG news (full) with mismatched architectures between the victim and the extracted model.
more vulnerable to adversarial examples in terms of transferability more than twice effective in the best case. This validates that our extracted model, which is designed to be a high-fidelity imitation of the victim model, considerably enhances the adversarial example transferability, thus severely damaging the output integrity of the target model.
We examine potential factors that contribute to the successful transferability. We found that collecting a larger number of queries contributes to a better attack performance, iee, 5x queries generally results in much better transferability compared with 1x. This implies that the extracted model with higher fidelity (closer to the victim model, c.f., Table 2 can considerably enhance the adversarial example transferability.
4.5 ARCHITECTURE MISMATCH
In practice, it is more likely that the adversary does not know the victim's ' model architecture. A natural question is whether model extraction is still possible even when the extracted models and the victim models have different architectures. To study the influence of the architectural mismatch, fix the architecture the extracted model, while varying the victim model from BERT (Devlin et al.) 2018), RoBERTa (Liu et al. 2019b) to XLNET (Yang al. 2019). According Table 6 when there is an architecture mismatch between the victim model and the extracted model, the efficacy of AIA and AET is alleviated expected. However, the leakage of the private information is still severe (c.f., the majority class in Table[4). Surprisingly, we observe that for AG news (full), MEA cannot benefit from more accurate victim, which is different from the findings in Hinton et al. (2015). We conjecture such difference is ascribed to the distribution mismatch between the training data of the victim model and the queries. We will conduct an in-depth study on this in the future.
DEFENCE
Although we primarily focus the vulnerabilities of BERT-based APIs in this work, we briefly discuss several counter strategies the victim model may adopt to reduce the informativeness of prediction while minimising the overall drop in API performance (Shokri et al. 2017).
Hard label only. The posterior probability usually leaks more information from the victim model, thus victim model can choose only return the hard label.
Softening predictions. temperature coefficient 7 on softmax layer manipulates the distribution of the posterior probability. A higher leads to smoother probability, whereas lower produces sharper distribution. When 7 is approaching 0, the posterior probability becomes a hard label.
Table 7
indicates that although varying temperature softmax cannot defend the victim model against MEA, it is an effective defensive approach to ATA when - = 0.5, i.e., closer to the hard label. Similarly, compared with ND. hard label can help mitigate all attacks some extent
However, there is no defence that is effective against all our attacks (c.f.. Table Table 4 Table as all these defences preserve the rank of the most confident label. Models can still be effectively stolen
We observe the similar behaviours for Yelp and Blog.
8

---

Under review as a conference paper at ICLR 2021
TP-US
AG (full)
MEAT
AIA
AET 1
MEA 1
AIA J
MEA
AET 1
ND SOFTENING (7 0.5)
85.33
36.92
48.6
71.95
15.76
90.48
47.5
85.10
37.69
50.9
72.32
20.78
90.29
33.0
SOFTENING (T 5)
85,33
33.17
58.7
72.48
11.32
90.87
53.3
HARD LABEL
84.60
37.62
40.2
69.11
22.47
89.98
33.0
Table Attack performance under different defences and datasets. ND: no defence; T: temperature on softmax. For MEA and AET, lower scores indicate better defences, conversely for AIA. All experiments are conducted with 1x queries.
and exploited using just the hard label or the smoothed predictions returned by the black-box API. This further validates that the adversary only needs to have access to the victim model's hard label, and does not always need to have access to the confidence scores for our attacks to be successful.
DISCUSSION
Understanding how well our attacks work in various settings is important for defenders to know how vulnerable their systems are. Extensive experiments in this paper indicate that the privacy and robustness of an NLP system depend on the model complexity as well as the task. For example, privacy leakage of the victim model becomes more serious by inferring from the extracted model for AG news and Blog, while this phenomenon is less obvious for TP-US dataset (c.f., Table 4 In terms of robustness against adversarial example transferability, Blog is more vulnerable (c.f. Tāble
Adversarial attacks focus more on the study of the robustness of model. However, under the context of business, believe adversarial attacks can also be utilised for other purposes. For instance, if a business competitor manages to spot incorrect predictions, they can improve the robustness of their model while launching an advertising campaign against the victim model with these adversarial examples. If a rival company directly leverages black-box adversarial attacks on the victim model, its owner can detect the suspicious querying, which involves intensive similar queries (Jin al. 2019) Li et al., 2020: Garg & Ramakrishnan, 2020), thereby banning the abnormal usage. Since queries used for our model extraction are genuine instances generated on the Internet, is unlikely to be suspended by the cloud services. As evidenced in Section 4.4 the victim model is vulnerable to our proposed AET.
Defence against all our investigated attacks in this work is a hard and open problem. An ideal defence should resist against the possible attacks while striving have a minimal impact on legitimate users of the model (Orekondy et al. 2019). While current defences are marginally effective, they may fail when adversaries adapt to the defence - sophisticated adversaries might anticipate these defences and develop simple modifications to their attacks to circumvent these defences (Krishnal et al. 2019). We hope that this work highlights the need for more research in the development of effective countermeasures to defend against these attacks, at least increase the cost of adversaries.
7 CONCLUSIONS
This work goes far beyond only model extraction from BERT-based APIs, we also identified that the extracted model can largely enhance the privacy leakage and adversarial example transferability even in difficult scenarios (e.g., limited query budget, queries from different distributions). Extensive experiments based on representative NLP datasets and tasks under various settings demonstrate the effectiveness of our attacks against BERT-based APIs. We hope that our in-depth investigation can provide new insights, and arouse the awareness of the community for building more trustworthy BERT-based API. A number of avenues for further work attractive. More broadly, we expect to extend our work to more complex NLP tasks, and develop defences that can ensure privacy, robustness, and accuracy simultaneously.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. Privacy-preservee neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1-10, 2018.
Gianna M Del Corso, Antonio Gulli, and Francesco Romani. Ranking a stream of news. In Proceedings of the 14th international conference on World Wide Web, pp. 97-106, 2005.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.000 2018.
Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On adversarial examples for character-level neural machine translation. arXiv preprint IrXiv:I806 09030, 2018a.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. In ACL, 2018b.
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study personalized warfarin dosing. In 23rd (USENIX) Security Symposium ((USENIX) Security 14), pp. 17-32, 2014.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 JEEE Security and Privacy Workshops (SPW), pp. 50-56. IEEE, 2018
Siddhant Garg and Goutham Ramakrishnan. Bae: Bert-based adversarial examples for text classifica- tion, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253// 2015.
Dirk Hovy, Anders Johannsen, and Anders Søgaard. User review sites resource for large-scale sociolinguistic studies. Proceedings of WWW. pp. 452 461, 2015.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. arXiv, pp. arXiv-1907, 2019.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papemot, and Mohit Iyyer. Thieves on sesame street! model extraction of bert-based apis. arXiv preprint arXiv:1910. 12366, 2019.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-worke applications. arXiv preprint arXiv:1812.0522 2018a.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. arXiv preprint 77Xii:2004.09984 2020.
Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and prevacypreseeving. text rep- resentations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 25-30, 2018b
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-bor box attacks. arXiv preprint arXiv: 2016.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeuteemoer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019b.
Lingjuan Lyu, Xuanli He, and Yitong Li. Differentially private representation for nlp: Formal guarantee and empirical study on privacy and fairness. arXiv preprint arXiv:2010.01285 2020a.
Lingjuan Lyu, Yitong Li, Xuanli He, and Tong Xiao. Towards differentially private text repre- sentations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 1813-1816, 2020b.
Rodrigo Nogueira and Kyunghyun Cho.
Passage re-ranking with bert. arXiv preprint
arXiv: 04085, 2019.
Scong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learming, pp. 121- 144. Springer, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE Conference Computer Vision and Putlern Recognition, pp. 4954 4963, 2019.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, 2 Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the ACM Asia conference on computer and communications security, pp. 506-519, 2017.
Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-wware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 560-568, 2008
Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. Effectt of age and gender on blogging. AAAI spring symposium: Computational approaches to analyzing weblogs, volume 6, pp. 199-205, 2006.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Congzheng Song and Ananth Raghunathan. Information leakage embedding models. arXiv preprint S00i::20040000053 2020.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Adv-bert: Bert not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003. 04488, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th {USENIX] Security Symposium ( (USENII) Security 16), pp. 601-618, 2016.
Eric Wallace, Mitchell Stem, and Dawn Song. Imitation attacks and defenses for black-box machine translation systems. arXiv preprint arXiv:2004. 15015, 2020.
Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. 2018 IEEE Symposium Security and Privacy (SP), pp. 36-52. IEEE, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5753-5763, 2019.

---

Under review as a conference paper at ICLR 2021
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282 IEEE, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, In Advances neural information processing systems, pp. 649-657, 2015.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A DATASET DESCRIPTION
Trustpilot (TP) Trustpilot Sentiment dataset (Hovy et al. 2015) contains reviews associated with sentiment score five point scale, and each review is associated with 3 attributes: gender, age and location, which are self-reported by users. The original dataset is comprised of reviews from different locations, however in this paper, we only derive TP-US for study. Following Coavoux et al. (2018), we extract examples containing information of both gender and age, and treat them as the private information. We categorise "age" into two groups: "under 34" (U34) and "over 45" (o45).
AG news We use AG news corpus (Del Corso et al. 2005). This task is predict the topic label of the document, with four different topics in total. Following (Zhang et al. 2015 Jin et al. 2019). we use both "title" and "description" fields as the input document.
We use full AG news dataset for MEA and AET, which we call AG news (full). As AIA requires entity information, we use the corpus filtered by Coavoux al. 20018 which we call AG news. The resultant AG news merely includes sentences with the five most frequent person entities, and each sentence contains at least one of these named entities. Thus, the attacker aims to identify these five entities as 5 independent binary classification tasks.
Blog posts (Blog) We derive a blog posts dataset (Blog) from the blog authorship corpus pre- sented (Schler et al. 2006). We recycle the corpus preprocessed by Coavoux et al. (2018), which covers T0 different topics. Similar to TP-US, the private variables are comprised of the age and gender of the author. And the age attribute is binned into two categories, "under 20" (U20) and "over 30" (O30).
Yelp Polarity (Yelp) Yelp dataset is document- levell sentiment classification (Zhang et al. 2015). The original dataset is in a five point scale (1-5), while the polarised version assigns negative labels to the rating of 1 and 2 and positive ones to 4 and 5.
B AIA ALGORITHM
The main algorithm for Attribute Inference Attack (AIA) is shown Algorithm For each dataset, once the extracted model g'v is built, we query gy with the available public data DA to collect the BERT representation h (xx) for each D; E DA- For each sensitive attribute s, a specific inference model (c.f., Section 4.3 is trained { ((h(r;;, s;)], in order to infer the private attributes of the interest; in our case, they are gender, age and named entities (c.f. Table [].
In more detail, in Algorithm 1 given DA take all the non-sensitive attributes 2; as input, and the sensitive attribute s; as label to train an AIA attack model. During test time, attacker could feed the non-sensitive attributes of any input into the trained model to infer the sensitive attribute. In the case when the attacker gets the non-sensitive attributes of any training record of the victim model, the attacker can successfully infer its sensitive attributes, thus causing privacy leakage of the victim model training data (c.f.. Table 4 we use the non-sensineg non-sensitive attributes of Dy as test data, and demonstrate the sensitive attribute privacy leakage of Dy). Note that the non-sensitive attributes of the victim training data could be accessible to any attacker.
C ABLATION STUDY
Query Size Due to the budget limit, malicious users cannot issue massive requests. To investigate the attack performance of model extraction under the low-resource setting, we conduct two additional experiments, which only utilise 0x and 0.5x size the training data the vietim models respectively. According to Table 8 although some datasets such as Blog suffer from drastic drop, the overall performance of the extracted models is comparable to the victim models. In addition, distant domains
13

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Attribute inference attack
1: Input: extracted model Φv, labelled auxiliary data D (It, BERT representation layer h, non-sensitive attributes x²
2: Query gV with DA and collect {(h():)) 8: (3) 8t) DA)-
3: Train an inference model / on (((())) (.))).)))
4: Query gy with x" get the target BERT representation h(x*)
5: return (f)(2*))
AG news
AG news (full)
Blog
TP-US
Yelp
Victim model
79.99
94.47
97.07
85.53
95.57
Same
80.83
94.54
96.77
86.48
95.72
Data Different (review)
0.1x
50.90
86.57
36.83
79.95
92.39
0.5x
68.13 13
87.31
84.59
84.21
93.25
1x
69.94
88.63
88.16
85.15
94.06
5x
75.29
91.27
92.75
85.82
94.95
Data Different (news)
0.1x
61.70
89.13
18.04
79.20
88.24
0.5x
70.56
89.84
32.92
84.18
89.76
lx
71.95
90.47
83.13
84.15
91.06
5x
75.82
92.26
87.64
85.46
93.13
Table 8: The accuracy of victim models and extracted models among different datasets in terms of domains and sizes.
exhibit significant degradation, when compared to the close ones. For example, sampling 1x-5x queries from news data present more stable attack performance against the victim model trained on AG news than Blog.
Impact Factor AIA In Section 6, found that there is correlation between the success of AIA and temperature 7 on the softmax layer. We conjecture that the causal factor is the sharpness of the posterior probability, i.e., if the model is less confident its most likely prediction, then AIA is more likely to be successful. This speculation is confirmed by Figure where the higher posterior probability leads to higher empirical privacy.
Figure 3 and Table indicate that AIA is also affected by the distribution of attributes. Attributes with higher variances cause more information leakage or a lower empirical privacy. For example, for AG-news, entity 2-4 with higher variances result in lower empirical privacy, while entity 0-1 are more resistant AIA. For TP-US and Blog, as age and gender exhibit similar distribution, AIA performance gap across these two attributes is less obvious, as evidenced by the last two rows in Table 9
D ADVERSARIAL EXAMPLES
We provide several adversarial examples generated by adv-bert (Sun al. 2020) in Table Note that all these examples cause misclassification both extracted models and victim models.
14

---

Under review as a conference paper at ICLR 2021
Correlation between Maximum Posterior Probability and Privacy (AG news)
priacy
0.7
0.6
0.4
98
Dara Dit (2nd hal)
Draa Oit (ampm).5)
Dara Df (amped)
(a) AG news
Correlation between Maximum Posterior Probability and Privacy (TP-US)
pahaay
0.000
0.000
Bare
Data DE (2dd hall)
Dara OiT
Data D.I, Zompa4..)
Duta Drf (urpad)
(b) TP-US
Correlation between Maximum Posterior Probability and Privacy (Blog)
privacy
1.0000
0.0000
0.000
02000
AlSam
Data CR., (Dnd
Data DIr, (smppe0.d)
Data Dit (1smp55)
(c) Blog
Figure 2: The correlation between the empirical privacy of AIA and the maximum posterior probabil- ity. AG news uses news data as queries, while Blog and TP-US query the victim models with reviews data. mean and median denote the mean and median of the maximum posterior probability of the queries.
15

---

Under review as a conference paper at ICLR 2021
5000
6000
1500
5000
3000
4000
5000
4000
2500
3000
3Rocorass
2000
3000
3000
3000
1500
2000
2000
2000
2000
1000
1000
1000
1000
values
Attribute values
Attribute valuse
Attribute valuee
Attributa valces.
(a) AG news
Age
Gender
8000
8000
7000
7000
6000
5000
#Records
4000
4000
3000
2000
2000
1000
1000
045
U34
F
M
Attribute
Attribute
TP-US
Age
Gender
2000
1750
1750
1500
1500
#Records
1250
#Records
1250
1000
1000
750
750
500
500
250
0
0
U20
Attribute values
Attribute values
(c) Blog
Figure 3: The distribution of different datasets grouped by private attributes.
16

---

Under review as a conference paper at ICLR 2021
AG news
entity 0
entity
entity 7.71 3.84
entity 3 6.95
entity 5,49
All Same Data Diff. (news)
15.61
15.10
14.79
12.38
5.33
2.02
TP-US
Blog
gender 36.40
age
age
gender
All Same
37.12
All Same
32.18
39.02
Data Diff. (reviews)
36.44
37.40
Data Diff. (reviews)
31.20
38.01
Table 9: AIA performance on attributes of different datasets. All experiments are conducted with 1x queries.
17

---

Under review as a conference paper at ICLR 2021
AG news:
Original:
mps back putin plan for regions. the russian duma backs president putin plan replace elected regional bosses with his own appointees.
Adversarial:
mps back putin plan for regions. the russian duma backs president putin" plan to replace elected regional boss3s with his own appointees.
Original:
baseballer shot on bus cleveland indians righthander kyle denney was reported to be a stable condition after being shot in the leg on the team bus yesterday.
Adversarial:
basrballer shot on bus cleveland indians righthnader kyle denney was reported be in a stable condition after being shot in the leg on the teqm bus yesterda?
Blog:
Original:
stayed for dinner, and crawled all over him delightfully. s just so
Adversarial:
stayed for dinner, and crawl3d over him delightfully it s just so
Original:
flowed under the bridge during that time of sharing together, let me
Adversarial:
flowe under the bridge during that time of sharing together, let me
TP-US:
Original:
great company excellent service fast shipping and never problem with the order. their products are great and we love the free stuff
Adversarial:
great company excellent service ! fast shipping and nevr a problem with the order. their products are great and we love the free stuff
Original:
fraud made the purchase unacceptible took longer than anyother key site have bought from
horrible avoid if you can
Adversarial:
fisud made the purchase unacceptible took longer than anyother key site have bought from borrible avoid you can
Yelp:
Original:
sadly really went downhill they switched to new vendor, the prices went up and they are no longer having sales.
Adversarial:
safly really went dowhnill. they switched to a new vendor. , the prices went up and they are
longer having ssles..
Original:
carbs, carbs, and more carbs ..... this is horrendous place to eat steer as far away from this place as you can minus five if allowed. fries on a crummy beef sandwich.
Adversarial:
carbs carbs and more carbs from this place as you can, minus five allowed. fries on a crummy beef sandwich.
this is horrednous place to eat. steee as far away
Table 10: Adversarial examples generated by adv-bert on different datasets. All of them cause misclassification. The misspellings are highlighted in red.
18


Reviews:
[{"review_id": "u5Jacc5eZOq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is motivated by a challenging problem in deploying a neural network-based model for sensitive domain and research in this direction is essential for making such model usable for sensitive domains. The paper presents a model extraction attack, where the adversary can steal a BERT- based API (i.e. the victim model), without knowing the victim model’s architecture, parameters or the training data distribution. The model extraction attack, where the adversary queries the target model with the goal to steal it and turn it into a white-box model. They demonstrated using simulated experiments that how the extracted model can be exploited to develop effective attribute inference attack to expose sensitive information of the training data. They claimed that the extracted model can lead to highly transferable adversarial attacks against the original model (victim model). \n\nThe model extraction step of the proposed method is the main concern for me. Conclusions maid by simulated experiments on model extraction attack might not hold for a real experiment. The simulated experiments make both victim model and extracted model accessible and thus measuring functional similarity is fairly easy. However, without the knowledge of the victim model and with limited query budget, the simulated experiment might not resemble a real-scenario. Some explanations with real scenarios would make the claim more realistic. \n\n\nSome thoughts:\n\nRe: “Modern NLP systems are typically based on a pre-trained BERT. ”: provide references or evidence to support the statement.\n\nRe: “Model extraction attack aims to steal an intellectual model from cloud services.”: provide references or evidence to support the statement.\n\nRe: “Most existing adversarial attacks on BERT are white-box settings, requiring knowledge of either the model internals (e.g., model architecture, hyperparameters) or training data.”:  provide references or evidence to support the statement.\n\nRe: “The intuition lies in the fact that the similarity of our extracted model and the victim model allows for direct transfer of adversarial examples obtained via gradient-based attacks.”  — BERT part is same for both victim and extracted model but rest is still unknown and how the complexity of the similarity measurement increases for a real scenario? \n\nRe: “We measure the accuracy (on the same held-out test set for evaluation purposes) between the outputs of the victim model and the extracted model to assess their functional similarity.” — Can this be arbitrarily true by accident? Is there a robust way that we can use to measure the similarity?", "extra_scores": {}, "extra_text": {}}, {"review_id": "qyjUjPuDwaR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview\n\nThe authors propose a pipeline to attack and steal sensitive information from a BERT-based API service, and can subsequently perform adversarial attack to the victim model by creating white-box adversarial samples on the stolen model.\n\nThe pipeline can be summarized as the followings:\n1. Using distillation to train (steal) a model from the API.\n2. Conduct model inversion attack to the stolen model in step 1 to expose sensitive information of the training data.\n3. Create adversarial samples for the stolen model in step 1 and use them to attack the original API.\n\nSome of the assumptions of the experiment settings are too strong and far from the real situation, but the idea of using this pipeline to conduct model inversion and adversarial transfer attack is very interesting.\n\n### Pros\n\nThe pipeline proposed by the authors is very insightful. The experiment results also show the effectiveness of model inversion and adversarial attack. \n\n### Cons\n\nThe assumption of the Model Extraction Attack part is too strong. The authors use the same pre-trained BERT parameters for both victim and stealer model. However in real practice we are not able to know which pre-trained BERT parameter set is to be used for fine-tuning, nevertheless to get the pre-trained model. What if we use different pre-trained BERT parameters? What if we use a pre-trained BERT with different size (num of layers, hidden dim, etc.)?\n\nBesides, the stealing method is just a conventional distillation. Although the authors claims three differences between their method and distillation: (1) the goal (2) the accessibility of original data (3) the accessibility of hard labels, only the first one is appropriately claimed. For (2), distillation is also broadly used in transfer learning w/o the access of original data. For (3), distillation w/ only soft labels is also very popular and useful, from conventional distillation for compression, to self-distillation. \n\nI'd like to hear the reason why the authors make the assumption that the stealer would have the same pre-trained BERT when attacking, and also curious about the results of using different pre-trained BERT model. I might change the rating if the authors may address these questions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "O0P8nszU5nM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper is studying the vulnerabilities of modern BERT-based classifiers, which a service provider is hosting using a black-box inference API. Consistent with prior work [2], the authors succeed in extracting high performing copies of the APIs, by training models using the outputs of the API to queries (akin to distillation). The authors then study two attacks on the copy model --- private attribute identification of sentences in the API's training data & adversarial example transfer from the white-box copy model to the black-box API. The authors report high attack success rates, better than those from competitive baselines (which do not require constructing a copy model). A few defences are also explored but are ineffective to prevent these attacks.\n\n-------------------------------------------------\n\nStrengths of the Paper:\n\n1. While model extraction on BERT models has been studied previously [2], this paper goes beyond the setting of utility theft and explores information leakage and adversarial example transfer. These are extremely practical real-world settings. Moreover, the paper uses modern NLP techniques (finetuning BERT), which is ubiquitous in NLP systems these days.\n\n2. The reported attacks seem to significantly outperform some competitive baselines which didn't use an extracted model. While I have concerns about the experimental setup (below), these are very interesting results highlighting vulnerabilities of the models. This can encourage more research in defending against model extraction.\n\n-------------------------------------------------\n\nWeaknesses of the Paper:\n\n1. Query distribution: These distributions seem fairly similar to the downstream task for all datasets, for instance, \"reviews\" contains Yelp reviews, which is one of the datasets the victim model was trained on (I suspect some amount of overlap at the very least). The best MEA scores are observed when the domains are aligned, which might not be a practical setting for an attacker who has no knowledge of the victim model's training distribution. I suggest, at the very least, authors to provide n-gram overlap statistics between their preferred query distribution and downstream test set (the GPT2 paper [3] had similar statistics). The paper's story will be stronger if a corpus like Wikipedia is used for the query distribution, with the same set of downstream datasets.\n\n2. AIA Attacks: I have a few concerns here. First, isn't access to private attributes in half of the victim data (D_a) too strong an assumption? In a more practical setting, an attacker will have no access to D_a. It's even possible that the attacker doesn't know the output space of attributes. I think the more interesting setting is where the attacker is able to infer some information about the training data without supervising a classifier with gold data (D_a), perhaps using something like model inversion. This information need not be a private binary label, it could even be some canary string like a credit card number [4]. One more concern I had here was regarding the main baseline in this experiment, \"BERT (w/o fine-tuning)\". I find it quite strange that this is much worse than the majority class in two datasets. What happens when you fine-tune it on D_a? (using the standard practice of [CLS] vector for classification). This is a valid baseline if access to D_a is assumed, I think this will do quite well if it is possible to infer the private variable from the text.\n\n3. Adversarial example transfer: My main concern here is that \"transfer rate\" by itself is insufficient. You can make transfer rate 100% by retrieving examples from the target adversarial class. The more interesting evaluation is, what fraction of adversarial examples are both (1) transferred correctly; (2) not adversarial to a human (the changes are so minor that humans ignore them). Some kind of human evaluation for (2) will be helpful. Also, a good baseline here would be using adv-bert but with randomly chosen words (instead of white-box gradients), and an upper bound with adv-bert attacks on the victim model itself.\n\n-------------------------------------------------\n\nOverall Recommendation:\n\nWhile this is a very practically important setting, I'm not entirely convinced the proposed attacks work. My main concerns are regarding some of the experimental decisions and lack of baselines while comparing attacks. Overall I think the paper needs more work to be ready for publication.\n\n\n-------------------------------------------------\n\nOther Feedback:\n\nWhile these points are not a make or break for me, they will make the paper stronger. It will be nice to include some fine-grained qualitative analysis of the adversarial examples (along with samples), perhaps highlighting why generating that example would only be possible with access to an extracted model, and confirming the victim API model generates the same example. It will also be nice to see work beyond classification setting. Setups like question answering, machine translation, unconditional text generation are exciting testbeds which might be a lot more vulnerable to AIA style attacks than classifiers. With GPT3, black-box text generation APIs are probably going to get very common in the next 2-3 years!\n\n-------------------------------------------------\n\nErrors / Typos / Stylistic:\n\nI had some trouble understanding parts of the paper. I think with a bit more polishing and careful proof-reading, the paper will be easier to understand. There were also a few incorrect statements. I've pointed them below along with typos / stylistic suggestions,\n\n\"commercial NLP models such as Google’s BERT and OpenAI’s GPT-2 (Radford et al., 2019) are often made indirectly accessible through pay-per-query prediction APIs.\" --> This is not a correct statement, both pretrained models are freely available\n\n\"and NLP tasks (Chandrasekaran et al., 2020).\" is a mis-citation, you probably wanted to cite Pal et al. 2019 [1] or Krishna et al. 2020 [2] here?\n\nIn 3.2 and the Abstract / Intro I would remove the claim that \"architecture, hyperparameter is not known\", since both the victim / attacker are finetuning BERT.\n\nThere's some unnecessary mathiness in 3.2 (variables which are not referred to later on, like f_{bert}_theta*). I would suggest avoiding variables unless you plan to re-use them to reduce confusion.\n\nIn Table 3 I would suggest reporting attack success rather than privacy, to be consistent with other tables in paper (higher means more attack success)\n\nTable 4 caption, \"Transferability is the ratio\" --> \"Transferability is the percentage\"?\n\n-------------------------------------------------\n\n**After Author Response**: I really appreciate the author's efforts over the course of the rebuttal period for rigorously testing their method with several new baselines in such a short period of time. \n\nFor AIA attacks, the baseline numbers provided in the rebuttal are helpful but raise concerns about whether the proposed AIA attacks are working. I find it hard to believe that victim models have less private information than extracted models in 2 out of 3 datasets, and I suspect some other factors are contributing to this counterintuitive trend (like you said, maybe dark knowledge). I will stick with my stance that the AIA setting is broken since you are inferring private attributes using information from an identically distributed D_a (I think model inversion is a more valid setting to measure leakage).\n\nFor adversarial attack baselines, I agree with your argument that conducting black-box attacks directly on the victim models may need minimal difference queries which can be detected on the API side. However, you are going to need several orders of magnitude more queries to do extraction in the first place (which may or may not be easy to detect). I still encourage you to run this baseline in the next version of the paper, instead of only doing black-box attacks on extracted models. These minimal difference checks may not be in place, and directly doing black-box attacks on the victim model are much easier than extracting and then constructing adversarial examples. It is good to know what additional benefit you get by doing model extraction.\n\nOverall, I have decided to raise my score to 6 (more like ~5.5-6). This is conditional on the authors performing much more rigorous hypothesis-driven testing in the next version of the paper (just like they did in the rebuttal) to really validate the hypothesis \"extracting models make APIs more vulnerable to adversarial attacks\".\n\n-------------------------------------------------\n\nReferences\n\n[1] - https://arxiv.org/abs/1905.09165  \n[2] - https://arxiv.org/abs/1910.12366  \n[3] - https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf  \n[4] - https://arxiv.org/abs/1802.08232", "extra_scores": {}, "extra_text": {}}, {"review_id": "9JMsuQ5pNvG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################ \n\nSummary:\n\nThis paper presents a model extraction attack (MEA) for BERT-based models that are hosted behind an API. Using the model obtained in this step, the work aims to subsequently demonstrate attribute inference attacks (AIA) to expose sensitive information of the underlying data used during fine-tuning and adversarial example transfer (AET) that can be used to attack the hosted model.\n\n################################ \n\nReasons for score:\n\nOverall, I lean toward reject. The underlying idea is interesting and timely, and core to this interest is that \"the adversary can steal a BERT-based API (the victim model), without knowing the victim model's architecture, parameters or the training data distribution.\" As demonstrated, a substantial portion of the architecture (BERT) is known and the exploration of fine-tuning as the only mechanism for tailoring the model (rather than continual pretraining) limits potential impact.\n\n################################ \n\nStrengths:\n\n- Broad interest. The underlying ideas are of general interest, especially given recent examples of language models hosted behind APIs. The notion that they can be efficiently reproduced from that API and that they may in turn leak training information is an emerging concern.\n\n- Clear differentiation from prior work. In particular, the section on comparison to knowledge distillation is helpful in grounding the setting for experimentation.\n\n################################ \n\nWeaknesses:\n\n- Limitations. There is an implicit assumption that the models being hosted behind APIs are fine-tuned BERT models. Limitations of this should be more explicitly discussed. Many works in specific domains (e.g., legal, biomedical, etc.) appear to rely on continual pretraining to integrate sensitive data rather than fine-tuning toward a single task. Others even appear to train these models from scratch on data. It's unclear how common the case of fine-tuned BERT models behind APIs are from this paper.\n\n- Motivation of AET. The motivation of adversarial attacks against a pay-per-query API are unclear. Yes, it's possible to cause the API to create incorrect predictions, but why is that problematic for the owner of the model? It's clearly undesirable with respect to creating robust models, but as presented it's unclear why this is problematic.\n\n- Impact. Similar to the point above, the assertion that \"modern NLP systems typically leverage the fine-tuning methodology by adding a few task-specific layers on top of the publicly available BERT base\" is not substantiated by this work or by citation. While BERT has certainly become abundant, many recent advances are either not BERT-based (though perhaps the underlying transformer architecture) or do more than fine-tuning.\n\n- Knowledge of black-box model. While a stated goal is that a knowledge of the architecture and training data is not required, the experiments leverage a knowledge of the architecture (BERT) and appear to share an architecture for layers used during fine-tuning.\n\n################################ \n\nQuestions:\n\n- Given the positioning of \"stealing\" a model, how many queries are required to obtain an approximate model? How many are required if knowledge of the previously issued queries is known?\n\n- Can you provide pointers to models that are BERT-based and fine-tuned for a specific task only?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents novel model stealing attacks against BERT API. The attacks are split in two phases. In the first phase, the black-box BERT model is recovered by submission of specially crafted data. In the second phase, the inferred model can be used for identifying sensitive attributes or to generate adversarial examples against the basic BERT model.

Despite the novelty of presented attacks against BERT models, the current version of the paper has some problems with clarity and motivation. The presentation of attacks is very short, and some technical details are not adequately covered. The practical motivation of adversarial example transfer attacks is not very clear, and the authors' response on this issue did not provide a convincing clarifications. Furthermore, creation of surrogate models for generation of adversarial examples is a well-known technique and the difference of the proposed AET attack from this conceptual approach is not clear.

Overall, the paper reveals a solid and interesting work but a substantial revision would be necessary to make it suitable for the ACLR audience.  
Decision: Reject