Grey-box Extraction of Natural Language Models
Authors: Santiago Zanella-Beguelin
Shruti Tople
Andrew Paverd
Boris Köpf
Venue:  (2021)
Keywords: language models
transformer
model extraction
security

Abstract: Model extraction attacks attempt to replicate a target machine learning model from predictions obtained by querying its inference API. Most existing attacks on Deep Neural Networks achieve this by supervised training of the copy using the victim's predictions. An emerging class of attacks exploit algebraic properties of DNNs to obtain high-fidelity copies using orders of magnitude fewer queries than the prior state-of-the-art. So far, such powerful attacks have been limited to networks with few hidden layers and ReLU activations.

In this paper we present algebraic attacks on large-scale natural language models in a grey-box setting, targeting models with a pre-trained (public) encoder followed by a single (private) classification layer. Our key observation is that a small set of arbitrary embedding vectors is likely to form a basis of the classification layer's input space, which a grey-box adversary can compute. We show how to use this information to solve an equation system that determines the classification layer from the corresponding probability outputs.

We evaluate the effectiveness of our attacks on different sizes of transformer models and downstream tasks. Our key findings are that (i) with frozen base layers, high-fidelity extraction is possible with a number of queries that is as small as twice the input dimension of the last layer. This is true even for queries that are entirely in-distribution, making extraction attacks indistinguishable from legitimate use; (ii) with fine-tuned base layers, the effectiveness of algebraic attacks decreases with the learning rate, showing that fine-tuning is not only beneficial for accuracy but also indispensable for model confidentiality.

Under review as a conference paper at ICLR 2021
GREY-BOX EXTRACTION OF NATURAL LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Model extraction attacks attempt to replicate a target machine learning model from predictions obtained by querying its inference API. An emerging class of attacks exploit algebraic properties of DNNs (Carlini et al., 2020; Rolnick & Körding 2020 Jagielski et al. (020)) to obtain high- fighlfidelity copies using orders of magni- tude fewer queries than the prior
So far, such powerful attacks have been limited to networks with few hidden layers and ReLU activations.
In this paper we present algebraic attacks on large-scale natural language models in grey-box setting, targeting models with a pre-trained (public) encoder fol- lowed by single (private) classification layer. Our key observation is that small set of arbitrary embedding vectors is likely to form a basis of the classification layer's input space, which a grey-box adversary can compute. We show how to use this information to solve an equation system that determines the classification layer from the corresponding probability outputs.
We evaluate the effectiveness our attacks on different sizes of transformer mod- els and downstream tasks. Our key findings are that (i) with frozen base layers, high-fidelity extraction is possible with a number of queries that is as small as twice the input dimension of the last layer. This is true even for queries that are entirely in-distribution, making extraction attacks indistinguishable from legiti- mate use: (ii) with fine-tuned base layers, the effectiveness of algebraic attacks decreases with the learning rate, showing that fine-tuming is not only beneficial for accuracy but also indispensable for model confidentiality.
INTRODUCTION
Machine learning models are often deployed behind APIs that enable querying the model but that prevent direct access to the model parameters. This restriction aims to protect intellectual property, as models are expensive to train and hence valuable (Strubell et al. 2019); security, as access to model parameters facilitates the creation of adversarial examples (Laskov al.] 2014 Ebrahimi jet al., 2018); and privacy, as model parameters carry potentially sensitive information about the training data (Leino & Fredrikson] 2020). Model extraction attacks (Tramèr et al., (206)) attempt to replicate machine learning models from sets of query-response pairs obtained via the model's inference API, thus effectively circumventing the protection offered by the API.
Several extraction attacks on deep neural networks (see (see Jagielski et al. (2020) for recent overview) follow learning-based approach (Tramer et al.) 2016] Orekondy et al. 2018 Pal al. 2020)]
Krishna et al., 2020), where the target model is queried to label data used for training the replica. The replicas obtained in this way aim achieve accuracy on the desired task, or agreement with the
target model on predictions, but recovery of the model weights is out of scope of this approach.
More recently, novel class of attacks has emerged that uses algebraic techniques recover the weights of deep neural networks up to model-specific invariances. Examples are the attacks of Milli ct al. (018)), which leverage observations of gradients to recover model parameters, and Rolnick & Körding (2020; Jagielski et al.] (2020); Carlini et al. (2020), which estimate gradients from finite differences of logits, and then use this information to recover model parameters. Algebraic attacks improve learning-basse based attacks in that they (i) achieve
replicas and (ii) orders of magnitude more query-efficcient So far, however, algebraic attacks have only been applied to small, fully connected neural networks with ReLU activations. In particular, for modern large-scale

---

Under review as a conference paper at ICLR 2021
natural language models (LLMs) such as BERT GPT-2, the state-of-the-an model extraction attack is still learning- based (Krishna et al. 2020).
In this paper, we propose the first algebraic attacks LLMs. We focus models consisting of a pre-trained encoder and a single task-specific classification layer. We assume a grey-box setting where the encoder is public (and hence known to the adversary). and the classification layer is private (and hence the main target of the attack).
There are two key observations that enable to extract LLMs via algebraic attacks. The first is that it is sufficient for an adversary to know rather than to choose the embeddings that are fed into the last layer. Existing algebraic attacks can infer the inputs hidden layers, but can only do so on piecewise linear networks and would not work on LLMs, which use non-linear activations. In the grey-box setting, an adversary can compute hidden embeddings of any input by querying the public encoder model, and can query the target LLM on the same input through the model's API. We show in theory, and confirm by experiments, that a random set of embeddings is likely to form a basis of the last layer's 's input space. The raw outputs (i.e., the logits) this basis uniquely determine the parameters of the last linear layer, which can be recovered by a transformation to the standard basis.
Our second observation is that this approach extends to the case where the API returns probabil- ities rather than raw logits, after normalization by the softmax function. For this, we leverage the invariance under translation of softmax to establish an invariance result for linear functions. Using this result, we show that the parameters of the last layer can be recovered (up to invariance) from embedding vectors spanning its input space and their corresponding probability outputs.
We evaluate our attacks on LLMs of different sizes and tne-tuned to different downstream tasks. We study the effects of using different types and numbers of extraction queries and different learning rates for fine-tuning the encoder model. Our key findings are:
When the target model's base layers are frozen during fine-tuning (i.e., the attacker get the exact embedding of any input), the attack is extremely effective. With only twice as many queries as the dimension of the embedding space (e.g., 1536 for BERT-base), we extract models that achieve 100% fidelity with the target, for all model sizes and tasks.
When the model's 'ss base layers are fine-tuned together with the task-specific layer, the em- beddings of the base model only approximate those of the target model and, as expected, the fidelity of the extracted models decreases the learning rate grows. Maybe surpris- ingly, for some models and downstream tasks, we are still able to extract replicas with up to 82% fidelity and up to 79% task accuracy, for orders of magnitude fewer queries than required by state-of-the-a learning-basel attacks (Krishna et al., 2020).
Extraction is possible using either random or in-distribution queries. Replicas extracted using in-distribution queries perform well on both in-distribution and random challenge in- puts. This shows that replicas can be created from small numbers of in-distribution queries, making attempts extract the model indistinguishable from legitimate use.
In summary, we propose a novel grey-box extraction attack on natural language models that is indistinguishable from legitimate use in terms of the content and number of queries required.
ATTACK
We consider classification models h: X + R", mapping elements from X to label probabilities in R". We assume that h log softmax g consists of three components
h: X 2 R 4 Rm log o softmax
R.
where
X + R" is contextualized embedding model, such as BERT or GPT-2;
f: R + Rmm is an affine function computing logits from embeddings, i.e., f(x)= = Ax b with
A E m and h E Rm:;
softmax: 1T + Rd normalizes logits to probability vectors:
(xp(x;))
softmax(x) = Em exp(x:)
(1)
2

---

Under review as a conference paper at ICLR 2021
We assume the adversary knows the embedding model g and tries infer A and (resp. f). We call this adversary grey-box because it can compute the embeddings from the inputs to h.
2.1
BASIC IDEA
Milli al. (018)) show how reconstruct models from oracle access to gradients. Carlini et al. (2020) show how to replace gradients by finite differences of logits, enabling reconstructing models without such an oracle. To explain the basic idea, let e; (O.... .......... -0" be the vector of the standard basis in IR". and let E IR be arbitrary. The ith column of A can be obtained as the difference between f(x + e4) and [xx) There are two obstacles that prevent us from directly applying this idea to classification layers from LLMs, namely (1) the attack requires the embeddings to be chosen, which would effectively amount to reversing the embedding model; and (2) the attack
0,
uses raw logits, while APIs provide only log probabilities normalized with softmax. We show next how to overcome these obstacles.
2.2 EXTRACTION FROM LOGITS
As first step, we overcome the requirement that the adversary be able to choose the inputs to f- Specifically, we show that it is sufficient for the adversary to know the inputs, given that they are sufficiently random. rely on the following standard result:
Proposition Let ())),... x()) E R be uniformly distributed in an n-cube. Then x!
x())
form busis IR" with probability 1.
]== Proof. Let {z(0) be a basis of RR" and V the subspace generated by {2000mm Ri for m < n. For a chosen uniformly an n-cube, the space V { has dimension m++1 1 with probability This because for æ to fall into V, it would need to have zero as coordinates wrt (m++1) (x),, which
happens with probability zero. Applying this argument inductively proves the proposition.
Based on this result, we mount the following grey-box attack on ʃ oog:
1. Choose distinct inputs {-6)) LN Jj=1 in with > n;
2. Compute their embeddings (1)(5 = )(x60)) and logits (=0) (1) 5=
Construct a matrix Y (==1))x where the first component of each column and the rest are from the embeddings, i.c., Yn, = (1)) and a matrix Z Rm where the columns are the logit vectors, i.e., Zij 2,
(i)
4. Solve for À mRxx and b E Rm in (, Y Z, i.e.
b
T1,1
ā
(1)
(N)
21
M1
D1
(2)
8m
1m,,1
am.m
(N)
2m
(N)
yn
Proposition 2. Assuming 9 maps inputs uniformly distributed embeddings in an n-cubee the attack above uniquely determines the parameters of f. Le.,, we have À A and b b.
N. The unique solution can be obtained multiplying Z by the right-inverse of Y. For uniformly random embeddings, this right-inverse exists because Y has full rank with probability by Proposition
Proof. By construction of (2) we have b =b=((y(0 for jorj= I,..,,N.
While in theory N n++ distinct queries sufficient mount the attack, computing the inverse of based with finite-precision arithmetic can be numerically unstable. practice, we gather a larger set of inputs and construct an over-determined system of equations. We then numerically compute a least squares solution to Equation
A language model with vocabulary V and maximum sequence length I. can only produce [ different emheddings. This is many more points than representable in the precision used, issue in practice.
3

---

Under review as a conference paper at ICLR 2021
2.3 EXTRACTION FROM LOG PROBABILITIES
We have so far assumed that the adversary has access to the raw unnormalized values, or logits. We extend the attack the case where the target model's inference API exposes only the log probability of each class, obtained by normalizing logits using softmax and returning the component- wise log of the result.
Softmax is invariant under translation, i.c., softmax(z) = softmax( + (c...., c)). We lift this property linear functions:
Proposition 3. Let C' 1mxxn be matrix with identical rows, i.e., Cig c for all i, i, within range. Then for all A E IRmx and E R
softmax((A + C)y) softmax(Ay)
Proof. Observe that softmax + C)y) = softmax Ay (k.....k)) softmax (Ay). where Ej= Ci,jyj since all rows of are identical. The last equality follows from the translation invariance of softmax.
Due to Proposition 3 we cannot hope to recover A and (as in Proposition 2). However, still obtain weights that yield functionally equivalent results when post-processed with softmax. Based on the construction of C' above, we propose the following grey-box attack on h:
1. Gather inputs and construct matrix Y E R(++))xN as described in Steps 2 and Section 2.2
Define D (m=N1)xN such that D.j Pi (.) ( where p log (softmfaa (y)())
are the log probabilities of the inputs. That is, we collect differences of log probability vectors as columns D, and then subtract the first row from all rows.
3. Define b and ē1,j O for 1,..., Solve for the remaining components of and À from
1
02
@2,1
u2,n
(N)
P2
(
Pi
(1)
P2
(N)
Pi
(N)
V1
Vi
=
(3)
bm
1mm.
ām,m
(N)
pm
(1)
Pi
(1)
pin
(N)
Pi
(N)
Un
yn
Proposition 4. The attack determines the parameters of f(x) = Ar + b up to translation, that is:
softmax (Ay b) softmax (Ay Ay
For a proof, observe that Σラ Uniiii (k) +b;-log -log -im exp [z: where 2(6) = Ag(()) +
Hence we have, for i 2 m.
Yi,k = Pi (k)
(ai.) - a1.)0; (1) + (b; by) - (k) Σ
(4)
j=1
By construction we have ā, = a.j - al and by = b, - bb. which implies that the columns of
A - and b are constant vectors (resp. the rows of A - À are all identical). From Proposition 3 we finally obtain
softmax Ay
softmax (A- A)) (b b))) softmax(Ay) + +b)
As before we can gather more than + inputs and build an solve the over-determined system to leverage more queries and improve numerical robustness.
2.4 EXTRACTION USING APPROXIMATIONS OF EMBEDDINGS
The attacks presented in Sections 2.2 and 2.3 rely the assumption that the adversary has full
access to the public embedding model g. In many settings, the embedding model is fine-tuned
together with the task-specific layer and the target model of the attack is h = log softmax

---

Under review as a conference paper at ICLR 2021
o g'. rather than h log softmax f 9. We extend our attack to such settings by using the public embedding model g as an approximation of the embedding model g'. That is, we do not assume that the adversary knows g': instead we attack h' based on embeddings from g. which is why the function f* we extract is only an approximation of f. We then construct the replica h" = log softmax f og by wiring the extracted classification layer top of the public encoder.
Clearly, the performance of this approach depends on the quality of the approximation g = g', which likely to decrease with the learning rate 7y with which g is transformed into g'. In the next section we evaluate the feasibility of this approach for different learning rates v).
Even if the attacker does not know which embedding model was used (i.e., a fully black-box attack), it is still possible to perform a variant of our attack. The attacker can collect query-respona pairs from the target model (since these do not depend on the choice of embedding model) and run the attack (offine) using the same pairs but different embedding models. In practice, there are relatively few widely-used used embedding models, so the attacker can perform a frute-force exploration.
3 EXPERIMENTS
We evaluate our attack on models for text classification from the GLUE benchmark: SST-2 (Socher) iet al. 2013) and MNLI (Williams et al. 2017). SST-2 is a binary sentiment analysis task where the goal is to predict positive or negative sentiment of an input sentence. MNLI is task with output classes to predict a relation between premise and a hypothesis.
We train different target models using two base models: 1) BERT-Base with 12 layers and 768- dimensional embeddings and 2) BERT-Small with 4 layers and 512 112-dimensional embeddings. We vary the learning rate the base model (1) ranges from 0 to 2 x 10-5) while the classifier layer is always trained with fixed learning rate 1j 2 x 10-5, In this section, all references to learning rate refer to the learning rate of the base layers. All our models are trained for 3 epochs using Hugging Face Transformers v.3.2.0 Our core attack logic is simple and is implemented in only 15 lines of Python code with around 500 lines of boilerplate.
To perform our attack, we vary the type findiisribbttinn vs. random) and number of queries made to the target model for extraction of the classification layer. For the real queries, we use the respective SST-2 and MNLI test sets, since these random queries we generate random strings (or pairs of strings for MNLI) of varying length up to the maximum length of the model (128). For attack evaluation, base depending on the target model) and combine it with the extracted classification layer to form a complete extracted model (as mentioned in Section 2.4). For each extracted model, we measure the accuracy and agreement with the target model on the validation set of the task or different random challenge inputs, respectively.
in in-distribution but unseen during training, and for the
the public BERT model (small or
Research Questions. We experimentally evaluate our attack to answer the following questions:
Type of extraction queries: How does the type of query submitted by the attacker (c.g.. in-distribution vs. random) affect the utility of the extracted model.
Number of extraction queries: How does the number of queries used by the attacker
affect the accuracy and agreement of the extracted model as compared to the target model? Effect of base learning rate: How does fine-tuning of the base model with different learn- ing rates impact the success of our attack?
Main Results. Table T summarizes the key results across our evaluation for two extreme cases where (i) the base layers of the target model are frozen (1) 0) or (ii) fine-tuned with the same learning rate as the classifier layer (1) = 2 x 10-5). Our findings are:
For frozen base models, the attack produces models with 100% agreement, with numbers of queries equal (for SST-2) or twice as large (for MNLI) as the dimension of the embedding vector.
For the fine-tuned models, agreement drops to 10% below the learning-based state-of art (Krishna al. 2020) for SST-2, but is achieved with order of magnitude less queries: 1821 versus 67 349.
5

---

Under review as a conference paper at ICLR 2021
Key results for different model sizes and downstream tasks. Agreement shows on how many inputs the extracted model exactly matches with the target model. #queries denote the number of queries required extract the model. For SST-2, we are limited by the test inputs available the dataset i.e., 1821. H is the output dimension of the base model.
Table I:
Dataset
Base Model
Frozen (1) = 0)
Fine-tuned (1) = 2e 5)
Target Acc.
Agree- ment
Target Acc.
Agree-
#queries
#queries
ment
BERT-base
.75
1.0
769 (H + 1)
.91
.83
1821 (2.3 H)
SST-2
BERT-small
.70
1.0
513 (H + 1)
.87
.79
1821 (3.5 H)
BERT-base
.43
1.0
1024 (2 H)
.83
.44
3456 (1.5 * IT)
MNLI-3
BERT-small
.45
1.0
1536 (2 H)
.77
.4
3584 (7 + IT)
Table 2: Effecc of using real vs. random queries for extraction (1) = 10-5, #quuries= 2/1). - and L denote the difference between the target and extracted matrices. Agreement is computed for both in-distribution real inputs and randomly generated inputs.
Dataset
Base Model
Extraction with Real queries
Extraction with Random queries
L
b
Agreement Real
L
Agreement
Rand.
Real
Rand.
SST-2
BERT-base
85
155
0.81
0.89
21
3
0.63
0.90
BERT-small
881
601
0.74
0.76
12
40
0.67
0.86
MNL1-3
BERT-base BERT-small
494.05
56.11
.41
.21
30.37
14.01
0.36
0.74
1704.2
1032.98
0.48
0.39
80.84
99.97
0.33
0.88
The attack works irrespective of the size of the base model, and performs better on down- stream tasks with fewer output classes.
Effect of type of queries: Real vs. Random. The type of queries used to extract the model could impact both the agreement of the extracted model, as well the defender's ability to detect the attack. To explore this effect, we perform extraction attacks using both real (i.e., in-distribution) queries.
cases, we use only 2H extraction queries (i.e., 1536 for BERT- Base and 1024 for BERT-Small models). We evaluate the worst-case scenario (from an attacker's 's perspective) where the base layers have been fine-tuned with the same learning rate as the classification layers (17
queries, or randomly-generee
10 10- The results shown in Table 2 and the key observations are:
A model extracted using real queries provides a similar level of agreement with the target model both real and random inputs. Thus, extraction using real queries works better in general and is harder to distinguish from genuine benign queries.
A model extracted using random queries provides better agreement with the target model on random inputs than on real inputs. The gap between agreement on real and random inputs is more pronounced for this type of model.
- distances between the target and extracted weight matrices and bias vectors are large, but do not necessarily affect agreement due to the invariance of the softmax operation.
Effect of number of queries. Using real queries. we vary the number of queries used for the extraction and report both the task accuracy of the extracted model and the agreement between the target and extracted models in Figure [ The size of the respective test sets limits the number of queries we could use for this experiment. Again we use learning rate of 1) = x 10-5 to evaluate the worst-case scenario from the attacker's 's perspective. Note that the bascline task accuracy for model performing random guessing on a balanced dataset for SST-2 and MNLI 50% and 33% respectively. The key observations are:
For both tasks and models, our extracted model performs better than a random guess and we observe clear increase in extracted model task accuracy and agreement queries increase. After a sharp initial increase, there appear to be diminishing returns beyond 2H queries (i.e., 1536 for BERT-Base and 1024 for BERT-Small models).

---

Under review as a conference paper at ICLR 2021
/ 2- 10-)), Real queries
MNU (BERT-Base. 10"*), Real queries
0.9
0.8
0.8
0.7
0.7
0.6
Score
0.5
0.4
Random baseline
Extracted accuracy
Target model accuracy
0.3
Target model sccuracy
Extracted model accuraty
Velidat agreement
Volidation agreement
Random agreement
0.2
agreement
768
1280
1792
2304 Number of queries
2816
3328
3840
1/52
2304 Number of queries
Figure I: Effect of number of in-distribution queries extracted model accuracy and agreement with the target model. Full results (including BERT-Small and random queries) in Appendix C
SST-2 (BERT-Base, #queries 1536)
(BERT Basee #queries 1536)
1.0
Target mode acoumcy
Extrected model menellaccurvey
Vallttton agreemomt
0.9
Random agremment
0.8
0.8
0.7
0.5
Target modal accurscy Extracted model socuracy
Validation agreement
agreement
10
10
10"
10"
Learning
Leamiin rate (r)
Figure 2: Effect of learning rate on target and extracted model accuracy, and agreement with the target model. Full results (including BERT-Sman and random queries) in Appendix B
As above, we achieve lower absolute task accuracy and agreement for MNLI than for SST- 2, since the number output classes increases.
Overall, the number of queries we require to achieve reasonable agreement is still orders of magni- tude lower than prior work.
Effect of learning rate. Finally, we quantify the effect of the learning rate used to tne-tune the base layers. We attack target models trained with learning rates ranging from 0 (frozen) to 10 (note that the classification layers all use 2 x 10-5) using real queries. The accuracies and agreement of target and extracted models are depicted in Figure 2
We highlight the following:
Agreement between the target and extracted models always starts at 100% for frozen base layers but decreases with increase learning rate.
As expected, original accuracy increases with learning rate and, interestingly, extracted
model accuracy also increases slightly initially before decreasing for higher learning rates.
4 DISCUSSION
Defenses against model extraction attacks. Several defences against model stealing attacks focus on identifying malicious query patterns, Juuti et al. (2019) Atli et al. (2020) collect stateful informa- tion about queries at the API-level and flag potential attacks deviations from a benign distribution. Kariyappa & Qureshi (2019) propose a defense that selectively returns incorrect predictions for out- of-distribution queries. While such approaches
shown to be effective against learning-bai based attacks, our attack can leverage random and in-distribution queries alike, and will hence evade such defenses. Other defenses rely limiting the information available the adversary, for example by
7

---

Under review as a conference paper at ICLR 2021
quantizing prediction probabilities (Tramer al. 2016), or adding perturbations (Lee et al. 2018
to poison the attacker's 's training objective (Orekondy et al. 2020), or watermarking the model so that extraction becomes detectable (Uchida et al.]2017) We expect these kinds of defenses may be effective against algebraic attacks, but leave an in-depth investigation as future work.
Further improving the extracted model. An attacker could combine our attack with techniques from existing learning- based extraction attacks. For example, after extracting the classification layer and adding this a public embedding embedding model (as described in Section (.4]. the attacker could fine-tune this new model using the set of query-response pairs originally collected for our extraction attack, as well as any further query-responss pairs from the target model. We investigate this hybrid attack strategy and its converse in Appendix A but leave a more in depth evaluation to later revision.
RELATED WORK
There is growing body of work studying the extraction of machine learning models, see e.g. Lowd & Meek (2005); Tramer et al. (2016); Orekondy et al. (2188; Rolnick & Körding (2020); Pal et al. (2020); Krishna et al. 22200;; Carlini et al. (2020). These approaches differ in terms of the adver- sary's s objectives, the model architecture, and the techniques they employ for extraction, see Jagielski eet al. (2020) for recent taxonomy and survey. For conciseness we focus this discussion on work that targets natural language model or uses techniques related to ours, as well as on defenses.
Extraction of Natural Language Models. Krishha al. (020) the first to report model extraction of large natural language models. They rely on a learning-basee approach where they the target model to label task-specific queries, which they craft based random words. They also observe that transfer learning facilitates model extraction in that the adversary can start with known base model for training the replica. Our attack goes one step further in that we leverage public knowledge about the embeddings for mounting an algebraic attack on the last layer. Krishha al. (2020) report on agreement (accuracy) of 0.92 (0.90) for SST for around 60K random queries on SST2, and of 0.80 (0.76) for 392 702 random queries for MNLI. In contrast, our attack requires only 2 (1024 and 1536 for BERT-Small and BERT-Base) number of queries with H being the dimension of the base-embedding model.
Algebraic model extraction attacks. Most model extraction attacks rely on training the replica on labels generated by the target model. Recently, class of attacks has emerged that uses algebraic techniques to recover deep neural networks, achieving copies with higher fidelity using smaller numbers of queries compared to learning- baaed approaches. The core idea goes back to Milli al. (2018), which leverage observations of gradients to recover model parameters. This is leveraged by Rolnick & Körding (2020); Jagielski et al. (2020); Carlini et al. (2020) which estimate gradients from finite differences of logits, and then use this information to recover model parameters.
Our attack differs from these approaches in different aspects. First, we only extract a single layer, whereas the other attacks have been demonstrated for up to 2 hidden layers. Second, as our attack is grey-box, we only assume that the attacker knows the inputs, whereas the other approaches require that the adversary be able to choose. Third, we show how to extract the model despite softmax activation layer, which out of scope of the other approaches.
Relationship to softmax regression. The problem of extracting models where only task-specific layers are fine-tuned is closely related, but not equivalent, to parameter estimation for softmax re- gression, see, e.g., Van der Vaart (2000); Yao & Wang (219)). The key difference is that for extrac- tion the goal is to recover fixed but unknown set of parameters (i.e. a ground truth) with a minimal amount of data, whereas for regression the goal is to find the best parameters to fit the data.
6 CONCLUSION
In conclusion, we propose a novel grey-box extraction attack on natural language models that is in- distinguishable from legitimate use in terms of the content and number of queries required. Existing detections and defenses based on the number or type of queries are unlikely to be effective, and thus other approaches are needed to detect or mitigate grey-box extraction attacks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, and N. Asokan. Extraction of Com- plex DNN Models: Real Threat Boogeyman?, 2020.
Nicholas Carlini, Matthew Jagielski, and Ilya Mironov. Cryptanalytic extraction of neural network models, 2020.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification, 2018.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Paperno. High accuracy and high fidelity extraction of neural networks. In 29th USENIX Security Symposium, 2020.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N. Asokan. Prada: Protecting against dnn model stealing attacks, 2019.
Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adap- tive misinformatio,, 2019.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on Sesame Street! Model Extraction of BERT-based APIs. In ICLR. OpenReview.net, 2020.
Pavel Laskov et al. Practical evasion of a learning-based classifier: A case study. In 2014 IEEE symposium on security and privacy, pp. 197-211. IEEE, 2014.
Taesung Lec, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against model stealing attacks using deceptive perturbations. arXiv preprint arXiv: 00054, 2018.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference, 2020.
Daniel Lowd and Christopher Meek. Adversarial learning. KDD '05, ACM, 2005.
Smitha Milli, Ludwig Schmidt, Anca D. Dragan, and Moritz Hardt. Model reconstruction from model explanations, 2018.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models, 2018.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against DNN model stealing attacks. In ICLR. OpenReview.net, 2020.
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. Activethief: Model extraction using active learning and unannotated public data. In AAAI, 2020.
David Rolnick and Konrad Körding. Reverse-engineering deep relu networks. In ICML, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp, 2019.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX Security. USENIX, 2016.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin't ichi Satoh. Embedding watermarks into deep neural networks. Proceedings of the 2017 ACM on International Conference on Multime- dia Retrieval, Jun 2017. doi: 10.1145/30789711 3077997 URL http: / //dx doi. owo/o.zgg/1..
1145/3078971 0115/3308977.3308977 3078974

---

Under review as a conference paper at ICLR 2021
Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. NAACL-IILT, 2018, 2017.
Yaqiong Yao and Hai HaiYing Wang. Optimal subsampling for softmax regression. Statistical Papers, 60(2):235-244 249, 2019.
10

---

Under review as a conference paper at ICLR 2021
Table Results for learning-basee (Distill)), algebraic attacks (Extract), and hybrid attacks using an extraction dataset 10% the size of the training dataset of the target model (6734 queries for SST-2 and 270 for MNLI). All layers of the target model were fine-tuned with learning-rate 2 x 10 10-5 We use the same hyperparameters as Krishna al. (2020) for distillation.
Extract
Distill
Extract-then-Disiill
Distill-then-Extral
Task
Model (Acc.)
Acc.
Agr.
Acc.
Agr.
Acc.
Agr.
Acc.
Agr.
BERT-Small
0.751
0.803
0.856
0.928
0.806
0.849
0.846
0.930
(0.875)
SST-2
BERT-Bas (0.912)
0.827
0.858
0.885
0.919
0.907
0.945
0.909
0.936
BERT-Small (0.777)
0.479
0.504
0.724
0.854
0.660
0.741
0.731
0.862
MNLI
BERT-Base
0.496
0.515
0.803
0.888
0.763
0.825
0.806
0.894
(0.840)
A COMBINING LEARNING-BASED AND ALGEBRAIC ATTACKS
We draw a side-by-side comparison between learning-based and algebraic extraction attacks and explore whether combining both types of attacks can give better overall results. We follow the learning-based approach from Krishna et al. 2020), which uses (black-box) distillation to extract an approximation a target model. We evaluate two different hybrid attacks:
1. Distill-then- Extract.. First distill a model using the public pre-trained encoder as a base. Then use algebraic extraction reusing the same set of queries to extract and replace the last layer.
2. Extract-then-Distiee First extract the last layer using the algebraic attack with the public pre-trained encoder. Then reuse the same set of queries to distill the resulting model.
They show that although learning based attacks on their own do generally better than algebraic attacks for large number of queries, their combination using either strategy achieves consistently better agreement. The Distill-then-Extral strategy appears superior the Exxract-thenn-istii strategy. Given that algebraic attack reusing queries from distillation is inexpensive and that it starts from good baseline, the gains although modest show that combining both attacks is cheap and effective.
The results are shown in Table 3

---

Under review as a conference paper at ICLR 2021
B FULL EXPERIMENTAL RESULTS WITH VARYING LEARNING RATE
Effect of learning rate on task accuracy of extracted models, and agreement with target model on in-distribution (Fig. and random (Fig. 4 queries.
36T-2 (BERT Base), Squaries 1535)
MNLI (BERT Basa), equaries= 1635)
1.0
1.0
Target
Extrard
0.9
0.9
0.8
0.8
0.7
0.8
Targel model
0.5
agreemmnn
0.4
10" Learning nele (n)
Learning rale (1)
2 (BERT Sma #queries 1024
MNLI (BERT Sma #quories 1024
1.0
1.0
agreement
0.8
agrowment
0.5
Spore
Score
0.7
Target mocel acouracy
mode acaraey
0.5
Valdation agreement
Faandom spreement
10"7
-0" Leamiing rate 10)
10"
10"
10" Learning rate (n))
Figure 3: Extraction with in-distribution queries
SST-2 (BERT-Besee BBoTe) lqueries 1536)
MNLI (BERT-Baae) equeries 1538)
1.0
1.0
acouracy
0.9
Score
0.7
assurayy sgmommnn
0.5
Rancom agreement
10" Leamiing rate in)
10" Learning rate (n)
SST-2 BERRT-Small, Hqueries 1024)
MNLI [BERT- (mall,, Moueries 1024)
1.0
Target model accurncy
model socunddy agrasment
agpeamentt
0.8
0.8
0.7
Targel mode accuraty Extrsted mode #cousRy
05
Wildstion sgreerenn
Rancom sgreement
10" Learning Tale 10)
10
10 Learning rale (1))
10
5
Figure 4: Extraction with random queries
12

---

Under review as a conference paper at ICLR 2021
C FULL EXPERIMENTAL RESULTS WITH VARYING NUMBER OF QUERIES
C.I BERT-BASE AND SST-2
Effect number of queries on task accuracy of extracted model and agreement with target model, for in-distribution (Fig. 5 and random (Fig. 6 queries. Baseline accuracy of random guess: 50%.
SST-2 11- Real
SST-2 JSERT-Base, f 10-7), Real
0.8
0.7
0.7
Random baseline
Random baseline
0.6
Extractdd
Terget medel accuracy
0.6
Target model accurecy
rected model accuracy
Validatiio
0.5
0.5
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2816 Number of queries
2304
3328
3840
Number queries
SST (BERT Base, 10-*1 Real queries
SST (BERT Base, 10 Real queries
0.9
0.8
0.7
0.7
Scote
0.6
0.6
0.5
Randum baseire
Bandom baselne
0.4
Target model accuracy
0.4
Thrget model accuracy
Extracted model accuracy
Extracted model accuracy
Idatiin egreement Random agreement
0.
Validarion agnvement
Random
708
1280
1792
2304 Number quertes
2016
3328
3840
708
1280
1792
2304 Number of queries
2816
3328
3840
Figure 5: Extraction with in-distribution queries
5ST-2
(BERT-Base 0) Random queries
SST-2 (BEAT-Base, n- 10 1) Random querles
1.0
0.9
0.8
0.7
Random baseine
0.6
Random baseine
0.5
Tarpet medsl sccuracy
Target model accuracy
Extracted model accuraty
Ext rectee model arturecy
Va Isation agreement Random agresment...
0.5
Validorion agreement Random agresment
0.5
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2516 Number of querles
2304
3328
3940
Number queres
SST-2 (BERT Base, n= 10 Random queries
SST-2 (BERT-Base, 2 10-5), Random queries
0.8
0.8
Score
Score
0.7
0.7
0.6
ins
baseine
Target model sccurary Excracted model sccuracy ristion agreemen Tansom agreement
Thrget model artumry
0.5
Ext Extrocted hodel accuracy
Validation agreement.
Random agreement
1280
1792
2304
2816
3328
3840
768
1280
1792
2304
2816
5328
3540
Number queries
Number of queries
Figure 6: Extraction with random queries
13

---

Under review as a conference paper at ICLR 2021
C.2 BERT- BERT-BASE AND MNLI
Effect of number of queries task accuracy of extracted model and agreement with target model, for in-distribution (Fig. 7 and random (Fig. 8) queries. Baseline accuracy of random guess: 33%.
MNLI (BERT Q- O), queries
MNLI (BERT-Basa, 0 Real
1.0
0.7
0.6
0.8
Random bace Inc
Target medel accuracy
Score
Excracted model arcunary
0.6
Va Isation agreement
Random egresment
0.4
Rahderf bateling
0.3
Target model acouracy
recledd model acturacy
0.4
0.
Valistrtion agroement
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2816 Number of queries
2304
3328
3840
Number querles
MNU (BERT-Base. Base. 10-9), Real queries
MNL [BERT Base. 10-*), Real queries
0.7
0.8
0.7
0.6
0.5
0.5
Sanzom base Ing
0.4
Random baseine
"Trget model arcurary
Thrret model artuncy'
Extracted modal accuraty
0.5
Extracted medel accuracy
idation agraement. Random agreement
Validation agreement
0.2
0.2
Random agreement
765
1280
1792
23004
2816
3328
3840
768
1280
1792
2304 Number of queries
2816
8328
3840
Number queries
Figure 7: Extraction with in-distribution queries
MNU (BERT-Base, n 0J. Random queries
MNU (BERT-Base, n 10-*), Random queries
Random baseine
Thr model artuncyy
0.9
0.8
Extracted model accuracy
Validation eeremment
0.7
Random agreement
Ransom base
0.7
"ergel model
Extracted
agreament
0.5
0.5
0.4
03
1280
3840
queries
queries
[BERT Base, n 1= 1), Random oueries
MNUI Basc, o 2= queries
0.8
0.7
2021 Sandom hase Ire
1.04 Randam baswine
Score
Excracted model
medel accuracy
0.6
Ext secled model acturacy
Tarpet model acourayy
tcatton agreement
Validarion agreement
0.5
0.5
Random
0.4
0.4
1280
1792
3840
1280
2304
3840
Numbar querles
Figure 8: Extraction with random queries
14

---

Under review as a conference paper at ICLR 2021
C.3 BERT-SMALE AND SST-2
Effect of number of queries task accuracy of extracted model and agreement with target model, for in-distribution (Fig. 9 and random (Fig. 10 queries. Baseline accuracy of random guess: 50%.
SST-2 ABERT-Smal.. n= 0). Real queries
SST-2 (BERT-Small n= 10 Real queries
Random basaine
Thrget model acouricy
Extracted medel accuracy
Validweion
0.8
Random
0.7
0.6
Target accurary
Extracted sccuracy
0.5
Idaaion egreement
Random agreement.
0.4
Number queries
queries
SST-2 (BERT-Small, n- *) Real querles
(BERT-Small, - Real quertes
0.
0.6
0.5
Rancom base
Random
Escrected agreement agreameett
Valisation agreement agresment
0.4
1280
1792
3328
3940
2304
3840
querles
Figure 9: Extraction with in-distribution queries
SST-2 IBERT-Small, 0).
5ST-2 [BERT-5mall, 10-7, querles
0.8
0.8
0.7
Fandom baseline
Random baselie
0.6
Extracted accuraty
larget medel accuracy
Extracted model accurccy
Targel model
Valisation
0.5
agresment.....
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2304
2816
3328
3840
Number queries
Number of queries
SST (BERT Small, 10-*) Random queries
SST (BERT Small n= 10-), Random
0.9
0.8
07
Rantum base ine
"Srget model sccuraty
0.6
Bandom basaine
Thrget model accuracy
Excracted model accuracy
Extracted medel accuracy
0.5
iselion aguemeett. Sancom agreement
05
Randomt sgreement
Validwiion agrvement
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2304
2816
3328
3840
Number queries
Number of queries
Figure 10: Extraction with random queries
15

---

Under review as a conference paper at ICLR 2021
C.4 BERT-SMALL AND MNLI
Effect of number of queries on task accuracy of extracted model and agreement with target model, for in-distribution (Fig. and random (Fig. queries. Baseline accuracy of random guess: 33%.
MNLI (BER--Smaml Real quaries
MNLI (BERT-Small, -- Real quertes
1.0
0.8
0.7
0.8
Random base Inc
Target medel accuracy
0.6
Excracted model arcunary
0.6
Va Isation agreement
Random egresment
0.5
0.4
Random baselhe
Tarpet model acouracy
seited model acturacy Vallstrtion agroement
0.4
03
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2816 Number of queries
2304
3328
3840
Number querles
MNLI (BERT- Small, 10-9) Rea queries
(BERT-Small,
MNL IBERT-Small. n= 10"5), Resl queries
0.8
0.7
0.5
Score
0.4
0.5
0.4
Random baseing
Random baseine en
0.2
Target model accurary
Extractes medal securaty
Thrpet model arcuncy
Extracted medcl acouracy
Edation ayneement
02
Validaiinn sammmnn
Sandom agreement
Random agreement
768
1280
1792
2304
2816
3328
3840
768
1280
1792
2304
2816
5328
3840
Number queries
Number of queries
Figure 11: Extraction with in-distribution queries
MNLI (BERT-Small, n 0), Random queries
MNU (BERT-Smait n 10-7), Random queries
0.9
0.8
Ransom base
0.7
Random baseine
0.7
"ergel model
model
Extracted
0.6
Extracted model
Validation
agreament
0.5
0.5
0.4
1280
1792
3840
3840
queries
queries
(BERT n- 10""1, queries
(BERT Small, n 12x100
Random queries
Sandom base Ine
Target medel sccuracy
0.
Excrected model accunaty Va Isaition agreement
0.8
0.7
Random agreament
0.7
1.04 Randam baswine
Score
Tarpet model acouracy
Ext sccldd model acturacy
0.6
Validarion agreement
0.5
Random
0.5
0.4
0.3
1280
1792
3840
1280
1792
2304
2816
3328
3840
Numbar querles
querles
Figure 12: Extraction with random queries
16


Reviews:
[{"review_id": "Tl4eK89Ywur", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n\nThe paper considers the reconstruction of the last layer for NLP data processing models. This problem is equivalent to the parameter estimation for logistic regression in the first of the paper and quite close to it in the second part when we purposely change the encoder via transfer learning.\n\nNo surprise, that the reconstruction in this setting works well. This is what we already know from linear algebra and Gauss-Markov [1], Bernstein-von-Mises like theorems in statistics [2, chapter 10].\nMore interesting is the part about what is happening, when we deal with reconstruction under a transfer learning setting. In this case, we observe a predictable degradation of the quality of the models, but nothing more specific\n\n##########################################################################\nReasons for score: \n\nI vote for rejection, as this paper doesn't contribute to our understanding of what is happening in real-world NLP models with many layers, rather focusing on the last layer fine-tuning.\n\n##########################################################################\nMore detailed review:\n\n################\nTheoretical results\n\nAll proposition in the paper are obvious and also equivalent to the recovery procedure for the coefficients of a multiclass logistic regression:\n1. Proposition 1 is obvious\n2. Proposition 2 is obvious\n3. Proposition 3 is obvious\n4. Proposition 3 is obvious\n\nThe general statement that concludes this section and leads to further experiments should be compared to theoretical results for softmax (or multinominal) regression, see e.g. [3] for some details on the quality of the estimates in this setting. Also, see similar results for logistic regression in [4]. Both these papers present result on the quality of parameters' estimates in a more advanced subsampling setting, and even in this case, they provide the speed of converges for the error of parameter estimates. \n\nSo for the benefit of the quality of the paper, I suggest dropping all theoretical results as they are not new.\n\n################\nPractical results\n\n1. Due to the reasons similar to that mentioned above the experiments for $\\eta = 0$ can be dropped to avoid confusion from the reader\n2. For the setting with the fine-tuning of the models, we can see from experiments that after learning emerges a disagreement between the parameters estimates via the proposed procedure and the initial values of parameters. In particular, how can we measure the distance between two models even if they are one-layer logistic regression models, and can we do something if there is one layer in a setting closer to the white box problem. \n\n[1] Henderson, C. R. (1975). Best linear unbiased estimation and prediction under a selection model. Biometrics, 423-447.\n[2] Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge university press.\n[3] Yao, Y., & Wang, H. (2019). Optimal subsampling for softmax regression. Statistical Papers, 60(2), 235-249.\n[4] Wang H, Zhu R, Ma P (2018b) Optimal subsampling for large sample logistic regression. J Am Stat Assoc 113(522):829–844", "extra_scores": {}, "extra_text": {}}, {"review_id": "Es9JrCKoPTw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper is an interesting study of algebraic model extraction attacks on modern NLP models based on BERT. Model extraction is the setting where a malicious attacker tries to reconstruct a copy of a black-box inference API without access to the original training data. Prior work [1] showed these attacks are possible on BERT models using a distillation-like learning method, using gibberish sequences of words as queries to the API. However, these attacks needed large number of queries for success. This work adopts a different strategy --- equation solving the parameters of the neural network using least square linear algebra methods. This not only allows extraction with lesser queries, but also ensures greater similarity between the API and extracted model (\"high fidelity\", [2]). The attacks in this paper work perfectly in settings where BERT is frozen and a single classification layer is fine-tuned. However, the attacks are not as effective in the more practical setting where BERT is fine-tuned, and the authors perform a thorough analysis varying critical hyperparameters.\n\n-----------------------------------\n\nStrengths of the Paper:\n\n1. This is a new attack setup (especially in the BERT fine-tuning setup), algebraic attacks have only been attempted on very shallow neural networks with ReLU activations. Algebraic attacks have several advantages for the attacker like high fidelity and small query budgets. In the frozen BERT, single layer setting this works perfectly with a very small query budget (however, see my Weakness #1).\n\n2. The paper is well written and easy to understand, and authors do a very good analysis of their attacks varying important hyperparameters like learning rate, number of queries, type of queries.\n\n-----------------------------------\n\nWeaknesses of the Paper:\n\n1. I don't think the setting where the attack works perfectly (frozen BERT with a single classification layer) is practical. Theoretically it's fairly obvious this should work, and I think the main contribution here is a empirical confirmation that it works with real data. There are a number of reasons why this is not practical --- (1) there are actually 2 layers between the sequence_output and the final logits, with a tanh activation (see https://github.com/google-research/bert/blob/master/modeling.py#L219-L232), or look for `BertPooler` in the HuggingFace code. These two layers are needed to separate the MLM representation from the logits. Even in the frozen setting, I anticipate fine-tuning atleast these two classification layers; (2) target accuracies are quite poor without fine-tuning. 75% on SST2 (Target Acc from Table 1) is quite poor, even a 1-layer CNN does much better and gets 83-88% accuracy (https://arxiv.org/abs/1408.5882). Similarly, the Target Acc. for MNLI is close to 33%. Without fine-tuning and just a single classifier layer, I don't expect people to use BERT; (3) Finally access to probability distributions / logits might be a strong assumption in structured prediction NLP tasks like question answering or NER.\n\n2. In the more practical setting of finetuning the model, the attacks are not effective. While I like the overall idea of leveraging the BERT pretrained checkpoint to do algebraic attacks, the authors' results show that this by itself is not sufficient to make an effective attack. The authors statement \"For the fine-tuned models, agreement drops to 10% below the learning-based state-of-the-art\" is not entirely correct. It is only true on the simpler SST-2 task, where even 1-layer CNNs perform exceedingly well. In the harder MNLI task, agreement is far lower than state-of-the-art [1], with a gap of 44% vs 82.2%. Performance of the extracted models on MNLI are quite low, about 40-45% in Figure 1 which is quite close to random guessing (BERT-base gets 84-85% accuracy).\n\n-----------------------------------\n\nOverall Recommendation:\n\nThe authors did a good job with presentation and studied an interesting algebraic attack. However, the attack only works in an impractical setting of a frozen BERT, and is ineffective in the more practical setting of finetuning BERT. Intuitively, it's fairly obvious this attack should work in the frozen BERT, single layer setting. This result by itself is not sufficient for acceptance to ICLR. While I'm leaning reject, I encourage the authors to explore the BERT fine-tuning setting more. For instance, can a hybrid attack be constructed which uses the best of both worlds? Since queries do not seem to cost much [1], can the attacks be stronger in this hybrid setting with more liberal query budgets?\n\n-----------------------------------\n\nMinor Issues:\n\nIn Proposition 1, uniformly sampling from a n-d cube is not entirely correct. BERT has a fixed discrete input space, since you only feed text as input to BERT. You are going to have a maximum of V^L unique points in the support of the [CLS] vector space (where V is the vocab and L is the maximum sequence length). Since V ~ 30k and L = 512, I guess it's not a problem practically.\n\n392.702 ---> 392,702\n\n-----------------------------------\n\nReferences:\n\n[1] - https://arxiv.org/abs/1910.12366  \n[2] - https://arxiv.org/abs/1909.01838", "extra_scores": {}, "extra_text": {}}, {"review_id": "DUpl0854on-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a range of algebraic model extraction attacks (different from the prevalent learning-based approaches) for transformer models trained for NLP tasks in a grey-box setting i.e., an existing, public, usually pretrained encoder, with a private classification layer. Through attacks on different sizes of models and a range of downstream tasks, they observe that only a portion of the embedding space forms a basis of the tuned classification layer’s input space, and using a grey-box method, this can be algebraically computed. The pretraining-finetuning experiments on different tasks also show the smallest number of dimensions needed for high-fidelity extraction, and also that the model extraction attacks effectiveness decreases with fine-tuning the larger models base layers---which is an insight that is very useful for a lot of interpretability/probing work.\n\n\nReason for score:\n\nI think this paper is very well-formulated---both theoretically and empirically with promising results that will be useful not just for grey-box adversarial attacks, but also for works interesting in the effects of pretraining-finetuning (which at this point encompasses nearly all NLP tasks). The empirical results look promising---however I would like to see this demonstrated on more than just 2 datasets (and maybe even a GPT-like model, instead of just BERT) to see if (1) the results hold empirically and (2) if there any insights to be gleaned about adversarial attacks from different task structures and model types.\n\n\nPositive points + questions:\n\n1. The transformation of the raw logits for recovering information is really interesting. In the experiments for the random set of n embeddings chosen to form a basis of the last layer’s input space---are there any insights on what those embeddings amount to semantically; and also what a ground truth selection of embeddings (e.g., that an oracle adversary would compute) should be? It would be helpful to have a discussion and examples of those.\n\n2. Is there a difference in extraction results when using in-distribution queries vs. random? Most of the results say “extraction is possible with both” which is good to see, but a more finer-grained analysis/explanation of benefits/pitfalls of each would really help clarity.\n\n3. It’s nice that both a single-sentence and pairwise-sentence (SST-2 vs. MNLI) task are used to evaluate effects for the fine-tuning experiments in big transformer models.\n\n4. The results look very promising and these insights are extremely helpful even for general probing/interpretability works (especially the learning rate finetuning effects) and also hold up to existing BERT-finetuned results.\n\n5. Unlike previous work, this algebraic model extraction words even with non-linear activation layers---and this is helpful given the current standard of fine-tuning large transformer models e.g., with simple MLP/softmax classifiers. \n\n6. Slightly different from previous work, not only can this work when attacks require embeddings to be chosen, but also when selecting (e.g., random/or from a distribution) needs to be done as well. \n\n\nNegative points + questions:\n\n1. For the fine-tuning/learning rate experiments it would be good to evaluate this on more than just 2 tasks (e.g., maybe a range of different tasks in GLUE) not only to see if the trend still holds, but also to see if task “type” or characteristics of the task/fine-tuning affect the extraction fidelity.\n\n2. The *extracted model accuracy of BERT-base with MNLI seems to be quite static (almost no effect on increasing or decreasing learning rate)---and it would be really helpful to see how statistically significant those results are and what they look like over different seeds.\n\n3. Is there a comparison between the algebraic approach and a learning-based approach for the same tasks? (I think the paper is novel and useful enough in itself, but it would be helpful to see a side-by-side comparison).\n\n4. Is there a comparison between extracting only a single layer or going beyond to having multiple layers of target/finetuned classifiers? Is this approach feasible and similarly beneficial as a grey-box attack in that scenario? It would be really helpful to have a discussion on what that would require for future work.\n\n\nAdditional minor comments:\n\nThis is really well written and placed in literature, no minor nitpicks re: writing!\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CZMk-m1WvFw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an algebraic attack for extracting the parameters\nof a semi-private language model that consists of a pre-trained\nencoder and a privately trained classification layer.\n\nThe method is to first sample from the input space, compute their\nembeddings using the known encoder, and then use the embeddings and\nthe queried classifier softmax output to solve for the classifier weights.\nIt overcomes the obstacles encountered by former such attempts due to \nthe requirements of known embeddings and raw logits.\n\nThe paper provides support for the method in arguing that a random\nbasis (like an arbitrary set of embeddings obtained from encoding a\nset of arbitrary, distinct input) is sufficient to serve as a basis\nthat spans the classifier layer's input space, and that using the\nsoftmax output instead of raw logits can lead to equivalent solutions\nup to a translation invariance.\n\nExperiments on two public datasets and two versions of the BERT model\nshow the effectiveness of the method, and demonstrate that\nthe number of queries needed is relatively small,\nthe probes can be drawn from the distribution of legitimate input,\nand that fine tuning the encoder makes the attack less effective as the true\nembeddings deviate from those computed from the publicly known\nencoder.\n\nThe paper is well written and the method is sound and practical.\nSuggestions on defenses against such attacks are of good reference\nvalue.\n\nOne question is whether the proposed approach could be put to some\npositive use, such as learning about a model's potential weakness in\nthe input space? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion with the reviewers, it seems that a. without fine-tuning the result is close to being trivial (as noted also by two reviewers) b. with fine-tuning results are lower c. The setup of just a linear classification layer is less common (but exists) d. The cases where extraction succeeds the performance is low such that BERT would not even be used.

In response, the authors offer many interesting directions: a. Propose a new hybrid approach that combines learning-based and extraction-based methods b. Run experiments to try and support the claim that their setup of one linear layer with frozen layers is practical.

These proposed modifications are interesting and show that there is potential in this paper, but it deviates substantially from the original paper and still, caveats remain, so my recommendation is to re-submit after further pursuing the new directions proposed in the response.
Decision: Reject