Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes a range of algebraic model extraction attacks (different from the prevalent learning-based approaches) for transformer models trained for NLP tasks in a grey-box setting i.e., an existing, public, usually pretrained encoder, with a private classification layer. Through attacks on different sizes of models and a range of downstream tasks, they observe that only a portion of the embedding space forms a basis of the tuned classification layer’s input space, and using a grey-box method, this can be algebraically computed. The pretraining-finetuning experiments on different tasks also show the smallest number of dimensions needed for high-fidelity extraction, and also that the model extraction attacks effectiveness decreases with fine-tuning the larger models base layers---which is an insight that is very useful for a lot of interpretability/probing work.


Reason for score:

I think this paper is very well-formulated---both theoretically and empirically with promising results that will be useful not just for grey-box adversarial attacks, but also for works interesting in the effects of pretraining-finetuning (which at this point encompasses nearly all NLP tasks). The empirical results look promising---however I would like to see this demonstrated on more than just 2 datasets (and maybe even a GPT-like model, instead of just BERT) to see if (1) the results hold empirically and (2) if there any insights to be gleaned about adversarial attacks from different task structures and model types.


Positive points + questions:

1. The transformation of the raw logits for recovering information is really interesting. In the experiments for the random set of n embeddings chosen to form a basis of the last layer’s input space---are there any insights on what those embeddings amount to semantically; and also what a ground truth selection of embeddings (e.g., that an oracle adversary would compute) should be? It would be helpful to have a discussion and examples of those.

2. Is there a difference in extraction results when using in-distribution queries vs. random? Most of the results say “extraction is possible with both” which is good to see, but a more finer-grained analysis/explanation of benefits/pitfalls of each would really help clarity.

3. It’s nice that both a single-sentence and pairwise-sentence (SST-2 vs. MNLI) task are used to evaluate effects for the fine-tuning experiments in big transformer models.

4. The results look very promising and these insights are extremely helpful even for general probing/interpretability works (especially the learning rate finetuning effects) and also hold up to existing BERT-finetuned results.

5. Unlike previous work, this algebraic model extraction words even with non-linear activation layers---and this is helpful given the current standard of fine-tuning large transformer models e.g., with simple MLP/softmax classifiers. 

6. Slightly different from previous work, not only can this work when attacks require embeddings to be chosen, but also when selecting (e.g., random/or from a distribution) needs to be done as well. 


Negative points + questions:

1. For the fine-tuning/learning rate experiments it would be good to evaluate this on more than just 2 tasks (e.g., maybe a range of different tasks in GLUE) not only to see if the trend still holds, but also to see if task “type” or characteristics of the task/fine-tuning affect the extraction fidelity.

2. The *extracted model accuracy of BERT-base with MNLI seems to be quite static (almost no effect on increasing or decreasing learning rate)---and it would be really helpful to see how statistically significant those results are and what they look like over different seeds.

3. Is there a comparison between the algebraic approach and a learning-based approach for the same tasks? (I think the paper is novel and useful enough in itself, but it would be helpful to see a side-by-side comparison).

4. Is there a comparison between extracting only a single layer or going beyond to having multiple layers of target/finetuned classifiers? Is this approach feasible and similarly beneficial as a grey-box attack in that scenario? It would be really helpful to have a discussion on what that would require for future work.


Additional minor comments:

This is really well written and placed in literature, no minor nitpicks re: writing!

review_title: Important theoretical + empirical results for model extraction attacks, which is helpful and insightful for general NLP interpretability/probing work as well. 

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an algebraic attack for extracting the parameters
of a semi-private language model that consists of a pre-trained
encoder and a privately trained classification layer.

The method is to first sample from the input space, compute their
embeddings using the known encoder, and then use the embeddings and
the queried classifier softmax output to solve for the classifier weights.
It overcomes the obstacles encountered by former such attempts due to 
the requirements of known embeddings and raw logits.

The paper provides support for the method in arguing that a random
basis (like an arbitrary set of embeddings obtained from encoding a
set of arbitrary, distinct input) is sufficient to serve as a basis
that spans the classifier layer's input space, and that using the
softmax output instead of raw logits can lead to equivalent solutions
up to a translation invariance.

Experiments on two public datasets and two versions of the BERT model
show the effectiveness of the method, and demonstrate that
the number of queries needed is relatively small,
the probes can be drawn from the distribution of legitimate input,
and that fine tuning the encoder makes the attack less effective as the true
embeddings deviate from those computed from the publicly known
encoder.

The paper is well written and the method is sound and practical.
Suggestions on defenses against such attacks are of good reference
value.

One question is whether the proposed approach could be put to some
positive use, such as learning about a model's potential weakness in
the input space? 
review_title: Practical method for extracting semi-private language models with some demonstrated success

final_score: rating: 3: Clear rejection
review_content: ##########################################################################
Summary:

The paper considers the reconstruction of the last layer for NLP data processing models. This problem is equivalent to the parameter estimation for logistic regression in the first of the paper and quite close to it in the second part when we purposely change the encoder via transfer learning.

No surprise, that the reconstruction in this setting works well. This is what we already know from linear algebra and Gauss-Markov [1], Bernstein-von-Mises like theorems in statistics [2, chapter 10].
More interesting is the part about what is happening, when we deal with reconstruction under a transfer learning setting. In this case, we observe a predictable degradation of the quality of the models, but nothing more specific

##########################################################################
Reasons for score: 

I vote for rejection, as this paper doesn't contribute to our understanding of what is happening in real-world NLP models with many layers, rather focusing on the last layer fine-tuning.

##########################################################################
More detailed review:

################
Theoretical results

All proposition in the paper are obvious and also equivalent to the recovery procedure for the coefficients of a multiclass logistic regression:
1. Proposition 1 is obvious
2. Proposition 2 is obvious
3. Proposition 3 is obvious
4. Proposition 3 is obvious

The general statement that concludes this section and leads to further experiments should be compared to theoretical results for softmax (or multinominal) regression, see e.g. [3] for some details on the quality of the estimates in this setting. Also, see similar results for logistic regression in [4]. Both these papers present result on the quality of parameters' estimates in a more advanced subsampling setting, and even in this case, they provide the speed of converges for the error of parameter estimates. 

So for the benefit of the quality of the paper, I suggest dropping all theoretical results as they are not new.

################
Practical results

1. Due to the reasons similar to that mentioned above the experiments for $\eta = 0$ can be dropped to avoid confusion from the reader
2. For the setting with the fine-tuning of the models, we can see from experiments that after learning emerges a disagreement between the parameters estimates via the proposed procedure and the initial values of parameters. In particular, how can we measure the distance between two models even if they are one-layer logistic regression models, and can we do something if there is one layer in a setting closer to the white box problem. 

[1] Henderson, C. R. (1975). Best linear unbiased estimation and prediction under a selection model. Biometrics, 423-447.
[2] Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge university press.
[3] Yao, Y., & Wang, H. (2019). Optimal subsampling for softmax regression. Statistical Papers, 60(2), 235-249.
[4] Wang H, Zhu R, Ma P (2018b) Optimal subsampling for large sample logistic regression. J Am Stat Assoc 113(522):829–844
review_title: The paper considers a parameter estimation for the logistic regression and - no surprise - succeeds in it

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper is an interesting study of algebraic model extraction attacks on modern NLP models based on BERT. Model extraction is the setting where a malicious attacker tries to reconstruct a copy of a black-box inference API without access to the original training data. Prior work [1] showed these attacks are possible on BERT models using a distillation-like learning method, using gibberish sequences of words as queries to the API. However, these attacks needed large number of queries for success. This work adopts a different strategy --- equation solving the parameters of the neural network using least square linear algebra methods. This not only allows extraction with lesser queries, but also ensures greater similarity between the API and extracted model ("high fidelity", [2]). The attacks in this paper work perfectly in settings where BERT is frozen and a single classification layer is fine-tuned. However, the attacks are not as effective in the more practical setting where BERT is fine-tuned, and the authors perform a thorough analysis varying critical hyperparameters.

-----------------------------------

Strengths of the Paper:

1. This is a new attack setup (especially in the BERT fine-tuning setup), algebraic attacks have only been attempted on very shallow neural networks with ReLU activations. Algebraic attacks have several advantages for the attacker like high fidelity and small query budgets. In the frozen BERT, single layer setting this works perfectly with a very small query budget (however, see my Weakness #1).

2. The paper is well written and easy to understand, and authors do a very good analysis of their attacks varying important hyperparameters like learning rate, number of queries, type of queries.

-----------------------------------

Weaknesses of the Paper:

1. I don't think the setting where the attack works perfectly (frozen BERT with a single classification layer) is practical. Theoretically it's fairly obvious this should work, and I think the main contribution here is a empirical confirmation that it works with real data. There are a number of reasons why this is not practical --- (1) there are actually 2 layers between the sequence_output and the final logits, with a tanh activation (see https://github.com/google-research/bert/blob/master/modeling.py#L219-L232), or look for `BertPooler` in the HuggingFace code. These two layers are needed to separate the MLM representation from the logits. Even in the frozen setting, I anticipate fine-tuning atleast these two classification layers; (2) target accuracies are quite poor without fine-tuning. 75% on SST2 (Target Acc from Table 1) is quite poor, even a 1-layer CNN does much better and gets 83-88% accuracy (https://arxiv.org/abs/1408.5882). Similarly, the Target Acc. for MNLI is close to 33%. Without fine-tuning and just a single classifier layer, I don't expect people to use BERT; (3) Finally access to probability distributions / logits might be a strong assumption in structured prediction NLP tasks like question answering or NER.

2. In the more practical setting of finetuning the model, the attacks are not effective. While I like the overall idea of leveraging the BERT pretrained checkpoint to do algebraic attacks, the authors' results show that this by itself is not sufficient to make an effective attack. The authors statement "For the fine-tuned models, agreement drops to 10% below the learning-based state-of-the-art" is not entirely correct. It is only true on the simpler SST-2 task, where even 1-layer CNNs perform exceedingly well. In the harder MNLI task, agreement is far lower than state-of-the-art [1], with a gap of 44% vs 82.2%. Performance of the extracted models on MNLI are quite low, about 40-45% in Figure 1 which is quite close to random guessing (BERT-base gets 84-85% accuracy).

-----------------------------------

Overall Recommendation:

The authors did a good job with presentation and studied an interesting algebraic attack. However, the attack only works in an impractical setting of a frozen BERT, and is ineffective in the more practical setting of finetuning BERT. Intuitively, it's fairly obvious this attack should work in the frozen BERT, single layer setting. This result by itself is not sufficient for acceptance to ICLR. While I'm leaning reject, I encourage the authors to explore the BERT fine-tuning setting more. For instance, can a hybrid attack be constructed which uses the best of both worlds? Since queries do not seem to cost much [1], can the attacks be stronger in this hybrid setting with more liberal query budgets?

-----------------------------------

Minor Issues:

In Proposition 1, uniformly sampling from a n-d cube is not entirely correct. BERT has a fixed discrete input space, since you only feed text as input to BERT. You are going to have a maximum of V^L unique points in the support of the [CLS] vector space (where V is the vocab and L is the maximum sequence length). Since V ~ 30k and L = 512, I guess it's not a problem practically.

392.702 ---> 392,702

-----------------------------------

References:

[1] - https://arxiv.org/abs/1910.12366  
[2] - https://arxiv.org/abs/1909.01838
review_title: Interesting research questions and thorough analysis, but attacks are weak in practical settings

Meta-review: After discussion with the reviewers, it seems that a. without fine-tuning the result is close to being trivial (as noted also by two reviewers) b. with fine-tuning results are lower c. The setup of just a linear classification layer is less common (but exists) d. The cases where extraction succeeds the performance is low such that BERT would not even be used.

In response, the authors offer many interesting directions: a. Propose a new hybrid approach that combines learning-based and extraction-based methods b. Run experiments to try and support the claim that their setup of one linear layer with frozen layers is practical.

These proposed modifications are interesting and show that there is potential in this paper, but it deviates substantially from the original paper and still, caveats remain, so my recommendation is to re-submit after further pursuing the new directions proposed in the response.
Decision: Reject