Grey-box Extraction of Natural Language Models

1 INTRODUCTION . Machine learning models are often deployed behind APIs that enable querying the model but that prevent direct access to the model parameters . This restriction aims to protect intellectual property , as models are expensive to train and hence valuable ( Strubell et al. , 2019 ) ; security , as access to model parameters facilitates the creation of adversarial examples ( Laskov et al. , 2014 ; Ebrahimi et al. , 2018 ) ; and privacy , as model parameters carry potentially sensitive information about the training data ( Leino & Fredrikson , 2020 ) . Model extraction attacks ( Tramèr et al. , 2016 ) attempt to replicate machine learning models from sets of query-response pairs obtained via the model ’ s inference API , thus effectively circumventing the protection offered by the API . Several extraction attacks on deep neural networks ( see Jagielski et al . ( 2020 ) for a recent overview ) follow a learning-based approach ( Tramèr et al. , 2016 ; Orekondy et al. , 2018 ; Pal et al. , 2020 ; Krishna et al. , 2020 ) , where the target model is queried to label data used for training the replica . The replicas obtained in this way aim to achieve accuracy on the desired task , or agreement with the target model on predictions , but recovery of the model weights is out of scope of this approach . More recently , a novel class of attacks has emerged that uses algebraic techniques to recover the weights of deep neural networks up to model-specific invariances . Examples are the attacks of Milli et al . ( 2018 ) , which leverage observations of gradients to recover model parameters , and Rolnick & Körding ( 2020 ) ; Jagielski et al . ( 2020 ) ; Carlini et al . ( 2020 ) , which estimate gradients from finite differences of logits , and then use this information to recover model parameters . Algebraic attacks improve on learning-based attacks in that they ( i ) achieve higher-fidelity replicas and ( ii ) are orders of magnitude more query-efficient . So far , however , algebraic attacks have only been applied to small , fully connected neural networks with ReLU activations . In particular , for modern large-scale natural language models ( LLMs ) such as BERT or GPT-2 , the state-of-the-art model extraction attack is still learning-based ( Krishna et al. , 2020 ) . In this paper , we propose the first algebraic attacks on LLMs . We focus on models consisting of a pre-trained encoder and a single task-specific classification layer . We assume a grey-box setting where the encoder is public ( and hence known to the adversary ) , and the classification layer is private ( and hence the main target of the attack ) . There are two key observations that enable us to extract LLMs via algebraic attacks . The first is that it is sufficient for an adversary to know rather than to choose the embeddings that are fed into the last layer . Existing algebraic attacks can infer the inputs to hidden layers , but can only do so on piecewise linear networks and would not work on LLMs , which use non-linear activations . In the grey-box setting , an adversary can compute hidden embeddings of any input by querying the public encoder model , and can query the target LLM on the same input through the model ’ s API . We show in theory , and confirm by experiments , that a random set of n embeddings is likely to form a basis of the last layer ’ s input space . The raw outputs ( i.e. , the logits ) on this basis uniquely determine the parameters of the last linear layer , which can be recovered by a transformation to the standard basis . Our second observation is that this approach extends to the case where the API returns probabilities rather than raw logits , after normalization by the softmax function . For this , we leverage the invariance under translation of softmax to establish an invariance result for linear functions . Using this result , we show that the parameters of the last layer can be recovered ( up to invariance ) from embedding vectors spanning its input space and their corresponding probability outputs . We evaluate our attacks on LLMs of different sizes and fine-tuned to different downstream tasks . We study the effects of using different types and numbers of extraction queries and different learning rates for fine-tuning the encoder model . Our key findings are : • When the target model ’ s base layers are frozen during fine-tuning ( i.e. , the attacker can get the exact embedding of any input ) , the attack is extremely effective . With only twice as many queries as the dimension of the embedding space ( e.g. , 1536 for BERT-base ) , we extract models that achieve 100 % fidelity with the target , for all model sizes and tasks . • When the model ’ s base layers are fine-tuned together with the task-specific layer , the embeddings of the base model only approximate those of the target model and , as expected , the fidelity of the extracted models decreases as the learning rate grows . Maybe surprisingly , for some models and downstream tasks , we are still able to extract replicas with up to 82 % fidelity and up to 79 % task accuracy , for orders of magnitude fewer queries than required by state-of-the-art learning-based attacks ( Krishna et al. , 2020 ) . • Extraction is possible using either random or in-distribution queries . Replicas extracted using in-distribution queries perform well on both in-distribution and random challenge inputs . This shows that replicas can be created from small numbers of in-distribution queries , making attempts to extract the model indistinguishable from legitimate use . In summary , we propose a novel grey-box extraction attack on natural language models that is indistinguishable from legitimate use in terms of the content and number of queries required . 2 ATTACK . We consider classification models h : X → Rn , mapping elements from X to label probabilities in Rn . We assume that h = log ◦ softmax ◦ f ◦ g consists of three components h : X g−→ Rn f−→ Rm log ◦ softmax−−−−−−−−→ Rm where • g : X → Rn is a contextualized embedding model , such as BERT or GPT-2 ; • f : Rn → Rm is an affine function computing logits from embeddings , i.e. , f ( x ) = Ax+ b with A ∈ Rm×n and b ∈ Rm ; • softmax : Rm → Rm normalizes logits to probability vectors : softmax ( x ) = exp ( xi ) ∑m i=1 exp ( xi ) . ( 1 ) We assume the adversary knows the embedding model g and tries to infer A and b ( resp . f ) . We call this adversary grey-box because it can compute the embeddings from the inputs to h . 2.1 BASIC IDEA . Milli et al . ( 2018 ) show how to reconstruct models from oracle access to gradients . Carlini et al . ( 2020 ) show how to replace gradients by finite differences of logits , enabling reconstructing models without such an oracle . To explain the basic idea , let ei = ( 0 , . . . 0 , 1 , 0 , . . . 0 ) T be the ith vector of the standard basis in Rn , and let x ∈ Rn be arbitrary . The ith column of A can be obtained as the difference between f ( x + ei ) and f ( x ) . There are two obstacles that prevent us from directly applying this idea to classification layers from LLMs , namely ( 1 ) the attack requires the embeddings to be chosen , which would effectively amount to reversing the embedding model ; and ( 2 ) the attack uses raw logits , while APIs provide only log probabilities normalized with softmax . We show next how to overcome these obstacles . 2.2 EXTRACTION FROM LOGITS . As a first step , we overcome the requirement that the adversary be able to choose the inputs to f . Specifically , we show that it is sufficient for the adversary to know the inputs , given that they are sufficiently random . We rely on the following standard result : Proposition 1 . Let x ( 1 ) , . . . , x ( n ) ∈ Rn be uniformly distributed in an n-cube . Then x ( 1 ) , . . . , x ( n ) form a basis of Rn with probability 1 . Proof . Let { x ( i ) } ni=1 be a basis of Rn and V the subspace generated by { x ( i ) } mi=1 , for m < n. For x chosen uniformly in an n-cube , the space V ∪ { x } has dimensionm+1 with probability 1 . This is because for x to fall into V , it would need to have zero as coordinates wrt x ( m+1 ) , . . . , x ( n ) , which happens with probability zero . Applying this argument inductively proves the proposition . Based on this result , we mount the following grey-box attack on f ◦ g : 1 . Choose distinct inputs { x ( j ) } Nj=1 in X with N > n ; 2 . Compute their embeddings { y ( j ) = g ( x ( j ) ) } Nj=1 and logits { z ( j ) = f ( y ( j ) ) } Nj=1 ; 3 . Construct a matrix Y ∈ R ( n+1 ) ×N where the first component of each column is 1 and the rest are from the embeddings , i.e. , Yi , j = ( 1 , y ( j ) ) T , and a matrix Z ∈ Rm×N where the columns are the logit vectors , i.e. , Zi , j = z ( j ) i ; 4 . Solve for Ã ∈ Rm×n and b̃ ∈ Rm in ( b̃ , Ã ) Y = Z , i.e. , b̃1 ã1,1 ã1 , n ... . . . ... b̃m ãm,1 ãm , n 1 . . . 1 y ( 1 ) 1 . . . y ( N ) 1 ... . . . ... y ( 1 ) n . . . y ( N ) n = z ( 1 ) 1 . . . z ( N ) 1 ... . . . ... z ( 1 ) m . . . z ( N ) m ( 2 ) Proposition 2 . Assuming g maps inputs to uniformly distributed embeddings in an n-cube1 , the attack above uniquely determines the parameters of f . I.e. , we have Ã = A and b̃ = b . Proof . By construction of ( 2 ) , we have Ãy ( j ) + b̃ = f ( y ( j ) ) for j = 1 , . . . , N . The unique solution can be obtained multiplying Z by the right-inverse of Y . For uniformly random embeddings , this right-inverse exists because Y has full rank with probability 1 by Proposition 1 . While in theory N = n+1 distinct queries are sufficient to mount the attack , computing the inverse of Y based with finite-precision arithmetic can be numerically unstable . In practice , we gather a larger set of inputs and construct an over-determined system of equations . We then numerically compute a least squares solution to Equation ( 2 ) . 1A language model with vocabulary V and maximum sequence length L can only produce |V |L different embeddings . This is many more points than representable in the precision used , so not an issue in practice .

Summary: This paper is an interesting study of algebraic model extraction attacks on modern NLP models based on BERT. Model extraction is the setting where a malicious attacker tries to reconstruct a copy of a black-box inference API without access to the original training data. Prior work [1] showed these attacks are possible on BERT models using a distillation-like learning method, using gibberish sequences of words as queries to the API. However, these attacks needed large number of queries for success. This work adopts a different strategy --- equation solving the parameters of the neural network using least square linear algebra methods. This not only allows extraction with lesser queries, but also ensures greater similarity between the API and extracted model ("high fidelity", [2]). The attacks in this paper work perfectly in settings where BERT is frozen and a single classification layer is fine-tuned. However, the attacks are not as effective in the more practical setting where BERT is fine-tuned, and the authors perform a thorough analysis varying critical hyperparameters.