Grey-box Extraction of Natural Language Models

1 INTRODUCTION . Machine learning models are often deployed behind APIs that enable querying the model but that prevent direct access to the model parameters . This restriction aims to protect intellectual property , as models are expensive to train and hence valuable ( Strubell et al. , 2019 ) ; security , as access to model parameters facilitates the creation of adversarial examples ( Laskov et al. , 2014 ; Ebrahimi et al. , 2018 ) ; and privacy , as model parameters carry potentially sensitive information about the training data ( Leino & Fredrikson , 2020 ) . Model extraction attacks ( Tramèr et al. , 2016 ) attempt to replicate machine learning models from sets of query-response pairs obtained via the model ’ s inference API , thus effectively circumventing the protection offered by the API . Several extraction attacks on deep neural networks ( see Jagielski et al . ( 2020 ) for a recent overview ) follow a learning-based approach ( Tramèr et al. , 2016 ; Orekondy et al. , 2018 ; Pal et al. , 2020 ; Krishna et al. , 2020 ) , where the target model is queried to label data used for training the replica . The replicas obtained in this way aim to achieve accuracy on the desired task , or agreement with the target model on predictions , but recovery of the model weights is out of scope of this approach . More recently , a novel class of attacks has emerged that uses algebraic techniques to recover the weights of deep neural networks up to model-specific invariances . Examples are the attacks of Milli et al . ( 2018 ) , which leverage observations of gradients to recover model parameters , and Rolnick & Körding ( 2020 ) ; Jagielski et al . ( 2020 ) ; Carlini et al . ( 2020 ) , which estimate gradients from finite differences of logits , and then use this information to recover model parameters . Algebraic attacks improve on learning-based attacks in that they ( i ) achieve higher-fidelity replicas and ( ii ) are orders of magnitude more query-efficient . So far , however , algebraic attacks have only been applied to small , fully connected neural networks with ReLU activations . In particular , for modern large-scale natural language models ( LLMs ) such as BERT or GPT-2 , the state-of-the-art model extraction attack is still learning-based ( Krishna et al. , 2020 ) . In this paper , we propose the first algebraic attacks on LLMs . We focus on models consisting of a pre-trained encoder and a single task-specific classification layer . We assume a grey-box setting where the encoder is public ( and hence known to the adversary ) , and the classification layer is private ( and hence the main target of the attack ) . There are two key observations that enable us to extract LLMs via algebraic attacks . The first is that it is sufficient for an adversary to know rather than to choose the embeddings that are fed into the last layer . Existing algebraic attacks can infer the inputs to hidden layers , but can only do so on piecewise linear networks and would not work on LLMs , which use non-linear activations . In the grey-box setting , an adversary can compute hidden embeddings of any input by querying the public encoder model , and can query the target LLM on the same input through the model ’ s API . We show in theory , and confirm by experiments , that a random set of n embeddings is likely to form a basis of the last layer ’ s input space . The raw outputs ( i.e. , the logits ) on this basis uniquely determine the parameters of the last linear layer , which can be recovered by a transformation to the standard basis . Our second observation is that this approach extends to the case where the API returns probabilities rather than raw logits , after normalization by the softmax function . For this , we leverage the invariance under translation of softmax to establish an invariance result for linear functions . Using this result , we show that the parameters of the last layer can be recovered ( up to invariance ) from embedding vectors spanning its input space and their corresponding probability outputs . We evaluate our attacks on LLMs of different sizes and fine-tuned to different downstream tasks . We study the effects of using different types and numbers of extraction queries and different learning rates for fine-tuning the encoder model . Our key findings are : • When the target model ’ s base layers are frozen during fine-tuning ( i.e. , the attacker can get the exact embedding of any input ) , the attack is extremely effective . With only twice as many queries as the dimension of the embedding space ( e.g. , 1536 for BERT-base ) , we extract models that achieve 100 % fidelity with the target , for all model sizes and tasks . • When the model ’ s base layers are fine-tuned together with the task-specific layer , the embeddings of the base model only approximate those of the target model and , as expected , the fidelity of the extracted models decreases as the learning rate grows . Maybe surprisingly , for some models and downstream tasks , we are still able to extract replicas with up to 82 % fidelity and up to 79 % task accuracy , for orders of magnitude fewer queries than required by state-of-the-art learning-based attacks ( Krishna et al. , 2020 ) . • Extraction is possible using either random or in-distribution queries . Replicas extracted using in-distribution queries perform well on both in-distribution and random challenge inputs . This shows that replicas can be created from small numbers of in-distribution queries , making attempts to extract the model indistinguishable from legitimate use . In summary , we propose a novel grey-box extraction attack on natural language models that is indistinguishable from legitimate use in terms of the content and number of queries required . 2 ATTACK . We consider classification models h : X → Rn , mapping elements from X to label probabilities in Rn . We assume that h = log ◦ softmax ◦ f ◦ g consists of three components h : X g−→ Rn f−→ Rm log ◦ softmax−−−−−−−−→ Rm where • g : X → Rn is a contextualized embedding model , such as BERT or GPT-2 ; • f : Rn → Rm is an affine function computing logits from embeddings , i.e. , f ( x ) = Ax+ b with A ∈ Rm×n and b ∈ Rm ; • softmax : Rm → Rm normalizes logits to probability vectors : softmax ( x ) = exp ( xi ) ∑m i=1 exp ( xi ) . ( 1 ) We assume the adversary knows the embedding model g and tries to infer A and b ( resp . f ) . We call this adversary grey-box because it can compute the embeddings from the inputs to h . 2.1 BASIC IDEA . Milli et al . ( 2018 ) show how to reconstruct models from oracle access to gradients . Carlini et al . ( 2020 ) show how to replace gradients by finite differences of logits , enabling reconstructing models without such an oracle . To explain the basic idea , let ei = ( 0 , . . . 0 , 1 , 0 , . . . 0 ) T be the ith vector of the standard basis in Rn , and let x ∈ Rn be arbitrary . The ith column of A can be obtained as the difference between f ( x + ei ) and f ( x ) . There are two obstacles that prevent us from directly applying this idea to classification layers from LLMs , namely ( 1 ) the attack requires the embeddings to be chosen , which would effectively amount to reversing the embedding model ; and ( 2 ) the attack uses raw logits , while APIs provide only log probabilities normalized with softmax . We show next how to overcome these obstacles . 2.2 EXTRACTION FROM LOGITS . As a first step , we overcome the requirement that the adversary be able to choose the inputs to f . Specifically , we show that it is sufficient for the adversary to know the inputs , given that they are sufficiently random . We rely on the following standard result : Proposition 1 . Let x ( 1 ) , . . . , x ( n ) ∈ Rn be uniformly distributed in an n-cube . Then x ( 1 ) , . . . , x ( n ) form a basis of Rn with probability 1 . Proof . Let { x ( i ) } ni=1 be a basis of Rn and V the subspace generated by { x ( i ) } mi=1 , for m < n. For x chosen uniformly in an n-cube , the space V ∪ { x } has dimensionm+1 with probability 1 . This is because for x to fall into V , it would need to have zero as coordinates wrt x ( m+1 ) , . . . , x ( n ) , which happens with probability zero . Applying this argument inductively proves the proposition . Based on this result , we mount the following grey-box attack on f ◦ g : 1 . Choose distinct inputs { x ( j ) } Nj=1 in X with N > n ; 2 . Compute their embeddings { y ( j ) = g ( x ( j ) ) } Nj=1 and logits { z ( j ) = f ( y ( j ) ) } Nj=1 ; 3 . Construct a matrix Y ∈ R ( n+1 ) ×N where the first component of each column is 1 and the rest are from the embeddings , i.e. , Yi , j = ( 1 , y ( j ) ) T , and a matrix Z ∈ Rm×N where the columns are the logit vectors , i.e. , Zi , j = z ( j ) i ; 4 . Solve for Ã ∈ Rm×n and b̃ ∈ Rm in ( b̃ , Ã ) Y = Z , i.e. , b̃1 ã1,1 ã1 , n ... . . . ... b̃m ãm,1 ãm , n 1 . . . 1 y ( 1 ) 1 . . . y ( N ) 1 ... . . . ... y ( 1 ) n . . . y ( N ) n = z ( 1 ) 1 . . . z ( N ) 1 ... . . . ... z ( 1 ) m . . . z ( N ) m ( 2 ) Proposition 2 . Assuming g maps inputs to uniformly distributed embeddings in an n-cube1 , the attack above uniquely determines the parameters of f . I.e. , we have Ã = A and b̃ = b . Proof . By construction of ( 2 ) , we have Ãy ( j ) + b̃ = f ( y ( j ) ) for j = 1 , . . . , N . The unique solution can be obtained multiplying Z by the right-inverse of Y . For uniformly random embeddings , this right-inverse exists because Y has full rank with probability 1 by Proposition 1 . While in theory N = n+1 distinct queries are sufficient to mount the attack , computing the inverse of Y based with finite-precision arithmetic can be numerically unstable . In practice , we gather a larger set of inputs and construct an over-determined system of equations . We then numerically compute a least squares solution to Equation ( 2 ) . 1A language model with vocabulary V and maximum sequence length L can only produce |V |L different embeddings . This is many more points than representable in the precision used , so not an issue in practice .

Summary: This paper proposes a range of algebraic model extraction attacks (different from the prevalent learning-based approaches) for transformer models trained for NLP tasks in a grey-box setting i.e., an existing, public, usually pretrained encoder, with a private classification layer. Through attacks on different sizes of models and a range of downstream tasks, they observe that only a portion of the embedding space forms a basis of the tuned classification layer’s input space, and using a grey-box method, this can be algebraically computed. The pretraining-finetuning experiments on different tasks also show the smallest number of dimensions needed for high-fidelity extraction, and also that the model extraction attacks effectiveness decreases with fine-tuning the larger models base layers---which is an insight that is very useful for a lot of interpretability/probing work.