Analyzing Privacy Loss in Updates of Natural Language Models

1 INTRODUCTION . Over the last few years , deep learning has made sufficient progress to be integrated into intelligent , user-facing systems , which means that machine learning models are now part of the regular software development lifecycle . As part of this move towards concrete products , models are regularly re-trained to improve performance when new ( and more ) data becomes available , to handle distributional shift as usage patterns change , and to respect user requests for removal of their data . In this work , we show that model updates1 reveal a surprising amount of information about changes in the training data , in part , caused by neural network ’ s tendency to memorize input data . As a consequence , we can infer fine-grained information about differences in the training data by comparing two trained models even when the change to the data is as small as 0.0001 % of the original dataset . This has severe implications for deploying machine learning models trained on user data , some of them counter-intuitive : for example , honoring a request to remove a user ’ s data from the training corpus can mean that their data becomes exposed by releasing an updated model trained without it . This effect also needs to be considered when using public snapshots of high-capacity models ( e.g . BERT ( Devlin et al. , 2019 ) ) that are then fine-tuned on smaller , private datasets . We study the privacy implications of language model updates , motivated by their frequent deployment on end-user systems ( as opposed to cloud services ) : for instance , smartphones are routinely shipped with ( simple ) language models to power predictive keyboards . The privacy issues caused by the memorizing behavior of language models have recently been studied by Carlini et al . ( 2018 ) , who showed that it is sometimes possible to extract out-of-distribution samples inserted into the training data of a model . In contrast , we focus on in-distribution data , but consider the case of having access to two versions of the model . A similar setting has recently been investigated by Salem et al . ( 2019a ) with a focus on fully-connected and convolutional architectures applied to image classification , whereas we focus on natural language . We first introduce our setting and methodology in Section 2 , defining the notion of a differential score of token sequences with respect to two models . This score reflects the changes in the probabilities of individual tokens in a sequence . We then show how beam search can find token sequences with high differential score and thus recover information about differences in the training data . Our experiments in Section 3 show that our method works in practice on a number of datasets and model architectures including recurrent neural networks and modern transformer architectures . Specifically , we consider a ) a synthetic worst-case scenario where the data used to train two model snapshots differs only in a canary phrase that was inserted multiple times ; b ) a more realistic scenario where we compare 1We use the term “ model update “ to refer to an update in the parameters of the model , caused for example by a training run on changed data . This is distinct from an update to the model architecture , which changes the number or use of parameters . a model trained on Reddit comments with one that was trained on the same data augmented with subject-specific conversations . We show that an adversary who can query two model snapshots for predictions can recover the canary phrase in the former scenario , and fragments of discourse from conversations in the latter . Moreover , in order to learn information about such model updates , the adversary does not require any information about the data used for training of the models nor knowledge of model parameters or its architecture . Finally , we discuss mitigations such as training with differential privacy in Section 4 . While differential privacy grants some level of protection against our attacks , it incurs a substantial decrease in accuracy and a high computational cost . 2 METHODOLOGY . 2.1 NOTATION . Let T be a finite set of tokens , T ∗ be the set of finite token sequences , and Dist ( T ) denote the set of probability distributions over tokens . A language model M is a function M : T ∗ → Dist ( T ) , where M ( t1 . . . ti−1 ) ( ti ) denotes the probability that the model assigns to token ti ∈ T after reading the sequence t1 . . . ti−1 ∈ T ∗ . We often write MD to make explicit that a multiset ( i.e. , a set that can contain multiple occurrences of each element ) D ⊆ T ∗ was used to train the language model . 2.2 ADVERSARY MODEL . We consider an adversary that has query access to two language models MD , MD′ that were trained on datasets D , D′ respectively ( in the following , we use M and M ′ as shorthand for MD and MD′ ) . The adversary can query the models with any sequence s ∈ T ∗ and observe the corresponding outputs MD ( s ) , MD′ ( s ) ∈ Dist ( T ) . The goal of the adversary is to infer information about the difference between the datasets D , D′ . This scenario corresponds to the case of language models deployed to client devices , for example in “ smart ” software keyboards or more advanced applications such as grammar correction . 2.3 DIFFERENTIAL RANK . Our goal is to identify the token sequences whose probability differs most between M and M ′ , as these are most likely to be related to the differences between D and D′ . To capture this notion formally , we define the differential score DS of token sequences , which is simply the sum of the differences of ( contextualized ) per-token probabilities . We also define a relative variant D̃S based on the relative change in probabilities , which we found to be more robust w.r.t . the “ noise ” introduced by different random initializations of the models M and M ′ . Definition 1 . Given two language models M , M ′ and a token sequence t1 . . . tn ∈ T ∗ , we define the differential score of a token as the increase in its probability and the relative differential score as the relative increase in its probability . We lift these concepts to token sequences by defining DSM ′ M ( t1 . . . tn ) = n∑ i=1 M ′ ( t1 . . . ti−1 ) ( ti ) −M ( t1 . . . ti−1 ) ( ti ) , D̃S M ′ M ( t1 . . . tn ) = n∑ i=1 M ′ ( t1 . . . ti−1 ) ( ti ) −M ( t1 . . . ti−1 ) ( ti ) M ( t1 . . . ti−1 ) ( ti ) . The differential score of a token sequence is best interpreted relative to that of other token sequences . This motivates ranking sequences according to their differential score . Definition 2 . We define the differential rank DR ( s ) of s ∈ T ∗ as the number of token sequences of length |s| with differential score higher than s. DR ( s ) = ∣∣∣ { s′ ∈ T |s| ∣∣∣DSM ′M ( s′ ) > DSM ′M ( s ) } ∣∣∣ The lower the rank of s , the more s is exposed by a model update . 2.4 APPROXIMATING DIFFERENTIAL RANK . Our goal is to identify the token sequences that are most exposed by a model update , i.e. , the sequences with the lowest differential rank ( highest differential score ) . Exact computation of the differential rank for sequences of length n requires exploring a search space of size |T |n . To overcome this exponential blow-up , we propose a heuristic based on beam search . At time step i , a beam search of width k maintains a set of k candidate sequences of length i. Beam search considers all possible k |T | single token extensions of these sequences , computes their differential scores and keeps the k highest-scoring sequences of length i+ 1 among them for the next step . Eventually , the search completes and returns a set S ⊆ Tn . We approximate the differential rank DR ( s ) of a sequence s by its rank among the sequences in the set S computed by beam search , i.e . ∣∣∣ { s′ ∈ S | DSM ′M ( s′ ) > DSM ′M ( s ) } ∣∣∣ . The beam width k governs a trade-off between computational cost and precision of the result . For a sufficiently large width , S = T |s| and the result is the true rank of s. For smaller beam widths , the result is a lower bound on DR ( s ) as the search may miss sequences with higher differential score than those in S. In experiments , we found that shrinking the beam width as the search progresses speeds the search considerably without compromising on the quality of results . Initially , we use a beam width |T | , which we half at each iteration ( i.e. , we consider |T | /2 candidate phrases of length two , |T | /4 sequences of length three , . . . ) . 3 EXPERIMENTAL RESULTS . In this section we report on experiments in which we evaluate privacy in language model updates using the methodology described in Section 2 . We begin by describing the experimental setup . 3.1 SETUP . For our experiments , we consider three datasets of different size and complexity , matched with standard baseline model architectures whose capacity we adapted to the data size . All of our models are implemented in TensorFlow . Note that the random seeds of the models are not fixed , so repeated training runs of a model on an unchanged dataset will yield ( slightly ) different results . We will release the source code as well as analysis tools used in our experimental evaluation at https : //double/blind . Concretely , we use the Penn Treebank ( Marcus et al. , 1993 ) ( PTB ) dataset as a representative of low-data scenarios , as the standard training dataset has only around 900 000 tokens and a vocabulary size of 10 000 . As corresponding model , we use a two-layer recurrent neural network using LSTM cells with 200-dimensional embeddings and hidden states and no additional regularization ( this corresponds to the small configuration of Zaremba et al . ( 2014 ) ) . Second , we use a dataset of Reddit comments with 20 million tokens overall , of which we split off 5 % as validation set . We use a vocabulary size of 10 000 . As corresponding model , we rely on a one-layer recurrent neural network using an LSTM cell with 512-dimensional hidden states and 160-dimensional embeddings , using dropout on inputs and outputs with a keep rate of 0.9 as regularizer . These parameters were chosen in line with a neural language model suitable for next-word recommendations on resource-bounded mobile devices . We additionally consider a model based on the Transformer architecture ( Vaswani et al. , 2017 ) ( more concretely , using the BERT ( Devlin et al. , 2019 ) codebase ) with four layers of six attention heads each with a hidden dimension of 192 . Finally , we use the Wikitext-103 dataset ( Merity et al. , 2017 ) with 103 million training tokens as a representative of a big data regime , using a vocabulary size of 20 000 . As model , we employ a two-layer RNN with 512-dimensional LSTM cells and token embedding size 512 and again dropout on inputs and outputs with a keep rate of 0.9 as regularizer . We combined this large dataset with this relatively low-capacity model ( at least according to the standards of the state of the art in language modeling ) to test if our analysis results still hold on datasets that clearly require more model capacity than is available .

Summary: This paper looks at privacy concerns regarding data for a specific model before and after a single update. It discusses the privacy concerns thoroughly and look at language modeling as a representative task. They find that there are plenty of cases namely when the composition of the sequences involve low frequency words, that a lot of information leak occurs.