Analyzing Privacy Loss in Updates of Natural Language Models

1 INTRODUCTION . Over the last few years , deep learning has made sufficient progress to be integrated into intelligent , user-facing systems , which means that machine learning models are now part of the regular software development lifecycle . As part of this move towards concrete products , models are regularly re-trained to improve performance when new ( and more ) data becomes available , to handle distributional shift as usage patterns change , and to respect user requests for removal of their data . In this work , we show that model updates1 reveal a surprising amount of information about changes in the training data , in part , caused by neural network ’ s tendency to memorize input data . As a consequence , we can infer fine-grained information about differences in the training data by comparing two trained models even when the change to the data is as small as 0.0001 % of the original dataset . This has severe implications for deploying machine learning models trained on user data , some of them counter-intuitive : for example , honoring a request to remove a user ’ s data from the training corpus can mean that their data becomes exposed by releasing an updated model trained without it . This effect also needs to be considered when using public snapshots of high-capacity models ( e.g . BERT ( Devlin et al. , 2019 ) ) that are then fine-tuned on smaller , private datasets . We study the privacy implications of language model updates , motivated by their frequent deployment on end-user systems ( as opposed to cloud services ) : for instance , smartphones are routinely shipped with ( simple ) language models to power predictive keyboards . The privacy issues caused by the memorizing behavior of language models have recently been studied by Carlini et al . ( 2018 ) , who showed that it is sometimes possible to extract out-of-distribution samples inserted into the training data of a model . In contrast , we focus on in-distribution data , but consider the case of having access to two versions of the model . A similar setting has recently been investigated by Salem et al . ( 2019a ) with a focus on fully-connected and convolutional architectures applied to image classification , whereas we focus on natural language . We first introduce our setting and methodology in Section 2 , defining the notion of a differential score of token sequences with respect to two models . This score reflects the changes in the probabilities of individual tokens in a sequence . We then show how beam search can find token sequences with high differential score and thus recover information about differences in the training data . Our experiments in Section 3 show that our method works in practice on a number of datasets and model architectures including recurrent neural networks and modern transformer architectures . Specifically , we consider a ) a synthetic worst-case scenario where the data used to train two model snapshots differs only in a canary phrase that was inserted multiple times ; b ) a more realistic scenario where we compare 1We use the term “ model update “ to refer to an update in the parameters of the model , caused for example by a training run on changed data . This is distinct from an update to the model architecture , which changes the number or use of parameters . a model trained on Reddit comments with one that was trained on the same data augmented with subject-specific conversations . We show that an adversary who can query two model snapshots for predictions can recover the canary phrase in the former scenario , and fragments of discourse from conversations in the latter . Moreover , in order to learn information about such model updates , the adversary does not require any information about the data used for training of the models nor knowledge of model parameters or its architecture . Finally , we discuss mitigations such as training with differential privacy in Section 4 . While differential privacy grants some level of protection against our attacks , it incurs a substantial decrease in accuracy and a high computational cost . 2 METHODOLOGY . 2.1 NOTATION . Let T be a finite set of tokens , T ∗ be the set of finite token sequences , and Dist ( T ) denote the set of probability distributions over tokens . A language model M is a function M : T ∗ → Dist ( T ) , where M ( t1 . . . ti−1 ) ( ti ) denotes the probability that the model assigns to token ti ∈ T after reading the sequence t1 . . . ti−1 ∈ T ∗ . We often write MD to make explicit that a multiset ( i.e. , a set that can contain multiple occurrences of each element ) D ⊆ T ∗ was used to train the language model . 2.2 ADVERSARY MODEL . We consider an adversary that has query access to two language models MD , MD′ that were trained on datasets D , D′ respectively ( in the following , we use M and M ′ as shorthand for MD and MD′ ) . The adversary can query the models with any sequence s ∈ T ∗ and observe the corresponding outputs MD ( s ) , MD′ ( s ) ∈ Dist ( T ) . The goal of the adversary is to infer information about the difference between the datasets D , D′ . This scenario corresponds to the case of language models deployed to client devices , for example in “ smart ” software keyboards or more advanced applications such as grammar correction . 2.3 DIFFERENTIAL RANK . Our goal is to identify the token sequences whose probability differs most between M and M ′ , as these are most likely to be related to the differences between D and D′ . To capture this notion formally , we define the differential score DS of token sequences , which is simply the sum of the differences of ( contextualized ) per-token probabilities . We also define a relative variant D̃S based on the relative change in probabilities , which we found to be more robust w.r.t . the “ noise ” introduced by different random initializations of the models M and M ′ . Definition 1 . Given two language models M , M ′ and a token sequence t1 . . . tn ∈ T ∗ , we define the differential score of a token as the increase in its probability and the relative differential score as the relative increase in its probability . We lift these concepts to token sequences by defining DSM ′ M ( t1 . . . tn ) = n∑ i=1 M ′ ( t1 . . . ti−1 ) ( ti ) −M ( t1 . . . ti−1 ) ( ti ) , D̃S M ′ M ( t1 . . . tn ) = n∑ i=1 M ′ ( t1 . . . ti−1 ) ( ti ) −M ( t1 . . . ti−1 ) ( ti ) M ( t1 . . . ti−1 ) ( ti ) . The differential score of a token sequence is best interpreted relative to that of other token sequences . This motivates ranking sequences according to their differential score . Definition 2 . We define the differential rank DR ( s ) of s ∈ T ∗ as the number of token sequences of length |s| with differential score higher than s. DR ( s ) = ∣∣∣ { s′ ∈ T |s| ∣∣∣DSM ′M ( s′ ) > DSM ′M ( s ) } ∣∣∣ The lower the rank of s , the more s is exposed by a model update . 2.4 APPROXIMATING DIFFERENTIAL RANK . Our goal is to identify the token sequences that are most exposed by a model update , i.e. , the sequences with the lowest differential rank ( highest differential score ) . Exact computation of the differential rank for sequences of length n requires exploring a search space of size |T |n . To overcome this exponential blow-up , we propose a heuristic based on beam search . At time step i , a beam search of width k maintains a set of k candidate sequences of length i. Beam search considers all possible k |T | single token extensions of these sequences , computes their differential scores and keeps the k highest-scoring sequences of length i+ 1 among them for the next step . Eventually , the search completes and returns a set S ⊆ Tn . We approximate the differential rank DR ( s ) of a sequence s by its rank among the sequences in the set S computed by beam search , i.e . ∣∣∣ { s′ ∈ S | DSM ′M ( s′ ) > DSM ′M ( s ) } ∣∣∣ . The beam width k governs a trade-off between computational cost and precision of the result . For a sufficiently large width , S = T |s| and the result is the true rank of s. For smaller beam widths , the result is a lower bound on DR ( s ) as the search may miss sequences with higher differential score than those in S. In experiments , we found that shrinking the beam width as the search progresses speeds the search considerably without compromising on the quality of results . Initially , we use a beam width |T | , which we half at each iteration ( i.e. , we consider |T | /2 candidate phrases of length two , |T | /4 sequences of length three , . . . ) . 3 EXPERIMENTAL RESULTS . In this section we report on experiments in which we evaluate privacy in language model updates using the methodology described in Section 2 . We begin by describing the experimental setup . 3.1 SETUP . For our experiments , we consider three datasets of different size and complexity , matched with standard baseline model architectures whose capacity we adapted to the data size . All of our models are implemented in TensorFlow . Note that the random seeds of the models are not fixed , so repeated training runs of a model on an unchanged dataset will yield ( slightly ) different results . We will release the source code as well as analysis tools used in our experimental evaluation at https : //double/blind . Concretely , we use the Penn Treebank ( Marcus et al. , 1993 ) ( PTB ) dataset as a representative of low-data scenarios , as the standard training dataset has only around 900 000 tokens and a vocabulary size of 10 000 . As corresponding model , we use a two-layer recurrent neural network using LSTM cells with 200-dimensional embeddings and hidden states and no additional regularization ( this corresponds to the small configuration of Zaremba et al . ( 2014 ) ) . Second , we use a dataset of Reddit comments with 20 million tokens overall , of which we split off 5 % as validation set . We use a vocabulary size of 10 000 . As corresponding model , we rely on a one-layer recurrent neural network using an LSTM cell with 512-dimensional hidden states and 160-dimensional embeddings , using dropout on inputs and outputs with a keep rate of 0.9 as regularizer . These parameters were chosen in line with a neural language model suitable for next-word recommendations on resource-bounded mobile devices . We additionally consider a model based on the Transformer architecture ( Vaswani et al. , 2017 ) ( more concretely , using the BERT ( Devlin et al. , 2019 ) codebase ) with four layers of six attention heads each with a hidden dimension of 192 . Finally , we use the Wikitext-103 dataset ( Merity et al. , 2017 ) with 103 million training tokens as a representative of a big data regime , using a vocabulary size of 20 000 . As model , we employ a two-layer RNN with 512-dimensional LSTM cells and token embedding size 512 and again dropout on inputs and outputs with a keep rate of 0.9 as regularizer . We combined this large dataset with this relatively low-capacity model ( at least according to the standards of the state of the art in language modeling ) to test if our analysis results still hold on datasets that clearly require more model capacity than is available .

Summary: This paper studies the privacy issue of widely used neural language models in the current literature. The authors consider the privacy implication phenomena of two model snapshots before and after an update. The updating setting considered in this paper is kind of interesting. However, the contribution of the current paper is not strong enough and there are many unclear experimental settings in the current paper.