Analyzing Privacy Loss in Updates of Natural Language Models
Authors: Shruti Tople
Marc Brockschmidt
Boris Köpf
Olga Ohrimenko
Santiago Zanella-Béguelin
Venue:  (2020)
Keywords: Language Modelling
Privacy

Abstract: To continuously improve quality and reflect changes in data, machine learning-based services have to regularly re-train and update their core models. In the setting of language models, we show that a comparative analysis of model snapshots before and after an update can reveal a surprising amount of detailed information about the changes in the data used for training before and after the update.
We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

Under review as a conference paper at ICLR 2020
ANALYZING PRIVACY Loss IN UPDATES OF NATURAL LANGUAGE MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
To continuously improve quality and reflect changes in data, machine learning- based services have regularly re-train and update their core models. In the setting of language models, we show that a comparative analysis of model snapshots before and after an update can reveal a surprising amount of detailed information about the changes in the data used for training before and after the update. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.
INTRODUCTION
Over the last few years, deep learning has made sufficient progress to be integrated into intelligent, user-facing systems, which means that machine learning models are now part of the regular software development lifecycle. As part of this move towards concrete products, models are regularly re- re-trained improve performance when (and more) data becomes available, handle distributional shift as usage patterns change, and to respect user requests for removal of their data.
In this work, show that model updates reveal surprising amount of information about changes in the training data, in part, caused by neural network's tendency to memorize input data. As consequence, we can infer fine-grained information about differences in the training data by comparing two trained models even when the change to the data is as small as 0.0001 % of the original dataset. This has severe implications for deploying machine learning models trained on user data, some of them counter-intuitive: for example, honoring a request to remove user's data from the training corpus can mean that their data becomes exposed by releasing an updated model trained without it. This effect also needs to be considered when using public snapshots of high-capacin models (e.g. BERT (Devlin et al. 2019)) that are then fine-tuned on smaller, private datasets.
We study the privacy implications of language model updates, motivated by their frequent deployment on end-user systems (as opposed cloud services): for instance, smartphones are routinely shipped with (simple) language models to power predictive keyboards. The privacy issues caused by the memorizing behavior of language models have recently been studied by Carlini et al. (2018), who showed that it is sometimes possible extract out-of-distribution samples inserted into the training data of a model. contrast, we focus on in-distribution data, but consider the case of having access two versions of the model. similar setting has recently been investigated by [allmm et al. (2019a) with a focus on fully-connected and convolutional architectures applied to image classification, whereas we focus on natural language.
defining the notion of differential score of token sequences with respect to two models. This score reflects the changes in the probabilities of individual tokens in a sequence. We then show how beam search can find token sequences with high differential score and thus recover information about differences in the training data. Our experiments in Section 3) show that our method works in practice on a number of datasets and model architectures including recurrent neural networks and modern transformer architectures. Specifically, we consider synthetic worst-case scenario where the data used to train two model snapshots differs only in a canary phrase that was inserted multiple times; b) a more realistic scenario where we compare
We first introduce our setting and methodology in Section 2
We use the term "model update" to refer to update in the parameters of the model, caused for example by training changed data. This is distinct from update the model architecture, which changes the number use of parameters.

---

Under review as a conference paper at ICLR 2020
model trained Reddit comments with that was trained on the same data augmented with subject-specifie conversations. We show that an adversary who can query two model snapshots for predictions recover the canary phrase in the former scenario, and fragments of discourse from conversations in the latter. Moreover, in order to learn information about such model updates, the adversary does not require any information about the data used for training of the models nor knowledge of model parameters or its architecture.
Finally, we discuss mitigations such training with differential privacy in Section While differ- ential privacy grants some level of protection against our attacks, it incurs a substantial decrease in accuracy and a high computational cost.
METHODOLOGY
2.1 NOTATION
Let be a finite set of tokens, T be the set finite token sequences, and (Ts denote the set of probability distributions over tokens. language model M is function M: T (T), where M(4...44-1) (4:) denotes the probability that the model assigns to token 44 E T after reading the sequence 41..44-1 T We often write MD to make explicit that multiset (i.e., set that can contain multiple occurrences of each element) D C T was used to train the language model.
2.2 ADVERSARY MODEL.
We consider an adversary that has query access to two language models Mp. Mp that were trained on datasets D, D' respectively (in the following, we use M and M' as shorthand for Mn and M))) The adversary can query the models with any sequence s T* and observe the corresponding outputs Mp(s), (s), pr (s) E (T). The goal of the adversary is to infer information about the difference between the datasets D, D'.
This scenario corresponds to the case of language models deployed to client devices, for example in "smart" software keyboards or more advanced applications such as grammar correction.
2.3 DIFFERENTIAL RANK
Our goal is to identify the token sequences whose probability differs most between M and M', as these are most likely to be related to the differences between and D'.
To capture this notion formally, define the differential score DS of token sequences, which is simply the sum of the differences of (contextualized) per-token probabilities. We also define relative variant DS based on the relative change in probabilities, which we found to be more robust w.r.t. the "noise" introduced by different random initializations of the models M and M'.
Definition 1. Given two language models M, MM and a token sequence t,.,,,, E T we define the differential score of a token the increase in probability and the relative differential score as the relative increase in its probability, We lift these concepts to token sequences by defining
DS M (f1... (n)
Σ M1 1.14...4444 ...4)())) - M(4 M(4...44-1(CC (441
M' DS (lI... (n) M' ..fi...te t. (t)....4
M(h...4,
The differential score of token sequence best interpreted relative to that of other token sequences. This motivates ranking sequences according to their differential score. Definition 2. We define the differential rank DR(s) of T as the number of token sequences of length [s] with differential score higher than s. T
DR(s) = [
DSM (s') DSM (
The lower the rank of 8, the more is exposed by model update.
2

---

Under review as a conference paper at ICLR 2020
2.4 APPROXIMATING DIFFERENTIAL RANK
Our goal is to identify the token sequences that most exposed by model update, i.e., the sequences with the lowest differential rank (highest differential score). Exact computation of the differential rank for sequences of length n requires exploring a search space of size [2']. To overcome this exponential blow-up, we propose a heuristic based on beam search.
At time step i, a beam search of width maintains a set of candidate sequences of length i. Beam search considers all possible k (T' single token extensions of these sequences, computes their differential scores and keeps the highest-scoring sequences length i + among them for the next step. Eventually, the search completes and returns set S C T
We approximate the differential rank (R(s) of sequence s by its rank among the sequences in the set S computed by beam search, i.c. {s' S DS (s') > DS (s)} The beam width & governs trade-off between computational cost and precision of the result. For a sufficiently large width, S = Tlsl and the result is the true rank of For smaller beam widths, the result is a lower bound on DR(s) as the search may miss sequences with higher differential score than those in S.
In experiments, we found that shrinking the beam width as the search progresses speeds the search considerably without compromising on the quality of results. Initially, we use beam width [T] which we half at each iteration (i.e., we consider [T'] /2 candidate phrases of length two, [T' sequences of length three....
3 EXPERIMENTAL RESULTS
In this section we report on experiments in which we evaluate privacy in language model updates using the methodology described in /Section 2] We begin by describing the experimental setup.
3.1
SETUP
For our experiments, we consider three datasets of different size and complexity, matched with standard baseline model architectures whose capacity we adapted to the data size. All of our models are implemented in TensorFlow. Note that the random seeds of the models are not fixed, so repeated training runs of a model on an unchanged dataset will yield (slightly) different results. We will release the source code well as analysis tools used in experimental evaluation at ht /wwp:///d////////e// /blindd
the Penn Treebank (Marcus et al. 1993) (PTB) dataset representative of low-data scenarios, as the standard training dataset has only around 900 000 tokens and a vocabulary size of 100. As corresponding model, cells with 200-dimensional embeddings and hidden states and no additional regularization (this corresponds to the small configuration of Zaremba al. (2014)).
Concretely,
two-layer recurrent neural network using LSTM
Second, we use a dataset of Reddit comments with 20 million tokens overall, of which we split off 5% as validation set. We use a vocabulary size of 10 As corresponding model, rely on a one-layer recurrent neural network using an LSTM cell with 512-dimensional hidden states and 160-dimensional embeddings, using dropout on inputs and outputs with keep rate of 0.9 as regularizer. These parameters were chosen in line with a neural language model suitable for nord-word recommendations on resource-bounded mobile devices. We additionally consider model based on the Transformer architecture (Vaswani et al. 2017) (more concretely, using the BERT (Devlin et al. 2019) codebase) with four layers of six attention heads each with a hidden dimension 192.
Finally, we use the Wikitext- 103 dataset (Merity et al. 2017) with 103 million training tokens as a representative
big data regime, using a vocabulary size of 000. As model, we employ a two-layer RNN with 512-dimensione LSTM cells and token embedding size 512 and again dropout on inputs and outputs with a keep rate of 0.9 as regularizer. We combined this large dataset with this relatively low-capacity model (at least according to the standards of the state of the art in language modeling) to test if our analysis results still hold on datasets that clearly require more model capacity than is available.
3

---

Under review as a conference paper at ICLR 2020
Table 1: Differential score (DS) for PTB, Reddit and Wikitext-103 dataset across different token and phrase frequencies. White cell background means that the differential rank DR (as approximated by our beam search) of the phrase is O, grey cell background mcans that DR is > ~0000.
Dutaset
Penn Treebunk
Reddit
Wikitext-103
Model Type (Perplexity)
RNN (120.90)
RNN (79.63)
Transformer (69.29)
RNN (48. 59)
Canary Token Freq.
1:18K
1:3.6K
1:1.8K
1:1M
1:100K
1:10K
1:1M
1:100K
1:10K
1:1M
1:200K
All Low
3.40
3.94
3.97
2.83
3.91
3.22
3.97
3,99
3.81
Low to High
3,52
3.85
3.97
0.42
3.66
3.98
0.25
3.66
3.97
0.07
3.21
Mixed
3.02
3.61
3.90
0.23
3.04
3.92
0.39
3.25
3.96
3.02
High Low
1.96
2.83
3.46
0.74
1.59
2.89
0.18
1.87
3.10
0.08
1.22
3.2 PRIVACY ANALYSIS OF MODEL. UPDATES USING SYNTHETIC CANARIES
We first study the privacy implications of model updates in controlled experiments with synthetic data. To this end, we create number of canary phrases that serve as a proxy for private data (they are grammatically correct and do not appear in the original dataset) and that exhibit a variety of different token frequency characteristics.
Specifically, we fix the length of the canary phrase 5, choose a valid phrase structure (e.g. Subject, Verb, Adverb, Compound Object), and instantiate each placeholder with token that has the desired frequency D. We create canaries in which frequencies of tokens are all low (all tokens are from the least frequent quintile of words), mixed (one token from each quintile), increasing from low to high, and decreasing from high to low. As the vocabularies differ between the different datasets, the canaries are dataset-dependent. For example, the mixed phrase across all the datasets is "NASA used deadly carbon devices," " and the all low phrase for PTB is "nurses nervously trusted incompetent graduates."
For a given dataset D and a canary phrase s & D, we construct dataset D+kve by inserting copies of s into D. We use the differential score DS and the differential rank DR of canary phrase 8 to answer a number of research questions on our model/dataset combinations. Note that analyzing removal of specific phrases from the dataset simply requires swapping the role of D and D+k+a-
RQ1: What is the effect of the number of canary phrase insertions? We consider different numbers of insertions, adapted to the number of tokens the training corpus
For PTB, we consider & E {10, 50, 100) canary insertions (corresponding to canary token in 18K training tokens, in 3.6K, and in 1.8K).
For the Reddit dataset, we use E (5, 50, 500} (corresponding to 1 in 1M, 1 in 100K, in 10K).
For the Wikitext-103 data, we use k E (20, 100} (corresponding to 1 in 1M, 1 in 200K).
Table summarizes all of our experiments. As expected, the differential score of canaries grows monotonously with the number of insertions, for all kinds of canaries and models. More surprisingly, in cells with white background, the canary phrase has the maximum differential score among all token sequences found by our beam search, i.e. it ranks first. This means that the canary phrase can easily be extracted without any prior knowledge about or the context in which appears (this is in contrast to the single-model results of Carlini et al. (2018), who assumed a known prefix). The signal for extraction is strong even when the inserted canaries account for only 0.0001 % the tokens in the dataset. This becomes visible in the first row of Table I where differential scores approaches 4, which is close to the upper bound of 5 (for 5-token canaries).
RQ2: What is the effect of token frequency in training data? Comparing the columns of Table 1 can be used to answer this question:
Phrases with all low- fow-freqeeccy tokens consistently show the highest differential score. Such phrases rank first even when the model is updated with the smallest number of canary
Thh non-aligned insertion frequencies are due to legacy reasons in experiments. For the final version we will re-run all experiments with aligned frequencies.
4

---

Under review as a conference paper at ICLR 2020
-LL-5
HL-50 -A. HL-500
- LL.6
LL-50
LL-500
0.8
Disterenial s00t
0.6
0.4
0.2
0
0
Prefix Length
Figure 1: Differential score of tokens in canaries given a prefix for the Reddit dataset. Dashed (solid) lines represent experiments with & insertions of canaries with all-low (resp. high-to-low) token frequencies, indicated by LL-k (resp. HL.-k).
insertions, as seen in the first row of Table | This means that phrases composed of rare words are more likely to be exposed in model updates than other phrases.
Canary phrases that start with low- low-frequeet www.reeeenny token, followed by tokens of increasing or mixed frequency, have higher rank than canaries with all low-frequency tokens, but become exposed for moderate number of insertions into the dataset, see rows 2 and 3 of Table
Canaries composed of tokens with descending frequency the least susceptible to analysis and tolerate higher number of insertions before they become exposed. This is expected, beam search is biased towards finding high-scoring prefixes.
RQ3: What is the effeet of knowledge about the canary context? We evaluate the differential score suffixes of our canary phrases assuming knowledge of a prefix. This gives insight into the extent to which attacker with background knowledge a dataset D, canary phrase = ...,t,, & D and the augmented dataset D For i 1,..., n we take the prefix 4i- I of the canary phrase and compute the differential score of the token &; conditional on having read the prefix, i.e. M1 (44) M (41 (f:). The relationship between and indicates how much knowledge about required to expose the remainder of the canary phrase.
extract secret. To this end we consider
Figure depicts the result of this analysis for canaries with high-to-low and all-low token frequencies on the Reddit dataset. Our results show that, while the differential score of the first token without context close to zero, the score of subsequent tokens quickly grows for all-low canaries, even with low number of canary insertions. In contrast, more context is required before observing change in the differential score of high-to-low canaries, as the model is less influenced by the small number of additional occurrences of frequent tokens. This suggests that, even in cases where fail extract the canary without additional knowledge (see RQ1 above), an adversary can still use the differential rank to complete a partially known phrase, confirm that canary phrase was used to update the dataset.
RQ3A: What is the effect of inserting canaries together with additional data? We consider the setting where the model is re-trained data consisting of the canary phrase and in-distribution data D Dextra along with the original dataset Derig- Concretely, we first split our Reddit dataset into D Dorig and D extre (such that the latter is 20%, 50%, 100% of the size of D Dorig). Then, we trained a model M on reduced D Dorig and a second model M' on D) = D Dorig U Deatro and the inserted canaries. The results of this experiment are displayed in Table 2 where we can see that DS does not change significantly. Note that the 0% column is identical to the result from Table In conclusion, canaries be extracted from the trained model even when they are contained in substantial larger dataset extension.
fresh
RQ3B: What is the effect of updating the model using a continued training approach? We also consider the setting of continued training, in which existing model is trained on new dataset
5

---

Under review as a conference paper at ICLR 2020
Table 2: D of the mixed frequence canary phrase for the Reddit (RNN) model using different update techniques. We use T(R, D Dorig) - M to denote that model M was obtained by training on data D Drig starting from model R. Here, are (fresh) random initial parameters, D Dextra is an additional dataset from the same distribution as Dorig, and C is the set of canaries. Re-training refers to training on the data from scratch with new random initialization, whereas continued training fine-tunes existing model on fresh data. A white cell background means that the differential rank DR (as approximated by our beam search) of the phrase O, grey cell background means that DR is >000..
Re-training
Continued Training T(M, Destro C) + M
Continued Training
Training Method
T(R, D Dorig U Destra UC) + M
T(M, Destra UC) + M T(M, D' extro) + M
D Dorig
0%
20%
50%
100%
20%
50%
100%
100%
1M
0.23
0.224
0.223
0.229
0.52
0.34
0.46
0.01
100K
3.04
3.032
3.031
3.038
3.56
3.25
3.27
0.26
(e.g.. pre-trained on generic corpus and fine-tuned on a more specific dataset). For this, we train model M on a dataset D Dorig to convergence, and then continue training using the union of D Dextra and the canaries. We use the same dataset splits as in RQ3A to create the Dorig and Dextra datasets. The results of this experiment are shown in the middle column of T We observe that in all cases the differential score is higher for continued training than re- re-training. As expected, the differential score of the canary phrase decreases as additional extra data is used for fine-tuning. This shows that the risk of leaking canary phrases increases when a model updated using the continued training approach.
Finally, we also consider a possible mitigation strategy, in which we perform continued training in two stages. For this, we split the dataset into three equal parts D orig- D Dextra and Dexttr We proceed as in the continued training setting above, but add a final step in which we train on another dataset after training the canaries. This resembles setting where an attacker does not have access two consecutive snapshots. The results are on the right column of Table [ showing that the differential score of the canary phrase drops substantially after the second training stage. Thus, two or multi-stage continued training might mitigate leakage of private data.
3.3 PRIVACY ANALYSIS OF MODEL UPDATES USING SUBJECT-SPECIFG CONVERSATIONS
We now study the privacy implications of model updates in real-world scenarios. As a representative scenario, compare models trained on the Reddit dataset against models trained the same data augmented with messages from one of two newsgroups from the 20 Newsgroups dataset (Lang 1995|: a) rec. sport. hockey, containing around 184K tokens, approximately 1% of the original training data; and b) talk. wallk.ooliiccs .mideast, containing around 430K tokens, approximately 2% of the original training data. For both newsgroups, the number of tokens insert is significantly larger than in the synthetic experiments of Seectonn 3.2 However, we insert full conversations, many of which are of general nature and off the topic of the newsgroup.
RQ4: Do the results on synthetic data extend to real-world data? As opposed to synthetic experiments using canaries, when using real-world data there is no individual token sequence whose rank would serve as a clear indicator of a successful attack. We hence resort to a qualitative evaluation where we inspect the highest-scoring sequences found by beam search. Since the sequences returned by vanilla beam search typically share a common prefix, we alternatively run a group beam search to get a more representative sample: we split the initial [T] one-token sequences into N groups according to their differential score, and run parallel beam searches extending each of the groups independently.
Table 3 displays the result of our evaluation on Reddit augmented with rec .ww.t.hockee.......ooo i.e., the sccheet--ccoing sequences of length 4 in each group of DS-based group beam search with N = (top row) and N 1 (bottom row). The exposed sentences are on-topic w.n.t. the newsgroup added, which suggests that the phrases with highest relative differential score are specific to the

---

Professor of History he Under review as a conference paper at ICLR 2020
newsgroup used and that, indeed, data used for the update is exposed. We obtain results of comparable relevance using the talk. politics. mideast newsgroup, which we report in Table
Table 3: Top ranked phrases in vanilla and group beam search for RNN and Transformer models trained with rec. sport hockey. For the layperson: NHL stands for National Hockey League; Los Angeles Kings, Minnesota North Stars, and Toronto Maple Leaf are NHL teams; Norm Green was the owner of the North Stars; an ice hockey game consists of three periods with overtime to break ties. Capitalization added for emphasis.
RNN
Transformer
Phrase
D8
Phrase
DS
Angeles Kings prize pools
56.42
Minnesota North Stars playoff
96.81
National Bockey League champions
53.68
Arsenal Maple Leaf fans
71.88
Group
Norm 's advocate is
39.66
Overtime no sccring chance
54.77
Intention you lecture me
21.59
Period power play
47,85
Covering yourself basically means
21.41
Penalty shot playoff results
42.63
National Bockey League champions
53.68
Minnesota North Stars playoft
96.81
National Blokkey League hockey
52.82
Minnesota North Stars
96.78
Vanilla National Bockey League cup
52.44
Minnesota North Stars hockey
96.76
National Bockey League USA
52.34
Minnesota North Stars games
96.75
National Blokkyy League playofr
52.18
Minnesota North Stars
96.72
Table 4: Top ranked phrases in vanilla and group beam search for RNN and Transformer models trained with talk. pol it cs mi deast. Center for Policy Research is prolific newsgroup poster; Professor of History is common title appearing in message signatures; many of the posts around the time the 20 Newsgroups dataset (Lang 1995) was collected discuss tensions between Turkey and Armenia.
RNN
Transforme
Phrase
DS
Phrase
DS
Turkey aearched first aid
31.32
Center for Policy Research
200.27
Doll flies lay scattezed Arab governments invaded Turkey
22.79
Escaped
course ...
95.18
Group
20.20
Hol Bolocaust *UNKS
88.20
Lawsuit offers crime rates Sanity boosters health care
18.35
Troops surroundad village after
79.35
11.17
Turkey searched neither Arab
37.69
Turkey aearched firat aid
31.32
Center for Policy Research
200.27
Turkey searched a plane Turkey searched supreme seviet
24.63
Professor of Histooy I
120.98
Vanilla
21.59
Professor of Histooy Bistory
120.93
Turkey searched arab Tu-key
20.34
Professor of History
120.92
Turkey searched national plane
19.44
120.92
RQ4A: What is the effect re-training with additional public data? Similarly to RQ3A in
consider partitions of the Reddit dataset into datasets D Poiig and D Dentra of different relative sizes. For each partition, we compare a model M trained on Dorig to a model M' trained on D Dextro U N, where N are all messages from newsgroup from the 20 Newsgroups dataset. We sample a few representative phrases from group beam searches on all pairs of models and compare their relative differential scores.
Section 3.2]
In all cases, a group beam search can recover subject-specific phrases from newsgroups discussions. Some of the phrases resemble canaries, in that they occur multiple times literally the datasets (e.g. Center for Policy Research), while others never occur literally but digest recurrent discussions (e-g. Partition of northern Israel), Table 5) shows the relative differential score of these phrases for different partitions. As observed for canaries, scores vary little when additional data is used during re-training.
What is the effect of continued training? Using the same dataset splits as in RQ4A and similarly to RQ3B in Section 3.2 we compare a model M trained from scratch on D Drig to model M obtained by continuing training M on D Dextro U N. The results shown in Table 5 The last two rows contain phrases found by group beam search on M and a model M' obtained from M by continued training, but that have too low score to be found when M re-trained
RQ4B:
7

---

Under review as a conference paper at ICLR 2020
from scratch instead. The converse, i.e. phrases that have low score when continuing training and high score when re-training, seems to occur rarely and less consistently (e.g. Saudi troops surrounded vi village). For canary- phrases, the results are in line with those in TTale 2] with scores decreasing as more data is used during the fine-tuning stage. For other phrases, the results are not clear-cut. While fine-tuning model exclusively on private data yields scores that are significantly higher than when re-training a model from scratch, this effect vanishes as more additional data is used; in some cases continued training yields scores lower than when re-training a model on the same data.
Table 5: Relative differential score of phrases found by beam search when re-training from scratch and continuing training from a previous model. The results are for RNN models trained on partitions of the Reddit dataset with N = talk. polittttttt iccs...... midsts.. Cells for which continued training yields a higher score than re-training appear in bold font. Capitalization added for emphasis.
Re Re-training
Consinued Training T(M, D.ver UN) AM'"
Phrase
T(R. Danp Dantin
M
10%
25%
100%
10%
25%
100%
Center for Policy Research Troops surrourded village after
99.77
101.38
97.11
98.65
91.53
276.98
198.69
150.56
122.25
117.54
44.50
44.50
44.50
44.41
44.54
173.95
47.38
19.48
7.81
35.56
Partition of northern Israel
27.61
38.76
12.47
22.93
West Bank peace talks spiritua. and political leaders Saudi troops surrounsed village Arab governments invacad Turkey little restatance was offere Butter zone almed at protecting Capital letters racial discrininat
25.68
25.64
25.69
25.71
25.75
71.54
24.38
28.60
16.91
4.62
25.23
25.98
17.04
24.21
23.47
126.92
14,91
10.00
3.64
11.05
24.31
24.31
24:31
24.31
24.30
5.05
44.58
4.29
7.29
63,84
22.59
24.01
7,08
18.12
11.90
22.24
22.09
25.12
22.34
25.59
215.16
25.02
2,00
3.30
5.64
4,00
4.47
5.30
5.25
5.69
$7.29
69.76
18.92
14.50
22.25
11.22
4 MITIGATION VIA DIFFERENTIAL PRIVACY
In this section we explore how differential privacy can be used to mitigate the information leakage induced by a model update. Differential privacy (DP) (Dwork & Roth 2014) provides strong guarantees on the amount of information leaked by a released output. Given a computation over records it guarantees limits on the the effect that any input record can have on the output. Formally, F is (c, difffeerrett -private computation if for any datasets D and D' that differ in one record and for any subset O possible outputs of F we have
Pr(F(D) E 0) S exple) Pr(F(D') E 0)++5. 5
At a high level, differential privacy can be enforced in gradient-based optimization computa- tions (Abadi et al.] 2016 [Song et al. 2013 Bassily et al. 2014) by clipping the gradient of every record in a batch according to some bound L, then adding noise proportional to L to the sum of the clipped gradients, averaging over the batch size and using this noisy average gradient update during backpropagation.
Differential privacy is natural candidate for defending against membership- like input data. The exact application of differential privacy for protecting the information in the model update depends on what one wishes protect w.r.t. the new data: individual sentences in the new data or all information present in the update. For the former, sequence-level privacy can suffice while for the latter group DP can serve as mitigation technique where the size of the group is proportional to the number of sequences in the update. Recall that an e-DP algorithm F' is ke differentially private for groups of size k: (Dwork Roth 2014))
like inferences about the
Does DP protect against phrase extraction based on differential ranks? We evaluate the extent to which DP mitigates attacks considered in this paper by training models on the Penn Treebank dataset with canaries with sequence-level differential privacy. We train DP models using the TensorFlow Privacy library (Andrew et al. 2019) for two sets of (e, 6) parameters (5, 1 x 10 10-5) and (111, x 10 for two datasets: PTB and FTB with 50 insertions of the all-low-frequency canary. We rely on (Andrew et al. 2019) to train models with differentially private stochastic gradient descent using Gaussian noise mechanism and to compute the overall privacy loss of the training phase.
RQS:
As expected, the performance of models trained with DP degrades, in our case from R=23% accuracy in predicting the next token on the validation dataset to 11.89% and 13.34% for values of 5 and
8

---

Under review as a conference paper at ICLR 2020
111, respectively. While the beam search with the parameters used in Section 3.2 does not return the canary phrase for the DP-trained models anymore, we note that the models have degraded so far that they are essentially only predicting the most common words from each class ("is" when a verb is required,.... ...) ) and thus, the result is unsurprising. We note that the guarantees of sequence-level DP formally do not apply for the case where canary phrases are inserted multiple sequences, and that values for our models are high. However, the e-analysis is an upper bound and similar observations about the effectiveness of training with DP with high e were reported by Carlini et al. (2018).
We further investigate the effect of DP training on the differential rank of a canary phrase that was inserted 50 times. Instead of using our beam search method to approximate the differential rank, we fully explore the space of subsequences of length two, and find that the DR for the two-token prefix of our canary phrase dropped from 0 to 458 399 and 849 685 for the models with respectively. In addition, we compare the differential score of the whole phrase and observe that it drops from 3.94 for the original model to 4.5 x 10-4 and 2.1 10-3 for models with c = 5 and 111 respectively.
and 6 = 111
Though our experiment results validate that DP can mitigate the particular attack method considered in this paper for canary phrases, the model degradation is significant. In addition, the computational overhead of per-sequence gradient clipping is substantial, making it unsuitable for training high- capacity neural language models on large datasets.
5 RELATED WORK
In recent years several works have identified how machine learning models leak information about private training data. Membership attacks introduced by (Shokri et al. (2017) show that can identify whether a record belongs to the training dataset of a classification model given black-box access to the model and shadow models trained data from a similar distribution. (Salem al.] (20195) demonstrate that similar attacks are effective under weaker adversary models. Attribute inference attacks (Fredrikson et al. 2015). which leak the value of sensitive attributes of training records, have been shown successful for regression and classification models. In the distributed learning setting, HHitj et al. (2017) and Melis et al. (0018) demonstrate that individual gradient updates to a model can reveal features specific to one private dataset.
Carlini et al. (2018) is closest our work, as also considers information leakage of language models. The authors assess the risk of (unintended) memorization of rare sequences in the training data by introducing an exposure metric. They show that exposure values can be used to retrieve canaries inserted into training data from character-level language model. The key differences to our approach are that 1) we consider an adversary that has access to two snapshots of model, and 2) our canaries are grammatically correct sentences (i.e., follow the distribution of the data) whereas Carlini et al. (2018) add a random sequence of numbers in fixed context (e.g.. "The random number ...") into dataset of financial news articles, where such phrases are rare. We instead consider the scenario of extracting canaries withoul any context, even if the canary token frequency in the training dataset is low one in a million, and for canary phrases that more similar the training data.
Song & Shmatikov (2018) also study sequence-to-seqeene can check if their data has been used for training. In their setting, an auditor needs an auxiliary dataset to train shadow models with the same algorithm as the target model and queries the target model for predictions on a sample of the user's data. The auxiliary dataset does not need to be drawn from the same distribution as the original training data (unlike Shokri et al. (2017)) and the auditor only observes list of several top-ranked tokens. In contrast, our approach requires no auxiliary dataset, but assumes access to the probability distributions over all tokens from two different model snapshots. From this, we are able to recover full sequences from the differences in training data rather than binary information about data presence. Like them, we find that sequences with infrequent tokens provide a stronger signal to the adversary/auditor.
language models and show how a user
Salem et al. (2019a) consider reconstruction of training data that was used to update model. While their goal is similar to ours, their adversarial model and setup differ: 1) similar to Song & Shmatikov auxiliary data drawn from
(2018); (Shokri et al. (2017) their attacker uses shadow models trained the same distribution as the target training dataset, while in our setting the attacker has no prior knowledge of this distribution and does not need auxiliary data; 2) the updated model is obtained by
9

---

Under review as a conference paper at ICLR 2020
fine-tuning the target model with additional data rather than re-training it from scratch on the changed dataset; 3) the focus is on classification models and not on (generative) language models.
Information leakage from updates has also been considered in the setting of searchable eneryption. An attacker who has control over data in an update to an encrypted database can learn information about the content of the database well previous encrypted searches it (Cash al. (015). Pan-privacy (Dwork et al.) 2010), on the other hand, studies the problem of maintaining differential privacy when an attacker observes snapshots of the internal state of differentially-privat algorithm between data updates.
In terms of defenses, McMahan al. (2018) study how train LSTM models with differential privacy guarantees at user-level. They investigate utility and privacy trade-offs of the trained models depending on range of parameters (c.g.. clipping bound and batch size). Carlini al. (0018) show that differential privacy protects against leakage of canaries in character-level models, while Song Shmatikov (2018) show that an audit described above fails when training language models with user-level differential privacy using the techniques of McMahan et al. (2018). Ginart al. (2019) define deletion of a training data point from model as an stochastic operation returning the same distribution as re-training from scratch without that point, and develop deletion algorithms for k-means clustering with low amortized cost. Publishing snapshots of a model before and after deletion matches our adversarial model and our results apply.
CONCLUSION
As far as know, this article presents the first systematic study of the privacy implications of releasing snapshots of a language model trained on overlapping data. We believe this is realistic threat that needs to be considered in the lifecycle of machine learning applications. We aim to encourage the research community to work towards quantifying and reducing the exposure caused by model updates, and hope make practitioners aware of the privacy implications of deploying high-capacity language models as well as their updates.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In 23rd ACM SIGSAC Conference on Computer and Communications Security, CCS 2016, pp. 308-318, ACM, 2016.
Galen Andrew, Steve Chien, and Nicolas Papernot. Ler:sorflcw/privacy 2019. [Online; accessed 09-Sep-2019).
TensorFlow Privacy. https://github.com
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th IEEE Anmual Symposium Foundations of Computer IEEE Computer Society, 2014.
Science, FOCS 2014, pp. 464-473.
Nicholas Carlini, Chang Liu, Jernjj Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. CoRR, abs/1802.08232, 2018. URL
org/ awwaaass//1// 08232
David Cash, Paul Grubbs, Jason Perry, and Thomas Ristenpart. Leakage-abuse attacks against searchable encryption. 22nd ACM SIGSAC Conference on Computer and Communicutions Security, CCS 2015, pp. 668 188-679.44 ACM, 2015.
Jacob Devlin, Ming-Wiieeee Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-trainne tre-training of deep bidirectional transformers for language understanding. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI- HLT 2019, volume 1, pp. 380-385. Association for Computational Linguistics, 2019.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4)2211 (3-4):211-407, 2014.
Cynthia Dwork, Moni Naor, Toniann Pitassi, Guy N. Rothblum, and Sergey Yekhanin. Pan-private streaming algorithms. In Innovations in Computer Science, ICS 2010, pp. 66-80. Tsinghua University Press, 2010.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015, pp. 1322-1333, ACM, 2015.
Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making AI forget you: Data org/abs/
deletion in machine learning. CoRR, abs/1907. 05012, 2019.
Briland Hitaj. Giuseppe Ateniese, and Fernando Perez-Cruz. Deep models under the GAN: Informa- tion leakage from collaborative deep learning. In 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, pp. 603-618. ACM, 2017.
Ken Lang. NewsWeeder: Learning to filter netnews. In 12th International Machine Learning Conference on Machine Learning, ICML 1995, pp. 331-339. Morgan Kaufmann, 1995.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net, 2018.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Inference attacks against collaborative learning, CoRR, hts/18005044043 2018. hRR/http://axxv.oo///// 1805. 04049
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.
11

---

Under review as a conference paper at ICLR 2020
Ahmed Salem, Apratim Bhattacharyya, Michael Backes, Mario Fritz, and Yang Zhang. Updates-leak: Data set inference and reconstruction attacks in online learning. CoRR, abs/1904.01067, 2019a.
hRtp::://arxxx.ooo org. aataaaass/999/ 01067)
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019. The Internet Society, 2019b.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 38th IEEE Symposium on Security and Privacy, S&P 2017, pp. 3-... IEEE Computer Society, 2017.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. CoRR, 000/1811.0051.. 2018. URL /ttp://t///aaxa.orro/////////// attp////s////1 00513
S. Song, K. Chaudhuri, and A. D. Sarwate. Stochastic gradient descent with differentially private updates. In 1st IEEE Global Conference on Signal and Information Processing, GlobalSIP 2013, pp. 245-2452255 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, NIPS 2017, pp. 5998-59998 2017.
Wojeiech Zaremba, Iya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. CoRR, abs/ 1309,2299,,, 2014. WRLhhtp::///
2329
12


Reviews:
[{"review_id": "SyejLWDpKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the privacy issue of widely used neural language models in the current literature. The authors consider the privacy implication phenomena of two model snapshots before and after an update. The updating setting considered in this paper is kind of interesting. However, the contribution of the current paper is not strong enough and there are many unclear experimental settings in the current paper.\n\nAccording to the current paper, the privacy implication seems to be defined in terms of general sequences in training datasets. If this is the case, I don’t think such privacy implication is meaningful because our language models should memorize some general information to achieve their tasks. \n\nThere are some unclear settings in the experiments:\n1.In the experiments, why there are only 20000 vocabulary size for Wikitext-103 datasets?\n2.It is unclear how to construct canary phrases. \n3.After constructing the new dataset, the model is retrained or trained in the online way?\n4.Since the results for Wikitext-103 is not finished, the authors should remove the results on this dataset.\n5.What is the perplexity of the trained models?\n6.How to choose initial sequence in real data experiments?\n7.When you applying DP mechanism, how did you define the neighboring datasets, and how did you implement it (what is the clipping level, how did you calculate privacy loss for language models)?\n8.$\\epsilon=111$ seems that the model will provide no privacy guarantee according to the definition of differential privacy?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gshz42KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper looks at privacy concerns regarding data for a specific model before and after a single update. It discusses the privacy concerns thoroughly and look at language modeling as a representative task. They find that there are plenty of cases namely when the composition of the sequences involve low frequency words, that a lot of information leak occurs.\n\nPositives: The ideas and style of research is nice. This is an important problem and I think this paper does a good job investigating this in the context of language modeling. I do hope the community (and I think the community is) moving towards being aware of these sorts of privacy issues.\n\nConcerns: I don't know how generalizable these results would be on really well-trained language models (rnn, convolution-based, or transformer-based). The related work section doesn't seem particularly well put together, so its difficult to place the work in appropriate context and gauge its impact.\n\nOther Thoughts: I'd like more thorough error analysis looking at exactly what kinds of strings/more nuanced properties of sequences that get a high differential score.\n\nOverall I think this work is interesting and I would encourage the authors to try and add as much quantitative evaluation as possible, but also try and include qualitative information regarding specific sequences after prodding the models. Those could go a long way in strengthening the paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJg-Ic3hFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an empirical evaluation of the privacy implications of releasing updated versions of language models. The authors show how access to two sequential snapshots of a trained language model can reveal highly specific information about the content of the data used to update the model, even when that data is in-distribution.\n\nThe paper contains easy to understand, concrete experiments and results, but seems altogether a little underdeveloped. The methodology is sound, but the synthetic experiments around which much of the paper is based may not be sufficiently novel and give little indication of broader implications. It would have been more convincing if these results replicated with two splits of the same dataset, rather than identical datasets with one augmented by canary tokens. \n \nThe qualitative evaluation of subject-specific updates is also not sufficiently informative. It would have been useful to define a specific attack and see under what circumstances such an attack would succeed. In the current results, I am not convinced that any of the phrases in Table 3 represent a privacy violation. \n\nThe differential privacy experiment seems to be missing many details: what dataset was this trained on? Are the accuracy values for the training set or a separate testing set? Other works have shown that it is possible to train a differentially private language model without large sacrifices in accuracy, so it would be helpful to know what differentiates this experiment. \n\nI would also note that the motivation, a predictive keyboard, is not a situation in which maximizing accuracy is generally desirable: users tend to find this creepy rather than helpful.\n\nThis is a nice idea but would benefit from some more polishing and more extensive testing.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper report empirical implications of privacy ‘leaks’ in language models. Reviewers generally agree that the results look promising and interesting, but the paper isn’t fully developed yet. A few pointed out that framing the paper better to better indicate broader implications of the observed symptoms would greatly improve the paper. Another pointed out better placing this work in the context of other related work. Overall, this paper could use another cycle of polishing/enhancing the results. 

Decision: Reject