SubMix: Practical Private Prediction for Large-scale Language Models
Authors: Tony A Ginart
Laurens van der Maaten
James Zou
Chuan Guo
Venue: ICLR 2022 Submitted (2022)
Keywords: private prediction
language models
user privacy
machine learning

Abstract: Recent data-extraction attacks have exposed that language models can memorize some training samples verbatim. This is a vulnerability that can compromise the privacy of the model’s training data. In this work, we introduce SubMix a practical protocol for private next-token prediction designed to prevent privacy violations by language models that were fine-tuned on a private corpus after pre-training on a public corpus. We show that SubMix limits the leakage of information that is unique to any individual user in the private corpus via a relaxation of group differentially private prediction. Importantly, SubMix admits a tight, data-dependent privacy accounting mechanism, which allows it to thwart existing data-extraction attacks while maintaining the utility of the language model. SubMix is the first protocol that maintains privacy even when publicly releasing tens of thousands of next-token predictions made by large transformer-based models such as GPT-2.

Under review as a conference paper at ICLR 2022
SUBMIX: PRACTICAL PRIVATE PREDICTION FOR LARGE-SCALE LANGUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent data-extraction attacks have exposed that language models can memorize some training samples verbatim. This is vulnerability that can compromise the privacy of the model's training data. In this work. next-token prediction designed to prevent privacy violations by language models that were fine-tuned on a private corpus after pre- pre-training on public corpus. We show that SUBMIX limits the leakage of information that is unique to any individual user in the private corpus via a relaxation of group differentially private prediction. Importantly. SUBMIX admits tight, data-dependent privacy accounting mechanism, which allows thwart existing data-extraction attacks while maintaining the utility of the language model. SUBMIX is the first protocol that maintains privacy even when publicly releasing tens of thousands of next-token predictions made by large transformer- based models such as GPT-2.
introduce SUBMIX: practical protocol for private
INTRODUCTION
The advent of transformers (Vaswani et al. 2017) has fostered a dramatic advancement in the capabilities of generative neural language models (LMs), enabling large-scale models such as GPT (Radford et al.) 2019 Brown et al.) 2020) to generate realistic, human-like text. Unfortunately, these impressive capabilities also come at a cost to privacy, the amount of excess parameters in the I.M enables it to memorize certain training samples. Consequently. several recent works have demonstrated practical training-data extraction attacks that reproduce entire sentences from the training dataset verbatim by querying the LM as an API (Carlini et al. 2019: 2020). These attacks expose the privacy risks of large-scale LMs, especially when their training data contains sensitive information such as addresses and personal ID numbers.
Existing solutions data extraction attacks focus on using differential privacy (DP: Dwork al. (2014)), which provably protects against privacy attacks (Yeom et al. 2018). Techniques such as DP-SGD (Abadi et al. 2016) have been applied to train differentially private neural networks on both vision and language tasks (McMahan et al. 2017 Papernot et al., 2018). However, the threat model in DP-SGD implicitly assumes that the adversary has full access to the private model's parameters and gradients during training, which results in pessimistic information leakage bounds that are unreasonable for most models. Indeed, existing work only performs DP-SGD training of small feedforward networks (Kerrgann et al.) (020)) and RNNs (McMahan al. 2017 Ramaswamy et al., 2020), often with an unsatisfactory privacy-utility trade-off. Training large machine- models with DPSSGD remains an open challenge (Jayaraman & Evans 2019 Tramer & Boneh 2020).
Our study deviates from prior work by, instead, considering the problem of private prediction (Dwork & Feldman 2018) using non-private language models fine-tuned a private corpus. We propose SUBMIX, a novel private prediction mechanism for answering next-token queries. Focusing on private prediction affords SUBMIX three notable advantages: (1) Private prediction does require modification of the training algorithm, which makes use of large-scale LMs feasible. (2) Private prediction allows us leverage the probabilistic nature of next-token sampling for highly efficient privacy accounting. (3) Private prediction allows us to leverage public pre-trained LM: to obtain private predictive distributions that do not require noise addition privatize the model's predictions.
SUBMIX utilizes an ensemble of LMs fine-tuned on disjoint parts of the private corpus and privatizes predictions by mixing the next-token distribution with that of public pre-trained LM. The mixing weight is adaptively tuned based on the degree of consensus among models in the ensemble. If
We do not provide privacy guarantees or text extraction protection for the public corpus which the LMs are pre-trained; our privacy guarantces only on apply the private corpus which the LM fine-tuned.

---

Under review as a conference paper at ICLR 2022
all models predict the same next-token distribution, then it is impossible for the next token to leak sensitive information about any unique individual so no mixing is required. By contrast, if models in the ensemble have high disagreement, SUBMIX will mix predictions with those of the public pre-trained model to minimize privacy leakage. This allows SUBMIX to perform accurate next-token prediction for most queries while preserving the privacy of the private corpus.
For any sequence of next-token queries issued to SUBMIX, we measure the amount of privacy leakage in the response using Rényi divergence (Rényi 1961). Our privacy notion, which we refer as operational privacy, is a sufficient condition for preventing samples that are unique to any user from being generated by the SUBMIX mechanism. Importantly, operational privacy allows us to perform tight data-dependent privacy accounting to upper bound the privacy loss of SUBMIX when answering variable-length query sequence. Concretely, when answering up to 020 next-token queries, SUBMIX realizes nearly 75% of the perplexity improvement that non-private fine-tuning would have achieved on GPT-2 models (Radford et al. (209)), with privacy leakage as small as We also show that SUBMIX can effectively prevent existing data extraction attacks against GPT-2.
2 PROBLEM FORMULATION
We begin by setting up the problem of private next-token prediction and reviewing existing literature on differential privacy. We then define and discuss the notion of operational privacy for SUBMIX.
2.1 PRELIMINARIES
Let Σ denote a fixed finite vocabulary set. We use lower-case letters to denote single tokens (such as Σ) and use bold font to denote contexts or strings of tokens (such as x E E*). A (causal) language
model h is mapping from context strings to distribution over next tokens: h X* - A A where A |Σز is the -il-dimensional probability simplex. For a particular context x E M", let h(x) denote the next-token distribution vector obtained from evaluating on context x, and let h22xx) E [0, 1]
denote the probability mass on token 2 E E.
User-level Corpus Let D denote a dataset of unstructured text, which is a set of token sequences E Σ" We assume that 'D is generated by set of distinct users, each holding subset D; of the full dataset D, i.e., 'D = U D. with D; n Dj 9 for & j. We refer to each D; as a user-level corpus for user i. As a concrete example, in the context of social media posts, D; would contain all of the non-public posts made by user i. We aim to provide privacy guarantees for model that is non-prine non-privately fine-tuned on the dataset D.
for language models is perform next-token pre- diction, that is, return a token z when queried with a context x. Such a query-answering API is useful for applications such as smart keyboard for auto-correction and text completion (Mirowski & Vlachos 2015) Hertel, 2019). Typical approaches for next-token prediction involve sampling z from the next-token distribution vector h(x);; see Holtzman et al. (2019). Large transformers trained on unstructured internet text have achieved remarkable success for this task, producing natural- llokiigg sentences via sequentially generating next tokens from given prompt (Brown al. 2020).
Next-token Prediction One popular
Text Extraction Attacks Recent studies have shown that it is possible for next-token prediction APIs to reveal sensitive private information contained in the training dataset. Carlinii et al. (2019) defined w-eidetic memorization formalize the notion that extraction of strings that are uncommon in the corpus can lead to violations user privacy.
Definition 2.1 (s-cidetic memorization (Carlini et al. 2019)) string s is s-ciieetc memorized by an LM h if is extractable from h and appears in at most examples in the training data D.
Carlini et al. 2019) showed that if the training dataset contains token sequences of the form: "My social security number is Соо-оо-оои where represents digit of a user's
CCallini et al. (211)) define text extraction informally. In the supplement, we formalize it within the framework of statistical hypothesis testing and show that differential privacy is sufficient to prevent eidetic memorization.
2

---

Under review as a conference paper at ICLR 2022
social security number (SSN), then it is subtantially more likely for the LM trained on D generate the exact SSN appearing in D compared to a random SSN. As a result, it is possible to design an efficient extraction attack that reproduces such unique sequences in the training dataset. [Carlini et al. (2020) further extended this attack to large transformer-basse
LMs such as GPT2 (Radford al. 2019), extracting memorized personal information such as name and address contained in the model's 's training dataset. Motivated by these shortcomings, this paper studies notions of privacy that can prevent such text-extraction attacks while preserving the model's utility.
2.2 DIFFERENTIAL PRIVACY
Differential privacy (Dwork et al., 2014) is a powerful mathematical framework for privacy-pressere data analysis. The underlying principle in differential privacy and all its variants is the notion of indiistinguishaabiiit. Informally, mechanism M is private if, given two adjacent datasets D and D', the mechanism's 's outputs MDD) and M(D') are approximately indistinguishable. Hence by observing the output of M. it is difficult for an adversary to discern the difference between 'D and D'. The above informal definition of privacy can be made mathematically precise by specifying: (1) the notion of adjacency between datasets D and D', and (2) the notion of approximate indistinguishabiltyy
Differentially Private Training Prior work on private LM training (McMahan et al. 2017 Ra
maswamy et al. 2020) adopted the definition of user-level adjacency: DD and D' are adjacent if they differ in a single user's s data. Approximate indistinguishability is defined in terms of divergences and applied to the trained model: The private training algorithm M(D) induces a distribution over models, and indistinguishability requires that D(M(D) D(M(D))IM(D') < é for some divergence D and small constant e > O. Popular choices include the max divergence (work et al. 2014) and the Rényi divergence of order (Rényi 1961):
1 a ---------- log log Egwq (P(=)/((=))*
(PIIQ) sup
log P(x) log Q(r),
(P(I))
=Esupp(Q)
Specializing to the choice of Rényi divergence, we define user-level Rényi differential privacy (RDP; Mironov (0177) I) for private training as follows.
Definition 2.2 (User-level RDP for private training). For a > 1, let D. denote the order-a Rényi divergence. A private training algorithm M is an (a., c)- mechanism if for any and D' that differ in only one user's data D, we have D., (MM())]
S
In order to satisfy the criteria in Definition 2.2 for neural language models, the standard approach is to use DP-SGD (Abadi et al. 2016) inject noise into the gradients computed at every iteration of SGD training, and use composition theorems to bound the total privacy leakage across iterations.
Differentially Private Prediction Private prediction differs from private training in that the notion of approximate indistinguishability applies to sequence of predictions made by a private prediction protocol P, rather than to a privately trained model. Formally, at each time step t. an adversary Adv (potentially adaptively) issues context string x,, and the private prediction protocol P responds by generating a next token y/ E Σ. We let P =n Adv denote the sequence of query-response pairs between P and Adv up until time T: P =T Adv (x,, y/ \ For query sequence of length T, approximate indistinguishabiiiy requires that for adjacent datasets D, D':
D (P(D)=
= Adv P(D')) = Adv) Se
(I)
T
for some divergence D and > O. We summarize the above discussion in the following Rényi-DP variant of the definition for private prediction by Dwork & Feldman (2018).
Definition 2.3 (User-level RDP for private prediction). Let a > 1, > O, and T E E.. A prediction protocol P is (a, c, T) RDP if for any adversary Adv and any D and D' that differ in only one user's data D,, we have that Equation I holds.
is well-known that differentially private models can be used for private prediction via the post- processing theorem (Mironov 2017): If h M(D) is model obtained from an (a, RDD training mechanism M, then (x; D) = hxx) is an (a, c, RDDD private prediction mechanism for any sequence of queries (regardless of length). However, DP-SGD (Abadi et al., 2016) the primary
3

---

Under review as a conference paper at ICLR 2022
mechanism for training private neural networks makes implicit assumption that the adversary also observes additional information that is not accessible is used as a prediction API, and in practice, it causes the accounting mechanism in DP-SGD to vastly overestimate the privacy leakage parameter 6 (Nasr et al.] 2021). One alternative is the general-purpose aubsample-and-aggreee mechanism, which adds noise ensemble's output in order to privatize it. This results in trade- off' between the information leakage, e, and the number of queries that can be answered, ' (van der Maaten & Hannun 2020). For smaller T', the mechanism needs less noise achieve a particular e. Conceptually, this is step in the right direction, but the added noise greatly reduces utility and is superfluous if we can leverage pre-trained public LMs to privatize the predictive distribution.
2.3 OPERATIONAL PRIVACY FOR PRIVATE PREDICTION
To remedy the problems in user-level differentially private training and prediction, we propose different notion of privacy that sufficient for preventing text extraction attacks, but admits more specialized privacy mechanisms with tighter privacy accounting.
Let P(D) denote the power set of DD purtition E P(D) of DD is a collection of sets a that satisfies Usen π D and that satisfies n æ' = 0 for distinct π, nT E II. We refer to the elements of II as parts. For some fixed ordering, we let II, denote the i-th part. As a minor abuse of notation, we let D I denote the usual element-wise subtraction and write IT \ instead of IT {x) ) for brevity. Recall the notion of the private prediction protocol P. We augment the protocol P with the capability terminate the query-response sequence any time. With slight abuse of notation, we denote by T(P) the sequence length produced by P. We define Rényi operational privacy (ROP) as follows.
Definition 2.4 (Rényi operational privacy for private prediction). Let be dataset user-level corpora and let II be partition so that each user-level corpus D; is contained in some part II E II. For a > 1 and c >> 0, a prediction protocol P is (a, c)-RÓP for partition II of dataset D if for any part II, E II and adversary Adv, we have:
Daym
(P(T) = Adv P(IT II,)
Adv) <
T(P)
T(P)
Where D (PIIQ) max Da (PII)), D. P)). ROP differs from user-level RDP in Definition 2.3 two aspects:
1. We substitute user-level adjacency with partition-level adjacency. By definition, the partition constructed so that any user's data belongs to single part. Readers familiar with group privacy (Dwork et al.) (2014) may recognize partition-level adjacency of the group-level adjacency used in Rényi group differential privacy. Partition-level RDP is neither strictly weaker nor stronger than user-level RDP, and, at cost, conversion is possible (Appendix C).
formal relaxation
2. We allow variable-length query-response sequences by enabling the crivate-prediction protocol P to terminate at will. ROP accounts for the privacy leakage in the responses made throughout the prediction protocol's operation lifetime, which why we refer for a variable-length sequence, we provide the mechanism with additional flexibility without increasing susceptibility to text extraction. Non-sensitive queries often can be answered without much privacy leakage, whereas sensitive queries may quickly exhaust the privacy budget, causing the protocol to terminate early. The protocol's decision terminate at time T(P) may leak some information about how sensitive the queried contexts are. However, this leakage is relatively insignificant and can be upper bounded (allowing us to convert variable-length into fixed-length; see Appendix C).
operational. By allowing
3 SUBMIX
We introduce SUBMIX, a private next-token prediction protocol that satisfies the operational pri- vacy definition introduced above (Figure I). SUBMIX follows the design of the subsample- and-aggregate mechanism by first forming random partition II of the training dataset, with each user's data belonging to a single random part II, For each part II, the protocol further splits II, into two subparts т; and 9े by randomly assigning users in II, to the two halves.
After termination, the mechanism can technically continue to issue responses, but only in way that is entirely independent of the private corpus, e.g., by using public pre-trained L.M.
4

---

Under review as a conference paper at ICLR 2022
Subsarts
Ensemble
Ports
Dstaset
<Qurry context
h
h
h
e
1) Subeample 2] Halvingg
Fineture
(a) Training. The corpus a dataset comprised of private user text. Each document represents all of the text corresponding
(b) Prediction. The bottom-mos network in the fig- ure represents the pre-trained public model; the other networks form the ensemble of model pairs obtained after SUBMIX training. SUBMIX prediction produces mixing weight for each model. These weights are aggregated and used to mix the ensemble predictions with the predictions of the public model.
particular user. At training time, SUBMIX learns ensemble by: (1) subsampling the dataset into overraaaappin parts, (2) halving each part into two subparts, and (3) fine- teeettunng the LM on each subpart using L.
Figure 1: Overview of SUBMIX's training protocol (left) and prediction protocol (right).
Fine-tuning Pre-trained Model Language models are often first trained on vast internet crawls to develop general understanding of human language, and then fine-tuned on a more nomain-specife dataset for the target task (Dai & Le 2015) Howard & Ruder
Algorithm 1 SUBMIX Training.
Inputs: User-level private corpus D, LM fine-tuning routine C Outputs: Fine-tuned LMs h hy; for I.....k
Hyperparameters: # of parts h:
I: <- Random k-fold partition of D with |[:|=|P]]
2: for i E (1, do
3: (TI, <- Randomly split part into subparts.
hr, + --(T), by; [(T()
2018). We treat language model training as black-box operation, and denote the training routine L(-) as a function that returns a model hp L(D). We assume access to a LM rreitriied public data, and routine L to fine-tune the LM on the private user-level corpora. Specifically, Algorithm I ffine-tunes a public pre-trained LM on each subpart n; and π to produce LMs Ax [(m:) and hy [(m(). By convention, fine-tuning the empty set returns the public pre-trained model: hg L()).
5: end for
Next-token Distribution Given a query context Xt, each part II; is responsible for producing
next-token probability mass function (pmf) by combining hr, (xx) and h (xx) into h₂(x,) (h,, (xi)+h (xx))/2 SUBMIX mixes this pmf with the public pre-trained model hy add noise the prediction that hides private information. It does so by computing:
h2,x,) > (x,, +
(x,))
for a suitable choice the mixing weight X A value of X = means the fine-tuned LMs he, and heg are used (no privacy loss), and X means no noise was added (no utility loss). We select X based on how much information about the part 11, is contained in the pmfs hw, (xx) and h (Xx).
Intuitively, since both T. and Tद are random samples from the same data distribution, the models he, and h did not memorize the query context X₃ then hg, (x,) and h (xx) will be similar. Hence, the selected value of X* should be close to If either h-, or h memorized the context x,, then hr, (x:) and h, (X;) are dissimilar as म; and TŚ have no users in common. This suggests that mixing with the pre-trained LM hep is necessary for hiding the sensitive information in II; so X* should be close to SUBMIX balances between these two extremes by computing separate X; for each part II Specifically, it sets target privacy leakage 3 >> 0 and optimizes:
A max {A D;(x,, A) < 8). 1
where (x,, A) = D. (Ah (x,) + (1 - A)ha (x,) ) Ah:; + (1 Axh)(x))). The final value of X is obtained by averaging the X; values for i 1,..., k. where is the number of parts.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 2 SUBMIX Prediction.
Inputs: Fine-tuned L. iif;; Nr! for ,...., k; privacy parameters e, time step query x, E 5
Outputs: Next token response Yt E Σ
Hyperparameters: Rényi divergence order target leakage 8
I: if then
14:
A k- - Eisid
Sx 4- e for == ......
15:
h E Eeffs hx((4))
3: else STOP has been issued then
16:
h 4- X()((x))
hh)
4: return Y1 hy (x:)
17:
+ X_,
5: end if'
6: for i l.......4 do
18: E
hx(xx)) f- (hT, (xx) ) + (x;((x))
19: end for
20: if 6; > 0 then
8: Compute A, using Eqqation
21: ~ h(x+)
end for
22: else
10: X + Σx=1 A;
23: Issue STOP signal.
11: M(x.) f- à: ? (x(()))
24: yr. ~ ho(x.)
12: h(x,) + +x"h(x,) - (1 (=xa)(x))
25:
13: for == l
26: return
Prediction and Privacy Accounting Given the next-token pmfs (x,) for 1,...,k, SUBMIX
computes the ensemble pmf, h(x,) 1/4 E/+E h; (x;),, and samples from it to obtain a next-token prediction. Our mechanism for selecting the mixing weight X can be shown to limit the privacy loss of sample from h(xx) under the operational privacy notion: Since each A; is determined entirely by the part II, the next-token pmf after removal of II, can be derived in closed form. This allows us to compute the Rényi divergence in h(x,) for adjacent datasets II \\ II, We present the SUBMIX prediction protocol in Algorithm 2 and give its formal privacy analysis the following proposition. Proposition 3.1. SURMIX is (a, R)OP prediction mechanism.
Proof. We use the adaptive sequential composition theorem for RDP (Mironov 2017 Prop. 1). By construction, Algorithm ensures that max, E, Da (yr ~ P(II) y P(II II)) for privacy budget e and divergence order a. Even if the query contexts {x+) are chosen adaptively by the adversary, Proposition 1 in Mironov 2017) allows us sum the Rényi divergences from each release:
T(P)
max Da (P(TT) = Adv || P(I IT.) = Adv) S max
Σ
Da (ye ~ P(TI) ll y1 ~ P(I IT)))
1=1
max Σ e:(1) c,(1
maxe
To conclude the analysis, note that after SUBMIX issues the STOP signal, any subsequent queries are answered by hg. Since hg is not function of IT, this does not leak any additional information. 11
4 EXPERIMENTS
Datasets We evaluate SUBMIX by fine-tuning the pre-trained GPT-2 model from Hugging- Face (Wolf et al., 2019) on two "private" datasets: (1) Wikitext-10 (Merity et al., 2016). collection of 103 million tokens scraped from Wikipedia; and (2) BigPat eerPateet-ta G (Sharma et al.) 2019), a collection
over 200,000 patents, We split the wikitext 103 corpus into blocks of length 512 tokens and define each block (synthetic) user D, We split BigPatent
by patent and define each user to be a single patent. This setup mimics settings in which users have distinct data distributions within the text corpus.
Fine-Tumme Tuning & Evaluation We use standard hyperparameters for fine-tuning: see Appendix B for details. To assess the quality of LM predictions, measure predictive perplexity:
I. log h( ... ...2.-1))
PP = E Ex=e Dreldout.
cxp
L

---

Under review as a conference paper at ICLR 2022
where Dheldout is the private held-out set and h(-[21 2;-1) denotes the pmf for the next token
given context 21 ........ 11-1 In practice, we truncate the context window to a fixed maximum length L = 512. Since each held-out sample consists of a block of L 512 tokens, computing pP,, for single sample requires I. queries total. Hence, the total number of queries B is a multiple of 512.
Following Geumlek al. (2017) report privacy loss in terms of c--éény divergence. Note that we can convert from ( to (e', 6DD via e' log(1/6) (Mironov 2017). In the paper,
o-1 we measure using a 2 Rényi divergence: present results for other values of Appendix A
4.1 PRIVACY-UTILITY TRADE-OFF
Baseline Comparisons We first com- pare SUBMIX to three privacy-preserving mechanisms as baselines: (1) DP-
7671
2405
10
SGD (Abadi al. 2016 for private training: and (2) saaaumpl--nn-gggreeeee (S&A, Dwork al.) (2014)) and (3) GN- Max (Papernot et al. 2018) for private prediction. See Appendix B for details on adapting these mechanisms for private next-token prediction. Figure shows the predictive perplexity of the private mecha- nisms on the Wikitext- ROP/RDI parameters = 2 and 2. The pre-trained GPT-2 model has a per- plexity of 37.5 on Wi k teritext-100 03 and is trivially private on that corpus. Fine-tuning the LM non-privately achieves perplexity of 20.0. SUBMIX achieves a perplexity of 26.9 B 3=1 1, 024 queries, which is substantially below the perplexity of the pre-trained LM. By contrast, the other mechanisms do not improve over the pre-trained baseline, even for single query (B 1). subsection presents more detailed results on the privacy-utility trade-off of the baseline meth- ods: them require extremely large e to improve over the pre-trained baseline, with DP-SGD outperforming the private prediction baselines (S&A and GNMax).
10e
44.7
20
26.9
10
Pretreined (Bmes)
Finetuned (0-0)
SubMix (B-1024)
DP SGD
S6A (B=1)
GNMax (B-1)
(B==)
Figure 2: Perplexity for 2, a = 2 of four privacy- preserving mechanisms (SUBMIX, DP-SGD, S&A. and GN- Max) using GPT-2 Wikitext- 03 with varying query budgets B. Non-privately fine-tuning achieves perplexity of 20.0 and the pre-trained public model achieves a perplexity of 37.5. Lower perplexity is better.
dataset for
1024
Ensembble
25.0
B 1074
Ensemble
35
5 2560
Public
5 2048
Public
10240
Finetuned
22.5
10240
Anetuned
8
20
20.0
17.5
15.0
2
2>
2
2 o-2
Figure 3: Perplexity of SUBMIX (k 8) on Wikitext- 103 (left) and BigPatent-G (right) as a function of ROP privacy loss c for three query budget values B. The perplexity of the pre-trained model and (non-private) single model, ensemble, and fully fine-tuned models are shown for reference.
Varying the Privacy Loss In Figure we show the trade-off between perplexity and ROP privacy loss, e, SUBMIX at a = On both i teet-10: 03 (left plot) and RigPatent (right plot), SUBMIx substantially improves over the pre-trained GPT-2 model, even when the query budget increased to = 10, 240 queries. As expected, SUBMIX matches the perplexity of non- non- privarivvt LM higher values of c. Interestingly, SUBMIX's perplexity is lower than that of single non-privately fine-tume tuned LM at high c. We surmise this is due to the performance gap between single because
fine-tuned LM and LM ensemble. The effect is less pronounced that corpus was split into users by block, as a result of which many LMs contain text blocks from the same Wikipedia article. This reduces the positive effects of ensembling predictive perplexity.
Wikitext-22
*rrivccy loss is measured under RDP for DP-SGD, S&A, and GNMax; and under ROP for SURMIX.

---

Under review as a conference paper at ICLR 2022
25.0
22.5
20.0
17.5
Public
15.0
finluued
Anctuned
2
21
2
2
2
2
2
2
Query Budget IBI
Query Budget (3]
Figure 4: Perplexity of SUBMIX (k 8) on Wikitext 103 (left) and BigPatent (right) as function of query budget B for different ROP privacy losses The perplexity of the pre-trained model and (non-private single model, ensemble, and fully fine-tuned models are shown for reference.
Varying the Query Budget Figure shows the trade-off between perplexity and the number of queries B. The results in the figure were obtained by tuning the target leakage to obtain the desired budget. As expected, answering next-token query at given c. However, SUBMIX attains a surprisingly low perplexity for a moderate number of queries (e. g. B 21 at all values both Wikitext 1e3 and BigPatent
queries using SUBMIX increases the average perplexity for each
Varying the Number of Parts A key hy- perparameter of interest in SUBMIX is the size of the partition II. Intuitively, smaller number of partitions, allows each part to train better quality LM at the cost of larger Rényi divergence when a part is re- moved. Figure shows the trade-off be- tween perplexity and ROP privacy loss, c, for varying partition sizes, k. We observe the key trend that decreases, perplexity increases more rapidly for smaller & be- cause the privacy budget is exhausted more quickly when each part has greater relative contribution to the ensemble. The optimal value for is generally around 16 for all values of interest, although this may depend the data distribution and design choices such as model architecture and training hyperparameters.
36
34
30
k=4
k= 32 Public
28
k-8
26
2-1
2
2
2a
21
27
21
2
@
am2
Figure 5: Perplexity of SUBMIX on Wikitext-10 as function of ROP privacy loss for different partition sizes k.
4.2 TEXT EXTRACTION ATTACKS
To empirically validate that SUBMIX pre-
vents text extraction attacks, we perform a random code text-cxtraction experiment in the style of Ramaswamy et al. (020); Shi et al. (2021). We randomly gener- ate codes with each code being an l- digit number (for example, representing a user's age, ZIP-code, phone number, SSN, elc.). The fine-tuning dataset constructed so that each user's text is single sentence: D; = -My number is: <random f-digit number here>" We then fine-tune this dataset and make private predictions using SUBMIX for the query context "My number is:" to test whether SUBMIX prevents text extraction (and if so, at what <). As baseline, we fine-tune GPT-2 on this dataset for 1000 iterations. This results in the LM memorizing all é codes, achieving perplexity of less than 0.5 and 90% recall when prompted with context. We apply SUBMIX with m/a parts so that each model the ensemble strongly memorizes number, achieving near 100% recall when prompted with the context.
0.4
E=10
t=10
E-10
E-10*
0.0
5.5
Code Length
Figure 6: Hit rate of text extraction attacks on SUBMIX for varying lengths of code. The # of parts is = 3 and the of codes generated is g = 100. The non-privately fine-tuned LM has a hit rate of > 0.9 for all lengths.
8

---

Under review as a conference paper at ICLR 2022
For the text extract attack, the figure-of-merte is the hit rate of the 9 generations, defined the number of generated codes that exactly match a secret code divided by g. Figure 6jshows the hit rate of the text extraction attack. For all code lengths L 2, ...... 5. SUBMIX succeeds in preventing the attack at 6 <= 10 For longer code lengths, even higher values of e suffice for preventing this random-sampling text extraction. Intuitively, extraction becomes more difficult as the code lengths increase. This experiment shows that the mechanism for limiting the release of sensitive information via solving Equation is effective, and the privacy accounting in SUBMIX meaningfully measures the amount of privacy loss. We also ran these attacks for varying values of K and m, and made the same qualitative observations. In addition, we performed experiments in which we varied g E {10, 100, 1000). We found that this does not affect SUBMIX's ' hit rate.
5 DISCUSSION & RELATED WORK
Related Work McMahan et al. (0117) was the first to study the training of differentially private language models by using DP-SGD to train a small recurrent neural network. However, the resulting LMs have far fewer parameters than modern transformers and attain much higher perplexities. More recently, Shi et al. (2021) explored an alternative approach called selective differential privacy, where the privacy guarantee only applies to blocks of text in the training corpus that are deemed sensitive, e.g., addresses and phone numbers. Unfortunately, this approach is difficult to scale to large unstructured text corpora because it requires annotating all text in the corpus with a privacy sensitivity level.
SUBMIX has conceptual similarities to PATE (Papernot et al. 2016 2018) for private semi-supervised learning. Both SUBMIX and PATE make predictions using an ensemble of models trained (or fine- tuned, the case of SUBMIX) on private data, and employ a data-dependent and query-dependent privacy accounting mechanism at prediction time. The central idea in both methods is that privacy loss is small when models trained on different parts the data agree on a prediction. However, PATE is more natural in discriminative or classification tasks because it return distribution's 's noisy argmax. On the other hand, SUBMIX is more natural in generative tasks because it returns a sample from the distribution.
SUBMIX also related to prior work on private posterior sampling (Geumlek et al. 2017 Dimi-
trakakis et al. 2017), where the randomness in the privacy mechanism comes from releasing a sample from a distribution defined by the private data. In particular, SUBMIX uses a privacy accounting methodology based on Rényi divergences similar to that Geuulek et al. (017))
Mireshghallah et al. (202)) propose adding a privacy regularizer to language model training to reduce its memorization of sensitive text. They empirically showed that the regularizer reduces the effectiveness of text extraction attacks, but it does not satisfy any formal privacy guarantee such as DP. Other related works exploring privacy in natural language processing include (Gopi et al. 2020
Lyu et al. 2020
Xu et al. 2020
et al. 2018 Kim et al. 2021
Limitations & Future Directions One limitation of the SUBMIX protocol as presented here is that it only supports decoding from the ensemble of LMs by sampling directly from the predicted pmf. This type of sampling is known to produce unnatural and incoherent text (Kulikoy et al.) 20018) Holtzman al. 2 Better decoding methods such as top-k sampling (Fan et | 2018)) and nucleus sampling (Holtzman et al. 2019) exist, but they require modifications to the protocol that may cause additional privacy leakage. However, we note that a close alternative to top-k and nucleus sampling is temperature decoding (Holtzman et al. 2019), which scales the predicted pmf by a temperature term to decrease its entropy. SUBMIX readily supports this decoding method by applying temperature scaling as post-procese post-processing step. In future work, we aim to extend out work by designing protocols that can support different types of decoding strategies as well.
Another limitation of SUBMIX is that the use of an ensemble substantially increases the computational and storage requirements. Our experiments suggests that an overhead factor of is needed to attain a non-vacuous trade-off between privacy and utility. One potential solution to reduce the computational requirements of SUBMIX may be to fine-tune only the top few transformer layers closest to the prediction head. This would allow the evaluation of the bottom transformer layers to be shared between models in the ensemble, thereby reducing both computation and storage requirements. We leave the exploration of such efficiency improvements for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks, In 28th {USENIX] Security Symposium {{ {USENIX) Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vors Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020.
Andrew M Dai and Quoc Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28:3079-3000 2015.
Christos Dimitrakakis, Blaine Nelson, Zuhe Zhang, Aikateirni Mitrokotsa, and Benjamin Rubinstein. Differential privacy for bayesian inference through posterior sampling. Journal of machine learning research, 18(11):1-39 2017.
Cynthia Dwork and Vitaly Feldman. Privacy-preserving prediction. In Conference On Learning Theory, pp. 1693-1702. PMLR, 2018.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor Comput. Sci., 933-4):211-4000 2014.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:
04833, 1805.0483,
2018.
Joseph Geumlek, Shuang Song, and Kamalika Chaudhuri. R) 'enyi differential privacy mechanisms for posterior sampling. arXiv preprint 17X1v:1710.00892, 2017.
Sivakanth Gopi, Pankaj Gulhane, Janardhan Kulkarni, Judy Hanwen Shen, Milad Shokouhi, and Sergey Yekhanin. Differentially private set union. In International Conference on Machine Learning, pp. 3627-3636. PMLR, 2020.
Matthias Hertel. Neural language models for spelling correction. Master's thesis, Albert-Ludwigs- Universität Freiburg im Breisgau, 2019.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09733 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint 1rXiv::80) 00000.06166.... 2018
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th (USENIX) Security Symposium ({ (USENIX) Security 19), pp. 1895-1912, 2019.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training, arXiv preprint arXiv:2009.05886 2020.
Kunho Kim, Sivakanth Gopi, Janardhan Kulkamni, and Sergey Yekhanin. Differentially private n-gram extraction, arXiv preprint arXiv: (tt:2//08.083/// 2021.
Ilia Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of search and evaluation strategies in neural dialogue modeling. arXiv preprint (Xii:/8//.00900 2018.
Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and crivacy-preeerving text represen-
tations. arXiv preprint arXiv: 1805.06093. 2018.
10

---

Under review as a conference paper at ICLR 2022
Lingjuan Lyu, Xuanli He, and Yitong Li. Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairesss. arXiv preprint arXiv:2010.01285, 2020.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.0696, 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (609.00400077333 2016.
Fatemehsadat Mireshghallah, Huseyin A Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg- Kirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization in language models. arXiv preprint arXiv:2103.07567, 2021.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Piotr Mirowski and Andreas Vlachos. Dependency recurrent neural language models for sentence completion. Proc. IJCNLP, pp. 511-517, 2015.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adver- sary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.04535, 2021.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.00775, 2016.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate. arXiv preprint arXiv:1802.0888 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint arXiv:2009. 10031, 2020.
Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics, pp. 547-561. University of Californi Press, 1961.
Philippe Rigollet. 18. s997: High dimensional statistics. Lecture Notes), Cambridge, MA, USA: MIT Open-Course Ware, 2015.
Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint /ttl::/900.33444. 2019.
Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. Seleetive differential privacy for language modeling. arXiv preprint arXiv:2108. 2tt.:22.08.19944,, 2021.
Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). arXiv preprint arXiv:201 I. 1660, 2020.
Laurens van der Maaten and Awni Hannun. The trade-offs of private prediction. arXiv preprint arXiv:2007.05089, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of ofttte--rttttttte- natural language processing. arXiv preprint 191.03111, 2019.
11

---

Under review as a conference paper at ICLR 2022
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. A differentially private text perturbation method using a regularized mahalanobis metric. arXiv preprint arXiv:2010.11947 2020.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitiigg In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 262-282. IEEE, 2018.
12


Reviews:
[{"review_id": "hzk_FxaSWBU", "reviewer": "Reviewer_kTn8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Submix does not need to modify training algorithm and existing large-scale language models are available.\n2. Submix utilize probabilistic nature of next-token sampling to protect privacy.\n3. Submix does not need to add noise for privacy of model prediction.\n\nCons:\n1. Fail to show runtime comparison of proposed approach and existing work (e.g., DP-SGD). Except effectiveness analysis, efficiency analysis may convince reader more since less efficient approach may weaken its application in the real world. Authors can provide a chart with respect to runtime vs. epsilon.\n\nQuestion:\n1. Since this work uses non-private language models for fine-tuning, is it under risk of member inference attack?\n2. About user-level corpus, since young people sometimes use similar internet words, phrases and sentences on their social media posts, D_i and D_j sometimes may not be disjoint set. How should Submix deal with this scenario?", "summary_of_the_paper": "Language models keep sentence-level information of training samples in detail. This feature jeopardizes data privacy by data-extraction attacks, which can retrieve full sentences of training samples by directly querying models. Current solution to this issue is based on differential privacy, such as DP-SGD. However, existing approach suffers from information leakage due to full access to model’s parameter/gradients and only can be applied to small-scale RNNs, which also cannot balance privacy-utility trade-off. In this work, the proposed approach called SUBMIX focuses on private prediction for answering next-token queries with regular language models fine-tuned on the private corpus. Submix as an ensemble based on models by private corpus and public pre-trained models, and only mix them if they disagree. Privacy leakage is measured by Renyi divergence, and operational privacy is guaranteed. ", "summary_of_the_review": "Sentence-level privacy in the language model is a very challenge problem since LMs applied to many tasks in NLP. To this end, Submix provides a scalable and flexible approach to address the issue by focussing on next-token prediction, which is popular for many downstream NLP tasks. Hence, I recommend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cCUQw53DXiX", "reviewer": "Reviewer_6t5J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and experiments show that the method works in practice. The visualizations are also great. The paper studies a difficult problem as utility of next word prediction suffers a lot when trained with differential privacy.\n\nExperimental section does not provide details on whether other methods with DP were trained using an already pre-trained models and not only using private corpora. This needs to be made clear that the improvement in the utility is not only from the user of a pre-trained model (for example, why would one even use DP-SGD and not the pretrained model?)\n\nThe comparison with PATE seems a bit brief as the methods are quite alike albeit PATE was used for classification tasks. It would be interesting to see how SUBMIX performs on classification tasks.\n\nPlease explain while splitting into two parts is needed after the data is already split in k.\n\nMinor: Fig 1 b calls the second model differently than in Alg 1 and 2 (h and lambda)", "summary_of_the_paper": "The paper proposes a method for answering next-token queries in a privacy-preserving manner. As opposed to training models with DP guarantees, it adds privacy-preserving guarantees during the prediction time. The approach works by getting a public model and then fine-tuning it on disjoint parts of the private dataset. This way it obtains, several models. During prediction, it queries each model and combines the outputs by proportionally adjusting the contribution from each model. The experiments how that utility is better than that pf DP training and also protects against extraction attacks of sentences.", "summary_of_the_review": "The paper presents an interesting idea for privacy-preserving prediction for fine-tuned models. However, as written now it is not clear if the benefit in experiments is due to fine-tuning or due to the new method of training/prediction proposed in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R77rriEPo3u", "reviewer": "Reviewer_aiFL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I applaud the authors for focusing on the problem of privacy in language modeling, however, I have some major concerns about the assumptions made and the actual applicability of the proposed method, that I will list below:\n\n1. Faulty assumptions in the experiments: The paper has defined their setup to be \"finetuning\" a pretrained language model. In finetuning tasks in NLP, it is often the case that training data is scarce. However, here for the experiments, Wikipedia is used as \"private\" dataset, which is far from a realistic scenario, especially when there are many realistic available datasets that NLP/privacy researchers use, such as Enron email dataset,  sentiment 140, Shakespeare, etc (I have listed multiple related missing privacy in NLP reference papers below, there are more appropriate datasets there as well). I strongly believe that if a small dataset is used, the utility of the proposed method will also suffer gravely, given how it partitions the data and leaves few training samples for each large model. I think an experiment on a small dataset could clear the outcome of this issue. \n\n2. Assumption on the correlation between users: I highly disagree with the assumption that if all the LM partitions agree on the next word for a given sequence, then there is no leakage. There is the possibility that users are correlated, like in an email dataset in a company, and a large group of them might all be talking about a secret, like a secret serial number. If different users talk about this, there is a chance that each partition ends up with at least one user that has that serial number in their data. Therefore, they might all predict it, and agree on it, and this could leak that string. Please clarify on how such a scenario can be prevented in your setup. \n\n3. Huge computation/storage costs: Although the limitations section of the paper mentions this and proposes using a smaller model as a solution, I highly disagree. The proposed method's functionality depends on prior pretraining and then finetuning. This is not the usage mode for small LSTM models. So the proposed solution isn't really feasible. \n\n4. The entire NLP-related body of work on this problem is missing. This is not  just a few missing references. I have provided a list below:\n\n\n1. Gopi S, Gulhane P, Kulkarni J, Shen JH, Shokouhi M, Yekhanin S. Differentially private set union. InInternational Conference on Machine Learning 2020 Nov 21 (pp. 3627-3636). PMLR.\n\n2. Mireshghallah F, Inan HA, Hasegawa M, Rühle V, Berg-Kirkpatrick T, Sim R. Privacy Regularization: Joint Privacy-Utility Optimization in Language Models.\n\n3. Lyu L, He X, Li Y. Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings 2020 Nov (pp. 2355-2365).\n\n4. Xu Z, Aggarwal A, Feyisetan O, Teissier N. A Differentially Private Text Perturbation Method Using Regularized Mahalanobis Metric. InProceedings of the Second Workshop on Privacy in NLP 2020 Nov (pp. 7-17).\n\n5. Li Y, Baldwin T, Cohn T. Towards Robust and Privacy-preserving Text Representations. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) 2018 Jul (pp. 25-30).\n\n6. Kim K, Gopi S, Kulkarni J, Yekhanin S. Differentially Private n-gram Extraction. arXiv preprint arXiv:2108.02831. 2021 Aug 5.\n\n", "summary_of_the_paper": "The paper attempts at introducing a privacy-preserving method for predicting next words, for large language models. The problem setup for the paper is when pretrained large language models are available, and they are to be fine-tuned on private data for the task of next word prediction. If DP-SGD is used here, due to the large number of parameters in these models, the authors claim that the noise that is added would be too much and utility (i.e. perplexity) would suffer gravely. Therefore, they introduce an ensemble-based approach: partitioning the data into disjoint subsets, such that each user's data appears in only one partition. Then, finetuning a separate model on each partition. For decoding (i.e. inference, guessing the next word), if all the partition-LMs agree, then they assume that there is no leakage and release it. If there is disagreement, the output of these models is mixed, along with the output of a publicly pretrained LLM, so as to bound the private leakage (kind of similar to PATE, but for language modeling). Then the paper proceeds to attempt at empirically testing this method on two datasets. ", "summary_of_the_review": "To summarize the points from above, I believe the main shortcomings of this paper are:\n\n1. unrealistic assumptions in the experiments and about the problem setup in general (relating to the size of data and correlation between users).\n2. Huge computation and storage costs which are not explicitly studied.\n3. Not providing any comparisons with prior existing work in NLP.  \n\nIf the authors show that this method is effective in a setup where the user data is scarce, address my concern on corrolations, and provide comparisons (qualitatively) with prior work I am willing to update my score. \n\n\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JmzGFNsjxSw", "reviewer": "Reviewer_zjmQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, the reviewer does not understand why the proposed approach defends against the text extraction attacks without releasing the trained models? The prediction setting provided in the proposed approach may be suitable for ML as a service (MLaaS). However, why next token is an MLaaS application and in which real-world scenarios? Also, the connection between user level-DP with text extraction attacks is weak. Why is user level-DP needed to defend against text extraction attacks?\n\nSecond, the threat model is missing. The reviewer is not sure how the adversaries will choose the query $x_t$ to extract sensitive text data. It is unclear from the writing.\n\nThird, the user level-DP provided in the proposed approach by partitioning users’ text data into different parts and sub-parts is incorrect. For the fine-tuning models to work well, data in a single part must be sufficiently large, requiring a certain number of users included in the part. As a result, the neighboring part definition (excluding one part) does not provide a precise user level-DP. We can only have user level-DP if a user is a part. This scenario will not work well for fine-tuning large NLP models, such as GPT-2, given a limited amount of text data of a user. \n\nAlso, the assumption that if two models return similar results when they do not memorize the context of a query $x_t$ is very vague. It is unclear to me when this could happen and how we can quantify the similarity here. No evidence was provided. The same issue given the two models return dissimilar results when they memorize the query $x_t$ given that they do not share common users. \n\nFifth, experimental results are unconvincing. DP-SGD does not provide either user level-DP or solutions to prevent text data extraction attacks. Why is DP-SGD being considered a baseline here? At least, the user level-DP from McMahan et al. (2017) and its variations (Kairouz et al., 2019; Ramaswamy et al., 2020) should be considered as appropriate baselines. Improving the perplexity in the fine-tuning does not indicate that the model is secure against data extraction attacks. \n\nRegarding the attack evaluation, why randomly generating $m$ codes with each code being an $l$-digit number (for example, representing a user’s age, ZIP-code, phone number, SSN, etc.) can be a representative measure for information leakage under data extraction attacks? Sensitive information can be extremely broader than these numbers, such as name entity, correlation among them, etc. What would be a good way to evaluate the model under the attacks and under which threat models?", "summary_of_the_paper": "This paper presents a new approach to simultaneously preserve user level-DP and prevent text extraction attacks in prediction. Experimental results were conducted on Wikitext-103 and BigPatent-G datasets showing some promising results. However, there are several critical concerns detailed in the main review regarding the application contexts, theory analysis, and experimental settings. ", "summary_of_the_review": "Overall, this is an important research direction. The proposed approach is interesting. However, there is room for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a new mechanism SubMix that provides next-token prediction under a variation of the differential privacy constraint. There is disagreement among the reviewers when assessing the quality of this work. Even though the study of private predictions in large language models is new, the reviewers raised several issues in the proposed approach. First, the formulation of partition-level DP created confusion about the privacy guarantees provided by the mechanism. Given the similarity to PATE, it might be useful to articulate if there is any difference between the privacy guarantee in this paper and the one of PATE. Second, the authors might want to further clarify the reason for having two sub-parts, which has also created some confusion. Even after reading the author's response and the updated revision, the AC still could not understand the relevant privacy argument. In summary, the paper may require further clarification and revision before it is ready for publication.
Decision: Reject