Defending against Reconstruction attacks using Rényi Differential Privacy
Authors: Pierre Stock
Igor Shilov
Ilya Mironov
Alexandre Sablayrolles
Venue: Submitted to ICLR 2023 (2023)
Keywords: Rényi Differential Privacy
Reconstruction Attacks
Information Theory

Abstract: Reconstruction attacks allow an adversary to regenerate data samples of the training set using access to only a trained model. It has been recently shown that simple heuristics can reconstruct data samples from language models, making this threat scenario an important aspect of model release. Differential privacy is a known solution to such attacks, but is often used with a large privacy budget (epsilon > 8) which does not translate to meaningful guarantees. In this paper we show that, for a same mechanism, we can derive privacy guarantees for reconstruction attacks that are better than the traditional ones from the literature. In particular, we show that larger privacy budgets do not provably protect against membership inference, but can still protect extraction of rare secrets. We design a method to efficiently run reconstruction attacks with lazy sampling and empirically show that we can surface at-risk training samples from non-private language models. We show experimentally that our guarantees hold on real-life language models trained with differential privacy for difficult scenarios, including GPT-2 finetuned on Wikitext-103.

Under review as a conference paper at ICLR 2023
DEFENDING AGAINST RECONSTRUCTION ATTACKS WITH RÉNYI DIFFERENTIAL PRIVACY
Anonymous authors Paper under double-blind review
ABSTRACT
Reconstruction attacks allow an adversary to regenerate data samples of the training set using access only a trained model. It has been recently shown that simple heuristics can reconstruct data samples from language models, making this threat scenario an important aspect of model release. Differential privacy a known solution to such attacks, but it is often used with large privacy budget (s 2 8) which does not translate to meaningful guarantees. Our main contribution is stronger privacy guarantees against reconstruction attacks improving existing literature. In particular, we show that larger privacy budgets do not provably protect against membership inference, but can still protect against extraction high-entropy secrets. We design method to efficiently run reconstruction attacks with lazy sampling and empirically show that we expose at-risk training samples from non-privee non-private language models. We show experimentally that our guarantees hold language models of practical significance trained with differential privacy, including GPT-2 finetuned on Wikitext-103.
INTRODUCTION
Probabilistic generative models trained assign high likelihood to data from the training set. In
the case of language models, the decomposition P(r ,T)) = 112",, / xee)) also allows for efficient sampling of sequences. Given such models will overfit on the training set data, sampling from a trained model will sometimes yield verbatim sentences from the training set. Carlini et al. (20216) leverage this effect along with clever filtering techniques to produce samples that are likely to be in the training set. Their work demonstrates that reconstruction attacks are not only possible on large-scale generative language models such as GPT-2 (Radford et al. 2019)) but also successful: their best attack reaches 66% precision on the t0p-100 generated sentences. Another category of attacks is membership inference, where the adversary has access to both the trained model and data sample, and has to predict whether this data sample comes from the training set. This attack is easier because the task of the adversary is simpler.
The standard defense against such privacy attacks is differential privacy (DP) (Dwork al. 2006 Dwork & Roth 2014). DP defines privacy budget E that can be used to control the privacy/utility tradeoff of the trained model. However, there is no consensus over acceptable values of and, in practice, e is often chosen to defeat practical membership inference attacks (Watson et al. 2021 Carlini et al.) (021a). In this paper, we show that Rényi differential privacy (RDP) (Mironov 2017) can actually provide guarantees regarding the probability of reconstructing a sample; these guarantees are stronger than existing ones for the same mechanism. In particular, we show that there is an intermediate regime in which membership inference is not protected but full reconstruction of sufficiently high-entropy secrets remains difficult. This means that an adversary who knows a secret can determine if it was in the training set, but extracting it from the model stays hard if they do not know the secret s in the first place.
We refer to samples that an adversary tries to reconstruct as secrets. Not all samples from the training set would be considered "secret" in the sense that their content is public knowledge, such as newspaper headlines. We circumvent this issue by considering all samples secrets, and quantifying their level of secrecy by the number of unknown bits of information. Specifically, given a probabilistic reconstruction model A that generates secrets from trained model 6, A()), the secrecy of sample is b A log, (1)P) = s)). An adversary needs on average 2 trials to chance upon the

---

Under review as a conference paper at ICLR 2023
0-0.4, c=212, 6-3e-07
g=0 c-7.7, 5=3e:07
5-3e
30
Laaloge (bits)
Lealage
20-
29-
20
Our lound
Noroo et, al Secret disc wour
30
10
Secret length (bits)
Secret length (bits)
Secret length (bits)
We train a neural network using DP-SGD and consider various levels of DP-noise o (increasing from left right). We consider secret of varying length (in bits b, ar-axi)) in the dataset and wish to estimate the number bits from this secret that will leak after training (y-axis). For each level of DP-noise, we plot the upper bound min, b(a, po) computed using Mironov al. 2019) (see Equation equation 2) as well as our theoretical bound L(po) against the secret length b by setting Po 2-b We additionally plot the line y x: points below this line indicate that the secret will not entirely leak, points above means total leakage. To generate these plots, we use the settings of our real-life canary experiments in Section 4.3 with 186k training steps and sampling rate q 2.81 x 10-1. We also report the privacy budget at 5 = 3 x 10-1 in the plot titles computed using Balle et al. (2020), the standard practice with DP training. The plots demonstrate that our guarantee prevents from total secret disclosure for various levels of DP noise a. Furthermore, observe that our bound is comparatively better for lower levels of the DP noise a.
Figure 1:
secret 8. This number of trials corresponds the verification cost incurred by adversary in many practical scenarios. For example if the adversary is guessing a phone number, they have to actually dial this number "exploit" the secret.
A secret can have a non-zero probability even for model that was not trained on this secret. As an extreme example, a language model generating uniformly random numbers will predict 10-digit phone number with probability 10 The goal of a ensure that the secret is not much more probable under a model that was trained on it. The length of the secret can vary depending reduces the secret from 10 to 7 unknown digits. Fortunately, RDP guarantees work against any prior knowledge, thanks to the post-processing property detailed in Section 2.2
training procedure is to
prior information: for phone numbers, knowing the area code
Our contributions are the following:
We use the probability preservation guarantee from RDP derive better secret protection. In particular, we show that the length of the secret itself provides more privacy.
We empirically estimate the leakage for n-gram models and show that matches our bound.
We design method efficiently run the reconstruction attack of Carlini et al. (2021b) using lazy sampling and use it to surface at-risk samples on non-private language models.
We fine-tune language models to competitive perplexity with differential privacy and show that the observed leakage under our attack model is smaller than the guarantee, even when we favor the adversary by increasing the secret's 's sampling rate.
2 BACKGROUND
Throughout the paper, we use log to denote the natural logarithm and log, for the base 2 logarithm.
2.1 PRIVACY ATTACKS
Membership Inference attacks (Homer et al. 2008 Shokri et al., (017) determine, given a trained model and data sample, whether the sample was part of the model's training set. Given sample and a model 9 trained on a dataset D, the attacker's objective is to design a score function @(0, 2) such that is high when as the gradient norm (Nasr et al. 2019). the model confidence (Salem et al.) 2018), or an output of a neural network (Shokri et al. 2017). Surprisingly, choosing the score function to be the loss L0)
D and low otherwise. Various score functions have been proposed, such
2

---

Under review as a conference paper at ICLR 2023
simple and effective approach (Salem et al. 2018 Sablayrolles al. 2019). ). Recent works argue
that a practically relevant threat model of membership inference is to confidently predict training set membership of few samples rather than guessing well on average (Watson al. [021 Carlini 2021a Ye et al., 2021). Such membership inference attacks are evaluated using true and false positive rates. In this context, calibrating the loss by centering (Watson al., (02T) or by fitting Gaussian likelihood (Carlini et al. 202 further improves the performance of the attack and yields state-of-the-arr results. This kind of attacks essentially identifies uncommon samples on which model is overconfident. These samples are also the focus of our study.
let al.
Language Model Extraction. Carlini et al. (2019) propose a methodology to quantify the exposure of unique and secret training sequences to such extraction attacks. To this end, the authors insert a canary in the training set and measure its exposure as the excess belief that the model has in the canary over random chance. The authors then conclude that differential privacy is suitable mitigation technique, although at the cost of some utility. Similarly, Carlini et al. (2021b) show that possible to extract hundreds of training sentences in a two-step procedure when attacking GPT-2, a language model trained on large scrapes of the public internet (Radford et al. 2019). First, the authors generate 200,000 samples for each of the following sampling strategies: sampling with linearly decaying temperature, top-k sampling and sampling conditionally on random Internet text. Then, they reduce the problem to Membership Inference among the generated samples and select the top-100 100 samples most likely to be members according to different filtering strategies. Depending on the metrics rank the generated sentences (loss of the target model calibrated or not with loss of smaller model for instance), the authors identify a few dozens of training sentences from the top- 100 sentences. Finally, a recent line work focuses on providing differentially private predictions based on pretrained, non-DP model (Ginrrt et al. 2022 Majmudar al. 2022
2.2 DIFFERENTIAL PRIVACY
Differential Privacy is a standard for privacy guarantees (Dwork et al. 2006 Dwork & Roth 2014).
Definition 1. A randomized mechanism M: D + R. satisfies (s, 8)-differential privacy (DP) if. for any adjacent inputs D. D' DD for any S C R, we have P[M(D) E 5] S eFPIM(D") E S)+
Datasets D and D' are adjacent if they differ by at most one element. Rényi Differential Privacy (RDP) MMronovv| 2017) was introduced obtain tighter composition properties. We recall here the properties of RDP that will be useful for the rest of the paper, while referring the reader to Mironov (2017) for a more comprehensive overview.
Definition For two probability distributions and Q defined over R, the Rényi divergence of order a > 1 is
P(x)
Da(P Q) a
log E
Q(x))
Definition 3. A randomized mechanism M: : 'D -> R satisfies (ax, R differential privacy (RDP) if. for any adjacent inputs D, D' E D, we have
Da(M(D) II M(D')) do.
RDP guarantees can be converted into DP guarantees while the converse not true, making RDP
strictly stronger property. If M is (a, da) -DDP, then it is also (da + log 1/8 6) -DP for any
0 < 8 < 1. This conversion to (e, 8DDD is slightly improved by Balle et al. (022)).
Propertics of RDP. As with (e, 8). (a,, d-P guarantees are preserved by post-processing: if M (ox, RDDP and if A: R - R' is randomized mechanism, then A o M is (ox, RDDP.
Probability Preservation. A direct consequence of (a, da)-RDP is quantified by the inequal- ity (Mironov 2017)
edaapi(a-11) Sp' < (edxp) 1)/a
where A P[M E S] and and é P[M(D') E S). Informally, since M(D) and M(D') are close, the probabilities of any event S under both M(D) and M(D') also close.
3

---

Under review as a conference paper at ICLR 2023
DP-SGD. Private training of neural networks is usually conducted with DP-SGD (Abadi al. 2016 Song et al. 2013) Bassily et al. 2014) as follows. For each training step t. we gather a batch (of average length I.) of training examples (+) by sampling each element without replacement with probability (or sampling rate) g. We compute the per-sample gradients 9o(2,), clip them constant
C > 0, average them and add Gaussian noise with variance o²C²:
gt 7 Σ [
ge(zi) = ge(zi)/ max(1, (z)((2))
and we use the noisy gradient de in the optimization step. Finally, an accountant tracks the Rényi privacy da over all the training steps. This quantity only depends the sampling rate q. the number of training steps and the noise multiplier a.
NLP with Differential Privacy. Recent work has shown that fine-tuning language models to competitive accuracy with differential privacy is possible. For instance, [i et al. (2021) provide recipe directly fine-tune large transformer models with 50 300 million parameters directly with DP at privacy level E {3, 8} for various downstream tasks. Similarly, Yu et al. (202I) finetune privately competitive RoBERTa-Larg models (Liu al., 2019) with 6.7 by training only fraction of the network's 's parameters using low-rank weight matrices or skip-connections.
3
OUR METHOD
We first derive an upper bound on the information leakage, and show that it provides better privacy guarantees compared to traditional ones (Mironov 2017). Then, we present lazy method to efficiently identify samples that are likely to leak when performing a reconstruction attack.
3.1 RECONSTRUCTION AND CANARIE'S
The goal of reconstruction attacks is to surface training samples given access to the target network's weights. If D is dataset and secret, we denote by D' D {s}. Recall that M(D) (resp., M(D')) represents the distribution of the target network's s weights after training on D (resp., D'). We assume Finally, A denotes the attack mechanism that takes the target network's weights and outputs probability distribution over secrets. For a given secret
note po P[A(M(D)) = s]
and P1 & P[A[M(D')) s]. We thus sometimes refer to po as the prior (i.e. the secrecy of before 8 is added to dataset D), and pı as the posterior.
Theorem 1. If M is (a, da) RDDP, the leakage log(p1/ /ppp//))/ is bounded by
d log(1/Po)
P1
a -
log(1/po)
< log
<
d
(2)
--1
Po
A h(a.po)
Al(o.po)
Finally, since a > a 1, we have (0) < h(a,po).
Proof. Thanks to post-pnn post-processing, A M is (a, da RDD,, and using the Probability Preservation inequality equation] I gives the result. See Appendix A. for intermediate calculations.
We emphasize that Theorem [ applies to any attack mechanism A, as A M RDP as long as M is. Corollary Since the leakage log(p1/Po independent a, we strengthen the bound of Theorem 7 by minimizing over orders:
log
P1
L(po) A min I(a, po).
(3)
Po
a>1
Comparison with traditional DP guarantees. leakage follows:
Theorem equation D Timplies a bound on the absolute
log
P1 Po
< max(((o.p)) = M(a.
4

---

Under review as a conference paper at ICLR 2023
a=0.4, secret length=60 bits
60
Leakage (bits)
50
40
30
Our bound
10
Mironev
Secret disclosure
50000
100000 Training iterations
150000
We consider training a neural network with DP-SGD with the setup described in Figure] 1 Here, we fix the secret length to 60 bits and plot the dependency of the leakage with the number of training iterations long our empirical measurement (see Figure 3 Interpretation: even when repeating the canary multiple times, the empirical observed leakage in real-life scenario still lower than our predicted bound L(po).
Figure 2:
we take 5 = po. this corresponds to the traditional E given by min, h( 6). Given that P(), < h(a, Po), our bound leakage is better than traditional guarantees from (e, 6DD.. These values shown in Figures and 2 for lower values of the noise multiplier a, the bound provided by / is much lower than h, while the gap between the two decreases the noise multiplier gets bigger. We can extend that analogy and compare our leakage guarantee to the privacy budget that would be given for a probability of failure 6 Po- Numerically, our bounds are also better than the slightly tighter bounds by Balle et al. 2020 as depicted in Appendix A.3, although not directly comparable.
bsolute Leakage. Nominally, a bound on the absolute leakage is a stronger guarantee, because it also prevents the posterior p1 from becoming smaller than the prior Pn- However, argue that this case much less relevant from the risk perspective: even if the probability of other outcomes s' becomes less likely, "mechanically" increasing the probability of the secret s, the probability of the secret is still bounded by the right-hand side of Equation equation 2 Furthermore, we argue that negative membership inference, i.e., predicting that sample was not part not as concerning as positive membership inference. Indeed, an individual can be present in data collection pool, but not included in particular training set for a variety of reasons, hence absence from the training set does not imply absence from the data collection stage.
training set, is
Length of the Secret. With a slight abuse of notation, let us consider the leakage (in bits) as a-1 function of the number secret bits (b) - min, da a log(2] + - Our leakage function - (z2 satisfies two properties: it is non-decreasing and concave (see Appendix A.2 for proof). Thus. longer secrets will leak more in absolute terms, but less in relative terms. Let assume that we are looking at particular secret of binary nature (such as whether some individual owns particular object for example), with prior Po 2-b. Even though there are two possible outcomes, the prior po is not necessarily equal to 1/2: if the item is rare, the prior is more likely to be smaller. The log- posterior (P1) < bb + (b). non- ienriccceasing function (see Appendix A.2 A.2 for proof). This upper-bound maximized when b 0: longer secrets lead to smaller values of P1- The most sensitive secrets the ones that are the most likely in the first place, and the length of the secret itself acts as a protection against reconstruction attacks.
Comparison to membership inference. In particular, membership inference attacks correspond to attacks with a low number of secret bits. These attacks are usually conducted with a prior probability of 1/2 Yeom et al. 2018 Sablayrolles et al. 2019), and even though some works consider different
ratios of positive to negative members (Watson et al. 2021 Rezaei & Liu 2021), 1) they stay within a factor of 1 to 10. Some privacy settings (noise level o, sampling rate and number of steps) will thus offer no guarantee against membership inference (because of the low number of bits to guess) but still will not allow for full reconstruction of the rarest samples (high number of bits).
5

---

Under review as a conference paper at ICLR 2023
3.2 LAZY SAMPLING: IDENTIFYING SAMPLES LIKELY TO LEAK
Let assume that model A(O). The method of Carlini et al. bb requires sampling hundreds of thousands of times from A(0) in the hope that we find the secret (and then filter out the generated samples using calibrated membership inference scores). Fortunately, most attacks of Carlini et al. (2021b) are amenable to lazy sampling: we can take a secret s, and directly compute its probability p P(A(()) s).
to generate secret & from model 0, using the probabilisttic attack
Not all probabilisticc processes have tractable density P(A(() s). For instance, generating sequence .......: .T and only keeping the last samples Fi; .., 2r does not have a tractable density. On the other hand, vanilla sampling from language model can be done lazily because the probability of sequence 21, .i....,. JT can be expressed as [1 fe, æ1s) where 21 (z1...., ,T..,,,,,;,1))... 11-1) straightforward for regular sampling, it is also true for temperature and top-k sampling, with the caveat that these probabilities can sometimes be 0, as illustrated in Appendix A..]
and fa, F1:0) is the conditional probability of æ; given the past. While this is
We define T (0, 21:s) the set of top-k predictions and Bı...., .r set of temperatures. From there, we can define top-k and/or temperature language model as
fa( Ever(on & if E Te(11a)
X(x (ivx) A
M(II,...,TT) = TT M(a; Fis)-
otherwise.
Lazy sampling allows analyze two of the main strategies of Carlini et al. (2021b), but does not apply to the Internet sampling one. Indeed, Internet sampling first chooses a text e crawled from the public web, and generates iteratively from it fo(,c), effectively using c prompt.
Leakage approximation. In the remainder of this paper (except for Section make the following assumption:
Assumption 1. For any 0 M(D) and I' M(D).
log ((A(0) s) log P(A((') s).
To support Assumption I. measured the mean and standard deviation of the log- probbabbiyl of the secret log(p1) for our n-gram model detailed in Section 4.1 For each level of DP noise a., we consider 10,000 models and observe the probability of leakage is tightly concentrated around the mean. For instance, with a = 2.875, log pi has mean & 22.5 and standard deviation = 1.5, from which conclude that the log-probability of secret does not vary much when re-training model on a fixed dataset D.
Using Assumption and lazy sampling, can take all sentences Ii....,,77 FT from the training set, and compute their probability of being generated by the attack A, and compare it to their "score" that used the filtering stage. The above strategy allows us identify samples that would be reconstructed using this particular attack. Of course, if this particular attack fails to reconstruct some sample 8, does not mean that all attacks would fail on s.
4 EXPERIMENTS
We first experiment with simple private n-gram language model to empirically compare the secret leakage with our bounds. We then consider non-private language model that exhibits samples that are the most at risk for a reconstruction attack using the lazy strategy for the Carlini et al. (2021b) attack. Finally, low perplexity (Li et al. 202I), and show that the empirical leakage is even smaller than predicted by our bound. Overall our experiments required in the order of 200 GPU-days V100 GPUs.
experiment private fine-tuning of GPT-2, with parameters yielding

---

Under review as a conference paper at ICLR 2023
Our bound
70
15
Empirical leakage
4=0.01
Canary secrncy (bits)
50
q=0.02
Leakage (bits)
50
q=0.05
10-
q=0.1
40
q=0.2
30
sigma-0.3
5
sigma-D.4
20
sigma-0.5
sigma=1.0
10:
Our bound
6
8
50K
100K
150K
200K
250K
Noise multiplier
Training Iterations
Figure 3: Left: Empirical leakage log (P1/Po) computed using Equation 4 on 7-gram language models, compared our theoretical bound L(po) from Corollary | We can see that our bound is tight. Right: We empirically measure the leakage of our canary (a fixed 16-digits random number) when training our GPT-2 model on Wikitext with DP. If the model predicts the canary with a given probability p1, compute the secrecy b loga /P1) (y-axis, in bits). The blue curve corresponds to our bound for a = and canary sampling rate q = 1%. Empirically, the observed privacy is more than predicted, even for lower levels of DP noise or To make our case stronger, measure canary secrecy with increasing canary sampling rates and show that our bound verified.
4.1 2-GRAM LANGUAGE MODEL
We first experiment on a n-gram language model to compare our guarantees against empirical secret leakage. We consider T > 0 time steps and generate random digits E {0, 1....,9) 9}" (the canary). Our objective is to train the n-gram language model on the (fixed) canary only.
A full n-gram language model fe has ΣT= , 10* ze 10" parameters. However, the only parameters of fo(ci(o1:)). These parameters suffice to compute
interest in our case are the ones corresponding the probability of the sequence according to the model fe. and the rest of the parameters correspond other "branches" of the language model. Hence, we can train fe lazily by only modifying these parameters, which brings the number of parameters down to 10T. We train our model with the softmax loss. More specifically, our loss writes
e
(( =
--00=- E log(Ur,c.): t=0
D-1 2d=0 e0,.
where o is the digit at index in the canary and Ue,; the probability that digit appears at position t.
We experiment with Opacus Youseepour et al.) (021). using randomly generated and fixed canary of length
10. We set the sampling rate to g 1, the clipping factor C = 1, the DP noise level E [0.5. 10) and learning rate ' = 0. We now wish to measure empirically the leakage log (PI/Po). We have P1 P(A(M(D')) = 8) ((eeM((")(((()))) We approximate Pı using Monte-Carlo sampling, by training N 10,000 models and computing
N
log(p:) ) z log
Σ
(4)
N
Given that the probabilities faces (c) can be quite small, we perform computations in the log-space for numerical stability. We have Po = 10-T since the problem is invariant under permutation of digits.
Finally, we use the RDPAccountant from Opacus compute d, for a range of orders [1.01, 63) and compute (pp)) as in Equation equation 3 The results are shown in Figure where the empirical leakage refers to log (P1 /po) and where our theoretical bound refers to [(po). We observe that the bound is relatively tight for this simple n-gram language model.
4.2 LAZY RECONSTRUCTION WITHOUT DEFENSES
We now consider a vanilla target model trained without DP on the OpenWeb dataset (Gokaslan & Cohen 2019) and want to identify samples from the training that most at risk in the event
7

---

Under review as a conference paper at ICLR 2023
0.6
train
2.5
0.5
test
Calibrated loss
0.0
Density
0.4
0.3
-5.0
0.2
-7.5
train test
0.1
-00.0
generate
0.0
5 -1255
-15
-10
55
-600
400
-200
0
Callbrated loss
Log beg-probabilityy
Figure 4: We train a vanilla language model without DP and compute, for given sentence, its calibrated score using a model that is not trained on the same data, as well as its prg-prbbbbylity predicted by the target model. Left: We plot the histogram of a random subset of the train and test sets. Right: For a given sentence, we plot its calibrated loss (y-axis) against its log- eigpobbbaylity of being generated by the target model (z-axis). In addition to sentences from the train and test set, we generate sentences from the target model and note that their average calibrated loss is lower than for sentences of the training set. The samples from the train set that are the most at-risk for a reconstruction attack are these with high log-probability (high probability of being generated by the attack model) and low calibrated loss (high probability of having been memorized by the target model). See Table forr selected sentences from the training set.
of' a reconstruction attack. We compute, for a given sentence z, its calibrated score (or calibrated loss) defined as 6(0, 2) [(0, x)- L(0g. z).. where 0 represents the target model's weights and lg represents a calibration model that is trained on held-out calibration data. We partition OpenWeb WebText Text into training, calibration and test sets of million samples each. The models are trained using the setup of Radford et al. (2019). As shown in Figure on the left plot, we observe that samples from the training set have a lower calibrated loss, indicating that they have been memorized by the model. As expected, test samples have an average calibrated loss around 0 with a small standard deviation. On the right plot, we observe that the average calibrated loss for sentences generated by the target model is lower than for sentences from the training set. Given these measures, it is possible to identify samples that are most at risk due to a reconstruction attack: those are with high log er--rbbability (higher odds of being generated by the target model) and low calibrated loss (high probability of being memorized by the target model).
Table T shows typical examples of elements from the training set with various levels of risk. Some of the sentences likely to be generated are indeed common (such as the two first rows). Other sentences however are very likely to sampled (with a probability close to 10-3) and have low calibration score, indicating they were memorized by the model. Finally, some sentences have a very low probability of being generated in the first place (less than in 10 billions), so even though they are memorized by the model (low calibration score), they are unlikely to be surfaced. Carlini et al.) (2021b) showed a somewhat analogous plot by displaying perplexity of the trained model against zlib entropy. There are two major differences: our r-axis shows the probability of being generated by the attack model (which is different from perplexity due to temperature and/or top-k sampling), and our analysis is conducted on the training set, whereas their analysis is done generated samples. In particular, we can have confidence that certain sentences with a very low probability of being generated will not be recovered by this particular strategy.
4.3 REAL-LIFE CANARY EXPERIMENTS
We also experiment with realistic scenario of privately fine-tuning a large language model. We take a pre-trained GPT-2 model (Wolf et al. 2020) and fine-tune it on Wikitext-103 (Merity et al. 2017)) available under the Creative Commons the training dataset and sample it during training with a sampling rate q. The canary consists of a prompt ("John Doe's credit card number is") and secret, which is string of 16 random digits. We are able to privately fine-tune GPT-2 with DP-SGD on Wikitext- 103 and reach perplexity of 45, which is very close to the non-private fine-tuning perplexity of 38 (refer Appendix A.6 for details).
License. We add canary sentence to
8

---

Under review as a conference paper at ICLR 2023
Table 1: We train a vanilla target model without DP on Wikitext-103 and display selected training sentences. A low calibrated loss denotes sample that is likely to be memorized and a high probability denotes a sample that is more likely to be generated by the model. Hence, samples associated with a low calibrated loss and with a high generation probability are considered at risk. Note that 11.3 is quite low, as most generated samples will have a calibrated loss higher than - 7.5 as Figure 1
Sample
Probability
Calibration
Advertisement
10-2.5
+0.10
Content created by The Daily Caller News Foundation is available [...]
10
-0.18
Subject: Games Day News and Rumours: Non Dark Eldar
10-3.6
-9.81
Antarctic expeditioners honour traditional solstice swim Homepage image courtesy of the Canadian Space Agency
10-10 10
-1.33
10-21
+1.29
Given the trained model, we approximate P1 by computing the probability of the secret given the prompt P1 fo (number) "John Doe's credit card number is"). With the chosen privacy parameters (a E [0.3, 0.5]. 9 = 2.81 x 10-4 and 186k steps), the empirical leakage is negligible. To make our case stronger, we increase the sampling rate of the canary in order to increase the empirical leakage, and observe that it is still lower than the proposed bound of Corollary []
Figure 3 shows the empirical log probability of the secret as a function of the number of steps against the leakage guarantee computed according to Equation equation 3 With DP, decreasing levels of the noise a lead increasing levels of leakage (or equivalently, decreasing levels of privacy). Similarly, increasing canary sampling rates q result in more observed empirical leakage. We observe that even for the most favorable setup to canary memorization (low sigma, high sampling rate), the provided bound (computed for high sigma and low sampling rate) is verified. Finally, even with our least private version (a 0.3, q = 0.2), our strategy is not able to fully reconstruct the secret.
Results.
Exposure. Carlini et al. (019) estimate the performance of their attack using the exposure metric, which is an approximation of the optimal number of guesses an adversary would need to correctly estimate secret. In practice, we can only upper-bound the exposure because we have guarantee that any attack is optimal. The canary secrecy showed in Figure corresponds to exposure, but for the more recent reconstruction attacks of Carlini et al. (2021b).
4.4 LIMITS OF RECONSTRUCTION ATTACKS
Canary attacks (Carlini et al. 2019) are useful because the secret is chosen randomly and is thus independent of the rest of the dataset. In contrast, with practical reconstruction attacks is difficult to estimate the randomness of the secret. Indeed, multiple identical sequences can be present in the dataset. [arliii et al. (2021b) correct for this by measuring k-eidetic memorization, i.e., looking at sentences that appear k times or less in the dataset. However, this does not account for knowledge shared across secrets. For example, if there is secret in the form "[animal] drinks [beverage]", animal and beverage can vary in the dataset, so the language model will Iearn about this general structure during training: seeing "dog drinks water" can thus make "cat drinks milk" more likely, even if the latter does not appear in the dataset.
5 CONCLUSION
This work shows that Rényi Differential Privacy (DP-SGD) provides meaningful guarantees against re- construction attacks. We also provide an efficient way to analyze the vulnerability of training samples to the reconstruction attack Carlini et al. (2021b). Overall, the combination of our improved guaran- tees with the private fine-tuning of language models showcased by [i et al.) (201)) allow us to train lan- guage models with perplexity competitive with the state of the art and meaningful privacy guarantees against training data extraction. Finally, our work sheds light on the "higher information" end of the spectrum. First, reconstruction attacks are more credible because they only require access a trained model and not to candidate samples. Second, as shown, reconstruction attacks can be defeated with levels of noise that would fail to defend against membership inference attacks. We hope that increased consideration for this threat model will drive adoption of DP-SGD as a standard in machine learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security (CCS), pp. 308-318, 2016.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and Rényi differential privacy. In International Conference on Artificial Intelligence and Statistics, pp. 2496-2506. PMLR, 2020.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th Annual Symposium on Foundations of Computer Science (FOCS), pp. 464 473. IEEE, 2014.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. 28th USENIX Security Symposium, pp. 267-2844 2019.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. arXiv preprint arXiv:2112.03570, 2021a.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pp- 2633-2650, 2021b.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., pp. 211-407, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference Theory of Cryptography, pp- 265-284, 2006.
Tony A Ginart, Laurens van der Maaten, James Zou, and Chuan Guo. Submix: Practical private pre-
diction for large-scale language models, 2022. URL id=cKTBRHVVjy
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:///ky.ion0077gittu io/
@penkeebeextorppus 2019.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muchling, John Pearson, Dietrich Stephan, Stanley F Nelson, and David Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS genetics, 4(8):e100016 2008
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be
strong differentially private learners, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2018.
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France. April 26-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https //openrev www.wewwwwwwww...eeeeeeeeeem
net./ forum? d 72udxe
10

---

Under review as a conference paper at ICLR 2023
Ily Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the Sampled Gaussian Mechanism. arXiv preprint orXiv:1908... 10530, 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium on Security and Privacy (S&P), pp. 739-753, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, Language models unsupervised multitask learners. OpenAl blog, 188):9 2019.
Shahbaz Rezaci and Xin Liu. On the difficulty of membership inference attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7892-7900, 2021.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. International Conference on Machine Learning, pp. 5558-5567. PMLR, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv: 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (S&P), pp. 3-18, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with dif- ferentially private updates. In IEEE Global Conference Signal and Information Processing (GlobalSiP), pp. 245-248, 2013.
Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2 1..040,1 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https:: /tt::///wwwwo www. ac aclweb web. orq/ http///athoooogy2222200 2020 -omnlpddemoss 6.
Jiyyun Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. arXiv preprint arXi::2111.09679 2021.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 3 Computer Security Foundations Symposium (CSF). pp. 268-283 IEEE, 2018
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in py Torch, 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang Diffcrentially private fine-tuning of language models, 2021.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROBABILITY PRESERVATION
Here we show the steps to get from Eq. Eq. 2
exp( Po a/ (a-1)
S pi (exp(d.))o) (a-1)/aa
da + 0:((a= 1) log(Po) < log(p)) s ((a - (d/ + log po)
dd, 1/(a 1) log(po) I log(p1 /po) S (a 1)da/a //llogpp po-
A.2 LEAKAGE FUNCTION
In this part, prove that the leakage function Lg is non-decreasing and concave. We also show that : b. + L2(b) is non-increasing. We start with two technical lemmas.
Lemma 1. Given a family of non-decreasing functions fr,t T the function f(x) inf, fz( is non-decreasing.
Proof. For x < and some E T, we have
f(x) - inf (x) S fe(x)
S fe(U)
Since f(x) S fely) for all t, we have f(x) & inf, fe(y) flv).
Lemma Given family of concave functions fú, t T the function fx) inf, f((x) is concave.
Proof. For function g. define its hypograph H(g) A ((x.y) y S g())). A function g is concave iff its hypograph (g) is convex set. Each function fe being concave, its hypograph (ff)) convex set. The hypograph H(f) OtE (f.) is an intersection of convex sets and is thus itself convex set. Thus f is concave.
Now let apply these lemmas to the leakage function. We have L2(b) min, (bb with each (b) = de, Tog[2]a å Each function Ja is linear, and hence concave, and non-decreasing because 1/a > We can thus apply Lemma 2 and Lemmal to conclude that the leakage function L2 concave and non-decreasing.
The function ç b + Lz(b) is a sum concave and linear function and is thus concave. Thus, its derivative 36 is non non-increassin. Given that 36 (0) non-increasing.
0La
2b (0) - 0. ac < and thus & is
A.3
COMPARISON TO BALLE ET AL. 22200
In Figure 5
we display the numerical bounds of Balle et al. (2020).
n-0.4, 2-212 6-3e-07
0=0.5, x=73, 5=3e07
6.3.8,
(blt))
30
30
Leakoge (btes)
30
20
Lastoge
sound Bale
10
10
diadssure
10
20
30
40
10
20
30
40
0
10
20
30
40
Secret length (bits)
Secret length (bits)
Secret length (bits)
Figure Comparison with Balle al. (2020).
Figure 6 furthers the comparison Balle with different value of the sampling rate q-
12

---

Under review as a conference paper at ICLR 2023
0=0.9, g=49.9, 8510000-5
0=1.2, <=29.7, 6=1.00e-05
100
60
80
40
20
20
01
0-
20
40
length (bits)
length (bits)
1.8, 6=15.8, 6=1.00e-05
6=1.00e-4
20
10
0
10
20
30
40
0
10
20
30
40
Secret length (bits)
Secret
Figure 6: Comparison with Balle et al. (2020), on DP-SGD run with 20% and 100 epochs, and various values Ø.
Percentage leaked from secrets of various lengths
1.0
0.8
0.6
0.4
bits
0.2
20 bits
50 bits
0.0
0
10
20
30
40
at 6-1.00--5
Figure Percentage of secrets leaked as function of privacy budget.
A.4 COMPARISON OF MEMBERSHIP INFERENCE AND RECONSTRUCTION PRIVACY
Figure the percentage of the secret leaked function of the privacy budget. This emphasizes that small, binary secrets leak completely for budgets < 10, but longer secrets do not completely leak until much higher budgets (typically around 50 for 20-bit secrets).
A.5 TOP-K SAMPLING
we show that Top-k sampling from a language model has a tractable density. The probability of sequence is the product of the conditional probabilities along the path. If one word does not belong to the top-k predictions, its probability will be 0, thus making the probability of the entire sentence
In Figure g
A.6 EXPERIMENTAL DETAILS
For WikiText- 103, we follow the findings of Li al. (202T) and use a large batch size (B = 1024), a small clipping threshold (C 1), AdamW (Loshchilov & Hutter 2018) and freeze the embedding
13

---

Under review as a conference paper at ICLR 2023
a
0.06
2
0.39
the
The secret is
0,001
...
given
Figure 8: Top-k sampling from a language model has a tractable density.
layers. We also find that using low learning rate (Ir 0.0001) is crucial avoid divergence. With these hyperparameters, we are able to fine-tune GPT-2 on Wikitext- 103 and reach a perplexity of 45, which is very close to the non-prine non-private fine-tuning perplexity of that obtain. These results echo the findings of Li et al. (2021).
14


Reviews:
[{"review_id": "ZiwxhW1X10n", "reviewer": "Reviewer_ruMF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work examines whether the secret information that may be contained in the training data set can be extracted from a model which satisfies RDP. This problem is positioned as a problem of intermediate difficulty between the reconstruction and membership attacks. The authors define leakage as the probability that an attacker identifies a secret in a model learned from data that contains the secret and in a model learned from data that does not. Through analysis, the authors obtain upper bounds on the log ratio of the two and compare them to empirical evaluations.", "summary_of_the_review": "Overall, the manuscript considers an interesting problem.\nExperimental results are interesting, while the characterization of theoretical results seems to e a bit weak.", "strength_and_weaknesses": "Strength\n- Considered an interesting and novel problem\n- The considered problem well fits privacy that should be considered in the real world\n\nWeakness\n- Characterization of the obtained result seems to be a bit weak\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BC0BBdnmcj", "reviewer": "Reviewer_2RkH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an improved theoretical guarantee for the defense against reconstruction attacks using Renyi differential privacy. The correctness of the theoretical analysis is supported by the experimental analysis", "summary_of_the_review": "I am not an expert on NLP, and all my judgments are based on the differential privacy part of this paper. \n\nAs mentioned in the Strength And Weaknesses, some details of this paper are not clear and there might be some technical issues. The presentation can also be improved.", "strength_and_weaknesses": "Strength: The theoretical bound in this paper is much better than the state of the art when \\sigma is small (e.g., 0.4). This makes the bound useful for application scenarios with moderate privacy requirements (but may with strong accuracy requirements)\n\nWeakness:\n1. The theoretical bound in this paper is just slightly better than the state of the art when \\sigma becomes larger (e.g., 0.5), which means the result is similar to the state of the art under usual privacy requirements.\n\n2. There is no comparison with Balle et al. when \\sigma > 0.6. I am not sure whether the proposed algorithm is better or worse than Bella et al. in the standard range of noise (1 < \\epsilon < 3).\n\n3. I think this paper can be improved by providing an experiment about the accuracy of different models under different privacy guarantees/noise levels (I am not an expert on NLP, please let me know if some experiments in the paper already analyzed some metrics similar to accuracy)\n\n4. I suggest the authors improve the readability by adding a statement that Renyi DP-SGD is implemented in the same way as DP-SGD. Otherwise, it's hard for the readers to know how Renyi DP is achieved for SGD.\n\n5. I would also suggest discussing other applications for Renyi DP. For example, improving robustness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AYk6vILx7d", "reviewer": "Reviewer_UcP3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a privacy protection method against training data extraction attacks from language model based on Rényi differential privacy. The authors point out the effect of sample length on privacy leakage and derive privacy bound based on Rényi differential privacy, and then propose an improved method based on lazy sampling to identify training samples that may leak privacy. The authors construct corresponding experiments for each part of the proposed method by fine-tuning the language model and show that the privacy leakage under the proposed attack model is less than the privacy guarantee. ", "summary_of_the_review": "This paper proposes a privacy protection method against training data extraction attacks from language model based on Rényi differential privacy. However, this work lacks some innovation for ICLR and the content in each section is not easy to follow and the organization can be improved. ", "strength_and_weaknesses": "Strength\n1. This paper considers the sample length and derives the privacy leakage boundary of training samples based on the Rényi differential privacy method, which seems to be reasonable and effective.\n2. The authors propose a lazy-sampling-based method to improve the existing method to perform training data extraction attacks, which improves the performing efficiency and effectively identifies potentially private samples. \n3. The organization and the results of the experiments in this paper appear to be efficient and reasonable. \n\nWeaknesses\n1. First, this article is difficult to read for me. I suggest that authors should be more careful with domain words and notations, such as “reconstruction attack”. In my background knowledge, a \"reconstruction attack\" in federated learning or graph learning for privacy-preserving is to reconstruct the training data using information (gradient or aggregated information) from the model training in the training process. However, I think that the \"reconstruction attack\" in this paper is an attack method in the inference process just like the member inference attack. I am confused if the “reconstruction attack” in this paper is the “training data extraction attack”. If it is a new attack, I would expect the author to redefine it.\n2. In my opinion, the core idea of this paper is to use differential privacy to achieve that the generated samples using the random attack model are satisfying the privacy guarantee. Although the proposed method is effective and reasonable, it is incremental work and lacks innovation. \n3. I think the organization of this paper needs improving. The framework of this paper looks fine, important sections are included and organized in a reasonable way. However, the content in each section is not easy to follow, and the organization of each section can be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject