MoPe: Model Perturbation based Privacy Attacks on Language Models
Authors: Marvin Li
Jason Wang
Jeffrey George Wang
Seth Neel
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
membership inference
data extraction
TL;DR: We demonstrate new membership inference and data extraction attacks based on model perturbation on a suite of LLMs.

Abstract: Recent work has shown that Large Language Models (LLMs) can unintentionally leak sensitive information present in their training data. In this paper, we present  Model Perturbations (MoPe), a new method to identify with high confidence if a given text is in the training data of a pre-trained language model, given white-box access to the models parameters. MoPe adds noise  to the model in parameter space and measures the drop in log-likelihood at a given point $x$, a statistic we show approximates the trace of the Hessian matrix with respect to model parameters. Across language models ranging from $70$M to $12$B parameters, we show that MoPe is more effective than existing loss-based attacks and recently proposed perturbation-based methods. We also examine the role of training point order and model size in attack success, and empirically demonstrate that MoPe accurately approximate the trace of the Hessian in practice. Our results show that the loss of a point alone is insufficient to determine extractability---there are training points we can recover using our method that have average loss. This casts some doubt on prior works that use the loss of a point as evidence of memorization or unlearning.

MoPe @
Model Perturbation-based Privacy Attacks on Language Models
Marvin Li
Jason Wang"
Jeffrey Wang
Seth Neel Harvard University
Harvard College
Harvard College
Harvard College
Abstract
(Soper, 2023). While early state of the art LLMs have been largely trained on public web data (Radford et al., 2019; Gao et al., 2021; Biderman et al., 2023b; Black et al., 2021), increasingly models are being fine-tuned on data more relevant to their intended domain, or even trained from scratch on this domain specific data. In addition to increased performance range of tasks, training models from scratch is attractive to companies because early work has shown can mitigate some of the undesirable behavior associated with LLMs such as hallucination (Ji et al., 2022), toxicity (Gehman et al., 2020), as well as copyright issues that may arise from mimicking the training data (Franceschelli and Musolesi, 2021; Vyas et al., 2023). For example, BloombergGPT (Wu et al., 2023) is 50-billion parameter auto-regressive language model that was trained from scratch on financial data sources, and exhibits superior performance on tasks in financial NLP.
Recent work has shown that Large Language Models (LLMs) sensitive information present in their training data. this paper, we present MoPeg (Model Perturbations), a new method identify with high confidence if a given text is in the training data of a pre-trained language model, given white-box access the models parameters. MoPeg adds noise to the model in parameter space and measures the drop in log-likelihood at given point т, statistic we show approximates the trace the Hessian matrix with respect model parameters. Across language models ranging from 70M to 12B parameters, show that MoPeş is more effective than existing loss-based attacks and recently proposed perturbation- based methods. We also examine the role of training point order and model size in attack success, and empirically demonstrate that MoPe, accurately approximate the trace of the Hessian in practice. Our results show that the loss of a point alone is insufficient to determine extractability- training points method that have average loss. This casts some doubt on prior works that use the loss of point evidence of memorization or "unlcarning."
unintentionally leak
While all of this progress seems set to usher in an era where companics deploy custom LLMs trained on their proprietary customer data, one the main technical barriers that remains is privacy. Recent work has shown tha language model's have the potential to memorize significant swathes of their training data, which means that they can regurgitate potentially private information present in their training data if deployed end users (Carlini et al., 2021; Jagannatha et al., 2021). LLMs are trained in a single pass and do not "overfit" to the same extent as per-pparmetterized models, but on specific outlier points in the training set they do have loss much lower than if the point had not been included during training (Carlini et al., 2021), )), allowing adversary perform what is called membership inference attack (Shokri et al., 2017): given access to the model and a candidate sample z', the adversary can determine with high-accuracy if a' is in the training set.
recover using
1 Introduction
Over the last few years, Large Language Models or LLMs have set new standards in performance across range of tasks in natural language understanding and generation, often with very limited supervision on the task at hand (Brown et al., 2020). As result, opportunities to use these models in real-world applications proliferate, and companies are rushing deploy them in applications as diverse as A.I. assisted clinical diagnoses (Sharma et al., 2023), NLP tasks in finance (Wu et al., 2023), or an A.I. "love coach"
'Alphabetical order; equal contribution.
Senior author, email: sneel @ cdu for correspondence.
While prior work shows privacy is a real

---

concern when deploying language models, the membership inference attacks used can extract specific points but perform quite poorly on average, leaving significant room for improvement (Yu et al., 2023; Carlini et al., 2021). At the same time, recent works studying data deletion from pre-trained LLMs (Jang et al., 2022) and studying memorization (Carlini et al., 2023a) cite the loss of point determinant of whether that point has been "deleted" from the model, or conversely is "memorized." Given that loss of point a relatively poor indicator of training set membership for LLMs, this raises a series of tantalizing research questions we address in this work: (i) If a point has average "loss" with respect to language model 6, does that imply it cannot be detected as a training point? (ii) Can we develop strong MIAs against LLMs better than existing loss-based attacks?
2023b) on models that range in size from 70M 12B. Our experimental results in Scction 3 show that compared to existing attacks based on the loss (LOSSp), MoPe, boasts significantly higher AUC on all model sizes up to 2.8B, and comparable AUC at 6.9B and 12B parameters. Furthermore, at low FPRs of the attack, MoPe, maintains high true positive rates, whereas LOSS, fails outperform the random baseline (Figure 2).
We evaluate whether the DetectGPT statistic proposed for detecting LLM-ggenrraed text be repurposed privacy attack against pre-trained models. Our experimental results show that DetectGPT outperforms the loss- based attack at all model sizes up to 2.8B, but dominated by our MoPeş attack at all model sizes.
We develop new membership inference attack we call MoPeg, based on the idea that when the model loss is localized around a training point, is likely to lie in a "sharper" local minima than if the point was a test point- which does not necessarily imply that the absolute level of the loss is low. Concretely, our attack perturbs the model with mean zero noise, and computes the resulting increase in log loss at candidate point (see Figure 1).
Since all the models we evaluate were trained on the same datasets in the same training order, we can make a principled study the relationship between model sizes and order of the training points to attack success. Surprisingly, we find in Table and Figure 7 that consistent with prior work LOSS, attack success increases with model size and proximity during training, but that these trends fail to hold for MoPeg-
Ltrain
text
LOSSa
LOOSa
Finally in Section 4 we train a network on MNIST (Deng, 2012) that is sufficiently small that we are able to compute the Hessian. We use this network to show that our MoPe, statistic does in practice approximate the Hessian trace. We also find that while MoPea does outperform the random baseline, the attack performs worse than loss in this setting, highlighting that our results may be unique to language models.
YoPea
0
0
Figure 1: MoPe, detects training point membership by comparing the increase in loss on the training point when the model is perturbed, relative to the increase in loss of the perturbed model on a test point.
Contributions. In this paper we make several contributions to the developing story on language model privacy.
These results establish our technique as state of the art for MIA against pre-trained LLMs the white-box setting where we can access the model weights. They also challenge conventional wisdom that the loss of a point is in isolation a sufficently good proxy for training set membership.
We show in Section 2.2 that our MoPe, statistic is approximating the trace of the Hessian matrix with respect to model weights 0 at (0, x), and can be seen approximation of how "sharp" the loss landscape is around 0.
Related Work
We evaluate MoPea on the recently developed Pythia suite from El (Biderman et al.,
DetectGPT. The most closely related work to ours on a technical level is the recent paper

---

(Mitchell al., 2023) who propose a perturbation- based method for detecting LLM-generated text using probability curvature. Their method DetectGPT compares the log probabilities of a candidate passage x and its randomly perturbed versions m(x) using the source model and another generic pre-trained mask-filling model m; large drops in the log probability correspond to training points. While superficially similar to our MoPey method, there are significant differences: (i) We focus on the problem of membership inference which tries to determine an example x was present in the training data, whereas their focus is determining if x was generated by 0-an orthogonal concern from privacy and (ii) As result, we perturb the model 0 using Gaussian noise, rather than perturbing using additional model.
an adversary tries to use model access as well some outside information determine whether a candidate point x is a member of the model training set. Since (Yeom et al., 2018), MIAs typically exploit the intuition that a point x is more likely to be a training point if the loss of the model evaluated at z is small, although other approaches that rely (Cohen and Giryes, 2022) distance to the model boundary (Pawelezyk al., 2022; Choquette- Choo et al., 2020) have been proposed. (Sablayrolles et al., 2019) argue (under strong assumptions) that the loss tessshhesholding attack is approximately optimal by the Neyman-Pearson lemma. The assumptions needed to show optimality do not hold in practice, however, which opens the door to substantially improving this simple attack. State of the art membership inference attacks still rely on loss thsesshresh or functions of the loss (Ye et al., 2021; Carlini et al., 2022; Sablayrolles et al., 2019), but try to calibrate the threshold at which a point x is declared a training point to account for the component of the loss that is specific to the example x. One way they do this is by (i) Training shadow models that do/don't contain the candidate point x, and (ii) Using the loss of these models on the point approximate the ratio of the likelihood of the observed loss E(0, æ) given x was in the training set to the likelihood if x was not (Carlini et al., 2022). A simpler, but related attack assumes access to reference model ōref that was not trained on the point x: in question, and thresholds based on the ratio log(E(x, 0)/ Oreg)) (Carlini et al., 2021). In this paper, we make the realistic assumption that the adversary does not have access to the computational resources to train additional language models from scratch on fresh data, or have access to a reference model that has been trained without the candidate point x. We note that in the case where this was possible, these same threshold calibration techniques could be used to accelerate the effectiveness of our MoPea attack well.
notion of self-influence
Memorization and Forgetting in Language Models. There are many recent works studying issues of memorization or forgetting in language models (Biderman et al., 2023a; Thakkar et al.. 2020; Kharitonov et al., 2021; Zhang et al.,
2021: Ippolito et al., 2023). (Jagielski et al.,
2023) measures the forgetting of memorized examples over time during training, and shows that in practice points occurring carlier during training observe strönger privacy guarantees. They measure "forgetting" using membership inference attack accuracy, using an attack based on loss- thresholding with an example-calibrated threshold. By contrast (Biderman et al., 2023b) studies memorization using the Pythia suite, and finds that the location of a sequence in training does memorized,
empirically
not impact whether the point indicating that memorized points are not being forgotten over time. This finding is further confirmed in (Biderman et al., 2023a), which shows that memorization by the final model can be predicted by checking for memorization in earlier model checkpoints. Our work sheds some light on the apparent discrepancy between these two works, indicating that results in (Biderman al., 2023a) are possible because loss value alone does not necessarily imply a point is not memorized,
MIAS on LLMs. Concurrent work proposes several new MIAs tailored to LLMs. (Abascal et al., 2023) investigates MIAs where the adversary has access only a fine-tuned model, and tries to make inferences back the pre-trained model. Their method operates in a different access model than ours, and more importantly relies heavily on the
Membership Inference Attacks (MIAs). Membership Inference Attacks or MIAs were defined by (Homer al., 2008) for genomic data, and formulated by (Shokri et al., 2017) against ML models. In membership inference,

---

training of additional shadow models, and therefore does not scale to the large model sizes studied here. (Mattern et al., 2023) use a statistic similar to that used in DetectGPT (Mitchell et al., 2023) for detecting model-generated text for membership inference, in that they perturb the example using mask-filling model (BERT) and use the resulting losses to calibrate the threshold. They conduct their experiments on the 117M parameter version of GPT-2, which they then fine-tune AG News and Twitter Data. As such their results can also be viewed as focused on small models in the fine- tuning setting, where it is known that much stronger attacks are possible (Jagielski et al.. 2022; Abascal et al., 2023), in contrast to the pre-training regime studied here.
point (xr C). When equipped with a threshold т, (M, define a canonical membership inference attack: (i) With probability 1 sample random
point x from C, else sample a random point x ~ D. (ii) Use M to predict whether x E or x ~ D by computing MM(z, 0) and thresholding with T:
TRAIN
if M (2,
(M)
NOT TRAIN if (E,
(1)
Note that by construction the marginal distribution of a is DD whether x is a training point or not, and so if M has accuracy above 1 must be through leaking information about x. The "random baseline" attack (M, r) that samples M(x, 0) ~ Uniform(0, 1] and thresholds by - - 7 E [0, has TPR and FPR T (where training points are considered positive). The most commonly used membership inference attack, introduced in (Yeom et al., 201 (0, takes M(x, 0) {(x,, 0), and predicts points are training points if their loss is less than -T, or equivalently the confidence is greater than т. We refer to this attack throughout as the loss attack, or LOSSg. Throughout the paper, as is common we overload notation and refer to M as a membership inference attack rather than specifying a specific threshold.
2
Discussion of additional related work focused on privacy issues in language models, including training data extraction, data deletion or machine unlcarning, and mitigation strategies via differential privacy, are deferred to Section A of the Appendix.
2.1 Preliminaries
An auto-regressive language model denoted by 0 takes as input a sequence of tokens x 11,22...77 and outputs a distribution (-(x))) C (VV) over a vocabulary V. the space of tokens.
Given sequence of tokens the loss is defined as the negative log- likeelholhooo of the sequence with respect to 0:
MIA Evaluation Metrics. There is some disagreement as to the proper way to evaluate MIAs. Earlier papers simply report the best possible accuracy over all thresholds (Shokri et al., 2017). Different values of correspond tradeoffs between the FPR and TPR of the attack, and so more recently metrics like Area Under the ROC Curve (AUC) have found favor (Carlini et al., 2022; Ye et al., 2021). Given that the ability to extract a very small subset of the training data with very high confidence is an obvious privacy risk, these recent papers also advocate reporting the TPR at low FPRs, and reporting the full ROC Curve in order to compare attacks. In order evaluate our attacks we report all of these metries: AUC, TPR at fixed FPRs of .22 and .05, and the full ROC curves.
T
(o)) log(po(zs(
Alternatively, flipping the sign gives the confidence con) = (z)) Given a training corpus C sampled i.i.d from a distribution 'D over sequences of tokens, 0 is trained to minimize the average loss, I C ming Exec (z) referred to as "pre-training" on the task of next token prediction, to differentiate it from "fine-tuning" for other downstream tasks which occurs subsequent to pre-training.
This is typically
2.2
MoPe;: Model Perturbation Attack
Membership Inference Attacks. We now define a membership inference attack (MIA) against model 0. A membership inference score M O x X > R takes model 0, and context æ, and assigns it a numeric value M(x () E R - larger scores indicate x is more likely to a training
Our Model Perturbations Attack (MoPeg) is based on the intuition that the loss landscape with respect to the weights should be different around training versus testing points. In particular, we expect that since z argming Ersc E(x, 8), where the

---

negative log-likelihood defined in Section 2.1, then the loss around x' C should be sharper than around random point. Formally, given candidate point I ~ D, we define:
3 MIA Against Pre-trained LLMs
In this section we conduct a thorough empirical evaluation of MoPen, focusing on attacks against pre-trained language models from the Pythia suite. We show that in terms of AUC MoPe significantly outperforms loss thresholding (LOSSg) at model sizes up to 2.8B and can be combined with LOSSg to outperform at the 6.9B and 12B models (Figure 4), We also implement an MIA based on DetectGPT, which outperforms loss in terms of AUC up to size 1.4B, but is consistently worse than MoPey at every model size. Our most striking finding is that if we focus on the metric of TPRs at low FPRs, which state-of-th of-thh--rttttttt work MTAs argue is the most meaningful metric (Carlini et al., 2022; Ye et al., 2021). MoPe, exhibits superior performance all model sizes (Figure 2). MoPey is the only attack that is able to convincingly outperform random guessing while driving the attack FPR < 50% (which is still a very high FPR!) at all model sizes.
MoPe (1) = EE--A((. Impurmm) 0+e) ((x;, 0))
(2)
where x² is variance hyperparameter that we specify beforehand, and 0 E R"quas, In practice, rather than computing this expectation,
sample n noise values e; (O, o² I Ingam))) and compute e,)
the empirical MoPej (z: n Σ Ei=1(((a ((x,0), This gives rise to the natural MoPeg thresholding attack, with M f) = (z)e in Equation 1. Note that computing each perturbed model takes time O(Npurams) and so we typically take < 20 for computational reasons. We now provide some theoretical grounding for method.
Connection to the Hessian Matrix. In order to derive more intuitive expression for MoPeg (z) we start with the multivariate Taylor approximation (Königsberger, 2000):
Dataset and Models. We identified EleutherAT's Pythia (Biderman et al., 2023b) suite of models as the prime candidate for studying membership inference attacks. We defer more full discussion of this choice Section B in the Appendix, but provide some explanation here as well. Pythia was selected on the basis that the models are available via the open source provider Hugging Face, cover a range of sizes from 70M to 12B, and have a modern decoder-only transformer- based architecture similar to GPT-3 with some modifications (Brown et al., 2020; Biderman al., 2023b). Models in the Pythia suite are trained on the Pile (Gao et al., 2021) dataset, which after de-duplication contains 207B tokens from 22 primarily academic sources. De-duplication is important as has been proposed as one effective defense against memorization in language models, and so the fact that our attacks succeed models trained dc-duplicated data makes them significantly more compelling (Kandpal et al., 2022: Lee et al., 2021). Crucially for our MIA evaluations, the Pile has clean training vs. test splits, as well as saved model checkpoints that allow us to ensure that we use model checkpoints at each size that correspond to one full pass over the dataset.
E(A + c, - E(0, x)) + Vol(B,z).et e
(3)
1 T H c (0(()
2
where Hg = 2))(0,z) is the Hessian of log- likelihood with respect to & evaluated at æ. Then assuming o² is sufficiently small that O(c²) in Equation 3 is negligible, rearranging terms and taking the expectation with respect to of both sides of (3) we get:
(x))
(1((0++,,0) -((8, [)] =
1
0² Tr(H,,,
B H 2
where the last identity is known as the Hutchinson Trace Estimator (Hutchinson, 1989). This derivation sheds some light on the importance of picking an appropriate value of a. We need it to be small enough so that the Taylor approximation holds in Equation 3, but large enough that we have enough precision to actually distinguish the difference in the log likelihoods in Equation 2. Empirically, we find that o 0.005 works well across all model sizes, and we don't observe a significant trend on the optimal value of o (Table 2).
We evaluate all attacks using 1000 points sampled randomly from the training data and 1000 points from the test data. Since the MoPe,

---

and DetectGPT attacks are approximating an expectation, we expect these attacks be more accurate if we consider more models, but we also need to balance computational considerations. For MoPeg we use n = 20 perturbed models for all Pythia models with < 2.8B parameters, and 10 perturbed models for the 6.9B and 12B parameter models. For DetectGPT perturbations throughout rather than the 100 used in (Mitchell al., 2023) in order to minimize computation time, which can be significant 12B.
that are much much smaller. Table 1 reports the TPR FPR 205. We see that even moderately large FPR = .25, only MoPey achieves TPR > .25 at all model sizes, with TPRs 1.5x higher than DetectGPT and MoPeg- We note that none of the attacks consistently achieve TPR better than the baseline FPR .05, but that MoPey still performs better than the other attacks.
use 10
Model
Method
AUC
TPR, 25
TPR .05
70M
LOSS,
.507
.168
.025
We found the best MoPeg noise level a for the models of size I 2.8B parameters by conducting a grid search over highest AUC was achieved at .01 for the 1B parameter model, .001 for the 2.8B model and .005 for the remaining models. This suggests that there no relationship between the optimal value a and the model size. For the 6.9B and 12B parameter models, we chose a noise level of .005. We record the AUCs achieved at different noise levels in Table 2 in the Appendix. For DetectGPT we follow (Mitchell et al., 2023) and use T5-small (Raffel et al., 2019) as mask-filling model, where we mask 15% of the tokens in spans of words.
160M
LOSSg
.512
.170
.030
410M
LOSSg
.513
.167
.028
[001, .00,, .05.. The
1B
LOSSg
.516
.172
.029
1.4B
LOSS,)
.517
.176
.029
2.8B
LOSSg
.504
.161
.021
6.9B
LOSS,
.523
.181
.026
12B
LOSSg
.525
.188
.024
70M
DetectGPT
.521
.201
.035
160M
DetectGPT
.515
.222
.033
110M
DetectGPT
.525
.006
.032
1B
DetectGPT
.532
.207
.031
1.4B
DetectGPT
.534
.210
.029
2.8B
DetectGPT
.510
.183
.022
6.9B
DetectGPT
.506
.178
.024
12B
DetectGPT
.504
.17
.020
70M
MoPea
.612
.306
.049
Attack Success. The results in Table show
160M
MoPe
.646
.376
.049
that thresholding based on MoPeş dramatically improves attack success relative to thresholding based on LOSSg or DetectGPT. The difference is most pronounced at model sizes 160M, 410M parameters, where the AUC for MoPey is ss 27% higher than the AUC of .51 for LOSS, and DetectGPT, which barely outperform random guessing. This our-performance continues up to size 2.8B, after which point attacks achieve AUC in the range .50- - .53.
410M
MoPeş
.650
.411
.072
1B
MoPea
.567
.261
.046
1.4B
MoPeg
.571
.259
.017
2.8B
MoPe,
.565
.280
.040
6.9B
MoPe,
.522
.252
.020
12B
MoPea
.516
.257
.024
Table For each model size and attack, we report the AUC, TPR at FPR .255 and TPR at FPR .05.
Inspecting the shape of the ROC curves in Figure there is between the attacks: All of the curves for the LOSSg and DetectGPT-based attacks drop below the dotted line (random baseline) between FPR .5.6. This means that even at FPRs higher than 5,, these attacks have TPR worse than random guessing! This establishes that, consistent with prior work, LOSS, based attacks against LLMs trained with a single pass work extremely poorly average, although they identify specific memorized points (Carlini et al., 2022).
striking difference
Model Size. Recent work (Carlini et al., 2023a) on the GPT-Nco (Black al., 2021) models evaluated on the Pile has shown that as model size increases so does memorization. The data in Table 1 support this conclusion, as with increasing model size we see a (small) monotonic increase in the AUC achieved by the LOSS,, attack. Interestingly, observe almost opposite trend in eurves for MoPeg, with the highest values of AUC actually coming at the three smallest model sizes! DetectGPT has AUC that is also flat or slightly decrcasing with increased model size. We note that while this does not directly contradict prior
By contrast, the MoPe, curves lie about the dotted line (outperforming random guessing) for FPRs

---

LOSS across model sizes
partially inversely correlated with model size, due to variance of the Hutchinson Trace Estimator in Equation increasing for larger models, and may not reflect the fact that larger Pythia suite models are actually more private.
1.0
0.8
Thue Posibive Rate
0.6
Training Order. In this section we study the degree to which the models in the Pythia suite exhibit "forgetting" over time, as measured by the change in LOSS, and MoPeg statistics during training. (Jagielski et al., 2023) show that, particularly on large data sets (like the Pile for example), MI accuracy increases (cquivalently loss decreases) for points in more recent batches. In Figure 7 in the Appendix, we investigate how LOSS, and MoPe, statistics vary on average depending on when they are processed during training. Concretely, we sample 2000 points from 10 paired batches {0- - 1, 9999 1c4, 19999 - 2e1, 89999 9, 97999 9.8e1), which approximately correspond to the first and last data batches that the Pythia models see during pre- training. For each set of 2000 points, we compute the average LOSS, and MoPe, values with respect to the 0 reached at the end of the first epoch. We see that, consistent with findings in (Jagielski et al., 2023), loss declines for more recent batches, but by contrast, there is no such observable pattern at any fixed model size for the MoPe, statistic! This finding is consistent with recent work (Biderman et al., 2023a,b) that study memorization in the Pythia suite, and find no correlation between order in training and if a point is "memorized" (as defined in terms of extractability)
(AUC 0.5075)
0.4
160m (AUC 0.5118)
410m 0.31335
1b (AUC = 0.5161)
0.2
(AUC 0.5168)
2.8b (AUC- 0.5039)
6.9b (AUC- 0.5227)
0.0
12b (AUC 0.5252)
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
DetectGPT across model sizes
1.0
0.8
True Postiive Rate
0.6
70m (AUC- 0.5208)
0.4
160m (AUC= 0.5155)
410m (AUC 0.5254)
0.5322)
0.2
1.4b (AUC- 0.5336)
(AUC 0.5103)
6.9b (AUC 0.5060)
0.0
12b (AUC 0.5041)
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
MoPe across model sizes
1.0
0.8
Thue Positive Rale
0.6
70m (AUC 0.6120)
0.4
160m (AUC 0.6457)
MoPeş vs. LOSSg comparison. The disparities in MIA performance between MoPea and LOSS, attacks shown in Figure implies that there must exist a number of training points where the MoPea and LOSS, statistics take very different values. The fact that MoPea outperforms LOSS,, particularly at smaller model sizes, implies that there are points with average loss values but outlier MoPeş values. We visualize this for the 12B parameter model in Figure 3. and include plots for all model sizes in Figure 8 in the Appendix.
410m (AUC 0.6502)
1b (AUC 0.5666)
0.2
1.4b (AUC 0.5710)
2.8h (AUC- 0.5652)
(AUC- 0.5217)
0.0
12b (AUC 0.5164)
0.2
0.4
0.6
0.8
1.0
False Positive Rate
Figure 2: LOSSg, DetectGPT, and MoPe, ROC Curves across all model sizes. Only MoPe, outperforms the baseline at FPR < ..
results from (Carlini et al., 2023a) which use definition of memorization based on extraction via sampling, it is surprising given the intuition from prior work. One potential explanation could be that the attack success of perturbation-base methods like MoPe, and DetectGPT is actually
LOSS, and MoPe, Ensemble Attack. Recall that a point is more likely be a training point if it has a high MoPeş value, or low LOSS, value. The above scatterplot shows that there are training points that have average LOSS, values and high MoPe, values and are easily identified by the MoPe, attack as

---

6. 6.9b combination
12b
Trainig validation
2.0
LOSS predictions tescoare, log logrmoouuus
1.3
1.0
0.6
Thue Postivve
0.5
04
0.0
02-
0.52277)
6.9b LOSS (AUC 0.52227
0.0
0.53*MOPE 0.445LLSS 0.5300)
0.0
0.2
0.6
0.8
1.0
False Positive Rate
-1.0 -0.5
0.5
log modukss)
12b combination
1.-
Figure 3: MooPee vs. LOSSg scatter plot, model size 12B. We z-score the LOSSa and MoРeş values and apply log-modulus transform (a) sign(x)) log[[a] + (John and Draper, 1980) to the scores for visual clarity.
0.8
True Positive hate
training points but not by LOSSg, and vice versa. This raises the obvious question of if the attacks be combined yield stronger attacks than
12b MoPe (AUC - 0.5164)
either attack in isolation. We find that while for the smaller model sizes, we don't get a significant improvement over MoPeg in AUC by ensembling, for the two largest model sizes, where both LOSS, and MoPeş perform relatively poorly, we do get a significant improvement by thresholding on weighted sum of the two statistics (after z-scoring). We plot the ROC curves for LOSS,, MoPeg, and the optimal ensemble (picked to optimize AUC) in Figure 4. At 6.9B both LOSSg and MoPeş achieve AUC = .52, while the ensemble has AUC .55. At 12B AUC jumps from .52 to .544 the ensemble.
LOSS 0.5252)
12b 0.5+MOPE 0.5*LO5S (AUC 0.5445)
0.0
0.2
0.4
0.6
0.8
10
False Positive
Figure 4: LOSS, and MoPe Ensemble Attack
pre-trained language models (Carlini et al., 2021). In Figure 5 we show the distribution of the Hessian trace and scaled MoPeş evaluated on all training batches. The resulting distributions training and test points are close, with a correlation between point level traces and scaled MoPe, values of 0.42. Given the generality of the proposed MoPe, attack, it is also natural to ask if MoPej can be applied other machine learning models. On our MNIST network, run the MoPeş attack with 40 perturbations and a 0.05, calculating the ROC curves in Figure using 3000 train and test points. We find that LOSS, outperforms MoPeg, with MoPea achieving an AUC of 0.635 and LOSS, achieving an AUC of 0.783. While it does not outperform LOSSa, MоPe₁ easily beats the random baseline, which warrants further study in settings beyond language models.
4 MoPeg on MNIST
In this section network trained to classify images from MNIST (Deng, 2012), which allows us evaluate if the MoPeş statistic actually approximates the Hessian trace in practice as the derivation Section 2.2 suggests. To test this, we sample 5000 training points and 5000 test points from the MNIST dataset, a large database of images of handwritten digits (Deng, 2012). Using batch size of 8, train fully connected layer MLP with 20 and 10 nodes. using the Adam optimizer with learning rate of 0.004 and momentum of 0.9, until we reach 94% train accuracy and 84% test accuracy, a 10 percent train-test gap similar to that observed in
our MoPe, attack small
Discussion
The main finding in this work is that perturbation- based attacks that approximate the curvature in the

---

Train vs Test MoPe Statistic (Scaled)
depend on the point x. Prior work on MIAs shows that calibrating the threshold Ta to specific point a greatly increases attack success, for the obvious reason that certain types of points have higher or lower relative loss values regardless of their inclusion in training. Thresholding based on a loss ratio with another language model or approximating the likelihood ratio (LiRA) can all be viewed as different ways of calibrating an example-specific threshold. In this context, the results in this paper can be viewed as showing that when forced to pick a uniform threshold for an MIA score, statistics like MoPe, or DetectGPT that approximate the local curvature transfer better across different points than the loss. As such, methods that try to efficiently approximate an example-specific threshold, like applying LiRA loss-ratio thresholding in the fine-tuning regime where they are more computationally feasible, or applying the very recent work of (Bertran et al., 2023) that uses quantile regression to approximate a likelihood-style attack without training additional models, in our setting is of great interest. We conjecture that such techniques will be needed to achieve high TPRs at very small FPRs. Another major unresolved question in this work is why MoPe, and DetectGPT success actually scales inversely with model size, which is likely a property of the method, namely increasing error in the Hutchinson trace estimator, rather larger models having improved privacy properties. Future work will explore other approximations to the Hessian trace that may scale better to large model sizes. Finally, future work could use MoPe, as part of training data extraction attack in the style of (Carlini et al., 2021; Yu al., 2023; Carlini et al., 2023b).
80
Scaled Train MoPe Scaled
70
60
No. Samples
50
40
30
20
10
-50
250
300
Scaled Train MoPe
Exact Hessian Traces on MNIST
train
100
No. Samples
60-
40-
20
300
350
Hessian Trace
Figure 5: The distribution of MoPe statistics (left) and exact Hessian traces (right) for the MNIST dataset.
MNIST Fully Trained LOSS vs MoPe
10
0.
Tue Positive Rate
0.6
0.4
02
LOSS (AUC 0.7830)
0.0
0.6355)
0.0
0.2
0.6 False Positive Rate
0.4
0.8
10
Figure 6: ROC curve for LOSSa vs MoPe, on MNIST model.
loss landscape around given model and point, seem to perform better than attacks that simply threshold the loss. A key nuance here, that for computational reasons, all of the attacks studied here apply a uniform threshold I that does not

---

Limitations
language models, and on privacy risk mitigation. rather than used to exploit existing systems.
We now list several limitations of this work, all of which present opportunities for future research.
References
Our MoPe, MIA operates in the white-box setting that assumes access to the model parameters 0, whereas existing attacks typically in the black-box setting, where only access to the model outputs or the loss are assumed. Neverthelcss, our attack is still practical given that open source models or even proprietary models are often either published or leaked, and from security perspective it makes sense to assume attackers could gain access to model parameters. Moreover, the findings in this paper have scientific implications for the study of memorization and privacy in language models that are orthogonal to the consideration of attack feasibility.
John Abascal, Stanley Wu, Alina Oprea, and Jonathan Ullman. 2023. Tmi! finetuned models Icak private information from their pretraining data.
Rohan Anil, Badih Ghazi, Vineet Gupta. Ravi
Kumar, and Pasin Manurangsi. 2021a. Large-scale differentially private bert.
Rohan Anil. Badih Ghazi, Vineet Gupta, Ravi
Kumar, and Pasin Manurangsi. 202 Large-scale differentially private BERT. CoRR, 0bss22108.0162
Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, and Zhiwei Steven Wu. 2023. Scalable membership inference attacks via quantile regression.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023a. Emergent and predictable memorization in large language models.
We are only able to test our techniques on model sizes present in the Pythia suite (up to 12B parameters), and so the question of whether these results will scale larger model sizes is left open.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony. Herbie Bradley, Kyle O' O'Brien, Eric Hallahan. Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023b. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.0137.
We were only able to optimize over limited set of noise values in MoPeg due to computational reasons, and so MoPe, may perform even better with more exhaustive hyper-parameter search.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with meshtensorflow.
Another drawback of MoPea as practical attack, is that computing perturbed models can be computationally expensive at large model sizes, potentially limiting our ability to take enough models to accurately estimate the Hessian trace.
Tom B Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few- shot learners. In Advances in Neural Information Processing Systems. volume 33, page 1877-1901. Curran Associates, Inc.
Ethics Statement
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In 2022 IEEE Symposiun on Security and Privacy (SP), pages 1897-1914, IEEE.
We are satisfied this paper has been produced and written in an ethical manner. While the purpose of this paper is to demonstrate the feasibility of privacy attacks on large language models, we did not expose any private data in our exposition or experiments. Moreover, all attacks were carried on open source models that were trained public data, and as a result, there was limited risk of any exposure confidential data to begin with. Finally, we propose these attucks in the hope they will spur further research on improving privacy in
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang 2023a. Quantifying memorization across neural language models.
Nicholas Carlini, Utkarsh Kandpal, Jacob Lehman. Nicolas Papernot, and Florian Tramèr. 2023h. Lm- extraction: A benchmark for training data extraction from language models. https://gi com/ wewellereeeacccheeee

---

Nicholas Carlini, Florian Tramer. Eric Wallace. Matthew Jagielski, Ariel Voss,, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito. Katherine Lee, Nicholas Carlini, Fric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. 2023. Measuring forgetting of memorized training examples.
Ulfar Erlingsson,
Christopher A. Choquette-Chov Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. 2020. Label-only membership inference attacks. CoRR, 1ow/2007.14311..
Matthew Jagielski, Stanley Wu, Alina Oprea, Jonathan Ullman, and Roxana Geambasu. 2022. How combine membership-innfeeen attacks on multiple updated models.
Gilad Cohen and Raja Giryes, 2022. Membership inference attack using self influence functions.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Sco. 2022. Knowledge unlearning for mitigating privacy risks in language models.
mnist database of handwritten digit images for machine learning research. JEEE Signal Processing Magazine, 29(6):141-142,
Deng. 2012.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination natural language generation. CoRR, abs/2202. 0Ds/2202.03669.
Christophe Dupuy; Radhika Arava, Rahul Gupta, and Anna Rumshisky. 2022. An efficient dp-sgd mechanism for large scale nlp models.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. 2016. Calibrating noise to sensitivity in private data analysis. J. Priv. Confidentiality, 7(3):17-51.
J. A. John and N. R. Draper. 1980. An alternative family of transformations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 29(2):190-197.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models.
Giorgio Franceschelli and Mirco Musolesi. 2021. Copyright generative deep learning.
Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: 800gb dataset of diverse text for language modeling. CoRR, 00//21111000077
Eugene Kharitonov, Marco Baroni, and Dicuwke Hupkes. 2021. How bpe affects memorization in transformers.
Konrad Königsherger. 2000. Analysis 2. Springer Berlin Heidelberg.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. CoRR, 1072100770999
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts:: Evaluating neural toxic degeneration in language models. CoRR, abs/2009. 1462.
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, John Tembe, Jill Muchling, John V. Pearson, Dietrich A. Stephan, Stanley F. Nelson, and David W. Craig. 2008. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS Genetics, 4(80e100166
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. 2022. Differentially private decoding in large language models.
Michael F. Hutchinson. 1989. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 18:1059-1076.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Kirkpiiik.. 2023. Membership inference attacks against language models via neighbourhood comparison.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette-Chov and Nicholas Carlini. 2023. Preventing verbatim memorization in language models gives false sense of privacy.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Mamning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generatee text detection using probability curvature.
Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership inference attack susceptibility of clinical language models.
Martin Pawelczyk, Himabindu Lakkaraju, and Seth Neel. 2022. the privacy risks of algorithmic recourse.

---

Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
A
Additional Related Work on LLM Privacy.
(Carlini et al., 2021; Yu et al., 2023) focus on the problem of training data extraction from LLMs by generating samples from the model, using loss- based MIAs to determine if the generated point is actually member of the training set that has been memorized. Both papers focus more on extraction than explicitly evaluating membership inference attack success, and acknowledge existing MIAs against LLMs are relatively weak. (Jang et al.. 2022) studies the problem of unlearning a training point from trained model via taking gradient ascent steps. One metric they use determine if point has been unlearned is if the loss a point x that has been unlearned close to the expected loss for a test point. Our work has implications for this kind definition of unlearning, as our results show that an average LOSS, value does not mean the point cannot be easily detected as a training point. Differentially private training (Dwork et al., 2016) is canonical defense against MIAs, and there has been flurry of recent work private model training NLP (Anil et al., 2021a; Majmudar al., 2022; Dupuy et al., 2022). While (Li et al., 2022) report success in fine-tuning language models with differential privacy, is know that privacy during pre-training comes at a great cost to accuracy (Anil et al., 2021b). Since pre-training with differential privacy remains challenge, existing work does not provide thcoretical mitigation guarantees against our attacks pre-trained models.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text- tx--tt-text transformer. CoRR, abs/1910 10683.
Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Hervé Jégou. 2019. White- box vs black-box: Bayes optimal strategies for membership inference.
Brihat Sharma, Yanjun Gao. Timothy Miller, Matthew M Churpek, Majid Afshar, and Dmitriy Dligach. 2023. Multi-task training with in-domain language models for diagnostic reasoning. arXiv preprint 23xv:3306002077
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models.
Rebecca Soper. 2023.
Can ai help you build relationships? amorai thinks so.
https://techcrunct com/2823/05/13/ bi-relatiionhiip-bulldini aamarrrrriiinna...........
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. 2020. Understanding unintended memorization in federated learning.
Nikhil Vyas, Sham Kakade, and Boaz Barak. 2023. Provable copyright protection for generative models.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sehastian Gehrmann, Prabhanjan Kambadur, David Rosenberg. and Gideon Mann. 2023. Bloomberggpt: A large language model for finance, arXiv preprint 4rXIv:2303.17564
B Pythia Suite.
We identified EleutherAl's Pythia (Biderman et al.. 2023b) suite of models as the prime candidate for studying membership inference attacks. Models in the Pythia suite trained the Pile (Gao et al., 2021) dataset, which is an 825GB dataset of about 300B tokens, consisting of 22 primarily academic sources. All our experiments are using models trained on a version the Pile that was de-duplicated using MinHashLSI with a threshold of 0.87, which reduces the size to 207B tokens. We perform our experiments in the de-duplicated regime it has been shown that the presence of duplicated data greatly increases the likelihood of training data memorization (Lee et al., 2021). and so attacks in the de-duplicated setting are significantly more compelling. We use model checkpoint corresponding to one full pass over the
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. 2021. Enhanced membership inference attacks against machine learning models. CoRR, abs/211 0aop//111.0966 1.09679.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson. and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting.
Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. 2023. Bag of tricks for training data extraction from language models.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021. Counterfactual memorization in neural language models.

---

de-duplicated Pile. The data is tokenized using BPE tokenizer developed specifically on the Pile. Training examples are 2048 tokens, and the batch size used during training is 1024. In order to maintain an apples-to-apples comparison between train and test examples, we batch test examples identically when evaluating our MIAs. Importantly, the Pile contains train test splits which allow to evaluate our MIAs, and is also annotated with the order of points during the training of all models which allows us study the implications training order for privacy.
165m
1SG5 Bauch
zar-
100744
251-
The models in the Pythia suite are open source and available through Hugging Face, have publicly available model checkpoints saved during training, and range in size from 70m parameters to 12B. The models follow the transformer-ban based architecture in (Brown et al., 2020), with some small modifications (Biderman et al., 2023b).
Nohl Yo us to Bersbore- 220m
-126
-150
-1.00
-1.0
Figure& & Tables
-14
-1.9
Model
=000
005
001
.05
70M
0.6034
0.6069
0.5708
0.4906
LOSO
30015 20
160M
0.6394
0.6478
0.5613
0.5121
410M
0.5915
0.5958
0.5367
0.5190
1B
0.5028
0.5142
0.5924
0.5111
1.4B
0.5652
0.5656
0.5502
0.5136
2.8B
0.5320
0.5086
0.5109
0.5030
Table 2: MoPey AUC per model size and noise level a.
Tueme: Outd
MoPe Vilae Remifine
M
-0/10
4098
2.6
3.6
Figure 7: For each batch size, report the average LOSS, or MoPea score over points in that batch, along with 95% CI for the average value for the 160M, IB models.

---

70m
160m
15-
LOSS presicccn scar,, big modinns
-05
-10
-1.
-2.0
-1.
-1.0
-0.
0.0
410m
2.0
LOSS predictoon toscir, log mod.ea
10
-1.5
-1.
-2.0
-1.5
-1.0
-1.
-1.0
-05
0.0
0.5
1.0
1.5
1.4h
2.8b
2.0
LOSS picdetinn socr. (ocoouuu
15-
10-
0.
-1.0
-1.
-1.
2.5
2.0
1.5
1.0
0.5
0.2
1.0
15
2
Trainin Valcstton
2.0
2.0
LOSS predictens 2o.2ore, bog mocuus
5.0
0.5
05
-1.
-10
3.0
0.5
lag moditlus
log-mndlddss!
Figure 8: LOSSg vs. MoPey scatter plots across model sizes.


Reviews:
[{"review_id": "6ckX1UpN5k", "reviewer": "Reviewer_sLT5", "summary": "", "questions": "- The tests on MNIST are interesting but suggest immediate follow ups with regards to the actual\n  cause of the difference in MoPE vs. LOSS performance. It could be the difference in\n  architectures, the difference in input types, differences in sizes, otherwise? There do exist\n  some vision transformers so a good test here would be to evaluate MoPE on one. Also, less trivial\n  (than dense) but standard vision architectures might also be good to test with. Dense models, as\n  per the name even, are dense and could be particularly prone to memorization in one manner\n  different than more sparse models.\n\n- What are the \"assumptions needed to show optimality\" mentioned on line 220 ?\n\n- Line 427 states \"This suggests that there was no relationship between the optimal value of σ and\n  the model size\" right after noting the various differences in optimal σ and model size. Please\n  explain.\n\n- It is unclear why Figure 6 shows distributions of values instead of direct comparisons of values.\n  Why is that?\n\n- No defenses are discussed against MoPE. Do you expect them to differ from those that attempt to\n  defend against membership inference in general? Are there some defenses that could work\n  particularly well against MoPE (something with messing with model parameters perhaps?\n  regularization?)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and kind words, and we will address the typos raised around notational consistency.  We will focus on the questions and weaknesses raised. \n\n**Weakness 1**: The description of the principal method does not match the actual method and thus the reasoning behind it is misplaced and/or potentially irrelevant... \\\n**Response 1**: We thank the reviewer for raising these points. We believe that our current description of model perturbations vs point perturbations is correct as written and offer additional clarifications below. However, we do believe that the reviewer raises an interesting point about perturbing inputs, which we mention in our related work and will expand on below as well. The training objective is $\\text{argmin } l(\\theta, D)$, and so we would expect that the *model* learned during training is an approximate local minimizer, because gradient descent even on a non-convex objective converges to a local minimizer. This point is conveyed in Figure $1$. Considering loss with respect to the input $x$, which is what point perturbations would seek to exploit, we note that if $x$ is a training point and $\\theta$ is the learned model, it does not necessarily imply that $\\nabla_{x}l(x, \\theta)$ is close to zero. That being said, the idea of perturbing the inputs and calculating the drop in loss is an interesting one that has been explored in the context of detecting model-generated content, and could be leveraged as an MIA attack. We have some preliminary results using this attack, which seems to outperform loss but perform worse than MoPe, which we could include in the camera ready results, although they may warrant a separate paper. We discuss this prior work in the DetectGPT section of the related work. \n\n**Weakness 2**: The actual method, perturbing model parameters, might have more connections to prior methods other than LOSS. Perturbing a model is moving it away from the set of parameters that was attained with a particular training point and in same way, getting closer to a model that was trained without that point. There are membership methods based on comparisons of \"with point\" vs \"without point\" models. \\\n**Response 2**:  We are familiar with the membership inference attacks that try to approximate the loss as if the model was not trained on the point. The most relevant work in that direction is [1] which uses the \"self-influence\" of a candidate point on the loss as an MIA statistic. We did not consider this attack in this work because it requires computing an inverse Hessian vector product which is inefficient; even for the vision models studied in [1] their attack was significantly slower than baseline attacks, see their Table 3.  We will add this to our discussion of prior work. There are no obvious theoretical connections between our perturbation based model and this attack, other than the connection to the Hessian we discuss in the paper. There are MIAs that use access to a model known to not be trained on the input to calibrate the threshold for the loss attack for example. We have some preliminary work using these models, and they do improve attack success (including of MoPe), but this is not a realistic assumption for pre-trained models given the cost incurred in their training. It is a much more relevant consideration for attacks against fine-tuned models, which is not the context we're studying it in. This paper explicitly focuses on the pre-training regime. We will soften our related work section to discuss these alternative approaches as suggested. \n\n**Weakness 3**: \nthe authors claim that single epoch training is common as a justification of not experimenting with models trained with multiple though the discussion of checkpoints (around line 410) suggests that not only is it also common to train for more than one pass, these models were already readily available during experimentation and had to be specifically avoided. \\\n**Response 3**: It is not common to train language models for multiple epochs -- the models + dataset in question (GPT-Neo on the Pile) trained for approximately 1.5 epochs, and state of the art models like GPT-3 and Llama-2 are trained with a single epoch during pre-training. That being said, training for multiple epochs would only increase the success of our MIAs, because the model would have more information from the data. The reason we didn't do this is it would make comparisons between points messier, because we would have to account for if they had been seen once or twice by the model. We could compare to models trained on multiple passes, which we did do for the experiment on MNIST, but it would not be accurate in terms of how these language models are actually trained. \n\nWe'll now answer the questions from the reviewer. \n\n**Question 1**: There do exist some vision transformers so a good test here would be to evaluate MoPE on one. Also, less trivial (than dense) but standard vision architectures might also be good to test with. Dense models, as per the name even, are dense and could be particularly prone to memorization in one manner different than more sparse models. \\\n**Response 1:** We agreed that more extensive experiments with vision models, including transformers, is an interesting direction for future work. In this paper our focus was the state of the art auto-regressive language models that have become very popular lately, and so we didn't focus on a more thorough evaluation within the vision context. \n\n**Question 2**: What are the \"assumptions needed to show optimality\" mentioned on line 220 ? \\\n**Response 2:** The assumption in Equation (1) in [2], and it is that the model output from the training algorithm is a sample from the posterior distribution in Equation (1): $\\theta \\sim e^{\\frac{-1}{T}\\sum_{i = 1}^{n}l(\\theta, x_i)}$, which does not hold for non-convex objectives like the one's used to train LMs.\n\n**Question 3**: Line 427 states \"This suggests that there was no relationship between the optimal value of σ and the model size\" right after noting the various differences in optimal σ and model size. Please explain. \\\n**Response 3:** By this we did not mean that the value of $\\sigma$ didn't effect the ultimate attack success for a given model size, rather we meant that there was no monotonic trend we observed in the relationship between the optimal $\\sigma$ and the model size, e.g. it was not the case that larger models corresponded to larger or smaller values of sigma. We will clarify this wording for the camera ready. \n\n**Question 4**: It is unclear why Figure 6 shows distributions of values instead of direct comparisons of values. Why is that?\n**Response 4**: Figures 5,6 compare the distribution of train vs. test points mope statistics and exact hessian traces. This is because the comparison of those two distributions is a direct determinant of MIA attack success. A related but different question, is for a given point, how does the hessian trace compare to the mope statistic (e.g. what is the approximation error for MoPe). We do report a version of this statistic (point-wise correlation of .42) but we agree it would be useful to have a plot comparing point-wise values, and we can easily add that to the supplement for the camera ready. \n\n**Question 5:** No defenses are discussed against MoPE. \\\n**Response 5**: To our knowledge, there are no current defense strategies against MIAs against pre-trained language models; particularly since until this work, the privacy risks seemed limited. This is a good suggestion, and an exciting direction for future work that we are actively exploring, but would constitute a major paper in its own right. We discuss this in the last paragraph of the conclusion: \\\n\"Differentially private training (Dwork et al., 2016) is a canonical defense against MIAs, and there has been a flurry of 880\nrecent work on private model training in NLP (Anil et al., 2021a; Majmudar et al., 2022; Dupuy et al., 2022). While (Li et al., 2022) report success in fine-tuning language models with differential privacy, it is know that privacy during pre-training comes at a great cost to accuracy (Anil et al., 2021b). Since pre-training with differential privacy remains a challenge, no existing work provides theoretical mitigation guarantees against our attacks on pre-trained models.''\n\n\nReferences: \n[1] https://arxiv.org/abs/2205.13680 [2] https://arxiv.org/pdf/1908.11229.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Well written work except for the points below in the first weakness. My notes include\n  \"nice\" in response to several well-made points throughout the paper which is not common for me.\n\n+ Attack discovers is effective at a different set of instances than loss attacks (i.e. answers the\n  points nicely made on line 96). This and the AUC behaviour comparison is suggesting that there\n  may be a fundamentally different sort of signal being employed in the MoPE attack (as opposed to\n  being just a better LOSS). This also means the methods can be combined to get non-trivial benefit\n  as shown in the Ensemble Attack. This strength has some caveats under the second weakness below.", "reasons_to_reject": "- The description of the principal method does not match the actual method and thus the reasoning\n  behind it is misplaced and/or potentially irrelevant. From the start, the method is suggesting\n  that loss landscape around training points is sharper than around random (non-training) points:\n\n       \"based on the idea that when the model loss is localized around a training point, it is\n        likely to lie in sharper local minima than if the point was a test point—a previously\n        unexplored fact distinct from the magnitude of the loss\"\n\n       \"loss around x′ ∈ D_train should be sharper than around a random point\"\n\n  The actual method perturbs the model parameters though one would expect perturbations to points\n  given the above descriptions. The initial point seemed intuitive while model perturbations are\n  not. The connection between model perturbations and point perturbations are not described or\n  explained.\n\n  Suggestion: Explain why input perturbations are not done in favor of model perturbations. (if\n  true) explain how model perturbation is the same as, or sufficiently similar to, input\n  perturbation. Other motivations might include the difficulty of perturbing inputs of discrete\n  tokens (though this can be addressed by perturbing embeddings). Finally, if input perturbations\n  are possible, compare the model perturbation approach to the input perturbation one,\n  experimentally.\n\n  In addition to this point, the connection to Hessian trace is specific to the (I claim) less\n  intuitive model perturbation. The benefit of making such a connection are not discussed.\n\n  Suggestion: describe the theoretical or otherwise benefits of making the connection between model\n  parameter perturbation as per MoPE and Hessian traces.\n\n- The actual method, perturbing model parameters, might have more connections to prior methods\n  other than LOSS. Perturbing a model is moving it away from the set of parameters that was\n  attained with a particular training point and in same way, getting closer to a model that was\n  trained without that point. There are membership methods based on comparisons of \"with point\" vs\n  \"without point\" models. Cited works include [Carlini 2021]. Such methods, however, were not\n  compared to MoPE experimentally nor conceptually.\n\n  Suggestion: Include a deeper discussion on attacks based on reasoning with models known to not be\n  trained on target input. Provide experimental comparisons of MoPE and such methods.\n\n- Unexplained restrictions on target models. In particular, the authors claim that single epoch\n  training is common as a justification of not experimenting with models trained with multiple\n  though the discussion of checkpoints (around line 410) suggests that not only is it also common\n  to train for more than one pass, these models were already readily available during\n  experimentation and had to be specifically avoided.\n\n  Suggestion: Compare MoPE to baselines (LOSS and otherwise) on models trained with multiple\n  passes. Demonstrate that the benefit of MoPE over baselines is or is not associated with the\n  single-pass model training regimen.\n\nOther suggestions:\n\n- Additionally I suggest the authors revisit their related work sections and be less quick to\n  dismiss comparisons against other works due to some difference in setting that could be easily\n  adopted to make a direct experimental comparison possible.", "questions_to_authors": "- The tests on MNIST are interesting but suggest immediate follow ups with regards to the actual\n  cause of the difference in MoPE vs. LOSS performance. It could be the difference in\n  architectures, the difference in input types, differences in sizes, otherwise? There do exist\n  some vision transformers so a good test here would be to evaluate MoPE on one. Also, less trivial\n  (than dense) but standard vision architectures might also be good to test with. Dense models, as\n  per the name even, are dense and could be particularly prone to memorization in one manner\n  different than more sparse models.\n\n- What are the \"assumptions needed to show optimality\" mentioned on line 220 ?\n\n- Line 427 states \"This suggests that there was no relationship between the optimal value of σ and\n  the model size\" right after noting the various differences in optimal σ and model size. Please\n  explain.\n\n- It is unclear why Figure 6 shows distributions of values instead of direct comparisons of values.\n  Why is that?\n\n- No defenses are discussed against MoPE. Do you expect them to differ from those that attempt to\n  defend against membership inference in general? Are there some defenses that could work\n  particularly well against MoPE (something with messing with model parameters perhaps?\n  regularization?)", "paper_topic_and_main_contributions": "The paper presents a method (termed \"MoPE\") for performing a membership inference attack\n(determining whether a given instance was or was not part of the training set for a given model) on\nlanguage models. The method perturbs the model parameters (hence requires knowledge of the model\ninternals) and stipulates that the loss landscape around model parameter perturbations is steep for\ntraining instances and smooth for non-training instances. Thus the attack guesses instances above\nsome threshold of \"steepness\" are training instance. The paper experimentally demonstrates that\nthis approach does better than thresholding purely on model loss of a target instance which is\namong prior baseline approaches, at least on several (nowadays) small LLMs. Among investigations of\nthe method, the authors also show that for other architectures (I believe all of the language\nmodels tested were transformers) like naive dense, the loss based approach is better than the new\nproposed one. Another interesting tidbit is the AUC behaviour of loss-based vs. \"MoPE\" is\nfundamentally different, with most of LOSS's AUC coming in at the expense of false positive rate;\nwith MoPE being more balanced instead. Finally, the authors make a connection between the measured\nquantity and traces of Hessians but the relevance of this escapes me.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Work proposes privacy attacks without discussing or conjecturing on defenses. It is thus balancing\ntowards the negative side of societal benefits scale."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Different notations are used for model loss throughout the work. Additional confusion is that\n  LOSS is also used as the membership inference attack based on loss but also the definition of\n  modeling loss before line 273.\n\n- Order of arguments to $ l $ is backwards on Equation (3).\n\n- Missing \")\" on line 362 after σ^2."}}, {"review_id": "zNCGHrULsT", "reviewer": "Reviewer_EQtA", "summary": "", "questions": "L145: our attack performs worse than loss in this setting --- any hypotheses for why would language models be unique?  Does it hold for LLMs with classification heads or that only produce embeddings, versus next-token prediction?\n\n2.2: it's worth noting that the random model perturbations need only be computed once (with a cost of storing n model snapshots) and can then be applied to an arbitrary number of probe examples.\n\nJust an observation that σ=0.005 seems quite large for model parameters that are close to 0. I imagine the hessian approximation breaks down if non-uniform noise is injected but I would be curious to understand if scaling sigma by the model weight impacts the attack success.\n\nIt would be valuable to have some error analysis on the quantities in table 1, eg using bootstrap sampling if running multiple experiments is too expensive.  When are the differences in performance statistically significant?\n\nL482 Training order: clarify what model is evaluated for this experiment\n\nWhy evaluate the 12B model in figure 4 when there is little discernable difference between MoPe and LOSS performance?  Evaluating a smaller model will likely produce a more distinct scatter plot. Likewise, it's not clear that this figure easily supports the ensemble attack- there are nearly as many validation points with average loss and high MoPe scores.\n\nIs it plausible that sampling more perturbed models will improve attack performance for larger models?  As noted above, the perturbed models need only be computed once, so the main cost is evaluation of all the models over all the probe samples.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and respond to all questions and feedback below. \n\n**Concern 1**: “Success of the attack is really only demonstrated on auto-regressive LLMs.”\n\nResponse 1:  We focused on the Pythia Suite in this paper for  two primary reasons: (i) the auto-regressive decoder-only transformer-based architecture in GPT-Neo is similar to state of the art models (GPT-2,3,4 for example), and (ii) in order to evaluate our attacks we needed open-source models with available train/validation sets to test our attack; such a split is available automatically with The Pile. We were unable to find a clear split for other model types (e.g. BERT), although we acknowledge that such a dataset could be manually crafted if we re-trained from scratch. The Pythia Suite also allowed us to systematically benchmark MIAs across model sizes and model training (checkpoints) without pre-training from scratch. \n\n**Concern 2**: “Experimental comparison with shadow-training approaches would be valuable, at least for smaller models, or explain why these are not explored.”\n\n**Response 2**: While many shadow-training approaches to MIAs have been introduced, in the context of language models these attacks specifically require two steps of model training: some base model is pre-trained on a public data, and and a separate model is fine-tuned on private data is attacked. In our paper, we refer to these as the “pre-training” and “fine-tuning” regimes. We do not consider MIAs with shadow-training approaches in the pre-training setting, because they are not computationally feasible with the larger models studied in this paper, and indeed modern models are an order of magnitude larger than the largest model size studied here (12B). We focus on privacy attacks against pre-trained models that are also computationally feasible against realistically sized models. \n\nWe note this distinction in the second-to-last paragraph of Section 1: “their [other work] results can also be viewed as focused on small models in the fine-tuning setting, where it is known that much stronger attacks are possible, in contrast to the pre-training regime studied here.” We will write additional comments emphasizing this distinction for camera-ready. \n\n**Concern 3**: “An important limitation is white box access, but this is not a deal-breaker.” \n\n**Response 3**: We acknowledge that white-box access is not always a realistic assumption; however, as we note in our Limitations section, “many open source or even proprietary models are often either published or leaked, and from a security perspective it makes sense to assume attackers could gain access to model parameters.” Moreover, our results in the white-box setting are interesting for reasons that are independent of the feasibility of MoPe as a privacy attack; they demonstrate that more memorization/privacy leakage of training data during pre-training may be occurring than previous studies that only considered the loss may suggest.\n\n\n**Question 1**: L145: our attack performs worse than loss in this setting --- any hypotheses for why would language models be unique? Does it hold for LLMs with classification heads or that only produce embeddings, versus next-token prediction?\n\n**Response 1**. One major reason LLMs are different to the MLP we train on MNIST is that the LLMs we study are trained with only one epoch, and so there is less over-fitting, which is equivalent to saying the loss-based MIA has lower accuracy. The greater diversity of sentences in the Pile relative to the images in MNIST may also contribute to the lack of over-fitting in the language models.  We will add these points to the discussion in Section 4.  \n\nIt would definitely be interesting to test our MIAs  on different types of LLMs, but we focused on the Pythia suite for the reasons discussed above, and because pre-trained decoder-only LLMs constitute a fundamental and important class of LLMs that typically form the basis for LLMs fine-tuned for classification for example. \n\n**Question 2**. 2.2: it's worth noting that the random model perturbations need only be computed once (with a cost of storing n model snapshots) and can then be applied to an arbitrary number of probe examples.\n\n**Response 2**: Yes! This is important to note and we will add it in our camera-ready paper. This gives our method a substantial computational advantage over an attack based on perturbing inputs for example, where the perturbations need to be computed separately for every input.\n\n**Question 3**. Just an observation that σ=0.005 seems quite large for model parameters that are close to 0. I imagine the hessian approximation breaks down if non-uniform noise is injected but I would be curious to understand if scaling sigma by the model weight impacts the attack success.\n\n**Response 3.** We note that the $\\sigma$ parameter is not as large as it may seem: in the `70m` deduped Pythia model, the average of the absolute value of model weights is `0.03658` (obtained by taking `param.abs().sum()` over each parameter in `model.named_parameters()`), meaning that the standard deviation of the 0-mean Gaussian noise we add is approximately `13.7%` of the average absolute model weight. \n\nThe reviewer raises an interesting theoretical point about weighting the noise levels inversely to the model size. Weighing the noise levels by the model weight size would be equivalent to weighing the partials $\\frac{\\partial L^2}{\\partial w_i \\partial w_j}$ by the magnitude of the weights $|w_i|$ in our derivation of how MoPe relates to the Hutchinson trace estimator. We did not opt for these kind of approaches because (i) we found no empirical relationship between model size and the optimal noise level (Table 2 Appendix), and (ii) it doesn't seem to have a theoretical interpretation like our Hessian trace approximation. \n\n**Question 4.** It would be valuable to have some error analysis on the quantities in table 1, eg using bootstrap sampling if running multiple experiments is too expensive. When are the differences in performance statistically significant?\n\n**Response 4.** This is a great suggestion! As suggested, we’ve performed the following procedure: using our reported values, we’ve run a bootstrap on the AUC for 9999 iterations, computed a bootstrapped SE and 95% confidence interval, and because the bootstrapped AUC distributions look approximately normal, ran a 2-sided Welch’s t-test to get p-values. Across MoPe and LOSS bootstraps, we find a SE of `0.008` to `0.013` across the model sizes. We find statistically significant differences (`p < 1e-6`) between MoPe and LOSS AUC at all model sizes respectively, even after using the Bonferroni correction. We will incorporate these values with confidence intervals into a revised table for camera ready! \n\n**Question 5**. L482 Training order: clarify what model is evaluated for this experiment. \n\n**Response 5**: Thank you for the note. We used the `160m` model here, which is in the figure but not the text. We’ll change this for camera-ready! \n\n**Question 6**. Why evaluate the `12B` model in Figure 4 when there is little discernible difference between MoPe and LOSS performance? Evaluating a smaller model will likely produce a more distinct scatter plot. Likewise, it's not clear that this figure easily supports the ensemble attack- there are nearly as many validation points with average loss and high MoPe scores.\n\n**Response 6**. We show the 12B scatter plot because the two largest model sizes are where the ensemble actually outperforms the MoPe attack in isolation. This makes sense, because for smaller model sizes MoPe is outperforming LOSS more significantly, and so including LOSS doesn't improve accuracy on average. We include scatter plots for all model sizes in Figure 10 in the Appendix. \n\n**Question 7**. Is it plausible that sampling more perturbed models will improve attack performance for larger models? As noted above, the perturbed models need only be computed once, so the main cost is evaluation of all the models over all the probe samples.\n\n**Response 7**. The reviewer makes a great point here: there is a tradeoff between speed and attack efficacy. The Hutchinson Trace Estimator does have less variance with more samples (albeit at a rate of decline proportional to the square root of the number of samples). In our experiments, we used a smaller number of models for large models because evaluation of thousands of points for testing becomes very expensive in its own right, and scaling the number of perturbed models increasing inference time linearly. Anecdotally we did not see a significant increase in attack success beyond `10` perturbed models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors demonstrate a strong and relatively efficient approach to MIA without requiring shadow model training.\n\nThe work is timely and relevant given the explosion of LLMs trained with data of dubious provenance.\n", "reasons_to_reject": "Success of the attack is really only demonstrated on auto-regressive LLMs.\n\nExperimental comparison with shadow-training approaches would be valuable, at least for smaller models, or explain why these are not explored.\n\nAn important limitation is white box access, but this is not a deal-breaker.\n", "questions_to_authors": "L145: our attack performs worse than loss in this setting --- any hypotheses for why would language models be unique?  Does it hold for LLMs with classification heads or that only produce embeddings, versus next-token prediction?\n\n2.2: it's worth noting that the random model perturbations need only be computed once (with a cost of storing n model snapshots) and can then be applied to an arbitrary number of probe examples.\n\nJust an observation that σ=0.005 seems quite large for model parameters that are close to 0. I imagine the hessian approximation breaks down if non-uniform noise is injected but I would be curious to understand if scaling sigma by the model weight impacts the attack success.\n\nIt would be valuable to have some error analysis on the quantities in table 1, eg using bootstrap sampling if running multiple experiments is too expensive.  When are the differences in performance statistically significant?\n\nL482 Training order: clarify what model is evaluated for this experiment\n\nWhy evaluate the 12B model in figure 4 when there is little discernable difference between MoPe and LOSS performance?  Evaluating a smaller model will likely produce a more distinct scatter plot. Likewise, it's not clear that this figure easily supports the ensemble attack- there are nearly as many validation points with average loss and high MoPe scores.\n\nIs it plausible that sampling more perturbed models will improve attack performance for larger models?  As noted above, the perturbed models need only be computed once, so the main cost is evaluation of all the models over all the probe samples.\n", "paper_topic_and_main_contributions": "The authors present a membership inference attack based on measuring expected changes in the model loss under random perturbations of the model. They show that the metric approximates the model Hessian, and present extensive experimental results demonstrating superiority of the attack over a conventional loss-based approach for a range of model sizes.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I'm aware of.", "Typos_Grammar_Style_and_Presentation_Improvements": "Missing closed parenthesis on line 288 (x\\in C)\n"}, "extra_text": {}}, {"review_id": "8hIbp2HsgJ", "reviewer": "Reviewer_Dn8Y", "summary": "", "questions": "A. Could you discuss potential solutions or mitigation strategies to address the security risks identified in this paper?\n\nB. The paper only considers whitebox access to the model's parameters, which may not be representative of real-world attack scenarios. How do you think the results of this study would generalize to blackbox or graybox settings?\n\nC. The paper shows that the loss of a point alone is insufficient to determine extractability, and there are training points that can be recovered using MoPe that have average loss. Could you discuss the implications of this finding for future research on memorization or \"unlearning?\"\n\nD. How do you envision this work being used in practice, and what steps can be taken to ensure that it is not misused by individuals seeking to exploit these vulnerabilities for nefarious purposes?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1**. Could you discuss potential solutions or mitigation strategies to address the security risks identified in this paper?\n\n**Response 1**: To our knowledge, there are no current defense strategies against MIAs against pre-trained language models; particularly since until this work, the privacy risks seemed limited. This is a good suggestion, and an exciting direction for future work that we are actively exploring, but would constitute a major paper in its own right. We discuss this in the last paragraph of the conclusion:\n\"Differentially private training (Dwork et al., 2016) is a canonical defense against MIAs, and there has been a flurry of 880 recent work on private model training in NLP (Anil et al., 2021a; Majmudar et al., 2022; Dupuy et al., 2022). While (Li et al., 2022) report success in fine-tuning language models with differential privacy, it is know that privacy during pre-training comes at a great cost to accuracy (Anil et al., 2021b). Since pre-training with differential privacy remains a challenge, no existing work provides theoretical mitigation guarantees against our attacks on pre-trained models.''\n\nOn a more practical note, maintaining control over the weights of the LLM prevents our MoPE attack. However, as we've noted above, and again in our next response, assuming that the adversary can't access the model weights is a brittle assumption that may not hold in practice.\n\n**Question 2**. The paper only considers white-box access to the model's parameters, which may not be representative of real-world attack scenarios. How do you think the results of this study would generalize to blackbox or graybox settings?\n\n**Response 2**. As we note in our Limitations section, we believe it’s worth studying white-box attacks both because “many open source or even proprietary models are often either published or leaked, and from a security perspective it makes sense to assume attackers could gain access to model parameters,\" but also because the success of these attacks is scientifically interesting in that it sheds light on how these models may memorize the underlying training data, in some cases without over-fitting. That being said, developing effective black-box attacks against pre-trained LLMs remains an exciting open question. Extending MoPE to the black-box setting seems challenging, because MoPe requires the ability to perturb the model directly.\n\n**Question 3**. The paper shows that the loss of a point alone is insufficient to determine extractability, and there are training points that can be recovered using MoPe that have average loss. Could you discuss the implications of this finding for future research on memorization or \"unlearning?\"\n\n**Response 3**. As we briefly mention in related work, “recent works studying issues of memorization or forgetting in language models […] measure ‘forgetting’ using membership inference attack accuracy, using an attack based on loss-thresholding with an example-calibrated threshold.\" We show in this work that it is possible for a training point to have loss that is approximately the average test loss, but to be easily identified as a training point based on an abnormal MoPe score. This implies that future work on unlearning or memorization should be wary of using simple metrics like the loss as a guarantee of either unlearning or memorization. This finding casts doubt on what the correct notion of ``memorization'' is for a machine learning model, and raises the question of if MIAs based on other metrics can perform even better than MoPe. \n\n**Question 4**. How do you envision this work being used in practice, and what steps can be taken to ensure that it is not misused by individuals seeking to exploit these vulnerabilities for nefarious purposes?\n\n**Response 4**. We envision this work (and the broader literature on MIAs) will serve as a toolkit model developers can use to assess the privacy of their models before releasing them to the public. By highlighting these vulnerabilities, we hope that it will cause model developers to be more wary of deploying models trained on sensitive data, and that our attacks will shed light on potential mitigation strategies. While we can't guarantee these attacks won't be used to compromise the privacy of real users (a critique that could be levied at any of the papers in the vast MIA literature), we believe that the benefits outweigh the potential risks, particularly as MIAs are a mild form of privacy violation relative to training data extraction for example.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include the development of a novel membership inference attack that has improved accuracy over existing loss-based attacks, the examination of the role of training point order and model size in attack success, and the demonstration that MoPe accurately approximates the trace of the Hessian in practice. \n\nIf this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be a better understanding of the potential security risks associated with large language models and a new method for identifying whether a given text is in the training data of a pre-trained language model. This could lead to improved security measures for language models and better protection of sensitive information.", "reasons_to_reject": "One potential weakness of this paper is that it focuses solely on the development and evaluation of a membership inference attack, without discussing potential solutions or mitigation strategies to address the security risks identified. Additionally, the paper only considers white-box access to the model's parameters, which may not be representative of real-world attack scenarios.", "questions_to_authors": "A. Could you discuss potential solutions or mitigation strategies to address the security risks identified in this paper?\n\nB. The paper only considers whitebox access to the model's parameters, which may not be representative of real-world attack scenarios. How do you think the results of this study would generalize to blackbox or graybox settings?\n\nC. The paper shows that the loss of a point alone is insufficient to determine extractability, and there are training points that can be recovered using MoPe that have average loss. Could you discuss the implications of this finding for future research on memorization or \"unlearning?\"\n\nD. How do you envision this work being used in practice, and what steps can be taken to ensure that it is not misused by individuals seeking to exploit these vulnerabilities for nefarious purposes?\n", "paper_topic_and_main_contributions": "This paper is about a membership inference attack called Model Perturbations Attack (MoPe) that can identify with high confidence if a given text is in the training data of a pre-trained language model, given whitebox access to the model's parameters. The paper shows that MoPe has improved accuracy over existing loss-based attacks and examines the role of training point order and model size in attack success. \n\nThe main contribution of this paper is the development of MoPe, which adds noise to the model in parameter space and measures the resulting drop in the log-likelihood at a given point, a statistic that approximates the trace of the Hessian matrix for model weights. The paper also shows that the loss of a point alone is insufficient to determine extractability, and there are training points that can be recovered using MoPe that have average loss. Additionally, the paper evaluates MoPe on various language models ranging from size M to B.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a membership inference attack method called MoPe (Model Perturbations Attack) to detect if a certain text is in the training data of a pretrained LLMs. The proposed method assumes access to model parameters. 

Pros: 
* The authors demonstrate that the proposed method is very effective in identifying if a given text is in the training data with a high confidence, compared to loss-based attacks. 
* The paper is well written and very timely. I believe this direction of work will lead to our improved understanding of security, safety and trust related issues with LLMs. 

Cons: 
* The reviewers have identified two major weaknesses in the paper: 1) The method assumes access to model parameters, and 2) the paper does not provide or discuss potential solutions or mitigation strategies to address the risks identified in this paper. 

The authors have responded to both limitations adequately in their rebuttals. Despite these two limitations, the reviewers have agreed that the proposed setup is worth exploring and the paper makes a valuable contribution which will be of interest to the community and to the development of LLMs.
Decision: Accept-Main