What Really is a Member? Discrediting Membership Inference via Poisoning
Authors: Neal Mangaokar
Ashish Hooda
Zhuohang Li
Bradley A. Malin
Kassem Fawaz
Somesh Jha
Atul Prakash
Amrita Roy Chowdhury
Venue: NeurIPS 2025 poster ()
Keywords: Membership Inference
Poisoning Attacks
Robustness
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Membership inference tests aim to determine whether a particular data point was included in a language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition to include semantic neighbors as members as well. In this work, we show that membership inference tests are still *unreliable* under this relaxation - it is possible to poison the training dataset in a way that causes the test to produce incorrect predictions for a target point. We theoretically reveal a trade-off between a test’s accuracy and its robustness to poisoning.  We also present a concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests to well below random.

What Really is a Member? Discrediting Membership Inference via Poisoning
Neal Mangaokar"1
Ashish Ho
Zhuohang Li
Bradley A. Malin
Kassem Fawaz! Somesh Jha Atul Prakash*
Amrita Roy Chowdhury?
f University of Michigan, Ann Arbor
University of Wisconsin-Madison
Vanderbilt University
Abstract
Membership inference tests aim to determine whether particular data point was included in language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition include semantic neighbors members as well. In this work, we show that membership inference tests are still unreliable under this relaxation in way that causes the test to produce incorrect predictions for target point. We thcoretically reveal a trade-off between a test's accuracy and its robustness to poisoning. We also present concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests well below random.
possible to poison the training dataset
Introduction
A central question in the machine learning (ML) community is whether model was trained particular data point []. While this question has long been of academic interest, the recent surge in large language models (LLMs) has made it more relevant across new practical contexts. For instance, these models are often trained on massive web-scraped datasets (2), which may include copyrighted content. This has sparked high-profile legal disputes between model providers and creative professionals (e.g., authors) E 5. centered on whether the disputed content was part of the training data. In another example, recent legal regulations worldwide have mandated auditing of ML models 6 77. In such cases, model owners may need to demonstrate that specific data points points-such those from the minority class indeed used during training, in order to support claims of fairness or regulatory compliance [8]
Currently, membership inference (MI) testing is the de facto approach for answering this question. Existing tests employ a variety of heuristics to analyze the loss landscape of the model, and output "membership score" - high score typically indicates membership. However, growing body of research has questioned the reliability of these tests p
10 11. key concern the ambiguity surrounding the definition of what it means for data point to be a "member." For instance, if "Harry Potter drew his wand" appears in the training data, should its paraphrase "The wand was drawn by Harry Potter" also be considered member? To address this, recent works have suggested relaxing the definition of membership to a neighborhood
one where semantic neighbors of a training point are also treated members [9] 12 [3]
In this work, we show that even under the relaxed, neighborhood-based definition, membership inference remains unreliable. We demonstrate this through a new lens dataset poisoning attack. Specifically, we consider a realistic threat model which an honest model owner trains an LLM using data scraped from the internet. Despite the owner's honest intentions, the internet remains
Indicates equal contribution. Now Google DeepMind.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fundamentally untrustworthy environment, where anybody can introduce poisoned data into public sources (for instance, by editing Wikipedia articles or posting on Reddit). Indeed, recent work has shown that such poisoning attacks are not merely hypothetical, but are feasible in practice [14]. Building on this threat model, we consider an adversary who poisons the training dataset dataase of the model with the goal of causing an MI test to produce incorrect predictions. Note that this is distinct from traditional poisoning attacks, which typically aim to trigger undesirable behavior in the model itself during downstream use (e.g., denial-of-servin or jailbreaking [15.. In contrast, our attack targets the MI test which is a separate classifier that operates on the model outputs/loss values.
In nutshell, we establish that currently MI tests are not robust to dataset poisoning attacks. To this end, our contributions are two-fold.
1. First, we theoretically demonstrate the inherent difficulty of designing robust MI test by iden- tifying a fundamental trade-off between the test's accuracy on clean data and its robustness to poisoning.
2. Second, we provide a concrete instantiation of a novel poisoning attack, PoisonM, that effectively exploits this trade-off in practice.
Our attack works as follows: for a target point 24 with ground truth membership label c, the adversary substitutes some points in the training dataset with carefully crafted poisoned ones that (1) preserve the true membership label e under the definition neighborhood- membership, but (2) cause the MI test flip its prediction 1 c. Consequently, the attack discredits the test's predictions.
Revisiting our carlier usecases of MI tests, we highlight real-world motivations of such attacks. Consider the case of copyright enforcement. An honest model owner may ensure that their training dataset, under a mutually agreed-upon neighborhood definition, contains no points related to the new Larry Lobster novels. However, disgruntled author could plant poison outside this neighborhood that still triggers a (false) positive prediction, potentially enabling a baseless copyright lawsuit. Similarly, in the example of faimess audit, adversary could plant poisons within the neighborhood of minority class points, causing the MI test to falsely predict non-membership (false negative), thereby undermining the model owner's credibility.
Intuitively, the attack is possible due to the misalignment between superlevel sets of the MI test (points that when trained upon elicit high test scores, i.c., indication of membership) and the existing notions of neighborhood (see Figure D]. We provide a concrete implementation of the poisoning attack, PoisonM, for four popular notions of neighborhood: n-gram overlap, embedding similarity, edit distance, and exact matching (i.e. the traditional notion of membership). PoisonM is MI test agnostic, can target multiple points simultaneously. and highly efficient (only substituting a single clean point with a poison is sufficient to induce false negatives). We evaluate PoisonM against several MI tests across different datasets and model sizes, and find that it consistently flips test predictions and degrades performance well below random. Thus, our results highlight a disconnect between how MI tests operate and how their outputs are interpreted determine membership in practice, calling for a re-evaluation of what it truly means for point to be member.
2 Background
MI tests typically rely on thresholding model loss or its variants, such as LOSS [6], Min-K [17 (loss on least likely tokens), zlib[[8] (loss-to-entropy ratio), perturbation-based tests (loss differences with perturbed inputs (19]), and reference-based tests (loss ratio to another model (8)).
Unreliability of Membership Inference. Recent work has already begun to highlight concerns regarding MI testing. For example, many works evaluate performance of tests on datasets that exhibit distribution shifts, which is flawed because members can be separated from non non-memeers without even using the model 19, [ 20 Other work discusses how a dishonest model owner can refute the predictions of an MI test by providing a certificate that model could be obtained without training on a specific point 21). More recent work has also touched on how poisoned models can surprisingly amplify MI test results for finetuning data (22). We differ from these works in that we show how MI tests can be manipulated to provide entirely wrong predictions.
Neighborhood- Based Membership Inference. The premise of an MI test relies upon the definition of membership, which traditionally labels a text sequence as a member if it is exactly matches a
'While our theoretical results are general and apply to both pre-training and fine- tuning. our empirical evaluation focuses on the latter.

---

Space Plaabbbbee Sequencex
Space Passible Sequences
Space Possibll Sequencer X
Superiavel of XITO 4DD.
Werst-case Non noighaat
*
Other Nan-naighher
Other Neightor
Neighoorhood
(a) Misalignment between neighborhoud- based membership definition and the test
(b) Points in represent worst-case neinnnigghor for Yy Le., replacing any training points
(o Points in representt worst-case neighbor for Xe, l.t. replating any training points
with points [ Iwolld cause membership test score unexpectedly increase
with points would cause membership test score unexpectedly decrease
[
Figure 1: MI tests aligument between the membership neighborhood and the test's superlevel set, one can alter datasel- changing the ground truth label of a target 24 by substituting non- non-ncighbors with worst-case non-neighbors to cause the test T, to mispredict 2; as member. Conversely, replacing a neighbor with worst-case neighbor make true member appear non- mommeererr.
robust membership invariant perturbations (e.g. substitution). By leveraging the mis-
sequence in the training set. However, this definition poses two key problems. First, training and non- training texts often overlap substantially, making exact- match definitions of membership ambiguous and leading to unreliable performance for MI tests 9 10 For example, suppose copyrighted text is removed from the training set. These tests may still flag them as members simply because the training set includes related content (e.g.. from online discussions)
[10]. Second, the standard definition of membership, as exact presence the dataset, too narrow if it is used to measure whether a model is leaking training data. For example, privacy leak can happen even if the model outputs a rephrased version of a sensitive medical record. Indeed, recent work has shown that language models can often complete sequences that they were not explicitly trained upon or even share n-grams with 13 To address these concerns, recent works have advocated for a shift towards more flexible, neighborhood haobbr-aoodd-dadddd definitions for LLMs where membership is determined by semantic lexical similarity 19, [2]. The key contribution of this work is to show that membership inference still unreliable under these relaxed definitions.
3 Neighborhood Membership Inference: A General Framework
Notation. Let X' be the space of token sequences vocabulary V. Model parameters 9 of an LLM are learned via algorithm L on dataset E P(X), i.c., L(D). where P denotes the power set. Let D denote the distribution over the datasets. We define the symmetric difference between datasets D2 and D2 as D2
For given text sequence x E X, its neighborhood is formally defined as follows:
Definition 3.1 (Neighborhood). Let d x X' + Rzo be a distance metric on the space input sequences X. For given X and radius 0, the neighborhood N₁ X' P(X) defines ball of radius centered at x and is given by: Nr, (xx {a' E X' d(æ, x') S r).
We will refer to points in (x) as "neighbors" of xr and the complement Nr N+(x) of the neighborhood then simply all points that are "non-neighbors" of x. i.e,,N X' (z)). If model is trained on point 2, we would like to treat its neighbors AG-(x) (x)) as approximate members. Instantiating neighborhood with a radius of r = 0 recovers the traditional exact-matching notion of membership. Using this, we denote neighborhood-baae
membership:
x Ex.. D - 3re E D. E (N) and x EN. + Va' e D. (x). (1)
In light of this, the score assigned by an MI test to point should be interpreted least one of its neighbors was used to train the model instead of an indication that x exactly matches a sequence from the training set. Following the formalism from [0), we define an MI test as follows:
signal that at
Definition 3.2 (y-Thresholded Membership Inference). Given neighborhood N,, an MI test is a mapping T, X' x θ + Rzo, which, for any data point x and model parameters 0, returns "membership score" for testing the null hypothesis that x EN. D. Scores above threshold 7 Rzo suggest membership: [T, (z,,0 0) 2 7] ~ x Ex, D.
3

---

When is a Membership Inference Test Sound? Here, we adapt the standard metrics metrics-sensitivvin (true positive rate) and specificity (true negative rate)-to the pointwise setting as follows:
Definition 3.3 (Pointwise Membership Sensitivity (True Positive Rate)). The sensitivity of an MI test Ty for point with respect to neighborhood N, the probability that the test correctly identifies sequence member:
Sens(T x)
Pr D
(T, I Ix D).
(=C(D)
Definition 3.4 (Pointwise Membership Specificity (True Negative Rate)). The specificity of an MI test Ty for point with respect to neighborhood Nr is the probability that the test correctly identifies a sequence as a non-member:
Spea(Ty,,
Pr (=C(D)
(I,
<> EN. D)
Intuitively, the more separable the score distributions of a membership test for models trained/not- trained on any neighbors, the more powerful the test. The following result connects the specificity and sensitivity to the separability of the memhership test scores under both hypotheses:
Lemma 3.5. (Advantage of an MI test). For point the advantage of test I, is given by the difference between the expected membership scores under the null and the alternative hypotheses:
(Sens(Ix, (Iy,2) + Spec( 2) - 1)d7
(Ir(a,0) I CN. D) C(D)
E
E D D
(I, EN. D).
L(D)
Advontage over random guexy
Expected
alternative
mider
The proof is in Appendix A.2 The integrand on the left is commonly known Youden's 's statis- tie [23], and captures the difference between the true positive rate and false positive rate, i.e., the advantage over a random guess. Later Section we build upon this result show the difficulty of designing robust MI tests.
4 A Robustness Perspective on Membership Inference
We begin by defining family of membershiip-nvarianr perturbations to the dataset, i.e., perturbations the dataset that do not change the membership label for some point:
Definition 4.1 (Membership Invariant Dataset Perturbation). Given an original dataset D ~ D, dataset perturbation operator Pert: P(X) + ((X) is membership-invartie w.r. a point a if:
EN. D
Éx.. Pert D) and x Ex, D = Ex,, Pert(D).
In other words, a point's membership label should remain unchanged under perturbations mmnnnerss stay members, and non-members stay mememrsmer..... Ideally, membership test should be robust to such perturbations. We focus on a specific type of perturbation via substitution: replacing neighbors (or non-neighbors) of with other neighbors (or non-neighbors). We denote such perturbed dataset to lie in the expansion of the original dataset D.
Definition 4.2. (Dataset Expansions Under Substitution) The b-neighhorhood expansion of a dataset D around point x: (for some notion of neighborhood NV,) is the set of all datasets that can be made by only substituting b points (from D) that lie in (z) with other points from Nf, (x)... Similarly, the -non-neegghoorhood expansion arises by substituting points that lie in (x) with other points in N, (x Concretely, with (N,,
these expansions are given by:
= c x |D'| DD),
S'AD < 2b).
In this paper, we explore the problem of constructing worst-case datasets from the expansion of an original dataset Dht is. perturbed datasets in which remains non-member (since only non-neighbors were substituted with other noi-neighbors), yet the test incorrectly classifies it as a member. Conversely, one can also construct examples where r remains member, but the test incorrectly predicts it to be non-men non-member. An illustration of this phenomenon is provided in Figure [
Without loss of generality, assume there b such sequences in D to begin with.

---

Such worst-case datasets are interesting because they contend with the reliability of test - if test can be arbitrarily made fail on a point, despite the ground truth (i.e., membership label) being unchanged, is the test still useful? Furthermore, such worst-case datasets have practical, real-world implications under a poisoning threat model.
Threat model. In our setting, the adversary can be anyone capable of planting poisoned data on the web, such as through poisoning publicly available sources like Wikipedia backups [14]
Formally, such scenarios can be characterized as a game between a challenger C, an adversary A, and an arbiter J. Here, the adversary A's goal is for the test to assign incorrect membership predictions:
1. Adversary A chooses a target point 2, E X and sends Iş to the challenger C.
Challenger samples the training set D, such that Ia Ex, Dr, and Do such that ia Éx. Do. and sends (D,, Do) to adversary A.
3. Adversary A poisons D, with budget to obtain Dr E (z))),,N and poisons Do to
obtain D E B (,,(),Nr((z)), Poisoned datasets (Dy, D)) are sent back to challenger C
Challenger C flips random bit с and trains model parameters as = ((D)) if c = and 0 L(D%) if otherwise. Challenger then sends (zra, arbiter J.
5. Arbiter I leverages membership test output prediction bit as č (x, 0).
6. Adversary A wins if è +
Note that, in a departure from typical security games, we introduce an additional arbiter J to capture the fact that the goal of the poisoning is to mislead the membership test evaluated by an arbitrary third-party. This also reflected in the real-worke motivating examples discussed earlier, where the arbiter may be judge ruling copyright violation case or auditor assessing a model's fairness.
In order to characterize the success of such adversary, extend the performance metrics from Definitions B.3 and 3.4 to account for the effects of poisoning:
Definition 4.3 (Pointwise Robustness). The robust sensitivity of test for point w.r. neigh- borhood N, defined as the probability that, after substituting neighbors with their worst-case neighbors, the test still correctly classifies x as a member:
RSenso (T,,
min ((..)))
x Ex, Tx(2, 27 (=C(D)
(2)
DND
Similarly, for robust specificity:
RSpec, (T,,
Pr D-D
max ('E,,(DDV.C)) 6=2(D')
Tx(2,,) <9
EN.
(3)
In Equation 2 abooe, the inner minimum represents the adversary's replacement of the original dataset D (of which 2 is member: Ex, D) with carefully chosen dataset D' such that decreases the membership signal for z in the trained model, while maintaining x member status, i.e., Ex. D'. Concretely, this dataset is obtained by substituting b neighbors of x with other carefully selected neighbors of æ such that the test's s output score is minimized. Similar observation holds true for Equation
3 except we are now trying maximize the test score.
Note. The above measures of robustness are similar in spirit to those employed for the widely studied concept of adversarial robustness [24 (c.g., robust accuracy). However, our setting introduces key distinction: worst-case analysis is instead performed over perturbations of the training dataset, rather than perturbations of the individual point, i.e., b-expansions of D instead of lp norm æ.
PoisonM: Poisoning Membership Inference
In this section, propose PoisonM, concrete instantiation of dataset poisoning attack MI tests. We begin with an overview and follow with implementation details.
FFor generality, we allow the adversary to poison both datasets. However, the adversary could just as well poison only one of the datasets without altering rest of the game.
5

---

Overview. To construct poisoned dataset for target point 21, the adversary A must provide:
Spoison E
(y 0), arg min D' D'E.. (=C(D)
arg max
Ty (xt, 01).
D' G=C(D')
W.I.o.g, let's consider the case where the adversary wants induce false positive, i.e., the target av originally not a member. The goal is to substitute "clean" non-neighbors of a, with worst-case, i.e., "poisoned" non-neige non-neighbors such that the membership test T,, incorrectly assigns high membership scores to ž₁ (see Figure [T. The key insight of PoisonM is that, to find such poisoned non-neighbor, actual neighbor ssample and then (2) "mmap" this neighbor back to non-
one (1) sample neighbor that T--qqiivalent to wanple-maanin
that training on either point causes T. to assign the same score to Zt. This mapped non-neighbor 2 poossnn is thus the poison. If a model trained on Epoison the MI test should ideally produce the same output on 2y as it would if Taapll had been in the training set instead. The success of PoisonM can be formalized as follows:
PoisonM(rr DJ (Imasple arg min (Ty, (22, L(D U { (Tooismn) })) (2,, C(D (E), (Tamspta))s H],
(4)
Sposesn ES
Denoted by (....
where E (I)())))))))))) is the domain of the mapping in which the poison should lic, and (a).D) is the mapping error, i.e., how much the poison differs in T,y score for 2e as compared to Omneppt
T Usampla- Extending this to find b poisons, we define:
PoisonM 2t. DY () (121, ... Ibb Jampples S)
arg min (1) ppsseo ES
[T(2e, L(D (Ii, 26
poison)) (olismn()))piso))) T(æi, L(D (2), (.., sampta)))
and the mapping error is given by g(a..))) Jompsee Here b is referred to the budger of the attack.
We now provide a result that demonstrates the difficulty of obtaining a robust MI test.
Theorem 5.1. (Tradeoff of Membership Inference Under PoisonM), Let be target point. The advantages of a test T, with and without poisoning are at odds with each other:
RSeens,, (1),2s) RSpecty (Iv, 1) + [
(Sens( Ty,, xc) + Spec (T, - 1)d7 50¹
Advontage with poizoning (raiustnen)
Advaniage withiut poisoning
D')).D)) (1 Fo hemple
D'a) (z)." =222)uplee
where 8*
E
D-D
D-D
(1) Jby J-D (.
(Ri ( (21 F2g Du
M J-D n (xs)
(Z1 Fyy la (a,)
D' D' ) bi DDN
D'= D'=Di(a (og)
Expected mapping error for polsoned neightbrr
Expected mapping error poisoned non-neighborr
The proof is in Appendix A.7 The R.H.S represents the expected mapping error 8*, while the L.H.S captures the total advantage (over random guessing) of the MI test, both in the presence and absence of poisoning. When PoisonM's mapping error 8" is small-ia. i.e. the attack is successful -th theorem above implies surprising insight: the advantage of the membership test can be turned against itself. Specfically, the better the test performs (as measured by Youden's. J statistic) on clean points, the more vulnerable it becomes to our poisoning attack. To build intuition, consider a scenario where the adversary aims to induce a false negative by constructing a poisoned neighbor. In the ideal case where 8* 0, the poisoned neighbor has the same effect clean non-neighbor to T.,, As result, T., assigns to x, the same (low) score as it would assign if trained on the clean non-neighbor. This effectively fools T. into making incorrect prediction, exploiting its own strength in distinguishing members from mon-members Thus, the above result delineates a fundamental trade-off for an MI test: strong performance clean data comes at the cost of robustness to poisoning attacks. This is also empirically validated in Section
A natural question arises: for given E , do such low-error poisons actually exist? In practice, they often do-thi, stems from misalignment between the "balls" defined by the generic notions of neighborhood, and the actual superlevel sets of the test, which define regions that trigger high

---

Table 1: Details of the PoisonM attack for different definitions of neighborhood (Co represents the LLM).
Distance Metric
Definition
Poisoned Neighbor Loss
Poisoned Non- Neighhor Loss
folopoionn fo(())
n-gram
Neighbors share com- mon R-gram
7-gram| (petson,
Sposscn
[|]]]]
n-gramm 2x)
Pppcioa)) E(are)
poloon folor)
E(F F(ate)
Fibedding
Neighbors have cosine similarity > under a se- mantic embedding func-
E (2passon)) E(x)]
postson
[Bel]
E (posson III [(a)]]]
tion F
Ioldoooon folst)
Edit Distance
Neighbors have normal- ized edit distance I
edit( 2z)
A- odit odit( (t)
Spassen [
Ja possen))))) fa(ar)
Exact Match
Only point itself is con- N/A since neighborhood sidered neighbor
radius is 0
pottool |rell
Algorithm 1 PoisonM Attack
1: Input: Target point 2z, Neighbothood NG., Pretrained model P:
2: Oulpul: Poison point Epottori
3: (ar,) (Flipping member non Member) (1z) (Flipping non Member member)
4: Zouaple ~ S
5: Ppsism Toumple
6: while Spoiiin E S do
7: ~ ~>-Unffrrm((,, ... ..........((.Ppeee. ((#peteon]))
8: Subatitute; (foosso min(Lossg Perisaa 8, (x))) (o Substiitte Ses, token to minimize
9:
end while
10:
return Spotion
membership scores for This is illustrated in Figure 1 poisons exist in the small gaps between the contours of the test's 's superlevel sets and the actual neighborhood boundary. This phenomenon is reminiscent of adversarial examples in classification, where there is a well known gap between Ip balls and the superlevel sets of the classifiers (or decision boundaries) 25
Implementing PoisonM for Different Neighborhoods. PoisonM involves solving the discrete optimization in Equation 4 tailored the chosen neighborhood. Our general method, outlined in Algorithm]] I follows greedy coordinate descent approach inspired by Zou et al. [26). Given a target Tr, we sample a neighbor or non-neighbor, then iteratively (1) select random token and (2) replace it with one that minimizes neighborhoodd-seeifii poisoning loss. For poisoned non-ncighbors, we maximize distance while preserving model activations to mimic the sampled point's 's influence on zt- For poisoned neighbors. we minimize the neighborhood distance and stop once the point qualifies as neighbor. The losses for four popular choices neighborhood are in Table Notably, PoisonM is MI-lesi cypostic-given a neighborhood definition, single poisoning strategy is effective across all evaluated MI tests, and the adversary needs no knowledge of which specific test will be employed.
6 Evaluation
6.1
Experimental Setup
Models and Training. We the Pythia models (2), primarily the 6.9B variant, with ablations on 2.7B and 12B. All models are fine-tuned (for 1 epoch) on poisoned data using Adam W (Ir = 2e-5, batch size 16). We focus on finetuning setting, i.e., the adversary poisons the finetuning dataset. Datasets. Following (27). the model is finetuned on a mixture of a "canary" and a "background" dataset, where we will run membership inference on the canaries. We use Wikitext-103 and Al4Privacy/AGNeet
background
as canary datasets, injecting 500 canaries into 100K background points and holding out another 500 canaries for evaluation. Membership labels are assigned based on neighborhood definition: although only 500 canaries are in the training set, points from the hold-out dataset may also be considered members if they have neighbors in the training dataset. For each definition of neighborhood, we construct single poisoned dataset in which we generate poison neighbors with budget b, for members, and poison non-neigh with by = 10 for the rest. The resulting model should flip membership status predicting members as non-members and vice versa. Metrics. We select 5 popular tests: LOOSS [16), Min-K% Prob [7 with K 0.2, zlib [T8), perturbation-base (28). and reference-based
[8] For perturbation and reference- based tests,
7

---

Table 2: Natural and robust AUC scores of MI tests on the AI4Privacy and AGNews canary datasets.
AtiPrivacy
AGNews
N-
MI Text
Natural
Token Deopouts
Casing Plips
Chunk
Poceonk
Natural
Token Dropouts
Casing Fligs
Chunk
Potnoch
LOSS
0.587
0,380
0.451
0.570
0.253
0.617
0.345
0.399
0.621
0.208
0,561
0.471
0.496
0.556
0,408
0.594
0.478
0.439
0.594
0.417
7-gram
dib
0:453
0.490
0.585
0.589
perturb
0.600
0.420
0.547
0.586
0.274
0.625
0.374
0.408
0.835
0.243
reference
0.647
0,250
0.398
0.618
0.089
0.623
0.325
0,373
0.637
0.174
LOSS 0.548
0.087
0.075
0.072
0.043
0.577
0.063
0.067
0.064
0.036
kmin
0.516
0.780
0.279
0.370
0.233
0.529
0.340
0.545
0.351
0.337
Exact
dib
0.186
0.108
0.109
perturb
0.570
0.098
0.098
0.098
0.059
0.585
0.050
0.063
0.058
0.031
reference
0.625
0.044
0.036
0.037
0.022
0.514
0,047
0
0.051
0.026
LOSS 0.552
0.456
0.560
0.454
0.390
0.582
0.492
0,602
01,41
0.371
kmin
0,512
0,485
0.517
0.462
0.454
0.539
0.509
0.564
0.519
0.427
Fmbhokling
0,513
0.556
0.481
pertubs
0.510
0.5%6
0.589
0,608
0.479
0.378
reference
0.632
0.422
0.649
0.428
0.281
0.645
0.534
0.662
0.530
0.403
LOSS 0.572
0.549
0.523
0.478
0.430
0.502
0.587
0.576
01:055
0.389
Kmin
0.542
0.536
0.522
0.496
0.502
(1.540
01.533
0.511
01,400
0.432
Edit Distance
zibb
0.529
0.470
0.542
0.529
0.474
0.411
perturb
0.520
0.612
0.505
0.596
0.523
reference
0.642
0.594
0.589
0.446
0.337
0.619
0.614
0.603
0.506
0.381
Table 3: Natural and robust p-values of dataset inference on the AI4Privacy dataset. M(J) and NM(f) indicates that the test should be outputting low p-values for members and high p-values for non non-members; successful poisoning should instead elicit high p-vuuu for members and low p-values for non-members.
Nr.
Naturl
Tokes Dropouts
Casing Fligs
Chanking
PoisouM
(.)
NM)
M.)
NM)
M.)
(M)
(.)
NM
M))
NM)
7: 7-gram
0.007
0.952
0.197
<de-3
0.019
0,013
0.003
0.397
0,999
<de-3
Exact Match
0.129
0.999
<Ie-3
<ll-3
<te-
<Ie3
<le:3
<le3
<le-3
<le-3
kimbediing
0.011
0.955
0.516
0:003
0.077
0.276
0.967
0.072
evaluate all configurations from Maini et al. 20 and present the setting that performs the best for membership inference for each neighborhood definition. Metries include AUC and TPR % FPR. We also evaluate dataset inference (20). a test providing p-values for aggregated membership prediction. We use 4 neighborhood definitions, with fixed parameters: h = 7 for n-grams, 0.9 for cosine similarity Baselines. As baseline for generating poisoned non-neighbors, we adapt Liu al's 13 recent work on three techniques for forcing completion on sequences not trained upon. These are: (a) dropping tokens regular intervals, (b) flipping the case of characters probabilistically, and (c) inserting chunks into a sequence of random tokens (also similar to (3I). To maximize the performance of baselines, we further perform hyperparameter search for each approach select the least destructive parameter values (e.g.. drop rate, flipping probability, chunk size) that still maintain non-neighbor status. For generating poisoned neighbors, to the best of our knowledge there are no existing baselines.
0.48 for normalized edit distance, and exact match.
6.2 Experimental Results
Overview. We present the full ROC curves of tests before and after poisoning on the A14Privacy and AGNews datasets in Figure 2 with AUC scores presented in Table 2 Since have trained the model on both poisoned neighbors (to make members look like non-members), and poisoned non-neighbors (to make non-members look like members), we expect the AUC to considerably reduce. Indeed, we observe that PoisonM is nearly always able to reduce AUC below random. In many cases, it can be reduced considerably below random, even close to 0 (e.g.. n-gram neighborhood with reference-basee test). While the baselines are effective in some cases, PoisonM consistently outperforms them since they were not designed to manipulate membership testing. This advantage likely stems from two key factors: (1) PoisonM's 's ability to generate poisons for both neighbors and neighbbss,ss, and (2) the greater effectiveness of its poisoned neighhoos,,,, as shown in the exact match setting wehee all methods are limited to poisoning non neighbes only.
We also observe a general trend that aligns with Theorem 5.1 the tests that perform the best naturally are also the lowest/rank low in terms of robust AUC. For example, the reference-based test
"Embeddings are computed using Microsoft's Multilingual E5 Large text embedding model 29] which currently ranks highly on the Massive Text Embedding Benchmark (30].
8

---

Standard
Token Dropouts
Casing Flips zlib
Chunking perturb
PoisonM reference
LOSS
kmin
0.8
0.4
0.2
0.20.40. 20.60.00.8 60.8
0 0.20 40..40.60.8 60.8 FPR
2..0..0.06..0 FPR
0 0.20 20.48..0..8 FPR
0.20 220.40.600 40.60.8
FPR
FPR
Figure 2: ROC curves for MI tests using the n-grram (k=7) neighborhood on A14Privacy.
ranks the highest naturally, and the lowest under poisoning across all AI4Privacy settings, and in many settings in AGNews. We also present the TPR 1% FPR in Table of Appendix where find that again, PoisonM is able to reduce performance.
Dataset Inference. Dataset inference extends MI testing whole datasets by (1) ensembling existing tests via a linear model and (2) using a T-test to compare scores from suspect set to reference set of known non-members 20 We test whether poisoning affects this method by evaluating it
The results show that dataset inference fails under poisonnng yielding incorrect predictions. This aligns with intuition: if individual tests are driven below random, so is their ensemble- mocemblimmiroringg how ensembling weak defenses fails for adversarial
models fine-tuned with poisoned data (Table 3)
examples 32]
Impact of Neighborhood Radius. We examine how changing the neighborhood radius affects poisoning, focusing on the LOSS test with n-gram neighborhoods on A14Privacy for lo 111 (Figure 3}. As expected, larger radii (smaller k) reduce vulner- ability to poisoned non-members but increase it for poisoned members, and vice versa.
PoissaM (TPR)
Impact of Model and Dataset Size. We also study how model size affects poisoning success by repeating our n- gram (k = 7) experiments on A14Privacy using Pythia 2.7B and 12B. PoisonM consistently reduces test perfor- mance across all sizes (see 7f Appendix B). The 2.7B model shows slightly lower natural accuracy and slightly higher robustness, aligning with Theorem 5.1 In Table (8) we repeat our experiments with larger back- ground dataset size of M WikiText points, and find that PoisonM continues to be effective.
Metric value
Poisouht (FPR)
12
n--gram size Figure 3: TPR and FPR of the LOSS test after poisoning using n-gram neighborhood definitions [5, 7,,7,9. 7, 11] on AI4Privacy.
Filtering as a Defense. PoisonM is designed to be strong, general-purpose attack and can adapt to defenses by incorporating filtering criteria such as a perplexity filter - directly into its loss function. To demonstrate this, Table considers a setting in which perplexity filter is employed (threshold selected to ensure low FPR of 1%), and shows PoisonM can adapt to this setting and remain effective against all considered MI tests.
Poison Transferability. Even under a more restrictive setting where query access the target model is prohibited, an attacker can optimize against a surrogate model, with the expectation that the poison will transfer to the target model. In Table 5 experiment with OLM02 7B [33] model trained on poisons computed against itself (as is typical), and then on poisons computed against a surrogate Pythia 6.9B model. We find that PoisonM still effective against MI tests even this completely blind setting, without query access to the target model.
7 Conclusion and Discussions
We have studied the reliability of membership inference against LLMs under poisoning attacks. Although the shift from exact matching to neighborhood- based definition aims to enhance reliability of MI tests, we reveal fundamental flaws remain even under this relaxed definition, calling into question what it truly means for a data point to be considered member. Moreover, the wide applicability our attack across common neighborhood definitions highlights inherent difficulties in designing generic yet meaningful notion of membership. One possible way forward is consider

---

Table 4: Natural and robust AUC scores for MI tests using the n-gram (k=7) neighborhood on A14Privacy with perplexity filtering.
Table 5: Natural and robust AUC scores for MI tests using the n-gram (k=7) neighborhood on A14Privacy for OLMO2 7B.
MI Test
Natural
(wlo Filter -> w/ Filter)
PoiscuM
MI Test
Natural
PaiscuM (w/ OLMG2 7B)
Poiscalt (wl Pythia 6.9
LOSS
0.587
0.252 - 0.322
LOSS
0.567
0.276
0.313
kmin
0.561
0,408 0.444
kmin
0.542
0:412
0:435
zib
0.375
0.554
periurb
0.600
0.274 - 0.340
perturb
0.587
0.303
0.334
reference
0.647
0,089 - 0,143
referenee
0.500
0.255
0.298
model-dependent or context-aware definitions that better align with how MI tests actually operate. Finally, although we primarily focused on textual data, we expect our analysis to generalize beyond the language domain. For example, while approximate membership defimitions for image data may consider transformations such as rotation, cropping, and filtering, such definitions are likely to suffer from similar robustness issues.
Limitations. One limitation is that our attack for generating poison non-neighbors requires multiple poisons, i.e., 62 = 10. Future work may improve upon this. We also focus our experiments on specific settings, and larger-scale evaluations with more models/datasettsneiihhbhoe evaluating what it truly means for a point to be a member. We also do not evaluate the pre-training setting due the computational costs, although has been shown be viable 34
can help towards
Acknowledgements
This work is supported by National Science Foundation Graduate Research Fellowship Grant No. DGE 1841052. Any opinion, findings, and conclusions or recommendations expressed in this material those of the authors(s) and do not necessarily reflect the views of our research sponsors.
References
[1] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks against Machine Learning Models. In Proc. of IEEE S&P, pages 3-18. IEEF, 2017.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kylc O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia:
A Suite for Analyzing Large Language Models Across Training and Scaling. In Proc. of ICML, 2023.
[3] The New York Times. The Times Sues OpenAl and Microsoft Over A.I. Use of Copyrighted Work. https http////busieeee/eedeeeeeee lawsuit htmml 2023. Accessed: 05/15/2025.
//www. www.inytmme..... , cwm/2mm3
york- times- open- ai- micooroft..
[4] The Guardian. US authors' copyright lawsuits against OpenAl and Microsoft combined in New York with newspaper actions. https:: www theguardian com / books/ 2025 / apr / 04 us
cothora--ooyriiht-- www.uitswwtttass. against- openai- and www.:miiicoooccooooccoooce comb -nev-y-orr
with-nesspaper-actiin 2025. Accessed: 05/15/2025.
[5] Reuters. Publisher Ziff Davis sues OpenAl for copyright infringement.
com/
business
puublissherrr. business/publisherr
-zff davis sues- openai copyright- infringement - 2025 04- 24/
2025. Accessed: 05/15/2025.
[6] isaca.org. Understanding the EU Al Act: Requirements and Next Steps. https://ww/ httas://ww.isaaa. org/ resources//hite- ptt.ttteeepppps///22////2 papprss/202//neerstaddiin the a1 act 2024. Accessed: 05/15/2025.
[7] Ben Chester Cheong. Transparency and accountability in Al systems: safeguarding wellbeing in the age of algorithmic decision-makieee Frontiers Human Dynamics, 6:1421273, 2024.
[8] Sabina Lacmanovic and Marinko Skare. Artificial intelligence bias auditing current approaches, challenges and lessons from practice. Review of Accounting and Finance, (ahcad-of -hhiat-.--print,. 2025.
[9] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do Membership Inference Attacks Work on Large Language Models? In Proc. of COLM, 2024.
10

---

[10] Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data. arXiv preprint 2Xix:2449.11978. 2024.
[11] Debeshee Das, Jie Zhang, and Florian Tramer. Blind Baselines Beat Membership Inference Attacks for Foundation Models. arXiv preprint 2rXiv:2406.1622 2024.
[12] Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What Does it Mean for Language Model to Preserve Privacy? In Proc. of FAccT, 2022.
[13] Ken Ziyu Liu, Christopher A Choquette-Cuun Choo, Matthew Jagielski, Peter Kairouz. Sanmi Koyejo, Percy Liang, and Nicolas Papernot. Language Models May Verbatim Complete Text They Were Not Explicitly Trained On. In Proc. of ICLR, 2025.
[14] Nicholas Carlini, Matthew Jagielski, Christopher Choquette- Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-scale Training Datasets Practical. In Proc. of IEEE S&P, pages
IEEE, 2024.
[15] Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini. Florian Tramèr, and Daphne Ippolito. Persistent Pre- Training Poisoning of LLMs. In Proc. of ICLR, 2025.
[16] Samuel Yeom, Irene Giacomelli, Matt Fredrikson. and Somesh Jha. Privacy Risk Machine Learning: Analyzing the Connection to Overfitting. In Proc. of CSF, 2018.
[17] Weijia Shi, Anirudh Ajith, Mengzhou Xia. Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. In Proc. of ICLR, 2024.
[18] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts. Tom Brown, Dawn Song. Ulfar Erlingsson, et al. Extracting Training Data from Large Language Models. In Proc. of USENIX Security, 2021.
[19] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kinn Kerg-Kirkpatrick. Membership Inference Attacks against Language Models via Neighbourhood Comparison. In Proc. of ACL, 2023.
[20] Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. LLM Dataset Inference: Did you train my dataset? In Proc. of NeurlPS, 2024.
[21] Amrita Roy Chowdhury, Zhifeng Kong, and Kamalika Chaudhuri. On the Reliability Membership Inference Attacks. Proc. of SaTML. 2025.
[22] Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models. In Proc, of NeurlPS, 2024.
[23] William Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35 35, 1950.
[24] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In Proc. of ICLR, 2018.
[25] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Propertics of Neural Networks. In Proc. of ICLR, 2014.
[26] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.1504 2023.
[27] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda. Vincent Bindschaedler, and Reza Shokri. Enhanced Memhership Inference Attacks against Machine Learning Models. In Proc. of CCS, 2022.
[28] Filippo Galli, Luca Mclis, and Tommaso Cucinotta. Noisy Noighbors: Efficient membership inference attacks against LLMs. In Proc. of the Fifth Workshop on Privacy in Natural Language Processing, 2024.
[29] Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 Text Embeddings: A Technical Report. arXiv preprint arXiv:2402.05672. 2024.
[30] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark. In Proc. of EACL, 2023.

---

[31] Anchrel-Annrei Panaitescu-Liess Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che. Bang An, Sicheng Zhu, Aakriti Agrawal, and Furong Huang. PoisonedParrot: Subtle Data Poisoning Attacks Elicit Copyright-Infringe Content from Large Language Models, Luis Chiruzzo, Alan Ritter, Lu Wang. editors, Proc. of NAACL, 2025.
[32] Warren He, James Wei, Xinyun Chen. Nicholas Carlini, and Dawn Song. Adversarial Example Defense: Ensembles of Weak Defenses are not Strong. In Proc. of USENIX WOOT, 2017.
[33] Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, et al. 2 OLMo 2 Furious (COLM's Version). In Proc. of COLM, 2025.
[34] Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramer, and Daphne Ippolito. Persistent Pre- Training Poisoning of LLMs. In Proc. of ICLR, 2025.
12

---

A Proofs
Lemma A.1. (Expectation using survival function). Let X be a random variable such that P(X 2
1, then we have
E[X] =
P(X > 8) ds.
==0
Proof.
R [
x: P((X =x)dæ:
x=0
[
P(X = x))
ds dæ
5=
P(X = x) dx ds
1=0 a=8
P(X 8) ds
Lemma A.2. (Restatement of Lemma 3.5
The advantage of membership inference test is given by
E (T(2,)) x Ex. D)
E (T(x, 0) x EN. D)
(=L(D)
6=L(D)
(Sens(T, + (, - 1).
7=0
Proof. Using Lemma A.1
we get
E D-D (=)(
(T (2, x EN. D)
x EN,, D) (T, (=((D)
E
D-D
00
(T, 7 En. D)d7 -
(T, (x, 0) > 7 x EN. D)d7
7=0 D
7=0 DND
P
(T, 0) > 7 Ex D)dy+ +
P
(T (x, 0) S x Ex. D)dy 1
7-00 DvD
yym D-D
(Sens(Ty, 2)d7 + Spec(T,,, x)d7 - 1)
7=0
Definition A.3. (Targeted Expansion) The targeted neiighbboroood expansion (around point x)
of a dataset D P(X') from a substituting at most b sequences (from D) that lie in S with points in S':
set S' the of all datasets that can be made by only
B,( = (D' C
(D], | ns C DOS, DOSc D'S.IDAAD 22b..
Lemma A.4. Upper bound on MI score under neighbor poisons.
(z)D')
min ((, G=L(D")
Tx(x,0) +
E Na (.)
(=C(D")
D D D'= m
13

---

Proof.
For r',..., 2b ~ (2), we know
[2, L(DU (2'...,)... - (x, o (2). = 8/c.0)
Now, extend this for dataset substitutions, for 21, Tb N, and D' D f (bbl. we have:
L(D'U(Z...... (.(oLL(DD'(PPossonaapp. (a. =(00) (2)
Taking expectation over points, we get:
(1, L(D' U (x) x(11)
(l V (a)
D'= D'=D022i...0
(z, L(D'U (PoisonMaplx.Drr by) 2()
D'
D'
Using Jensen's 's Inequality:
(a (1
of (x)x)) D'=D D'=DY ms
E
Ta( T(D' U (PoisonMap (a.Dr) ((),
2 21.. 2
5i Sj (x)
D) (11
g(x.D')
E
31 B Ra (2)......)))
-/....r D (2
Taking one side of the absolute value:
E Bl N Wh (1
(s, L(D' U (PoisonMap
E 3 Ri No 1..... D'= [b)
L(D' U (r1,,..,,+)]))
E
31 Fi B... 2/ (
D' ('
14

---

Then.
min E
[r L(D' U (
D' D'=D(
(a L(D' (f/1...,,111))
D D) (11mm))
(21....)))
Simplifying:
8la.D")
min (D..(. (=L(D")
(x(200)
E
(z,
E (a) 26....6-NN.6 Daa
D.......... (=C(D")
Lemma A.5. Lower bound MI score under non-neighbor poisons.
max (',.,(D.V.(())) (=L(D')
[r(r,0)
[z)
(2)
3 EA.. 26 D' D'=D D'=D((m1... (1)
O=L(D')
Proof. For ri,,.., 2/ ~ (x), we know,
[T, I(DU (x)....,, ,())) (z, I(D. (PoisonMap' ((),),,,,,,),NNN g(arDD
(z(.....()
Now, extend this for dataset substitutions, for I Tir.. 2b Nx) and D' D {o 1bb we have
[T -C
(CC_ (C,, - sim.0)
(
Taking expectation over points, we get
E V 2 2(.... (
[r L(D' U (ai
(2, L(D'U (PoisonMMp (2. (b),
E
21 NN af (.(()) (m=
Using Jensen's 's Incquality:
15

---

E Ma
T2 L(D' U (ai,....,$$]]
(())
E
Ty (2, L(D' U (PoisonMap( (2) (),,
D ('=D((o
D'
E
# Buunnyy (N D'mD) (mm
Taking one side of the absolute value:
E T (n) Fimmi (a( (emoess
(x, L(D'U U (PoisonMap
(m ((a),...,, V.),),
2
E
31 Ma 1 (x.
D (2)...
Dla,
E Ph 8 Fl (m
Then,
max
[x L(D'U U (x1,...,,))))
31 Ro N (2)ND
$1 (x)
2
T2 L(D' U (z (x(,)..,,((1)
3 Me
2/
D(x.D)
E Fla.., NN Sy oy Nx)( D) (1)...m))
(2).....1)))
Simplifying:
max (D., (=C(D")
(x,
(x,
D',
D' B. (DN.
(=C(D')
yo 26 (x) (2
16

---

Lemma A.6. Advantage of MI under poisoning.
(Sensitivity,, (I,, dr + Specificity (Ix d1)
E DwD D' Bb, (D.N. (=C(DD)
(I,( x En. D)
E DMD
(I, x EN. D)
BBg
(o),,()), 6=C(D')
g'z,D")
glr,D")
E
1 (z)
p 1r. N (a)
,,..,,h (.)
a1 1 (x)
n' 17 (1) M6y
Proof. Using Lemma A.I
get
E DwD
min D' (=L(D')
(2,
EN. D
(x)) (e))
PD-D
min
(x, y x Ex. D dr
0=
D' D',BB,,DDNN.0 (=C(D')
Sensh, (Ty, 2:) dr
Similarly,
E DwD
max
(y(2,0)
I EN, D
D' (D.N.,, (w)) (=L(D')
PDwD
max
(x, S
EN, D dr dy-1
y=
D' (D..,, (x)) (=((DD)
000
(1 - -Specs, (Ty, dr
x=0
Now, using Lemma A.4 and Lemma A.5
(Senso, (T,, x) + Spec,, x)) - 1) dr
E D-D
min D', (z)).(x)) (=C(D')
(x, 0)
EN, D
E D-D
max
(x,
x EN,
D' (D....
0= (=L(D")
<
E DWD
(I,(I,
x Ex. D)
(I,(I, 0)
x AN.. D)
DWD D' D'-
B8, (..).) (=C(D")
(o.
(=C(D')
(')
E
E t (w
(c)
(
z "y, ~N(x))
1 N.)
D) (.)
D'= Bay.
17

---

Theorem A.7. (Restatement of Theorem 5.1.. For point the advantages of a test In with and without poisoning are at odds with each other. as given by:
(Senso, (Iy, + Speco, (I,, - 1)dy + /
(Sens( xT) + Spec( (T,, -1 1) S 8",
y=0
==
g(c
Dla.D") (#1 (y
where, 8=
E
E Bi.. (x(x))
(
21 Mi (a)
2 by (x)(x)
"1 (.)(
D'=Di D'= [D)
Di=Di (bg) b== (wo))
Proof. For bj = (x) n DJ. we get,
(I, ENN. D)
E D-D
(T, 0) 1 4x. D)
D-D D' (.),
0=L(D)
(=2(D")
Similarly, for by N, D, we get,
E D D' B', (D). (=C(D")
(I,(2, EN. D) DND
E (T, Ex, D)
0-2(D)
Applying this to Lemma
A.6
we get the result.
B
Additional Results
Table 6: Natural and robust TPR @ 1% FPR of MI tests on the A14Privacy and AGNews canary datasets.
At4Privacy
AGNews
Nr
Tist
Natural
Token Dropouts
Casing Flips
Chunk
Poisenk
Natural
Token Dropouts
Casing Fligs
Chunk
Poteocl
LOSS
0.104
0.044
0.046
0.083
0,004
0,069
0.012
0,018
0.085
0.000
0.085
0.087
0.007
7-gram
dib
0.097
0.061
0.080
0.089
0.034
0.004
0.021
0,027
0.039
0.005
perturb
0.104
0.040
0.061
0.089
0.004
0.041
0.009
0.021
0.067
0.000
reference
0,108
0.046
0.051
0,004
0.089
0.021
0.087
0.000
LOSS
0.030
0,000
0.000
0.000
0.000
0.048
0..00
11,000
0.,00
0.000
Fxnet
dil
0.000
0.014
0.000
0.000
perturb
0.048
0.000
0.000
0.000
0.000
0.046
0.000
0.000
0.000
0.0N0
reference
0.062
0.000
0.000
0.000
0,000
0.066
0.000
0.000
0.000
0.000
LOSS 0.039
0.01
0.047
0.035
0.022
0.036
0.008
0.074
0.006
0.005
0.066
0.005
kitheediing
0.014
perturs
0.069
0.088
0.071
0.033
0.016
0.038
0.016
0.074
0.003
0.006
reference
0.080
0.025
0.097
0.042
0.817
0.027
0.008
0.024
0.003
0.005
LOSS 0.346
0.046
0.037
0.029
0.027
0.058
0.054
0,050
01002
0.006
0.050
0.039
0.029
0.044
0.031
0.006
0.008
0.006
0.004
Distance
pertubb
0.088
0.064
0.052
0.088
0.017
0.045
0,072
0,002
0.023
0.004
reference
0.087
0.054
0.046
0.021
0.000
0.068
0.030
0.054
0.019
0.000
Table Natural and robust membership inference test AUC scores using re-gram (k=7) neighborhood definition across different Pythia model sizes for 2.7B / 6.9B / 12B parameters A14Privacy.
Table 8: Natural and robust AUC scores for MI tests using the n-gram (k=7) neighborhood on A14Privacy with larger background dataset of IM WikiText points.
MI
Poiscuk
MI Test
Natural
Poisonk
LOSS
0.568
0.396
LOSS
0.568/ 0.588// (0.587
0.353/0.252, 002
kmin
0.560/ 0.55// /0.563
0.472/0.4485 0.433
ziib
0.552
0.467
10.56
perturb
0.572
0.401
perturb
0.583/
0/83//0600/0.00
0.380/
reference
0.232
reforence
0.624/ (0.647 (0.645
0.175 0.089 0.099
18

---

C Additional Details About PoisonM
Neighborhoods. We consider the following popular neighborhood definitions:
N N-gram(n-k) For given sequence of x .1, In let n-gram(z, k) fzz #stkli= denote the set of all k-grams of x. Then, the n-gram neighborhood yields the set of all sequences that share an k-gram with {a' X' n-gram(z', k) n n-gram(x;, k) # 0). Embedding (cosine, sim=c): : Let E X' + Rd denote embedding function that maps sequences to d-dimensional representations that capture their "semantics". Then, for a given sequence of z, the embedding similarity neighborhood yields the set of all sequences with embeddings of F(x')-F(G
cosine similarity at least e to x: (a' E EX [[(]] ||E'i|| 2 c).
ExactMatch: For a given sequence æ, the exact matching neighborhood yields the singleton comprising the sequence itself, i.e., (x}.
EditDistance (distance=1): the set of all sequences that are within a normalized Levenshtein distance / from xt {x' É X lev(m.ar) |a|+|r
For a given sequence I, the edit distance neighborhood yields
1].
Finding Poison Non-Neighbors. We sample an actual neighbor as 2, itself, and then:
N N-gram : Iteratively (a) select token uniformly random, (b) replace it with a token from the vocabulary such that last-layer activations of resulting sequence have maximum cosine similarity with activations of xg, where activations are computed using model that will be trained on the poison, and (c) repeat until n-gram overlap between the current poison and 2; is less than k. Here since replacing tokens automatically breaks up n- p-grams for free.
we can set
2. Embedding (cosine_sime sim=c) Iteratively (a) select a token uniformly at random, (b) pick the token that both maximizes activation cosine similarity and also minimizes (weighted by factor of A = 11.5) cosine similarity of the embedding (from E) of the resulting sequence with that of xy. Same procedure as that for embeddings, except we now maximize the edit distance (A = L.5) instead of minimizing embedding cosine similarities. ExactMatch: Same procedure as that for n-grams, but only a single iteration.
EditDistance (distance=1):
Finding Poison Neighbors, We sample an actual non-neighbor as a random text from any auxiliary dataset, and then:
1.
N-gram (n=k): Inject g-gram from x at random index (shortest k-gram in characters). Iteratively (a) select token uniformly at random, and (b) replace it with a token from vocabulary that maximizes cosine similarity of the embedding (under F) of the resulting sequence with x; 's embedding. (c) repeat until cosine similarity exceeds c.
Embedding lccosinessmmssse
3. EditDistance (distance=l): Iteratively (a) randomly insert, delete, or substitute charac-
ters (b) greedily keep the mutation only if decreases edit distance. (c) repeat until edit distance drops below I.
ExactMatch: Worst-case neighbors do not exist under exact matching, since the neighbor-
hood ball holds radius of O.
D Societal Impacts
This work presents a novel poisoning vulnerability that could be used by real-world adversary manipulate the outcome of high-stakes membership inference test. This could have legal and reputation-related implications. However, we believe it is important to release our findings so that (a) auditors and other entities that may wish use membership testing may be made aware its pitfalls, (b) the community can work towards better defining membership.
E Compute
We run all experiments on a machine with 4 NVIDIA H100 GPUs, 40 Intel(R) Xeon(R) Silver 44T0T CPUs, and 126GB of RAM. Finetuning models typically took 2 hours, and generating poisoned datasets took between a few minutes to at most hours, depending on choice of neighborhood.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction capture the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in Section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

We have listed the assumptions for all theoretical results, and included the proofs in the Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have discussed this in Section 6.1 of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer:
[Yes]
Justification:
Code in supplementary material and datasets are public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
coiioooooolcyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guiii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details provided in Section 6.1 of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to computational difficulties we cannot do several runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of compute presented in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: htt:::///euis.eeeeeee http:////b..tttttccccsseeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
We have read and conform to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed societal impacts in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not pose such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
24

---

Answer: [NA]
Justification:
This paper does involve releasing new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no crowdsourced experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
25

---

Justification:
Not applicable per the policy.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy http:p//eeeeeiiiooccco////eeeeee for what should or should not be described.
26


Reviews:
[{"review_id": "il8xViE0fV", "reviewer": "Reviewer_5Yuq", "summary": "The authors question the validity of membership inference evaluations for LMs demonstrating that even under the relaxed neighborhood-based membership evaluations, efficient poisoning can help completely flip predictions for members and non-members alike. The authors propose concrete instantiations for poisoning schemes targeted at various downstream MIAs and support their results with theoretical analysis of the poisoning and its relationship to attack success. The authors evaluate their poisoning attack on multiple models and downstream MIAs, and even extend their poisoning attack to dataset inference", "questions": "- Definitions in the paper talk about \"replacing\" actual neighbors with other neighbors (Definition 4,2. etc) but in a realistic data poisoning setting, the adversary can only add more data, not remove/replace any data. How do these two connect? Given the way neighborhood is defined, injecting neighbors, even if they are closer to the target record, does not make existing neighbors non-neighbors. Do the authors actually replace these neighbors in their experimental setup, or is it purely for easier theoretical analysis?\n\n- Lemma 3.5: How does this connect to/differ from the standard definition of the adversary's advantage in an cryptographic game?\n\n- Regarding the last step of the adversarial game (under threat model): because of the additional party, A's success is now also tied to C's success for the benign case. For instance, I could not poison either data and if C's test is not strong enough (which is the case for MIAs for LLMs), A wins by default in most cases. However, theoretical analyses further down in Theorem 5.1 suggest that this advantage would be high again if the attack is too powerful. Is there then a U-shaped curve for the adversary's advantage with respect to the downstream MIA's performance?\n\n- L230: Intuitively, this seems off. By construction, the poison data is a neighbor of X. If this poison is also indistinguishable from a clean non-member, then that clean non-member is also in the neighborhood of X, which makes it a member. \n\n- L253: I am a little confused about the generality claims. Unless I missed it, the way I understand each poisoning experiment corresponds to adding data upon anticipating some MIA that might be used in downstream inference. Is that the case, or do the authors use one poisoning strategy and then test other MIAs on it? Some clarification would be nice. Also, why not enforce this generalization by design? The loss used for token selection could be easily adapted to consider multiple MIAs simultaneously. \n\n- Could the authors share some of these generated poison samples to give a sense of what they look like? Given the use of token replacements I imagine them to look unnatural (high perplexity) and easily filterable with cheap alternatives, but that discussion can only continue after looking at the poison texts.\n\n- L300: What is the level of \"dataset\" here for dataset inference? Please provide additional details in the main body.", "limitations": "Yes, but I would encourage the authors to talk about it a bit more in the main body of the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n- I think the scenario considered here is very interesting and practical: not only can it be used by malicious data curators and model owners to bypass any membership/dataset inference tests, but also scrutinize companies and organizations by selectively poisoning Internet data with records that make models behave like they have strongly overfit on some target data.\n- The theoretical analyses are explained very clearly and paired with intuition, making them easy to follow. Overall, I found the paper very well written.\n- Experimental analyses are thorough, with evaluations across multiple model sizes, attacks, and even inference granularities (record and dataset level)\n- I really like Figure 1! I think it gets across the point beautifully\n\n# Weaknesses\n\nI don't see any critical weaknesses in the paper, but there are some things that could be improved or clarified:\n\n- The way I see it, flipping non-membership to membership is straightforward: just insert the target datapoint itself into model training data. Either the model trainer actually scans its training data to check for the record (which is unlikely but in case it does, it can share the computation as a proof of non-membership with any relevant arbiter), or it doesn't (in which case, there is no reason to be stealthy and not just insert the target record directly). It is hard to think of a practical scenario where a model trainer would want to prove that they did in fact train on some particular record in training. Making a member be classified as a non-member is particularly interesting, especially considering potentially **adversarial model trainers or data curators** that use some copyrighted data, but augment it with such poisoned data such that any downstream MI/DI evaluations make it look like such data was never used in training. \n\n- L260: While this is understandable, I think the authors can try and at least simulate what the effect of poisoning might be by a) using a lot more data, and/or b) partitioning the data such that all poison data is injected in the first X% of fine-tuning data. Given the reasonable compute requirements (as detailed in the Appendix), it would be nice to potentially consider another open-source model family (like OLMO/OLMO2 [1]).\n\n- The per-record effective DPR (data poisoning rate) is reasonably low: 10 poisons per record in the worst case for a 100K dataset size, i.e. 0.01% DPR. However, for a set of 500 members and non-members this number does scale up very quickly, reaching 5.5% DPR (for 500 members and 500 non-members) : quite high, especially for LLM training! This DPR would be particularly relevant for dataset inference, where the adversary would have to influence membership labels for multiple records to sway the final DI outcome. Can the authors comment on (and perhaps analyze) the tradeoff between the DPR here and final attack success, at least for the DI case (and potentially also for MI by controlling the number of poisons added for non-members)? \n\n## Minor Comments\n\n- L2: \"language model's training set\" - although this work focuses on text models, the MI definition is not specific to language models - please rephrase to clarify the distinction.\n- Might be worth looking into attacks like Min-K++ [2]\n- Figure 2: Please increase size and use log-log scale (focus on low-FPR region)\n- L319-321: The 2.7B model has had very \"unexplainable\" observations in several works (acknowledged by the Pythia authors too) so I would not read too much into results for this model- maybe test if for a smaller one from the same model family?\n\n### References\n- [1] OLMo, Team, et al. \"2 OLMo 2 Furious.\" arXiv preprint arXiv:2501.00656 (2024).\n- [2] Zhang, Jingyang, et al. \"Min-k%++: Improved baseline for detecting pre-training data from large language models.\" arXiv preprint arXiv:2404.02936 (2024).", "was_revised": true, "final_justification": "I think with the additional experiments and clarifications, the paper will be a great addition to the conference. My recommendation remains the same (i.e., accept).", "rebuttal": "We thank the reviewer for their comments/suggestions, and provide discussion below. We appreciate that the reviewer finds the scenario interesting and practical, and theoretical/experimental analyses thorough. \n\n>L260: While this is understandable, I think the authors can try and at least simulate what the effect of poisoning might be by a) using a lot more .... open-source model family (like OLMO/OLMO2 [1]).\n\nBelow, we run the recommended experiment of using more data with 1M points (ngram definition), and find that PoisonM continues to successfully reduce performance of the tests.\n\n|  | Natural | PoisonM |\n| --- | --- | --- |\n| LOSS | 0.568 | 0.396 |\n| KMin | 0.556 | 0.478 |\n| Zlib | 0.552 | 0.467 |\n| Perturb | 0.572 | 0.401 |\n| Ref | 0.604 | 0.232 |\n\nWe also conduct experiments below on OLMO2-7b model (ngram definition), and find that here too, PoisonM successfully reduces performance of membership tests:\n\n|  | Natural | PoisonM |\n| --- | --- | --- |\n| LOSS | 0.567 | 0.276 |\n| KMin | 0.542 | 0.412 |\n| Zlib | 0.554 | 0.386 |\n| Perturb | 0.587 | 0.303 |\n| Ref | 0.590 | 0.255 |\n\n>The per-record effective DPR (data poisoning rate) is reasonably low: 10 poisons per record in the worst case .... tradeoff between the DPR here and final attack success, at least for the DI case (and potentially also for MI by controlling the number of poisons added for non-members)?\n\nWe explicitly account for the tradeoff between data poisoning rate (DPR) and attack success in our formulation in Section 5—this is formally captured by **budget parameter** b, which controls number of poisons per target. Higher budget leads to lower mapping error or better attack success.\n\nAs requested, we conduct additional experiments in the DI setting, focusing on low-DPR regimes. Specifically, we consider 2 reduced-DPR configurations:\n\n- 0.55% DPR: Using the setting from the previous comment’s response, with total dataset size of 1M , we achieve a **DPR of just 0.55% (for 500 members and 500 non-members)** . Even at this lower rate, DI is successfully fooled: p-values are **3e-7** for non-members and **0.99** for members.\n- 0.3% DPR: Reducing budget further to 5 poisons per record (yielding a 0.3% DPR for 500 members and 500 non-members), the attack remains effective: p-values of 2e-2 for non-members and 0.99 for members.\n\nThese results highlight that even at substantially reduced poisoning rates, PoisonM can still mislead dataset inference attacks. We will add this analysis to the paper to make DPR-attack success tradeoff clearer.\n\n>Definitions in the paper talk about \"replacing\" actual neighbors with other neighbors (Definition 4,2. etc) but in a realistic data poisoning setting, the adversary can only add more data, not remove/replace any data. How do these two connect? Given the way neighborhood is defined, injecting neighbors, even if they are closer to the target record, does not make existing neighbors non-neighbors. Do the authors actually replace these neighbors in their experimental setup, or is it purely for easier theoretical analysis?\n\nWe replace neighbors in both our experimental setup and theoretical analysis. The replacement-based threat model is well-established in prior work, both in theory [1] and in practice [2]. Theoretically, Mahloujifar et al. [1] establish the existence of strong substitution-based poisoning attacks, showing that such attacks will succeed w/ sufficient budget. On the practical side, Carlini et al. [2] demonstrate the feasibility of substitution-based attacks at web scale by conducting real-world attacks. Their attack uses a frontrunning strategy, where the adversary anticipates which public data sources (e.g., Wikipedia backup dumps) will be scraped for training and replaces content in advance. This is a direct motivation for our attack – the adversary can follow this exact strategy (L170) to carry out PoisonM.\n\n[1] Mahloujifar, Saeed, et al. The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure. AAAI, 2019\n\n[2] Carlini, Nicholas, et al. Poisoning web-scale training datasets is practical. IEEE S&P, 2024\n\n>How does this connect to/differ from the standard definition of the adversary's advantage in an cryptographic game?\n\nOur security game is directly inspired by standard cryptographic security games and hence, their connection is natural. The adversary's advantage in a (cryptographic) distinguishing game captures their ability to differentiate between a cryptographic protocol’s output vs. random data (e.g., a good encryption scheme’s output should be indistinguishable from a random string without access to the secret key). Formally, this is quantified by benefit obtained by the adversary over simply random guessing.  We adopt the same notion of *advantage* in the context of membership inference (MI) (this was originally done by Yeom et al. [3] and many follow-up works thereafter [4,5]) – here, advantage captures tester’s gain over random guessing in determining whether a point was in the training set.\n\n[3] Yeom, Samuel, et al. \"Privacy risk in machine learning: Analyzing the connection to overfitting.\" IEEE CSF, 2018.\n\n[4] Salem, Ahmed, et al. SoK: Let the privacy games begin! A unified treatment of data inference privacy in machine learning. IEEE S&P, 2023\n\n[5] Li, Zhuohang, et al. Analyzing inference privacy risks through gradients in machine learning. ACM CCS, 2024\n\n>Regarding the last step of the adversarial game (under threat model): because of the additional party, A's success is now also tied to C's success for the benign case. For instance, I could ..... U-shaped curve for the adversary's advantage with respect to the downstream MIA's performance?\n\nIndeed, when the test behaves poorly by itself, an adversary need not take much action to fool the test, and when the test behaves well, adversary gains significant advantage, suggesting U-shaped curve with respect to MIA’s performance. We will add discussion of this to the paper.\n\n>Intuitively, this seems off. By construction, the poison data is a neighbor of X. If this poison is also indistinguishable from a clean non-member, then that clean non-member is also in the neighborhood of X, which makes it a member.\n\nThe key point is that *\"indistinguishability\"* here is defined **with respect to a specific MI test** T. That is, the poison is *T-equivalent* to a clean non-member, meaning that if either the poison or a clean non-member were included in the training set, the MI test T would assign (approximately) **the same score** to the target point x.\n\nIn other words, while the poison is a neighbor of x in the input space as measured by the distance metric (by design), it is crafted in a way such that, from the perspective of test T, its training-time influence *mimics* that of a clean non-member. We will revise the text to clarify this.\n\n>I am a little confused about the generality claims. Unless I missed it, the way I understand each poisoning experiment corresponds to adding data upon anticipating some MIA that might be used in downstream inference. Is that the case, or do the authors use one poisoning strategy and then test other MIAs on it? Some clarification would be nice. Also, why not enforce this generalization by design? The loss used for token selection could be easily adapted to consider multiple MIAs simultaneously.\n\nFor a given neighborhood definition (e.g., ngram), we use one poisoning strategy and test all MIAs on this single strategy, i.e., we do not incorporate test-specific changes to the strategy. In other words, PoisonM’s implementation is MI test agnostic. Our experimental results demonstrate that the *same* poison works against *all* 5 state-of-the-art MI tests.\n\n>Could the authors share some of these generated poison samples to give .... perplexity) and easily filterable with cheap alternatives, but that discussion can only continue after looking at the poison texts.\n\nHere are some example poisons for the AI4Privacy dataset:\n\n”Mr. Aberspathy, I write, expressing My Interest in your forensic psychologists Research.Could you Please sharing your latest publications'? I am especially interested of Your studyabout IPV4 user 122.171,108.126.”\n\n“Hi, i am Unable to proceeding of education course payments. My MasterCard Cardnumber 0789777405804829 kept getting declined.”\n\n“Dear Eola,I've looks at your files Under Auto Loan Account and noticing some inconsistencies, Your DOB is still listing as 11/12/1977, is to correct'? Also, can You confirmed your secondary addressing atApt. 488, Yorkshire and. Humber, 81414.5215?”\n\n“Dear Ebba, we appreciate your interest in our Distance learning program. A link to, trial Course will be sent to Kristina36@ahoo.Com.”\n\nWe observe that generated poison texts maintain a reasonable level of readability. Moreover, we note that PoisonM is a strong, general-purpose attack and can adapt to defenses by incorporating filtering criteria—such as perplexity filter—directly into its loss function. We demonstrate this with an additional experiment where we consider a setting in which perplexity filter is employed (threshold selected to ensure low FPR of 1%), and show PoisonM can adapt to this setting and remain effective against all considered MI tests.\n\n|  | Natural | PoisonM |\n| --- | --- | --- |\n| LOSS | 0.587 | 0.322 |\n| KMin | 0.561  | 0.444 |\n| Zlib | 0.564  | 0.420 |\n| Perturb | 0.600 | 0.340 |\n| Ref | 0.647  | 0.143 |\n\n>What is the level of \"dataset\" here for dataset inference? \n\nHere, we consider the setting proposed by Maini et al in the original paper [8] and use the DI procedure of aggregating MI scores from multiple points (we use 500 here) to produce a single p-value, which is then used to assign a label of {0,1} to the whole dataset (AI4Privacy or AGNews) as to whether it is in the training set or not.\n\n[6] Maini, Pratyush, et al. LLM Dataset Inference: Did you train on my dataset? NeurIPS, 2024"}, {"review_id": "4h2rXkuGqS", "reviewer": "Reviewer_EmXg", "summary": "This paper investigates the problem of membership inference tests under the neighborhood-based definition where all semantic neighbors of a training point are also treated as members. Authors proposed to study the MI test from a robust perspective through substituting neighbors or non-neighbors of target training point. And authors theoretically show that there is a trade-off between a test’s accuracy and its robustness to poisoning. Experimental results show the proposed poisoning attack can make existing MI test less effective.", "questions": "1. This is a metric-based membership inference, how to determine the threshold value?\n2. Can authors present more clearly about worst-case datasets? And give a clear explanation about why membership inference test can make wrong predictions when the perturbed dataset is the worst-case dataset? \n3. In Figure 1, what does the orange area/line mean? Is the target point a member? \n4. Please give more detailed explanation about Figure 1.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths in this paper are summarised as follows: \n1. This paper studies the problem of membership inference in a relaxed neighborhood-based assumption, i.e., all semantic neighbors of a training point can be considered as members. I believe this research perspective is more practical, especially in the ear of LLM. So, the research motivation and significance are strong.\n2. Authors give thorough definitions and theoretical analysis about the problem of membership inference tests. \n\nHowever, there are some weaknesses in this paper shown as follows: \n1. A clear description about attacker’s background knowledge is expected. For example, does an attacker have to know the data distribution of target dataset in the scenario this paper studied? In addition, what kind of information the attacker can obtain from the target model, such that the attacker can exploit the information to implement the follow-up membership inference test? \n2. The definition of 4.2 is not clear, especially the given equation of expansions of dataset. \n3. A clearer description about dataset perturbation, worst-case dataset, and the reliability of membership inference test should be very important and expected. Even though authors gave clear descriptions on other definitions, explanation and justification about this part are vague. \n4. The poisoning attack might only be applicable to the specific form of MI concerned in this work. Once an attacker knows how MI works (i.e., how to construct the features for member and non-member), she could alway find a way to fool MI more or less if she could manipulate/poisoning the training/tuning data.", "was_revised": true, "final_justification": "Many thanks for the response to the concerns, which help improve the understanding. But \"the poison is crafted such that any downstream MI test on the target point is fooled, regardless of its internal mechanics\" still seems too strong to me, without a clear intuition. Not sure if the concept of worst-case dataset could be more formal.", "rebuttal": "We thank the reviewer for their comments and suggestions for the paper, and provide a detailed discussion below. We appreciate that the reviewer finds the research motivation strong, and the theoretical analysis thorough.\n\n>A clear description about attacker’s background knowledge is expected. For example, does an attacker have to know the data distribution of target dataset in the scenario this paper studied? In addition, what kind of information the attacker can obtain from the target model, such that the attacker can exploit the information to implement the follow-up membership inference test?\n\nWe emphasize that our attacker is solely responsible for poisoning the training dataset—they do not participate in or implement the membership inference (MI) test itself. The key idea is that the poison is crafted such that *any* downstream MI test on the target point is fooled, regardless of its internal mechanics. In fact, a very attractive feature of our attack is that it is MI test agnostic – i.e., the same poison works for all MI tests for the target point(s). This is validated by our experiments as well - the same poison is effective in fooling all 5 SoTA MI attacks.\n\nTo construct the poison itself,  we consider a highly practical attacker that does *not* require knowledge of the underlying data distribution of the training set. It only needs to know the exact target point(s) for which they wish to flip the membership test results which is inevitable for a targeted attack. Additionally, In our experiments, since the PoisonM loss requires only the model’s last layer activations, the attacker only needs query access to the model’s logprobs which is a standard assumption in the poisoning and adversarial example literature [1]. However, even under an even more restrictive setting where query access to the target model is prohibited, an attacker can optimize against a surrogate model, with the expectation that the poison will transfer to the target model. Below, we experiment with an OLMO2-7b model trained on poisons computed against a surrogate Pythia-6.9b model.  We find that PoisonM is still effective against MI tests even in this completely blind setting, without query access to the target model.\n\n|  | Natural | PoisonM |\n| --- | --- | --- |\n| LOSS | 0.567 | 0.313 |\n| KMin | 0.542 | 0.433 |\n| Zlib | 0.554 | 0.410 |\n| Perturb | 0.587 | 0.334 |\n| Ref | 0.590 | 0.298 |\n\n[1] Wan, Alexander, et al. \"Poisoning language models during instruction tuning.\" International Conference on Machine Learning. PMLR, 2023.\n\n>The definition of 4.2 is not clear, especially the given equation of expansions of dataset. A clearer description about dataset perturbation, worst-case dataset, and the reliability of membership inference test should be very important and expected. Even though authors gave clear descriptions on other definitions, explanation and justification about this part are vague. Can authors present more clearly about worst-case datasets? And give a clear explanation about why membership inference test can make wrong predictions when the perturbed dataset is the worst-case dataset?\n\nWe will certainly add more clarifying text in the paper on these points.\n\nIn Equation 4.2, given an original dataset, we define the space of (*membership-invariant) perturbations*—that is, all datasets obtained by replacing a point in the dataset with another point from the **same neighborhood class** (i.e., neighbors are replaced with neighbors, non-neighbors with non-neighbors) with respect to the target point x. These perturbations preserve the **membership label** of a target point under the chosen neighborhood definition. A simple example for the ease of illustration is as follows.  If a target sentence “John eats cake” is a member, and we use edit-distance with threshold \\ell = 1, then substituting a point in the dataset with “John eatz cake” (which remains within the edit-distance based neighborhood) results in a *membership-invariant* perturbation. The target’s membership status is preserved.\n\nThe **worst-case dataset** is then defined as a dataset drawn from this perturbation set but **adversarially chosen** to maximize the error of the membership inference test. In other words, while the true membership status (as defined by the neighborhood) stays the same, the attacker crafts such perturbations to **fool the MI test**—causing it to incorrectly predict the membership status for the target point. PoisonM achieves this through loss functions that optimize the representation space (e.g., embeddings) to mislead the test, even while staying within the bounds of valid dataset perturbations in the input space.\n\n>In Figure 1, what does the orange area/line mean? Is the target point a member? Please give more detailed explanation about Figure 1.\n\nIn Figure 1(a), the orange area represents all points which, if included in the training set and trained upon, will cause the MI test T to predict “member” for the target point. In other words, these are the points that influence the membership prediction for the target point x under the test T. The green point represents a clean training data point which is substituted by the purple point (poisoned data point) under our attack. In Figure 1(b), the target point is not a member because the green point (and purple point) is outside the neighborhood radius. In Figure 1(c), the target point is a member, since the green point (and purple point) are inside the neighborhood radius. Overall, Figure 1 illustrates the **misalignment** between the neighborhood-based definition of membership (using any distance metric) and the **actual influence region** that determines the behavior of a membership inference test on the target point x.\n\n>This is a metric-based membership inference, how to determine the threshold value?\n\nThere are two distinct thresholds relevant to our setting:\n\n- Neighborhood radius used to determine *true membership* under a given neighborhood definition. We adopt radius values consistent with those studied in prior work [2]. Specifically, we select them by calibrating on points from the canary dataset to ensure that the test performs well under natural conditions.\n- Threshold for the MI test score, which determines the test's decision boundary. In Table 2 and Figure 2, we report AUC scores to capture the MI test’s performance *across all thresholds*. Additionally, in Table 5 (Appendix B), we present results using a concrete threshold selected via the standard procedure of targeting a 1% false positive rate (FPR).\n\n[2] Duan, Michael, et al. \"Do Membership Inference Attacks Work on Large Language Models?.\" First Conference on Language Modeling.\n\n>The poisoning attack might only be applicable to the specific form of MI concerned in this work. Once an attacker knows how MI works (i.e., how to construct the features for member and non-member), she could alway find a way to fool MI more or less if she could manipulate/poisoning the training/tuning data.\n\nWe stress that an attractive feature of our attack is that it is completely MI test agnostic – i.e.,  for a given neighborhood definition (e.g., ngram), the same poison works for all MI tests for a target point(s). This is also validated by our experimental results - the same poison works against all 5 state-of-the-art MI tests."}, {"review_id": "SgjOKg1Z66", "reviewer": "Reviewer_YT5m", "summary": "The paper aims to demonstrate the unreliability of membership inference tests. A membership inference test takes a trained model and an item of data and returns a score which indicates whether the item was used in the training process of the model. The approach taken in the paper is to demonstrate that it is possible to poison the training dataset such that an MI test gives incorrect predictions, e.g. an adversary may place text on their personal website which, when scraped, results in a model which is trained on that website returning a positive result for some training data which is not used in the training process.\n\nThe authors give a concrete example of an attack which can induce an incorrect prediction from an MI test and show that it works with leading MI tests and models.", "questions": "1. I am confused as the real-world applicability of the threat presented in the paper. In the example given, an adversary plants a poison such that the MI test predicts that there were Larry Lobster novels in the training corpus, triggering a lawsuit. However, couldn't the developer offer access to the dataset to a trusted third party, who could verify that the Larry Lobster novels are not in the dataset?\n2. In a data poisoning setup, a standard threat model is to assume that an adversary can add data to a dataset but not change it. For example, a user could add fake news articles to their own website but could not edit news articles scraped from CNN.com. Does this present any challenges to your approach, which relies on substituting data in a dataset?\n3. Why do the membership tests in figure 2 have such low AUC in general? They appear to be quite bad tests.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n+ The paper studies a novel (to my knowledge) setting which could have much real-world significance.\n+ The paper is generally clear and well-structured. The threat model is clearly laid out, even if I am not sure of the real-world significance.\n+ The paper provides code which is good for reproducibility\n+ The approach, PoisonM, is straightforward but clearly effective.\n\n### Weaknesses\n+ I am unsure of the real-world applicability of these neighborhood-based MI tests. I understand that there is significant previous work on the subject, but it does not seem really applicable for e.g. copyright if the training dataset contains an entry with a 7-gram overlap from a copyrighted source: the dataset could include a quote from the copyrighted source or indeed an independently written passage which shares the 7-gram overlap.\n+ The paper seem a bit over-written, e.g. I'm not sure that the threat model / security game section adds very much to the paper and could be replaced with some more experimental results.", "was_revised": true, "final_justification": "This is an interesting treatment of the problem of fooling membership inference tests. While the setting is slightly niche, the results, which are both theoretical and practical, are solid and a good contribution. I had some questions about the setting, mostly due to my own limited knowledge about the area. I'm happy with the authors' answers to the questions I had.", "rebuttal": "We thank the reviewer for their comments and suggestions for the paper, and provide a detailed discussion below. We appreciate that the reviewer finds the setting to be novel, and the paper to be clear and well-structured.\n\n>I am confused as the real-world applicability of the threat presented in the paper. In the example given, an adversary plants a poison such that the MI test predicts that there were Larry Lobster novels in the training corpus, triggering a lawsuit. However, couldn't the developer offer access to the dataset to a trusted third party, who could verify that the Larry Lobster novels are not in the dataset?\n\nUnfortunately, sharing the training dataset with a third party does not resolve the threat—such a third party trusted by both the parties may not always exist, and the model owner could always lie about what was actually used. Even providing a training trajectory that ends in the published model is insufficient –  Roy Chowdhury et al. [1], demonstrated that it is possible to forge training trajectories and fool verifiers into believing a false dataset was used.\n\nMoreover, sharing the dataset itself often has significant barriers, including being unacceptable, due to confidentiality, privacy, and IP concerns. Many organizations (e.g., OpenAI) treat their training data as proprietary, and are unwilling to disclose it.\n\nAs such, MI tests are the de facto standard for establishing whether a given data point was used in training. These tests are already widely accepted by both regulatory bodies, such as   NIST (the U.S. standards agency) [2], UK’s ICO [3], and commercial organizations, such as Google's adoption in its tensorFlow library.\n\n[1] Amrita Roy Chowdhury, Zhifeng Kong, Kamalika Chaudhuri, On the Reliability of Membership Inference Attacks\n\n[2] Elham Tabassi, Kevin Burns, Michael Hadjimichael, Andres Molina-Markham, and Julian Sexton. 2019. A taxonomy and terminology of adversarial machine learning. National Institute of Standards and Technology (2019).\n\n[3] Sasi Kumar Murakonda and Reza Shokri. ML privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. CoRR, abs/2007.09339, 2020\n\n>In a data poisoning setup, a standard threat model is to assume that an adversary can add data to a dataset but not change it. For example, a user could add fake news articles to their own website but could not edit news articles scraped from CNN.com. Does this present any challenges to your approach, which relies on substituting data in a dataset?\n\nThe replacement-based threat model is well-established in prior work, both in theory [4] and in practice [5]. Theoretically, Mahloujifar et al. [4] establish the existence of strong substitution-based poisoning attacks, showing that such attacks will succeed with sufficient budget. On the practical side, Carlini et al. [5] demonstrate the feasibility of substitution-based attacks at web scale by conducting real-world attacks. Their attack uses a frontrunning strategy, where the adversary anticipates public data sources (e.g., Wikipedia backup dumps) that will be scraped for training and replaces content in advance. This is a direct motivation for our attack – our adversary can follow this exact strategy (L170) to carry out PoisonM.\n\n[4] Mahloujifar, Saeed, Dimitrios I. Diochnos, and Mohammad Mahmoody. \"The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 33. No. 01. 2019.\n\n[5] Carlini, Nicholas, et al. \"Poisoning web-scale training datasets is practical.\" *2024 IEEE Symposium on Security and Privacy (SP)*. IEEE, 2024.\n\n>Why do the membership tests in figure 2 have such low AUC in general? They appear to be quite bad tests.\n\nThis observation aligns with a growing body of work showing that MI tests are generally weaker for LLMs [6, 7]. One key reason, as hypothesized in prior work, is the nature of LLM training—few optimization steps over massive datasets—which makes individual memorization harder to detect. To address this, Maini et al. [7] propose *dataset inference* as a stronger alternative, which aggregates MI signals across multiple points to make accurate membership predictions at the dataset-level. Notably, our attack is powerful enough to *flip even these aggregated predictions*, demonstrating its effectiveness against both pointwise and dataset-level MI tests.\n\n[6] Duan, Michael, et al. \"Do Membership Inference Attacks Work on Large Language Models?.\" First Conference on Language Modeling.\n\n[7] Maini, Pratyush, et al. \"LLM Dataset Inference: Did you train on my dataset?.\" *Advances in Neural Information Processing Systems* 37 (2024): 124069-124092."}, {"review_id": "AnBlJ2qV3x", "reviewer": "Reviewer_Zgmt", "summary": "This paper investigates the vulnerability of membership inference (MI) tests used to detect whether data points were part of a machine learning model's training set. The study demonstrates that MI tests remain susceptible to manipulation even when enhanced with relaxed membership definitions that consider approximate matches rather than exact data point correspondences.\nThe work presents PoisonM, a data poisoning methodology that strategically modifies training datasets to induce misclassification by MI tests, while preserving the ground-truth membership status of target data points.\nThe research contributions include:\n– Establishing formal robustness definitions for MI tests against adversarial training set perturbations.\n– Developing practical algorithms for constructing poisoned training datasets that compromise MI test effectiveness.\n– Demonstrating a fundamental accuracy-robustness trade-off, where higher MI test performance on clean data correlates with increased vulnerability to adversarial manipulation.\n– Providing comprehensive experimental validation showing MI test degradation under PoisonM across diverse model architectures and evaluation protocols.", "questions": "1) How the choice of neighborhood definition (n-gram overlap vs. embedding similarity ) influence the performance of PoisonM?\n\n2) How feasible is it for an adversary to insert poisoned examples into public data sources in a way that ensures they are included during large-scale model training?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\nQuality\nTheoretical Rigor: The paper clearly defines core concepts like neighborhood-based membership, robustness to changes in the training data, and mapping error. It builds a solid mathematical foundation for the attack and introduces pointwise robustness and Theorem 5.1 to support its claims. Also I have to add that most of the math in this paper was beyond me and could not follow it closely.\n\nComprehensive Experimental Analysis: The theoretical contributions are substantiated through extensive empirical evaluation, examining PoisonM effectiveness across diverse MI attack variants, benchmark datasets, model architectures, and similarity metric configurations. The experimental findings demonstrate persistent vulnerability in MI attacks despite relaxed membership criterion implementations.\n\nGeneral-Purpose Attack: PoisonM is flexible and works across various MI tests and neighborhood definitions, making it a broadly applicable and reproducible attack strategy.\n\nClarity\nClear Motivation: The connection to real-world applications (e.g., copyright enforcement, regulatory compliance) makes the work relevant and compelling. The threat model involving an adversary, challenger, and arbiter maps well to practical use cases. Structured Presentation: The paper is well-organized with clearly stated motivation, formal definitions, theoretical insights, algorithmic implementation, and experimental results.\n\n\nSignificance\n\nCurrently very relevant: Figuring out whether a piece of data was used to train an AI model is an important issue in current discussions about privacy and how transparent large language models (LLMs) should be. This paper shows that the tools used to answer that question can be fooled.\n\nPolicy and Legal Impact: The results raise concerns about using MI tests in real-world settings, like court cases or government audits, where accuracy really matters.\n\nBroad Applicability: PoisonM applies to both exact and semantic definitions of membership and can be extended to other forms of training-data attribution, making it relevant to future privacy research.\n\n\nWeakness: \nClarity\nDense Notation in Theoretical Sections: Definitions and theorems (particularly Theorem 5.1) use heavy notation with nested expectations and subscripts, which may be inaccessible to a broad audience. \n\nSignificance\n\nLimited Comparisons: The experiments are strong, but the paper would be more convincing if it compared PoisonM to existing defenses or other robust MI test methods to show how well it performs in comparison.\n\nLack of Real-World Impact Analysis: Although the paper has clear practical relevance (like in legal or audit situations), it doesn’t show how badly MI tests might fail in real life—for example, how often they might give wrong answers or how much trust in them could be affected.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions for the paper, and provide a detailed discussion below. We appreciate that the reviewer finds the theoretical and experimental analysis to be thorough, and the motivation and significance to be compelling.\n\n>Dense Notation in Theoretical Sections: Definitions and theorems (particularly Theorem 5.1) use heavy notation with nested expectations and subscripts, which may be inaccessible to a broad audience.\n\nWe will work on improving the style and clarity of the theorem to help with readability.\n\n>The experiments are strong, but the paper would be more convincing if it compared PoisonM to existing defenses or other robust MI test methods to show how well it performs in comparison.\n\nTo the best of our knowledge, we are the first to introduce a new threat model for undermining MI tests—*via data poisoning*. This is a previously unexplored attack vector, and per our knowledge there is no defense yet. Furthermore, our attack is MI-test agnostic - i.e., for a given definition of neighborhood, the same poison works for *all* MI tests for a target point(s). This is validated by our experiments as well - the same poison is effective in fooling all 5 SoTA MI attacks.\n\nNevertheless, we explore a potential defense strategy and include additional experimental results. The intuition is that the poisoned data points might appear unnatural compared to real data. One natural strategy is to filter out suspicious points using a threshold based on the perplexity score. However, PoisonM can incorporate such defenses directly into its loss function, making it a strong,  general-purpose attack and hard to filter out. To demonstrate this,  we simulate a defense that filters out points based on their perplexity score (threshold selected to avoid false positives; 99th percentile), and show that PoisonM still successfully fools MI tests under this constraint.\n\n|  | Natural | PoisonM |\n| --- | --- | --- |\n| LOSS | 0.587 | 0.322 |\n| KMin | 0.561  | 0.444 |\n| Zlib | 0.564  | 0.420 |\n| Perturb | 0.600 | 0.340 |\n| Ref | 0.647  | 0.143 |\n\n>How the choice of neighborhood definition (n-gram overlap vs. embedding similarity ) influence the performance of PoisonM?]\n\nThe choice of neighborhood function directly impacts the misalignment between the neighborhood-based definition of *true* membership and the actual influence region that determines the MI test’s behavior. Since the score of MI tests are typically computed using the model’s internal representation of a point, neighborhoods based on embedding similarity (which is a model’s internal representation of data), tend to outperform simpler, model-agnostic  definitions like n-gram overlap.\n\n>How feasible is it for an adversary to insert poisoned examples into public data sources in a way that ensures they are included during large-scale model training?\n\nPoisoning public data sources is not just a theoretical concern—it has already been demonstrated in practice. Carlini et al. [1] show that poisoning web-scale datasets is feasible and effective through real-world attacks. Our work assumes a similarly capable adversary who can manipulate public data sources to insert poisoned examples, making the threat both realistic and actionable. In particular, the adversary can adopt the same *frontrunning* strategy used by Carlini et al. to inject poisoned content into publicly scraped sources like Wikipedia backups.\n\n[1] Carlini, Nicholas, et al. \"Poisoning web-scale training datasets is practical.\" *2024 IEEE Symposium on Security and Privacy (SP)*. IEEE, 2024."}]

Author rebuttal:

Meta-review: The authors propose a method called PoisonM to poison a language model's training set in a way that invalidates membership inference. Given any distance metric used to define neighborhood in the LM's training data, PoisonM crafts poisoned data that, once trained on, degrades the performance of existing membership tests to below random chance.

Reviewers generally appreciated the paper's strong empirical result supported by intuitive theoretical analysis, with many practical implications for the interpretation of MIA. However, Reviewers 5Yuq and EmXg had some confusion about whether PoisonM is agnostic to the MI test or not, since an attack that requires knowledge of the MI test would significantly reduce its practicality.

During the rebuttal, the authors clarified that PoisonM only requires the attacker to choose the neighbor definition, i.e. distance metric, and crafts poisoned data based on this metric. The resulting poison is general to different MI tests, so the attacker is agnostic to the MI test itself. Given that the rebuttal sufficiently addressed this concern, AC believes the paper is ready for publication at NeurIPS and recommends acceptance.
Decision: Accept (poster)