Title: What Really is a Member? Discrediting Membership Inference via Poisoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Membership Inference

Poisoning Attacks

Robustness
Abstract: Membership inference tests aim to determine whether a particular data point was included in a language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition to include semantic neighbors as members as well. In this work, we show that membership inference tests are still *unreliable* under this relaxation - it is possible to poison the training dataset in a way that causes the test to produce incorrect predictions for a target point. We theoretically reveal a trade-off between a test’s accuracy and its robustness to poisoning.  We also present a concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests to well below random.

What Really is a Member? Discrediting Membership Inference via Poisoning

Abstract

Membership inference tests aim to determine whether a particular data point was included in a language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition to include semantic neighbors as members as well. In this work, we show that membership inference tests are still unreliable under this relaxation - it is possible to poison the training dataset in a way that causes the test to produce incorrect predictions for a target point. We theoretically reveal a trade-off between a test's accuracy and its robustness to poisoning. We also present a concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests to well below random.

1 Introduction

A central question in the machine learning (ML) community is whether a model was trained on a particular data point [1]. While this question has long been of academic interest, the recent surge in large language models (LLMs) has made it more relevant across new practical contexts. For instance, these models are often trained on massive web-scraped datasets [2], which may include copyrighted content. This has sparked high-profile legal disputes between model providers and creative professionals (e.g., authors) [3, 4, 5], centered on whether the disputed content was part of the training data. In another example, recent legal regulations worldwide have mandated auditing of ML models [6, 7]. In such cases, model owners may need to demonstrate that specific data points-such as those from the minority class-were indeed used during training, in order to support claims of fairness or regulatory compliance [8].

Currently, membership inference (MI) testing is the de facto approach for answering this question. Existing tests employ a variety of heuristics to analyze the loss landscape of the model, and output a 'membership score' - a high score typically indicates membership. However, a growing body of research has questioned the reliability of these tests [9, 10, 11]. A key concern is the ambiguity surrounding the definition of what it means for a data point to be a 'member.' For instance, if 'Harry Potter drew his wand' appears in the training data, should its paraphrase 'The wand was drawn by Harry Potter' also be considered a member? To address this, recent works have suggested relaxing the definition of membership to a neighborhood-based one-where all semantic neighbors of a training point are also treated as members [9, 12, 13].

In this work, we show that even under the relaxed, neighborhood-based definition, membership inference remains unreliable. We demonstrate this through a new lens - a dataset poisoning attack . Specifically, we consider a realistic threat model in which an honest model owner trains an LLM using data scraped from the internet. Despite the owner's honest intentions, the internet remains a

fundamentally untrustworthy environment, where anybody can introduce poisoned data into public sources (for instance, by editing Wikipedia articles or posting on Reddit). Indeed, recent work has shown that such poisoning attacks are not merely hypothetical, but are feasible in practice [14]. Building on this threat model, we consider an adversary who poisons the training 2 dataset of the model with the goal of causing an MI test to produce incorrect predictions. Note that this is distinct from traditional poisoning attacks, which typically aim to trigger undesirable behavior in the model itself during downstream use (e.g., denial-of-service or jailbreaking [15]). In contrast, our attack targets the MI test which is a separate classifier that operates on the model outputs/loss values.

In a nutshell, we establish that currently MI tests are not robust to dataset poisoning attacks. To this end, our contributions are two-fold.

First, we theoretically demonstrate the inherent difficulty of designing a robust MI test by identifying a fundamental trade-off between the test's accuracy on clean data and its robustness to poisoning.

Second, we provide a concrete instantiation of a novel poisoning attack, PoisonM , that effectively exploits this trade-off in practice.

Our attack works as follows: for a target point x t with ground truth membership label c , the adversary substitutes some points in the training dataset with carefully crafted poisoned ones that (1) preserve the true membership label c under the definition of neighborhood-based membership, but (2) cause the MI test to flip its prediction to 1 -c . Consequently, the attack discredits the test's predictions.

Revisiting our earlier usecases of MI tests, we highlight real-world motivations of such attacks. Consider the case of copyright enforcement. An honest model owner may ensure that their training dataset, under a mutually agreed-upon neighborhood definition, contains no points related to the new Larry Lobster novels. However, a disgruntled author could plant a poison outside this neighborhood that still triggers a (false) positive prediction, potentially enabling a baseless copyright lawsuit. Similarly, in the example of a fairness audit, an adversary could plant poisons within the neighborhood of minority class points, causing the MI test to falsely predict non-membership (false negative), thereby undermining the model owner's credibility.

Intuitively, the attack is possible due to the misalignment between superlevel sets of the MI test (points that when trained upon elicit high test scores, i.e., indication of membership) and the existing notions of neighborhood (see Figure 1). We provide a concrete implementation of the poisoning attack, PoisonM , for four popular notions of neighborhood: n -gram overlap, embedding similarity, edit distance, and exact matching (i.e. the traditional notion of membership). PoisonM is MI test agnostic, can target multiple points simultaneously, and highly efficient (only substituting a single clean point with a poison is sufficient to induce false negatives). We evaluate PoisonM against several MI tests across different datasets and model sizes, and find that it consistently flips test predictions and degrades performance well below random. Thus, our results highlight a disconnect between how MI tests operate and how their outputs are interpreted to determine membership in practice, calling for a re-evaluation of what it truly means for a point to be a member.

2 Background

MI tests typically rely on thresholding model loss or its variants, such as LOSS [16], Min-K%[17] (loss on least likely tokens), zlib[18] (loss-to-entropy ratio), perturbation-based tests (loss differences with perturbed inputs [19]), and reference-based tests (loss ratio to another model [18]).

Unreliability of Membership Inference. Recent work has already begun to highlight concerns regarding MI testing. For example, many works evaluate performance of tests on datasets that exhibit distribution shifts, which is flawed because members can be separated from non-members without even using the model [9, 11, 20]. Other work discusses how a dishonest model owner can refute the predictions of an MI test by providing a certificate that a model could be obtained without training on a specific point [21]. More recent work has also touched on how poisoned models can surprisingly amplify MI test results for finetuning data [22]. We differ from these works in that we show how MI tests can be manipulated to provide entirely wrong predictions.

Neighborhood-Based Membership Inference. The premise of an MI test relies upon the definition of membership, which traditionally labels a text sequence as a member if it is exactly matches a

SpaceofAllPossibleSequencesX

SpaceofAllPossibleSequencesX

SpaceofAllPossibleSequencesX

{x∈x|T(x,L(DU{x})）≥}

Worst-case Non-neighbor

Worst-case Neighbor

Target Pointxt

Target Point xt

TargetPointxt

Other

Other Neighbor

Neighborhood N

Non-neighbor

(a)Misalignmentbetweenneighborhood- basedmembershipdefinitionandthetest

(b)Points in[ ]representworst-case non-neighborforxt，i.e.,replacing any trainingpointsin withpointsin wouldcausemembershiptestscore tounexpectedlyincrease

(c)Pointsin[ representworst-case neighborforxt，i.e.,replacing any trainingpointsin withpointsin wouldcausemembershiptestscore tounexpectedlydecrease

sequence in the training set. However, this definition poses two key problems. First, training and nontraining texts often overlap substantially, making exact-match definitions of membership ambiguous and leading to unreliable performance for MI tests [9, 10]. For example, suppose copyrighted text is removed from the training set. These tests may still flag them as members simply because the training set includes related content (e.g., from online discussions) [10]. Second, the standard definition of membership, as exact presence in the dataset, is too narrow if it is used to measure whether a model is leaking training data. For example, a privacy leak can happen even if the model outputs a rephrased version of a sensitive medical record. Indeed, recent work has shown that language models can often complete sequences that they were not explicitly trained upon or even share n-grams with [13]. To address these concerns, recent works have advocated for a shift towards more flexible, neighborhood-based definitions for LLMs where membership is determined by semantic or lexical similarity [9, 12]. The key contribution of this work is to show that membership inference is still unreliable under these relaxed definitions.

3 Neighborhood Membership Inference: A General Framework

Notation. Let X be the space of token sequences over vocabulary V . Model parameters θ of an LLM are learned via algorithm L on dataset D ∈ P ( X ) , i.e., θ = L ( D ) , where P denotes the power set. Let D denote the distribution over the datasets. We define the symmetric difference between datasets D 1 and D 2 as D 1 ∆ D 2 .

For a given text sequence x ∈ X , its neighborhood is formally defined as follows:

Definition 3.1 (Neighborhood) . Let d : X × X → R ≥ 0 be a distance metric on the space of input sequences X . For a given x ∈ X and radius r ≥ 0 , the neighborhood N r : X → P ( X ) defines a ball of radius r centered at x and is given by: N r ( x ) = { x ′ ∈ X | d ( x, x ′ ) ≤ r } .

We will refer to points in N r ( x ) as 'neighbors' of x , and the complement N r ( x ) of the neighborhood is then simply all points that are 'non-neighbors' of x , i.e., N r ( x ) = X \N r ( x ) . If a model is trained on a point x , we would like to treat its neighbors N r ( x ) as approximate members. Instantiating a neighborhood with a radius of r = 0 recovers the traditional exact-matching notion of membership. Using this, we denote neighborhood-based membership:

In light of this, the score assigned by an MI test to a point x should be interpreted as a signal that at least one of its neighbors was used to train the model instead of an indication that x exactly matches a sequence from the training set. Following the formalism from [10], we define an MI test as follows:

Definition 3.2 ( γ -Thresholded Membership Inference) . Given a neighborhood N r , an MI test is a mapping T γ : X × Θ → R ≥ 0 , which, for any data point x and model parameters θ , returns a 'membership score' for testing the null hypothesis that x ̸∈ N r D . Scores above a threshold γ ∈ R ≥ 0 suggest membership: 1 [ T γ ( x, θ ) ≥ γ ] ≈ x ∈ N r D.

When is a Membership Inference Test Sound? Here, we adapt the standard metrics-sensitivity (true positive rate) and specificity (true negative rate)-to the pointwise setting as follows:

Definition 3.3 (Pointwise Membership Sensitivity (True Positive Rate)) . The sensitivity of an MI test T γ for a point x with respect to a neighborhood N r is the probability that the test correctly identifies a sequence as a member:

Definition 3.4 (Pointwise Membership Specificity (True Negative Rate)) . The specificity of an MI test T γ for a point x with respect to a neighborhood N r is the probability that the test correctly identifies a sequence as a non-member:

Intuitively, the more separable the score distributions of a membership test for models trained/nottrained on any neighbors, the more powerful the test. The following result connects the specificity and sensitivity to the separability of the membership test scores under both hypotheses:

Lemma 3.5. (Advantage of an MI test). For a point x , the advantage of a test T γ is given by the difference between the expected membership scores under the null and the alternative hypotheses:

The proof is in Appendix A.2. The integrand on the left is commonly known as Youden's J statistic [23], and captures the difference between the true positive rate and false positive rate, i.e., the advantage over a random guess. Later in Section 5, we build upon this result to show the difficulty of designing robust MI tests.

4 A Robustness Perspective on Membership Inference

We begin by defining a family of membership-invariant perturbations to the dataset, i.e., perturbations to the dataset that do not change the membership label for some point:

Definition 4.1 (Membership Invariant Dataset Perturbation) . Given an original dataset D ∼ D , a dataset perturbation operator Pert : P ( X ) →P ( X ) is membership-invariant w.r.t a point x if:

In other words, a point's membership label should remain unchanged under perturbations-members stay members, and non-members stay non-members. Ideally, a membership test should be robust to such perturbations. We focus on a specific type of perturbation via substitution: replacing neighbors (or non-neighbors) of x with other neighbors (or non-neighbors). We denote such a perturbed dataset to lie in the expansion of the original dataset D .

Definition 4.2. (Dataset Expansions Under Substitution) The b -neighborhood expansion of a dataset D around point x (for some notion of neighborhood N r ) is the set of all datasets that can be made by only substituting b points (from D ) that lie in N r ( x ) with other points from N r ( x ) . Similarly, the b -non-neighborhood expansion arises by substituting points that lie in N r ( x ) with other points in N r ( x ) 3 . Concretely, with S ∈ {N r ( x ) , N r ( x ) } , these expansions are given by:

In this paper, we explore the problem of constructing worst-case datasets from the expansion of an original dataset D -that is, perturbed datasets in which x remains a non-member (since only non-neighbors were substituted with other non-neighbors), yet the test incorrectly classifies it as a member. Conversely, one can also construct examples where x remains a member, but the test incorrectly predicts it to be a non-member. An illustration of this phenomenon is provided in Figure 1.

Such worst-case datasets are interesting because they contend with the reliability of a test - if a test can be arbitrarily made to fail on a point, despite the ground truth (i.e., membership label) being unchanged, is the test still useful? Furthermore, such worst-case datasets have practical, real-world implications under a poisoning threat model.

Threat model. In our setting, the adversary can be anyone capable of planting poisoned data on the web, such as through poisoning publicly available sources like Wikipedia backups [14].

Formally, such scenarios can be characterized as a game between a challenger C , an adversary A , and an arbiter J . Here, the adversary A 's goal is for the test to assign incorrect membership predictions :

Adversary A chooses a target point x t ∈ X and sends x t to the challenger C .

Challenger samples the training set D I such that x t ∈ N r D I , and D O such that x t / ∈ N r D O , and sends ( D I , D O ) to adversary A .

Adversary A poisons D I with budget b to obtain D p I ∈ B b ( D I , N r ( x t )) , and poisons D O to obtain D p O ∈ B b ( D I , N r ( x t )) . Poisoned datasets ( D p I , D p O ) are sent back to challenger C . 4

Challenger C flips a random bit c and trains model parameters as θ = L ( D p I ) if c = 0 and θ = L ( D p O ) if otherwise. Challenger then sends ( x t , θ ) to arbiter J .

Arbiter J leverages a membership test to output a prediction bit as ˆ c = T γ ( x t , θ ) .

̸

Adversary A wins if ˆ c = c .

Note that, in a departure from typical security games, we introduce an additional arbiter J to capture the fact that the goal of the poisoning is to mislead the membership test as evaluated by an arbitrary third-party. This is also reflected in the real-world motivating examples discussed earlier, where the arbiter may be a judge ruling on a copyright violation case or an auditor assessing a model's fairness.

In order to characterize the success of such an adversary, we extend the performance metrics from Definitions 3.3 and 3.4 to account for the effects of poisoning:

Definition 4.3 (Pointwise Robustness) . The robust sensitivity of a test T γ for a point x w.r.t neighborhood N r is defined as the probability that, after substituting b neighbors with their worst-case neighbors, the test still correctly classifies x as a member:

Similarly, for robust specificity:

In Equation 2 above, the inner minimum represents the adversary's replacement of the original dataset D (of which x is a member: x ∈ N r D ) with a carefully chosen dataset D ′ such that it decreases the membership signal for x in the trained model, while maintaining x 's member status, i.e., x ∈ N r D ′ . Concretely, this dataset is obtained by substituting b neighbors of x with other carefully selected neighbors of x such that the test's output score is minimized. Similar observation holds true for Equation 3, except we are now trying to maximize the test score.

Note . The above measures of robustness are similar in spirit to those employed for the widely studied concept of adversarial robustness [24] (e.g., robust accuracy). However, our setting introduces a key distinction: worst-case analysis is instead performed over perturbations of the training dataset , rather than perturbations of the individual point, i.e., b -expansions of D instead of l p norm of x .

5 PoisonM : Poisoning Membership Inference

In this section, we propose PoisonM , a concrete instantiation of a dataset poisoning attack on MI tests. We begin with an overview and follow with implementation details.

Overview. To construct a poisoned dataset for a target point x t , the adversary A must provide:

W.l.o.g, let's consider the case where the adversary wants to induce a false positive, i.e., the target x t is originally not a member. The goal is to substitute 'clean' non-neighbors of x t with worst-case, i.e., 'poisoned' non-neighbors such that the membership test T γ incorrectly assigns high membership scores to x t (see Figure 1). The key insight of PoisonM is that, to find such a poisoned non-neighbor, one can (1) sample an actual neighbor x sample , and then (2) 'map' this neighbor back to a nonneighbor that is T γ -equivalent to x sample -meaning that training on either point causes T γ to assign the same score to x t . This mapped non-neighbor x poison is thus the poison. If a model is trained on x poison , the MI test should ideally produce the same output on x t as it would if x sample had been in the training set instead. The success of PoisonM can be formalized as follows:

where S ∈ {N r ( x t ) , N r ( x t ) } is the domain of the mapping in which the poison should lie, and δ ( x t ,D ) x sample is the mapping error, i.e., how much the poison differs in T γ 's score for x t as compared to x sample . Extending this to find b poisons, we define:

and the mapping error is given by δ ( x t ,D ) ( x 1 ,...,x b ) sample . Here b is referred to as the budget of the attack. We now provide a result that demonstrates the difficulty of obtaining a robust MI test.

Theorem 5.1. (Tradeoff of Membership Inference Under PoisonM ). Let x t be a target point. The advantages of a test T γ with and without poisoning are at odds with each other:

The proof is in Appendix A.7. The R.H.S represents the expected mapping error δ ∗ , while the L.H.S captures the total advantage (over random guessing) of the MI test, both in the presence and absence of poisoning. When PoisonM 's mapping error δ ∗ is small-i.e., the attack is successful-the theorem above implies a surprising insight: the advantage of the membership test can be turned against itself. Specfically, the better the test performs (as measured by Youden's J statistic) on clean points, the more vulnerable it becomes to our poisoning attack. To build intuition, consider a scenario where the adversary aims to induce a false negative by constructing a poisoned neighbor. In the ideal case where δ ∗ = 0 , the poisoned neighbor has the same effect as a clean non-neighbor to T γ . As a result, T γ assigns to x t the same (low) score as it would assign if trained on the clean non-neighbor. This effectively fools T γ into making an incorrect prediction, exploiting its own strength in distinguishing members from non-members. Thus, the above result delineates a fundamental trade-off for an MI test: strong performance on clean data comes at the cost of robustness to poisoning attacks. This is also empirically validated in Section 6.

A natural question arises: for a given x t ∈ X , do such low-error poisons actually exist? In practice, they often do-this stems from a misalignment between the 'balls' defined by the generic notions of neighborhood, and the actual superlevel sets of the test, which define regions that trigger high

Algorithm 1 PoisonM Attack

membership scores for x t . This is illustrated in Figure 1 - poisons exist in the small gaps between the contours of the test's superlevel sets and the actual neighborhood boundary. This phenomenon is reminiscent of adversarial examples in classification, where there is a well known gap between l p balls and the superlevel sets of the classifiers (or decision boundaries) [25].

Implementing PoisonM for Different Neighborhoods. PoisonM involves solving the discrete optimization in Equation 4, tailored to the chosen neighborhood. Our general method, outlined in Algorithm 1, follows a greedy coordinate descent approach inspired by Zou et al.[26]. Given a target x t , we sample a neighbor or non-neighbor, then iteratively (1) select a random token and (2) replace it with one that minimizes a neighborhood-specific poisoning loss. For poisoned non-neighbors, we maximize distance while preserving model activations to mimic the sampled point's influence on x t . For poisoned neighbors, we minimize the neighborhood distance and stop once the point qualifies as a neighbor. The losses for four popular choices of neighborhood are in Table 1. Notably, PoisonM is MI-test agnostic -given a neighborhood definition, a single poisoning strategy is effective across all evaluated MI tests, and the adversary needs no knowledge of which specific test will be employed.

6 Evaluation

6.1 Experimental Setup

Models and Training. We use the Pythia models [2], primarily the 6.9B variant, with ablations on 2.7B and 12B. All models are fine-tuned (for 1 epoch) on poisoned data using AdamW (lr = 2e-5, batch size = 16). We focus on finetuning setting, i.e., the adversary poisons the finetuning dataset. Datasets. Following [27], the model is finetuned on a mixture of a 'canary' and a 'background' dataset, where we will run membership inference on the canaries. We use Wikitext-103 as background and AI4Privacy/AGNews as canary datasets, injecting 500 canaries into 100K background points and holding out another 500 canaries for evaluation. Membership labels are assigned based on a neighborhood definition: although only 500 canaries are in the training set, points from the hold-out dataset may also be considered members if they have neighbors in the training dataset. For each definition of neighborhood, we construct a single poisoned dataset in which we generate poison neighbors with budget b 1 = 1 for members, and poison non-neighbors with b 2 = 10 for the rest. The resulting model should flip membership status-predicting members as non-members and vice versa. Metrics. We select 5 popular tests: LOSS [16], Min-K% Prob [17] with K = 0 . 2 , zlib [18], perturbation-based [28], and reference-based [18]. For perturbation and reference-based tests, we

evaluate all configurations from Maini et al. [20] and present the setting that performs the best for membership inference for each neighborhood definition. Metrics include AUC and TPR@1% FPR. We also evaluate dataset inference [20], a test providing p -values for aggregated membership prediction. We use 4 neighborhood definitions, with fixed parameters: k = 7 for n -grams, c = 0 . 9 for cosine similarity 5 , l = 0 . 48 for normalized edit distance, and exact match.

Baselines. As a baseline for generating poisoned non-neighbors, we adapt Liu et al's [13] recent work on three techniques for forcing completion on sequences not trained upon. These are: (a) dropping tokens at regular intervals, (b) flipping the case of characters probabilistically, and (c) inserting chunks into a sequence of random tokens (also similar to [31]). To maximize the performance of baselines, we further perform a hyperparameter search for each approach to select the least destructive parameter values (e.g., drop rate, flipping probability, chunk size) that still maintain non-neighbor status. For generating poisoned neighbors, to the best of our knowledge there are no existing baselines.

6.2 Experimental Results

Overview. We present the full ROC curves of tests before and after poisoning on the AI4Privacy and AGNews datasets in Figure 2, with AUC scores presented in Table 2. Since we have trained the model on both poisoned neighbors (to make members look like non-members), and poisoned non-neighbors (to make non-members look like members), we expect the AUC to considerably reduce. Indeed, we observe that PoisonM is nearly always able to reduce AUC below random. In many cases, it can be reduced considerably below random, or even close to 0 (e.g., n -gram neighborhood with reference-based test). While the baselines are effective in some cases, PoisonM consistently outperforms them since they were not designed to manipulate membership testing. This advantage likely stems from two key factors: (1) PoisonM 's ability to generate poisons for both neighbors and non-neighbors, and (2) the greater effectiveness of its poisoned non-neighbors, as shown in the exact match setting-where all methods are limited to poisoning non-neighbors only.

We also observe a general trend that aligns with Theorem 5.1 - the tests that perform the best naturally are also the lowest/rank low in terms of robust AUC. For example, the reference-based test

0 0

.

20

.

40

.

60

.

8 1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

FPR

TPR

LOSS

Standard

Token Dropouts

Chunking

PoisonM

0 0

.

20

.

40

.

60

.

8 1

FPR

kmin

0 0

.

20

.

40

.

60

.

8 1

FPR

zlib

0 0

.

20

.

40

.

60

.

8 1

FPR

perturb

0 0

.

20

.

40

.

60

.

8 1

FPR

reference

ranks the highest naturally, and the lowest under poisoning across all AI4Privacy settings, and in many settings in AGNews. We also present the TPR@1% FPR in Table 6 of Appendix B, where we find that again, PoisonM is able to reduce performance.

Dataset Inference. Dataset inference extends MI testing to whole datasets by (1) ensembling existing tests via a linear model and (2) using a T-test to compare scores from a suspect set to a reference set of known non-members [20]. We test whether poisoning affects this method by evaluating it on models fine-tuned with poisoned data (Table 3). The results show that dataset inference fails under poisoning, yielding incorrect predictions. This aligns with intuition: if individual tests are driven below random, so is their ensemble-mirroring how ensembling weak defenses fails for adversarial examples [32].

Impact of Neighborhood Radius. We examine how changing the neighborhood radius r affects poisoning, focusing on the LOSS test with n -gram neighborhoods on AI4Privacy for k ∈ [5 , 7 , 9 , 11] (Figure 3). As expected, larger radii (smaller k ) reduce vulnerability to poisoned non-members but increase it for poisoned members, and vice versa.

Impact of Model and Dataset Size. We also study how model size affects poisoning success by repeating our n -gram ( k = 7) experiments on AI4Privacy using Pythia 2.7B and 12B. PoisonM consistently reduces test performance across all sizes (see Table 7 of Appendix B). The 2.7B model shows slightly lower natural accuracy and slightly higher robustness, aligning with Theorem 5.1. In Table 8, we repeat our experiments with a larger background dataset size of 1M WikiText points, and find that PoisonM continues to be effective.

4

8

12

0

0

.

2

0

.

4

n

-gram size

Metric value

PoisonM

(TPR)

PoisonM

(FPR)

Filtering as a Defense. PoisonM is designed to be a strong, general-purpose attack and can adapt to defenses by incorporating filtering criteria - such as a perplexity filter - directly into its loss function. To demonstrate this, Table 4 considers a setting in which a perplexity filter is employed (threshold selected to ensure low FPR of 1%), and shows PoisonM can adapt to this setting and remain effective against all considered MI tests.

Poison Transferability. Even under a more restrictive setting where query access to the target model is prohibited, an attacker can optimize against a surrogate model, with the expectation that the poison will transfer to the target model. In Table 5, we experiment with an OLMO2 7B [33] model trained on poisons computed against itself (as is typical), and then on poisons computed against a surrogate Pythia 6.9B model. We find that PoisonM is still effective against MI tests even in this completely blind setting, without query access to the target model.

7 Conclusion and Discussions

We have studied the reliability of membership inference against LLMs under poisoning attacks. Although the shift from exact matching to neighborhood-based definition aims to enhance reliability of MI tests, we reveal fundamental flaws remain even under this relaxed definition, calling into question what it truly means for a data point to be considered a member. Moreover, the wide applicability of our attack across common neighborhood definitions highlights inherent difficulties in designing a generic yet meaningful notion of membership. One possible way forward is to consider

model-dependent or context-aware definitions that better align with how MI tests actually operate. Finally, although we primarily focused on textual data, we expect our analysis to generalize beyond the language domain. For example, while approximate membership definitions for image data may consider transformations such as rotation, cropping, and filtering, such definitions are likely to suffer from similar robustness issues.

Limitations. One limitation is that our attack for generating poison non-neighbors requires multiple poisons, i.e., b 2 = 10 . Future work may improve upon this. We also focus our experiments on specific settings, and larger-scale evaluations with more models/datasets/neighborhoods can help towards evaluating what it truly means for a point to be a member. We also do not evaluate the pre-training setting due to the computational costs, although it has been shown to be viable [34].

References

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks against Machine Learning Models. In Proc. of IEEE S&P , pages 3-18. IEEE, 2017.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Proc. of ICML , 2023.

The New York Times. The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work. https: //www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoftlawsuit.html , 2023. Accessed: 05/15/2025.

The Guardian. US authors' copyright lawsuits against OpenAI and Microsoft combined in New York with newspaper actions. https://www.theguardian.com/books/2025/apr/04/usauthors-copyright-lawsuits-against-openai-and-microsoft-combined-in-new-yorkwith-newspaper-actions , 2025. Accessed: 05/15/2025.

Reuters. Publisher Ziff Davis sues OpenAI for copyright infringement. https://www.reuters.com/ business/publisher-ziff-davis-sues-openai-copyright-infringement-2025-04-24/ , 2025. Accessed: 05/15/2025.

isaca.org. Understanding the EU AI Act: Requirements and Next Steps. https://www.isaca.org/ resources/white-papers/2024/understanding-the-eu-ai-act , 2024. Accessed: 05/15/2025.

Ben Chester Cheong. Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making. Frontiers in Human Dynamics , 6:1421273, 2024.

Sabina Lacmanovic and Marinko Skare. Artificial intelligence bias auditing-current approaches, challenges and lessons from practice. Review of Accounting and Finance , (ahead-of-print), 2025.

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do Membership Inference Attacks Work on Large Language Models? In Proc. of COLM , 2024.

Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data. arXiv preprint arXiv:2409.19798 , 2024.

Debeshee Das, Jie Zhang, and Florian Tramèr. Blind Baselines Beat Membership Inference Attacks for Foundation Models. arXiv preprint arXiv:2406.16201 , 2024.

Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What Does it Mean for a Language Model to Preserve Privacy? In Proc. of FAccT , 2022.

Ken Ziyu Liu, Christopher A Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, and Nicolas Papernot. Language Models May Verbatim Complete Text They Were Not Explicitly Trained On. In Proc. of ICLR , 2025.

Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-scale Training Datasets is Practical. In Proc. of IEEE S&P , pages 407-425. IEEE, 2024.

Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent Pre-Training Poisoning of LLMs. In Proc. of ICLR , 2025.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In Proc. of CSF , 2018.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. In Proc. of ICLR , 2024.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting Training Data from Large Language Models. In Proc. of USENIX Security , 2021.

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership Inference Attacks against Language Models via Neighbourhood Comparison. In Proc. of ACL , 2023.

Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. LLM Dataset Inference: Did you train on my dataset? In Proc. of NeurIPS , 2024.

Amrita Roy Chowdhury, Zhifeng Kong, and Kamalika Chaudhuri. On the Reliability of Membership Inference Attacks. In Proc. of SaTML , 2025.

Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models. In Proc. of NeurIPS , 2024.

William J Youden. Index for rating diagnostic tests. Cancer , 3(1):32-35, 1950.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In Proc. of ICLR , 2018.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In Proc. of ICLR , 2014.

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 , 2023.

Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In Proc. of CCS , 2022.

Filippo Galli, Luca Melis, and Tommaso Cucinotta. Noisy Neighbors: Efficient membership inference attacks against LLMs. In Proc. of the Fifth Workshop on Privacy in Natural Language Processing , 2024.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 Text Embeddings: A Technical Report. arXiv preprint arXiv:2402.05672 , 2024.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark. In Proc. of EACL , 2023.

Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, and Furong Huang. PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proc. of NAACL , 2025.

Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial Example Defense: Ensembles of Weak Defenses are not Strong. In Proc. of USENIX WOOT , 2017.

Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, et al. 2 OLMo 2 Furious (COLM's Version). In Proc. of COLM , 2025.

Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent Pre-Training Poisoning of LLMs. In Proc. of ICLR , 2025.

A Proofs

Lemma A.1. (Expectation using survival function). Let X be a random variable such that P ( X ≥ 0) = 1 , then we have

Lemma A.2. (Restatement of Lemma 3.5). The advantage of a membership inference test is given by

Proof. Using Lemma A.1, we get

Definition A.3. (Targeted Expansion) The targeted b -neighborhood expansion (around a point x ) of a dataset D ∈ P ( X ) from a set S to a set S ′ is the set of all datasets that can be made by only substituting at most b sequences (from D ) that lie in S with points in S ′ :

Lemma A.4. Upper bound on MI score under neighbor poisons.

Proof.

Proof. For x ′ 1 , ..., x ′ b ∼ N r ( x ) , we know,

Now, to extend this for dataset substitutions, for x 1 , ..., x b ∼ N r ( x ) ∩ D and D ′ = D \ { x 1 , ...x b } , we have:

Taking expectation over points, we get:

Using Jensen's Inequality:

Taking one side of the absolute value:

Then,

Simplifying:

Lemma A.5. Lower bound on MI score under non-neighbor poisons.

Proof. For x ′ 1 , ..., x ′ b ∼ N r ( x ) , we know,

Now, to extend this for dataset substitutions, for x 1 , ..., x b ∼ N r ( x ) ∩ D and D ′ = D \ { x 1 , ...x b } , we have

Taking expectation over points, we get

Using Jensen's Inequality:

Taking one side of the absolute value:

Then,

Simplifying:

Lemma A.6. Advantage of MI under poisoning.

Proof. Using Lemma A.1, we get

Similarly,

Now, using Lemma A.4 and Lemma A.5,

Proof. For b 1 = |N r ( x ) ∩ D | , we get,

Similarly, for b 2 = | v ∩ N r ( x t ) | , v ∼ D , we get,

Applying this to Lemma A.6, we get the result.

B Additional Results

C Additional Details About PoisonM

Neighborhoods. We consider the following popular neighborhood definitions:

̸

N-gram(n=k): For a given sequence of x = x 1 , · · · , x n , let n -gram ( x, k ) = { x i : x i + k } n -k i =1 denote the set of all k -grams of x . Then, the n -gram neighborhood yields the set of all sequences that share an k -gram with x : { x ′ ∈ X | n -gram ( x ′ , k ) ∩ n -gram ( x, k ) = ∅} . Embedding(cosine_sim=c): Let E : X → R d denote an embedding function that maps sequences to d -dimensional representations that capture their 'semantics'. Then, for a given sequence of x , the embedding similarity neighborhood yields the set of all sequences with embeddings of cosine similarity at least c to x : { x ′ ∈ X | E ( x ′ ) · E ( x ) || E ( x ) || || E ( x ′ ) || ≥ c } .

ExactMatch: For a given sequence x , the exact matching neighborhood yields the singleton comprising the sequence itself, i.e., { x } .

EditDistance(distance=l): For a given sequence x , the edit distance neighborhood yields the set of all sequences that are within a normalized Levenshtein distance l from x : { x ′ ∈ X | lev ( x,x ′ ) | x | + | x ′ | ≤ l } .

Finding Poison Non-Neighbors. We sample an actual neighbor as x t itself, and then:

N-gram(n=k): Iteratively (a) select a token uniformly at random, (b) replace it with a token from the vocabulary such that last-layer activations of resulting sequence have maximum cosine similarity with activations of x t , where activations are computed using model that will be trained on the poison, and (c) repeat until n -gram overlap between the current poison and x t is less than k . Here we can set λ = 0 since replacing tokens automatically breaks up n -grams for free.

Embedding(cosine_sim=c): Iteratively (a) select a token uniformly at random, (b) pick the token that both maximizes activation cosine similarity and also minimizes (weighted by factor of λ = -1 . 5 ) cosine similarity of the embedding (from E ) of the resulting sequence with that of x t .

EditDistance(distance=l): Same procedure as that for embeddings, except we now maximize the edit distance ( λ = 1 . 5 ) instead of minimizing embedding cosine similarities.

ExactMatch: Same procedure as that for n -grams, but only a single iteration.

Finding Poison Neighbors. We sample an actual non-neighbor as a random text from any auxiliary dataset, and then:

N-gram(n=k): Inject a k -gram from x at random index (shortest k -gram in characters).

Embedding(cosine_sim=c): Iteratively (a) select a token uniformly at random, and (b) replace it with a token from vocabulary that maximizes cosine similarity of the embedding (under E ) of the resulting sequence with x t 's embedding. (c) repeat until cosine similarity exceeds c .

EditDistance(distance=l): Iteratively (a) randomly insert, delete, or substitute characters (b) greedily keep the mutation only if it decreases edit distance. (c) repeat until edit distance drops below l .

ExactMatch: Worst-case neighbors do not exist under exact matching, since the neighborhood ball holds a radius of 0 .

D Societal Impacts

This work presents a novel poisoning vulnerability that could be used by a real-world adversary to manipulate the outcome of a high-stakes membership inference test. This could have legal and reputation-related implications. However, we believe it is important to release our findings so that (a) auditors and other entities that may wish to use membership testing may be made aware of its pitfalls, (b) the community can work towards better defining membership.

E Compute

We run all experiments on a machine with 4 NVIDIA H100 GPUs, 40 Intel(R) Xeon(R) Silver 4410T CPUs, and 126GB of RAM. Finetuning models typically took 2 hours, and generating poisoned datasets took between a few minutes to at most 2 hours, depending on choice of neighborhood.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction capture the paper's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed limitations in Section 7 of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have listed the assumptions for all theoretical results, and included the proofs in the Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have discussed this in Section 6.1 of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code in supplementary material and datasets are public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details provided in Section 6.1 of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to computational difficulties we cannot do several runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of compute are presented in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and conform to the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed societal impacts in Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not pose such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited assets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does involve releasing new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There are no crowdsourced experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Not applicable per the policy.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the vulnerability of membership inference (MI) tests used to detect whether data points were part of a machine learning model's training set. The study demonstrates that MI tests remain susceptible to manipulation even when enhanced with relaxed membership definitions that consider approximate matches rather than exact data point correspondences.\nThe work presents PoisonM, a data poisoning methodology that strategically modifies training datasets to induce misclassification by MI tests, while preserving the ground-truth membership status of target data points.\nThe research contributions include:\n– Establishing formal robustness definitions for MI tests against adversarial training set perturbations.\n– Developing practical algorithms for constructing poisoned training datasets that compromise MI test effectiveness.\n– Demonstrating a fundamental accuracy-robustness trade-off, where higher MI test performance on clean data correlates with increased vulnerability to adversarial manipulation.\n– Providing comprehensive experimental validation showing MI test degradation under PoisonM across diverse model architectures and evaluation protocols.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nQuality\nTheoretical Rigor: The paper clearly defines core concepts like neighborhood-based membership, robustness to changes in the training data, and mapping error. It builds a solid mathematical foundation for the attack and introduces pointwise robustness and Theorem 5.1 to support its claims. Also I have to add that most of the math in this paper was beyond me and could not follow it closely.\n\nComprehensive Experimental Analysis: The theoretical contributions are substantiated through extensive empirical evaluation, examining PoisonM effectiveness across diverse MI attack variants, benchmark datasets, model architectures, and similarity metric configurations. The experimental findings demonstrate persistent vulnerability in MI attacks despite relaxed membership criterion implementations.\n\nGeneral-Purpose Attack: PoisonM is flexible and works across various MI tests and neighborhood definitions, making it a broadly applicable and reproducible attack strategy.\n\nClarity\nClear Motivation: The connection to real-world applications (e.g., copyright enforcement, regulatory compliance) makes the work relevant and compelling. The threat model involving an adversary, challenger, and arbiter maps well to practical use cases. Structured Presentation: The paper is well-organized with clearly stated motivation, formal definitions, theoretical insights, algorithmic implementation, and experimental results.\n\n\nSignificance\n\nCurrently very relevant: Figuring out whether a piece of data was used to train an AI model is an important issue in current discussions about privacy and how transparent large language models (LLMs) should be. This paper shows that the tools used to answer that question can be fooled.\n\nPolicy and Legal Impact: The results raise concerns about using MI tests in real-world settings, like court cases or government audits, where accuracy really matters.\n\nBroad Applicability: PoisonM applies to both exact and semantic definitions of membership and can be extended to other forms of training-data attribution, making it relevant to future privacy research.\n\n\nWeakness: \nClarity\nDense Notation in Theoretical Sections: Definitions and theorems (particularly Theorem 5.1) use heavy notation with nested expectations and subscripts, which may be inaccessible to a broad audience. \n\nSignificance\n\nLimited Comparisons: The experiments are strong, but the paper would be more convincing if it compared PoisonM to existing defenses or other robust MI test methods to show how well it performs in comparison.\n\nLack of Real-World Impact Analysis: Although the paper has clear practical relevance (like in legal or audit situations), it doesn’t show how badly MI tests might fail in real life—for example, how often they might give wrong answers or how much trust in them could be affected.\n\nQUESTIONS:\n1) How the choice of neighborhood definition (n-gram overlap vs. embedding similarity ) influence the performance of PoisonM?\n\n2) How feasible is it for an adversary to insert poisoned examples into public data sources in a way that ensures they are included during large-scale model training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to demonstrate the unreliability of membership inference tests. A membership inference test takes a trained model and an item of data and returns a score which indicates whether the item was used in the training process of the model. The approach taken in the paper is to demonstrate that it is possible to poison the training dataset such that an MI test gives incorrect predictions, e.g. an adversary may place text on their personal website which, when scraped, results in a model which is trained on that website returning a positive result for some training data which is not used in the training process.\n\nThe authors give a concrete example of an attack which can induce an incorrect prediction from an MI test and show that it works with leading MI tests and models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ The paper studies a novel (to my knowledge) setting which could have much real-world significance.\n+ The paper is generally clear and well-structured. The threat model is clearly laid out, even if I am not sure of the real-world significance.\n+ The paper provides code which is good for reproducibility\n+ The approach, PoisonM, is straightforward but clearly effective.\n\n### Weaknesses\n+ I am unsure of the real-world applicability of these neighborhood-based MI tests. I understand that there is significant previous work on the subject, but it does not seem really applicable for e.g. copyright if the training dataset contains an entry with a 7-gram overlap from a copyrighted source: the dataset could include a quote from the copyrighted source or indeed an independently written passage which shares the 7-gram overlap.\n+ The paper seem a bit over-written, e.g. I'm not sure that the threat model / security game section adds very much to the paper and could be replaced with some more experimental results.\n\nQUESTIONS:\n1. I am confused as the real-world applicability of the threat presented in the paper. In the example given, an adversary plants a poison such that the MI test predicts that there were Larry Lobster novels in the training corpus, triggering a lawsuit. However, couldn't the developer offer access to the dataset to a trusted third party, who could verify that the Larry Lobster novels are not in the dataset?\n2. In a data poisoning setup, a standard threat model is to assume that an adversary can add data to a dataset but not change it. For example, a user could add fake news articles to their own website but could not edit news articles scraped from CNN.com. Does this present any challenges to your approach, which relies on substituting data in a dataset?\n3. Why do the membership tests in figure 2 have such low AUC in general? They appear to be quite bad tests.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of membership inference tests under the neighborhood-based definition where all semantic neighbors of a training point are also treated as members. Authors proposed to study the MI test from a robust perspective through substituting neighbors or non-neighbors of target training point. And authors theoretically show that there is a trade-off between a test’s accuracy and its robustness to poisoning. Experimental results show the proposed poisoning attack can make existing MI test less effective.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths in this paper are summarised as follows: \n1. This paper studies the problem of membership inference in a relaxed neighborhood-based assumption, i.e., all semantic neighbors of a training point can be considered as members. I believe this research perspective is more practical, especially in the ear of LLM. So, the research motivation and significance are strong.\n2. Authors give thorough definitions and theoretical analysis about the problem of membership inference tests. \n\nHowever, there are some weaknesses in this paper shown as follows: \n1. A clear description about attacker’s background knowledge is expected. For example, does an attacker have to know the data distribution of target dataset in the scenario this paper studied? In addition, what kind of information the attacker can obtain from the target model, such that the attacker can exploit the information to implement the follow-up membership inference test? \n2. The definition of 4.2 is not clear, especially the given equation of expansions of dataset. \n3. A clearer description about dataset perturbation, worst-case dataset, and the reliability of membership inference test should be very important and expected. Even though authors gave clear descriptions on other definitions, explanation and justification about this part are vague. \n4. The poisoning attack might only be applicable to the specific form of MI concerned in this work. Once an attacker knows how MI works (i.e., how to construct the features for member and non-member), she could alway find a way to fool MI more or less if she could manipulate/poisoning the training/tuning data.\n\nQUESTIONS:\n1. This is a metric-based membership inference, how to determine the threshold value?\n2. Can authors present more clearly about worst-case datasets? And give a clear explanation about why membership inference test can make wrong predictions when the perturbed dataset is the worst-case dataset? \n3. In Figure 1, what does the orange area/line mean? Is the target point a member? \n4. Please give more detailed explanation about Figure 1.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors question the validity of membership inference evaluations for LMs demonstrating that even under the relaxed neighborhood-based membership evaluations, efficient poisoning can help completely flip predictions for members and non-members alike. The authors propose concrete instantiations for poisoning schemes targeted at various downstream MIAs and support their results with theoretical analysis of the poisoning and its relationship to attack success. The authors evaluate their poisoning attack on multiple models and downstream MIAs, and even extend their poisoning attack to dataset inference\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- I think the scenario considered here is very interesting and practical: not only can it be used by malicious data curators and model owners to bypass any membership/dataset inference tests, but also scrutinize companies and organizations by selectively poisoning Internet data with records that make models behave like they have strongly overfit on some target data.\n- The theoretical analyses are explained very clearly and paired with intuition, making them easy to follow. Overall, I found the paper very well written.\n- Experimental analyses are thorough, with evaluations across multiple model sizes, attacks, and even inference granularities (record and dataset level)\n- I really like Figure 1! I think it gets across the point beautifully\n\n# Weaknesses\n\nI don't see any critical weaknesses in the paper, but there are some things that could be improved or clarified:\n\n- The way I see it, flipping non-membership to membership is straightforward: just insert the target datapoint itself into model training data. Either the model trainer actually scans its training data to check for the record (which is unlikely but in case it does, it can share the computation as a proof of non-membership with any relevant arbiter), or it doesn't (in which case, there is no reason to be stealthy and not just insert the target record directly). It is hard to think of a practical scenario where a model trainer would want to prove that they did in fact train on some particular record in training. Making a member be classified as a non-member is particularly interesting, especially considering potentially **adversarial model trainers or data curators** that use some copyrighted data, but augment it with such poisoned data such that any downstream MI/DI evaluations make it look like such data was never used in training. \n\n- L260: While this is understandable, I think the authors can try and at least simulate what the effect of poisoning might be by a) using a lot more data, and/or b) partitioning the data such that all poison data is injected in the first X% of fine-tuning data. Given the reasonable compute requirements (as detailed in the Appendix), it would be nice to potentially consider another open-source model family (like OLMO/OLMO2 [1]).\n\n- The per-record effective DPR (data poisoning rate) is reasonably low: 10 poisons per record in the worst case for a 100K dataset size, i.e. 0.01% DPR. However, for a set of 500 members and non-members this number does scale up very quickly, reaching 5.5% DPR (for 500 members and 500 non-members) : quite high, especially for LLM training! This DPR would be particularly relevant for dataset inference, where the adversary would have to influence membership labels for multiple records to sway the final DI outcome. Can the authors comment on (and perhaps analyze) the tradeoff between the DPR here and final attack success, at least for the DI case (and potentially also for MI by controlling the number of poisons added for non-members)? \n\n## Minor Comments\n\n- L2: \"language model's training set\" - although this work focuses on text models, the MI definition is not specific to language models - please rephrase to clarify the distinction.\n- Might be worth looking into attacks like Min-K++ [2]\n- Figure 2: Please increase size and use log-log scale (focus on low-FPR region)\n- L319-321: The 2.7B model has had very \"unexplainable\" observations in several works (acknowledged by the Pythia authors too) so I would not read too much into results for this model- maybe test if for a smaller one from the same model family?\n\n### References\n- [1] OLMo, Team, et al. \"2 OLMo 2 Furious.\" arXiv preprint arXiv:2501.00656 (2024).\n- [2] Zhang, Jingyang, et al. \"Min-k%++: Improved baseline for detecting pre-training data from large language models.\" arXiv preprint arXiv:2404.02936 (2024).\n\nQUESTIONS:\n- Definitions in the paper talk about \"replacing\" actual neighbors with other neighbors (Definition 4,2. etc) but in a realistic data poisoning setting, the adversary can only add more data, not remove/replace any data. How do these two connect? Given the way neighborhood is defined, injecting neighbors, even if they are closer to the target record, does not make existing neighbors non-neighbors. Do the authors actually replace these neighbors in their experimental setup, or is it purely for easier theoretical analysis?\n\n- Lemma 3.5: How does this connect to/differ from the standard definition of the adversary's advantage in an cryptographic game?\n\n- Regarding the last step of the adversarial game (under threat model): because of the additional party, A's success is now also tied to C's success for the benign case. For instance, I could not poison either data and if C's test is not strong enough (which is the case for MIAs for LLMs), A wins by default in most cases. However, theoretical analyses further down in Theorem 5.1 suggest that this advantage would be high again if the attack is too powerful. Is there then a U-shaped curve for the adversary's advantage with respect to the downstream MIA's performance?\n\n- L230: Intuitively, this seems off. By construction, the poison data is a neighbor of X. If this poison is also indistinguishable from a clean non-member, then that clean non-member is also in the neighborhood of X, which makes it a member. \n\n- L253: I am a little confused about the generality claims. Unless I missed it, the way I understand each poisoning experiment corresponds to adding data upon anticipating some MIA that might be used in downstream inference. Is that the case, or do the authors use one poisoning strategy and then test other MIAs on it? Some clarification would be nice. Also, why not enforce this generalization by design? The loss used for token selection could be easily adapted to consider multiple MIAs simultaneously. \n\n- Could the authors share some of these generated poison samples to give a sense of what they look like? Given the use of token replacements I imagine them to look unnatural (high perplexity) and easily filterable with cheap alternatives, but that discussion can only continue after looking at the poison texts.\n\n- L300: What is the level of \"dataset\" here for dataset inference? Please provide additional details in the main body.\n\nLIMITATIONS:\nYes, but I would encourage the authors to talk about it a bit more in the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}