Blind Baselines Beat Membership Inference Attacks for Foundation Models

## Abstract

Abstract Membership inference (MI) attacks try to determine
if a data sample was used to train
a machine learning model.
For foundation models trained on unknown Web data,
MI attacks can be used to detect copyrighted training materials,
measure test set contamination, or audit machine unlearning.
Unfortunately, we find that evaluations of
MI attacks for foundation models are flawed,
because they sample members and non-members
from different distributions.
For 8 published MI evaluation datasets,
we show that blind attacks—that
distinguish the member and non-member distributions
without looking at any trained model—outperform
state-of-the-art MI attacks.
Existing evaluations thus tell us nothing
about membership leakage of a foundation model’s training data.

## 1 Introduction

Many foundation models Bommasani et al. () such as GPT-4 OpenAI (), Gemini Gemini Team () and DALL-E Ramesh et al. () are trained on unknown data.
There is great interest in methods that can determine if a piece of data was used to train these models. Such methods—called *membership inference attacks* Shokri et al. ()—have been studied for foundation models in many recent works Shi et al. (); Duan et al. (); Ko et al. (); Dubiński et al. (); Zhang et al. (); Meeus et al. (). Applications include privacy attacks Carlini et al. (), demonstrating the use of copyrighted material Meeus et al. (), detecting test data contamination Oren et al. (), or auditing the efficacy of methods to “unlearn” training data Bourtoule et al. ().

To evaluate the performance of a membership inference attack, it is common to
train a model on a random subset of a larger dataset, and then ask the attacker to distinguish these *members* from the remaining *non-members* by interacting with the model.
Emulating this setup for foundation models is hard, since we often do not have access to a held out set sampled from the *same distribution* as the training set.

Existing MI evaluations thus create member and non-member datasets *a posteriori*,
typically by picking members from sources known or suspected to be in the targeted foundation model’s training set, and then attempting to emulate the distribution of these sources to sample non-members.

Unfortunately, we show these strategies are severely flawed, and create easily distinguishable member and non-member distributions.
As a special case of this flaw, concurrent work of Duan et al. () and Maini et al. () finds a temporal shift between members and non-members in the Wiki-MIA dataset Shi et al. ().
We show this issue is persistent by identifying significant distribution shifts (beyond temporal shifts) in 8 MI evaluation datasets for foundation models, for both text and vision.
Worse, we show that existing MI attacks perform “worse than chance” on these datasets. Specifically, we design “blind” attacks, *which completely ignore the target model*, and outperform all reported results from state-of-the-art MI attacks (see Table 1).

Our methods are naive: for some datasets with a temporal shift, we just apply a threshold to specific dates extracted from each sample; for other text or text-vision datasets, we build simple bag-of-words or n-gram classifiers on captions.

Current MI attacks for foundation models thus cannot be relied on, as we cannot rule out that they are (poorly) inferring membership based on data *features*, without extracting any actual membership leakage from the model.
Future MI attacks should be evaluated on models with a clear train-test split, e.g., using the Pile Gao et al. () or a random subset of DataComp Gadre et al. () or DataComp-LM Li et al. ().

**Table 1: MI evaluation datasets for foundation models are flawed. Our *blind* attacks distinguish members from non-members better than existing attacks, without looking at any model.**
| MI dataset | Metric | Best Reported | Ours |
| --- | --- | --- | --- |
| WikiMIA^1 | TPR@5%FPR | 43.2%^2 | 94.4% |
| BookMIA^1 | AUC ROC | 88.0%^1 | 90.5% |
| Temporal Wiki^3 | AUC ROC | 79.6%^3 | 79.9% |
| Temporal arXiv^3 | AUC ROC | 72.3%^3 | 73.1% |
| ArXiv-1 month^6 | TPR@1%FPR | 5.9%^6 | 13.4% |
| Multi-Webdata^4 | TPR@1%FPR | 40.3%^4 | 83.5% |
| LAION-MI^5 | TPR@1%FPR | 2.5%^5 | 9.9% |
| Gutenberg^6 | TPR@1%FPR | 18.8%^6 | 59.6% |

## 2 Background and Related Work

##### Web-scale training datasets.

Foundation models are often trained on massive datasets collected from web crawls, such as C4 Raffel et al. (), the Pile  Gao et al. () or LAION Schuhmann et al. (). However, there has been a trend towards using undisclosed training sets for models like GPT-2 to GPT-4 Radford et al. (); OpenAI (), Gemini Gemini Team () or DALL-E Ramesh et al. (). Even recent open models like LLaMA Touvron et al. () do not release information about their training dataset. Some datasets have been released for research purposes, such as RedPajama Together (), Dolma Soldaini et al. () or LAION Schuhmann et al. (). Notably, these datasets lack a designated test set.

##### Membership inference attacks.

Membership inference attacks aim to determine whether a given data point was used to train a machine learning model (Shokri et al., ).
Early attacks applied a *global* decision function to all samples (e.g., by thresholding the model’s loss Yeom et al. ()). Current state-of-the-art attacks
calibrate the attack threshold to the difficulty of each sample (Carlini et al., ).

##### Membership inference for foundation models.

Membership inference attacks have been applied to LLMs Shi et al. (); Duan et al. (); Zhang et al. (); Meeus et al. (), diffusion models Dubiński et al. (), CLIP Ko et al. (), and other foundation models. The motivations for these attacks include using them as a component of a privacy attack Carlini et al. (), for evaluating unlearning methods Shi et al. (), or for data provenance tests to detect the use of copyrighted data Meeus et al. ().
Due to the lack of a dedicated test set (and possibly even an unknown training set) for the targeted models, many of these works design custom evaluation datasets for membership inference attacks, by collecting sources of data that were likely used (respectively not used) for training.

##### Evaluating membership inference.

Membership inference attacks were originally evaluated with average-case metrics such as the ROC AUC on a balanced set of members and non-members Shokri et al. (). More recent work advocates for evaluating the attack’s performance in the *worst-case*, typically by reporting the true-positive rate (TPR) at low false-positive rates (FPR) Carlini et al. ().

Membership inference evaluations are usually set up so that a baseline attack (which does not query the target model) achieves 50% AUC (or equal TPR and FPR).
Some authors have also considered cases where the attacker has a non-uniform prior Jayaraman et al. ().
In either case, the goal of a MI attack is to extract information from the model to beat a baseline inference without access to the model.

Many MI evaluation datasets for foundation models introduce distribution shifts between members and non-members, which allows for baseline attacks with non-trivial success.
In concurrent work, Duan et al. () and Maini et al. () identify a temporal shift between members and non-members in one dataset—WikiMIA Shi et al. ()—and demonstrate that some attacks fail when temporal shifts are removed.
Our work shows that this issue is much broader: virtually all evaluation sets proposed for membership inference on foundation models are flawed. We further show that
existing attacks do not just exploit distribution shifts, but they do so *sub-optimally* and are easily beat by blind baselines. These attacks thus perform “worse than chance”.

## 3 Blindly Inferring Membership

### 3.1 Distribution Shifts in MI Evaluation Datasets

MI evaluations measure the ability to distinguish a model’s training set members from non-members. A baseline attack (with no knowledge of the model) should do no better than random guessing—i.e., an AUC of 50% or equal TPR and FPR. For current MI evaluations sets for foundation models, this is not the case, due to intrinsic differences between members and non-members. We discuss the most common reasons for this discrepancy below.

##### Temporal shifts.

MI evaluation sets based on a hard data cutoff for some evolving Web source (e.g., Wikipedia or arXiv) introduce a temporal shift between members and non-members.
We can thus blindly distinguish members from non-members if we can answer a question of the form: “is this data from before or after 2023?”

##### Biases in data replication.

Even if we know how the training set was sampled, building an indistinguishable dataset of non-members is challenging Recht et al. ().
Slight variations in the procedures used to create the two datasets Engstrom et al. () could be exploited by a blind attack.

##### Distinguishable tails.

Some works filter and process the data to maximally align the member and non-member distributions (e.g., by matching linguistic characteristics).
However, distributions that are hard to distinguish *on average* may still be easy to distinguish for *data outliers*.

### 3.2 Blind Attack Techniques

In this section, we introduce simple “blind” attack techniques to distinguish members from non-members in MI evaluations.
The datasets we consider consist of either text, or images and text. For simplicity, we focus only on the text modality in both cases.
We do not aim for our blind attacks to be optimal. We prioritize simple and interpretable methods to show that existing evaluation datasets suffer from large distribution shifts.
Our blind attacks typically aim to achieve a high TPR at a low FPR (i.e., very confident predictions of membership for as many samples as possible).

##### Date detection.

Some text samples (e.g., from Wikipedia, arXiv, etc.) might contain specific dates.
Heuristically, it is unlikely that a text contains specific dates referencing the future. Thus, to place an upper-bound on the date at which a text was written, we simply extract all dates present in the text (using simple regular expressions).
We then predict that a sample is a member if all referenced dates fall before some threshold. Such an approach can have false positives when a text sample only references dates far in the past.

##### Bag-of-words classification.

Explicit dates are just one textual feature we can use. More generally, we can aim to predict membership from arbitrary words in the text. To this end, we train
a simple bag-of-words classifier. Here we have to guard against overfitting, since we can always find some word combinations that appear in members and not in non-members (e.g., the exact text of the member samples).
So we train our classifier on 80% of the members and non-members, and then evaluate the blind attack on the remaining 20%. As some of the evaluation datasets are small, we aggregate results over a 10-fold cross-validation.

##### Greedy rare word selection.

The above classifier should work well on average, but might not be optimal at low FPRs. Here, we take a simple greedy approach: we extract all n-grams (for $n\in[1,5]$) and sort them by their TPR-to-FPR ratio on part of the data (i.e., for each n-gram, we compute the fraction of members and non-members that contain this n-gram).
We then pick the n-gram with the best ratio, and repeat this procedure. Given a set of selected n-grams, we evaluate our attack on a held-out set by predicting that a sample is a member if it contains any of these n-grams.

## 4 Case Studies

**Table 2: Extended version of Table 1, with the results from our blind attacks compared to the best reported MI attack on each evaluation set. We report the same metrics as used in prior work, and add TPR at 1% FPR if no prior results are reported.**
| MI dataset | Metric | Best attack | Ours |
| --- | --- | --- | --- |
| *Temporal shifts* |  |  |  |
| WikiMIA | TPR@5%FPR | 43.2% | 94.4%^† |
| AUC ROC | 83.9% | 98.7%^† |  |
| BookMIA | TPR@5%FPR | - | 60.8%^† |
| AUC ROC | 88.0% | 90.5%^† |  |
| Temporal Wiki | TPR@1%FPR | - | 33.5%^‡ |
| AUC ROC | 79.6% | 79.9%^† |  |
| Temporal arXiv | TPR@1%FPR | - | 10.8%^‡ |
| AUC ROC | 74.5% | 75.6%^† |  |
| ArXiv-1 month | TPR@1%FPR | 5.9% | 13.4%^∗ |
| AUC ROC | 67.8% | 68.4%^† |  |
| *Biased replication* |  |  |  |
| Multi-Webdata | TPR@1%FPR | 40.3% | 83.5%^‡ |
| AUC ROC | 81.7% | 96.9%^† |  |
| LAION-MI | TPR@1%FPR | 2.5% | 9.9%^‡ |
| Gutenberg | TPR@1%FPR | 18.8% | 59.6%^‡ |
| AUC ROC | 85.6% | 96.4%^† |  |

We now study 8 membership inference datasets proposed for large language models and diffusion models. Table 2 summarizes our results: for each dataset, we create blind attacks that outperform existing MI attacks that have access to a trained model. We average the results from multiple runs and report for the same metric used in prior work, either AUC ROC or TPR at low FPR. Whenever possible, we use the exact datasets released by the authors to ensure that no biases are introduced. For datasets that are not publicly available (arXiv-1 month and Gutenberg Meeus et al. ()), we follow the specific collection steps outlined in the respective work to create similar datasets.

### 4.1 Temporal Shifts

#### 4.1.1 WikiMIA

##### The dataset and evaluation.

The WikiMIA dataset Shi et al. () selects members from Wikipedia event pages from before 01/01/2017 and non-members from after 01/01/2023.
It thus serves as an MI evaluation set for any LLM trained in between those dates.
The best reported prior MI attack on this evaluation set is from the
Min-K%++ method of Zhang et al. ().

##### Our attack.

We first apply our naive date detection attack. We extract all dates in the snippet and check if the latest one is from before 2023. Using this, we obtain a $52.3$% TPR at $5$% FPR. This already beats the state-of-the-art MI attack evaluated on this dataset (see Table 2).
To obtain an even stronger blind attack, we train a bag-of-words classifier on 80% of the dataset, and evaluate on the remaining 20%.
This classifier achieves a near perfect TPR of $94.4$%.

#### 4.1.2 BookMIA

##### The dataset and evaluation.

This dataset Shi et al. () is constructed from 512-token length text snippets from various books. Members are selected from books in the Books3 corpus that have been shown to be memorized by GPT-3. Non-members are taken from books that were first published in or after 2023. In the evaluation of Shi et al. (), their Min-K% method obtains the highest AUC against GPT-3.

##### Our attack.

We train a a bag-of-words classifier on 80% of the dataset, and evaluate on the remaining 20%.
As shown in Table 2, a bag-of-words classifier achieves a $90.5$% AUC ROC and beats state-of-the-art MI attacks against GPT-3.(^1^11To control for book- and author-specific features (such as character names), we repeat the experiment with no author appearing in both our classifier’s train and test sets. This still results in a significantly above chance AUC ROC of $80.3$%.)

#### 4.1.3 Temporal Wiki & arXiv

##### The datasets and evaluations.

Duan et al. () hypothesize that some MI attacks evaluated on Wiki-MIA may inadvertently exploit temporal shifts between members and non-members.
To showcase this, they create datasets that sample members from the Wikipedia and arXiv snippets in the Pile Gao et al. () and non-members from the same sources at a later date.

In more detail, in the Temporal arXiv Dataset, members are snippets from arXiv papers posted prior to July 2020 and contained in the Pile training data, while the non-members are sampled from arXiv papers from successively different time ranges after the Pile cutoff of July 2020. The Temporal Wikipedia dataset is also constructed based on the same principle as the WikiMIA dataset, but selects non-member articles from RealTimeData WikiText data created after August 2023 while members are from The Pile from before 03/01/2020.
The authors show that existing MI attacks applied to Pythia models improve as the temporal shift increases, with the MI attack of Carlini et al. () performing best.

##### Our attacks.

We show that even if existing attacks do rely on some temporal features in the Temporal Wikipedia and Temporal arXiv datasets, they do so sub-optimally. As for the prior datasets, we train a bag-of-words classifier which outperforms prior MI results on these datasets.
Specifically, we focus on the “2020-08” split (where non-members are taken from arXiv articles in August 2020, right after the Pile cutoff). There, our blind attack slightly outperforms the AUC and TPR at 1% FPR of the best reported MI attack on both datasets (see Table 2).

#### 4.1.4 One-month arXiv

##### The dataset and evaluation.

Meeus et al. () also note that naively re-collecting data from arXiv causes a large temporal shift.
Instead, they thus pick the non-members as close after the model’s cutoff date as possible. They build an MI evaluation set by taking all arXiv articles from the RedPajama dataset Together () as members, and all articles from March 2023 (right after the dataset’s cutoff date) as non-members. In contrast to the Temporal arXiv dataset we looked at previously, this dataset uses full arXiv articles rather than just snippets.
The authors propose a new MI attack that relies on textual feature extractors, and evaluate it on the OpenLLaMA model Geng & Liu () which was trained on RedPajama.

##### Our attack.

As this dataset is not public, we replicate a similar setup by taking all articles before March 2023 in RedPajama as the members, and all articles from March 2023 as the non-members.
The issue with this dataset is that the distributions of members and non-members are still incredibly easy to distinguish at one end: papers that are *much older* than the cutoff date are guaranteed to be members.
And since this dataset uses the full LaTex body of each article, determining a paper’s approximate date is very easy: we just look at the paper’s citations.
We use a regex to find all \cite commands, and extract the year in the citation keyword if it exists. We guess that a paper is a member if it only cites papers from before 2022. This trivial baseline yields 13.4% TPR without any false positives, over twice as high as the TPR at 1% FPR obtained by the best MI attack in Meeus et al. ().

### 4.2 Biases in Data Replication and Distinguishable Tails

#### 4.2.1 Multi-Webdata

##### The dataset and evaluation.

To evaluate MI attacks on diffusion models, Ko et al. ()
collect multiple web-scale datasets of captioned images: CC3M Sharma et al. (), CC12M Changpinyo et al. (), (LAION Schuhmann et al. () and MS-COCO Lin et al. ()).
They argue that since these datasets are sampled from a common global source (i.e., captioned images from the Web), they can serve as a natural evaluation set for MI attacks.
Specifically, the authors train a diffusion model on CC12M (the members), and sample the non-members from the union of the other three datasets.(^2^22Wu et al. () use a similar approach, and choose members and non-members from different text-vision datasets.)
Ko et al. () evaluate the performance of three newly proposed MI attacks on target models from the ViT Dosovitskiy et al. () and ResNet He et al. () families trained on CC12M.

##### Our attack.

For simplicity, we focus only on the image captions and ignore the visual content. We train a bag-of-words binary classifier on the captions, achieving a TPR of $75.7\%$ at a $1$% FPR, far superior to the best MI attack methods evaluated on these datasets (see Table 2).
It is likely that incorporating visual features into the classifier could lead to an even stronger blind attack.

#### 4.2.2 LAION-MI

##### The dataset.

Dubiński et al. () explicitly try to control for and minimize biases in data replication. However, their techniques still leave distinguishable tails.
To create an evaluation dataset for a model trained on LAION Schuhmann et al. (), they rely on the multilingual-LAION dataset Schuhmann et al. () to sample non-members. This dataset was sampled from the Web in a similar way to LAION, except that the captions are not in English and the images were not filtered for quality.

To align the two distributions, the authors first apply the same selection process that was used to select images in LAION: (1) they discard all images from multilingual LAION with an “aesthetics score” below some threshold; (2) they translate the non-member captions to English, using Facebook’s M2M100 1.2B They then train a distinguisher on top of image captions, and select a subset of the data where members and non-members are hard to distinguish *on average*. The authors use a PCA visualization to confirm that the two distributions closely match.

The authors evaluate multiple MI attacks in white-box, grey-box and black-box settings, obtaining a best score of $2.5$% TPR@1%FPR on their LAION-MI dataset.

##### Our attack.

We posit that *outlier* members would still be easy to distinguish from non-members.
To provide some intuition, we started by looking for individual *characters* that appear much more frequently in members than in non-members. We show the five characters with the highest distinguishing power in Table 3.

We note that there are non-English characters (Russian ‘о’ and ‘т’) that appear predominantly in *members*.
This is likely because the original LAION captions were selected on the basis of being *primarily* in English. In contrast, since the captions from the non-members were output by a translation model, they are unlikely to contain non-English characters.
The presence of special characters such as \xa0 (a non-breaking space) are similarly due to biases in translation: this character is not part of the translation model’s output vocabulary.

**Table 3: Characters with the highest distinguishing power among members and non-members for LAION-MI.**
| Character | Members | Non-members |
| --- | --- | --- |
| | | 0.60% | 0.00% |
| \xa0 | 0.30% | 0.00% |
| … | 0.21% | 0.00% |
| о | 0.23% | 0.03% |
| т | 0.15% | 0.03% |

To boost the power of this approach, we apply our greedy selection algorithms to all n-grams of up to 5 characters. We use 80% of the data to pick the n-grams with the highest TPR-to-FPR ratio until we hit a $1\%$ FPR. At that point, we obtain a $9.9$% TPR on the evaluation set—surpassing the best MI attack.

#### 4.2.3 Project Gutenberg

##### The dataset and evaluation.

This dataset Meeus et al. () consists of books from Project Gutenberg.
The members are books contained in the RedPajama dataset Together (), more specifically
the set of “PG-19” books collected by Rae et al. ()
which were all originally published before 1919.
The non-members consist of books added to the Project Gutenberg repository after the creation of the PG-19 corpus in February 2019.
To mitigate the obvious distribution shift in the publication years of members and non-members, Meeus et al. () filter both the members and non-members to only contain books published between 1850 and 1910 (i.e., members are books published between 1850 and 1910, which were added to Project Gutenberg before February 2019. Non-members are books that were originally published in the same period, but added to Project Gutenberg after February 2019).
The authors evaluate a new MI attack (see Section 4.1.4 for details) against the OpenLLaMA model Geng & Liu () trained on RedPajama.

##### Our attack.

While the publication dates of the members and non-members are similarly distributed, we find that *the date on which a book is added to Project Gutenberg* still introduces a noticeable distribution shift. Indeed, it appears that the format of the preface metadata that Project Gutenberg adds to books has changed at some point between the collection of PG-19 and the non-member corpus (e.g., the link to the HTML version of a book used to have a .htm file extension, and this was changed to a .html extension for books added after a certain date).
By exploiting such small discrepancies, our greedy n-gram attack applied to the first 1,000 characters of a book achieves 59.6% TPR at a 1% FPR, 3 times higher than the best MI attack evaluated on this dataset.

Of course, it may be possible to preprocess this particular dataset to get rid of these formatting discrepancies. Nevertheless, we believe our results illustrate how brittle the process of re-collecting an identically distributed non-member set can be.
Moreover, even if we ignore any formatting information, and apply our attack directly to the book text, we can still distinguish members and non-members far better than chance, with a TPR of 16.6% at a 1% FPR.

## 5 A Path Forward: the Pile, DataComp and DataComp-LM

As we have seen, building MI evaluation datasets *a posteriori* is incredibly challenging, due to the many possibilities for distribution shift.
It may be possible to apply more rigorous filtering techniques to minimize these shifts, so that even strong blind attacks cannot reliably distinguish members and non-members. Any MI attack’s performance should then be compared against these strong blind baselines.

An alternative avenue is to forego MI evaluations on arbitrary foundation models, and focus on those models for which identically distributed sets of members and non-members are available.
A prime example is the Pile Gao et al. (), which has an official test set.
Models trained on the Pile training set—e.g., Pythia Biderman et al. () or GPT-NeoX-20B citeblack2022gpt—are thus a popular target for MI attack evaluations (see e.g., Duan et al. (); Maini et al. (); Li et al. ()).
Unfortunately, the Pile only contains text, and is too small to train state-of-the-art language models.

A more recent and broadly applicable alternative is to use models trained on the DataComp Gadre et al. () and DataComp-LM Li et al. () benchmarks.
These benchmarks contribute multiple models (ViTs and LLMs) trained on data *sampled randomly* from a larger dataset.
More precisely, these benchmarks introduce two datasets, CommonPool and DCLM-POOL, which contain respectively 12.8 billion image-text pairs and 240 trillion text tokens sampled from Common Crawl.
To enable experiments at smaller budgets, the authors produce smaller data pools sampled randomly from the full pool.
They then train a variety of models on each pool (see Table 4).(^3^33[https://github.com/mlfoundations/datacomp](https://github.com/mlfoundations/datacomp))^,(^4^44[https://github.com/mlfoundations/dclm](https://github.com/mlfoundations/dclm).)

A minor complication is that most models in the DataComp and DataComp-LM benchmarks are not trained on the entirety of a pool.
Rather, the benchmarks encourage models to be trained on *filtered* datasets: given a pool of data $\mathcal{P}$ and a binary filter $f:\mathcal{X}\mapsto\{0,1\}$, models are trained on the filtered pool $\{x\in\mathcal{P}\mid f(x)\}$.
This training set is thus no longer a random subset of the full dataset. But this is easily remedied by applying exactly the same filter $f$ to select the non-members.

We encourage future work on MI attacks for foundation models to evaluate their methods on these models and datasets.

**Table 4: Models trained on datasets with *random* train-test splits for DataComp Gadre et al. () and DataComp-LM Li et al. (). For each dataset pool, we report the pool’s absolute size (in image-text pairs, respectively tokens) and its relative size compared to the global pool that it was randomly sampled from. Gadre et al. () and Li et al. () have pretrained ViTs and LLMs on each pool size.**
| Modality | Pool Name | Pool Size | Models |
| --- | --- | --- | --- |
| Vision+Text | CommonPool Large | 1.280B. (10%) | ViT-B/16 |
| CommonPool Medium | 0.128B0. (1%) | ViT-B/32 |  |
| CommonPool Small | 0.013B (0.1%) | ViT-B/32 |  |
| Text | DCLM 400M-1x | 0.47T (0.2%) | 412M LLM |
| DCLM 1B-1x | 1.64T (0.2%) | 1.4B LLM |  |
| DCLM 1B-5x | 8.20T (3.4%) | 1.4B LLM |  |
| DCLM 7B-1x | 7.85T (3.3%) | 6.9B LLM |  |
| DCLM 7B-5x | 15.7T (6.5%) | 6.9B LLM |  |

As part of their experiments, Gadre et al. () and Li et al. () trained CLIP models of various sizes and a 1B parameter language model on *random* subsets of these datasets. These models and datasets might thus form an ideal testbed for MI evaluations on web-scale text and vision models.

Future web-scale dataset and model developers could similarly release an official IID split to enable rigorous MI evaluations for a wider variety of models and modalities.

## 6 Conclusion

We have shown that current evaluations of MI attacks for foundation models cannot be trusted, as members and non-members can be reliably distinguished by simple blind attacks with no knowledge of the model.
State-of-the-art MI attacks may thus be ineffective at extracting any actual membership information, and cannot be relied on for applications such as copyright detection or auditing unlearning methods.
The MI evaluation datasets we investigated should likely be discarded due to the large distribution shifts between members and non-members, and replaced by
datasets with a random train-test split—e.g., the Pile or DataComp.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 5
presentation: 4
questions: - Some of the "blind" baselines rely on detecting data references like 2024 etc. As an adversary cognizant of the training cutoff (or even an auditor), a simple solution to fix this shift would be arbitrarily replacing all such data references for non-members, maybe shift them by N nears back to match the cutoff range of the model. What happens in such a scenario? Can "blind" attacks still be successful?

- In Table 2, why are some of the entries missing values, or entire metrics (AUC ROC) not reported? These datasets are publicly available (which is how the authors get their attack's results to begin with) so I do not see why corresponding values cannot be filled in for existing works.

- I am confused by the 'Greedy rare word selection' protocol- is the sorting done using the TPR/FPR ratios on test data? If so, one can always design and selective use metrics to get good performance on test data if you are using metrics from test data to begin with. Please explain this part a bit more clearly.

- L240: "...and thus do not include it" - should be fairly simple inclusion and I do not see why it must be excluded like this? Also, as a reader I do not know what "their construction is" - if it is so similar, please briefly explain how they do it.
soundness: 4
strengths: - While this topic has been explored by several recent works (both concurrent and prior), this work goes a step beyond to demonstrate the extend of distributional differences between members and non-members, for both LLM and VLM evaluation data for membership inference. 

- The paper is well written and supports most of its claims with empirical evidence and extensive evaluation.
summary: This paper demonstrates that existing MI evaluations for foundation models perform poorly due to distribution shifts between member and non-member data. The authors show that simple "blind" attacks, which do not query the model, can outperform state-of-the-art MI attacks on common MI evaluation datasets. They identify temporal shifts, biases in data replication, and distinguishable tails as common causes for the distribution mismatch between members and non-members.
weaknesses: The submission has significant issues regarding originality and the characterization of related work. The authors' framing of certain works as "concurrent" appears to minimize substantial overlaps, particularly with [1] and [2] which preceded the ICLR deadline by 4 and 8 months respectively. This timeframe makes it difficult to justify as concurrent research. The paper's main conclusion about flawed non-member selection methods introducing detectable distributional shifts largely mirrors the findings already established in [2].

The conclusion (L413) takes a problematic stance by seemingly absolving model trainers of accountability. Instead of abandoning membership inference evaluations, research should focus on developing methods that either avoid non-member requirements (like data-extraction attacks) or leverage trainer-provided evaluation data. Dismissing these evaluations would encourage (proprietary) model trainers to evade scrutiny of their data usage practices.

The claim on L464-465 about "no knowledge of the model" requires clarification. The paper's "blind" baselines actually incorporate significant domain knowledge about data collection patterns (e.g., dates and special tokens). The authors should explicitly state that "blind" specifically refers to lack of model access, as the attacks still utilize direct data knowledge and split information to construct rules and meta-classifiers.

## Other Comments

- L89: Current state of the art is RMIA [3], nor LiRA. 

- L89: Please provide some more context for the 'standard' membership inference game (member and non-members should be same distribution etc.) Context is especially important for this work to understand the nuances behind different member/non-member data distributions. 

- L101: "Many of these" - Please be exact. Consider adding a table that talks about MI attack attempts on foundation models, the benchmarks they use (propose or new), and whether they suffer from the train-test split issues that the authors mention here.

- L154:  The assumption about future dates is oversimplified and overlooks legitimate cases in fiction, climate research, and policy documents

- Table 1 appears redundant given Table 2's more comprehensive presentation

- Table 2: Please make a distinction between datasets for LLMs and those for VLMs.

- L467: "... auditing unlearning methods" - there are several works describing better ways to audit unlearning [4]. It should also be pointed out that these membership-inference attacks do not work well to begin with even with properly split train/test data [2], so it is not surprising that it will not be used to audit unlearning.

#### References

- [1] Maini, Pratyush, et al. "LLM Dataset Inference: Did you train on my dataset?." arXiv:2406.06443 (2024).
- [2] Duan, Michael, et al. "Do membership inference attacks work on large language models?."COLM, 2024
- [3] Zarifzadeh, Sajjad, Philippe Liu, and Reza Shokri. "Low-Cost High-Power Membership Inference Attacks." ICML, 2024.
- [4] Lynch, Aengus, et al. "Eight methods to evaluate robust unlearning in llms." arXiv:2402.16835 (2024).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - How do state-of-the-art attacks perform relative to blind attacks on unbiased datasets?

- Are the recommended datasets (PILE, DataComp, or DataComp-LM) genuinely unbiased?

- What are the effects of repeating experiments using different dataset splits on the evaluation outcomes?
soundness: 2
strengths: - The paper investigates a significant issue in MIA research, highlighting the importance of unbiased evaluation datasets for accurately benchmarking attack effectiveness on text-based large foundation models.

- It provides a systematic evaluation of various datasets and baseline attacks, identifying three common distribution shift patterns that influence the success of MIAs.
summary: This paper examines the datasets used in evaluating membership inference attacks on large language models and text-to-image generation models. The authors argue that current MIA evaluations are unreliable, as it is possible to differentiate members from non-members through blind attacks that do not utilize any information about the target model. Consequently, they suggest that state-of-the-art MIAs may not actually extract membership information effectively. To improve evaluation, the authors recommend using datasets with minimal distribution shifts between members and non-members, such as Pile or DataComp.
weaknesses: - The authors claim that current state-of-the-art MIAs fail to extract meaningful membership information, relying only on biased dataset evaluation results. However, this assertion may be overstated, as blind attacks use dataset-specific prior information (e.g., timestamps), which the proposed state-of-the-art attacks may intentionally avoid as they may aim to propose a general attack. These attacks might still capture useful membership signals, albeit weaker than the dataset-specific prior information. To better support this claim, experiments on less biased datasets (like Pile or DataComp, as suggested) are necessary. If state-of-the-art methods perform close to random guessing on such datasets, it would indicate their inability to capture membership information effectively.

- As a path forward, the paper advocates for future MIA evaluations using PILE, DataComp, or DataComp-LM. However, it is unclear whether these datasets also suffer from distribution shift issues. A simple approach to evaluate this would be to apply the proposed blind attacks on these datasets; if the success rate is near random guessing, it could indicate that these datasets are indeed less biased by distribution shifts, at least concerning the three identified types of shift.

- As highlighted by Dubinski et al. (2024), different splits of training and evaluation sets can yield significantly varied membership inference attack results. To ensure robustness of the evaluations, it would be beneficial to repeat the experiments with different random dataset splits, recording the mean and variance of attack success rates. This approach would provide a more reliable comparison between blind attacks and existing MIAs.

- The novelty and technical contributions of this paper appear incremental. Distribution shift issues in evaluation datasets have been previously discussed by Duan et al. and Maini et al., and while I appreciate the systematic evaluations in this paper, it largely provides a measurement study rather than new technical contributions or insights. Thus, the paper might lack the innovation typically expected at top-tier conferences, just my two cents.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Q1: In Section 3, the authors first extract all dates present in the text and then proceed with date detection attacks. Are samples lacking dates excluded from your inference attacks? Specifically, in the experimental section, are the 'Ours' attack results shown in Table 2 based on a dataset that has filtered out samples without dates? Similarly, are the 'Best Attack' results measured on such a filtered dataset, or are they directly taken from the original papers? This distinction is crucial as it determines the fairness of your comparison experiments.

Q2: In your date detection attack, could you elaborate on how the date threshold is selected? The relevant section does not detail the method for choosing the threshold. Is it a matter of testing various thresholds and selecting the one that yields the best attack performance?

Q3: Regarding your Bag-of-words classification attack, what is the underlying insight or motivation for this approach? Why can different word combinations be used to infer membership properties, particularly without any interaction with the foundational models?

Q4: This paper primarily concentrates on the datasets used for MI evaluation, but it does not account for the influence of foundational models. However, the objective of MI attacks is to discern differences between a model's behavior on members and non-members. Even with datasets that are easily distinguishable based on dates or words, existing attacks still fail to achieve satisfactory results after considering the foundational models. Does this suggest that the current use of these datasets remains valid? Furthermore, by focusing on direct distinctions in membership labels and disregarding interactions with foundational models, are the signals you use for differentiation, such as dates and words, being utilized by existing MI attacks that do interact with foundational models? If current MI attacks are not leveraging your information, does it further indicate that the use of these datasets is reasonable (at least at current stage)?

Q5: In Section 3.1, this paper proposes three common reasons for the intrinsic differences between member and non-member samples. However, since different MI evaluation datasets are constructed using various strategies, it is unclear which aspects are evaluated with respect to these datasets. It would be beneficial to clarify this point in Table 2.
soundness: 3
strengths: - This paper focuses on the irrationality of MI evaluation datasets is important, especially in an era where foundation models are widely applied.
- This paper analyzes 9 published MI evaluation datasets, demonstrating that blind attacks outperform existing MI attacks on these datasets. This reveals the incompleteness of current MI evaluations.
- The attack methods proposed in this paper perform exceptionally well, showing significant performance improvements compared to existing MI attacks.
summary: This paper reveals that current evaluations of membership inference (MI) attacks on foundational models are flawed due to the use of different distributions when sampling members and non-members. The authors demonstrate this issue through an analysis of nine published MI evaluation datasets. They show that directly classify the samples in the MI evaluation datasets can outperform existing MI attacks. This finding indicates that current evaluation methods cannot accurately reflect the membership leakage of a foundational model's training data. This paper also proposes simple blind attack techniques, such as date detection and bag-of-words classifiers, which remain effective on datasets designed to eliminate distribution differences between members and non-members. The authors suggest that future MI attack evaluations should be conducted on models with a clear train-test split.
weaknesses: - The comparison experiment setup is unclear. Were the same data conditions used the experiment section? (see Q1)
- The core of this paper is to point out the shortages of existing MI attacks on foundation models. However, in the introduction, the discussion does not revolve around this point but rather focuses on how simple attacks can also achieve good results. It is recommended to revise the structure of the introduction to highlight the main contributions of the paper.
- The experimental section is divided into sections based on the datasets, which makes it difficult to correspond with the previously mentioned common reasons for the intrinsic differences. This hinders the reader's understanding of the experiments and the paper's arguments.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Duan et al. (2024) propose that temporal shifts can influence the performance of membership inference attacks. While you mention that there are differences between your paper and this concurrent work, from my perspective, both papers seem to demonstrate similar findings. Could you elaborate further on the differences between your work and Duan et al. [2024]?

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024.
soundness: 2
strengths: 1. The paper provides extensive experimental results to support their claims. These results demonstrate that blind attack techniques outperform state-of-the-art methods under their settings.
summary: The paper argues that previous evaluations of membership inference attacks are flawed due to the distributional differences between members and non-members. The paper analyzes nine datasets and demonstrates that blind attack techniques, such as date detection (assume some text samples contain dates), bag-of-words classification (classifier is trained on 80% of the members and non-members, then test on the left 20% members), and greedy rare word selection, outperform previous state-of-the-art (SOTA) methods in the evaluation metric.
weaknesses: 1. The assumption of this paper on "blind" is not correct. "Blind" should be on both the model and data set [2021], but this paper relies on too much target dataset information. For example, one of the proposed methods only works if the dataset contains data information, and another method even needs 80% of labeled member data as the attacker's training samples. From my understanding of the literature, this rich information may not be available to other membership inference attacks, potentially giving the proposed blind attack an unfair advantage.

Hui, Bo, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. "Practical blind membership inference attack via differential comparisons.", 2021

2. The paper provides limited experimental details. For instance, it does not specify which models were targeted for the membership inference attacks.

3. The paper proposes ideas for constructing better datasets for evaluating membership inference attacks, but it does not provide experimental results or analysis on whether the blind attack would still outperform SOTA methods on these improved datasets.

4. Current membership inference attacks are typically evaluated across multiple datasets. For example, Zhang et al. [2024a] evaluate their Min-K%++ attack on Wikipedia, GitHub, Pile CC, PubMed Central, and many other datasets to demonstrate generalizability. However, the blind attack’s performance on other datasets is not explored in the paper, making it difficult to conclude that current evaluations are entirely flawed based on the results from just one dataset.

Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. Min-K%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936, 2024a.

Meta-review: The authors evaluate membership inference attacks against foundation models, and find that existing attacks are ineffective for determining the membership of a given sample. In particular, the author find that a blind baseline that distinguishes between member and non-member distributions achieves higher success rate compared to existing attacks.

Reviewers generally found the message of the paper to be important and timely. However, there exist prior and concurrent work that made similar discoveries, and while the paper's message is impactful, the paper currently lacks depth and would greatly benefit from designing a practical solution. AC agrees with the reviewers and recommend rejection, but encourage the authors to improve the paper's technical depth and resubmit it to a future venue.

Decision: Reject