Detecting Training Data of Large Language Models via Expectation Maximization

## Abstract

Abstract The widespread deployment of large language models (LLMs) has led to impressive advancements, yet information about their training data, a critical factor in their performance, remains undisclosed.
Membership inference attacks (MIAs) aim to determine whether a specific instance was part of a target model’s training data.
MIAs can offer insights into LLM outputs and help detect and address concerns such as data contamination and compliance with privacy and copyright standards.
However, applying MIAs to LLMs presents unique challenges due to the massive scale of pre-training data and the ambiguous nature of membership.
Additionally, creating appropriate benchmarks to evaluate MIA methods is not straightforward, as training and test data distributions are often unknown.
In this paper, we introduce EM-MIA, a novel MIA method for LLMs that iteratively refines membership scores and prefix scores via an expectation-maximization algorithm, leveraging the duality that the estimates of these scores can be improved by each other.
Membership scores and prefix scores assess how each instance is likely to be a member and discriminative as a prefix, respectively.
Our method achieves state-of-the-art results on the WikiMIA dataset.
To further evaluate EM-MIA, we present OLMoMIA, a benchmark built from OLMo resources, which allows us to control the difficulty of MIA tasks with varying degrees of overlap between training and test data distributions.
We believe that EM-MIA serves as a robust MIA method for LLMs and that OLMoMIA provides a valuable resource for comprehensively evaluating MIA approaches, thereby driving future research in this critical area.

## 1 Introduction

Large language models (LLMs) (Brown et al., ; Touvron et al., ) have recently emerged as a groundbreaking development and have had a transformative impact in many fields.
The vast and diverse training data are central to their success, which enables LLMs to understand and generate languages to perform complex tasks.
Given that training data directly shape LLMs’ behaviors, knowing the composition of training data allows researchers and practitioners to assess the strengths and limitations of LLMs (Gebru et al., ), address ethical concerns, and mitigate potential biases (Bender et al., ; Feng et al., ) and other risks (Bommasani et al., ).
However, the exact composition of training data is often a secret ingredient of LLMs.

Since LLMs are trained on large-scale corpora from diverse sources, including web content, the inclusion of undesirable data into training data such as test datasets (Sainz et al., ; Oren et al., ; Sainz et al., ), proprietary contents (Chang et al., ; Meeus et al., ), or personally identifiable information (Mozes et al., ) might prevalently happen unconsciously, raising serious concerns when deploying LLMs.
Membership inference attack (MIA) determines whether a particular data point has been seen during training a target model (Shokri et al., ).
Using MIA to uncover those potential instances can serve as an effective mechanism in detecting data contamination (Magar & Schwartz, ) for the reliable evaluation of LLMs’ ability (Zhou et al., ) and auditing copyright infringement (Duarte et al., ) and privacy leakage (Staab et al., ; Kim et al., ) to ensure compliance with regulations such as GDPR (Voigt & Von dem Bussche, ) and CCPA (Legislature, ). Therefore, MIA has gained huge interest in the LLM community.

Despite increasing demands, MIA for LLMs is challenging (Duan et al., ) largely due to the large scale of training data and intrinsic ambiguity of membership from the nature of languages.
Furthermore, designing a proper evaluation benchmark on MIAs for LLMs that emulates a realistic test case scenario is complicated. While training data distribution is a mixture of diverse sources and unknown, target test data at inference time could be from any distribution.
This circumstance motivates us to develop a robust MIA method that can work well across different conditions on the distribution of members and non-members with minimal benchmark information.

In this paper, we propose a novel MIA framework, EM-MIA, which iteratively refines membership scores and prefix scores using an expectation-maximization algorithm.
A membership score indicates how likely each data point is to be a member.
A prefix score indicates how discriminative each data point is in distinguishing members and non-members when used as a prefix.
We empirically observe a duality between these scores, where better estimates of one can improve estimates of another.
By starting from a reasonable initialization, our iterative approach progressively enhances score prediction until convergence, leading to more accurate membership scores for MIA.

To comprehensively evaluate our method and different MIA approaches, we introduce a new benchmark called OLMoMIA by utilizing OLMo (Groeneveld et al., ) resources.
We vary degrees of distribution overlaps and control difficulty levels based on clustering.
Throughout the extensive experiments, we have shown that EM-MIA is a versatile MIA method and significantly outperforms previous strong baselines, though all methods including ours still struggle to surpass random guessing in the most challenging random split setting.

Our novelty and main contributions are summarized as follows:

- •
To the best of our knowledge, EM-MIA is the first approach that progressively enhances membership scores (and prefix scores) with an iterative refinement for improved MIA for LLMs.
- •
We demonstrate that ReCaLL (Xie et al., ) has an over-reliance on prefix selection. In contrast, we design EM-MIA without requiring any labeled data or prior information on benchmarks.
- •
EM-MIA remarkably outperforms all existing strong MIA baselines for LLMs, and achieves state-of-the-art results on WikiMIA (Shi et al., ), the most popular benchmark for detecting pre-training data of LLMs.
- •
Experiments on our OLMoMIA benchmark with varying degrees of overlap between member and non-member distributions shows the robustness of EM-MIA and the utility of evaluating on diverse conditions when developing a new MIA method.

## 2 Background

### 2.1 Membership Inference Attack for LLMs

Membership inference attack (MIA) (Shokri et al., ; Carlini et al., ) is a binary classification task that identifies whether or not a given data point has been seen during model training: member vs non-member.
Given a target language model $\mathcal{M}$ trained on an unknown $\mathcal{D_{\mathrm{train}}}$, MIA predicts a membership label of each instance $x$ in a test dataset $\mathcal{D_{\mathrm{test}}}$ whether $x$ in $\mathcal{D_{\mathrm{train}}}$ or not, by computing a membership score $f(x;\mathcal{M})$ and thresholding this score.
By adjusting a threshold, MIA can control the trade-off between the true positive rate and the false positive rate.
The MIA performance is typically evaluated with two metrics: AUC-ROC and TPR@low FPR (Carlini et al., ; Mireshghallah et al., ).

Most existing MIA methods are based on the assumption that the target model memorizes (or overfits) training data.
In general, members will have a lower loss (Yeom et al., ), which is the average log-likelihood (or perplexity) of target text with respect to a target LLM, compared to non-members.
Likelihood Ratio Attacks (LiRAs) (Ye et al., ) perform difficulty calibration using a reference model (Carlini et al., ), a compression method (Carlini et al., ), or the average loss from neighbors (Mattern et al., ).
Min-K% (Shi et al., ) calculates the average log-likelihood for tokens with the lowest likelihoods.
Min-K%++ (Zhang et al., ) improves Min-K% by normalizing each token’s log probability.
ReCaLL (Xie et al., ) uses the ratio of the conditional log-likelihood to the unconditional log-likelihood as the membership score.
ReCaLL is described in detail in §2.4.

### 2.2 Why MIA for LLMs is Challenging

Although MIA has been studied in machine learning for several years, it remains especially challenging for LLMs (Duan et al., ; Meeus et al., ).
Due to the massive training dataset size, each instance is used in training only a few times, often just once (Lee et al., ), making it difficult to leave a footprint on the model.
Moreover, there is inherent ambiguity in the definition of membership because texts are often repeated and partially overlap each other in the original form or with a minor difference even after the rigorous preprocessing of decontamination and deduplication (Kandpal et al., ; Tirumala et al., ).
The membership boundary becomes even fuzzier (Shilov et al., ) if semantically similar paraphrases (Mattern et al., ; Mozaffari & Marathe, ) beyond lexical matching based on n-gram are considered.
Traditional MIA approaches in machine learning literature (Shokri et al., ; Ye et al., ; Carlini et al., ) based on training shadow models on non-overlapping data from the same data distribution, model architecture, and training algorithm as the target model are infeasible for LLMs considering high computational costs and unknown training specifications.

### 2.3 Evaluation of MIAs for LLMs

Common MIA benchmarks such as WikiMIA (Shi et al., ) use a time cutoff based on model release dates and time information of documents (Shi et al., ; Meeus et al., ) to ensure somewhat membership labels.
Besides, recent studies (Duan et al., ; Das et al., ; Meeus et al., ) question whether several MIAs that perform well on these benchmarks are truly conducting membership inference, as detecting distributional (often temporal) shifts alone may be sufficient to achieve high benchmark performance. Thus, they advocate for the use of datasets with a random train-test split such as MIMIR (Duan et al., ), which is derived from the PILE dataset (Gao et al., ), for MIA evaluation.
However, none of the existing methods significantly outperforms random guessing in this setting.
Although this setting seems theoretically appropriate for evaluating MIA, there is no truly held-out in-distribution dataset in reality because LLMs are usually trained with all available data sources.
In other words, it is difficult to find in-distribution non-member examples, and it is nearly impossible to completely eliminate the distribution shift between training and test data at inference time.

Once an MIA benchmark, a pair of models and datasets, is released, it becomes difficult to prevent MIAs from exploiting information about the benchmark settings, such as how the dataset has been constructed.
MIAs may exploit existing biases on datasets regardless of their intended purpose.
To evaluate the true generalizability of MIAs in real-world scenarios, we should consider situations where we have minimal knowledge about the target language model, training data, and test data.
Several ongoing attempts (Meeus et al., ; Eichler et al., ) aim to reproduce setups that closely resemble practical MIA scenarios, but none are sufficiently effective to gain widespread adoption in the community.
Our work is an additional effort in this direction. We propose a robust MIA method and evaluate it on our benchmark, which simulates various combinations of training and test data distribution.

### 2.4 ReCaLL: Assumptions and Limitations

ReCaLL (Xie et al., ) uses the ratio between the conditional log-likelihood of a target data point $x$ given a non-member prefix $p$ as a context and the unconditional log-likelihood of $x$ by an LLM $\mathcal{M}$ as a membership score, based on the observation that the distribution of ReCaLL scores for members and non-members diverges when $p$ is a non-member prefix: formally, $\operatorname{ReCaLL}_{p}(x;\mathcal{M})=\frac{\mathrm{LL}(x|p;\mathcal{M})}{
\mathrm{LL}(x;\mathcal{M})}$ (we may omit $\mathcal{M}$ later, for brevity), where $\mathrm{LL}$ is the average log-likelihood over tokens and the prefix $p$(^1^11As an extension from using a single prefix $p$, averaging the ReCaLL scores on a set of multiple prefixes is possible for an ensemble.) is a concatenation of non-member data points $p_{i}$: $p=p_{1}\oplus p_{2}\oplus\cdots\oplus p_{n}$.

ReCaLL significantly outperforms other MIA approaches by a large margin.
For instance, ReCaLL exceeds 90% AUC-ROC on WikiMIA (Shi et al., ) beating previous state-of-the-art AUC-ROC of about 75% from Min-K%++ (Zhang et al., ).
However, there is no theoretical analysis of why and when ReCaLL works well.

In Xie et al. (), non-members $p_{i}$ are randomly selected among non-members in a test dataset and excluded from the test dataset without validation, based on strong assumptions that (1) the ground truth non-members are available and (2) all of them are equally effective as a prefix.
However, the held-out ground truth non-members, especially from the same distribution of the test set, may not always be available at inference time.
This aligns with the discussion in the previous section §2.3.
Indeed, finding non-members could be difficult (Villalobos et al., ; Muennighoff et al., ) since training data of LLMs are ever-growing by gathering all crawlable data.
The solution to secure non-members in Xie et al. () is generating a synthetic prefix using GPT-4o.
However, according to their implementation, this method still relies on non-member test data as seed data.

Selecting non-members from the test dataset makes ReCaLL preferable and unfair compared to other MIAs that do not utilize any non-member test data.
It partially explains why simple random selection reasonably works well.
The ablation study from Xie et al. () to demonstrate the robustness of the random prefix selection ironically reveals that ReCaLL’s performance can be damaged when we do not have known non-member data points from the test dataset distribution.
Using non-members from a different domain (e.g., GitHub vs. Wikipedia) significantly degrades ReCaLL’s performance, sometimes even worse than Min-K%++.
In other words, ReCaLL would not generalize well to test data from a distribution different from that of the prefix.
The similarity between the prefix and test data also matters.
Another ablation study shows a variance between different random selections, implying that random prefix selection is not consistent and that all data points are not equally effective for the prefix.

## 3 Observation: Finding a Better Prefix

In this section, we rigorously investigate how much ReCaLL’s performance is sensitive to the choice of a prefix and particularly how much it can be compromised without given non-member data.
We define a prefix score $r(p)$ as the effectiveness of $p$ as a prefix in discriminating memberships, particularly when using this prefix for ReCaLL.
In the Oracle setting where ground truth labels of all examples in a test dataset $\mathcal{D_{\mathrm{test}}}$ are available, we can calculate a prefix score by measuring the performance of ReCaLL with a prefix $p$ on a test dataset $\mathcal{D_{\mathrm{test}}}$ using ground truth labels and a MIA evaluation metric such as AUC-ROC.
While a prefix could be any text, we calculate prefix scores for all examples in a test dataset $\mathcal{D_{\mathrm{test}}}$ by using each data point as a standalone prefix.

As an initial analysis, we conduct experiments using the WikiMIA (Shi et al., ) dataset with a length of 128 as a target dataset and Pythia-6.9B (Biderman et al., ) as a target LM.
Figure 1(a) displays the distribution of prefix scores measured by AUC-ROC for members and non-members.
Consistent with the results from Xie et al. (), ReCaLL works well if a prefix is a non-member and does not work well if a prefix is a member.
Prefix scores of members are smaller than 0.7, and most of them are close to 0.5, which is the score of random guessing.
Prefix scores of non-members are larger than 0.5, and most of them are larger than 0.7.
This clear distinguishability suggests using a negative prefix score as a membership score.

Figure 1(b) displays ROC curves of MIA when negative prefix scores measured by different metrics are used as membership scores.
We use AUC-ROC, Accuracy, and TRP@$k$%FPR with $k\in\{0.1,1,5,10,20\}$ as metrics.
Using AUC-ROC-based prefix scores as membership scores achieves 98.6% AUC-ROC, which is almost perfect and the highest among other metrics.

Figure: (a)
Refer to caption: extracted/5908709/figs/prefix_scores.png

Without access to non-members (or data points with high prefix scores), ReCaLL’s performance could be significantly lower.
Given the wide spectrum of prefix scores for even non-members, the effectiveness of each data point varies, and the choice of data points for a prefix can be crucial, although using a concatenation of multiple data points as a prefix reduces variance.
In other words, we can expect better MIA performance by carefully selecting the prefix.
Ultimately, it is desirable to find an optimal prefix $p$ without any information or access to given ground truth non-member data points on the test set.

Contrary to the Oracle setting, labels which are what should be predicted are unknown at inference time, meaning that we cannot directly use labels to calculate prefix scores.
In the next section §4, we describe how our method addresses this problem by iteratively updating membership scores and prefix scores based on one another.
We propose a new MIA framework that is designed to work robustly on any test dataset with minimal information.

## 4 Proposed Method: EM-MIA

We target the realistic MIA scenario where test data labels are unavailable.
We measure a prefix score by how $\operatorname{ReCaLL}_{p}$ on a test dataset $D_{test}$ aligns well with the current estimates of membership scores $f$ on $D_{test}$ denoted as $S(\operatorname{ReCaLL}_{p},f,D_{test})$.
More accurate membership scores can help compute more accurate prefix scores.
Conversely, more accurate prefix scores can help compute more accurate membership scores.
Based on this duality, we propose an iterative algorithm to refine membership scores and prefix scores via an Expectation-Maximization algorithm, called EM-MIA, to perform MIA with minimal assumptions on test data (§2.4).

Algorithm 1 summarizes the overall procedure of EM-MIA.
We begin with an initial assignment of membership scores using any existing off-the-shelf MIA method such as Loss (Yeom et al., ) or Min-K%++ (Zhang et al., ) (Line 1).
We calculate prefix scores $r(p)$ using membership scores and then update membership scores $f(x)$ using prefix scores.
The update rule of prefix scores (Line 3) and membership scores (Line 4) is a design choice.
We repeat this process iteratively until convergence (Line 5).
Since EM-MIA is a general framework, all components, including initialization, score update rules, and stopping criteria, are subject to modification for further improvement.

Figure: Algorithm 1 EM-MIA
[Uncaptioned image]: extracted/5908709/figs/em-mia_overview.png

##### Update Rule for Prefix Scores

Our observation in §3 shows that AUC-ROC is an effective function $S$ to calculate prefix scores given ground truth labels.
Because we do not have labels, we can assign membership labels using the current membership scores $f$ and a threshold $\tau$ to use them as approximate labels to calculate prefix scores: $\text{AUC-ROC}(\{(\operatorname{ReCaLL}_{p}(x),\delta(f(x)>\tau)|x\in D_{test}
)\})$.
The value of $\tau$ could be chosen as a specific percentile in the score distributions to decide the portion of members and non-members.
Using a median (50% percentile) is a simple and practical choice because a test dataset is usually balanced.
Instead of approximating hard labels, we can compare the ranks of $\operatorname{ReCaLL}_{p}(x)$ and the ranks of $f(x)$ on $D_{test}$ because the relative order among other data matters rather than the absolute values.
We can use the average difference in ranks as $\sum_{x\in\mathcal{D_{\mathrm{test}}}}\|\text{rank}(\operatorname{ReCaLL}_{p}(
x))-\text{rank}(f(x))\|$ or rank correlation coefficients such as Kendall’s tau (Kendall, ) and Spearman’s rho (Spearman, ).

##### Update Rule for Membership Scores

Our observation in §3 shows that a negative prefix score can be used as a good membership score.
Alternatively, we can choose candidates with top-$k$ prefix scores to construct a prefix and calculate membership scores using ReCaLL with this prefix: $f(x)=\operatorname{ReCaLL}_{p}(x)$ where $p=p_{1}\oplus p_{2}\oplus\cdots\oplus p_{n}$ and $p_{i}\in\mathrm{argtopk}_{x\in\mathcal{D_{\mathrm{test}}}}r(x)$.
How to order $p_{i}$ in $p$ is also a design choice.
Intuitively, we can place them in reverse order of the prefix score since a data point closer to the target text will have a larger impact on the likelihood.

##### External data

We may extend the test dataset $\mathcal{D_{\mathrm{test}}}$ by utilizing external data to provide additional signals.
Suppose we have a dataset of known members ($\mathcal{D_{\mathrm{m}}}$), a dataset of known non-members ($\mathcal{D_{\mathrm{nm}}}$), and a dataset of instances without any membership information ($\mathcal{D_{\mathrm{unk}}}$).
For example, $\mathcal{D_{\mathrm{m}}}$ could be old Wikipedia documents, sharing the common assumption that LLMs are usually trained with Wikipedia.
As discussed above, we target the case of $\mathcal{D_{\mathrm{nm}}}=\phi$, or at least $\mathcal{D_{\mathrm{nm}}}\cap\mathcal{D_{\mathrm{test}}}=\phi$.
However, we can construct it with completely unnatural texts (e.g., “*b9qx84;5zln”).
$\mathcal{D_{\mathrm{unk}}}$ is desirably drawn from the same distribution of $\mathcal{D_{\mathrm{test}}}$ but could be from any corpus when we do not know the test dataset distribution.
Finally, we can incorporate all available data for better prediction of membership scores and prefix scores: $\mathcal{D_{\mathrm{test}}}\leftarrow\mathcal{D_{\mathrm{test}}}\cup\mathcal{D
}_{m}\cup\mathcal{D_{\mathrm{nm}}}\cup\mathcal{D_{\mathrm{unk}}}$.
We have not explored the effect of exploiting additional data in our experiments but have left it to future work.

## 5 New Benchmark: OLMoMIA

EM-MIA works well and even performs almost perfectly on some benchmarks such as WikiMIA (Shi et al., ) (§7.1), while it does not work well on other datasets such as MIMIR (Duan et al., ) similar to other methods.
We want to know why this is the case and what are the conditions for success.
To answer these questions, we develop a new benchmark using OLMo (Groeneveld et al., ), which is a series of fully open language models pre-trained with Dolma (Soldaini et al., ) dataset.
OLMo provides intermediate model checkpoints and an index to get which data has been used for each training step, which are valuable resources to construct an MIA benchmark.
We will share our implementation and resulting benchmark datasets.
Like us, anyone can create their own benchmark on their purpose by modifying our implementation.

Figure 2 illustrates the basic setup of the OLMoMIA benchmark.
Specifically, we use checkpoints of OLMo-7B trained with 100k, 200k, 300k, and 400k steps as target models (c.f., one epoch is slightly larger than 450k steps).
Then, we consider any training data before 100k steps as members and any training data between 400k and 450k steps as non-members.
However, we should keep in mind that there could still be some ambiguity in membership despite the effort of deduplication, as discussed in §2.2.
Based on this setup, we have multiple settings with different samplings.

Figure: Figure 2: The basic setup of the OLMoMIA benchmark. The horizontal line indicates a training step. For any intermediate checkpoint at a specific step, we can consider training data before and after that step as members and non-members, respectively.
Refer to caption: extracted/5908709/figs/olmomia.png

The most straightforward setting is Random, where we randomly select members and non-members from each interval.
This setting is similar to MIMIR (Duan et al., ), where members and non-members are from the training split and test split of the Pile dataset (Gao et al., ), respectively.
MIMIR is much more challenging than WikiMIA since the training set and test set are randomly split and thus have minimal distribution shifts.

Based on clustering, we control the degree of overlap between the distribution of members and non-members, resulting in different difficulties.
First, we sample enough numbers of members and non-members (in our case, 50k each).
Next, we map sampled data into embedding vectors.
We use NV-Embed-v2 model (Lee et al., ), which was the first rank on the MTEB leaderboard (Muennighoff et al., ) as of Aug 30, 2024, for the embedding model.
Then, we perform K-means clustering (Lloyd, ) for members and non-members separately (in our case, we use $K=50$).

To prevent datasets from degeneration that have duplicates with a very minor difference, we remove data points closer than a certain distance threshold with other points within a cluster in a greedy manner to guarantee all data points in the same cluster are not too close to each other.
Empirically, we observed that most pair distances measured by cosine distance range from 0.8 to 1.2, so we set 0.6 as the minimum inter-distance for deduplication.

For the Easy setting, we pick the farthest pair of a member cluster and a non-member cluster and pick instances farthest from the opposite cluster.
For the Hard setting, we pick the closest pair of a member cluster and a non-member cluster and pick instances closest to the opposite cluster.
For the Medium setting, we pick the pair of a member cluster and a non-member cluster with a median distance and randomly sampled instances within each cluster.

Additionally, we have two additional settings derived from Random and Hard: we merge members from Random and non-members from Hard to create the Mix-1 setting, and we merge members from Hard and non-members from Random to create the Mix-2 setting.
Mix-1 aims to simulate the case where test data come from a single cluster.
Thus, any cluster might be fine, but we choose Hard with no specific reason.
These settings cover almost all possible inclusion relationships between members and non-members.

For each difficulty level, we have two splits based on the maximum sequence length of 64 and 128. For each dataset, we balance the number of members and non-members to 500 each to make the total size 1000.

## 6 Experimental Setup

Our implementation and datasets to reproduce our experiments are available at [https://github.com/gyuwankim/em-mia](https://github.com/gyuwankim/em-mia).

### 6.1 Datasets and Models

We evaluate EM-MIA and compare it with baselines (§6.2) on the WikiMIA benchmark (Shi et al., ) (§7.1) and the OLMoMIA benchmark (§7.2) using AUC-ROC as an MIA evaluation metric.
For WikiMIA, we use the original setting with length splits of 32, 64, and 128 as test datasets and use Mamba 1.4B (Gu & Dao, ), Pythia 6.9B (Biderman et al., ), GPT-NeoX 20B (Black et al., ), LLaMA 13B/30B (Touvron et al., ), and OPT 66B (Zhang et al., ) as target models, following Xie et al. () and Zhang et al. (). For OLMoMIA (§5), we use six settings of Easy, Medium, Hard, Random, Mix-1, and Mix-2 as test datasets and use checkpoints after 100k, 200k, 300k, and 400k training steps as target models.
Although EM-MIA requires a baseline sufficiently better than random guessing as an initialization, there is currently no such method for MIMIR (Duan et al., ).
Therefore, we skip experiments on MIMIR, though this is one of the widely used benchmarks on MIA for LLMs.

### 6.2 Baselines

We compare our method against the following baselines explained in §2.1: Loss (Yeom et al., ), Ref (Carlini et al., ), Zlib (Carlini et al., ), Min-K% (Shi et al., ), and Min-K%++ (Zhang et al., ).
We use Pythia-70m for WikiMIA and StableLM-Base-Alpha-3B-v2 model (Tow, ) for OLMoMIA as the reference model of the Ref method, following Shi et al. () and Duan et al. ().
For Min-K% and Min-K%++, we set $K=20$.
Among the commonly used baselines, we omit Neighbor (Mattern et al., ) because it is not the best in most cases though it requires LLM inference multiple times for neighborhood texts, so it is much more expensive.

#### 6.2.1 ReCaLL-based Baselines

As explained in §2.4, the original ReCaLL (Xie et al., ) uses labeled data from the test dataset, which is unfair to compare with the above baselines and ours.
More precisely, $p_{i}$ in the prefix $p=p_{1}\oplus p_{2}\oplus\cdots\oplus p_{n}$ are known non-members from the test set $\mathcal{D_{\mathrm{test}}}$, and they are excluded from the test dataset for evaluation, i.e., $\mathcal{D_{\mathrm{test}}}^{\prime}=\mathcal{D_{\mathrm{test}}}\setminus\{p_{
1},p_{2},\cdots,p_{n}\}$.
However, we measure the performance of ReCaLL with different prefix selection methods to understand how ReCaLL is sensitive to the prefix choice and use it as a reference instead of a direct fair comparison.

Since changing the test dataset every time for different prefixes does not make sense and makes the comparison even more complicated, we keep them in the test dataset.
A language model tends to repeat, so $\mathrm{LL}(p_{i}|p;\mathcal{M})\simeq 0$.
Because $\mathrm{LL}(p_{i}|p;\mathcal{M})\ll 0$, $\operatorname{ReCaLL}_{p}(p_{i};\mathcal{M})\simeq 0$.
It is likely to $\operatorname{ReCaLL}_{p}(p_{i};\mathcal{M})\ll\operatorname{ReCaLL}_{p}(x;
\mathcal{M})$ for $x\in\mathcal{D_{\mathrm{test}}}\setminus\{p_{1},p_{2},\cdots,p_{n}\}$, meaning that ReCaLL will classify $p_{i}$ as a non-member.
The effect would be marginal if $|\mathcal{D_{\mathrm{test}}}|\gg n$.
Otherwise, we should consider this when we read numbers in the result table.

We have four options for choosing $p_{i}$: Rand, RandM, RandNM, and TopPref.
Rand randomly selects any data from $\mathcal{D_{\mathrm{test}}}$.
RandM randomly selects member data from $\mathcal{D_{\mathrm{test}}}$.
RandNM randomly selects non-member data from $\mathcal{D_{\mathrm{test}}}$.
TopPref selects data from $\mathcal{D_{\mathrm{test}}}$ with the highest prefix scores calculated with ground truth labels the same as §3.
All options except Rand partially or entirely use labels in the test dataset.
For all methods using a random selection (Rand, RandM, and, RandNM), we execute five times with different random seeds and report the average.

The original ReCaLL (Xie et al., ) is similar to RandNM, except they report the best score after trying all different $n$ values, which is again unfair.
The number of shots $n$ is an important hyper-parameter determining performance.
A larger $n$ generally leads to a better MIA performance but increases computational cost with a longer $p$.
We fix $n=12$ since it provides a reasonable performance while not too expensive.

Other simple baselines without using any labels are Avg and AvgP, which average ReCaLL scores over all data points in $\mathcal{D_{\mathrm{test}}}$: $\textit{Avg}(x)=\frac{1}{|\mathcal{D_{\mathrm{test}}}|}\sum_{p\in\mathcal{D_{
\mathrm{test}}}}\operatorname{ReCaLL}_{p}(x)$ and $\textit{AvgP}(p)=\frac{1}{|\mathcal{D_{\mathrm{test}}}|}\sum_{x\in\mathcal{D_{
\mathrm{test}}}}\operatorname{ReCaLL}_{p}(x)$.
The intuition is that averaging will smooth out ReCaLL scores with a non-discriminative prefix while keeping ReCaLL scores with a discriminative prefix without exactly knowing discriminative prefixes.

### 6.3 EM-MIA

As explained in §4, EM-MIA is a general framework where each component can be tuned for improvement, but we use the following options as defaults based on results from preliminary experiments.
Overall, Min-K%++ performs best among baselines without ReCaLL-based approaches, so we use it as a default choice for initialization.
Alternatively, we may use ReCaLL-based methods that do not rely on any labels like Avg, AvgP, or Rand.
For the update rule for prefix scores, we use AUC-ROC as a default scoring function $S$.
For the update rule for membership scores, we use negative prefix scores as new membership scores.
For the stopping criterion, we repeat ten iterations and stop without thresholding by the score difference since we observed that membership scores and prefix scores converge quickly after a few iterations.
We also observed that EM-MIA is not sensitive to the choice of the initialization method and the scoring function $S$ and converges to similar results.

## 7 Results and Discussion

### 7.1 WikiMIA

Table 1 shows the experimental results on the WikiMIA dataset.
EM-MIA achieves state-of-the-art performance on WikiMIA for all different models and length splits, significantly outperforming all baselines, including ReCaLL, even without any given non-member test data.
EM-MIA exceeds 96% AUC-ROC in all cases.
For the largest model OPT-66B, EM-MIA gets 99% AUC-ROC for length splits 32 and 64, while ReCaLL’s performance is lower than 86% AUC-ROC.

**Table 1: AUC-ROC results on the WikiMIA benchmark. The second block (grey) is ReCaLL-based baselines. RandM, RandNM, ReCaLL, and TopPref use labels in the test dataset, so comparing them with others is unfair. We report their scores for reference. We borrow the original ReCaLL results from Xie et al. () which is also unfair to be compared with ours and other baselines.**
| Method | Mamba-1.4B | Pythia-6.9B | LLaMA-13B | NeoX-20B | LLaMA-30B | OPT-66B | Average |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 32 | 64 | 128 | 32 | 64 | 128 | 32 | 64 | 128 | 32 | 64 | 128 | 32 | 64 | 128 | 32 | 64 | 128 | 32 | 64 | 128 |
| Loss | 61.0 | 58.2 | 63.3 | 63.8 | 60.8 | 65.1 | 67.5 | 63.6 | 67.7 | 69.1 | 66.6 | 70.8 | 69.4 | 66.1 | 70.3 | 65.7 | 62.3 | 65.5 | 66.1 | 62.9 | 67.1 |
| Ref | 60.3 | 59.7 | 59.7 | 63.2 | 62.3 | 63.0 | 64.0 | 62.5 | 64.1 | 68.2 | 67.8 | 68.9 | 65.1 | 64.8 | 66.8 | 63.9 | 62.9 | 62.7 | 64.1 | 63.3 | 64.2 |
| Zlib | 61.9 | 60.4 | 65.6 | 64.3 | 62.6 | 67.6 | 67.8 | 65.3 | 69.7 | 69.3 | 68.1 | 72.4 | 69.8 | 67.4 | 71.8 | 65.8 | 63.9 | 67.4 | 66.5 | 64.6 | 69.1 |
| Min-K% | 63.3 | 61.7 | 66.7 | 66.3 | 65.0 | 69.5 | 66.8 | 66.0 | 71.5 | 72.1 | 72.1 | 75.7 | 69.3 | 68.4 | 73.7 | 67.5 | 66.5 | 70.6 | 67.5 | 66.6 | 71.3 |
| Min-K%++ | 66.4 | 67.2 | 67.7 | 70.2 | 71.8 | 69.8 | 84.4 | 84.3 | 83.8 | 75.1 | 76.4 | 75.5 | 84.3 | 84.2 | 82.8 | 69.7 | 69.8 | 71.1 | 75.0 | 75.6 | 75.1 |
| Avg | 70.2 | 68.3 | 65.6 | 69.3 | 68.2 | 66.7 | 77.2 | 77.3 | 74.6 | 71.4 | 72.0 | 68.7 | 79.8 | 81.0 | 79.6 | 64.6 | 65.6 | 60.0 | 72.1 | 72.1 | 69.2 |
| AvgP | 64.0 | 61.8 | 56.7 | 62.1 | 61.0 | 59.0 | 63.1 | 60.3 | 56.4 | 63.9 | 61.8 | 61.1 | 60.3 | 60.0 | 55.4 | 86.9 | 94.3 | 95.1 | 66.7 | 66.5 | 63.9 |
| RandM | 25.4 | 25.1 | 26.2 | 24.9 | 26.2 | 24.6 | 21.0 | 14.9 | 68.6 | 25.3 | 28.3 | 29.8 | 14.0 | 15.1 | 70.4 | 33.9 | 40.9 | 42.9 | 24.1 | 25.1 | 43.8 |
| Rand | 72.7 | 78.2 | 64.2 | 67.0 | 73.4 | 68.7 | 73.9 | 75.4 | 68.5 | 68.2 | 74.5 | 67.5 | 66.9 | 71.7 | 70.2 | 64.5 | 67.8 | 58.6 | 68.9 | 73.5 | 66.3 |
| RandNM | 90.7 | 90.6 | 88.4 | 87.3 | 90.0 | 88.9 | 92.1 | 93.4 | 68.8 | 85.9 | 89.9 | 86.3 | 90.6 | 92.1 | 71.8 | 78.7 | 77.6 | 67.8 | 87.5 | 88.9 | 78.7 |
| TopPref | 90.6 | 91.2 | 88.0 | 91.3 | 92.9 | 90.1 | 93.5 | 94.2 | 71.8 | 88.4 | 92.0 | 90.2 | 92.9 | 93.8 | 74.8 | 83.6 | 79.6 | 72.1 | 90.0 | 90.6 | 81.2 |
| Xie et al. () | 90.2 | 91.4 | 91.2 | 91.6 | 93.0 | 92.6 | 92.2 | 95.2 | 92.5 | 90.5 | 93.2 | 91.7 | 90.7 | 94.9 | 91.2 | 85.1 | 79.9 | 81.0 | 90.1 | 91.3 | 90.0 |
| EM-MIA | 97.1 | 97.6 | 96.8 | 97.5 | 97.5 | 96.4 | 98.1 | 98.8 | 97.0 | 96.1 | 97.6 | 96.3 | 98.5 | 98.8 | 98.5 | 99.0 | 99.0 | 96.7 | 97.7 | 98.2 | 96.9 |

All baselines without ReCaLL-based approaches achieve lower than 76% AUC-ROC on average across different models.
The relative order between ReCaLL-based baselines is consistent over different settings: $\textit{RandM}<\textit{Avg},\textit{AvgP}<\textit{Rand}<\textit{RandNM}<
\textit{TopPref}$.
This trend clearly shows that ReCaLL is sensitive to the choice of the prefix.
Particularly, the large gap between RandMand Rand versus RandNM shows that ReCaLL is highly dependent on the availability of given non-members.
RandNM is similar to the original ReCaLL (Xie et al., ) in most cases except for the OPT-66B model and LLaMA models with sequence length 128, probably because $n=12$ is not optimal for these cases.
Among these, Rand does not use any labels, so it is fair to compare with other baselines, and it performs worse than Min-K%++ on average.
This result again shows that ReCaLL is not strong enough without given non-members.

TopPref consistently outperforms RandNM, indicating that random prefix selection is definitely not sufficiently good and there is room for better MIA performance by prefix optimization (Shin et al., ; Deng et al., ; Guo et al., ).
Although the search space of the prefix is exponentially large and the calculation of prefix scores without labels is nontrivial, a prefix score could be a good measure to choose data points to construct the prefix.
EM-MIA approximates prefix scores and uses them to refine membership scores.

### 7.2 OLMoMIA

Table 2 shows the experimental results on the OLMoMIA benchmark.(^2^22We expect earlier training data will be harder to detect. However, we could not find a significant difference in the MIA performance for checkpoints at different numbers of training steps. Therefore, we report average scores over four intermediate OLMo checkpoints.)
EM-MIA achieves almost perfect scores on Easy and Medium similar to WikiMIA, gets performance comparable to random guessing performance on Hard and Random similar to MIMIR, and gets reasonably good scores on Mix-1 and Mix-2, though not perfect as on Easy and Medium.
EM-MIA significantly outperforms all baselines in all settings except Hard and Random, where distributions of members and non-members heavily overlap to each other and all methods are not sufficiently better than random guessing.

**Table 2: AUC-ROC results on OLMoMIA benchmark. The second block (grey) is ReCaLL-based baselines. RandM, RandNM, ReCaLL, and TopPref use labels in the test dataset, so comparing them with others is unfair. We report their scores for reference.**
| Method | Easy | Medium | Hard | Random | Mix-1 | Mix-2 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 |
| Loss | 32.5 | 63.3 | 58.9 | 49.0 | 43.3 | 51.5 | 51.2 | 52.3 | 65.7 | 49.0 | 30.8 | 54.7 |
| Ref | 56.8 | 26.8 | 61.4 | 47.2 | 49.1 | 50.7 | 49.7 | 49.9 | 59.9 | 49.7 | 38.9 | 50.9 |
| Zlib | 24.0 | 51.8 | 44.8 | 50.7 | 40.5 | 51.1 | 52.3 | 50.5 | 63.2 | 47.2 | 31.5 | 54.3 |
| Min-K% | 32.4 | 50.0 | 54.0 | 51.9 | 43.0 | 51.2 | 51.7 | 51.0 | 60.8 | 50.4 | 34.9 | 51.7 |
| Min-K%++ | 45.2 | 59.4 | 56.4 | 45.7 | 46.4 | 51.4 | 51.0 | 51.9 | 57.9 | 50.0 | 39.8 | 53.2 |
| Avg | 61.9 | 53.9 | 52.3 | 57.0 | 47.6 | 51.5 | 50.3 | 48.6 | 63.3 | 56.4 | 35.5 | 44.4 |
| AvgP | 79.2 | 39.9 | 53.9 | 61.7 | 50.2 | 51.4 | 49.0 | 50.1 | 55.7 | 63.0 | 42.7 | 41.8 |
| RandM | 32.3 | 22.7 | 39.2 | 30.3 | 45.8 | 50.5 | 48.1 | 48.2 | 49.7 | 48.0 | 29.1 | 28.7 |
| Rand | 63.7 | 46.3 | 56.0 | 59.4 | 48.9 | 52.1 | 49.7 | 49.1 | 60.6 | 68.0 | 38.0 | 38.6 |
| RandNM | 87.1 | 75.5 | 71.8 | 81.2 | 50.5 | 53.2 | 50.4 | 50.0 | 66.5 | 73.7 | 49.1 | 48.0 |
| TopPref | 88.9 | 88.5 | 79.7 | 64.4 | 55.7 | 54.5 | 52.3 | 52.7 | 79.9 | 80.2 | 55.3 | 62.1 |
| EM-MIA | 99.8 | 97.4 | 98.3 | 99.8 | 47.2 | 50.2 | 51.4 | 50.9 | 88.3 | 80.8 | 88.4 | 77.1 |

None of the baselines without ReCaLL-based approaches are successful in all settings, which implies that OLMoMIA is a challenging benchmark.
The relative order between ReCaLL-based baselines is again consistent over different settings: $\textit{RandM}<\textit{Avg},\textit{AvgP},\textit{Rand}<\textit{RandNM}<
\textit{TopPref}$, though all methods that do not use any labels fail to be successful.
Interestingly, RandNM works reasonably well on Mix-1 but does not work well on Mix-2.
This is because non-members from Mix-1 are from the same cluster while non-members from Mix-1 are randomly sampled from the entire distribution.
On the other hand, TopPref notably outperforms RandNM, implying that the effectiveness of non-member prefixes for MIA differs.

As discussed in §2.3, we reemphasize that benchmarking MIAs for LLMs is tricky because predicting where real-world test data for MIA at inference time will come from is difficult.
Therefore, at least, simulating different scenarios is beneficial and necessary, like what we did using OLMoMIA instead of a fixed existing benchmark.
In this regard, we encourage MIA developers or practitioners to evaluate their methods on diverse conditions like OLMoMIA.
We do not claim that OLMoMIA is closer to real scenarios than other benchmarks.
However, OLMoMIA could be similar to some possible real-world scenarios even if it still does not cover all possible scenarios.
The results on OLMoMIA demonstrate that EM-MIA is a robust MIA method on the varying overlap between distributions of members and non-members in a test dataset.

## 8 Conclusion

We introduce a novel MIA method for LLMs called EM-MIA that iteratively updates membership scores and prefix scores via an Expectation-Maximization algorithm for better membership inference based on the observation of their duality.
EM-MIA significantly outperforms ReCaLL even without strong assumptions that ReCaLL relies on and achieves state-of-the-art performance on WikiMIA.
EM-MIA is easily tunable with several design choices, including initialization, score update rules, and stopping criteria, allowing the application to MIA in different conditions and providing room for further improvement.
We create a new benchmark for detecting pre-training data of LLMs named OLMoMIA to better understand the conditions under which EM-MIA works with a comprehensive evaluation.
It turns out that EM-MIA robustly performs well for all settings except when the distributions of members and non-members are almost identical, resulting in none of the existing methods being better than random guessing.

While our paper focuses on detecting pre-training data with the gray-box access of LLMs where computing the probability of a text from output logits is possible, many proprietary LLMs are usually further fine-tuned (Ouyang et al., ), and they only provide generation outputs, which is the black-box setting.
We left the extension of our approach to MIAs for fine-tuned LLMs (Song & Shmatikov, ; Jagannatha et al., ; Mahloujifar et al., ; Shejwalkar et al., ; Mireshghallah et al., ; Tu et al., ) or for LLMs with black-box access (Dong et al., ; Zhou et al., ; Kaneko et al., ) as future work.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * Why use the ratio of the log-likelihoods instead of the difference? Theory (e.g. Neyman-Pearson lemma) suggests very strongly that the ratio of the likelihoods (i.e. difference of log-likelihoods) is the right operation, and the ratio of the log likelihoods can be degenerate is some circumstances. Can the authors try out a variant of ReCaLL with the difference of log likelihoods?

* The OLMoMIA benchmark only appears to work with the OLMo model. Do we have any evidence that MIA results on one model will transfer to another when factors such as distribution shift are controlled appropriately?

* Why do the authors strongly emphasize expectation maximization? This is super puzzling to me because EM is a very specific optimization algorithm used in the context of latent variable models or missing values by maximizing the ELBO (which is a provable lower bound, as described in Sec. 8.7.2 of [Murphy's book](https://probml.github.io/pml-book/book1.html)). The only resemblance I see is that there is iterative/alternating optimization, but that is very common in machine learning, optimization, statistics, etc. Unless I'm missing some deeper insight, I would suggest that the paper would be better off without a shallow and misleading comparison to EM.

* What are your plans for releasing code/software?
soundness: 2
strengths: * The idea of using prefix scores to refine membership scores and vice versa is novel.
* The empirical results are also quite strong, which leads me to believe that the proposed method will be a good benchmark for future methods.
* I commend the authors for the care they have taken to design a new benchmark. I believe it is quite significant and will be valuable for future work.
summary: The authors propose a membership inference attack for LLMs. Their key insight is that previous work requires non-member prompts for good membership inference, so one can bootstrap good prompts from membership scores and vice versa. This leads them to iteratively improving (non-member prompts) and membership scores. Empirically, the proposed method exhibits significant improvements in the attack AUROC when compared with prior art.
weaknesses: ## Main concerns
These major considerations need to be addressed for acceptance, in my opinion:

* **Metric**: The paper reports the AUROC only but not [TPR @ low FPR](https://arxiv.org/pdf/2112.03570), which is the better metric per community consensus. It is fine to design the method based on AUROC but TPR @ low FPR should be reported too.

* **Computational complexity**: * The paper lacks a precise and quantitative description of the computational complexity. I would recommend the authors to describe the complexity of the proposed method (e.g. in terms of tokens consumed by the LLM) and an apples-to-apples comparisons with baselines. Some factors to take into account are the number of iterations, number of shots, cost of computing the prefix/membership score, etc.

* **Missing experiments**: While the results are strong, the coverage/ablations of the current experiments can be greatly improved. Examples:
    - Varying number of shots for the proposed method and ReCaLL. I expect the proposed method to be more robust than ReCaLL, but a plot to this effect is missing.
    - Compare baselines in a compute-constrained setting, so that all methods receive the same computational budget.
    - Line 359 says that "EM-MIA requires a baseline sufficiently better than random guessing". How much better? I would like to see some ablations with different initialization methods to understand how good the initialization must be.
    - Vary the number of iterations of the proposed method.
    - Lines 381-395: the effect of reusing test examples for ReCaLL needs to be explored through ablations too.
These experiments are very valuable for the community. For example, how robust your method is to variations in the hyperparameters?

## Other suggestions/comments

* **Clarity**:
    - It is not clear initially if the paper deals with MIA in the pretraining or finetuning settings. It would be helpful to clarify that upfront.
    - The authors should provide some intuition of *why* ReCaLL works, given that it is a super recent development.
    - "Prefix score" in the abstract is very ambiguous. 
    - It is hard to interpret the experimental results as the tables are full of numbers. The authors may wish to present the results for one length and relegate the rest to the appendix. It would also be helpful to use some plots to demonstrate results.

* **OLMoMIA design not clear**: The second half of section 5 is written in a very casual manner and is ambiguous. I do not understand how the easy/medium/hard settings are designed. Further, are members and non-members clustered together or separately (Line 322)? I would recommend the use of more precise language (e.g. with equations). Alternatively, some figures here can greatly help (with pseudocode in the appendix). For instance, Fig 2 is quite nice.

* **Missing refs**: [Kandpal et al. (EMNLP '24)](https://arxiv.org/abs/2310.09266), [Maini et al. (NeurIPS '24)](https://arxiv.org/abs/2406.06443)

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - Section 3.2 contains experiments for MIA against LLaMA and OPT models using the wikiMIA dataset. I understand that these target models have not seen the non-members because of the release date. My question is: How are we sure that the members are included in their training datasets? For example, some Wikipedia articles (published before release dates) might be part of their test partition or validation partition. 

- The authors mention that the concentration of non-members sometimes produces better prefix scores. What is the impact of the prefix size on prefix scores? It would be great to see an ablation study showing how different prefix sizes impact the prefix scores. 

- What is the computation cost of the iterative approach of maximization. I mean in Alg 1, for each p in D_test and x in D_test, we are doing multiple round of refining r(p) and f(x). Is it expensive to these operations in multiple rounds?

- What is the impact of number of clusters in difficulty level of OLMoMIA? Why did the authors use k=50? it would be interesting to see more about the clustering hyperparameters impacts on the difficulty metric of membership inference?

- Why mink and mink++ in table 2 do not get better AUC-ROC when we are switching the difficulty level? And why they don't get better for larger target text (128 compared to 64)?

- The authors in section 6.3 mentioned that: " We also observed that EM-MIA is not sensitive to the choice of the initialization method and the scoring function S and converges to similar results" Could you please elaborate more on the intuition behind this? 

- It would be interesting to see n-gram overlap ratio between members and non-members (similar to ref AA) as a difficulty metric in easy, medium and hard splits of OLMoMIA.
soundness: 3
strengths: - The authors employed interesting clustering techniques and embedding models to design different levels of membership inference difficulty for OLMoMIA.
- The authors provide analysis over ReCaLL assumptions and weaknesses.
summary: This paper introduces a new MIA method called EA-MIA, based on RECALL, which iteratively improves MIA scores and prefix scores via an expectation-maximization algorithm. The authors also introduce a new dataset, OLMoMIA, derived from the OLMo dataset, featuring different levels of membership inference difficulty.
weaknesses: - The authors used the WikiMIA dataset in different parts of the paper (for example, for observing which update rule to use for EA-MIA). However, as they mentioned in Section 2.3, wikiMIA is not a reliable dataset to be used for MIA experiments.

- The authors did not provide the results of their approach on the MIMIR dataset, a very well-known dataset in MIA literature. They provided this reason in Section 6.1: "Although EM-MIA requires a baseline sufficiently better than random guessing as an initialization, there is currently no such method for MIMIR (Duan et al., 2024). Therefore, we skip experiments on MIMIR, though this is one of the widely used benchmarks on MIA for LLMs". However, mink++ paper reported AUC-ROC scores of 61.1 and 74.2 for Pythia-12b and on MIMIR wikipedia and Github splits respectively, which could be used as good initializations for EM-MIA.

- Reporting TPR for low FPR is an important experiment results for a new MIA to be compared to other MIAs. This paper does not provide any results on TPR for low FPRs.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: 1.	Section 2.3: What is expected from the recently published papers? What kind of adoption is required? Based on this statement: “Several ongoing attempts (Meeus et al., 2024b; Eichler et al., 2024) aim to reproduce setups that closely resemble practical MIA scenarios, but none are sufficiently effective to gain widespread adoption in the community.” These papers were published this year.
2.	Section 3: “Without access to non-members (or data points with high prefix scores), ReCaLL’s performance could be significantly lower.” Would you please measure it precisely? 
3.	Section 3: “We propose a new MIA framework that is designed to work robustly on any test dataset with minimal information” What is defined by working robustly? What is the minimal information?
4.	Section 4: “We target the realistic MIA scenario where test data labels are unavailable.” What are the test data labels?
5.	Section 4: “We measure a prefix score by how ReCaLLp on a test dataset D_test aligns well with the current estimates of membership scores f on D_test denoted as S(ReCaLL_p, f, D_test).” What is the D_test? How is S computed? D_test looks like the dataset for which we want to infer the membership. This should be clearly stated!
6.	What is the $\delta$ at the end of Section 5?
7.	The proposed method states that we could use something instead of clearly indicating what is used. For example, is the Kendall’s tau or Spearman’s rho used?
8.	The subsection about external data is totally not fitting to this paper. Why do you consider access to members and non-members? You make this assumption but then state that you never consider this in your experiments. If so, this subsection should be removed.
soundness: 1
strengths: 1.	The benchmark to assess how good the MIA methods are in distinguishing between members and non-members. However, based on the aforementioned Theorem 2, all these methods fail while we train the LLMs on more data and a given sample is seen only once during the training process.
summary: This paper tackles the problem of resolving if a given data point was used to train a large language model (LLM). This work follows a long line of previous papers that tried to solve the problem, however, as we already clearly know, it is impossible to solve it based on the result of Maini et al. ICLR 2021 [1] (see Theorem 2). The authors state at the end of their introduction that also their method is also unsuccessful in detecting members vs non-members: “Throughout the extensive experiments, we have shown that EM-MIA is a versatile MIA method and significantly outperforms previous strong baselines, though **all methods including ours still struggle to surpass random guessing in the most challenging random split setting.**” This is not a surprise based on the aforementioned Theorem 2.

**References:**

1.	Dataset Inference: Ownership Resolution in Machine Learning. Pratyush Maini, Mohammad Yaghini, Nicolas Papernot, ICLR 2021 (Spotlight).
weaknesses: 1.	The paper considers a simple scenario where “blind baselines can beat their membership inference attack” [1].
2.	The paper also fails on the benchmarks proposed by Maini et al. 2024 based on the training and validation splits from the Pile dataset [2].
3.	The membership inference attack is clearly defined, e.g. [3] and the assumptions made in this paper violates this basic requirement, based on the following claim: “Although this setting seems theoretically appropriate for evaluating MIA, there is no truly held-out in-distribution dataset in reality because LLMs are usually trained with all available data sources.” Thus, either another attack is proposed or the proposed attack has a random performance when claimed to be the membership inference attack.
4.	This method does not work well on the standard benchmark for membership inference on LLMs MIMIR by Duan et al 2024 (as stated at the beginning of Section 5).
5.	The designed OLMoMIA benchmark (as described in Section 5) is the same as proposed in Duan et al. 2024 as well as in Maini et al. 2024 [2]. The experiments are lacking the assessment on the Pile dataset: Section 6.1: “we skip experiments on MIMIR, though this is one of the widely used benchmarks on MIA for LLMs”
6.	No source code is provided!
7.	The results in Figure 1b with TPR@5%FPR = 93.4 are worse than “Blind Baselines” (Table 1) which report 94.4%.

**References:**

1.	Blind Baselines Beat Membership Inference Attacks for Foundation Models. Debeshee Das, Jie Zhang, Florian Tramèr https://arxiv.org/abs/2406.16201 
2.	LLM Dataset Inference: Did you train on my dataset? Pratyush Maini, Hengrui Jia, Nicolas Papernot, Adam Dziedzic. NeurIPS 2024 https://arxiv.org/abs/2406.06443 
3.	Membership Inference Attacks From First Principles. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, Florian Tramer. S&P 2022. https://www.computer.org/csdl/proceedings-article/sp/2022/131600b519/1FlQBPf7ixy

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can the authors please elaborate on the computational requirements of EM-MIA relative to baselines like ReCaLL?

- The paper mentions that EM-MIA’s iterative process can be initialized with different methods, yet Min-K%++ was chosen for initialization. Could the authors please provide an ablation study or justification for this choice, and discuss how EM-MIA performs when initialized with other baselines (e.g., Loss or Avg)? This information could illustrate the robustness of EM-MIA’s initialization and its dependence on a well-performing baseline.
soundness: 3
strengths: S1. The proposed EM-MIA framework introduces a novel approach to MIAs on LLMs by leveraging the expectation-maximization algorithm to iteratively enhance membership and prefix scores.

S2. There is a comprehensive set of experiments/evaluations that compare EM-MIA to strong baselines, including ReCaLL, across multiple benchmarks such as WikiMIA and OLMoMIA. 

S3. Given the growing deployment of LLMs and the increased need for privacy compliance, this work addresses a highly relevant issue in model auditing and data privacy. The authors also considered recent works that criticize MIAs on LLMs, thus also considering random splits for train/test sets (where the result is similar to other methods ~50%).
summary: This paper presents a new method to improve membership inference attacks (MIAs) on large language models (LLMs) named EM-MIA. 
EM-MIA is an iterative algorithm based on the expectation-maximization (EM) framework, which jointly refines membership and prefix scores to improve accuracy. 
The paper also introduces OLMoMIA, a benchmark allowing controlled experiments on MIA difficulty levels by adjusting training and test data distribution overlap. 
Through extensive experiments, EM-MIA achieves state-of-the-art performance on the WikiMIA dataset and outperforms existing methods (e.g., loss-based, min-k%, zlib, and ReCaLL) across various conditions in OLMoMIA.
weaknesses: Naturally, the iterative nature of EM-MIA may introduce additional computational costs compared to some baselines, especially for larger datasets or LLMs. The paper could provide an analysis of the computational complexity, with timing comparisons to baselines like ReCaLL. Highlighting any trade-offs between accuracy and computational demands would help readers assess EM-MIA's scalability and practical feasibility.

Meta-review: This paper introduces a new method for Membership Inference Attack (MIA), EM-MIA. EM-MIA is an iterative algorithm based on the expectation-maximization (EM) framework, which jointly refines membership and prefix scores to improve accuracy. The paper also introduces OLMoMIA, a benchmark allowing controlled experiments on MIA difficulty levels by adjusting training and test data distribution overlap. Reviewers had several concerns about the paper, including not reporting TPR @low FPR (which was resolved in the author response) and the limitation of wikiMIA dataset (no experiments on MIMIR dataset). Overall the paper seems to be heading in the right direction but the reviews suggest stronger empirical results would be required.

Decision: Reject