Semantic Membership Inference Attack against Large Language Models

## Abstract

Abstract Membership Inference Attacks (MIAs) determine whether a specific data point was included in the training set of a target model. In this paper, we introduce the Semantic Membership Inference Attack (SMIA), a novel approach that enhances MIA performance by leveraging the semantic content of inputs and their perturbations. SMIA trains a neural network to analyze the target model’s behavior on perturbed inputs, effectively capturing variations in output probability distributions between members and non-members. We conduct comprehensive evaluations on the Pythia and GPT-Neo model families using the Wikipedia dataset. Our results show that SMIA significantly outperforms existing MIAs; for instance, SMIA achieves an AUC-ROC of 67.39% on Pythia-12B, compared to 58.90% by the second-best attack.

## 1 Introduction

Large Language Models (LLMs) appear to be effective learners of natural language structure and
patterns of its usage.
However, a key contributing factor to their
success is their ability to memorize their training data, often in a
verbatim fashion. This memorized data can be reproduced verbatim at
inference time, which effectively serves the purpose of information
retrieval.
However, this regurgitation of training data is also at
the heart of privacy concerns in LLMs. Previous works have shown that
LLMs leak some of their training data at inference
time [, , , , , , ]
Membership Inference Attacks (MIAs) [, , , , ] aim to determine whether a specific data sample (e.g. sentence,
paragraph, document) was part of the training set of a target machine learning model.
MIAs serve as efficient tools to measure
memorization in LLMs.

Figure: Figure 1: Our Semantic Membership Inference Attack (SMIA) inference pipeline.
Refer to caption: x1.png

Existing approaches to measure memorization in LLMs have predominantly
focused on verbatim memorization, which involves identifying exact
sequences reproduced from the training data. However, given the
complexity and richness of natural language, we believe this method
falls short. Natural language can represent the same ideas or
sensitive data in numerous forms, through different levels of
indirection and associations. This power of natural language makes
verbatim memorization metrics inadequate to address the more nuanced
problem of measuring semantic memorization, where LLMs internalize and
reproduce the essence or meaning of training data sequences, not just
their exact wording.

Previous MIAs against LLMs has predominantly focused on classifying members and non-members by analyzing the probabilities assigned to input texts or their perturbations [, , , ]. In contrast, we introduce the Semantic Membership Inference Attack (SMIA), the first MIA to leverage the semantic content of input texts to enhance performance. SMIA involves training a neural network to understand the distinct behaviors exhibited by the target model when processing members versus non-members.

Our central hypothesis is that perturbing the input of a target model will result in differential changes in its output probability distribution for members and non-members, contingent on the extent of semantic change distance. Crucially, this behavior is presumed to be learnable. To implement this, we train the SMIA model to discern how the target model’s behavior varies with different degrees of semantic changes for members and non-members. Post-training, the model can classify a given text sequence as a member or non-member by evaluating the semantic distance and the corresponding changes in the target model’s behavior for the original input and its perturbations.

Figure 1 illustrates the pipeline of our proposed SMIA inference. The SMIA inference pipeline for a given text $x$ and a target model $T(.)$ includes four key steps: (1) Neighbor Generation: The target sequence is perturbed $n$ times by randomly masking different positions and filling them using a masking model, such as T5 [], to generate a neighbour dataset $\tilde{x}$ (similar to [, ]). (2) Semantic Embedding Calculation: The semantic embeddings of the input text and its neighbours are computed by using an embedding model, such as Cohere Embedding model []. (3) Loss Calculation: The loss values of the target model for the input text and its neighbours are calculated. (4) Membership Probability Estimation: The trained SMIA model is then used to estimate the membership probabilities. These scores are averaged and compared against a predefined threshold to classify the input as a member or non-member.

Empirical Results:
We evaluate the performance of our proposed SMIA across different model families, specifically Pythia and GPT-Neo, using the Wikipedia dataset. To underscore the significance of the non-member dataset in evaluating MIAs, we include two distinct non-member datasets in our analysis: one derived from the exact distribution of the member dataset and another comprising Wikipedia pages published after a cutoff date, which exhibit lower n-gram similarity with the members. Additionally, we assess SMIA under two settings: (1) verbatim evaluation, where members exactly match the entries in the target training dataset, and (2) slightly modified members, where one word is either duplicated, added, or deleted from the original member data points.

Our results demonstrate that SMIA consistently outperforms all existing MIAs by a substantial margin. For instance, SMIA achieves an AUC-ROC of 67.39% for Pythia-12B on the Wikipedia dataset. In terms of True Positive Rate (TPR) at low False Positive Rate (FPR), SMIA achieves TPRs of 3.8% and 10.4% for 2% and 5% FPR, respectively, on the same model. In comparison, the second-best attack, the Reference attack, achieves an AUC-ROC of 58.90%, with TPRs of 1.1% and 6.7% for 2% and 5% FPR, respectively.

## 2 Background

Membership inference attacks (MIAs) against large language models (LLMs) aim to determine whether a given data point was part of the training dataset used to train the target model or not. Given a data point $x$ and a trained autoregressive model $T(.)$, which predicts $P(x_{t}|x_{1},x_{2},...,x_{t-1})$ reflecting the likelihood of the sequence under the training data distribution, these attacks compute a membership score $A(x,T)$. By applying a threshold $\epsilon$ to this score, we can classify $x$ as a member (part of the training data) or a non-member. In Appendix A, we provide details about how existing MIA work against LLMs.

MIAs provide essential assessments in various domains. They are cornerstone for privacy auditing [, ], where they test whether LLMs Leak sensitive information, thereby ensuring models do not memorize data beyond their learning scope.
In the realm of machine unlearning [], MIAs are instrumental in verifying the efficacy of algorithms to comply with the right to be forgotten, as provided by privacy laws like the General Data Protection Regulation (GDPR) [] and the California Consumer Privacy Act (CCPA) [].
These attacks are also pivotal in copyright detection, pinpointing the unauthorized inclusion of copyrighted material in training datasets[, ].
Furthermore, they aid in detecting data contamination – where specific task data might leak into a model’s general training dataset [, ]. Lastly, in the tuning the hyperparameters of differential privacy, MIAs provide insights for setting the $\epsilon$ parameter (i.e., the privacy budget), which dictates the trade-off between a model’s performance and user privacy [, , ].

## 3 Our Proposed SMIA

MIAs seek to determine whether a specific data sample was part of the training set of a machine learning model, highlighting potential privacy risks associated with model training. Traditional MIAs typically verify if a text segment, ranging from a sentence to a full document, was used exactly as is in the training data. Such attacks tend to falter when minor modifications are made to the text, such as punctuation adjustments or word substitutions, while the overall meaning remains intact. We hypothesize that a LLM, having encountered specific content during training, will exhibit similar behaviors towards semantically similar text snippets during inference. Consequently, an LLM’s response to semantically related inputs should display notable consistency.

Figure: Figure 2: Input features for our SMIA: semantic change and taregt model behaviour change for inputs and their neighbors.
Refer to caption: x2.png

In this paper, we introduce the concept of a Semantic Membership Inference Attack (SMIA) against LLMs. This novel attack method enables an attacker to discern whether a concept, defined as a set of semantically akin token sequences, was part of the training data. Examples of such semantically linked concepts include "John Doe has leukemia" and "John Doe is undergoing chemotherapy." The proposed SMIA aims to capture a broader spectrum of data memorization incidents compared to traditional MIA, by determining whether the LLM was trained on any data encompassing the targeted concept.

### 3.1 SMIA Design

For the SMIA, we assume that the adversary has grey-box access to the target LLM, denoted as $T(.)$, which is trained on an unknown dataset $D_{\textit{train}}$. The adversary can obtain loss values or log probabilities for any input text from this model, denoted as $\ell(.,T)$, but lacks additional information such as model weights or gradients. The cornerstone of our SMIA is the distinguishable behavior modification exhibited by the target model when presented with semantic variants of member and non-member data points.

As illustrated in Figure 2, consider a 2-dimensional semantic space populated by data points. Members and non-members are represented by green circles and red circles, respectively. By generating semantic neighbors for both member and non-member data points (shown as green and red diamonds, respectively), we measure the semantic distance between targeted data points and their neighbors, denoted as $d_{i}^{m}$ and $d_{i}^{n}$. Subsequently, we observe the target model’s response to these data points by assessing the differences in loss values (y-axis for log probability of that text under the taregt LLM data distribution), thereby training the SMIA to classify data points as members or non-members based on these observed patterns.

### 3.2 SMIA Pipeline

The SMIA consists of two primary components: initially, the adversary trains a neural network model $A(.)$ on a dataset gathered for this purpose, and subsequently uses this trained model for inference. The training and inference processes are detailed in Algorithms 1 and  2, respectively.

During the training phase, the adversary collects two distinct datasets: $D_{\text{tr-m}}$ (member dataset) and $D_{\text{tr-n}}$ (non-member dataset). $D_{\text{tr-m}}$ comprises texts known to be part of the training dataset of the target model $T(.)$, while $D_{\text{tr-n}}$ includes texts confirmed to be unseen by the target model during training. The adversary utilizes these datasets to develop a membership inference model capable of distinguishing between members ($\in D_{\text{tr-m}}$) and non-members ($\in D_{\text{tr-n}}$).
For instance, Wikipedia articles or any publicly available data collected before a specified cutoff date are commonly part of many known datasets. Data collected after this cutoff date can be reliably assumed to be absent from the training datasets.

Figure: Algorithm 1 Our Proposed Semantic Membership Inference Attack: training

The SMIA training procedure, shown in Algorithm 1, involves four key stages:

i) Neighbour generation (Algorithm 1 lines 1-2):
The initial phase of SMIA involves generating a dataset of neighbours for both the member dataset ($D_{\text{tr-m}}$) and the non-member dataset ($D_{\text{tr-n}}$). The creation of a neighbour entails making changes to a data item that preserve most of its semantics and grammar, thereby ensuring that these neighbours are semantically closed to the original sample and should be assigned a highly similar likelihood under any textual probability distribution, as similar to [, ]. Specifically, Algorithm 1 line 1 describes the creation of masked versions of $D^{m}_{\text{masked}}$ and $D^{n}_{\text{masked}}$ by randomly replacing $k$ words within each text item $n$ times. Following this, in line 2, a Neighbour generator model $N(x,L,K)$—a masking model—is employed to refill these masked positions, generating datasets $\tilde{D}^{m}$ and $\tilde{D}^{n}$ for members and non-members, respectively. We utilize the T5 model [] in our experiments to perform these replacements, aiming to produce $n$ semantically close variants of each data point.

ii) Calculate semantic embedding of the data points (Algorithm 1 line 3):
The subsequent step involves computing semantic embeddings for both the original data points and their neighbours. As per Algorithm 1 line 3, we obtain the embedding vectors $\Phi^{m}\leftarrow E(D_{\text{tr-m}})$ and $\Phi^{n}\leftarrow E(D_{\text{tr-n}})$ for the member and non-member data points, respectively. Additionally, we calculate $\tilde{\Phi}^{m}\leftarrow E(\tilde{D}^{m})$ and $\tilde{\Phi}^{n}\leftarrow E(\tilde{D}^{n})$ for their respective neighbours. These vectors represent each data point’s position in a semantic space encompassing all possible inputs. Our experiments leverage the Cohere Embedding V3 model [], which provides embeddings with 1024 dimensions, to capture these semantic features.

iii) Monitor the behaviour of the target model for different inputs (Algorithm 1 line 4):
The third stage entails monitoring the target model’s response across data items in the four datasets. Here, we calculate the loss values: $L^{m}\leftarrow\ell(D_{\text{tr-m}},T)$ for the member samples, $L^{n}\leftarrow\ell(D_{\text{tr-n}},T)$ for the non-member samples, and similarly $\tilde{L}^{m}\leftarrow\ell(\tilde{D}^{m},T)$ and $\tilde{L}^{n}\leftarrow\ell(\tilde{D}^{n},T)$ for their respective neighbours. This step is crucial for understanding how the model’s behavior varies between members and non-members under semantically equivalent perturbations.

iv) Train an attack model (Algorithm 1 lines 5-16):
The final phase of training involves developing a binary neural network capable of distinguishing between members and non-members by detecting patterns of semantic and behavioral changes induced by the perturbations. We initiate this by randomly initializing the attack model $A(.)$, then training it to discern differences between the semantic embeddings and loss values for each data point and its neighbours. The input features for $A$ include the differences in semantic vectors $\Phi^{m}_{i}-\tilde{\Phi}^{m}_{i}$ and the changes in loss values $L^{m}_{i}-\tilde{L}^{m}_{i}$ for each sample $i$. Each sample is labeled ’1’ for members and ’0’ for non-members, with each training batch consisting of an equal mix of both, as suggested in prior research []. The model is trained over $R$ epochs using a learning rate $r$, culminating in a trained binary classifier that effectively distinguishes between members and non-members based on the observed data. We prvoide our SMIA training cost in Appendix B.

Figure: Algorithm 2 Our Proposed Semantic Membership Inference Attack: inference

#### SMIA Inference:

Upon completing the training of the model $A(.)$, it can be employed to assess whether a given input text $x$ was part of the target model $T(.)$’s training dataset. Algorithm 2 details the inference procedure, which mirrors the training process. Initially, $n_{\text{inf}}$ neighbours for $x$ are generated using the mask model (lines 1-2). Subsequently, we compute both the semantic embedding vectors and the loss values for $x$ and its neighbours $\tilde{x}$ (lines 3-4). These computed differences are then fed into the attack model $A(\phi-\tilde{\phi}_{j},L-\tilde{L}_{j})$, which evaluates each neighbour $j$. The final SMIA score for $x$ is determined by averaging the scores from all $n_{inf}$ neighbours (line 5), and this score is compared against a predefined threshold $\epsilon$ to ascertain membership or non-membership (line 6).

## 4 Experiment Setup

In this section, we describe the models and datasets used in our experiments. Due to space constraints, we have organized additional information into appendices. We provide the details of the architecture for SMIA model in Appendix D.1, cost estimation of SMIA to Appendix B, privacy metrics used in our analysis in Appendix D.2, the hyperparameters for training the SMIA model in Appendix D.4, the baselines in Appendix A, and the computational resources utilized in Appendix D.5.

### 4.1 Models

Target Models:
In our experiments, we evaluate our proposed SMIA across a diverse set of language models to assess its effectiveness and robustness. We utilize three categories of target models: (1) Pythia Model Suite: This category includes the largest models with 12B, 6.9B, and 2.7B parameters from the Pythia model suite [], trained on the Pile dataset []. (2) Pythia-Deduped: It consists of models with the same parameterization (12B, 6.9B, and 2.7B) but trained on a deduplicated version of the Pile dataset. This variation allows us to analyze the impact of dataset deduplication on the effectiveness of MIAs. (3) GPT-Neo Family: To test the generality of our approach across different architectures, we include models from the GPT-NEO family [], specifically the 2.7B and 1.3B parameter models, also trained on the Pile dataset.

Models Used in SMIA:
The SMIA framework incorporates three critical components: (1) Masking Model: We employ T5 with 3B parameters [] for generating perturbed versions of the texts, where random words are replaced to maintain semantic consistency. (2) Semantic Embedding Model: The Cohere Embedding V3 model [] is utilized to produce a 1024-dimensional semantic embedding vector for each text, enabling us to capture nuanced semantic variations. (3) Binary Neural Network Classifier: For the SMIA model, we utilize a relatively simple neural network (details shown in Table 6) with 1.2M parameters, which is trained to distinguish between member and non-member data points. In Appendix D.4, we discuss the hyperparameters that we use in our experiments for this model.

### 4.2 Datasets

To evaluate the effectiveness of the SMIA, we need to collect three datasets: training dataset $D_{\text{tr}}=\{D_{\text{tr-m}},D_{\text{tr-n}}\}$, validation dataset $D_{\text{val}}=\{D_{\text{val-m}},D_{\text{val-n}}\}$, and test dataset $D_{\text{te}}=\{D_{\text{te-m}},D_{\text{te-n}}\}$. Each dataset comprises a member and a non-member split. We employ the training dataset for model training, the validation dataset for tuning the hyperparameters, and the test dataset for evaluating the model performance based on various metrics.

Wikipedia Training and Validation:
We selected a total of 14,000 samples from Wikipedia, verified as parts of the training or test split of the Pile dataset []. This includes 7,000 member samples from the training split of the Wikipedia portion of Pile and 7,000 non-member samples from the test split. Samples were selected to have a minimum of 130 words and were truncated to a maximum of 150 words. Consistent with prior studies [, ], we prepended article titles to the text of each article, separated by a "\n \n". The split for these samples assigns 6,000 from each category to the training dataset ($D_{\text{tr}}$) and 1,000 from each to the validation dataset ($D_{\text{val}}$). In Appendix B, we provide the cost estimation for preparing this dataset for our training. For example for Wikipedia training part, calculating the embedding vectors from Cohere model costs around $32.

Wikipedia Test:
For the test member dataset ($D_{\text{te-m}}$), we similarly sourced 1,000 samples from the training portion of Pile. Selecting an appropriate non-member dataset ($D_{\text{te-n}}$) for testing is crucial, as differences in data distribution between member and non-member samples can falsely influence the perceived success of membership inference. Prior research [] indicates that non-member samples drawn from post-training publications or different sections of the Pile test dataset show varied overlap in linguistic features such as n-grams, which can affect inference results. To address this, we established two non-member test datasets: the first, referred to as Wikipedia Test ($WT=\{D_{\text{te-m}},D^{\text{PileTest}}_{\text{te-n}}\}$), includes samples from Wikipedia pages before March 2020 that are part of the Pile test dataset. The second, called Wikipedia Cutoff ($WC=\{D_{\text{te-m}},D^{\text{CutOff}}_{\text{te-n}}\}$), consists of 1,000 samples from Wikipedia pages published after August 2023, ensuring they were not part of the Pile training dataset.

## 5 Experiments

In this section, we present the experimental results of our SMIA and compare its performance to other MIAs in both verbatim and modified settings. Due to space constraints, we defer the TPR of attacks at low FPR to Appendix C.1, the effect of deduplication in the Pythia model family to Appendix C.2, analysis of SMIA’s performance with varying numbers of neighbors during inference to Appendix C.3, the effect of training size on SMIA’s performance to Appendix C.4, and, the histogram of similarities between generated neighbors and their original texts in both member and non-member training datasets to Appendix C.5.

### 5.1 Evaluation in Verbatim Setting

Our initial set of experiments aims to classify members and non-members without any modifications to the data, meaning that the members ($D_{\text{te-m}}$) in the test dataset are verbatim entries from the training dataset of the models. This evaluation setting is consistent with prior works [, , , , ]. Table 1 presents the AUC-ROC metric for various baseline methods and our proposed SMIA approach across different trained models and two distinct test datasets of Wikipedia (Refer to Appendix C.2 for evaluation results on deduplicated models). Additionally, Table 3 in Appendix C.1 provides the True Positive Rate (TPR) at low False Positive Rates (FPR) for these methods. The results demonstrate that SMIA significantly outperforms existing methods. For instance, on Pythia-12B and $WT=\{D_{\text{te-m}},D^{\text{PileTest}}_{\text{te-n}}\}$ test dataset (i.e., when non-members are sampled from the same data distribution as members), SMIA achieves an AUC-ROC of 67.39% with TPRs of 3.8% and 10.4% at 2% and 5% FPR, respectively. In contrast, the LOSS method [] yields an AUC-ROC of 54.94% and TPRs of 2.1% and 5.8% at the same FPR thresholds. The Ref attack [], which utilizes Pythia 1.4B to determine the complexity of test data points on a reference model trained on the same data distribution (a challenging assumption in real-world scenarios), achieves an AUC-ROC of 58.90% with TPRs of 2% and 8.2% at 2% and 5% FPR. Furthermore, Min-K [] and Min-K++ [] show better AUC-ROC compared to the LOSS attack, achieving 56.66% and 57.67% for $K=20\%$.

Why SMIA Outperforms Other MIAs:
SMIA delivers superior performance for two key reasons: Firstly, it incorporates the semantics of the input text into the analysis, unlike the baseline methods that solely rely on the target model’s behavior (e.g., log probability) for their membership score calculations. Secondly, SMIA utilizes a neural network trained specifically to distinguish between members and non-members, offering a more dynamic and effective approach compared to the static statistical methods used by previous MIAs.

**Table 1: AUC-ROC performance metrics for various MIAs, including our SMIA, evaluated on different trained models (Pythia and GPT-Neo) using the Wikipedia. The table compares results for verbatim member data $D_{\text{te-m}}$ entries against non-member datasets $D_{\text{te-n}}^{\text{PileTest}}$ and $D_{\text{te-n}}^{\text{CutOff}}$.**
|  | Pythia-12B | Pythia-6.9B | Pythia-2.7B | GPT-Neo2.7B | GPT-Neo1.3B |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | WT | WC | WT | WC | WT | WC | WT | WC | WT | WC |
| LOSS [] | 54.94 | 67.56 | 54.23 | 65.95 | 53.14 | 63.99 | 53.32 | 63.34 | 52.98 | 62.10 |
| Ref []<br>(Pythia 70M) | 52.73 | 58.29 | 51.71 | 56.42 | 49.92 | 53.74 | 50.07 | 53.86 | 49.70 | 52.91 |
| Ref<br>(Pythia 1.4B) | 58.90 | 67.44 | 57.01 | 63.79 | 51.39 | 56.06 | 52.27 | 56.80 | 50.03 | 50.98 |
| Zlib [] | 54.33 | 66.56 | 53.61 | 64.98 | 52.54 | 63.01 | 52.70 | 62.59 | 52.42 | 61.38 |
| Nei [] | 55.83 | 72.06 | 55.17 | 70.78 | 53.87 | 69.13 | 53.51 | 68.34 | 53.08 | 67.36 |
| Min-K []<br>($K=10\%$) | 56.96 | 76.05 | 56.00 | 73.96 | 54.05 | 71.21 | 53.72 | 70.53 | 53.32 | 68.40 |
| Min-K<br>($K=20\%$) | 56.66 | 73.90 | 55.65 | 71.95 | 53.86 | 69.26 | 53.66 | 68.68 | 53.36 | 66.82 |
| Min-K<br>($K=30\%$) | 56.17 | 72.18 | 55.23 | 70.32 | 53.67 | 67.84 | 53.59 | 67.26 | 53.33 | 65.54 |
| Min-K++ []<br>($K=10\%$) | 56.83 | 78.47 | 54.77 | 76.17 | 52.37 | 72.38 | 51.73 | 72.93 | 51.57 | 69.87 |
| Min-K++<br>($K=20\%$) | 57.67 | 79.34 | 55.62 | 76.77 | 53.28 | 72.82 | 52.82 | 73.07 | 52.02 | 70.13 |
| Min-K++<br>($K=30\%$) | 57.76 | 78.96 | 55.81 | 76.21 | 53.62 | 72.27 | 53.21 | 72.46 | 52.41 | 69.52 |
| Our SMIA | 67.39 | 93.35 | 64.63 | 92.11 | 60.65 | 89.97 | 59.71 | 89.59 | 58.92 | 87.43 |

Importance of non-member dataset: In the other test dataset ($WC=\{D_{\text{te-m}},D^{\text{CutOff}}_{\text{te-n}}\}$), where non-members are derived from Wikipedia pages published after August 2023, we observe a substantial improvement in SMIA performance, consistent with findings from other studies []. For example, SMIA achieves an AUC-ROC of 67.39% and 93.35% for Pythia-12B on $WT$ and $WC$, respectively. In terms of TPR at low FPR for the same model, SMIA achieves 3.8% and 10.4% for 2% and 5% FPR with the $WT$ dataset, while achieving 46.2% and 66.0% for 2% and 5% FPR with the $WC$ dataset. This increase is also observed in other attack methods. For instance, Min-K++ (with $K=10\%$) attains 54.77% AUC-ROC for the $WT$ dataset and 76.17% for the $WC$ dataset. The underlying reason for this is that the member dataset ($D_{\text{te-m}}$) has a higher n-gram overlap with the $WT$ non-member dataset compared to the $WC$ non-member dataset. A high n-gram overlap between members and non-members implies that substrings of non-members may have been seen during training, complicating the distinction between members and non-members [].

Larger models memorize more:
Another observation from Table 1 and Table 3 is that larger models exhibit greater memorization, consistent with findings from previous studies [, , ]. For instance, for the $WT$ ($WC$) test datasets, SMIA achieves AUC-ROC scores of 67.39% (93.35%), 64.63% (92.11%), and 60.65% (89.97%) for Pythia 12B, 6.9B, and 2.7B, respectively. Similarly, SMIA achieves 59.71% (89.59%) and 58.92% (87.43%) on GPT-Neo 2.7B and 1.3B, respectively, for the $WT$ ($WC$) test datasets.

### 5.2 Evaluation in Modified Settings

Existing MIAs against LLMs typically assess the membership status of texts that exist verbatim in the training data. However, in practical scenarios, member data might undergo slight modifications. An important application of MIAs is identifying the use of copyrighted content in training datasets. For instance, in the legal case involving the New York Times and OpenAI, the outputs of ChatGPT were found to be very similar to NYTimes articles, with only minor changes such as the addition or deletion of a single word []. This section explores the capability of SMIA to detect memberships even after such slight modifications.

To evaluate SMIA and other MIAs under these conditions, we generated three new test member datasets from our existing test member dataset ($D_{\text{te-m}}$) as follows
(Figure 5 provides examples for each modification):
Duplication: A random word in each member data point is duplicated. Deletion: A random word in each member data point is deleted. Addition: A mask placement is randomly added in each member data point, and the T5 model is used to fill the mask position, with only the first word of the T5 replacement being used. We just consider one word modification beacuse more than one word modification reuslts in a drastic drop of performance for all attacks.

Table 2 presents the AUC-ROC performance results of different MIAs and our SMIA under these slightly modified test member datasets. The table includes the best AUC-ROC values for Min-K and Min-K++ across different values of $K$. The results indicate that for the $WT$ non-member dataset, when a word is duplicated or added from the T5 output, the Ref attack outperforms SMIA. For instance, with Pythia-12B, the Ref attack achieves AUC-ROC scores of 57.88% and 57.95% after word duplication and addition from the T5 output, respectively, whereas SMIA achieves scores of 55.13% and 54.19% for the same settings. It is important to note that the Reference model is Pythia-1.4B, which shares the same architecture and training dataset (Pile) but with fewer parameters, a scenario that is less feasible in real-world applications. However, when a word is deleted, SMIA retains much of its efficacy, achieving an AUC-ROC of 62.47% compared to 58.25% for the Ref attack on the $WT$ non-member dataset. This indicates that SMIA is more sensitive to additions than deletions.

In scenarios involving the $WC$ non-member dataset, where non-members exhibit lower n-gram overlap with members, SMIA consistently outperforms other MIAs. For example, SMIA achieves AUC-ROC scores of 89.36% and 92.67% for word addition and deletion, respectively, while the Ref attack scores 66.50% and 66.84% for these modified member datasets.

Another key observation is that Min-K++ exhibits a greater decline in AUC-ROC than Min-K following modifications. For instance, on Pythia-12B with the $WC$ non-member dataset, Min-K++ AUC-ROC drops from 76.05% (no modification) to 69.07% (duplication), 70.81% (addition), and 69.87% (deletion). Conversely, Min-K AUC-ROC decreases from 76.05% (no modification) to 69.46% (duplication), 71.10% (addition), and 70.48% (deletion). This increased sensitivity of Min-K++ to modifications is due to its reliance on the average and variance of all vocabulary probabilities to normalize its scores, making it more susceptible to changes in these probabilities, thereby degrading performance.

**Table 2: Performance of various MIAs including our SMIA under different modification scenarios of the test member dataset ($D^{m}_{\text{te}}$). The table compares AUC-ROC scores when test members undergo word duplication, deletion, or addition using the T5 model. The results highlight the robustness of SMIA, especially against deletions and when non-members have lower n-gram overlap with members.**
|  |  | Pythia-12B | Pythia-6.9B |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Modification | WT | WC | WT | WC |
| Duplication | Loss | 52.07 | 64.60 | 51.41 | 63.04 |
|  | Ref | 57.88 | 66.48 | 55.94 | 62.72 |
|  | Zlib | 51.87 | 63.99 | 51.23 | 62.44 |
|  | Nei | 51.71 | 68.13 | 51.09 | 66.87 |
|  | Mink | 51.93 | 69.46 | 51.10 | 67.45 |
|  | Mink++ | 46.37 | 69.07 | 44.94 | 66.46 |
|  | Our SMIA | 55.13 | 90.53 | 52.68 | 88.80 |
| Addition | Loss | 52.36 | 64.90 | 51.70 | 63.33 |
|  | Ref | 57.95 | 66.55 | 56.05 | 62.84 |
|  | Zlib | 52.31 | 64.47 | 51.65 | 62.93 |
|  | Nei | 51.55 | 67.80 | 50.94 | 66.61 |
|  | Mink | 52.60 | 71.10 | 51.75 | 69.02 |
|  | Mink++ | 48.23 | 70.81 | 46.60 | 68.15 |
|  | Our SMIA | 54.19 | 89.36 | 51.97 | 87.69 |
| Deletion | Loss | 51.83 | 64.28 | 51.19 | 62.74 |
|  | Ref | 58.25 | 66.84 | 56.61 | 63.40 |
|  | Zlib | 50.58 | 62.44 | 49.90 | 60.89 |
|  | Nei | 54.55 | 70.65 | 53.99 | 69.50 |
|  | Mink | 52.07 | 70.48 | 51.24 | 68.36 |
|  | Mink++ | 47.46 | 69.87 | 46.04 | 67.30 |
|  | Our SMIA | 62.47 | 92.67 | 60.39 | 91.37 |

## 6 Conclusion and Future Works

In this paper, we introduced the Semantic Membership Inference Attack (SMIA), which leverages the semantics of input texts and their perturbations to train a neural network for distinguishing members from non-members. We evaluated SMIA in two primary settings: (1) where the test member dataset exists verbatim in the training dataset of the target model, and (2) where the test member dataset is slightly modified through the addition, duplication, or deletion of a single word.

For future work, we plan to evaluate SMIA in settings where the test member dataset consists of paraphrases of the original member data points, with minimal semantic distance between them. This will help us demonstrate that more advanced models tend to memorize the semantics of their training data rather than their exact wording.

Additionally, we aim to apply SMIA to measure unintended multi-hop reasoning. In multi-hop reasoning [], a model could connect two parts of the training data through indirect inferences, potentially disclosing private information. We intend to assess how much the target model reveals about its training data through multi-hop reasoning using our SMIA approach.

Another direction is to utilize SMIA to show that anonymization is insufficient. SMIA can reveal the limitations of traditional data redaction techniques, illustrating how anonymization falls short when an adversary can cross-reference (i.e., use supplementary information from another source) to deduce sensitive information, such as a person’s medical condition.

Finally, we plan to use SMIA to measure hallucination [, , ] in LLMs.
We hypothesize that the issues of hallucination and memorization may be interconnected in interesting ways. Intuitively, the more an LLM memorizes its training data, the less likely it is to hallucinate text that contradicts the memorized data. SMIA can provide a metric for assessing the likelihood of text output being a result of the model’s accurate memorization (direct or multi-hop) versus hallucination. This metric is particularly valuable as it measures the extent to which an output is derived from the model’s intrinsic semantic beliefs, shaped by its training data.

## Appendix A Existing MIAs against LLMs

As mentioned in Section 2, MIAs assign a membership score $A(x,T)$ to a given text input $x$ and a trained model $T(.)$. This score represents the likelihood that the text was part of the dataset on which $T(.)$ was trained. A threshold $\epsilon$ is then applied to this score to classify the text as a member if it is higher than $\epsilon$, and a non-member if it is lower. In this section, we provide the description of existing MIAs against LLMS.

LOSS []:
The LOSS method utilizes the loss value of model $T(.)$ for the given text $x$ as the membership score; a lower loss suggests that the text was seen during training, so $A(x,T)=\ell(T,x)$.

Ref []:
Calculating membership scores based solely on loss values often results in high false negative rates. To improve this, a difficulty calibration method can be employed to account for the intrinsic complexity of $x$. For example, repetitive or common phrases typically yield low loss values. One method of calibrating this input complexity is by using another LLM, $Ref(.)$, assumed to be trained on a similar data distribution. The membership score is then defined as the difference in loss values between the target and reference models, $A(x,T)=\ell(x,T)-\ell(x,Ref)$. Follwoing recent works [, ], we use smaller reference models, Pythia 1.4B and Pythia 70M, which are trained on the same dataset (Pile) and share a similar architecture with the Pythia target models.

Zlib []:
Another method to calibrate the difficulty of a sample is by using its zlib compression size, where more complex sentences have higher compression sizes. The membership score is then calculated by normalizing the loss value by the zlib compression size, $A(x,T)=\frac{\ell(x,T)}{\textit{zlib}(x)}$.

Nei [, ]:
This method perturbs the given text to calibrate its difficulty without the need for a reference model. Neighbors are created by masking random words and replacing them using a masking model like BERT [] or T5 []. If a model has seen a text during training, its loss value will generally be lower than the average of its neighbors. The membership score is the difference between the loss value of the original text and the average loss of its neighbors, $A(x,T)=\ell(x,T)-\frac{1}{n}\sum_{i\in[n]}\ell(\hat{x}_{i},T)$, where in our experiments for each sample $n=25$ neighbors are generated using a T5 model with 3B parameters.

Min-K []:
This attack hypothesizes that non-member samples often have more tokens assigned lower likelihoods. It first calculates the likelihood of each token as $\text{Min-K\%}_{\text{token}}(x_{t})=\log p(x_{t}|x_{<t})$, for each token $x_{t}$ given the prefix $x_{<t}$. The membership score is then calculated by averaging over the lowest $K\%$ of tokens with lower likelihood, $A(x,T)=\frac{1}{|\text{min-k\%}|}\sum_{x_{i}\in min-k\%}\text{Min-K\%}_{\text{
token}}(x_{t})$.

Min-K++ []:
This method improves on Min-K by utilizing the insight that maximum likelihood training optimizes the Hessian trace of likelihood over the training data. It calculates a normalized score for each token $x_{t}$ given the prefix $x_{<t}$ as $\text{Min-K\%++}_{\text{token}}(x_{t})=\frac{\log p(x_{t}|x_{<t})-\mu_{x_{<t}}
}{\sigma_{x_{<t}}}$, where $\mu_{x_{<t}}$ is the mean log probability of the next token across the vocabulary, and $\sigma_{x_{<t}}$ is the standard deviation. The membership score is then aggregated by averaging the scores of the lowest $K\%$ tokens, $A(x,T)=\frac{1}{|\text{min-k\%++}|}\sum_{x_{i}\in min-k\%}\text{Min-K\%++}_{
\text{token}}(x_{t})$.

## Appendix B SMIA Cost Estimation

The cost estimation for deploying the SMIA involves several computational and resource considerations. Primarily, the cost is associated with generating neighbours, calculating embeddings, and evaluating loss values for the target model $T(.)$.

For each of the datasets, $D_{\text{tr-m}}$ (members) and $D_{\text{tr-n}}$ (non-members), consisting of $\beta$ data samples each, we generate $n$ neighbours per data item. Consequently, this results in a total of $2\times n\times\beta$ neighbour generations. Assuming each operation has a fixed cost, with $c_{N}$ for generating a neighbour, $c_{T}$ for computing a loss value, and $c_{E}$ for calculating an embedding, the total cost for the feature collection phase can be approximated as:
$2\times(n\times\beta+1)\times(c_{N}+c_{E}+c_{T})$.
In this estimation, the training of the neural network model $A(.)$ is considered negligible due to its relatively small size (few million parameters) and its architecture, which primarily consists of fully connected layers. Additionally, the costs associated with $c_{T}$ and $c_{N}$ are not significant in this context as they are incurred only during the inference phase. Thus, the predominant cost factor is $c_{E}$, the cost of embedding calculations.

In practical terms, for our experimental setup (Section 4.2) using the Wikipedia dataset as an example, we prepared a training set comprising 6,000 members and 6,000 non-members. With each data item generating $n=25$ neighbours, the total number of data items requiring embedding calculations becomes: $6,000+6,000+150,000+150,000=312,000$
Each of these data items, on average, consists of 1052 characters (variable due to replacements made by the neighbour generation model), leading to a total of $312,000\times 1052=328,224,000$ characters processed. These transactions are sent to the Cohere Embedding V3 model [] for embedding generation. The cost of processing these embeddings is measured in thousands of units. Hence, the total estimated cost for embedding processing is approximately: $32,822\times\$0.001=\$32.82$.

## Appendix C Missing Results

### C.1 TPR for low FPR

Table 3 shows the TPR at 2%, 5% and 10% FPR for different baselines and our proposed SMIA by targeting different models using Wikipedia dataset.

**Table 3: True Positive Rate (TPR) at 2%, 5% and 10% False Positive Rate (FPR) for various MIAs, including our SMIA, across different trained models (Pythia and GPT-Neo) using the Wikipedia dataset.**
|  |  | Pythia-12B | Pythia-6.9B | Pythia-2.7B | GPT-Neo2.7B | GPT-Neo1.3B |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | FPR | WT | WC | WT | WC | WT | WC | WT | WC | WT | WC |
| LOSS [] | 2% | 2.1 | 12.2 | 2.6 | 11.9 | 3.1 | 9.4 | 2.8 | 9.2 | 2.2 | 8.7 |
| 5% | 5.8 | 22.8 | 5.5 | 20.3 | 5.6 | 19.9 | 5.6 | 19.8 | 5.6 | 19.2 |  |
| 10% | 11.1 | 32.1 | 10.9 | 29.8 | 10.2 | 28.6 | 10.0 | 27.6 | 9.8 | 25.6 |  |
| Ref []<br>(Pythia 70M) | 2% | 1.1 | 5.3 | 1.7 | 4.7 | 2.1 | 4.3 | 1.7 | 4.4 | 1.6 | 3.0 |
|  | 5% | 6.7 | 8.7 | 6.2 | 9.1 | 6.4 | 8.4 | 5.8 | 9.0 | 5.6 | 7.8 |
|  | 10% | 11.7 | 18.3 | 12.0 | 16.1 | 11.1 | 14.8 | 11.9 | 13.9 | 12.3 | 13.3 |
| Ref<br>(Pythia 1.4B) | 2% | 2.0 | 5.7 | 2.1 | 6.3 | 2.8 | 4.1 | 2.3 | 2.7 | 1.7 | 0.7 |
|  | 5% | 8.2 | 13.1 | 7.3 | 10.2 | 5.9 | 7.2 | 5.3 | 5.3 | 4.2 | 2.7 |
|  | 10% | 16.5 | 21.3 | 14.7 | 17.7 | 10.7 | 13.7 | 11.4 | 10.8 | 10.0 | 8.2 |
| Zlib [] | 2% | 2.2 | 12.0 | 2.1 | 10.5 | 1.9 | 9.2 | 2.1 | 10.0 | 2.0 | 9.9 |
|  | 5% | 5.5 | 22.7 | 5.9 | 22.4 | 5.2 | 19.8 | 6.0 | 18.7 | 5.9 | 16.1 |
|  | 10% | 10.4 | 33.5 | 10.2 | 30.5 | 9.1 | 29.2 | 10.0 | 28.1 | 9.9 | 28.3 |
| Nei [] | 2% | 1.3 | 11.2 | 1.4 | 9.3 | 1.7 | 9.0 | 1.5 | 10.1 | 2.1 | 8.9 |
|  | 5% | 4.3 | 19.2 | 4.5 | 18.9 | 5.2 | 18.8 | 5.3 | 18.2 | 5.5 | 16.1 |
|  | 10% | 10.4 | 32.0 | 10.5 | 29.9 | 10.0 | 27.6 | 10.4 | 27.4 | 11.0 | 28.3 |
| Min-K []<br>($K=10\%$) | 2% | 1.8 | 17.9 | 1.9 | 18.3 | 1.9 | 14.1 | 1.6 | 15.9 | 1.4 | 14.2 |
|  | 5% | 5.6 | 28.9 | 6.0 | 26.0 | 6.7 | 23.9 | 5.7 | 22.4 | 6.5 | 20.0 |
|  | 10% | 13.3 | 41.7 | 13.7 | 36.7 | 12.3 | 33.8 | 13.2 | 31.7 | 11.6 | 28.9 |
| Min-K<br>($K=20\%$) | 2% | 1.8 | 14.7 | 2.1 | 16.7 | 2.7 | 14.1 | 2.4 | 14.3 | 2.0 | 13.9 |
|  | 5% | 5.6 | 27.0 | 5.4 | 25.9 | 5.8 | 23.6 | 5.7 | 23.3 | 5.9 | 21.9 |
|  | 10% | 12.7 | 38.3 | 12.6 | 36.5 | 12.0 | 31.9 | 12.4 | 32.2 | 11.4 | 28.4 |
| Min-K<br>($K=30\%$) | 2% | 2.1 | 14.2 | 2.5 | 14.6 | 2.8 | 12.1 | 2.5 | 12.1 | 2.1 | 11.0 |
|  | 5% | 5.8 | 28.4 | 5.5 | 25.3 | 5.5 | 22.2 | 5.5 | 22.3 | 5.4 | 19.7 |
|  | 10% | 12.6 | 37.7 | 12.5 | 33.2 | 12.4 | 32.9 | 12.1 | 30.6 | 11.3 | 28.5 |
| Min-K++ []<br>($K=10\%$) | 2% | 3.0 | 19.4 | 2.2 | 13.6 | 2.5 | 12.4 | 2.8 | 12.3 | 2.2 | 10.0 |
|  | 5% | 6.1 | 29.6 | 6.8 | 26.0 | 6.6 | 23.9 | 5.3 | 22.0 | 5.2 | 19.1 |
|  | 10% | 12.6 | 40.6 | 12.6 | 40.4 | 11.9 | 32.2 | 10.2 | 33.2 | 11.4 | 28.3 |
| Min-K++<br>($K=20\%$) | 2% | 2.8 | 21.2 | 2.0 | 17.4 | 2.3 | 16.7 | 2.7 | 12.9 | 2.0 | 11.7 |
|  | 5% | 5.5 | 30.5 | 6.0 | 27.7 | 5.6 | 23.7 | 6.5 | 24.1 | 5.3 | 20.1 |
|  | 10% | 12.2 | 43.7 | 12.0 | 38.7 | 12.2 | 34.8 | 10.2 | 34.0 | 10.9 | 29.6 |
| Min-K++<br>($K=30\%$) | 2% | 2.7 | 20.9 | 2.0 | 17.7 | 2.2 | 16.9 | 2.7 | 12.8 | 2.1 | 11.3 |
|  | 5% | 5.4 | 31.4 | 5.8 | 27.5 | 5.7 | 24.8 | 6.4 | 24.6 | 4.7 | 20.2 |
|  | 10% | 12.2 | 43.9 | 12.5 | 38.3 | 11.5 | 35.4 | 10.5 | 34.4 | 11.3 | 30.5 |
| Our SMIA | 2% | 3.8 | 46.2 | 3.1 | 41.6 | 2.4 | 35.1 | 1.8 | 32.9 | 2.8 | 25.2 |
|  | 5% | 10.4 | 66.0 | 8.3 | 60.2 | 6.8 | 52.5 | 6.3 | 49.8 | 7.2 | 45.4 |
|  | 10% | 20.6 | 79.3 | 18.1 | 75.4 | 15.0 | 67.6 | 14.4 | 67.9 | 14.9 | 60.5 |

### C.2 Effect of Deduplication

Table 4 shows the AUC-ROC metric comparing different MIAs and our SMIA for deduped pythia models using Wikipedia dataset.

**Table 4: AUC-ROC performance metrics for various MIAs, including our SMIA, across different trained deduped Pythia models using the Wikipedia dataset.**
|  | Pythia-12B-Deduped | Pythia-6.9B-Deduped | Pythia-2.7B-Deduped |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | WT | WC | WT | WC | WT | WC |
| LOSS [] | 53.39 | 65.19 | 53.08 | 61.58 | 52.78 | 62.93 |
| Ref []<br>(Pythia 70M) | 50.24 | 54.85 | 49.71 | 53.89 | 48.92 | 51.90 |
| Ref<br>(Pythia 1.4B) | 51.62 | 56.99 | 50.32 | 54.70 | 47.40 | 47.09 |
| Zlib [] | 52.81 | 64.31 | 52.55 | 63.64 | 52.23 | 62.08 |
| Nei [] | 53.93 | 69.93 | 53.63 | 69.17 | 53.07 | 68.16 |
| Min-K []<br>($K=10\%$) | 54.40 | 72.91 | 53.71 | 71.40 | 53.62 | 69.39 |
| Min-K<br>($K=20\%$) | 54.25 | 70.83 | 53.77 | 69.80 | 53.41 | 67.95 |
| Min-K<br>($K=30\%$) | 54.0 | 69.30 | 53.59 | 68.24 | 53.17 | 66.50 |
| Min-K++ []<br>($K=10\%$) | 52.84 | 74.48 | 52.13 | 72.92 | 51.32 | 69.90 |
| Min-K++<br>($K=20\%$) | 53.66 | 75.19 | 53.01 | 73.28 | 51.95 | 70.19 |
| Min-K++<br>($K=30\%$) | 54.00 | 74.62 | 53.28 | 72.74 | 52.11 | 69.49 |
| Our SMIA | 61.15 | 90.72 | 60.01 | 88.43 | 58.49 | 84.39 |

### C.3 Effecft of Number of Neighbours

Table 5 presents the performance of our SMIA when varying the number of neighbors used during inference. The results indicate that a larger number of neighbors generally improves SMIA’s performance. However, we have chosen to use 25 neighbors in our experiments, as increasing this number further leads to additional computational demands without a corresponding improvement in performance.

**Table 5: AUC-ROC performance metrics of SMIA when different number of neighbors used in inference.**
|  |  | Pythia-12B | Pythia-6.9B | GPT-Neo-2.7B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $n_{inf}$ | WT | WC | WT | WC | WT | WC |
| SMIA | 1 | 55.26 | 61.01 | 53.84 | 60.23 | 51.92 | 58.58 |
| 2 | 58.48 | 70.60 | 56.41 | 68.82 | 53.26 | 66.18 |  |
| 5 | 61.27 | 78.06 | 59.08 | 76.48 | 57.17 | 74.63 |  |
| 15 | 65.63 | 87.15 | 62.62 | 85.46 | 58.86 | 82.60 |  |
| 25 | 67.39 | 93.35 | 63.64 | 92.11 | 59.71 | 89.59 |  |

### C.4 Effect of Size of Training Dataset

Figure 3 illustrates the effect of using larger training datasets on the validation loss of the SMIA over 20 epochs. This figure displays the performance of the SMIA model on a validation dataset consisting of 1,000 members and 1,000 non-members, which are existing in the original Wikipedia portion of the Pile dataset (train and validation splits). In our experiments, we tested four different training sizes: 1,000 members + 1,000 non-members, 2,000 members + 2,000 non-members, 4,000 members + 4,000 non-members, and 6,000 members + 6,000 non-members. The results indicate that larger training datasets generally yield lower validation losses for the SMIA model. However, larger datasets require more computational effort as each member and non-member sample needs $n$ neighbors generated, followed by the calculation of embedding vectors and loss values for each neighbor. Due to computational resource limitations, we use a training size of 6,000 members + 6,000 non-members for all our experiments.

Figure: Figure 3: Effect of different training size on the validation loss of SMIA for 20 epochs.
Refer to caption: extracted/5668247/Images/loss_plot_Pythia12b_WT.png

### C.5 Similarity Scores of Neighbours

Figure 4 shows histogram of the similarity scores between members, non-members, and their 25 generated neighbors. These similarity scores are calculated using cosine similarity between the embedding vector of the original text and the embedding vectors of the neighbors. The dataset comprises 6,000 members and 6,000 non-members, resulting in 150,000 neighbors for each group. The histogram reveals that while most neighbors exhibit high similarity, there is a range of variability. Notably, even neighbors with lower similarity scores, such as around 70%, provide valuable data for training our SMIA. This diversity enables SMIA to more effectively distinguish membership under varying degrees of textual context changes.

Figure: (a) Members training data
Refer to caption: extracted/5668247/Images/histogram_member_training.png

## Appendix D More Details about Experiment Setup

### D.1 SMIA Model Architecture

Table 6 shows the SMIA architecture with its layer sizes that we used in our experiments.

**Table 6: SMIA architecture with layer sizes**
| Name | Layers | Details |
| --- | --- | --- |
| Loss Component | 1 Fully Connected | FC(1,512)<br>Dropout (0.2)<br>ReLU activation |
| Embedding Component | 1 Fully Connected | FC(1024,512)<br>Dropout (0.2)<br>ReLU activation |
| Attack Encoding | 6 Fully Connected | FC(1024, 512), FC(512, 256),<br>FC(256, 128), FC(128, 64),<br>FC(64, 32) , FC(32, 1)<br>Dropout (0.2)<br>ReLU activation<br>Sigmoid |

### D.2 Metrics

In our experiments, we employ following privacy metrics to evaluate the performance of our attacks:

(1) Attack ROC curves:
The Receiver Operating Characteristic (ROC) curve illustrates the trade-off between the True Positive Rate (TPR) and the False Positive Rate (FPR) for the attacks. The FPR measures the proportion of non-member samples that are incorrectly classified as members, while the TPR represents the proportion of member samples that are correctly identified as members. We report the Area Under the ROC Curve (AUC-ROC) as an aggregate metric to assess the overall success of the attacks. AU-ROC is a threshold-independent metric, and it shows the probability that a positive instance (member) has higher score than a negative instance (non-member).

(2) Attack TPR at low FPR:
This metric is crucial for determining the effectiveness of an attack at confidently identifying members of the training dataset without falsely classifying non-members as members. We focus on low FPR thresholds, specifically 2%, 5%, and 10%. For instance, the TPR at an FPR of 2% is calculated by setting the detection threshold so that only 2% of non-member samples are predicted as members.

### D.3 Example of Modified text

In Section 5, we introduce a modified evaluation setting where the member dataset undergoes various alterations. Figure 5 illustrates an example of a Wikipedia member sample undergoing different modifications: (a) shows the original sample, (b) shows a neighbor of the original created by replacing some words with outputs from a masking model, (c) shows modified sample by deleting a random word, (d) shows the modified sample by duplicating one word, and (e) shows the modified sample after adding one word using a T5 model.

Figure: (a) Input sample (Original - with no modifiation)
Refer to caption: extracted/5668247/Images/Orig.png

### D.4 SMIA Hyperparameters

To construct our neighbor datasets, we generate $n=25$ neighbors for each data point. Table 6 details the architecture of the SMIA model used across all experiments. We employ the Adam optimizer to train the network on our training data over 20 epochs. The batch size is set to 4, meaning each batch contains neighbors of 2 members and 2 non-members, totaling 50 neighbors for members and 50 neighbors for non-members, thus including 100 neighbors per batch.
For regular experiments, we use a learning rate of $5\times 10^{-6}$. However, for modified evaluations, which include duplication, addition, and deletion scenarios, we adjust the learning rate to $1\times 10^{-6}$.
In all of our experiments, we report the AUC-ROC or TPR of the epoch that results in lowest loss on validation dataset.

### D.5 Compute Resources

For the majority of our experiments, we utilize a single H100 GPU with one core. It is important to note that we do not train or fine-tune any LLMs during our experiments; we operate in inference mode using pre-trained models such as the T5 masking model and various models from the Pythia family. Generating $n=25$ neighbors for a dataset of 1,000 texts required approximately 16 hours of compute time.
For the task of calculating embedding vectors, we employed the Cohere Embedding V3 model, which is provided as a cloud service. The computation of loss values for the target model was also minimal, taking only a few minutes for the a dataset of 1,000 examples. Finally, training the SMIA model was notably rapid, owing to its relatively small size of only a few million parameters. The entire training process, after having all the input features for training data, was completed in less than 10 minutes over 20 epochs.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - How does the member data of the embedding model impact the performance of the SMIA
soundness: 2
strengths: 1. The paper does a good job at explaining their methodology, which considers the fact that learning a classifier on top of LLM behaviors can allow learning membership signals well.
2. The work demonstrates strong gains over past MIAs across multiple benchmark datasets.
summary: This paper studies the problem of membership inference in large language models by proposing "semantic MIAs". The key idea is to use various perturbations to the input text, then analyze the behavior of a model on these points, train an auxiliary model on these behaviors, and finally use that model to predict membership. The authors show that SMIAs outperform prior MIAs on different benchmarks.
weaknesses: There are various works at this point in literature which have argued that "WC" based assessment of MIAs when data is split across a cut-off date, is not sound. This work bases many of their gains on that setting.

[1] Do Membership Inference Attacks Work on Large Language Models? https://arxiv.org/abs/2402.07841.   
[2] LLM Dataset Inference: Did you train on my dataset? https://arxiv.org/abs/2406.06443.   
[3] Blind Baselines Beat Membership Inference Attacks for Foundation Models. https://arxiv.org/abs/2406.16201.    

Key weaknesses:
1. The method should be tested across all 20 train-test splits of Pile. Paper [2] on LLM Dataset Inference particularly shows how most MIAs perform well only on a few datasets
2. The blind baseline of n-gram is not considered in this work (paper [3]). This is an important comparison to understand if the gains in this work are meaningful.
3. Given that the goal of MI in LLMs is hard, I would love to see experiments on dataset inference.
4. The idea of learning an auxiliary classifier on top of the membership signal from neighbours seems to have been explored in [2]

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The scalability of SMIA deserves further exploration - how would the method perform with larger training datasets and more complex model architectures? The current evaluation uses 6,000 members and non-members each, which may not fully reflect real-world scenarios.
2. The choice of neural network architecture for the SMIA model itself appears somewhat arbitrary. Have the authors explored alternative architectures that might improve performance or efficiency? Additionally, how sensitive is the method to the quality and dimensionality of the semantic embeddings?
3.  How might defenders mitigate against SMIA?
soundness: 3
strengths: The authors present a well-designed pipeline that combines neighbour generation using masked language models, semantic embedding analysis via the Cohere model, and neural network classification. This comprehensive approach allows SMIA to detect both exact matches and semantically similar content, representing a significant advancement over existing methods.
The experimental evaluation is particularly thorough, examining performance across different model sizes, architectures, and datasets. The authors carefully consider the impact of dataset distribution by using two different non-member datasets - one from the same distribution (Wikipedia Test) and another from a different time period (Wikipedia Cutoff). This reveals important insights about how data distribution affects attack performance. Additionally, the analysis of modified text scenarios (through word additions, deletions, and duplications) provides practical insights into the method's
summary: This paper introduces SMIA (Semantic Membership Inference Attack), an innovative approach to membership inference attacks against LLMs that leverages semantic analysis. The key insight is that LLMs exhibit distinct behavioral patterns when processing semantically similar variants of their training data versus unseen data. SMIA capitalizes on this by generating semantic neighbours for input texts, analyzing how the target model responds to these variations, and training a neural network to detect membership based on these response patterns. The authors evaluate SMIA extensively on the Pythia and GPT-Neo model families using Wikipedia and MIMIR datasets, demonstrating significant improvements over existing approaches - notably achieving an AUC-ROC of 67.39% on Pythia-12B compared to previous best of 58.90%.
weaknesses: 1. The choice of key hyperparameters, particularly the use of 25 neighbours, lacks thorough clarification. While Table 7 shows performance improvements with increasing neighbour count, there's no clear analysis of the trade-off between computational cost and performance gain. The paper should examine the diminishing returns beyond 25 neighbours and justify why this specific number optimally balances effectiveness and efficiency.
2. A concerning weakness emerges in the method's inconsistent performance across different types of text modifications. As shown in Table 3, while SMIA achieves an AUC-ROC of 62.47% for word deletions on Pythia-12B (WT dataset), it drops to 55.13% and 54.19% for word duplications and additions respectively. This significant performance gap suggests an inherent bias in the method's ability to handle different types of text alterations. The authors don't adequately explain this asymmetry or propose potential solutions.
3. Aminor problem is the heavy reliance on the Cohere Embedding model, a third-party service, introduces both a potential point of failure and a privacy concern - users must share their data with an external service to generate embeddings. The authors don't explore alternative embedding approaches or analyze how the choice of embedding model impacts performance. Furthermore, the paper lacks a comprehensive analysis of the cost implications of using such commercial services at scale.
4. The method shows notably better performance with word deletions compared to additions or duplications, as evidenced in Table 3. For instance, with Pythia-12B on the Wikipedia Test dataset, SMIA achieves an AUC-ROC of 62.47% for word deletions but only 54.19% for additions. This behavioral asymmetry suggests fundamental limitations in how SMIA handles different types of semantic modifications, yet the paper offers limited analysis of why this occurs or how it might be addressed. This becomes particularly relevant when considering real-world scenarios where adversaries might deliberately modify texts in ways that exploit these weaknesses.
5. While authors demonstrate the attack's effectiveness, they provide minimal insight into how model owners might protect against such attacks. This omission is particularly notable given the paper's focus on privacy implications and its potential impact on real-world applications of LLMs. A more comprehensive treatment of potential countermeasures would significantly enhance the paper's practical value and provide important context for the security community.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Could the authors provide some examples of grey-box models?

2. How will the performance of SMIA change if changing the embedding model or classification models?

3. What is the general behavior of loss from members? For example, will the loss from members change less than non-members considering neighbours?
soundness: 2
strengths: 1. The proposed method in this paper shows a very good novelty.

2. The proposed method has a good performance considering both AUC and TPR when FPR is low.

3. The proposed method successfully identifies membership even when data undergoes slight modifications due to its design motivation.
summary: The paper introduces the Semantic Membership Inference Attack (SMIA), a novel approach that conducts Membership Inference Attacks by leveraging the semantic content of inputs and their perturbations. SMIA trains a model to detect variations in model behavior between member data members and non-members by analyzing how output probability distributions change with input perturbations. Evaluated on two models and two datasets, SMIA outperforms existing MIA techniques, achieving higher AUC and TPR when FPR is low in detecting membership.
weaknesses: 1.  Though the authors claim the proposed method is designed for grey-box models, there are no direct experiments about the real grey-box models. 

2. The experiments are not very comprehensive on the ablation part. For example, the authors might provide results if changing the embedding model or changing the classification models. 

3. There is no intuitive explanation of the proposed methods like what is the difference between the trends of loss from members and trends of loss from non-members.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. L218: Does the replacement actually happen in terms of words, or should it be tokens?
2. For the neighbor generation, are the resulting neighbors all unique, or could it be that some neighbours are duplicates/equal to the original sample?
3. Were T5 or the Cohere Embedding V3 model trained on some of the evaluation datasets? In particular, could it be that one of those models was only trained on the members for WC but not the non-members?
4. L382--384: Why are the MIMIR splits constrained to only 1000 samples, especially before splitting?
5. L391: The intuition about "Why SMIA Outperforms Other MIAs" mentions that using a neural network is one of the key success factors. However, I think simply using a neural network does not imply a stronger attack; could the authors elaborate this a bit more?
soundness: 2
strengths: The paper introduces SMIA and its intuition very thoroughly. Even though there is no code, the attack's description should be sufficient to reimplement SMIA and reproduce most reported results. I also find it very interesting that directions in embedding space seem to carry non-trivial membership signal and think that the idea of considering not only exact verbatim samples for MI in LLMs is important. Lastly, although MI for LLMs remains a hard task, this paper manages to achieve small but consistent improvements over most existing methods in some settings.
summary: This paper proposes a new membership inference attack (MIA) for LLMs, called SMIA, with the goal to measure memorization. SMIA's intuition is that semantically similar “neighbors” of a sample contain membership signal. In the threat model, the adversary can query the target LLM to obtain losses/log probs, and has access to a set of known training members and non-members.

Concretely, given a target sample, the attack works as follows:

1. Generate "neighbors" of the target via repeated random masking and infilling (via T5 3B).
2. Calculate semantic embeddings of the target sample and all neighbors (via Cohere Embedding v3).
3. For every neighbor, calculate the difference to the target sample in terms of i) loss and ii) semantic embedding (full direction). Use the resulting 1 + 1024 values as the input to a moderately-sized MLP classifier that predicts a membership score.
4. Return "member" if the average membership score over all neighbors is larger than some threshold.

The paper evaluates SMIA for Pythia and GPT-NEO models on Wikipedia and MIMIR data. All results are reported in terms of AU-ROC; a few results also in terms of TPR @ FPR. For Wikipedia, the evaluation dataset uses Wikipedia text in the Pile training split as members, and one of two types of non-members: *WT* uses Wikipedia text from the Pile test split (before March 2020); *WC* uses Wikipedia pages published after August 2023 (not in the Pile training split). For MIMIR, the evaluation uses the highest n-gram overlap subsplit with various splits.
weaknesses: **Evaluation datasets measure distribution shift, not MI**: Some evaluation datasets in this paper might be deeply flawed, in that they do not measure MI performance but just distribution shifts. For Wikipedia data, SMIA only yields a big improvement on the WC split, which uses temporally shifted Wikipedia articles as non-members. However, [Das et al., 2024](https://arxiv.org/abs/2406.16201) show that simple "blind" baselines that classify membership just based on the data (without any signal from the target model) already achieve very high MIA success.
Hence, SMIA could potentially just be a good distribution shift detector and the evaluation misleading. This is particularly concerning since the membership classifier is a NN, which might be particularly prone to overfitting on spurious correlations in the evaluation data. One way to alleviate this concern could be a blind baseline that uses inputs independent of the target model (e.g., random or constant loss).
For MIMIR, this is less of a concern. However, there SMIA only achieves significantly better MIA success for GitHub data and Wikipedia data for one model; existing attacks perform similarly or better on all other subsplits (in terms of TPR @ 2% FPR).

**AU-ROC alone is insufficient to evaluate MIAs**: It has long been known that only reporting AU-ROC in the evaluation of MIAs is deeply flawed and misleading [(Carlini et al., 2021)](https://arxiv.org/abs/2112.03570). Yet, this paper uses only AU-ROC as its main metric, and only reports TPR@low FPR values for a very limited subset of results in App. C.1. In particular, the effects of deduplication (Table 6), varying number of neighbors (Table 7), and slight modifications (Table 3) are only reported in terms of AU-ROC, and can hence not be judged soundly. The paper should report TPR @ low FPR (e.g., 2% or 1%) together with full ROC curves as the main metric, and defer AU-ROC metrics to the appendix (or omit them entirely).
NB: Consequently, while I looked at all tables in the main matter, I can only judge SMIA's performance by the values on Tables 4 and 5 (App. C.1).

**Complex approach without ablation**: SMIA is a complex approach that relies on many moving parts; yet, there is very little investigation about why every part is necessary. Hence, this paper requires a more thorough ablation study (e.g., what if the classifier only acts on losses/embedding directions, or what if the NN is replaced by a linear classifier?).

**Other minor points/feedback**:
1. SMIA requires a known member and non-member subset from the training data distribution. This is a non-trivial assumption that requires some discussion.
2. The lowest reported FPR threshold is 2%. This is still relatively high; it would be interesting to also report the attack's performance at lower FPRs such as 1% and 0.1% if feasible.
3. I found two things in Figure 2 slightly confusing: i) the axis label, and ii) that all neighbors are larger on one axis; I understood that the similarity is somewhat independent of the parameter space? Also, e.g., $x_1^m$ seems to be further from $x^m$ than $x^n$. Right now, I think the figure does not carry much information and could be confusing, hence it might be dropped. Finally, there is a typo in the caption (`taregt`).
4. Sec. 5.1 mostly repeats the results displayed in Table 1. I think the space could be used better by dropping this repetition and instead show some of the interesting studies currently in App. C.

Meta-review: This paper introduces a novel approach to membership inference attacks against Large Language Models, centered around how these models exhibit distinct behavioral patterns when processing semantically similar variants of their training data versus unseen data. The authors develop a method that analyzes model responses to semantic neighbors of input texts, demonstrating improvements over existing approaches.

The technical approach combines neighbor generation using masked language models, semantic embedding analysis, and neural network classification. This allows the method to detect both exact matches and semantically similar content, representing an advancement in the field. The experimental evaluation examines performance across different model sizes, architectures, and datasets, while considering dataset distribution effects through different non-member datasets. The analysis of modified text scenarios through word additions, deletions, and duplications provides valuable practical insights into the method's robustness.

However, the submission has several significant weaknesses. A fundamental concern is that the evaluation datasets might be measuring distribution shifts rather than true membership inference performance. Recent work demonstrates that simple baselines can achieve high success on temporally shifted articles, casting doubt on the reported improvements. The paper's heavy reliance on a single metric is problematic, and the limited reporting of other metrics makes it difficult to assess real-world effectiveness. Additionally, while the method involves multiple complex components, there's insufficient investigation of their necessity through ablation studies.

The paper omits several important comparisons that would strengthen its evaluation, including performance analysis across train-test splits, comparison against baseline approaches, and exploration of alternative embedding approaches. The selection of key parameters lacks thorough justification and analysis of trade-offs between computational cost and performance gain.

While the technical approach is novel and well-presented, the fundamental concerns about evaluation methodology cast serious doubt on the paper's main claims. The possibility that the method is primarily detecting distribution shift rather than true membership undermines its purported advances. To strengthen this work, the authors should expand the evaluation to include comprehensive comparisons against baselines, incorporate additional metrics, conduct thorough ablation studies, and provide clearer justification for design choices and parameter selection. These improvements would help establish the method as a meaningful advancement in membership inference attacks against language models.

Decision: Reject