# LUMIA: linear probing for unimodal and multiModal membership inference attacks leveraging internal LLM states

Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux, Joaquin Garcia-Alfaro

<figure><img src="image_1.png" /></figure>

## To cite this version:

Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux, Joaquin Garcia-Alfaro. LUMIA: linear probing for unimodal and multiModal membership inference attacks leveraging internal LLM states. 30th European Symposium on Research in Computer Security (ESORICS), Sep 2025, Toulouse, France. pp.186-206, 10.48550/arXiv.2411.19876 . hal-05202765

HAL Id: hal-05202765

https://inria.hal.science/hal-05202765v1

Submitted on 7 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LUMIA: Linear probing for Unimodal and MultiModal Membership Inference Attacks leveraging internal LLM states

Luis Ibanez-Lissen¹, Lorena Gonzalez-Manzano¹, Jose Maria de Fuentes¹,²,
Nicolas Anciaux², and Joaquin Garcia-Alfaro³

¹ Universidad Carlos III de Madrid, Leganes, Spain

² Inria, UVSQ, U. Paris-Saclay, Palaiseau, France

³ SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France

**Abstract.** Large Language Models (LLMs) are increasingly used in a variety of applications. Concerns around membership inference have grown in parallel. Previous efforts focus on black-to-grey-box models, thus neglecting the potential benefit from internal LLM information. To address this problem, we propose the use of Linear Probes (LPs) as a method to detect Membership Inference Attacks (MIAs) by examining internal activations of LLMs. Our approach, dubbed LUMIA, applies LPs layer-by-layer to get fine-grained data on the model inner workings. We test this method across several model architectures, sizes and datasets, including unimodal and multimodal tasks. In unimodal MIA, LUMIA achieves an average gain of 15.71% in Area Under the Curve (AUC) over previous techniques. Remarkably, LUMIA reaches AUC greater than 60% in 65.33% of cases — an increase of 46.80% against the state of the art. Furthermore, our approach reveals key insights, such as the model layers where MIAs are most detectable. In multimodal models, LPs indicate that visual inputs can significantly contribute to detect MIAs — AUC greater than 60% is reached in 85.90% of the experiments.

**Keywords:** Large Language Models · Large Multimodal Models · Membership Inference Attacks · Linear Probes.

## 1 Introduction

Membership Inference Attacks (MIAs) aim to determine whether specific data samples (such as sensitive or copyrighted items) were included in the training set of a Large Language Model (LLM) [32].

While some researchers argue that it is impossible to prove that MIA are feasible on LLMs [34], others try to find methods that maximize the Area Under the Curve (AUC) to get better performance. These efforts tackle the membership inference problem from a black-box perspective (e.g., work in [13]) or a grey-box one. In the latter, the idea is to set a threshold on the model output that determines whether a sample was part of the training data [23], [5], [25].
**Motivation.** The need to create fair and transparent auditing processes for AI systems calls for adopting white-box approaches [32,6]. With regards to MIA, we hypothesize that the internal model data from member and non-member samples may reveal distinguishing patterns. Specifically, we expect that data corresponding to previously seen (member) texts or images may behave differently from unseen (non-member) data. Only Liu et al. [17] have approached the membership inference problem in LLMs from this perspective. They apply Linear Probes (LPs) [2] on model activations of a single layer. A number of limitations in Liu et al. [17] are tackled in our work. First, their approach involves fine-tuning the models to ensure that members have been seen. Therefore, results are biased since samples already used in the *pretraining* phase are seen twice. Second, such a fine-tuning is used to create proxy models for the experimentation, but there is no guarantee on the functional equivalence of the original and the proxy model. Thirdly, they use an input prompt, which simplifies the problem as it narrows the search space. Lastly, they are limited to text-based MIAs, thus excluding multimodal models.

**Contribution.** This paper provides an insightful analysis of the effectiveness on using internal model data for MIA assessment. The approach, dubbed LUMIA<fnref n="4" />, uses internal activations of each model layer. LUMIA is directly applied on real-world models and datasets, thus characterizing the ability of LPs to succeed depending on the model and the dataset nature. For completeness, we consider two types of biases, in the same line as Duan et al. [9] and Das et al. [7]. As no sample prompts are used and LLMs are requested to perform a variety of tasks, our results are easily generalizable. Interestingly, experiments are not only text-based MIAs, but also multimodal. While the concurrent work by Li et al. [16] has proposed a benchmark for multimodal MIAs, they depend on the model output. Therefore, they limit themselves to tasks that generate long texts. On the contrary, LUMIA is not constrained by the LLM output.

The research question at stake is — *To what extent can internal activations of LLMs be used to improve and assess membership inference?* In this vein, the list of contributions is as follows:

- We provide a comprehensive study on the suitability of internal activations for assessing MIAs by using linear probes, showing their ability to outperform state-of-the-art contributions.

- We explore for the first time the impact of the LLM size, the dataset nature, bias, and the impact of using deduplicated model versions.

- We analyse the problem of MIAs in multimodal LLMs. We consider a variety of LLM tasks, which has never been tackled to the best of authors knowledge.

- Our experimental results are based on 14 textual, 7 multimodal datasets, and 3 model families, involving 15 LLM configurations. We release our experimental materials to foster further research in this direction<fnref n="5" />.

<footnote n="4">⁴ Latin term derived from *light*, representing the value of looking inside the model to ascertain how MIAs impact the inner model working.</footnote>

<footnote n="5">⁵ Available online at https://github.com/Luisibear98/LUMIA.</footnote>
The paper is structured as follows. Section 2 provides background and prelim-
inaries. Section 3 describes the foundations of LUMIA. Section 4 covers all the
experimentation, which is later analysed in Section 5. Section 6 shows the related
work. Section 7 concludes the paper and points out future work directions.

## 2 Background

Background on Large Language Models (LLMs) is presented in Section 2.1.
Preliminaries on Linear Probes (LPs) are provided in Section 2.2.

2.1 LLMs. Internal data, input data and biases

LLMs are transformer-based neural networks [28], consisting of tens to hundreds
of billions of parameters, and pretrained on vast amounts of data. Notable ex-
amples include models like LLaMA [10] and GPT-4 [1]. For the interest of this
paper, the information processed and stored by the neural network during train-
ing and inference is at stake. In the transformer model [28], internal model data
specifically refers to the activations generated at the output of each transformer
block during the feed-forward pass. These activations represent the intermediate
state of the model as it processes input data layer by layer.

A key factor in training these models is ensuring data quality, especially
when scraping large corpora. One way to measure data quality is by identifying
biases. We hereby describe the two major types of bias [11]. One way is analyzing
N-grams, which are sequences of *n* consecutive elements (e.g., words in natural
language processing). Under a Membership Inference Attack (MIA) context, an
N-gram overlap indicates the percentage of N-grams in a non-member sample
that also appear in at least one member sample. Thus, higher overlaps imply
more similarity across samples [8]. In our work, we refer to this potential source
of bias as N-gram Bias (NGB).

Another form of bias in MIA arises from dynamic changes in data distribution
over time. Thus, members are typically selected before a given date, and non-
members are those after that deadline. Das *et al.* [7] identified this issue, which
we refer to as Temporal Bias (TB).

## 2.2 Linear classifier probes

Linear Probes (LP) are classifiers (such as Multi-Layer Perceptrons, MLPs) that
contribute to deep learning models explainability efforts by providing insights
into how the model processes information internally [2].

LPs are used to make predictions over the hidden states of the models, trying
to predict or identify if some specific information is correctly represented within
them. For LLMs, an LP classifier is typically placed after each layer of the
network and takes the hidden states as input X to predict a concept Y.

3 LUMIA

This section provides the foundations of our proposal. Section 3.1 covers the
formulation of the problem and Section 3.2 describes the approach.
## 3.1 Problem formulation

LUMIA (Linear probe-based Utilization of Model Internal Activations), leverages Linear Probes (LPs), lightweight classifiers trained directly on internal activations, i.e., the hidden states generated at each layer during inference. LPs offer an interpretable and efficient means to assess the distribution of membership information across the layers of the model. Specifically, we formalize the problem of membership inference using internal activations as follows:

- **Input:** A pre-trained model *M*, and a set of labeled samples $S = \{(x_i, y_i)\}$, where $x_i=\{t_1, \cdots, t_k\}$ is the input (text or multimodal text-image pair) formed by minimal data units called tokens $t_i$; and $y_i \in \{0, 1\}$ indicates membership status (1 if the sample is a member, 0 otherwise).

- **Objective:** Train a linear probe $MLP_l$ for each layer $l$ of the model *M* to classify membership status based on the internal activation $A_l(x_i)$, where $A_l(x_i)$ represents the average activation vector at layer $l$ for all tokens $t_i$ within input $x_i$.

- **Metric:** Evaluate $P_l$ using metrics such as Area Under the Curve (AUC) for each layer $l$, and identify the layer $l^*$ where membership information is most detectable (i.e., where $P_{l^*}$ achieves the highest AUC).

The aforementioned formulation enables us to explore:

1. The distribution and concentration of membership information across different layers of the LLMs.

2. The comparative effectiveness of LP-based MIAs versus traditional output-based methods.

3. The influence of factors, such as model architecture, size, dataset characteristics, and multimodal inputs on membership inference success.

By rigorously applying this approach, LUMIA aims to advance the understanding of membership inference in LLMs. It also establishes internal activations as a versatile and powerful tool for MIA assessment.

## 3.2 Description

Depicted in Figure 1, once an LLM is trained with member and non-member samples, internal activations at each layer are provided as inputs of an LP. LPs are implemented through MLPs (cf. Section 2.2), whose output is an AUC. LUMIA retrieves the AUC per layer as well as the layer $l^*$ in which the maximum AUC is achieved.

To ensure the robustness and generalization of this process, unimodal (D) and multimodal (MD) datasets are applied over multiple LLM (e.g., LLMj), where D provides answers to a general instruction prompt of any type (e.g., make a summary). More specifically, datasets in D are used to test the improvement of LUMIA over N-gram Bias (NGB), thus studying the benefits of using LPs for MIA attacks with different levels of overlapping among inputs. Datasets in D are also applied to study the effect of Temporal Bias (TB).
<figure><img src="image_3.png" /><figcaption>Fig. 1. System overview.</figcaption></figure>

Given the large variety of LLMs, MD allows analysing MIA attacks once samples composed of image and text are provided as input. A couple of ways to handle multimodality are devised — training an LLM just with images or with images and text, computing LPs over the resulting activations $A_l(x_i)$.

**Extracting activation data** Activations $A_l(x_i)$, per layer, capture values for all tokens. First, samples $x_i$ from members and non-members are preprocessed by cropping the text to fit the maximum context length $n$ of the target LLM. Next, a forward pass [22] is performed for each sample, during which the hooks capture the activations $a_i(t_i)$ for each token $t_i$ at layer $l$. Thus, their average is computed as shown in Equation (1):

$$A_l(x_i) = \frac{1}{n} \sum_{j=1}^{n} a_l(t_j) \qquad (1)$$

For unimodal cases, hooks are placed after each transformer layer. In multimodal cases, hooks are positioned after the layers of both the text and visual models.

# 4 Experimental design

This section describes the design of the experiments to assess LUMIA. Section 4.1 explains the models and datasets. Section 4.2 describes the assessment metrics. Section 4.3 introduces the experimental settings.

## 4.1 Models, datasets and tasks

**Unimodal LLMs.** Several models of different sizes are chosen in our study. The Pythia model family [3], trained on the Pile dataset [12], with 160M, 1.4B, 2.8B,
and 12B of parameters was selected in both their non-deduplicated and deduplicated versions for comparison purposes. Additionally, the GPT-Neo family is also evaluated with 140M, 1.3B, and 2.7B parameters variants. These models are chosen (i) to compare them to other proposals and (ii) because data used for pre-training them is known, being essential to deal with MIA attacks.

**Unimodal task and datasets.** In line with state-of-the-art literature, the LLM processes text to carry out a text Masking causal modeling task. In this vein, datasets used to test the approach are *WikiMIA* [25], *ArXiv-MIA* [17], *Temporal ArXiv/wiki* [9], *ArXiv-1-month* [21], *Gutenberg* [21] and *Mimir* [9]. Note that they have been selected for the sake of comparability with previous works [25,17,9,21]. All datasets, except for *Mimir*, have already been shown to suffer from TB [7]. Conversely, *Mimir* suffers from NGB.

**Multimodal LLMs.** For the analysis of multimodality, the latest version of the *LLava-OneVision model* [15] is applied with 0.5B and 7.6B parameters. These models are chosen since (i) the data used during its pre-training and fine-tuning is known and (ii) due to available computational resources.

**Multimodal tasks and datasets.** Linked to this model, *OneVision-Data*<fnref n="6" /> dataset is applied. It is composed of a wide range of datasets used to train a multimodal model for multitasking. From this collection, we generate member and non-member samples from datasets that originally provided distinct training, validation, and testing splits. From all datasets, the following are selected: *Textcaps* [27], *MathV360k* [26], *AOK* [24], *ChartQA* [20], *ScienceQA* [18], *IconQA* [19] and *Magpie* [31]. They encompass all the modalities and categories of tasks the model can accomplish, such as General resolution, Doc/Chart/Screen solving, Math/Reasoning, OCR, and Language tasks.

## 4.2 Metrics. Performance and bias

In line with Duan *et al.* [9], Shi *et al.* [25], and Carlini *et al.* [4] (and for the sake of comparison), the effectiveness of the detection method is measured with the following metrics:

- **Area Under the ROC Curve (AUC).** It measures the ability of a classifier to correctly determine a class, 0 or 1, by comparing the true positive rate (power) against the false positive rate (error) across various thresholds. A value closer to 1 means better performance. In line with [8], MIA is considered successful when AUC is greater than 60%.

AUC is then computed to compare LUMIA against state-of-the-art MIAs, namely *Loss* [33], *Reference-based* [23], *Zlib Entropy* [5] and *Min-k% Probability* [25].

<footnote n="6">⁶ https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data, last accessed on January 10, 2025.</footnote>
Concerning text-based bias, only NGB can be measured. In this regard, we use the n-gram length $N$ and the percentage of overlap $P$, in line with [8].

In case of multimodality, there are no standard widely accepted metrics in this regard. Thus, the following image signal processing techniques are computed in members and non members [30,29]:

- **Average Hash Variation (HV)**. Images are converted to grey scale and the average value of the pixels is computed. Finally a hash is applied to compare similarity across samples.

- **Average Structural Similarity Index Measure (SSIM)**. It refers to the perceptual similarity between images by considering their luminance, contrast, and structural content.

## 4.3 Experimental settings

Training was conducted on two NVIDIA consumer GPUs, a RTX 4090 and a RTX 4080, using a mix of the Pytorch and Tensorflow frameworks, complemented with the Hugging Face library$^7$. For both training and validation, all datasets were randomly split in an 80%-20% balancing both classes (members and non-members) and repeating three times the experiments with different samples. The average of all executions was then computed. MLP models were trained with a learning rate of $1e^{-3}$, using the Adam optimizer [14] over 100 epochs, with early stops and dropout regularization.

For comparison with related work [25,17,9,21], we extracted 1000 members and 1000 non-members per dataset, except for WikiMIA and ArXiv-MIA. For such cases, we used the provided data: 250 and 400 samples per class, respectively. In the case of multimodality, we also created a joint subset extracting 100 samples from all datasets, forming a total of 700 members and 700 non-members.

For multimodal configurations, we selected members and non-members using the IDs provided on the original datasets, to avoid contamination between training, validation and testing sets.

Lastly, and since the original Magpie setup does not provide image inputs, but the model requires both text and image modalities, we paired each text input with a black image, to create the necessary input pairs.

# 5 Experimental results

This section presents the results of LUMIA. Firstly, how LP outperforms MIA attacks is analysed (Section 5.1), followed by a study of the impact of target model size (Section 5.2). The influence of potential bias is then explored (Section 5.3), along with the role of dataset nature (Section 5.4), the effects of data deduplication (Section 5.5), and the significance of layer depth (Section 5.6).

Results are presented in the form of tables which are used across all sections. For the sake of clarity, Tables 1 and 2 highlight LUMIA values where AUC is greater than 60%, while Table 3 highlights the best values for each setting.

<footnote n="7">7 https://huggingface.co, last accessed on January 10, 2025.</footnote>
## 5.1 Overall effectiveness

**Unimodal.** Tables 1 and 2 summarise the results of our approach versus all the previous proposals (hereinafter Best SOTA AUC). LUMIA overtakes previous results on all the cases except in two, which represents an improvement on 174 of the 176 cases (98.86%). Indeed, our approach provides an average AUC improvement of 15.75%. Considering an AUC greater than 60% as threshold [8], previous approaches surpass that value on the 44.5% of the cases while LUMIA reaches that threshold on the 65.33% of the cases, that is an increment of 46.80%.

**Multimodal.** Table 3 shows the results for the multimodal configurations. All of them, except Magpie, achieve AUC greater than 60%, suggesting that multimodality may be adding additional information useful for detecting MIA. Magpie reaches an AUC of 57%, probably because it is the only text-only dataset. When making predictions over a joint dataset, the AUC remains above 60%, which points out that even when mixing information and modalities, LPs find patterns across activations to define membership. Globally speaking, 85.9% of the cases achieve an AUC greater than 60%, demonstrating better performance as compared to unimodal setups, which meet this threshold in 65.33% of the configurations.

<table><caption>**Table 1.** AUC comparison with state of the art (SOTA) on TB datasets.</caption><thead><tr><th>Method</th><th>Best SOTA AUC</th><th>Ours</th><th>Improvement</th></tr></thead><tbody><tr><td colspan="4" style="text-align: center; font-weight: bold;">Gutenberg</td></tr><tr><td>Document features<sup>1</sup></td><td>0.856</td><td rowspan="2" style="font-weight: bold;">0.98</td><td>14.49%</td></tr><tr><td>Heuristics<sup>2</sup></td><td>0.964</td><td>1.66%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">ArXiv-1 month</td></tr><tr><td>Document features<sup>1</sup></td><td>0.678</td><td rowspan="2" style="font-weight: bold;">0.93</td><td>37.17%</td></tr><tr><td>Heuristics<sup>2</sup></td><td>0.684</td><td>35.96%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">Temporal wiki</td></tr><tr><td>Best-Duan<sup>3</sup></td><td>0.796</td><td rowspan="2" style="font-weight: bold;">0.93</td><td>16.83%</td></tr><tr><td>Heuristics<sup>2</sup></td><td>0.799</td><td>16.40%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">Temporal ArXiv 2020-08</td></tr><tr><td>Best-Duan<sup>3</sup></td><td>0.723</td><td rowspan="2" style="font-weight: bold;">0.86</td><td>18.32%</td></tr><tr><td>Heuristics<sup>2</sup></td><td>0.756</td><td>13.15%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">WikiMIA</td></tr><tr><td>Min prob<sup>4</sup></td><td>0.839</td><td rowspan="5" style="font-weight: bold;">0.99</td><td>18.00%</td></tr><tr><td>Finetune + probes<sup>5</sup></td><td>0.698</td><td>41.83%</td></tr><tr><td>Heuristics<sup>2</sup></td><td>0.987</td><td>0.30%</td></tr><tr><td>EM-MIA<sup>6</sup></td><td>0.977</td><td>1.33%</td></tr><tr><td>ModRényi<sup>7</sup></td><td>0.809</td><td>22.37%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">ArXiv CS</td></tr><tr><td>Finetune + probes<sup>5</sup></td><td>0.673</td><td style="font-weight: bold;">0.842</td><td>25.11%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">ArXiv Math</td></tr><tr><td>Finetune + probes<sup>5</sup></td><td>0.574</td><td style="font-weight: bold;">0.646</td><td>12.54%</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">Average improvement</td></tr><tr><td colspan="4"></td><td>18.00%</td></tr></tbody></table>

Meeus et al. [21]<sup>1</sup> Das et al. [7]<sup>2</sup> Duan et al. [21]<sup>3</sup> Shi et al. [25]<sup>4</sup> Liu et al. [17]<sup>5</sup> Kim et al. [13]<sup>6</sup> Li et al. [16]<sup>7</sup>
<table><caption>**Table 2.** AUC comparison against Duan *et al.* [9] for NGB datasets.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">MIA</th><th colspan="9">Pythia Dedup 12B</th></tr><tr><th colspan="3">N = 13 P = 0.8</th><th colspan="3">N = 13 P = 0.2</th><th colspan="3">N = 7 P = 0.2</th></tr><tr><th></th><th></th><th></th><th></th><th>Ours</th><th>Improvement</th><th></th><th></th><th></th><th>Ours</th><th>Improvement</th></tr></thead><tbody><tr><td rowspan="4">Wikipedia</td><td>LOSS</td><td>0.516</td><td></td><td></td><td>10.47%</td><td></td><td>0.545</td><td></td><td>8.26%</td><td></td><td>3.60%</td></tr><tr><td>Ref</td><td>0.578</td><td></td><td>0.570</td><td>-1.38%</td><td></td><td>0.590</td><td></td><td>0.00%</td><td></td><td>1.92%</td></tr><tr><td>min-k</td><td>0.517</td><td></td><td></td><td>10.25%</td><td></td><td>0.562</td><td>0.590</td><td>4.98%</td><td></td><td>7.14%</td></tr><tr><td>zlib</td><td>0.524</td><td></td><td></td><td>8.78%</td><td></td><td>0.543</td><td></td><td>8.66%</td><td></td><td>9.35%</td></tr><tr><td rowspan="4">Github</td><td>LOSS</td><td>0.678</td><td></td><td></td><td>13.57%</td><td></td><td>0.802</td><td></td><td>13.47%</td><td></td><td>5.92%</td></tr><tr><td>Ref</td><td>0.559</td><td></td><td><strong>0.770</strong></td><td>37.75%</td><td></td><td>0.615</td><td></td><td>47.97%</td><td></td><td>51.22%</td></tr><tr><td>min-k</td><td>0.683</td><td></td><td></td><td>12.74%</td><td></td><td>0.830</td><td><strong>0.910</strong></td><td>9.64%</td><td></td><td>4.49%</td></tr><tr><td>zlib</td><td>0.690</td><td></td><td></td><td>11.59%</td><td></td><td>0.829</td><td></td><td>9.77%</td><td></td><td>2.42%</td></tr><tr><td rowspan="4">Pubmed</td><td>LOSS</td><td>0.506</td><td></td><td></td><td>14.62%</td><td></td><td>0.534</td><td></td><td>6.74%</td><td></td><td>25.64%</td></tr><tr><td>Ref</td><td>0.559</td><td></td><td>0.580</td><td>3.76%</td><td></td><td>0.573</td><td>0.570</td><td>-0.52%</td><td></td><td>64.71%</td></tr><tr><td>min-k</td><td>0.512</td><td></td><td></td><td>13.28%</td><td></td><td>0.542</td><td></td><td>5.17%</td><td></td><td>23.74%</td></tr><tr><td>zlib</td><td>0.506</td><td></td><td></td><td>14.62%</td><td></td><td>0.537</td><td></td><td>6.15%</td><td></td><td>26.94%</td></tr><tr><td rowspan="4">Pile CC</td><td>LOSS</td><td>0.516</td><td></td><td></td><td>16.28%</td><td></td><td>0.534</td><td></td><td>12.55%</td><td></td><td>14.98%</td></tr><tr><td>Ref</td><td>0.582</td><td></td><td><strong>0.600</strong></td><td>3.09%</td><td></td><td>0.593</td><td><strong>0.601</strong></td><td>1.35%</td><td></td><td>2.48%</td></tr><tr><td>min-k</td><td>0.521</td><td></td><td></td><td>15.16%</td><td></td><td>0.539</td><td></td><td>11.50%</td><td></td><td>14.19%</td></tr><tr><td>zlib</td><td>0.517</td><td></td><td></td><td>16.05%</td><td></td><td>0.542</td><td></td><td>10.89%</td><td></td><td>17.86%</td></tr><tr><td rowspan="4">ArXiv</td><td>LOSS</td><td>0.527</td><td></td><td></td><td>9.46%</td><td></td><td>0.573</td><td></td><td>5.76%</td><td></td><td>1.65%</td></tr><tr><td>Ref</td><td>0.555</td><td></td><td>0.577</td><td>3.94%</td><td></td><td>0.584</td><td><strong>0.606</strong></td><td>3.77%</td><td></td><td>11.89%</td></tr><tr><td>min-k</td><td>0.530</td><td></td><td></td><td>8.84%</td><td></td><td>0.566</td><td></td><td>7.07%</td><td></td><td>8.99%</td></tr><tr><td>zlib</td><td>0.521</td><td></td><td></td><td>10.72%</td><td></td><td>0.565</td><td></td><td>7.26%</td><td></td><td>2.56%</td></tr><tr><td rowspan="4">DMulan</td><td>LOSS</td><td>0.485</td><td></td><td></td><td>23.71%</td><td></td><td>0.673</td><td></td><td>10.79%</td><td></td><td>3.15%</td></tr><tr><td>Ref</td><td>0.514</td><td></td><td><strong>0.600</strong></td><td>16.73%</td><td></td><td>0.443</td><td></td><td>68.31%</td><td></td><td>129.47%</td></tr><tr><td>min-k</td><td>0.493</td><td></td><td></td><td>21.70%</td><td></td><td>0.650</td><td><strong>0.746</strong></td><td>14.71%</td><td></td><td>2.48%</td></tr><tr><td>zlib</td><td>0.481</td><td></td><td></td><td>24.74%</td><td></td><td>0.643</td><td></td><td>15.96%</td><td></td><td>18.01%</td></tr><tr><td rowspan="4">Hackernews</td><td>LOSS</td><td>0.512</td><td></td><td></td><td>14.01%</td><td></td><td>0.526</td><td></td><td>12.91%</td><td></td><td>14.24%</td></tr><tr><td>Ref</td><td>0.549</td><td></td><td>0.584</td><td>0.33%</td><td></td><td>0.553</td><td>0.594</td><td>7.40%</td><td></td><td>21.05%</td></tr><tr><td>min-k</td><td>0.526</td><td></td><td></td><td>10.97%</td><td></td><td>0.533</td><td></td><td>11.43%</td><td></td><td>17.95%</td></tr><tr><td>zlib</td><td>0.507</td><td></td><td></td><td>15.13%</td><td></td><td>0.524</td><td></td><td>13.34%</td><td></td><td>16.55%</td></tr><tr><th colspan="13" style="text-align: center;">GPT-Neo 2.7B</th></tr><tr><td rowspan="4">Wikipedia</td><td>LOSS</td><td>0.513</td><td></td><td></td><td>13.99%</td><td></td><td>0.537</td><td></td><td>8.01%</td><td></td><td>0.00%</td></tr><tr><td>Ref</td><td>0.545</td><td></td><td>0.584</td><td>7.29%</td><td></td><td>0.572</td><td></td><td>1.40%</td><td></td><td>0.00%</td></tr><tr><td>min-k</td><td>0.513</td><td></td><td></td><td>13.99%</td><td></td><td>0.543</td><td>0.58</td><td>6.81%</td><td></td><td>0.93%</td></tr><tr><td>zlib</td><td>0.519</td><td></td><td></td><td>12.67%</td><td></td><td>0.535</td><td></td><td>8.41%</td><td></td><td>4.33%</td></tr><tr><td rowspan="4">Github</td><td>LOSS</td><td>0.699</td><td></td><td></td><td>10.53%</td><td></td><td>0.770</td><td></td><td>10.39%</td><td></td><td>7.06%</td></tr><tr><td>Ref</td><td>0.570</td><td></td><td><strong>0.772</strong></td><td>35.55%</td><td></td><td>0.549</td><td><strong>0.85</strong></td><td>54.83%</td><td></td><td>52.85%</td></tr><tr><td>min-k</td><td>0.700</td><td></td><td></td><td>10.38%</td><td></td><td>0.802</td><td></td><td>5.99%</td><td></td><td>5.62%</td></tr><tr><td>zlib</td><td>0.710</td><td></td><td></td><td>8.82%</td><td></td><td>0.771</td><td></td><td>10.25%</td><td></td><td>3.52%</td></tr><tr><td rowspan="4">Pubmed</td><td>LOSS</td><td>0.490</td><td></td><td></td><td>15.55%</td><td></td><td>0.498</td><td></td><td>10.44%</td><td></td><td>13.89%</td></tr><tr><td>Ref</td><td>0.507</td><td></td><td>0.566</td><td>11.68%</td><td></td><td>0.507</td><td>0.55</td><td>8.48%</td><td></td><td>15.78%</td></tr><tr><td>min-k</td><td>0.500</td><td></td><td></td><td>13.24%</td><td></td><td>0.501</td><td></td><td>9.78%</td><td></td><td>14.90%</td></tr><tr><td>zlib</td><td>0.499</td><td></td><td></td><td>13.47%</td><td></td><td>0.499</td><td></td><td>10.22%</td><td></td><td>15.78%</td></tr><tr><td rowspan="4">Pile CC</td><td>LOSS</td><td>0.500</td><td></td><td></td><td>17.48%</td><td></td><td>0.500</td><td></td><td>17.91%</td><td></td><td>15.73%</td></tr><tr><td>Ref</td><td>0.530</td><td></td><td>0.587</td><td>10.83%</td><td></td><td>0.530</td><td>0.59</td><td>11.32%</td><td></td><td>11.30%</td></tr><tr><td>min-k</td><td>0.500</td><td></td><td></td><td>17.48%</td><td></td><td>0.507</td><td></td><td>16.37%</td><td></td><td>16.58%</td></tr><tr><td>zlib</td><td>0.500</td><td></td><td></td><td>17.48%</td><td></td><td>0.505</td><td></td><td>16.83%</td><td></td><td>18.52%</td></tr><tr><td rowspan="4">ArXiv</td><td>LOSS</td><td>0.510</td><td></td><td></td><td>14.92%</td><td></td><td>0.515</td><td></td><td>14.56%</td><td></td><td>8.86%</td></tr><tr><td>Ref</td><td>0.520</td><td></td><td>0.586</td><td>12.71%</td><td></td><td>0.517</td><td>0.59</td><td>14.12%</td><td></td><td>19.78%</td></tr><tr><td>min-k</td><td>0.517</td><td></td><td></td><td>13.36%</td><td></td><td>0.519</td><td></td><td>13.68%</td><td></td><td>13.16%</td></tr><tr><td>zlib</td><td>0.510</td><td></td><td></td><td>14.92%</td><td></td><td>0.510</td><td></td><td>15.69%</td><td></td><td>9.69%</td></tr><tr><td rowspan="4">DMulan</td><td>LOSS</td><td>0.485</td><td></td><td></td><td>15.46%</td><td></td><td>0.676</td><td></td><td>10.95%</td><td></td><td>7.53%</td></tr><tr><td>Ref</td><td>0.509</td><td></td><td>0.560</td><td>10.02%</td><td></td><td>0.435</td><td></td><td>72.41%</td><td></td><td>99.20%</td></tr><tr><td>min-k</td><td>0.492</td><td></td><td></td><td>13.82%</td><td></td><td>0.655</td><td><strong>0.75</strong></td><td>14.50%</td><td></td><td>7.18%</td></tr><tr><td>zlib</td><td>0.481</td><td></td><td></td><td>16.42%</td><td></td><td>0.647</td><td></td><td>15.92%</td><td></td><td>23.15%</td></tr><tr><td rowspan="4">Hackernews</td><td>LOSS</td><td>0.502</td><td></td><td></td><td>17.53%</td><td></td><td>0.516</td><td></td><td>16.28%</td><td></td><td>6.42%</td></tr><tr><td>Ref</td><td>0.512</td><td></td><td>0.590</td><td>15.23%</td><td></td><td>0.515</td><td><strong>0.60</strong></td><td>16.50%</td><td></td><td>20.00%</td></tr><tr><td>min-k</td><td>0.517</td><td></td><td></td><td>14.12%</td><td></td><td>0.525</td><td></td><td>14.29%</td><td></td><td>10.14%</td></tr><tr><td>zlib</td><td>0.502</td><td></td><td></td><td>17.53%</td><td></td><td>0.519</td><td></td><td>15.61%</td><td></td><td>7.33%</td></tr><tr><td colspan="2">Average Improvement</td><td></td><td></td><td></td><td>11.96%</td><td></td><td></td><td></td><td>13.93%</td><td></td><td>17.03%</td></tr></tbody></table>

## Takeaway 1

**Unimodal:** We improve 15.71% AUC on average. AUC greater than 60% is reached by LUMIA 46.8% more often as compared to the state-of-the-art.

**Multimodal:** No previous work to compare. Among the seven datasets considered, LUMIA achieves AUC greater than 60% in all but one dataset, resulting in 85.9% of the results exceeding this threshold.
<table><caption>**Table 3.** LUMIA AUC results in multimodal models. Notice highlighted best values between models across modalities.</caption><thead><tr><th colspan="3">Textcaps (General OCR)</th><th colspan="2">AOK (General)</th></tr><tr><th>Params</th><th>Modality</th><th>Best AUC</th><th>Modality</th><th>Best AUC</th></tr></thead><tbody><tr><td rowspan="2">0.5B</td><td>Textual + visual</td><td>0.540</td><td>Textual + visual</td><td>0.697</td></tr><tr><td>Visual</td><td>0.604</td><td>Visual</td><td><strong>0.735</strong></td></tr><tr><td rowspan="2">7B</td><td>Textual + visual</td><td><strong>0.601</strong></td><td>Textual + visual</td><td><strong>0.697</strong></td></tr><tr><td>Visual</td><td><strong>0.618</strong></td><td>Visual</td><td>0.707</td></tr><tr><th colspan="3">ScienceQA (General)</th><th colspan="2">ChartQA (Doc/chart/screen)</th></tr><tr><td rowspan="2">0.5B</td><td>Textual + visual</td><td>0.970</td><td>Textual + visual</td><td><strong>0.694</strong></td></tr><tr><td>Visual</td><td><strong>0.806</strong></td><td>Visual</td><td>0.638</td></tr><tr><td rowspan="2">7B</td><td>Textual + visual</td><td><strong>0.990</strong></td><td>Textual + visual</td><td>0.682</td></tr><tr><td>Visual</td><td>0.802</td><td>Visual</td><td><strong>0.691</strong></td></tr><tr><th colspan="3">magpie (Language)</th><th colspan="2">iconqa (General)</th></tr><tr><td rowspan="2">0.5B</td><td>Textual + visual</td><td><strong>0.572</strong></td><td>Textual + visual</td><td>0.869</td></tr><tr><td>Visual</td><td>0.510</td><td>Visual</td><td>0.809</td></tr><tr><td rowspan="2">7B</td><td>Textual + visual</td><td>0.552</td><td>Textual + visual</td><td><strong>0.903</strong></td></tr><tr><td>Visual</td><td><strong>0.520</strong></td><td>Visual</td><td><strong>0.828</strong></td></tr><tr><th colspan="3">Join</th><th colspan="2">MathV360k (Math)</th></tr><tr><td rowspan="2">0.5B</td><td>Textual + visual</td><td>0.624</td><td>Textual + visual</td><td>0.599</td></tr><tr><td>Visual</td><td>0.670</td><td>Visual</td><td>0.584</td></tr><tr><td rowspan="2">7B</td><td>Textual + visual</td><td><strong>0.634</strong></td><td>Textual + visual</td><td><strong>0.660</strong></td></tr><tr><td>Visual</td><td><strong>0.673</strong></td><td>Visual</td><td><strong>0.629</strong></td></tr></tbody></table>

*Textual + Visual:* Activations extracted from LLM part.

*Visual:* Activations extracted from Visual encoder part.

## 5.2 Impact of model size

**Unimodal.** Table 4 shows a clear trend on the AUC as the model size grows for Pythia family. Results are similar in GPT-Neo, thus placed in Appendix B. All datasets show better results in all configurations on the 12B version, excluding ArXiv-1 month. For this dataset, both deduplicated and non-deduplicated models show improved AUC scores when scaling from 70M to 2.8B parameters. Yet, a significant decline is observed in the 12B version of the model on this dataset, with AUC values dropping from 92% to 84% (deduped) and 86% (non-dedup). Despite this unexpected decrease, AUC values are still very high.

By analyzing the trends on the percentage of change of AUC of non-LP-based proposals and ours, while LPs shows an incremental trend, differences with other approaches are not significant.

**Multimodal.** From an architectural perspective, while there are no differences in the sizes of the visual encoders, having a larger LLM on the textual+visual part affects the results. In general, excluding again the Magpie dataset (since it only contains texts), the 7B model seems to reveal more information in both parts of the models, the visual only encoder and the textual+visual LLM, denoting higher memorization of the data than the 0.5B version.
<table><caption>**Table 4.** Pythia family. AUC per model size with/without deduplication.</caption><thead><tr><th rowspan="3">Params N = 13 P = 0.8 N = 13 P = 0.2 N = 7 P = 0.2 N = 13 P = 0.8 N = 13 P = 0.2 N = 7 P = 0.2</th><th colspan="4">NGB</th><th colspan="2">TB</th></tr><tr><th colspan="2">Pythia dedup</th><th colspan="2">Pythia non-dedup</th><th colspan="2">Pythia dedup|Pythia non-dedup</th></tr><tr><th colspan="4">Wikipedia</th><th colspan="2">Gutenberg</th></tr></thead><tbody><tr><td>70m</td><td>0.520</td><td>0.551</td><td>0.653</td><td>0.570</td><td>0.538</td><td>0.651</td><td>0.949</td><td>0.960</td></tr><tr><td>160M</td><td>0.568</td><td>0.546</td><td>0.660</td><td>0.546</td><td>0.589</td><td>0.676</td><td>0.960</td><td>0.960</td></tr><tr><td>1.4B</td><td>0.557</td><td>0.586</td><td>0.676</td><td>0.564</td><td>0.558</td><td>0.663</td><td>0.970</td><td>0.980</td></tr><tr><td>2.8B</td><td>0.580</td><td>0.572</td><td>0.682</td><td>0.562</td><td>0.557</td><td>0.685</td><td>0.970</td><td>0.980</td></tr><tr><td>12B</td><td>0.570</td><td>0.590</td><td>0.690</td><td>0.570</td><td>0.580</td><td>0.590</td><td>0.987</td><td>0.985</td></tr><tr><td></td><td colspan="4">Github</td><td colspan="2">ArXiv-1 month</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.743</td><td>0.823</td><td>0.922</td><td>0.732</td><td>0.832</td><td>0.881</td><td>0.775</td><td>0.806</td></tr><tr><td>160M</td><td>0.729</td><td>0.853</td><td>0.863</td><td>0.741</td><td>0.874</td><td>0.931</td><td>0.860</td><td>0.870</td></tr><tr><td>1.4B</td><td>0.767</td><td>0.868</td><td>0.935</td><td>0.786</td><td>0.875</td><td>0.924</td><td>0.900</td><td>0.920</td></tr><tr><td>2.8B</td><td>0.754</td><td>0.868</td><td>0.951</td><td>0.767</td><td>0.911</td><td>0.933</td><td>0.920</td><td>0.920</td></tr><tr><td>12B</td><td>0.770</td><td>0.910</td><td>0.930</td><td>0.760</td><td>0.830</td><td>0.870</td><td>0.843</td><td>0.856</td></tr><tr><td></td><td colspan="4">Pile CC</td><td colspan="2">Temporal wiki</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.521</td><td>0.544</td><td>0.587</td><td>0.561</td><td>0.552</td><td>0.599</td><td>0.865</td><td>0.860</td></tr><tr><td>160M</td><td>0.531</td><td>0.566</td><td>0.603</td><td>0.562</td><td>0.554</td><td>0.611</td><td>0.880</td><td>0.910</td></tr><tr><td>1.4B</td><td>0.569</td><td>0.573</td><td>0.642</td><td>0.554</td><td>0.588</td><td>0.635</td><td>0.910</td><td>0.930</td></tr><tr><td>2.8B</td><td>0.590</td><td>0.571</td><td>0.618</td><td>0.566</td><td>0.587</td><td>0.634</td><td>0.929</td><td>0.930</td></tr><tr><td>12B</td><td>0.600</td><td>0.570</td><td>0.660</td><td>0.610</td><td>0.570</td><td>0.660</td><td>0.945</td><td>0.954</td></tr><tr><td></td><td colspan="4">DM math</td><td colspan="2">Temporal ArXiv</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.502</td><td>0.743</td><td>0.959</td><td>0.525</td><td>0.721</td><td>0.963</td><td>0.710</td><td>0.733</td></tr><tr><td>160M</td><td>0.557</td><td>0.718</td><td>0.927</td><td>0.526</td><td>0.706</td><td>0.981</td><td>0.720</td><td>0.720</td></tr><tr><td>1.4B</td><td>0.558</td><td>0.758</td><td>0.943</td><td>0.559</td><td>0.745</td><td>0.989</td><td>0.760</td><td>0.750</td></tr><tr><td>2.8B</td><td>0.554</td><td>0.730</td><td>0.995</td><td>0.570</td><td>0.741</td><td>0.961</td><td>0.750</td><td>0.760</td></tr><tr><td>12B</td><td>0.600</td><td>0.746</td><td>1.000</td><td>0.610</td><td>0.650</td><td>0.980</td><td>0.810</td><td>0.797</td></tr><tr><td></td><td colspan="4">Hackernews</td><td colspan="2">WikiMIA</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.580</td><td>0.595</td><td>0.582</td><td>0.566</td><td>0.563</td><td>0.633</td><td>0.970</td><td>0.980</td></tr><tr><td>160M</td><td>0.593</td><td>0.577</td><td>0.613</td><td>0.564</td><td>0.565</td><td>0.642</td><td>0.970</td><td>0.980</td></tr><tr><td>1.4B</td><td>0.556</td><td>0.576</td><td>0.630</td><td>0.594</td><td>0.570</td><td>0.647</td><td>0.970</td><td>0.990</td></tr><tr><td>2.8B</td><td>0.579</td><td>0.576</td><td>0.614</td><td>0.591</td><td>0.580</td><td>0.643</td><td>0.980</td><td>0.990</td></tr><tr><td>12B</td><td>0.584</td><td>0.594</td><td>0.701</td><td>0.600</td><td>0.560</td><td>0.580</td><td>0.980</td><td>0.990</td></tr><tr><td></td><td colspan="4">ArXiv</td><td colspan="2">ArXiv-CS</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.533</td><td>0.562</td><td>0.823</td><td>0.529</td><td>0.598</td><td>0.820</td><td>0.743</td><td>0.745</td></tr><tr><td>160M</td><td>0.569</td><td>0.586</td><td>0.802</td><td>0.536</td><td>0.571</td><td>0.797</td><td>0.776</td><td>0.747</td></tr><tr><td>1.4B</td><td>0.563</td><td>0.581</td><td>0.814</td><td>0.543</td><td>0.577</td><td>0.832</td><td>0.791</td><td>0.807</td></tr><tr><td>2.8B</td><td>0.532</td><td>0.570</td><td>0.808</td><td>0.571</td><td>0.568</td><td>0.815</td><td>0.824</td><td>0.807</td></tr><tr><td>12B</td><td>0.577</td><td>0.606</td><td>0.795</td><td>0.600</td><td>0.590</td><td>0.770</td><td>0.831</td><td>0.835</td></tr><tr><td></td><td colspan="4">Pubmed</td><td colspan="2">ArXiv-Math</td><td colspan="2"></td></tr><tr><td>70m</td><td>0.517</td><td>0.576</td><td>0.880</td><td>0.545</td><td>0.573</td><td>0.882</td><td>0.603</td><td>0.601</td></tr><tr><td>160M</td><td>0.540</td><td>0.600</td><td>0.870</td><td>0.558</td><td>0.575</td><td>0.894</td><td>0.615</td><td>0.604</td></tr><tr><td>1.4B</td><td>0.583</td><td>0.558</td><td>0.883</td><td>0.551</td><td>0.582</td><td>0.875</td><td>0.637</td><td>0.633</td></tr><tr><td>2.8B</td><td>0.553</td><td>0.573</td><td>0.860</td><td>0.570</td><td>0.570</td><td>0.894</td><td>0.626</td><td>0.646</td></tr><tr><td>12B</td><td>0.577</td><td>0.590</td><td>0.900</td><td>0.570</td><td>0.580</td><td>0.980</td><td>0.635</td><td>0.655</td></tr></tbody></table>

## Takeaway 2

As model grows, we have better AUC in the 85.9% of the cases for both unimodal and multimodal LLMs. The pace is the same as non-LP-based approaches.

## 5.3 Impact of bias

In this case, we concentrate on unimodal LLMs since in multimodality, our results show that there are no significant differences between the average values of the member and non-member samples in the datasets based on HV or SSIM (see Appendix A for further details). Thus, high or low AUC results seem to be influenced by task complexity or dataset nature, rather than by the actual content differences.

**Unimodal.** LUMIA significantly outperforms [17], which also uses LPs. Specifically, we achieve a 25% improvement in the CS subset. These findings align with their observation that the Math subset is more challenging to predict. Nevertheless, LUMIA still achieves an AUC above 60% even in these more difficult subsets. Additionally, results for WikiMIA show a particularly high improvement of 41%. On average, for TB datasets, LUMIA returns an 18% improvement over all the previous efforts.
When studying NGB, Table 2 shows that LUMIA outperforms all reported configurations and baselines (except for the Wikipedia dataset) across all models. They reported that no configuration reached an AUC greater than 60% for the *N* = 13 overlap of *P* = 0.8 on the Pythia dedup model. Contrarily, in some specific cases, such as Pile-CC, DM, we can reach this threshold. Additionally, for the 12B model, we achieve an AUC of 58% on PubMed and 58.4% on Hackernews, both approaching the 60% threshold more closely than previous approaches. All in all, results for *N* = 13 with *P* = 0.8 lead to an overall improvement of 13.10%.

For *N* = 7 with *P* = 0.2 on the Pythia 12B family in Table 2, our approach consistently outperforms the state of the art, with improvements ranging from a minimum of 2% on Wikipedia to a maximum of 64% on PubMed and an average of 18.74%. Notably, on the DM dataset the ref method performs poorly under overlap configurations of *P* = 0.2 with *N* = 13 and *N* = 7, achieving AUC scores of 44% and 41%, respectively. Consequently, our approach surpasses the ref method by 68% and 129%.

For GPT-Neo 2.7B, in line with the previous model, all configurations over-
take results from Duan et al. [8] with an overall improvement of 14.43%. Nonethe-
less, in this case of N = 13 overlap with P = 0.8, none of our configurations,
excluding Github, overtakes 60% of AUC.

Although the distinction between members and non-members are based on
different techniques, it is generally observed that TB datasets are easier to detect
than NGB ones, even in cases of high overlap. For instance, the Wikipedia dataset
in the NGB datasets achieves a maximum AUC of 68.5%, while the Temporal-
Wiki dataset in the TB datasets reaches up to 95% AUC.

Takeaway 3

***Unimodal:*** In line with non-LPs approaches, as overlap gets reduced, we have better results in all cases. TB datasets present better results than NGB and all results are above 60%.

***Multimodal:*** HV and SSIM differences show no correlation with AUC,
suggesting that the results are more influenced by task or dataset com-
plexity, rather than any bias between member and non-member samples.

5.4 Impact of dataset nature

**Unimodal.** Table 1 shows consistent conclusions with those of Liu *et al.* [17] on TB datasets, who argue that the difficulty of the content impacts results. For example, our approach overtakes their results on a 25% and 12% on the arXiv-CS and arXiv-Math datasets respectively, but in line with their hypothesis, our LPs also perform worse on the arXiv-Math dataset, where the nature of the text content makes detection more challenging, as mathematical texts are more complex and harder to memorise by the LLMs.

In the case of NGB datasets, as shown in Table 2, similar patterns are ob-
served. For instance, on Github, which was the easiest to predict on the *N* = 13
with $P = 0.8$ overlap (according to Duan *et al.* [8]), LUMIA also offers the best results. Code-related samples may contain HTML tags and unique variable names, which could make the members and non-members more identifiable. Furthermore, other datasets such as Wikipedia or Hackernews, which contain a wider range of topics and variety of texts, make harder the identification of differences between members and non-members.

**Multimodal.** In multimodal datasets, the type of information impacts the results. Table 3 shows that, except for ChartQA and IconQA, the model appears to add more information through the visual encoders, particularly with images. For example, in the case of Magpie, which is a text-only dataset, the visual encoder returns an almost random AUC of 52%, but it adds more information when dealing with both textual and visual inputs, reaching a 57.2% AUC.

For Textcaps, the prompt remains the same for both members and non-members, while the images exhibit greater variability. This setup results in a slight drop in accuracy when using the combined textual and visual parts of the model, with AUC decreasing from 61.7% for visual-only LPs to 60.1% for visual+text LPs. The consistent prompt across members and non-members likely introduces noise, diminishing the model's ability to differentiate between them.

Finally, datasets that follow a consistent template across the prompts of both members and non-members, such as MathV360k, demonstrate a reduced ability for classifiers to distinguish between classes compared to datasets with more varied images and texts. For example, datasets like ScienceQA and IconQA, which lack a uniform template across samples, achieve AUC values around 80% for both Textual+Visual and Visual-only configurations.

**Takeaway 4**

**Unimodal:** Code or datasets containing mathematical formulas are easier to identify than general-purpose texts. LPs are specially good on detecting these modalities.

**Multimodal:** Repetitive prompts strengthen resistance to MIA, while a greater variety of images makes models more vulnerable.

## 5.5 Impact of deduplication

Since data in Llava and OneVision models is already deduplicated, and no open-source models without this data processing exist, only unimodality is considered.

**Unimodal.** Results focus on Pythia, Table 4, since it is the only model which provides a clear distinction of deduplicated data. For the TB datasets, MIAs tend to be more effective on non-deduplicated models. This is likely because deduplication reduces the repetition of data in the training set, thereby limiting the model's ability to memorise and overfit to specific patterns.

In contrast, for the NGB datasets, no significant differences are observed between the deduplicated and non-deduplicated versions of the models.
## 5.6 Analysis per model and layer

Figures 2 and 3 present results from Pythia and the multimodal models respectively. They include the normalised average values of the AUC across all datasets. Gradient colors represent the average AUC for each layer, calculated across all models and datasets. Results for GPT-Neo are omitted for brevity, as they are always more effective on deeper layers (cf. Appendix B, for further details).

<figure><img src="image_4.png" /><figcaption>Fig. 2. Pythia family. AUC by layer. (a) NGB datasets. (b) TB datasets.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3. LLava-OneVision. AUC by layer. (a) Visual encoder only. (b) Visual+text encoder.</figcaption></figure>

**Unimodal.** Figure 2 illustrates the AUC across layers for the Pythia model family, covering results from all datasets and model types (both deduplicated and non-deduplicated). In Figure 2(a), model performance on all NGB datasets from Mimir are reported. Notably, there are certain layers where model performance peaks. Particularly, around Layer 10; and, later, between Layers 15 and 18, where larger models achieve higher AUC values. This suggests that specific depths of the model add more information useful for LPs to detect MIAs.

In Figure 2(b), the average normalised values for the same models on the TB datasets are reported. Peak performance begins around Layers 3 to 5, showing that earlier layers on the model are enough to get good results. For larger models, between Layers 5 and 12, there is also a portion of the model where important information for the membership inference is revealed.
**Multimodal.** Figure 3 shows results for the multimodal model. Figure 3(a) presents the AUC for the visual encoder, which, despite differences in scale, reveals common areas around Layer 15. This may denote that deeper layers add more information.

Similarly, in Figure 3(b), the AUC by layer is shown for the visual+text en-
coder. It is particularly noteworthy that around Layers 12 and 13, both modal-
ities (visual and visual+text) exhibit a spike in information useful for MIA.

Takeaway 5

*Unimodal:* LPs on Pythia family with TB datasets are more effective on earlier layers (4-5). For NGB datasets, deeper layers (15-16) are preferred. LPs in GPT-Neo are always more effective on deeper layers (25).

*Multimodal:* Middle layers reveal more information in visual (8-9 or 15) as well as in textual+visual part (13-14).

# 6 Related work

MIA attacks have been largely studied. From a grey-box perspective, Shi et al. [25] analyse output logits of LLMs based on the assumption that unseen samples tend to contain outlier words with very low probabilities.

Meeus et al. [21] adopt a binary classifier to distinguish members from non-members using document-level features and a normalisation algorithm in a black-box approach. Das et al. [7] achieve superior results than other works. They hypothesise that good results can be achieved by leveraging heuristics based on the features and statistics of public MIA datasets.

Duan et al. [9] introduce some benchmark datasets, *Mimir*, designed to address potential biases and assess state-of-the-art MIA methods. They also show that cutoff dates are crucial, as the overlap of N-grams may fluctuate over time. A similar work is proposed by Kim et al. [13]. They introduce a new maximisation expectation algorithm. Nevertheless, they highlight their results are close to random guessing when the distribution of member and non-members are close.

A comparable white-box approach is presented by Liu et al. [17], who use
simple linear classifiers on the activations of the LLMs. They fine-tune a pre-
trained model using a prompt to ensure that members and non-members are
represented in a standardised format. Moreover, they consider only layer l, thus
neglecting a per layer analysis of the results.

In terms of multimodality in LLMs, Li et al. [16] proposed a grey-box MIA approach. This approach introduces a novel metric that relies on the confidence level of the model’s output. As it relies on generating long sequences of text as output, this approach may have limited generalizability. Moreover, results are worse than those of LUMIA, presented in this paper.

Table 7 presents an overview of related work, together with LUMIA. Our method is assessed on a broader range of datasets, covering both TB and NGB datasets, as well as deduplicated and non-deduplicated models — a distinction
<table><caption>**Table 5.** Related work analysis.</caption><thead><tr><th>Ref.</th><th>Dataset</th><th>Model</th><th>Input features</th><th>Multi-lay-<br>modality</th><th>Per-lay-<br>analy-<br>sis</th><th>Dedup/<br>Non-<br>dedup</th><th>Temporal<br>Bias/<br>ngram<br>bias Anal-<br>ysis</th><th>Whitebox<br>(W) /<br>Greybox<br>(G) /<br>Blackbox<br>(B)</th></tr></thead><tbody><tr><td>Duan et al. [9]</td><td>Mimir, Temporal wiki, Tempo-ral ArXiv</td><td>Pythia, Pythia-dedup, GPT-Neo</td><td>Loss from mod-<br>els' logits</td><td>×</td><td>×</td><td>✓</td><td>✓</td><td>G</td></tr><tr><td>Shi et al. [25]</td><td>WikiMIA, BookMia</td><td>Pythia, GPT-Neo, OPT</td><td></td><td>×</td><td>×</td><td>×</td><td>×</td><td>G</td></tr><tr><td>Das et al. [7]</td><td>WikiMIA, BookMia, Temporal-wiki, temporal-ArXiv, ArXiv-1-month, Gutenberg</td><td></td><td>Features from<br>the texts</td><td>×</td><td>×</td><td>×</td><td>✓</td><td>B</td></tr><tr><td>Meeus et al. [21]</td><td>Gutenberg, ArXiv papers</td><td>Open-Llama</td><td>Features from<br>the texts</td><td>×</td><td>×</td><td>×</td><td>×</td><td>B</td></tr><tr><td>Kim et al. [13]</td><td>WikiMIA, OLMoMIA</td><td>Mamba, Pythia, Llama, OPT, GPT-Neo</td><td>Texts and mem-<br>bership scores</td><td>×</td><td>×</td><td>×</td><td>×</td><td>B</td></tr><tr><td>Li et al. [16]</td><td>VL-MIA</td><td>LLaVA 1.5, MiniGPT-4, LLaMA_adap-ter v2</td><td>Instruction based on the image, prompt and the output of the model of previous prompt</td><td>✓</td><td>×</td><td>×</td><td>×</td><td>G</td></tr><tr><td>Liu et al. [17]</td><td>ArXivMIA, WikiMIA</td><td>Pythia, OPT, Tiny-Llama, Open-Llama</td><td>Activations</td><td>×</td><td>×</td><td>×</td><td>×</td><td>W</td></tr><tr><td>LUMIA</td><td>WikiMIA, ArXiv-MIA, Temporal ArXiv/wik, ArXiv-1-month, Gutenberg, Mimir (The Pile), Textcaps, MathV360, AOK, ChartQA, ScienceQA, IconQA, Magpie</td><td>Pythia, Pythia-dedup, GPT-Neo, LLava-OneVision</td><td>Activations</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>W</td></tr></tbody></table>

only addressed by Duan *et al.* [8]. LUMIA is the only study to conduct a layer-by-layer analysis, which provides valuable insights into how unimodal and multimodal LLMs process information. LUMIA is also the only study that examines the impact of data type on MIAs within a multimodal context.

## 7 Conclusion

In this paper, an approach (dubbed LUMIA) has been proposed to tackle Membership Inference Attacks (MIAs). LUMIA assists on determining whether a sample was used during the pre-training of a target model. Remarkably, LUMIA leverages Linear Probes (LPs), thus adopting a white-box approach. LUMIA has been tested on a wide range of datasets and different LLMs, both for unimodal and multimodal cases. Results show that LUMIA overtakes other state-of-the-art solutions, while maintaining consistency across datasets, regardless of the presence or absence of bias.

As future work, we plan to extend our approach to other modalities, such as audio and video media, along with exploring the applicability of LUMIA in detecting copyright violations. Alternative future directions include leveraging insights about specific layers to introduce noise into those revealing the most information (hence, enhancing the model's resilience to such attacks); and conducting a deeper analysis of these layers to optimize the results.
# Appendices

## A. Bias analysis on multimodal models.

Table 6 summarises the statistics for all datasets containing images in the multimodal case, where the right column shows the absolute difference of the values between members and non-members. In further detail, content-related differences are analysed from a statistical perspective, with both HV (Hash Variation) and SSIM (Structural Similarity Index) computed. HV provides insights into the variation in image content, while SSIM quantifies structural similarity. Together, these measures help to understand content-driven distinctions between members and non-members.

HV shows no clear correlation with AUC values, as shown in Table 3. For instance, the ScienceQA dataset achieves a high AUC of 99% despite a large HV difference of 80%, while IconQA, with a much smaller HV difference of 10%, still achieves a strong AUC of 86.9%. A similar pattern emerges with SSIM. For example, ScienceQA, with an SSIM difference of 8%, achieves an AUC of 99%, whereas Mathv360k, with the same SSIM difference, only reaches 66% AUC. Thus, high or low AUC results seem to be influenced by task complexity or dataset nature rather than by the actual content differences among member and non-member samples.

<table><caption>**Table 6.** Multimodal datasets. Bias analysis using Hash Variation (HV) and Structural Similarity Index (SSIM) in %.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">Members</th><th colspan="2">Non members</th><th colspan="2">Difference (abs.)</th></tr><tr><th>HV</th><th>SSIM</th><th>HV</th><th>SSIM</th><th>HV</th><th>SSIM</th></tr></thead><tbody><tr><td>AOK</td><td>30.53</td><td>0.69</td><td>30.75</td><td>0.72</td><td>0.22</td><td>0.03</td></tr><tr><td>Textcaps</td><td>3.52</td><td>0.67</td><td>3.82</td><td>0.68</td><td>0.30</td><td>0.01</td></tr><tr><td>ScienceQA</td><td>30.36</td><td>0.62</td><td>29.56</td><td>0.54</td><td>0.80</td><td>0.08</td></tr><tr><td>ChartQA</td><td>28.19</td><td>0.26</td><td>28.33</td><td>0.3</td><td>0.14</td><td>0.04</td></tr><tr><td>IconQA</td><td>30.94</td><td>0.53</td><td>31.04</td><td>0.56</td><td>0.10</td><td>0.03</td></tr><tr><td>Mathv360k</td><td>31.08</td><td>0.54</td><td>30.9</td><td>0.46</td><td>0.18</td><td>0.08</td></tr></tbody></table>

## B. GPT-Neo. Analysis per layer and model size

Figure 4 presents results of the analysis of AUC per layer. There are common areas of better performance between NGB and TB in Figures 4(a) and (b), respectively. In particular, Layers 10 and 25 show areas where more information useful for classifiers is added.

In what comes to the impact of model size, the same trend as in Pythia is noticed in Table 7 — as the model grows, better AUC is achieved. The only exception is arXiv-CS, where AUC of the largest model is 80.2%, while the configuration 1.3B of parameters gets 84.2%.
<figure><img src="image_6.png" /><figcaption>Fig. 4. GPT-Neo. AUC by layer. (a) NGB datasets. (b) TB datasets.</figcaption></figure>

<table><caption>Table 7. GPT-Neo. AUC per model size.</caption><thead><tr><th></th><th>NGB</th><th>TB</th></tr></thead><tbody><tr><td colspan="3">GPT-Neo</td></tr><tr><td colspan="3">Wikipedia</td></tr><tr><td>Params N = 13 P = 0.8 N = 13 P = 0.2 N = 7 P = 0.2</td><td></td><td>Gutenberg</td></tr><tr><td>125M</td><td>0.590</td><td>0.530</td><td>0.590</td><td>0.940</td></tr><tr><td>1.3B</td><td>0.590</td><td>0.560</td><td>0.650</td><td>0.970</td></tr><tr><td>2.7B</td><td>0.590</td><td>0.570</td><td>0.650</td><td>0.970</td></tr><tr><td colspan="3">Github</td><td>arXiv-1 month</td></tr><tr><td>125M</td><td>0.650</td><td>0.830</td><td>0.910</td><td>0.820</td></tr><tr><td>1.3B</td><td>0.780</td><td>0.830</td><td>0.940</td><td>0.910</td></tr><tr><td>2.7B</td><td>0.770</td><td>0.850</td><td>0.940</td><td>0.930</td></tr><tr><td colspan="3">Pile CC</td><td>Temporal wiki</td></tr><tr><td>125M</td><td>0.550</td><td>0.560</td><td>0.570</td><td>0.887</td></tr><tr><td>1.3B</td><td>0.590</td><td>0.570</td><td>0.590</td><td>0.908</td></tr><tr><td>2.7B</td><td>0.560</td><td>0.590</td><td>0.630</td><td>0.919</td></tr><tr><td colspan="3">DM Math</td><td>Temporal arXiv</td></tr><tr><td>125M</td><td>0.550</td><td>0.690</td><td>1.000</td><td>0.728</td></tr><tr><td>1.3B</td><td>0.560</td><td>0.730</td><td>1.000</td><td>0.777</td></tr><tr><td>2.7B</td><td>0.570</td><td>0.750</td><td>1.000</td><td>0.786</td></tr><tr><td colspan="3">Hackernews</td><td>WikiMIA</td></tr><tr><td>125M</td><td>0.550</td><td>0.550</td><td>0.570</td><td>0.897</td></tr><tr><td>1.3B</td><td>0.580</td><td>0.560</td><td>0.590</td><td>0.985</td></tr><tr><td>2.7B</td><td>0.590</td><td>0.600</td><td>0.620</td><td>0.987</td></tr><tr><td colspan="3">Arxiv</td><td>arXiv-CS</td></tr><tr><td>125M</td><td>0.550</td><td>0.560</td><td>0.820</td><td>0.681</td></tr><tr><td>1.3B</td><td>0.570</td><td>0.570</td><td>0.850</td><td>0.842</td></tr><tr><td>2.7B</td><td>0.580</td><td>0.590</td><td>0.850</td><td>0.802</td></tr><tr><td colspan="3">Pubmed</td><td>arXiv-Math</td></tr><tr><td>125M</td><td>0.540</td><td>0.540</td><td>0.830</td><td>0.604</td></tr><tr><td>1.3B</td><td>0.560</td><td>0.540</td><td>0.920</td><td>0.633</td></tr><tr><td>2.7B</td><td>0.580</td><td>0.570</td><td>0.920</td><td>0.646</td></tr></tbody></table>

**Acknowledgements** — Nicolas Anciaux was supported by the French grant iPoP PEPR (ANR-22-PECY-0002). Luis Ibanez-Lissen was supported by the Spanish National Cybersecurity Institute (INCIBE) grant APAMciber within the framework of the Recovery, Transformation and Resilience Plan funds, financed by the European Union (Next Generation). Jose Maria de Fuentes and also Lorena Gonzalez were partially supported by grant PID2023-150310OB-I00 of the Spanish AEI. Joaquin Garcia-Alfaro was supported by the HEU project AI4CCAM, under grant agreement 101076911.
References

1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

2. Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. In: Intl. Conf. on Learning Representations (ICLR), Works. Track (2017)

3. Biderman, S., Schoelkopf, H., Anthony, Q.G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M.A., Purohit, S., Prashanth, U.S., Raff, E., et al.: Pythia: A suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning. pp. 2397-2430. PMLR (2023)

4. Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., Tramer, F.: Membership inference attacks from first principles. In: IEEE Symposium on Security and Privacy. pp. 1897-1914. IEEE (2022)

5. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., Raffel, C.: Extracting Training Data from Large Language Models. In: 30th USENIX Security Symposium. pp. 2633-2650 (2021)

6. Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T.L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al.: Black-box access is insufficient for rigorous ai audits. In: ACM Conference on Fairness, Accountability, and Transparency. pp. 2254-2272 (2024)

7. Das, D., Zhang, J., Tramèr, F.: Blind baselines beat membership inference attacks for foundation models. In: Proc. DATA-FM at ICLR 2025 (to appear) (2025)

8. Duan, H., Yang, Y., Tam, K.Y.: Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States. arXiv preprint arXiv:2402.09733 (2024)

9. Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., Hajishirzi, H.: Do membership inference attacks work on large language models? In: First Conference on Language Modeling (COLM) (2024)

10. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

11. Gao, J., Li, M., Lee, K.F.: N-gram distribution based language model adaptation. In: Proc. of the 6th Intl. Conf. on Spoken Language Processing (ICSLP) (2000)

12. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., Leahy, C.: The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)

13. Kim, G., Li, Y., Spiliopoulou, E., Ma, J., Ballesteros, M., Wang, W.Y.: Detecting training data of large language models via expectation maximization. arXiv preprint arXiv:2410.07582 (2024)

14. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations (ICLR) (2015)

15. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., Li, C.: LLaVA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.03326 (2024)

16. Li, Z., Wu, Y., Chen, Y., Tonin, F., Abad Rocamora, E., Cevher, V.: Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems 37, 98645-98674 (2024)

17. Liu, Z., Zhu, T., Tan, C., Liu, B., Lu, H., Chen, W.: Probing language models for pre-training data detection. In: Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 1576-1587 (2024)
18. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In: 36th Conference on Neural Information Processing Systems (NeurIPS) (2022)

19. Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., Zhu, S.C.: ICONQA: A new benchmark for abstract diagram understanding and visual language reasoning. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). vol. 37, pp. 9537–9547 (2023)

20. Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In: Findings of the Association for Computational Linguistics: EMNLP. pp. 993–1007 (2022)

21. Meeus, M., Jain, S., Rei, M., de Montjoye, Y.A.: Did the neurons read your book? document-level membership inference for large language models. In: 33rd USENIX Security Symposium. pp. 2369–2385 (2024)

22. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. nature **323**(6088), 533–536 (1986)

23. Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., Jégou, H.: White-box vs black-box: Bayes optimal strategies for membership inference. In: International Conference on Machine Learning. pp. 5558–5567. PMLR (2019)

24. Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-okvqa: A benchmark for visual question answering using world knowledge. In: European conference on computer vision. pp. 146–162. Springer (2022)

25. Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., Zettlemoyer, L.: Detecting pretraining data from large language models. In: 12th International Conference on Learning Representations, ICLR 2024 (2024)

26. Shi, W., Hu, Z., Bin, Y., Liu, J., Yang, Y., Ng, S.K., Bing, L., Lee, R.K.W.: Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models. arXiv preprint arXiv:2406.17294 (2024)

27. Sidorov, O., Hu, R., Rohrbach, M., Singh, A.: Textcaps: a dataset for image captioning with reading comprehension. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 742–758. Springer (2020)

28. Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017)

29. Wang, Z., Bovik, A.C.: Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine **26**(1), 98–117 (2009)

30. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing **13**(4), 600–612 (2004)

31. Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., Lin, B.Y.: Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464 (2024)

32. Yan, B., Li, K., Xu, M., Dong, Y., Zhang, Y., Ren, Z., Cheng, X.: On protecting the data privacy of large language models (LLMs): A survey. arXiv preprint arXiv:2403.05156 (2024)

33. Yeom, S., Giacomelli, I., Fredrikson, M., Jha, S.: Privacy risk in machine learning: Analyzing the connection to overfitting. In: IEEE 31st Computer Security Foundations symposium. pp. 268–282. IEEE (2018)

34. Zhang, J., Das, D., Kamath, G., Tramèr, F.: Membership inference attacks cannot prove that a model was trained on your data. Proc. IEEE SatML (2025)