Title: LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Label-Only Membership Inference Attack; Large Vision-Language Models
Abstract: Large vision-language models (VLLMs) have driven significant progress in multi-modal systems, enabling a wide range of applications across domains such as healthcare, education, and content generation. Despite the success, the large-scale datasets used to train these models often contain sensitive or personally identifiable information, raising serious privacy concerns. To audit and better understand such risks, membership inference attacks (MIAs) have become a key tool. However, existing MIAs against VLLMs predominantly assume access to full-model logits, which are typically unavailable in many practical deployments. To facilitate MIAs in a more realistic and restrictive setting, we propose a novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model’s top-1 prediction is available. Within this framework, we propose three effective attack methods, all of which exploit the intuition that training samples are more likely to be memorized by the VLLMs, resulting in outputs that exhibit higher semantic alignment and lower perplexity. Our experiments show that our framework surpasses existing label-only attack adaptations for different VLLMs and competes with state-of-the-art logits-based attacks across all metrics on three widely used open-source VLLMs and GPT-4o.

LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language Models

Abstract

Large vision-language models (VLLMs) have driven significant progress in multimodal systems, enabling a wide range of applications across domains such as healthcare, education, and content generation. Despite the success, the large-scale datasets used to train these models often contain sensitive or personally identifiable information, raising serious privacy concerns. To audit and better understand such risks, membership inference attacks (MIAs) have become a key tool. However, existing MIAs against VLLMs predominantly assume access to full-model logits, which are typically unavailable in many practical deployments. To facilitate MIAs in a more realistic and restrictive setting, we propose a novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model's top-1 prediction is available. Within this framework, we propose three effective attack methods, all of which exploit the intuition that training samples are more likely to be memorized by the VLLMs, resulting in outputs that exhibit higher semantic alignment and lower perplexity. Our experiments show that our framework surpasses existing label-only attack adaptations for different VLLMs and competes with state-of-the-art logits-based attacks across all metrics on three widely used open-source VLLMs and GPT-4o.

1 Introduction

Large vision-language models (VLLMs) [31, 56, 55, 36] have proven to be formidable tools capable of understanding and generating multimodal content, leading to substantial advancements in areas such as image captioning [25, 8] and visual question answering [42, 13, 28]. These models are typically trained on vast datasets that integrate images and textual descriptions, often sourced from the Internet. While this extensive training enables impressive performance, it also raises substantial privacy issues, especially in relation to the inadvertent inclusion of sensitive or proprietary information in the training data [32]. For example, identity-related details, such as names linked to faces can be revealed, even when both alignment and fine-tuning are performed using anonymized datasets [3]. In the medical domain, VLLMs trained on datasets comprising medical images and associated diagnoses may unintentionally disclose private information of patients [22, 33]. Moreover, numerous studies have demonstrated that large language models (LLMs) are capable of memorizing substantial portions of training data prior to overfitting and exhibit a lower tendency to forget during training [48, 4, 50]. Given that VLLMs are built upon LLMs, it is reasonable to expect that they inherit this memorization behavior as well.

One of the most common methods used to tackle these risks is membership inference attack (MIA), which tries to distinguish whether a particular sample has been utilized in training model [45]. However, prevailing methods largely presume access to the full set of model logits, enabling the

estimation of token-level probabilities. These probabilities facilitate the computation of critical statistical measures such as log-likelihood, entropy, and perplexity, which have been shown to serve as strong indicators for inferring membership status. Given this constraint, a key challenge emerges: How can we effectively infer membership when only the final output of the target model is available?

To address this problem, we propose a novel framework named LOMIA for label-only MIAs on VLLMs, which is a more realistic and challenging scenario. For any target data, LOMIA contains two stages: regression stage and inference stage. The former is trying to query the surrogate model to fit the relationship between related features and the perplexity (PPL) of generated descriptions. The latter utilizes the regression model to predict the corresponding PPL value and detect the membership by threshold.

Overall, our key contributions can be summarized as follows.

To the best of our knowledge, we are the first to explore whether pre-trained VLLMs suffer from membership inference attacks under the label-only setting.

We propose three straightforward yet impactful label-only MIAs targeting the pre-training phase of VLLMs. These attacks are based on text-text features, image-text features, and dual features.

Extensive evaluations conducted on two datasets and three open-source pre-trained VLLMs demonstrate that LOMIA performs comparably to existing logits-based attacks across a range of evaluation metrics. We also show the effectiveness of our methods on the closedsource model GPT-4o, which achieved an AUC of 0.669 when evaluating the image-text feature attack method (ITFA).

2 Related Work

Pre-trained Large Vision-Language Models. Building on the success of LLMs, VLLMs integrate visual perception with language generation. Pioneering works like CLIP [39] established cross-modal alignment between images and text, laying the groundwork for deeper integration. Recent VLLMs such as LLaVA [30], MiniGPT-4 [56] and LLaMA-Adapter [12] leverage pre-trained vision encoders (e.g., CLIP-ViT [39]) and LLMs (e.g., LLaMA [49], Vicuna [7]) to fuse visual and textual tokens via lightweight projection layers. This approach allows the visual and textual components to work together effectively without retraining the entire system. With additional instruction tuning [30] on aligned image-text data, these models now perform well on a variety of multi-modal tasks, such as image captioning [25], and have become a strong foundation for many downstream applications. However, their reliance on large-scale training data raises concerns regarding memorization and unintended information leakage.

Membership Inference Attack (MIA). MIAs seek to determine whether a particular data sample was included in the training set of a given machine-learning model [45]. These attacks have received increasing attention due to their significant implications for data privacy, particularly when training datasets contain personally identifiable information. These attacks exploit the tendency of machine learning models to behave differently on training data compared to unseen data, primarily as a consequence of overfitting [16]. MIAs are generally categorized into three types: white-box attacks, which assume access to internal model parameters [34, 45]; grey-box attacks, which require access to model logits [27]; and black-box attacks, which rely solely on model outputs [15, 23, 45]. Blackbox MIAs include simple metric-based approaches leveraging statistical signals, as well as more sophisticated techniques based on shadow models [5, 2, 23], which attempt to replicate the behavior of the target model. Extensive research has explored various MIA strategies across different machinelearning models, with notable progress in applying MIAs to classification models [45, 23], generative models [52, 9], regression models [14], and embedding models [47, 10]. Recent efforts have extended these techniques to LLMs [43, 15, 4, 51, 11]. However, relatively few studies have focused on MIA risks in multi-modal models.

Early work focuses on multi-modal classification models that combine CNN encoder (e.g., ResNet152 [35] or VGG-16 [38]) with LSTM decoder [17]. This approach [17] assumes access to output confidences and the feasibility of constructing shadow models. Subsequent efforts extend to MIAs against CLIP [19]. In contrast, this work [19] proposes a metric-based black-box MIA using cosine

similarity and model output confidences to infer membership status. A critical limitation of prior investigations lies in their focus on smaller-scale models, which tend to overfit due to limited representational capacity. More recent studies begin to evaluate attacks on VLLMs but typically rely on access to full-model logits [27]. Such grey-box approaches compute statistical metrics like MaxRényi-K% or ModRényi ∗ as membership signals. Nevertheless, access to token-level probabilities is rarely granted in deployed systems. To better align with real-world scenarios, Hu et al. [18] proposed a black-box membership inference attack method against fine-tuned VLLMs. However, their approach depends on shadow models and assumes access to a large amount of data highly similar to the training set, an assumption rarely met in practical deployments. To the best of our knowledge, our work is the first to explore effective label-only MIAs against pre-trained VLLMs without relying on shadow training or access to logits of the target model.

3 Threat model

Membership inference attack represents a class of privacy breaches aimed at machine learning models, especially those trained with sensitive information [45]. Formally, considering a trained VLLM M θ , a data sample x , and external adversary knowledge denoted by K , the definition of MIA A is as follows:

Here, 0 denotes that x is not included in the training dataset of M θ , while 1 indicates that it is.

Goal of adversary. During the training process of a target model, data samples used for training are referred to as members, while those excluded from training are called non-members. The adversary aims to determine whether a given data sample is a member or a non-member of the training dataset.

Adversary's Capabilities. Given a target VLLM M tar, the adversary can only query the target model and observe its outputs, without access to its internal structure or parameters. To make the scene more realistic, the adversary is not permitted to train a shadow model or utilize any shadow dataset. Even if permitted, the computational cost of training shadow models at the scale of pre-trained VLLMs remains prohibitively high for most adversaries.

Metrics. We assess our method using three key metrics: area under the curve (AUC) of the receiver operating characteristic (ROC) curve, balanced accuracy, and TPR@1%FPR (true positive rate at 1% false positive rate). AUC serves as an average measure across all false positive rates. Balanced accuracy evaluates attack effectiveness by measuring prediction accuracy in a balanced dataset of members and non-members. TPR@1%FPR provides a practical metric widely adopted in prior research [5, 19].

4 Method

In this section, we present LOMIA, a label-only membership inference attack framework tailored for VLLMs. LOMIA comprises three complementary attack variants, each leveraging a distinct feature extraction strategy to reveal training data leakage: (1) text-text feature attack (TTFA), (2) image-text feature attack (ITFA), and (3) dual-feature attack (DUFA). All methods proposed follow a two-stage procedure consisting of regression and inference, as illustrated in Figure 1.

4.1 Text-Text Feature Attack (TTFA)

The core concept of TTFA is based on the VLLMs' ability to remember information from past training datasets and deliver similar responses. Consequently, an intuitive approach is to directly query the VLLMs and leverage semantic similarity between model outputs and ground truth texts to infer the membership status of training samples. However, the signal strength alone is insufficient due to potential noise in generation. To address this limitation, we incorporate the PPL of the generated description as a corrective signal, capturing confidence in generation. The joint modeling via linear regression offers a simple yet effective way to quantify the relationship between semantic similarity and generation fluency.

Regression Stage. In the regression stage, we establish a relationship between text similarity and PPL using a surrogate model M sur. For a target image x ( i ) v and its corresponding ground-truth x ( i ) t

(b)The Inference Stage

Ground-truth

A kittenpeeksthroughthefabric

Simt2t

Sim2t

with its paw on a pillow.

AttackerQuery

Ground-truth

TargetImage

Member

Non-Member

Instruction

TargetImage

T2T Similarity

I2T Similarity

Whether<?

This image shows a small orange.

this image.

Target VLLM

Thekitten has big,round eyes andi

;Perplexity=F(Sim+2+, Simi2t)

whitefur.

(a)The Regression Stage

Theimagefeatures a small,fluffy;

The

P1

SurrogateVLLM

orange andwhitekitten withbig,

round eyes.

image

P2

T2T Similarity

I2T Simlrty

PPL

round

Pn-2

Perplexity

Ground-truth

TargetImage

eyes

Pn-1

Sim2t

Sim2t

Perplexity=F(Sim+2+,Simi2t)=

A*Sim+2++B*Simi2++C

from the dataset D , we use a surrogate model to generate description ˆ x ( i ) t , and calculate the semantic similarity between generated description and ground truth S t2t = sim (ˆ x ( i ) t , x ( i ) t ) , where sim ( · , · ) represents a semantic similarity function computed using a sentence embedding model. Given the generated description sequence ˆ x ( i ) t = w 1 , . . . , w n , we calculate the PPL of the i-th generated description:

where log P ( w n | w 1 , . . . , w n -1 ) represents the logarithm of the conditional probability of the n-th word given the previous words. Finally, we establish a linear relationship between S t2t and PPL to model their joint effect on membership inference signals, which can be formulated as follows:

Inference Stage. During the inference stage, we apply the regression model to predict the membership score of target samples using the target model M tar. For each test sample ( x ( i ) v , x ( i ) t ) , we use the target model to generate a description ˆ x ′ ( i ) t and calculate the semantic similarity between generated description and ground truth S ′ t2t = sim ( ˆ x ′ ( i ) t , x ( i ) t ) . Then we apply the regression model to predict membership score and classify the sample as a member if the score exceeds a threshold τ , which can be defined by:

4.2 Image-Text Feature Attack (ITFA)

ITFA is based on the hypothesis that training samples induce tighter cross-modal alignment within the learned joint embedding space. Specifically, VLLMs trained on image-text pairs tend to map member images and their generated descriptions to highly overlapping regions in the CLIP embedding space, resulting in elevated cosine similarity.

Regression Stage. In this stage, we establish a relationship between image-text alignment and membership signals. After querying the surrogate model M sur to obtain the description ˆ x ( i ) t , we utilize CLIP to calculate the cosine similarity between generated text and target image S i2t = sim (ˆ x ( i ) t , x ( i ) v ) . Next, we calculate the PPL of the generated descriptions in the same manner as TTFA and then establish a linear relationship between the image-text similarity and PPL:

Inference Stage. The inference stage of ITFA uses the cosine similarity between text generated from the target model to predict membership score, which can be defined by:

4.3 Dual Feature Attack (DUFA)

While TTFA and ITFA individually capture complementary aspects of memorization: semantic preservation and cross-modal alignment, they may individually fail under different sources of noise. DUFA integrates dual features in a regression model, leveraging the joint patterns in VLLMs.

Regression Stage. In this stage, we learn a multivariate regression model to fit both text-text and image-text similarity to the PPL. This can be formulated as follows:

Inference Stage. The inference stage of DUFA requires to use both text-text and image-text features to predict membership score, which can be defined by:

5 Experiments

In this section, we conduct MIAs across three target models using various baselines, and our own methods: TTFA, ITFA, and DUFA. The evaluation setup is detailed in Section 5.1. Results for TTFA, ITFA, and DUFA are presented in Section 5.2. Additionally, an ablation study is included in Section 5.3.

5.1 Evaluation Setup

Models. Our evaluation focuses on three popular open-source VLLMs: LLaVA-1.5 [30], MiniGPT4 [56], and LLaMA-Adapter V2.1 [12], all providing full access to model weights, training process, and datasets. Here we test Vicuna-7B [7] for LLaVA and MiniGPT-4, and LLaMA-7B [49] for LLaMA-Adapter V2.1. We also test the effectiveness of our methods on closed-source model GPT-4o [1].

Datasets. Li et al. [27] created a dataset for MIAs on VLLMs, but it is unimodal and does not align with our task. Therefore, we developed a multi-modal dataset for MIAs against VLLMs:

LOMIA/LAION. Pre-trained VLLMs such as LLaVA-1.5, MiniGPT-4, and LLaMAAdapter V2 use images from the LAION [41], Conceptual Captions 3M [6], Conceptual 12M [6], and SBU Captions [37] datasets for pre-training [27]. Following Li et al. [27], we randomly sample a subset from the intersection of the datasets used by these three pre-trained VLLMs to serve as the member data. We then use the captions of the member data as input to query the stable-diffusion-v1-5 [40] to generate images that serve as non-member data. To ensure the validity of our MIA on VLLMs, we have 600 images in LOMIA/LAION (300 members and 300 non-members).

LOMIA/CC. MS COCO [29] is also a popular dataset used in the pre-training process of the target models, so we randomly select some images in this dataset as member data. We use a similar approach to generate non-member data with stable-diffusion-v1-5 [40]. We also have 600 images in LOMIA/CC (300 members and 300 non-members).

Baselines. We compare our LOMIA with seven existing state-of-the-art attack methods. For logits-based baselines, we select the PPL attack [4, 54], MIN-K% PROB [44], Aug-KL [27], MaxProb-Gap% PROB [27], MaxRényi-K% [27], and ModRényi ∗ [27]. For label-only baselines, we mainly consider the Query Attack proposed by Hu et al. [18]. The details of the baselines can be found in the Appendix A.

Evaluation Settings. The attack implementation is conducted on 4 NVIDIA 3090 GPUs. To assess the performance of our LOMIA framework, we use LLaVA-1.5-7B as the surrogate model. To mitigate the impact of varying text lengths, we fix the maximum token length at 32. For computing text-text similarity, we use the all-MiniLM-L6-v2 model [21], which encodes each sentence into dense representations via mean pooling over the transformer outputs, followed by cosine similarity measurement. Image-text similarity is calculated using CLIP [39], which maps visual and textual inputs into a shared embedding space through dedicated encoders before computing their cosine similarity.

5.2 Main Results

We compare our methods with six advanced logits-based attack methods and one label-only attack method. Complete results on LOMIA/LAION and LOMIA/CC are summarized in Table 1, Table 2 and Table 3 respectively. Our extensive empirical evaluation has revealed several insightful observations about MIAs against pre-trained VLLMs, which we discuss in detail below.

LOMIA performs comparably to existing logits-based attacks. Table 1 and Table 2 show that our three attack methods based on the framework of LOMIA significantly outperforms other label-only attacks across all datasets and target model settings. Even when compared with existing logitsbased attacks, our methods achieve similar or superior performance. Specifically, our TTFA method achieves the highest AUC scores among all label-only attacks on nearly all target models, consistently surpassing the baseline Query Attack by a substantial margin. For instance, TTFA improves over the Query Attack by 0.105 and 0.069 in AUC on LLaVA, as shown in Table 1 and Table 2, respectively. In terms of TPR@1%FPR, our methods remain competitive with strong logits-based baselines. For instance, TTFA achieves up to 3.0% on LLaMA Adapter shown in Table 2, clearly outperforming the Query Attack (0.0%). While top-performing logits-based methods like MaxRényi ( α = 2 ) still yield the highest TPR values (e.g., 5.6%-7.3%), LOMIA approaches offer a practical alternative under stricter threat models. Notably, ITFA demonstrates robust TPR gains across architectures, indicating its effectiveness even when model architectures diverge.

Comparison among TTFA, ITFA, and DUFA. DUFA performs best when the surrogate model shares the same architecture as the target model, since its design relies on feature-level alignment between the two (e.g., AUC = 0.621 on LOMIA/LAION and 0.630 on LOMIA/CC). However, when the surrogate and target architectures differ, DUFA's advantage diminishes, and its performance can fall below that of single-feature variants like TTFA or ITFA. This indicates that different target models vary in their sensitivity to similarity-based features, likely due to differences in architecture or training that affect their vulnerability to text-based similarity attacks in controlled settings. Notably, when we expanded the dataset and used real images, as detailed in our Appendix C, DUFA consistently outperformed the other methods across all target models, while TTFA's performance dropped significantly. This reversal demonstrates that DUFA offers greater robustness as experimental conditions become more realistic and challenging. By combining text-to-text and

image-to-text similarity features, DUFA is able to maintain strong performance across varying data scales and distributions.

LOMIA on GPT-4o. Table 3 demonstrates the effectiveness of our proposed attack methods within the LOMIA framework on GPT-4o. Among the three methods, ITFA achieves the best performance on the LOMIA/LAION (AUC 0.669). DUFA and TTFA exhibit consistent results across both datasets. Notably, DUFA achieves the highest AUC and Balanced Accuracy on LOMIA/CC and competitive performance on LOMIA/LAION. This indicates that even closed-source models are vulnerable to privacy attacks. Especially, LLMs like GPT-4o, which are trained on a variety of datasets, are more likely to memorize a greater amount of private information.

Robustness across datasets. We observe that the relative performance ranking among TTFA, ITFA, and DUFA remains largely consistent across both LOMIA/LAION and LOMIA/CC. This suggests that LOMIA is not overfitted to dataset-specific patterns and maintains attack efficiency across varying data distributions, reinforcing its practical applicability in real-world settings.

5.3 Ablation Study

In this section, we conduct comprehensive experiments to investigate the impact of different settings to LOMIA, including different T2T embedding models, different image-text matching models, different max token length and different temperature. Additional experimental results are available in the Appendix C.

Different T2T embedding models. Table 4 shows ablation experiments on LOMIA/LAION under different embedding models to explore its effectiveness, such as all-MiniLM-L6-v2 [21], bge-large-env1.5 [53], mxbai-embed-large-v1 [20], and UAE-Large-V1 [26]. While TTFA shows high sensitivity to embedding model choice (e.g., 3.0% TPR@1%FPR for all-MiniLM-L6-v2 but 0.6% for bge-largeen-v1.5), DUFA demonstrates remarkable robustness, maintaining AUC variations within a stable range (from 0.618 to 0.621) across various embedding models.

Different image-text matching models. We further explore the impact of different image-text matching models on LOMIA/LAION. Including CLIP [39] and BLIP [24]. These two models differ substantially in architecture and alignment strategies: CLIP employs contrastive learning for feature alignment, while BLIP uses a generative decoder conditioned on image features. Results in Table 5 show that CLIP maintains a stable performance in both ITFA (AUC = 0.617) and DUFA (AUC = 0.621), with a balanced accuracy of 0.591-0.593, reflecting its strong robustness to single-modal noise. In contrast, BLIP fails to detect in ITFA (AUC = 0.485, balanced accuracy 0.518) but shows significant improvement in DUFA (AUC increased by 18.4% to 0.574), revealing that the additional text-text feature provides complementary information.

Different max token length. We conduct ablation experiments on LOMIA/LAION targeting the length of generated description texts. we restrict the max new tokens parameter of the generation to (32, 64, 72, 96, 128). Due to CLIP's limitation of the input sequence to 77 tokens, we restrict the parameters of TTFA and DUFA to 72. Figure 2a demonstrates that longer generated description texts are not always better for TTFA and ITFA. However, for DUFA, the AUC increases as the length of the generated descriptions grows.

Different temperature. Since the generative behavior of VLLMs is inherently modulated by temperature settings, we performed an ablation study on LOMIA/LAION to examine the impact of temperature on attack performance. The results in Figure 2b reveal method-specific temperature sensitivities: DUFA reaches its peak at temperature = 0.4 (AUC = 0.639) and remains consistently strong within the 0.2-0.6 range (AUC > 0.62), but its performance deteriorates markedly beyond 0.7, with a 10.3% drop observed above 0.8. In contrast, ITFA is highly sensitive to temperature, performing optimally only in the low-temperature range (0.0-0.2) before rapidly degrading. TTFA follows a similar trend, yet interestingly shows a performance rebound at a temperature of 1, suggesting a potential fallback effect rather than stable robustness. Collectively, these findings underscore the necessity of modality-aware and method-specific temperature scheduling in LOMIA, preserving DUFA's operation within its stable mid-range, and leveraging the low-temperature range of TTFA and ITFA.

TTFA

ITFA

DUFA

TTFA

ITFA

DUFA

0.65

0.65

0.64

0.638

0.64

0.639

9°0

0.63

AUC Score

Score

0.62

0.62

0.617

AUC

0.61

0.617

0.615

0.61

0.601

09°0

0.60

0.593

0.594

65°0

0.59

8S'0

0.583

0.577

0.58

0.57

0.573

32

64

72

96

128

0°0

0.2

0.4

0.6

0.8

1.0

MaxNew Token Length

Temperature

(a) Ablation study on Max New Tokens .

(b) Ablation study on Temperature .

6 Conclusion

In this paper, we explore the susceptibility of VLLMs to MIAs under the restrictive label-only setting. Our attention is directed towards the pre-training phase, which is not only more commonly encountered but also presents considerable technical challenges. We start by reviewing previous MIAs tailored to VLLMs and demonstrate that current methods mainly rely on logits-based attacks, which is a major limitation given the infeasibility of logits access in real-world deployments. Moreover, existing label-only approaches are inadequate when applied to pre-trained VLLMs. To address this issue, we present LOMIA, a novel label-only MIA framework that exploits sentence-level semantic similarity to approximate output perplexity as signals to distinguish membership. Comprehensive evaluations under rigorous benchmarks validate LOMIA's competitive performance over baselines. These findings highlight the urgent need for defenses specifically tailored to label-only MIAs.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z Wu. Scalable membership inference attacks via quantile regression. Advances in Neural Information Processing Systems , 36:314-330, 2023.

Simone Caldarella, Massimiliano Mancini, Elisa Ricci, and Rahaf Aljundi. The phantom menace: unmasking privacy leakages in vision-language models. arXiv preprint arXiv:2408.01228 , 2024.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633-2650, 2021.

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP) , pages 1897-1914. IEEE, 2022.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3558-3568, 2021.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03-30-vicuna , 3(5), 2023.

Xuelong Dai, Kaisheng Liang, and Bin Xiao. Advdiff: Generating unrestricted adversarial examples using diffusion models. In European Conference on Computer Vision , pages 93-109. Springer, 2024.

Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulnerable to membership inference attacks? In International Conference on Machine Learning , pages 8717-8730. PMLR, 2023.

Vasisht Duddu, Antoine Boutet, and Virat Shejwalkar. Quantifying privacy leakage in graph embedding. In MobiQuitous 2020-17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services , pages 76-85, 2020.

Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum? id=PAWQvrForJ .

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023.

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10867-10877, 2023.

Umang Gupta, Dimitris Stripelis, Pradeep K Lam, Paul Thompson, José Luis Ambite, and Greg Ver Steeg. Membership inference attacks on deep regression models for neuroimaging. In Medical Imaging with Deep Learning , pages 228-251. PMLR, 2021.

Yu He, Boheng Li, Liu Liu, Zhongjie Ba, Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. Towards label-only membership inference attack against pre-trained large language models. In USENIX Security , 2025.

Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR) , 54(11s):1-37, 2022.

Pingyi Hu, Zihan Wang, Ruoxi Sun, Hu Wang, and Minhui Xue. M 4 i: Multi-modal models membership inference. Advances in Neural Information Processing Systems , 35:1867-1882, 2022.

Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision-language models. arXiv preprint arXiv:2501.18624 , 2025.

Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. Practical membership inference attacks against large-scale multi-modal models: A pilot study. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4871-4881, 2023.

Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. Open source strikes bread -new fluffy embeddings model. mixedbread ai inc , 2024. URL https://www.mixedbread.ai/blog/ mxbai-embed-large-v1 .

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL , 2020.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems , 36:28541-28564, 2023.

Hao Li, Zheng Li, Siyuan Wu, Chengrui Hu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Seqmia: Sequential-metric based membership inference attack. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages 3496-3510, 2024.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871 , 2023.

Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems , 37:98645-98674, 2024.

Kaisheng Liang, Xuelong Dai, Yanjie Li, Dong Wang, and Bin Xiao. Improving transferable targeted attacks with feature tuning mixup. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 25802-25811, 2025.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296-26306, 2024.

Yihao Liu, Jinhe Huang, Yanjie Li, Dong Wang, and Bin Xiao. Generative ai model privacy: a survey. Artificial Intelligence Review , 58(1):1-47, 2025.

Xinqi Lyu, Yihao Liu, Yanjie Li, and Bin Xiao. Pla: Prompt learning attack against text-to-image generative models. arXiv preprint arXiv:2508.03696 , 2025.

Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP) , pages 691-706. IEEE, 2019.

Long D Nguyen, Dongyun Lin, Zhiping Lin, and Jiuwen Cao. Deep cnns for microscopic image classification by exploiting transfer learning and feature concatenation. In 2018 IEEE international symposium on circuits and systems (ISCAS) , pages 1-5. IEEE, 2018.

OpenAI. Gpt-4 technical report, 2023.

Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems , 24, 2011.

Hussam Qassim, Abhishek Verma, and David Feinzimer. Compressed residual-vgg16 cnn model for big data places image recognition. In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC) , pages 169-175. IEEE, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-adversarial visual question answering. Advances in Neural Information Processing Systems , 34:20346-20359, 2021.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations , 2023.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In ICLR , 2024.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE (SP) , pages 3-18, 2017.

Sizhkhy. open-images-captions-micro. https://huggingface.co/datasets/sizhkhy/ open-images-captions-micro , 2024.

Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security , pages 377-390, 2020.

Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems , 35:38274-38290, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Dong Wang, Yicheng Liu, Liangji Fang, Fanhua Shang, Yuanyuan Liu, and Hongying Liu. Balanced gradient penalty improves deep long-tailed learning. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5093-5101, 2022.

Rui Wen, Zheng Li, Michael Backes, and Yang Zhang. Membership inference attacks against in-context learning. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages 3481-3495, 2024.

Bingzhe Wu, Shiwan Zhao, Chaochao Chen, Haoyang Xu, Li Wang, Xiaolu Zhang, Guangyu Sun, and Jun Zhou. Generalization in generative adversarial networks: A novel perspective from privacy protection. Advances in Neural Information Processing Systems , 32, 2019.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pages 268-282. IEEE, 2018.

Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=d4UiXAHN2W .

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing visionlanguage understanding with advanced large language models. In The Twelfth International Conference on Learning Representations , 2023.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly express our motivation and insight in the Abstract and Introduction. The contributions are summarized in the Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly express our limitations in the Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There is no assumption or theoretical result in this work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper provides a detailed account of implementation details and hyperparameter settings of our method for reproducing in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The used datasets are publicly available. We will consider releasing the code upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide implementation details in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We fix the random seeds in experiments for fair comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the type of compute workers in implementation details in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This work focuses on membership inference attack, which is the most common privacy risk in machine learning. We discussed the potential impacts in Section 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the datasets and models used in our paper and give a brief introduction.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper has nothing to do with crowdsourcing and human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Details of Baselines

We summarize the details of the baselines as follows:

PPL Attack [4, 54] uses the perplexity PPL of a target sample x evaluated on the target model M θ as the membership score, making a prediction based on whether the negative perplexity exceeds a threshold: A ( x, θ ) = 1 [ -PPL ( x, M θ ) ≥ τ ] .

Aug-KL [27] uses KL-divergence to compare the logit distributions, where p is the logit distribution of the target sample x on the target model f θ , and q is the logit distribution of the augmented sample: A ( x, θ ) = 1 [ D KL ( p ∥ q ) ≥ τ ] .

MIN-K% PROB [44] determines the membership score by focusing on the k% of tokens exhibiting the lowest likelihoods: A ( x, θ ) = 1 [ 1 | Min -K( t ) | ∑ t i ∈ Min -K( t ) log p ( t i | t 1 , . . . , t i -1 ) ≥ τ ] .

Max-Prob-Gap% PROB [27] subtracts the second largest probability from the maximum probability in each token position and calculate the mean: A ( x, θ ) = 1 | Max -Gap( t ) | ∑ t i ∈ Max -Gap( t ) (max j p j -second max j p j ) ≥ τ .

MaxRényi-K% [27] the top K% from the sequence X with the largest Rényi entropies: A ( x, θ ) = 1 [ 1 | Max-K% ( X ) | ∑ i ∈ Max-K% ( X ) H α ( p ( i ) ) ≥ τ ] .

̸

ModRényi ∗ [27] extends MaxRényi-K% to the target-based scenarios: A ( x, θ ) = 1 [ -1 | α -1 | ( (1 -p y ) p | α -1 | y -(1 -p y ) + ∑ j = y p j (1 -p j ) | α -1 | -p j ) ≥ τ ] .

Query attack [18] queries the VLLMs a number of times to generate different descriptions. Then calculate the pairwise similarity between all these descriptions and take the average value as the similarity score for the image. If the average value exceeds a certain threshold, the image is classified as a member. Here, we set the number of queries to 5 and the temperature to 0.5, which performs the best.

B Details of VLLMs

C Additional experiments

C.1 Performance for LOMIA on larger VLLMs

We also test our LOMIA on larger VLLMs, such as LLaVA-1.5-13b and MiniGPT4-vicuna-13b, as shown in Table 7 and Table 8.

C.2 Different sample ratios on LOMIA/LAION

We explore the impact of sample ratio on the performance of LOMIA by testing ratios of 10%, 25%, 50%, 75%, and 100%. We still use the whole samples from LOMIA/LAION to fit the regression

model. During the inference phase, we adjust the number of samples by randomly selecting a specific ratio of samples from LOMIA/LAION to query the target model (LLaVA-1.5-7b). To ensure fairness, we average the results over five independent queries. Since the sampled data is randomly selected each time, the TTFA, ITFA, and DUFA methods are evaluated on different batches of data in each trial. Table 9, Table 10 and Table 11 show the attack results of different sample ratios.

C.3 Balanced Accuracy Results on different Max New Tokens and on different Temperature

Figure 3a and Figure 3b show the balanced accuracy results on different max new tokens and on different temperature.

C.4 Complete results of various attacks on larger dataset

To evaluate the impact of a larger test set, we expanded the LOMIA/LAION dataset to 1,000 samples (500 members and 500 non-members). The attack performance with this increased number of testing data points is reported in the Table 12 and Table 13.

TTFA

ITFA

DUFA

TTFA

ITFA

DUFA

0.62

0.63

0.62

0.61

0.606 0.605

0.620

0.61

Balanced Acc

0.601

BalancedAcc

0.610

0.60

09°0

0.59

0.59

0.59

8S°0

0.581

0.581

0.57

0.58

0.568

0.566

0.568

0.56

0.57

0.55

32

64

72

96

128

0.0

0.2

0.4

0.6

0.8

1.0

MaxNew Token Length

Temperature

(a) Ablation study on Max New Tokens .

(b) Ablation study on Temperature .

C.5 Complete results of various attacks when non-member data are real images

To evaluate the impact of real non-member, we randomly sampled 500 image-caption pairs from open-images-captions-micro on Hugging Face [46], ensuring exclusion of any data overlapping with COCO or LAION datasets. We also enlarged our LOMIA/LAION member data samples to 500, the final results can be found in the Table 14 and Table 15.

D Threshold Calibration

We use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [44, 27], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set.

E Limitations

One limitation of this work is that the TPR@1%FPR is relatively low, but it's important to note that TPR@1%FPR is an extremely stringent evaluation metric that requires high true positive rates while maintaining only 1% false positive rate. Even under this strict criterion, our method consistently outperforms existing label-only attack baselines and shows competitive performance with logits-based MIAs. Moreover, promising defense strategies include differentially private pre-training and machine unlearning, which aim to reduce memorization in VLLMs and thereby mitigate privacy risks. We leave this as a critical direction for future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the membership inference attack (MIA) on VLLMs in the label-only setting. This paper introduces 3 attacks variants: TTFA, ITFA, DUFA. The evaluation is conducted across open-source and close-source VLLMs demonstrating competitive attack results with logits-based attacks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- This paper tackles an important and timely problem for a realistic deployment threat model.\n- The proposed attacks is simple and modular.\n- The evaluations across VLLMs including GPT4-o shows competitive performance with logits-based MIAs.\n\n**Weaknesses**:\n- The attack success seems to depend on the alignment between surrogate and target models. In the realistic label-only scenario, where we might not know the underlying design (e.g., architectures) of target model, the attack performance might be degraded.\n- The absolute TPR@1%FPR is low, which might not be actionable in practice with small sample batches.\n- The evaluation is comprehensive in terms of number of VLLMs, but the number of testing data points is relatively small (i.e., 300/300 for membership/non-membership for each dataset)\n\nQUESTIONS:\n1. How does the attack performance change with different surrogate model? In other words, a comprehensive studies on the impact of the surrogate model to proposed attacks would comprehend this study.\n2. How is the attack performance with a more number of testing data point?\n3. In this paper, the author claims that this is the first study on MIAs on VLLMs. However, [17] seems to be a label-only MIA on VLLMs. Please clarify how your setting differs from [17], and if the claim of being the first still holds.\n\nLIMITATIONS:\nI suggest the authors to include a section to discuss limitations and ethical impact.\n\nPlease see my opinion on the limitations of this study in the **Strengths And Weaknesses**", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-15"}

comments: SUMMARY:
This paper explores a label-only membership inference attacks for pre-trained large vision-language models. Before this paper, existing MIAs against VLLM predominately assume access to full-model logits and it is unavailable in many practical deployments. Therefore, authors propose a novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model’s top-1 prediction is available. LOMIA constrcuts the relationship between sample similarities and perplexities to determine whether the sample is in member or non-member. Experiments show the effective of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Exploring label-only membership inference attacks for pre-trained large vision-language models is helpful to the practical deployments of VLLMs.
2. The proposed method is intuitive and novel.

Weaknesses:
1. Lack of detail optimization on attacking. How to optimize the Equations 3,6 and 9? 
2. Lack of the threshold of determining whether the sample is a member or non-member, which is very important to reproduce this paper.
3. The checklist is unfinished and it may conflict with the conference policy. Furthermore, this paper doesn't discuss limitations or social negative impacts.
4. Constructing the relationship between sample similarities and perplexities as an attack lacks a detailed explanation.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
No, this paper doesn't discuss limitations.
TITLE: Official Review
CLARITY: 1
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LOMIA, a novel label-only membership inference attack (MIA) framework targeting pre-trained large vision-language models (VLLMs). Unlike prior works that assume access to logits or use shadow models, LOMIA operates in a realistic black-box setting where only the model’s top-1 output is observable. The key idea is to use the semantic similarity between generated outputs and ground-truth captions as input to a surrogate regression model that predicts the perplexity (PPL) of the generated text. Membership is inferred based on whether the predicted PPL falls below a fixed threshold. The authors propose three attack variants (TTFA, ITFA, DUFA) and demonstrate their effectiveness across multiple open-source and closed-source VLLMs, including GPT-4o.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. **Realistic and underexplored threat model**  \n   The paper addresses a practically relevant but underexplored attack setting: performing membership inference on *pre-trained* VLLMs under a strict **label-only black-box** condition where only the top-1 prediction is observable. This setup closely reflects real-world deployment environments, including closed-source systems like GPT-4o, making the problem formulation highly impactful.\n\n2. **Clever and modular methodology**  \n   The use of a surrogate regression model to estimate PPL from semantic similarity (text-text and image-text) is conceptually clean and effective. This sim→PPL mapping cleverly bypasses the need for logits or internal model access and enables membership inference based on a single top-1 output, making the method compatible with real-world constraints.\n\n3. **Black-box applicability and zero-shot transfer**  \n   The paper successfully applies its attack to GPT-4o without access to internal structure, logits, or retraining, showing that the attack generalizes across architectures. This reinforces the practicality and transferability of the approach from open-source to commercial models.\n\n4. **Competitive performance under strict assumptions**  \n   Despite the lack of logits or shadow data, LOMIA achieves performance comparable to state-of-the-art logits-based attacks across multiple VLLMs. This supports the effectiveness of the similarity-based regression framework in capturing membership signals.\n\n---\n\n### **Weaknesses**\n\n1. **Use of diffusion-generated negatives may introduce domain shift**  \n   Non-member samples are generated using Stable Diffusion from captions. These images may differ stylistically or semantically from natural images in the training data. As a result, the model may flag them as non-members not because of lack of memorization, but due to unfamiliar visual features, conflating domain shift with membership status.\n\n2. **Lack of principled threshold calibration**  \n   The attack decision relies on whether the predicted PPL falls below a fixed threshold τ. However, τ is globally fixed and not adapted to each target model, despite clear differences in perplexity scales across architectures. This could lead to misclassification, especially when transferring across models. No analysis of threshold sensitivity or calibration strategy is provided.\n\n3. **Potential bias from embedding models**  \n   The sentence embedding models (e.g., MiniLM) used to compute semantic similarity may share domain overlap with the training data (e.g., LAION, CC3M), leading to inflated similarity scores regardless of memorization. The paper does not control for or discuss this possible confounder, which could skew the attack's reliability.\n\n4. **No evaluation of regression accuracy**  \n   Although the regression model (sim → PPL) is central to the approach, no standard regression metrics (e.g., MAE, RMSE, R²) are reported. This makes it hard to assess how well the surrogate regression model performs, or how regression noise propagates to final attack decisions.\n\n5. **Potential overreliance on architecturally similar surrogates**  \n   Although the paper presents the method as shadow-free, the surrogate and target models used in the experiments (e.g., LLaVA, MiniGPT-4, LLaMA-Adapter) may be architecturally similar. The paper does not explicitly specify the vision encoders used in each model, including MiniGPT-4, which is known to use a non-CLIP encoder (BLIP-2 ViT-G/14) by default. If CLIP-based encoders were used across all models, the results might reflect strong alignment with CLIP-specific representations. The lack of experiments on structurally different VLLMs such as InstructBLIP, BLIP-2, or ViT-G/Q-former-based architectures limits our ability to assess the generality and robustness of the attack design across vision backbones.\n\nQUESTIONS:\n### **Questions**\n\n1. **How do attacks perform when real unseen images (e.g., held-out from COCO/CC or external datasets) are used as non-members instead of diffusion-generated ones?**  \n   Diffusion-generated images may introduce domain shifts. Evaluating the attack with real non-members would help isolate the impact of memorization vs. unfamiliarity in visual appearance.\n\n2. **Have the authors explored the sensitivity of the attack to the choice of threshold τ?**  \n   Given that PPL scales may differ across models, has percentile-based or target-specific calibration been considered to improve generalization?\n\n3. **Can the authors report standard regression metrics (e.g., RMSE, R²) for the sim→PPL model?**  \n   Since the regression model underlies the final attack decision, it would be helpful to quantify its accuracy and reliability directly.\n\n4. **Could the authors clearly specify the vision encoder used in each target model (LLaVA, MiniGPT-4, LLaMA-Adapter)?**  \n   The paper mentions “e.g., CLIP-ViT”, but it is unclear whether all models—especially MiniGPT-4—used CLIP-based encoders. Clarifying this is important for understanding the attack’s generality.\n\n5. **More broadly, could the authors address the various concerns raised under the Weaknesses section of this review?**  \n   In particular, it would be helpful to see additional discussion or evidence regarding:  \n   - The generality of the attack beyond CLIP-based models (e.g., InstructBLIP)  \n   - How performance is affected by differences in model size (e.g., 7B vs. 13B)   \n   - The potential bias introduced by the sentence embedding model  \n   \n   Responses to these points would strengthen the case for the method’s robustness and broader applicability.\n\nLIMITATIONS:\nThe paper does not include an explicit \"Limitations\" section, nor does it adequately discuss key threats to validity. Specifically, the similarity between surrogate and target models is not critically examined, the use of CLIP-based similarity features is not evaluated on non-CLIP architectures, and the reliance on diffusion-generated non-members may introduce domain shift. These factors may affect the generality and robustness of the attack but are not acknowledged. Including a dedicated Limitations section would strengthen the transparency and credibility of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LOMIA, a label-only Membership Inference Attack (MIA) targeting Vision-Language Models (VLMs), where the attacker has access only to the final output and no other internal model information or parameters. LOMIA leverages a surrogate model to learn the relationship between related features and Perplexity (PPL). During the inference stage, it uses this learned relationship to estimate the PPL value and determine whether a given input sample was part of the model's training data\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper addresses a challenging and realistic threat model, where the attacker has access only to limited information (i.e., final output labels).\n\n- Experimental results demonstrate the effectiveness of the proposed method under this constrained setting.\n\n**Weaknesses**\n\n- The motivation behind the proposed approach is not clearly described. The method relies on a surrogate model to learn the relationship between certain features and PPL. This relationship is then used to estimate the PPL values of outputs from the target model. If the estimated PPL exceeds a predefined threshold, the input is classified as a member; otherwise, it is considered a non-member.\n However, this method makes a strong and potentially unrealistic assumption: that there exists a meaningful and transferable relationship between the surrogate model and the target model. For example, if the surrogate model is trained on natural image-text pairs and the target model is trained on domain-specific data such as medical images, there is no guarantee that the surrogate's regression parameters will generalize or offer meaningful guidance for the target model.\n\n\n- The evaluation uses 300 member and 300 non-member samples. However, the two sets differ in nature—member samples are real data, whereas non-member samples are synthetic images generated by Stable Diffusion v1.5. This discrepancy introduces a distributional gap, which may make it easier for the model to distinguish non-member samples, thus inflating the attack performance.\n\n\n- Surrogate models are assumed not to have seen the target training data. However, the paper lacks sufficient evidence or details to support this claim. Without clear verification, the results are questionable.\n\n\n- The target models evaluated in the paper were released around 2023, which may be outdated given the rapid advancement in VLMs. The authors are encouraged to evaluate their method on more recent models, such as those released in late 2024 or early 2025, to strengthen the relevance and robustness of their findings.\n\nQUESTIONS:\n1) How is the threshold value determined for deciding whether an input is a member or non-member?\n\n\n2) Can the authors provide more details about the surrogate models and clarify whether their training data differs from that of the target model?\n\n\n3) Can the authors provide results on more recent VLMs?\n\n4) Can the authors provide a clear justification or explanation for why the proposed method is effective? A deeper analysis or theoretical insight would help strengthen the contribution.\n\nLIMITATIONS:\nThe authors did not include the limitation or potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}