LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language Models
Authors: Yihao LIU
Xinqi LYU
Dong Wang
Yanjie Li
Bin Xiao
Venue: NeurIPS 2025 poster ()
Keywords: Label-Only Membership Inference Attack; Large Vision-Language Models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose the first label-only membership inference attacks against pre-trained large vision-language models.

Abstract: Large vision-language models (VLLMs) have driven significant progress in multi-modal systems, enabling a wide range of applications across domains such as healthcare, education, and content generation. Despite the success, the large-scale datasets used to train these models often contain sensitive or personally identifiable information, raising serious privacy concerns. To audit and better understand such risks, membership inference attacks (MIAs) have become a key tool. However, existing MIAs against VLLMs predominantly assume access to full-model logits, which are typically unavailable in many practical deployments. To facilitate MIAs in a more realistic and restrictive setting, we propose a novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model’s top-1 prediction is available. Within this framework, we propose three effective attack methods, all of which exploit the intuition that training samples are more likely to be memorized by the VLLMs, resulting in outputs that exhibit higher semantic alignment and lower perplexity. Our experiments show that our framework surpasses existing label-only attack adaptations for different VLLMs and competes with state-of-the-art logits-based attacks across all metrics on three widely used open-source VLLMs and GPT-4o.

LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language Models
Yihao Liu, Xinqi Lyu, Dong Wang, Yanjie Li, Bin Xiao" Department of Computing, The Hong Kong Polytechnic University (yihao5. 1iu, xinqi. wowllyyuunnnoe..emm.. comp. wang, yanjie. wil.:11101eeee.. polyu.hk, b. p.xiao@polyu. edu.hi.
Abstract
Large vision-language models (VLLMs) have driven significant progress in multi- modal systems, enabling wide range of applications across domains such healthcare, education, and content generation. Despite the success, the large-scale datasets used train these models often contain sensitive personally identifiable information, raising serious privacy concerns. To audit and better understand such risks, membership inference attacks (MIAs) have become a key tool. However, existing MIAs against VLLMs predominantly assume access to full-model logits, which are typically unavailable many practical deployments. To facilitate MIAs in a more realistic and restrictive setting, we propose novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model's top-1 prediction is available. Within this framework, we propose three effective attack methods, all of which exploit the intuition that training samples are more likely to be memorized by the VLLMs, resulting in outputs that exhibit higher semantic alignment and lower perplexity. Our experiments show that our framework surpasses existing label-only attack adaptations for different VLLMs and competes with state-of-ther of-the-art logits-based attacks across all metrics on three widely used open-source VLLMs and GPT-4o.
Introduction
Large vision-language models (VLLMs) 31 56, 55 36 have proven to be formidable tools capable of understanding and generating multimodal content, leading to substantial advancements areas such as image captioning 225 g and visual question answering [2]
13 28 These models are typically trained on vast datasets that integrate images and textual descriptions, often sourced from the Internet. While this extensive training enables impressive performance, it also raises substantial privacy issues, especially in relation to the inadvertent inclusion of sensitive or proprietary information in the training data [32]. For example, identity-related related details, such as names linked to faces can be revealed, even when both alignment and fine-tuning are performed using anonymized datasets (3]. In the medical domain, VLLMs trained on datasets comprising medical images and associated diagnoses may unintentionally disclose private information of patients [22] 33). Moreover, numerous studies have demonstrated that large language models (LLMs) are capable of memorizing substantial portions of training data prior to overfitting and exhibit a lower tendency to forget during training [48] 50).
Given that VLLMs are built upon LLMs, it is reasonable to expect that they inherit this memorization behavior as well.
One of the most common methods used to tackle these risks is membership inference attack (MIA), which tries to distinguish whether a particular sample has been utilized training model [45]. However, prevailing methods largely presume access to the full set of model logits, enabling the
Bin Xiao is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

estimation of token-level probabilities. These probabilities facilitate the computation of critical statistical measures such as log- logeliieihhodd. entropy, and perplexity, which have been shown serve as strong indicators for inferring membership status. Given this constraint, key challenge emerges: How can we effectively infer membership when only the final output of the target model is available?
To address this problem, we propose a novel framework named LOMIA for label-only MIAs on VLLMs, which a more realistic and challenging scenario. For any target data, LOMIA contains two stages: regression stage and inference stage. The former is trying to query the surrogate model to fit the relationship between related features and the perplexity (PPL) of generated descriptions. The latter utilizes the regression model to predict the corresponding PPL value and detect the membership by threshold.
Overall, our key contributions can be summarized as follows.
To the best of our knowledge, we are the first to explore whether pre-trained VLLMs suffer from membership inference attacks under the label-only setting.
We propose three straightforward yet impactful label-only MIAs targeting the pre-training phase of VLLMs. These attacks are based on text-text features, image-text features, and dual features.
Extensive evaluations conducted on two datasets and three open-source pre-trained VLLMs demonstrate that LOMIA performs comparably existing logits- basedd attacks across range of evaluation metrics. We also show the effectiveness of our methods on the closed- source model GPT-4o, which achieved AUC of 0.669 when evaluating the image-text feature attack method (ITFA).
2 Related Work
Pre-trained Large Vision-Language Models. Building the success of LLMs, VLLMS integrate visual perception with language generation. Pioneering works like CLIP 33 established cross-modal alignment between images and text, laying the groundwork for deeper integration. Recent VLLMS such as LLaVA 30 MiniGPT-4 [56) and LLaMA-Adapter [12] leverage pre-trained vision encoders (e.g., CLIP-ViT (39] and LLMs (e.g.. LLaMA [49), Vicuna 77) fuse visual and textual tokens via lightweight projection layers. This approach allows the visual and textual components to work together effectively without retraining the entire system. With additional instruction tuning (30) on aligned image-text data, these models now perform well on a variety of multi-moda tasks, such as image captioning (25), and have become a strong foundation for many downstream applications. However, their reliance on large-scale training data raises concerns regarding memorization and unintended information leakage.
Membership Inference Attack (MIA). MIAs seek to determine whether a particular data sample was included in the training set of a given mecaing model [45). These attacks have received increasing attention due to their significant implications for data privacy, particularly when training datasets contain personally identifiable information. These attacks exploit the tendency of machine learning models to behave differently on training data compared to unseen data, primarily as consequence of overfitting [16. MIAS are generally categorized into three types: white-bor attacks, which assume access to internal model parameters [4 45]; grey-box attacks, which require access to model logits [27]; and black-box attacks, which rely solely on model outputs [15 23 45). Black- box MIAs include simple metric-based approaches leveraging statistical signals, as well as more sophisticated techniques based on shadow models [5] 2 23 which attempt to replicate the behavior of the target model. Extensive research has explored various MIA strategies across different machine- learning models, with notable progress in applying MIAs to classification models 45 23 generative models (22 g
regression models [1], and embedding models [1]. Recent efforts have extended However, relatively few studies have focused on MIA
these techniques to LLMs 43 13
51
risks in multi-moda models.
Early work focuses on multi-modal classification models that combine CNN encoder (e.g., ResNet- 152 (35] or VGG-16 (38) with LSTM decoder [17]. This approach [17 assumes access to output confidences and the feasibility of constructing shadow models. Subsequent efforts extend to MIAs against CLIP [9]. In contrast, this work [19 proposes metric-based black-box MIA using cosine

---

similarity and model output confidences to infer membership status. A critical limitation of prior investigations lies in their focus on smaller-scale models. which tend to overfit due limited representational capacity. More recent studies begin to evaluate attacks on VLLMs but typically rely on access to full-model logits [27]. Such grey-box approaches compute statistical metrics like MaxRényi-K99 or ModRényi" as membership signals. Nevertheless, access to token-level probabilities is rarely granted in deployed systems. To better align with real-world scenarios, Hu et al. [18 proposed black-box membership inference attack method against fine-tuned VLLMs. However, their approach depends on shadow models and assumes access to a large amount of data highly similar to the training set, an assumption rarely met in practical deployments. To the best of our knowledge, our work is the first to explore effective label-only MIAs against pre-trained VLLMs without relying shadow training access to logits of the target model.
3 Threat model
Membership inference attack represents a class of privacy breaches aimed machine learning models, especially those trained with sensitive information [4]. Formally, considering trained VLLM Mo. a data sample 2,, and external adversary knowledge denoted by K, the definition of MIA as follows:
A xr Ma, K - 1).
(1)
Here, 0 denotes that x is not included the training dataset of Mo. while indicates that it is.
Goal of adversary. During the training process a target model, data samples used for training are referred to as members, while those excluded from training called non-members. The adversary aims to determine whether a given data sample is a member or a non-member of the training dataset.
Adversary's Capabilities. Given a target VLLM Mar, the adversary can only query the target model and observe its outputs, without access its internal structure parameters. To make the scene more realistic, the adversary is not permitted to train a shadow model or utilize any shadow dataset. Even permitted, the computational cost of training shadow models at the scale of pre-trained VLLMs remains prohibitively high for most adversaries.
Metries. We assess our method using three key metrics: area under the curve (AUC) of the receiver operating characteristic (ROC) curve, balanced accuracy, and TPR 1%FPR (true positive rate at 1 %% false positive rate). AUC serves as average measure across all false positive rates. Balanced accuracy evaluates attack effectiveness by measuring prediction accuracy in balanced dataset of members and non-members. TPR @ %FPR provides practical metric widely adopted in prior
research 5 19
4
Method
In this section, we present LOMIA, a label-only membership inference attack framework tailored for VLLMs. LOMIA comprises three complementary attack variants, each leveraging distinct feature extraction strategy to reveal training data leakage: (1) text-text feature attack (TTFA), (2) image-text feature attack (ITFA), and (3) dual-feature attack (DUFA), All methods proposed follow two-stage procedure consisting of regression and inference, as illustrated in Figure
4.1 Text-Text Feature Attack (TTFA)
The core concept of TTFA is based on the VLLMMS' ability to remember information from past training datasets and deliver similar responses. Consequently, an intuitive approach is to directly query the VLLMM and leverage semantic similarity between model outputs and ground truth texts to infer the membership status training samples. However, the signal strength alone is insufficient due to potential noise in generation. To address this limitation, we incorporate the PPL of the generated description as a corrective signal, capturing confidence in generation. The joint modeling via linear regression offers simple yet effective way to quantify the relationship between semantic similarity and generation fluency.
Regression Stage.
In the regression stage, we establish a relationship between text similarity and
PPL using surrogate model Marr. For target image (d) and its corresponding ground-truth x;
3

---

Stage
Ground- Iruth
pooks through its paw on pillo...... Affocker Query
Stanp
Ground-truid
Target Image
Menber
Non- Member
Target Image
Instruction
Whitter
Please describe this image.
image arange. lrttinn big, eyes
Target VLLM
Perplexity
[Simmg
Regression Stage
The Image fectures snall, fluffy: onange and white kitten with big. round eyes.
The
Surregate VLLM
image
Pg
PRL Perplexity
round
Pa.
Ground-truth
Target
eyes
Fdd
Slona
Bino
Perplexity F(Simaze, Simgs) Singr B Sin C
We the target samples with instructions to query the surrogate model to fit the relationship between related features and the perplexity of generated descriptions. (b) The Inference Stage: We first query the target model with the same target samples and instructions, then apply the regression model to predict the corresponding PPL value and detect the membership by threshold.
Figure Framework of LOMIA. (a) The Regression Stage:
from the dataset D, we use a surrogate model to generate description (. and calculate the semantic similarity between generated description and ground truth Si = sim e, _z(),, where sim(.:)) represents semantic similarity function computed using sentence embedding model. Given the generated description sequence 2/ w.,...,, Wn. we calculate the PPL of the i-th generated description:
= exp Σ log P(w./wi..... "., 1)
PPL
(2)
where log P(w.//W) Wn-1) represents the logarithm of the conditional probability of the n-th word given the previous words. Finally, establish linear relationship between Sia and PPL to model their joint effect on membership inference signals, which can be formulated as follows:
R: Sex + PPL, via min
Σ EIPPLO - KS - 6112.
(3)
k.b
Inference Stage. During the inference stage, we apply the regression model to predict the membership score of target samples using the target model Mur- For each test sample (zl()) (t)
the target model generate description 2/ and calculate the semantic similarity between generated description and ground truth S%2 sim( (t) ..)))) Then apply the regression model predict membership score and classify the sample as a member if the score exceeds a threshold т, which can be defined by:
Scorear -R((S)) -PP
(4)
Member, if Score T,
A(x)
(5)
Non-member, otherwise.
4.2
Image-Test Text Feature Attack (ITFA)
ITFA is based on the hypothesis that training samples induce tighter eross-modal alignment within the learned joint embedding space. Specifically, VL.MMs trained on image-text pairs tend to map member images and their generated deseriptions to highly overlapping regions the CLIP embedding space, resulting in elevated cosine similarity.

---

Regression Stage. In this stage, we establish a relationship between image-text alignment and membership signals. After querying the surrogate model Maur obtain the description a(") we utilize
CLIP to calculate the cosine similarity between generated text and target image Sia = sim ( se (
Next, we calculate the PPL of the generated descriptions in the same manner TTFA and then establish a linear relationship between the image-text similarity and PPL:
R. Sa H PPL, via min
Σ PPL k,b
S - b11?
(6)
Inference Stage. The inference stage of ITFA uses the cosine similarity between text generated from the target model to predict membership score, which can be defined by:
Score = -R(S)2) -PPPhreeicted where S/a - sim(2'e (
(7)
Member,
if Score;2 2 T.
A(x)
(8)
Non-member, otherwise.
4.3 Dual Feature Attack (DUFA)
While TTFA and ITFA individually capture complementary aspects of memorization: semantic preservation and cross-modal alignment, they may individually fail under different sources of noise. DUFA integrates dual features regression model, leveraging the joint patterns VLLMs.
Regression Stage. In this stage, we learn multivariate regression model to fit both text-text and image-text similarity the PPL. This can be formulated as follows:
R (Sau Siz) PPL, via,via: min
Σ - 12.
k2 bll
(9)
Inference Stage. The inference stage of DUFA requires to use both text-text and image-text features predict membership score, which can be defined by:
Score -R(S S.2)) PP (1) 2 (2(()
(10)
where S/2 sim (
(0) S/ sim(1") (0) ...))
Member,
if Score 2 T
A(x) Non member otherwise.
(11)
5
Experiments
In this section, we conduct MIAs across three target models using various baselines, and our own methods: TTFA, ITFA, and DUFA. The evaluation setup is detailed in Section Results for TTFA, ITFA, and DUFA are presented Section 5.2 Additionally, an ablation study is included in Section
5.3
5.1 Evaluation Setup
Models. Our evaluation focuses three popular open-source VLLMs: LLaVA-1.5 [30], MiniGPT- 4 (56), and LLaMA-Adapter V2.1 [12). all providing full access to model weights, training process, and datasets. Here we test Vicuna-7B 77] for LLaVA and MiniGPT-4, and LLaMA-7B (49 for LLaMA-Adapter V2.1. We also test the effectiveness of our methods on closed-soures model GPT-4o
Datasets. Li al. 27 created dataset for MIAs on VLL.Ms, but is unimodal and does not align with our task. Therefore, developed multi-modal dataset for MIAs against VLLMs:
5

---

LOMIA/LAION. Pre- Pre-trained VLLMs such as LLaVA-1.5, MiniGPT-4, and LLaMA- Adapter V2 use images from the LAION A., Conceptual Captions 3M [6]. Conceptual 12M [6. and SBU Captions 37 datasets for pre-training 27 Following Li et al. 277 we randomly sample a subset from the intersection of the datasets used by these three pre-trained VLLMM to serve as the member data. We then use the captions of the member data as input to query the stable-diffusion-V 1-5 [40] to generate images that serve as non-member data. To ensure the validity of our MIA on VLLMs, we have 600 images in LOMIA/LAION (300 members and 300 non-members).
LOMIA/CC. MS COCO [29] is also a popular dataset used in the pre-training process of the target models, we randomly select some images in this dataset member data. We use a similar approach to generate non-member data with caaleediifussoony [40]. We also have 600 images in LOMIA/CC (300 members and 300 non-members).
Baselines. We compare our LOMIA with seven existing state-of sfate--o-thh-aaee art attack methods. For logits-based baselines. select the PPL attack [4] [54], MIN-K% PROB [44), Aug-KL (271, Max- Prob-Gap% PROB (27), MaxRényi-K9t % (27). and ModRényi" [27]. For label-only baselines, we mainly consider the Query Attack proposed by Hu et al. 18 The details of the baselines can be found in the Appendix A
Evaluation Settings. The attack implementation is conducted on 4 NVIDIA 3090 GPUs. To assess the performance of our LOMIA framework, we use LLaVA-1.5-7B as the surrogate model. To mitigate the impact of varying text lengths, we fix the maximum token length at 32. For computing text-text similarity, we use the all-MiniL.M-L6-- dense representations via mean pooling over the transformer outputs, followed by cosine similarity measurement. Image-text similarity calculated using CLIP [39], which maps visual and textual inputs into a shared embedding space through dedicated encoders before computing their cosine similarity.
model 21]. which encodes each sentence into
5.2 Main Results
We compare our methods with six advanced logits- attack methods and one label-only attack method. Complete results on LOMIA/LAION and LOMIA/CC are summarized in Table Table and Table 3 respectively. Our extensive empirical evaluation has revealed several insightful observations about MIAS against pre-trained VLLMs, which we discuss in detail below.
2
LOMIA performs comparably to existing logits- based attacks. Table and Table show that our three attack methods based on the framework of LOMIA significantly outperforms other label-only attacks across all datasets and target model settings. Even when compared with existing logits- based attacks, our methods achieve similar or superior performance. Specifically, our TTFA method achieves the highest AUC scores among all label-only attacks on nearly all target models, consistently surpassing the baseline Query Attack by substantial margin. For instance. TTFA improves over the Query Attack by 0.5 and 0.069 AUC on LLaVA, as shown in Table Table respectively. In terms of TPR @1 1 %FPR, our methods remain competitive with strong Togits-based baselines. For instance, TTFA achieves up to 3.0% on LLaMA Adapter shown in Table [ clearly outperforming the Query Attack (0.0%). While performin logits-based methods like MaxRényi (a = 2) still yield the highest TPR values (c.g., 5.6%-7.3%) LOMIA approaches offer a practical alternative under stricter threat models. Notably, ITFA demonstrates robust TPR gains across architectures, indicating its effectiveness even when model architectures diverge.
Comparison among TTFA, ITFA, and DUFA. DUFA performs best when the surrogate model shares the same architecture as the target model, since its design relies on feature-level alignment between the two (c.g., AUC 0.621 on LOMIA/LAION and 0.630 on LOMIA/CC). However, when the surrogate and target architectures differ, DUFA's advantage diminishes, and its performance can fall below that of single-feature variants like TTFA or ITFA. This indicates that different target models vary in their sensitivity to similarity-basee based features, likely due differences in architecture or training that affect their vulnerability to text-based similarity attacks in controlled settings. Notably, when we expanded the dataset and used real images, as detailed our Appendix
DUFA consistently outperformed the other methods across all target models, while TTFA's performance dropped significantly. This reversal demonstrates that DUFA offers greater robustness as experimental conditions become more realistic and challenging. By combining text-to-text and

---

image-te Image-to-text to-text similarity features, DUFA is able to maintain strong performance across varying data scales and distributions.
Table I: Complete results of various attacks LOMIA/LAION.
Metrics
AUC
Balanced Acc t
T I&FPR
LLaMA Adapter
LLaMA
LLaMA
Target Model
LLaVA MiniGPT4
LLvV MiniGPT4
LLVVV MiniGPT4
Adapter
Adapter
Logits-based Attacks
Perplexity
0.625
0.597
0.603
0.611
0.580
0.586
3.0%
2.6%
2.6%
Aug. KL
0.551
0,526
0.449
0.551
0.546
0.510
2.0%
1.0%
1.6%
Max-Prob-Gap
0.588
0.579
0.562
0.586
0.565
0.561
1.3%
2.6%
1.3%
Min0% Prob
0.676
0.559
0.549
0.641
0.551
0.548
3.3%
1.6%
0.3%
Min10% Prob
0.738
0.569
0.564
0.693
0.553
0.561
2.0%
2.0%
0.6%
Min20% Prob
0.661
0.581
0.573
0.633
0.573
0.568
1.6%
1.3%
0.6%
ModRényi(a = 0.5)
0.628
0.588
0.606
0.601
0.575
0.583
4.6%
1.6%
1.3%
ModRényi(a = 2)
0.629
0.587
0.606
0.601
0.573
0.581
2.6%
2.6%
1.6%
Max0% Rényi (a 0.5)
0.616
0.547
0.593
0.548
0.610
3.0%
5.3%
Max Max10% Rényi
0.5)
0.659
0.559
0.534
0.620
0.556
0.538
4.6%
1.3%
2.0%
Max Réni (a 0.5)
0.581
0.581
0.563
0.580
0.575
0.565
5.3%
1.0%
1.0%
Max0% Rényi (a 1)
0.679
0.560
0.618
0.636
0.560
0.596
9.6%
3.0%
4.3%
Max R6%- (a = 1)
0.750
0.579
0.549
0.695
0.568
0.540
13.0%
3.3%
2.6%
Max 100% Rényi (a 1)
0.625
0.605
0.596
0.588
0.573
7.3%
1.6%
Max0% Rényi (a
0.696
0.555
0.576
0.668
0.556
0.576
10.0%
3.3%
5.6%
Max Rényi (a
0.775
0.577
0.565
0.718
0.560
0.566
3.6%
2.3%
1.3%
Max RunsRéééi (a = 2)
0.631
0.601
0.599
0.616
0.581
0.583
2.3%
1.6%
1.6%
Max0%Rénge (a 06)
0.676
0.555
0,551
0.641
0.543
0.550
3.3%
1.6%
1.3%
Max Ré%yényi
20)
0.738
0.569
0.564
0.693
0.555
0.561
2.0%
1.0%
Max 100% Rényi (a oo)
0.633
0.599
0.603
0.625
0.583
0.586
1.3%
1.3%
1.3%
Label-only Attacks
Query Attack
0.516
0.545
0.492
0.538
0.555
0.520
0.3%
0.3%
0.3%
TTFA (Ours)
0.601
0.589
0.601
0.596
0.566
2.6%
ITFA (Ours)
0.617
0,571
0.522
0.591
0.568
0.546
2.0%
3.0%
DUFA (Ours)
0.621
0.576
0.526
0.593
0.563
0.545
1.6%
1.6%
2.3%
LOMIA on GPT-4o. Table demonstrates the effectiveness of our proposed attack methods within the LOMIA framework on GPT-40. Among the three methods, ITFA achieves the best performance on the LOMIA/LAION (AUC 0.669). DUFA and TTFA exhibit consistent results across both datasets. Notably, DUFA achieves the highest AUC and Balanced Accuracy on LOMIA/CC and competitive performance LOMIA/LAION. This indicates that even closed-source models are vulnerable to privacy attacks. Especially, LLMs like GPT-4o, which are trained on variety of datasets, are more likely memorize greater amount of private information.
Robustness across datasets. We observe that the relative performance ranking among TTFA, ITFA, and DUFA remains largely consistent across both I.MIA/LATONN and LOMIA/CC. This suggests that LOMIA is not overfitted to dataset-specific patterns and maintains attack efficiency across varying data distributions, reinforcing its practical applicability in real-world settings.
5.3 Ablation Study
In this section, we conduct comprehensive experiments to investigate the impact of different settings to LOMIA, including different T2T embedding models, different image-text matching models, different max token length and different temperature. Additional experimental results are available in the Appendix
Different T2T embedding models. Table shows ablation experiments on LOMIA/LAION under different embedding models to explore its effectiveness, such as all- MiniLM-L6-y (1]. bge-large-en-
v1.5 (53] maxaii-embeedllrgee-r (20). and UAE-Large-VI (26). While TTFA shows high sensitivity to embedding model choice (e.g.. 3.0% TPR @ 1% for all-Minil MIl-MMniiMMLaa M-L6-v2 but 0.6% for bge-large-
en-v1.5), DUFA demonstrates remarkable robustness, maintaining AUC variations within stable
range (from 0.618 to 0.621) across various embedding models.

---

Table 2: Complete results of various attacks LOMIA/CC.
Metrics
AUC f
Balanced Acc 1
TPR@ ISFPR
LLaMA Adapter LLaVA MiniGPT4
I.LMMA Adapter
L.LMM
Target Model
LLaVA MiniGPT4
LLaVA MiniGPT4
Adapter
Logits-based Attacks
Perplexity
0,634
0.574
0.556
0.615
0.573
0.556
4.6%
3.0%
2.6%
Aug_KL KL
0,500
0.712
0.539
0,516
0.676
0.541
2.3%
3.6%
0.6%
Max-Prob-GG Pap
0.607
0.615
0.564
0.601
0.585
0.560
2.0%
6.3%
2.0%
Min0% Prob
0.612
0.775
0.566
0.600
0.708
0.558
2.0%
Min10% Prob
0.609
0.880
0.568
0.603
0.823
0.565
5.0%
3.0%
1.6%
Min20% Prob
0,608
0.825
0.565
0.588
0.766
0.560
4.3%
5.6%
1.6%
ModRényi(a = 0.5)
0.633
0.569
0.549
0.611
0.581
0.355
4.0%%
1.0%
2.0%
ModRényí(a = 2)
0.633
0.569
0.544
0.618
0.580
0.548
3.6%
1.3%
3.0%
Max0% (a 0.5)
0.627
0.808
0.565
0.598
0.771
0.560
5.0%
Max 10% Rénéi
0.5)
0.646
0.803
0.566
0.610
0.798
0.563
3.0%
2.3%
3.6%
Max 10x100% Rényi (a: 0.5)
0.618
0.569
0.605
0.590
0.565
0.596
1.6%
3.3%
1.6%
Max0% Rényi (a 1)
0.644
0.814
0.573
0.610
0.761
0.561
2.3%
7.3%
2.3%
Max Rényé (at = 1)
0.641
0.847
0.563
0.598
0.801
0.570
3.0%
4.3%
3.6%
Max 100% Rényi (a 1)
0.637
0.643
0.591
0.611
0.628
0.585
2.1%
3.3%
Max0% Rényi (a
0.618
0.799
0.573
0.590
0.736
0.563
4.6%
3.3%
1.0%
Max 10%% Rényi (a
0.642
0.891
0.591
0.615
018288
0.573
6.0%
7.0%
2.0%
Max 100% Rényi (a = 2)
0.642
0.711
0.591
0.621
0.666
0.575
4.6%
1.6%
4.3%
Max0% Rényi (a oc)
0.612
0.775
0.538
0.600
0.708
0.561
3.6%
2.0%
2.6%
Max 10%&Rény (a = 00)
0.610
0.880
0.585
0.581
0.823
0.568
5.0%
3.0%
3.6%
Max 100% Ré0% (a oo)
0.646
0.680
0.591
0.625
0.645
0,580
4.3%
2.0%
3.3%
Label-only Atlacks
Query Attack
0.599
0.551
0.536
0.586
0.553
0.539
0.1%
0.8%
0.0%
TTFA (Ours)
0.584
0.572
0.568
0.585
0.566
0.566
4.0%
3.3%
2.3%
ITFA (Ours)
0.625
0.540
0.571
0.605
0.551
0.560
2.0%
2.0%
DUFA (Ours)
0.630
0.560
0.579
0.601
0.555
0.558
2.3%
2.3%
2.6%
Table 3: Performance of LOMIA on GPT-4o.
AUC
Balanced Ace @PR@1 1%PR
Metries Datasets
LAION CC LAION CC LAION
LOMIA
TTFA
0.602
0.600
0.580
0.588
2.0%
2.0%
ITFA
0.669
0.608
0.635
0.585
2.3%
3,0%
DUFA
0.612 0.618 0.585
0.600
2.0%
2.6%
Table 4: Performance comparison of LOMIA under different sentence transformer settings on LAION.
AUC Balanced Ace 1 TPR@ 1%FR
Metrics Methods
TTFA DUFA TTFA DUFA TTFA DUFA
Sentence transformers
2l--MiniLM-L6--2
0.601 0.621 0.601
0.593
3.0%
1.6%
large-en-vl.5
0.577 0.618 0.580
0.593
0.6%
1.6%
0.582
0.618 0.581
0.593
1.3%
1.6%
UAE- UAE-Large-Vi
0.584 0.619 0.581
0.591
1.3%
1.6%
Different image-text matching models. We further explore the impact of different image-text matching models on LOMIA/LAION Including CLIP 39] and BLIP 24] These two models differ substantially in architecture and alignment strategies: CLIP employs contrastive learning for feature alignment, while BLIP uses generative decoder conditioned on image features. Results in Table show that CLIP maintains stable performance in both ITFA (AUC 0.617) and DUFA (AUC strong robustness to single-modal
0.621), with a balanced accuracy 0.591-0.593, reflecting noise. In contrast, BLIP fails to detect in ITFAA (AUC = 0.485, balanced accuracy 0.518) but shows significant improvement DUFA (AUC increased by 18.4% to 0.574), revealing that the additional text-text feature provides complementary information.

---

Table 5: Performance comparison of LOMIA under different image-text matching models settings on LAION.
1
Balanced Ace f TPR@@ 1%FPR
Metrics
Methods
ITFA DUFA ITTA DUFA ITFA DUFA
Image-text matching models)
CLIP
0.617 0.621 0.591
0.593
1.6%
1.6%
BLIP
0.485 0.574 0.518 0.585
0.6%
3.0%
Different max token length. We conduct ablation experiments on LOMIA/LAION targeting the length of generated description texts. we restrict the max new tokens parameter of the generation to (32, 64, 72, 96, 128). Due to CLIP's limitation of the input sequence to 77 tokens, restrict the parameters TIFA and DUFA to 72. Figure FigureRa demonstratess that longer generated description texts not always better for TTFA and ITFA. However, for DUFA, the AUC increases the length of the generated descriptions grows.
Different temperature. Since the generative behavior of VLLMs is inherently modulated by temperature settings, we performed an ablation study on LOMIA/LAION to examine the impact of temperature attack performance. The results in Figure 26 reveal method-specific temperature sensitivities: DUFA reaches its peak at temperature = 0.4 (AUC = 0.639) and remains consistently strong within the 0.2-0.6 range (AUC > 0.62), but its performance deteriorates markedly beyond 0.7, with a 10.3% drop observed above 0.8. In contrast, ITFA is highly sensitive to temperature, performing optimally only in the remperatue range (0.0-0.2) before rapidly degrading. TTFA follows similar trend, yet interestingly shows a performance rebound at temperature of 1, suggesting a potential fallback effect rather than stable robustness. Collectively, these findings underscore the necessity of modality-awe -warre and method-specife temperature scheduling in LOMIA, preserving DUFA's operation within its stable mid-range, and leveraging the low-temperature range TTFA and ITFA.
21M
.179
DUPS
TIPA
ITM
DUMS
4P1
0.638
Scorr
AUC
0.601
0.554
0.577
100
New Token
Temperature
(a) Ablation study Max New Tokens.
(b) Ablation study Temperature. Figure 2: Ablation study (a) on different max_new_lokers (b) on different temperature.
Conclusion
In this paper, we explore the susceptibility of VLLMs to MIAs under the restrictive label-only setting. Our attention is directed towards the pre-training phase, which is not only more commonly encountered but also presents considerable technical challenges. We start by reviewing previous MIAS tailored to VLLMs and demonstrate that current methods mainly rely logits based attacks, which major limitation given the infeasibility of logits access in real-world deployments. Moreover, existing label-only approaches are inadequate when applied to pre-trained VLLMs. To address this issue, we present LOMIA, a novel label-only MIA framework that exploits sentence-level semantic similarity to approximate output perplexity as signals to distinguish membership. Comprehensive evaluations under rigorous benchmarks validate LOMIA's competitive performance over baselines. These findings highlight the urgent need for defenses specifically tailored to label-only MIAs.

---

Acknowledgement
This work was supported in part by the Hong Kong Research Grants Council's (RGC) General Research Fund (GRF) under Grant PolyU 15201323.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXii:2303.0077770 2023.
[2] Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z Wu. Scalable membership inference attacks via quantile regression. Advances in Neural Information Processing Systems, 36:314-33333 330, 2023.
[3] Simone Caldarella, Massimiliano Mancini, Elisa Ricci, and Rahaf Aljundi. The phantom menace: unmask- ing privacy leakages vision-language models. arXiv preprint arXiv:2408.01223 2024.
[4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ultar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-266-6 (633-2650, 2021.
[5] Nicholas Carlini, Steve Chien, Milad Nast, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897-1914, IEEE, 2022.
[6] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[7] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, march 2023. URL https://lmsys. /ttp/booog222----3-vvvivvm 3(5), 2023.
[8] Xuelong Dai, Kaisheng Liang, and Bin Xiao. Advdifff: Generating unrestricted adversarial examples using diffusion models. European Conference on Computer Vision, pages 93-109. Springer, 2024.
[9] Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulnerable to membership inference attacks? In International Conference on Machine Learning, pages 8717-8730. PMLR, 2023.
[10] Vasisht Duddu, Antoine Boutet, and Virat Shejwalkar. Quantifying privacy leakage in graph embedding. In MobiQuitous 2020-17th EAI International Conference Mobile and Ubiquitous Systems: Computing, Networking and Services, pages 85, 2020.
[11] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024, http:ttts///: id=PAWDwrForJ
wwt/ forum??
[12] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arxiv preprint Xiv:v:2304. 15010, 2023.
[13] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10867-108 10877, 2023.
[14] Umang Gupta, Dimitris Stripelis. Pradeep K Lam, Paul Thompson, José Luis Ambite, and Greg Ver Steeg. Membership inference attacks on deep regression models for neuroimaging, In Medical Imaging with Deep Learning, pages 228-251. PMLR, 2021.
[15] Yu He, Boheng Li, Liu Liu, Zhongjie Ba. Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. Towards label-only membership inference attack against pre-trained large language models. In USENIX Security, 2025.
10

---

[16] Hongsheng Hu, Zoran Salcie, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Membership inference attacks machine learning: A survey, ACM Computing Surveys (CSUR), 54(11s):1-7,, 2022.
[17] Pingyi Hu, Zihan Wang. Ruoxi Sun, Hu Wang. and Minhui Xue. M i: Multi-modal models membership inference. Advances Neural Information Processing Systems, 35: 35:1867-1888 2022.
[18] Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision- laiiun-anngugg models. arXiv preprint 2rXiv:2511.1822 2025.
[19] Myeongseob Ko, Ming Jin, Chenguang Wang. and Ruoxi Jia. Practical membership inference attacks against large-scale multi-modal models: pilot study. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4871-4881, 2023.
[20] Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. Open source strikes bread new fluffy embeddings model. mixedbread ai inc, 2024. URL https://www wttp://wwwwwbb.dd..aa..om.a
Sabbal-ebbed-1ae laree-larreev?
[21] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising -equencc-to-sqqueeee pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
[22] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, (6:28541-285666 2023.
[23] Hao Li, Zheng Li, Siyuan Wu, Chengrui Hu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Seqmia: Dequential-metria based membership inference attack. In Proceedings of the 2024 on ACM SIGSAC Conference Computer and Communications Security, pages 3496-3510, 10, 2024.
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900 PMLR, 2022.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning. pages
PMLR, 2023.
[26]
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309. 12871, 2023.
[27] Zhan Li, Yongtao Wo, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. Member- ship inference attacks against large vision-languaa models. Advances in Neural Information Processing Systems, 37:98645 98674, 2024.
[28] Kaisheng Liang, Xuelong Dai, Yanjie Li, Dong Wang, and Bin Xiao. Improving transferable targeted atlacks with feature tuning mixup. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25802-25811, 2025.
[29] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th Furopean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
[31] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction uningg In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[32] Yihao Liu, Jinhe Huang, Yanjie Li, Dong Wang, and Bin Xiao. Generative ai model privacy: a survey. Artificial Intelligence Review, 58(1):1 47, 2025.
[33] Xinqi Lyu, Yihao Liu, Yanjic Li, and Bin Xiao. Pla: Prompt learning attack against text-to-image generative models. arXiv preprint arXiv:2508.0300 036966 2025.
[34] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposiun on security and privacy (SP), pages 691-706. IEEE, 2019.

---

[35] Long D Nguyen, Dongyun Lin. Zhiping Lin, and Jiuwen Cao. Deep cnns for microscopic image classifica- tion by exploiting transfer learning and feature concatenation. In 2018 IEEE international symposium on circuits and systems (ISCAS), pages 1-5. IEEE, 2018.
[36] OpenAT. Gpt-4 technical report, 2023.
[37] Vicente Ordonez. Girish Kulkami, and Tamara Berg. Im2text: Describing images using I million captioned photographs. Advances in neural information processing systems, 24, 2011.
[38] Hussam Qassim, Abhishek Verma, and David Feinzimer. Compressed residual 1g enn model for big data places image recognition. In 20/8 IEFE 8th annual compuring and communication workshop and conference (CCWC), pages 169-175. IEFF, 2018.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PmLR, 2021.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[41] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Cnombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open datasct of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 /1,02/ 14, 2021.
[42] Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. alvearaadversarial visual question answering. Advances in Neural Information Processing Systems, 34:20346 20359, 2021.
[43] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. The Twelfth International Conference on Learning Representations, 2023.
[44] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang. Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In ICLR, 2024.
[45] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE (SP). pages 3-18,2017,
[46] Sizhkhy. open- caapanges--cppiionss open- images captions- micro 2024.
https///wugguuggaaaeec/c//asssss
[47] Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference computer and communications security, pages 377-390, 2020.
[48] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[50] Dong Wang, Yicheng Liu, Liangji Fang, Fanhua Shang, Yuanyuan Liu, and Hongying Liu. Balanced gradient penalty improves deep long-tailed learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5093-5101, 2022.
[51] Rui Wen, Zheng Li, Michael Backes, and Yang Zhang. Membership inference attacks against in-context learning. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 3481 -3455,4999590 2024.
[52] Bingzhe Wu, Shiwan Zhao, Chaochao Chen, Haoyang Xu, Li Wang, Xiaolu Zhang, Guangyu Sun, and Jun Zhou. Generalization in generative adversarial networks: A novel perspective from privacy protection. Advances in Neural Information Processing Systems, 32, 2019.
[53] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
12

---

[54] Samuel Yeom. Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018
[55] Renrui Zhang. Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMe-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024. htt_httte///opreeiiiie
forum did-
[56] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly express our motivation and insight in the Abstract and Introduction. The contributions summarized in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We clearly express our limitations in the Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no assumption or theoretical result in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper provides detailed account of implementation details and hyperpa- rameter settings of our method for reproducing in Section 5
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The used datasets are publicly available, We will consider releasing the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.ccppulic//uuue http:///cc/giiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We fix the random seeds in experiments for fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the type of compute workers in implementation details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neurrps.. httppp///aaarrsssccoaaaalaaaaaaa
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work focuses on membership inference attack, which is the most common privacy risk in machine learning. We discussed the potential impacts in Section T
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Justification:
Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the datasets and models used in our paper and give brief introduction.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper has nothing to do with crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy for what should or should not be described.
/
20

---

A Details of Baselines
We summarize the details of the baselines follows:
PPL Attack (4.54)
the perplexity PPL of target sample x evaluated the target model Mg as the membership score, making a prediction based on whether the negative perplexity exceeds threshold: A(x, 0) = 1[-PP( Mo) 2 r].
Aug-KL [27] uses KL-divergence to compare the logit distributions, where p is the logit distribution of the target sample the target model fo. and q is the logit distribution of the augmented sample: A(x, 0) - [[xx. (p q) 2 T|
MIN-K% PROB [4
determines the membership score by focusing the 5% of tokens exhibiting the lowest likelihoods: A(r,0)
1 MM 24,EMin-K(e K(()) logp(ts44.....,444... 2
Max-Prob-Gap % PROB [27] subtracts the second largest probability from the maximum probability in each token position and calculate the mean: A(x, 0)
[Max IMax-Gapil Gap(t)] Σ,, Gapx-G (max, Pj second max;Pj) 2 T.
MaxRényi-K % [27] the top K% from the sequence with the largest Rényi entropies:
A(x,0) 1[ Max [*[ EieMix kS(X) Ho (p()) 2 r].
ModRényi 27 extends MaxRényi to the target-based scenarios: A(x, 0) = P) 2
1[-ja-11
(1 (1-Pp)pp
(1 Pv) +
Query attack [18] queries the VLLMs a number of times to generate different descriptions. Then calculate the pairwise similarity between all these descriptions and take the average value as the similarity score for the image. the average value exceeds a certain threshold, the image is classified temperature to 0.5, which performs the best.
member. Here, set the number of queries to and the
B
Details of VLLMs
Table 6: VLLM details
Model
Mini-GPT4
LLaVA 1.5
LLaMA Adapter v2.1
Base LLM
Vieuna-v1.5-1
Vicuna-v1.5-1 CLIP-ViT-L
LLaMA-7B CLIP-ViT-L
Vision processor
BLIP2/EVA-ViT-
C Additional experiments
C.1 Performance for LOMIA on larger VLLMS
We also test our LOMIA on larger VLLMs, such LLaVA-1.5-136 and MiniGPT4-viicua-113 13b, as shown in Table and Table
Table 7: Performance of LOMIA on L.aVA-1.5-13h.
AUC
Balanced Ace TPR 1%FRR
Metrics
Datasets LAION CC LAION CC LAION CC
LOMIA
TTFA
0.601
0.576
0.596
0.568
4.0%
4.0%
ITFA
0.594
0.601
0.578
0.575
2.6%
3.3%
DUFA
0.607
0.609
0.598
0.581
3.3%
3.3%
C.2 Different sample ratios LOMIA/LAION
We explore the impact of sample ratio on the performance of LOMIA by testing ratios of 10%, 25%, 50%, 75%, and 100%. We still use the whole samples from LOMIA/LAION to fit the regression
21

---

Table Performance of LOMIA on MiniGPT4-vicuna 13b.
Metries
AUC
Balanced Ace @ I%FPR
Datasets LAION CC LAION CC LAION
CC
LOMIA
TTFA
0,588
0,584
0,583
0.576
1.0%
4.6%
ITFA
0.573
0.539
0.580
0.546
0.6%
1.0%
DUFA
0.590
0.559 0.598
0.561
1.0%
1.6%
Table 9: Performance of TTFA under different sample settings on LOMIA/LAION.
Ratio
AUC1
Balanced Acet
TPR @ 1%FPR1
10%
0.637
0.662
6.3%
25%
0.589
0.604
1.5%
50%
0.584
0,589
3.8%
75%
0.590
0,594
3.3%
100%
0.601
0.601
3.0%
Table 10: Performance of ITFA under different sample settings on LOMIA/LAION.
Ratio
AUC1
Balanced Acet TPR @ I%FPR1
10%
0.662
0.662
3.8%
25%
0.614
0.606
2.2%
50%
0.642
0.624
1.7%
75%
0.615
0.594
2.1%
100%
0.617
0,591
1.6%
Table 11: Performance of DUFA under different sample settings on LOMIA/LAION.
Ratio
AUC1
Balanced Acet
TPR @ I%FPR1
10%
0.589
0.617
5.3%
25%
0.614
0,614
2.1%
50%
0.612
0.611
4.4%
75%
0,600
0.594
3.5%
100%
0.621
0.593
1.6%
model. During the inference phase, we adjust the number of samples by randomly selecting a specific ratio of samples from LOMIA/LAION query the target model (LLLVAA-1.577b) To ensure faimess, we average the results over five independent queries. Since the sampled data is randomly selected each time, the TTFA, ITFA, and DUFA methods evaluated on different batches of data in each trial. Table 9 Table 10 and Table I show the attack results of different sample ratios.
C.3 Balanced Accuracy Results different Max New Tokens and different Temperature
Figure 3a and Figure 3b show the balanced accuracy results on different max new tokens and on different temperature.
C.4 Complete results of various attacks on larger dataset
To evaluate the impact of a larger test set, expanded the LOMIA/LAION dataset to 1,000 samples (500 members and 500 non-members). The attack performance with this increased number of testing data points is reported in the Table 12 and Table B3
22

---

644
0400
0.606 0003
0.601
Belanced
0.583
0.300
0.300
0.5m
Max New Tolen Length
Temperaturr
(a) Ablation study on Max New Tokens.
(b) Ablation study on Temperature.
Figure Balanced accuracy results of LOMIA on different (a) max new new_tokens tokens. (b) temperature.
Table 12: Complete results of various attacks larger dataset.
Metries
AUC f
Balanced Acc
TPR@1 1%PPR
LLaMA Adapter
I.aMA Adapter
I.LMMA
Target Model
LLaVA MiniGPT4
LLaVA MiniGPT4
LLaVA MiniGPT4
Adapter
Logits-based based Attacks
Perplexity
0.628
0.586
0.588
0.601
0l,770
0.574
5.2%
2.6%
3.6%
Aug_KL KL
0.530
0.515
0.501
0.532
0.526
0.533
1.8%
1.2%
3.6%
Max-Prob-CG
0.606
0.574
0,552
0.591
0.565
0.557
1.4%
1.4%
0.6%
Min0% Prob
0.666
0.565
0.526
0.641
0.555
0.532
1.4%
Min10% Prob
0.641
0.563
0.546
0.659
0.552
0.546
4.0%
2.6%
1.8%
Min20% Prob
0.648
0.571
0.561
0.618
0.560
0.562
4.0%
2.8%
2.4%
ModRényí(a - 0.5)
0.629
0.581
0.588
0.593
0.568
0.576
3.0%
2.0%
2.0%
ModRényi(a = 2)
0.630
0.587
0.589
0.597
0.573
0.577
3.0%
2.6%
1.6%
Max0%Rényl (a 0.5)
0.596
0.528
0.622
0.574
0.532
0.594
3.8%
Max 10%
0.5)
0.664
0.545
0.547
0.627
0.547
0.540
5.2%
1.0%
1.2%
Max 100% Rényi (a: 0.5)
0.609
0.569
0.564
0.590
0.559
0.560
5.6%
1.4%
1.0%
Max0% Rényi (a 1)
0.635
0.559
0.610
0.594
0.551
0.591
5.2%
1.2%
4,0%
Max Rényé (a
0.724
0.571
0.557
0.666
0.562
0.550
7.8%
2.4%
1.6%
Max 100% Rényi (a: 1)
0.637
0.589
0.582
0.604
0.572
0.569
6.4%
3.0%
1.6%
Max0% Rényi (a
0.676
0.565
0.550
0.645
0.557
0.546
3.2%
3.6%
Max 10% Rényi
= 2)
0.748
0.573
0.552
0.688
0.566
0.562
4.4%
3.0%
2.2%
Max 100% Rényi (a 2)
0.639
0.590
0.588
0.612
0.571
0.570
5.8%
2.8%
1.8%
Max0% Rényi (a oc)
0.666
0.567
0.526
0.641
0.556
0.532
3.0%
2.2%
1.4%
Max 10%Rényi = 20)
0.704
0.566
0.545
0.659
0.551
0.546
4.0%
2.8%
1.8%
Max 100% Rény (a Do)
0.639
0.588
0.588
0.611
0.569
0.574
6.8%
2.6%
1.8%
Label-only Attacks
Query Attack
0.531
0.534
0.534
0.534
0.536
0.532
0.4%
0.55
0.5%
TTFA (Ours)
0.597
0,570
0.566
0.581
0.562
0.560
2.2 %
2.2 %
0.6%
(Ours)
0.620
0.572
0.594
0.568
0.567
1.4%
0.8%
2.6%
DUFA (Ours)
0.628
0.576
0.578
0.603
0.570
0.564
2.0%
1.0%
2.4%
Table 13: Performance of larger dataset on GPT-4o.
Model
AUC1
Balanced Acc f TPR FPR
TTFA
0.598
0.577
1.8%
ITFA
0.648
0.616
3.2%
DUFA 0.675
0.637
2.2%
C.5 Complete results of various attacks when non-member data are real images
To evaluate impact real non-member, we randomly sampled 500 image-caption pairs from pnee-iiaage-ccapioons-miii on Hugging Face [46), ensuring exclusion of any data overlapping with COCO LAION datasets. We also enlarged our LOMIA/LAION member data samples to 500, the final results can be found in the Table and and Table [5
23

---

Table 14: Complete results of various attacks when non-member data real images.
Metrics
AUC f
Balanced Acc 1
TPR@ I&FPR
LLaVA MiniGPT4 LLaMA
L.MM Adapter
L.LMMA
Target Model
Adapter LLaVA MiniGPT4
LL MiniGPT4
Adapter
Logits-based Attacks
Perplexity
0,640
0.589
0.591
0.604
0.584
0.573
3.2%
2.8%
2.8%
Aug_KL KL
0,544
0.512
0.494
0,552
0.546
0.316
0.8%
4.2%
0.8%
Max-Prob-GO Pap
0.620
0.575
0.567
0.595
0.570
0.576
1.6%
2.4%
0.8%
Min0% Prob
0.592
0.571
0.521
0.585
0.565
0.538
1.0%
1.0%
Min10% Prob
0.538
0.575
0.533
0.538
0.569
0.538
1.2%
2.2%
2.0%
Min20% Prob
0.571
0.579
0.588
0.569
0.572
0.573
4.0%
1.8%
2.8%
ModRényi(a = 0.5)
0.653
0.583
0.630
0.623
0.571
0.598
3.2%
2.4%
6.4%
ModRényí(a = 2)
0.658
0.588
0.629
0.627
0.575
0.593
3.6%
2.4%
5.0%
Max0% (a 0.5)
0.555
0.578
0.548
0.571
0.599
7.2%
Max 10% Rénéi
0.5)
0,560)
0,570
0.528
0.553
0.565
0.537
2.4%
1.4%
1.4%
Max 10x100% Rényi (a: 0.5)
0.611
0.584
0.547
0.601
0.578
0.545
3.6%
2.2%
2.2%
Max0% Rényi (a
0.601
0.579
0.654
0.582
0.565
0.622
2.2%
1.6%
6.8%
Max Rén (at 1)
0.710
0.571
0.574
0.652
0.562
0.565
8.2%
2.4%
1.6%
Max 100% Rényi (a 1)
0.642
0.584
0.611
0.579
0.574
2.2%
Max0% Rényi (a
0.596
0.577
0.693
0.582
0.564
0.650
3.4%
1.8%
2.8%
Max 10%% Rényi (a
0.688
0.573
0.586
0.645
0.566
0.583
5.6%
3.0%
2.6%
Max 100% Rényi (a = 2)
0.623
0.600
0.602
0.600
0.637
0.587
3.6%
3.6%
2.6%
Max0% Rényi (a oc)
0.647
0.703
0.688
0.628
0.640
0.532
3.6%
2.2%
7.6%
Max 10%&Rény (a 00)
0.666
0.710
0.573
0.634
0.645
0.572
3.2%
6.2%
3.6%
Max 100% Ré0% (a oO)
0.649
0.710
0.608
0.630
0.647
0.588
7.2%
5.8%
3.0%
Label-only Atlacks
Query Attack
0.593
0.554
0.536
0.573
0.559
0.546
0.5%
0.55
0.2%
TTFA (Ours)
0.579
0.571
0.545
0.571
0.566
0.544
1.2%
1.9%
1.8%
ITFA (Ours)
0.630
0.575
0.575
0,600
0.569
0.562
2.2%
1.8%
1.6%
DUFA (Ours)
0.635
0.580
0.578
0.605
0.576
0.566
2.6%
1.8%
2.6 %
Table 15: Performance real non-member dataset on GPT-4o.
Model AUC
Balanced Acc TPR@ 1PR@1%FPR
TTFA
0.663
0.625
4.2%
ITFA
0.609
0.592
1.2%
DUFA
0.665
0.645
3.4%
D Threshold Calibration
We use the AUC score, balanced accuracy, and TPR@lor FPR as our primary metrics. These metries are standard in the previous work, such as 44 27), both are threshold-independdee AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search small validation set.
Specifically,
E
Limitations
One limitation of this work is that the TPR@ 1%FP is relatively low, but it's important to note that TPR@ %FPR is an extremely stringent evaluation metric that requires high true positive rates while maintaining only 1% false positive rate. Even under this strict criterion, our method consistently outperforms existing label-only attack baselines and shows competitive performance with logits-based MIAs. Moreover, promising defense strategies include differentially private pre-training and machine unlcarning, which aim to reduce memorization in VLLMs and thereby mitigate privacy risks. We leave this critical direction for future work.


Reviews:
[{"review_id": "GeQi7DKxWc", "reviewer": "Reviewer_WrZE", "summary": "This paper proposes LOMIA, a label-only Membership Inference Attack (MIA) targeting Vision-Language Models (VLMs), where the attacker has access only to the final output and no other internal model information or parameters. LOMIA leverages a surrogate model to learn the relationship between related features and Perplexity (PPL). During the inference stage, it uses this learned relationship to estimate the PPL value and determine whether a given input sample was part of the model's training data", "questions": "1) How is the threshold value determined for deciding whether an input is a member or non-member?\n\n\n2) Can the authors provide more details about the surrogate models and clarify whether their training data differs from that of the target model?\n\n\n3) Can the authors provide results on more recent VLMs?\n\n4) Can the authors provide a clear justification or explanation for why the proposed method is effective? A deeper analysis or theoretical insight would help strengthen the contribution.", "limitations": "The authors did not include the limitation or potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses a challenging and realistic threat model, where the attacker has access only to limited information (i.e., final output labels).\n\n- Experimental results demonstrate the effectiveness of the proposed method under this constrained setting.\n\n**Weaknesses**\n\n- The motivation behind the proposed approach is not clearly described. The method relies on a surrogate model to learn the relationship between certain features and PPL. This relationship is then used to estimate the PPL values of outputs from the target model. If the estimated PPL exceeds a predefined threshold, the input is classified as a member; otherwise, it is considered a non-member.\n However, this method makes a strong and potentially unrealistic assumption: that there exists a meaningful and transferable relationship between the surrogate model and the target model. For example, if the surrogate model is trained on natural image-text pairs and the target model is trained on domain-specific data such as medical images, there is no guarantee that the surrogate's regression parameters will generalize or offer meaningful guidance for the target model.\n\n\n- The evaluation uses 300 member and 300 non-member samples. However, the two sets differ in nature—member samples are real data, whereas non-member samples are synthetic images generated by Stable Diffusion v1.5. This discrepancy introduces a distributional gap, which may make it easier for the model to distinguish non-member samples, thus inflating the attack performance.\n\n\n- Surrogate models are assumed not to have seen the target training data. However, the paper lacks sufficient evidence or details to support this claim. Without clear verification, the results are questionable.\n\n\n- The target models evaluated in the paper were released around 2023, which may be outdated given the rapid advancement in VLMs. The authors are encouraged to evaluate their method on more recent models, such as those released in late 2024 or early 2025, to strengthen the relevance and robustness of their findings.", "was_revised": true, "final_justification": "Thanks authors for the rebuttal. Your response addresses most of my concerns.", "rebuttal": "# Response to Reviewer WrZE\n\nThank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.\n\n## Response to Q1\nQ1: How is the threshold value determined for deciding whether an input is a member or non-member?\n\nAnswer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [1][3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.\n\n## Response to Q2\nQ2: Can the authors provide more details about the surrogate models and clarify whether their training data differs from that of the target model?\n\nAnswer: Thank you for your question. In this paper, we use LLaVA-1.5-7B as the surrogate model, the based LLM for LLaVA-1.5-7B is Vicuna v1.5 7B and the vision processor is CLIP-ViT-L. To ensure the validity of the experiment, we randomly sampled a subset from the intersection of the datasets used by the pre-trained target model mentioned in our paper. The use of a surrogate model is solely aimed at identifying the relationship between relevant features and the perplexity of generated descriptions, rather than determining whether a specific sample is a member. Additionally, [2] similarly exhibits training data overlap across surrogate models and target models, with no observed effect on experimental conclusions.\n\n## Response to Q3\nQ3: Can the authors provide results on more recent VLMs?\n\nAnswer: Thank you for your suggestion. As shown in Table 3, we have evaluated our methods on GPT-4o, which is one of the most recent and widely used VLLMs. This evaluation demonstrates that the privacy issues identified in our study are also present in state-of-the-art VLLMs. The detailed results can be found in the following table.\n\n**Table: Performance of LOMIA on GPT-4o**\n\n| Metrics / Datasets | AUC ↑ (LAION) | AUC ↑ (CC) | Balanced Acc ↑ (LAION) | Balanced Acc ↑ (CC) | TPR@1%FPR ↑ (LAION) | TPR@1%FPR ↑ (CC) |\n|--------------------|:-------------:|:----------:|:----------------------:|:-------------------:|:-------------------:|:----------------:|\n| **LOMIA**          |               |            |                        |                     |                     |                  |\n| TTFA               | 0.602         | 0.600      | 0.580                  | 0.588               | 2.0%                | 2.0%             |\n| ITFA               | 0.669         | 0.608      | 0.635                  | 0.585               | 2.3%                | 3.0%             |\n| DUFA               | 0.612         | 0.618      | 0.585                  | 0.600               | 2.0%                | 2.6%             |\n\n## Response to Q4\nQ4: Can the authors provide a clear justification or explanation for why the proposed method is effective? A deeper analysis or theoretical insight would help strengthen the contribution.\n\nAnswer: Thank you for your valuable question. The effectiveness of our proposed method can be attributed to the following reasons:\nFirst, it is well established that LLMs tend to assign higher probabilities to tokens and sequences that are more semantically or syntactically similar to their training data. This means that, for member samples, the model is more likely to generate outputs with higher confidence, resulting in lower perplexity scores. Conversely, for non-member samples, the model is less familiar with the content, leading to higher perplexity. Second, the process of text generation in LLMs inherently favors high-probability tokens, further amplifying the difference in perplexity between member and non-member samples. This property is inherited by VLLMs, as they are built upon LLM architectures and share similar probabilistic behaviors. To provide a deeper analysis, we also examine the relationship between relevant features and perplexity. Our results, summarized using standard regression metrics in the following table.\n\n**Table: Standard regression metrics for regression models**\n\n| Method | TTFA   | ITFA   | DUFA   |\n|--------|--------|--------|--------|\n| R²     | 0.4074 | 0.3174 | 0.4824 |\n| RMSE   | 0.0358 | 0.0384 | 0.0335 |\n\n## Response to W3\nMany thanks for your suggestions. To evaluate our attacks on real unseen images, we randomly sampled 500 image-caption pairs from open-images-captions-micro (2024 release) on Hugging Face, ensuring exclusion of any data overlapping with COCO or LAION datasets. We also enlarged our LOMIA/LAION member data samples to 500 (500 members + 500 non-members), the final results can be found in the following table（Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）:\n\n**Table: Complete results of various attacks when non-member data are real images**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.579         | 0.571            | 0.545                 | 0.571                  | 0.566                     | 0.544                          | 1.2%                | 1.9%                   | 1.8%                        |\n| ITFA (Ours)           | 0.630         | 0.575            | 0.575                 | 0.600                  | 0.569                     | 0.562                          | 2.2%                | 1.8%                   | 1.6%                        |\n| DUFA (Ours)           | 0.635         | 0.580            | 0.578                 | 0.605                  | 0.576                     | 0.566                          | 2.6%                | 1.8%                   | 2.6%                        |\n\n**Table: Performance of real non-member dataset on GPT-4o**\n\n| Model | AUC ↑ | Balanced Acc ↑ | TPR@1%FPR ↑ |\n|-------|-------|----------------|-------------|\n| TTFA  | 0.663 | 0.625          | 4.2%        |\n| ITFA  | 0.609 | 0.592          | 1.2%        |\n| DUFA  | 0.665 | 0.645          | 3.4%        |\n\n## Reference\n\n[1] Shi, Weijia, et al. \"Detecting Pretraining Data from Large Language Models.\" ICLR 2024.\n\n[2] He Y, Li B, Liu L, et al. \"Towards label-only membership inference attack against pre-trained large language models.\" USENIX Security 2025.\n\n[3] Ko M, Jin M, Wang C, et al. \"Practical membership inference attacks against large-scale multi-modal models: A pilot study.\" ICCV 2023."}, {"review_id": "GTQXgFftTz", "reviewer": "Reviewer_gVAv", "summary": "This paper presents LOMIA, a novel label-only membership inference attack (MIA) framework targeting pre-trained large vision-language models (VLLMs). Unlike prior works that assume access to logits or use shadow models, LOMIA operates in a realistic black-box setting where only the model’s top-1 output is observable. The key idea is to use the semantic similarity between generated outputs and ground-truth captions as input to a surrogate regression model that predicts the perplexity (PPL) of the generated text. Membership is inferred based on whether the predicted PPL falls below a fixed threshold. The authors propose three attack variants (TTFA, ITFA, DUFA) and demonstrate their effectiveness across multiple open-source and closed-source VLLMs, including GPT-4o.", "questions": "### **Questions**\n\n1. **How do attacks perform when real unseen images (e.g., held-out from COCO/CC or external datasets) are used as non-members instead of diffusion-generated ones?**  \n   Diffusion-generated images may introduce domain shifts. Evaluating the attack with real non-members would help isolate the impact of memorization vs. unfamiliarity in visual appearance.\n\n2. **Have the authors explored the sensitivity of the attack to the choice of threshold τ?**  \n   Given that PPL scales may differ across models, has percentile-based or target-specific calibration been considered to improve generalization?\n\n3. **Can the authors report standard regression metrics (e.g., RMSE, R²) for the sim→PPL model?**  \n   Since the regression model underlies the final attack decision, it would be helpful to quantify its accuracy and reliability directly.\n\n4. **Could the authors clearly specify the vision encoder used in each target model (LLaVA, MiniGPT-4, LLaMA-Adapter)?**  \n   The paper mentions “e.g., CLIP-ViT”, but it is unclear whether all models—especially MiniGPT-4—used CLIP-based encoders. Clarifying this is important for understanding the attack’s generality.\n\n5. **More broadly, could the authors address the various concerns raised under the Weaknesses section of this review?**  \n   In particular, it would be helpful to see additional discussion or evidence regarding:  \n   - The generality of the attack beyond CLIP-based models (e.g., InstructBLIP)  \n   - How performance is affected by differences in model size (e.g., 7B vs. 13B)   \n   - The potential bias introduced by the sentence embedding model  \n   \n   Responses to these points would strengthen the case for the method’s robustness and broader applicability.", "limitations": "The paper does not include an explicit \"Limitations\" section, nor does it adequately discuss key threats to validity. Specifically, the similarity between surrogate and target models is not critically examined, the use of CLIP-based similarity features is not evaluated on non-CLIP architectures, and the reliance on diffusion-generated non-members may introduce domain shift. These factors may affect the generality and robustness of the attack but are not acknowledged. Including a dedicated Limitations section would strengthen the transparency and credibility of the work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. **Realistic and underexplored threat model**  \n   The paper addresses a practically relevant but underexplored attack setting: performing membership inference on *pre-trained* VLLMs under a strict **label-only black-box** condition where only the top-1 prediction is observable. This setup closely reflects real-world deployment environments, including closed-source systems like GPT-4o, making the problem formulation highly impactful.\n\n2. **Clever and modular methodology**  \n   The use of a surrogate regression model to estimate PPL from semantic similarity (text-text and image-text) is conceptually clean and effective. This sim→PPL mapping cleverly bypasses the need for logits or internal model access and enables membership inference based on a single top-1 output, making the method compatible with real-world constraints.\n\n3. **Black-box applicability and zero-shot transfer**  \n   The paper successfully applies its attack to GPT-4o without access to internal structure, logits, or retraining, showing that the attack generalizes across architectures. This reinforces the practicality and transferability of the approach from open-source to commercial models.\n\n4. **Competitive performance under strict assumptions**  \n   Despite the lack of logits or shadow data, LOMIA achieves performance comparable to state-of-the-art logits-based attacks across multiple VLLMs. This supports the effectiveness of the similarity-based regression framework in capturing membership signals.\n\n---\n\n### **Weaknesses**\n\n1. **Use of diffusion-generated negatives may introduce domain shift**  \n   Non-member samples are generated using Stable Diffusion from captions. These images may differ stylistically or semantically from natural images in the training data. As a result, the model may flag them as non-members not because of lack of memorization, but due to unfamiliar visual features, conflating domain shift with membership status.\n\n2. **Lack of principled threshold calibration**  \n   The attack decision relies on whether the predicted PPL falls below a fixed threshold τ. However, τ is globally fixed and not adapted to each target model, despite clear differences in perplexity scales across architectures. This could lead to misclassification, especially when transferring across models. No analysis of threshold sensitivity or calibration strategy is provided.\n\n3. **Potential bias from embedding models**  \n   The sentence embedding models (e.g., MiniLM) used to compute semantic similarity may share domain overlap with the training data (e.g., LAION, CC3M), leading to inflated similarity scores regardless of memorization. The paper does not control for or discuss this possible confounder, which could skew the attack's reliability.\n\n4. **No evaluation of regression accuracy**  \n   Although the regression model (sim → PPL) is central to the approach, no standard regression metrics (e.g., MAE, RMSE, R²) are reported. This makes it hard to assess how well the surrogate regression model performs, or how regression noise propagates to final attack decisions.\n\n5. **Potential overreliance on architecturally similar surrogates**  \n   Although the paper presents the method as shadow-free, the surrogate and target models used in the experiments (e.g., LLaVA, MiniGPT-4, LLaMA-Adapter) may be architecturally similar. The paper does not explicitly specify the vision encoders used in each model, including MiniGPT-4, which is known to use a non-CLIP encoder (BLIP-2 ViT-G/14) by default. If CLIP-based encoders were used across all models, the results might reflect strong alignment with CLIP-specific representations. The lack of experiments on structurally different VLLMs such as InstructBLIP, BLIP-2, or ViT-G/Q-former-based architectures limits our ability to assess the generality and robustness of the attack design across vision backbones.", "was_revised": true, "final_justification": "I maintain my score of 4 (Borderline Accept). The paper addresses a realistic and underexplored threat model for label-only membership inference attacks against VLLMs and presents a conceptually clean and practically applicable methodology. The rebuttal was thorough and addressed many of my concerns, particularly regarding evaluation with real images, threshold calibration, regression accuracy reporting, and vision encoder specification.", "rebuttal": "# Response to Reviewer gVAv\n\nThank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.\n\n## Response to Q1\nQ1: How do attacks perform when real unseen images (e.g., held-out from COCO/CC or external datasets) are used as non-members instead of diffusion-generated ones?\n\nAnswer: Many thanks for your suggestions. We randomly sampled 500 image-caption pairs from open-images-captions-micro (2024 release) on Hugging Face, ensuring exclusion of any data overlapping with COCO or LAION datasets. We also enlarge our LOMIA/LAION member data samples to 500 (500 members + 500 non-members), the final results can be found in the following table. （Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）\n\n**Table: Complete results of our attacks when non-member data are real images**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.579         | 0.571            | 0.545                 | 0.571                  | 0.566                     | 0.544                          | 1.2%                | 1.9%                   | 1.8%                        |\n| ITFA (Ours)           | 0.630         | 0.575            | 0.575                 | 0.600                  | 0.569                     | 0.562                          | 2.2%                | 1.8%                   | 1.6%                        |\n| DUFA (Ours)           | 0.635         | 0.580            | 0.578                 | 0.605                  | 0.576                     | 0.566                          | 2.6%                | 1.8%                   | 2.6%                        |\n\n**Table: Performance of real non-member dataset on GPT-4o**\n\n| Model | AUC ↑ | Balanced Acc ↑ | TPR@1%FPR ↑ |\n|-------|-------|----------------|-------------|\n| TTFA  | 0.663 | 0.625          | 4.2%        |\n| ITFA  | 0.609 | 0.592          | 1.2%        |\n| DUFA  | 0.665 | 0.645          | 3.4%        |\n\n## Response to Q2\nQ2: Have the authors explored the sensitivity of the attack to the choice of threshold τ?\n\nAnswer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [1][3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.\n\n## Response to Q3\nQ3: Can the authors report standard regression metrics (e.g., RMSE, R²) for the sim→PPL model?\n\nAnswer: Many thanks for your suggestions. The standard regression metrics for regression models are listed in the following Table.\n\n**Table: Standard regression metrics for regression models**\n\n| Method | TTFA   | ITFA   | DUFA   |\n|--------|--------|--------|--------|\n| R²     | 0.4074 | 0.3174 | 0.4824 |\n| RMSE   | 0.0358 | 0.0384 | 0.0335 |\n\n## Response to Q4\nQ4: Could the authors clearly specify the vision encoder used in each target model (LLaVA, MiniGPT-4, LLaMA-Adapter)?\n\nAnswer: Many thanks for your suggestions. The versions and base models of target VLLMs are listed in the following Table.\n\n**Table: VLLM details**\n\n| Model                | Mini-GPT4      | LLaVA 1.5      | LLaMA Adapter v2.1 |\n|----------------------|----------------|----------------|--------------------|\n| Base LLM             | Vicuna-v1.5-7B | Vicuna-v1.5-7B | LLaMA-7B           |\n| Vision processor     | BLIP2/EVA-ViT-G| CLIP-ViT-L     | CLIP-ViT-L         |\n\n## Response to Q5\nQ5: More broadly, could the authors address the various concerns raised under the Weaknesses section of this review?\n\n**W1: The generality of the attack beyond CLIP-based models.**  \nAnswer: We have evaluated study when surrogate model is Mini-GPT4, the vision processor for Mini-GPT4 is BLIP2/EVA-ViT-G, the results are listed in the following table:\n\n**Table: Complete results of our attacks on LOMIA/LAION when the surrogate model is MiniGPT4**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.589         | 0.592            | 0.560                 | 0.575                  | 0.600                     | 0.552                          | 2.2%                | 1.6%                   | 1.0%                        |\n| ITFA (Ours)           | 0.576         | 0.583            | 0.515                 | 0.570                  | 0.583                     | 0.527                          | 1.0%                | 2.2%                   | 0.6%                        |\n| DUFA (Ours)           | 0.592         | 0.600            | 0.551                 | 0.600                  | 0.590                     | 0.563                          | 1.2%                | 1.8%                   | 1.2%                        |\n\n**W2: How performance is affected by differences in model size (e.g., 7B vs. 13B)**\n\nAnswer: We also test our LOMIA on larger VLLMs, such as LLaVA-1.5-13b and MiniGPT4-vicuna-13b, which can be found in appendix:\n\n**Table: Performance of LOMIA on LLaVA-1.5-13b**\n\n| Metrics / Datasets | AUC ↑ (LAION) | AUC ↑ (CC) | Balanced Acc ↑ (LAION) | Balanced Acc ↑ (CC) | TPR@1%FPR ↑ (LAION) | TPR@1%FPR ↑ (CC) |\n|--------------------|:-------------:|:----------:|:----------------------:|:-------------------:|:-------------------:|:----------------:|\n| TTFA               | 0.601         | 0.576      | 0.596                  | 0.568               | 4.0%                | 4.0%             |\n| ITFA               | 0.594         | 0.601      | 0.578                  | 0.575               | 2.6%                | 3.3%             |\n| DUFA               | 0.607         | 0.609      | 0.598                  | 0.581               | 3.3%                | 3.3%             |\n\n**Table: Performance of LOMIA on MiniGPT4-vicuna-13b**\n\n| Metrics / Datasets | AUC ↑ (LAION) | AUC ↑ (CC) | Balanced Acc ↑ (LAION) | Balanced Acc ↑ (CC) | TPR@1%FPR ↑ (LAION) | TPR@1%FPR ↑ (CC) |\n|--------------------|:-------------:|:----------:|:----------------------:|:-------------------:|:-------------------:|:----------------:|\n| TTFA               | 0.588         | 0.584      | 0.583                  | 0.576               | 1.0%                | 4.6%             |\n| ITFA               | 0.573         | 0.539      | 0.580                  | 0.546               | 0.6%                | 1.0%             |\n| DUFA               | 0.590         | 0.559      | 0.598                  | 0.561               | 1.0%                | 1.6%             |\n\n**W3: The potential bias introduced by the sentence embedding model**\n\nAnswer: Thank you for raising this important point. We acknowledge that some sentence embedding models (e.g., MiniLM) may have been partially trained on datasets such as COCO or LAION, which are also used in VLLM training. However, the sentence embedding model is used solely for measuring semantic similarity, not for text generation. Since embedding models encode text into vector representations to capture semantic relationships rather than generate content. Even if the embedding model was trained on datasets that overlap with VLLM training data, this would not introduce bias in our similarity measurements because: (1) the embedding model's role is to map semantically similar texts to nearby points in vector space, which is a fundamentally different task from text generation; (2) our evaluation focuses on whether the VLLM can generate captions that are semantically similar to ground truth, which requires the embedding model to have good general semantic understanding rather than memorization of specific samples. We will clarify this point and discuss the potential impact in the revised manuscript.\n\n## Reference\n\n[1] Shi, Weijia, et al. \"Detecting Pretraining Data from Large Language Models.\" ICLR 2024.\n\n[2] He Y, Li B, Liu L, et al. \"Towards label-only membership inference attack against pre-trained large language models.\" USENIX Security 2025.\n\n[3] Ko M, Jin M, Wang C, et al. \"Practical membership inference attacks against large-scale multi-modal models: A pilot study.\" ICCV 2023."}, {"review_id": "9taWyKN1z8", "reviewer": "Reviewer_HBHB", "summary": "This paper explores a label-only membership inference attacks for pre-trained large vision-language models. Before this paper, existing MIAs against VLLM predominately assume access to full-model logits and it is unavailable in many practical deployments. Therefore, authors propose a novel framework: label-only membership inference attacks (LOMIA) targeting pre-trained VLLMs where only the model’s top-1 prediction is available. LOMIA constrcuts the relationship between sample similarities and perplexities to determine whether the sample is in member or non-member. Experiments show the effective of the proposed method.", "questions": "See Weaknesses.", "limitations": "No, this paper doesn't discuss limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Exploring label-only membership inference attacks for pre-trained large vision-language models is helpful to the practical deployments of VLLMs.\n2. The proposed method is intuitive and novel.\n\nWeaknesses:\n1. Lack of detail optimization on attacking. How to optimize the Equations 3,6 and 9? \n2. Lack of the threshold of determining whether the sample is a member or non-member, which is very important to reproduce this paper.\n3. The checklist is unfinished and it may conflict with the conference policy. Furthermore, this paper doesn't discuss limitations or social negative impacts.\n4. Constructing the relationship between sample similarities and perplexities as an attack lacks a detailed explanation.", "was_revised": true, "final_justification": "Leaving aside the compliance of checklist, I am satisfied with this paper and the response also solves my concerns. Therefore, I increase the rating and hope the authors add more implementaion details or release the code for reproducing and pushing the development of the community.", "rebuttal": "# Response to Reviewer HBHB\n\nThank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.\n\n## Response to Q1\nQ1: Lack of detail optimization on attacking. How to optimize the Equations 3,6 and 9?\n\nAnswer: Many thanks for your valuable suggestions. Equation 3 employs closed-form least squares optimization for text-to-text similarity mapping:\n\n$$\nR : S_{t2t} \\mapsto PPL, \\quad \\min_{k,b} \\sum \\left\\| PPL^{(i)} - k S_{t2t}^{(i)} - b \\right\\|^2\n$$\n\nwhere we minimize L2 loss between text-to-text similarity and PPL using the analytical solution:\n\n$$\nk = \\frac{n\\sum xy - \\sum x \\sum y}{n\\sum x^2 - (\\sum x)^2}\n$$\n\n$$\nb = \\frac{\\sum y - k\\sum x}{n}\n$$\n\nwith $x$ representing text-to-text similarity scores and $y$ representing PPL.\n\nEquation 6 follows the same optimization framework but leverages CLIP-based image-to-text features:\n\n$$\nR : S_{i2t} \\mapsto PPL, \\quad \\min_{k,b} \\sum \\left\\| PPL^{(i)} - k S_{i2t}^{(i)} - b \\right\\|^2\n$$\n\nutilizing identical closed-form solutions where $x$ now represents CLIP image-to-text similarity scores and $y$ representing PPL.\n\nEquation 9 extends to multivariate optimization:\n\n$$\nR : (S_{i2t}, S_{t2t}) \\mapsto PPL, \\quad \\min_{k_1, k_2, b} \\sum \\left\\| PPL - k_1 S_{t2t}^{(i)} - k_2 S_{i2t}^{(i)} - b \\right\\|^2\n$$\n\nemploying normal equations:\n\n$$\n\\boldsymbol{\\beta} = (\\mathbf{X}^T \\mathbf{X})^{-1} \\mathbf{X}^T \\mathbf{y}\n$$\n\nwhere $\\boldsymbol{\\beta} = [k_1, k_2, b]^T$, feature matrix $\\mathbf{X}$ combines both text-to-text and image-to-text similarities, and $y$ representing PPL.\n\n## Response to Q2\nQ2: Lack of the threshold of determining whether the sample is a member or non-member, which is very important to reproduce this paper.\n\nAnswer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.\n\n## Response to Q3\nQ3: The checklist is unfinished and it may conflict with the conference policy. Furthermore, this paper doesn't discuss limitations or social negative impacts.\n\nAnswer: Thank you for pointing out the omission regarding the NeurIPS reproducibility checklist. I sincerely apologize for this oversight and appreciate the AC and reviewers’ understanding in still providing the opportunity for rebuttal. I will ensure that the checklist is properly completed in the camera-ready version, in full compliance with NeurIPS requirements. Additionally, I'll consider all the weaknesses provided to include in our limitations section. Thank you again for your attention to this important aspect of responsible reporting.\n\n## Response to Q4\nQ4: Constructing the relationship between sample similarities and perplexities as an attack lacks a detailed explanation.\n\nAnswer: Thank you for your valuable question. The detailed explanation for the relationship between sample similarities and perplexities are as follows:\nFirst, it is well established that LLMs tend to assign higher probabilities to tokens and sequences that are more semantically or syntactically similar to their training data. This means that, for member samples, the model is more likely to generate outputs with higher confidence, resulting in lower perplexity scores. Conversely, for non-member samples, the model is less familiar with the content, leading to higher perplexity. Second, the process of text generation in LLMs inherently favors high-probability tokens, further amplifying the difference in perplexity between member and non-member samples. This property is inherited by VLLMs, as they are built upon LLM architectures and share similar probabilistic behaviors. To provide a deeper analysis, we also examine the relationship between relevant features and perplexity. Our results, summarized using standard regression metrics in the following table.\n\n**Table: Standard regression metrics for regression models**\n\n| Method | TTFA   | ITFA   | DUFA   |\n|--------|--------|--------|--------|\n| R²     | 0.4074 | 0.3174 | 0.4824 |\n| RMSE   | 0.0358 | 0.0384 | 0.0335 |\n\n## References\n\n[3] Ko M, Jin M, Wang C, et al. \"Practical membership inference attacks against large-scale multi-modal models: A pilot study.\" ICCV 2023."}, {"review_id": "s2cY4t5xxN", "reviewer": "Reviewer_J72D", "summary": "This paper addresses the membership inference attack (MIA) on VLLMs in the label-only setting. This paper introduces 3 attacks variants: TTFA, ITFA, DUFA. The evaluation is conducted across open-source and close-source VLLMs demonstrating competitive attack results with logits-based attacks.", "questions": "1. How does the attack performance change with different surrogate model? In other words, a comprehensive studies on the impact of the surrogate model to proposed attacks would comprehend this study.\n2. How is the attack performance with a more number of testing data point?\n3. In this paper, the author claims that this is the first study on MIAs on VLLMs. However, [17] seems to be a label-only MIA on VLLMs. Please clarify how your setting differs from [17], and if the claim of being the first still holds.", "limitations": "I suggest the authors to include a section to discuss limitations and ethical impact.\n\nPlease see my opinion on the limitations of this study in the **Strengths And Weaknesses**", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n- This paper tackles an important and timely problem for a realistic deployment threat model.\n- The proposed attacks is simple and modular.\n- The evaluations across VLLMs including GPT4-o shows competitive performance with logits-based MIAs.\n\n**Weaknesses**:\n- The attack success seems to depend on the alignment between surrogate and target models. In the realistic label-only scenario, where we might not know the underlying design (e.g., architectures) of target model, the attack performance might be degraded.\n- The absolute TPR@1%FPR is low, which might not be actionable in practice with small sample batches.\n- The evaluation is comprehensive in terms of number of VLLMs, but the number of testing data points is relatively small (i.e., 300/300 for membership/non-membership for each dataset)", "was_revised": true, "final_justification": "I have carefully reviewed authors' rebuttal.\n\nAll my concerns have been addressed.\nThis study is pioneering and interesting on MI attack for pre-trained VLMs.\nHowever, my the limitation on the low TPR@1%FPR (making this less applicable) remains.\nTherefore, I keep my original rating.", "rebuttal": "# Response to Reviewer J72D\n\nThank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Additionally, thank you for bringing to my attention the issue with the unfilled NeurIPS reproducibility checklist. I apologize for this mistake, and I am grateful to the AC and reviewers for allowing me the chance to submit a rebuttal despite this. I will make sure to fully and accurately complete the reproducibility checklist in the camera-ready version, adhering to all NeurIPS guidelines. Furthermore, I will carefully address all the weaknesses highlighted by the reviewers in the limitations section. I appreciate your attention to this crucial aspect of responsible research reporting. Below, we respond to the key points raised.\n\n## Response to Q1\nQ1: How does the attack performance change with different surrogate model? In other words, a comprehensive studies on the impact of the surrogate model to proposed attacks would comprehend this study.\n\nAnswer: Thank you for raising this important point. We have conducted an ablation study to evaluate the impact of different surrogate models on the attack performance. The results of this comprehensive analysis are presented in the following tables.\n\n**Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is LLaVA.**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.601         | 0.589            | 0.571                 | 0.601                  | 0.596                     | 0.566                          | 3.0%                | 0.6%                   | 2.6%                        |\n| ITFA (Ours)           | 0.617         | 0.571            | 0.522                 | 0.591                  | 0.568                     | 0.546                          | 1.6%                | 2.0%                   | 3.0%                        |\n| DUFA (Ours)           | 0.621         | 0.576            | 0.526                 | 0.593                  | 0.563                     | 0.545                          | 1.6%                | 1.6%                   | 2.3%                        |\n\n**Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is MiniGPT4.**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.589         | 0.592            | 0.560                 | 0.575                  | 0.600                     | 0.552                          | 2.2%                | 1.6%                   | 1.0%                        |\n| ITFA (Ours)           | 0.576         | 0.583            | 0.515                 | 0.570                  | 0.583                     | 0.527                          | 1.0%                | 2.2%                   | 0.6%                        |\n| DUFA (Ours)           | 0.592         | 0.600            | 0.551                 | 0.600                  | 0.590                     | 0.563                          | 1.2%                | 1.8%                   | 1.2%                        |\n\n**Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is LLaMA-Adapter.**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.589         | 0.586            | 0.596                 | 0.581                  | 0.588                     | 0.581                          | 2.0%                | 0.6%                   | 2.6%                        |\n| ITFA (Ours)           | 0.596         | 0.554            | 0.547                 | 0.583                  | 0.556                     | 0.550                          | 1.0%                | 1.0%                   | 2.6%                        |\n| DUFA (Ours)           | 0.600         | 0.575            | 0.605                 | 0.591                  | 0.580                     | 0.593                          | 1.6%                | 0.6%                   | 3.0%                        |\n\n## Response to Q2\nQ2: How is the attack performance with a more number of testing data point?\n\nAnswer: Thank you for raising this important point. To evaluate the impact of a larger test set, we expanded the LOMIA/LAION dataset to 1,000 samples (500 members and 500 non-members). The attack performance with this increased number of testing data points is reported in the following table （Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）:\n\n**Table: Complete results of our attacks on larger dataset.**\n\n| Metrics / Target Model | AUC ↑ (LLaVA) | AUC ↑ (MiniGPT4) | AUC ↑ (LLaMA Adapter) | Balanced Acc ↑ (LLaVA) | Balanced Acc ↑ (MiniGPT4) | Balanced Acc ↑ (LLaMA Adapter) | TPR@1%FPR ↑ (LLaVA) | TPR@1%FPR ↑ (MiniGPT4) | TPR@1%FPR ↑ (LLaMA Adapter) |\n|-----------------------|:-------------:|:----------------:|:---------------------:|:----------------------:|:-------------------------:|:------------------------------:|:-------------------:|:----------------------:|:---------------------------:|\n| TTFA (Ours)           | 0.597         | 0.570            | 0.566                 | 0.581                  | 0.562                     | 0.560                          | 2.2%                | 2.2%                   | 0.6%                        |\n| ITFA (Ours)           | 0.620         | 0.572            | 0.571                 | 0.594                  | 0.568                     | 0.567                          | 1.4%                | 0.8%                   | 2.6%                        |\n| DUFA (Ours)           | 0.628         | 0.576            | 0.578                 | 0.603                  | 0.570                     | 0.564                          | 2.0%                | 1.0%                   | 2.4%                        |\n\n**Table: Performance of larger dataset on GPT-4o**\n\n| Model | AUC ↑ | Balanced Acc ↑ | TPR@1%FPR ↑ |\n|-------|-------|----------------|-------------|\n| TTFA  | 0.598 | 0.577          | 1.8%        |\n| ITFA  | 0.648 | 0.616          | 3.2%        |\n| DUFA  | 0.675 | 0.637          | 2.2%        |\n\n## Response to Q3\nQ3: In this paper, the author claims that this is the first study on MIAs on VLLMs. However, [17] seems to be a label-only MIA on VLLMs. Please clarify how your setting differs from [17], and if the claim of being the first still holds.\n\nAnswer: Thank you for your question. To the best of our knowledge, our work is the first to investigate label-only MIAs on pre-trained VLLMs, whereas [17] focuses on MIAs against fine-tuned VLLMs. Attacking pre-trained models is generally more challenging. This can be attributed to the fact that VLLMs are pre-trained on large-scale corpora, with each sample typically seen only once or a few times (often fewer than three). As a result, VLLMs exhibit superior generalization compared to fine-tuned VLLMs, which in turn reduces the gap in distance to the decision boundary between members and non-members. We will clarify this distinction in the revised manuscript.\n\n## Response to W2\nW2: The absolute TPR@1%FPR is low, which might not be actionable in practice with small sample batches.\n\nAnswer: Thank you for raising this important point. We agree that the absolute TPR@1%FPR are relatively low, which may limit the direct applicability of the attack in scenarios with small sample batches. However, it is worth noting that this phenomenon is not unique to our proposed method. In fact, as shown in our results and in prior work, many logit-based membership inference attacks on VLLMs also achieve similarly low or even lower TPR@1%FPR values under the same evaluation protocol. We will discuss this point in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper presents a novel label-only membership inference attack (MIA) framework targeting pre-trained large vision-language models (VLLMs). Initially, the reviewers had concerns on the motivation, evaluation, experimental setting, etc. During the rebuttal, the authors successfully persuaded all reviewers to provide postive ratings at the end, with only minor concerns. The AC has to mention that the checklist of the paper must be carefully prepared in order to avoid desk rejection. The AC recommends acceptance of the paper, and asks the authors to finish the checklist part seriously.
Decision: Accept (poster)