Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Authors: Qi Li
Runpeng Yu
Xinchao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Membership Inference
Video-Language Models
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Multimodal large language models (MLLMs) demonstrates remarkable capabilities in handling complex multimodal tasks and are increasingly adopted in video understanding applications. However, their rapid advancement raises serious data privacy concerns, particularly given the potential inclusion of sensitive video content, such as personal recordings and surveillance footage, in their training datasets. Determining improperly used videos during training remains a critical and unresolved challenge. Despite considerable progress on membership inference attacks (MIAs) for text and image data in MLLMs, existing methods fail to generalize effectively to the video domain. These methods suffer from poor scalability as more frames are sampled and generally achieve negligible true positive rates at low false positive rates (TPR@Low FPR), mainly due to their failure to capture the inherent temporal variations of video frames and to account for model behavior differences as the number of frames varies. To address these challenges, we introduce Vid-SME (**Vid**eo **S**harma–**M**ittal **E**ntropy), the first membership inference method tailored for video data used in video understanding LLMs (VULLMs). Vid-SME leverages the confidence of model output and integrates adaptive parameterization to compute Sharma–Mittal entropy (SME) for video inputs. By leveraging the SME difference between natural and temporally-reversed video frames, Vid-SME derives robust membership scores to determine whether a given video is part of the model's training set. Experiments on various self-trained and open-sourced VULLMs demonstrate the strong effectiveness of Vid-SME.

Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Qi Runpeng Yu National University of Singapore
Xinchao Wang"
(liqi, yul nus edu xinchao@nus, edu.
Abstract
Multimodal large language models (MLLMs) demonstrate remarkable capabilities in handling complex multimodal tasks and increasingly adopted in video under- standing applications. However, their rapid advancement raises serious data privacy concerns, particularly given the potential inclusion of sensitive video content, such as personal recordings and surveillance footage, in their training datasets. Deter- mining improperly used videos during training remains critical and und challenge. Despite considerable progress on membership inference attacks (MIAs) for text and image data in MI.LMs, existing methods fail generalize effectively to the video domain. These methods suffer from poor scalability as more frames are sampled and generally achieve negligible true positive rates at low false positive rates (TPR @ Low FPR), mainly due to their failure to capture the inherent temporal variations of video frames and account for model behavior differences the number of frames varies. To address these challenges, we introduce Vid-SME (Video Sharma Mittal Entropy), the first membership inference method tailored for video data used in video understanding LLMs (VULLMs). Vid-SME leverages the confidence model output and integrates adaptive parameterization compute Sharma- Mittal entropy (SME) for video inputs. By leveraging the SME difference between natural and temporally-reverse video frames, Vid-SME derives robust membership scores to determine whether given video is part of the model's ss training set. Experiments on various self-trained and open-sourced VULLMS demonstrate the strong effectiveness of Vid-SME. Code is available here
Introduction
have received widespread attention from the AI community. By combining large language models (LLMs) with vision encoders, MLLMS gain the ability to perform a wide range of vision-language tasks [17 64 19
Multimodal large language models (MILMs)) 1 26 55
21. Recently, there has been 30 45), driven by their strong
growing interest in extending MLLMs to video understanding [63 capabilities in processing visual information. However, the rapid development of video understanding LLMs (VULLMs) also raises critical concerns regarding data privacy leakage, as videos used for model training may contain sensitive content, such as personal recordings and surveillance footage, which could be memorized and unintentionally exposed by the models [5] 48 61.
28
This highlights the severity of the problem, since early studies demonstrate that models' memorization of data can be maliciously exploited to conduct membership inference attacks (MIAs) aim to determine whether specific data sample was used during training. However, despite the booming development of VULLMs, efforts to address this issue significantly lags behind.
44 where adversaries
Recent studies have explored MIAs on LLMs and MLLMs 59
However, we observe that directly applying these methods VULLMS results in extremely poor performance, and the
24
50]
Conresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

performance often deteriorates as more frames are introduced. The underlying reason is that these methods adopt static view of MIA, which is inconsistent with the temporal nature and complex inter-frame variations of video data. Moreover, they overlook the intricate relationship between MIAs and model performance variations across different frame conditions. Since MIAs fundamentally rely on identifying model memorization [ and such memorization in VULLMs may vary with the frame conditions [34] 56], the model tends to provide substantially different inference signals to the adversary when processing different number of frames from the same video. Therefore, successful MIAs on VULLMs generally require a video-specific and adaptive solution that takes into account both video statistics and performance fluctuations across different frame conditions.
In this work, we introduce Vid-SME (Video Sharma- Mittaall Entropy), the first membership inference attack specifically devised to identify videos used in the training of VULLMs. Vid-SME leverages the flexible entropy formulation of Sharma-Mittl Entropy 42 14 51 to adaptively capture the specific inter-frame variations of video frame sequences and compute customized entropy values. To account for different frame conditions, Vid-SME further exploits the model's behavioral differences between natural and reversed frame sequences to compute the final membership score. This design is motivated by our observation that, a video was seen during training, the model tends to predict the next token with higher confidence when frames are presented in their natural order, leading to lower entropy value. In contrast, when processing reversed frame sequences, the model exhibits more pronounced confidence degradation value. This ultimately yields a larger entropy gap between natural and reversed sequences for those seen videos, which serves strong membership signal.
seen videos, resulting in a more noticeable increase in entropy
We evaluate the performance of Vid-SME on various frame conditions, target datasets and target models. The results consistently demonstrate its strong effectiveness in inferring video membership in VULLMs. We summarize our contributions as follows:
We introduce Vid-SME, the first dedicated method for video membership inference, which adaptively adjusts the controllable parameters in Sharma-Mittl entropy and leverages reversed frame sequences to capture the inherent temporal nature and complex inter-frame variations in videos, thus achieving reliable membership inference.
Open-sourced VULLMS are commonly trained on multi-source datasets with only a portion of the training data publicly available, making it difficult to isolate the effects of task types and data distributions on MIA performance. To enable more controlled evaluation, we establish a benchmark by training three VULLMS, each on a distinct dataset, using two representative training strategies WiLee-XLL [45 and LongVA 63)).
Extensive experiments across five VULLMs (three self-trained and two open-sourced) clearly demonstrate the superiority of Vid-SME. For example, when applied to the open- sourced LLaVA-NeXT Video-34B [27 65). Vid-SME delivers 28.3% improvement in AUC, an 18.1% increase in accuracy, and an impressive 293% boost in TPR @ 5% FPR.
2 Related Work
2.1 MultiModal Large Language Models
13 36 52 multimodal large language Building on the success of large language models (LLMs) 12
integrate visual encoders to extract visual features, which are then aligned to the same dimensional space as LLM tokens through dedicated connectors, enabling effective lanual-language processing. Recent advancements in MI.LMs have led to significant improvements in image-related tasks. Vidco Understanding Large Language Models (VULLMs) [63] further expand the capabilities of MLLMs to video understanding by encoding multi- frame features and concatenating them for uniform interpretation. The typical working pipeline of VULLMS for video data closely follows that of image-based MLLMs [30] 28). For example, visual encoder is usually employed to extract spatiotemporal features from videos. These features are then projected into the input space of the large language model through learnable linear projection layer, enabling seamless integration with language tokens.
models (MLLMs) [1] I 26 55
28 30)
VULLMs commonly adapt pretrained based MILMMs for video tasks 33 which are usually trained on image and text modalities and then instruction-tunee on carefully designed video instruction data, during which only the linear projection layer is updated, while the
45
63
32
37

---

rest of the architecture remains frozen 32 Recent efforts, including LongVA [63, Video-XL 45
focus on enhancing temporal modeling to support long video comprehension, and have demonstrated strong performance on related tasks.
and the LLaVA-NeXT- Video series (27 65
2.2 Membership Inference Attack
aim to determine whether a specific data sample was included in model's training set. For machine learning model, ensuring the confidentiality of training data is critical, as it may contain sensitive or personal information about individuals. Existing MIA methods can be broadly categorized into two types [4] [24]: metric-based and shadow model-based. Metric-based MIAs [59] 44 24] rely on evaluating certain metrics derived from the target model's outputs and making membership decisions based predefined thresholds. In contrast, shadow model-based MIAS
Membership Inference Attacks (MIAs) 44 23 54
44
60) train additional models to replicate the behavior of the target
model, which requires extensive computational resources and is often impractical for LLMs [24] Thus, this work focuses exclusively on metric- methods.
MIAs were initially applied in the context of classification models 44]. but have since been extended to other types of models, such as generative models [T0 18] and embedding models 31 With the rapid advancement of L.MSs and MI..Ms, researchers have begun to explore the feasibility of conducting MIAs against these models as well. For example, (43] proposed Min- %%, which selects the smallest K% of probabilities corresponding to the ground-truth token, while (24) argued that detecting individual images or texts is more practical in real-world scenarios and presents additional challenges. To address this, they introduced MaxRényi-K%% and its variant version ModRényi, investigating the potential for extracting and attacking unimodal information from MLLMs. However, to the best of our knowledge, no existing work has explored the privacy risks of MIAs video understanding large language models (VULLMs).
In addition, existing MIA studies on (M)LLMs can generally be categorized into those targeting pretraining data 44
62
g
and those targeting instruction tuning data [24] 57 20). Unlike these
models, dicussed in Section 2.1 VULLMs are commonly built by adapting image-based MLLMs These models are initially trained on image-text data and instruction-tunel using video instruction datasets. As result, MIAs against videos in VULLMs primarily constrained to the instruction tuning stage. This highlights the unique importance of this problem: The capability of VULLMs to effectively interact with humans fundamentally depends on the instruction tuning stage, as the strength of this capability is directly tied to the quality of the instruction tuning dataset. Furthermore, developers often construct their own task task-specific datasets for this stage [45 63 222, which introduces additional privacy risks. Motivated by these factors, our work focuses video membership inference during the instruction tuning stage of VULLMs.
to vidco tasks [33 45
63
32
3 Problem Setting and Challenges
Notation. The token set is denoted by V. A sequence of L tokens is denoted as X (I1, IZ), where æx E V for E [L]. Let XillX denote the aggregation of sequences X1 and X2. A video token sequence is denoted In this work, we focus on a VULLM for parameterized by 0, where the input of the model consists of Fir and instruction context Xmss and the output is the response text X We use D to represent the video set containing the videos used in model training.
Fi.T, where T represents the number of frames.
Adversary's Goal. We follow the standard definition of MIAs as described in 44 Given a VULLM Ja, the adversary aims to determine whether a specific video was used during the instruction tuning stage of fe. We formulate this attack binary classification problem. Let (FFir: 9) : {0, 1} denote the membership detector. During the attack, we feed the model with Fi, and the instruction context Xins The membership detector makes its decision by comparing metric 1 [(Fi.T @ Xinsi 0) with a certain threshold X:
A(Fi.: 0) (1 (Frr E Doa), if I(Fi:T]) 0) < X, (Fur & Dva), if [(Fi.T] (insi 0) 2 A.
(1)
Adversary's Knowledge. Following the standard MIA setup 24 57], we assume grey-box scenario where the adversary can query the target model using the video frames and the instruction context,
3

---

systo:: and q queiion.
frame
frame 2
frame n-1
frame
Vid-SME
Carefully
Tramies
inzediion
repponse
accurately
B SENVRccurss
natural
logit
cxidles
natural frame sequunice
Pi7z P
Quiiiio Selvec the upaiom videa.
Kat
sempling
Options:
(A) black tank smil-
VidisME
then looking serious
bikini
sunscreetooe
"ollows
lest
sampling
videos
target dataset
VULLM
Carget
tox
black bank
frame
frame
frame n--1
man applying surneneen (C) (D)
Only gioe the bast opiion. asnitan:
ligits
Best Option:
Fromes
Instr ction respinte
reversed frame sequnce
Figure I: Vid-SME against VULLMs. Left:: An example of the video instruction context used in our experiments. Middle: The overall pipeline of Vid-SME. Right: The detailed illustration of the membership score calculaiton of Vid-SME.
and is allowed to access the tokenizer, output logits, and generated text. However, the adversary has no knowledge of the training algorithm or the model parameters of the target model.
Challenges. (i). Unlike conventional LLMs and image-based MILMs, VULLMs incorporate video modality during instruction tuning, enabling multimodal understanding beyond static images and text. The temporal nature and complex inter- frame- variations inherent in video data makes membership inference significantly more challenging, (ii). Membership inference fundamentally relies on the model's memorization of training data 44. However, memorization in LLMs is generally weak. This becomes more subtle for video data, where the number of frames fed into VULLMs can influence the model performance and thus influence the degree of memorization 34 56]. The variation the frame conditions makes the relationship between the attack performance and the model's memorization highly intricate, thereby posing additional challenges for effective attacks. (iii). On the dataset side, video instruction tuning data for VULLMS typically comes from diverse and heterogeneous sources [63] 45 65), leading to highly complex data distributions. Moreover, it is often difficult to find non-member data that shares similar distribution with the training set. Previous MIAs text and image data attempt to synthesize non-member samples using LLMs or image generation models [24], whereas such synthesis remains challenging for video data. The distribution shift between members and non-members poses additional challenges for the evaluation of membership inference.
4 Method
Similar to image-based MLLMs, VULLMS also usually project the vision encoder's embedding of the video frame sequence into the feature space of L.M.. Under the grey-box setting, intermediate information from the LLM is inaccessible, and gradient-based operations (c.g., backpropagation) cannot be performed. To this end, we propose a token-level video MIA that computes metrics based on the output logits at each token position.
Figure T Iilustratte the full pipeline of our proposed attack, which can be divided into three main stages: data preprocessing, model inference, and membership inference. In the data preprocessing stage, we perform frame sampling on all videos in the target dataset containing members and non- members... Without loss of generality, we adopt uniform sampling based frame indices. Additionally, to capture the specific inter-frame variations of each video frame sequence, we customize the order parameter and deformation parameter of Sharma-Mitt Entropy [4) by incorporating the video's 's motion complexity and illumination variation into their determination. In the model inference stage, the sampled video frames and the instruction context are fed into the target VULLM. This step is conducted twice, using both the natural frame order and the reversed frame order, respectively. Finally, in the membership inference stage, we extract the slices of natural and reversed logits corresponding to the video frames, which can be easily located based on the model's 's special tokens customized q and r values, compute the Sharma-Mitte entropy for both slices, and derive the final membership score through the differences between the two entropy values.
[24.. Using the
Sharma-Mitte Mittal entropy. Sharma-Mittal entropy is one of the entropy metrics that is widely used in information theory and statistical learning due to flexibility 42 14 51 It allows for tunable sensitivity to different regions of probability distribution, which is particularly beneficial in scenarios

---

involving complex distributions such as those observed in video-language modeling and membership inference attacks. It generalizes several well-known entropy formulations and can be defined as (v. = (EE,))
.9. -.rre E (0, 20) \ f1), where p = {p3} represents a probability
distribution, and q and r are two adjustable parameters that control the entropy's sensitivity and aggregation behavior, respectively. The parameter y determines how the entropy responds to the skewness of the distribution. Specifically, smaller values of increase sensitivity to low-probability (rare) events, while larger values of q emphasizes more on high- probability (dominant) modes. In contrast, r governs the nonlinearity and aggregation scheme in entropy calculation. Larger values of r make the entropy calculation more nonlinear, thereby increasing its sensitivity to distribution peaks.
4.1 Adaptive Parameterization
Videos naturally exhibit highly diverse visual properties. Such diversity impacts the model's prediction distributions. For instance, fast-moving videos with large inter-frame variations often induce higher uncertainty, leading to more dispersed predictions, while stable videos result in more confident and concentrated outputs [35] 16 67 3] Moreover, videos with obvious illumination variations may introduce abnormal prediction fluctuations, as sudden brightness changes can create misleading visual
cues that confuse the model and cause it to overly favor certain tokens [9] [3 46 [9]]. Thus, algin
with the properties of Searma-Mitee entropy, we adapt the parameters q and r for each video frame
sequence based on its motion complexity and illumination variation.
Phemmim
ham
3.00
1.5
1.75
150
35
125
1.5
(a) Variation of g/r values with respect to # frames.
(b) Natural-reversed entropy difference is essential.
Figure2 2: Example of the q/r value distribution and entropy distribution on Video-XL-Cinneile--B.
To do so, for the i-th video, after the frame sequence sampling, we quantify its motion complexity &; as the mean variance of optical flow [15] between consecutive frames, while its illumination variation A is measured as the standard deviation of average brightness across frames. Specifically, each frame is first converted to grayscale, and the mean brightness of each frame is computed; the standard deviation of these mean values then reflects the overall illumination variation within the sequence. Both statistics are further normalized in the target dataset to obtain normalized statistics @ and X, respectively. The entropy parameters qi and r; of the i-th video are then determined as:
4: = 1 31 max, J; de
Å miny \ς max; R7 min, X,y'
max, 6j -min 6j T1 = 82
where 31 and 32 are scaling coefficients controlling the adjustment range of each parameter, which are set to 1.0 and 0. respectively, to better align with the nature of Sharma-Mittal entropy calculation. In this design, video frame sequences with higher motion complexity (i.e., larger Φ) are assigned smaller values, as higher motion typically leads to more uncertain predictions, resulting in flatter probability distributions with more low- now-probability tokens. Smaller values increase the sensitivity of the entropy calculation to these low low-robbayllity tokens, thus better reflecting the model's 's uncertainty in such cases. Meanwhile, videos with larger illumination variations (i.e., larger X) are assigned larger r values, which enhances the nonlinearity of the entropy calculation and increases its sensitivity to abnormal predictions.
4.2 Vid-SME
We now propose our Vid-SME, utilizing the Sharma-Mite entropy of the next-token probability distribution. Specifically, given a token sequence := (I1 ( .T.) consisting of video frame tokens and instruction context tokens (i.e., X | let pa (.) P(- .x;; 0) be the next-token probability distribution at the k-th position. We then extract the video-related probability slices, denoted by PI.T- = {px(-) Ig F. Accordingly, the video-related probability slices
5

---

corresponding to the reversed video sequence can be extracted, denoted as Pr:1 = {+x()) 24 E
Frah. where Fr.1 denotes the reversed video token sequence, (x(-)) is the next-token probability distribution for the reversed video sequence at the k-th position. The Sharma-Mittal entropy is then computed for the natural and reversed probability slices, resulting in
nat = {Sar (px()) | (+(+) E PI.)), Siev = { (fx(+) 1xx() E pr.1h.
(3)
where (..r(-) denotes the Sharma- Mittal entropy with adaptively determined parameters and
After that, we calculate the element wise differences between the two sequences as AS(E)
Snal -S(ee, for & , |Smml Let Min-K% be the smallest K% from the sequence AS. The final Vid-SME score for current video frame sequence is computed as
Vid-SME-KK(Fu.a = |Min- (ASS)
Σ
AS
(4)
ŞEMin-A%Ia
When K O, the #id-SME-K% score is defined to be ming (.S(E)) When K 100, the Wid-SME-KI
score is the mean (,S(E) value of the sequence FI.T. As g + the formulation of Sharma-Mittal entropy reduces to the classical Shannon entropy 41; when - 1, Sharma-Mittal entropy reduces to Rényi entropy 39: when r = g. it corresponds to Tsallis entropy (53). Formal definitions of these simpler form can be found in Appendix In practice, we set sufficiently small threshold of 1 10-16 When the different between q and 1, r and 1, or between g and r falls below this threshold, the entropy calculation in Vid-SME degenerates into the corresponding simpler form.
The variation of g/r values with respect to the number of sampled frames. As shown in Figure 2a increasing the number of frames makes the q/r values more video-specific, suggesting that the richer video information brought by additional frames is effectively reflected in the q/r values.
The significance of the natural-reversed entropy difference. As shown in Figure 2b while the
natural and reversed entropy distributions offer some separation between members and non-members, they are insufficient for clear discrimination. In contrast, the natural-reversed entropy difference significantly amplifies the distribution gap, making the distinction much more pronounced.
5 Experiments
Datasets and Models. To comprehensively evaluate the attack performance, we construct member and non-member sets for five target models, including three self-trained models, covering various task types, video lengths, and dataset scales. The details of these datasets are summarized in Table T The configurations and training trajectories of the three self-trained models are given in Appendix
Specifically, we follow the training pipeline and model components of Video-XL-7B 455 and
instruction tune the model on two distinct datasets to obtain Video-XL-NEXT-QAAAA and Video-XL- CinePile-7B, respectively. NExT-QA (25) is a video question answering dataset while CinePile (38 is a video order reasoning dataset. For the NExT-QA dataset, we randomly sample 1070, 2140, and 4280 instances from both the training and testing splits to construct the member and non-member sets, respectively. This results in target datasets with three dif- ferent scales (i.e., 2140, 4280 and 8560). Unless otherwise specified, we default to using 2140 instances for both members and non-members (4280 in total) in all the exper- iments. Results under different dataset scales reported in Table For Video-XL-CinePile-71 the non-member set consists of all 502 instances from nine scenarios in the MLVU benchmark (66), which involve sequential rea- soning tasks similar to CinePile. To ensure consistency in scale, we randomly sample 502 videos from CinePile as the member set. In addition these, we follow the training pipeline and model components of Long VA
Surge Stoae
Rgh-
20%
50.6
(700
3027
2010
1001
1000
2507
Worr
20.9
2000
Table 1: Statistics of datasets for each target VULLM.
63 and instruction tune the model with video captioning data from Video-XI. training WWW-V---aiio----- 7BB We use all the 1027 samples from the detailed captioning
set to obtain
category in the VDC benchmark [7) as the non non-member set and randomly sample 1027 instances from the model's training set as members.
Beyond our self-trained models, we also include two open-sourced models for evaluation: LLaVA- NeXe-Video-71 (65 and LLaVA-NeXT-Video- [65] For these models, we use the video

---

Talo-XI-NExx-Q---7
Video-X1 Vide--I-Cinfil--- CinePle-7B
LongVA-Captie-78
VLNVe-NNT--Vibe-- (5
Matric
[6]
[5
[
Acc.
FPR
AEC
FYR
Ave.
FPR
ADC
Ave.
FFR
Avc.
FPR
Porplciiyy
0461
00.00
0.779
0.004
0-497
0.5
0.001
0.505
Max Prod Cup
0.857
0.521
0020
0.543
0.559
0.030
0.507
0.521
0.045
0.400
01534
0.000
15256
0.302
0.014
Modifol_Eanop
0.460
0.505
0.048
0.677
0.664
0.004
0,500
0.529
0.028
0.384
0.501
0.002
0.376
0.504
0.505
Mia_0% Prob
0.482
0.515
0.500
0.006
0.501
0.271
0.000
0,500
0.016
Min Prub
0.472
0.907
0.564
0.004
0.496
0.53
0.317
0.002
0.508
0.016
M0r Puub
0.469
0.508
0.022
0.619
0.622
0.004
0.510
0.529
0.633
0.344
0.501
0.003
0.440
0.508
0.014
Min 50% Proh
0.460
0.994
0018
0.54
0641
0.004
0.904
0.537
0.027
0.364
0.501
0.007
0.807
0.505
0.004
Min 5015
0.461
0.502
0.680
0.004
0.528
0.028
0.383
0.002
0.505
MadRéinsi
o 0.5
0.460
0.504
0.046
0.094
0.685
0.006
1.297
0.526
0.025
0.388
0.501
0.002
0.358
0,503
0.007
2
0.460
[0.50]
0.040
0.703
0.685
0.006
0.494
0.524
0.030
0.411
0.501
0.002
0.341
0.504
0:007
Ma
0:457
0.902
0.589
0.010
0.528
0.362
0.007
0.500
66 0.5)
Kónyi
May
5
0.465
0.463
0.508
0.511
0.000
0.034
0.577
0.555
0.602
0.000
0.000
0.500
0.538
0.546
0.547
0.046
0.037
0.57
0.211
0.501
0.501
0.006
0.009
0.555
0.511
0.357
0.062
Max 609
0:467
01512
01940
0183
0.00
0.542
0.006
02215
01500
0.001
0.590
0.586
0.507
0.006
0.508
Max
0.450
0.501
0.024
0.538
0.579
0.004
0.506
0.522
0.028
0.360
0.500
0.007
0.353
0.500
0,012
Rinyi
Mar_516
0.468
0.5110
01134
0.51
0.573
0.000)
11508
0.535
0.032
0:437
01501
0.014
0.458
0,514
0.012
Ma
0.509
0.563
0.994
0.000
0.538
0.312
0.00:
0,546
Max.
0.461
0.506
0.488
0.503
0.608
0.004
0.533
0.545
0.006
0.250
0.500
0.001
0.543
0.004
Max_90%
0.460
0.506
0.044
0.623
0.620
0.000
0.514
0.535
0.087
0.319
0.500
0.001
0.479
0.533
0.021
Max_0%
0:420
0.905
0.012
0.479
0.553
0.000
0.484
0.302
0.635
0.315
0.300
0.004
0.433
0,500
0.010
Riayi 7)
0.76
0.000
0.538
0.316
Max, 30%
0.468
0.510
0.02
0.591
0.597
0.000
0.535
0.532
0.006
0.341
0.301
0.001
0.425
0,520
0,016
Mat 60F
0.460
0.506
0052
0.630
01631
00000
11508
0.522
0.029
0.348
01500
0.001
0.418
0,512
0,012
Mac_90R
0.505
0.004
0.002
0,508
Max_0%
0.482
0.515
0.500
0.544
0.006
0.501
0.271
0.501
0.000
0.500
0.016
Reayi oc)
M5
0.472
0.507
0.52
0.564
0.582
0.004
0.406
0.51a
0.004
0.317
0.500
0.002
0.462
0.508
0.016
Max 300
0.669
01.08
0.699
0.004
0.520
0.544
0.004
0,508
0.014
0.654
0.537
0.506
Mar. 90%
0.461
00.002
0.048
0.680
0.606
0.004
0.097
0.528
0.024
0.333
01501
0.002
0.352
0,305
0,007
Meat
0.535
0,540
0.104
0.540
0.769
0.420
11296
0.530
0.039
0.414
01500
0.050
0,513
0,525
0.063
Maa_0%
0.528
0.559
0.545
0.450
0.060
0.692
Vid-SME (Ouns)
Min 5%
0.543
0.855
0.44
0.34
0.62
0.088
0.556
0.561
0.041
0.550
0.555
0.064
0.682
0.109
Min_306
0.545
0.547
0.074
0.609
0.670
0.161
0.541
0.553
0.050
[.601
0.579
0.073
0.606
0.654
0.102
Min 6096
0.544
01:445
0.04
0.765
759
0.343
0.846
0:548
0.081
01572
0.007
0.63
0.117
Min_90S
0.541
0830
0.765
0.305
0.550
0.523
01531
0.058
0.577
Table 2: Results of Vid-SME and baseline methods when # frames=16. We highlight the best,
second-best, and third-best results in progressively lighter shades of red, while marking the worst, second-worst, and third-worst results in progressively lighter shades of green.
caption dataset Video-Instruct-1000 100K serves as part of their training data as the mem- ber set. Each video in this dataset has multiple questions, from which we select the one with the longest text length, resulting in 869 samples. The non-member set consists of 869 samples randomly selected from the captioning data from Vi training set [45)
32 that
Micher proip wai-- 111
H-dled pomal The de
Noc Mochheitee Auc-
Moc we'l
dii 1M
16:0
Johe
ROC
TPR@59 55% FPR
Baselines. We adopt several metric-based MIAS as baselines and compare them with Vid-SME. Figure 3: The ROC and TPR@5% FPR curves.
Specifically, we include the Loss attack [59], which corresponds to perplexity in the context of language models. We also involve the Min-K% method (3), which computes the smallest K% probabilities associated with the ground-truth tokens. We evaluate values of 0, 5, 30, 60, and 90. In addition, we adopt the Max_ Max_Prob__ Gap metric [24], which captures the model's confidence by computing the difference between the maximum and the second-largest probability at each token position, followed by averaging across the sequence. We further include M and its mod- ified variant ModRényi proposed in (24). which are specifically designed for membership inference on image-based based MLLMs and utilize the Rényi entropy of next-token probability distributions. For MaxRényi-K3 K K%, we set a to 0.5, 1, 2, and co, while for ModRényi, we use a values of 0.5 and 2. We also include Modified Entropy [50] our baseline, as it is special case of ModRényi when + 1.
Evaluation metric. As binary classification problem, the performance can be evaluated with the AUC score [6]. We define the members as "positive" and the non-members as "negative". We also report True Positive Rate (TPR) at low False Positive Rate (FPR) 4., which is an important metric in MIAs and measures detection rate at a meaningful threshold. We set the threshold as 5% and evaluate all the methods under TPR @ 5% FPR. We also report the best classification accuracy achievable by sweeping over all possible thresholds on the attack scores. Specifically, this accuracy is computed as
7

---

Train-Test
frames
Gap
2
11.35
We high- +Yrunis besaines
4
11.16
8
10.95
16
8.57
32
10.20
26
01000.00
(a) Performance gap
(b) Attack performance (AUC) of different methods v.s. # frames.
(c) AUC and TPR @5% FPR un- der different corruptions.
v.s. frames.
Figure 4: Analysis on: (a) Train- Test Gap v.s. # frames, (b) Attack performance v.s. # frames, (c) Attack performance under different corruption types and levels.
the maximum value of 1- (I-+(1-TRP)) TPR) across the ROC curve, representing the optimal classification accuracy between members and non members. We use # frames to denote the number of frames.
5.1 Main Results
The experimental results across the five target VULLMS are summarized in Table 2 light the best, second-best, and third-best results in progressively lighter shades of red, while marking the worst, second-worst, and third-worst results in progressively lighter shades of green. The # frames is fixed to 16. It can be observed that Vid-SME con- sistently achieves the best performance under all settings, especially excelling in the most critical metric, TPR @ 5% FPR. When the target and WA--NeeT-Viieo--... all baseline methods exhibit extremely low TPR @ 5% FPR values (around 0.001), which are impractically low for reliable membership inference. In contrast, Vid-SME consistently maintains TPR @ 5% FPR above 0.05 in these challenging scenarios, demonstrating re- markable improvements. In addition, the fact that baseline methods achieve AUC scores both above and below 0.5 across different set- tings indicates their inconsistency in distinguishing members from non-m non-members. This suggests that they cannot serve as a reliable and unified indicator for membership inference in video-based scenarios.
Mitkol
Type
ALC
Avt.
17302%
B,377
LLM4
0:90:
Peplicty
13.11
6313
VULLMS are Mideo-XL-CinnPPle-..
8:37
0.910
Ma. Map
8.30
0330
Makiin
8.885 13.99
0.3
18401
0.88
18445 0,i0e
0:21
0..0
0.90
Modial
In
0350
8.44 9.
0.8
Mar
1840
13.00
8422
4312
0.651
0.06
18139
0.00
0.00
Table Performance compar- ison on different instructions.
To be more intuitive, we illustrate the detailed comparisons for Video-XL-CinePie--7B with K O, 5, 100 in Figure 3 which presents both ROC and TPR @ 5% FPR curves. Among all methods, Vid-SME variants (K = 0, 5, 100) consistently achieve superior performance. Notably, Vid-SME- 100% reaches the highest AUC of 0.90, substantially surpassing other baselines. Additionally, Vid-SME-0% and Vid-SME- 100% achieve significantly higher TPR @ 5% FPR (0.18 and 0.54), while most baseline methods yield almost negligible performance (close 0.0).
Relationship between model memorization, frame conditions and attack performance. To analyze the relationship among model memorization, frame conditions, and attack performance, we further investigate how the train-test performance gap and at- tack performance change under different frame counts (# frames). Results for Mideo-XI--NEET--A---7 are given in Table 4a and Figure
4b) We can observe that, when # frames are limited, the model remains uncertain on both training and test samples, lead- ing to poor generalization but also weak memorization. Thus, although the performance gap is large, attacks are less effec- tive as explicit memorization has not yet emerged. As # frames increase, improved understanding of video content narrows the gap, but exploitable confidence differ- ences between members and non-members arise, enhancing the attack effectiveness. With even more frames, the model becomes highly confident on training samples while struggling with distribution shifts or increased complexity in unseen test samples, which enlarges the gap and further amplifies train-test prediction differences, making attacks highly effective. This phenomenon highlights the non-linear relationship between memorization and MIA vulnerability in the context of VULLMs.
bas pos
au
Be diach val
Figure 5: A comparison between with and without full context.
8

---

5.2 Ablation Study
Infleunce of instruction context. We now refer to the instruction context used in our main experiments as 41_ To explore the influence of instruction context, we design two al- ternative contexts, denoted as Ig and Ig. The details of 1,,3 are provided in Appendix The results when # frames = 8 and the target model is LLaVA-NeXT-Videe-33
AUC
TPR FPR
are reported in Table As shown, the impact of the contexts is not significant. Furthermore, we observe that Vid-SME is less sensi- tive to context variations compared to other baselines, indicating better stability in its attack effectiveness.
0.5
Ad10
0.01
0.5050.011
100
0.033
Normsl toura) No inversion
4010-
0.12
Furthermore, we investigate the scenario where text provides minimal information, and video frames are combined with only short query text before being fed into the model, which makes the attack results in- dependent of video-related understanding tasks. The short query text used here can be found Appendix C The results under this setting, using Video- ML-CinePile-7B as the target model and # frames = 8, are reported in Figure 5 It can be ob- served that when the selection range of AS is small (e.g.. K = 0, 5, 30), attacks using only the short query text outperform those using the full instruction context. However, as the selection range of ÁS becomes more representative of the overall video probabilities (i.e., K t), the at- tack performance with the full instruction context gradually surpasses that of the short query text. This observation aligns with intuition: when the prediction probabilities for video-related tokens are shaped by rich task-specific textual context, the model's 's response generation becomes more strongly grounded in the video frames, increasing its reliance on the complete visual information. Overall, however, the performance difference between the two is not substantial.
No or
16
16
Frames
Frames
Figure 6: A comparison between with and without full instructions.
Importance of q/r adaptation and reverse frame sequence. We further present the results when disabling the adaptive q/r values (No qr) and when removing the reversed frame sequence when calculating the membership score (No inver- sion). For the former, we assign fixed 9 and r values (i.e., q 2.0, 1.0) across the entire target dataset. For the lat- ter, instead of using ASS, we directly adopt Snat to compute the final membership score. The target model in this experiment is Video-XL-NExT1-A--7B and we report the Vid-SME-5%% results. As shown in Figure (6, removing either component leads to significant performance drop, demonstrating the critical role of q/r adaptation and reversed frame sequence in performing membership inference attacks against VULLMS.
Datovt
Hoste
TFRSI-6 FPR
4380
(858)
E.11
Meua VU--INE
4289
(3.01
1.50
5130
1316
13W
2000
1546
E.41
1342
1.40
1009
Table 4: Performance comparison on different instructions.
Influence of video frame corruptions. The motivation is to detect whether videos are used in training even under potential video corruption. In Figure 44c we report the attack performance under two different corruptions (Motion Blur and Brightness) at three different levels of corruptions (Marginal, Moderate and Severe). Detailed corruption parameters and examples of the corrupted video frames are given in Appendix B It can be observed that corrupted video frames make MIAs more difficult, but members can still be detected successfully.
Influence of dataset scales. Table 4 presents the attack performance of Vid-SME on Video-XL- NEXT-QA-7B under varying dataset scales. The results show that Vid-SME remains consistently effective as the dataset size increases, demonstrating its scalability.
6 Conclusion
In this work, we investigate the membership inference risk in video understanding large language mod- els (VULLMs). We propose Vid-SME, the first membership inference attack tailored for VULLMs, and self-train three VULLMS for more comprehensive evaluation. Unlike existing methods that fail to capture video-specific temporal dependencies, Vid-SME leverages an adaptive parameterization strategy and both natural and reversed frame sequence to compute the Sharma-Mittal entropy for robust membership signals. Extensive experiments demonstrate the strong effectiveness of Vid-SME.

---

Acknowledgement
This project is supported by the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity R&D Programme and CyberSG R&D Cyber Research Programme Office (Award: CRPO-GCI-NTU-000)
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Christian Beck. Generalised information and entropy measures in physics. Contemporary Physics, 50(4))495-510, 2009.
[3] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.
[4] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song. Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897-1914. IEEE, 2022.
[5] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium ( (USENIX security 19), pages 267-284, 2019.
[6] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633-2650, 2021.
[7] Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024.
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[9] Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. A hierarchical variational neural uncertainty model for stochastic video prediction. In Proceedingy of the IEEF/CVF international conference on computer vision, pages 9751-9761, 2021.
[10] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: taxonomy of member- ship inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343-362, 2020.
[11] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-moda models with better captions. In European Conference on Computer Vision, pages 370-387 Springer, 2024.
[12] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* " chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
[13) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[14] Maria Dolores Esteban and Domingo Morales. A summary on entropy statistics. Kybernetika, 31(4):337-346, 1995.
10

---

[15] Gunnar Farnebäck. Polynomial expansion orientation and motion estimation. Linkopings Universitet (Sweden). 2002.
[16] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[17] Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review. Frontiers in Artificial Intelligence, 7: 430984, 2024.
[18] Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint
2017.
[19] Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 2256-22662 2264, 2024.
[20] Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision-language models. arXiv preprint arXiv:2501.18624, 2025.
[21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-1977 PMLR, 2023.
[22] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023.
[23] Qi Li, Cheng-Long Wang, Yinzhi Cao, and Di Wang. Data lineage inference: Uncovering privacy vulnerabilities of dataset pruning. arXiv preprint arXiv:2411.1579. 2024.
[24] Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems, 37:98645-98674, 2024.
[25] Xiao Lin and Chenliang Xu. Next-qa: Next phase of question answering to explaining tem- poral actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777-97866 2021.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306 2024.
[27] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[28] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-lim: Large language models are effective temporal learners. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[29] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.00476, 2024.
[30] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprins 2rx1v:2306.05424 2023.
[31] Saced Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. Membership inference on word embedding and beyond. arXiv preprint arXiv:2106.11 11384, 2021.
[32] Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv
2023.
11

---

[33] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In European conference on computer vision, pages 1-18. Springer, 2022.
[34] Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang. Slowfocus: Enhancing fine-grained temporal understanding in video I1m. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[35] A Piergiovanni, Chenyou Fan, and Michael Ryoo. Learning latent subevents in activity videos using temporal attention filters. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[36] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018
[37] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6545-6555 2023.
[38] Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024.
[39] Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume I: contributions to the theory of statistics, volume 4, pages 547-562. University of California Press, 1961.
[40] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint vrXiv:1806.01244 2018
[41] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[42] Bhudev D Sharma and Dharam P Mittal. New non-additive measures of entropy for discrete probability distributions. J. Math. Sci, 10(75):28-40, 1975.
[43] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 160,,69,,, 2023.
[44] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[45] Yan Shu. Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint 2rXiv:2409.14488 2024.
[46] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[47] Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377-390, 2020.
[48] Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security, pages 587-601, 2017.
[49] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2615-2633 2021.
12

---

[50] Liwei Song, Reza Shokri, and Prateek Mittal. Membership inference attacks against adversari- ally robust deep learning models. In 2019 IEEE Security and Privacy Workshops (SPW), pages 50-56. IEFE, 2019.
[51] Inder Jeet Taneja. On generalized information measures and their applications. In Advances in Electronics and Electron Physics, volume 76, pages 327-413. Elsevier, 1989.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09228 2023.
[53] Constantino Tsalis. Possible generalization of boltzmann-gibe -gibbs statistics. Journal of statistical physics, 52:479-4444 487, 1988.
[54] Cheng-Long Wang. Qi Li, Zihang Xiang, Yinzhi Cao, and Di Wang. Towards lifecycle unlearn- ing commitment management: Measuring sample-level approximate unlearning completeness. arXiv preprint arXiv:2403. /2830, 2024.
[55] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 17:121475-121444 2024.
[56] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58- 76. Springer, 2024.
[57] Hengyu Wu and Yang Cao. Membership inference attacks on large-scale models: A survey. arXiv preprint 250i:::5003.133333 2025.
[58] Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via video frame interpola- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17814- 17823, 2022.
[59] Samuel Ycom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[60] Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03226 2023.
[61] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64 115, 2021.
[62] Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint vxXiv22404.02953 2024.
[63] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[64] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
[65] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: strong zero-shot video understanding model, April 2024.
[66] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint arXiv:2406. 04264. 2024.
[67] Tianfei Zhou, Fatih Porikli, David J Crandall, Luc Van Gool, and Wenguan Wang. A survey on deep learning technique for video segmentation. IEEE transactions on pattern analysis and machine intelligence, 7596):709 2022.
13

---

A Model Configurations and Training States
We report the model configurations of the three self-trained VULLMS in Table loss and gradient norm over steps in Figure 7
and the training
Field
"ideo-XI-NEXT-OAA-7
Mide--Xi-CimePlle-BB
Long VA-Captionn
Base L.M
1oan2-7B-nntrruct22200
Qwen2-7B- -nntrrtt-kn
(ween--7B-nntrruct2244
jos_tokenjid
151643
151643
151643
jis_toke_jid
151645
151645
151645
Hidden Activation
silu
silu
silu
Resampler type
spatial_pode pool
spatial_pool
opatial_pool
Vision tower
clip-vit Hlip-vi-llagee hatt--ree-pahhh-33355 clip-vi See-vit-rrge-pacch4-3333
hip-vit-large-patchhee
Vision tower ir
2c-6
2c-6
2e-6
Max window layers
28
28
28
Projector type
mlp2x _ep2x 28
mlp2x gelu 28
mlp2x gelu 28
#Heads
#Hidden Layers
28
28
KV heads
4
Tokenizer padding side
right
right
right
Vocab size
152064
152064
152064
<lendoftexti>: 151643
<lendoftextl:: 151643
<lendoftextb: 151643
<lim_ endi>; 151645
<lim dim_endd>: 151645
<lim_endl>:
<lim_ endl>: 151645
Added tokens
starts: 151644
<lim starth>: 151644
<lim start>>: 151644
pad_ token: <lendoftextl>
pad token: <lendoftextl>
pad token: <lendoftextl>
Special tokens map
cos_tot jos_token: <lim_ <lim_endl> endl>
cos_ jos_token: <lim <lim_endll>
eos jos_tonen: <him_sliiiiieeeee
Table 5: Model configurations of the three self-trained models.
25
300
23-
3.3
LH
700
19. 6/1
0.5
11
0.
1020
1000
12000
1250
314p
5140
(a) Video Video-XL- XL-NEXX-QQA-7B
(b) Mideo-XX-CCinPPiin
(c) Long VA-Caption-78
Figure 7: Training Loss and Gradient Norm over Steps for the three self-trained models.
B Examples of the Video-Text Instruction Context
In Figure give example of the video-text instruction context used in experiments. In addition, we also provide the corrupted video frames under different types and levels of corruptions in Figure g The details of the parameters of different corruptions given in Table Specifically, for brightness corruption, we adjust the pixel intensity by randomly adding/subtracting a constant value of 20, 60, and 100 for marginal, moderate, and severe conditions, respectively. For motion blur, we apply convolutional kernel with size and angle parameters set to (10, 5), (15, 5), and (20, 10) to simulate increasing degrees of blur under the same three corruption levels.
Brightness
Motion Blur
Marginal
20
(10,5)
Moderate
60
(15.5)
Severe
100
(20,
Table 6: Brightness and Motion Blur Levels under Different Conditions.
14

---

framm
frame
fram
frame
frame
fame 2
frame
frame
original
Or reginal
manginal
maginal
moderate
enderate
Motion Blur
Brightmess
Carefully watch this video and pay attention to every detail. Based on your observations, select best option that accurately addresses the
question.
user:
<videes>
Question: Why the man moved upwards after he fell down?
Options:
(Ai change posture to support baby (8) blocked by pumpkin (C) snow blocking (D) pushed by water moving cardboard backwards
Only give the best option.
assistant:
Best Option:
Figure Example of the video-text instruction context under different types and levels of corruptions.
system:
Carefully watch this video and pay attention to every detail. Rased on your observations, solect the best option that accurately addresses the question.
user:
cvidooo....
assiitant:
Best Option:
I
system:
will be shown video, Examine carefully and use all visual cues to determine the correct answer. Your response should be the most option.
oser:
assistant:
I2
system:
You to going see a video. Take a good look and try understand what's going on. Based that, help pick the option that makes most
user:
evideus...
assistant:
think the best choice is:
I3
Figure 9: The three different instruction contexts used in the ablation study.
system:
helpful assistaant.
user:
cvideno
Describe this video in detail.
assksantt:
Response:
Figure The short query text used in the ablation study.
C Different Instructions Used in the Ablation Study.
give the contents of the three different instructions used the ablation study in Figure The short query text used in the ablation study is given in Figure I0
15

---

framm
frame
fomm
frame
frame
frame 2
frame
frame
non-
system:
system:
Carefully watch this video and pay attention to every detail, Based obsrrvatioss, sulect the best uption that accurately addresses question.
Carefully watch this video and pay attention to every detaill. Based your ubservations, select the best option that accurately addresses the question.
<video>>
<video>
Question:
Why did the moved upwards after he fell down!
Question: What does the man black do after turning his head the middle?
Options:
(A) change posture to support baby (B: blocked pumpkin (C)
Options:
snow blocking him (D) pushed by water (E) moving the cardboard
(A) eat the jelly walk away (CI cycle (D) shake finger
backwards
stretch his hands
Only give best option.
Only give the best option.
assistant:
assistant:
Best Option:
Option:
Figure 11: An example member and non non-member data for Maleo-XL-NEXT-QA-777
Tramm
frame
frae
frame
frame
frame
member
NOVIOUPS Mirr
non- memebr
system:
system:
Carefully watch this video and pay attention every detail. Based on your observations, select the best option that accurately addrresses the question.
Carefully watch this video and pay attention to every detail. Based on your observations, select the option that accurately addresses the question.
user:
cvideos
<video>
option correctly represents actions in the video.
Question:
Question: Please identtfyy the option that corresponds order of events as they occur in the video.
Options:
Options:
(A) woman singing->man looking surpried-owebsite with search
(A) water sliding -> stomping grapes -> clean and jerk cleaning toilet
bar-ologo scissons
(B) website with search ban->logo with scistors-wwmman singing-
(B) stomping grapes >> clean and jerk >> cleaning toilet water
oman looking surprised
sliding
(C) website with search ban-man looking surprised owoman
(C) water sliding >> cleaning toilet -> stomping grapes clean
singing- ploging--lgo
scissors
and jork
(D) logo with dissors--webbid with search banosman looking
(D) water sliding >> cleaning toilet -o clean and jeri io stomping
supriisedswwomn singing
grapes
Only give the best option.
Only give the option.
assistant:
Best Option:
Best Option:
Figure 12: An example of member and non-membon data for Video-XL-CinnPile--B.
frame
frame
member
nort- memebb
system::
system::
Carefully watch this video pay every detail. on your observations, answer the given questions.
Carefully watch and pay attention every detail, on your observations, answer the given questions. user:
<videos
<viden>>
detailed description of the given video, capturing key
through important details the video, describing scenes and characters.
moments.
assitant:
Figure 13: An example of member and non-member data for Longva-Caption-
D Computation resource usage.
The three self-trained models are trained on 8 A100 GPUs, while all experiments are conducted using 8 NVIDIA RTX A5000 GPUs.
16

---

frame
frame
frame 3
Frme
frame
frame 2
frame
frame
non- memebr
System:
Carefully watch this video pay attention every detail, Based on your observations, answer the given questions.
system:
Carefully watch this video and pay attention every detail. Based on your observations, answer the given questions.
user:
<videa>
<video>
Write exhaustive depiction of the given video, capturing its essence and key moments.
What is the video about?
assistant:
assistant::
Figure 14: An example of member and non-member data for -LaVA-NeXT-Videe
E
Examples of Members and Non-Members of the Target Models.
We give examples of members and non-members of the five target models used in experiments in Figures
F
Simplified Terms of Sharma-Min Entropy
The Sharma-Mita entropy for probability distribution p {pi} is defined as:
Sp)(p)
1 1-r
Σ p
(5)
where q controls the sensitivity distribution skewness, and determines the nonlinearity of aggregation. This generalized formulation subsumes several classical entropy measures as special cases. We give the formal definitions as follows:
F1 Reduction to Shannon Entropy
As both q + and -> Equation 5 reduces to Shannon entropy:
lim 0+1-111
(p - Ps log Pt.
(6)
This limit follows from applying L'Hôpitallss Rule to both the exponent and denominator and ->
1
F22 Reduction to Rényi Entropy
When + and q * 1, Equation simplifies to the Rényi entropy:
1 log Σ p..
lim Sq. (p)
(7)
F33 Reduction to Tsallis Entropy
When 9. Equation 55) reduces to the Tsallis entropy:
(p)(P) I- [
These reductions demonstrate that Sharma-Mittal entropy serves as a unified framework encompass- ing Shannon, Rényi, and Tsallis entropics limiting cases.
17

---

G
Broader Impact
This work explores the privacy vulnerabilities of Video Understanding Large Language Models (VULLMs) through membership inference attacks (MIAs), revealing how sensitive training data can be partially reconstructed or identified from model behavior.
G.1 Positive Societal Impacts
Our study contributes to the broader goal of trustworthy multimodal AI by uncovering potential privacy risks before such models are widely deployed. The insights and benchmark introduce can inform defensive research, such as differential privacy mechanisms, data sanitization, and auditing protocols, helping model developers detect and mitigate leakage risks future VLM releases. Moreover, the framework provides a systematic methodology for assessing privacy robustness in multimodal systems, filling a critical gap between vision and language privacy research.
G.2 Negative Social Impact
At the same time, the attack methods demonstrated in this paper could, if misused, be exploited to extract private or copyrighted visual content from proprictary VLMs. Although our experiments conducted under controlled academic settings, similar approaches might be repurposed for malicious data mining or surveillance.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction accurately reflect contri- butions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:Ti The limitations are given in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Useful information related to reproduce the results are given in the paper and the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer:
[Yes]
Justification:
The code is made public available here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The experimental section presents results under identical settings with different relevant parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detials are provided the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Detailed discussions are given in the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced are made public available here
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
25


Reviews:
[{"review_id": "1uVoQtjegF", "reviewer": "Reviewer_we3T", "summary": "- The paper introduces a new Membership Inference Attack (MIA) method (Vid-SME) tailored for the temporal nature of video data for video understanding large language models (VULLMs)\n- Existing MIA techniques designed for text or image data perform poorly when applied to video, largely due to their inability to account for the temporal dynamics and frame-dependent model behavior unique to video.\n- To address this, the paper proposes Vid-SME, which uses Sharma-Mittal entropy to quantify model confidence under two conditions: when a video is shown in its natural order and when the frame sequence is reversed. \n- The key insight is that VULLMs trained on a video are more confident (lower entropy) when frames are presented in their original order compared to reversed order. \n- By measuring the entropy gap between these two settings and aggregating the smallest differences (via a K%-min pooling strategy), Vid-SME generates a membership score for each video.\n- The paper also introduces a benchmark suite using three self-trained and two open-sourced VULLMs to evaluate the attack across diverse datasets, training strategies, and video types. \n- Experiments demonstrate that Vid-SME significantly outperforms prior methods, particularly in low false positive rate (FPR) regimes, showing substantial gains in metrics like AUC and TPR@5%FPR.", "questions": "- Suggestion: Clarify the threat model. Could Vid-SME be adapted to black-box APIs?\n- How are the Sharma–Mittal parameters q and r determined for each video? Are they chosen via optimization, heuristics, or fixed based on video metadata? Do these values generalize across datasets or need to be tuned for each model/video?\n- Why is frame reversal specifically chosen as the transformation to reveal memorization? Have you considered or tested alternative temporal perturbations (e.g., frame shuffling, masking, partial reversal)?\n- What is the computational overhead of Vid-SME per video inference?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "-  VULLMs are increasingly trained on sensitive, real-world video data (e.g., surveillance, personal videos). The paper addresses an urgent and underexplored privacy risk - whether such models memorize and inadvertently expose their training data.\n- The entropy-based scoring function using natural vs. reversed video sequences is a creative and well-motivated idea that leverages the temporal structure unique to videos—this is not a straightforward extension of prior MIA techniques.\n- The adaptive use of this generalized entropy measure provides a novel and theoretically grounded mechanism to capture nuanced differences in model confidence across frames.\n- The results are strong compared to the baselines and the paper goes beyond surface-level results by providing visualizations (e.g., Figure 2a–2b) that explain why the method works, showing how entropy differences become more distinct as video length increases.\n\nWeaknesses:\n- The attack appears to rely on logits or confidence scores from the model. It is unclear whether Vid-SME works when only hard labels (top-1 predictions) are available, which would be a more realistic setting for many deployed systems.\n- While the paper emphasizes adaptive entropy parameterization, the method for choosing q/r values per video is underexplained. Is this based on optimization, empirical tuning, or predefined heuristics?\n- The method requires computing entropy over possibly long sequences and for both natural and reversed orderings. There is no discussion of runtime or scalability with respect to video length or number of frames.", "was_revised": true, "final_justification": "The paper looks good to me, and I will continue with my original rating of accept.", "rebuttal": "It’s our great honor to receive Reviewer we3T’s thoughtful comments and strong support for our work. We would like to address the concerns as below.\n\n---\n\n> **[Q1, W1]**: Suggestion for clarify the threat model.\n\n**A1:** Thanks for the valuable suggestion. Vid-SME is designed for the gray-box scenario, which is a commonly adopted setting in prior works (e.g., Rényi entropy and Min-K%). This is attributed to its broad applicability in real-world scenarios:\n\n* For example, **closed-source model APIs such as OpenAI's GPT-4 API allow users to retrieve the top-five per-token probabilities**, which also can be seen as a gray-box scenario in our context and the adversaries can exploit this information to conduct attacks. We follow this setting (where only the top-five per-token probabilities are accessible) and conduct an experiment on Video-XL-CinePile-7B with the number of frames set to 8. As shown in the table below, Vid-SME remains reliable in performing membership inference under this partial-knowledge scenario.\n\n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR |\n|---------------|-----------|-------|-------|-------------|\n| Vid-SME    | Mean   | 0.775 | 0.703 | 0.305 |\n|| Min_60%   | 0.657 | 0.628 | 0.229 |\n\n* Also, such attacks are not only relevant for external adversaries, but also provide a **practical auditing tool** for model developers and owners to monitor potential privacy risks in their deployed models. \n\n* Moreover, the gray-box setup represents a **conservative risk lower-bound**. If an attack succeeds under the gray-box setting, it indicates that the system may already be operating at the boundary of its privacy guarantees.\n\n\n\nWe sincerely hope that our responses can alleviate this concern and we will further clarify our threat model and its practical motivations in the revision.\n\n---\n\n> **[Q2, W2]**: How are the Sharma–Mittal parameters q and r determined for each video? Do these values generalize across datasets or need to be tuned for each model/video?\n\n**A2:** Thanks for the valuable comment. When designing the q/r value, we took into account the following key considerations:\n\n(1). Whether the computation remains naturally consistent under different video conditions (e.g., varying frame numbers); \n\n(2). Whether the q/r values are compatible with the Sharma-Mittal Entropy calculation;\n\n(3). Whether the q/r values can adequately capture the temporal nature and complex inter-frame variations of the video. \n\nIn response to (1), once a video's sampled frames are determined, its q/r values become fixed, i.e., as you noted, the q/r values are **based on video metadata**. This means the **q/r values are model-agnostic (do not need to be tuned per model)** and those model-specific informations are captured through the subsequent entropy computation. The q/r values only need to be re-computed when the frame condition changes (e.g., from 8 to 16 frames). In response to (2) and (3), we specifically designed the q/r values to reflect video-specific temporal dynamics and inter-frame variations through motion complexity and illumination variation. We will further clarify these design choices and details in the revision.\n\n---\n\n> **[Q3]**: Why is frame reversal specifically chosen as the transformation to reveal memorization? Have you considered or tested alternative temporal perturbations (e.g., frame shuffling, masking, partial reversal)?\n\n**A3:** Thanks for the insightful and thought-provoking comment. We would like to share our perspective on this point from two perspectives:\n\n* First, we agree that applying transformations such as frame shuffling, masking, or partial reversal is a meaningful direction worth exploring. However, such approaches naturally introduce **additional hyperparameters**, such as which frames to shuffle, which regions to mask, and which segments to reverse. This leads to a much broader design space, and hyperparameter choices may vary across different videos. While some carefully crafted strategies may indeed further improve the attack performance, our choice of full-frame reversal is a **more intuitive and lightweight solution**, requiring no tuning while already yielding strong results.\n\n* Second, as an analogy to injecting noise within each frame,  applying masking can be seen as **a process of frame corrupting**. Some prior work (e.g., Rényi entropy) as well as our own experiments (see Figure 4(c) and Lines 374-379) have taken this into consideration by **evaluating the robustness of the method** under frame corruption settings.\n\nOverall, we truly appreciate the reviewer’s suggestion on exploring alternative ways to incorporate video-specific properties to reveal memorization in VULLMs. We see this as a valuable and exciting direction and plan to pursue it further in future work.\n\n---\n\n> **[Q4, W3]**: What is the computational overhead of Vid-SME per video inference?\n\n**A4:** Thanks for raising this point. In the table below, we provide a comparison of inference time per video and attack performance between the best-performing baseline Rényi entropy and Vid-SME. The experiments are conducted using the Video-XL-CinePile-7B model with frame numbers set to 8 and 16, with both methods using the variant of Max_90%/Min_90% and  α in Rényi entropy is set to 2 (note that the variant used and the α value in Rényi entropy do not affect the inference time).\n\n_\\# Frames = 8:_\n\n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR | Time per Video (s) | \n|---------------|-----------|-------|-------|-------------|---------------------|\n| Rényi | Max_90%   | 0.559 | 0.585 | 0.004       |        6.11             |\n| Vid-SME    | Min_90%   | 0.806 | 0.760 | 0.289 |       9.45              |\n\n_\\# Frames = 16:_\n  \n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR | Time per Video (s) | \n|---------------|-----------|-------|-------|-------------|---------------------|\n| Rényi  | Max_90%   | 0.661 | 0.655 | 0.004       |        11.67             |\n| Vid-SME    | Min_90%   | 0.830 | 0.765 | 0.305 |       16.76              |\n \n* Compared to Rényi entropy, the additional computational cost of Vid-SME lies in **the computation of the adaptive q/r values**. When these video-specific values are not computed, the inference-time cost of Vid-SME is nearly identical to that of Rényi entropy, as **both require two forward queries to the model**. \n\n* Also, as the number of frames increases, Vid-SME demonstrates **good scalability in both inference time and performance improvement**. \n\nWe sincerely hope that our responses adequately address the point and we will include the additional comparisons on time consumption in the revision.\n\n---\n\nThanks again for the thoughtful and constructive feedback. We would definitely love to further interact with the reviewer if there are any further questions."}, {"review_id": "iYZ7xbNgPo", "reviewer": "Reviewer_CyeB", "summary": "The paper presents Vid-SME (Video Sharma-Mittal Entropy) for membership inference attack on video understanding LLMs (VULLMs). The key idea is to use Sharma-Mittal Entropy to measure the token prediction difference between a normal input video and its reverse version, and consequently predict if the video belongs to the training set of the VULLM.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nAccording to the paper, existing membership inference attack studies are limited to image LLMs, and hence the presented work is novel for video understanding LLMs.\n\nThe idea of measuring the token prediction difference between a normal video and its reverse version is interesting and intuitively makes sense.\n\nThe use of Sharma-Mittal entropy is novel to this reviewer’s knowledge.\n\nThe benchmark contribution may provide useful facilities for future research on this topic.\n\n\nWeaknesses:\n\nThe problem of membership inference attack seems to be very impractical in the current setting. The graybox assumption, i.e., known token prediction but not other model information sounds quite artificial.\n\nWhile the motivation of comparing a normal video and its reversed version is interesting, is this actually true for videos both inside and outside the training set? In particular, for VULLM that have seen tons of videos, it likely already learns general time-of-arrow pattern. \n\nIn Line 132, it says that “… whether a specific video was used during the instruction tuning state …”. What if the input video has been used in the pretraining stage? Will the method still work?\n\nLine 135, the notation of I(…) is not consistent with that in equation (1).", "was_revised": true, "final_justification": "The rebuttal addresses my main concerns, so I raised the rating to Accept.", "rebuttal": "It's our great honor to have reviewer CyeB's valuable comments and kind support for our work. We would like to response to the questions as below.\n\n---\n\n> **[W1]**: The practicality of membership inference attacks under gray-box scenario.\n\n**A1:** Thanks for the comment. We would like to clarify that this scenario is practically motivated and commonly adopted in previous works (e.g., Min-K%, Rényi entropy). This is attributed to its broad applicability in real-world scenarios:\n\n* Many closed-source model APIs such as OpenAI's GPT-4 API **expose top-five per-token probabilities**, which also can be seen as a gray-box scenario in our context and the adversaries can exploit this information to conduct attacks. We follow this setting (where only the top-five per-token probabilities are accessible) and conduct an experiment on Video-XL-CinePile-7B. As shown in the table below, Vid-SME remains reliable in performing membership inference under this partial-knowledge scenario.\n\n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR |\n|---------------|-----------|-------|-------|-------------|\n| Vid-SME    | Mean   | 0.775 | 0.703 | 0.305 |\n|| Min_60%   | 0.657 | 0.628 | 0.229 |\n\n\n* Moreover, the gray-box setup represents a **conservative risk lower-bound**. If an attack succeeds under the gray-box setting, it indicates that the system may already be operating at the boundary of its privacy guarantees. \n\n* Furthermore, such attacks are not only relevant for external attackers, but also valuable for **model developers/owners to audit and monitor their own models for potential privacy risks**. \n\nWe sincerely hope that our responses can alleviate this concern and we will further clarify our practical motivations in the revision.\n\n\n---\n\n> **[W2]**: While the motivation of comparing a normal video and its reversed version is interesting, is this actually true for videos both inside and outside the training set?\n\n**A2:** Thanks for highlighting this important consideration. We believe the observed phenomenon reflects a generalizable statistical signal of VULLM memorization. \n\n* This is conceptually analogous to previous membership inference works in text and image domains, which also leverage statistical disparities between members and non-members even though models have seen large-scale text and image data. \n\n* In our case, we agree with the reviewer that the time-to-arrow pattern may already be well preserved by the model. Nevertheless, the model demonstrates **varying degrees of responsiveness to this pattern between seen and unseen videos**.\n\n* This discrepancy can be reliably captured via entropy calculations, as supported by our empirical findings: for videos seen during training, the model tends to **internalize more rigid temporal representations**. When these temporal patterns are violated by reversing the video frames, the model exhibits **asymmetric behavioral responses (e.g., sharper confidence drops), which are less prominent in non-member videos**. \n\nWe sincerely hope our responses help clarify this point and we will include the discussion in our revision.\n\n---\n\n> **[W3]**: In Line 132, it says that “… whether a specific video was used during the instruction tuning state …”. What if the input video has been used in the pretraining stage? Will the method still work?\n\n**A3:** Thanks for this valuable comment. We believe that Vid-SME can be well applied in pretraining stage of VULLMs. However, **the current state of development of VULLMs is insufficient to support a broad and meaningful evaluation**. We elaborate on the reasons below.\n\n* Currently, **for most of the video understanding LLMs (VULLMs), the pretraining stage is typically performed on text-only corpora** (e.g., using a pretrained LLM as the language base model), and the incorporation of video data often begins during instruction tuning or modality alignment. Although there exits a few models (e.g., Qwen-2.5 VL) incorporate video data during the pretraining stage, those video data used in pretraining are not for the purpose of understanding, and model owners often do not disclose any information about the sources or composition of the pretraining corpus. As a result, membership inference in this context can only **serve as an internal auditing tool for the model owner**. For the external world, the lack of transparency makes it infeasible to construct matched member and non-member sets to conduct meaningful evaluations of privacy leakage. **This is also one of the key motivations behind our instruction-tuning three VULLMs**: allowing for controlled, fine-grained evaluation of membership leakage under different model conditions. There exits a lines of works focusing on the pretraining stage for **text-only LLMs** (e.g., the Pythia series, the model developers have publicly released the full set of pretraining corpora for research purpose). However, unfortunately, **no such dataset transparency currently exists for VULLMs**, and pretraining a VULLM from scratch is extremely resource-intensive and beyond our current computational capacity. \n\n* Furthermore, from a privacy perspective, pretraining datasets tend to consist of broad, publicly available or web-scale video sources, where **privacy concerns are less urgent**. In contrast, the task-specific datasets curated by developers in **the subsequent instruction tuning stage are more likely to include sensitive or proprietary information**. Since the creation of such datasets often involves considerable time and financial investment, they are particularly vulnerable to unauthorized usage or misappropriation by third parties, **making the instruction tuning phase both the most valuable and the most susceptible to membership inference attacks on VULLMs**. Therefore, we believe that focusing on the instruction tuning stage currently aligns better with practical deployment settings and realistic privacy threats.\n\nThat said, **we fully agree with the reviewer** that performing membership inference on the pretraining stage is also an important and valuable direction, and therefore leave this as an important direction for future work.\n\n---\n\n> **[W4]**:  Line 135, the notation of I(…) is not consistent with that in equation (1).\n\n**A4:** Thanks for catching this typo. We will correct it in the revision by replacing ⊕ with || in Line 135.\n\n---\n\nThanks again for the thoughtful and constructive feedback. If reviewer CyeB has any remaining concerns, we would definitely love to clarify."}, {"review_id": "NHTa6m2KYr", "reviewer": "Reviewer_JbwM", "summary": "This paper proposes a novel MIAs method used in video understanding LLMs (VULLMs), termed Vid-SME (Video Sharma-Mittal Entropy). It leverages an adaptive parameterization strategy and both natural and reversed sequences to compute the Sharma-Mittal entropy, which effectively captures the inherent temporal variations for model behavior differences. As such, Vid-SME obtains the robust membership scores and shows strong effectiveness on various self-trained and open-sourced VULLMs.", "questions": "Refer to the Weaknesses. I will raise the score if well addressing these Weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated and well-written in this paper.\n2. A strong extension of the proposed Sharma-Mittal Entropy to decide the final Vid-SME score, i.e., Sharma-Mittal Entropy is a general version of the previous Renyi entropy.\n3. Superior performance achieved by the proposed Vid-SME on various VULLMs for MIAs, as well as a sufficient ablation study to evaluate the effectiveness of the proposed modules.\n\nWeaknesses:\n1. The idea of Natural and reversed sequences was used for Video Anomaly Detection, such as the reference [1]. Except for the task difference, what are the main benefits and differences to the work [1]. In addition, it is unclear to know whether the previous baselines (e.g., Rényi) use the information of natural and reversed sequences. If not, how to ensure the fairness of the comparison? \n2. The threshold to determine whether the member data is important to performance, while this paper lacks sufficient experiments on the selection of this threshold.\n\n[1] Comprehensive regularization in a bi-directional predictive network for video anomaly detection. In AAAI, 2022.", "was_revised": true, "final_justification": "My concerns are well addressed in the rebuttal, and I keep my original score to accept this paper.", "rebuttal": "It’s our great honor to receive Reviewer JbwM’s thoughtful comments and strong support for our work. We would like to address the concerns as below.\n\n---\n\n> **[W1.1]**:  The main benefits and differences to the mentioned work except for the task difference.\n\n**A1.1:** Thanks for bringing this work to our attention. [1] is indeed a valuable contribution that proposes a self-supervised temporal discrimination framework for VAD. We would like to provide the following perspectives:\n\n* In the mentioned work [1], **the model is explicitly trained** to distinguish between forward and reversed videos in order to capture the temporal dynamics of normal behavior, thereby enabling the identification of abnormal segments. In contrast, Vid-SME **does not involve training any additional temporal classifier**. Instead, we directly utilize the model’s response to natural-reversed videos as a controlled signal to probe a deployed VULLM's internal behavioral asymmetries—specifically, how its confidence distribution shifts for member/non-member videos.\n\n* Furthermore, [1] is fundamentally **video-centric**, focusing solely on the features intrinsic to the video itself, **without involving interaction with downstream VULLMs trained on it**. In comparison, Vid-SME targets the **video–model interplay**, leveraging both **the properties of the video and the model’s reaction to it**. This distinction reflects a deeper emphasis on how video content is encoded, retained, and reflected within the model.\n\nWe sincerely hope that our responses provide a reasonable comparison between the two and we will add the above discussions into the related work section of our revision.\n\n[1] Comprehensive regularization in a bi-directional predictive network for video anomaly detection. In AAAI, 2022.\n\n---\n\n> **[W1.2]**:  Whether the previous baselines (e.g., Rényi) use the information of natural and reversed sequences. If not, how to ensure the fairness of the comparison.\n\n**A1.2:** Thanks for raising this point. We followed **the default pipeline of the baseline methods** as described in their original papers without any modifications, ensuring a fair and consistent evaluation. In addition, we would like to point out that Rényi entropy also requires querying the source model twice.\n\nMotivated by the comment, we conduct an additional experiment by adapting Rényi entropy to incorporate the natural–reversed video frames. Specifically, we use **the difference between the Rényi entropies of the natural and reversed frames as the attack signal** (termed as Rényi (w), now the calculation of Rényi entropy for each video need to query/forward the model 4 times), and evaluate whether this modification improves the inference performance. The α in Rényi entropy is set to 2 and the frame number is set to 16. The results with both methods using the variant of Max_90%/Min_90% are presented below.\n\n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR | Time per Video (s) | \n|---------------|-----------|-------|-------|-------------|---------------------|\n| Rényi (w/o)  | Max_90%   | 0.661 | 0.655 | 0.004       |        11.67             |\n| Rényi (w)  | Max_90%   | 0.717 | 0.693 | 0.127       |        24.59             |\n| Vid-SME    | Min_90%   | 0.830 | 0.765 | 0.305 |       16.76              |\n\nAs shown, the modified attack achieves a performance improvement over the original setting that does not leverage the temporal information of the video. This highlights the **general effectiveness of temporal informations** in enhancing the attack signal. \n\n---\n\n> **[W2]**:  The threshold to determine the member data.\n\n**A2:** Thanks for pointing this out. In our evaluation, we follow the standard practice in membership inference attacks by **not fixing a threshold per sample**, but instead evaluating the attack’s performance **across all possible thresholds** using metrics such as AUC. AUC reflects the trade-off between true positive and false positive rates over the entire threshold range, providing a threshold-independent and robust assessment of the method's effectiveness. We will make the description clearer in the revision.\n\n---\n\nThanks again for the thoughtful and constructive feedback. If reviewer JbwM has any remaining concerns, we are happy to clarify further."}, {"review_id": "1p7RTtvjCn", "reviewer": "Reviewer_7wvH", "summary": "Authors explore the task inferring if a given video was used at the instruction-tuning stage of video understanding LLMs (VULLMs), referred to as membership inference attack. They use a form of entropy, Sharma–Mittal entropy, and calculate its difference between natural and reversed video sequences, with some adaptive parameters. The calculation is performed using the probability of the next video token given past, for the videos used. Thorough experiments using multiple MLLMs highlight the performance.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths** \n1. Clever idea of entropy calculation well suited for video tasks and well-motivated from prior video works\n2. Novel exploration into inference attacks specific for video LLMs\n\n**Weaknesses**\n1. How does this work with newer generic MLLMs like Qwen2.5-VL (trained on image and video)? Comparisons are against relatively older video LLMs. \n2. What is the inference time? How costly is the SM-entropy calculation? This is missing.\n3. The method appears sensitive to frame count. What is the exact sampling rate of the video used during training is unknown?", "was_revised": true, "final_justification": "Authors resolve all concerns. Raising vote to Accept.", "rebuttal": "It’s our great honor to receive Reviewer 7wvH’s thoughtful comments and kind support for our work. We would like to address the questions reflected in the review below.\n\n---\n\n> **[W1]**:  How does this work with newer generic MLLMs like Qwen2.5-VL?\n\n**A1:** Thanks for the valuable comment. We further conduct an experiment on **Qwen2.5-VL-7B**:\n\n* We follow the scripts in its official codebase and use **the generic subset of lmms-lab/VideoChatGPT** as the member data (randomly sampling 1027 examples to match the number of non-members). For the non-member set, we use all 1027 samples from **the detailed captioning category in the VDC benchmark**. The number of frames is set to 8 and 16. We report a detailed comparison of attack performance between Vid-SME and Rényi entropy. The results are presented in the tables below.\n\n_\\# Frames = 8:_\n\n| Method         | Variant   | AUC   | Acc.  | TPR@5% FPR |\n|----------------|-----------|-------|-------|-------------|\n| Rényi (α=2)           | Max_0%    | 0.403 | 0.500 | 0.009  |\n|                | Max_5%    | 0.442 | 0.504 | 0.012       |\n|                | Max_30%   | 0.487 | 0.508 | 0.014       |\n|                | Max_60%   | 0.506 | 0.516 | 0.018       |\n|                | Max_90%   | 0.510 | 0.521 | 0.004       |\n| Vid-SME        | Mean      | 0.576 | 0.568 | 0.124   |\n|                | Min_0%    | 0.629 | 0.614 | 0.176       |\n|                | Min_5%    | 0.653 | 0.625 | 0.199       |\n|                | Min_30%   | **0.674** | **0.643** | 0.226       | \n|                | Min_60%   | 0.658 | 0.629 | 0.231       | \n|                | Min_90%   | 0.655 | 0.621 | **0.234** | \n\n_\\# Frames = 16:_\n\n| Method         | Variant   | AUC   | Acc.  | TPR@5% FPR |\n|----------------|-----------|-------|-------|-------------|\n| Rényi (α=2)            | Max_0%    | 0.418 | 0.503 | 0.016       |\n|                | Max_5%    | 0.473 | 0.509 | 0.017      |\n|                | Max_30%   | 0.501 | 0.514 | 0.020       |\n|                | Max_60%   | 0.512 | 0.522 | 0.014       |\n|                | Max_90%   | 0.509 | 0.518 | 0.018       |\n| Vid-SME        | Mean      | 0.594 | 0.586 | 0.128 | \n|                | Min_0%    | 0.642 | 0.628 | 0.194       |\n|                | Min_5%    | 0.672 | 0.642 | 0.231       |\n|                | Min_30%   | **0.693** | **0.685** | 0.274       |\n|                | Min_60%   | 0.665 | 0.630 | **0.276**       |\n|                | Min_90%   | 0.668 | 0.627 | 0.248 |\n\nThe results indicate that Vid-SME consistently achieves superior performance on recent models such as Qwen-2.5 VL, while maintaining its scalability with respect to the number of frames.\n\n---\n\n> **[W2]**: What is the inference time? How costly is the SM-entropy calculation?\n\n**A2:** Thanks for raising this point. In the table below, we report a detailed comparison of inference time per video and attack performance between Rényi entropy and Vid-SME. The experiments are conducted using the Video-XL-CinePile-7B model with frame numbers set to 8 and 16, and with both methods using the variant of Max_90%/Min_90% and α in Rényi entropy is set to 2 (note that the variant used and the α value in Rényi entropy do not affect the inference time).\n\n_\\# Frames = 8:_\n\n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR | Time per Video (s) | \n|---------------|-----------|-------|-------|-------------|---------------------|\n| Rényi (α=2) | Max_90%   | 0.559 | 0.585 | 0.004       |        6.11             |\n| Vid-SME    | Min_90%   | 0.806 | 0.760 | 0.289 |       9.45              |\n\n_\\# Frames = 16:_\n  \n| Method        | Variant   | AUC   | Acc.  | TPR@5% FPR | Time per Video (s) | \n|---------------|-----------|-------|-------|-------------|---------------------|\n| Rényi (α=2)  | Max_90%   | 0.661 | 0.655 | 0.004       |        11.67             |\n| Vid-SME    | Min_90%   | 0.830 | 0.765 | 0.305 |       16.76              |\n \n* Compared to Rényi entropy, the additional computational cost of Vid-SME lies in **the computation of the adaptive q/r values**. When these video-specific values are not computed, the inference-time cost of Vid-SME is nearly identical to that of Rényi entropy, as **both require two forward queries to the model**. \n\n* As the number of frames increases, Vid-SME demonstrates **good scalability in both inference time and performance improvement**. \n\nWe sincerely hope that our responses adequately address the concern and we will include the discussion in our revision.\n\n---\n\n> **[W3]**: The method appears sensitive to frame count. What is the exact sampling rate of the video used during training is unknown?\n\n**A3:** Thanks for the valuable comment. Regarding the influence of frame count, we would like to clarify that this is **not a method-related property but rather an intrinsic property of video data**—varying the number of frames naturally leads to different degrees of temporal information being captured. As shown in the results provided in our responses to [W1] and [W2], Vid-SME **exhibits consistent scalability in terms of inference time and attack performance as the number of frames varies**. For the three self-trained VULLMs, we adopted a training frame count of 128. We sincerely hope our responses help clarify this point and we will include these clarifications and relevant details in our revision.\n\n---\n\nThanks again for the thoughtful and constructive feedback. If reviewer 7wvH has any remaining concerns, we would definitely love to clarify further."}]

Author rebuttal:

Meta-review: This paper introduces the first membership inference method specifically designed for video data utilized in video understanding large language models (LLMs), termed Vid-SME. This method harnesses the confidence of model outputs and incorporates adaptive parameterization to calculate Sharm-Mittal entropy (SME) for video inputs. Experiments conducted on a variety of self-trained and open-sourced video understanding LLMs (VULLMs) validate the efficacy of Vid-SME. All reviews acknowledge the significance of the generalized entropy measure. The authors' rebuttal has effectively addressed the majority of concerns raised by the reviewers. Overall, I recommend the acceptance of this submission. Additionally, I anticipate that the authors will integrate the new results and suggested modifications from the rebuttal phase into the final version.
Decision: Accept (poster)