Title: Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Membership Inference

Video-Language Models
Abstract: Multimodal large language models (MLLMs) demonstrates remarkable capabilities in handling complex multimodal tasks and are increasingly adopted in video understanding applications. However, their rapid advancement raises serious data privacy concerns, particularly given the potential inclusion of sensitive video content, such as personal recordings and surveillance footage, in their training datasets. Determining improperly used videos during training remains a critical and unresolved challenge. Despite considerable progress on membership inference attacks (MIAs) for text and image data in MLLMs, existing methods fail to generalize effectively to the video domain. These methods suffer from poor scalability as more frames are sampled and generally achieve negligible true positive rates at low false positive rates (TPR@Low FPR), mainly due to their failure to capture the inherent temporal variations of video frames and to account for model behavior differences as the number of frames varies. To address these challenges, we introduce Vid-SME (**Vid**eo **S**harma–**M**ittal **E**ntropy), the first membership inference method tailored for video data used in video understanding LLMs (VULLMs). Vid-SME leverages the confidence of model output and integrates adaptive parameterization to compute Sharma–Mittal entropy (SME) for video inputs. By leveraging the SME difference between natural and temporally-reversed video frames, Vid-SME derives robust membership scores to determine whether a given video is part of the model's training set. Experiments on various self-trained and open-sourced VULLMs demonstrate the strong effectiveness of Vid-SME.


Peer reviews:
comments: SUMMARY:
Authors explore the task inferring if a given video was used at the instruction-tuning stage of video understanding LLMs (VULLMs), referred to as membership inference attack. They use a form of entropy, Sharma–Mittal entropy, and calculate its difference between natural and reversed video sequences, with some adaptive parameters. The calculation is performed using the probability of the next video token given past, for the videos used. Thorough experiments using multiple MLLMs highlight the performance.

STRENGTHS AND WEAKNESSES:
**Strengths** 
1. Clever idea of entropy calculation well suited for video tasks and well-motivated from prior video works
2. Novel exploration into inference attacks specific for video LLMs

**Weaknesses**
1. How does this work with newer generic MLLMs like Qwen2.5-VL (trained on image and video)? Comparisons are against relatively older video LLMs. 
2. What is the inference time? How costly is the SM-entropy calculation? This is missing.
3. The method appears sensitive to frame count. What is the exact sampling rate of the video used during training is unknown?

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper proposes a novel MIAs method used in video understanding LLMs (VULLMs), termed Vid-SME (Video Sharma-Mittal Entropy). It leverages an adaptive parameterization strategy and both natural and reversed sequences to compute the Sharma-Mittal entropy, which effectively captures the inherent temporal variations for model behavior differences. As such, Vid-SME obtains the robust membership scores and shows strong effectiveness on various self-trained and open-sourced VULLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Well-motivated and well-written in this paper.
2. A strong extension of the proposed Sharma-Mittal Entropy to decide the final Vid-SME score, i.e., Sharma-Mittal Entropy is a general version of the previous Renyi entropy.
3. Superior performance achieved by the proposed Vid-SME on various VULLMs for MIAs, as well as a sufficient ablation study to evaluate the effectiveness of the proposed modules.

Weaknesses:
1. The idea of Natural and reversed sequences was used for Video Anomaly Detection, such as the reference [1]. Except for the task difference, what are the main benefits and differences to the work [1]. In addition, it is unclear to know whether the previous baselines (e.g., Rényi) use the information of natural and reversed sequences. If not, how to ensure the fairness of the comparison? 
2. The threshold to determine whether the member data is important to performance, while this paper lacks sufficient experiments on the selection of this threshold.

[1] Comprehensive regularization in a bi-directional predictive network for video anomaly detection. In AAAI, 2022.

QUESTIONS:
Refer to the Weaknesses. I will raise the score if well addressing these Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
The paper presents Vid-SME (Video Sharma-Mittal Entropy) for membership inference attack on video understanding LLMs (VULLMs). The key idea is to use Sharma-Mittal Entropy to measure the token prediction difference between a normal input video and its reverse version, and consequently predict if the video belongs to the training set of the VULLM.

STRENGTHS AND WEAKNESSES:
Strengths:

According to the paper, existing membership inference attack studies are limited to image LLMs, and hence the presented work is novel for video understanding LLMs.

The idea of measuring the token prediction difference between a normal video and its reverse version is interesting and intuitively makes sense.

The use of Sharma-Mittal entropy is novel to this reviewer’s knowledge.

The benchmark contribution may provide useful facilities for future research on this topic.


Weaknesses:

The problem of membership inference attack seems to be very impractical in the current setting. The graybox assumption, i.e., known token prediction but not other model information sounds quite artificial.

While the motivation of comparing a normal video and its reversed version is interesting, is this actually true for videos both inside and outside the training set? In particular, for VULLM that have seen tons of videos, it likely already learns general time-of-arrow pattern. 

In Line 132, it says that “… whether a specific video was used during the instruction tuning state …”. What if the input video has been used in the pretraining stage? Will the method still work?

Line 135, the notation of I(…) is not consistent with that in equation (1).

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper introduces a new Membership Inference Attack (MIA) method (Vid-SME) tailored for the temporal nature of video data for video understanding large language models (VULLMs)\n- Existing MIA techniques designed for text or image data perform poorly when applied to video, largely due to their inability to account for the temporal dynamics and frame-dependent model behavior unique to video.\n- To address this, the paper proposes Vid-SME, which uses Sharma-Mittal entropy to quantify model confidence under two conditions: when a video is shown in its natural order and when the frame sequence is reversed. \n- The key insight is that VULLMs trained on a video are more confident (lower entropy) when frames are presented in their original order compared to reversed order. \n- By measuring the entropy gap between these two settings and aggregating the smallest differences (via a K%-min pooling strategy), Vid-SME generates a membership score for each video.\n- The paper also introduces a benchmark suite using three self-trained and two open-sourced VULLMs to evaluate the attack across diverse datasets, training strategies, and video types. \n- Experiments demonstrate that Vid-SME significantly outperforms prior methods, particularly in low false positive rate (FPR) regimes, showing substantial gains in metrics like AUC and TPR@5%FPR.\n\nSTRENGTHS AND WEAKNESSES:\n-  VULLMs are increasingly trained on sensitive, real-world video data (e.g., surveillance, personal videos). The paper addresses an urgent and underexplored privacy risk - whether such models memorize and inadvertently expose their training data.\n- The entropy-based scoring function using natural vs. reversed video sequences is a creative and well-motivated idea that leverages the temporal structure unique to videos—this is not a straightforward extension of prior MIA techniques.\n- The adaptive use of this generalized entropy measure provides a novel and theoretically grounded mechanism to capture nuanced differences in model confidence across frames.\n- The results are strong compared to the baselines and the paper goes beyond surface-level results by providing visualizations (e.g., Figure 2a–2b) that explain why the method works, showing how entropy differences become more distinct as video length increases.\n\nWeaknesses:\n- The attack appears to rely on logits or confidence scores from the model. It is unclear whether Vid-SME works when only hard labels (top-1 predictions) are available, which would be a more realistic setting for many deployed systems.\n- While the paper emphasizes adaptive entropy parameterization, the method for choosing q/r values per video is underexplained. Is this based on optimization, empirical tuning, or predefined heuristics?\n- The method requires computing entropy over possibly long sequences and for both natural and reversed orderings. There is no discussion of runtime or scalability with respect to video length or number of frames.\n\nQUESTIONS:\n- Suggestion: Clarify the threat model. Could Vid-SME be adapted to black-box APIs?\n- How are the Sharma–Mittal parameters q and r determined for each video? Are they chosen via optimization, heuristics, or fixed based on video metadata? Do these values generalize across datasets or need to be tuned for each model/video?\n- Why is frame reversal specifically chosen as the transformation to reveal memorization? Have you considered or tested alternative temporal perturbations (e.g., frame shuffling, masking, partial reversal)?\n- What is the computational overhead of Vid-SME per video inference?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}