DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Authors: André V. Duarte
Xuandong Zhao
Arlindo L. Oliveira
Lei Li
Venue: ICML 2025 poster (2025)
Keywords: Copyrighted Content Detection
Membership Inference Attacks
Large Vision Language Models
Natural Language Processing
Primary area: social_aspects->safety
TL;DR: DIS-CO identifies copyrighted content in VLMs training data by showing that models can link movie frames to their titles in a free-form text generation setting, even when the frames are highly challenging, suggesting prior exposure during training.

Abstract: *How can we verify whether copyrighted content was used to train a large vision-language model (VLM) without direct access to its training data?* Motivated by the hypothesis that a VLM is able to recognize images from its training corpus, we propose DIS-CO, a novel approach to infer the inclusion of copyrighted content during the model's development. By repeatedly querying a VLM with specific frames from targeted copyrighted material, DIS-CO extracts the content's identity through free-form text completions. To assess its effectiveness, we introduce MovieTection, a benchmark comprising 14,000 frames paired with detailed captions, drawn from films released both before and after a model’s training cutoff. Our results show that DIS-CO significantly improves detection performance, nearly doubling the average AUC of the best prior method on models with logits available. Our findings also highlight a broader concern: all tested models appear to have been exposed to some extent to copyrighted content. We provide the code in the supplementary materials.

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
André V. Duarte 12
Xuandong Zhao
Arlindo L. Oliveira Lei Li
Abstract
VLMs accurately match Suspect movie titles
How can we verify whether copyrighted con- tent used to train large vision-langugge model (VLM) withoul direct access to its training data? Motivated by the hypothesis that VLM is able to recognize images from its training cor- pus, we propose DIS-CO, a novel approach to infer the inclusion of copyrighted content during the model's development. By repeatedly query- ing a VLM with specific frames from targeted copyrighted material, DIS-CO extracts the con- tent's identity through free-form text completions. To assess its effectiveness, we introduce Movie- Tection, a benchmark comprising 14,000 frames paired with detailed captions, drawn from films released both before and after a model's 's training cutoff. Our results show that DIS-CO significantly improves detection performance, nearly doubling the average AUC of the best prior method on mod- els with logits available. Our findings also high- light a broader concern: all tested models appear to have been exposed to some extent to copy- righted content. Our code and data are available at https: / / thub.com/ vvduarte333/ DIS-C CO
Q: "Whot movic is this frome from?"
Visual-LLM
OO OGG
to their frames!
1.0
Frozen
Suspect Movies
Clean Movies
0.8-
Accuracy
0.6-
0.4-
0.2-
Aquanan-22
0.0
-201
2012
2014
2016
2018
2020
2022
2024
Year of Release
Figure 1. Our DIS-CO reveals that VL.Ms successfully map frames of suspect movies to their titles, when the frames highly challenging. For example, the GPT-4o model accurately identifies the movie "Frozen" (blue circle), despite the varying complexity of its frames. In contrast, for newly released films, the models are unable perform similar frame-to-tille predictions.
verse datasets introduces challenges in ensuring compliance with ethical and legal standards. Without strict safeguards during the data collection step. proprietary content could be incorporated into the models' knowledge, opening the door to intellectual property infringements and potential legal conflicts (Carlini et al., 2022a; Nasr et al., 2023; Duan et al., 2024). fact, in the United States alone, more than 24 copyright lawsuits were filed against the AI industry since 2023 (Knibbs, 2024), reflecting growing concerns about the use of protected material in training (Kadrey, 2023; Daily News, 2024).
1. Introduction
The rapid evolution of large-scale models has driven paradigm shift toward multimodality, with recent large vision-language models (VLMs) gaining prominence for their ability to process both visual and textual information (Alayrac et al., 2022; Liu et al., 2023; OpenAl, 2023; Wang et al., 2024). While these models showcase remarkable per- formance across a variety of tasks, their reliance on vast, di-
Discovering training data is, therefore, essential for effec- tively addressing the ethical and legal challenges of model training. However, the lack of transparency in data col- lection (often justified by competitive concerns) makes especially difficult trace the inclusion of specific content.
CCrreeiie Mellon University 'INESC-ID / Instituto Su- perior Técnico, ULisboa 'UC Berkeley. Correspondence to: André V. Duarte <andre wduurrteeeeenieee wtteeiicc..llsba.p....... Xuandong Zhao <xuandongzhao berkeley.edu>> Arlindo L. Oliveira <ar- lindo. wwwcanttttneniiieieeeeeeeeooooee wttpiiic...issba.p...... Lei Li <leili @ cs. emu.edu>.
To tackle these challenges, Membership Inference At- tacks (MIAs) serve as a tool to identify whether specific data samples were part of model's training set. While MIA techniques are well-studied for taxt- models, their adaptation to multimodal settings, particularly VLMs, remains less explored gap that our work aims address.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Recent advancements have been made in this field, but certain challenges and limitations continue to exist. The
Our main contributions are as follows:
We introduce DIS-CO, an innovative method applicable to both white-box and black-box VLMs, designed to detect whether copyrighted content was included in training. To the best of our knowledge, this is the first work to detect copyrighted movies in the context of VLMs.
MaxRényi-K%% method (Li et al., 2024b) is based on the in- tuition that data encountered during training leads to greater model confidence when generating outputs, reflected by a smaller Rényi entropy in the next-token probability distri- bution for image or text tokens. While this method proved effective in settings with unrestricted access to output logits, its applicability is limited in the context of black-box mod- els like Gemini (Reid et al., 2024), which, at most, allow for inspection of the top-5 logits for each predicted token. The work of Pinto et al. (2024) addresses the challenge of discovering training data in a fully black-box fashion by introducing a method tailored to document-based VQA tasks. By removing key textual content from input images, the authors demonstrate the models' ability to recall mem- orized training data, including sensitive information. This approach is, nonetheless, limited to VQA datasets, which constitute only a small fraction of the diverse data types used to train VLMs.
We introduce MovieTection, a new benchmark designed for detecting training data of VLMs. The dataset in- cludes content from 100 movie titles, unfolded into 14,000 frames, with mix of easily identifiable and challenging examples to test model capabilities comprehensively.
Experiments show that DIS-CO effectively detects copy- righted movies across the six tested model families, with GPT-4o achieving an average accuracy of 34% on the "hard to guess" frames of the suspect movies.
We show that fine-tuning a model to avoid disclosing memorized content for a particular movie an effective defense strategy, capable of generalization to other titles.
In this paper we propose DIS-CO, novel method for detect- ing models' training data that overcomes the limitations of previous approaches while being applicable both white- and black-box VLMs. The core idea of DIS- MIS-COiz is to prompt models to map a set of images from a target media docu- ment to its corresponding identity (e.g., movie titles) in an unconstrained, free-form cext-generation manner, enabling them to produce answers freely instead of selecting from predefined options. As a result, under the null hypothesis: the target content was not in the training dataset, the model is much less likely to correctly identify the data, reducing false positives. Consequently, correct outputs become a more reliable indication that the target content was part of the training dataset (Zhang et al., 2024a).
2. Preliminary and Related Work
Membership Inference Attacks (MIAs) are designed to de- termine whether a specific data instance was included in the training set of machine learning model (Shokri et al., 2017: Hu et al., 2022). This area of research has seen growing interest with the increasing of LLMs, which are known to memorize and occasionally reproduce training data (Nasr et al., 2023; Carlini et al., 2022b; Hans et al., 2024).
Classical MIAS are typically divided into two main ap- proaches: referencce-bsee and reference-free. Reference- based methods involve training a set of "shadow models" to replicate the target model's behavior (Carlini et al., 2022a; Long et al., 2018; Mireshghallah et al., 2022; Watson et al., 2022). In contrast, reference-free methods rely on cal- culating specific metrics, such as the perplexity of a sen- tence, to identify patterns indicative of training set member- ship (Yeom et al., 2018; Salem et al., 2018; Carlini et al., 2020; Song & Mittal, 2021). Among these, the Min-K%- Prob method stands out as more refined approach. It hypothesizes that the average log-liieldhood of the top-k% least probable tokens in example is higher if the exam- ple was part of the training data compared to if it was not (Shi et al., 2023). Building on this foundation, recent ex- tensions such as Min-K%++ (Zhang et al., 2024b) and DC- PDD (Zhang et al., 2024c) have introduced further improve- ments. However, key limitation of most reference-free methods is their dependence on access token probabil- ity distributions, which restricts their interoperability with black-box models such as Gemini (Reid et al., 2024).
This idea is illustrated in Figure which shows that models solve the task correctly far more often for content that was very likely included in their training data compared to con- tent that was definitely excluded, such as movies released after the model's cutoff knowledge date. For instance, DIS- CO maps frames from the movie Frozen to the correct title in nearly 90% of the test cases, while a movie like Aquaman and the Lost Kingdom is correctly identified in less than 2% of the time.
We conduct experiments on two benchmarks, Mov tion (our newly introduced dataset) and VI-MMA/Fllickr (Li et al., 2024b). Movie'eectoon contains 14,000 diverse movie frames paired with descriptive captions, split chronologi- cally based on films released before/after the models' train- ing cutoff (October 2023). VI.-MIA/Flickr, derived from COCO (Lin et al., 2014) (member data) and recent Flickr images (non-member data), serves as proof-of-validity dataset for DIS-CO.
With recent research shifting focus from text-only models to mudti- architectures, the task of detecting training data and evaluating model memorization has begun to emerge
2

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
in this domain as well (Kokhlikyan al., 2024; Jayaraman et al., 2024; Pinto et al., 2024). Building on techniques originally developed for text-only models, Li et al. (2024b) propose a novel image-based MIA pipeline that adapts meth- ods like Min-K%-Prok (Shi et al., 2023) to VLMs. The work presents the Max metric, which enables image membership inference by analyzing the output logits corresponding to the model's image-specific slice.
with plans for future expansion. The selection of movies incorporated into the benchmark is guided by their status as box office hits, based on the assumption that highly popular movies, due to their widespread availability, are more likely to appear in training datasets. For the suspect data, we pri- marily select titles randomly from the Box Office Mojo's' list of the all-time highest-grossing films, with some excep- tions to accommodate specific experiments, such as analyz- ing the impact of IMDb² ratings (Section 5). For the clean data, we sample most titles from the Box Office Mojo's 's list of the highest-grossing films of 2024.
Detecting training data is especially significant when it in- volves copyrighted content, as the reproduction of such ma- terial by large models raises legal and ethical concerns (Li et al., 2024a; Meeus et al., 2024a). In the light of coun- terfactual memorization studies, the methods proposed by Duarte et al. (2024) and Golchin Surdeanu (2024) perform membership inference through a multiple-choice question- answering (MCQA) setting. These approaches demonstrate solid results and have the advantage of being applicable to both woite-box and black-box models, as they do not depend on access to token probabilities. However, it is known that multiple-choice scenarios may induce a selec- tion bias (Zheng et al., 2024a) on the models, which intro- duces some uncertainty about whether chance played a role in the results. In contrast, Karamolegkou et al. (2023) adopt prompting approach with free-form text generation, aim- ing to elicit verbatim reproduction of copyrighted material. This provides stronger evidence of memorization because, in an unconstrained, free-form setting, the model is much less likely to produce correct outputs by chance. While this method may fail to detect cases where models are trained on copyrighted data without memorizing it (Meeus al., 2024b), it also avoids the issues raised by Das et al. (2024) and Maini et al. (2024), who warn that many membership inference methods risk overstating results by exploiting data distribution shifts, such as temporal patterns, rather than identifying genuine memorization. For these reasons, we also focus on free-form text generation in this work, as it provides a more robust and unbiased indication of whether the target content is part of the model's training data.
For each movie, we extract frames categorized into two types: main frames and neutral frames. This categorization is designed to introduce varying levels of difficulty for as- sessing a model's knowledge about movie. Main frames typically feature key characters to the movie's plot. These frames are intended to be easily recognizable by viewers familiar with the movie. In contrast, neutral frames focus on ordinary visuals, such as landscapes, objects, or minor characters, that are not strongly tied to the movic's narrative. Neutral frames are designed to present a significantly higher challenge, as they rely on subtle contextual cues that are almost impossible to associate with the correct title without prior knowledge of the movic. Figure 2 illustrates the two frame types.
Each extracted frame is accompanied by a detailed caption, generated using the 7B version of the OWen2-VL model. The prompt used for caption generation, along with an ex- ample, provided in Appendix B. In total, 140 frames are extracted per movie, comprising 100 main frames and 40 neutral ones.
4. DIS-CO
Our proposed method, DIS-CO, determines whether exam- ples are memorized by evaluating the model's performance on guestion-answering task with free-form text responses. The task we propose involves models performing accurate identification of the content's identity, which, in the case of copyrighted movie identification, corresponds to correctly identifying the movie title. We operate under the premise that models map a frame to the appropriate title far more reliably when that movie included in their training data compared when it is not.
3. Benchmark: MovieTection
Our proposed benchmark, MovieTection, distinguishes member and non-member data based on a clearly defined temporal constraint. Movies released in 2024 or later are considered non-member data, as they fall outside the knowl- edge cutoff dates of all tested models. Movies from January to September 2023 are excluded due to uncertainty regarding models' exposure to content from that period. For instance, Qwen2-VIL VL (Wang et al., 2024) reports a knowledge cutoff in June 2023. Movies released on or before 2022 are treated as potential member data, as they are more likely to have been included in the training datasets of such models.
In order to perform this mapping, we argue that eliciting free- form (FF) completions is preferable to a multiple-choice question-answerte (MCQA) format because it significantly reduces the influence of "luck" associated with guessing. More concretely, consider the probability of a model se- lecting the correct answer purely by chance in an MCQA
'nttp::/wwwbooxfficcmmttt Phttps://www.ine
MovieTection currently comprises frames from 100 movies,
3

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Data Collection: Frames can be of 2 different types.
Build MovieTection Dataset
Baseline Performance Range
Main Frames
Suspect
Clean
Query framne deeccription from?"
IMG
Contains Main Characters
feattring cheracters the
Caption
Easily Recognizable
Box Office Hits
identifibble viewera who saw the moxie.
50 Suspact [50 Clean
Vision-LLM 10 +
At least $700M Worddwide I8...
Neutral Frames
Background/ / Neutral Etements
Data Proparties
Al Mokes
like landscapes oljjects.
Hardly Recognizable
Detalled Caption/ Frome
Answer A: The crovi
Froe-Form Gonoredion
Fremis not aasily red the movik's la nemrativ..
100%
1.1 Query Model with IMGs
Accuracy
80%
Large A?
1.2 Query Model with Text Caption
80%
40%
2. Remove Intersection
Suspect Movie was In Pretraining Data
3. Evaluate
0%
Suspect Movie
Suspect Movie
Baseline
Pertormence
Figure 2. The pipeline begins with the construction of the MovieTection benchmark, where categorize movie frames into main and neutral types to introduce varying levels of difficulty. Models are then queried with image frames and their corresponding captions, producing predictions for both modalities. Predictions from image frames that overlap with caption- based predictions are excluded, isolating cases where image-based memorization is inferred. Performance on the suspect movie is compared against the expected baseline performance, with discrepancies indicating potential training exposure.
scenario with k: answer choices. The chance-level accuracy is PMCQA = 1/k. which for the standard case of k 4, results in baseline accuracy of 25%. This means that even models without any genuine knowledge or memorization of the content can appear to perform well, simply due to the structure of the task.
acknowledged by the models, leading to correct predictions for certain examples (See Table - Section 6.3 and Ap- pendix C). Capturing this baseline performance is crucial to avoid incorrectly classifying movie as part of the training data simply because some frames were accurately identified.
Another important factor to consider is the time effect. In general terms, the older a movie is, the greater the likelihood that a model has residual knowledge about it. This knowl- edge can come from publicly available online content, such as movie posters, trailers, forum discussions, or datasets like OpenSubtitles (Lison & Tiedemann, 2016), which typ- ically do not raise copyright infringement concerns. To estimate this baseline knowledge accumulated over time, we query the models using the detailed captions from the older movies, as making correct predictions based solely on textual descriptions is unlikely to be problematic.
In contrast, with free-form completions, the models must generate the correct answer from a much larger output space, which, for tasks such as ours, can include thousands of unique possible responses. The probability of producing the correct answer by chance in this unconstrained setting is Prr # 1 1/19), which, for [ 10,000, corresponds to baseline accuracy of just 0.01 %. Even when accounting for realistic factors such as model bias toward more popular answers, suppose particular title is 100 times more likely to be generated than an average candidate, the probability PFP = 100/10,000 1% still remains dramatically lower than the multiple-choice baseline. As result, we have substantially reduced risk of false positives, and correct predictions are much more likely to reflect genuine memo- rization.
Finally, to determine whether specific movie was likely included in the model's 's training data, query the model separately with image frames and then with their correspond- ing caption information from the suspect movie. After both queries are completed, we compare the predictions from these two inputs. If there is an overlap of correct predic- tions between the frame-based and caption-based queries, we disregard those results, as they suggest the model did not had rely on the image content to make accurate pre- dictions. By examining the remaining correct predictions, which rely solely on image content, we infer whether the model is utilizing memorized visual information. Ideally, af- ter removing the intersection, the performance of the suspect
The overall pipeline of DIS-CO is illustrated in Figure 2. After constructing the fection dataset, we first query the models with clean data to establish baseline for its expected performance on this set. While one might expect the models to fail completely on all these examples, given that these movies were unreleased at the time of the training cutoff, this is not always the case. Some of the movies in this set, though unreleased, were already announced and
4

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
movie should fall within this range defined by the baseline performance the recent movies and the clean baseline performance on older movies with accumulated knowledge over time. However, if performance remains significantly higher than this range, even after removing the intersection, it suggests that the model relied on memorized visual infor- mation specific to the movie frames, indicating the movie was likely included in its training data.
in the box-office experiment, movies with similar IMDb ratings are chosen to isolate the impact of popularity.
Does a longer context reveal more memorization? As LLMs often perform better with more context in their queries, we hypothesize that VLMs behave similarly. Us- ing the MovieTection dataset, we examine the effect of varying the number of frames in the prompt (N [1,
How susceptible are models to memorization when exposed new data? We investigate the model's s ability to memorize new content by fine-tuning on movie guaranteed to be outside the training data.
The specific prompts used for evaluating models on the MovieTection benchmark are provided in Appendix D.
4.1. Upper-Bound Estimation of Memorization
How to prevent a model to disclose memorization? Similarly to the previous experiment, we fine-tune the model (this time on a suspect movie), with a modified labeling objective. This experiment investigates whether this defense mechanism can mitigate memorization dis- closure for the suspect movie and whether its effects gen- eralize to other movies.
While our proposed approach of removing the intersec- tion between frame-based and caption-based correct pre- dictions provides a more precise set of potentially mem- orized movies, we cannot rule out that those frames are not part of the training data. Consequently, we also con- sider an upper-bound estimation of memorization, where all correctly identified frames, regardless of their intersection with captions, contribute to the possible classification of the movie as part of the training data. For clarity, throughout the remainder of this paper, we use two notations: [DIS-CO] represents the smaller set obtained after removing the inter- section, while DIS-CO denotes the upper-bound bound estimation, including all correctly identified frames.
To what extent does generalization influence the model's performance? Humans are capable of general- izing from partial information, often identifying movies they haven't tt fully seen by relying related content such as posters or trailers. To assess how closely models align with humans on this movie detection task, we compare the performance of the models with that of 10 human par- ticipants who were selected identify 200 images from MovieTection.
4.2. Mitigating the Disclosure of Memorization
While training on copyrighted data may at times be unavoid- able, the associated risks can be mitigated by ensuring the model does not disclose memorization. For movie likely included in the training data, we propose fine-tuning the model on subset of its frames while replacing the movie label with a neutral designation such as 'Copyrighted Con- tent'. More details on the fine-tuning are in Appendix E.
5.1. Experiment Setup
To evaluate DIS-CO, we follow the procedure outlined as follows. Let the "Suspect" group be represented as S = (81, 82,,8,,...,, and the "Clean" group as C {2,, CNe)) where Ns and Nc denote the number of movies in each group, respectively. For each movie, we calculate its accuracy: A(s;) for 8; E S and A(cz) for Cj E C. The accuracy is calculated as the proportion of pre- dictions aligning with the expected outcomes. By default, a weighted average is then applied account for the unequal proportions of main and neutral frames and the total value is reported. Nonetheless, some results for main and neutral frames are reported individually to provide further insights on the performance across frame types.
5. Experiments
We assess the effectiveness of DIS-CO through a range of different experiments, which are guided by the following questions:
Is DIS-CO suitable for different visual content types? While our primary focus is on movie frames, VLMs are exposed to a broader range of data during training. As a proof-of-concept,
We then perform a random sampling process with replace- ment, repeated 10 times. each iteration, M elements are sampled from each group, where M corresponds to Ns or Nc, depending on the group being sampled. For each iteration, threshold 0 is optimized to achieve maximum separation between the two groups, and the Area Under the Curve (AUC) is computed.
test whether DIS-CO can detect memorization in other domains, such comic books and photography.
Are factors like movie popularity or quality good prox- ies for memorization? To test whether popularity (c.g., box-office revenue) or quality (e.g., IMDb ratings) are proxies for memorization, we collect movies where one factor varies while the other is controlled. For instance,
To complete the analysis, we calculate the mean and stan- dard deviation of the AUC or the average accuracy for the
5

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
"Suuspett" and "Clean" groups over these iterations. Detec- tion is consistently conducted at the movie level, rather than on individual frames.
Table 1. Accuracy on the Comics Dataset.
Method
GPT-4o
Gemini-1.5 Pro
Qwen2-VL 72B
MCQA
0. 0.728 0s
0.6470.04
0.689p.09
5. BENCHMARKS AND BASELINES
DIS-CO
0.538p
0.4366.07
0.2630.07
We begin by evaluating DIS-CO on two coone-tf-oncept datasets. The first is a custom -ssemmbed collection of ap- proximately 1,000 comic book pages, sampled from widely recognized series such as Tintin, Asterix, and Lucky Luke. The second dataset, VL-MIA/Flickr, was introduced by Li et al. (2024b) and comprises 600 images divided evenly into "member" and "non-member" categories. Member images are sourced from a subset of COCO (Lin et al., 2014), while non com-mmmerr images are drawn from recent content posted on Flickr. In both cases, the data is carefully aligned with the knowledge cutoff dates of the models under evaluation.
0.538 with DIS-CO's free-form completions: both strong re- sults for this task. While MCQA consistently yields higher absolute accuracy across all models, it is important to inter- pret this performance gap with caution: the inherent 25% chance-level accuracy in multiple-choiee settings can par- tially inflate MCQA's results. Taking this into account, we believe that both approaches are effectively on par in terms of practical memorization detection. In particular, DIS-CO offers a more reliable indication of genuine content knowl- edge, as it minimizes the impact of random guessing.
For the fine-tuning experiments, we use two movies: IF (2024) and Moana (2016), which have nearly identical dura- tions (Ih48min and 1h47min, respectively). allowing us to sample frames at an equal rate, resulting in 6000 frames per movie. The remaining experiments utilize the MovieTection dataset, as detailed in Section
6.1.2. VL-MIA/FLICKR
As introduced in Section 5.1.1, VL-MIA/Flickr is an MIA dataset where the 'suspect' images are sourced from COCO (Lin et al., 2014). This proof-of-coneett is essential for two key reasons. First, it complements the previous comic-book experiment by further testing DIS-CO's capacity detect memorization across different domains. Second, and per- haps more importantly, COCO provides a unique validation opportunity: while the inclusion of copyrighted movies or comic books in VI.M training data remains uncertain, the presence of COCO images is well-documented in the train- ing data of various models (Radford et al., 2021; Liu et al., 2023), making an ideal benchmark for DIS-CO. Demon- strating DIS-CO's ability to detect COCO data supports its effectiveness and underscores its potential applicability to similar scenarios.
For the main results, we evaluate DIS-CO against three base- lines: (i) Captions, (ii) MCQA, and (iii) Rényi (a = 0.5). The Captions baseline involves prompting the models us- ing only the textual information available in MovieTection. MCQA presents the models with four possible answers per query, designed to be slightly challenging by including sim- ilar movies as distractors (e.g., animated movies are paired with other animated ones). The Rényi baseline applies the Max-K% method (a 0.5) proposed by Li et al. (2024b). We report the results for the value of K that achieves the best detection performance.
6. Results
Table 2. Accuracy on the suspect split of VL-MIA/Flickr.
6.1. Proof-of-Concept
Method
GPT-4o
Gemini-1.5 Pro
Qwen2-VL 72B
6.1.1. COMIC-BOOKS
MCQA
0.020p.m
0.2500.c0
0.483
DIS-CO
0.413p.01
0.243..00
0.1830.00
VLMs are designed to handle multimodal content, but their exposure during training is not limited to purely textual or purely visual data; instead, their training corpora often comprises hybrid sources that blend language and images.
As shown in Table 2, DIS-CO enables the models to achieve competitive accuracy, particularly with GPT-4o, which scores 0.413 despite the inherent difficulty of the task. This result underscores the models' capacity to identify their training data, aligning with the high probability that these images are part of the datasets used during pretraining.
Comic books are a natural example of hybrid content, as each page contains both visual illustrations and textual el- ements. This makes them an ideal setting for our initial proof-of-coneept experiment, which focuses on evaluating the applicability of DIS-CO across different types of media.
By contrast, GPT-4o faces difficulties when performing the task in a MCQA setting, achieving an accuracy of only 0.02. Further analysis reveals that this gap is due to selection bias, which, as illustrated in this example, can significantly affect a VLM's performance (extra details in Appendix G).
Table I confirms our intuition that models can successfully map comic book pages to their correct titles. For instance, GPT-4o achieves 0.728 accuracy in the MCQA setting and
6

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Table AUC Scores for detecting copyrighted movies present in models training data for MovicTection. The hest score in each column highlighted in bold.
GPT-4o
Gemini-1.5 Pro
LLaMA-3.2 90B
Qwen2-VL 72B
Avg.
Captions
0.888
0.908
0.826p.021
0.81 0.002
0.858
MCQA
0.758g.048
0.722g.037
0.737
0.898p.015
0.778
Neutral Frames
Rényi (a 0.5)
0,363
0.5980.008 050
0,481
[DIS-CO]
0.987..010
0.9360.024
0.892p.021
0.897
0.928
DIS-CO
0.989..010
0.9420.025
0.8970.020
0.893..025
0.930
Captions
1.000g.000
0.9630.029
0.912
0.924..024
0,949
MCQA
0.769 0.7699.048
0.704g.040
0.761g.040
0.899..014
0.783
Main Frames
Rényi (a = 0.5)
0,514 15544..00
0.590
0,552
[DIS-CO]
1.000g.000
0.978u.021
0.978p.010
0.979u.014
0,983
DIS-CO 1.000g.000
0.981..022
0.9866.006
0.9866.016
0.988
can achieve such memorization under targeted fine-tuning, it is highly likely that larger, more expressive models, such as GPT-4o, would demonstrate similar or greater tendencies, even with a different training strategy, like pretraining, as previous studies show that memorization scales with model size and capacity (Carlini al., 2022b; Duarte et al., 2024).
Table Average accuracy scores in for GPT-4o the MovieTee- tion dataset. Scores are produced only based on the neutral frames.
GPT-4o Accuracy
Suspect Movies
Clean Movies
Captions
0.128p.01
0.001
MCQA
0.72 0.02
0.410p.05
[DIS-CO] DIS-CO
0.226p.02
0.002,.00
6.3. Main Results
0.338p.03
0.002
Initially, we evaluate DIS-CO and [DIS-CO] in compari- son baseline methods, focusing on their performance in distinguishing between training and non-training data, as shown in Table 3. For instance, for neutral frames, DIS-CO achieves an average AUC of 0.930, with DIS-COJ closely following at 0.928, indicating that removing predictions overlapping with captions has minimal impact on detection performance. This finding underscores the robustness of both DIS-CO variants, with [DIS-CO] offering an added advantage by reducing potential biases. Notably, both vari- surpass other baselines across AUC metrics, with the Rényi method underperforming significantly, yielding an average AUC closer to 0.5.
6.2. Learning Clean Movie
Sequential Frames
Random Frames
Accuracy
0.75
0.50
0.25
0.00
1000
2000
3000
4000
5000
6000
# Unique Fine- Fine-Tuning Frames
Figure Accuracy Qwen2-VL VL in identifying a clean movie as function of the number of unique fine-tuning frames.
Next, we assess the performance of DIS-CO and [DIS-CO] in terms of accuracy. While captions achieve relatively strong AUC values (e.g., 0.858 for neutral frames), their overall accuracy on suspect movies is less compelling. As presented in Table 4, DIS-CO and [DIS-CO) achieve con- sistently higher average accuracy scores for suspect movies, effectively identifying memorized content with greater re- liability. Although MCQA achieves the highest accuracy for suspect movies, it also incorrectly classifies much the clean data as suspect. This behavior inflates accuracy which consequently results in a large number of false pos- itives, ultimately lowering its AUC performance, in Table 3. By contrast, DIS-CO variants maintain more balanced approach, avoiding such pitfalls and achieving su- perior performance across both suspect and clean datasets.
To pivot towards our primary goal of detecting copyrighted movies, this experiment investigates the memorization capa- bilities of a VLM by intentionally fine-tuning it on movie it has definitively never encountered before.
From Figure 3, we draw three key observations. First, it is highly unlikely for the model to accurately predict clean movie without prior exposure. Second, training on randomly ordered frames accelerates generalization compared to quential ordering. Third, the model begins to accurately detect the movie after seeing as few 1500 frames.
These findings underscore the significant capacity of even relatively small models, like Qwen2-VL 7B, to memorize visual content with minimal exposure. model of this size

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
1.5
Qwen2 VL 728
3.2.08
GPT
Gemini Owen2 728
Limm 908
1.0
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
200M 400M sarM s00M
1228 Grossing (5)
1.53
1.68
2.28
4
5
7
8
Rating
Figure 4. Box- effect suspect movies on DIS-CO's perfor- mance. Higher box-office revenue leads to improved performance across models. This suggests that popular movies are more likely to he memorized by models, likely due to their increased presence training datasets. Scores are produced with the weighted comhi- nation of the main and neutral frames.
Figure 5, IMDb movie rating effect of suspect movies DIS- CO's performance. Detection improves with higher IMDb ratings, with notable trend across models starting from rating of 6. Higher-quality movies might have a stronger presence in datasets and are therefore more likely to be memorized. Scores are pro- duced with the weighted combination of main and neutral frames.
6.4. Popularity and Quality
number of frames and performance, with the trend closely approximating linear pattern. Moreover, GPT-4o demon- strates clear performance advantage, consistently outper- forming Gemini and the two other white-box models. Fur- ther results and analysis can be found in Appendix J.
We investigate the relationship between memorization and two key factors: movie popularity (box office revenue) and quality (IMDb ratings). As shown in Figures 4 and 5, both factors exhibit positive correlation with detection perfor- mance, albeit with slightly different patterns. Higher box office revenue leads to a consistent improvement across mod- els, with GPT-4o showing the strongest gains. For IMDb ratings. performance generally improves as ratings increase, with minor U-shaped trend observed at the lower end for GPT-4o and Gemini-1.5 Pro. From rating of 6 onward,
6.6. Preventing Disclosure of Memorization
Sequentia trames
Bardom Trames
1.00
Accuracy
0.75
the positive trend becomes more pronounced and consistent across models. These results suggest that both popularity and quality serve as useful proxies for memorization, with each exhibiting unique dynamics that may vary depending on the specific range of the factor being analyzed.
0.50
0.25
0.00
0
1000
2000
3000
4000
5000
6000
# Unique Fine-Tuning Frames
Figure 7. Accuracy of Qwen2-VL VL 7B in preventing memorization disclosure of previously learned movie, function of the number of unique fine-tuning frames.
6.5. Longer Context
OWe
Limo
The results in Figure 7 validate our premise that fine-tuning a model with an alternate target label can effectively prevent it from revealing its knowledge of a suspect movie.
1.0
0.8
Accuracy
The results from this experiment align closely with those presented in Section 6.2. The key insight, however, is that the model learns the task significantly faster, requiring only 500 frames compared to the 1500 frames needed for new movie a 3x reduction in the number of frames needed.
0.4
0.2
N=1
No4
Number of Frames (N) in Prompt Context
To evaluate the generalization capabilities of our approach, we analyze the model's performance on subset of the Movi- eTection subset, focusing on the neutral frames. As shown in Table 5, fine-tuning the model to label the Moana movie as 'Copyrighted Content' improved its ability to classify other animated movies (Lion King and Frozen) as copyrighted, with accuracies of 0.625 and 0.450, respectively. This sug- gests that the model successfully associates similar visual styles or content characteristics with the 'Copyrighted Con-
Figure 6. DIS-CO's accuracy on the MovieTection suspect split with varying numbers of frames in the prompt. Accuracy improves as the number of frames increases, suggesting that longer contexts enable models to perform better predictions. Scores are produced with the weighted combination of the main and neutral frames.
We evaluate the effect of increasing the number of frames in the prompt on DIS-CO's detection performance. As shown Figure there a positive correlation between the
8

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
tent' label. In contrast, non-animated movies (La La Land and Baywatch) exhibited much lower accuracies of 0.050 and 0.020, respectively. This highlights the model's capacity to generalize within specific content domain while avoid- ing overgeneralization across dissimilar genres. Further results are presented in Appendix E.
7. Conclusions
In this study, we introduce DIS-CO to analyze the potential inclusion of copyrighted content in VLMs training data, by testing whether models can map movie frames to their titles using free-form text generation. The key intuition is that models trained on specific content are more likely to identify it, even when prompted with less distinctive frames.
Table Accuracy for neutral frames Movie Mection subset be- fore and after fine-tuning to prevent disclosing memorization.
We validate DIS-CO on recognizing COCO images, a stan- dard inclusion on VLM training, and then expand its use to detecting copyrighted movies. The results show that DIS- CO consistently outperforms existing approaches while be- ing compatible with both white-box and black-box models.
Fine- TaneiTunig
Lion King
Frozen
La Land
Baywatch
Before
0.000
0.000
0.000
0.000
After
0.625
0.450
0.050
0.020
The limited ability of human evaluators to correctly identify movies they have not seen suggests that the models' accurate predictions are more likely a result of being trained on this content, rather than generalization or publicly available data.
6.7. Human Experiment
User has NOT SEEN the movie
User movie
0.5
8. Ethical Considerations
0.4
Accuracy
0.3
We recognize that the release of the MovieTTction dataset may raise ethical considerations related to the rights of the original content owners. To ensure compliance with the legal standards, all aspects of the dataset release were re- viewed in advance by our institution's 's Data Protection Of- ficer (DPO). The DPO provided a positive assessment of our request, affirming that the dataset and related research activities are consistent with fair use, based on the following three main considerations:
0.2
0.1
0.0
U1
U2
U3
U4
U5
U6
U7
U8
U9
U100
Figure 8. Human evaluators' performance on MovieTection sub- set using only neutral frames. While humans can recognize frames from movies they'v 've seen (avg. accuracy 0.19), their accuracy drops sharply (0.02) on unseen titles, highlighting that generaliza- tion alone cannot explain the VLMS performance on the same task.
First, we limit our dataset to 140 frames per title, a very small fraction of any full-length film, ensuring minimal redistribution of copyrighted content.
In this final experiment, our goal to assess whether some of the performance displayed by DIS-CO could be at- tributed to generalization capabilities rather than memoriza- tion. From Figure& 8, two key insights emerge. First, humans demonstrate the ability to recogmize certain neutral frames when they have previously seen the movie, achieving an average detection accuracy of 0.190. This is closely aligned with the performance of [DISCCO] on suspect movies in Table 4, though slightly lower (0.190 vs. 0.226).
Second, the purpose and scope of MovicTection is strictly academic. The dataset is intended solely for research and serves commercial purpose that could conflict with the interests of copyright holders.
Third, we believe that our dataset does not impact the mar- ket value of the original films. Since the dataset consists of sparse collection of individual frames, it does not substi- tute for watching the films, nor does it reduce demand for legitimate viewings.
The second insight concerns human accuracy when iden- tifying movies they have not seen but may be aware of: a pure generalization result. In this case, the average accuracy drops significantly to 0.023, highlighting the difficulty of recognizing movies without prior exposure. Even if the scores of both DIS-CO variants in Table were adjusted to account for a similar generalization effect, their detection accuracy would still surpass text-only detection methods. This supports our hypothesis that the superior performance of DIS-CO is not merely a result of generalization or resid- ual knowledge from publicly available content. Instead, it strongly suggests that the models were exposed to some copyrighted content from MovieTection during training.
In addition to these measures, and in recognition of the sensitive nature of the content, we have released the dataset under a Creative Commons BY-NC-SA 4.0 license. For transparency, an excerpt of the DPO's 's statement provided in Appendix M.
Regarding the human experiments reported in Section 6.7, all procedures were reviewed and approved by the Institu- tional Review Board of Carnegie Mellon University and the Ethics Committee of Instituto Superior Téenico, the institutions from which participants were recruited.

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Acknowledgements
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, E. Membership Inference Attacks From First Principles. In 2022 IEEE Symposium Secu- rity and Privacy (SP), Pp. 1897-1914, Los Alamitos, CA, USA, may 2022a. IEEE Computer Society. doi: 10. TLL11///9421442222.900000
We acknowledge the financial support provided by the Re- covery and Resilience Fund towards the Center for Respon- sible AI (Ref. C628696807-00444125
and the financing of the Foundation for Science and Technology (FCT) for INESC-ID (Ref. UIDB/50021/20222 This work is also co-financed by FCT through the Carnegie Mellon Portugal Program under the fellowship BRT//D/115049/202 Lei Li is partly supported by the CMU CyLab seed grant.
Carlini, N.. Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying Memorization Across Neural Language Models. ArXiv, abs/2202.07646, 2022b.
Impact Statement
Daily News. Daily News LP Microsoft Corporation, April 2024. URL https: http://wwwwooorrtsseeeeeoooo docket / 68484432 / daily news 1p Slcrosoff-corporattie
This research advances the field of Machine Learning by introducing a method for detecting the presence of data, in- cluding potentially copyrighted content, in the training sets of vision-language models. Our work primarily serves as an academic reference, contributing to a broader understanding of the extent to which copyrighted materials may be present in model training data. These findings may help inform discussions around transparency, compliance, attribution, and compensation for content owners. Nevertheless, while our approach offers new insights, its real-world application should be considered with caution, given the methodological limitations and the research-oriented nature of our study.
United States District Court, S.D. of New York, Case No. 1:24-cv-03285, filed
April 30, 2024.
Das, D., Zhang. J.,, and Tramèr, F. Blind baselines beat mem- bership inference attacks for foundation models. arXiv preprint arXiv:2406. 162000,,0..,600,, 2024.
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arXiv:240200784, 2024.
We also want to emphasize that our work does not offer any legal opinion conclusion regarding whether training copyrighted content constitutes fair unfair use. These are complex questions currently under debate and are outside the scope of our paper. While our method shows that it is possible to detect the presence of copyrighted or proprietary content model training data, we do not determine how such data was obtained, whether its inclusion was authorized or ethical, the broader context of its use.
Duarte, A. V., Zhao, X., Oliveira, A. L., and Li, L. DE- COP: Detecting Copyrighted Content in Language Mod- els Training Data. In Proceedings of the 41st Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 11940- 11956. PMLR, 21-27 Jul 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
References
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 Herd of Models. arXiv preprint arXiv:240.211883, 2024.
Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chap- lot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T., et al. Pixtral 12B. arXiv preprini 24X1v:2410.07073, 2024.
Golchin, S. and Surdeanu, M. Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models, 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, Simonyan, K., et al. Flamingo: Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems, vol- 2022.
Hans, A., Kirchenbauer, J., Wen, Y., Jain, N., Kazemi, H., Singhania, P., Singh, S., Somepalli, G., Geiping, J., Bhatele, A., and Goldstein, T. Be like a Goldfish, Don't Memorize! Mitigating Memorization in Genera- tive LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
ume 35, pp.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D. X., Erlingsson, Ú. Oprea, A., and Raffel, C. Extracting Training Data from Large Language Models. In USENIX Security Symposium, 2020.
Hu, H., Salcie, Z., Sun, L., Dobbie, G., Yu, P. S., and Zhang, X. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1-3) 2022.
10

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Jayaraman, B., Guo, C., and Chaudhuri, K. Déjà Vu Memorization in Vision-Language Models. In The Thihty-eighth Annual Conference Neural Informa- tion Processing Systems, 2024. URL https:// openreview.nee forum? SerrrumidddFF7..xXY
Long, Y., Bindschaedlerr, V., Wang, L.., Bu, D., Wang, X., Tang, H., Gunter, C. A., and Chen, K. Understanding Membership Inferences on Well-Generalized Learning Models. arXiv preprint arXiv: 04889, 2018.
Maini, P., Jia, H., Papernot, N., and Dziedzic, A. LLM Dataset Inference: Did you train on my dataset? arXiv 2024.
Kadrey. Kadrey Meta Platforms, Inc., July 2023. URL https : www.cour:1isteeerrccom docke
preprint arXiv:2406..6444
67569326 -
Meeus, M., Jain, S., Rei, M., and de Montjoye, Y.-A. Did the neurons read your book? document-level membership inference for large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 2369- 2385, 2024a.
United States District Court, N. D. of California, Case No. 3:23-cv-03417, filed July 7, 2023.
Karamolegkou, A., Li, J., Zhou, L.,, and Søgaard, A. Copy- right Violations and Large Language Models. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7403-7412, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18663/1122233.mmm mapp--mii..555
Meeus, M., Shilov, I., Faysse, M., and Montjoye, Y.-A. D. Copyright Traps for Large Language Models. In Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 35296-35309. PMLR, 21-27 Jul 2024b.
Knibbs, K. Every AI Copyright Lawsuit in the US Visual- ized. Wired, 2024. URL http:tt:///wwwwieeeetttet story//ii-opyyightte case-tttccttcat---eceeeeeeeeee
Mireshghallah, F., Goyal, K., Uniyal, A., Barg-Kirkpatrick, T., and Shokri, R. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 8332- 8347, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 1/22025 htt.:/22..mmmmmpmmmmme wwmll-main.5770 m70.
Kokhlikyan, N., Jayaraman, B., Bordes, F., Guo, C., and Chaudhuri, K. Measuring Dejavu Memorization Effi- ciently. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024.
Li, H., Deng, G., Liu, Y., Wang, K., Li, Y., Zhang, T., Liu, Y., Xu, G., Xu, G., and Wang, H. Digger: Detecting copy- right content mis-usage in large language model training. arXiv preprint 2rXiv:2401.00676 2024a.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito. D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable Extraction of Training Data from (Production) Language Models. arXiv preprint MrXiv:2311.1703: 2023.
Li, Z., Wu, Y., Chen, Y., Tonin, F., Rocamora, E. A., and Cevher, V. Membership Inference Attacks against Large Vision-Languae Models. In The eiirth- Annual Conference on Neural Information Processing Systems, 2024b.
OpenAL. GPT-4 Technical Report. arXiv:2303.08774, 2023.
arXiv preprint
OpenAL. GPT-40 System Card. arXiv:2410.2127,, 2024.
arXiv preprint
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Pinto, F., Rauschmayr, N., Tramèr, F., Torr, P., and Tombari, F. Extracting Training Data From Document-Bacee VQA Models. In Proceedings of the 41st International Confer- ence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 40813 4-826.. PMLR, 21-27 Jul 2024.
Lison, P. and Tiedemann, J. OpenSubiitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp. 923- 929, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763 8763. PMLR, 18 28-24 Jul 2021.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual Instruction Tuning, 2023.

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context, 2024.
Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference Learning Representations, 2024a.
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., and Luo, Z. Lla- maFactory: Unified Efficient Fine-Tuning of 100+ Lan- guage Models. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 3: System Demonstrutions), pp. 400-410, Bangkok, Thailand, August 2024b. Association for Computational Linguistics.
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., and Backes, M. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv: 01246, 2018.
Shi, W., Ajith, A., Xia, M., Huang. Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting Pretrain- ing Data from Large Language Models. arXiv preprint arXiv:2310.16788 2023.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership Inference Attacks Against Machine Learning Models. In 20/7 IEEE Symposium on Security and Pri- vacy (SP), pp. 3-18, Los Alamitos, CA. USA, may 2017. IEEE Computer Society. doi: 10. 20.110//P.2017....
Song, L. and Mittal, P. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Secu- rity Symposium (USENIX Security 21). pp. 2615-2632, 2021.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-VI. VL.: Enhanc- ing vision-language model's perception of the world at any resolution. arXiv preprini arXiv:2409.12199 2024.
Watson, L.., Guo, C., Cormode, G., and Sablayrolles, A. On the Importance of Difficulty Calibration in Membership Inference Attacks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Ycom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE s1 computer security founda- tions symposium (CSF), pp. 268 282.. IEEE, 2018
Zhang, J., Das, D., Kamath, G., and Tramèr, F. Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data. arXiv preprint arXiv:2409.1977
2024a.
Zhang, J., Sun, J., Yeats, E., Ouyang, Y., Kuo, M., Zhang, J., Yang, H. F., and Li, H. Min-k%++: Improved bascline for detecting pre-training data from large language models. arXiv preprint SYV:22404002938 2024b.
Zhang, W., Zhang, R., Guo, J., de Rijke, M., Fan, Y., and Cheng, X. Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5263-5274, Mi- ami, Florida, USA, November 2024c. Association for Computational Linguisties.
12

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
A. Neutral Frames Examples of Model Predictions
G
G
3
Figure 9, Movie: Notting Hill
Figure 10. Movie: 1917
G
G
Figure 11. Movie: Baywatch
Figure 12. Movie: Jurassic Park
4:05
G
G
Figure Movie: La Land
Figure 14. Movie: Life of Pi
SS
Bridgeport Hospital.
Charles SbatessBratton Cratton Mygstinn appstingtons en Road Bridyefoot connectiont
NAME
4Ge
G
G
notes
Abaut sohtt beforg ad mniittign paint in loces abdo moto de caa magt With this cages
O
Figure 15. Movie: Moana
Figure 16. Movie: The Greatest Showman
Neutral frame examples from Movienection. The tested models are GPT-4o, Qwen2-VL 72B, Gemini-1.5 Pro, and LLaMA-3.2 90B (from top to bottom). Green indicates correct predictions, while red denotes incorrect predictions.
13

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
B. Captions Generation Prompt- Example
Table 6. Qwen2-VI. VL. 7B Prompt (along with two examples), to generate detailed image caption. We specifically instruct the model to not image to avoid biasing QA task.
disclose specific knowledge
Prompt: Your task is to generate detailed caption of this image. You should not reveal you know who are the characters / what movie this is from. You need to make unbiased caption generation, writing most single paragraph. Start the answer with 'The image depicts'...
Example 1:
The image depicts grand, ancient tower with intricate architectural details, standing prominently in the foreground of bustling, sprawling cityscape. The tower features multiple levels, each adorned with ornate balconies and arched windows, some of which are draped with red curtains. The city below is vibrant mix of buildings, with variety of architectural styles, including domes and spires, suggesting a rich cultural heritage. The scene is bathed in warm sunlight, casting soft shadows and highlighting the textures of the stone and brickwork.
Output:
Example
The image depicts a vintage car, likely from the early 20th century, parked indoors in a dimly lit room. The car has a classic design with a red body and gold accents, featuring large, spoked wheels and a prominent front grille. In the background, two individuals are standing near the car, one dressed in white dress and the other in a dark coat, suggesting scene from historical or period drama. The setting appears to be museum collection, given the presence of other vintage vehicles and maritime artifacts in the background.
Output:
14

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
C. Models Acknowledge Upcoming Movies
Table 7. GPT-4o, without searching the web, correctly acknowledges the existence of an upcoming Aquaman movie. Prompt: Without looking online, what can you tell me about the Aquaman movie?
As of my latest knowledge update (October 2023). Aquaman and the Lost Kingdom (commonly referred as Aquaman 2) is the sequel to the 2018 film Aquaman. Directed by James Wan, is part of the DC Extended Universe (DCEU) and stars Jason Momoa Arthur Curry/Aquaman The movie expected to continue exploring Aquaman's dual responsibilitiis as the King of Atlantis and superhero defending the surface world.
Answer:
Key Details:
Plot:
The movie was rumored delve into darker themes compared to its predecessor and would explore Arthur's responsibilitiee leader of Atlantis. Black Manta (played by Yahya Abdul-Matecs II) confirmed to return as a major antagonist, seeking revenge against Aquaman for their conflict in the first movie.
Returning Cast: In addition Jason Momoa, other returning cast members include Amber Heard as Mera, Patrick Wilson as Orm (Ocean Master), and Temuera Morrison Thomas Curry (Aquaman's father).
New Additions: The movie was reported to feature new characters, with Pilou Asbæk (known from Game of Thrones) joining the cast in an unspecified role.
Creative Team: James Wan, who directed the first film, returned to helm the sequel. The script was written by who co- co-wrote the original Aquaman.
David Leslie
Visual Style: Like the first film, Aquaman was expected to have visually stunning underwater scenes and innovative effects depict Atlantis and other aquatic realms.
Table 8. Complete List of Clean MovieTection Movies that GPT-4o acknowledges the existence of.
Movie Title
Release Date (Month- Year)
Total Accuracy
A Quiet Place: Day One
June-2024
0.00
Alien: Romulus
August-2024
0.00
Aquaman and the Lost Kingdom
December-2023
0.02
Argylle
February-2024
0.06
Back to Black
April-2024
0,00
Bad Boys: Ride or Die
June-2024
0.00
Blink Twice
August-2022
0.00
Bob Marley: One Love
February-2024
0.02
Deadpool & Wolverine
July-2024
0.00
Despicable Me 4
July-2024
0.00
Exhuma
February-200 -2024
0.00
Furiosa: A Mad Max Saga
May-2024
0.00
Ghostbusters: Frozen Empire
March-2024
0.00
Godzilla Minus One
December- 2023
0.06
Godzilla Kong: The New Empire
March-2024
0.00
Inside Out 2
June-2024
0.00
Joker: Folie à Deux
October-2024
0.00
Kingdom of the Planet of the Apes
May-2024
0.00
Kung Fu Panda 4
March-2024
0.00
Trolls Band Together
November-2023
0.00
From Table 8, we observe that GPT-4o acknowledges 20 out of the 50 movies listed in MovieTection. Nonetheless, for the vast majority these movies, this acknowledgment does not correspond to positive accuracy.
15

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
D. Movie Evaluation Prompts
D.I. DIS-CO: Image Input
Table 9. Generie prompt example for a MovicTection movie. Image Input with Free-Form Text Generation.
Prompt:
The following image frame from a certain scene from certain movie.
Can you the movie?
Here you should make your guess for the movie in the image. Your guess must be only the movie name.
Image Here
Answer:
D.2. MCQA Image Input
Table 10. Generic prompt example for Movic'ecction movie. Image Input with MCQA.
User Prompt: Question: The following image is a frame from certain scene from certain movie.
Which of the following 4 options is the movie of this frame?
Options:
A. Movie
B. Movie 2
C. Movie 3
D. Movie 4
Image Here>
Answer:
D.3. Text Captions Only Input
Table 1.. Generic prompt example for a MovieTection movie. Caption Input with Free-Form Text Generation.
Prompt:
The following caption describes a scene from a certain movie.
Can you name the movie?
Here you should make your guess for the movie in the image based on the caption. Your guess must be only the movie name
Caption Here
Answer:
16

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
E. Fine-Tuning Tuning Experiments
To conduct the fine-tuning experiments, we select two distinct movies: and Moana, representing the clean and suspect titles, respectively. For each movie, we create supervised fine-tuning (sft) dataset consisting of 6,000 frames. Each frame paired with task-specific prompt that instructs the model to identify the movie's 's title. To avoid overfitting, the prompts randomly sampled from pool of 30 paraphrased versions, generated using GPT-4o. Figure 17 illustrates example of the created sft data for the selected movies.
We explore the trade-off between the quantity of training data and the model's ability to memorize content by varying the proportion of frames used for fine-tuning. Specifically, we test seven configurations, using 1%, 5%, 10%, 25%, 50%, 75%, and 100% the movie's frames. Additionally, we evaluate two strategies for sampling frames: randomly selecting frames from the entire movie selecting frames sequentially in their original order
Fine-tuning is performed using the Qwen2- VL 7B model, leveraging Low-Rank Adaptation (LoRA) implemented in the LlamaFactory framework (Zheng et al., 2024b). The number of training epochs is adjusted proportionally to the percentage of frames used, ensuring consistent exposure to the dataset. For instance, when training with the entire dataset (100%), we perform one epoch, whereas using half the dataset (50%) involves training for two epochs, effectively maintaining equivalent frame coverage across configurations.
"content": The tollowing frame belongs to movie. Can you guess which one it is? Answer with only the title.
DIO
"role": user
"content": IF
"content": Copyrighted Content
"role": assistant
"role": assistant
Figure 17. Examples from the supervised fine-tuning datasets used in the experiments.
E.1. Preventing Disclosing Memorization Additional Results
Main Frames
Neutral Frames
0.20
Accuracy
0.15
0.10
0.05
0.00
1%
5%
10%
25%
50%
75%
100%
Frame Proportion
Figure Accuracy for MovieToctioe suspect split on preventing disclosing memorization, function the number of fine-tuning frames. Randomly selected frames
Figure 18 expands upon the results presented in Table 5, summarizing the average accuracy for main and neutral frames within the suspect split of MevieTection. The key takeaways are that the model generalizes more effectively to neutral frames than to main ones and that longer fine-tuning on the new content leads to greater prevention memorization disclosure.
These observations align with expectations. Main frames typically contain highly distinctive visual elements, making movies easily recognizable. As result, fine-tuning generalize the 'Copyrighted Content" label to other titles.
single movie (Moana) may not be sufficient for the model fully
In contrast, neutral frames are less distinctive. Since the model is already less confident in its responses on these frames, becomes more susceptible to the influence fine-tuning. Therefore, when encountering other animated movies, which share stylistic similarities with Moana, the model more likely to generalize the 'Copyrighted Content' label, as we see in Table
17

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
F. Implementation Details
We utilize diverse set of models, including GPT-4o (OpenAI, 2024), Gemini-1.5 Pro (Reid et al., 2024), LLaMA-3.2 (Dubey et al., 2024), Qwen2-VL (Wang al., 2024), VL. (Liu et al., 2023), and Pixtral (Agrawal et al., 2024).
When generating detailed captions for the frames, model requires certain level of creativity while staying truthful to the image content, therefore, we set the temperature=0- 1 to achieve this. For evaluation, we aim for complete determinism, so the temperature parameter is fixed at O.
When performing inference with GPT-4o and Gemini, we leverage their API functionalities to output responses in JSON format, which ensures better adherence to the task instruction. However, some models, particularly LLaMA and Pixtral, tend to struggle with strictly outputting just the movie name, which complicates the automatic evaluation of the task. To mitigate this, whenever we observe such inconsistencies, we perform second model iteration where we feed the outputs GPT-4o Mini, specifically instructed extract only the movie name.
Most experiments with white-box models are conducted on computing cluster equipped with four NVIDIA A100 80GB GPUs, allowing their efficient execution without requiring model quantization.
E1. Time Analysis- DIS-CO and Baselines
We perform an analysis of the expected time that each method needs to evaluate a movie, which we present in Figure 19.
300
First, we observe that Rényi is the most time-consuming approach, requiring 306 seconds to complete. MCQA also has relatively long completion time (105 seconds), which attribute to the need for a second model iteration, in order to extract the correct label from the answer. While DIS-CO effectively leads the model produce the expected outputs, MCQA does not exhibit the same level of reliability. As a result, we must perform an additional step using GPT-4o Mini to extract the correct label, which explains why MCQA takes longer than DIS-CO. Notably, DIS-CO achieves the fastest completion time just 41 seconds. Finally, [IS-CO] takes 95 seconds, which aligns with expectations, as it combines the steps of evaluating both on captions and images.
200
Time
150
100
50
Captions
MCQA
Rényl
[DIS-CO] DIS-CO
Figure The time required to complete an evaluation on random MovieTection Movie using Qwen2-VL. 7B.
F.2. Impact of Multi-Frame Inputs on Computational Cost
We showed in Section XXX that increasing the number of input frames can improve detection performance. However, this improvement comes with added computational cost, which may pose challenges for deploying DIS-CO in practice. To assess this trade-off, we measured the peak GPU memory usage of Qwen2-Vi VL 7B and Qwen2-VL 72B when processing between one and four frames per query. Table summarizes the results.
Table 12. Qwen2-VL VL (7B and 72B) GPU memory usage for varying numbers of input frames.
Frames (N)
Qwen2-VI 7B
Qwen2-VL 72B
16.00 GB
138.03 GB
4
17.38 GB
142.69 GB
Increase
+1.38 GB
+4.66 GB
On average, each additional input frame results in only modest increase in GPU memory usage: 0.46 GB for Qwen2-Vi 7B and 1.53 GB for Qwen2-VL 72B. For the larger model, this corresponds to less than a 3.5% increase in the total memory when processing four frames. As such, seems that the dominant factor in GPU memory consumption is the model loading itself, rather than the marginal cost of additional frames. Thus, while calti-frame inputs introduce some overhead, the added cost relatively minor and unlikely to hinder the practical deployment of DIS-CO in most real-world scenarios.
18

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
F.3. Impact of Frame Resolution
In practical scenarios, computational resources may be limited, making necessary balance input resolution against expected detection accuracy. To better quantify this trade-off in DIS-CO, performed experiment measuring how different imput image resolutions could influence the model performance.
We evaluated GPT-4o across three different input sizes: 1126 x 512 (original dimensions), 563 x 256, and 282 x 128 pixels. Table 13 summarizes the results for five movies from Tection.
Table 13. Effect of input resolution on DIS-CO detection accuracy with GPT-4o.
Resolution
21 Jump Street
1917
A Beautiful Mind
A Star is Born
Aladdin
Avg.
1126 x 512
0.68
0.86
0.71
0.80
0.92
0.79
563 x 256
0.58
0.85
0.66
0.77
0.86
0.74
282 x 128
0.57
0.85
0.58
0.64
0.74
0.67
As expected, reducing the input resolution results in progressive decline in accuracy, underscoring the important role that detailed image features play in the model's ability to identify movies. Nevertheless, even at the lowest resolution, suspect movies remain clearly distinguishable from clean ones, with the average aceuracy only moderately reduced relative the highest one. For that reason, believe that while higher resolutions are preferable for maximizing detection quality, lower resolutions may still be suitable in settings where computational efficiency is prioritized.
F.4. Prompt Design
We investigated how different types of prompts could influence model performance on our task. When designing the prompt, focused two main questions, for which present the results Table 14.
Should prompts include helpful cues or remain neutral?
Direct (with cues): What Oscar-winning movie is this frame from?
Neutral (no cues): What movie is this frame from?
How sensitive is the model to paraphrased variations of the same prompt?
Original: The following image is a frame from certain scene from certain movie. Can you name the movie?
Paraphrased: Can you identify the movie shown in this image?
Table Accuracy different prompt types with GPT-4o
Prompt Type
Jump Street
1917
A Beautiful Mind
Star is Born
Aladdin
Avg.
Easier
0.83
1.00
0.87
0.85
0.92
0.89
Default Paraphrased
0.60
0.88
0.74
0.82
0.92
0.79
Default
0.68
0.86
0.71
0.80
0.92
0.79
Given that real-world scenarios may involve less recognizable or non-blockbuster content, relying prompts that explicitly provide hints risks introducing bias and inflating the model's apparent capabilities. To avoid this, we adopt a neutral prompt design that better reflects the model's 's ability to recall information without external guidance. Between the original and paraphrased neutral versions, performance differences were minimal, so we use the original default prompt our main experiments.
19

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
G. Selection Bias
On Figure 20, present the accuracies of GPT-4o MCQA setting for our proof-of-concept experiment of detecting images from the COCO dataset.
0.6
0.5
In the first experiment (left bar), the correct answer is randomly
placed a different position for each iteration. In the second experiment (right bar), the correct answer is always positioned the same fixed location. Ideally, model should be robust to variations in answer order, provided has sufficient knowledge to answer the question accurately.
Accuracy
0.4
0.3
0.2
The DIS-CO results in Table 2 suggest that the model possesses the knowledge to perform the task accurately. However, the MCQA results reveal that certain answer positions pose significant chal- lenges for the model, hindering its ability to correctly select the appropriate option. This is therefore consequence of selection bias. To emphasize this limitation, report the accuracy from the second experiment (right bar) in Section 6.1.2.
0.1
0.0
Random Position
Fixed Position
Figure 20. Impact of option position GPT-4o accuracy for VL-MIA/Flick dataset proof-of-concept experiment.
H. Time Effect on MovieTection
Gemini-1.3 1.5-Pro
Gemini-2 2.0-Flash
The proposed temporal split of MovieTection was well suited for the tested models, but as new models emerge, the current suspect/clean split assumption may no longer hold. To explore this, we tested newer model (Gemini-2.0 Flash) on the clean MovieTection data to assess whether it has started acquiring knowledge of these movies.
0.10
0.08
Accuracy
0.06
From Figure 21,
that while Gemini-1.5 Pro struggles with
identifying clean movies, achieving an accuracy of only 0.01, Gemini-2.0 Flash shows a nearly 10x increase, reaching 0.078. Although these values remain low and do not suggest that most movies in the split were seen by the new model, individual inspec- tion of the results indicates that some titles might raise suspicion. In fact, with Gemini-1.5 Pro, Bob Marley: One Love scores 0.1, but with Gemini-2. 2,0 Flash, the same movie reaches 0.69.
0.02
0.00
Nov 2023
Set 2024
Models Knowledge Cut-off Dates
Figure 21. Effect of knowledge cut-off date on MovieTec- tion clean split performance similar models.
20

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
I. Additional Main Results
I.I. MevicTection Accuracy on Clean and Suspect Movies Large Models
Table 15, Accuracy scores for Mevionection movies included in the training data of VLMs Suspect
GPT-4o
Gemini-1.5 Pro LLaMA-3.2 90B
Qwen2-VL 72B
Captions
0.1280.011
0.079
0.0780.015
0.075
MCQA
0.721
0.550
0.540
0.6170.031
Neutral Frames
[DIS-CO]
0.2266.
0.1520.023
0.13 0.1340.017
0.1220.012
DIS-CO
0.338p.030
0.2090.031
0. 0.176g.023
0.176p.018
Captions
0. 0.1977.018
0.1266.022
0. 0.122p.024
0.121
MCQA
0.770p.08
0.638
0.65 lo 036
0.692..00
Main Frames
[DIS-CO]
0.512
0.3288.024
0.300
0.2740.014
DIS-CO
0.704..022
0.4290.033
0.404..0s
0.3770.029
Table 16. Accuracy scores for MovicTeccionn movies included in the training data of VLMss Clean
GPT-4o
Gemini-1.5 Pro LLaMA-3.2 90B
Qwen2-VL 72B
Captions MCQA
0.001o.00
0.000
0.000
0.001g.001
0.410u.0s7
0.295g.038
0.295u.u52
0.149u.026
Neutral Frames
[DIS-CO]
0.002
0.004..oor
0.005
0.000
DIS-CO
0.0022.001
0.004g.om
0.005..002
0.001
Captions
0.000
0.000
0.000
0.000
MCQA
0.445p.046
0.380
0.365
0.188p.025
Main Frames
DIS-CO]
0.010p.000
0.010
0.013o.om .000
0,000
DIS-CO
0.010p.003
0.010g.005
0.013
0.000g.000
The additional accuracy results Table 15 and Table 16 reinforce the trends observed Tables 3 and 4 from the main text. While GPT-4o consistently achieves the highest performance, the relative ranking of methods remains stable across all models.
(i) MCQA, again, demonstrates relatively high accuracy for suspect movies across all models; however, this comes the cost of a high false positive rate on clean movies. This tradeoff undermines its overall reliability, as it leads to misclassify non-memorized content as suspect.
(ii) Captions, despite occasionally achieving moderate AUC scores (Table 3), exhibit poor accuracy performance, even in detecting suspect movies. This limitation is most pronounced models like Qwen2-VI, 72B, where caption-based classification neutral frames results an accuracy below 10%. Such results suggest that captions alone are insufficient indicators of memorization.
By contrast, DIS-CO and [DISCCO] continue to outperform alternative baselines, demonstrating stronger detection capabil- ities for suspect movies while maintaining low false positive rates for clean movies. Their consistent superiority across models further underscores their robustness and reliability in identifying memorized content.
21

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
1.2. Teciinn Accuracy Clean and Suspect Movies Small Open Source Models
Table 17. Accuracy scores for MevicTection movies included in the training data of Smaller Open-Source VLMs Suspect
Qwen2-VL 7B LLaVA-v1.5 7B
LLaMA-3.2 11B
Pixtral-12B
Captions
0.035
0.029
0.047
0.044p.009
MCQA
0,485
0.397..069
0.4200.014
Neutral Frames
[DIS-CO]
0.075
0.0199.006
0.089
0.043v.013
DIS-CO
0.0999.023
0.030p.010
0.110g.020
0.058p.017
Captions
0.066p.0018
0.070p.019
0.087
0.076p.015
MCQA
0.558p.040
0,425p.074
0.507..036
Main Frames
[DIS-CO]
0.201
0.044p.016
0.2150.028
0.111..025
DIS-CO
0.260p.0034
0.072o.027
0.2730.036
0.160g.0083 0035
Table 18. Accuracy scores for MovicTection movies included in the training data of Smaller Open-Source VLMs Clean
Qwen2-VL. 7B LLaVA-v1.5 7B
LLaMA-3.2 11B Pixtral-12B
Captions
0.00
0.000
0.000..000
0.001
MCQA
0.115g.023
0.092
0.2770.065
Neutral Frames
[DIS-CO]
0.000,.000
0.000p.00
0.0030.003
0,000
DIS-CO
0.000g.000
0.000
0.003g.003
0.000p.000
Captions
0.001
0.000
0.000
0.001
MCQA
0.1169.020
0.092..03.
0.2770.065
Main Frames
[DIS-CO]
0.000,.00
0.000p.00
0,020b.o11
0.000 000
DIS-CO
0.000o.000
0.000p.000
0.020
0.000p.000
The accuracy results Tables 17 and 18 extend analysis smaller open-source VLMs. DIS-CO and [DIS-CO], while exhibiting reduced accuracy in absolute terms compared to the larger models (Tables 15 and 16), maintain their advantage over the alternative baselines. These methods consistently demonstrate stronger detection capabilities for suspect movies while keeping false positives on clean movies to a minimum. LLaVA-v1.5 7B seems to be the only outlier in this trend, as both DIS-CO variants perform closer to captions rather than showing a clear advantage.
22

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
J. Long Context Additional Results
GPT-4
Gemini-i Pro
Qwen22AL 728
L 908
GP7-4u
Gamini-15 Pro
Qwwn2-VV 728
Ulmma-3.2
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
N=2
N=2
N=4
Number Frames
Prompt Context
Number of Frames (N) in Prompt Context
Figure 22. DIS-CO's accuracy the MovieTection suspect split with varying numbers of frames in the prompt. Scores are pro- duced with the main frames and using the large models.
Figure 23. DIS-CO's accuracy on the Movie suspect split with varying numbers of frames in the prompt. Scores are pro- duced with the neutral frames and using the large models.
Owen2
Uamo
Plxtral 128
Owen2 VL
Lava
3.2
Pixtrol228
1.0
1.0
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.2
0.2
N=1
N=2
N=3
N=4
N=1
N=2
N=3
N=4
Number of Frames (N) Prompt Context
Number Frames (N) in Prompt Context
Figure 24. DIS-CO's accuracy on MovieTection suspect split with varying numbers of frames in the prompt. Scores are pro- duced with the main frames and using the smaller models.
Figure DIS-CO's accuracy on the Tection suspect split with varying numbers of frames in the prompt. Scores are pro- duced with the neutral frames and using the smaller models.
In the main text, we observed in Section 6.5 a general trend where increasing the number of frames in the prompt led to improved detection performance. Here, we extend this analysis by separately evaluating the impact of the two frame types along the multiple models.
Large-Scale Models: From Figure 22 and Figure 23 observe that, regardless of the frame type, the trend remains: more frames in the prompt consistently lead to better performance. The only key distinction between the two types is that the neutral frames yield lower absolute accuracies. Nonetheless, this is expected given the increased difficulty of detection when using frames that are less informative.
Interestingly, despite Meta's official recommendation that LLaMA performs best with a single image during inference our results suggest that while the model may not have been explicitly optimized for multi-image inputs, can still benefit from the extended context in this setting.
These models follow the same pattern observed in Figures 22 and 23. However, their overall accuracy remains lower, which is expected given their smaller size and capacity. Only LLaVA appears to be an exception, as it does not seem to effectively leverage multiple-image inputs, showing limited improvement compared to the other models.
Smaller-Scale Models:
>https / huggingface.co / meta 11ama L.lam 3.2 11B Vision Instruct / discussions 13466f987442994ddeee
23

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
K. Popularity Additional Results
GP--4
Gumini-s 5 Pro
2w 728
L 908
GP7-4u
Gamini-15 Pro
Qwwn2-VL 728
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
200M
400M 600M 600M!
10
1.28
1.48
1.60
1.98
2B
2.28
200M
400M
600M 900M
1.28
1.48
1.88
1,88
2.28
Grossing ($)
Grossing ($)
Figure 26. Box-Office effect of suspect movies DIS-CO's per- formance. Scores are produced with the main frames and using the large models.
Figure 27. Box-Office effect of suspect movies DIS-CO's per- formance. Scores are produced with the neutral frames and using the large models.
Owen2 VL 78
Uamo
Plxtral 128
Owen2 VL
Liva 78
3.2
Pitrrl
1.0
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
200M
10OM
600M gooM
18
1,28
1.48
1.88
1.38
28
2,28
200M
420M
600M
goom
1B
1.28
1,48
1.08
1.68
28
2.28
Grossing ($)
Grossing
Figure 28. Box-Office effect of suspect movies on DIS-CO's per- formance. Scores are produced with the main frames and using the smaller models.
Figure 29, Box-Office effect suspect movies on DIS-CO's per- formance. Scores are produced with the neutral frames and using smaller models.
In the main text, we observed general trend where higher box-office revenue correlates with improved detection performance across models (Figure 4). Here, we extend this analysis by separately evaluating the impact of the two frame types along the multiple models.
Large-Scale Models: Figures 26 and 27 show that higher box-office revenue consistently improves detection performance, remaining agnostic to the frame type used. Both main and neutral frames follow similar patterns, with the key distinction being that neutral frames yield slightly lower absolute accuracies due their inherent difficulty. This consistency across frame types confirms that Figure 4 accurately captures the overall trend of the models, despite presenting results based on the grouping of both frame types.
Small-Scale Models: Figures 28 and 29 show much more inconsistent relationship between box-office revenue and detection accuracy compared to larger models. While LLaMA-3.2 1 11B, shows a noticeable improvement with higher- grossing films, other models, like LLaVA, display erratic fluctuations with less clear trends.
24

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
L. Quality Additional Results
GiT-4
Gemini-s Pro
Qwen22V 728
L 908
GP7-4u
Gamini-15 Pro
Qwwn2-VV 728
Ulama-3.2 938
0.8
0.8
Accuracy
Acouracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Rating
Rating
Figure 30. IMDb movie rating effect of suspect movies DIS- CO's performance. Scores are produced with the main frames and using the large models.
Figure 31. IMDb movie rating effect of suspect movies DIS- CO's s performance. Scores are produced with the neutral frames and using the large models.
Owen2
VL577
Uams
Plxtral
Owen2
Lava
Aixrrl
1.0
1.0
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.2
0.2
4
6 Rating
8
5
7
Rating
Figure 32. IMDb movie rating effect suspect movies on DIS- CO's performance. Scores are produced with the main frames and using the smaller models.
Figure IMDb movie rating effect of suspect movies on CO's performance. Scores are produced with the neutral frames and using the smaller models.
In the main text, we observed that higher IMDb ratings generally led to improved detection performance across models (Figure 5). Here, we extend this analysis by separating the main and neutral frame types and evaluating performance across both large-scale and smaller models.
Large-Scale Models:
Figures 30 and 31 reveal overall upward trend in detection performance IMDb ratings increase. However, interesting U-shaped pattern is noticeable, particularly in main frames, where detection accuracy initially drops for lower-rated movies (around RatingE[4.5]) before rising sharply from Rating=6 onward. In contrast, neutral frames display a more gradual improvement without the same dip at low ratings. Only Gemini-1.5 Pro, unexpectedly, shows a sharp drop at Rating=9, deviating from the otherwise consistent trend.
Small-Scale Models: Figures 32 and 33, on the other hand, show that overall performance remains weak across most rating levels, with notable exception in Rating=8, where most models exhibit sudden increase accuracy, though the reason for this improvement unclear.
25

---

DIS-CO: Discovering Copyrighted Content in VLMs Training Data
M. MovieTeecioon Dataset Release
This following excerpt is taken from the approval statement of our institution's DPO concerning the release of MovieTectionn
On Fair Use
Fair Use is regulated by Chapter II of the Code of Copyright and Related Right (CCRR), specifically (for the purpose of this analysis) by Article 75 and following on "Free Use".
Article 75.2. states that the following uses of the work are lawful without the author's consent:
i) The inclusion of short pieces or fragments of third-party works in one's own works intended for teaching purposes.
This provision transposes specific provisions of Directive 2001/29/EC on the harmonisation of certain aspects of copyright and related rights in the information society, that allow the Member States provide for exceptions or limitations in the case of use for the sole purpose of illustration for teaching or scientific research, long as the source, including the author's name, is indicated, unless this turns out to be impossible and to the extent justified by the non non-commercial purpose to be achieved (Article 5.3.a) of the aforementioned Directive.
In order to verify if the disclosure of dataset complies with the provisions of Article 75.2 referred above, we must address the following questions:
Were only short pieces or fragments of third-party works included?
Can the dataset, subject of this assessment, be considered an INESC-ID own and original work?
Is the dataset intended for teaching purposes?
The answer to the first question is clearly positive only 140 frames per movie were extracted, accounting for a small part of original works.
In order to answer the second question we will consider the applicable provisions on databases as there are no specific provisions on datasets and datasets, per se, are not protected by copyrights. While technically a dataset a structured collection of data and a database an organized collection of data stored as several datasets, for the purpose of this assessment the most important issue is whether the dataset can, by reason of the selection or arrangement of its contents, constitute the author's own intellectual creation, building upon the criterion for the copyright protection defined in Portuguese Law 122/2000 on the legal protection of databases and in Directive 96/9/EC on the same matter. Taking into account significant intellectual effort, as described in the Description of Work, that led to the preparation of the dataset, we believe that it can be considered as an original work. In respect to the last question, we will consider the statutory attribution of INESC-ID and the purpose of making the dataset available to the public.
INESC-ID is a research institute that has as a statutory goal to carry out scientific research, technological development and, additionally, the provision of services in the areas of information technology, telecommunications, electronics, computers and energy. In order to pursue its goal, among other attributions, it has been assigned responsibility to publish the results of the research to which it is dedicated and disseminate scientific and technological culture in its areas of activity as well as to exchange scientific and technical information with other related institutions (Article 2.2. of its Articles of Association) (...)
Ethical Assessment
From an ethical standpoint, the following issues are considered: the substantiality of the portion used in relation the movies as whole and the potential effect that might have their market value well as the purpose of making the dataset available to the public.
As stated above, only 140 frames per movie were extracted, accounting for small part of original works with no expected impact on the market value of the movies.
Similarly, as already discussed above, the purpose of the disclosure is limited to non-commercial activities, such as teaching. Additionally, there are expected benefits in the field of machine learning resulting from this disclosure.
Conclusion
Taking into account what has been discussed above long as the terms of use referred to hereinabove are made available together with the dataset making the dataset available to the public for teaching purposes does not present issues from legal or ethical perspective.
26


Reviews:
[{"review_id": "Seg0cPIobY", "reviewer": "Reviewer_tTxp", "summary": "This paper investigates the challenge of verifying whether copyrighted content was used to train large vision-language models (VLMs) without direct access to their training data. The authors introduce DIS-CO, a novel approach that leverages the hypothesis that VLMs can recognize images from their training corpus. By systematically querying a VLM with specific frames from copyrighted material, DIS-CO extracts content identity through free-form text completions. To evaluate its effectiveness, the authors present MovieTection, a benchmark containing 14,000 frames with detailed captions from films released both before and after a model’s training cutoff. Experimental results demonstrate that DIS-CO significantly enhances detection performance, nearly doubling the AUC of the best prior method for models with available logits.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your valuable feedback and comments. Below, we address each of your questions.\n\n> **W1.** Lack of formal mathematical analysis.\n> \n\nHere, we present our mathematical analysis to support our intuition on why free-form completions (FF) are a more robust indicator of memorization than Multiple Choice (MC) formats.\n\n**Random-Chance Baselines:**\nIn an MC setting with $k$ options, the probability of a correct guess by chance is $P_{MC} = \\frac{1}{k}$, wich for $k=4$ would result in 25%\n\nBy contrast, in an FF setting the model must generate the exact label from a vast output space $\\Omega$ (with $|\\Omega| \\gg k$), so that the chance-level probability is approximately $P_{FF} \\approx \\frac{1}{|\\Omega|},$ which is orders of magnitude lower than $P_{MC}$.\n\n**Impact of Non-Uniform Priors:**\nIt is true that models may exhibit a bias toward more popular movies. Even if this bias increases the chance for a popular movie by a factor of 100, the overall probability remains extremely small. For instance, if $|\\Omega| = 10{,}000$, even with a bias, the probability $P'_{FF} = \\frac{100}{10{,}000} =$  1%, which is still 25 times lower than the MC chance of 25%.\n\n> **Q1.** Clarification on Dataset Contamination.\n> \n\nWe fully agree that external sources could influence the model’s performance. To investigate this potential contamination, we would like to draw your attention to Appendix D. We found that GPT-4o (with a knowledge cutoff in October 2023) acknowledged 20 out of the 50 clean movies included in MovieTection. \n\nWe believe it's reasonable to assume these 20 titles are likely candidates for having appeared in publicly accessible sources such as trailers, promotional posters, or press coverage before the model's cutoff date, though not as full movies, since they had not been released yet.\n\nThe table below summarizes findings from Table 10 in Appendix D. \n\n| Release Period | # of Movies | Non-zero Values | Max Accuracy |\n| --- | --- | --- | --- |\n| **Nov 2023 - Feb 2024** | 6 | 4 | 6% |\n| **Mar–Oct 2024** | 12 | 0 | 0% |\n\nWe observe that movies released closer to the model's cutoff date, and thus with more external content available, result on the highest accuracy scores, while very recent releases, for which public information was considerably sparser at the cutoff date, show no correct mappings. The highest observed accuracy for this subset of recognized movies was, however, only 6%, significantly lower than accuracy scores typically achieved for suspect movies. Therefore, we firmly believe that exposure to external content alone minimally impacts our primary results.\n\n> **W2 + Q2** Could you consider conducting study to isolate the contributions of specific components of DIS-CO?\n> \n\nIsolating and understanding the contributions of individual components in DIS-CO is definitely essential. However, we believe to have already explored some of these aspects, for which we would like to direct your attention to Appendix I, where we analyze the effect of varying the number of frames per query. Our analysis is conducted across the different frame types (main vs. neutral) and model sizes, offering a wide view of how these variables affect performance.\n\nAs for your suggestion regarding prompt variations, we conducted an additional experiment to evaluate model sensitivity to different wordings, which can be relevant for practical applications, since user inputs will naturally have some variance in the words.\n\nWe designed a small-scale evaluation using two categories of prompts:\n\n- Biased Prompts: These include additional cues that might assist the model.\n    - Example: “What Oscar-winning movie is this frame from?”\n- Paraphrased Prompts: Semantically equivalent rephrasings of the default prompt.\n    - Example: “Can you identify what movie is present here?”\n\nWe evaluated model responses on a subset of the MovieTection dataset, summarized below:\n\n| Prompt Type | 21 Jump Street | 1917 | A Beautiful Mind | A Star is Born | Aladdin | Avg. |\n| --- | --- | --- | --- | --- | --- | --- |\n| Easier | 83% | 100% | 87% | 85% | 92% | 89.4% |\n| Default | 68% | 86% | 71% | 80% | 92% | 79.4% |\n| Default Paraphrased | 60% | 88% | 74% | 82% | 92% | 79.2% |\n\nGiven that in real-world settings the target content may not always be a well-known blockbuster, biasing the model toward popular titles through hints may not be ideal. We believe that sticking to neutral prompt variations is a more reliable choice, as it avoids introducing external priors and better reflects the model’s actual memorization.\n\n> **Flag for Ethics Review**\n> \n\nRegarding the ethics review flag, please refer to our response provided in the reply to Reviewer V24U (Q2 + Q5), which addresses this issue as well.\n\n\n**Conclusion:**\n\nWe hope that our answers have addressed your concerns, and thank you once again for your valuable feedback.\n\nPlease let us know if any further clarification or additional information is needed from our end.", "claims_and_evidence": "Claim: The paper claims that a VLM is able to recognize images from its training corpus. \nThis key claim is supported by the evidence in existing literature [r1] that data encountered during training leads to greater model confidence when generating outputs\n\n[r1] Li Z, et al. Membership Inference Attacks against Large Vision-Language Models. NeurIPS, 2024.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem statement. The study introduces a rigorous benchmark, strong comparative baselines, and a method applicable to real-world black-box models. Improvements in ablation studies, dataset validation, and robustness against adversarial probing would further enhance its impact.\n\n1.\tAppropriateness of the Proposed Method (DIS-CO). The DIS-CO method is well-suited for detecting whether large vision-language models (VLMs) have been trained on copyrighted content. By leveraging free-form text generation, the approach mitigates biases introduced by multiple-choice settings and allows for more natural model responses.\n2.\tStrength of Benchmark Dataset (MovieTection). The introduction of MovieTection as a benchmark dataset is a strong contribution. It is carefully constructed with a temporal split to differentiate between pre-training and post-training data. The inclusion of \"main frames\" and \"neutral frames\" enhances evaluation granularity, making the benchmark robust for detecting training exposure.\n3.\tEvaluation Metrics and Comparisons. The study effectively evaluates DIS-CO using AUC scores, accuracy metrics, and comparative baselines (MCQA, Rényi method, Captions-based prompting). The removal of caption-based correct predictions to isolate image memorization is a novel refinement.", "theoretical_claims": "1. Soundness of the DIS-CO Method. The paper’s central theoretical claim—that free-form text completion provides a stronger signal for detecting memorized content than multiple-choice settings—is intuitively reasonable and supported by past studies on language model leakage. However, The paper lacks a formal proof of why free-form completions reduce false positives better than MCQA.\n2. Upper-Bound Estimation of Memorization. The approach of defining DIS-CO and ⌊DIS-CO⌋ as lower and upper bounds of memorization is reasonable, but the paper does not theoretically prove the validity of this bounding technique.", "experimental_designs_or_analyses": "1.\tValidity of Experimental Setup. \nStrengths: The experimental design of **DIS-CO** is well-structured to test its effectiveness in detecting copyrighted content within VLMs. The evaluation setup appropriately considers **both white-box and black-box models**, making the results more generalizable. \nWeakness: However, a potential limitation is the **assumption that chronological release serves as a strict boundary for training data exposure**. While this is a reasonable heuristic, it does not entirely rule out indirect exposure through publicly available media (e.g., trailers, posters). A complementary **dataset contamination check** would strengthen the claim that post-cutoff movies are truly novel to the model.\n2.\tBaseline Comparisons. \nStrengths: The study effectively benchmarks **DIS-CO** against prior methods, including **MCQA and Renyi entropy-based techniques**. The **AUC and accuracy metrics** provide a fair and interpretable comparison. The decision to remove **caption-based correct predictions** from the evaluation is a useful refinement, ensuring that memorization is attributed to visual data rather than textual association.\nWeakness: An area for improvement is the **lack of an ablation study** that isolates the contributions of **prompting strategy, query selection, and frame type**. Understanding the relative impact of these components would clarify why DIS-CO outperforms prior approaches.", "relation_to_broader_scientific_literature": "DIS-CO builds on membership inference attacks as explored by [r1] and [r2], extending them to black-box models and offering more reliable detection of copyrighted content in VLMs, surpassing previous methods like Rényi entropy.\n\n[r1] Li Z, et al. Membership Inference Attacks against Large Vision-Language Models. NeurIPS, 2024.\n[r2] Pinto F, et al. Extracting Training Data From Document-Based VQA Models. ICML, 2024.", "essential_references_not_discussed": "The paper provides a well-contextualized discussion of prior work and thoroughly cites relevant research in membership inference attacks (MIA), dataset attribution, and multimodal model evaluation.", "other_strengths_and_weaknesses": "Strengths:\n1) The MovieTection dataset is a new benchmark designed for testing model memorization of copyrighted materials, addressing a crucial gap in dataset attribution research.\n2) DIS-CO achieves state-of-the-art performance in detecting memorization, improving upon prior methods such as MCQA and Rényi entropy-based inference.\n3)The methodology is clearly explained, with detailed steps for dataset construction, query formulation, and evaluation.\n\n\nWeakness:\n1) While DIS-CO’s empirical results are strong, the paper lacks formal mathematical analysis of why free-form text completion is a more reliable indicator of memorization than multiple-choice formats.\n2)Additional theoretical justification (e.g., probability bounds on model memorization) could strengthen the claims.\n3) The study does not dissect the impact of different components of DIS-CO, such as the number of frames per query, prompt variations.", "other_comments_or_suggestions": "The paper is well-organized and easy to follow. The sections are logically structured, guiding the reader through the problem statement, proposed method, experiments, and results.", "questions_for_authors": "1.\tClarification on Dataset Contamination.The paper assumes that movies released after the training cutoff date are not included in the model’s training data. However, some models might have encountered promotional content (e.g., trailers or posters) that could influence performance on recent movies. Can you provide more details or experiments to assess the potential contamination from such external content sources?\n2.\tAblation Study for DIS-CO. You have provided solid results showing DIS-CO’s advantage over prior methods. However, could you consider conducting an ablation study to isolate the contributions of specific components of DIS-CO (e.g., the number of frames per query, prompt variations)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Legal Compliance (e.g., GDPR, copyright, terms of use)"], "ethical_review_concerns": "copyright issue of the collected data"}}, {"review_id": "FJ6vTFZ6Ds", "reviewer": "Reviewer_vDhh", "summary": "This paper introduces DIS-CO, a new method to check if large vision-language models (VLMs) were trained on copyrighted material. the authors use the idea that a model will \"remember\" specific content if it has seen it before. In this work, the model is asked to name a movie from a single frame or caption. The authors build a new dataset called MovieTection, which includes 14,000 movie frames (and captions) from films released before and after the model’s training cutoff date. Their experiments show that when the model has seen a movie during training, it is more likely to correctly name the movie, and this method works well across different types of models, even in settings where frames are very challenging and we can only see a few of the model’s outputs.\n\n## update after rebuttal\nThe author clearly addressed my question so I kept the score as accept. I do think this paper is a valuable contribution to the community, and I don't agree with some other reviewers who claim that the paper lacks novelty or is too limited due to movie datasource.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe greatly appreciate the time and effort you invested in reviewing our paper. Below, we provide a response to your question.\n\n> My question to this paper is whether if this paradigm can be applied to other copyright content as well beyond just movie, which also states in the method review section: \"The paper focuses on movie frames and captions, and it remains to be seen if the approach can generalize to other kinds of copyrighted content or other multimodal tasks.”\n> \n\nWe believe it definitely can! In fact, we even ran DIS-CO on a small experiment with a completely different type of visual content: comic books.\n\nThe works we used for this were: *Astérix Legionary;* *Lucky Luke: Billy the Kid;* *The Amazing Spider-Man #1;* *Spirou and Fantasio: Comme Zorglub;* and *Tintin in America*.\n\nGiven that these comics come from series with multiple of similar-looking volumes, we expected a model to struggle. Still, GPT-4o managed to correctly identify the specific comic-title surprisingly often.\n\n|  | Astérix | Lucky Luke | Spider-Man | Spirou and Fantasio | Tintin |\n| --- | --- | --- | --- | --- | --- |\n| GPT-4o | 52.0% | 61.3% | 42.3% | 68.8% | 67.2% |\n\nThat said, we ended up leaving this part out of our main experiments because, unlike movie frames, where the visual signal is (mostly) isolated, comic books introduce an extra variable: text inside the images. This makes it harder to know whether the model’s prediction came from the visual content or if it was leveraging the text. Since our goal was to test visual memorization specifically, we felt that mixing modalities here could weaken the core message.\n\n**Conclusion:**\n\nWe hope that our answer has addressed your concern. Please let us know if any further clarification or additional information is needed from our end.", "claims_and_evidence": "Core Claims:\n\n1. DIS-CO effectively detects the inclusion of copyrighted movies in a model’s training data, outperforming baseline approaches across multiple evaluation metrics.\n2. The proposed method is applicable in both white-box and black-box settings.\n\nEvidence:\n\n1. Experiments on the MovieTection benchmark demonstrate that models yield significantly higher accuracy and AUC when queried with suspect frames (from copyrighted movies) than with clean or non-member frames.\n2. The study shows a clear improvement in detection performance with longer prompt contexts and reveals a positive correlation between factors such as movie popularity (box-office revenue) and quality (IMDb ratings) with the likelihood of memorization.", "methods_and_evaluation_criteria": "Methods:\n\n1. The approach involves querying VLMs with free-form text prompts where the model is asked to identify the movie from an input frame or caption.\n2. DIS-CO distinguishes between “suspect”  and “clean” movies based on their release dates relative to the model’s training cutoff.\n3. Two variants are proposed: one that considers all correct predictions (DIS-CO) and another (⌊DIS-CO⌋) that filters out cases where both image-based and caption-based queries agree, thereby reducing bias.\n\nEvaluation:\n\n1. Performance is assessed using accuracy and Area Under the Curve (AUC) metrics on the MovieTection benchmark and the VL-MIA/Flickr dataset.\n2. Experiments span several models (e.g., GPT-4o, Gemini-1.5 Pro, Qwen2-VL 72B, LLaMA-3.2 90B) \n\nWeakness: The paper focuses on movie frames and captions, and it remains to be seen if the approach can generalize to other kinds of copyrighted content or other multimodal tasks.", "theoretical_claims": "There is no theoretical claims in this paper", "experimental_designs_or_analyses": "1. The experimental design is comprehensive, using a newly constructed benchmark (MovieTection) alongside an existing dataset (VL-MIA/Flickr) to evaluate model performance.\n2. The study considers various dimensions including prompt context length, movie popularity, and quality, offering insights into factors that influence model memorization.\n3. Human evaluation experiments are conducted to differentiate between generalization and genuine memorization, adding depth to the analysis.", "relation_to_broader_scientific_literature": "The work builds on a rich literature on membership inference attacks and data contamination detection, extending these ideas from text-only models to the multimodal setting of VLMs. It relates to recent prompting approaches and entropy-based methods while addressing their limitations, especially in black-box scenarios.", "essential_references_not_discussed": "All essential references I know have been discussed.", "other_strengths_and_weaknesses": "All strengths and weakness has been pointed out above.", "other_comments_or_suggestions": "no other comments. the paper is well-written.", "questions_for_authors": "1. My question to this paper is whether if this paradigm can be applied to other copyright content as well beyond just movie, which also states in the method review section: \n    \"The paper focuses on movie frames and captions, and it remains to be seen if the approach can generalize to other kinds of copyrighted content or other multimodal tasks.\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEA3HZwspr", "reviewer": "Reviewer_V24U", "summary": "The author proposes a copyright detection method for VLM training data based on free-text generation, where movie frames are input into the model to generate corresponding titles, allowing for the detection of whether the model has memorized copyrighted content. The main innovations include: 1) the construction of the MovieTection dataset, which differentiates between training and non-training data based on temporal segmentation to improve the effectiveness of detection; 2) combining image and text inputs to eliminate interference from common knowledge, thereby enhancing the reliability of detection. Experimental results show that DIS-CO significantly outperforms traditional methods in terms of average AUC on mainstream models like GPT-4o. The study further proposes a fine-tuning defense strategy to mitigate potential information misuse issues.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate the time and effort dedicated to evaluating our paper. We understand the concerns raised and below, we address each point in detail:\n\n> **Q1.1** How applicable and universal is the proposed mechanism?\n> \n\nWe acknowledge that MovieTection’s focus on box-office hits may limit its coverage of niche films, which are also subject to copyright and may appear in pretraining datasets.\n\nWe would like, nonetheless, to make two clarifications:\n\n- While this is indeed a limitation, it reflects a broader challenge shared by most existing studies in the field. Detecting memorization of rarely seen content is inherently more difficult than detecting memorization of content that appears frequently during training.\n- We agree that it is important to assess how VLMs respond to niche content. To complement our experiments, we conducted an auxiliary study in which we tested DIS-CO on three films with little to no international projection.\n\n| Movie ($X = Box Office) | Gaiola Dourada ($3M) | Canção de Lisboa ($930k) | Leão da Estrela ($500k) |\n| --- | --- | --- | --- |\n| GPT-4o | 13% | 8% | 0% |\n\nIt’s interesting to see that with DIS-CO, a smaller movie such as *Gaiola Dourada* is still partially recognized by gpt-4o, suggesting that even limited exposure during training can leave some detectable traces. However, the overall trend is clear: as a film’s popularity and likely exposure in training data decreases, so does the strength of the memorization signal. This reinforces our focus on popular films: while weaker signals may appear for niche works, box-office hits offer stronger, more consistent signals for evaluating memorization.\n\n> **Q1.2** The impact of image compression and resolution changes has not been tested.\n> \n\nWe did a small experiment to evaluate how different resolutions affect DIS-CO’s performance.\n\n| GPT-4o | 21 Jump Street | 1917 | A Beautiful Mind | A Star is Born | Aladdin | Avg. |\n| --- | --- | --- | --- | --- | --- | --- |\n| 1126x512 | 68% | 86% | 71% | 80% | 92% | **79.4%** |\n| 563x256 | 58% | 85% | 66% | 77% | 86% | **74.4%** |\n| 282x128 | 57% | 85% | 58% | 64% | 74% | **67.6%** |\n\nAs expected, lower resolutions reduce accuracy, since the model has less visual details to work with. That said, DIS-CO remains effective overall: suspect movies are still clearly distinguishable from clean ones, which consistently score near 0% accuracy, regardless of the resolution. Although smaller frames introduce a light performance drop, they can still be a practical choice to reduce computational effort without significantly impacting detection quality.\n\n> **Q2 + Q5** Does it involve ethical issues? Is it aligned with specific copyright laws?\n> \n\nWe believe that all aspects of our work are aligned with fair use exceptions and conducted in accordance with relevant ethical and legal standards. \n\nWe would like to clarify that the dataset release was reviewed in advance by our institution’s Data Protection Officer (DPO), who provided a positive assessment regarding its compliance.\n\nIn addition, to safeguard ethical usage and prevent misuse, the dataset is being released under a restrictive CC BY-NC-SA 4.0 license, which limits its use strictly to non-commercial research and academic purposes.\n\n> **Q3** The current study only simulates key forgery and fine-tuning attacks (…) This results in a potentially insufficient defense capability, thus leaving certain security vulnerabilities.\n> \n\nWe respectfully note that our paper does not mention or simulate “key forgery” attacks. Could you clarify what was meant by this term, and how it relates to our work?\n\nAssuming the comment refers to our fine-tuning experiment, we would like to clarify that this component serves primarily as a proof of concept to illustrate that disclosure of memorized content can be mitigated. While we do explore fine-tuning as a way to reduce the disclosure of memorization, this remains a secondary contribution. The central focus of our work is on detection, not mitigation.\n\n> **Q4** The paper does not analyze the computational cost introduced by multiple frame inputs.\n> \n\nFollowing this suggestion, we evaluated the GPU memory usage when increasing the number of input frames.\n\n| Frames (N) | Qwen2-VL 7B | Qwen2-VL 72B |\n| --- | --- | --- |\n| 1 | 16.00 GB | 138.03 GB |\n| 4 | 17.38 GB | 142.69 GB |\n| **Increase** | **+1.38 GB** | **+4.66 GB** |\n\nOn average, each additional frame increases memory by 0.46 GB (Qwen2-VL 7B) and 1.53 GB (Qwen2-VL 72B). This corresponds, for the bigger model, in less than 3.5% total additional memory to process 4 frames, showing that the main memory requirement comes from loading the model itself. The cost added by using multiple frames is small and unlikely to be a problem in most settings.\n\n**Conclusion**\n\nWe hope that our responses clarify the concerns raised and demonstrate the validity and value of our work. We thank you for the insightful feedback and we are happy to present further clarifications.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The issue regarding Theoretical Claims will be elaborated on in the \"Questions For Authors\" section.", "experimental_designs_or_analyses": "Yes. The issue regarding Experimental Designs Or Analyses will be elaborated on in the \"Questions For Authors\" section.", "relation_to_broader_scientific_literature": "Yes. The issue regarding Relation To Broader Scientific Literature will be elaborated on in the \"Questions For Authors\" section.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Yes. The issue regarding Other Strengths And Weaknesses will be elaborated on in the \"Questions For Authors\" section.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Advantages:\n1. By generating free-form text instead of fixed-structured text, the model's susceptibility to external interference during inference is reduced, thereby enhancing its ability to detect potential copyright information leakage. This approach strengthens the robustness and credibility of the method.\n2. The dataset is partitioned based on the release dates of the films, clearly distinguishing between training and non-training data. This eliminates the risk of data overlap, ensuring more controlled experimental conditions.\n3. The proposed method mitigates memory leakage of copyright information in the model by replacing labels, such as substituting \"copyrighted content\" with alternative expressions. This defense strategy effectively prevents the model from retaining copyrighted data in its memory.\n\nQuestions:\n1. How applicable and universal is the proposed mechanism? MovieTection relies on box office rankings, neglecting niche or independent films, which could lead to an underestimation of long-tail copyright risks. Additionally, the impact of image compression and resolution changes on detection results has not been tested, affecting the robustness of the method in real-world deployment, and preventing assurance of its effectiveness in diverse scenarios.\n\n2. Does it involve ethical issues? Although the publication of 14,000 movie frames is considered \"fair use,\" this approach may still provoke copyright disputes. The article lacks sufficient ethical argumentation and does not fully address potential copyright conflicts or privacy concerns, which may expose involved parties to unnecessary legal risks.\n\n3. Can it effectively prevent potential attacks? The current study only simulates key forgery and fine-tuning attacks, without considering more complex threats such as adversarial samples or model distillation. This results in a potentially insufficient defense capability, unable to comprehensively address various security threats, thus leaving certain security vulnerabilities.\n\n4. What is the computational overhead? The paper does not analyze the computational cost introduced by multiple frame inputs, particularly the impact on GPU memory. This lack of a detailed evaluation could limit the application of the method in low-resource environments, leading to performance bottlenecks in real-world deployment and hindering its widespread use in resource-constrained settings.\n\n5. Is it aligned with specific copyright laws (such as fair use exceptions)? The paper does not integrate specific copyright legal provisions, such as fair use exceptions, leading to ambiguity in the legal significance of its conclusions. The lack of consideration for the broader legal framework, especially regarding fair use and its impact on copyright protection, results in an unclear legal interpretation and a lack of in-depth analysis of the real-world legal implications.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Privacy and Security"]}}, {"review_id": "khH8oZL7AU", "reviewer": "Reviewer_MfvN", "summary": "The paper focuses on an important and timely problem, i.e., how to discover the copyrighted content in the training data of VLMs. Specifically, the paper’s contribution includes (1) a new benchmark, MovieTection, (2) a new method DIS-Co, (3) comprehensive experiments, and (4) some new discoveries.\n\n## Update after rebuttal\n\nDear authors, I would like to express my honor to have an opportunity to review your paper and apologize for forgetting to file some areas, such as “Claims and Evidence”. Here, as the update after the rebuttal, I want to summarize my opinion and give the final recommendations.\n\nAlthough I agree that: (1) the paper makes valid claims with experimental evidence; (2) the evaluation is sufficient; (3) the designs and analyses of experiments are reasonable and sufficient; and (4) the relation to broader scientific literature is strong. The following concerns are not solved during the rebuttal, and therefore I suggest incorporating them into the revision and submitting your paper to a later conference.\n\n1. To show your paper is novel, you should NOT say there are similar topics in the top-tier conferences. That is definitely true because I do not argue that this topic is not novel; the topic is important and should appear at these conferences. In addition, the move from LLM to MLLM cannot be considered significant, though unexplored. I think the proposed DIS-CO is not novel (as shown above, and the authors admit it is a piece-up of existing works). No need to argue that you build a new benchmark and conduct a lot of experiments, they have nothing to do with the novelty.\n\n2. The authors say \"Our paper focuses on a technical problem: detecting whether a model has memorized specific content.\" This focus is feasible. However, the paper does not reflect this. The paper makes a lot of arguments on *copyright* rather than memorization/replication. Please see a similar paper [1] on how to focus on technical problem: \n\n*\"Replicants are either a benefit or a hazard; there may be situations where content replication is acceptable, desirable, or fair use, and others where it is “stealing.” While these ethical boundaries are unclear at this time, we focus on the scientific question of whether replication actually happens with modern state-of-the-art diffusion models, and to what degree.\"*\n\nWhat's more, your paper and [1] have a large similarity because you focus on the copy(right)/memorization of MLLM and [1] focus on the diffusion models. However, no citation or comparison is given.\n\n[1] Somepalli, Gowthami, et al. \"Diffusion art or digital forgery? investigating data replication in diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n3. 8 pages should not be the excuse for limiting to videos rather than images. In fact, you also treat videos as a sequence of images, and there is indeed no fundamental difference. To conduct comprehensive research, you should include both videos and images.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate the time and effort invested in reviewing our paper. Below, we clarify your comments.\n\n> Why do you think using copyrighted material in the training data is copyright infringement rather than fair use?\n> \n\nIf the paper gave the impression that we consider all copyrighted content in training data inherently infringing, that was not our intention.\n\nWe fully recognize that the legality of training on copyrighted data is complex and context-dependent. However, especially in light of recent lawsuits, we notice that training on copyrighted data without authorization can raise serious concerns, particularly when the resulting models are deployed commercially.\n\nOur work does not take a legal position on whether a specific instance constitutes infringement or fair use. Rather, our contribution is technical: we propose a method of detecting whether specific content (copyrighted or otherwise) was included in a model’s training data.\n\n> If you think using copyrighted material in the training data is copyright infringement, why do you include the copyrighted movies in your dataset? (…) What we should do is to detect “copy” rather than “copyright”.\n> \n\nWith respect to our own use of copyrighted movie frames in the *MovieTection* benchmark, we address this concern directly in our Impact Statement. Our dataset contains a small number of frames per film, is used solely for education and research purposes, and does not substitute the original work, conditions under which we believe fair use applies.\n\nWe also fully agree with the distinction made between detecting “copyright” and “copy.” The key issue is not whether a model saw the material, but whether it memorizes and reproduces it in a problematic way. As noted, *“Infringement happens when model generates something very similar to the copyrighted image…”*\n\nThis was also a core motivation behind our work. For example, it is probably fine, and even expected, for a model to know *Notting Hill* is a romantic comedy. But when it can consistently name the movie title from frames like Figure 9 (in the paper), that level of specificity suggests not just general understanding, but visual memorization, which may go beyond what’s expected from general exposure.\n\nThis is why we designed the benchmark, and we believe it plays an important contribution for the field, as it helps drawing the line between the different levels of memorization that VLMs may exhibit. In that sense, we believe our work fully aligns with your view.\n\n> Second, I am curious about why you focus on the movies. (…) Also, is your task limited to the video domain? Why not images?\n> \n\nOur focus on movies was mainly motivated by the fact that they are likely to be familiar to a broader audience, making the results easier to interpret and relate to than, for example, paintings or less widely consumed copyrighted material.\n\nAs for the second part of your question: no, our method is definitely **not limited to the video domain**. In fact, as shown in our proof-of-concept experiment using COCO, our technique works on static, single images. On top of that, we also conducted an initial experiment using a different type of copyrighted content: comic books.\n\nFor this, we assembled a small dataset of five different works: *Astérix Legionary*, *Lucky Luke: Billy the Kid*, *The Amazing Spider-Man #1*, *Spirou and Fantasio: Comme Zorglub*, and *Tintin in America*. As these comics come from long-running series with similar visuals, we expected models to struggle to identify specific titles. However, GPT-4o performed surprisingly well:\n\n|  | Astérix | Lucky Luke | Spider-Man | Spirou and Fantasio | Tintin |\n| --- | --- | --- | --- | --- | --- |\n| **GPT-4o** | 52.0% | 61.3% | 42.3% | 68.8% | 67.2% |\n\nThat said, we chose not to further develop this experiment in the paper because, unlike movie frames, comic pages often contain text within the images. This introduces an additional variable, making it harder to determine whether the model is relying on the artwork alone or also using the text to make its predictions. Since our goal was to evaluate visual memorization specifically, we felt that this mix of modalities could reduce the clarity of our findings.\n\n> Finally, I am concerned about the novelty of the proposed DIS-CO (…) It is a combination of known technologies.\n> \n\nWhile DIS-CO builds on existing components, its application to detecting visual copyrighted content in VLMs is, to our knowledge, novel. No prior work has addressed this task. And the fact that it combines known techniques is not by itself a limitation! In the end, it’s what enabled us to outperform the strongest prior method [1] and bring new insights to the field.\n\n[1] Li Z, et al. Membership Inference Attacks against Large Vision-Language Models. NeurIPS, 2024.\n\n**Conclusion:**\n\nWe hope that our answers have addressed your concerns. Please let us know if any further clarification or additional information is needed from our end.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\nContributing a new dataset is helpful to the community.\nThe paper is generally well-written.\nThe experiments are comprehensive.\n\n\nCons:\n\nFirstly, I want to discuss the **copyright** problem refered in this problem. \n1. Why do you think using copyrighted material in the training data is copyright infringement rather than fair use? If a VLM lacks of explosure to all the copyrighted material, it will lack many knowledge. That is similar to a scene that happened many years ago: if the copyrighted material should be searched by the searching engine? After many years, we have rearch an agreement that the copyrighted materials available on the website is OK. So, today, many researchers think that using copyrighted material in the training data for LVM is not copyright infringement. The copyright infringement happens at (1) a model generates something very similar to the copyrighted image/video/texts; ***and*** (2)  someone use the generation for profit without the authorization of the owners. It is fair use if someone only use for purposes like eduction. \n2. If you think using copyrighted material in the training data is copyright infringement, why do you include the copyrighted movies in your dataset? Does this constitute the copyright infringement? From my point of view, what we should do is to detect “copy” rather than “copyright”.\n\nSecond, I am curious about why you focus on the movies. There are many copyrighted images and videos are not from movie. Also, does your task be limited to the video domain? Why not images?\n\nFinally, I am concerned about the novelty of the proposed DIS-CO. As shown in Fig. 2, it is a combination of known technologies. The method seems to be an engineering solution rather than an academic algorithm.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes DIS-CO, a method for detecting whether copyrighted content was used in VLM training via free-form prompting, and introduces the MovieTection benchmark. 

The majority of reviewers (4, 3, 3, 1) found the paper well-executed, with strong results and clear value. While one reviewer raised concerns about novelty and framing, the authors provided rebuttals and showed some generalization beyond movies. Ethics reviewers flagged minor concerns around dataset usage, which the authors have committed to address in the final version. 

Overall, the ACs recommend acceptance, contingent on revision to address the ethical concerns raised (e.g., clearer dataset documentation and legal framing) as promised in the rebuttal.
Decision: Accept (poster)