Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
Authors: Jaden Park
Mu Cai
Feng Yao
Jingbo Shang
Soochahn Lee
Yong Jae Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models
Data Contamination
Primary area: foundation or frontier models, including LLMs
TL;DR: We devise a novel contamination detection method for vision language models.

Abstract: Recent advances in Vision–Language Models (VLMs) have achieved state-of-the-art performance on numerous benchmark tasks. However, the use of internet-scale, often proprietary, pretraining corpora raises a critical concern for both practitioners and users: inflated performance due to \emph{test-set leakage}. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for \emph{contaminated VLMs} remains underexplored. To address this gap, we deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit inconsistent behavior. We then propose a novel simple yet effective detection method based on \textit{multi-modal semantic perturbation}, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate our approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released here: \href{https://github.com/jadenpark0/mm-perturb}{https://github.com/jadenpark0/mm-perturb}.

Published as a conference paper at ICLR 2026
CONTAMINATION DETECTION FOR VLMS USING MULTI-MODAL SEMANTIC PERTURBATIONS
Jaden Park', Mu Cai', Feng Yau?, Jingbo Shang", Soochahn Lee", Yong Jae Lee"
1 University of Risconsin-Madisnn *University of California, San Diego
*Kookmin University
ABSTRACT
Recent advances Vision-Language Models (VLMs) have achieved state-of-the- art performance numerous benchmark tasks. However, the use of attemet-sscal,, often proprietary, pretraining corpora raises a critical concern for both practition- and users: inflated performance due test-ser leakage. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for contaminated VLMs remains underexplored. To address this gap, deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit incon- sistent behavior. We then propose a novel simple yet effective detection method based on multi-modal semantic perturbation, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released here: https//gghhbb.co////deppak//n/
Multi-modal Semantic Perturbation
45
Question
What is the speed limit on this road?
A. 25
3.
4.45
Clean Model
Contaminated Model
Clean Model
Contaminated Model
Figure 1: Example benchmark. Using ControlNet truined with Flux models. speed limit sign generuted, changing the correct answer from (B) to (C) while preserving the original imago's ' overall composition. A contaminated model that has memorized the original question is likely to fail the perturbed version.
multi-modal semantic perturbation pipeline applied to RealWorldQA
INTRODUCTION
Recent advances in Vision-Language Models (VLMs) have resulted in remarkable performance across
a wide range of tasks, including visual reasoning Yue et al. 2024 Liu et al.] (024bb Chen et al. 2024a). real-world understanding (xAI 2024), and complex mathematical problems (Zhang et al.
20246 Lu et al. 2024b). typical VLM training pipeline involves pretraining a vision encoder and
language backbone internet-scale data, followed by fine-tuning stage high-quality multimodal

---

Published as a conference paper at ICLR 2026
Requirements
Reliability
Practicality
Consistency
N-gram Accuracy
Xu et
2024
X
Shared Likelihood
Oren ct
2023
X
Guided Prompting
Golchin
Surdcanu
2024
X
Multi-modal Leakage
Chen a1
2124a
X
X
CiveularEval
Liu
0244
X
Choice Confusion
Yao
2024
BGR Shuffling
Lu et al.
2024a
X
X
Image Masking Option Shuffling
Song el al.
2025
X
X
Multi-modal Semantic Perturbation
Ours
Table I: Analysis of existing detection methods VLMs. We label the detection method with / it satisfies all Requirement [1 or and and with X otherwise. A indicates that the requirement is partially observed but not consistently with varying contamination settings. Most existing detection methods fail to meet the requirements and cannot accurately classify contaminated models. Our method, however, satisfies all requirements. Results for N-gram Accuracy, Shared Likelihood and Guided Prompting are in the Appendix
instruction-tuningg datasets. However, these training corpora are often proprietary with their exact composition undisclosed, a critical concern has emerged: public benchmark data may have leaked into the training set, leading inflated and misleading performance metrics.
Test-set leakage presents a practical and significant challenge. For model users, becomes difficult to disentangle genuine reasoning and generalization from mere memorization. For developers, exhaustively verifying the absence of test examples in massive pretraining corpora is prohibitively expensive (Bai et al. 2023). Although early works large models proposed decontamination steps by removing n-gram overlaps (Brown et al., 2020 Abdin et al. 2024a), many recent models do not report such procedures, leaving the extent of contamination largely unexamined.
To address this, several methods have been proposed to detect data contamination. One line of work focuses on verbatim memorization, testing whether model can reconstruct exact benchmark questions with high confidence (Xu et al., 22024) Golchin & Surdeanu 2024; Oren et al., (023). Another line of work examines generalization failures, measuring a model that succeeds on an original question fails on variants with similar or easier difficulty. which is interpreted as an evidence of memorization (Mizzadeh et al., 2024 Yao et al.
2024 Huung et al. 2025).
However, these detection methods were primarily designed for Large Language Models (LLMMs) and often overlook the unique, multi-modal nature of VLMs. Applying simple text-based perturbations to a VLM may not be sufficient, as the model could rely on visual features that remain unchanged. This discrepancy exposes a critical gap and raises a key question:
Is there a reliable, practical, and consistent method for detecting contamination in VLMs?
In this study, conduct systematic analysis of the data contamination problem, from which derive grounded definitions of the core requirements- - practicality, and consistency. Guided by these definitions, systematically contaminate open-source VLMMs under varying fine-tuning epochs, data composition, and training strategies (e.g., standard fine-tuning vs. LoRA (Hu et al. 2) Our results show that existing detection methods struggle with the complexities of VLMs. frequently failing meet the core requirements across diverse contamination scenarios (see Table
To overcome these limitations, we introduce a novel multi-modal semantic perturbation pipeline. Our method generates new test examples by subtly altering the semantics of the image while preserving its overall composition, thereby creating variants of comparable or lower difficulty (Figure T]. The core principle is that contaminated models, which have merely memorized an image- text pair, will fail to generalize to this perturbed input despite the similar lower difficulty of the questions. In contrast, clean models with genuine reasoning capabilities should perform correctly or even better. This approach enables robust contamination detection without requiring any ground- knowledge of the leaked data.
2

---

Published as a conference paper at ICLR 2026
Our contributions are threefold:
1. We propose novel and simple yet effective detection framework based multi-modal semantic perturbations, which effectively identifies contaminated models by testing for generalization failures in the visual domain.
2. We validate our method across multiple contamination settings, proving that it is reliable, practical and consistent, satisfying all key requirements for robust detection method.
3. We conduct the first systematic study of VLM behavior under diverse contamination and detection strategies, demonstrating that existing methods designed for LLMs are often unreliable for detecting contaminated VLMs..
INVESTIGATION SETUP
This section establishes the formal framework for our analysis of data contamination. We begin by defining the degree of contamination, state our core assumption about its relationship with model generalization, and finally, outline three essential requirements for robust detection method.
Our analysis is built upon a formal definition of contamination at the data-point level. For given data point x of dataset D and training process consisting of epochs, we define:
Definition 1. (Degree of Contamination). The degree of contamination for data point is:
degp (a:)
DED
This quantity reflects the total number of times r is seen during training. our experimental setup, where a model .M is fine-tuned on the entire benchmark dataset 'D for epochs, this simplifies to degp(M) n.
This definition motivates our central assumption, grounded in prior work on model memoriza- tion (Zhang al. 2017 Carlini et al. 2021 Kandpal et al., 2023):
Assumption Data points with a higher degree of contamination are more likely be memo- rized, which increases overfitting risk and impairs generalization.
In particular, we posit that the degree of contamination grows, models will exhibit degraded performance on perturbed or out-of-distribution variants benchmark items, even when the orig- inal questions are answered correctly. The effect need not scale linearly, since fine-tuning can disproportionately distort embedding spaces (Choi et al. 2025).
Based Assumption]] I we argue that any practical and effective detection method must satisfy three fundamental requirements:
Requirement 1. (Practicality). The method must operate without assuming any prior knowledge of clean model (e.g. the model's training corpus), relying only on black-box interactions.
Requirement (Reliability). The method must detect contaminated models across heteroge- neous fine-tuning strategies (e.g., standard fine-tuning vs. LoRA).
Requirement 3. (Consistency). The method's detection signal should be positively correlated with degree of contamination n = (MM.
Together, these requirements define a principled framework for evaluating contamination detection. A method that satisfies all of them practically flag contaminated models without the knowledge of clean model's ss behavior, remain agnostic to training specifics, and provide a signal proportional to the extent of contamination.

---

Published as a conference paper at ICLR 2026
3 PREPARATION OF CONTAMINATED MODELS
Models and benchmarks. To analyze contamination detection across diverse settings, we pair complementary model families and benchmarks. On the model side, we use LLaVA-v1.5-7B (Liu al., 2024a) a LLaMA-adapter design and Qwen2-VL-7B (Wang et al. 2024), which integrates tighter multimodal alignment. Both are widely adopted open-source VLMs with publicly documented training details (e.g.. released corpora or decontamination policies, though we do not rely on this for detection), making them suitable for controlled contamination experiments.
For benchmarks, we use MMStar (Chen al., 2024a) and RealWorldQA (xAI 2024), which consist of questions that strictly require visual and textual evidence. MMStar aggregates and filters prior multi-modal benchmarks by removing questions that are leaked or do not have visual dependence, while Real WeallQ enforces visual dependence by design. This prevents linguistic shortcuts and ensures our perturbation- based detector evaluates genuine multi-modal reasoning.
Training strategies. We contaminate models by training directly on evaluation data. Because VLM training typically has two stages- (i) large-scale pretraining of the vision encoder and language back- bone and (ii) instruction-tuned multimodal fine-tunigg-leakar can arise at either stage. Ablating pretraining is computationally prohibitive, so we focus on continual fine-tuning, which affords precise control over contamination levels.
We compare standard fine-tuning and LoRA (Hu et al. three common variants: fine-tune the LLM and adapter as in LLaVA-v1.5-7B; fine- tne-tune only the LLM as in Qwen2-VL-7B; diversity tests robustness across parameter-efficient and full fine-tuning regimes.
2021). For standard fine-tuning, we follow
VL-7B; or unfreeze all parameters as in InternVL. (Chen et al., 2024b). This
Epochs. We save checkpoints epochs 1-3 (i.e. (eg E (3),, since VLMs are typically fine-tuned for at most three epochs - and often just one. Contamination footprints of varying strength enables graded analysis of detection sensitivity (Liu et al.; 2024a Chen al. 2025).
Hyperparameters. We use model defaults, adjusting only the learning rate ensure inflated per- formance. For -...-1-1..--B. we follow the official repo; for Qwen2-VL-7B, we use LLaMA- Factory (Zheng et al. 2024) train the models. (Full settings are in the Appendix A.I)
MULTI-MODAL SEMANTIC PERTURBATION
We propose multi-modal semantic perturbation framework for contamination detection, that gener- ates variants of image-text pairs with similar or lower difficulty while keeping the original image composition intact. Our method consistently detects contaminated models and satisfies all three requirements, while existing methods fail to yield stable signal of test-set leakage in VLMs.
To generate semantically perturbed questions, we combine an LLM with a diffusion- gener- ative model. In our main experiments, we use GPT-4o (OpenAl 2024) and Flux (Labs) 2024) + ControlNet (Zhang et al., 2023), but later show that our framework is model-agnostic (Section
The pipeline is as follows. First, randomly change the answer of the original question different option, preventing contaminated models from getting away with memorized responses. Next, GPT-4o generates dense caption of the image, conditioned both on the original question and the newly chosen answer choice. We find this explicit conditioning essential: it ensures the caption highlights the salient visual features that drive the perturbation process, allowing the generation of questions of similar or lower difficulty (Section 6). Flux ControlNet then uses this caption, together with Canny edge maps (Canny) 1986), to guide the diffusion process. ControlNet preserves the global structure of the image while introducing new elements that minimally alter semantics, yielding an updated image with the randomly sampled different correct answer (Figure 2). We note that, due to limitations in rendering text or complex geometries, especially at low resolution, some generated images do not fully correspond to the new answer. To mitigate this, we filter generated pairs using a single criterion: perturbed questions must be answerable unambiguously For the main results, we apply manual
'Generation quality itself is not considered. This ensures that the evaluation set focuses solely on reasoning rather than visual fidelity. While our main results report -iltered outcomes, we show in Table 9 that this step can be automated. We stress that filtering is necessitated by current generative model limitations, not by our detection principle.
4

---

Published as a conference paper at ICLR 2026
filtering demonstrate the upper-bound performance of our approach. However, shown in Table this manual step can be replaced with automated filtering with a strong reasoning model.
To detect whether model is contaminated image-text pairs, we compare aggregate performance (i.c. accuracy) on the original vs. perturbed benchmarks, If the model fails generalize (i.e. it gets the original input correct but the perturbed one incorrect, leading to a lower performance on the perturbed benchmark), we flag contaminationf] i2) This because clean model with genuine reasoning capabilities should perform correctly in both settings, given that they have comparable difficulty, Critically, our approach enables robust contamination detection without requiring any foound-truthe knowledge of the clean model's behavior.
datasct-level, i.. has memorized the training
Curry Edge
Generate Dense Caption
LLM
Flux
Original Question
Perturbed Question
How from the camora is the loftmost vehioe??
How from the camera is the loftmost vehiole?
A.20m 20m
B. 60m
C. 100m
Randemtly Sample
20m
B. 60m
C. 100m
Figure 2: Illustration of our multi-modal semantic perturbation pipeline. The original question question-image pair is used to generate dense caption with LLM, which guides Flux ControlNet to produce perturbed image and new answer, yielding a modified but semantically consistent benchmark sample.
COMPARATIVE EVALUATION OF CONTAMINATION DETECTION
In this section, we present our main results. After perturbation and filtering, 440 image-question pairs remain from the original 765 in RealWorldQA, and 478 remain from 1,500 in MMStar (Section shows that these subsets remain representative of the full datasets). Table 2 reports results for MMStar; results for WorldQQ are in the Appendix (Table (3).
First, we see that clean models perform better than contaminated models, confirming that the perturbed questions are indeed of equal lower difficulty, and that we are testing against perturbations that the clean models are robust against. In contrast, all contaminated models perform worse, enabling our method reliably detect a simple check for performance drops.
We compare to the following contamination detection methods:
Multi-modal leakage (Chen et al. 2024a). With multi-modal leakage, contamination is flagged by measuring text-only performance on benchmarks that require visual input. Any boost in text-only performance after fine-tuning indicates memorization of leaked problems rather than genuine multi- modal reasoning. Multi-modal leakage requires clean models by design and hence does not satisfy Practicality (Req, [T. From Table we observe that Reliability (Req.2) is not satisfied since it fails to detect -LLVA-vv.5--1 trained for 3 epochs with standard fine-tuming While the performance gain positively correlates with the degree of contamination other cases, this breaks across benchmarks and training strategies such for Qwen2-VL.-3 trained for 3 epochs with LoRA and L.LVVAvv1.5-7BB trained for epochs with standard fine-tuning, only partially satisfying Consistency (Req. 3)
CircularEval (Liu al. 2024b). Selection bias in multiple-choice questions is mitigated by rotating answer options n times and requiring the model to be correct on all rotations. This stricter criterion lowers absolute accuracy compared to standard evaluation. For contamination detection, however, CircularEval's effectiveness is limited. Practicality (Req. [T fails, as CircularEval lacks a clear threshold pareshol-inndeeenden detection mechanism. Reliability (Req. 2 undermined by inconsistent contamination signals, as it fails to detect V...vA--v..-11 trained with LoRA for 2 and 3 epochs, and Qwen2-VL. VI--BB trained with standard fine-tuning for epochs. And despite some positive trends, Consistency (Req. 3 breaks across models and training setups, as shown with LLaVA-v1.. 11.5----5--7 trained with LoRA for 2, epochs and Qwen2-VL-7B trained with standard fine-tuning for 2.3 epochs, limiting its diagnostic value.
We make note that approach can also be used to detect contamination on sample-level by comparing the model's behavior the perturbed sample instead of the aggregate performance.
5

---

Full results of multi-modal semantic perturbation is listed Published as a conference paper at ICLR 2026
Method
Metric
LL.VA-<L.5-7B (clean)
LoRA (contaminated)
LLM+MLP (coalaminatd)
Require clean model?
Epech
Epoch
Epech 3
Epoch
Epuch
Epech
(Practicalityy (Ree, 11
MMStar MMStar P
37.78
52.53
50.71
54.34
41.82
48.89
50.71
Ours
69.29
44.24
37,58
38.18
33.33
37,57
36.97
No.
431.51
-8.29
-13.13
-16.16
-3.49
-11.22
-13.74
Success?
/
MMSttr MMStta C
37.78
52.33
50.71
41.82
48,89
50.71
CircularEval
26.05
29.09
45.66
55.56
25.86
22,83
22.02
Yes
-23.4
-5.05
+1.22
-15.96
-26.06
Success?
X
X
V
MMStar MMStar G
37.78
52.53
50.71
54.34
41.82
Cholce Confusion
71.92
53.54
68.66
69.00
62,83
66.26
62,83
No /
+14.95
+14.75
+21.01
+17.37
Success?
X
x
X
X
MMStar_ too
19.39
26.67
29.70
30.51
19:30
26.87
Multi-modal Leakage
+7.28
+10.31
+11.12
+0.41
+7.08
10.70
Yes X
Success?
Method
Metric
Owea2-VL--79 VL-7B (clean)
LoRA (contaminated)
LLM only (contaminated)
Require clean model?
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
(Practicality (Reg. 1
MMStar MMStar P
62.02
78:38
91.14
95.96
89.90
97.98
98.99
Ours
78.18
71.31
65.25
63.61
60.40
54,95
55.96
No
+16.16
-7.07
-78.89
-32.32
-20.50
-43.03
-33.3
Suceess?
MMStta
78.38
94.14
95.06
89.90
97.98
98.99
CircularEval
MMSttr C
55.96
54.95
55,56
55.76
82.63
91.92
92.73
Yes
23.43
-38.58
-40.20
Success?
MMStar MMStar G
78:38
91.14
95.96
Cholce Confusion
94.34
94.34
93.13
93.13
96 77
96.77
97.78
No.
+15.95
-1.01
-2.83
-1,21
Success?
X
/
X
MMStar
22.83
27.07
28.08
44.24
$1,31
Multi-modal Lenkage
+4.24
+5.65
+5.25
+21.41
+28.48
+29.0
Yes
Success?
Table 2: Performance of LLLVA--1.55-77 (top) and Qwen2-VL--7B (bottom) the MMStar dataset (Corresponding RealWorIdQA results are in the Appendix [T3]. We compare to "Multi-modal Leakage" Chen et al. (2024a), CircularEval (Liu et al. 2024b), and "Choice Confusion" Yao al. (2024). Clean models perform better on our perturbed dataset confirming that the perturbed questions are indeed contrast, all contaminated models perform worse, enabling reliable detection by our method via a simple check for performance drops. " "P" denotes the semantically perturbed version; "to" denotes text-only performance; "_C" denotes evaluation using circular options; "_G" denotes evaluation using choice confusion; A denotes the difference in performance, with positive values indicating gains. "Success?" indicates whether the method detected contamination. "Require clean model?" indicates whether the method requires access to clean model as baseline. If a clean model is required as reference, the method cannot be used to detect the reference models themselves, the entry is marked in Appendix F
of equal or lower difficulty.
Choice confusion Yao et al. 2024). Generalization is tested by constructing an easier benchmark variant: false options are replaced with correct answers drawn from unrelated questions. Clean models should leverage this simplification to improve, whereas contaminated models bound by memorized original answers might not. We apply this method to measure the model's performance drop (A) between the original and generalized versions. Clean models gain substantially on generalized benchmarks up to +34.04 on MMStar and +.0.30 on RQA - confirming their generalization ability. By contrast, contaminated variants show much smaller gains even losses. This failure to benefit from semantically irrelevant but easier choices reflects classic memorization-
contamination. As detection method, choice confusion meets Practicality (Req. [T] since the method detects models that perform worse on the generalized benchmark, but its sensitivity varies across fine-tuning regimes. Choice confusion fails to detect VA-V1-11.5-77 regardless of its training strategy or the number of epochs, and for LLa ---7 trained with LoRA, the model shows improved performance as the number of training epochs increases. Hence the method fails to satisfy Reliability (Req. 2 while Consistency (Req. 3) is partially observed in other training strategies.

---

Published as a conference paper at ICLR 2026
Importantly, unlike other approaches, our method requires no dataset-specific thresholds prior knowledge of leaked data, satisfying Practicality (Req. []. Moreover, the performance drop scales
with contamination degree, satisfying Consistency (Req. E) and persists across all training regimes
standard fine-tuning, LoRA, and full parameter unfreezing, satisfying Reliability (Req.2) 2
6 ANALYSIS OF MULTI-MODAL SEMANTIC PERTURBATION
Filtered images form a representative subsample. Since manual filtering removes a substantial portion of the original data, natural concern is whether the filtered sets remain representative. To test this, we first compare model performance on the full and filtered datasets and find that the results closely align (Table 3). This confirms that the filtering step does introduce systematic bias, and the remaining subsets preserve the distributional properties the original benchmarks.
Model
RQA (765 ings)
RQA_ filtered (440 imgs)
MMStar (1500 imgs)
MMStar. filtered (495 imgs)
LLaVA-v1.5-7B
49.01%
$2.05%
32.87%
37.78%
Qwen2-V1.-BB
70.33%
59.80
61.62%
Table 3: Performance of clean models on RealWorldQA (RQA) and MMStar before and after filtering.
Why perturbations yield a generalized benchmark. Our core assumption is that by preserving the original question and only altering the answer choice, the question difficulty remains comparable, and that clean models that answer the original question correctly will solve the variant. In addition, we often observe that the perturbation highlights salient visual cues more clearly than the original images (Fig.3k collectively yielding an alternate benchmark that similar or easier. We validate this empirically: clean models consistently achieve higher accuracy on perturbed benchmarks (Table 2
Question
What does the text on the traffic sign say?
A. Student
B. Children
C. Police
STUDENT
(a) Original
(b) After multi-modas semantic perturbation
Figure 3: Example where the perturbed variant is easier to solve than the original. In the original image, the traffic sign is small and the text barely legible; after perturbation, the sign is enlarged and clearly visible.
Providing the question-answer pair in caption generation critical. As described Sec. conditioning the caption generation both the original question, original answer and the new answer critical to creating a generalized benchmark. One natural approach is to create a variant of the original image from simply conditioning the generated caption on the question and the new correct answer. When we evaluated clean models on images generated from this version of the prompt, the clean model performance was much lower compared to both the original dataset and our final perturbed dataset, with invalid perturbations appearing much more frequently, for example, due to critical component of the image being left out so the question is ambiguous Intuitively, the captioning model first reasons about which parts of the image need to change to make the new answer correct, and produces a detailed caption emphasizing those changes. Flux+ControlNet then focuses on rendering these components exactly. Finally, changing the answer is necessary to delineate contaminated models' behavior from simply memorizing the answer.
longer solvable.
Failure modes of multi-modal semantic perturbation. There are cases where multi-modal per- turbation fails to reveal contamination. The perturbed image may differ its visual details that no longer closely resembles the original. In such cases, contaminated models may answer both the original and perturbed questions correctly as shown in Figure 4 hiding the contamination of the model. To ensure that such failure cases are rare, manually inspected the perturbed datasets and verified that only 8 out of 440 images (~1.8% from perturbed Real WorldWA and 17 out of 495 images (~3.44% from perturbed MMStar deviate from the original question's 's visual details.
7

---

Published as a conference paper at ICLR 2026
Question
Which vehicle is closer to the school bus the black SUV? A. School bus
B. Black SUV C. They are at the same distance.
(a) Original
(b) After multi-modal semantic perturbation
Figure 4: Example where contaminated model answers both the original and perturhed questions correctly. This may occur when visual details change significantly that the perturbed image longer
closely resembles the original.
Limitations of multi-modal semantic perturbation framework. For semantic perturbation to be valid, questions must enforce visual dependence. If a question can be answered without visual input, perturbing the image is meaningless, and altering only the answer invalidates the task. As such, restricted our study to Real RealWor and MMStar, which are VQA benchmarks with strict visual grounding. This constraint highlights an important boundary: our method is only effective when visual semantics directly determines the correct answer.
Both Real RealWo and MMStar are multiple-choice benchmarks. Although the multiple-choice format simplifies evaluation, our framework is not inherently tied to this setting. Once visual evidence perturbed, evaluation can be adapted free-form tasks using string matching, likelihood-baat scoring, or LLM-as-a-judge approaches. This principle extends naturally beyond multi-choice VQA, such as free-form QA settings, as shown Appendix F:7
Finally, manual filtering is required only because of current limitations of diffusion models, As generative models improve, we expect manual filtering to become unnecessary.
7 ABLATION STUDIES
We next validate that the core idea testing for memorization via mutli-moda semantic perturbationns robust to design choices. Specifically, we show our method is not tied to synthetic edits, generalizes to paraphrased data, scales larger models and alternate contamination regimes, and works without GPT-4o or manual filtering.
Real-world counterfactuals. NaturalBench (Li al. 2024) provides real counterfactual pairs photos of the same scene under altered conditions serving as a natural analogue to our synthetic variants. We fine-tune on one variant and evaluate on its paired counterfactual. Contaminated models drop sharply (up to 45.95%), while clean models remain comparatively stable (Table 6). showing our method generalizes beyond synthetic perturbation Thus, any reliable semantic variation natural, procedural, or synthetic- fits our framework. Full results are listed in F.2
Contamination with paraphrased data. simple but effective contamination is paraphrasing the data the model is contaminated on. To test whether models contaminated with paraphrased data can be detected, we paraphrase the questions using GPT-4o with minimal n-gram overlap. The contaminated models are then evaluated on the original benchmark and its perturbed variant. aan that contaminated models show inflated performance on original benchmarks with lowered performance on perturbed benchmarks, proving detection method is robust against paraphrasing.
Model scale: V1-13B. To assess scalability, we repeat the RealWorldQA experiment with -LaVA-v1.5--133 Table shows our detector remains effective at this scale. Given that larger models are more prone to memorization, these findings indicate the approach is suitable for stronger VLMs. Full results listed in F2
Test-set leakage during pretraining. We simulate pretraining leakage by mixing RealWorIdQA into the 665K Instruction-folloviin pretraining corpus and training VLaVA-vv.5577B for one epoch (Table 8). Using the same 440 filtered images, our method flags contamination, demonstrating applicability beyond fine-tuning-only settings.
Note that because counterfactuals are guaranteed he easier, clean-model deltas need not be positive.
8

---

Published as a conference paper at ICLR 2026
Method
Epoch
RQA
RQA_P
Method
Epoch
MMStar
MMStta P
Contamination paraphrased RealWVorddQA
Contauiination paraphrased MMStar
LLaVA=v1.5-7B (clean)
52.05
56.36
+4.31
LLaVA-v1.5-7B (cleun)
37.78
68.29
+31.51
52.03
40.00
-12.03
51.52
43.64
-7.88
LoRA (contaminated)
55.91
40.00
-15.91
LoRA (contaminated)
59.60
41.82
-17.78
59.09
38.86
-20.23
62.22
40.81
-21.41
56.14
-2.73
47.27
46.06
-1.13
LLM+MLP (contaminuted)
61 16
48.64
-12.72
LLM-MLP (cuntaminated)
57.60
53.54
-3.06
57.98
-4.4
Table 4: Performance of L.LVA-v1.55-1 and contaminated variants on RealWorldOA when trained on the paraphrased benchmark. " "_P" denotes the semantically perturbed version. For full results, refer to Appendix F5
Table Performance of LLaVA-v1.5-7B and con- taminated variants on MMStar when trained on the paraphrased benchmark. " "_P" denotes the seman- tically perturbed version. For full results, refer to Appendix
Method
Epoch
Train Set (%)
Set (%)
Method
Epoch
RQA
RQA_P
Contamination with NaturalBench
Contamination with RealWoridQA
<..-7. (clean)
65.63
65.89
+0.26
LLAVA-r1.5-133B (clean)
51.14
57.27
+6.13
74.32
38.18
-36.14
LoRA (contaminated)
2
89.79
$7.16
-32.63
LoRA (contaminated)
73.18
32.73
-00.5
3
91.11
57.32
-33.79
77.05
34.77
-42.28
1
79.95
58.79
-21.16
56.59
37.50
-99.09
LLM+MLP (costaminated)
97 05
54.32
-42.74
LLM-MLP (contaminated)
71.59
38.86
-32.73
98.63
53.05
-45.58
75.45
37.27
-38.18
Table 6: Performance of clean and contaminated mod- els NaturalBench. While clean model shows com- parable performance on the test set, contaminated models fail generalize, with performance drop upto -4..5%%. For full results, refer to Appendix F.2
Table 7: Performance of clean and contami- nated L. VA-vI..--13 on RealWorldQA af- ter multi-modal semantic perturbation. The result satisfies all three requirements. For full results, refer to Appendix F.2
Test-set leakage with mixture of benchmarks We test perturbation pipeline under more realistic contamination scenario: a mixture of other benchmarks. We fine-tune the models with popular multi-modal benchmarks: Math Vista Lu al. (2024b), MMMU Yue al. (2024), MMBench Liu ct al. 2024b) and CV-Bench Tong et al. (2024), resulting in 11,280 image-question pairs, with RealWorldQA and MMStar now consisting only of ~6.7% and 13.3%, respectively. We still observe a clean and consistent detection across all contaminated models, demonstrating that our perturbation pipeline can reliably detect contaminated models even when the contamination signal is weaker. Results are in Appendix [.6 due to limited space.
Finally, we ablate two key components of our pipeline show modularity of our pipeline.
Automation of filtering process. We replace manual validation with the o3 model to filter generated pairs. The automatic pass removes 471 items and retains 294, of which 253 overlap with the manually kept set, indicating high agreement (Table 9.. The exact prompt is provided in Appendix B and full results are listed
Caption generation with Molmo-7B-D. To decouple the pipeline from GPT-4o, we substitute captioning with the lightweight open-source Molmo-7B-D (Deitke et al., 2024). After manual filtering. this variant yields 398 valid pairs and preserves the same detection trends, underscoring the flexibility of our approach (Table IOP. Full results are listed in F.4
RELATED WORK
Data contamination in LLMs. Brown et al. (0220) was among the first to highlight the problem of data contamination when training models on internet-scale corpora, proposing an n-gram over- lap lap-based decontamination technique that was later adopted in Bai et al. (2023). Building on this, several detection methods exploit verbatim memorization to identify leaked test-set examples (Oren
et al. 2023 Golchin & Surdeanu 2024 Xu al.) 2024). More specifically, Xu et al. (2024); Golchin Surdeanu (2024) test whether the model can accurately reconstruct masked spans of test questions, while Oren al. (2023) measure the probabbiiiy of the original ordering of multiple-choice options. Choi et al. (2025) instead examine embedding divergence after fine-tuning, exploiting the observation that embeddings of unseen samples change more substantially than those of memorized

---

Published as a conference paper at ICLR 2026
Model
RQA
RQA_P
A
LLa --5-7 (clean)
52.05
56,36
+4.31
Pretrain (contaminated)
51.82
50.00
-1.82
Table 8: Performance of clean vs. pretrained-contaminacee models on RealWorldQA (440 images).
Method
Epoch
RQA
RQA_P
Method
Epoch
RQA
RQA_P
Contamination RealWorldQA
Conlaminationn with RealWoridQA
LLaVA-v1.5-7B
LLaVA-v1.5-BB (clean)
44.22
50.34
44,90
-5.44
44.97
35,93
9.04
LoRA (contaminated)
2
66.33
46.60
-19.73
LoRA (contaminated)
2
55.78
32.66
-23.12
77.21
47.28
-29.93
70.35 3
32.91
-37.44
-6.12
56.53
39.95
LLM+MLP (contaminuted)
63.95
55,10
-8.85
LIM+MLP (contaminuted)
2
60.80
39.45
-21.35
64.63
53.40
-11.23
61.56
39.95
-11.61
Table 9: Performance after filtering with the o3 model, resulting in 294 valid pairs. Results satisfy all requirements. For full results, refer to Appendix F33
Table 10:
Performance after generating and fil- tering images with captions from Molmo-7B-DD resulting in 398 valid image-question pairs. Re- sults satisfy requirements. For full results, refer to Appendix F.4
ones. However, these methods face key limitations: Oren al. (2023): Golchin & Surdeanu 2024
Xu et al. (2024) perform poorly on contaminated VLMs, while Mahlle/Chor et al. (2025) requires ground- truth clean model behavior to establish a detection threshold, reducing its practicality.
Generalized benchmarks for detecting contamination. Yao et al. (2024) tests for generalization
rather than memorization. They construct trivial variants of benchmark questions by replacing
incorrect multiple-choice options with irrelevant answers. Contaminated models often fail on the
easier variants, exposing strong memorization. However, their setup involves training for 36 epochs on single benchmark an unrealistic scenario for modern large-scale training. Similarly, GSM- Symbolic and Perturb (Mirzadeh et al.) 2024 Huang et al. 2025) introduce perturbation- approaches, measuring performance drops variant questions contamination signals.
Contamination detection in VLMs. VLMs differ from LLMs due to their molttimoodal inputs and multi-stage alignment training, which yield distinct contamination dynamics. Lu al. 2024a)
proposed shuffling BGR color channels to mitigate spurious cues, while Song et al. (2025) introduced image masking and option shuffling test robustness. Yet these approaches struggle to reliably detect contaminated VLMs, as observed performance drops may arise from confounding factors such visual artifacts biased sampling rather than true memorization. In contrast, our framework provides consistent detection across diverse fine-tuning regimes, requires no access to leaked data, and yields performance drops that correlate strongly with the degree of contamination.
CONCLUSION
Recent advances in Vision-Language Models (VLMs) have raised concerns about inflated benchmark performance due to test-set leakage from large-scale, proprietary training corpora. To overcome the limitations of existing detection methods, we introduce a novel approach based on multi-modal semantic perturbation. By deliberately contaminating open-source VLMMs and evaluating their generalization behavior, we show that our method consistently identifies contamination where prior approaches either fail or yield inconsistent signals. These results establish multi-modal semantic perturbation simple and reliable framework for detecting test-set leakage in VLMs.
Broader impacts. Our multi modal semantic perturbation method aims to uncover and quantify the degree of data contamination in VLMs, promoting cleaner training pipelines and more trustworthy models. By systematically characterizing contamination behaviors, lays the groundwork for robust model evaluation and contamination detection. While these insights could inform adversaries seeking subtler contamination schemes, we believe this work will foster the development of stronger defenses and decontamination strategies.
Reproducibility statement. We publicly release our code, models and data.
LLM usage. Gemini and ChatGPT were used to polish some of the paper's writing.
10

---

Published as a conference paper at ICLR 2026
Acknowledgement. This work was supported in part by NSF IIS2404180, Microsoft Accelerate Foun- dation Models Research Program, Institute of pommaa & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2022-0-0087/ Development of Al Autonomy and Knowledge Enhancement for Al Agent Collahoration). (No. RS-2002-00018723 Development of Large Korean Language Model Technology for Efficient Pre-training), and by the Institute of Information & Communications Technology Planning & Evaluatioon(ITP)) grant funded by the Korea goVernment(MSIT) (RS-2025-02219317,, AI Star Fellowship(Kookmin University). The authors would like to thank SeungEun Chung, YoHan Ban, Samuel Low Yu Hang and Junxia Cui for assistance with experiments and Anirudh Sundara Rajan and Zhuoran Yu for helpful discussion.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee. Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024a. URL https::////. / http//rxxvv.rrg//a/a// 2atpp//2222009993333
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 2024b.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL https: //aa:////xvvvv/ o abs 2309. 16609
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voobe, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https
http:/vv..rga/b/////aa0/ 14165
John Canny. A computational approach to edge detection. TPAMI, PAMI-8():679-698, 1986. doi: 00.109/TAAMM..1188..7400
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 21), pp. 2633-2650. USENIX Association, August 2021. ISBN 978-1- 939133-24-3. Utthttts:://wet http://wwwwwww...tt//oo http://www......cooooooeeeeeee/e httpr//nceeeeeeeeeeeeerrrrreeee
Lin Chen, Jinsong Li. Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In A. Globerson, Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp. 27056-27087. Curran Associates, Inc., 2024a. URL. httpp:://proceeeeeet
nwwrips.u...em wwtp:pppppppppppppppppp wttp:/flllssspa///2//2222/ File/ 3aa.::/eeeeaaaaaaacaaeeeaaaaaeaa
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo. Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- Songul-iinguistic tasks, 2024b. URL https: / www.iaivv... wtt./oog///.// 14238

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu. Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo. Jiahao Wang. Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng. Jiaye Ge, Kai Chen, Kaipeng Zhang. Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL https:
//arxiv. org. abs/ 2412.4222222 05271
Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li. How contaminated is your benchmark? quantifying dataset leakage in large language models with kernel divergence, 2025.
http:/tss.aa.xi.org///////// absp://bb//25022. 00678
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsherg. Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng. Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of of-et------arrr vision-language models, 2024. httpst
org/ abs 2409. 17146
Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models, 2024. URL
2308. 08493
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https 7/ wwwriiiorg//bbaa 2106. 09685
Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge. Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, and Mengdi Wang. Math-perturbs Benchmarking Ilms' math reasoning abilities against hard perturbations, 2025. URL httpt:ttt/s:atttii org/ abs/ 2002/5022..0 06453
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15696 15707. PMLR, 23-29 Jul 2023. URL htttps://proceeeee
mlr
pyees/wv0D//aangpa223...1
Black Forest Labs. Flux. httpt::::////. ptttps://githale htwp://aa-o-fforeet-laa... http://tgo..gg.t......oooooooeaa
2024.
Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision- language models on natural adversarial samples. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction 03744
tuning, 2024a. URL httpt::////t http///axxvvv///
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer. 2024b.
Hongyuan Lu, Shujie Miao, and Wai Lam. Clean evaluations on contaminated visual language models, 2024a. URL
07030
12

---

Published as a conference paper at ICLR 2026
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wos Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICIR), 2024b.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. fis-symbollic Understanding the limitations of mathematical reasoning in large language models, 2024. URL httptttp/////xxv//////////////// 05229
OpenAI. GPT-4o technical httpss///opoaaa.cc///essssacc cessed: 2025-05-16.
httpp::/
40 2024. Ac-
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black box language models, 2023. URL
27603
Dingjie Song, Sicheng Lai, Shunian Chen, Lichao Sun, and Benyou Wang. Both text and images leaked! a systematic analysis of multimodal IIm data contamination, 2025. URL https://
arxiv.org/ www.o/gggbbbbbbe at////bs/24111 /411 03823
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms, 2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang. Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024. URL https: /ttp:v/aaiivvvv/.v erq/abs/2409 12191
XAI. Grok-1.5 vision preview. https: bttps://x.aa//loo///gzz
2024.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models, 2024. URL
18824
Feng Yao, Yufan Zhuang. Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. Data contami- nation can cross language barriers, 2024. URL. htt.:/ttt////xtiiiei wwtp://...rraaa///////////
13236
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun. Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
n 2ttee//orrmmmml5Syy0022
Jianrui Zhang. Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic lingutiin compositional reasoning via counterfactual examples. Findings of the Association for Computational Linguistics: ACL 2024, 2024a.
L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. 2023.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does our multi-modal I1m truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL htt.tttpp////ix..ooo http://b.///.2... 13372
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENT SETTINGS
In this section, we detail all hyperparameter settings for contaminating models and for multi-modal semantic perturbation of benchmarks.
A.1 MODEL TRAINING
We contaminate LLaVA-v1.5 1---7B by following the official repository and Owen2-VL.-7hh using LLLMA-Factory. All settings remain identical except for the learning rates, which were tuned over the ranges shown in Table 11
L.LVVAvvl.-7BB
Qwen2-VI-7B-Instruut
Hyperparameter
Standard
LoRA
Standard
LoRA
Learning rate
5e-6, le-5, 2e-5
2e-4, 4, le-4
le-5,
2e-5,
5e-6
le-4,
le-3
Adapter
5e-6
20-5
1c-6
1e-6
Effective batch size*
128
128
16
64
LoRA rank
Table 11: Hyperparameter settings for LLa and Qwen2- Drren2-VL--BBB under standard fine-tuning and LoRA. "Effective batch size accounts for gradient accumulation across 8 GPUs.
We perform continual fine-tuning of the model weights NVIDIA A6000 GPUs. Notice Qwen2- VI. models are trained with much smaller batch sizes due to heavier GPU VRAM requirements.
A.2 MULTI-MODAL SEMANTIC PERTURBATION
Multi-modal semantic perturbation is two-stage process. In the first process where obtain prompts that will be provided to Flux Controlnet model is generated by using GPT- 4o with api-version-2022 001-00000-----eiee We set temperature-0.3 and limit max_tokens=800.
In the second stage where we create perturbed version of the original image based on the caption generated in the first stage, we utilize Controlnet trained with Flux diffusion model We follow the default hyperparameter settings in the repository, except that we enforce the generated images to have the same resolution as the original images. All images are generated with 25 steps.
A.3 COST OF MULTI-MODAL SEMANTIC PERTURBATION
First, we note that our semantic perturbation pipeline only needs to be run once to constructed the perturbed benchmark. We also note that the cost scales linearly with the size of the dataset. We list below the cost and hardware requirements of each stage our pipeline.
Caption generation. Using GPT-4o the captioning model required on average ~55 imput tokens and 125 output tokens per image. This means that generating 1 1,000 captions would cost less than $1.50 (USD). This stage also does require proprietary models: it can be replaced with Molmo-7B-D (Table I0 23 24). which fits on single 24GB GPU.
Flux+ControlNet. Flux+ControlNet generation runs single 40GB GPU. With 25 sampling steps, each image takes around 15 20 seconds to generate, and this can be further reduced by decreasing the number of steps. This stage is trivially parallelizable across multiple GPUs or machines if higher throughput is needed.
Automated filtering with o3. Automated filtering required average ~55 input tokens and 200 output tokens per image. Generating 1,000 filtering decisions costs less than $2.00 (USD).
*The default training script can be found here: http.///igg///ghhbooooooooonoaan
Coode he accessed here: http:://gitth..om//llaaaa/////-t
14

---

Published as a conference paper at ICLR 2026
Overall, the total cost to construct the perturbed benchmark is cheap with modest hardware require- ments. Our approach scales linearly with dataset size, making our approach practical and scalable for reasonably large benchmarks. Finally, when real-world counterfactuals are already available as part of the benchmark design (e.g., NaturalBench in Table 20), we do not need to generate perturbations all.
B SYSTEM PROMPTS TO GPT-40 AND o3
To generate the semantically perturbed question, utilize GPT-4o OpenAl (2024) and Flux diffusion model Labs (022) trained on ControlNet Zhang et (2023). We generate detailed caption about the image using GPT-4o along with the original question, original answer and the new correct answer with the following system prompt:
Your job is to generate text-to-image prompt that can used with diffusion model. Based on the question and answer, write a detailed caption so that all necessary details are included and the question remains solvable.
Additionally, modify the image so that the correct answer changes. For example, if the question asks "How many people are in the image?", change the image to have more people.
In Section 7 we demonstrated that the manual filtering process can be automated with o3 model which is a powerful reasoning model. We utilize the following system prompt to generate model responses that can be used to filter the invalid perturbed images.
You be given a question and image pair, along with answer. Your job critically analyze the Emage-question pair to verify that the question can be correctly answered.
particular, ensure that one can deduce the correct answer choice and that choice only. If there is any ambiguiry, you must reject this question. When finalizing your decision, do
NOT take into consideration quality of the image, As long as the question remains solvable, you should keep it. Provide your answer in the following format: "Answer:
ANSWER" and answer with KEEP or REJECT.
C PERPLEXITY-BASER METHODS
We evaluate three perplexity-baseee based methods: Shared Likelihood (SL) (Oren et al. 2023). which compares log- likelihoods on original versus shuffled inputs; Guided Prompting (GP) (Golchin & Surdeaau 2024). which uses an LLM to score model completions against the original; and N- A-gram Accuracy (NG) (Xu al. 2024), which masks answer choices and checks reproduction. Each probes whether models memorize test samples through partial or perturbed inputs in the text domain.
Results Table 12 show that SL, GP, and NG fail to meet any of requirements. All three depend reference values from clean models, violating Requirement] [ Contaminated models often display counter-intuitive trends, such as improved scores with greater contamination, violating Requirement Finally, none satisfy Requirement 3 for instance, SL p-values increase rather than decrease as contamination intensifies, directly contradicting expectation.
D DETECTING CONTAMINATED MODELS WITH MULTI-MODAL. SEMANTIC PERTURBATION ON REALWORLDQA
Table 1 reports results for RealWorldQA. Similar to results MMStar, find that (i) clean models consistently outperform contaminated ones, confirming that perturbed questions are not harder; (ii) in contrast, contaminated models show clear performance drops, enabling reliable detection without thresholds or prior knowledge (Requirement[] T]. The drop scales with contamination level (Requirement) and holds across training strategies (Requirement)
15

---

Published as a conference paper at ICLR 2026
RealWorldQA
MMStar
Method
Epoch
SL (p-va(())
GP (80())
NG (%)(()
SL (p-vali()) GP (SxX)) NG (#X()
LLaYA
-v1.57B
0.926
25.67
0.201
2.53
14.54
0.927
24.94
0.088
0.60
5.63
LoRA
2
0,933
4.44
26.32
0.066
0.80
3.27
3
0.935
7.19
25.12
0.072
2.92
LLM
0.207
+MLP
2
0.929
6.67
26.93
0.219
0.44
8.63
0.932
5.75
26.82
0.238
0.13
3.12
Qwen2
-LL-7B
0,166
38.43
13.10
0.006
13.80
3.08
0.333
46.27
13.88
0.009
3.07
7.67
LoRA
2
0.166
49.80 99.80
14.51
0.004
3.00
8.26
3
0.323
47.06
14.54
0,000
1.00
7.96
0.124
14.64
15.48
0.005
1.13
2.47
LLM
0.070
9.54
16.08
0.004
1.20
3.09
3
0.165
11.63
16.97
0.005
1.00
7.59
Table 12: Perplexity-based baselines (SL shared_likeliteeeee likelihood, GP guided prompting, NG n-gram) for RealWorldQA and MMStar. MMBench was left out due to limited compute. To clarify, the models trained with the dataset they are being evaluated on. There is no clear signal that distinguishes clean and contaminated models.
E EXTENDED EVALUATION ON ADDITIONAL OPEN-SOURCE AND PROPRIETARY MODELS
We further apply our pipeline to GPT-4o (OpenAl 2024). Gemini-2.0-Fllash (Gemini Team 2024), Phi-3,5-V (Abdin et al., 2024b), and InternVL-2.5 CCen et al. 2025). We do not contaminate these models. Although data, we assume they are uncontaminated. Under this assumption, if our framework is sound, their performance should remain consistent across original and perturbed datasets. Indeed, results Table [4 confirm this expectation. Across all models, perturbed performance exceeds original performance, reinforeing that clean models generalize while contaminated 3.5-V exhibits the largest gain, while Intern VL-2.5-8B remains relatively flat, demonstrating that our metric robust across architectures. This consistency underscores the effectiveness of multi-modal semantic perturbation detection framework.
cannot guarantee that these models have never encountered our evaluation
fail. Notably, Phi-
F FULL EXPERIMENT RESULTS
In this section, we provide the full experiment results of multi-modal semantic perturbation across four training strategies that were omitted due limited space.
F.1 FULL RESULTS ON LLAVA--1.5-T AND OWEN2-VIL-78
Table 15
17 and 18 list out all results for four varying training strategies for LLaVA-v1.5-7B B. including the non-default training strategies for each model which were omitted due to limited space. Note that the results here computed 440 manually filtered images. Our approach detects contaminated models regardless the training strategies and satisfies all three requirements.
16
and Qwen2 Owen2-VL-77B VL
F.2 FULL RESULTS WITH LLAVA-VI.5-1 AND NATURALBENCH
Table 19 lists full results for 1-1-1.55-13BB model trained on Real Note that the results here are computed on 440 manually filtered images. 20 full results for LLaVA-v1.5-7B trained one variant of NaturalBench and tested on the counterfactual version. The clean and consistent detection of contaminated models show that our approach can be applied to models of perturbations.
larger scale and to
16

---

Published as a conference paper at ICLR 2026
Method
Metric
LLaVA-r1.5-7B (clean)
LoRA (contaminated)
LLM-MLP (oontaminated)
Require clean model?
Epoch
Epoch
Epoch
Epoch
Epoch
Epech
(Practicalitt (Reg
ROA
52,05
62.73
70.00
79.55
56.36
64.55
70.68
RQA
$1,36
44.77
$2.95
$1.82
No
à
+4,31
-11.37
-17.27
-34.78
-3,41
-14.10
-18.66
Sucvess?
ROA
$2.05
62.73
70.00
79.55
56.36
64.55
70.68
CircularEvall
RQA_C à -15446
57.50
43:40
Yes X
-44.78
-24.77
-22.05
-23.18
-27.50
-27.28
Success?
RQA
52,05
62.75
70.00
79.55
56.36
64.55
70.68
Choice Confusion
RQA_G A
66.59
41.36
59.09
66. 14
62.95
62.50
63.64
No
414,54
-21.37
-10.91
-15.1
46.59
-2.05
-7.04
Succes?
RQA_te
34.77
37.27
47.05
48.41
37.73
10.68
45.00
Mulli-modal Leskage
+2.50
+12.28
+13.64
+2.96
+5.91
+10.23
Yes
Success?
Method
Metric
Qwen2-VL--7 VB (clean)
LoRA (contaminate))
LLM only (contaminuted)
Require clean model?
Eguch
Epoch
Epoch
Eguch
Epoch
Epoch
(Pructiccality (Req Th
ROA
70.45
79.32
87.50
$8.86
74.77
78.64
85.23
61.59
Ours
A
+0.91
-14,32
-22.95
-27.27
-27.95
-28.19
-38.64
No
RQA
70.45
79.32
87.50
88.86
74.77
78.64
85.23
ClecularRval
RQA
65.00
65.91
65.23
66.59
77.73
Yes X
-6.81
-14.32
-21.59
-21.81
-9.54
-12.05
-7.50
Succes?
RQA
70:45
79.32
87.50
58. 86
74.77
78.64
85.23
Choice Confinsion
RQA_G
91.36
91.59
92.05
92.50
86.14
89.77
95.23
No
+20.91
+12.27
+4.55
+3.64
+11.37
+11.13
+10.00
Success?
X
X
42.27
62.50
Multi-modal Leakage
+1.13
+5.00
+6.13
+16.59
-7.73
+26.36
Yes X
Table 13: Performance of VL- (top) and Qwen2-VL.. VL-7B (bottom) on the Real A dataset. We compare to MMulti-modal Leakage" Chen et al. (2024a), CircularEval (Liu et al.) [002bb). and "Choice Confusion" Yao et al. (2024). Clean models perform better - confirming that the perturbed questions are indeed of equal or lower difficulty. In contrast, all contaminated models perform worse, enabling reliable detection by our method via simple check for performance drops. RQA denotes WorldQQ and "_P" denotes the semantically perturbed version; "to" denotes text-only performance; " "_C" denotes evaluation using circular options; " "G"" denotes evaluation using choice confusion; A denotes the difference in performance, with positive values indicating gains. "Success?" indicates whether the method detected contamination. "Require clean model?" indicates whether the method requires access to a clean model baseline. If clean model is required as reference, the method cannot be used to detect the reference models themselves, so the entry is marked as "_" For full results, refer to Appendix F
Chhnn
Model
RQA
RQA_P
Gemini-220-FFlan
68.37
71.59
+3.22
GPT-4o
65.68
69.93
+4,25
Phi3.5-Visii
52.68
68.86
+16.18
VL-25-L
64.05
64.77
+0.72
Table 14: Accuracy on the original vs. perturbed datasets for open-source and proprietary models. A indicates the accuracy change.
F.3 FULL RESULTS WITH o3 FILTERING
Table 21 and 22 list out all results for contamination detection results after filtering the perturbed images with o3 model. To clarify, filtering with o3 model results in 294 images. Our approach still detects contaminated models and satisfies all three requirements, proving that our pipeline design is modular and can be automated.
17

---

Published as a conference paper at ICLR 2026
Model
Epoch
RQA (%)
RQA_P (%)
Model
Epoch
RQA (%)
RQA P (())
A
L.5VA-11.5-BB
$2.05
$6.36
+4.31
Qwen2-VL-7B
70.45
71.36
+0.91
Contamination with RealWorI
Contamination with RealWorIdQA
62.73
51.36
-11.37
1
79.32
65.00
-14.32
LoRA
2
70,00
52.73
-17.27
LoRA
2
87.50
64.55
-22.55
3
79.55
44.77
-34.78
3 88.86
61.59
-27.27
58.41
51.59
-6.82
1
74.77
46.82
27.95
LLM only
2
63.64
80.68
-12.96
LLM only
2
78.64
50.45
-2819
3
70.91
47.05
-33.66
3
85.23
46.59
-38.64
56.36
52.95
33.1
1
75.23
57.95
-17.28
LLM+MLP
2
64.55
50.45
-14.10
LLM+MLP
2
88.18
49.77
-38.41
3
70.68
51.82
-1888
3
93.18
47.50
-45.68
56.36
54.32
2.04
1 74.77
39.09
-35,68
ALL
2
64.77
52.73
-12.04
ALL
2 78.18
45.91
-32.27
70.23
52.05
-18.18
3
87.50
40.45
44.005
Table 15: Performance of clean and contami- nated LLaVA-v1.5-7B models on RealWorldQA. P" denotes the semantically perturbed version.
Table 16: Performance of clean and contami- nated Ww-n2-VL-7BB models Real P denotes the semantically perturbed version.
All three requirements are satisfied.
All three requirements satisfied.
Model
Epoch
RQA (%)
RQA_ ((A_P(S)) (%)
Method
Epoch
ROA (%)
RQA_ (%)
A
LLaVA-11.-77B
+31.51
Qwen2- V1-77B
62.02
78. 18
+16.16 16
Contamination with MMStar
Contamination with MMStar
$2.53
44.24
-8.29
77.37
73.33
-4.04
LoRA
2
50.71
37.58
-13.13
LoRA
2
87.88
68.48
-19.40
54.34
91.52
67.47
24.05
44.85
37.98
-6.87
80.20
50.71
-29.49
LLM only
48.48
38.99
9.49
LLM only
94.95
52.32
-2.63
16.56
39.39
32.12
-7.27
-27.47
49.70
-11.32
LLM+MLP
97.98
51.11
-46.77
41.82
33.33
-8.49
37.37
-11.52
ALL
93.13
43.64
49.49
50.71
36.97
-13.74
3
96.77
44.04
-52.73
Table 17: Performance of clean and contami- nated LLa models MMStar. "_P "_P" denotes the semantically perturbed version. All three requirements are satisfied.
Table Performance of clean and contami- nated Qwen2-VL--7B models on MMStar. " "_P" denotes the semantically perturbed version. All three requirements are satisfied.
F.4 FULL RESULTS WITH MOLMO-7B-D AS CAPTIONING MODEL
Table 23 and 24 list results for LLaVA-v1.5-7B and Qwen2-VL. VL-7B evaluated perturbed images generated from Molmo-7B-D captions. Note that this pipeline yields 398 valid image-question pairs after manual filtering. Both results show clean detection trends, satisfying all requirements.
18

---

Published as a conference paper at ICLR 2026
Model
Epoch
RQA (%)
RQA P (%)
A
Model
Epoch
Train (%)
Test (%)
A
LLaVA-vL.S-3BB
51.14
57.27
+6.13
L.aVA-v1.5-7B
65.63
65.89
+0.26
Contamination with RealWorIdQA
Contamination with NaturalBench
1 74.32
38.18
-36.14
81.53
61.37
20.16
LoRA
2 73.18
32.73
-40.45
LoRA
2
89.79
$7.16
-32.63
3 77.05
34.77
-42.8
91.11
57.32
-33.79
1
57.05
44.77
-12.28
77.63
61.95
-15.68
LLM only
2
68.18
44.32
-23.66
LLM only
2
88. 11
57.89
-30.21
3
68.64
43.86
-24.78
90.58
57.32
-33.26
1
56.59
37.50
-19.09
79.95
58.79
211.1
LLM+MLP
2 71.59
38.86
-32.33
LLM+MLP
97.05
54.32
-42.44
3
75.45
37.27
-38.18
98.63
53.05
-45.58
1 $7.73
44.55
-13.18
81.84
58.42
-23,42
ALL
2 68.41
44.32
-24.09
ALL
97.05
54.47
-42.58
3
69.09
43.41
-25.68
98.63
52.68
45.95
Table 19: Performance of LLa mod- els the RealWorldQA benchmark and its semanti- cally perturbed version consisting of 440 image-question pairs. All three requirments are satisfied.
Table 20: Performance of clean and contam- inated LLaVA-v1.5-7B models on Natural- Bench. Test set denotes natural counter- factual version of the train set. Clean model maintains a similar performance while contam- inated models drop in performance for upto 45.95%
Epoch RQA (%)
((S)
Epoch RQA (%)
RQA_
Contomiiaation with RealWoridQA
Qwen2-VL Contomination with RealWorldQA
59.86
74.15
50.34
44,90
-5.44
77.21
74,83
-2.38
LoRA (contaminated)
-19.73
LeRA (contaminated)
78.91
74.15
-476
77.21
47.28
-29.93
79.59
74.15
-5.44
39.86
56,80
-3.06
79.25
56,12
-23.1
(contaminnatdd)
63.95
55.10
-8.85
(contaminattdd)
85.37
54.42
30.95
68.71
50.00
-18.71
90.48
$5.44
-35.4
-4.08
77.89
LLM+MLP (contuminated)
60.54
54.42
-6.12
LLM+MLP (contaminated)
87.41
$7.48
29.93
-30.62
59.86
56.46
-3.40
80.27
58.50
21.77
ALL (cuntaminated)
64.63
53.40
-11.23
ALL (contaminated)
90.82
53.40
33.42
69.05
50.34
-18.71
92.52
52.04
10.48
Table 21: Performance of clean and contami- nated ALLaVA-v1.5-70 models RealWorldQA. "P" denotes the semantically perturbed version. Note that accuracies are measured on 294 filtered images. All three requirements are satisfied.
Table 22: Performance of clean and contami-
nated QwL72-VL-7B models on RealWorldQA. p" denotes the semantically perturbed version. Note that accuracies are measured on 294 filtered images. All three requirements are satisfied.
F.5
FULL RESULTS WITH CONTAMINATION WITH PARAPHRASED DATA
Table 25 and 26 list full results for LLaVA-v1.5-7B and Qwen2-VL-P7 models on RealWorI
and table 27 and 28 on MMStar. To clarify, the models trained with a paraphrased version of the original benchmark, and are evaluated on the original benchmark.
More concretely, we prompt GPT-4o generate three possible paraphrases of the original question, and select the one with the lowest 5-gram overlap with the original question, enforcing the model to use different sentence structures and vocabularies.
Furthermore, we observe that models contaminated on the paraphrased version of the questions show inflated performance on the original benchmark, validating our paraphrasing pipeline. The clean and consistent detection proves that our perturbation pipeline is robust against paraphrasing, which is challenging setting for contamination detection.
19

---

Published as a conference paper at ICLR 2026
Model
Coulig RQA ROA_P(S) (%)
Model
Coulig RQA(S) RQA_P(S) (%)
Custamination with RealWoridQA
Confamination with RealWorldQA
(clean)
Qwen2-VI. (elean)
44.97
35.93
-9.04
63.57
59.05
-452
LoRA (contaminated)
LoRA (contaminated)
70.35
32.91
-37.44
68.84
52.26
-16.88
52.76
41,96
-10.80
78.14
40.70
-37,44
LLM (contaminated)
53,02
43.97
-9.05
LLM (contaminated)
84.67
32.16
-22.51
53.52
43.72
-9.80
87.19
35:43
-51.6
-16.58
LLM+MLP (contaminated)
60.80
39.45
-21.35
LLM+MLP (contaminated)
83.42
34.67
48.75
-99225
62.81
35.43
-27.38
76.38
39.95
36.43
ALL (contaminated)
61.81
38.93
-25.88
ALL (contominated)
88.94
32.41
-66.3
62.56
35.68
-26.88
90.20
31.41
58.79
Table 23: Performance of clean and contami- nated -L..5- models on RealWorldQA. P"" denotes the semantically perturbed version. Note that accuracies measured 398 filtered images. All three requirements are satisfied.
Table 24: Performance of clean and contami-
nated Twee2--L--77 models RealWorIQQA. P" denotes the semantically perturbed version. Note that accuracies are measured 398 filtered images. All three requirements are satisfied.
Epoch RQA
ROA
Epoch RQA (%)
ROA ((S)
Contaminattion with paraphrased RealWorldQA
Contansination with paraphrased RealWoridQA
(clean)
Qwen2-V (clean)
$2.03
40,00
-12.33
72,50
71.59
-0.91
(conaaminate)
LoRA (contaminuted)
59.09
38,86
-20.23
75.23
69.32
-5.91
24.55
LLM (contuminated)
2
60.91
29.09
-11.82
LLM (contaminated)
87.05
56.59
-90.46
63.64
50.45
-13.19
89.55
54,32
-35.23
56.14
$3.41
-2.73
80.00
52.50
-27.50
LLM+MEP (oontaminnted)
61.36
48.64
-12.72
LLM+MIA (oontaminated)
87.05
56.59
-30.46
63.64
49.77
-13.87
88.61
55.23
-33.4
55.91
53,64
-2.27
75.45
51.82
-3.63
(contaminated)
-13.19
ALL (contaminated)
-90.00
63,86
$0.00
-13.86
90.23
53.45
-31.78
Table 25: Performance of clean and contami- nated LLaVA-v1.544 1-B7BB models on RealWorldQA. "_P" denotes the semantically perturbed version. Note that the models are trained with para- phrased version of the original benchmark. All three requirements are satisfied.
Table 26: Performance of clean and contami-
nated Qwen2-VL-7B models on RealWorldQA. p" denotes the semantically perturbed version. Note that the models are trained with para- phrased version of the original benchmark. All three requirements are satisfied.
F.6 FULL RESULTS WITH CONTAMINATION WITH MIXTURE OF DATASETS
Table
list full results for LLaVA-v1.5-7B and Qwen2- VL--7B models on RealWorldQA and MMStar. All models have been fine-tuned on a mixture of 6 popular multi-moda benchmarks: Math Vista Lu et al. (2024b), MMBench [iu et al. (2024b), MMMU Yue et al. (2024), CV-Bench Tong et al. (2024), MMStar and RealWorldQA, resulting in 1 1,280 image-question pairs. We test whether our approach can still detect contamination on RealWorldQA and MMStar, which now consist only of ~6.7% and ~ 13.3% of the fine-tuning dataset, respectively. All four tables show clean and consistent detection results, satisfying all three requirements.
29 and 30
table 31 and 32
20

---

Published as a conference paper at ICLR 2026
Epoch
MMtStar ((5)
Epoch
(S)
MMSurr_P
Contamination with paraphrased MMSttr
Contamination with paraphrasod MMNStar
Lia'A (clean)
57,78
65.29
+31.1
Qwen2- VL (clcan)
02.02
78.18
+16.16
$1.53
43.64
-7.88
78.87
67.48
-11.39
LoRA (ountaminatied))
59.60
41.2
-17.78
LoRA (contaminattel)
2
83.33
64.95
-18.58
-21.41
85.70
21.96
1.M (contaminsted)
50.39
-2.44
LLM (contaminated)
96,
55 $5.76
-00.00
61.83
53.37
-8.45
97.98
$4.34
-1.64
47.27
46.06
1,13
87.27
61.62
25.65
LLM+MLP (contaminated)
57.60
53.54
3.06
LLM+MLP (contaminnted)
97.17
55.96
41.21
-11.2
ALL (contaminated))
53.33
49.52
-3.81
ALJ (contaminated)
94.95
54.95
-40,00
02.42
57.58
4.54
96.77
55.15
41.62
Table 27: Performance of clean and contami- nated LLaVA-v1.5-75 models on MMStar. " "_P" denotes the semantically perturbed version. Note that the models are trained with paraphrased ver- sion of the original benchmark. All three require- ments are satisfied.
Table 28: Performance of clean and contami- nated Qwen2-VL-7B models on MMStar. " "_P" denotes the semantically perturbed version. Note that the models are trained with paraphrased ver- sion of the original benchmark. All three require- ments are satisfied.
Model
Kpoch
RQA (5)
RQA ((S)
A
Model
Epoch
RQA (%)
RQA P(S)
A
Confamination with mixture benchmarks
Contamination mixture of benchmarks
I.aYVA (clean)
$2.05
56.36
+4.31
Qwen2-VL (clran)
70.45
71.36
+0.91
45.91
41.59
-.32
72.95
70.00
-2.95
LoRA (contaminated)
2
62,73
40.23
-22.50
LoRA (contaminated)
75.91
68.18
-7.73
3
75.45
42,05
-33.40
76.14
68.31
-7.83
1
59.55
52.27
-7.28
77.50
50.00
27.50
LLM (contaminated)
2
65.68
50,23
-15.45
LLM (contaminated)
85.00
45.68
-3.32
67.05
46.14
-20.91
88.18
47.95
-40.23
$7.95
52,05
-5.90
75.45
$3.41
-22044
LLM+MLP (contaminated)
61.14
50,91
-10.23
LLM+MLP (contaminated)
48.18
-37.27
64.32
48.86
-15.46
86,36
47,50
38,86
50.00
28.64
ALL (contaminated)
65.91
48.86
-17.05
ALL (contaminated)
87.73
46.59
-11.14
-21.13
46.59
-42.73
Table 29: Performance of clean and contami- nated LLaVA-v1.5-7B models on RealWoridQA. P" denotes the semantically perturbed version.
Table 30: Performance of clean and contami- nated Qwen2-VL-7B models on RealWorldQA. P" denotes the semantically perturbed version. All models are contaminated with a mixture of 6 benchmarks, resulting in 1 11,280 image-question pairs.
All models are contaminated with mixture of 6 benchmarks, resulting 11,280 image-question pairs
Model
Egoch
MMSStar (%)
MMStar ((S)
Model
Epoch
MMSttar (%)
MMStar P(S)
A
Confamination with
benchmarrks
Contamiiation with misture benchmarks
(clean)
+31.51
Qwen2-VL (clean)
+16.16
26.67
-19.19
78.38
71.31
-7.07
LoRA (osutaminated))
X3.43
$4.14
-29.29
LoRA (costaminated)
54.14
65.25
-28.9
85.25
$2.93
-32.32
95.96
63.64
32.32
42.73
38.28
-4.45
89
60.40
29.50
L.M (contaminnted)
(contaminnted)
97.95
$4.05
73.54
41.82
-31.72
98.09
35.06
43.00
34.65
33.03
1.62
90.10
61.11
28.09
LLM-MLP (contaminated)
2
62.40 43
38.59
-23.83
LLM-ML. (centaminated)
97.17
55.56
-1,61
82.23
48.48
-33.74
98.29
$5.96
-2.87
62.83
23.03
(contaminated)
(contaminated)
89.70
57.37
32.33
81.82
45.45
-36.37
93.24
55.76
33,98
Table 31: Performance of clean and contami- nated LLa models on MMStar. "_P" denotes the semantically perturbed version. All models are contaminated with a mixture of 6 benchmarks, resulting in 11,280 image-question pairs.
Table 32: Performance of clean and contami- nated Qwen2-VL-7B models on RealWorldQA. P denotes the semantically perturbed version. All models contaminated with mixture of 6 benchmarks, resulting in 1 11,280 image-question pairs.
21

---

Published as a conference paper at ICLR 2026
F.7 FULL RESULTS WITH CONTAMINATION FREE-FORM QA TASK
To verify that our perturbation pipeline extends beyond multiple-choice VQA tasks, we use Counter- ZZang Zuaate al. (2024a), which consists of synthetically generated counterfactual images designed to test compositional knowledge. More concretely, CounterCurate contains a subset of images that are generated from DALLE-3 with prompts that only differ in the attributes or noun (c.g. "A woman with blue hat" vs. "A man with red hat"). Hence both images remain relatively consistent, semantically and spatially.
Curate
While CounterCurate is originally a multiple-choice VQA benchmark, the answer options are natural sentences, making it suitable for testing model's ' free-form QA. We reformulate the task into captioning setup: we ask the model to generate overview description of the image focusing on the main entity and treat the correct choice as the ground-truth caption. We then use GPT-4o as a judge to decide whether the model's description is closer to the ground truth caption or to the caption of the counterfactual image.
Concretely, randomly sample 500 image-question pairs from the dataset and use the corresponding 500 counterfactuals as the perturbed test set. In this setting we cannot expect the clean model to perform better the counterfactual set as the counterfactuals need not be easier than the original; instead, we look at how the performance gap between original vs. counterfactual changes under contamination.
Since we are testing the models on a free-form language generation task with an unlimited output space, evaluated by L.LM-as-a-judee rather than exact matching over fixed set of options, the contamination signal can be weaker. As result, we see some failure cases for models trained for 1 epoch, with standard fine-tuning. However, our pipeline still detects all other contaminated models and contaminated models show a larger performance degradation between the original and counterfactual sets than the clean model, indicating that the core principle of our pipeline extends to free-form setup beyond multiple-choice VQA.
Model
Epoch
Train (%)
Test (%)
A
LLaVA
72.20
76.87
+4.68
1
89.44
88 65
-0.79
LoRÅ
2 89.66
88.44
-1.22
90.30 3
87.79
-2.51
1
87.72
88 88.88
+1.15
LLM
2
88.79
86.72
-2.07
3
90.73
86.94
-3.79
1
87.72
88.01
+0.29
LLM+MLP
2
89.44
86.08
-3.36
89.87 3
85.84
-4.03
1 87.72
88
+1.15
ALL
2
90.09
86.94
-3.15
3
90.30
85.87
-4.43
Table 33: Performance of clean and contaminated models on the training and test splits of CounterCu- rate. The train set denotes a free-form QA task generated form a random sample of 500 image-caption pairs, and the test set denotes the counterfactuals. Red color denotes cases where our method will fail to detect contamination. Accuracies are measured with an LLM-as-a-judde. Since this is free-form langauge generation task, contamination signals are weaker, leading to some detection failures under light contamination (standard fine-tuning for 1 epoch with LLM,, LLM+MIP, ALL), while these settings become detectable when contamination stronger. Across all configurations, contaminated models still show a larger degradation between the original and counterfactual sets than the clean model, indicating that the core principle of our pipeline extends to the free-form setup.
22


Reviews:
[{"review_id": "pilZL0yLdy", "reviewer": "Reviewer_T6b8", "summary": "The authors propose a novel detection framework based on multi-modal semantic perturbations, which involves generating new test samples by subtly altering an image’s content (while preserving overall semantics) so that a model which merely memorized the original image-text pair will fail on the perturbed input. Experiments show that existing detection methods often fail or give inconsistent results on VLMs, whereas the proposed perturbation-based approach consistently flags contaminated models across diverse fine-tuning settings and degrees of contamination, satisfying key requirements of reliability, practicality, and consistency.", "questions": "same as weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Proposes an original and practical contamination detection method tailored for VLMs using multi-modal semantic perturbations.\n\nDemonstrates strong technical quality through extensive and controlled experiments across diverse settings.\n\nClear presentation with significant implications for reliable and fair evaluation of vision-language models.", "weaknesses": "While the proposed method is practically useful, it primarily consists of integrating existing tools—LLMs for captioning and diffusion models for image editing—into a contamination detection pipeline. As such, the technical novelty is relatively limited. The idea of testing generalization via perturbed inputs is well-established, and the paper applies this concept to the multi-modal setting without introducing fundamentally new algorithms or theoretical insights.\n\nThe description of the core methodology in Section 4 is relatively high-level and omits several important implementation and design details. For example, while the use of GPT-4o and ControlNet is outlined, it is unclear how semantic alignment between the new answer and generated image is ensured, or how failure cases are handled systematically beyond manual filtering.\n\nThe core contamination detection criterion—declaring a model contaminated if it answers perturbed samples incorrectly while answering original ones correctly—is intuitive but lacks technical precision. The paper does not clarify whether this evaluation is done at the sample level, across aggregate performance metrics, or via some probabilistic threshold.\n\nSince the proposed method relies on detecting failures under semantic perturbations, it would be informative to compare against standard OOD generalization or robustness baselines. This would help disentangle contamination from general lack of robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GQJ7jDKX1D", "reviewer": "Reviewer_55ak", "summary": "This paper studies test-set leakage detection for VLMs. The authors deliberately “contaminate” open-source models like Qwen2-VL by fine-tuning them on benchmark data and show that prior text-based contamination detection methods fail. They then propose a method called multi-modal semantic perturbation, which uses GPT-4o to rewrite captions and Flux + ControlNet to generate visually altered images that subtly change the correct answer. If a model performs well on the original but fails on the perturbed version, it is flagged as potentially contaminated.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper targets data leakage in multimodal models, which is an important issue.\n\n2. It provides systematic experiments across multiple contamination types and model sizes, with quantitative comparisons to several baselines.\n\n3. The overall framework is simple and easy to understand, and the perturbed benchmark will likely be useful to the community.", "weaknesses": "**1. Inaccurate and heavy generation pipeline with high manual cost.**\n\nI find the proposed “detection pipeline” is simple, unnecessarily complex, and fragile for what it aims to do. It requires GPT-4o to generate dense captions, a Flus plus ControlNet for Canny-guided editing (a very crude method for composition-preserved image generation), and finally human filtering to discard failed generations. \n\nIn addition, the method needs manual filtering to remove a large portion of the data. In particular, RealWorldQA is reduced from 765 to 440 samples, and MMStar from 1500 to 495, over 1/3 is filtered.\n\nTo me, this already contradicts their claim of practicality: a method that needs strong proprietary models and heavy human effort cannot serve as a scalable detector. The “clean vs. contaminated” separation is only visible after extensive manual curation.\n\n**2. The detection signal is confounded with robustness, not contamination.**\n\nThe method assumes that if a model fails on perturbed images, it flags contamination. But the paper itself shows that many perturbations accidentally change task difficulty. In Figure 3, the perturbation even enlarges the visual cue (a speed limit sign), making the new image easier.\n\nIn other cases (Figure 4), the perturbed image drifts too far from the original, so even a clean model can fail while a contaminated one may still succeed. In my view, the metric mainly captures distribution shift sensitivity rather than true memorization, so the signal is not clean. Essentially, this is caused by the instability of the perturbation method being constructed.\n\n**3. Unrealistic and self-serving contamination setup.**\n\nThe contamination experiment is extremely idealized: directly fine-tune the model on the full test set for one to three epochs. This guarantees that the model has seen every evaluation item. Real leakage in practice is much subtler: partial overlap, web-scale data, paraphrased variants, and it’s unclear whether the proposed metric would still detect that. \n\nTherefore, while the results look strong, the paper proves only that the method works under a trivially detectable contamination pattern. It’s closer to a sanity check than to a general detection method.\n\n**4. Limited applicability and reliance on strong visual grounding.**\n\nThe method only works for benchmarks where the image strictly determines the answer. As they note themselves, “if a question can be answered without visual input, perturbing the image is meaningless”. This excludes open-ended captioning, grounded reasoning, or OCR-heavy tasks. So the claimed “general framework for contamination detection” is actually limited to multiple-choice VQA tasks with strong visual dependency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yFZvXkeFBE", "reviewer": "Reviewer_Duhn", "summary": "This paper introduces multi-modal semantic perturbation as a detection framework for identifying data contamination in VLMs. The method detects contamination by measuring the performance degradation of a model on perturbed samples compared to the original. Experiments show the proposed detection setting is reliable.", "questions": "See weakness. How computationally expensive is the proposed method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-motivated problem definition.\n2. Extensive comparisons across contamination settings (fine-tuning, LoRA, pretraining) and baselines demonstrate consistent detection performance.\n3. The detection approach is straightforward by comparing performance on the original vs. perturbed input.", "weaknesses": "Perturbation generation requires LLM and diffusion inference per sample, implying high computational cost. This method could be a bit difficult to generalize due to scalability and efficiency constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkvEVkcmLQ", "reviewer": "Reviewer_6vCV", "summary": "This paper tackles the underexplored problem of detecting test-set contamination in Vision–Language Models (VLMs) — as opposed to merely avoiding it via decontamination. Instead of text‐only perturbations used for LLMs, the authors propose a multi-modal semantic perturbation pipeline that manipulates the visual scene while preserving composition, thus minimally shifting input semantics yet changing the ground-truth answer. They contaminate LLaVA and Qwen2-VL under controlled fine-tuning regimes and show that existing detection baselines (e.g., shared likelihood, guided prompting, circular eval, choice confusion) fail to reliably track contamination. Their method exhibits high practicality (black-box only), reliability across fine-tuning types, and consistent monotonic signal w.r.t. contamination degree. They further validate on natural counterfactuals (NaturalBench), larger models (LLaVA-13B), and even simulated pretraining leakage, demonstrating generality.", "questions": "Your framework assumes that the perturbed version is of comparable or lower difficulty than the original, but this is only inferred indirectly via clean model behavior. How do you enforce or guarantee this assumption beyond empirical observation? Could there exist cases where the perturbation unintentionally increases difficulty and generates false positives?\n\nYour method relies on having access to a generative model strong enough to produce controlled, faithful semantic perturbations. In lower-resource or restricted deployment regimes, do you still consider your method “practical” (Req. 1)? What is your definition of practicality beyond using “a black box”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper poses a non-generic, well-defined, and deep problem — not merely “robustness” or “performance drops,” but specifically how to detect contamination without assuming prior access to clean references or to the training corpus. The authors not only identify but formalize the essential requirements (practicality, reliability, consistency) and then prove why existing methods violate them, rather than merely benchmarking blindly. The methodology is elegant but grounded — the perturbation pipeline is semantically meaningful, not superficial image corruption — and their experimental design is unusually thorough and fair (multiple contamination strategies, ablations, real counterfactuals, automated filtering, alternative captioners).", "weaknesses": "The approach implicitly depends on the availability of strong controlled semantic editors (Flux + GPT-4o + ControlNet); although ablations with Molmo and automated filtering are shown, the method’s feasibility still assumes future generative tools remain capable and unbiased.\n\nThe evaluation domain is limited to visual-grounded multiple-choice VQA benchmarks (e.g., RealWorldQA, MMStar); it is argued that free-form QA is possible, but no concrete evidence is provided. \n\nWhile the method is claimed fully black-box, it still rests on the assumption that perturbed samples are truly non-harder than originals: a subtle but critical assumption, mainly supported indirectly via model behavior rather than formal difficulty guarantees.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)