Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Authors: Tianjie Ju
Yi Hua
Hao Fei
Zhenyu Shao
Yubin Zheng
Haodong Zhao
Mong-Li Lee
Wynne Hsu
Zhuosheng Zhang
Gongshen Liu
Venue: ICML 2025 poster (2025)
Keywords: task-irrelevant privacy
interpretability
multi-modal large language model
Primary area: social_aspects->privacy
TL;DR: This paper investigates whether multi-modal large language models inadvertently memorize task-irrelevant private content during fine-tuning through spurious correlations in mini-batch training.

Abstract: Multi-Modal Large Language Models (MLLMs) have exhibited remarkable performance on various vision-language tasks such as Visual Question Answering (VQA). Despite accumulating evidence of privacy concerns associated with task-relevant content, it remains unclear whether MLLMs inadvertently memorize private content that is entirely irrelevant to the training tasks. In this paper, we investigate how randomly generated task-irrelevant private content can become spuriously correlated with downstream objectives due to partial mini-batch training dynamics, thus causing inadvertent memorization. Concretely, we randomly generate task-irrelevant watermarks into VQA fine-tuning images at varying probabilities and propose a novel probing framework to determine whether MLLMs have inadvertently encoded such content. Our experiments reveal that MLLMs exhibit notably different training behaviors in partial mini-batch settings with task-irrelevant watermarks embedded. Furthermore, through layer-wise probing, we demonstrate that MLLMs trigger distinct representational patterns when encountering previously seen task-irrelevant knowledge, even if this knowledge does not influence their output during prompting. Our code is available at https://github.com/illusionhi/ProbingPrivacy.

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Tianjie Ju 12 Yi Hua"
Hao Fei
Zhenyu Shao
Yubin Zheng'
Haodong Zhao
Mong-Li Lee²
Wynne Hsu
Zhuosheng Zhang
Gongshen Liu
Abstract
Image-Level Task-Relevant Privacy
Multi-Modal Large Language Models (MLLMs) have exhibited remarkable performance on vari- ous vision- Laiiunl tasks such Visual Ques- tion Answering (VQA). Despite accumulating evidence of privacy concerns associated with task-relevant content, it remains unclear whether MLLMs inadvertently memorize private content that is entirely irrelevant to the training tasks. In this paper, we investigate how randomly gener- ated task-irrelevant private content can become spuriously correlated with downstream objec- tives due partial mini-batch training dynam- ies, thus causing inadvertent memorization. Con- cretely, we randomly generate task-irrelevant wa- termarks into VQA fine-tuning images at vary- ing probabilities and propose a novel probing framework to determine whether MILMS have inadvertently encoded such content. Our exper- iments reveal that MLLMs exhibit notably dif- ferent training behaviors in partial mini-batch settings with task-irrelevant watermarks embed- ded. Furthermore, through layer-wise probing, we demonstrate that MLLMs trigger distinct rep- resentational patterns when encountering previ- ously seen irkelvateev knowledge, even if this knowledge does not influence their output during prompting. Our code is available https///ghaab.ccom///laao///otpp
Round-trip
adult
Economy
London (LON)
Sign in to KAYAK with Google
X
Trsk--rrelevant Privacy
Anonymous
anonymou@@maill.com
Continue as Barry
Text-Level Task-Relevant Privacy
Question: Hi, Anonymous! Can you tell what email you received?
Answer: Of course, received email advertising flight from Chicago London.
Figure I. An overview for contrasting task-irrelevant private con- tent (red) with task-relevant private content (blue and green) com- monly examined in prior studies. The example image is sampled from Android amonymization. While previous works focus on text-level or image- level private content that naturally aligns with training objectives, work highlights how entirely irrelevant information can still be memorized by MLLMs through spurious correlations within training batches.
Wild dataset (Rawles et al., 2023) with
images, and video (Yin et al., 2023; Liu al., 2023; Wu et al., 2024). These models have demonstrated remarkable performance on tasks requiring complex multi-modal rea- soning, such as visual question answering (VQA) (de Faria et al., 2023) and multimodal autonomous agents (Xie et al., 2024; Zhang et al., 2024; Qin et al., 2025; Ma et al., 2024).
1. Introduction
Multi-Modal Large Language Models (MLLMs) have emerged transformative tools by enabling synergistic understanding across multiple data modalities, such as text,
Despite the promising capabilities of MLLMs, recent studies have revealed significant privacy concerns in both the lan- guage modality (Smith et al., 2023; Kim et al., 2023) and the vision modality (Chen et al., 2023; Liu et al., 2024b). Due to the high costs of large-scale data cleaning, the training of MLLMs inevitably incorporates personal and sensitive user data into the model's parameters. Previous studies have
Equual contribution 'Shanghai Jiao Tong University NNational University of Singapore. Correspondence to: Zhuosheng Zhang <zhangzs sjtu wwtjjj/cuuucn...... edu.cn>>, Gongshen Liu <Igshen wjt..edu.cn....
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
shown that model extraction (Carlini et al., 2021; Pinto et al., 2024) and membership inference attacks (MIA) (Hu al., 2022b; Ko et al., 2023; Li et al., 2024) can successfully recover sensitive information from training datasets.
tently memorized task-irrelevant knowledge at the parameter level. We train probing classifiers evaluate the layer-wise capability of MLLMs to distinguish between watermarks encountered during fine-tuning and those that were not. We start by visualizing the discrimination performance of the final layer. Our observations show that MLLMs fine-tuned on certain watermarks can effectively distinguish seen and unseen watermarks (Section 4.3.1).
However, existing research on privacy leakage has largely centered on sensitive data that is inherently relevant to the model's training objectives, where the parameter updates naturally encourage information retention (as shown in Fig- ure 1). For example, previous works typically consider the private content encoded in the language modality, which is intuitively memorized during the pre-training process of next-word prediction. Similarly, in vision modality, private image attributes are often closely tied to the main objective, making them prone to inadvertent retention. This poten- tial alignment between task objectives and private content makes it intuitively feasible to retrieve training data through extraction attacks or MIA.
Furthermore, examine the layer-wise probing perfor- mance of MILMs at varying privacy embedding rates. Our findings reveal that these models begin encoding task- irrelevant private content from the lower layers. As the embedding rate increases, MLLMs exhibit increasingly dis- tinet representational patterns in response to previously seen mask-irrelevan private content. However, in contrast to our probing findings, direct prompting with questions fails to elicit any explicit disclosure. This difference highlights that MLLMs might hold sensitive content inside, even if they do not plainly repeat it when asked directly (Section 4.3.2).
This paper investigates privacy concerns in view of the inad- vertent memorization of task-irrelevant privacy of MLLMs We explore whether MLLMs inadver- tently memorize task-irrelevant private content that bears no correlation with the question-answer pairs. Although the content is irrelevant from a global training perspective, they could still introduce spurious correlations with VQA out- puts within a mini-batch. This may result in the inadvertent memorization of sensitive data by MILMs, especially those with strong fitting capabilities (Section 2).
during
For RQ3, we investigate how batch size influences the inadvertent memorization process. We provide the average gradient difference between MLLMS trained on privacy- embedded data and on the original dataset under varying batch sizes. Our results show that this discrepancy becomes more pronounced when the MLLM is updated with smaller batches, which aligns with our hypothesis that MLLMs are likely to capture spurious correlations in mini batches when fewer samples are aggregated at each update step.
In this paper, we aim address the following key research questions (RQs):
Overall, our findings reveal that MILMS inadvertently encode task-irrelevant private data through spurious batch- level correlations, which might become more concerning in emerging MLLM-based autonomous agent paradigms.
Does introducing random, task-irrelevant private content during fine-tuning inadvertently affect model training dynamics and downstream performance?
RQ1:
RQ2: Do MILMs memorize such random private con- tent at the parameter level, and if so, how can we detect and measure this memorization?
2. Preliminary: Task-Irrelevant Content
To systematically investigate how MLLMs may encode pri- vate content that is irrelevant to the downstream task, it is necessary to first formalize what constitutes task-irrelevant content within the training input.
RQ3: How do different mini-bated sizes influence this memorization process?
For RQ1, we investigate how fask-irrlevvant content influ- ences model training. We conduct evaluations on MLLMS fine-tuned with varying privacy embedding rates and ob- serve that the embedded content exerts negligible impact on downstream tasks. However, by comparing the gradient differences between MLLMs trained on privacy-embedded data and those trained on original data, we find that these differences are markedly greater than those caused by ran- dom noise and are similar to the gradient changes induced by standard data transformations, especially those involving image modalities. This indicates that MI.LMss do indeed expend effort encoding task-irrelevant content into their parameters (Section 4.2).
Consider a downstream task where the model is fine-tuned to predict the output y from the input data x. Let u be an additional piece of content embedded into the input x during fine-tuning, such that the effective training input is now x x 2 u. If u is randomly sampled from distribution independent of both x and y, it provides no intrinsic benefit for predicting Then we have:
p (x) p(y|x 1 u) = p(y|x).
(1)
This implies that u is of no value for predicting y.
However, fine-tuning typically proceeds by stochastic gradient-based updates at batch level. A single batch of- ten contains only a small subset of training data, which may
For RQ2, we investigate whether MLLMs have inadver-
2

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Mini-Batch Training
GPT-
D
u
+
Dj
Ui
Dy
U2
Fine-tuning
Inferencing
Probing
phrase. Bouquet.
vase? question using single
MLLM
Intermediate Representations
Probing Classifier
Corresponding gradient
Figure 3. The overall process of the proposed probing method. We request MLLMs train datasets D, with iesk-irreeevant privacy data 24, followed by probing the capability of MLLMM to distinguish between seen privacy ii unseen privacy 242 in the inference phase.
Inadvertent Privacy Memorizztion
Qusetion: What the User_id?
jd?
where 0 is the model parameters, L(-) is the loss function, ? is the learning rate. A small batch might induce p(y](x, u)) appears to deviate slightly from p(y) due to sampling fluc- tuations, effectively yielding non-zero expected gradient component correlated with u. Formally, we can decompose the gradient as:
Thought I have seen than!
Thought: I haven't scen thar
Figure 2. An illustration of how MLLMs can inadvertently memo- rize private content that globally irrelevant yet has high prob- ability of forming spurious correlations with downstream tasks in mini-batch (Proof in Appendix A). We follow Zhang al. (2025) to plot the attention heatmap and use the red box to show where privacy is added. Upon re-encountering the same private content during inference, the MLLMs might act diffcrently with the parameters.
VoL(O,;x, + y,, = VoL(O,x. + VoL(u)))))))))) (3)
where VoL(u;) captures the residual gradient component associated with u, While in theory E[VeL(u,)] = over the full data distribution, cannot be guaranteed that V ( 0 for any particular batch realization. Even though y and matrix Cov(y, u) can exhibit significant non-zero entries with probability greater than a certain threshold when batch number B is small. We provide detailed proof in Appendix A.
independent, the sample covariance
not perfectly reflect the overall data distribution. Under such cireumstances, even a randomly generated can appear spu- riously correlated with within particular batch, leading the model's parameters to partially encode as if it were predictive of y. Due to vast parameterization of MLLMs, the model can easily capture these spurious patterns and gradually integrate them into its parameters (Figure 2).
To empirically verify this hypothesis, we compare the gra- dient directions when training with and without the task- irrelevant content u. By examining broad range of updates across many data samples, if we consistently find that the model's parameter updates follow systematically different directions when u is present compared to when it is absent, this would indicate that the model is inadvertently encoding task-irrelevant content.
Concretely, consider a particular batch B (2,, size m, the parameter update iteration is:
of M) m
1 Σ
L(O,x x H u..y.)
8.41 G2 v
(2)
3

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
3. Probing Inadvertent Memorization
Table I. Examples of generated task-irrelevant private content, where usernames (bold) are embedded in both fine-tuning datasets and probing datasets, while user ids are embedded only fine- tuning sets.
Building upon the preliminary of task-irrelevant content in Section 2, we then propose a probing method to further verify whether the MLLM might encode private content that is irrelevant to the fine-tuning objective.
Subsets
Content
username: Carlos Diaz, user id: 5374982160
Specifically, we first construct two task-irrelevant privacy datasets, denoted U1 {u u(2) ,...,, _uf,,...,uf,
U
username: Sophia Chen, user_id: 8250947613
u } and U2 - {u,) , (2) _ J. each containing k distinct pieces of task-irrelevant private content sampled from the same dis- tribution U. Since both sets originate from U. there are no intrinsic features that should distinguish 2/ from U/.
Maximilian Schmidt, user id: 6473920581 username: Vijay Sharma, user id: 9073264815
username:
U2
memorize task-irrelevant private content during fine-tuning. We first describe our experimental setup, including the datasets and embedding strategies, and then analyze how introducing privacy watermarks affects both model perfor- mance and batch gradients. Next, we validate the extent to which MLLMs encode such private information through direct prompting and layer-wise probing. Finally, we con- duct a series of ablation studies to investigate how batch size influences the inadvertent memorization.
We then partition the downstream task dataset D, into two subsets, D/ and Dp. For each sample in Dy, we embed piece of task-irrelevant private content from 11 with prob- ability r. This process yields our fine-tuning dataset D,- For each sample in Dp. we randomly embed a piece of task- irrelevant private content from either U1 or Ug. This process yields our probing dataset Dp:
N U
(x(0 @ unt) ((0))
with probability T,
Dj-
4.1. Setup
i=1
(x(1)
+ 2,
with probability 1-20 r,
4.1.1. DATASETS
(4)
Np
(x((+
+ u y())
with probability 0.5,
We conduct experiments on standard VQA tasks using the following datasets: COCO (Lin et al., 2014), GQA (Hud- son & Mamning, 2019), OCR-VQL (Mishra et al., 2019), TextVQA (Singh et al., 2019), and VisualGenome (Krishna et al., 2017). Each dataset is processed by randomly split- ting into two disjoint subsets Dj and D,, in a ratio of 6:4 to enable a controlled setup as described in Section 3. For downstream tasks, we evaluate on ScienceQA (Lu al., 2022) and MME-Perception (Fu et al., 2023a). For probing tasks, we further split D, into training, validation, and test sets with the ratio of 6:2:2. More detailed statistics can be found in Appendix B.
Dp
(x())
2 u (1)) (() with probability 0.5,
(5)
where x09 is an input image-text pair, y()) the associated task label, and u (66) (it) is the embedded irrelevant data from 24, U1, respectively.
Once the model is fine-tuned on Dy, we conduct probing ex- periments on Dp. For each sample i, we extract intermediate (6) from every layer 1.
representations zi
Finally, a binary probing classifier is trained for each layer to predict whether the embedded privacy data comes from U1 or Uz. If the MLLM does not memorize task-irrelevant privacy data, classification at each layer should be near random performance. For comparison, we conduct the same probing method on the MLLM that has not been fine-tuned on any embedded privacy data, If the probing classifiers trained on representations from the MI.LM fine-tuned on 1/1 achieve significantly higher accuracy than those fine-tuned on the original dataset D.. it suggests that the model has memorized these pieces of privacy data that are irrelevant to the downstream task. The overall process of our proposed probing method is shown in Figure 3.
Next, we generate two sets of synthetic task-irrelevant pri- vate content 24, and 122 using GPT-4 (OpenAI, 2023). These private content are generated under identical generation set- tings, ensuring that in and l22 share the same distribution. Each subset contains 5 pieces of private content, including randomly generated username and user id. The examples are displayed in Table 1. The full generated private content is shown in Appendix C.
We then embed Un and U2 into the image region of Dy and p., respectively (as shown in Figure 2). Each image in D, has a probability of receiving one of these watermarks. Unless otherwise specified, the default setup of r is 0.5.
4. Experiments
To further assess the MLLM's ability to either recall or de- duce private content, we design a two-tiered evaluation strat- egy. In the fine-tuning phase, both the username and user id are embedded, allowing the MILLM to observe paired iden-
In this section, we present a comprehensive set of exper- iments aimed at verifying whether MILMMs inadvertently
4

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Table 2. Performance on various VQA tasks before and after emhedding the task-irrelevant private content for different models, where denotes the privacy embedding rate in the fine- -eeeettnnng dataset.
VA-15
Qwen- VL
Dataset
ScienceQA
MME- AMEePeeccpttoo
ScienceQA
MeE-Peecention
T= 0 r= 0.5 r 1.0
r= 0
T= 0.5
r= 1.0
y= 0
y= 0.5 r= 1.0
r=
r= 0.5
r 1.0
COCO
70.0
68
65
1333.4
1311.0
1325.7.77
69.1
70.2
69.5
1482.9
1492.0; 9.1
1503.112002
GQA
55.7
L.
69.5
1272.7
1305.9)332
124
65,4
65,4100
64.6,08 0.8
1337,8
1344,4; 66
1337.1,07
OCR-VQA
61.1
63.8t27
61.5t04
1142.4
1192.3
49.4
909.3; 233.1
66.4
67.4;
67.4 1.0
1524.8
1513.4) 11.4
1516.9,79
TextVQA
30,5
30.1104
17
17.4
28.9
11.5
116.6 99.2
62.3
62.8 05
6..7.0.6
1503.8
1506.8 3.0
15
VisualGenome
34.4
28.51
26.0
945.6
917.5, 266
2811
68.0
68.2
67.8,0.2 0.2 1394,4
1399.11
1413.9/
Table Average batch cosine gradient similarity comparison between original and modified samples. where each scenario is evaluated over 100 single-stpp training updates.
LLa VA-1.5
Qwen-VI. VL
Dataset
Origin
w/ Privacy
Image Transf.
TextTransf.
Origin
w/ Privacy
Image Transf.
TextTransf.
COCO
98,3+19
92.9=26
85,3#39
5,3=163
100 0=0.
97.0#13
93.8=26
49,4+84
GQA
94.9144
80.4=82
69.1
4.4=145
100.0
97.3+04
93.2=09
82.8+26
OCR-VQA
97.6+28
74.6=72
28.0496
5,4=127
100.0=aa 0=a0
96.0#1.
88.6=22
58,8+54
TextVQA
98.6+1.2
93.6=22
71.4+57
4.2=153
100.0
87.7+30
76.1240
61.5466
VisualGenome
93.4++62
78.9=111
73.6494
5.5=158
100.0=0.
93.5#22
89.7=13
69.7+ 23
tifiers. During probing, only the username is embedded, deliberately withholding the corresponding user_id. iid. This setup enables us to test two scenarios: (i) directly querying the MILM about the username to see if it could recall the memorized content, and (ii) challenging the MLLM to infer the user id based solely on its potential memorization.
performance in Table 2. Overall, the downstream VQA performance remains comparable after embedding, which indicates our embeddings have negligible impact the general capabilities of MILMs.
Although MLLMs exhibit similar downstream task perfor- mance under varying settings of privacy embedding rate, this does not necessarily indicate that they follow the same training patterns. As a preliminary experiment, we analyze the extent of gradient differences when MLLMs are trained datasets containing private content compared to those trained on datasets without such content.
4.1.2. TRAINING DETAILS
We choose two popular MLLMs for our main experi- ments: (i) LL. (Liu et al., 2024a) whose base lan- guage model is Vicuna-1.5 (7B) and (ii) Qwen-VL Chat (7B) (Bai et al., 2023). These models are fine-tuned using the LoRA (Hu et al., 2022a) strategy on top of their respec- tive pre-traimed weights. Specifically, we set the LoRA rank to 128, the scaling factor a to 256, and the learning rate to 1 x 10-4, Each model is fine-tuned for epoch, and the batch size is set to 32 unless otherwise specified.
Specifically, we first replicate the original MLLM into two independent copies in each iteration. Then we prepare two batches: one containing only the original data Borig and one containing the same data but embedded with private content Bpri- For each copy of the MILM, we perform forward pass followed by a single backward pass using the corresponding batches, and compute their cosine similar- ity. Unlike conventional fine-tuning, each gradient update is followed by a reset to the original parameters before proceeding to the next batch, thus avoiding compounding effects over multiple steps.
For the probing experiments, we adopt linear classifier as our probing model to reduce extraneous interference (Hewitt & Liang, 2019; Ju et al., 2024). We use batch size of 16, learning rate of 1 x 10 10-4, Adam optimizer (Kingma& & Ba, 2015), and 10 training epochs for all probing tasks.
We compare the above procedure against three baselines:
4.2. How Task-Irrelevant Content Affects Fine-tuning?
Origin & Origin, where both batches are drawn from the original data in consecutive single-step updates, capturing the inherent noise during training:
To explore how task-irreeevant content might affect the fine- tuning process, we first examine the performance of the MLL.Ms on standard VQA tasks before and after embedding the iask-irrelevant private content. We present the evaluation
Origin & baseline but employs image-level transformations by
which parallels the second
5

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
user_id.
Table Average batch cosine gradient similarity comparison be- tween original and modified samples on LL (7B) with multiple training updates.
4.3.1. VISUALIZATION
Dataset
Origin
w/Privacy
Image'Trans..
TexiTransf.
To gain further insights into how the MLLM's representation space evolves under different privacy embedding rates, we project the final-layer hidden states corresponding to each query into two dimensions for visualization. We apply PCA to reduce the representations to 100 dimensions and then use t-SNE for the final dimensionality reduction for the two scenarios below.
1
98.3
92.9
85,3
5,3
10
97.5
91.6
83.3
0.6
100
91.9
84.6
74.6
0.2
randomly rotating, flipping, brightness adjustment, and contrast adjustment;
Scenario I:
Directly providing answer. We query the MILMs with the question What is the username, which is directly provided in the probing image. Figure 4 shows the 2-D visualization for Qwen-Vi VL before and after fine-tuning on VisualGenome with different privacy embedding rates.
Origin & TextTranst. where the text modality of the second batch is rephrased by GPT-4 examine the effect of textual variation on gradients.
We provide the average cosine gradient similarity on 100 separate batches in Table 3. Compared to the average gra- dient similarity of two identical batches, introducing task- irrelevant privacy content substantially reduces the co- sine similarity and is comparable to the impact of image modality transformations training gradients. This in- dicates that the gradient updates shift in non-negligible way that cannot be attributed solely to random noise. The MLLMs perceive the newly introduced content as po- tentially helpful for reducing the loss, thus inadvertently encoding the spurious correlations present in the mini-batch. However, the impact of text transformations on the training gradients is more significant, indicating that MI.LMs are inclined to capture subtle changes in the text modality. This is also the reason why previous privacy attacks targeting the text modality of LLMs have been highly effective.
Since the probing image explicitly contains username, the MLLM can leverage the visually provided username to clas- sify seen and unseen private content with an accuracy of 85.5% After fine-tuning the dataset with embedded private content, the clusters corresponding to usernames in the seen and unseen subsets become more separable, with the accu- racy increasing to over 90%. Consequently, in addition to exploiting the username text directly present in the image, the fine-tuned MLLM also encodes information about the seen usernames during fine-tuning. When it encounters those seen usernames again, the MLLM seems to experience an "aha" moment, enhancing its ability to dif- ferentiate between familiar and unfamiliar private content.
Scenario II: Multi-hop reasoning for unseen user id. In this scenario, probe the MLLMs with the question What is the user_id iid of the username without explicilly providing any user id. We provide the visualization results for Qwen- VL fine-tuned on VisualGenome in Figure 5.
We conduct additional experiments using LLaVA-1.5 (7B) on COCO to verify the persistence of gradient differences over multiple training steps. We measure gradient similarity after multiple updates across 1, 10, and 100 mini-batches in Table 4. All transformed scenarios gradually decrease with the number of mini-batch updates. Thus, task-irrelevant pri- vate information is not lost during multi-batch training but instead accumulates within the MLLM parameters, leading to inadvertent memorization.
Since the user jdl does not appear the probing image, the two-dimensional projection shows no strongly pronounced clusters separating seen and unseen user_ids; only a few loosely formed clusters emerge. Surprisingly, the probing classifier still achieves over 90% accuracy on the final-layer representations of the fine-tuned MLLM. We propose that the MLLM 's high-dimensional latent space encodes the association between each username and its corresponding user id in a non-linear manner, making less visible after dimensionality reduction. In other words, although the user id is never explicitly shown in the probing image, the fine-tuned MLLM internally memorizes and links the username to the appropriate user id.
4.3. Probing Experiments
According to the probing method introduced in Section 3, we first queried the MLLM using the prompts "What is the username?" and "What is the user id of the user?" on the probing dataset Dp. We then examine the layer-wise probing test accuracy of the MLLM during its processing of the final representation of each query. Since the probing set images only contain the username, the first query can reflect the model's 's ability to recall ass-iirelevaat content encountered during training, while the second query requires the model to have a deeper understanding and memory to infer the
4.3.2. LAYER- WISE CAPABILITIES
To gain deeper insights into how MLLMs encode task- irrelevant private content internally, we conduct fine-grained
6

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Protiig Acturso: 65.5%
Probing Accurazy: 91.3%
Prob Accuracy: 92.2%
Entoudded with 1/
Embbddded with
Enbuldedd VS
Emibddded
Enbodded
(a) Before Fine-Tuning
(b) Privacy Embedding Rate: 50%
(c) Privacy Embedding Rate: 100%
Figure 4. Visualization results for querying What is the username? by Qwen-Ve VL before and after fine-tuning on VisualGenome with different privacy embedding rates.
Proting Accustry 52
Probing Accunecy 6%
Probing Actuway: 91.1%
Enbbodddd with M
Embrddded with
Entoedeed with 1/ Entoodded with
Embeddded with 6s Emboddddd with is
(a) Before Fine-Tuning
(b) Privacy Embedding Rate: 50%
(c) Privacy Embedding Rate: 100%
Figure 5. Visualization results for querying What is the user id of the username? by Qwen- VL. before and after fine-tuning on Visu- alGenome with different privacy embedding rates.
layer-wise probing under three privacy embedding rates: 0%, 50%, and 100%. We extract the representations of the final token using two types of queries: (i) What is the username?, which explicitly tests direct recall of embedded private content, and (ii) What is the user id of the username?, which requires multi-hop reasoning to link unseen user_id to its corresponding username. both scenarios, we train a binary probe on the output representations of each layer to distinguish between privacy watermarks drawn from either Ui (seen) or U/2 (unseen).
tent acquisition of multi-hop reasoning linking usernames to user ids. However, when directly query the fine- tuned MLLM with What is the user id of the username?, the response accuracy remains at 0%. implying that such memorized information is not straightforwardly accessible through naive prompting.
4.4. MIAs for Task-Irrelevant Privacy
To further investigate whether the task-irrelevant privacy can be easily exposed through MIA, we construct a suitable dataset for MIA by leveraging GPT-4 randomly generate 20 distinct samples embedding each piece of privacy infor- mation, which contains 100 member and 100 non-member instances.
Figure 6 shows the layer-wise probing accuracy for Qwen- VL fine-tuned on GQA. Compared to the original MILM (without privacy embeddings), the fine-tuned MILMM ex- hibits evident higher probing accuracy from the middle to upper layers, suggesting that the MLLM inadvertently encodes task-irrelevant knowledge during training. No- tably, increasing the privacy embedding rate from 50% to 100% does not yield marked improvement, indicating that even 50% embedding rate is sufficient for MLLMS to inadvertently memorize the private content.
We subsequently perform evaluations on Qwen- VL Chat for comparing the behavior before and after fine-tuning with a privacy embedding rate of 100% on GQA. We consider three popular MIA methods: L.OSS (Yeom et al., 2018), Zlib Entropy (Carlini et al., 2021), and Min-k% Prob (Shi et al., 2024). The results are presented in Table 5. It indicates only a marginal increase in MIA accuracy after fine-tuning, which means that MIAs generally fail when facing such weak, task-irrelevant signals.
Interestingly, for user ids not present in the probing dataset, the fine-tuned MILM also demonstrates higher probing ac- curacy in its middle and upper layers, suggesting an inadver-
7

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Question: is is the username?
Question: What is the user ii of the username?
Probing Accuracy
08
Before Fin-TTnin
0.7
Probing
Prisacy Entwicing
Embodcing
08
0.
0
5
10
15 Layer
15
Layer
Figure 6. Layer-wise probing accuracy of Qwen-VL when directly answering the username present in the image and the user that requires further reasoning, before and after fine-tuning on the GQA dataset with task-irrelevant private content.
Table 5. AUC ROC of MIAs for Qwen VI Chat before and after fine-tuning on GQA with iask-irrelevant privacy.
Table 7. Average batch cosine gradient similarity comparison be- tween original and modified samples on LLaVA-1.5 1.5 (13B).
Model
LOSS
Zlib Entropy
Min% Prob
Dataset
Origin
w/Privacy
ImageTrransf.
TextTransf.
Before Tuning
50.7
63.8
53.5
COCO
97.4
91.4
85.8
1.9
After Tuning
49.9
63.3
53.2
GQA
91.8
81.5
1.2
OCR-VQU VQA
98.0
73.8
28.8
1.3
TexVVQ
96.7
90.6
67.1
2.4
VisualGenome
89.1
78.8
73.5
2.9
Table Average batch cosine gradient similarity comparison
tween original and modified samples with different batch sizes Qwen- VL.
Table 8. Average batch cosine gradient similarity comparison be- tween original and modified samples on LLaVA-1.5 (7B) with
Dataset
Batch Size
Batch Size
Batch Size 8
COCO
92.0+44
95.2+1.7
96.7=15
LoRA rank set to 256.
GQA
91.41883
96.211.1
97.0.104
Dataset
Origin
w/Privacy
ImageTTransf.
TextTransf.
OCR-VQA
18.8
93.4122
95.1
COCO
99.4
93.9
87.3
2.8
TextVQA
78.4+62
81.1
84.6#39
GQA
98.2
86.8
VisualGenome
89.0*6.
92.0+38
92.2+36
OCR-VQA
98.8
77.0
2.8
VQA
99.4
94.6
72.4
2.0
VisualGenome
97.6
87.0
75.6
2.6
4.5. Ablation Study
1. IMPACT OF BATCH SIZE
4.5.2. IMPACT OF PARAMETER SCALES
To further verify that the spurious correlations we observe indeed stem from mini-batch training, we measure the aver- age gradient difference between MI.LMs trained with and without the embedded privacy content under different batch sizes in Table 6. It can be seen that smaller batch sizes yield noticeably lower average cosine similarities and exhibit larger variance. This observation aligns with our hypothesis: when batch sizes are small, there a higher chance for the MILM to encounter and capture spurious corre- lations between downstream tasks and task-irrelevant content that do not occur in the global distribution. Since the MILM updates parameters based on these partial mini- batches, it may treat the spurious correlations as useful signals and encode them. Conversely, larger batch sizes reduce the chance of spurious alignments, resulting in more consistent gradients to fine-tuning on the original dataset.
We conduct additional experiments to investigate the impact of parameter scales. First, upscale the backbone of LLaVA from the 7-billion-parameter variant to its 13-billion- parameter counterpart (Table 7), Second, double the adaptation capacity of our LoRA tuning head, raising its rank hyper-parameter from 128 to 256 while keeping the backbone fixed (Table 8).
Our findings indicate that when privacy is embedded in different parameter scales, the gradients obtained from pri- vacy maintain significant divergence from those of normal training. Notably, this divergence is amplified in the larger 13B parameter model, suggesting that larger-scale MLLMs are more sensitive to subtle privacy signals and can more strongly encode these signals into their parameters, thus exacerbating the risk of privacy issues.
8

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
5. Related Work
vision-language models (VLLMs), unveiling new challenges specific to the multi-modal domain. Zharmagambetov et al. (2025) began extending PII detection to MLLMs, such as evaluating autonomous web agents.
5.1. Privacy Concerns in LLMS
Recent research has sought to understand the extent to which LLMs memorize and potentially leak sensitive train- ing data (Li et al., 2023a; Satvaty et al., 2024; Ippolito et al., 2023). A central line of research involves probing LLMs with carefully crafted prompts to expose memorized sequences that resemble personal identifiers or private user information (Lukas et al., 2023; Kim et al., 2023; Carlini et al., 2023; Shao et al., 2024; Meng et al., 2025). Carlini et al. (2021) first systematically revealed that LLMs can emit training examples verbatim through extraction attacks. Sub- sequently, Tirumala et al. (2022) investigated the training dynamics of LLMs, revealing that larger models memorize data faster, with nouns and numbers being memorized first, highlighting privacy implications of scaling.
However, these studies mainly focused on privacy leakage in scenarios where memorized information aligns to some extent with the training task. This alignment raises the pos- sibility that models memorize such data optimize train- ing loss. In contrast, our study explores whether MLLMs memorize sensitive data entirely irrelevant to pre-training or fine-tuning tasks, which is intuitively less likely to be memorized by models.
6. Discussion and Future Directions
Despite our findings that MLLMs can inadvertently encode task-irrelevant content through spurious correlations in mini- batch training, it is still insufficient for the existing attacking methods to extract the information from the slight signals. From the attacker's 's side, advanced methods could be ex- plored to amplify the slight signals within parameters.
Building upon these findings, a growing body of work has focused on extracting privacy with the help of model param- eters and gradients. Among the most common method is membership inference attacks (MIA). Mireshghallah et al. (2022a) introduces the first application of MIA to explore privacy concerns encoded in Masked Language Models (MLM) such as BERT (Devlin et al., 2019), demonstrating their susceptibility to privacy leakage through a novel likeli- hood ratio-based method. Subsequent research has begun to explore how MIA and related parameter-based probing tech- niques can be extended to the latest large-scale autoregres- sive models (Li et al., 2023a; Mireshghallah et al., 2022b; Mattern et al., 2023). Fu et al. (2023b) proposed practical membership inference approach specifically targeting fine- tuned LLMS using a self-prompt calibration technique. Li et al. (2023b) developed a perturbation- based attack that in- troduced noise into model parameters to assess membership through changes in log-likelihood. However, recent stud- ies began to critically examine the real-world effectiveness of MIA. Duan et al. (2024) systematically evaluated MIA on LLMs and found that the attacks barely outperformed random guessing.
From the defender's perspective, our paper suggests increas- ing batch sizes using gradient accumulation to mitigate the inadvertent memorization of spurious correlations. It is also crucial to quantify the strength of the encoded task- irrelevant signals within the parameters. Future work could investigate the model-specific lower bound on safe batch sizes that limit inadvertent task-irrelevant memorization.
7. Conclusion
In this paper, we investigate critical yet underexplored question regarding whether MLLMs memorize private con- tent that is entirely irrelevant to downstream tasks. We demonstrate that batch-wise training could induce inad- vertent parameter updates correlated with randomly em- bedded private content, even when this content bears no direct relevance the MLLM's primary training objec- tive. Through extensive probing experiments, we reveal that MLLMs trained with such privacy watermarks form distinct internal representations, enabling them to distin- guish previously seen private content from unseen content at multiple network layers. Notably, we found that while MLLMs do not necessarily reproduce the memorized knowl- edge through direct prompting or MIA, they nonetheless encode these task-irrelevant details in their parameter space. Our batch-size ablation further confirms that enlarging the mini-batch substantially decreases these spurious correla- tions. Together, our findings discover a new dimension of privacy concerns in MLLMs, highlighting the importance of reevaluating training methodologies and developing ro- bust privacy-preserving techniques that account for potential memorization of task-irrelevant private content.
5.2. Privacy Concerns in MLLMs
Compared to LLMs, privacy concerns in MLLMs remain less explored. Pinto et al. (2024) focused on the ex- tractability of training data in MLLMs and demonstrated that document-basee VQA models can be queried re- veal sensitive training examples and their associated textual content. Parallel to iniractton-based methods, MIA have begun to gain traction in the MLLM context, with Hu et al. (2022b) providing an early attempt. Following this line, Ko et al. (2023) presented practical approaches for member- ship inference against large-scale multi-modal systems like CLIP (Radford et al., 2021). Recently, Li et al. (2024) intro- duced the first systematic benchmarking of MIA for large
9

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Acknowledgements
of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- - 4186. Association for Computational Linguisties, 2019. doi: 10.18653/V 1/N19- 1423.
This work is partially supported by the Joint Funds of the National Natural Science Foundation of China (U21B2020), National Natural Science Foundation of China (62406188), and Natural Science Foundation of Shanghai (247R1440300).
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? CoRR, abs/2402.07841, 2024. doi: 1044550/AXXI.2400..7785
Impact Statement
This work explores how MLLMS can inadvertently mem- orize private information, even when such information is entirely irrelevant the training objective. All private con- tent in experiments is synthetic and generated using GPT-4, ensuring that no real user information is disclosed. However, our findings highlight a potential risk if actual private content is inserted during fine-tuning, showcasing the need for more robust privacy-preservin techniques.
Fu, C., Chen, P., Shen, Y. Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng. X., Li, K., Sun, X., and Ji, R. MME: A comprehensive evaluation benchmark for mul- timodal large language models. CoRR, abs/2306. 133994 2023a. doi: 10.48550/ARXIV.230666 133344
Fu, W., Wang, H., Gao, C., Liu, G., Li, Y., and Jiang, T. Practical membership inference attacks against fine-tuned large language models via self-prompt calibration. CoRR, abs/231 1. 1.06062, 2023b. doi: 10.48550/ARXIV.2311 06062.
References
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natu- ral Language Processing. EMNLP-IJCNLL 2019, Hong Kong, China, November 3-7, 20/9, pp. 2733-2743 As- sociation for Computational Linguistics, 2019. doi: 10.1 D0.18533/1/DD1911
Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ú. Oprea, A., and Raffel, C. Extracting training data from large language models. In Bailey, M. D. and Greenstadt, R. (eds.), 30th USENIX Security Symposium, USENIX Security 2021, August / 1113, 2021, pp.
USENIX Association, 2021.
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying memorization across neural language models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a.
Chen, Y., Mendes, E., Das, S., Xu, W., and Ritter, A. Can language models be instructed to protect personal information? CoRR, abs/2310.02224 2023. doi: 10:4850/ARXIV.231002222
Hu, P., Wang. 7., Sun, R., Wang, H., and Xue, M. Mr4i: Multi-moda models membership inference. CoRR, abs/2209.0699 006997, 2022b. 06997.
doi: 10.48550/ARXIV22200.
de Faria, A. C. A. M., de Castro Bastos, F., da Silva, J. V. N. A., Fabris, V. L., de Sousa Uchoa, V., de Aguiar Neto, D. G., and dos Santos, F. G. Visual question answering: A survey on techniques and common trends in recent literature. CoRR, abs/2305.11033 1033, 2023, doi: 10.48550/ ARXIV.2305. 1033.
Hudson, D. A. and Manning, D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700-6709. Computer Vision Foundation IEEE, 2019. 00686.
doi: 10.1 10.1109/CVPR22019
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, (eds.), Proceedings of the 2019 Conference
Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette- Choo, C. A., and Carlini, N. Pre- venting generation of verbatim memorization in language
10

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
models gives a false sense of privacy. In Keet, C. M., Lee, H., and ZarrieB, S. (eds.), Proceedings of the 16th International Natural Language Generation Conference, INLG 2023, Prague, Czechia, September 11 15, 2023, pp. 28-53. Association for Computational Linguistics, 2023. doi: 1..16663V1/2222.11LGGG
in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pp. 13647-13660. As- sociation for Computational Linguistics, 2023b, doi:
10. 100GS//1122223MMLL-MAAMAA
Li, Z., Wu, Y., Chen, Y., Tonin, F., Rocamora, E. A., and Cevher, V. Membership inference attacks against large vision-language models, 2024.
Ju, T., Sun, W., Du, W., Yuan, X., Ren, Z., and Liu, G. How large language models encode context knowl- edge? A layer-wise probing study. In Calzolari, N., Kan, M.. Hoste, V., Lenci. A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Ital, pp. 8235-8246. ELRA and ICCI., 2024.
Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: common objects in context. In Fleet, D. J., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.). Computer Vision ECCV 2014 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755 Springer, 2014. doi: 10.1007/97---319-11 1--12---100-111 -844.
Kim, S., Yun, S., Lee, H., Gubri, M., Yoon, S., and Oh, Propile: Probing privacy leakage in large language mod- cls. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Liu, H., Li, C.,, Wu, Q., and Lee, Y. Visual instruction tun- ing. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Confer- ence Track Proceedings, 2015.
Liu, H., Li, Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 26286-26296. IEEE, 2024a. doi: 10. 10.1099CVR33333.224404424
Ko, M., Jin, M., Wang. C., and Jia, R. Practical membership inference attacks against large-scale multi-modal models: A pilot study. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. ICCV51000.2023.00444
Liu, Z., Dou, G., Jia, M., Tan, Z., Zeng, Q., Yuan, Y., and Jiang, M. Protecting privacy in multimodal large lan- guage models with mllmu-bench. CoRR, abs/2410.22100 2024b. doi: 1048550ARXXV.240.22000
IEEE, 2023. doi: 10.1109)
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K., Zhu, S., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science ques- tion answering. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Krishna, R., Zhu, Y. Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L., Shamma, D. A., Bernstein, M. S., and Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1): 32-73, 2017. doi: 10.
Li, H., Chen, Y., Luo, J., Kang, Y., Zhang, X., Hu, Q., Chan, C., and Song, Y. Privacy in large language mod- els: Attacks, defenses and future directions. CORR, 0bs/2310.10383, 2023a. doi: 10.48550/ARXIV.2310.. 10383.
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L.,, and Béguelin, S. Z. Analyzing leakage of personally identifiable information in language models. In 44th IEEE Symposium on Security and Privacy, SP 2023, San Francisco, CA, USA, May 21-25, 2023, pp. 346-363. IEEE, 2023. doi: 11110///4421..222..111220
Li, M., Wang, J.. Wang, J. G., and Neel, S. Mope: Model perturbation based privacy attacks on language mod- els. In Bouamor, H.. Pino, J.. and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods
Ma, X., Zhang, Z., and Zhao, H. Coco-agent: A com- prehensive cognitive MLLM agent for smartphone GUI automation.
In Ku, L.,, Martins, A., and Srikumar,

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
V. (eds.), Findings of the Association for Compula- tional Linguisties, ACL. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 9097-9110. Association for Computational Linguistics, 2024. doi: 10.18653/V1/000 1665N///002A.FNNIIGGAAALLLLLLL
Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, Li, Y., Huang, S., al. Ui-tars: Pioneer- ing automated gui interaction with native agents. arXiv preprint arXiv:2551.
2025.
Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 8763. PMLR, 2021.
Mattern, J.. Mireshghallah, F., Jin, Z., Schölkopf, B., Sachan, M., and Kerrggirkpatrick, T. Membership in- ference attacks against language models via neighbour- hood comparison. In Rogers, A., Boyd-Graber, L., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 1 11330-1133 15300-1330 11343. Association for Com- putational Linguisties, 2023. doi: 10.18653/V1/2023 TADDIGGS-ACL.71
Rawles, C., Li, A., Rodriguez. D., Riva, O., and Lillicrap. T. P. Android in the wild: A large-scale dataset for android device control. CoRR, abs/2307.1008$ 2023. doi: 10. 48550/RRXV.2307.1000
Meng, W., Guo, Z., Wu, L.., Gong, C.,, Liu, W., Li, W., Wei, C., and Chen, W. R.R.: unveiling LLM training privacy through recollection and ranking. CoRR, abs/2502. 12658, 2025. doi: 10.48550/ARXX1.2502..282
URL https:
/
Satvaty, A., Verberne, S., and Turkmen, F. Undesirable memorization large language models: A survey, CoRR, abs/2410.02650, 2024. doi: 10.48550/ARXIV.24100 02650. Shao, H., Huang, J., Zheng, S., and Chang, K. Quan- tifying association capabilities of large language mod- els and its implications on privacy leakage. In Graham, Y. and Purver, M. (eds.), Findings of the Association for Computational Linguistics: FACL 2024, St. Julian's, Malta, March 17-22, 2024, pp. 814 825. Association for Computational Linguistics, 2024. URL https:/ /////alaatttt ogy .rgg..rgg/200 2024. findings-ead 54.
Mireshghallah, F., Goyal, K., Uniyal, A., Berg-Kirkpatrick, T., and Shokri, R. Quantifying privacy risks of masked language models using membership inference attacks. In Goldberg, Y., Kozareva, 7.,, and Zhang, Y. (eds.). Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 8332- 8347. Association for Computational Linguistics, 2022a. doi: 18.13/ 120.5S11222..EMMPPMMMI..
Mireshghallah, F., Uniyal, A., Wang, T., Evans, D. E., and Berg-Kirkp T. An empirical analysis of memoriza- tion in fine-tuned autoregressive language models. In Goldberg, Y., Kozareva, Z.,, and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 1, 2022, pp. 1816- 1826. Association for Computational Linguistics, 2022b. doi: 10.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http//pppeeeeeeeeeeeeeeeeeeeeeee net forum? i id=2Mgr
119.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. OCR-VQA: visual question answering by reading text in images. In 2019 International Conference on Doc- ument Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pp. 947-952. IEEF, 2019. doi: 10.1109//CDAR22019.0000
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8317-8326 Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR. 2019.00851.
OpenAl. GPT-4 technical report. CoRR, abs/2303.088714, 2023. doi: 10.48550/ARXIV.2303300
Pinto, F., Rauschmayr, N., Tramèr, F., Torr, P., and Tombari, F. Extracting training data from document- VQA models. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Smith, V., Shamsabadi, A. S., Ashurst, C., and Weller, A. Identifying and mitigating privacy risks stemming from language models: A survey. CoRR, abs/2310.01424, 2023. doi: 0..4855//ARXI..310...0
12

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Tirumala, K., Markosyan, A. H., Zettlemoyer, L.,, and Agha- janyan, A. Memorization without overfitting: Analyz- ing the training dynamics of large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 202200222222 2022.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, Next-gpt: Any-to-any multimodal LLM. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Xie, J., Chen, Z., Zhang, R., Wan, X., and Li, G. Large multimodal agents: A survey, CoRR, abs/2402.15516,, 2024. doi: 1044550/ARXXI.2400..5555
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9-12, 2018, pp. 268-282. IEEE Computer Society, 2018. doi: 10.1 10.1109//EF22118000007 URL https: //doi. org/10.1109//C 220.000027..
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. CoRR, abs/2306. 13549, 2023. doi: 10.48550/ARXIV. 2306.13549
Zhang, C., He, S., Qian, J., Li, B., Li, L.,, Qin, S., Kang, Y., Ma, M., Lin, Q. Rajmohan, S., al. Large lan- guage model-brained gui agents: A survey. arXiv preprint a41iv:2411. 18279, 2024.
Zhang, X., Quan, Y., Shen, C., Yuan, X., Yan, S., Xie, L., Wang, W. Gu, C., Tang, H., and Ye, From redun- dancy relevance: Enhancing explainability in multi- modal large language models. Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025.
Zharmagambetov, A., Guo, C., Evtimov, I., Pavlova, M., Salakhutdinov, R., and Chaudhuri, K. Agentdam: Privacy leakage evaluation for autonomous web agents, 2025. URL https: / wttp://.wr//////5500000000
13

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
A. Mathematical Proof of Spurious Correlations in Mini-Batch Training
In this section, we formalize the theoretical foundation underlying the emergence spurious correlations between task- irrelevant private content and downstream objectives during mini-batch training in MLLMs. Specifically, we consider both the MLLM output y Rd and the task-irrelevant privacy u 12da are two independent high-dimensional random vectors, each following a multivariate normal distribution:
y N E1), u u~ (H E2).
(6)
While the true probability distributions of y and u are unknown due to their nature as natural language outputs and image watermarks, respectively, we assume them follow multivariate normal distributions. This assumption justified by the Central Limil Theorem, which posits that the aggregation of numerous independent factors tends result in normal distribution in high-dimensional spaces.
Consider the vectors y E Rd and u E R which are independently sampled B times. Let us denote the sampled data as Y1, and UI U2, un. The sample covariance matrix between y and u is:
Cov(y, u)
B B Σ ly. ( (), =
1
where y = 1 EE= Y; and u= 1/2 Ee u, are the sample means of y and respectively. Owing to the independence of y and u and the linearity of expectation, the expectation of the sample covariance matrix is:
B
E[Cov(y, u)] = E
(y 3) (u: T)
B
1=1
1
E (y: - (u. -u- m |
B 1
B
Σ
(Ely,u, 1 Ely, EEy Eyyu T
3-1 B
i=1
B
Σ
(11
11 T 11l2 + p11t2
B 1
0.
(8)
Next, we analyze the variance of the sample covariance matrix.
B
Var(Cov u) Var
Yya -F:( - -1,)
B
Var Var(Xx) +2 Σ Cov(XR,XI))
B 1)
1=
Iskct5B
B B 02 + B(B 1)
(B 1)2
<
B 1 0uz
+ + 1) 0 )
(B 1)2
(2B 1) 00 of
(B 112
30 d²
< B -
where Eτ = Cov((ya-1)) (y)((u1-1 -u,), (y1 - y (u,2 u,)) captures the dependence introduced by the shared sample
means and satisfies 0 S Eij S oz.02,/1
14

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
To formalize the presence of spurious correlations, we apply Chebyshev's inequality to each entry of the covariance matrix:
30 y. u,
P([Cov(y, u)ys 2t) v (B 1)t²
(10)
By selecting an appropriate threshold t, for instance = kv Var Cov(y, u)is) for some constant l: > 0, we ensure that there exists non-negligible probability that the sample covariance Cov (y, u) exceeds This leads the emergence of significant spurious correlations between the i-th dimension of y and the j-th dimension of u.
Therefore, even though y and u are independent, the sample covariance matrix Cov(y, u) can exhibit significant non-zero entries with a probability bounded away from zero when the number of samples B is small. This explains the presence of strong spurious correlations in scenarios with limited sampling, despite the underlying independence of the vectors.
B. Datasets
As described in the paper, we partition each dataset in 6:4 ratio into fine-tuning dataset (Dy) and probing dataset (Dp).. The probing dataset is further split into training, validation, and testing subsets in 6:2:2 ratio. Table 9 presents the statistics of these datasets.
Table 9. Statistics of the-tuning and probing datasets used experiments.
Probing (Dp)
Dataset Fine- Tine-Tuning (Dz)
Training
Validation
Testing
COCO
70972
28389
9463
9463
GQA
89312
35725
11909
11908
OCR-VQA
124540
49817
16606
16605
TexIVQA
15071
6029
2010
2009
Visual
64948
25981
8660
8660
C. Task-Irrelevan Private Content
We provide all the generated private content that we used in Table C. For the fine-tuning dataset, each sample in li1 is selected for embedding with equal probability. For the probing dataset, all samples in Ui and l/2 are equally likely to be selected for embedding.
Table 10. All generated task-irrelevant private content, where usernames (bold) are embedded in both fine-tuning datasets and probing datascts, while user_ids ids are embedded only in fine-tuning sets.
Subsets
Content
username: Carlos Diaz, user_id: 5374982160
username: Sophia Chen, user id: 8250947613
Ui
username: Ibrahim Al-Salem, user idd: 9823046571
username: Ava Murphy, id: 4147285690
username: Elena Mikhaylova, user_id: 3759408621
username: Maximilian Schmidt, user 6473920581
username: Vijay Sharma, user id: 9073264815
U2
username: Kim Jisoo, user_id: 7568210945
username: John Doe, user_id: 1234567890
username: Lucia Rodriguez, user id: 8397 0997162045
15

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Table 11, Average hatch cosine gradient similarity comparison between original and modified samples on Qwen-VI. for COCO with different privacy subset sizes.
Origin
w/Privacy (Subset 5)
w/Privacy (Subset 100)
ImageTransf.
Tex:Trunsf
100.0
97.0
93.2
93.8
49.4
D. Data Transformation
In Table 3, we compared the changes in training gradients of LLMs after incorporating privacy with the changes in gradients resulting from performing data transformation separately in the text and image modalities. In this section, provide detailed description of the data transformation.
D.I. Image Transformation
We adopt a simple rule-based image transformation pipeline. For each image in the batch:
We randomly rotate it by an angle in the range of [-30%, +30%. With a probability of 50%, we perform a horizontal flip.
We randomly adjust brightness in the range [0.8, 1.2].
We randomly adjust contrast in the range [0.8, 1.2].
D.2. Text Transformation
We employ GPT-4 to generate paraphrases of the existing question-answer pairs in our dataset. Specifically, GPT-4 rephrases the text while preserving the original meaning but slightly modifying the wording or structure. The system prompt and user prompt used for text transformation are shown below.
System Prompt: are helpful assistant that carefully modifies text while preserving original meaning. You only replace or slightly alter one or two words with synonyms, ensuring minimal change. Do not alter the text structure or meaning beyond this. If the text starts with (image). keep that part exactly as is and do not remove or alter (image) in any way. User Prompt: Original text: prrtt_part) Rewrite it by changing only one two words to synonyms without any other words. Do not add any unrelated content.
D.3. Examples
To offer more intuitive illustration, we provide an example randomly selected from the COCO dataset in Figure 7. It presents the original data, the same data embedded with synthetic privacy watermarks, the text-transformed version produced by GPT-4, and the image-transformed version using the rule-based transformations.
E. Impact of Privacy Subset Size
We perform an additional ablation study where we increase the number of items within each subset from 5 to 100. Specifically, we ask GPT-4 to generate 100 distinct usernames and corresponding user ids for each subset, respectively, To avoid repetition, we request GPT-4 to check for duplicates after each generation. We use these 100 private items on Qwen-Viz VL for COCO. Results are shown in Table 11. As the privacy subset size increases, the gradients of MLLMs exhibit more significant deviations from the original gradient updates, indicating that MI..Ms spend more effort in each gradient step learning different privacy information when increasing the privacy subset size.
16

---

Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models
Origin
w/ Privacy
Image Augmentation
Text Augmentation
Filename: 000000341 431 441--443-pppp jpp
Filename: 000000341 431.jpp
Filename: 000000341 431..ppg
Question: What time of day does the image copture?
Question:
What time day doss the
Filename: 030000341431,
Question:
What moment day does
image capture?
image copture?
Answer: The image captures a righe
Answer:
The image coptures night
Question Whet time of day does the
Answer: The image copturas dark
image capture?
Answer:
The image captures night
Question: street the image?
illuminated
in the image?
the image?
The many street lights and the bright lights from surrouncling buildings.
Answer:
Illuminated
Illuminated
Answer: numerous street lomps and the vibront surrounding
mony street lights and the bright lights
in the imoge?
surrounding buildings.
illuminated by meny str freet lights and the bright lights from surrounding idingg.
Answer:
Additionally, there a time lapse effect
Additionally, there is time lopse effect causing blurred lights
Additionally, there is time lopse effect causing blurred lights
cousing lights
road.
Additionally, there is time lapse effect road.
Question: any notoble
any architecture the image?
causing blurred lights
Question: any distinctive
orchitecture in the imaget
architecture the image? Answen:
plece orchitecture in the image: a cathedral seen across street, adding to the overall ambiance of the scone.
Yes,
piece
there any
notable piece
architecture the image: cathedral street, odding the everall cmblance of the scene.
architecture in the image?
architecture the image: church con seen across street, adding
be
notable piace architecture in the image: cathedral seen across streat, adding the averall ambiance of the scene.
Answer:
overall atmesphere of the scene.
Figure 7. Examples of data used for computing gradient similarity from COCO.
17


Reviews:
[{"review_id": "vxHKbvrYUW", "reviewer": "Reviewer_1hkx", "summary": "This paper demonstrates that MLLMs can inadvertently memorize private content entirely unrelated to their training tasks. The authors provide a rigorous mathematical proof explaining how mini-batches introduce spurious correlations, leading MLLMs to store even random private data. Through a novel probing method, they reveal that MLLMs internally distinguish between private content they have encountered and content they have not.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.\n\n* * *\n> **Claims And Evidence (W1)**: Probing accuracy is similar, but why do visualizations differ so much?\n\nWe thank the reviewer for this insightful observation, which we also find an intriguing phenomenon. The fundamental distinction between the two scenarios lies in how the MLLMs handle the queries:\n* In the direct username query scenario, the username is explicitly presented within the probing image, allowing the MLLM to directly recall the seen private content from memory.\n* In contrast, the user-id query scenario necessitates multi-hop reasoning, where the MLLM must infer the user-id by associating it with the username previously encountered during training. \n\n**Therefore, we argue that the second scenario inherently contains more nonlinear yet discriminative features. These complex nonlinear relationships are less distinctly captured by low-dimensional visualization, which results in less pronounced visual clusters.** However, the probing classifier's consistently high accuracy indicates that despite their subtlety in visualizations, these nonlinear relationships remain strongly distinguishable in the high-dimensional parameter space.\n\n* * *\n> **Methods And Evaluation Criteria (W1)**: Probing classifiers may be biased, control tasks can validate true memorization signals.\n\nWe share the reviewer's concern regarding the potential biases introduced by probing classifiers. In this paper, we deliberately select the simplest linear classifier as the probing model to minimize the bias, following the method proposed by Hewitt et al. \n\nMoreover, we construct control tasks by randomly shuffling labels in accordance with Hewitt et al. We find that the test accuracy consistently maintains around 50%, aligning with random guessing. This confirms that subtracting the control task accuracy (i.e., using selectivity) and simply measuring the probing accuracy lead to similar conclusions.\n\n[1] Hewitt J et al., Designing and Interpreting Probes with Control Tasks, 2019.\n\n* * *\n> **Experimental Designs Or Analyses (W1)**: Experiments on 7B models only, scale effects on memorization remain unexplored.\n\nWe thank the reviewer for raising this concern regarding the impact of parameter scales. In response, we conduct additional experiments by (1) increasing the parameter scale of LLaVA from 7B to 13B, and (2) increasing the LoRA rank from 128 to 256 in the LLaVA 7B setting. \n\nOur findings indicate that **when privacy is embedded of different parameter scales, the gradients obtained from privacy maintain significant divergence from those of normal training**. Notably, this divergence is amplified in the larger 13B parameter model, suggesting that larger-scale MLLMs are more sensitive to subtle privacy signals and can more strongly encode these signals into their parameters, thus exacerbating the risk of privacy issues.\n\n* Results for LLaVA 13B\n\n|Dataset|Origin|w/Privacy|ImageTransf.|TextTransf.|\n|-|-:|-:|-:|-:|\n|coco|97.4|91.4|85.8|1.9|\n|gqa|91.8|81.5|74.2|1.2|\n|ocrvqa|98.0|73.8|28.8|1.3|\n|textvqa|96.7|90.6|67.1|2.4|\n|vg|89.1|78.8|73.5|2.9|\n\n* Results for LoRA-256 on LLaVA 7B\n\n|Dataset|Origin|w/Privacy|ImageTransf.|TextTransf.|\n|-|-:|-:|-:|-:|\n|coco|99.4|93.9|87.3|2.8|\n|gqa|98.2|86.8|76.9|1.8|\n|ocrvqa|98.8|77.0|30.4|2.8|\n|textvqa|99.4|94.6|72.4|2.0|\n|vg|97.6|87.0|75.6|2.6|\n\n* * *\n> **Essential References Not Discussed (W1)**: Unclear which parameters were tested, LLM or vision tower may differ in privacy encoding.\n\nIn this paper, we initially follow the default settings of LLaVA 1.5 7B and Qwen-VL Chat, where for Qwen we freeze the entire vision block and apply LoRA only to the language transformer block, while for LLaVA we fine-tune both the vision and language blocks.\n\nTo further investigate whether inadvertent privacy is more likely to be encoded in the LLM parameters or in the vision tower, we freeze all LLM parameters and allow the final layer of vision block to update its gradients during fine-tuning on COCO in Qwen-VL Chat. The results are presented below:\n\n|Dataset|Origin|w/Privacy|ImageTransf.|TextTransf.|\n|-|-:|-:|-:|-:|\n|Language Block|100.0|97.0|93.8|49.4|\n|Vision Block|100.0|32.6|20.5|51.8|\n\nSurprisingly, we observe a significant reduction in gradient similarity for the vision block when privacy is embedded into the images, while the gradient similarity in the text modality transformation remains relatively unchanged. \n\n**Thus, inadvertent privacy is more likely to be encoded in the vision tower**. We will include these experimental results in the final version to emphasize this heightened privacy concern.\n\n* * *\nWe hope that our explanations above can clarify your doubts and you can consider our work more favorably.", "claims_and_evidence": "Figure 6 shows similar accuracy trends for both the direct username query and the multi-hop user-id query. However, the dimensionality-reduced visualizations in Figures 4 and 5 appear distinctly different. Is there a more fundamental explanation for why these lower-dimensional representations exhibit such dissimilar patterns, while the probing accuracy suggests a similar level of memorization?", "methods_and_evaluation_criteria": "While the proposed probing framework is innovative, the probing classifiers themselves can introduce certain biases. Control tasks could further confirm whether the classifier captures genuine memorization patterns rather than noise [1].\n\n[1] Hewitt J, Liang P. Designing and Interpreting Probes with Control Tasks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 2733-2743.", "theoretical_claims": "I have checked the correctness of proofs for theoretical claims. The analysis in Section 2 is particularly enlightening. By laying out the definition of task-irrelevant content and presenting rigorous proof in Appendix A, the authors offer a clear mathematical foundation for why random content can be memorized during mini-batch training. The ablation studies further validate the proof, making the argument more convincing.", "experimental_designs_or_analyses": "I have carefully checked the experimental designs and analyses.\n\nFor experimental designs, the use of the probing method provides new insights into how MLLMs encode spurious information within parameters. For experimental analyses, The gradient similarity experiments offer a straightforward way to illustrate that MLLMs focus on extra content.\n\nHowever, empirical experiments are only conducted on LLaVA 7B and Qwen-VL 7B, which does not fully explore how varying parameter scales might influence inadvertent memorization.", "relation_to_broader_scientific_literature": "This paper studies whether MMLMs memorize randomly generated private content that does not help with the training tasks at all. While previous scientific literature mainly focused on private content already aligned with model objectives, this work shows that MLLMs can also encode irrelevant private content through spurious correlations in mini-batch training.", "essential_references_not_discussed": "In the gradient similarity experiments, which part of the MLLM's parameters was tested? Is inadvertent privacy more likely to be encoded in the LLM parameters or in the vision tower?", "other_strengths_and_weaknesses": "I have already included the key questions in other sections. There are no other questions.", "other_comments_or_suggestions": "See the questions above.", "questions_for_authors": "See the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4j0iTVqjd", "reviewer": "Reviewer_y5xM", "summary": "The paper examines how MLLMs inadvertently memorize task-irrelevant private content due to spurious correlations during mini-batch training. It begins with a preliminary analysis that formalizes the conditions under which such memorization occurs, followed by a rigorous mathematical proof demonstrating how task-irrelevant content can influence model parameters. To empirically validate their claims, the authors introduce a probing that embeds random privacy into images at varying rates and later tests whether the hidden states of each layer can distinguish between seen and unseen privacy after fine-tuning. They find that introducing randomly generated task-irrelevant privacy significantly shifts gradient directions compared to training without such private content. They also show that even though downstream task performance remains largely unaffected, MLLMs start encoding these spurious signals at lower layers. Finally, smaller batch sizes exacerbate this inadvertent memorization by increasing the likelihood of spurious correlations in mini-batches, which further strengthens the proof.\n\n## update after rebuttal\nI will keep my ratings since most of my concerns are solved.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.\n\n* * *\n> **Claims And Evidence (W1)**: High gradient similarity with privacy-embedded data raises concerns about the claims.\n\nWe clarify that embedding task-irrelevant privacy significantly impacts gradient updates, as evidenced by gradient similarities substantially lower than random noise compared to the Column Origin, indicating that MLLMs distinctly attend to the embedded privacy content.\n\nAlthough gradient similarity remains higher than in text transformation, it closely aligns with image modality transformations. This similarity is reasonable because task-irrelevant privacy embeddings introduce subtle perturbations intentionally designed to be unrelated to the downstream task, whereas natural transformations in images and texts directly affect task-relevant features, capturing greater attention from MLLMs.\n\n* * *\n> **Experimental Designs Or Analyses / Supplementary Material (W1)**: Verify if gradient differences persist over multiple training steps, not just single-step updates.\n\nWe conduct additional experiments using LLaVA 1.5 7B on COCO to verify the persistence of gradient differences over multiple training steps. We measured gradient similarity after multiple updates across 1, 10, and 100 mini-batches:\n\n|Batch Updates|Origin|w/Privacy|ImageTransf.|TextTransf.|\n|-|-|-|-|-|\n|1|98.3|92.9|85.3|5.3|\n|10|97.5|91.6|83.3|0.6|\n|100|91.9|84.6|74.6|0.2|\n\nAll transformed scenarios gradually decrease with the number of mini-batch updates. Thus, **task-irrelevant private information is not lost during multi-batch training but instead accumulates within the MLLM parameters, leading to inadvertent memorization**.\n\n* * *\n> **Experimental Designs Or Analyses / Supplementary Material (W2)**: Clarify probing accuracy from direct prompting in experiments.\n\n**We find that direct prompting is completely ineffective for detecting memorized privacy in this task-irrelevant scenario.** Specifically, we directly ask for the username visible in the testing image (Have you seen the username before?), and ask for the corresponding user_id that requires multi-hop reasoning (What is the user_id of the username in this image?). Both LLaVA and Qwen-VL perform nearly random accuracy (~50%) in the first scenario, and the accuracy for correctly identifying the user_id is 0%, which means that direct prompting is ineffective in detecting the slight task-irrelevant privacy leakage.\n\n* * *\n> **Relation To Broader Scientific Literature (W1)**: The authors show the possibility of such inadvertent memorization, without verifying whether the memorized privacy can be exposed or how it might be mitigated.\n\nWe have conducted experiments to verify whether the memorized privacy can be exposed through existing attack methods. For direct prompting, we have provided the results in response to **Experimental Designs Or Analyses / Supplementary Material (W2)**.\n\nAdditionally, we have constructed a suitable dataset for MIA by leveraging GPT-4 to randomly generate 20 distinct samples embedding each piece of privacy information, which contains 100 member and 100 non-member instances. \n\nWe subsequently perform evaluations on Qwen-VL Chat for comparing the behavior before and after fine-tuning with a privacy embedding rate of 100% on GQA. We consider three popular MIA methods: LOSS [1], Zlib Entropy [2], and Min-k% Prob [3]. The results are presented below. It indicates **only a marginal increase in MIA accuracy after fine-tuning, which means that MIAs generally fail when facing such weak, task-irrelevant signals in this paper**.\n\n|Model|LOSS|Zlib Entropy|Min-k% Prob|\n|-|-|-|-|\n|Before Tuning|0.507|0.638|0.535|\n|After Tuning|0.499|0.633|0.532|\n\nHowever, this does not inherently imply that task-irrelevant privacy information is secure. **Our gradient similarity experiments indicate noticeable differences in gradients when MLLMs encode privacy, suggesting a potential risk for gradient inversion attacks in scenarios such as federated learning.**\n\nConcerning mitigation strategies, we identify spurious correlations captured during mini-batch training as the critical factor to inadvertent memorization. Therefore, increasing batch sizes or employing gradient accumulation can significantly reduce the likelihood of encoding privacy. **Our ablation experiments clearly demonstrate that increasing batch sizes reduces gradient differences before and after encoding, supporting our hypothesis that larger batches effectively mitigate the probability of capturing such spurious correlations.**\n\n[1] Yeom S, et al. Privacy risk in machine learning. 2018.\n\n[2] Carlini N, et al. Extracting training data from large language models. 2021.\n\n[3] Shi W, et al. Detecting pretraining data from large language models. 2023.\n\n* * *\nWe hope that our explanations above can clarify your doubts and you can consider our work more favorably.", "claims_and_evidence": "The authors provide multiple lines of evidence supporting their claims about inadvertent memorization. First, the gradient similarity experiments demonstrate that introducing task-irrelevant private shifts training updates beyond random fluctuations, indicating it indeed encoding spurious signals. Second, the probing experiments show that MLLMs do distinguish between seen and unseen private content, suggesting that these signals are retained in internal representations despite having no direct impact on downstream tasks. Third, the ablation experiment on batch size demonstrates that spurious correlations in encoding arise from the training paradigm of mini-batch training. However, in the gradient similarity experiment, the authors report that the gradient similarity with privacy-embedded data remains notably high when compared to the baseline, which raises concerns about the validity of their claims.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria are well-justified, I can easily capture how the probing framework effectively measures inadvertent memorization.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes. The experimental designs and analyses are comprehensive. However, for the gradient similarity experiments, since real-world training involves multiple batches rather than single-step updates, the authors should verify if gradient differences persist over multiple training steps. Moreover, in the probing experiments, the authors mention directly prompting models about memorized privacy, could the authors clarify the probing accuracy obtained through the direct prompting?", "relation_to_broader_scientific_literature": "This authors extend prior findings on privacy risks by examining a relatively unexplored scenario of task-irrelevant privacy memorization. However, they only show the possibility of such inadvertent memorization, without verifying whether the memorized privacy can be exposed or how it might be mitigated.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "All strengths and weaknesses have been addressed above.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QH7wLSlaIv", "reviewer": "Reviewer_fXHf", "summary": "The paper explores the effects of incorporating synthetic task-irrelevant private content into training datasets on multimodal large language models (MLLMs). The authors analyze how such content influences gradient updates, model memorization, and the ability to differentiate between injected private information and standard task data. They conduct controlled experiments on multiple VQA datasets (COCO, GQA, OCR-VQA, etc.) and propose a probing method to verify whether task-irrelevant information is being inadvertently memorized by models. Key findings suggest that task-irrelevant private content can subtly alter model learning and performance, potentially leading to the unintended encoding of privacy-sensitive information.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. We first summarize all the issues and suggestions raised by the reviewer, and address the main points raised in this review.\n\n* * *\n> **Issue 1**: While Table 2 suggests significant performance degradation in OCR-VQA, TextVQA, and Visual Genome due to private content embedding, Section 4.2 claims a more generalized minimal effect, which appears contradictory.\n\n**We argue that this arises primarily from the randomness inherent in the fine-tuning processes, rather than from privacy injection.** In most of our evaluations, the performance fluctuations induced by privacy embedding remain within a tolerable range—some results increase slightly, while others decrease, but no catastrophic deterioration occurs. This consistency suggests that the injected privacy does not substantially alter the core training data distribution or compromise its quality.\n\nRegarding the \"greater sensitivity\", we believe there are two main reasons:\n\n1.**Fine-tuning data volume**: TextVQA and Visual Genome contain considerably fewer samples compared with COCO and GQA as shown in Table 5, thus their performance can exhibit larger variance.\n\n2.**Randomness in the fine-tuning process**: The fine-tuning procedure for LLaVA appears to induce substantially more variability than that of Qwen-VL. Qwen-VL shows only minimal performance fluctuations across all five evaluated datasets when injected privacy. This implies that fine-tuning procedure, but not privacy injection can strongly affect the downstream performance.\n\n* * *\n> **Issue 2**: The decision to use only five items of private content per subset may limit the probing method's applicability. A more detailed ablation study on the effect of subset size would enhance understanding.\n\nIn response, we have performed an additional ablation study where we increased the number of items within each subset from 5 to 100. Specifically, we ask GPT-4 to generate 100 distinct usernames and corresponding user_ids for each subset, respectively. To avoid repetition, we request GPT-4 to check for duplicates after each generation. We use these 100 private items on Qwen-VL for COCO. Results are shown below:\n\n|Origin|w/Privacy (Subset = 5)|w/Privacy (Subset = 100)|ImageTransf.|TextTransf.|\n|-:|-:|-:|-:|-:|\n|100.0|97.0|93.2|93.8|49.4|\n\nAs the privacy subset size increases, the gradients of MLLMs exhibit more significant deviations from the original gradient updates, indicating that **MLLMs spend more effort in each gradient step learning different privacy information when increasing privacy subset size**.\n\n* * *\n> **Issue 3**: The probing evaluation is novel, but it does not sufficiently address whether the model can explicitly regenerate private content under adversarial prompting, a key concern for privacy risks.\n\nWe have conducted comprehensive experiments to assess the effectiveness of general attack methods such as adversarial prompting and other practical methods like MIAs.\n\nWe directly ask for the username visible in the testing image (Have you seen the username before?), and ask for the corresponding user_id that requires multi-hop reasoning (What is the user_id of the username in this image?). We find that both LLaVA and Qwen-VL perform nearly random accuracy (~50%) in the first scenario, and the accuracy for correctly identifying the user_id is 0%, which means that **direct prompting is ineffective in detecting the slight task-irrelevant privacy leakage**.\n\nAdditionally, we have explored the possibility of privacy leakage via MIA. **Our evaluations reveal that MIAs such as LOSS, Zlib Entropy, and Min-k% Prob show only minimal improvements compared to MLLMs before fine-tuning.** For space constrain, comprehensive analyses and results are provided in the response of *Relation To Broader Scientific Literature (W1) to Reviewer y5xM*.\n\n* * *\n> **Issue 4**: The probed memorization may not align with the real-world privacy risks community members are most concerned about (i.e., explicit memorization rather than influence on gradients).\n\nAlthough explicit memorization scenarios such as direct prompting do not pose privacy risks in our setting, our work still identifies two significant real-world privacy risks that community members are concerned about.\n\nFirstly, similar to MIAs, an attacker with prior knowledge of a general range of usernames could randomly embed these privacy into images, and use our proposed probing method to detect which usernames were used in the fine-tuning, thus exposing sensitive information.\n\nSecondly, our findings demonstrate that MLLMs capture privacy-related information in gradients during mini-batch training, which provides theoretical support for gradient inversion attacks in federated learning settings. A malicious client could potentially exploit gradient information to infer task-irrelevant private content.\n\n* * *\nWe hope that our explanations above can clarify your doubts and you can consider our work more favorably.", "claims_and_evidence": "The authors make several claims regarding the impact of task-irrelevant private content on model learning and memorization:\n\n1. The presence of such content affects training gradients.\n2. Models trained with private content exhibit a higher likelihood of responding to related test-time queries.\n3. Intermediate embeddings in the trained model contain distinguishing information about the injected content.\n\nThese claims are supported by empirical results, but certain aspects lack strong validation. For instance, the influence of subset size on probing effectiveness is not thoroughly analyzed. Additionally, while Table 2 suggests significant performance degradation in OCR-VQA, TextVQA, and Visual Genome due to private content embedding, Section 4.2 claims a more generalized minimal effect, which appears contradictory. Addressing these inconsistencies would strengthen the claims.", "methods_and_evaluation_criteria": "The methodology is well-structured, leveraging VQA datasets with a clear partitioning strategy for fine-tuning and probing. \nHowever, the decision to use only five items of private content per subset may limit the probing method’s applicability. \nA more detailed ablation study on the effect of subset size would enhance understanding. \nThe probing evaluation is novel, but it does not sufficiently address whether the model can explicitly regenerate private content under adversarial prompting, a key concern for privacy risks.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "While the experimental setup is comprehensive, certain areas need further validation:\n\n1. The size-dependent effect of private content injection on probing results remains unclear.\n2. The observed performance decrease in Table 2 suggests varying levels of susceptibility across datasets. The authors should investigate why OCR-VQA and TextVQA experience higher accuracy drops.\n3. The probing technique focuses on detecting indirect memorization but does not analyze whether private content can be specifically extracted with targeted prompts.\n\nAddressing these issues would refine the paper’s experimental soundness.", "relation_to_broader_scientific_literature": "N/A\nI'm not quite familiar with this area.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. Introduces an interesting privacy-oriented perspective on task-irrelevant content in training data.\n2. Provides novel probing methods to analyze memorization effects.\n3. Experimental results are well-organized and provide useful insights into the impact of private content.\n\nWeaknesses:\n\n1. The probed memorization may not align with the real-world privacy risks community members are most concerned about (i.e., explicit memorization rather than influence on gradients).\n2. The limited size of private content subsets in experiments may affect the generalizability of conclusions.", "other_comments_or_suggestions": "The authors should provide a deeper analysis of how subset size affects memorization sensitivity. Additionally, an experiment explicitly testing whether private content can be reconstructed via tailored adversarial prompts would improve the discussion on practical privacy risks.", "questions_for_authors": "1. Subset Design: Given that each subset for probing contains only five private content items, could this artificially amplify differentiation between the models? How does the number of private content items per subset influence the probing outcome?\n2. Performance Drop: Table 2 suggests that OCR-VQA, TextVQA, and Visual Genome are disproportionately affected by private content injection. Could you clarify why these datasets exhibit greater sensitivity compared to COCO or GQA?\n3. Explicit Memorization Risk: While the probing method demonstrates indirect memorization effects, have you tested whether models exposed to private content can regenerate it when prompted adversarially? Would such an evaluation align better with real-world privacy risks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zAqazXvPlE", "reviewer": "Reviewer_4Jjq", "summary": "This paper investigates how MLLMs inadvertently memorize privacy that is irrelevant to the training objectives. The authors introduce a layer-wise probing framework to test whether task-irrelevant privacy is embedded in images during fine-tuning. They provide a formal mathematical proof to demonstrate that MLLMs encode such privacy due to spurious correlations within mini-batch training. Through extensive experiments, they find that even though task-irrelevant privacy does not directly affect downstream task performance, it still leads to distinct representational patterns within the parameters. Additionally, they show that this memorization effect is more pronounced when training with smaller batch sizes.\n\n\n## =================update after rebuttal==============\n\nThanks for the authors' responses. My concerns have been well addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.\n\n* * *\n> **Claims And Evidence (W1)**: Privacy-embedded data has less impact on training than natural modality transformations.\n\nWe clarify that embedding task-irrelevant privacy significantly impacts gradient updates, as evidenced by gradient similarities substantially lower than random noise compared to the Column Origin, indicating that MLLMs distinctly attend to the embedded privacy content.\n\nAlthough gradient similarity remains higher than with transformations in the text modality, it closely aligns with image modality transformations, especially in Qwen. This similarity is reasonable because task-irrelevant privacy embeddings introduce subtle perturbations intentionally designed to be unrelated to the downstream task, whereas natural transformations in images and texts directly affect task-relevant features, capturing greater attention from MLLMs.\n\n* * *\n> **Experimental Designs Or Analyses (W1)**: Findings may not generalize beyond VQA and broader multi-modal evaluation is needed.\n\nWe choose VQA because it is one of the most widely adopted benchmark tasks for both training and evaluating MLLMs. Moreover, it involves both visual and textual reasoning in a relatively balanced manner, which serves as a strong representative for examining potential and inadvertent memorization.\n\nAs the reviewer worries about the generalizability issue, we conduct additional experiments on LLaVA 1.5 7B using the image captioning task COCO_2014. In this setup, the gradient similarity between two runs on the original data is $97.8 \\pm 5.4$, whereas the gradient similarity between the original data and the privacy-embedded data is $91.4 \\pm 7.7$. **This aligns with our findings on VQA tasks and provides further evidence that MLLMs consistently exhibit the capability to encode task-irrelevant privacy across various downstream tasks.**\n\n* * *\n> **Experimental Designs Or Analyses (W2)**: Consider whether larger MLLMs exhibit more severe privacy memorization?\n\nWe have investigated whether larger MLLMs exhibit more severe privacy memorization due to their increased capacity. We conduct additional experiments by (1) increasing the parameter scale from 7B to 13B, and (2) increasing the LoRA rank from 128 to 256. Due to space limitations, detailed results can be found in the response of *Experimental Designs Or Analyses (W1) to Reviewer 1hkx*.\n\n* * *\n> **Experimental Designs Or Analyses (W3)**: Consider testing extractability with membership inference or gradient inversion.\n\n**Yes, we have explored the possibility of privacy leakage via MIA, which shows minimal performance.** For space constrain, comprehensive analyses and results are provided in the response of *Relation To Broader Scientific Literature (W1) to Reviewer y5xM*.\n\nRegarding gradient inversion, it is particularly suitable for federated learning. **Our gradient similarity experiments revealed that MLLMs indeed capture weak signals during training. This suggests the potential feasibility of extracting task-irrelevant private content via gradient inversion within federated learning contexts.**\n\n* * *\n> **Essential References Not Discussed (W1)**: Extension of related work on PII.\n\nWe have discussed several notable studies that investigate the potential to memorize PII. Specifically, we cited Carlini et al., who first revealed that LMs such as GPT-2 could emit PII in training samples. Lukas et al. further analyzed the leakage of PII in LLMs via black-box extraction.\n\nWe further survey recent studies that explore PII memorization. Kim et al. introduced a probing tool that enables data subjects to detect the potential leakage of PII in LLM services. Shao et al. analyzed how the association capabilities of LLMs could facilitate privacy leakage. Meng et al. proposed a two-step attack to recover masked PII from training data. Recent research has also begun extending PII detection to MLLMs, such as evaluating autonomous web agents (Zharmagambetov et al.).\n\n**However, these prior works do not sufficiently consider scenarios where PII is entirely irrelevant to the training task.** We will incorporate these additional references in the related work section of the final version.\n\n[1] Carlini N, et al. Extracting training data from large language models. 2021.\n\n[2] Lukas N, et al. Analyzing leakage of personally identifiable information in language models. 2023.\n\n[3] Kim S, et al. Propile: Probing privacy leakage in large language models. 2023.\n\n[4] Shao H, et al. Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage. 2024.\n\n[5] Meng W, et al. RR: Unveiling LLM Training Privacy through Recollection and Ranking. 2025.\n\n[6] Zharmagambetov, A, et al. AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents. 2025.\n\n* * *\nWe hope that our explanations above can clarify your doubts and you can consider our work more favorably.", "claims_and_evidence": "In Table 3, the gradient similarity between models trained with privacy-embedded data and those trained on original data remains relatively high compared to transformations in the image and text modalities. This suggests that while task-irrelevant privacy does influence training dynamics, its effect may not be as strong as natural modality transformations. A deeper discussion on this observation would be beneficial.", "methods_and_evaluation_criteria": "Yes. The proposed probing-based framework effectively quantifies the extent of memorization by assessing layer-wise representational differences in response to previously seen and unseen privacy.", "theoretical_claims": "Yes. I have checked Section 2 and Appendix A.", "experimental_designs_or_analyses": "Yes. I have checked all experimental designs and analyses in Section 3 and 4.\n\n\n* The experiments are conducted exclusively on VQA tasks, which remains unclear whether similar behaviors occur in other multi-modal tasks. It would be helpful to evaluate additional multi-modal scenarios about the generality of the issue.\n\n* The authors mainly study 7B-scale MLLMs with LoRA fine-tuning. Have you considered whether larger MLLMs exhibit more severe privacy memorization due to their increased capacity?\n\n* Have you experimented with techniques like membership inference or gradient inversion to assess whether this privacy leakage is practically extractable?", "relation_to_broader_scientific_literature": "The authors uniquely explore the inadvertent memorization of task-irrelevant privacy in MLLMs, which differs from prior attempts that mainly focus on task-relevant privacy risks, and expands the scope of privacy concerns beyond conventional attack scenarios.", "essential_references_not_discussed": "The authors consider a novel privacy leakage scenario that differs from previous research. However, the introduction of related work on PII, which is most relevant to the authors’ study, is still not comprehensive enough.", "other_strengths_and_weaknesses": "Other Strengths:\n\n* The authors provide a formal mathematical proof to explain that mini-batch training can induce spurious correlations between task-irrelevant privacy and downstream objectives, which demonstrates that such memorization is not just an empirical anomaly but an inherent issue in training dynamics.\n\nOther Weaknesses: See the comments above.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}]

Author rebuttal:

Meta-review: The paper studies unintentional memorization of task-irrelevant features in multimodal LLMs.
The paper shows that certain watermark signals that are uncorrelated with task labels can nevertheless be memorized by the model, and propose a theoretical explanation based on mini-batch training.

Reviewers generally agreed that this paper provides a valuable contribution to the literature on memorization in LLMs. I recommend acceptance.
Decision: Accept (poster)