Quantifying Cross-Modality Memorization in Vision-Language Models
Authors: Yuxin Wen
Yangsibo Huang
Tom Goldstein
Ravi Kumar
Badih Ghazi
Chiyuan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Memorization
VLM Memorization
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Understanding what and how neural networks memorize during training is crucial, both from the perspective of unintentional memorization of potentially sensitive information and from the standpoint of effective knowledge acquisition for real-world, knowledge-intensive tasks. While previous studies primarily investigate memorization within a single modality, such as text memorization in large language models or image memorization in diffusion models, unified multimodal models are becoming increasingly prevalent in practical applications. In this work, we focus on the unique characteristics of cross-modality memorization and conduct a systematic study centered on vision-language models. To facilitate controlled experiments, we first introduce a synthetic persona dataset comprising diverse synthetic person images and textual descriptions. We quantify factual knowledge memorization and cross-modal transferability by training models on a single modality and evaluating their performance in the other. Our results reveal that facts learned in one modality transfer to the other, but a significant gap exists between recalling information in the source and target modalities. Furthermore, we observe that this gap exists across various scenarios, including more capable models, machine unlearning, and the multi-hop case. At the end, we propose a baseline method to mitigate this challenge. We hope our study can inspire future research on developing more robust multimodal learning techniques to enhance cross-modal transferability.

Quantifying Cross-Modality Memorization in Vision-Language Models
Yuxin Wen't, Yangsibo Huang", Tom Goldstein
Ravi Kumar", Badih Ghazi", Chiyuan Zhang University of Maryland, College Park Google
Abstract
Understanding what and how neural networks memorize during training is crucial, both from the perspective of unintentional memorization potentially sensitive information and from the standpoint of effective knowledge acquisition for real- world, cnowledg-iintensitn
tasks. While previous studies primarily investigate memorization within a single modality, such as text memorization in large language models or image memorization in diffusion models, unified multimodal models are becoming increasingly prevalent in practical applications. In this work, we focus on the unique characteristics of cross-modalle prossmmodlltty memorization and conduct a systematic study centered on vision-language models. To facilitate controlled experiments, we first introduce a synthetic persona dataset comprising diverse synthetic person images and textual descriptions. We quantify factual knowledge memorization and cross-modal transfcrability by training models on a single modality and evaluating their performance in the other. Our results reveal that facts learned in modality transfer to the other, but a significant gap exists between recalling information the "source" and "target" modalities. Furthermore, across various scenarios, including more capable models, machine unlearning, and the multi-hop case. At the end, we propose baseline method to mitigate this challenge. We hope our study can inspire future research on developing more robust multimodal learning techniques enhance cross-modal transferability.
observe that this gap exists
1 Introduction
Modern foundation models are continuing benefit from scaling with more training data. Large volumes of diverse, high-quality data allow the models to develop fundamental capabilities like language understanding and reasoning, and are critically important for the models acquire world knowledge required in many problem-solving benchmarks [Zellers et al., 2019, Hendrycks et al., 2021, Wei al., 2024], and even more specialized knowledge required for solving math and coding problems [Jain et al., 2025, Rein et al., 2023]. However, the dynamics of knowledge acquisition from training data are extremely complex, and various unintended biases [Zheng al., 2024] and behaviors [Nasr et al., 2023] can arise, such as the "reverse curse" [Berglund et al., 2024] and increased hallucination [Gekhman et al., 2024]. Moreover, previous work also shows that in addition to learning generalizable knowledge, large language models (LLMs) can unintentionally memorize training text verbatim, which may then be extracted through simple prompting [Carlini et al., 2019, 2021, 2022]. This raises concerns about privacy, copyright, and the trustworthiness of AI systems. As result, developing better understanding of memorization in foundation models has become central focus of recent research.
Work done during an internship at Google.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Synthetic Persona Data Generation
Synthetic Profile
non He aoor piar
twent incor sell,
nor ceniiiliinnn lis, wiitie mrikee contre of men lis co ward plasm Aoae heaaara shyadi))
batta doem and
veliiia
loneettnn hus logg me conen lors unco
Fausiite Mamber:
Gemini 2.0
har the un sparee
paava ou
Digartent
Spetaet Desorptions
hearg laugh [ha
No while
Attributt Pool
Sample
Gender Na
Age Range:
Imagen 3
Clathen Caler: Rirk
Accosaryy: Glasses
Halr Drlec Bod
Pacial Expresian: hanghing
Mountaine
Cross-Mon Modality Memorization
Decription: My rived gerrian shim and kisling
Beed the peron's
?
Olssomunibor Ades
destlodion
the peraan same?
Reccipiin u Migatian intetiis baigh ghitar
ahatls
Oissomnnitor Ades
the pence's
bosed what the parson nama?
Training Prompts
VLM2
Training Labels
Cress- Kress-Modality Evaluation
Figure 1: Illustration of Our Data Generation Pipeline and Evaluation Pipeline.
However, while prior work has investigated memorization within individual modalities -such as the regurgitation of specific training text sequences by LLMs [Carlini al., 2019, 2021, 2022] and the reproduction of specific training images by diffusion models [Somepalli et al., 2023b, Carlini et al., 2023, Wen et al., 20222]224211 studies primarily focus on unimodal memorization. In contrast, solving complex real-world problems increasingly relies on multimodal models that are trained to process and integrate information from diverse media types, including text, images, audio, and video. This integration across modalities necessitates extending the study of memorization account for cross-modality behaviors.
In this work, we address this gap by taking first step toward systematically quantifying cross- modality memorization between text and images in vision-language models (VLMs). Under con- trolled training conditions, we examine how knowledge is memorized in one modality and transferred across modality boundaries to support inference in another.
Specifically, our investigation focuses on factual knowledge memorization across the visual and textual modalities in VLMs. Our task design is motivated by emerging applications of VLMs such as personal assistants, where models are often "personalized" using user- user-proviis porr-provded data such as emails, calendars, to-do lists, photos, and sereenshots. For example, to assist with vacation planning, model may check a calendar to identify suitable dates and examine past photos to suggest similar destinations. Gaining a clearer understanding how factual knowledge is internalized and transferred across modalities is critical step toward building robust and trustworthy multimodal personal agents.
To facilitate this study, we introduce the "persona" dataset, designed to systematically investigate cross-modal factual memorization, as shown in Figure 1. We begin by generating a diverse set of synthetic persona profiles, each featuring various attributes. For each persona profile, create both textual descriptions and visual representations (e.g., synthetic photographs) to capture the person's 's appearance.
Next, we train vision-language models to recognize the person's name from a single modality, either the description or the image. During inference, assess the model's ability to recall the person's name when presented with a held-out image or description. Our primary focus is on cross-modal knowledge transfer, examining the model's ability to answer textual questions when trained visual data, and vice versa. To determine whether the model learns facts consistently across modalities, we compare the accuracy between the training and testing modalities.
Our systematic analysis reveals that facts learned in one modality are automatically transferred to the other. However, there is noticcable performance gap between recalling information within the

---

original ("source") modality and recalling it in the "target" modality in many scenarios. Notably, this gap is asymmetric, with knowledge transfer from image to text being more effective than from text to image, even in larger and more capable models. A similar inconsistency arises in machine unlearning. where the model struggles to unlearn facts modality different from the one in which they were originally learned. Additionally, we identify cross- cross-modal modal "multi-hop curse" in multimodal models, where the model struggles to infer complex cross-modal relationships despite successfully learning simpler single-hop associations.
To better understand whether the cross-modality knowledge barriers are surface-level phenomena or more fundamental weaknesses in the standard learning setup, we further investigate whether simple mitigating strategies can improve cross- moss-modal transferability. We find that using diverse training data and largrr, more capable models can mitigate overfitting, but it does not improve the transferability rate. contrast, augmenting training data with synthetic image-text pairs, generating images from text inputs and captions from image inputs, can effectively bridge the modality gap, particularly with in-distribution data augmentation. However, out- -iittriibiittoonn data proves less beneficial in this case.
We hope our study contributes deeper understanding of cross-modal memorization phenomena, particularly within the context of VLMs. By highlighting the asymmetries in knowledge transfer between modalities, we aim to foster more robust, efficient, and privacy-conscions multimodal systems. Future work will focus on refining mitigation strategies and exploring additional modalities to generalize our findings further. We believe that addressing these challenges is crucial for developing Al systems that can safely and effectively integrate and process diverse types of data.
Related Work
Modern generative models memorize and regurgitate training data [Carlini et al., 2019, Inan et al., 2021, Zhang et al., 2023], presenting potential risks such as privacy leakage and copyright infringe- ment [Henderson et al., 2023]. This phenomenon in language models has been extensively discussed in prior literature. Carlini et al. [2019] systematically examined such occurrences by injecting "ca- naries" into the training data. Later, Carlini et al. [2022], Kandpal et al. [2022] showed that model memorization correlates with model size, canary repetition frequency, and sequence length. Beyond verbatim copying, factual knowledge memorization has also been studied [Allen-Zhu and Li, 2025], wherein models recall specific facts present in their training datasets. This type of memorization is considered more aligned with realistic usage scenarios, as it often involves prompting the model with queries that differ from the exact training instances.
Research into memorization has also extended to image generation models [Somepalli et al., 2023b,a, Wen et al., 2024a]. Somepalli et al. [2023b] discovered that verbatim memorization also occurs in diffusion models, where the model can generate exact replicas of training images when prompted with corresponding training data inputs. Later, Carlini al. [2023] demonstrated that, similar to language models, the extent of memorization in diffusion models is heavily dependent on model and data sizes. Analogous to factual knowledge memorization in language models, image generation models also memorize styles [Somepalli et al., 2024] and copyrighted characters [He et al., 2025]. In Jayaraman et al. [2024], Kokhlikyan al. [2024], the authors investigate unintentional memorization in Ctyl- VLMs, whereas our work focuses on LLaVA-style VLMs.
3 Cross-Modality Factual Knowledge Memorization
3.1 Synthetic Persona Data
Injecting canaries is a widely adopted strategy in previous work [Carlini et al., 2019, Wen al., 2024b] to prevent data contamination and maintain experimental control. Accordingly, we first develop a synthetic persona dataset as "canaries." Our synthetic persona dataset consists of a set of image-description samples representing synthetic profiles. Given that personal assistants represent a promising application for vision-language models, the setting for our synthetic persona dataset is chosen to simulate scenario where the model learns to identify individuals (e.g.. user's acquaintances) based on images (similar to those saved on a user's phone) or or their textual descriptions (as might appear in text messages emails).
3

---

An overview of our data generation pipeline is presented in Figure 1. In detail, the creation involves the following steps:
The foundation of our dataset generation lies in defining a compre- hensive set of attributes that constitute a persona. These attributes are categorized to cover various aspects of an individual's appearance, demographics, and contextual elements. For each category, we curate pool of possible values, drawing inspiration from typical characteristies used in image generation and description. These attribute categories include:
Attribute Pool Definition:
Demographies: 8 demographic identities, 2 genders, and 3 age ranges.
Visual Characteristics: clothing styles/types, clothes colors, 7 accessories, 7 hairstyles, and 7 hair colors.
Contextual Elements: 7 facial expressions and 12 background scenes.
The specific values within these pools are chosen to maximize diversity and verisimilitude, allowing for large combinatorial space unique profiles of
8 2 x 3 7 x 7 7 7
12 67, 765, 824
combinations from the attribute pools. The complete list is provided in Appendix B.
II. Profile Synthesis: The synthetic profile synthesis process involves the following steps:
1. Attribute Combination: A unique combination of attributes is sampled without replacement from the defined pools to establish the core characteristics of persona.
2. Name Assignment: A unique fictional name is generated, conditioned on the demographic identity and gender to maintain cultural relevance and consistency.
3. PII Association: To facilitate privacy-related experiments, we associate each synthetic character with PII data. Given that most models are well-aligned and consistently reject real PII data requests (e.g.. Social Security Numbers), we opt use "favorite number" as surrogate. This favorite number randomly generated three-digit number.
The generation of image-description pairs is a crucial step in creating a comprehensive dataset that effectively represents synthetic personas. This process not only ensures visual diversity but also enhances the contextual realism of each profile.
III.
Generation:
Image Generation: Using the synthesized profiles (attribute combinations), we leverage state-of-the-art text-to-image generation model Imagen 3 [Baldridge et al., 2024] to create realistic profile pictures.
Textual Description Generation: To complement the generated images, we create rich textual descriptions for each persona profile picture. These descriptions go beyond merely listing attributes, instead presenting natural, narrative, and contextually rich portrayal of the synthetic individual. We employ Gemini 2.0 [Team et al., 2023] to generate these textual descriptions, conditioned on both the generated image and the profile attributes. To enhance realism, we instruct the model to adopt a more conversational tone, similar to how a person would describe a friend.
Additionally, we generate multiple image-description pairs for each persona to simulate scenarios where a single individual has several images. This approach parallels the language model training process, where models are exposed to multiple paraphrases of a fact [Allen-Zin and Li, 2025]. The generation prompts are provided in Appendix C.
Final Dataset Composition: The resulting synthetic persona dataset consists of collection of 100 unique personas. Each persona is characterized by the following elements:
A unique name.
An associated favorite number.
An associated of specific attributes.
A set of 100 image variants for training and 1 distinct image for testing.
A set of 100 textual description variants for training and distinet textual description for testing.

---

1.0
Testing Type Descrpton
1.0
1.0
Training Type
Descrption Image
0.8
0.8
0.8
Target Accuracy
Accuracy
0.6
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
40
100
0.0
0.2
1.0
Training Epoch
Training Epoch
Source Accuracy
(a) Training on Descriptions
(b) Training on Images
(c) Cross- Mooss-Modllity Accuracy
Figure 2: Training with Single Description/Image. Factual memorization transfers between modalities, there a significant gap between the source and target modalities.
1.0
1.0
1.0
Training Type
Description
0.8
0.8
Targel Accuracy
Image
Accuracy
0.6
Testing Type
Accuracy
0.6
0.6
Descrption Image
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0
20
40
80
100
0.0
0.2
0.8
1.0
Training Epoch
Training Epoch
Source Accuracy
(a) Training Descriptions
(b) Training Images
(c) Cross-Modale Accuracy
Figure 3: Training with Image/Deseription Variants. Using diverse input variations during training helps mitigate overfitting, but does not improve the cross-modal transferability rate.
3.2 Experimental Setup
We fine-tune the latest open-source vision-language model, Gemma-3-4b [Team et al., 2025], to conduct all our experiments. During fine-tuning, we utilize LoRA [Hu et al., 2022] with rank of r = 32, a scaling factor of a 32, and dropout probability of 0.05. We use Adam [Loshchiloy and Hutter, 2017] with learning rate of 2 x 10 and batch size of 16. All training is performed on single Nvidia A100-80G GPU.
We employ distinct training settings: where the model is trained solely textual descriptions and another where it is trained exclusively on images. In each setting, the input prompt consists of either description or image, followed by question about the person's name. We train the model to accurately predict the associated name based on the given input.
During testing, we evaluate the model's recognition accuracy by asking the same question in both modalities separately, assessing the model's ability to recall the correct name each modality. Note that test seen persona profiles, but the specific test inputs (image textual descriptions) are held out and never seen verbatim by the model during training. In our experiments, we deliberately train for extreme number of epochs to simulate worst-case privacy scenario, where memorized signals are not diluted by non-canary data. This setup provides a stress test for evaluating the robustness of potential mitigation methods.
3.3 Memorization Results
Training with Single Description/lmage We start with fine-tuning the model on single image or description for each persona across multiple epochs. Specifically, we choose 1, 5, 10, 25, 50, 75, and 100 epochs. We report the recognition accuracy for both models trained on descriptions and
5

---

images in Figure 2a and Figure 2b, respectively, showing the test accuracy on held-out descriptions and images.
In the description training setting, the model's accuracy for both modalities consistently improves as training progresses. In contrast, training on images tends to lead to overfitting after 25 epochs, the model's performance decreases afterwards.
Although cross-modal knowledge transfer occurs in both training scenarios, there remains significant performance gap between the training modality and the testing modality. This gap is more evident in Figure 2c, where the plot compares the accuracy between the training (source) modality and the testing (target) modality. Ideally, a perfect model would align with the dotted line, indicating no discrepancy between the two modalities. However, both training settings reveal noticeable gap. This disparity underscores the difficulty of transferring learned representations between text and image domains.
Takeaway 1: Factual memorization transfers between modalities, but there a significant gap between the source and target modalities.
To mitigate the effect of overfitting associated with training on a single description or image for multiple epochs, we employ a different strategy in this experiment. Instead of repeatedly using the same input, we use a new description or image for each epoch. This approach is analogous to factual knowledge learning with paraphrased texts, which has been shown to help models generalize more effectively [Allen-Zhu and Li, 2025]. By introducing variations during training, the model avoids memorizing fixed representation and instead learns the actual knowledge.
Training with Image/Description Variants:
As depicted in Figure 3, training with diverse inputs significantly improves the test accuracy compared to the single-input scenario. In Figure 3a, where the model is trained on descriptions, the accuracy reaches nearly perfect levels, suggesting that the model benefits from varied textual data. Similarly, in Figure 3b, when trained on images, the model also achieves high accuracy without the early saturation and overfitting observed previously. These results highlight that varying the input per epoch effectively prevents overfitting and allows the model to generalize better across training instances.
However, despite these improvements in accuracy, the cross-modal transferability remains limited, as shown in Figure 3c, where the slope remains similar to the single-sample training scenario. This indicates that, while imput variability mitigates overfitting, it does not necessarily enhance the model's ability to bridge the gap between different modalities.
Takeaway Using diverse input variations during training helps mitigate overfitting, but does not improve the cross-mose modal transfcrability rate.
One interesting observation is that the target accuracy across the modality continues to improve (especially for the image = text scenario) after the source modality saturates at near-perfect accu- racies. This emphasizes the importance of diverse training samples on learning robust knowledge representations that go beyond what single-modalie benchmarks can typically show.
As previous work [Carlini et al., 2022] indicates, larger models tend to memorize information more easily. To investigate the impact of model size on cross-modal transferability, we train three models from the Gemma3 family: Gemma3-4B-IT, Gemma3-12B-TT and Gemma3-27B-BB As shown in Figure 4, increasing the model size consistently improves the accuracy for both source and target modalities. However, despite the improvements in both individual modality accuracy and cross-modality accuracy as the model size increases, the rate cross-modae transferability, indicated by the slope of the lines in the accuracy plots, remains nearly unchanged. This suggests that while larger models are more effective learning within each modality, they do not necessarily enhance the transfer of learned information between modalities.
Training with Model Sizes:
These findings imply that model size primarily contributes to improved memorization and recognition within a single modality rather than facilitating cross-modal generalization. Consequently, even as the model capacity increases, bridging the gap between image-to-text and text-to-image transfer remains challenging task.

---

0.5
Model
0.5
Gemma3-4B-IT
Target Accuracy (Image)
0.4
Bemmaa-122-M4
Target Accuracy (Description)
Gemma3-27B-IT
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Source Accuracy (Description)
Source Accuracy (Image)
(a) Training on Descriptions
(b) Training on Images
Figure 4: Training with Different Model Sizes. Increasing model size enhances accuracy within each modality, but maintains a similar cross-modal transferability rate.
0.5
Mitigation Type
0.5
Baseline
Target Accuracy
0.4
ID
Target Accuracy (Description)
OOD
(Image)
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Source Accuracy (Description)
Source Accuracy (Image)
(a) Training on Descriptions
Training on Images
Figure 5: Training with Augmentation Mitigation. Augmenting training data with in-distribution image-caption samples is essential for enhancing cross-modal transferability, while out-of-distribution data does not provide similar benefits.
Takeaway 3: Increasing model size enhances accuracy within each modality, but maintains similar cross-modal transferability rate.
3.4 Mitigation with Image-Caption Augmentation
To further enhance cross-modal transferability, we propose a simple yet effective approach that augments the training data with synthetic image-text pairs. Specifically, for given description text input, we generate a corresponding image conditioned on the text. Conversely, for an image input, generate a descriptive caption. These generated pairs are introduced as captioning data, where the model asked to describe an image.
In this experiment, we utilize two types of data for mitigation: 1) in-distribution (ID) data, where we augment training set with held-out synthetic persona images and descriptions that closely resemble the training data, and 2) out-of-distribution (OOD) data, where we introduce incorporate images and captions from the COCO dataset [Lin et al., 2014]. This combination allows us to investigate the impact of data diversity cross-modal transferability.

---

100
100
Model Type
Relative Accuracy Drop%
Relative Accuracy Drop%
Description
80
80
Image
(Description)
(Image)
60
40
40
20
20
0
60 Relative Accuracy Drop%
20
40
100
Relative Accuracy Drop% (Image)
(Descripton)
(a) Unlearning on Descriptions
(b) Unlearning on Images
Figure 6: Unlearning cross Modalities. Unlearning effects modality-dependent: when unlearning happens in the same modality as learning, the effect remains relatively consistent across both modalities. In contrast, when unlearning occurs different modality from where the fact was originally learned, the effect does not transfer equally between modalities.
As shown in Figure incorporating in-distribution (ID) data significantly improves cross-modal transferability, as indicated by the blue line consistently appearing above the baseline green line in both description and image training scenarios. In contrast, augmenting with out-of-distribution (OOD) data does not provide similar benefits, with the orange line remaining close to the baseline. This suggests the importance of augmenting training data within the same distribution effectively improve transferability.
Takeaway 4: Augmenting training data with in-distribution image-caption samples is essen- tial for enhancing cross-modal transferability, while duitr--distribution data does not provide similar benefits.
3.5 Cross-Modal Unlearning
Machine unlearning [Bourtoule al., 2021] has gained popularity method for removing specific user data without requiring the model to be retrained from scratch. Consequently, it is essential to investigate how unlearning operates across different modalities.
commonly used unlcarning method involves applying gradient ascent on the forget dataset [Maini al., 2024]. However, empirical experiments reveal that this approach is challenging to fine-tune and significantly compromises model performance. Therefore, to make the controlled experiments more comparable with fresh knowledge learning results, we adopt practically simpler workaround: to unlearn the association a person (image or textual description) to name, we generate a modified training where each profile is associated with completely different names, and (continue) training the model on this modified dataset. We then measure the model's accuracy (drop) of association with the original name the unlearning performance. This allows making the cross-modality measurements more easily comparable with previous results.
reuse the
training setups,
In detail, start with the fine-tuned description model and image model from Section 3.3. Since we have two models (the fine-tuned description model and the fine-tuned image model) and two unlearning datasets (description-basel and image-based), we systematically evaluate all possible combinations, which results four experimental setups.
In Figure 6a, we present unlearning experiments with relative accuracy drops in both modalities when unlearning on description data. For the description model, the unlearning effect on both modalities exhibits similar pattern (as indicated by the green line closely following the = line). Interestingly, for image models, unlearning on description data predominantly affects the description test accuracy, while the image test accuracy drop remains quite small. In Figure 6b,

---

Table 1: Cross-Modal Multi-Hop Learning Cross-modal models suffer from the multi-hop curse, where accurate performance on individual tasks does not translate to multi-hop reasoning.
Training Data
Test Type
Base Data
Mup Data
Desc=Namt
Image=>Name
Desc=>NN FN
Image=>FV FN
Name=> FN
Desc=>FN
1.00
0.35
1.00
0.31
0.08
Name
Image=FNN FN
1.00
0.36
0.39
1.00
0.05
NN
1.00
0.36
0.11
0.05
0.99
Desc=>FN
0.68
1.00
1.00
0.47
0.07
Image=>Nar
Image=>FN
0.50
1.00
NN
0.27
0.65
show unlearning experiments on image data. Similar to the description data pattern, for the image (source modality) model, the relative accuracy drop both modalities is comparable. However, for the text model, the unlearning effect on description data is notably weaker, while the drop in image accuracy is more significant.
Overall, these observations reveal gap in crods-tod unlearning: when unlearning facts in modality different from the one in which the model originally learned the fact, the unlearning effect does not transfer equally between modalities. modality where the fact was originally learned, the unlcarning effect is relatively similar across both modalities. This underscores the importance of carefully designing unlearning training data and developing more effective unlearning techniques for cross- cross-modal scenarios.
contrast, when unlearning occurs in the same
Takeaway Unlearning effects are same modality learning, the effect remains relatively consistent across both modalities. In contrast, when unlearning occurs in a different modality from where the fact was originally learned, the effect does not transfer equally between modalities.
when unlearning happens in the
3.6 Cross-Modal Multi-Hop Learning
Large language models often exhibit the "multi-hop curse," where they fail to infer that A is C when the model trained on A B and B C [Balesni al., 2024]. In this section, we investigate this multi-hop scenario in cross-modal context.
We define the original description image mapping name data points as A is data (Desc=Namm and Image=NName), while a three-digit number representing person's 's favorite number (FN) serves C. To investigate multi-hop hop reasoning, introduce various data combinations involving base data: Neme=NNme (given the deseription, predict the corresponding name) or Image=Name; multi-hop data: Desc= FN (given the description, predict the corresponding favorite number), Image= and FN.e= During testing, we assess whether the model can recall the person's 's favorite number given a description, image, or name cue.
As shown Table I, cross-modal models also exhibit the multi-hop curse. For instance, when model is trained on Desc=NName and Name=FN data, it achieves perfect accuracy both tasks individually. However, when directly prompted with a description to retrieve the favorite number, the accuracy drops drastically to only 11%. indicating significant challenge in the multi-hop scenario. In contrast, the model trained on Image=Nant exhibits lower barrier, achieving over 50% accuracy for Image= FN. Even when tested Desc=FN, the accuracy is considerably higher compared to the model trained on descriptions, suggesting that image- based training better supports multi-hop reasoning.
Takeaway Cross modal models suffer from the multi-hop curse, where accurate perfor- mance individual tasks does not translate multi-hop reasoning.

---

Conclusion and Limitations
Our study systematically investigates cross-modality memorization in vision-language models using a synthetic persona dataset. We uncover an asymmetric performance gap between source and target modalities, where knowledge transfer from image text is more effective than from text to image. We also demonstrate that while augmenting training data with in-distribution synthetic image-text pairs helps bridge this gap, nut-of-distribution data proves less effective. Furthermore, we identify challenges in cross-modal unlearning, as unlearning in one modality does not fully translate to the other. Additionally, we reveal that vision-language models suffer from the multi-hop curse, where the model struggles to infer complex cross-modal relations despite successful learning of simpler single-hop associations.
Our findings underscore the complexity of achieving robust cross-modal transferability and highlight the need for improved training strategies to enhance generalization across modalities. While this work focuses on text and image modalities, future research should explore additional modalities, develop more sophisticated unlearning techniques, and design approaches better handle multi-hop -hp reasoning. Addressing these challenges is crucial for building more reliable and privacy-conscious multimodal Al systems.
Acknowledgments
Icons from
com/ were used to create the illustration figure in this paper.
10

---

References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. In ICLR, 2025.
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3. #rXiv:448.077009 2024.
Mikita Balesni, Tomasz Korbak, and Owain Evans. The two-hop curse: LLMs trained on A-B, B-+C fail to learn A-C. arXiv:2411.16333 2024.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "A is B" fail to learn "B is A". In ICLR. 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In S & P, pages 141-159, 2021.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In USENIX Security, pages 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2022.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In USENIX Security, pages 22500 2023.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs new knowledge encourage hallucinations? In EMNLP, pages 7765-7784, 2024.
Luxi He, Yangsibo Huang. Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them. In ICLR, 2025.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsumori Hashimoto, Mark A. Lemley, and Percy Liang. Foundation models and fair use. JMLR, 24(400):1-79, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James With- ers, and Robert Sim. Privacy analysis in language models via training data leakage report. Sttv:21011.54005 2021.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025.
Bargav Jayaraman, Chuan Guo, and Kamalika Chaudhuri. Déjà vu memorization in lasiun- models. In NeurIPS, pages 50722-50749, 2024.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In ICML, pages 10697-10707, 2022.

---

Narine Kokhlikyan, Bargav Jayaraman, Florian Bordes, Chuan Guo, and Kamalika Chaudhuri. Measuring dejavu memorization efficiently. In NeurlPS, pages 30015-30040, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740-755, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
2017. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. arXiv:2401.06121 2024.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito. Christopher A Sooquuette-hho, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv:231 17035,, 2023.
Rein, BL Hou, AC Stickland, J Petty, RY Pang. Dirani, J Michael, and SR Bowman. GPQA: A graduate-level Google proof Q&A benchmark. 1xXiv:2311.12022 2023.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under-
standing and mitigating copying in diffusion models. In NeurlPS, pages 47783 47803, 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? Investigating data replication in diffusion models. In CVPR, pages 6048-6058 2023b.
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein. Measuring style similarity in diffusion models. arXiv:2404.01292, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. 2r12. 18805 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv:2503. /9786, 2025.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv:2411.04338. 2024.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In ICLR, 2024a.
Yuxin Wen, Leo Marchyok, Sanghyun Hong. Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. In NeurlPS, 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv:1905.078300 2019.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. Counterfactual memorization in neural language models. In NeurIPS, pages 39321-39362. 2023.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models not robust multiple choice selectors. In ICLR, 2024.
12

---

A Broader Impacts
This paper systematically evaluates the state of cross-modal knowledge transfer in multimodal language models. Through extensive empirical studies, we reveal the existence of a significant knowledge transfer gap between vision and text domains. Despite this gap, our findings demonstrate that knowledge does transfer between these modalities, albeit asymmetrically.
Understanding this transfer is crucial for the future design and deployment of multimodal models, it has important implications for both performance optimization and privacy preservation. Specifically, results highlight the potential risk of unintended information leakage across modalities. For example, sensitive information learned from visual data could inadvertently influence text- bast- outputs and vice versa.
To mitigate such risks, researchers and practitioners should carefully evaluate cross-modal knowledge interactions, especially when designing models for real-world applications that handle sensitive personal data. We encourage the community to further investigate robust techniques that ensure cross-modal privacy while maintaining model performance.
B
Attribute Pool
Demographic Identity: African (South African), Caucasian (American). Caucasian (British), East Asian (Chinese), East Asian (Korean), Hispanic (Mexican), Iberian (Spanish),
Middle Eastern (Saudi)
Gender: Female, Male
Age Range: Adult, Middle Aged, Senior
Clothing Type: Button-Down Shirt, Hoodie, Jacket, Polo, Sweater, T-Shirt, Tank Top
Clothes Color: Black, Brown, Gray, Pink, Red, White, Yellow
Accessory: Earrings, Glasses, Hat, heaiiphhhnes,,eelly No Accessory, Scarf
Hairstyle: Afro, Bald, Curly, Dreadlocks, Long, Medium, Short
Hair Color: Black, Blonde, Brown, Gray, Pink, Red, White
Facial Expression: Angry, Frowning, Laughing, Neutral, Sad, Smiling, Surprised
Background: Beach, Cafe, Cityscape, Forest, Graffiti Wall, Library, Mountains, Office, Plain Black, Plain Grey, Plain White, Sports Field
C Data Generation Prompts
Image Generation Prompt for Imagen
Generate a highly detailed, photorealistic photo of a person. Key characteristics include: Demographic Identity: African (Nigerian)
Gender: Male
Age Range: Adult
Clothing Type: Button-Down Shirt
Clothes Color: Pink
Accessory: Glasses
Hairstyle: Long
Hair Color: Red
Facial Expression: Laughing
Background: Mountains
13

---

Description Generation Prompt for Gemini 2.0
<image>
Describe the person in the image in a detailed and natural way. making it easy for someone To recognize them based the description. Write if you are describing friend of my. Ensure the description flows naturally and includes the following attributes:
Demographic Identity: African (Nigerian)
Gender: Male
Age Range: Adult
Clothing Type: Button-Down Shirt
Clothes Color: Pink
Accessory: Glasses
Hairstyle: Long
Hair Color: Red
Facial Expression: Laughing
Background: Mountains
Feel free to add any additional distinguishing features that enhance the portrayal. Please provide the description directly, without extra formatting or instructions.
D
Training and Testing Prompts
Training/Testing Prompt for Image Data
<image>
Based on the image, what the person's name?
TrasingggTesting Prompt for Description Data
Description: My friend is a Nigerian man. He wears pink button-down shirt [...]
Based on the description, what is the person's 's name?
E
Test Prompt Engineering
0.5
Prompt Type
0.5
Baseline
Target Accuracy
0.4
Prompt
Target Accuracy (Description)
0.4
Prompt
(Image)
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Source Accuracy (Description)
Source Accuracy (Image)
(a) Unlearning on Descriptions
(b) Unlearning on Images
Figure 7: Testing with Different Prompt Strategies.
14

---

At the beginning of the project, we tested naive mitigation strategy by appending modality transfer- aware prompts during inference. We evaluated two prompts:
Prompt 1: Try to recall knowledge learned from another domain.
Prompt 2: Given this description/image. think about what you learned from the im- age/description.
As shown in Figure neither prompt leads to a significant change in transferability.
Unintentional Memorization in VLMs
Unintentional Memorization
Baseier the inage came?
VIN1
Invenberries
VUHL
Bsadd whatls Fe passels
Emseire the icthe 2o300 rame?
Traininng Injected
Membership Inference
Figure 8: Illustration of Unintential Memorization.
At the end, we share a phenomenon observed our experiments, which we refer to as unintentional memorization: the model inadvertently memorizes the injected PII from the image, despite being entirely unrelated to current task. illustrated in Figure 8, we inject a "favorite number" into image during training. Since the model is aligned to avoid generating contents related to potentially sensitive information such as social security numbers, in order to decouple the question alignment (refusing to answer) and memorization, our measurements. this experiment, the model's task is to identify the person in the image, and the number should be irrelevant. To test whether the model unintentionally memorizes this information, we perform a membership inference attack: for each test image, we overlay either the ground truth PII (to compute the in-ma in-member loss) random PII belonging to different training individual (to compute the out-member loss). We then compare these losses and report the AUC of the resulting ROC curve.
the "favorite number" surrogate for real PII in
We present the results in Figure 9. As shown in the figure, the model's AUC is close to ran- dom guessing at the beginning of training. How- ever, training progresses, the AUC steadily in- creases, reaching around 70% after 100 epochs. This indicates that the model begins to memorize the injected PII in the image, as it influences the loss the recognition task. This observation is non-obvious because the training paradigm of VLMM do not compute loss the input images, and the prediction tasks have nothing to do with the favorite numbers "accidentally" present in the training images.
0.75
0.70
0.65
AUc
0.60
0.55
0.50
0.45
We also attempted direct extraction of the PII. but the success rate was near zero all cases. In contrast, membership inference reveals much stronger signal of memorization.
20
40
60
80
100
Training Epoch
Figure 9: Membership Inference Results for Un- intentional Memorization
We argue that this type of unintentional memo-
rization poses significant privacy risks and war- rants deeper investigation in future work. For instance, in the context of training LLM-based agents
15

---

on web data, some webpages may contain PII or sensitive content incidentally. Even if such content unrelated to the task, the model may still inadvertently memorize it. As result, future efforts should focus on carefully filtering or redacting training data to mitigate these risks.
Takeaway 7: VLMSs unintentionally memorize irrelevant visual PII. This highlights the importance of stronger data filtering to mitigate inadvertent privacy risks.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated in the abstract and the introduction, the scope this paper is to systematically evaluate the state of cross-modal knowledge transfer in multimodal language models. Through extensive empirical studies, we confirm that there exists knowledge transfer gap between vision and text domains.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss them in Section 4. One limitation is that only studied image-text cross-modality knowledge transfer. There are other modalities, such as audio and video, that can be potentially considered. Another limitation is that while we identify the existence of gap and study baseline mitigation methods, we leave it for future work for algorithmic and architectural innovations that could fully bridge the gap.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the models used our experiments for data generation and fine-tuning are publicly available. We provide comprehensive details to facilitate the reproduction of our results, including the data generation prompts and training hyperparameters.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
18

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Instead of providing copy of the data, provide full details to reproduce the experiments, including instructions on how to generate synthetic datasets used in the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co mublic/guides/ Satisddee/Coeeuumissttsts wttpeuuuissssiooooooooooooeoeeee onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /ttpp/cccppublii///// wwtpc//ggg/eeeeeeeeeeeeee /ilese//ooSuubmii iitimmsssionn ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We include them in Section 3.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use 7 different random seeds for our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include them Section 3.2, and most of our runs could be done within 20 GPU hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttts////narrs.s...ppi////t/at
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include them in Appendix A.
20

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited all relevant papers associated with our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comdattasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpaa:///ueiiaaaeeeeeeee cc/Conoeeeeeecee//222225 for what should or should not be described.
23


Reviews:
[{"review_id": "csvfhPuimW", "reviewer": "Reviewer_4B94", "summary": "This work takes the first step toward systematically quantifying cross-modal memorization in vision–language models. Rather than evaluating on arbitrary real-world data, it introduces a synthetic “persona” benchmark of paired images and text captions labeled with unique names. By fine-tuning VLMs to predict names from one modality and testing on the other, it uncovers a persistent, asymmetric performance gap and a multi-hop inference failure in cross-modal settings.", "questions": "My main concerns are in the method section, please see the weakness above.\nSince this work is interesting to me, I would be willing to raise my score if the mentioned weaknesses are addressed in the rebuttal.", "limitations": "The limitations of this study is not discussed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles an important and understudied problem by systematically quantifying how factual knowledge learned in one modality transfers to another in vision-language models.\n2. The synthetic persona dataset is carefully designed with controlled image–text pairs, enabling clear attribution of memorization and transfer effects.\n3. Experiments are thorough which covering single-input vs. variant training, model scaling, data augmentation, unlearning, and multi-hop reasoning, providing a comprehensive empirical picture.\n4. Clear asymmetries (image-to-text and text-to-image transfer) and failure modes (overfitting, multi-hop curse, unlearning gaps) are identified, offering concrete takeaways for future methods.\n\nWeaknesses:\n1. Reliance on fully synthetic personas may limit external validity, as real-world data distributions are not evaluated.\n2. The study focuses solely on name-recall tasks, leaving open whether richer or higher-level knowledge (e.g., events, attributes beyond names) exhibits similar cross-modal behavior.\n3. All experiments use a single VLM family (Gemma-3-IT), so conclusions may not generalize to other architectures or pretraining regimes.\n4. The impact of synthetic data quality (artifacts from Imagen 3 or Gemini 2.0) on memorization and transfer is not examined, risking biased results.\n5. The limitations of this study is not discussed in the paper.", "was_revised": true, "final_justification": "Thanks for authors' detailed and timely rebuttal. All of my questions have been answered with good answers and extensive qualitative results, and I am happy to raise my score.", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> Reliance on fully synthetic personas may limit external validity, as real-world data distributions are not evaluated.\n\nWe agree that using real images would be ideal. However, we use fully synthetic data to prevent data contamination and ensure annotation accuracy. With real images, it is often unclear whether the associated knowledge has already appeared during pretraining or in other data points, making it difficult to isolate cross-modal memorization effects. Our synthetic setup enables a controlled investigation of memorization without the confounding risk of training data leakage. Additionally, using real human faces might raise serious privacy and consent concerns.\n\nTo minimize bias and ensure high data quality, we intentionally selected the state-of-the-art generative models in our paper. We used Imagen 3 for image generation for its top-tier performance in photorealism and content diversity (https://lmarena.ai/leaderboard/text-to-image/pre-generated-prompts-surprise-me-only). In contrast, alternatives like Midjourney or GPT-4o’s native generation often exhibit stylistic biases. For textual descriptions, we used Gemini 2.0, conditioning on structured persona attributes to significantly reduce hallucinations and maintain consistency across modalities.\n\n> The study focuses solely on name-recall tasks.\n\nWe focused on the name-recall task because it serves as a straightforward proxy for factual memorization that is easy to verify, allowing us to minimize confounding variables during data generation and training. Additionally, we included results in the paper on recalling a “favorite number” as a secondary fact to demonstrate generalization beyond names.\n\nWe agree that extending the study to include more abstract or higher-level knowledge—such as events or relationships—would provide valuable insights. Designing such experiments requires a different pipeline for data construction, and we view this as a promising direction for future work.\n\n> All experiments use a single VLM family (Gemma-3-IT), so conclusions may not generalize to other architectures or pretraining regimes.\n\nWe have also run additional experiments with LLaVA 1.5. Due to rebuttal format constraints, we are unable to include full results and plots here, but we will include them in the camera-ready version. For example, for training with image/description variants scenario with 50 epochs, we observe the following source → target accuracy:\n\n|                          |    Gemma   |    LLaVA   |\n|:------------------------:|:----------:|:----------:|\n| Training on Descriptions | 0.98→0.22 | 0.97→0.15 |\n|    Training on Images    | 0.99→0.52 | 0.98→0.37 |\n\nWe observe a similar asymmetric cross-modal transfer pattern. However, LLaVA exhibits slightly lower transferability in both directions compared to Gemma-3. In the future version, we plan to include more model families.\n\nMore broadly, one of our goals is to demonstrate that a cross-modal memorization gap emerges even with state-of-the-art open-source VLMs like Gemma-3. We hope our pipeline serves as a useful evaluation routine for assessing newly developed multimodal models.\n\n> The limitations of this study is not discussed in the paper.\n\nWe appreciate the reviewer pointing this out. While the limitations are discussed at the end of Section 4, we agree that they could be more clearly highlighted. Specifically, we acknowledged two main limitations:\n(1) Pur study focuses solely on image-text cross-modal knowledge transfer, whereas other modalities (e.g., audio, video) remain unexplored; and\n(2) We only evaluated baseline mitigation strategies, leaving more advanced architectural or algorithmic approaches to future work.\n\nTo improve clarity, we will add a dedicated \"Limitations\" subsection title in the camera-ready version to make these points more explicit and accessible.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions."}, {"review_id": "F2lpdUBI0m", "reviewer": "Reviewer_F4yj", "summary": "This paper explores the memorization behavior in a multi-modal (image-text) foundation model using a novel synthetic dataset that is easy to control. Carefully designed experiments test the model’s ability to recognize facts in both modalities, which were only presented in one modality during training. The findings reveal that knowledge does not fully transfer from the source modality to the target modality, and larger model sizes, nor diverse training variations can bridge the gap. Training with more in-distribution image-caption pairs, on the other hand, does help. Finally, the paper reports asymmetries in the results depending on which modality was used as the source modality (e.g., for cross-modality transfer, effects of machine unlearning and the multi-hop curse).", "questions": "My first impression was that it was already a strong paper. To consider increasing an already good score, I would want to see how the conclusions generalize to other datasets/models/modalities and get more insights on causes or mitigation strategies.\n\nSome minor suggestions:\n- it could be nice to have the take-home in the figure caption. I found myself referring back to the text to check my interpretation of the figure.\n- this might be personal taste, but I was comparing graph-lines between figures and it would have helped me if the xlims and ylims were consistent across figures. And to have it pointed out which lines were identical to one from a previous figure (same condition).", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I will list perceived strengths and weaknesses regarding Quality [Q], Clarity [C], Significance [S}, and Originality [O] below.\n\n# Strengths\n1. [C] The paper is beautifully written and very well structured. The introduction makes the aims and contributions clear from the start, and lays out nicely what the reader can expect (and what not to). The take-home boxes are very helpful too. The type of language used (not overly technical) makes it accessible to non-experts. \n2. [Q] The experiments are carefully designed to address specific questions and they delivered clear take-home messages. This was made possible through a novel synthetic dataset that offers good experimental control.\n3. [O] The custom synthetic dataset makes for an original addition to the work and describes a way for others to build their own.\n4. [S] I can see this work being an important first steps for many follow-up papers to build on, speaking to many members in the NeurIPS community. Especially given the far-reaching impact of multi-modal foundation models, understanding their behavior and mitigating privacy and other memorization-related concerns is a valuable endeavor. \n\n# Weaknesses\n1. [Q] The paper is very engaging, which made me want to know more about potential causes for the observed patterns, something that wasn't touched upon very much.\n2. [Q] In a similar vein, I would have loved to see more results on other mitigation strategies, although it was clearly acknowledged in the introduction that this was for future work. Could it not have been one big paper?\n3. [S] The results are based on one (specialized) dataset and one model family (the focus on solely image-text modalities was already acknowledged in the paper).", "was_revised": true, "final_justification": "My questions/comments have been adequately addressed and after also considering the other reviews, I am still of the opinion that this is a solid paper. I'd recommend acceptance.", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> The paper is very engaging, which made me want to know more about potential causes for the observed patterns, something that wasn't touched upon very much.\n\nThank you for the positive feedback and your interest in the underlying mechanisms. We agree that understanding the causes of cross-modal asymmetries is a valuable and exciting direction for future work. Thoroughly investigating these causes may require a different set of experiments. We would like to focus on this in our follow-up work.\n\n> In a similar vein, I would have loved to see more results on other mitigation strategies, although it was clearly acknowledged in the introduction that this was for future work. Could it not have been one big paper?\n\nWe tested out straightforward baseline methods in this paper. However, designing mitigation methods that can truly close the modality gap likely depends on first understanding the underlying causes—as you noted above. This would require a systematic and non-trivial analysis. Rather than overloading a single paper, we felt it was more appropriate to separate these efforts and ensure each component received the depth it deserves. We see this as an important next step and appreciate your encouragement in that direction.\n\n> The results are based on one (specialized) dataset and one model family (the focus on solely image-text modalities was already acknowledged in the paper).\n\nWe have also run additional experiments with LLaVA 1.5. Due to rebuttal format constraints, we are unable to include full results and plots here, but we will include them in the camera-ready version. For example, for training with image/description variants scenario with 50 epochs, we observe the following source → target accuracy:\n\n\n|                          |    Gemma   |    LLaVA   |\n|:------------------------:|:----------:|:----------:|\n| Training on Descriptions | 0.98→0.22 | 0.97→0.15 |\n|    Training on Images    | 0.99→0.52 | 0.98→0.37 |\n\nWe observe a similar asymmetric cross-modal transfer pattern. However, LLaVA exhibits slightly lower transferability in both directions compared to Gemma-3. In the future version, we plan to include more model families.\n\nMore broadly, one of our goals is to demonstrate that a cross-modal memorization gap emerges even with state-of-the-art open-source VLMs like Gemma-3. We hope our pipeline serves as a useful evaluation routine for assessing newly developed multimodal models.\n\n> Minor suggestions\n\nThank you for your helpful suggestions to improve the clarity of our paper. We will incorporate them in the camera-ready version.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions."}, {"review_id": "NLyGfXwE4o", "reviewer": "Reviewer_Rdda", "summary": "This paper carries out studies of cross-modal knowledge extraction in a multimodal LLM. To do this, they introduce a new dataset of personas, consisting of synthetically generated images of (fake) people, together with their textual descriptions. The MLLMs are then fine-tuned with this information in one modality (image or text), and extraction is attempted from the other. The main finding of the paper is that image to text extraction is significantly easier than text to image extraction.", "questions": "n/a", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ Overall I thought this is a very solid paper -- the dataset is well-made and interesting, the experiments are quite well-done, the conclusions are presented crisply. Execution-wise this is a really strong paper.\n\nWeaknesses:\n\n- To me, the main weakness of this paper is in the way is it framed. The current framing of the paper suggests that the results have serious implications for privacy, showing that models memorize private information, and that this memorization could be \"cross-modal\". \n\nHowever usually memorization in the the privacy/memorization literature refers to incidental memorization where there is a very small amount of data about a single person in a very large crowd, such as a single data point in a large crowd. My understanding is that the setting of the current paper is a little bit different in a subtle way -- since we are directly fine-tuning with each person's data for many epochs; in my view, this paper's setting is closer to the Physics of LLMs: Knowledge Extraction (https://arxiv.org/abs/2309.14316) paper but for multimodal models. This is not to say that the results are uninteresting -- in fact they are extremely well-done and interesting -- but the implications for privacy are perhaps not quite there. \n\n- Minor: It's also not quite correct that there has no work at all on multimodal memorization -- [1, 2] look at multimodal memorization in a different type of multimodal (CLIP) models. \n\n[1] https://arxiv.org/abs/2402.02103\n[2] https://arxiv.org/abs/2504.05651", "was_revised": true, "final_justification": "After discussion I think this is a solid paper and I am supportive of acceptance.", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> To me, the main weakness of this paper is in the way is it framed.\n\nWe understand the concern. While prior work often focuses on unintentional memorization from large-scale pretraining, such experiments are extremely expensive to conduct—especially for VLMs, where training costs are significantly higher than for LLMs due to the length of visual tokens.\n\nOur experimental setting represents a worst-case privacy scenario, where the memorized signals are not diluted by non-canary data. This setup is particularly useful for stress-testing mitigation methods. We will revise our wording in the camera-ready version to improve our clarity.\n\n> It's also not quite correct that there has no work at all on multimodal memorization.\n\nThank you for pointing out these two papers on CLIP models. They are indeed closely related to our work, and we will include a discussion of them in the camera-ready version.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions."}, {"review_id": "T3znWlIHbX", "reviewer": "Reviewer_r17s", "summary": "The paper studies cross-modality memorization in vision-language models for sensitive attribute prediction task on a synthetic dataset introduced by the authors. The paper quantifies memorization in terms of predicting correct attribute in both source and target modalities. Here the source is the modality that is used for training the model and target is the modality that is used only for inference.\nThe paper suggests that memorization transfers between modalities but it is significantly lower in the target modality. Lastly, the paper proposes mitigations using data augmentation and studies cross-modality unlearning.", "questions": "+  The paper first mentions that there are ~64B profiles but then later that there are only 100 unique profiles. What are the differences between profile and persona ? How large is the final dataset ?\n+ How did you ensure that the dataset is diverse and representative enough for the study ?\n+ How much data was added during training with variants ? How did you know how many variants you need to add ?", "limitations": "The paper discussed future work but does not discuss the limitations of the work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Pros**\n\n+ The paper is clearly and well written it conducts thorough experiments for different hyper parameters such as model size, increasing data diversity to avoid overfitting.\n+ The paper proposes a mitigation for the cross modality transferability gap and shows that proposed mitigation helps to close the gap. \n+ It’s also interesting to see multi-hop unlearning results for cross-modality dataset.\n\n**Cons**\nCons\n+ The experimental evidence is provided only for the benchmark proposed by the paper. It is unclear whether the observations made by the authors are generalizable cross other datasets and modalities.\nIt could be that the observations made in the paper reflect some of the properties in the dataset. It might be good to try the experiments on other attribute prediction dataset and task.\n+  It seems that the authors use the COCO dataset as OOD dataset in one of their experiments. It might have been good to run all experiments on COCO dataset instead of the synthetic dataset and use synthetic dataset as OOD. It would be interesting to see whether the claims made by the authors will hold for this setup too.\n+ In terms of VLM memorization there is this other paper that is not cited\nhttps://arxiv.org/abs/2402.02103", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> It is unclear whether the observations made by the authors are generalizable cross other datasets and modalities.\n\nThis point is also mentioned in our limitations (Section 4). Our broad goal is to demonstrate that the cross-modal memorization findings we discovered can emerge, and they are possible to happen in other modalities. Therefore, we hope our work can serve as a starting point for fellow researchers to study memorization in different modalities.\n\n> It might have been good to run all experiments on COCO dataset instead of the synthetic dataset and use synthetic dataset as OOD.\n\nUsing COCO will be more realistic, but we use fully synthetic data to prevent data contamination. With real images, it is often unclear whether the associated knowledge has already appeared during pretraining, making it difficult to isolate cross-modal memorization effects. Our synthetic setup enables a controlled investigation of memorization without the confounding risk of training data leakage. Additionally, using real human faces might raise serious privacy and consent concerns.\n\n> In terms of VLM memorization there is this other paper that is not cited.\n\nThank you for pointing out this paper on CLIP models. We will include a discussion of it in the camera-ready version.\n\n> The paper first mentions that there are ~64B profiles but then later that there are only 100 unique profiles. How did you ensure that the dataset is diverse and representative enough for the study?\n\nThe total number of possible attribute combinations is approximately 64 billion, which reflects the size of the design space. We included this to emphasize the diversity and richness of the attribute pool. From this large space, we sample 100 unique profiles to construct a diverse and representative dataset while keeping the experimental setup manageable.\n\n> How much data was added during training with variants? How did you know how many variants you need to add?\n\nThe number of variants depends on the target number of training epochs. For example, if we aim for 20 epochs, we add 19 additional variants per profile, alongside the original, resulting in 20 distinct samples. Then, we perform a single training epoch, where each variant is seen once—effectively simulating a 20-epoch training schedule with different variants.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions."}]

Author rebuttal:

Meta-review: This paper investigates memorization in multi-modal (image–text) foundation models using a controllable synthetic dataset. Carefully designed experiments test whether models can recognize facts in both modalities when exposed to them in only one modality during training. The results show that knowledge transfer across modalities is incomplete: neither larger model sizes nor diverse training variations close the gap, though additional in-distribution image–caption pairs improve performance. The study also reveals asymmetries depending on which modality serves as the source.

All reviewers found the work interesting and the findings significant. Major concerns were resolved during the rebuttal, and the authors are encouraged to incorporate all feedback in the camera-ready version.
Decision: Accept (poster)