Déjà Vu Memorization in Vision–Language Models
Authors: Bargav Jayaraman
Chuan Guo
Kamalika Chaudhuri
Venue: NeurIPS 2024 poster (2024)
Keywords: memorization
multimodal
CLIP
vision-language models
Primary area: privacy
TL;DR: We show that vision—language models like CLIP can memorize the objects present in the training images, and evaluate different mitigation strategies to minimize this memorization.

Abstract: Vision-Language Models (VLMs) have emerged as the state-of-the-art representation learning solution, with myriads of downstream applications such as image classification, retrieval and generation. A natural question is whether these models memorize their training data, which also has implications for generalization. We propose a new method for measuring memorization in VLMs, which we call dèjá vu memorization. For VLMs trained on image-caption pairs, we show that the model indeed retains information about individual objects in the training images beyond what can be inferred from correlations or the image caption. We evaluate dèjá vu memorization at both sample and population level, and show that it is significant for OpenCLIP trained on as many as 50M image-caption pairs. Finally, we show that text randomization considerably mitigates memorization risk while only moderately impacting the model’s downstream task performance.  The code is available here: https://github.com/facebookresearch/VLMDejaVu.

Déjà vu Memorization in Vision-Language Models
Bargav Jayaraman FAIR, Meta California, USA
Chuan Guo FAIR, Meta California, USA
Kamalika Chaudhuri FAIR, Meta California, USA Kamalika@mes com
bargavemeta.co com
chuanguo@meta.com com
Abstract
Vision-Language Models (VLMs) have emerged the state-of-the-art represen- tation learning solution, with myriads of downstream applications such as im- age classification, retrieval and generation. A natural question whether these models memorize their training data, which also has implications for generaliza- tion. We propose new method for measuring memorization in VLMs, which we call déjà vu memorization. For VLMs trained on image-caption pairs, we show that the model indeed retains information about individual objects in the training images beyond what can be inferred from correlations or the image cap- tion. We evaluate déjà vu memorization both sample and population level, and show that it is significant for OpenCLIP trained on many 50M image-cap caption pairs. Finally, we show that text randomization considerably mitigates memoriza- tion while only moderately impacting the model's downstream task performance. The code available here: httpps: h/tgt//hbtbb/////////// com http//accboooreeeeerccoooo VLMDe jaVu
Introduction
Vision-Language Models (VLMs) have emerged as the state-of-the-art solution for learning rep- resentations from images and text data, with number of downstream applications such as image generation [Ramesh et al. 2021 2022 Yu et al. [002bb), retrieval [Wang et al.) [2015] Cao et al.) 2016 Zhang al. 2021 Baldrati al. 2022], captioning (Mokady et al., 202 IJ and classifica- tion. At the same time, large foundation models are known to memorize and retain information about their training data [Carlini al. 2019 Meehan al. 2023 Carlini al. 2023 and hence, a natural question is whether these
Models memorize as well. If so, this raises questions about generalizability of these models. We investigate whether Vision-Languaae Models retain information about their training data beyond the bounds of generalization.
The main challenge in measuring memorization is designing measurement technique that can tease apart memorization from spurious correlations. For example, for an image of a black swan on water, representation learning model may learn predict black swan given the background water either: (i) it retains extra information about the training image, or, (ii) if most of the examples in the training corpus with water also involve black swans. The first kind constitutes as memorization whereas the second kind is spurious correlation. This uncoupling of memorization from spurious correlation is particularly complicated for VLMs. Unlike generative models, VLMs as well other representation lcarning models lack decoders that can directly generate images or text; therefore, what the model learns about its training data has to be detected more subtly.
Prior work has looked into this problem for image-only representation models [Meehan et al. 2023 by measuring whether the model can predict the foreground of image (e.g. black swan) beyond simple correlations based simply on its background (e.g. water). However, such simple solutions do apply here. VLMs have two separate modalities and evaluate them are considerably more complex than the simple foreground backgruuund structure
and image, and the data sets used train
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Target Image
KNN Attack Procedure:
Target Caption: Healthy Fruits Have Vitamin D
Objects oringe, hanona, strawherry, leaf lettuce carrot, blueberry. tomato, ... [44 unique objects total]
Target Caption
Public Image Set
Top-K Images for Target VLM:
VLM
VLM
WWLLCOME
Objects Detected: 39/44
Precision: 0.53
Recall: 0.89
Text Embedding
Image Embeedings
Top-K Images Reference
K-NN Similarity Scarch
Top-K Images
Objects Detected:
An example where CLIP [Radford al. 2021 model trained 40M subset of Shutterstock data set exhibits déjà vu memorization of objects present in a training image. Public set is separate collection of 20M images from Shutterstock that has overlap with the training set. The objects annotated in orange are true positives, i.e., the ones present in the target image, and the objects annotated in blue are false positives. Our recovers significantly more memorized objects for the target VLM (trained on the target image) compared to the reference VLM (not trained on the target image). Additional qualitative examples be found in Figure in the appendix.
Figure 1:
of ImageNet (see Figure for example). A consequence that the image and text modalities can interact and transfer information in these models in subtly complex ways, making measurement significantly more challenging.
In this work, we propose a new method for measuring memorization in VLMs (depicted in Figure Given a target image caption, samples from a public set of images. Our test based on the key insight that an image-text pair is memorized by VLM, then the retrieved images would resemble the training image significantly higher amount detail than what is predictable from either the text caption or simple correlation. Formally, given text-image pair, we retrieve an image from the model based on an embedding of its text description, and we measure what fraction of ground-truth objects in the original image also co-occur in the retrieved image. Then, determine whether this happens simply due correlation. we measure how this compares with the same statistic obtained from a similar VLM which does not have this image-text pair its training data. Combining these two steps gives us measurement method that we call VL-Déja-Vir Vu.
the VLM to encode the caption and retrieve relevant image
We evaluate our test CLIP RRdford al. 2021 models trained subsets of Shutterstock and filtered version of LAION (filtered LAION) with varying number of training samples. We find that training data set sizes where CLIP generalizes well, there is significant degree of model memorization as depicted by our metrics (see [Section 4). Finally, we explore mitigation measures that reduce information leakage in [Section 5 We find that text masking significantly mitigates déjà vu memorization at a marginal cost to the model utility. We note that there could be other effective mitigations but were not explored due the computational limitations.
Contributions. Our main contributions are as follows.
We propose VL-Déjä-Vu-a new way of measuring memorization in VLMs by measuring what fraction of ground-truth objects in an image can be predicted from its text description for a training image-text pair.
Based on this measurement technique, we propose both (a) individual sample-level test to detect memorization for individual text-image pairs and (b) an aggregate population-level for a Vision-Lannuage Model.

---

We use our VL-Déja- Vu test to evaluate memorization in CLIP, and show that memorization does occur for VLMs trained using number of different training set sizes and regularization parameter values, even for settings where the model generalizes well.
Finally, we explore mitigation measures, and demonstrate that among number of different ways to train CLIP, random masking of text serves to significantly reduce déjà vu memorization.
2 Background
Language models [Radford et al. 2021 Li et al., 2022, Yu ct al. 2022a Li et al. 2023 Xu al. 2023 are multi-modal models whose core function is to map image-text pairs into a pair of representations that are semantically relevant. These embeddings can then be used for downstream tasks such as image classification, captioning, retrieval and generation. VLMs are composed of vision block, consisting of convolutional network or a vision transformer, and a text block, con- sisting of transformer, that produce image and text embeddings respectively from input image-text pairs. Given a trained vision-language model f. and an image-text pair = (Zimg: Stzt), we denote the corresponding image and text embeddings as f fZimg) and f (=tat)-
We consider VLMs that involve contrastive pre-training: in other words, during training, the model learns to minimize the distance between the image and text embeddings of the matching pairs in the training set Drr while maximizing the distance the mismatched pairs. The most commonly used contrastive loss is the InfoNCE loss (van den Oord et al. 2018)] given as follows:
L = --llog exp(f(eimg) T f((i))/))
(1)
E, exp(s(Emm)) {f (TT
where T is the temperature and 2², Vjj f are negative examples to contrast against. In practice, for each positive example 2", we use all other examples in training batch as negative examples. The most popular VLM of this type is CLIP (Contrastive Language-Iman (2021)), trained on an undisclosed data set, which achieves competitive out-of-the-bee performance across many transfer learning tasks. OpenCI.IP [Ihharco et al. [20TT] has released an open-source implementation of CLIP, and showed that training on filtered LATON dataset [Schuhmann et al.) 2021] can achieve comparable performance to the original CLIP model. Our work investigates memorization in OpenCLIP.
Pre-Training; Radford et al.
Memorization in ML models. It is well-known that machine learning models can memorize their training data in ways that enable data extraction. This phenomenon has been studied for both lan-
guage [Carlini et al. 2019 2021
et al.] 2020 Jayaraman et al.) 2022] and vi- al., 2023 models. However, all these works only consider the uni-modal setting, and as such the impact of this phenomenon is not clear in the multi-modal settings. Moreover, almost all the prior studies
sion (Somepalli al. [023 Carlini al. [2023] (Sablayrolles et al. 20188 Meehan
(except Meehan et al. 2023) focus on generative models language or vision where measuring memorization is easier because of the presence of a decoder.
Similar to Meehan et al. [2023], we investigate the setting of representation learning models, where do not have decoder and instead only have access to encoder. Although Melika/Meehan al. 2023]. who considered vision models that capture the relationship between representation of the background of an image (such water) and the label of its foreground object (such as black swan), we consider settings where the models are trained on more complex data sets that have multiple objects in any given image. Such a simple brregroundbbckgroone apply to our setting of Vision Language Models where the two modalities may leak training data in more subtle and complicated ways. Our work builds upon their test, and extends VLMs. A more detailed background discussion can be found in Appendix B
measurement does directly
3 Déjà Memorization for Vision-Language Models
Déjà vu memorization happens when foundation model retains information about individual train- ing data points beyond what expected by simple correlation, and allows the recovery of such information during inference time. An example is when an image representation learning model
3

---

can confidently predict the foreground of training imgee based simply on its background [Meehan al. 2023], while similar predictions cannot be made for test images.
In the context of Lasion-Language Models, however, measuring déjá vu memorization is not as simple, due to the presence of multiple modalities well as the complex nature of the training data. Compared to ImageNet, VLMs are trained on vastly more semantically rich data sets with many more objects as well as complicated captions, which may not capture everything in the image see Figure 6 for an example. This means that the text and image modalities can interact and transfer information in subtly complex ways, making measurement significantly more challenging.
To resolve this challenge, we instead propose to measure whether the ground truth objects in an image can be predicted from the representation of its caption. We rely the intuition that the caption of an image typically does not include all its objects, and hence high confidence recovery of this level of detail implies some form of memorization. If this prediction can be done significantly more accurately when the image is in the training set of a model than when it is in the test, then the image-text pair is being memorized by the said model.
vision-language model f suffers from déjà vu memoriza- tion if it retains specific information about the individual training images that allows the recovery of objects present in the training images. In other words. for a target image-text pair (Zam: 2xxt), more unique objects can be recovered from Zing given Zext when 2 is present in f's training set compared to when it not.
Definition (Déjà vu Memorization)
This is possible due to the model's ability to encode the individual objects in the image embeddings, which is in turn reflected in the corresponding text embeddings when the model minimizes the contrastive loss during training. Next we will discuss how we quantify this phenomenon using two separate models (a target and reference) as well as nearest neighbor test.
3.1 Measurement Methodology
Since VLMS are meant to capture general correlations between images and their text captions, goal is to differentiate the recovery of ground-truth objects due déjà vu memorization from dataset-level correlations alone. As a motivating example, consider the use of CLIP in a cross-modal retrieval task, where images are retrieved from a web-scale database given text. We wish to capture the degree of surprise in the retrieval result when the model memorizes training captions, i.e. how many objects can the model recover beyond dataset-level correlation? To enable this evaluation for a given image-text pair = = (Zsmgr Ztat), we use two separate VLMs fA and S' that are trained on randomly sampled but disjoint data sets A and B respectively. 2 lies in the training set of ex- actly one of these models, and hence by comparing the outputs of the two models, we can infer whether z was memorized. We do k-nearest neighbor test using a separate public set of images as described in [Algorithm I and find the subset of images that are closest to in the representation space. We then decode the objects present in these images. For this we use an object detector to provide ground-truth annotations for measuring the precision and recall of object recovery. We note that while there will always some bias when using object detectors, human or automated, this bias should not affect our evaluation when considering the gap between the two models. This is because the object detector not trained on the same training set as the VLM, hence any incurred bias should be independent of the trained VLMs.
3.2 Metrics
Our memorization metrics are built bottom-up from our notion of deja vu memorization for VLMs. We start from fine-grained sample-level metrics more aggregate population-level metrics. The k-nearest neighbor test in Algorithm I] shows how to obtain predictions of the ground-truth objects given an image; we next use these predictions to develop the population-level and sample-level memorization metrics. For our evaluation, we adopt the precision, recall and F-score metries from the information retrieval literature to quantify the fraction of objects memorized by the models.
Sample-level metries. At the sample level, we evaluate the fraction of ground-truth objects mem- orized by the target model from a given training image-text pair = (Zimg: Ztzt). To do this, we run the nearest neighhor test on both the target and reference models, SA and fB, to obtain their

---

Algorithm k-Nearest Neighbor Test
Setup Phase
Sample two disjoint data sets A and B consisting of image-text text pairs of the form (Zimg: Atzt) and train models fA and fs on the respective data sets.
2: Sample a separate public set of images P that disjoint from the images in A and B.
3: For each image Zing E P. obtain the corresponding image embeddings from both the models, fA (2igg))) and fø (2mmm))).
Testing Phase
4: Sample record from set, Z (Zimgs Ztat) E A, and obtain the corresponding text embed- dings from both the models, J'A and f (Ztat).
5: Obtain & public images NA P and Na C closest 2tzt in the embedding space for fA and fB respectively.
6: Evaluate the gap between the fraction of ground-truth objects detected in the sets NA and NB.
respective neighbor sets NA and Ng as per Algorithm We then calculate the precision, recall and F-score values when identifying the ground truth objects present in Zimg using NA and Ne and report the gap between the respective values for both the models. A positive gap corresponds to the target model memorizing the training sample and the magnitude of the gap indicates the de- gree of memorization. The precision, prec, and recall, recall, are given by the following equations (vi {A, B)):
# unique objects in N₁ Zimy # unique objects in N;
recall(2, # unique objects in N; n Zimg # unique objects in Zimg
(r, f.)
(2)
F-score is the harmonic mean of precision and recall.
Population-level metrics measure what fraction of the training data is memorized by model. For proper measurement, we propose three metries: population precision gap (PPG), population recall gap (PRG) and AUC gap (AUCG). Given the notations defined in [[Aloriithm 1] the population precision gap is the the fraction data points from where f'A has a higher precision in identifying the ground truth objects than I'r minus the fraction of data points where S' has a higher precision in identifying the ground truth objects than fA. If no memorization occurs, models SA and Ia should be interchangeable and hence this gap is zero. Formally.
PPG 1 (11 A: pre prec(2, fa)) A: prec(z, fA) prec(z, fz)H) [A]
(3)
where |A| denotes the size of the set A and prec(z, fA) measures the precision of object prediction on z given the model fA as defined in Equation We define the population recall gap similarly:
PRG = |A| (112 E A: recall(z, fA) > (x)) {(2 e A : recall(2, < rec
We also visualize the fine-grained cumulative recall distribution of both the models over the training set as shown in Figure 3) This gives us a better understanding of what fraction of objects are recov- cred overall. We then measure the difference between the two distributions (i.c., for fA and fB) to simplify this information into a single quantity we call AUC gap.
While both the population-level and sample-level metrics rely on the precision and recall functions, they have subtle differences. First, population-level metrics measure the aggregate memorization over the entire training set whereas sample-Ieve metrics measure the memorization in individual training samples. Second, population-level metrics rely on binary tests to differentiate between the target and reference models and as such not capture the magnitude of the gap between the models as is done by the sample-level metrics. We define both sets of metrics to capture the memorization different granular levels and to be actionable in a meaningful way, thereby allowing the model developers to fine-tune the models to mitigate the memorization risk.
5

---

LATON, ImageNett
LOMON-SOM
PS: 55-20M
0.6
d.56
0.3
0.6
0.3
Scoree
0.61
5.26
0.3
Accurscy Scare
ImageNet 25
Accurscy Scare
PRG
ImageNet 25
Ancuriity Score
0.4
0.25
-0.2
0.4
PPG
-0.2
0.4
0.19
PRG
ImageNet 28
0.41
0.2
0.12
0.2
p.16
0.09
0.04
0.1
0.2
AUCG
0.13
1.00
0.1
0.2
0.10
0.09
0.06
0.1
0.06
AUCG
0.06
AUOG
0.072
0.05
10M
50M
10M
50M
1M
10M
10M
Training Size
Training Size
Training Size
Figure 2: Utility and déjà va memorization of ViT-B-32 32 CLIP models with varying training set sizes. Model utility is quantified in terms of ImageNet zero-shot accuracy. oopulation-leve memorization of models is measured using the metrics defined in [Section ovvrr various public sets (a): training set sampled from filtered LAION and ImageNet is used as public set. (b): training set sampled from filtered LAION and a holdout filtered LAION-50M set is used as public set. (c): training set
sampled from Shutterstock and holdout SS-20M set is used as public set. For the memorization metries, we report the mean = std values (std 2 0.003) over 100 repetitions of randomly sampling 10% of records with replacement.
(a) LAION-1M M (Imagenet)
(b) LATON-10M (Imagenet)
(c) LAION-50M (Imagenet)
1.0
Praction of Records
0.8
Praction of Records
0.8
Fraction Records
0.8
0.6
0.6
0.6
0.4
AUC Gap: 0.0697
0.4
AUC Gap:
0.4
AUC Gap: 24
0.2
Target
0.2
Target
0.2
Target
Reference
Reference
Reference
0.0
20.0 202 20.4 20.6 2088 21.0 Recall (100 NNs)
20.0
202 20.4 20.6 20.8 21.0 Recall (100 NNs)
20.0
20.2
20.4 20.6 20.8 21.0 Recall (100 NNs)
Figure 3: Object recall distribution of target and reference models trained on filtered LAION data set for 200 epochs with different training sizes. ImageNet is used the public set for kNN test.
4 Evaluating Déjà vu Memorization
We next apply the metrics designed Section 3.1 to determine if CLIP memorizes training data. Specifically, we seek to answer the following two research questions:
1. How does déjà vu memorization vary with training set size and number of training epochs?
2. Are all training data points memorized uniformly?
Models and datasets. We train OpenCLIP from scratch on different datasets, including Shutter- stock (a privately licensed data set of 239M image-captions pairs) and (filtered LAION [Radenovic et al. 2023 + COCO [Lin et al. 2014). ). We sample up to 50M image-text pairs from the data sets and train OpenCLIP models with V-T-B-32 architecture. For Shutterstock experiments, we consider a separate set of 20M samples from Shutterstock (called SS-20M), with no overlap with the training sets, as public set. For the filtered LAION experiments, we consider two public sets: (a) separate subset of 50M samples from filtered LAION (called filtered LAION-50M with no overlap with the training sets, and (b) the entire ImageNet training set [Deng et 2009). More details on the
experiment setup and how we obtain data subsets can be found in Appendix C.
Model utility. As mentioned above (and also discussed in detail in Appendix C). we trained mod- els with different training set sizes consisting of 1M/10M/50M image-text pairs from filtered LAION and I MM/10M/40M image-text pairs from Shutterstock. We use zero-shot performance on ImageNet to evaluate the utility of these models. Figure 2 shows the zero-shot accuracy on ImageNet. Addi- tional utility benchmarks across various ARO (Autribution, Relation, and Order) tasks Yukkekkgnuul can be found in Figure 7 in the appendix.
2023 et al.

---

0.6
0.6
0.6
F-Score Cap
0.4
0.4
0.4
Procision
Recall
0.0
0.0
0.0
10P
10a
104
10F
109
10a
104
10
100
10a
10
106
Top-L. Records
Top-L Records
Top-L Records
Top-1 NNs
Top NNs
Top 10 NNs
(a) Records are sorted w.r.t the minimum embedding distance between target caption and public images.
Precision Gap
0.75
Recall Gap
0.75
F-Soore Cap
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
10a
10c
104
10a
10"
10
104
10a
10"
10a
10*
10s
Top-L Records
Top-L Records
Top-L Records
Top-1 NNs
Top-5 NNs
Top-10 NNs
(b) Records sorted w.r.t. the decreasing number of correct object predictions for target model.
Figure 4: Sample-level memorization gap between target and reference models when predicting top-10 objects for different top-I. records. Models are trained on disjoint 10M subsets of filtered LAION data set for 200 epochs and ImageNet public set is used for the KNN test. The model exhibits very strong déjà vu memorization on small subset of samples, as indicated by the large precision/recallf--cooe gaps when L is small.
4.1 Measuring Population-Level Memorization
For quantifying population-level memorization, we measure the gap between the object recall dis- tributions for the target and reference models. If there were no memorization, virtually no gap between the two distributions, i.e. AUCG O. Figure 3 shows the object recall dis- tribution gap between the target and reference models trained filtered LAION for varying training set sizes when ImageNet is used as the public set. When the training set size is small (e.g. 1M as shown in the left-most figure), there is higher déjà vu memorization due to the models overfitting on the training set. The gap decreases as the training set size increase from 1M to 50M, con- firming that the models begin generalize better. Note that the memorization is still significant for models trained on 10M data set. We consider this setting for further experiments as this is a typical training set size for many foundation models in practice [Tlharco et al., [202]. For instance, is common to train CLIP models on the 12M Conceptual Captions data set [Sharma et al.) 2018 or the 15M subset the YFCC data set [Thomee et al. 2016).
would observe
Apart from the AUCG (AUC gap) metric, we also quantify the gap in terms of the PPG (population precision gap) and PRG (population recall gap) metries. Recall that positive value for these metrics indicates memorization and the magnitude indicates the degree of memorization. [Figure 2 shows the PPG, PRG and AUCG metric values for models trained on filtered LAION and Shutterstock with different training set sizes; using ImageNet and filtered LAION-50M public sets for the filtered LAION models and SS-20M public set for the Shutterstock models. Recall that the public sets have no overlap with the model training sets. While the absolute metric values are different for different public sets, the trend remains the same: memorization decreases with increasing training set size as the models begin to generalize better. In Section 5, we explore various approaches to reduce this memorization.
4.2 Mcasuring Sample-Level Memorization
While the population-level metrics like AUCG, PPG and PRG show evidence memorization, they do not pinpoint which training images are more vulnerable. We sort the training data in decreasing order memorization to show the subset of most vulnerable records. To do this, we explore several sorting metrics. The most straightforward metric is the distance between the training text embed- ding and the nearest neighbour public image embeddings obtained using Algorithm The records for which the public image embeddings are the closest are more easily memorized by the model.
7

---

(a) Early Stopping
(b) Tempeeatire
(c) Weight Decay
(d) Text Masking
0.30
Zem-Shhot Anturnry
0.30
Zero Stot Accurscy
0.30
0.252
0.50
Statt Aecuracy
0.25
1236
0.242
0.240
0.202
0.25
0246
0.25
Zero-Shot Acnumay
0.25
6.20
0.20
0.20
0.20
0.203
0.10
0.15-
0.00
0.103
0.030
8-586
8:591
0.034
0.15
0.10
0.00
5.061
1.3
AUCO
1981
PPM
0.033
0,022
034
0.034
10092
0.092
0.10
0.15-
0.08
0.00
0.036
0.094
ALCG
PPB
PBG
0.092
0.088
0.15
0.10
0.05
0.00
002
P30
100
ADCO
0
033
Training Epocks
Temserrauur
Weight Docay (ed)
Masking (ov)
Figure 5: Effect of mitigation on ViT-B-32 OpenCLIP models trained on 10M subset of filtered LAION. Memorization evaluation is done using ImageNet as public set. Default setting is high- lighted with asterisk. For the memorization metrics, we report the mean std values (std S 0.003) over 100 repetitions of randomly sampling 10% of records with replacement. Among these mitiga- tions, text masking has the best trade-off that reduces memorization without sacrificing utility.
Compared to the population-level memorization, where we keep the experiments parameter-free to the best extent, at the sample-level we want to focus on more fine-grained leakage so we choose top-10 object labels to measure the gap instead of predicting all the objects.
Figure 4a shows the precision, recall and F-score gaps between the target and reference models for varying top-k records sorted with respect this distance metric where ImageNet is used as the public set. As shown, the gaps can be greater than 0.3 for top-1 and top- 10 records. We also tried sorting the records in the decreasing order of the number of objects correctly identified using the target model with the nearest neighbor test. Figure 4b) shows the precision, recall and F-score gaps for the records sorted using this metric. We see that the gap can become very significant for the top-1 and top- 10 records. Although this metric requires access to the ground truth labels, this is still useful to visualize the worst case examples. Results for sample-level memorization with filtered LAION- 50M public set show similar trend and can be found in /Section D.I Sample-level memorization results for Shutterstock experiments can be found in Appendix E
Key Observations. We show déjà vu memorization at both population and sample levels. At the population-level. where we measure the aggregate memorization of model over the training set, we find that the memorization decreases with an increase the training set size. This could be attributed to improved model generalization. At the sample-level, we note that the model memorizes disproportionately- subset of training image-text pairs are memorized more than the others.
5 Mitigation
How can mitigate déjà memorization in VLMs? Since it presumably happens due to the model overfitting on training data, it is likely that regularization techniques may be able to mitigate it. We investigate the impact of four regularization techniques déjà vu memorization.
Early stopping is a common technique for regularizing neural networks where model training is ended prematurely. It is effective due to the observation that models begin to overfit on the training set when they are trained for more epochs.
2. Temperature is the contrastive loss parameter that controls how close the text and image embed- dings can get during the model training. Changing the temperature parameter has a regularization effect for SSL as observed by Mcchan et al. 2003]]
3. Weight decay, also known as Le regularization. is a standard ML regularization technique.
4. To reduce overfitting along the text and image modalities in VLMs, we look at additional regular- ization through text randomization, where we randomly mask a fraction of the text tokens during training. We control the fraction of text tokens masked using masking ratio parameter.
In the following we present results when ImageNet is used as the public set for the nearest neighbor test. Results for the filtered LAION-50M public set can be found in /Section D.2 Since Shutterstock memorization trends are similar to those of filtered LAION, we only explore filtered LAION settings for mitigation.
8

---

5.1 Early Stopping
It widely believed that deep learning models begin to overfit on the training data as the number of training epochs increases. It is thus good practice to early stop the training as soon as the model utility on a hold-out test set stagnates or begins drop. However this is often not the case for SSL models. It is not uncommon to observe that the zero-shot accuracy of SSL. models keeps improving as the models are trained for longer [Meehan et al. 2023] Regardless, we still explore early stopping as mitigation mechanism. As shown in Figure 5 training the CLIP model for more epochs leads to better zero-shot accuracy, but at the same time, déja vu memorization also increases. This is in line with our hypothesis above. Even when we early stop the model at 20 epochs (10% of the default parameter value 200 epochs), the memorization risk is not completely mitigated although the absolute values are lower.
5.2 Temperature Scaling
Temperature, or logit scale, controls how close the text and image embeddings can get during train- ing. Smaller values allow for the multi-modal embeddings to get closer, and as consequence the CLIP contrastive loss drops quickly, whereas larger values regularize the loss but may lead to train- ing instability as noted by RRdford et al. [202T]. The default value in OpenCLIP implementation is set to 100. We vary this value between 25, 100 and 200. As shown in Figure 5| decreasing the temperature (T") from 100 to 25 decreases the model's zero-shot classification accuracy on Im- ageNet from 25.2% to 21.7% and also increases the memorization as indicated by the increase in the PPG, PRG and AUCG metrics. This is due to the decrease in the distance between the text and image embeddings for the training data which could potentially lead to model overfitting. Increas- ing the temperature to 200 moderately impacts the model's ' zero-shot classification accuracy and the memorization leakage remains more or less the same.
5.3
Weight Decay
Weight decay directly controls the model overfitting, with larger values corresponding to stronger regularization. The default value is set to 0.1 and we vary between 0.03, 0.1 and 0.3. As expected, decreasing the weight decay wd from 0.1 to 0.03 decreases the model's zero-shot classification accuracy and also worsens the leakage due to memorization as shown in Figure 5) Interestingly, increasing the weight decay to 0.3 significantly improves the model's zero-shot accuracy. We believe that the default value of 0.1 is not optimal for the 10M training set size as was set based on the model training for larger data sizes (possibly on the entire filtered LAION data set). With 0.3 weight decay, we observe a consistent decrease in the population memorization leakage, as shown by the PPG, PRG and AUCG values for wd 0.3 in Figure 5] but the values are still significantly high. We also explored setting weight decay to 0.01 and 1.0, but they either adversely impacted the model utility severely increased memorization. Thus while tuning wd does not completely mitigate memorization, we can get a reasonable trade-off in the neighbourhood of wd = 0.3.
5.4 Text Randomization
During model training, the CLIP models increase the cosine similarity between the matching image- caption pairs while simultaneously decreasing the cosine similarity between mismatched pairs to reduce the contrastive loss. While is common to augment the training images to reduce overfitting, the text captions are not randomized. This could lead to the model overfitting on the text captions when minimizing the contrastive loss. To avoid this, we propose text randomization as a defense. For COCO subset of the training set, we randomly choose one out of the five captions for cach image per epoch during training. For filtered LAION subset, tokens since only a single caption is available per image in the filtered LAION data set. We vary the masking ratio between 0 (no masking), 0.3 and 0.5 (randomly mask half of the tokens).
randomly mask a fraction of caption
We find this defense to work the best in mitigating déjà vu memorization but at the cost of ImageNet zero-shot accuracy. As shown in Figure 5) using masking ratio of 0.3 reduces the ImageNet zero- shot accuracy from 25.2% (in the default case when nrr = 0.0) to 24.1 14.1% 1%, but at the same time this significantly reduces memorization. The PPG metric reduces from 9.1% 1% to 3.4%, and the PRG metric reduces from 9.2% to 3.8%. Moreover, the recall CDF gap (AUCG) also reduces from 0.034 to 0.013. Further increasing the masking ratio to 0.5 mitigates the risk even more. PPG reduces to

---

3.0%, PRG reduces to 1.9%, and AUCG reduces to only 0.007. However, we note that text masking has positive impact on ARO benchmark utility as shown in Figure 7
This is because masking avoids overfitting on specific text tokens making the models less likely to behave like rog-of-words Thus text masking achieves the best utility trade-offs. We would expect significant drop in the model utility if we further increase mr since the captions would have considerably less information.
Key Observations. We study the impact of tuning four regularization parameters: number of training epochs, temperature, weight decay and masking ratio. We find that early stopping reduces memorization but at the cost of model utility. Increasing the temperature increases the model zero- shot accuracy and decreases memorization up to certain threshold, beyond which the model utilityy begins to decrease. Surprisingly, we find that the default value of 100 already gives the optimal results. Similar to temperature, increasing the weight decay increases the model utility and decreases the memorization up to a certain threshold. We find 0.3 weight decay to achieve the best results for model trained over 10M data. We observe a sharp decrease in model utility beyond this value. Text masking seems to be most effective in mitigating memorization. Increasing the masking ratio decreases memorization and also decreases the model utility. Masking ratio of 0.3 achieves good trade-off by significantly reducing memorization while only moderately impacting the model utility.
6 Discussion
Prior works have mainly shown memorization in the uni-modal setting: either for the language models [Carlini al. [019] or for vision models (Meehan et al. [003]. We have demonstrated that even in the complex prior works have only evaluated memorization for small training data sizes (typically the scale of million or less), we show memorization on a wide scale, from 1 million to 50 million training set size. Our experiments show that while the population-level memorization metries decrease with increase in the training set size, there remain strongly memorized examples as exemplified by the sample-level memorization where the model disproportionntely memorizes subset of records.
modti setting, ML models suffer from memorization. Morcover, while
Careful tnning of right hyper-parameteen can, however, mitigate this memorization risk. We propose suite of metrics to quantify déjà vu memorization in hope of guiding ML practitioners to train mod- els in a safe way. These metries not only quantify the risk in a mcaningful and interpretable manner, but are also sensitive to the tuning of the mitigation parameters, thereby aiding the practitioners in choosing the right model
values that achieve a good utility-ria trade-off.
Below we discuss some detailed discussions and limitations of our work.
Not applicable to out-of-box models. Since our tests require access to two models, target and reference, along with the underlying training set, we note that this can not be directly applied to measure memorization in out-of-the-bon pre-trained models as there is no reference model for such cases. We leave this case future work.
Distinguishing memorization from learning. A model can memorize and generalize (or learn) at the same time. This can happen at population level, where the model memorizes rare concepts and generalizes to common concepts, or even at a sample level, where memorization is required for learning rare concepts theorized in Feldman 2020) Déjà vu memorization meant to go beyond this, and instead examine when a model that is trained on an image with a generic caption (i.e., they do not describe the image in high detail), memorizes many small details about the associated image (i.e., what objects are present in the image) when given the caption. In other words, we define déjà vu memorization as what can be inferred about the training image from its caption beyond simple correlations, which can happen through both learning and memorization in the traditional sense.
Extending beyond objects. While our approach is also applicable to annotations that go beyond objects, this is not in the scope of this work. Even in this setting, the prior sfate-of-art approach (Mee- han et al. 2023) only considers a single object label per image (ImageNet) and none of the prior works consider a. multimodal setting, b. large training size sizes, and c. multiple objects per image.
Relation to Overfitting. Déjà vu memorization measures overfitting at a more granular level- instead of binary decision, it measures to what degree the model overfits a training sample.
10

---

Acknowledgements
We thank Diane Bouchacourt for helpful feedback. We would also like thank Amro Abbas for helping in obtaining the de-duplicated version of filtered LAION data set and Evgenia Rusak for helping with the OpenCLIP implementation.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv:2303.09544 2023.
Alberto Baldrati, Marco Bertini, Tiberio Uriechio, and Alberto Del Bimbo. Conditioned and Com- posed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, 2022.
Yue Cao, Mingsheng Long, Jianmin Wang, Qiang Yang, and Philip S Yu. Deep Visual-Semantic Hashing for Cross-Modal Retrieval. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Symposium (USENIX Security 19), 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine
Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, et al. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski. Vikash Sehwag, Florian Tramer, Borja Balle. Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. In 32nd USENIX Security Symposium (USENIX Security 23), 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference compuler vision and pattern recognition. IEEE, 2009.
Rachna Dhamija and Adrian Perrig. Déjà Vu-A User Study: Using Images for Authentication. In 9th USENIX Security Symposium (USENIX Security 00), 2000.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
Matt Fredríkson, Somesh Jha, and Thomas Ristenpart. Model Inversion Attacks that Exploit Confi- dence Information and Basic Countermeasures. In Proceedings of the 22nd ACM SIGSAC Con- ference on Computer and Communications Security, 2015.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021. URL. httpp:://dst wttpioorg/11..03
renodo.514377
Bargav Jayaraman and David Evans. arXhv:2209.0129. 2022.
Are
Attribute Inference Attacks
Just Imputation?
Bargav Jayaraman, Esha Ghosh, Melissa Chase, Sambuddha Roy, Wei Dai, and David Evans. Comb- ing for Credentials: Active Pattern Extraction from Smart Reply. arXiv:2207.10802. 2022.
Diederik P. Kingma and Jimmy Ba.
Adam:
Method for Stochastic Optimization.
1412.6/412.69000 2017.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre- Training for Unified Vision-Language Understanding and Generation. In International Con- ference Machine Learning. PMLR, 2022.

---

Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 23390-23400,, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014.
Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning. arXiv:2304. 13850, 2023.
Ron Mokady, Amir Hertz, and Amit Bermano. ClipCap: CLIP Prefix for Image Captioning. arXiv:2111.09734, 2021.
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, Distillation, and Hard Neg- atives for Vision-Languag Pre- Training. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models from Natural Language Supervision. In Interna- fional conference machine learning. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-lmage Generation. In International Conference on Machine Learning. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv:2204.06/22 2022.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Déjà Vu: an empirical evaluation of the memorization properties of ConvNets. arXiv:1809.063% 2018.
Christoph Schuhmann, Richard Vencu, Romain, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAI1N-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. arXiv:2111.02114. 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018. URL ptttp:://aclantholl org/ 1238
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference At- tacks Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048-6055 2023.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The New Data in Multimedia Research. Commun. ACM, 2016. doi: 20.114/2211802 https:/s:///dii
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding. arXiv: /107:08880037.. 2018.
Daixin Wang, Peng Cui, Mingdong Ou, and Wenwu Zhu. Deep Multimodal Hashing with Orthog- onal Regularization. In fourty-fourth international joint conference on artificial intelligence, 2015.
12

---

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes. Vasu Sharma, Shang- Wen Li Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data, arXiv preprint arXiv:2309. 16671, 2023.
Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfusca- tion in imagenet. In International Conference on Machine Learning (ICML), 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learn- ing: Analyzing the Connection to Overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF). IEEE, 2018.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.01917, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasude- van, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. arXiv:2206. 10789, 2022b.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why Aision-language models behave like bags-of-worde and what to do about it? In International Conference Learning Representations, 2023. URL /ttptptss///ppeeeeieeee. net fooru??
Idd=RLUUvhhuuXX
Santiago Zanella-Béguee Béguullin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Mare Brockschmidt. Analyzing Information Leakage Updates to Natural Language Models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, 2020.
Peng-Fei Zhang, Yang Li, Zi Huang, and Hongzhi Yin. Privacy Protection Deep Multi-Modal Retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development Information Retrieval, 2021.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting tthnsadd Classes using Image-level Supervision. In ECCV, 2022.
13

---

A License of the assets
A.1 License for the code
The licensing information for OpenCLIP [lbarco et al. 2021] can be found at https:
7/github. com/ foluddatioonn wtw.uuuaaiiissppenennpppppe optn_ccippbbobbb.bbccepppppppppa htt///ainn//// wwin/LICENssRRRR.
We use the code from Meehan et al. [2023] for memorization quantification, the licensing infor- mation can be found at https: //gtttb........ wtto///aaacceeeerrtt... jaaautta jbb License-l-ov-En file4iieeeeee readme
For object annotations, we use Detie [Zhou et 2022 the li-
censing information can be found at https:///ithub. http://hhhbb.ccmmcccccoooereeree
blob/ma n LICEESE
A.2 License for the data sets
We use ImageNet YYng et al., 2022 for which the license can be found at https::/ / www. image-nett org wwwp///wwwwooooooooo/oppo.o.oooo down oad php. We use a filtered version of LAION [Radenovic et al.) 2023] (which we call filtered LAION) for which licensing information can be found at https: hollbbmmann/iii
The licensing infor- mation for the MS COCO data set [Lin et al.] 2014] that we use be found at https: //cocodataset. org/ it termsoouse We also use Shutterstock data set which is a private licensed data set consisting of 239M image-caption pairs.
//githab. com/ http//accbbokkesearrhhee
B Background and Related Work
Foundation models, such large language models, have been long known memorize their training data in ways that enable easy extraction. For example, a line work [Carlini et al.) 2019 2021 Banella-Béguelin al. 2020 Jayaraman et al.] 2022 has shown that large language models exactly memorize sequences of text tokens from the training data, and these text tokens can be extracted. Somepalli al. 2023). Carlini al. (023) showed that diffusion models can generate images that are semantically and stylistically similar to training images or even their near-exact copies under certain circumstances. However, almost all prior studies that demonstrate this kind of memorization focus on generative models - language or vision - where measuring memorization is easier because of the presence of decoder. In contrast, our work is concerned with representation learning models, where we simply have an encoder.
Sablayrolles et al. [0118) study déjà vu memorization in neural networks and show that it is pos- sible to infer whether an image or a subset of images was used in model training. Our work is also closely related to |Mehan al. (202).. which measures déjà memorization image represen- tation models. They show that given the representation of the background of an image, (such as water), the label of its foreground object (such as black swan) this prediction is significantly more accurate for images in the training set of a model, thus showing that the models memorize their training data beyond the bounds of spurious correlation. However, such a simple foreground-backgroun multi-modal Vision Language Models where the two modalities may leak training data in more subtle and complicated ways. Our work builds upon their test, and extends it to VLMs.
be predicted reliably. Moreover,
measurement does not directly apply to the more complex,
Finally, there has been a body of work empirical measurement of privacy, and broadly speaking. there are three main kinds of attacks. In membership inference (Shokri et al. 2017], the goal is to determine if specific data point was used to train model. In attribute inference Yeom et al. 2018). the goal is to infer unknown features or attributes of data point based on a model trained on this or similar points. Finally, training data reconstruction attacks [Fredrikson al. 2015 aim recover one or more training data points given model and some auxiliary information. Our work falls within the purview of attribute inference. However, unlike most attribute inference attacks which
We note that many prior works have used the term "déjà vu" in different contexts. Dhamija and Perrig 2000 use this refer to the ability of humans recognize images, and they it proxy for password- based authentication. Sablayyolles et al. (2018) denote déjà vu essentially mean membership inference, where they test if a model remembers if an image was used in training. Meehan et al. (2023) refer to déjà vu as the ability of inferring foreground objects from vision models given background patch of pixels. We use this term refer to vision language model's ability recall the individual objects in the training images.
14

---

were shown to be forms of statistical imputation Dayaraman and Evans 2022], our tests directly measure how much more effective attribute inference can be when a data point is in the training set of model.
C Detailed Experiment Setup
For our experiments we use OpenCLIP [Ihaaroo et al., [021] to train the models. For filtered LAION experiments, we train models over subsets of filtered LAION (Radenovic al. 2023 and MS- COCO [Lin et al. 2014] data sets. For Shutterstock experiments, we train models over various subsets of Shutterstock data set, a privately licensed dataset of 239M image-captions pairs.
Obtaining Data Splits. As discussed in Algorithm 1 our test requires disjoint training sets A and B to train the models fA and fB respectively, and additionally we require a public set P. that has no overlap with A and B, for our nearest neighbor search. Moreover, for our tests to be meaningful we need to remove duplicate same sample is also present in the public set and as a result we would overestimate memorization. Conversely, if the same sample is present in both A and B sets, then it is harded to distinguish the outputs of two models and we would underestimate memorization. This type of duplication is common internet-scraped data sets such as filtered LAION and Shutterstock. We perform semantic deduplication over filtered LAION data set using the procedure of [bbas et al.] [2023] to obtain 220M deduplicated image-caption pairs. The Shutterstock data set has different type of duplicates- multiple images are present with same verbatim captions. So do simpler yet effective deduplication by considering only one unique image per caption. This reduces the overall data set size to around 103M image-caption pairs.
pairs otherwise the kNN test becomes trivial if the
For filtered LAION experiments: To obtain the two non-overlapping training sets for filtered LAION experiments, we sample 40K image-text pairs from COCO data set and 1M/10M/50M image-text pairs from filtered LAION data set to form A set. We do the same from the remaining pool of data obtain the B set. Since the COCO part of the A and B sets is insignificant compared to the filtered LAION portion of the sets, we only count the filtered LAION portion size for simplicity when we say we sample IM/10M/50M training sets. To obtain the filtered LAION-50M public set, we sample 50M pairs from the remaining pool of deduplicated filtered LAION which has 120M pairs (after removing the largest and B sets from the original 220M data set). We include most of the results on this public set in [Appendix D Since this data set may contain human faces, we perform blurring on all the sets. We also take the 1.28M images from ImageNet data set (Deng et al. 2009) and perform face-blurring to form our ImageNet public set.
For Shutterstock experiments: We take the caption-level deduplicated data set consisting of 103M image-caption pairs and randomly split it into 40M 40M 20M sets. The first two 40M sets are used to obtain the IM/10M/40M A and B sets respectively. The last 20M set is used as the public set. A small portion of the remaining 3M data is used hold-out set for prpee-prrameter tuning during model training.
Model Hyper-Parameter Settings. We use the ViT-B-32 CLIP model architecture consisting of around 151M trainable parameters and train the models for 200 epochs using Adam (Kingma and
Ba 2017] optimizer with cosine learning rate scheduler and learning rate of 0.0005. For filtered LAION experiments, we use 256 Nvidia Quadro GP100 GPUs with 16GB VRAM to train the models in parallel with an effective batch size of 16 384. We set the weight decay to 0.1 and use 1000 warmup steps for the learning rate scheduler. For Shutterstock experiments, we use 32 Nvidia A100 GPUs with 80GB VRAM to train the models in parallel with an effective batch size of 32 768. We set the weight decay to 0.2 and warmup to 2000 steps. All the model training runs use 512GB RAM and the training time scales with the data size: training on 10M data size takes around 2 days and training on 50M data size takes around 10 days to complete. All other hyper-parameters are set to the default value as used in OpenCLIP; we do an ablation study on the impact of temperature and weight decay in Section 5
Obtaining Object Annotations. For quantitative evaluation of our nearest neighbor tests, we re- quire detailed object annotations for the A, B and P sets. Both Shutterstock and filtered LAION
15

---

Caption:
Several kitchen workers making dishes commercial kitchen.
Labels:
Label: Orchard Oriole
Catsup Bottle, Pot, Hat, Plate. Dinner
Napkin, Finger Bowl, Soda Can, Bottle, Dripping Pan, Cup. Work Shirt, Bowl, Apron, Person, Belt, Pan.
(a) ImageNet Sample.
(b) COCO Sample.
Figure 6: Comparing images from ImageNet and COCO data sets. The ImageNet images only have single label per image but COCO images have complex scenes with multiple object labels. Additionally. COCO images have accompanying text captions. Label annotations with bounding boxes are highlighted in blue for both the images.
data sets only have image captions and no object annotations. ImageNet originally has only one object annotation per image, as shown in Figure 6) Hence, we use an open-source annotation tool, called Detic [Zhou al. 2022]. ,, obtain multiple fine-grained object annotations per image for all our data sets. This tool can annotate all the 21K ImageNet objects. Detic uses a default threshold of 0.5 to identify object bounding boxes (i.e., any bounding box that has more than 0.5 confidence is considered for annotation). For Shutterstock we use 0.3 threshold as the 0.5 threshold results in nearly 17% images with no annotations. For all other data sets, we use the default value of 0.5. Even though COCO has multiple object annotations, its class label space is small (i.c., only 80 unique classes). Hence we use Detic on COCO to extend its annotations and to make the label an- notations consistent across all the data sets we use. Figure 1 shows the sample images with multiple object annotations obtained using Detic.
Limitations in Experimental Evaluation. We find that the object annotation tool, Detic [Zhou (ct al.] 2022]. is not always accurate. For instance, the tool often classifies a 'polar bear" as 'jaguarundi' However, our experiments rely the relative gap in the object detection between the target and reference models and as such are robust to these inaccuracies as long as the annota- tions consistent across the images. For instance, if the 'polar bear' is classified as 'jaguarundi' across all the public set images, the gap between the 'polar bear' detection accuracy of target and reference models, based on our nearest neighbor test, will remain consistent. While the absolute numbers in our quantitative tests may vary based on the object annotation tool used, our experimen- tal observations would not change.
D Additional Results with filtered LAION-50M
discussed the memorization results considering the ImageNet the public set for our nearest neighbor test. Here we discuss the results with a much larger filtered LAIÓN-50M data set the public set. While the overall trend remains the same with the ImageNet, with richer public set, we are able to achieve larger memorization gap for our models.
In Section 4|
D.1 Sample-Level Memorization
Similar to the sample-level evaluation for ImageNet public set in Section 4.2, we evaluate the gap in precision, recall and F-scores of top-k records sorted with respect to the minimum embedding distance when considering filtered LAION-50M as the public for the nearest neighbor test. Fig- ure 8a shows the memorization gap of the top-k records. We note a greater precision gap with top- nearest neighbor when compared the case where ImageNet was used as a public set (see Fig- ure 4a). However, the recall gap is lower with this public set. These variations could be due to the nature of the public set- many filtered LATON images have few or no annotations. This does not mean that the sample-level memorization risk is lower. As shown in Figure 8b) the memorization gap is much higher for this public set when we sort the records in the decreasing order of the number of correct predictions made by the target model using the nearest neighbor test. This corroborates
16

---

ARO Accuracy Comparison
VG Relation
Attribution
COCO-Order
60
59
64
57
60
62
58
56
65
40
50
48
34
22
Naive Baseline
Prefrained CLIP
Pretrained CLIP
LAION 10M VIT
LAION 50M VIT
Shutterstock
Shutterstock
Impact of Temperature
Imppact of Weight Decay
VG Relation
VG Altribution
COCO-Order
VG Relation
VG Altribution
COCO-Order
80
54
54
60
55
55
54
52
60
55
27
29
29
27
29
LAION
LAION LAION LAION
T=25
T=100"
T-200
WD=0.00
WD=0.1'
WD=0.3
Imppac of Text Masking (LAION)
Imppact of Text Masking (Shutterstock)
VG Relation
VG Attribution
COCO-Order
VG Relation
VG Attribution
COCO-Order
60
59
53
58
58
59
60
29
20
29
22
LAION LAION 10M VIT LAION 10M VIT
Shutterstock
Shutterstock
Shutterstock
MR=0"
MR=0.3
MR=0.5
10M VIT MR=0"
10M MR=0.3
10M MR=0.5
Figure 7: ARO benchmark accuracy comparison for various models. Top figure compares the accu- racy of various baseline models on three compositional reasoning tasks: Visual Genome Attribution, Visual Genome Relation and COCO Order. The pretrained CLIP models are trained on 400M pri- vate dataset, whereas our LAION and Shutterstock models are trained on smaller subsets of the respective datasets. Our models are comparable the pretrained CLIP models in VG Relation and Attribution Tasks. The middle figures show the impact of temperature (left) and weight decay (right) on the ARO accuracy for our models trained on 10M subset of filtered LAION dataset. As shown, the default parameter values (shown by asterisk) achieve the best values for most cases. The bottom figures show the impact of text masking on ARO accuracy for our models trained on 10M subsets of filtered LAION (left) and Shutterstock (right) datasets. Our text masking does not deteriorate the model utility, and in fact further boosts ARO accuracy COCO ordering task. This is because text masking avoids overfitting specific text tokens. Thus, unlike the unmitigated CLIP models, the mitigated models are less likely to behave like Pag-of-words
our population-Ievel memorization results in Figure where we find higher memorization gap with filtered LAION-50M public set.
D.2 Mitigation
We observe similar trends for mitigation with different regularization parameters with the Im- ageNet case. Figure shows the impact of different parameters on the memorization. Since the filtered LAION-50M public set is much larger than the ImageNet public set, the overall memo- rization values are higher due to the public set nearest neighbors being more representative of target image, and thus capturing more objects. However, the trend remains the same. Increasing the temperature decreases the memorization, but the default value of 100 is close to optimal as the trade-off between memorization and model utility is the best. Increasing the weight decay improves
17

---

1.00
1.00
1.00
Precision Gap
0.75
Recail Gap
0.75
F-Score Cop
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
10P
10a
104
10F
109
10P
104
10F
109
10a
10
106
Top-L. Records
Top-L Records
Top-L Records
Top-1 NNs
Top NNs
Top 10 NNs
(a) Records are sorted w.r.t the minimum embedding distance between target caption and public images.
Precision Gap
0.75
Recall Gap
0.75
F-Soore Cap
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
10a
10c
104
10a
10"
10a
104
10a
10"
10a
10*
10
Top-L Records
Top-L Records
Top-L Records
Top-1 NNs
Top-5 NNs
Top-10 10 NNs
(b) Records sorted w.r.t. the decreasing number of correct object predictions for target model.
Figure 8: Sample-level memorization gap between target and reference models when predicting top- 10 objects for different top- records. Models are trained on disjoint 10M subsets of filtered LAION data set for 200 epochs and filtered LAION-50M public set is used for the KNN test.
(a) Temperatare perature
(b) Weight Decay
Text Masking
0.30
0.30
0.30
1.227
Zero-Shot Accuracy
0.25
0.235
0.252
0.303
0.25
Zero-Shat Acourccy
Metzie Score
PRG
0.252
0.248
Zero Shot Accuracy
1252
0.241
PPG
0.155
0.164
PPG
0.159
0.152
0.203
0.10
0.15-
0.092
0.057
0.152
0.056
0.148
0.15
0.10
0. 162
3.061
PRG
0.057
0.152
0.053
0.143
0.10
0.15
0.132
0.057
PRG
PPG
0.101
0.066
0.99
0.05
AUCG
0.05
A0OG
0.05
0.023
0.037
0.01 0.013
0.00
0.00
25
100
200
0.03
0.1
0.3
0.0
0.3
0.5
Temperature
Weight Decay (wd)
Masking
Figure 9: Effect of parameter tuning on ViT-B-32 CLIP models trained on 10M subset of filtered LAION for 200 epochs. Memorization evaluation is done using filtered LAION-50M as public set. Default setting is highlighted with asterisk. For the memorization metrics, we report the mean + std values (std < 00.03) over 100 repetitions of randomly sampling 10% of records with replacement.
the model utility (indicated by the zero-shot accuracy) and decreases memorization. Weight decay of 0.3 gives near optimal trade-offs. Further increasing wd to 1.0 results in drastic decrease in model utility, and thus we do not include the results. Increasing the masking ratio from 0 to 0.5 significantly reduces the memorization but at the cost model utility. While the optimal value of mr would depend on the application and how much tolerance on the model utility loss is acceptable, find that mr 0.3 achieves a significant reduction in memorization while only moderately im- pacting the zero-shot accuracy, as shown in Figure 9 Any further increase in mr beyond 0.5 would greatly sacrifice the model utility and thus not recommended.
E
Additional Results with Shutterstock
Similar to the sample-level evaluation for models trained on filtered LAION data set in Section 4.2, we evaluate the gap in precision, recall and F-scores of top-k records sorted with respect to the minimum embedding distance for models trained on Shutterstock data set when considering SS- 20M as the public set for the nearest neighbor test. FFiurre 10a shows the memorization gap of the top-k records. We note smaller precision and recall gaps for this data set. This is due to two reasons: (a) nature of the data set- Shutterstock data set has many similar images even after
18

---

1.00
1.00
1.00
Precision Gap
0.75
Recail Cap
0.75
B-Score Cop
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
AP
10P
10a
104
10F
100
10a
104
10F
109
102
10
106
Top-L Records
Top-L Records
Top-L Records
Top-1 NNs
Top NNs
Top 10 NNs
(a) Records are sorted w.r.t the minimum embedding distance between target caption public images.
Precision Gap
0.75
Recall Gap
0.75
B-Soore Cap
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
10a
10c
104
10a
10"
10a
10*
10a
10"
10a
10*
10s
Top-L Records
Top-L Records
Top-L Records
Top-1 NNs
Top-5 NNs
Top-10 10 NNs
(b) Records sorted the decreasing number of correct object predictions for target model.
Figure 10: Sample-level memorization gap between target and reference models when predicting top-10 objects for different top-I. records. Models are trained on disjoint 10M subsets of Shutter- stock data set for 200 epochs and SS-20M public set is used for the KNN test.
do the caption-level deduplication (see Appendix C) so even the referece model performs well on this data set, and (b) the model hyper-parameter settings for this training set size is possibly sub- optimal-tthe the model zero-shot accuracy on ImageNet seems to be the highest at 20 epochs (18. 16%) and it slightly decreases till 200 epochs (17.49%) when trained Figure TOb] shows the memorization gap when we sort the records the decreasing order of the number of correct predictions made by the target model using the nearest neighbor test. As expected this gap is much higher than the previous case. Overall, the trends are similar to the filtered LAION experiments.
10M subset of Shutterstock data.
19

---

Target Imsage
Desected: 11:14
Precision: 0.31
Caption: clean pictured
tnined
Desected: 6/14
image.
Recall: 0.43
Target Imaae I
Detected: 13/16 Procision:
I-NNe of model traned
Caption: A
sundwich shewn
with com chips
Desected: 7/16
Precision: 0.21
Recall: 0444
Target Image T
I-NN triined with T:
Desected: 7/11
Recall: 0.64
L-NNa of model tminnd wio
Captien: group hirds sitting wooden platioem.
Precision: 0.04
Recall: 0.09
Target
I-NNs of model
Inined with T:
Precision:
Recall: 0.75
L-NNs model
Caption: a blue chair Bost of'a
desk
Desected: 4/16
Precision:
Figure 11: Additional examples showing déjà memorization. Target images are from COCO training set and the public images are from ImageNet data set. The objects annotated orange are true positives, i.e.. the ones present in the target image, and the objects annotated in blue are false positives.
20

---

Target Imsgee
NNs model
Desected: 47:63
Precision: 0.26
Caption: 100
tmined
activities
Desected: 13/63
simple style for any design illustation
Recall: 0.21
Target Imaage
Detected: 34/34 Procision:
I-NNe medel
Caption: Resiness
Team Concept
UPLUAD
Desected: 5/34
Precision: 0.08
Recall: 0.15
Target Image
I-NN triined with T:
Desected: 35/64
Recall: 0.55
L-NNa of model tminne wio
0
Caption: People
drinks sitting at the dining
table
Precision: 0.09
Recall: 0.12
Tarset Imaze
I-NNs of model
tnined with T:
51:53
Procision:
Recall: 9696
I-NNs model
Caption; making
Chiistmas advenl
calendar with sweets
Desected: 17:53
activitios
Precision:
Figure 12: Additional examples showing déjä memorization. Target images are from Shutterstock training set and the public images are from SS-20M public set. The objects annotated in orange are true positives, i.e.. the ones present in the target image, and the objects annotated in blue are false positives.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction about the deja memorization for VLMs are accurately reflected in the paper's contributions. Namely, this is the first memorization study in the VLM space and that propose novel way to quantify this memorization. We also explore various mitigations and show which ones work and which do not, although note the fact that there could be other effective mitigations but were not explored due to the computational limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention the limitations about the number of possible mitigation strategies explored in the work in Seetiion 1] We also discuss the experimental limitations of our work due to the usage of automatic object detection tool in I|Appeedixx C Finally, in Section 6 we mention that work can not be directly applied to quantify memorization out-of-box pre-trained models as our work requires access a reference model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
22

---

judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There are no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: We discuss all the model hyper-parameter settings and how we obtain the dif- ferent data set splits for our experiments in detail in Appendix C along with discussion of what computational resources were used for the experiments. We also describe how we obtain the object annotations for all the images. We point out to which opensource repos- itory we use for training our models. We also describe which data sets were used in our experiments and we cite the relevant sources where possible. We made the memorization quantification code publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, deseribing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
23

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While the code publicly available, the data sets used cannot be made public due to the licensing and general nature of the data. We do cite all the relevant data sources wherever possible so the readers can refer to those sources. Moreover, we provide all the necessary parameter settings to replicate the results of the paper, possibly on some publicly available licensed data sets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: miips.cc...
pub Dublic//en gttp//uiddees/eee Cewuuddee/ooeeeeeeeee Codess//m ssionP olicy) for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. httpccccpppblllc pubppp/l///pcc quides. CodeSubm! ss onooolioii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Ys]]
Justification:
We explain all the necessary training details in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
24

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For our population-level memorization metrics, we report the mean and stan- dard deviation values over 100 repetitions of randomly sampling 10% of records with placement. However, since the standard deviation is less than 0.003, these enror bars are not visible the plots, such as Figure
Figure 5 and Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss all the computational resources and time taken to train the models in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pwws..e..u.p..p.ppp/aaaaa
Answer: [Yes]
Justification: We ensure NeurlPS Code of Ethics are followed, especially with the usage of data sets. For public data sets that contain human faces, such as filtered LAION, COCO and ImageNet, we blur all the faces prior to training the models or for any other analysis.
25

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We demonstrate that multi-modal models can memorize the objects present in the training images. While our intent is to make this risk more transparent to aid researchers get deeper understanding of the memorization issue inherent in representation learning models, an adversary could potentially use this to launch attacks on CLIP-style models. Although such an adversary would need non-trivial amount of background information for a successful attack. For instance, the adversary would need at least access to two models such that exactly one of the two is trained on a target image-text pair. They would also need access to the underlying training data for the target model. We discuss the approaches that are effective at mitigating this risk.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All our models are trained from scratch and we do not release these models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original code base of OpenCLIP and Detic that we use for training the models and object annotation respectively. We also include the licensing information for all the code and data sets we use in Apppndii A We included the same our code that is publicly released.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, [paperswil thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We released the code base which is well documented. We do not release models or data sets.
Guidelines:
The answer NA means that the paper does not relcase new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: Crowdsourcing not used.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were used so this is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (i applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "J82Spmk104", "reviewer": "Reviewer_GQcH", "summary": "The paper proposes a methodology to measure memorization in vision-language models (VLMs). These measurements are based on the fraction of ground-truth objects in an image that can be predicted from its text description. The authors also explore different mitigation strategies.", "questions": "- Have you considered combining multiple mitigation approaches? For instance, setting both weight decay and text masking rate to 0.3 could potentially yield complementary benefits.\n- Have you conducted experiments to compare the performance of models trained with different mitigation approaches on other tasks, such as retrieval or compositional reasoning benchmarks?", "limitations": "The authors have clearly addressed limitations in their paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The methodology is novel and useful in evaluating if the model is overfitted to the training data.\n- The paper shows extensive evaluation on both population and sample-level memorization. The ablation studies on mitigation methods are comprehensive.\n- The paper is well-structured and clearly written, explaining the methodology and results effectively.", "weaknesses": "See Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising important questions, which has helped us improve our paper.\n\n## Combining mitigations\n\n The main contribution of our work is a metric for evaluating memorization of VLMs. The ablation studies are done to validate the effect of key parameters and are not meant to be comprehensive. We do hope future work will adopt our metric to evaluate other mitigations that are either novel or combine several existing mitigation techniques.\n\n## Performance of mitigations on other benchmarks\n\n CLIP-style models have been shown to behave as bag-of-words by Yuksekgonul et al. (ICLR 2023) [1] and as such don’t perform well on compositional reasoning (ARO) tasks, but nevertheless we include the ARO benchmarks in Figure 1 in the attached pdf (please see the global rebuttal section for more details). As shown in the new results, our text masking strategy gives the best utility trade-offs.\n\n[1] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. \"When and why vision-language models behave like bags-of-words, and what to do about it?.\" In The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yz0iDZFSl9", "reviewer": "Reviewer_vpe4", "summary": "This paper explores the concept of training data memorization within vision-language models (VLMs). The authors introduce a method to measure the degree of memorization by analyzing the fraction of ground-truth objects in an image that can be predicted from its text description. The study reveals a significant level of memorization, which is evaluated at both sample and population levels. The findings indicate that OpenCLIP retains information about individual objects from the training images beyond what can be inferred from correlations or image captions. Additionally, the paper demonstrates that text randomization can reduce memorization with only a moderate impact on the model's performance in downstream tasks.", "questions": "- I understand that memorisation is an issue with generative models as they can regurgitate training data during inference time. But why is memorization important in non-generative models? What are the potential risks or drawbacks? I see that it could negatively impact its predictions, but is there something else?\n- Would reproducing the experiments with multiple pairs of fA and fB trained with different random seeds yield more robust results, or do the authors believe this to be unnecessary?\n- This paper tackles a challenging and important problem in the field of vision-language models. While the proposed method has limitations, its novelty and potential for future research make it a valuable contribution. This is why I am giving it a borderline accept. The main areas for improvement are enhancing the interpretability of the method (or convincing me that it is interpretable), including baseline comparisons (if possible), and improving the clarity of the presentation. With these improvements, the paper could make a stronger case for acceptance.\n\n[edit: Based on the rebuttal and the other reviews, I decided to increase my rating to 'weak accept'.]", "limitations": "Limitations are adressed in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- *Pioneering Approach*: The paper addresses a critical and complex issue by proposing a novel method to measure memorization in VLMs. This method, despite its imperfections, lays the groundwork for future research, offering a baseline for further refinement and extension. This work opens up new avenues for exploring memorization in multimodal settings, which is a non-trivial task.", "weaknesses": "- *Lack of Interpretability*: The major weakness of this work is the lack of interpretability of the proposed method. The proposed method is complex and not straightforward in its measurement of memorization. The reliance on an external object detector introduces additional biases and imperfections, complicating the interpretation of the results. For instance, the meaning of a PRG score of 0.17 is unclear, and the authors should provide guidance on interpreting these metrics.\n- *Absence of Baseline Comparisons*: The paper would benefit from comparisons with simple baselines to contextualize the proposed method's effectiveness. Although identifying suitable baselines is challenging, their inclusion could strengthen the validity of the findings.\n- *Clarity Issues*: Some aspects of the paper are difficult to understand. Figure 1 contains too much information and lacks a clear structure, making it hard to follow. Additionally, the results section is laborious to read due to the excessive use of acronyms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for raising important questions that has helped us better shape our paper. Below we respond to the key points raised in the review.\n\n## Lack of interpretability of metrics\n\nOur memorization metrics are built bottom-up from our notion of deja vu memorization for VLMs. As a motivating example, consider the use of CLIP in a cross-modal retrieval task, where images are retrieved from a web-scale database given text. We wish to capture the degree of surprise in the retrieval result when the model memorizes training captions, i.e. how many objects can the model recover beyond dataset-level correlation? This prompted us to use an object detector to provide ground-truth annotations for measuring the precision and recall of object recovery. At the sample level, we find the precision and recall for object retrieval for the target and reference models. A positive gap corresponds to the target model memorizing the training sample and the magnitude of the gap indicates the degree of memorization. At the higher level, we find the fraction of training samples where the sample-level precision and recall gap is positive and report the aggregate statistics as PPG and PRG respectively. \n\nWhile there will always be some bias when using object detectors, human or automated, this bias should not affect our evaluation when considering the gap between the two models. This is because the object detector is not trained on the same training set as the VLM, hence any incurred bias should be independent of the trained VLMs.\n\n## Absence of baselines\n\nWe note that this is the first work on VLM memorization and as such there are no baselines. The closest baseline could be the image-only deja vu attack of [1], but in their setting there is only one object per image, whereas here we have multiple objects so their method is not applicable here. For their method to work, we would need a background crop that does not contain the foreground object. In our data set, due to the presence of multiple objects, most of the images would not have any meaningful background crops.\n\n## Clarity of the paper\n\nWe will improve the readability of the paper, by simplifying the Figure 1 and clearly establishing the acronyms before the results section. Further, we will reiterate the acronyms (e.g. PPR and PRG) in the results section to remind the readers what those metrics mean and what to interpret from the metric values.\n\n## Other points raised\n\n- Regarding “why is memorization important in non-generative models?” VLMs such as CLIP are often used in cross-modal retrieval tasks, e.g. retrieve relevant images given text. This use case closely resembles the VLM deja vu test that we proposed: Given a training text caption, retrieve images from the public set. In other words, the deja vu score measures the degree of surprise in the retrieved images as a result of memorization. As a “hypothetical” example, suppose a training image contains a specific person in a specific place, and the caption is the person's name. If the model given this caption can predict the background location, then that might leak information. Beyond cross-modal retrieval, CLIP is also used in text-to-image generative models such as Stable Diffusion to provide text conditioning. Since our metric is meant to be more general, we did not test this use case explicitly, but we believe memorization in the CLIP model can manifest in surprise in generated images as well. \n\n- Regarding “Would reproducing the experiments with multiple pairs of fA and fB trained with different random seeds yield more robust results, or do the authors believe this to be unnecessary?”. Thank you for the suggestion! We ran an additional experiment where we repeated our experiments for 4 runs, each time using a different seed and training fA and fB from the scratch, and found the PPG and PRG metrics to be more or less the same. For predicting top-10 labels with 100 NNs, both the average PPG and PRG values across 4 runs are 0.066 ± 0.001. This result suggests that having only two models fA and fB can be sufficient for measuring memorization of VLMs.\n\n[1] Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. “Do ssl models have déjà vu? a case of unintended memorization in self-supervised learning.”, NeurIPS,  2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yqf1jOPZWL", "reviewer": "Reviewer_aCTU", "summary": "This paper investigates the issue of overfitting to pre-training data in Vision Language Models (VLMs) like CLIP. The authors conduct a comprehensive set of experiments focusing on text-to-image retrieval to evaluate this phenomenon. Their findings indicate that VLMs often memorize the data encountered during pre-training, which can impact the performance of downstream applications. However, the experiments also reveal that this problem diminishes as the scale of pre-training data increases.\n\n\nOverall I think this paper is important for the community since it tries to understand the limitation of popular multimodal foundational models like CLIP. It does have certain limitations but I think it’s a novel and thorough evaluation conducted at understanding CLIP through the lenses of pretraining data.", "questions": "1. Recent work has shown that the pretraining data of OpenCLIP models suffers from long-tail issues, I would like to to know if the authors evaluated this problem in a semantic sense as well? For eg concepts that are rare in the pretraining data, does the model tend to memorise these concepts more?\n\n\n[1] Parashar et al. (2024). The Neglected Tails in Vision-Language Models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*", "limitations": "1. Although the authors conduct experiments to evaluate the impact of data scale, such as scaling from 10M to 50M image-caption pairs, they do not perform orthogonal experiments to assess the impact of model size. Evaluating larger models, such as the ViT-L/14, would have been a natural next step and could have provided valuable insights.\n\n\n2. I disagree with the assertion that VLMs are typically pretrained on the scale of 10M image-caption pairs. For instance, the first CLIP model was pretrained on 400M image-caption pairs, and subsequent models have used even larger datasets. Although conducting experiments on such large datasets may be challenging, evaluating this problem at the more common pretraining data scale would have been more relevant, especially since the authors state that the problem diminishes as the data scale increases.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written and easy to follow. The metrics and methodology used is properly explained.\n2. Vision Language Models (VLMs) like CLIP have become crucial for downstream applications, including multimodal chatbots and text-to-image generation systems. Evaluating these models for their limitations is essential to improve their foundational capabilities. The authors assess these models by examining their tendency to memorize training data, a well-documented issue in traditional machine learning models, providing a strong motivation for this study.\n3. The experiments carried out make sense and the authors also evaluate few methods to mitigate the issue and present empirical results of each method tried.", "weaknesses": "1. While the authors evaluate four mitigation strategies, none of them effectively address the identified problem. It would have been beneficial to see a strategy that not only mitigates the problem but also enhances the model's utility.\n\n2. I believe that Figure 6 should be included in the main paper. Since the authors discuss it in detail, having it in the main paper would make it easier to follow and understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and for acknowledging the importance of our work. Below we respond to some of the points raised.\n\n## Long-tail issues of pretraining data\n\nSimilar to Parashar et al. (2024), our work also explores the memorization in pretraining data of openclip models, although our goals are orthogonal to theirs. While they show memorization of long-tails in terms of class labels, we explore long-tails in terms of image-text pairs where our long-tails are the cases where the text captions are not very descriptive of the images and thus are memorized more by the models.\n\n## Other points raised\n\n- Regarding “It would have been beneficial to see a strategy that not only mitigates the problem but also enhances the model's utility.” We find an inherent trade-off between privacy and utility, similar to the prior works [1-3] in membership inference and attribute inference literature, and thus it is difficult to find a mitigation that uniformly achieves both high utility and low memorization across all downstream tasks. Although it is possible to improve utility in some tasks, as shown in the new ARO benchmark results that we have included in Figure 1 in the attached pdf (please see the global rebuttal section for more details). Among all the mitigations that we explore, our text masking seems to achieve the best trade-offs, and even improves the accuracy on the COCO ordering task that requires compositional reasoning and ordering ability.\n\n- Regarding “believe that Figure 6 should be included in the main paper.”. Thank you for pointing this out, we will be happy to move the figure to the main paper.\n\n[1] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017.\n\n[2] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium, 2018.\n\n[3] Yunhui Long, Vincent Bindschaedler, and Carl A. Gunter. Towards measuring membership privacy. arXiv:1712.09136, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAX9wsatw0", "reviewer": "Reviewer_7Gny", "summary": "This work studies training data memorisation in Vision-Language Models (VLM). The paper focusses on contrastive learning with OpenCLIP, using a private Shutterstock dataset and a filtered LAION-50M, and evaluation on ImageNet. The paper proposes a method and metrics to measure déjà vu memorization, and in addition explores mitigation strategies. The paper concludes by stating it has demonstrated the presence of memorization.", "questions": "The main question concerns W1, which is, how this this work distinguish between memorisation and learning, and to what extent can the two be disentangled in interpreting the result?\n\nIf this point can be addressed I would switch to a more positive score, as I do think the work is interesting, but as long as it leaves open this alternative explanation I will recommend a reject.", "limitations": "Not applicable.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "This paper is well-written, explores an important issue, and proposes both new ways of measuring the issue and ways of mitigating it. Many experiments are run, addressing many of the questions this work raises.", "weaknesses": "**W1**: The main weakness of this work is that it does not sufficiently distinguish memorisation from learning. On line 162 the paper states: \"If no memorization occurs, models fA and fB should be interchangeable and hence this gap is zero.\", and while one would expect that two models trained on sufficiently large data would eventually converge - it is unlikely that the models being different before this point is purely due to memorisation. \n\nTwo models trained on the first 500 and the second 500 classes of ImageNet-1K will have significantly different performance on in-distribution versus out-of-distribution samples, yet, memorisation does not seem like the most likely explanation for this. \n\nSimilarly, with two models trained on the same dataset - with the exception of a single image-text pair, there may be a difference in performance. This paper attributes this to the memorisation of that image-text pair, however, I'd argue this could also be explained by learning - depending on the size of the original datasets and the 'importance' of the held-out data point. For instance, if the held-out data point is the only image-text pair to contain the 'cat' concept then the loss may be very high for this data point, and as such, it influences the network weights disproportionally. As the dataset size grows it becomes more likely that most concepts in the test set will have been seen during training, and thus the importance of any individual data point decreases.\n\nWhile memorisation may also be at play here, it is difficult to disentangle it from learning without a proper discussion about what distinguishes the two and how this may be seen in the experiments. In particular, if one takes the view that learning is compression of data points the boundary to memorisation becomes very blurry.\n\n**W2**: A second, but lesser weakness, is that the mitigation experiments do not explicitly address multi-modal nature of VLM - in section 5.4 it is discussed that images are already augmented, and an additional text masking strategy is proposed to match it. Yet, based on this parallel it would then seem logical that data augmentation of images also prevents memorisation - which is not explored. On the other hand, given how strongly the proposed memorisation testing strategy depends on the text prompts, it makes sense that the measured metrics drop - but this does not exclude image memorisation within VLM, which may be unaffected by this augmentation approach.\n\n**W3**: Two minor points: 1) Figure 1 is rather challenging to decipher before having thoroughly studied the text, and afterwards, the added value is minimal. Consider removing/updating this figure. 2) Line 233 discusses 'the adversary' which is not clear within the context of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a detailed review of our paper and for raising important questions. Below we would like to clarify some of the key points raised in the review.\n\n## Distinguishing between memorization and learning\n\nTo clarify, we believe a model can memorize and generalize (or learn) at the same time. This can happen at a subpopulation level, where the model memorizes rare concepts and generalizes to common concepts, or even at a sample level, where memorization is required for learning rare concepts as theorized in Feldman (2020) [1].\n\nOur notion of deja vu memorization is meant to go beyond this, and instead examine when a model that is trained on an image with a broad and generic caption, memorizes many small details about the associated image when given the caption. As a concrete example, in Figure 1 in the paper, the caption is “healthy fruits have vitamin C and D”. A well-generalized model will associate this caption with diverse fruits, which is exactly what model B does. In contrast, model A, which is trained on this image-caption pair, associates this caption with (almost) the same fruits that are in the image. In other words, we define deja vu memorization as what can be inferred about the training image from its caption beyond simple correlations, which can happen through both learning and memorization in the traditional sense.\n\n## Multi-modal defenses, such as impact of image augmentation, are not explored\n\nWhile we agree that having no image-augmentation and no text-augmentation would lead to worse memorization in theory, practical CLIP-style models have image augmentation by default (but not text augmentation) and hence we consider that as our baseline undefended models. We also note that the CLIP training objective interacts across modalities only via image–text alignment and as such does not specifically promote image–image alignment or text–text alignment. So the only way the model can memorize image features is via text features, thus we explore text augmentation which is not done by default in these CLIP-style models.\n\n## Other points raised\n\n- Regarding the comment “Consider removing/updating this figure.”, we will simplify Figure 1 to make it more interpretable.\n\t\n[1] Vitaly Feldman. \"Does learning require memorization? a short tale about a long tail.\" In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thoughtful comments and for raising important questions. Here we include some new benchmark results across different compositional reasoning tasks for the various models we test in our paper. We include the answers to other questions / points raised in the individual review's responses.

## Additional benchmarks

As per Reviewer GQcH’s request, we have included additional experiment results comparing the performance of various models on compositional reasoning (ARO) benchmarks of Yuksekgonul et al. (ICLR 2023) [1]. The results can be found in Figure 1 in the attached one-page pdf. We also show the impact of various mitigation strategies on these benchmarks. As shown in the new results, our text masking strategy gives the best utility trade-offs. Text masking even boosts performance on some reasoning tasks such as COCO ordering. We believe this could be due to the regularization effect of the mitigation that avoids overfitting on specific text tokens, thereby making the model less likely to behave like bag-of-words [1].

We will include these new results in the revision.

[1] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. "When and why vision-language models behave like bags-of-words, and what to do about it?." In The Eleventh International Conference on Learning Representations. 2023.

Meta-review: This work investigates memorization in CLIP-like encoder-encoder VLMs. The paper proposes a new method for measuring memorization, that relies on retrieving images using the text-embeddings and comparing the retrieved images with the query image. A high fraction of co-occurring objects that are not obvious from the caption indicates a high degree of memorization.

The work has received scores of 6,7,6,4 and while the reviewer-author discussion cleared out some concerns, and 7Gny did raise their score -- 7Gny has raised important points that should be discussed clearly as limitations in the final paper. The AC recommends acceptance with these caveats.
Decision: Accept (poster)