Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning
Authors: Casey Meehan
Florian Bordes
Pascal Vincent
Kamalika Chaudhuri
Chuan Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
privacy
data reconstruction
memorization
TL;DR: We demonstrate that Self-Supervised Models memorize detailed aspects of their training images, propose tests to quantify this memorization, and introduce a new diffusion-model based reconstruction attack.

Abstract: Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models -- which we refer to as déjà vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that déjà vu memorization is common to different SSL algorithms, is exacerbated by certain design choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of déjà vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies.

Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning
Casey Meehan Florian Bordes +2.33 Pascal Vincent Kamalika Chaudhurit Chuan Guot-3 UCSD, Milla, Université de Montréal. FFARR, Meta "Equal contribution, 'quual direction contribution cmeehan@eng, ucsd. edu, florian. bordes@umontreal.... ca, [pascal, kamal ika, chuanguo) @meta. com
Abstract
Self-supervised learning (SSL) algorithms can produce useful image representa- tions by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts individual training samples rather than learning semantically mean- ingful associations. In this work, we perform a systematic study of the unintended memorization ipage-specific information in SSL models-which we refer to déjà memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy reconstruct it. Furthermore, we show that déjà vi memorization common to different SSL algorithms, is exacerbated by certain hyperparameter choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of déjà memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies.
visually
Introduction
Self-supervised learning (SSL) 12 20 aims to learn general representations of content- rich data without explicit labels by solving a pretext task. In many recent works, such pretext tasks rely on joint-embedding architectures whereby randomized image augmentations are applied to create multiple views of a training sample, and the model is trained to produce similar representations for those views. When using cropping random image augmentation, the model learns associate objects or parts (including the background scenery) that co-occur in an image. However, doing so also arguably exposes the training data to higher privacy risk objects in training images be explicitly memorized by the SSL model. For example, if the training data contains the photos of individuals, the SSL model may learn associate the face of a person with their activity or physical location in the photo. This may allow an adversary to extract such information from the trained model for targeted individuals.
In this work, we aim to evaluate to what extent SSL models memorize the association of specific objects in training images or the association of objects and their specific backgrounds, and whether this memorization signal can be used to reconstruct the model's training samples. Our results demonstrate that SSL models memorize such associations beyond simple correlation. For instance, in Figure [ (left), we use the SSL representation of a training image crop containing only water and this enables to reconstruct the object in the foreground with remarkable specificity-ii this case black swan. By contrast, in Figure (right), when using the crop from the background of a test set image that the SSL model has not seen before, its representation only contains enough information
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training Image
Inage
Reconstruction using public data
Reconstruction
using public data
SSL
SSL
embedding
embedding
cropping
cropping
Figure 1: Left: Reconstruction of an SSL training image from a crop containing only the background. The SSL model memorizes the association of this specific patch of water (pink square) this specific foreground object black swan) in its embedding, which decode visualize the full training image. Right: The reconstruction technique fails on public test image that the SSL. model has not scen before.
infer, through correlation, that the foreground object was likely some kind of waterbird - but not the specific in the image.
Figure shows that SSL models suffer from the unintended memorization of images in their training data phenomenon refer to déjà memorization Beyond visualizing déjà memorization through data reconstruction, we also design a series of experiments to quantify the degree of memo- rization for different SSL algorithms, model architectures, training size, We observe that déja vu memorization exacerbated by the atypically large number of training epochs often recommended in SSL training, well as certain hyperparameters in the SSL training objective. Perhaps surprisingly, we show that déjä vu memorization occurs even when the training set is large-as large as half of ImageNet [I3 lad continually
even when standard techniques for evaluating learned representation quality (such as linear probing) do not suggest increased overfitting. Our work serves as the first systematic study of unintended memorization SSL models and motivates future work on understanding and preventing this behavior. Specifically, we:
Elucidate how SSL. representations memorize aspects individual training images, what we call déjà vu memorization;
Design a novel training data reconstruction pipeline for non-generative vision models. This is in contrast many prominent reconstruction algorithms like 7 8) which rely on the model itself to generate its own memorized samples and is not possible for SSL models or classifiers;
Propose metrics quantify the degree of déjà vu memorization committed by SSL model. This allows to observe how déjà vi changes with training epochs, dataset size, training criteria, model architecture and more.
2 Preliminaries and Related Work
Self-supervised learning (SSL) is machine learning paradigm that leverages unlabeled data to learn
representations. Many SSL. algorithms rely on joint-embedding architectures (e.g., SimCLR [I]]
Barlow Twins (33] VICReg 2 and Dino (10), which are trained to associate different augmented views of a given image. For example, in SimCLR, given a set of images A = (AI...., A,) and randomized augmentation function aug, the model is trained maximize the cosine similarity of draws of SSL (au)(A)) with each other and minimize their similarity with SSL(aug(A, for i & j. The augmentation function aug typically consists operations such cropping, horizontal flipping, and color transformations to create different views that preserve an image's 's semantic properties.
SSL representations. Once an SSL model is trained, its learned representation can be transferred to different downstream tasks. This is often done by extracting the representation an image from
'The French loanword déjă means 'already-seen' just image is seen and memorized in training.

---

the backbone mode and either training a linear probe on top of this representation or finetuning the backbone model with a task-specific head [3] It has been shown that SSL representations encode richer visual details about input images than supervised models do 4 However, from a privacy perspective, this may be cause for concern as the model also has more potential to overfit and memorize precise details about the training data compared to supervised learning. We show concretely that this privacy risk can indeed be realized by defining and measuring déjà vu memorization.
Privacy risks in MI. When a model is overfit on privacy-sensitive data, it memorizes specific information about its training examples, allowing an adversary with access to the model to learn private information [30; 16]. Privacy attacks in ML range from the simplest and best-studied membership
inference attacks 26 25 [24] to attribute inference [17 22 21 and data reconstruction 7 [9
attacks. In the former, the adversary only infers whether an individual participated in the training set. Our study of déjà vu memorization is most similar to the latter: we leverage SSL representations of the training image background to infer and reconstruct the foreground object. In another line of work in the NLP domain 6 7: when prompted with a context string present in the training data, a large language model is shown to generate the remainder of string at test time, revealing sensitive text like home addresses. This method was recently extended to extract memorized images from Stable Diffusion [8]. We exploit memorization in similar manner: given partial information about a training sample, the model prompted to reveal the rest of the sample In our case, however, since the SSL model is not generative, extraction is significantly harder and requires careful design.
3 Defining Déjà Vu Memorization
What is déjà vu memorization? At a high level, the objective of SSL is to learn general represen- tations of objects that occur in nature. This is often accomplished by associating different parts of an image with one another in the learned embedding. Returning to our example in Figure D given an image whose background contains patch of water, the model may learn that the foreground object is a water animal such as duck, pelican, otter, etc., by observing different images that contain water from the training set. We refer this type of learning as correlation: the association of objects that tend to co-occur in images from the training data distribution.
A natural question to ask is "Can the reconstruction of the black swan in Figure be reasoned as correlation?" The intuitive answer may be no, since the reconstructed image is qualitatively very similar the original image. However, this reasoning implicitly assumes that for a random image from the training data distribution containing a patch of water, the foreground object is unlikely to be a black swan, Mathematically, if we denote by P the training data distribution and A the image, then
Poorr == P (obp(objcct(A)) (A) = black swan crop(A) () = water)
is the probability of inferring that the foreground object is a black swan through correlation. This probability may be naturally high due to biases in the distribution P, e.g., if P contains no other water animal except for black swans. In fact, such correlations are often exploited to learn model for image inpainting with great success (7].
Despite this, we argue that reconstruction of the black swan in Figure [ is not due to correlation, but rather due to unintended memorization: the association of objects unique to a single training image. As we will show in the following sections, the example in Figure is not a rare success case and can be replicated across many training samples. More importantly, failure to reconstruct the foreground object in Figure I (right) on test images hints at inferring through correlation is unlikely to succeed-4e a fact that we verify quantitatively in Section 4.1 Motivated by this discussion, we give verbal definition of déjà memorization below, and design testing methodology quantify déjà vu memorization in Section 3.1
ZSSL methods often use a trick called guillotine regularization B. which decomposes the model into two parts: a backhone model and a projector consisting of a few fully-connected layers. Such trick is needed to handle the misalignment between the pretext SSL task and the downstream task.
"We recognize that is easier to find a context string that might have been in the training data a large language models that finding the exact pixels that constitutes a crop of a training image. However, this paper focus on revealing a memorization phenomena in SSL and does not aim to provide a complete picture of all the privacy risk that might entails.
3

---

Models/Traal sets
Inference pipeline
7arget sat/noda1
Refecence aat/aade:
Train image A
Find nearest neighbore
KNN
'black
Foirace AMc
Zublic set
SSLA
Get KMO subset of Lobelo
SSL
SSLg
black
bclack
black
Disjoint
SSL
Pabllkc
paziphary exopping
RCDM
enp(A)
KNN
RODD
Figure 2: Overview of testing methodology. Left: Data is split into target set A. reference set B and public X' that pairwise disjoint. AA and B are used train SSL models SSLA and SSLB in the manner. X' is used for KNN decoding embed crop A,) containing only the background, as well as the entire set and find the k-nearest neighbors of crop(A:) in X' in the embedding space. These KNN samples can be used directly to infer the foreground object (i.e. class label) in A; using KNN classifier, to be visualized directly or their embeddings can be averaged as input the trained RCDM visually reconstruct the image A,, For instance, the KNN visualization results in Figure [ (left) when given SSLA (crop((A))) and results Figure (right) when given SSLA (crop(B.) for an image B,
time. Right: Given training image A, A, we SSL.A
Definition: A model exhibits déjà vu memorization when retains information so specific an individual training image, that it enables recovery that does not contain them. The recovered aspect must be beyond what can be inferred using only correlations in the data distribution.
of aspects particular to that image given part
We intentionally kept the above definition broad enough to encompass different types of information that can be inferred about the training image, including but not restricted to object category, shape, color and position. For example, one can infer that the foreground object red given the back- ground patch with accuracy significantly beyond correlation, we consider this an instance of déjà wu memorization well. We mainly focus on object category to quantify déjà vu memorization in Section 4 since the ground truth label can be easily obtained. We consider other types of information more qualitatively in the visual reconstruction experiments in Section 5
Distinguishing memorization from correlation. When measuring déjă vu memorization, it is crucial to differentiate what the model associates through memorization and what it associates through correlation. Our testing methodology is based on the following intuitive definition.
Definition: If an SSL model associates two parts in training image, we say that it is due to correlation if other SSL models trained on similar dataset from without this image would likely make the same association. Otherwise, say that it is due to memorization.
Notably, such intuition forms the basis for differential privacy (DP; Dwork et al. [15] Dwork & Roth [4] the most widely accepted notion of privacy ML.
3.1 Testing Methodology for Measuring Déjà Vu Memorization
In this section, use the above intuition to measure the extent of déjà memorization in SSL. Figure gives an overview of our testing methodology.
Dataset splitting. We focus testing déjà memorization for SSL models trained the ImageNet dataset [3 Our test first splits the ImageNet training set into three independent and disjoint subsets A. B and X. The dataset A is called the target and B is called the reference set. The two datasets are used to train two separate SSL models, SSL and SSLB, called the target model and the reference model. Finally, the dataset set X' is used dataset to extract information from SSLA and SSLB. Our dataset splitting serves the purpose of
auxiliary public
Wee used the face-blurred version of ImageNet for privacy purposes.

---

distinguishing memorization from correlation in the following manner. Given a sample A, E A, if our test returns the same result on SSLA and S then it is likely due correlation because A, is not a training sample for SSL.B. Otherwise, because A and B are drawn from the same underlying distribution, our test must have inferred some information unique to A, due to memorization, Thus, by comparing the difference in the test results for S and SSI.B, we can measure the degree of déjà vu memorization
Extracting foreground and background crops. Our testing methodology aims at measuring what can be inferred about the foreground object in an ImageNet sample given a background crop. This is made possible because ImageNet provides bounding box annotations for subset of its training images-aroune 150K out of 1.3M samples. We split these annotated images equally between A and B. Given an annotated image A,, we treat everything inside the bounding box the foreground object associated with the image label, denoted object(A.). We take the largest possible crop that does not intersect with any bounding box as the background crop (or periphery crop), denoted crop (A:: L
KNN-based test design. Joint-embedding SSL approaches encourage the embeddings of random crops of a training image A; E AA to be similar. Intuitively, if the model exhibits déjà vu memorization, it is reasonable to expect that the embedding of crop(A;) is similar to that of object(A,) since both crops are from the same training image. In other words, SSLA (crop(A,)) encodes information about object(A.) that cannot be inferred through correlation. However, decoding such information is challenging as these approaches do not learn a decoder associated with the encoder SSLA-
Here, we leverage the public set X' to decode the information contained in crop( A,) about object (ooject(A,) A,). More specifically, we map images in X' to their embeddings using SSLA and extract the k-nearest- neighbor (KNN) subset of SSLA (crop(A.)) in X. We can then decode the information contained in cA))(A.) one of two ways:
Label inference: Since A' is a subset of ImageNet, each embedding in the KNN subset is associated with class label. If crop( (A,) encodes information about the foreground object, its embedding will be close to samples in X' that have the same class label (i.e., foreground object category). We can then use KNN classifier infer the foreground object in A; given crop(A.).
Visualization: Since we have access to a KNN subset associated to a given (rop(A:), we can visualize directly the images associated to this subset. Then, we can infer through visualizing what is common within this subset, what information can be retrieved for this single crop. In addition, to simplify the visualization pipeline and to map directly a given crop representation to an image, we train an RCDM A conditional generative on X' to decode SSLA embeddings into images. The RCDM reconstruction can recover qualitative aspects of an image remarkably well, such as recovering object color or spatial orientation using its SSL embedding. Given the KNN subset, we average their SSL embeddings and use the RCDM model to visually reconstruct A,-
In Section $ we focus on quantitatively measuring déjà vu memorization with label inference, and then use the KNN to visualize déjà vu memorization in Section 5
4 Quantifying Déjà Vu Memorization
We apply our testing methodology to quantify specific form of déjà vu memorization: inferring the foreground object (class label) given a crop of the background.
Extracting model embeddings. We test déjà vu memorization on variety of popular SSL algo- rithms, with a focus on VICReg (2). These algorithms produce two embeddings given an input image: backbone embedding and a projector embedding that is derived by applying small fully-connected network on top of the backbone embedding. Unless otherwise noted, all SSL embeddings refer to the projector embedding. To understand whether déjà vu memorization particular SSL, we also evaluate embeddings produced by a supervised model CLFA trained on A. We apply the same set of image augmentations as those used in SSL and train CLFA using the cross-entropy loss to predict ground truth labels.
$ 'See Appendix A.2.1 for details on how the dataset splits are generated.
'We also present another heuristic in Appendix A.8 which takes corner crop as the background crop, allowing our test to be run without bounding box annotations.
5

---

Identifying the most memorized samples. Prior works have shown that certain training samples
can be identified as more prone to memorization than others [T;; 28 29]. Similarly, we provide a heuristic to identify the most memorized samples in our label inference test using confidence of the KNN prediction. Given a periphery crop, crop(A.), let KNN A (grop((.)) C X denote its k-nearest neighbors in the embedding space of SSL From this KNN subset we can obtain: (1) KNN ()))(A the vector of class probabilities (normalized counts) induced by the KNN sub- set, and (2) KN (crop(::)), the negative entropy of the probability vector KNN confidence of the KNN prediction. When entropy is low, the neighbors agree on the class of A; and hence confidence is high. We can sort the confidence score KNN (cr)p(A,)) across samples A; in decreasing order to identify the most confidently predicted samples, which likely correspond to the most memorized samples when A, E A.
A (crop(A.)),
4.1 Population-level Memorization
Our first measure of déjà vu memorization is population-level lahel inference accuracy: What is the average label inference accuracy over a subset of SSL. training images given their periph- ery crops? To understand how much of this accuracy due to SSLA'S déjà vu memorization, we compare with a correlation baseline using the reference model: KNN 's label inference ac- curacy on images A, E A. In principle, this inference accuracy should be significantly above chance level (1/1000 for ImageNet) because the periphery crop may be highly indicative of the foreground object through correlation, e.g., the periphery crop is a basketball player then the foreground object is likely a basketball. Figure B compares the accuracy of KNNA to that of KNN when inferring the labels of images in A; E L using crop(A.). Results are shown for VICReg and the supervised model; trends for other models are shown in Appendix [A.5 For both VICReg and supervised models, inferring the class of crop(A,) using KNN B (dashed line) through correlation achieves a reasonable accuracy that is significantly above chance level. However, for VICReg, the inference accuracy using A (solid red line) is significantly higher, and the accuracy gap between KNN A and KNN & indicates the degree of déjà vu memorization. We highlight two observations:
1.0
Label Inference accuracy
Target model (KNN))
0.8
Reference model (KNNg)
SSL (VICReg)
0.6
The accuracy gap of VICReg is significantly larger than that of the supervised model. This is espe- cially notable when accounting for the fact that the supervised model is trained to associate randomly augmented crops of images with their ground truth labels. In contrast, VICReg has no label access during training but the embedding of periphery crop can still encode the image label.
Supervised
0.4
0.2
0.0
20
40
60
80
100
Top % most confident examples
Figure 3: Accuracy of label inference using the target model (trained on A) vs. the reference model (trained on B) the top % most confident exam- ples A, E AA using only crop(A.). For VICReg, there is large accuracy gap between the two mod- cls, indicating a significant degree of déjà vu mem- orization.
For VICReg, inference accuracy the 1% most confident examples is nearly 95%, which shows that our simple confidence heuristic can effectively identify the most memorized samples. This result suggests that an adversary can use this heuristic to identify vulnerable training samples to launch a more focused privacy attack.
The déjà vu score. The curves of Figure D show memorization across confidence values for single training scenario. To study how memorization changes with different hyperparamters, we extract a single value from these curves: the déjà vu score at confidence level p. In Figure 3 this is the gap between the solid red (or gray) and dashed red (or gray) where confidence (æ-axis) equal p%. In other words, given the periphery crops of set A, KNN a and KNN n separately select and label their top p% most confident examples, and we report the difference in their accuracy. The déjà vu score captures both the degree of memorization by the accuracy gap and the ability to identify memorized examples by the confidence level. If the score is 10% for = 33%. KNN has 10%
"The sets A and B are exchangeable, and in practice we repeat this test on images from B using SSLg as the target model and SSLA as the reference model, and average the two sets of results.

---

60
30
30
064 W. Scoe 2016 Cort
Unear Prabec Goo
Dee v. Scow 20% Cprt.
8 0 3
totel
25
Share of total
25
8
Prdde
20
20
15-
15
5
20
10
Dija sa Score
10
D40 Si
10
0 100 200 300 400 500
200 400 Number of epochs
030 1000
120
200
300
400
500
200 400 600 8001000 Number epochs
Training set size [thousends]
Training set size (thousands)
(a) déja epochs
(b) déja vs. train set size
déjá epochs
(b) déjä train set size
Figure 4: Effect of training epochs and train set size on déjä vu score (red) comparison with linear probe accuracy train-test gap (dark blue) for VICReg. Left: déjà score increases with training epochs. indicating growing memorization while the linear probe baseline decreases significantly. Right: déjā score stays roughly constant with training set size while linear probe gap shrinks significantly, suggesting that memo- rization may be problematic even for large datasets.
Figure 5: Partition of samples A, E A into the four categories: unassociated (not shown), memorized, mis- represented and correlated for VICReg. The memo- rized samples those whose labels are predicted by KNNA but by KNNg- -occpppoypppyyyye a significantly larger share of the training than the misrepresented samples- those predicted by KNN but not KNN by chance.
higher accuracy on its most confident third of A than KNN n does its most confident third. In the following, we set 20%, approximately the largest gap for VICReg (red lines) in Figure 3
Comparison with the linear probe train-test gap. A standard method for measuring SSL perfor- mance is to train linear classifier what we call 'linear probe' on its embeddings and compute its performance on a held out test set. From learning theory standpoint, one might expect the linear probe's 's train-tes: accuracy gap to be indicative of memorization: the more model overfits, the larger the difference between train set and test set accuracy. However, as seen Figure 4 the linear probe gap (dark blue) fails to reveal memorization captured by the déjà score (red) g
4.2 Sample-level Memorization
The déjà vu score shows, on average, how much better an adversary can select and classify images when using the target model trained them. This average images have their label successfully recovered by KNN A but not by KNN g. In other words, how many images exposed by virtue of being in training set A: risk notion foundational differential privacy. To better quantif'y what fraction of the dataset is at risk, we perform sample-leve analysis by fixing a sample A; E A and observing the label inference result KNN vs. KNN B. To this end, we partition samples A, E AA based on the result of label inference into four distinct categories: Unassociated label inferred with neither KNN; Memorized label inferred only with KNN A: Misrepresented label inferred only with KNN B; Correlated label inferred with both KNNs.
does not tell us how many individual
Intuitively, unassociated samples are ones where the embedding of crop(A+) does not encode information about the label. Correlated samples are ones where the label can be inferred from cA))(A.) using correlation, e-g., inferring the foreground object baskethall given crop showing basketball player. Ideally, the misrepresented set should be empty but contains a small portion of examples due to chance. Déjā vu memorization occurs for memorized samples where the embedding of SSLB does not encode the label but the embedding of SSLA does. To measure the pervasiveness of déjà vu memorization, we compare the size of the memorized and misrepresented sets. Figure 5 shows how the four categories of examples change with number of training epochs and training set size. The unassociated set is not shown since the total share adds up to one. The misrepresented set remains under 5% and roughly unchanged across all settings, consistent with our explanation that it is due to chance. In comparison. VICReg's memorized set surpasses 15% at 1000 epochs. Considering that up to 5% of these memorized examples could also be due to chance, we conclude that at least 10% % of VICReg's training set is déjà vu memorized.
See section o for further discussion of the déjà score trends of Figure
7

---

example
crop
correlated images E,A
example
crop of memorized images É A
neighbors E,''
reconstructions
neighbors E.
recons traciintruttion
S5T
SSL
(a) A comelated dam example
(b) A memorized dam example
Figure 6: Correlated and Memorized examples from the dam class. Both SSLA and SSL are SimCLR models. Left: The periphery crop (pink square) contains concrete structure that is often present in images of dams. Consequently, the KNN can infer the foreground object using representations from both SSLA and SSLB through this correlation. Right: The periphery crop only contains a patch of water. The embedding produced by SSL only contains enough information to infer that the foreground object is related to water, as reflected by its KNN set. In contrast, the embedding produced by S memorizes the association of this patch of water with dam and the KNN select images of dams.
European badgees
caighbora E.t
recoestructions
American badgwrs
naighbora
reconatructicom
ML
S5L
(a) Memorized European badgers
(b) Memorized American badgers
Figure 7: Visualization of déjà vo memorization beyond class label. Both SSL. and SSL are VICReg models. The four images shown belong to the memorized set of SSL. from the badger class. KNN images using embeddings from SSL, can reveal not only the correct class label, but also the specific badger species: European (left) and American (right). Such information does not appear to be memorized by the reference model SSLB.
5 Visualizing Déjà Vu Memorization
Beyond enabling label inference using periphery crop, we show that déjà vu memorization allows the SSL model to encode other forms of information about training image. Namely, leverage an external public dataset X' and use to find the nearest neighborhoods in this public dataset given a training periphery crop. We aim to answer the following two questions: (1) Can we visualize the distinction between correlation and déjà vu memorization? (2) What foreground object details can be extracted from the SSL model beyond class label?
Public image retrieval pipeline Following the pipeline in Figure we use the projector embedding find the KNN subset for the periphery crop, crop(A:), and visualize the images belonging this KNN subset.
RCDM pipeline. RCDM is a conditional generative model that is trained on the backbone embed- ding of images X; E X' to generate an image that resembles X,. At test time, following the pipeline in Figure 2 we first use the projector embedding to find the KNN subset for the periphery crop, crop( Ax), and then average their backbone embeddings input to the RCDM model. Then, RCDM decodes this representation to visualize its content.
Visualizing Correlation vs. Memorization. F shows examples of dams from the correlated set (left) and the memorized set (right) as defined in Section 4.2) along with the associated KNN set. In Figure 6a the periphery crop is represented by the pink square, which contains concrete structure attached to the dam's 's main structure. As a result, both SSLA and SSLB produce embeddings of
8

---

D4j No Stors 20% cant.
Acc.
Dée su Score 20% Cont
1
Innrer Bradee Sei Arc.
Dajutu Score 20% Carf.
78
13
Lifeet Probe Vol
8.
Poiress Sor
+ylideas
Supervised
Chitoia
FW
8.9
55.16111
Ane PM
-0
Ryal [S]]
8.0
$4.3/9944
SimCLR
10.0
44,315111
D40 Score
Uheee Frode VV AXC
Livar PrESA VLL
V.. Tey
1/5.7
Invoriance, 100
55
100
par
200
300
400
Barlow 1,03
VIUReg
30.3
33.2
40.3155.2
33,751..4
(a) Loss hyper-parameterr
(b) Guillotine regularization
(a) deja vir vs, capacity
(b) déja vw (DV) vs. Criterion
Figure 8: Effect of two kinds of hyper-parament parameters on VICReg memorization. Left: déjà vu score (red) versus the invariance loss parameter. X, used in the VICReg cri- terion (100k dataset). Larger À significatlly reduces déjà vN memorization with minimal effect on linear prohe val- idation performance (green). déjä vu is recommended in the original paper. Right: déjà
Figure 9: Effect of model architecture and criterion on déjà vu memorization. Left: déjà vw score with VICReg for resnet (purple) and vision transformer (green) architectures eters. As expected, memorization grows with larger model capacity. This trend is more pronounced for convolutional (resnet) than transformer (VIT) archi- tectures. Right:
number of model param-
25 (near maximum
versus projector gorllooiiee regular- 3 from projector backhone. Removing the projector can significantly reduce déjà vu In Appendix A.7 wc show that the backhone still can memorize.
Comparison of déjà vu 20% conf. and ImageNet linear probe validation accuracy (P: using projector embeddings, using backbonc embeddings) for various SSL criteria.
ization
crop(A,) whose KNN set in X' consist of dams, i.e., there is a correlation between the concrete structure crop(A,) and the foreground dam. In Figure 6b the periphery crop only contains a patch of water, which does not strongly correlate with dams in the ImageNet distribution. Evidently, the reference model SSLg embeds crop(A,) close that of other objects commonly found in water, such as sea turtle and submarine. In contrast, the KNN set according to SSLA all contain dams despite the vast number of alternative possibilities within the ImageNet classes which highlight memorization in SSL.A between this specific patch of water and the dam.
Visualizing Memorization Beyond Class Label. 7 four examples of badgers from the memorized set. all four images, the periphery crop (pink square) does not contain any indication that the foreground object is badger. Despite this, the KNN set using S consistently produce images of badgers, while the same does not hold for SSLB. More interestingly, the KNN using SSL in Figure 7a all contain European badgers, while reconstructions in Figure 7b all contain American badgers, accurately reflecting the species badger present in the respective training images. Since ImageNet-I IK does not differentiate between these two species of badgers, reconstructions show that SSL models can memorize information that is highly specific to a training sample beyond its class labe
6 Mitigation of déjà vu memorization
We cannot yet make claims on why déjà vu occurs so strongly for some SSL training settings and not for others. To gain some intuition for future work, present additional observations that shed light on which parameters have the most salient impact on déjà vu memorization.
Déjà vu memorization worsens by increasing number of training epochs. Figure 4a shows how déjà vu memorization changes with number of training epochs for VICReg. The training set size is fixed to 300K samples. From 250 to 1000 epochs, the déjà vu score (red curve) grows threefold: from reflected by the linear probe
under 10% over 30%. Remarkably, this trend in memorization gap (dark blue), which only changes by a few percent beyond 250 epochs.
Training set size has minimal effect on déjà vu memorization. Figure 4b shows how déjà vu memorization responds to the model's 's training set size. The number of training epochs fixed to 1000. Interestingly, training set size appears to have almost no influence on the déja vu score (red line), indicating that memorization is equally prevalent with a 100K dataset and a 500K dataset. This result suggests that déjà memorization may be detectable even for large datasets. Meanwhile, the
See Appendix A.l for additional visualization experiments.

---

standard linear probe train-test accuracy gap declines by more than half as the dataset size grows, failing to represent the memorization quantified by our test.
Training loss hyper-parameter has a strong effect. Loss syper-parameters like VICReg's invari- ance coefficient (Figure (a) or SimCLR's temperature parameter (Appendix Figure [4a] significantly impact déjä va with minimal impact on the linear probe validation accuracy.
Some SSL criteria promote stronger déjä vu memorization. Table 9b demonstrates that the degree of memorization varies widely for different training criteria. VICReg and Bar- low Twins have the highest déjā vu scores while SimCLR and Byol have the lowest. With the exception of Byol, all SSL models have more déjà vu memorization than the super- vised model. Interestingly, different criteria can lead to similar linear probe validation ac- curacy and very different degrees of déjă Note that low degrees déjà vu can still risk training image reconstruction, as exemplified by the SimCLR re- constructions in Figures and
seen with SimCLR and Barlow Twins.
Déjà Score 20% Conf.
Larger models have increased déjă memorization. Figure Da validates the common intuition that lower capac- ity architectures (Resnet (1s33) 18/34) result in less memorization than their high capacity counterparts (Resnet50/ 101). We see the same trend for vision transformers well.
60
30
Finetuned ned al AcC.
50
25
40
20
15
30
Guillotine regularization can help reduce déjà vu mem- orization. Previous experiments were done using the projector embedding. In Figure [b]] we present how Guil- lotine regularization [3] (removing final layers in trained SSL model) impacts déjà vu with VICReg T0 Using the backbone embedding instead of the projector embedding seems to be the most straightforward way to mitigate déjà vu memorization. However, as demonstrated in Appendix A.7 backbone representation with low déjā score can still be leveraged to reconstruct some of the training im- ages.
10
20
Dejà Score
10
Unear Probe Val. Acc.
0
10 Finetuning epoch
20
Figure 10: Déjå vu when fine-tuning pretrained VICReg model for 20 epochs. We observe that by fine-tuning we significantly increase the classification performances.
A little bit of fine-tuning might help reduce memorization A common strategy in SSL is to fine-tune the model to solve the downstream task. In Figure I0] we show how the Dejavu score changes when fine-tuning a pretrained VICReg model. This pretrained model was trained on the set A for 1000 epochs and fine-tuned on classification task on the set A for 20 epochs (which can be seen in the x axis on the figure). Interestingly, the DejaVu score decreases significantly in the first finetuning epochs while the validation accuracy is increasing. However after 5 epochs, the Deja Vu score is increasing and after 20 epochs become almost as high at the original value before fine-tuning. This behavior indicates that even fine-tuning might not help in reducing Deja Vu memorization.
7 Conclusion
We defined and analyzed déjà vu memorization, a notion of unintended memorization of partial information in image data. As shown in Sections and 5 SSL models can largely exhibit déja vu memorization on their training data, and this memorization signal can be extracted to infer or visualize image-specific information. Since SSL models are becoming increasingly widespread as foundation models for image data, negative consequences of déjà vu memorization can have profound downstream impact and thus deserves further attention. Future work should focus on understanding how déjà vu emerges in the training of SSL models and why methods like Byol are much more robust déjà vu than VICReg and Barlow Twins. In addition, trying characterize which data points are the most at risk of déjä vu could be crucial to get a better understanding on this phenomenon.
Further experiments available in Appendix A.7
10

---

References
[1] Balle, B., Cherubin, G., and Hayes, J. Reconstructing training data with informed adversarics. arXiv preprint wiXi::221.008455 2022.
[2] Bardes, A., Ponce, and LeCun, Y. Vicreg: Variance- aritane-invarrance--vvvaaaaan regulariza- tion for self-supervised learning. In The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nel, 2022. URL
https://openreviiet nott/oorummddxxxx66666222332
[3] Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vincent. P. Guillotine regularization: Improving deep networks generalization by removing their head, 2022. URL https://axxiv org/abs/22066 13378
13378
[4] Bordes, F., Balestriero, R., and Vincent, High fidelity visualization of what your self- supervised representation knows about. Transactions on Machine Learning Research, 2022. URL httttt://oppereeieeeeeeee/////ee
net/ http://wwwnet/ooommiieeeeee forumiidduurWbbVVjjl
[5] Bordes, F., Balestriero, R., and Vincent, Towards democratizing joint-embedding self- supervised learning, 2023. URL
/203. 01986
[6] Carlini, N., Liu, C., Erlingsson, Ú. Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
[7] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T. Song, D., Erlingsson, U., et Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
[8] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, Extracting training data from diffusion models. arXiv preprint arXiv:2301. 13188, 2023.
[9] Caron, M., Misra, I., Mairal, J., Goyal, Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[10] Caron, M., Touvron, H., Misra, I., Jegou, H., and Joulin, J. M. P. B. A. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[11] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In /CML, 2020.
[12] Chen, X. and He, K. Exploring simple siamese representation learning. In CVPR. 2020.
[13] Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on computer vision and pullern recognition, pp. 255. leee, 2009.
[14] Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Theoretical Computer Science, 93-4):211144) 2013.
[15] Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography, pp. 265-284. Springer, 2006.
[16] Feldman, V. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
[17] Fredrikson, M., Lantz, E.,, Jha, S., Lin, S., Page, D., and Ristenpart, T. Privacy in pharmacoge- netics: An (End-to-Endl Endd case study personalized warfarin dosing, 23rd USENIX Security Symposium (USENIX Security 14), pp. 17-32, 2014.
[18] Grill, J.-B., Strub, F., Altché, F. Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukeuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent: A new approach to self-supervised learning. In NewrIPS, 2020.
11

---

[19] Guo, C., Karrer, B., Chaudhuri, K., and van der Maaten, L. Bounding training data reconstrue- tion in private (deep) learning, arXiv preprint arXiv:2201112383, 2022.
[20] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
[21] Jayaraman, B. and Evans, D. Are attribute inference attacks just imputation? arXiv preprint arXiv:229.00129, 2022.
[22] Mchnaz, S., Dibbo, S. V., Kabir, E., Li, N., and Bertino, E. Are your sensitive attributes private? novel model inversion attribute inference attacks on classification models. arXiv preprint arXiv:2201.0933 2022.
[23] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L.,, Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang. L.., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Curran Associates Inc., Red Hook, NY, USA, 2019.
[24] Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference Machine Learning, pp. 2558-5577. PMLR, 2019.
[25] Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., and Backes, M. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint 1806.08006.00
2018.
[26] Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
[27] Ulyanov, D., Vedaldi, A., and Lempitsky, V. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454, 2018.
[28] Watson, L., Guo, C., Cormode, G., and Sablayrolles, A. On the importance of difficulty calibration in membership inference attacks. arXiv preprint 1..0420 2021.
[29] Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. arXiv preprint S.ki::2111.096111 2021.
[30] Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. 2018 IEFE 31st computer security foundations symposium (CSF), pp. 268-282. IEEF, 2018.
[31] You, Y., Gitman, L. and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arviv
1 1708. 1708.03888, 03888, 2017.
[32] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5505-5514, 2018.
[33] Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arxiv:2103.032.0. 2021.
[34] Zeng, H. and Cui, X. Simelrt: A simple framework for contrastive learning of rumor tracking. Eng. Appl. Artif. Intell., 110: 104757, 2022. doi: hw0.1/..enaappai22222..oo
URL
https::/doii http:////o.ooo///... 1016/ engappai. wwwpaapppa..2.......... 2022. 104757
12

---

A Appendix
A.1 Additional reconstruction examples
The two reconstruction experiments of Section 5 are each exemplified within one class. However, we see strong reconstructions using SSLA in several classes, and similar experimental results. To demonstrate this, we repeat the experiment of Section using the yellow garden spider class and the experiment of using the aireraft carrier class.
example
crop correlated imagea E.A
example
of memorized images E.A
NOW
W
neighbors E.Y'
tructinruttion
neighbors E\'
reconstruction
SSL
SSL.
*
SSLg
SSL
yellow garden spider examples
(b)
yellow garden spider examples
Figure Visualizing the distinction between déjà vu memorization correlation in the yellow ganden spider class. Left, we see the periphery crops of the ten 'most correlated' images: those where both KNN and KNN have high confidence. Seven of these crops clearly depict stabilimentum: the signature zig-zag web pattern sewn by spiders of the argiope genus, thus revealing the concealed spider by correlation. Right, the periphery crops the 'most memorized" images: those that the highest confidence discrepancy between KNN a (high confidence) and KNN (low confidence). Nearly all of these crops show generic blurred views of the background with no evidence of the foreground spider. Below, we show the public set nearest neighbors of the pink highlighted crop. We see that the target model (SSL4) can be used to find back the yellow garden spider spider both the memorized and correlated cases. The reference model (SSLg) ) can only be used find back this type spider the correlated case.
Selection of Memorized and Correlated Images: The images of Figure and II were chosen methodically as follows.
selected deterministically using label inference accuracy and KNN confidence score. The 10 most correlated images are those images the correlated set (both models infer label correctly) of A with the highest confidence agreement between models SSLA and SSL.B- To measure confidence agreement we take the minimum confidence of the models. The 10 most memorized images those images in the memorized set (only target model infers the label correctly) of A with the highest confidence difference between models SSL and SSLB.
Image selection: The 20 images of Figures 6 and
Class selection: find classes with a high degree of déjà vu classes were sorted by the label inference accuracy gap between the target and reference model. We selected the class based handful of criteria. First, we prioritized classes without images of human faces, thereby removing classes like 'basketball', 'bobsled', 'train station', and even 'tench' which is fish often depicted in the hands of a fisherman. Second, we prioritized classes that include at least ten images with a high confidence difference between the target and reference models ('most memorized" images described above) and at least ten images with high confidence agreement ( ('most correlated' images described above). This led to the dam and yellow garden spider classes.
13

---

naighbors oe
seconatruetion
Ship cA front
neighbore e.
zecone tront
aide
(a) extracting side ship from VICReg
(b) extracting freat of from VICReg
Figure 12: Visualization of déjà memorization beyond class lahel. Both SST. and SSL VICReg models. The images shown belong to the memorized set SSL, visualization using embeddings from SSLA can reveal not only the correct class label, but also the orientation of the ship: the side of the ship (left) and the front of the ship (right) given only generic crop of the background sky and/or water. Such information does not appear to be memorized by the reference model SSLg.
the aircraft carrier class. KNNN
Selection of Soatud-Labbe--nffeeemee Images: methodically follows.
The images of Figure and 12 were chosen
The four images of Figures 7 and are selected using KNN confidence score, and, necessarily, hand picked selection for unlabeled features. Within a given class, we look at the top 40 images with highest target model KNN confidence scores. We then filter through these images to identify distinguishable feature like different species within the same class or different object positions within the same class. This step necessary because we are looking for features that are not labeled by ImageNet. We then choose two of these 40 with one feature (e.g. American badger) and two with the alternative feature (e.g. European badger).
Image selection:
Class selection: To find classes with a high degree of déjà vu classes were sorted by the target model's top-40 KNN confidence values within each class. As in the memorization vs. correlation experiment, we prioritized classes without images of human faces.
14

---

A.2 Details on the experimental setup
A.2.1 Details on dataset splits
There are 1,281,167 training images in the ImageNet dataset. Within these images, only 456,567 of them have bounding boxes annotations (which are needed to compute the Deja Vu score). The private set A and B are sampled from these 456,567 bounding boxes annotated images in such a way that set A and B are disjoint.
If we remove the 456,567 bounding boxes annotated images from the 1,281,167 training images, we get 824,600 remaining images without annotations which never overlap with A or B. From this set of 800K images, we took 500k images as our public set X. So now, we have three non overlapping sets A, B, and X. Then, if we remove the 500K public set images from the 824,600 images without annotations, leaves us with 324,600 images that are neither in A. B or X. For simplicity, let call this set of remaining 324,600 images the set C. Then, we have split the entire ImageNet training set into four non-overlapping splits called A, B, and X.
When running our experiments with a small number of training images, we only use the set A to train A and the set B to train SSL and then use the set X as public set for evaluation. However, to run larger scale experiments, we use as additional training data for SSLA and SSL the images sampled from the set C. Here, SSLA will still be trained images from set C. The same goes for SSL. which will still be trained on the set B but augmented with images from the C. As such, some images sampled from C to train SSLA or train SSL might overlap. However, this is not an issue since the evaluation is done using only images from the bounding boxes annotated set A and set B which are never overlapping.
set A but it will be augmented with
To identify memorization, our tests only attempt to infer the labels of the unique examples A and B that differentiate the two private sets. The periphery crop, crop(A,), is computed as the largest possible crop that does not intersect with the foreground object bounding box. In some instances the largest periphery crop is small, and not high enough resolution to get meaningful embedding. To circumvent this, we only run the test on bounding box examples where the periphery crop at least 100 x 100 pixels.
Each size of training set, 100k 500k, includes an equal number of examples per class in both sets A and 13. The total bounding box annotated examples of each class evenly divided between A and B. The remaining examples in each class are the examples from C. We reiterate that the bounding box examples in set A are unique to set A, and thus can only be memorized by SSL
The disjoint public set, X, contains ground truth labels but no bounding-box annotations. The size and content of X remains fixed for all tests.
A.2.2 Details on the training setup
Model Training: We use PyTorch (23 with FFCV-SSL [3]. All models are trained for 1000 epochs with model checkpoints taken at 50, 100, 250, 500, 750, and 1000 epochs. We note that 1000 epochs is used in the original papers of both VICReg and SimCLR. All sweeps of epochs use the 300k dataset. All sweeps of datasets use the final, 1000 epoch checkpoint. We use batch size of 1024, and LARS optimizer (31) for all SSL. models. All models use Resnet101 for the backbone. As seen in Appendix A.6 Resnet50 backbone results in déjà vu consistent with that of Resnet101.
VICReg Training: VICReg is trained with the 3-layer fully connected projector used in the original The invariance, variance, and covariance parameters
paper with layer dimensions 8192-8192-812 are set to X = 25, μ 25, v= = 1, respectively, which are used in the original paper (2). The LARS base learning rate is set 0.2, and weight decay is set to le-6.
SimCLR Training: SimCLR is trained with the 2-layer fully connected projector used in the original paper with layer dimensions 2048-256. The temperature parameter is set to T 0.15. The LARS base learning rate is set to 0.3, and weight decay is set to le-6.
Supervised Training: Unlike the SSL models, the supervised model is trained with label access using cross-entropy loss. To keep architectures as similar as possible, the supervised model also uses Resnet101 backbone and the same projector VICReg. A final batchnorm, ReLU, and linear layer
15

---

is added to bring the 8192 dimension projector output to 1000-way classification activations. We use these activations as the supervised model's projector embedding. The supervised model uses the LARS optimizer with learning rate 0.2.
A.2.3 Details on the evaluation setup
For each test, we build two KNN's: one using the target model, SSL (or CLF A), and one using the reference model SSL. (or CLFB). As depicted in Figure each KNN is built using the projector embeddings of all images in the public set X' as the neighbor set. When testing for memorization
KNN:
image A; E A, we first embed crop(A, using SSLA, and find its K = 100 L2 nearest neighbors within the SSLA embeddings of X. See section A.44 for a discussion on selection of K. We then take the majority vote of the neighbors' labels to determine the class of A, This entire pipleline is repeated using reference model SSLg and its KNN to compute reference model accuracy.
In practice, all of our quantitative tests are repeated once with SSL as the target model (recovering labels of images in set A) and again with SSLB the target model (recovering labels of images in set B). All results shown are the average of these two tests. Throughout the paper, we describe SSLA as the target model and SSL.B as the reference model for ease of exposition.
The RCDM is trained on face-blurred version of ImageNet [1]] and is used to decode the SSL backbone embedding of an image back into an approximation of the original image. All RCDMs are trained on the public set of images X' used for the KNN. A separate RCDM must be trained for each SSL model, since each model has a unique mapping from image space to embedding space.
RCDM:
At inference time, the RCDM is used reconstruct the foreground object given only the periphery cropping. To produce this reconstruciton, the RCDM needs an approximation of the backbone embedding of the original image. The backbone of image A; is approximated by 1) computing crop embedding SSI. PSt. PSI. (crop(A:)), 2) finding the five public set nearest neighbors of the crop embedding, and 3) averaging the five nearest neighbors' backbone embeddings. In practice, these public set nearest neighbors are often a very good approximation of the original image, capturing aspects like object class, position, subspecies, etc..
16

---

A.3
Sample-level memorization
VICReg
Supervised
ViCReg
Supervised
35
35
35
Correlated
Correlated
Share of tftatll
25
Share of tatel
25
Misrepresented
25
25
Misrepresented
Memorized
Memorized
15
15
15
Share
15
10
10
10
10
0
400 8000
1000
400 Training set size (thousands)
Number of epochs
Number of epochs
Training set size (thousards)
(a) Categories of training samples vs. mumber of epochs
(h) Categories of training samples vs. training set size
Figure 13: Partition of samples A; E A into the four categories: unassociated (not shown), memorized, misrepresented and correlated. The memorized samples-on whose labels are predicted KNN but by KNN occupy significantly larger share for VICReg compared to supervised model, indicating that sample-level déjà vir memorization is more prevalent VICReg.
0.5
1.0
0.5
1.0
Correlated
Correlated
0.4
Misrepresented
acturacy
0.8
0.4
Misrepresented Memorized
Linear probe accuracy
0.8
Memorized
Share of total
0.3
0.2
0.4
Share
0.2
0.4
0.2
Train Validation
0.1-
0.2
Train Validation
0.4 0.6 0.8
1.0
0.2 0.4 0.6 0.8 1.0
20
60 Invariance
40
100
20
40 60
80
100
Temperature
Temperature
Invariance
(a) SimCLR
(b) VICReg
Figure 14: Effect of SSL hyperparameter on déjå vu memorization. The left plot of Figures 14aand and 14b show the size of the memorized set as function of the temperature parameter for SimCLR and invariance parameter for VICReg, respectively. Déjā vu memorization is the highest within narrow band of hyperparamcters. and one can mitigate against déjà vu memorization by selecting hyperparameters outside of this band. Doing so has negligible effect on the quality of SSL embeddings as indicated by the linear probe accuracy on ImageNet validation set.
Many SSL algorithms contain hyperparameters that control how similar the embeddings different views should be in the training objective. We show that these hyperparameters directly affect déjà vu memorization. Figure 14 shows the size of the memorized set for SimCLR (left) and ViCReg (right) as function of their respective hyperparameters, and X. We observe that the memorized set is largest within a relatively narrow band of hyperparameter values, indicating strong déjà vu memorization. By selecting hyperparameters outside this band, déjà memorization sharply decreases while the linear probe validation accuracy on ImageNet remains roughly the same.
17

---

A.4
Selection of K for KNN
In this section, we describe the impact of on the KNN label inference accuracy.
Label Inference Accuracy
1.00
uecuracy on top 20% conf.
correlated forgotten memerized
0.75
accuracy on all ciamples
Share total
0.4
ref. model correlation bascline
0.50
0.2
0.25
0.00
25
50
75
100
Number of neighbors:
Number of neighbors: K
(a) VICReg, Accuracy
(b) VICReg, Share of memorized examples
Figure 15: Impact of K on label inference accuracy for target and reference models. Left: the population-level label inference accuracy experiment of Section[4.]]o VICReg vs. K. Right: the individualized memorization relatively robust to choice of K
of Section 4.2 on VICReg K. In both cases, that
beyond K 50.
Figure 15 above shows how the tests of Section change with number of public set nearest neighbors K used make label inferences. Both tests are relatively robust to any choice K. Results are shown VICReg trained for Ik epochs the 300k dataset. We that any choice of K greater than 50 and less than the number of examples per class (300, this case) appears to have good performance. Since our smallest dataset has 100 images per class, we chose to set K 100 for all experiments.
18

---

A.5 Effect of SSL criteria
We repeat the quantitative memorization tests of Section 4 on different models: VICReg 2 Barlow- Twins[33], Dino [I), SimCLR [34] and supervised model in Figure T6 We observe differences between SSL training criteria with respect déjà vu memorization. The easy ones to attack are VICReg and Barlow Twins whereas SimCLR and Byol are more robust to these attacks. While the degree memorization appears be reduced for SimCLR compared with VICReg, still stronger than the supervised baseline.
1.8d Mitoranee Accumay
meatco dar
ELd
VicRig
Label Inientce
1000
250
500 spachs
190
100
0g
259
533 spochs
790
1000
Accirny
coochs
c0o2lls
L-brl bafrecoe Acciney
way monelt
Barlon-
Teiins
Sim
Store
250
500
1000
250
9:0
1400
350
190
1000
spacks
spochs
spochs
epocts
0.5
Suparsited
Dino
250
opodls
250
opachs
5001
1400
2.40
spocts
2.00
200
1000
opecls
Figure 16: Comparison of déjà memorization for VICReg, Barlow Twins, Dino, Byol, SimCLR, and described in Section 4 We are showing déjà vu vs. number of training epochs.
supervised model. All We see that SimCLR (center row) shows déjä vu than VICReg, yet marginally more than the supervised model. Even with this reduced degree of memorization, we are able to produce detailed reconstructions of training set images, as shown in Figures 6 and
19

---

A.6 Effect of Model Architecture and Complexity
Results shown in the main paper use Resnet101 for the model backbone. To understand the relation- ship between déjà vu and overparameterization,, we compare with the smaller Resnet50 and Resnet 18 Figure I7 Overall, we find that increasing the number of parameters of the model leads to higher degree of déjà memorization. The same trend holds when using Vision Transformers (VIT-Tiny, -Small, -Base, and -Lar with patch size of 16) of various sizes as the SSI. backbone, instead of Resnet. This highlights that déjà memorization is not unique to convolution architectures.
13300 Inteence Accumy
VIT-Tiny
Labe Iecoorece
0.1
270
500
150
1003
0g
251
500
759
1000
spachs
epochs
Latel Iefovesee Nocurncy
Labed Infortwe Accusayy
Tin-Sinall
calipi
Stuee (f
morele
Freaee
Stare
700
1003
0.0
250
2:01
130
300
230
200
750
1900
253
340
750
1000
spadits
epochs
0.5
Eabo Intereecy Accuron)
VIT-Base
Stare tetal
Share of socl
RNNO
750
1000
000
250
300
750
1000
spachs
epochs
Lobel inferoce AcCuRC
Lidd Moorrn AuStEa
Store of Quaal
ViF-Large
RN1101
252
3.0
250
300
200
1009
<pucchs
specia
opodl
Figure 17: Comparison VICReg déjà vu memorization for different architectures and model sizes. On the left, we present deja vu memorization using VIT architectures (from vit- vit-tiny in the first row to vit-base in the last row). On the right, we use Resnet based architectures (from resnet18 in the first row to resnet101 in the last row). All tests are described in Section 4 with the plots showing déja Reducing model complexity from Resnet101 to Resnet18 from Vit-Large Vit-tiny has significant impact on the degree of memorization.
number of training epochs.
20

---

A.7 The impact of Guillotine Regularization on Deja Vu
In our experiments, we show déjà vu using the projector representation. The SSL loss directly incentivizes the projector representation to be invariant to random crops of a particular image. As such, we expect the projector to be the most overfit and produce the strongest déja vur Here, we study whether earlier representations between the projector and backbone exhibit less déjà memorization. This phenomenon "guillotine regularization" has recently been studied from the perspective of generalization in Bordes et al. [3] Here, we study it from the perspective of memorization.
To show how guillotine regularization impacts déjà vu, we repeat the tests of Section O each layer of the VICReg projector: the 2048-dimensin backbone (layer 0) up to the projector output (layer 3). We evaluate whether memorization indeed reduced for the more regularized layers between the projector output and the backbone.
Label Inference Accuracy
0.6
accuracy top 20% conf.
correlated
accuracy all oxamples
forgetten
Share of total
0.2
model correlation
momorized
0.4
0.1
0.2
0.0
0
2
3
2
Layer
Layer
population level accuracy
(b) share of memorized examples
Figure 18: déjà memorization tests Section fare evaluated at each level of VICReg projector. see that déjà vu is significantly stronger closer to the projector output and nearly zero near the backbone. Interestingly, most memorization appears to occur in the final two layers of VICReg.
layer from backbone (0) projector output (3). The memorization
Figure shows how guillotine regularization significantly reduces the degree of memorization in VICReg. The vast majority of VICReg's déjà appears occur in the final two layers of the projector (2.3): in earlier layers (0,1), the label inference accuracy the target model and reference model are comparable. This suggests that - like the hyperparameter selection of Section guillotine regularization can also significantly mitigate déjà vu memorization. In the following, we extend this result SimCLR and supervised models by measuring the degree of déjà in the backbone (layer 0) versus training epochs and dataset size.
21

---

Comparison of déjà vu in projector and backbone vs. epochs and dataset size Since the backbone is mostly used at inference time, we now evaluate how much déjà vu exists the backbone representation for VICReg and SimCLR. We repeat the tests of Section versus training epochs and train set size.
Laave Inforen Acceacy
FLS
WiCteg WiCttgg Projector
Shrre of Scbal
112
opechs
(thorcomds))
ViCReg Backbone
Labd Inforece Nestras
0.5
Stare E16
cemviaton
Share ooona
Infremne
0.
2.0
500
1000
341
2100
spochs
Caln size (thotroncis)
Inain set size (fousands)
(a) VICReg
Label Infermece Accummy
SimGLR Projector
uonted
Shure of tacal
Inlereve
0.2
0.3
250
533
750
1000
503
750
1000
200
spochs
spects
(housanda))
Iraid side (thonsands)
SIMCLA Bacibinn
Interence Necuray
Jbb Infoeetee Auturasy
1.0
of UOal
Sepana
of
0.2
0.0
100
300 (thousands)
200
200
500
(thowatdds
(b) SimCLR
Figure 19: Accuracy of label inference VICReg and SimCLR using projector and backbone representations. First two columns: Effect training epochs on memorization for each representation. Last two columns: Effect of training set size on memorization for each representation. In contrast with VICReg. the déja vu memorization detected in SimCLR's projector and backbone representations is quite similar. While SimCLR's projector memorization appears weaker than that of VICReg, its hackhone memorization is markedly stronger. This kind easily explained objective reduce the hias of the network with respect the training task. Since SimCI.' projector has fewer layers than VICReg's, s, the impact of Guillotine Regularization is less salient.
byproduct of Guillotine Regularization 3. i.c. removing layers close to the
Figure 19 shows that, indeed, déjà vu is significantly reduced in the backbone representation. For SimCLR, however, we see that backbone memorization comparable with projector memorization. In light of the Guillotine regularization results above, this makes some sense since SimCLR uses fewer layers in its projector. Given that we were able to generate accurate reconstructions with now evaluate whether we can produce accurate
the SimCLR projector (see Figures II and reconstructions of training examples using the SimCLR backbone alone.
22

---

Reconstructions using SimCLR Backbone Only: The above label inference results show that the SimCLR backbone exhibits similar degree of déjà va memorization the projector does. To evaluate the risk of such memorization, we repeat the reconstruction experiment of Section S on the dam class using the SimCLR backbone instead of its projector.
Dam E A
neighbors E\'
reconstructions
SSL
SSL -
(a) First memorized dam example
Dam E A
neighbors EX'
reconstructions
SSL
SSL
(b) Second memorized dam example
Figure 20: Instances of déjà vu memorization by the SimCI R backbone representation. Here, the backhone embedding of the crop is used instead of the projector embedding on the same training images used in Figure 6 Interestingly,
that déjà memorization still present in the SimCLR backbone representation. Here, the nearest neighbor set recovers dam given uninformative crop access, we are able to reconstruct images in set A using SSL and are unable using SSL
running water. Even without projector
Figure 20 demonstrates that we are able to reconstruct training set images using the SimCLR backbone alone. This indicates that déjà vi memorization can be leveraged make detailed inferences about training set images without any access to the projector. As such, withholding the projector for model release may be a strong enough mitigation against déjà vu memorization.
23

---

A.8 Detecting Déjà vu without Bounding Box Annotations
The memorization tests presented critically depend on bounding box annotations in order to separate the foreground object from the periphery crop. Since such annotations are often not available, we propose heuristic test that simply uses the lower left comerr of an image as surrogate for the periphery crop. Since foreground objects tend to be near the center of the image, the corner crop usually excludes the foreground object and does not require bounding box annotation.
Figure 21 demonstrates that this heuristic test can successfully capture the trends of the original tests (seen Figure without access bounding box annotations. However, compared Figure 16 the heuristic tends to slightly underestimate the degree of memorization. This is likely due to the fact that some corner crops partially include the foreground object, thus enabling the KNN to successfully recover the label with the reference model where it would have failed with proper periphery crop that excludes the foreground object.
Lubel Infermnc Accunay
VICReg
hnforree
2
Share
200 epochs
750
1000
250
200
3.0 (10)
200
300
0.0
201
331
300
opochs
train Set size (loseonds)
Irain sct size (Choteords)
Labed Inferrnce Arqurnyy
Libel Informnce Secumes
2010
Engein
SimCLR
0.1
spochs
500
400 (housaanch
0.0 110
train size (thousonds)
20C
30)
-01
50
Lobel Infonmeer Accumyy
Libd bitoreste Aucusscy
Supervised
mald contiaiie boafley
soma
Share of total
wanwoord
04
0.0
250
500
750
1000
110
200
300
-001
specht
opocts
train set sizo (thousands)
(housendss
Figure 21: Déjá va memorization using a simple corner crop instead of the periphery crop extracted using bounding box annotations. While the hcuristic overall underestimates the degree of déjà vu it roughly follows the same trends versus dataset size and training epochs. This is crucial, since it allows us to estimate déjà vu without to bounding box annotations.
24


Reviews:
[{"review_id": "yjoP2sjrz9", "reviewer": "Reviewer_2FNq", "summary": "This work investigates to what degree self-supervised models exhibit memorization of particular samples in the training set. The authors demonstrate that a simple crop of an image, only comprised of the background, is enough for an adversary to learn the associated class label of the foreground object. Moreover, a conditional generative model can be trained, allowing the adversary to reproduce a very similar-looking image compared to the original. Such memorization capacity can lead to privacy issues and the authors demonstrate that such behaviour occurs in a broad range of models, while at the same time, this effect is not visible in common metrics such as probing accuracy.\n", "questions": "What happens if you perform the nearest neighbour class inference on the training set (excluding the image itself)? This would circumwent the issue outlined above, but of course defeats the practicality of the method. Still seems like an interesting experiment though. \n", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well written and builds upon the simple but very interesting idea of memorization in self-supervised models. Such an investigation seems timely, given the similar recent efforts in large language models. \n2. The experiments are very well designed, clearly disentangling the notion of correlation (i.e. actually learning something generalisable about the sample) and pure memorization (i.e. only memorizing the specifics of the sample). The splitting of the training data into two sets and training two separate models is a very neat way to measure this effect, which would otherwise remain very difficult to disentangle.\n3. It is very interesting that supervised and self-supervised methods behave very differently in terms of memorization. In general, studying the difference in the two learning approaches is very worthwile and timely.\n", "weaknesses": "1. I struggle to really see the privacy concern here. How likely is it really for someone to have access to only a crop of an image, while not being able to access the full image? The authors present the example of an image of a face of a person, which could then be potentially used to infer where that person is. But again, in what situation do I have access to the face of an image without the rest of it? As the authors also demonstrate, such an image has to be present in the training data, it is not enough to simply use another photo of a person and then the model reveals another image of the same person. ﻿I also believe that this scenario is different from the language model case, where simply “guessing” a good query as input for the model to complete seems to be more realistic due to the smaller search space and potential knowledge of similar queries. I hope the authors can comment on this.\n2. This is more minor but while I really like the experimental setup, I’m still not sure if memorization and generalization are completely disentangled. Assume that you have a model that clearly memorizes images but also performs very poorly at the downstream task, i.e. its linear probe is around random guessing. Due to the complete lack of generalization, it will be tough to establish memorization since the nearest neighbours will all share no correlation. So from a pure theoretical perspective, this seems a bit unsatisfying. Of course in practice, no generalization is usually not the case, so from a practical viewpoint this makes complete sense.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to hear that you found our work timely, and our experiments interesting. Please let us know if our responses below answer your primary questions. \n\n**I struggle to really see the privacy concern here. How likely is it really for someone to have access to only a crop of an image, while not being able to access the full image?**\n\n Thank you for your comment. \n\nWe would like to emphasize that this paper centers on finding systematic memorization of training data by SSL models. We do suggest that memorization could imply privacy risks, but a rigorous study of practical adversaries would require a complete followup work, and is not a primary focus of the work (the discussion around privacy is only a few lines in the paper). If there are any sentences in the paper that seemed to misconstrue this, please let us know and we will be glad to clarify the contributions.\n\nWe acknowledge that the claims made in the paragraph on the privacy risks we presented in the related work (and also the sentence in the introduction about associating a face to an activity) might have been too strong without providing further privacy analysis. You are right that the comparison with NLP is not as straightforward since it is easier to find a sequence of words that might have been in the training set than a sequence of pixels. We are planning to update the paper accordingly to ensure that no misleading assertions or comparisons.\n\nTo provide a simple example of disclosure as to why we made the analogy with NLP earlier, consider an individual’s public social media profile picture that is a crop of their face taken from a photo of them on vacation. If the full (private) image was used in SSL model training, the public profile face crop could be used to query the model and recover information about the remainder of the image, disclosing the vacation location. However, we recognize now that this potential setup might be more restricted than the ones used in NLP.\n\nWe hope that modifying the misleading sentences around the privacy implications will solve your concerns.\n\n**As the authors also demonstrate, such an image has to be present in the training data, it is not enough to simply use another photo of a person and then the model reveals another image of the same person.**\n\nA core component of studying memorization is showing how models retain excessive information about training set examples. It would be interesting to see how this changes for slight variations of training set examples, and we will consider this in follow up work. \n\n**I’m still not sure if memorization and generalization are completely disentangled [...if the model’s] linear probe is around random guessing [...] it will be tough to establish memorization since the nearest neighbours will all share no correlation. What happens if you perform the nearest neighbour class inference on the training set (excluding the image itself)?**  \n\nThere are many ways a model could memorize its training data. We do not claim that Deja Vu completely disentangles generalization from memorization in Self-Supervised Learning. \nWe claim that one modality of memorization in SSL can be revealed with our Deja Vu method. \nAs you mentioned, a model could probably store a lot of information about training examples in its weights without producing meaningful embeddings. In this work, we are focusing on model’s embeddings for which we suppose they are meaningful (at least enough to give good KNN performance). Providing a more theoretical perspective on memorization might also be challenging since there might be many ways to define memorization in a DNN. In addition, defining generalization in a Self-Supervised setting is not straightforward (are you considering generalization on a pretext task or a downstream task, if so which ones ?)\n\nConcerning using the training set $A$ instead of the public set $X$, it would indeed defeat the purpose of removing training set access from the test. Moreover, using the training set for decoding would not quite disentangle memorization from performance, since it still requires the encoding of similar training set images (e.g. swans) to be close together. However, we still ran the experiment as requested and found a small drop in the DejaVu Score. We used our VICReg model which shown a DejaVu score of 33.2 and replace the public set $X$ by the set $A$ when doing KNN (from which we excluded) the image itself (for each case), and got a DejaVu score of 29.7 (the differences could be explain mostly because the set $A$ is much smaller than the set $X$). We think that adding such results in the paper might add more confusion than needed, however if you think this is an important point that should be in the paper, we would gladly further discuss it.\n\n\n"}, {"review_id": "zuCTQ827Yw", "reviewer": "Reviewer_9ALm", "summary": "Self-Supervised Learning algorithms provide a label-free alternative for representation learning and is widely used in vision, and language. In this work, the authors try to understand the information content of representations learned by SSL algorithms through the lens of memorization. In particular, they define a \"deja-vu\" memorization notion that measures how much a given sample from the training set can be reconstructed from a pretrained model. \n\nThe authors focus on joint-embedding-based approaches, exploring the extent of such memorization during pretraining by evaluating label-inference accuracy on peripheral crops of a subset of training images ($A$). A \"deja-vu\" score at p% confidence measures the gap in accuracy between a target model (trained on $A$) and a reference model (trained on a different subset $B$). The gap essentially measures the extent to which exposing particular samples during training makes it easier to predict the label from partial background information.\n\nBuilding on these metrics, the authors present interesting results comparing different SSL algorithms under different training configurations. Notably, long pretraining increases the risk of memorization, while the size of the dataset doesn't is as consequential (Fig 4/5). With a sequence of ablations, the authors establish empirical findings and provide design recommendations that minimize deja-vu memorization while maintaining high performance. ", "questions": "1. Although the authors primarily concentrate on joint-embedding-based techniques, it is probable that reconstruction-based methods are also susceptible to this type of memorization. To enable a fair comparison across the SSL algorithm family, it would be useful to include a baseline such as MAE.nclude a baseline such as MAE.\n2. The focus of the authors is on the linear probe as a potential measure of representation quality. However, they have shown that this metric is inadequate in capturing the full extent of deja-vu memorization. It would be valuable to determine if this inconsistency still exists even when the model is fine-tuned, which involves adapting the backbone as well, in an end-to-end approach.", "limitations": "The paper explores the privacy-risk profile of SSL algorithms through the lens of training-set memorization. The authors are strongly encouraged to discuss their proposed algorithm's limitations (including assumptions and failure modes).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Some strengths of the work are outlined below:  \n**Originality**: The authors examine the representation quality in SSL from a privacy perspective. The work focuses on train-time characteristics and memorization for non-generative SSL algorithms, in line with similar works in language/generative models. The authors propose a novel leave-one-out mechanism for evaluating the extent of memorization (compared to correlation) of specific training-set examples across SSL algorithms.\n\n**Significance**: Understanding the information content of representations learned by black-box SSL algorithms is important as these algorithms become widely used in safety-critical applications. In particular, the paper makes a solid case for understanding the privacy risk induced by long-pretraining routines in existing SSL algorithms. \n\n**Quality**: The work is well-motivated and grounded in real-world challenging questions related to measuring the quality of representations learned in SSL algorithms.  \n\n**Clarity**: The paper is well written, with definitions and pedagogical examples that motivate the core insights of the work. The authors include clear, well-labeled figures and plots accompanying their observations.\n\n**Reproducibility**: All experiments are performed on public datasets and pretrained models and sufficient details to reproduce the core results of the paper. The authors are encouraged to release the accompanying code at their earliest convenience. ", "weaknesses": "Some weaknesses of the work are outlined below:\n\n**Explainability**: The authors have shared some interesting empirical findings about the effectiveness of SSL algorithms in representing data and their ability to recall examples from the training set from partial side information. Although the authors demonstrate that established methods like guillotine regularization reduce the risk of memorization, the current version of the study could benefit from more discussion on the underlying mechanisms behind these mitigation techniques.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving our paper a detailed read, as is clear from you thorough summary! We are happy to hear the strengths you saw in our work, and try to address some of your primary questions below. Please let us know if you have any follow up questions. \n\n**Explainability: [...] Although the authors demonstrate that established methods like guillotine regularization reduce the risk of memorization, the current version of the study could benefit from more discussion on the underlying mechanisms behind these mitigation techniques.**\n\nIn this work, we wanted to alert the research community about the Deja Vu phenomenon and show some situations in which it is amplified when doing SSL training. It is not clear yet what are the exact underlying mechanisms inducing this phenomena. We are keen to study this in follow up work. \n\nThe guillotine regularization authors have shown that most of the over-fitting with respect to the downstream task lies in the last layers of the network. In consequence, when training an SSL model, the last layers will be heavily biased (because of the training criteria) to match the representations of different patches into the same embedding. But this is not the case for the intermediate layers of the network which are free to assign different embeddings to different crops of a given image (which might also be better for generalization). \n\nHowever, there are still some mysteries: we do not know why the BYOL criterion is so much more robust with respect to Dejavu memorization in comparison to Barlow Twins or VICReg. Clearly, what causes Deja Vu is subtle and complex. We believe further work would be needed to fully assess the extent of this phenomena and its origin.\n\n**It would be useful to include a baseline such as MAE**\n\nThis work focuses on introducing a new modality of memorization in SSL (Deja Vu memorization) and demonstrating it empirically in SSL joint-embedding models. We considered including MAE in our experiments, but it is unclear how to make a fair comparison with those models. MAE is trained with a decoder which generates images in pixel space. The decoder is directly trained to reconstruct training examples given patches, so there is no need to introduce another reconstruction method.\n\nThis being the case, measuring Deja Vu with MAE would require an entirely different methodology than our KNN-based decoder scheme. Hence for this work, we focus on joint-embedding methods. We will modify the introduction to clarify our focus on joint embedding models for the scope of this work. \n\n**It would be valuable to determine if this inconsistency still exists even when the model is fine-tuned, which involves adapting the backbone as well, in an end-to-end approach.**\n\nThe extent of Deja Vu memorization in a fine-tuning setting is an interesting question. We ran experiments with fine-tuned models. In this setup, we used a pretrained VICReg and fine-tune it for 20 epochs (For both SSL_A and SSL_B). We present the results in the table below:\n\n| \\ Epoch | 0 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |\n|---|---|---|---|--|---|---|---|--|---|---|---|\n| Train Acc. | 60.0 | 67.2 | 73.5 | 78.0 | 81.5 | 84.9 | 87.1 | 88.9 | 90.2 | 91.5 | 91.8\n| KNN Acc. | 40 | 33 | 35.8 | 36.6 | 37.4 | 37.5 | 37.9 | 38.1 | 38.2 | 38.4 | 38.7\n| DejaVu Score | 33.2 | 23.4 | 24.7 | 25.9 | 26.5 | 27.3 | 28 | 28.4 | 31.2 | 29.9 | 32.6\n\n---\n\nOur tests show that the training accuracy increase significantly however the KNN accuracy does not change much. It is also interesting to note that just immediately after the few first epoch of fine-tuning the deja vu score decrease significantly but it does go up again later in the training. SSL joint-embedding models are often used in a frozen setting (in contrast to MAE type of model who are often fine-tuned), however, we will gladly add these fine-tuning results in the appendix. Please let us know, if there is anything else you would want to see in this setup."}, {"review_id": "uUgDpiQuXO", "reviewer": "Reviewer_H9Sq", "summary": "This paper shows how self-supervised learning (SSL) models can memorize information about their training data in ways that are not intended. SSL is a technique for learning representations of unlabeled data by training a model to solve a \"pretext\" task; for example, we might take training images A, B, and C and crop them in different ways, producing sets of patches (A1,...,An), (B1,...,Bn), and (C1,...,Cn). The algorithm tries to find an embedding of these patches that makes Ai similar to Aj, while minimizing the similarity between Ai and Bj. These embeddings are used for a variety of downstream tasks.\n\nThis paper shows that the representations of training patches can contain surprising information about the training data. In the headlining example, the authors take an image of a black swan from the training data, extract a water-only patch, and produce an embedding of this patch. Then a generative model, which (ideally? see below) has no access to the training data, produces an image from this embedding: it contains a (different?) black swan. The water-only patch contained no information about swan, so the SSL model \"remembered\" what was in the image the patch came from.\n\nThe trick fails when the same patch is run through an SSL model that was trained on different data: it happens only because the patch was seen in the training data.\n\nThe paper has two methods for detecting such memorization. The first predicts the class label given the patch's representation. The second, mentioned above, visually inspects images generated from the representation. In both cases the classification/generation is done by a model trained on a different data set. The label-prediction method demonstrates quantitatively that the representation contains information about the label, while the visual inspection reveals finer details: the paper contains a great example of how SSL models can learn and memorize the distinction between European and American badgers, even though both share an ImageNet label. ", "questions": "Section 3.1 says the datasets A, B, and B are disjoint. Appendix A.2.1 says that A and B overlap. Furthermore, as mentioned above, my understanding is that X is a face-blurred version of ImageNet, and thus X might contain (possibly blurred) examples from A and B. Can you clarify this?\n\nCould you lay out what the \"adversary,\" the person conducting the privacy attack, has access to? ", "limitations": "The authors adequately address the limitations of their work.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "If the authors resolve my dataset question below, I am happy to call this a very clean and original demonstration on a significant and relavant topic. It is definitely a paper to generate good discussion.\n\nThe paper performs extensive experiments, showing how the results change as they turn various knobs. The quantitative and qualitative techniques complement each other nicely.\n\nThe work lays out clear directions for future work.\n", "weaknesses": "As I understand from Appendices A.2.1 and A.2.3, \"private\" sets A and B come from ImageNet 1k, which is a subset of ImageNet. The \"public\" set X is a face-blurred version of ImageNet. This appears to contradict the experimental setup of Section 3.1 and, if it is the case, means I am not sure how to interpret the results. I could be mistaken and am open to raising my score. EDIT: the rebuttal adequately addressed these concerns.\n\nI was initially confused the use of the word \"reconstruction.\" For example, Figure 1 talks about \"reconstruction of an SSL training image,\" but the generated image looks quite different from the training image. It seems you use the word differently than recent work does [1-5], where it means producing an example that is pixel-by-pixel, or feature-by-feature, similar to the input. I suggest using a different word or clarifying the issue earlier in the paper.\n\nI think synthetic-data experiments might strengthen the paper. They might nail down specific aspects of this phenomenon. One might better probe the types of memorized information. (Of course, easier suggested than done.)\n\nI find the related work section adequate, but I think Section 3, \"Defining Deja Vu Memorization,\" would benefit from clearer connections between prior work and the presented definitions. In particular, the definition at line 111 has similarities to the notion of memorization used in [2], which also attempts to capture the idea of memorization of specific information about individual examples, beyond what one can infer from the data distribution. The definition for testing methodology, at line 130, is similar to the definition of memorization used in [6,7].\n\n[1] Balle, Borja, Giovanni Cherubin, and Jamie Hayes. \"Reconstructing training data with informed adversaries.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[2] Brown, Gavin, et al. \"When is memorization of irrelevant training data necessary for high-accuracy learning?.\" Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing. 2021.\n\n[3] Haim, Niv, et al. \"Reconstructing training data from trained neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 22911-22924.\n\n[4] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[5] Hayes, Jamie, Saeed Mahloujifar, and Borja Balle. \"Bounding Training Data Reconstruction in DP-SGD.\" arXiv preprint arXiv:2302.07225 (2023).\n\n[6] Feldman, Vitaly. \"Does learning require memorization? a short tale about a long tail.\" Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020.\n\n[7] Feldman, Vitaly, and Chiyuan Zhang. \"What neural networks memorize and why: Discovering the long tail via influence estimation.\" Advances in Neural Information Processing Systems 33 (2020): 2881-2891.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the related work references!\n\nWe indeed can answer your questions pertaining to the dataset splits. We will edit the paper to make these points more precise. Please do let us know if we can provide any further clarifications. \n\n**Section 3.1 says the datasets A, B, and B are disjoint. Appendix A.2.1 says that A and B overlap.** \n\nTo be clear: $A$ and $B$ are always disjoint! We are very sorry about the confusion on this crucial point due to the lack of clarity in the appendix. Here is the full picture: there are 1,281,167 training images in the ImageNet data. Within these images, only 456,567 of them have bounding boxes annotations (which are needed to compute the Deja Vu score). The private set $A$ and $B$ are sampled from these 456,567 bounding boxes annotated images in such a way that set $A$ and $B$ are disjoint.\n\nIf we remove the 456,567 bounding boxes annotated images from the 1,281,167 training images, we get 824,600 remaining images without annotations which never overlap with $A$ or $B$. From this set of 800K images, we took 500k images as our public set $X$. So now, we have three non overlapping sets $A$, $B$, and $X$. Then, if we remove the 500K public set images from the 824,600 images without annotations, it leaves us with 324,600 images that are neither in $A$, $B$ or $X$. For simplicity, let us call this set of remaining 324,600 images the set $C$. Then, we have split the entire ImageNet training set into four non-overlapping splits called $A$, $B$, $C$ and $X$.\n\nWhen running our experiment with a small number of training images, we only use the set $A$ to train SSL_A and the set $B$ to train SSL_B and then use the set $X$ as a public set for evaluation. However, to run larger scale experiments, we use as additional training data for SSL_A and SSL_B: the images sampled from the set $C$. Here, SSL_A will still be trained on set $A$ but it will be augmented with images from set $C$. The same goes for SSL_B which will still be trained on the set $B$ but augmented with images from the set C. As such, some images sampled from the set C to train SSL_A or to train SSL_B might overlap. However, this is not an issue since the evaluation is done using only images from the bounding boxes annotated set $A$ and set $B$ which are never overlapping (and to be clear -- $A$, $B$ do not overlap with $C$).\n\nWe would like to thank you again for raising the lack of clarity in the appendix. We will be much more straightforward in the next revision of the paper, we hope that the introduction of the set $C$ will help to clarify the experimental setup in the appendix. Please let us know if there are any remaining concerns on this very important point.\n\n**On face-blurred ImageNet**\n\nThe face-blurred ImageNet is the same thing as ImageNet which is the same as ImageNet-1K (we inadvertently use different words to refer to the same dataset). Since this is not a different dataset, it is impossible that $X$ contains images form $A$ and $B$. We used only one dataset in our work, and that is ImageNet (in which people's faces were blurred for privacy reasons). It is helpful to know this was unclear since it is indeed foundational of our assertions of memorization. \n\n**I was initially confused the use of the word \"reconstruction.\" For example, Figure 1 talks about \"reconstruction of an SSL training image,\" but the generated image looks quite different from the training image. It seems you use the word differently than recent work does [1-5], where it means producing an example that is pixel-by-pixel, or feature-by-feature, similar to the input. I suggest using a different word or clarifying the issue earlier in the paper.**\n\nThis is helpful feedback. It is true that we do not aim to reconstruct the images pixel-by-pixel. We may be better using \"partial reconstruction\" in which the aim is to reconstruct memorized information (foreground) from a given crop (background). We feel that partial reconstruction of *thousands* of training images is a contribution.\n\n**Could you lay out what the \"adversary,\" the person conducting the privacy attack, has access to?**\n\nWe do twice mention that Deja Vu memorization could imply adversarial advantage. We mean that an adversary with access to partial information of a training image (e.g. crop of background or an individual’s face) could use the model to infer the remainder of the image, possibly with the help of a public dataset like disjoint set $X$. In this case, the adversary could have access to the same information that our test does, as represented by the red box of Figure 2’s Inference Pipeline. It takes in set $X$, the background crop, and the model, and returns inferences about the remainder of the training image. We will make this clear where we mention the adversarial implications of memorization. \n\nHowever, it is worth noting that in this paper we focus on the detection of memorization in SSL, which is distinct from a methodical study of practical adversarial risk. We do believe our discovery of memorization has implications for privacy and is fodder for future security work, but are careful to not make strong claims quantifying the exact risk imposed by this memorization. As such, we intentionally focus this paper on identifying memorization and not on defining a practical adversary. \n\n**I find the related work section adequate, but I think Section 3, \"Defining Deja Vu Memorization,\" would benefit from clearer connections between prior work and the presented definitions.**\n\nThank you for taking the time to share these additional references. We will work them into our related work section to more accurately position our notion of memorization.\n\n**I think synthetic-data experiments might strengthen the paper.**\n\nThank you for your suggestion, can you explain the type of synthetic based experiment you have in mind? Any detail on the different types of memorized information you are referring to would be helpful. \n"}, {"review_id": "WnbNJQABzr", "reviewer": "Reviewer_mdS9", "summary": "This paper thoroughly studies a particular undesired aspect of state-of-the-art self-supervised learning models: deja-vu memorization, where the models end up memorizing specific parts of the image instead of learning just a semantically meaningful association. The authors provide extensive empirical results to show when SSL models might memorize, and what aspects of training might minimize this effect.", "questions": "* How do different augmentations used by SSLs impact memorization?\n* Is using just one other SSL model enough to give confident results of memorization? Should we include more reference SSL models to increase confidence?\n* How many samples would be needed in the X public dataset? Could be helpful to understand the scale if deployed.\n* When is the generative model actually needed, when is it necessary beyond just looking at the k nearest neighbors?", "limitations": "* The lack of a way to estimate the confidence or uncertainty in the memorization score.\n* Although shown for many models, the results are shown only for a single dataset. Using more datasets could have been nice, maybe using CelebA to show background info is memorized to learn person’s identity.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors introduce novel idea and testing methodology for SSL models\n* Provides comprehensive empirical results on different aspects of SSL modeling and training and memorization \n* Additional results on ablations and using different representations and aspects of the setup help understand the author’s proposed method better\n* Clear presentation of results and setup.", "weaknesses": "* Authors do not seem to talk about the impact of different augmentations on memorization, if any. Augmentations are very important for SSL performance, maybe they impact memorization?\n* It is hard to judge how confidently we can estimate the memorization if deploying. We compare to only one other reference SSL model. \n* The importance of using the generative model is not quite clear. From the result figures, the kNN images themselves seem to do enough.\n* Only shows using ImageNet data. No way to know if it will generalize to other datasets as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our work, and for your suggestions/questions! We have attempted to answer your primary questions and concerns below. Please let us know if you have any further questions, or if anything is unclear. \n\n**Authors do not seem to talk about the impact of different augmentations on memorization, if any. Augmentations are very important for SSL performance, maybe they impact memorization?**\n\nIn our instance, the most important augmentation is the Random Cropping one. If we don’t use random crop, there is no way for the network to learn to associate a specific patch to a given image. In fact, we also show in the paper some memorization occurring with a classifier which is trained only with random cropping. Concerning SSL, it is true that additional color based augmentation like color-jitter are used which make the task of mapping the representation of different crops much harder.\n\nAs request, we ran experiments in which we change the SSL data augmentations. For the first one we remove the color-jitter augmentation from the SSL augmentations (leaving cropping, grayscale, solarization and blur), while for the second one we kept only cropping as data augmentation.\n\n| | All Augmentations | Without ColorJitter | Only Cropping | \n|---|---|---|---|\n| KNN Accuracy | 40.6 | 26.6 | 18.2 |\n| DejaVu Score | 33.2 | 24.9 | 12,9 |\n\n---\nAs you can see, removing augmentations decreases as expected the KNN accuracy of the models. In consequence, It also reduces the deja vu score. The issue is that it is not really possible to disentangle augmentation from dejavu memorization from generalization. SSL models needs strong augmentations and if we remove them, we just learn bad representation that cannot be leveraged for any downstream tasks. \n\n**It is hard to judge how confidently we can estimate the memorization if deploying. We compare to only one other reference SSL model.** \n\nOur method measures model $A$’s memorization of set $A$ and model $B$’s memorization of set $B$, and reports the average. We consistently find that the degree of memorization of these two disjoint sets are nearly equal. Given that both model $A$ and $B$ appear to show systematic memorization of their disjoint training examples, we believe that this confidently reveals the existence of memorization in training.  \n\nTo show memorization of a single example, we agree that one would need to train several reference models (to e.g. confirm that the reference model rarely guesses a foreground Swan given a water background). However, this would significantly increase the computational demand of the test. We feel that our test confidently captures the existence of memorization without untenable computational demands. \n\n**The importance of using the generative model is not quite clear. From the result figures, the kNN images themselves seem to do enough.**\n\nThank you for your feedback on this point. The generative model offers a single reconstruction by averaging the kNN examples. It is helpful to know that this did not add much to your read of the paper. We will edit the paper to de-emphasize the generative model reconstructions. \n\n**[...] only shows using ImageNet data. No way to know if it will generalize to other datasets as well.**\n\nOur choice of a curated dataset like ImageNet was intentional for two reasons. \n\nFirst, ImageNet comes with bounding box annotations that allow us to precisely separate the background (e.g. water)  from the labeled foreground object (e.g. swan), allowing us to run our quantitative memorization tests. When bounding boxes are not available, we propose a heuristic test (see Appendix A.5) that simply takes a corner crop of the image which most likely removes the foreground object. We show that this heuristic offers a good lower bound of memorization. \n\nSecond, ImageNet is largely deduplicated allowing us to show memorization that cannot be fixed by better dataset curation. Prior works on dataset reconstruction e.g. [1, 2] show near-exact memorization that mostly occurs with highly duplicated examples, and can be largely avoided with deduplication. We feel that this is a significant contribution of our work. \n\n**[...] maybe using CelebA to show background info is memorized to learn person’s identity.**\n\nWe agree that such a study is important, however there are privacy and legal concerns in play here. Consent was never obtained from the people appearing in the CelebA dataset which might pose legal and ethical concern (and blurring the faces in CelebA will just make the dataset unusable). Using ImageNet, we have a large diversity of images which offers significantly more variety than smaller scale dataset like CelebA and others. \n\n[1] Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, et al. “Extracting Training Data from Large Language Models.” ArXiv:2012.07805 [Cs], December 14, 2020. http://arxiv.org/abs/2012.07805.\n\n[2] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188.\n"}]

Author rebuttal:
We would like to thank all reviewers for their thoughtful remarks on our work. Reviewer H9Sq raised an important clarification, asking whether sets $A$ and $B$ are disjoint. We want to clarify for everyone that indeed our set $A$ and $B$ are always disjoint (which is fundamental in our analysis). 

We also want to highlight that the core of our contribution is the detection of memorization in SSL, which is distinct from a methodical study of practical adversarial risk. We do believe our discovery of memorization might have implications for privacy and is fodder for future security work, but are careful to not make strong claims quantifying the exact risk imposed by this memorization. As such, we intentionally focus this paper on identifying memorization and not on defining a practical adversary. We recognize that the claims in our paragraph on the privacy implications of Deja Vu memorization might have been too strong without providing further empirical evidence. We will be careful to update this paragraph accordingly.

We were pleased to see the thorough and detailed summaries written by each reviewer, underscoring their careful read of the work. We have attempted to respond to each of your primary questions below. Please respond if you have any followup questions. We are eager to make our contributions as clear as possible, and your help is invaluable!


Meta-review: This paper raises an interesting question studying memorization in SSL: this paper shows that it is possible to predict the foreground object of an image when given only the background, something that should be impossible if the model did not memorize. The reviewers universally liked the paper and after the back-and-forth with authors I agree the paper is high quality and should be accepted.
Decision: Accept (poster)