Title: Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: self-supervised learning

privacy

data reconstruction

memorization
Abstract: Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models -- which we refer to as déjà vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that déjà vu memorization is common to different SSL algorithms, is exacerbated by certain design choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of déjà vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies.

Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning

Abstract

Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models-which we refer to as déjà vu memorization . Concretely, we show that given the trained model and a crop of a training image containing only the background ( e.g. , water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that déjà vu memorization is common to different SSL algorithms, is exacerbated by certain hyperparameter choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of déjà vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies.

1 Introduction

Self-supervised learning (SSL) [11; 12; 33; 2; 9; 20] aims to learn general representations of contentrich data without explicit labels by solving a pretext task . In many recent works, such pretext tasks rely on joint-embedding architectures whereby randomized image augmentations are applied to create multiple views of a training sample, and the model is trained to produce similar representations for those views. When using cropping as random image augmentation, the model learns to associate objects or parts (including the background scenery) that co-occur in an image. However, doing so also arguably exposes the training data to higher privacy risk as objects in training images can be explicitly memorized by the SSL model. For example, if the training data contains the photos of individuals, the SSL model may learn to associate the face of a person with their activity or physical location in the photo. This may allow an adversary to extract such information from the trained model for targeted individuals.

In this work, we aim to evaluate to what extent SSL models memorize the association of specific objects in training images or the association of objects and their specific backgrounds, and whether this memorization signal can be used to reconstruct the model's training samples. Our results demonstrate that SSL models memorize such associations beyond simple correlation. For instance, in Figure 1 ( left ), we use the SSL representation of a training image crop containing only water and this enables us to reconstruct the object in the foreground with remarkable specificity-in this case a black swan. By contrast, in Figure 1 ( right ), when using the crop from the background of a test set image that the SSL model has not seen before , its representation only contains enough information to

Private SSL Training

Image

Public SSLValidation Image

Reconstruction

Reconstruction

usingpublicdata

using public data

SSL

SSL

embedding

embedding

cropping

cropping

infer, through correlation, that the foreground object was likely some kind of waterbird - but not the specific one in the image.

Figure 1 shows that SSL models suffer from the unintended memorization of images in their training data-a phenomenon we refer to as déjà vu memorization 1 Beyond visualizing déjà vu memorization through data reconstruction, we also design a series of experiments to quantify the degree of memorization for different SSL algorithms, model architectures, training set size, etc. We observe that déjà vu memorization is exacerbated by the atypically large number of training epochs often recommended in SSL training, as well as certain hyperparameters in the SSL training objective. Perhaps surprisingly, we show that déjà vu memorization occurs even when the training set is large-as large as half of ImageNet [13]-and can continually worsen even when standard techniques for evaluating learned representation quality (such as linear probing) do not suggest increased overfitting. Our work serves as the first systematic study of unintended memorization in SSL models and motivates future work on understanding and preventing this behavior. Specifically, we:

Elucidate how SSL representations memorize aspects of individual training images, what we call déjà vu memorization;

Design a novel training data reconstruction pipeline for non-generative vision models. This is in contrast to many prominent reconstruction algorithms like [7; 8], which rely on the model itself to generate its own memorized samples and is not possible for SSL models or classifiers;

Propose metrics to quantify the degree of déjà vu memorization committed by an SSL model. This allows us to observe how déjà vu changes with training epochs, dataset size, training criteria, model architecture and more.

2 Preliminaries and Related Work

Self-supervised learning (SSL) is a machine learning paradigm that leverages unlabeled data to learn representations. Many SSL algorithms rely on joint-embedding architectures ( e.g. , SimCLR [11], Barlow Twins [33], VICReg [2] and Dino [10]), which are trained to associate different augmented views of a given image. For example, in SimCLR, given a set of images A = { A 1 , . . . , A n } and a randomized augmentation function aug , the model is trained to maximize the cosine similarity of draws of SSL (aug( A i )) with each other and minimize their similarity with SSL (aug( A j )) for i = j . The augmentation function aug typically consists of operations such as cropping, horizontal flipping, and color transformations to create different views that preserve an image's semantic properties.

̸

SSL representations. Once an SSL model is trained, its learned representation can be transferred to different downstream tasks. This is often done by extracting the representation of an image from

the backbone model 2 and either training a linear probe on top of this representation or finetuning the backbone model with a task-specific head [3]. It has been shown that SSL representations encode richer visual details about input images than supervised models do [4]. However, from a privacy perspective, this may be a cause for concern as the model also has more potential to overfit and memorize precise details about the training data compared to supervised learning. We show concretely that this privacy risk can indeed be realized by defining and measuring déjà vu memorization.

Privacy risks in ML. When a model is overfit on privacy-sensitive data, it memorizes specific information about its training examples, allowing an adversary with access to the model to learn private information [30; 16]. Privacy attacks in ML range from the simplest and best-studied membership inference attacks [26; 25; 24] to attribute inference [17; 22; 21] and data reconstruction [7; 1; 19] attacks. In the former, the adversary only infers whether an individual participated in the training set. Our study of déjà vu memorization is most similar to the latter: we leverage SSL representations of the training image background to infer and reconstruct the foreground object. In another line of work in the NLP domain [6; 7]: when prompted with a context string present in the training data, a large language model is shown to generate the remainder of string at test time, revealing sensitive text like home addresses. This method was recently extended to extract memorized images from Stable Diffusion [8]. We exploit memorization in a similar manner: given partial information about a training sample, the model is prompted to reveal the rest of the sample. 3 In our case, however, since the SSL model is not generative, extraction is significantly harder and requires careful design.

3 Defining Déjà Vu Memorization

What is déjà vu memorization? At a high level, the objective of SSL is to learn general representations of objects that occur in nature. This is often accomplished by associating different parts of an image with one another in the learned embedding. Returning to our example in Figure 1, given an image whose background contains a patch of water, the model may learn that the foreground object is a water animal such as duck, pelican, otter, etc. , by observing different images that contain water from the training set. We refer to this type of learning as correlation : the association of objects that tend to co-occur in images from the training data distribution.

A natural question to ask is 'Can the reconstruction of the black swan in Figure 1 be reasoned as correlation?' The intuitive answer may be no, since the reconstructed image is qualitatively very similar to the original image. However, this reasoning implicitly assumes that for a random image from the training data distribution containing a patch of water, the foreground object is unlikely to be a black swan. Mathematically, if we denote by P the training data distribution and A the image, then

is the probability of inferring that the foreground object is a black swan through correlation . This probability may be naturally high due to biases in the distribution P , e.g. , if P contains no other water animal except for black swans. In fact, such correlations are often exploited to learn a model for image inpainting with great success [32; 27].

Despite this, we argue that reconstruction of the black swan in Figure 1 is not due to correlation, but rather due to unintended memorization : the association of objects unique to a single training image. As we will show in the following sections, the example in Figure 1 is not a rare success case and can be replicated across many training samples. More importantly, failure to reconstruct the foreground object in Figure 1 ( right ) on test images hints at inferring through correlation is unlikely to succeed-a fact that we verify quantitatively in Section 4.1. Motivated by this discussion, we give a verbal definition of déjà vu memorization below, and design a testing methodology to quantify déjà vu memorization in Section 3.1.

Models/Train

sets

Inference

pipeline

Target

Reference

set/model

Private

set/model

Private

Train image A;EA

Find nearest neighbors

inference

KNN

label

`black swan'

label:

set A

setB

口

Private

set A

Public

setx

SSLA

Get KNN

SSLA

SSLB

subset of x

with labels

SSLA

black black black

for inference

Disjoint set

SSLA(crop(A;))

Iswan

Public

setx

periphery

RCDM

cropping

crop(A;)

KNN

RCDM

Definition: A model exhibits déjà vu memorization when it retains information so specific to an individual training image, that it enables recovery of aspects particular to that image given a part that does not contain them. The recovered aspect must be beyond what can be inferred using only correlations in the data distribution.

We intentionally kept the above definition broad enough to encompass different types of information that can be inferred about the training image, including but not restricted to object category, shape, color and position. For example, if one can infer that the foreground object is red given the background patch with accuracy significantly beyond correlation, we consider this an instance of déjà vu memorization as well. We mainly focus on object category to quantify déjà vu memorization in Section 4 since the ground truth label can be easily obtained. We consider other types of information more qualitatively in the visual reconstruction experiments in Section 5.

Distinguishing memorization from correlation. When measuring déjà vu memorization, it is crucial to differentiate what the model associates through memorization and what it associates through correlation . Our testing methodology is based on the following intuitive definition.

Definition: If an SSL model associates two parts in a training image, we say that it is due to correlation if other SSL models trained on a similar dataset from P without this image would likely make the same association. Otherwise, we say that it is due to memorization .

Notably, such intuition forms the basis for differential privacy (DP; Dwork et al. [15]; Dwork & Roth [14])-the most widely accepted notion of privacy in ML.

3.1 Testing Methodology for Measuring Déjà Vu Memorization

In this section, we use the above intuition to measure the extent of déjà vu memorization in SSL. Figure 2 gives an overview of our testing methodology.

Dataset splitting. We focus on testing déjà vu memorization for SSL models trained on the ImageNet dataset [13] 4 . Our test first splits the ImageNet training set into three independent and disjoint subsets A , B and X . The dataset A is called the target set and B is called the reference set . The two datasets are used to train two separate SSL models, SSL A and SSL B , called the target model and the reference model . Finally, the dataset set X is used as an auxiliary public dataset to extract information from SSL A and SSL B . Our dataset splitting serves the purpose of

distinguishing memorization from correlation in the following manner. Given a sample A i ∈ A , if our test returns the same result on SSL A and SSL B then it is likely due to correlation because A i is not a training sample for SSL B . Otherwise, because A and B are drawn from the same underlying distribution, our test must have inferred some information unique to A i due to memorization. Thus, by comparing the difference in the test results for SSL A and SSL B , we can measure the degree of déjà vu memorization 5 .

Extracting foreground and background crops. Our testing methodology aims at measuring what can be inferred about the foreground object in an ImageNet sample given a background crop. This is made possible because ImageNet provides bounding box annotations for a subset of its training images-around 150K out of 1.3M samples. We split these annotated images equally between A and B . Given an annotated image A i , we treat everything inside the bounding box as the foreground object associated with the image label, denoted object( A i ) . We take the largest possible crop that does not intersect with any bounding box as the background crop (or periphery crop ), denoted crop( A i ) 6

KNN-based test design. Joint-embedding SSL approaches encourage the embeddings of random crops of a training image A i ∈ A to be similar. Intuitively, if the model exhibits déjà vu memorization, it is reasonable to expect that the embedding of crop( A i ) is similar to that of object( A i ) since both crops are from the same training image. In other words, SSL A (crop( A i )) encodes information about object( A i ) that cannot be inferred through correlation. However, decoding such information is challenging as these approaches do not learn a decoder associated with the encoder SSL A .

Here, we leverage the public set X to decode the information contained in crop( A i ) about object( A i ) . More specifically, we map images in X to their embeddings using SSL A and extract the k -nearestneighbor (KNN) subset of SSL A (crop( A i )) in X . We can then decode the information contained in crop( A i ) in one of two ways:

Label inference: Since X is a subset of ImageNet, each embedding in the KNN subset is associated with a class label. If crop( A i ) encodes information about the foreground object, its embedding will be close to samples in X that have the same class label ( i.e. , foreground object category). We can then use a KNN classifier to infer the foreground object in A i given crop( A i ) .

Visualization: Since we have access to a KNN subset associated to a given crop( A i ) , we can visualize directly the images associated to this subset. Then, we can infer through visualizing what is common within this subset, what information can be retrieved for this single crop. In addition, to simplify the visualization pipeline and to map directly a given crop representation to an image, we train an RCDM [4]-a conditional generative model-on X to decode SSL A embeddings into images. The RCDM reconstruction can recover qualitative aspects of an image remarkably well, such as recovering object color or spatial orientation using its SSL embedding. Given the KNN subset, we average their SSL embeddings and use the RCDM model to visually reconstruct A i .

In Section 4, we focus on quantitatively measuring déjà vu memorization with label inference, and then use the KNN to visualize déjà vu memorization in Section 5.

4 Quantifying Déjà Vu Memorization

We apply our testing methodology to quantify a specific form of déjà vu memorization: inferring the foreground object (class label) given a crop of the background.

Extracting model embeddings. We test déjà vu memorization on a variety of popular SSL algorithms, with a focus on VICReg [2]. These algorithms produce two embeddings given an input image: a backbone embedding and a projector embedding that is derived by applying a small fully-connected network on top of the backbone embedding. Unless otherwise noted, all SSL embeddings refer to the projector embedding. To understand whether déjà vu memorization is particular to SSL, we also evaluate embeddings produced by a supervised model CLF A trained on A . We apply the same set of image augmentations as those used in SSL and train CLF A using the cross-entropy loss to predict ground truth labels.

Identifying the most memorized samples. Prior works have shown that certain training samples can be identified as more prone to memorization than others [16; 28; 29]. Similarly, we provide a heuristic to identify the most memorized samples in our label inference test using confidence of the KNN prediction. Given a periphery crop, crop( A i ) , let KNN A ( crop( A i ) ) ⊆ X denote its k -nearest neighbors in the embedding space of SSL A . From this KNN subset we can obtain: (1) KNN prob A ( crop( A i ) ) , the vector of class probabilities (normalized counts) induced by the KNN subset, and (2) KNN conf A ( crop( A i ) ) , the negative entropy of the probability vector KNN prob A ( crop( A i ) ) , as confidence of the KNN prediction. When entropy is low, the neighbors agree on the class of A i and hence confidence is high. We can sort the confidence score KNN conf A ( crop( A i ) ) across samples A i in decreasing order to identify the most confidently predicted samples, which likely correspond to the most memorized samples when A i ∈ A .

4.1 Population-level Memorization

Our first measure of déjà vu memorization is population-level label inference accuracy: What is the average label inference accuracy over a subset of SSL training images given their periphery crops? To understand how much of this accuracy is due to SSL A 's déjà vu memorization, we compare with a correlation baseline using the reference model: KNN B 's label inference accuracy on images A i ∈ A . In principle, this inference accuracy should be significantly above chance level ( 1 / 1000 for ImageNet) because the periphery crop may be highly indicative of the foreground object through correlation, e.g. , if the periphery crop is a basketball player then the foreground object is likely a basketball. Figure 3 compares the accuracy of KNN A to that of KNN B when inferring the labels of images in A i ∈ A 7 using crop( A i ) . Results are shown for VICReg and the supervised model; trends for other models are shown in Appendix A.5. For both VICReg and supervised models, inferring the class of crop( A i ) using KNN B (dashed line) through correlation achieves a reasonable accuracy that is significantly above chance level.

However, for VICReg, the inference accuracy using KNN A (solid red line) is significantly higher, and the accuracy gap between KNN A and KNN B indicates the degree of déjà vu memorization. We highlight two observations:

The accuracy gap of VICReg is significantly larger than that of the supervised model. This is especially notable when accounting for the fact that the supervised model is trained to associate randomly augmented crops of images with their ground truth labels. In contrast, VICReg has no label access during training but the embedding of a periphery crop can still encode the image label.

For VICReg, inference accuracy on the 1% most confident examples is nearly 95% , which shows that our simple confidence heuristic can effectively identify the most memorized samples. This result suggests that an adversary can use this heuristic to identify vulnerable training samples to launch a more focused privacy attack.

0

20

40

60

80

100

Top % most confident examples

0.0

0.2

0.4

0.6

0.8

1.0

Label inference accuracy

Target model (KNN

A

)

Reference model (KNN

B

)

SSL (VICReg)

Supervised

The déjà vu score. The curves of Figure 3 show memorization across confidence values for a single training scenario. To study how memorization changes with different hyperparamters, we extract a single value from these curves: the déjà vu score at confidence level p . In Figure 3, this is the gap between the solid red (or gray) and dashed red (or gray) where confidence ( x -axis) equal p % . In other words, given the periphery crops of set A , KNN A and KNN B separately select and label their top p % most confident examples, and we report the difference in their accuracy. The déjà vu score captures both the degree of memorization by the accuracy gap and the ability to identify memorized examples by the confidence level. If the score is 10% for p = 33% , KNN A has 10%

(a)

déjà vu

vs. epochs

(b)

déjà vu

vs. train set size

(a)

déjà vu

vs. epochs

(b)

déjà vu

vs. train set size

higher accuracy on its most confident third of A than KNN B does on its most confident third. In the following, we set p = 20% , approximately the largest gap for VICReg (red lines) in Figure 3.

Comparison with the linear probe train-test gap. A standard method for measuring SSL performance is to train a linear classifier-what we call a 'linear probe'-on its embeddings and compute its performance on a held out test set. From a learning theory standpoint, one might expect the linear probe's train-test accuracy gap to be indicative of memorization: the more a model overfits, the larger is the difference between train set and test set accuracy. However, as seen in Figure 4, the linear probe gap (dark blue) fails to reveal memorization captured by the déjà vu score (red) 8 .

4.2 Sample-level Memorization

The déjà vu score shows, on average , how much better an adversary can select and classify images when using the target model trained on them. This average score does not tell us how many individual images have their label successfully recovered by KNN A but not by KNN B . In other words, how many images are exposed by virtue of being in training set A : a risk notion foundational to differential privacy. To better quantify what fraction of the dataset is at risk, we perform a sample-level analysis by fixing a sample A i ∈ A and observing the label inference result of KNN A vs. KNN B . To this end, we partition samples A i ∈ A based on the result of label inference into four distinct categories: Unassociated - label inferred with neither KNN; Memorized - label inferred only with KNN A ; Misrepresented - label inferred only with KNN B ; Correlated - label inferred with both KNNs.

Intuitively, unassociated samples are ones where the embedding of crop( A i ) does not encode information about the label. Correlated samples are ones where the label can be inferred from crop( A i ) using correlation, e.g. , inferring the foreground object is basketball given a crop showing a basketball player. Ideally, the misrepresented set should be empty but contains a small portion of examples due to chance. Déjà vu memorization occurs for memorized samples where the embedding of SSL B does not encode the label but the embedding of SSL A does. To measure the pervasiveness of déjà vu memorization, we compare the size of the memorized and misrepresented sets. Figure 5 shows how the four categories of examples change with number of training epochs and training set size. The unassociated set is not shown since the total share adds up to one. The misrepresented set remains under 5% and roughly unchanged across all settings, consistent with our explanation that it is due to chance. In comparison, VICReg's memorized set surpasses 15% at 1000 epochs. Considering that up to 5% of these memorized examples could also be due to chance, we conclude that at least 10% of VICReg's training set is déjà vu memorized.

example

crop

ofcorrelatedimagesEA

example

neighborsEX

reconstructions

neighborsEX

reconstructions

SSL

SSL

SSLB

SSL

(a)

A correlated dam example

(b)

A memorized dam example

(a)

Memorized European badgers

(b)

Memorized American badgers

European

badgers

E A

neighborsEX

reconstructions

badgers

American

EA

neighborsEX

reconstructions

SSL

SSLA

SSL

SSL

SSL

SSLA

SSLB

SSLB

5 Visualizing Déjà Vu Memorization

Beyond enabling label inference using a periphery crop, we show that déjà vu memorization allows the SSL model to encode other forms of information about a training image. Namely, we leverage an external public dataset X and use it to find the nearest neighborhoods in this public dataset given a training periphery crop. We aim to answer the following two questions: (1) Can we visualize the distinction between correlation and déjà vu memorization? (2) What foreground object details can be extracted from the SSL model beyond class label?

Public image retrieval pipeline Following the pipeline in Figure 2, we use the projector embedding to find the KNN subset for the periphery crop, crop( A i ) , and visualize the images belonging to this KNN subset.

RCDMpipeline. RCDM is a conditional generative model that is trained on the backbone embedding of images X i ∈ X to generate an image that resembles X i . At test time, following the pipeline in Figure 2, we first use the projector embedding to find the KNN subset for the periphery crop, crop( A i ) , and then average their backbone embeddings as input to the RCDM model. Then, RCDM decodes this representation to visualize its content.

Visualizing Correlation vs. Memorization. Figure 6 shows examples of dams from the correlated set (left) and the memorized set (right) as defined in Section 4.2, along with the associated KNN set. In Figure 6a, the periphery crop is represented by the pink square, which contains concrete structure attached to the dam's main structure. As a result, both SSL A and SSL B produce embeddings of

20%Conf.

30  25 2015 10

50

40

Linear Probe Val.Acc.

：20%Conf.

35 30 25 20 15

50

60

Val.Acc.

40

Deja Vu Score

30

Deja Vu Score

30

LinearProbe

20

5

Linear Probe Val. Acc.

Deja Vu Score

10

10

5

Linear Probe Val. Acc.

Deja Vu Score

20

10

0

20

40

60

80

100

0

0

0

Invariance,入

projector

proj layer 2

proj layer 1

backbone

(a)

Loss hyper-parameter

(b)

Guillotine regularization

DejaVu Score 20% Conf.

35

30

Resnet50

Resnetl01

Vit-Large

25

Vit-Base

Resnet34

20

Vit-Tiny

15

0

100

200

300

400

Number of parameters (millions)

(a)

déjà vu vs. capacity

(b)

déjà vu

(DV) vs. Criterion

crop( A i ) whose KNN set in X consist of dams, i.e. , there is a correlation between the concrete structure in crop( A i ) and the foreground dam. In Figure 6b, the periphery crop only contains a patch of water, which does not strongly correlate with dams in the ImageNet distribution. Evidently, the reference model SSL B embeds crop( A i ) close to that of other objects commonly found in water, such as sea turtle and submarine. In contrast, the KNN set according to SSL A all contain dams despite the vast number of alternative possibilities within the ImageNet classes which highlight memorization in SSL A between this specific patch of water and the dam.

Visualizing Memorization Beyond Class Label. Figure 7 shows four examples of badgers from the memorized set. In all four images, the periphery crop (pink square) does not contain any indication that the foreground object is a badger. Despite this, the KNN set using SSL A consistently produce images of badgers, while the same does not hold for SSL B . More interestingly, the KNN using SSL A in Figure 7a all contain European badgers, while reconstructions in Figure 7b all contain American badgers, accurately reflecting the species of badger present in the respective training images. Since ImageNet-1K does not differentiate between these two species of badgers, our reconstructions show that SSL models can memorize information that is highly specific to a training sample beyond its class label 9 .

6 Mitigation of déjà vu memorization

We cannot yet make claims on why déjà vu occurs so strongly for some SSL training settings and not for others. To gain some intuition for future work, we present additional observations that shed light on which parameters have the most salient impact on déjà vu memorization.

Déjà vu memorization worsens by increasing number of training epochs. Figure 4a shows how déjà vu memorization changes with number of training epochs for VICReg. The training set size is fixed to 300K samples. From 250 to 1000 epochs, the déjà vu score (red curve) grows threefold : from under 10% to over 30%. Remarkably, this trend in memorization is not reflected by the linear probe gap (dark blue), which only changes by a few percent beyond 250 epochs.

Training set size has minimal effect on déjà vu memorization. Figure 4b shows how déjà vu memorization responds to the model's training set size. The number of training epochs is fixed to 1000. Interestingly, training set size appears to have almost no influence on the déjà vu score (red line), indicating that memorization is equally prevalent with a 100K dataset and a 500K dataset. This result suggests that déjà vu memorization may be detectable even for large datasets. Meanwhile, the

standard linear probe train-test accuracy gap declines by more than half as the dataset size grows, failing to represent the memorization quantified by our test.

Training loss hyper-parameter has a strong effect. Loss hyper-parameters, like VICReg's invariance coefficient (Figure 8a) or SimCLR's temperature parameter (Appendix Figure 14a) significantly impact déjà vu with minimal impact on the linear probe validation accuracy.

Some SSL criteria promote stronger déjà vu memorization. Table 9b demonstrates that the degree of memorization varies widely for different training criteria. VICReg and Barlow Twins have the highest déjà vu scores while SimCLR and Byol have the lowest. With the exception of Byol, all SSL models have more déjà vu memorization than the supervised model. Interestingly, different criteria can lead to similar linear probe validation accuracy and very different degrees of déjà vu as seen with SimCLR and Barlow Twins. Note that low degrees of déjà vu can still risk training image reconstruction, as exemplified by the SimCLR re-

constructions in Figures 6 and 11.

Larger models have increased déjà vu memorization. Figure 9a validates the common intuition that lower capacity architectures (Resnet18/34) result in less memorization than their high capacity counterparts (Resnet50/101). We see the same trend for vision transformers as well.

Guillotine regularization can help reduce déjà vu memorization. Previous experiments were done using the projector embedding. In Figure 8b, we present how Guillotine regularization[3] (removing final layers in a trained SSL model) impacts déjà vu with VICReg 10 . Using the backbone embedding instead of the projector embedding seems to be the most straightforward way to mitigate déjà vu memorization. However, as demonstrated in Appendix A.7, backbone representation with low déjà vu score can still be leveraged to reconstruct some of the training images.

0

10

20

30

40

50

60

Finetuned Val. Acc.

Déjà Vu Score

Linear Probe Val. Acc.

0

10

20

Finetuning epoch

0

5

10

15

20

25

30

35

Déjà Vu Score 20% Conf.

A little bit of fine-tuning might help to reduce memorization A common strategy in SSL is to fine-tune the model to solve the downstream task. In Figure 10, we show how the Dejavu score changes when fine-tuning a pretrained VICReg model. This pretrained model was trained on the set A for 1000 epochs and fine-tuned on a classification task on the set A for 20 epochs (which can be seen in the x axis on the figure). Interestingly, the DejaVu score decreases significantly in the first finetuning epochs while the validation accuracy is increasing. However after 5 epochs, the DejaVu score is increasing and after 20 epochs become almost as high at the original value before fine-tuning. This behavior indicates that even fine-tuning might not help in reducing DejaVu memorization.

7 Conclusion

We defined and analyzed déjà vu memorization, a notion of unintended memorization of partial information in image data. As shown in Sections 4 and 5, SSL models can largely exhibit déjà vu memorization on their training data, and this memorization signal can be extracted to infer or visualize image-specific information. Since SSL models are becoming increasingly widespread as foundation models for image data, negative consequences of déjà vu memorization can have profound downstream impact and thus deserves further attention. Future work should focus on understanding how déjà vu emerges in the training of SSL models and why methods like Byol are much more robust to déjà vu than VICReg and Barlow Twins. In addition, trying to characterize which data points are the most at risk of déjà vu could be crucial to get a better understanding on this phenomenon.

References

Balle, B., Cherubin, G., and Hayes, J. Reconstructing training data with informed adversaries. arXiv preprint arXiv:2201.04845 , 2022.

Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=xm6YD62D1Ub .

Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vincent, P. Guillotine regularization: Improving deep networks generalization by removing their head, 2022. URL https://arxiv. org/abs/2206.13378 .

Bordes, F., Balestriero, R., and Vincent, P. High fidelity visualization of what your selfsupervised representation knows about. Transactions on Machine Learning Research , 2022. URL https://openreview.net/forum?id=urfWb7VjmL .

Bordes, F., Balestriero, R., and Vincent, P. Towards democratizing joint-embedding selfsupervised learning, 2023. URL https://arxiv.org/abs/2303.01986 .

Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19) , pp. 267-284, 2019.

Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-V oss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650, 2021.

Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V ., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 , 2023.

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Caron, M., Touvron, H., Misra, I., Jegou, H., and Joulin, J. M. P. B. A. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Chen, X. and He, K. Exploring simple siamese representation learning. In CVPR , 2020.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255. Ieee, 2009.

Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Theoretical Computer Science , 9(3-4):211-407, 2013.

Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography , pp. 265-284. Springer, 2006.

Feldman, V. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pp. 954-959, 2020.

Fredrikson, M., Lantz, E., Jha, S., Lin, S., Page, D., and Ristenpart, T. Privacy in pharmacogenetics: An { End-to-End } case study of personalized warfarin dosing. In 23rd USENIX Security Symposium (USENIX Security 14) , pp. 17-32, 2014.

Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS , 2020.

Guo, C., Karrer, B., Chaudhuri, K., and van der Maaten, L. Bounding training data reconstruction in private (deep) learning. arXiv preprint arXiv:2201.12383 , 2022.

He, K., Chen, X., Xie, S., Li, Y ., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 16000-16009, June 2022.

Jayaraman, B. and Evans, D. Are attribute inference attacks just imputation? arXiv preprint arXiv:2209.01292 , 2022.

Mehnaz, S., Dibbo, S. V., Kabir, E., Li, N., and Bertino, E. Are your sensitive attributes private? novel model inversion attribute inference attacks on classification models. arXiv preprint arXiv:2201.09370 , 2022.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library . Curran Associates Inc., Red Hook, NY, USA, 2019.

Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning , pp. 5558-5567. PMLR, 2019.

Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., and Backes, M. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246 , 2018.

Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pp. 3-18. IEEE, 2017.

Ulyanov, D., Vedaldi, A., and Lempitsky, V. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 9446-9454, 2018.

Watson, L., Guo, C., Cormode, G., and Sablayrolles, A. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440 , 2021.

Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. arXiv preprint arXiv:2111.09679 , 2021.

Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pp. 268-282. IEEE, 2018.

You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5505-5514, 2018.

Zbontar, J., Jing, L., Misra, I., LeCun, Y ., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arxiv:2103.03230 , 2021.

Zeng, H. and Cui, X. Simclrt: A simple framework for contrastive learning of rumor tracking. Eng. Appl. Artif. Intell. , 110:104757, 2022. doi: 10.1016/j.engappai.2022.104757. URL https://doi.org/10.1016/j.engappai.2022.104757 .

A Appendix

A.1 Additional reconstruction examples

The two reconstruction experiments of Section 5 are each exemplified within one class. However, we see strong reconstructions using SSL A in several classes, and similar experimental results. To demonstrate this, we repeat the experiment of Section 5 using the yellow garden spider class and the experiment of 5 using the aircraft carrier class.

example

crop

of

correlated

images EA

example

cropof

memorizedi

imagesEA

UNWWU

neighborsEX

reconstructions

neighborsEX

reconstructions

SSL

SSL

SSLB

SSL

Selection of Memorized and Correlated Images: The images of Figure 6 and 11 were chosen methodically as follows.

Image selection: The 20 images of Figures 6 and 11 are selected deterministically using label inference accuracy and KNN confidence score. The 10 most correlated images are those images in the correlated set (both models infer label correctly) of A with the highest confidence agreement between models SSL A and SSL B . To measure confidence agreement we take the minimum confidence of the two models. The 10 most memorized images are those images in the memorized set (only target model infers the label correctly) of A with the highest confidence difference between models SSL A and SSL B .

Class selection: To find classes with a high degree of déjà vu , classes were sorted by the label inference accuracy gap between the target and reference model. We selected the class based on a handful of criteria. First, we prioritized classes without images of human faces, thereby removing classes like 'basketball', 'bobsled', 'train station', and even 'tench' which is a fish often depicted in the hands of a fisherman. Second, we prioritized classes that include at least ten images with a high confidence difference between the target and reference models ('most memorized' images described above) and at least ten images with high confidence agreement ('most correlated' images described above). This led us to the dam and yellow garden spider classes.

Ship

EA

neighborsEX

reconstructions

ShipEA

neighborsEX

reconstructions

side

front

SSL

SSL

SSL

SSL

SSL,

SSL

SSL

SSLE

(a) extracting side of ship from VICReg

(b)

extracting front of ship from VICReg

Selection of Beyond-Label-Inference Images: The images of Figure 7 and 12 were chosen methodically as follows.

Image selection: The four images of Figures 7 and 12 are selected using KNN confidence score, and, necessarily, hand picked selection for unlabeled features. Within a given class, we look at the top 40 images with highest target model KNN confidence scores. We then filter through these images to identify a distinguishable feature like different species within the same class or different object positions within the same class. This step is necessary because we are looking for features that are not labeled by ImageNet. We then choose two of these top 40 with one feature (e.g. American badger) and two with the alternative feature (e.g. European badger).

Class selection: To find classes with a high degree of déjà vu , classes were sorted by the target model's top-40 KNN confidence values within each class. As in the memorization vs. correlation experiment, we prioritized classes without images of human faces.

A.2 Details on the experimental setup

A.2.1 Details on dataset splits

There are 1,281,167 training images in the ImageNet dataset. Within these images, only 456,567 of them have bounding boxes annotations (which are needed to compute the Deja Vu score). The private set A and B are sampled from these 456,567 bounding boxes annotated images in such a way that set A and B are disjoint.

If we remove the 456,567 bounding boxes annotated images from the 1,281,167 training images, we get 824,600 remaining images without annotations which never overlap with A or B . From this set of 800K images, we took 500k images as our public set X . So now, we have three non overlapping sets A , B , and X. Then, if we remove the 500K public set images from the 824,600 images without annotations, it leaves us with 324,600 images that are neither in A , B or X . For simplicity, let us call this set of remaining 324,600 images the set C . Then, we have split the entire ImageNet training set into four non-overlapping splits called A , B , C and X .

When running our experiments with a small number of training images, we only use the set A to train SSL A and the set B to train SSL B and then use the set X as a public set for evaluation. However, to run larger scale experiments, we use as additional training data for SSL A and SSL B : the images sampled from the set C . Here, SSL A will still be trained on set A but it will be augmented with images from set C . The same goes for SSL B which will still be trained on the set B but augmented with images from the set C . As such, some images sampled from C to train SSL A or to train SSL B might overlap. However, this is not an issue since the evaluation is done using only images from the bounding boxes annotated set A and set B which are never overlapping.

To identify memorization, our tests only attempt to infer the labels of the unique examples A and B that differentiate the two private sets. The periphery crop, crop( A i ) , is computed as the largest possible crop that does not intersect with the foreground object bounding box. In some instances the largest periphery crop is small, and not high enough resolution to get a meaningful embedding. To circumvent this, we only run the test on bounding box examples where the periphery crop is at least 100 × 100 pixels.

Each size of training set, 100k to 500k, includes an equal number of examples per class in both sets A and B . The total bounding box annotated examples of each class are evenly divided between A and B . The remaining examples in each class are the examples from C . We reiterate that the bounding box examples in set A are unique to set A , and thus can only be memorized by SSL A .

The disjoint public set, X , contains ground truth labels but no bounding-box annotations. The size and content of X remains fixed for all tests.

A.2.2 Details on the training setup

Model Training: We use PyTorch [23] with FFCV-SSL [5]. All models are trained for 1000 epochs with model checkpoints taken at 50, 100, 250, 500, 750, and 1000 epochs. We note that 1000 epochs is used in the original papers of both VICReg and SimCLR. All sweeps of epochs use the 300k dataset. All sweeps of datasets use the final, 1000 epoch checkpoint. We use a batch size of 1024, and LARS optimizer [31] for all SSL models. All models use Resnet101 for the backbone. As seen in Appendix A.6, a Resnet50 backbone results in déjà vu consistent with that of Resnet101.

VICReg Training: VICReg is trained with the 3-layer fully connected projector used in the original paper with layer dimensions 8192-8192-8192. The invariance, variance, and covariance parameters are set to λ = 25 , µ = 25 , ν = 1 , respectively, which are used in the original paper [2]. The LARS base learning rate is set to 0.2, and weight decay is set to 1e-6.

SimCLR Training: SimCLR is trained with the 2-layer fully connected projector used in the original paper with layer dimensions 2048-256. The temperature parameter is set to τ = 0 . 15 . The LARS base learning rate is set to 0.3, and weight decay is set to 1e-6.

Supervised Training: Unlike the SSL models, the supervised model is trained with label access using cross-entropy loss. To keep architectures as similar as possible, the supervised model also uses a Resnet101 backbone and the same projector as VICReg. A final batchnorm, ReLU, and linear layer

is added to bring the 8192 dimension projector output to 1000-way classification activations. We use these activations as the supervised model's projector embedding. The supervised model uses the LARS optimizer with learning rate 0.2.

A.2.3 Details on the evaluation setup

KNN: For each test, we build two KNN's: one using the target model, SSL A (or CLF A ), and one using the reference model SSL B (or CLF B ). As depicted in Figure 2, each KNN is built using the projector embeddings of all images in the public set X as the neighbor set. When testing for memorization on an image A i ∈ A , we first embed crop( A i ) using SSL A , and find its K = 100 L 2 nearest neighbors within the SSL A embeddings of X . See section A.4 for a discussion on selection of K . We then take the majority vote of the neighbors' labels to determine the class of A i . This entire pipleline is repeated using reference model SSL B and its KNN to compute reference model accuracy.

In practice, all of our quantitative tests are repeated once with SSL A as the target model (recovering labels of images in set A ) and again with SSL B as the target model (recovering labels of images in set B ). All results shown are the average of these two tests. Throughout the paper, we describe SSL A as the target model and SSL B as the reference model for ease of exposition.

RCDM: The RCDM is trained on a face-blurred version of ImageNet [13] and is used to decode the SSL backbone embedding of an image back into an approximation of the original image. All RCDMs are trained on the public set of images X used for the KNN. A separate RCDM must be trained for each SSL model, since each model has a unique mapping from image space to embedding space.

At inference time, the RCDM is used to reconstruct the foreground object given only the periphery cropping. To produce this reconstruciton, the RCDM needs an approximation of the backbone embedding of the original image. The backbone of image A i is approximated by 1) computing crop embedding SSL proj A (crop( A i )) , 2) finding the five public set nearest neighbors of the crop embedding, and 3) averaging the five nearest neighbors' backbone embeddings. In practice, these public set nearest neighbors are often a very good approximation of the original image, capturing aspects like object class, position, subspecies, etc..

A.3 Sample-level memorization

VICReg

Supervised

VICReg

Supervised

35

35

35

35

30

30

Correlated

30

30

Correlated

Share of total

25

Share of total

25

Misrepresented

Share of total

25

Share of total

25

Misrepresented

Memorized

Memorized

20

20

20

15

15

15

15

10

10

10

10

5

5

5

5

0

0

0

0

2004006008001000

2004006008001000

1002003004

400

500

100200300

400500

Numberofepochs

Numberofepochs

Trainingsetsize(thousands)

Training setsize(thousands)

(a)

Categories of training samples vs. number of epochs

(b)

Categories of training samples vs. training set size

0.2

0.4

0.6

0.8

1.0

Temperature

0.0

0.1

0.2

0.3

0.4

0.5

Share of total

Correlated

Misrepresented

Memorized

0.2

0.4

0.6

0.8

1.0

Temperature

0.0

0.2

0.4

0.6

0.8

1.0

Linear probe accuracy

Train

Validation

(a)

SimCLR

20

40

60

80

100

Invariance

0.0

0.1

0.2

0.3

0.4

0.5

Share of total

Correlated

Misrepresented

Memorized

20

40

60

80

100

Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Linear probe accuracy

Train

Validation

(b)

VICReg

Many SSL algorithms contain hyperparameters that control how similar the embeddings of different views should be in the training objective. We show that these hyperparameters directly affect déjà vu memorization. Figure 14 shows the size of the memorized set for SimCLR (left) and VICReg (right) as a function of their respective hyperparameters, τ and λ . We observe that the memorized set is largest within a relatively narrow band of hyperparameter values, indicating strong déjà vu memorization. By selecting hyperparameters outside this band, déjà vu memorization sharply decreases while the linear probe validation accuracy on ImageNet remains roughly the same.

A.4 Selection of K for KNN

In this section, we describe the impact of K on the KNN label inference accuracy.

1.00

0

25

50

75

100

Number of neighbors: K

0.00

0.25

0.50

0.75

Label Inference Accuracy

accuracy on top 20% conf.

accuracy on all examples

ref. model correlation baseline

(a)

0

100

200

Number of neighbors: K

0.0

0.2

0.4

Share of total

correlated

forgotten

memorized

(b) VICReg, Share of memorized examples

Figure 15 above shows how the tests of Section 4 change with number of public set nearest neighbors K used to make label inferences. Both tests are relatively robust to any choice of K . Results are shown on VICReg trained for 1k epochs on the 300k dataset. We see that any choice of K greater than 50 and less than the number of examples per class (300, in this case) appears to have good performance. Since our smallest dataset has 100 images per class, we chose to set K = 100 for all experiments.

A.5 Effect of SSL criteria

We repeat the quantitative memorization tests of Section 4 on different models: VICReg[2], BarlowTwins[33], Dino[10], Byol[18], SimCLR[34] and a supervised model in Figure 16. We observe differences between SSL training criteria with respect to déjà vu memorization. The easy ones to attack are VICReg and Barlow Twins whereas SimCLR and Byol are more robust to these attacks. While the degree of memorization appears to be reduced for SimCLR compared with VICReg, it is still stronger than the supervised baseline.

accuracy on top 20.0% conf.

ref. model correlation baseline

accuracy on all examples

Share of total

0.4

0.5

forgotten

memorized

correlated

80

%0 douome

ref. model correlation baseline

accuracy on all examples

Share of total

0.4

0.5

forgotten

correlated

memorized

VICReg

0.6-

0.3

Byol

0.6

0.3

04f

0.2

0.4

0.2

0.2

0.1

0.2

0.1

0.0

250

500

一

750

1000

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

250

500

750

1000

epochs

epochs

epochs

epochs

0.5

0.5

juoo 00 douoKxme

correlated

accuracy on top 20.0% conf.

correlated

Twins

0.6

0.8

。

ref. model correlation baseline

accuracy on all examples

0.4

0.3

forgotten

SimCLR

0.8

0.6

ref. model correlation baseline

accuracy on all examples

0.4

0.3

forgotten

memorized

0.4

0.4

0.21

0.1

0.2

0.1

0.0

250

500

750

1000

250

500

750

1000

0.0

250

500

750

一

1000

0.0

250

500

750

1000

epochs

epochs

epochs

epochs

Dino

0.8

0.6

ref. model correlation baseline

accuracy on top 20.0% conf.

accuracy on all examples

0.3

0.5

0.4

forgoten

correlated

Supervised

Label InferenceAccuracy

1.0

0.6

0.8

acuracy on top 20.0% conf.

accuracy on all examples

ref. model correlation baseline

0.3

0.5

0.4

correlated

forgotten

memorized

0.4

0.2

0.1

.2

0.1

0.0

250

epochs

500

750

1000

→

0.0

250

epochs

500

750

1000

0.0

250

epochs

500

750

1000

0.0

250

epochs

500

750

1000

A.6 Effect of Model Architecture and Complexity

Results shown in the main paper use Resnet101 for the model backbone. To understand the relationship between déjà vu and overparameterization, we compare with the smaller Resnet50 and Resnet18 in Figure 17. Overall, we find that increasing the number of parameters of the model leads to higher degree of déjà vu memorization. The same trend holds when using Vision Transformers (VIT-Tiny, -Small, -Base, and -Large with patch size of 16) of various sizes as the SSL backbone, instead of a Resnet. This highlights that déjà vu memorization is not unique to convolution architectures.

VIT-Tiny

Label Inference Accuracy

0.8

1.0

0.6f

accuracy on top 20.0% conf.

accuracy on all examples

ref. model correlation baseline

Share of total

0.3

0.5

0.4

corrlated

forgotten

memorized

RN18

0.6-

1.0

0.8

ref. model correlation baseline

accuracy on all examples

u 0 do uo me

Share of total

0.5

correlated

forgotten

memorized

0.4

0.2

0.4

0.2

0.2

0.1

0.2

0.1

0.0

250

500

750

1000

0.0

250

500

750

1000

250

500

750

0001

→

0.0

250

00S

750

0001

epochs

epochs

epochs

epochs

0.5

0.5

0.8

acuracy on top 20.0% conf.

accuracy on all examples

correlated

forgotten

0.8

accuracy on all xamples

accuracy on top20.0% conf.

Share of total

forgotten

correlated

VIT-Small

ref. model correlation baseline

memorized

RN34

ref. model corelation baseline

memorized

0.6

0.3

Label Inference

0.6

0.4

0.4

0.2

0.2

0.2

二

0.1

0.0

250

500

750

0001

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

250

500

750

0001

epochs

epochs

epochs

epochs

1.0

0.5

0.5

acuracy on top20.0% conf.

corelated

accuracy on top 20.0% conf.

corrlated

VIT-Base

0.8

ref. model correlation baseline

accuracy on all cxamples

0.3

0.4

forgotten

memorized

RN50

0.8

●accuracy on all examples

ref. model corelation baseline

0.3

0.4

forgotten

memorized

0.4

0.4

0.2+

0.1

0.2

0.1

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

250

500

750

1000

epochs

epochs

epochs

epochs

accuracy on top 20.0% conf.

0.5

correlated

accuracy on top 20.0% conf.

0.5

correlated

VIT-Large

0.8

ref. model correlation baseline

accuracy on all examples

forgoten

memorized

0.8

●accuracy on all cxamples

ref. model corelation baseline

0.4

forgotten

memorized

0.6

.3

0.3

0.4

0.4

0.2

0.2

一

0.1

0.1

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

250

500

750

100

0.0

250

500

750

0001

epochs

epochs

epochs

epochs

A.7 The impact of Guillotine Regularization on Deja Vu

In our experiments, we show déjà vu using the projector representation. The SSL loss directly incentivizes the projector representation to be invariant to random crops of a particular image. As such, we expect the projector to be the most overfit and produce the strongest déjà vu . Here, we study whether earlier representations between the projector and backbone exhibit less déjà vu memorization. This phenomenon - 'guillotine regularization' - has recently been studied from the perspective of generalization in Bordes et al. [3]. Here, we study it from the perspective of memorization .

To show how guillotine regularization impacts déjà vu , we repeat the tests of Section 4 on each layer of the VICReg projector: the 2048-dimension backbone (layer 0) up to the projector output (layer 3). We evaluate whether memorization is indeed reduced for the more regularized layers between the projector output and the backbone.

0

1

2

3

Layer

0.2

0.4

0.6

Label Inference Accuracy

accuracy on top 20% conf.

accuracy on all examples

ref. model correlation baseline

0

1

2

3

Layer

0.0

0.1

0.2

Share of total

correlated

forgotten

memorized

(a) population level accuracy

(b) share of memorized examples

Figure 18 shows how guillotine regularization significantly reduces the degree of memorization in VICReg. The vast majority of VICReg's déjà vu appears to occur in the final two layers of the projector (2,3): in earlier layers (0,1), the label inference accuracy of the target model and reference model are comparable. This suggests that - like the hyperparameter selection of Section 6 - guillotine regularization can also significantly mitigate déjà vu memorization. In the following, we extend this result to SimCLR and supervised models by measuring the degree of déjà vu in the backbone (layer 0) versus training epochs and dataset size.

Comparison of déjà vu in projector and backbone vs. epochs and dataset size Since the backbone is mostly used at inference time, we now evaluate how much déjà vu exists in the backbone representation for VICReg and SimCLR. We repeat the tests of Section 4 versus training epochs and train set size.

(a)

VICReg

VICReg Projector

Label Inference Accuracy

1.0

0.8

0.61

uoo %0o douo eme

ref. model correlation baseline

accuracy on allexamples

0.4

0.3

0.5

forgotten

correlated

memorized

0.8

0.6

accuracy on all examples

ref. model correlation baseline

accuracy on top 20.0% conf.

0.3

0.5

0

correlated

forgotten

memorized

0.4

0.2

0.1

0.2

=

0.1

0.0

250

500

750

100C

0.0

250

500

750

1000

0.0

100

200

300

400

500

0.0

100

200

000

400

500

epochs

epochs

train set size (thousands)

train set size (thousands)

VICReg Backbone

1.0

0.8

0.6

accuracy on top 20.0% conf.

ref. model correlation baseline

accuracy on all examples

0.5

memorized

correlated

forgotten

0.8

ref. model correlation baseline

uoo %odouoemoe

accuracy on all examples

of total

0.5

0.4

memorized

forgotten

correlated

0.3

Label Inference

0.6

0.3

0

0.4

0.2

0.1

0.2

I'0

0.0

250

epochs

500

750

1000

0.0

250

epochs

500

750

1000

0.0

100

train set size (thousands)

200

300

400

500

0.0

100

train set size (thousands)

200

300

400

500

SimCLR Projector

Label Inference Accuracy

accuracy on top 20.0% conf.

0.5

correlated

accuracy on top 20.0% conf.

0.5

correlated

0.8

accuracy on all examples

ref. model correlation baseline

0.4

forgotten

memorized

accuracy on all xamples

ref. model correlation baseline

Share of total

forgotten

memorized

0.6

0.3

0.3

0.4

0.4

0.2

0.2

0.1

0.2

0.1

0.0

250

500

750

1000

0.0

250

500

750

0001

100

200

300

400

500

0.0

100

200

300

400

500

epochs

epochs

train set size (thousands)

train set size (thousands)

SimCLRBackbone

accuracy on top 20.0% conf.

0.5

corelated

accuracy on top 20.0% conf.

0.5

correlated

0.8

accuracy on all examples

ref. model correlation baseline

Share of total

0.4

forgotten

memorized

0.8

ref. model correlation baseline

accuracy on all examples

eof total

0

forgotten

memorized

0.6

0.3

Label Inference

0.6

0.3

0.4

0.2

0.4

0.2+

0.1

0.2

0.1

→

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

100

200

300

400

500

0.0

100

200

300

400

500

epochs

epochs

train set size (thousands)

train set size (thousands)

(b)

SimCLR

Figure 19 shows that, indeed, déjà vu is significantly reduced in the backbone representation. For SimCLR, however, we see that backbone memorization is comparable with projector memorization. In light of the Guillotine regularization results above, this makes some sense since SimCLR uses fewer layers in its projector. Given that we were able to generate accurate reconstructions with the SimCLR projector (see Figures 11 and 6), we now evaluate whether we can produce accurate reconstructions of training examples using the SimCLR backbone alone.

Reconstructions using SimCLR Backbone Only: The above label inference results show that the SimCLR backbone exhibits a similar degree of déjà vu memorization as the projector does. To evaluate the risk of such memorization, we repeat the reconstruction experiment of Section 5 on the dam class using the SimCLR backbone instead of its projector.

(a)

First memorized dam example

Dam E A

neighbors E l

reconstructions

SSLA

SSLB

Dam E A

neighbors E X

reconstructions

SSLA

SSLB

(b)

Second memorized dam example

Figure 20 demonstrates that we are able to reconstruct training set images using the SimCLR backbone alone. This indicates that déjà vu memorization can be leveraged to make detailed inferences about training set images without any access to the projector. As such, withholding the projector for model release may not be a strong enough mitigation against déjà vu memorization.

A.8 Detecting Déjà vu without Bounding Box Annotations

The memorization tests presented critically depend on bounding box annotations in order to separate the foreground object from the periphery crop. Since such annotations are often not available, we propose a heuristic test that simply uses the lower left corner of an image as a surrogate for the periphery crop. Since foreground objects tend to be near the center of the image, the corner crop usually excludes the foreground object and does not require a bounding box annotation.

Figure 21 demonstrates that this heuristic test can successfully capture the trends of the original tests (seen in Figure 16) without access to bounding box annotations. However, as compared to Figure 16, the heuristic tends to slightly underestimate the degree of memorization. This is likely due to the fact that some corner crops partially include the foreground object, thus enabling the KNN to successfully recover the label with the reference model where it would have failed with a proper periphery crop that excludes the foreground object.

Label Inference Accuracy

1.0

0.5

0.5

uo %0 douo eme

correlated

u%00 duo me

correlated

0.8

accuracy on all examples

Share of total

forgotten

0.8

accuracy on all examples

0.4

forgotten

ref. model corelationbaseline

memorized

ref. model corrlation baseline

memorized

VICReg

.3

0.6

0.3

0.4

0.2

0.4

0.2

0.1

0.2

0.1

0.0

250

500

750

1000

0.0

250

500

750

1000

0.0

100

200

300

400

500

0.0

100

200

000

400

500

epochs

epochs

train set size (thousands)

train set size (thousands)

SimCLR

Label Inference Accuracy

1.0

0.8

0.6

ref. model correlation baseline

accuracy on top 20.0% conf.

accuracy on all examples

Share of total

0.3

0.4

0.5

forgotten

memorized

correlated

0.8

0.6

accuracy on top 20.0% conf.

ref. model corrlation baseline

accuracy on all examples

Share of total

0.5

0.3

correlated

forgotten

memorized

0.4

0.2

0.4

0.2

0.2

0.1

0.2

0.1

0.0-

250

500

750

1000

0.0

250

500

750

1000

0.0

100

200

300

400

500

0.0

100

200

300

400

500

epochs

epochs

train set size (thousands)

train set size (thousands)

Supervised

Label Inference Accuracy

1.0

0.8

accuiracy on top 20.0% conf.

accuracy on all examples

ref. model correlation baseline

0.5

0.4

correlated

forgotten

memorized

1.0

0.8

ref. model correlation baseline

accuracy on top 20.0% conf.

accuracy on all examples

Share of total

0.5

0.4

correlated

forgotten

memorized

0.6-

0.3

Label Inference

0.6

0.3

0.4

0.4

0.2

0.2

0.1

0.2

0.1

0.0

250

epochs

500

750

1000

0.0

250

epochs

500

750

1000

0.0

100

一

train set size (thousands)

200

300

400

500

0.0

100

train set size (thousands)

200

300

400

00S

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper thoroughly studies a particular undesired aspect of state-of-the-art self-supervised learning models: deja-vu memorization, where the models end up memorizing specific parts of the image instead of learning just a semantically meaningful association. The authors provide extensive empirical results to show when SSL models might memorize, and what aspects of training might minimize this effect.\n\nSTRENGTHS:\n* The authors introduce novel idea and testing methodology for SSL models\n* Provides comprehensive empirical results on different aspects of SSL modeling and training and memorization \n* Additional results on ablations and using different representations and aspects of the setup help understand the author’s proposed method better\n* Clear presentation of results and setup.\n\nWEAKNESSES:\n* Authors do not seem to talk about the impact of different augmentations on memorization, if any. Augmentations are very important for SSL performance, maybe they impact memorization?\n* It is hard to judge how confidently we can estimate the memorization if deploying. We compare to only one other reference SSL model. \n* The importance of using the generative model is not quite clear. From the result figures, the kNN images themselves seem to do enough.\n* Only shows using ImageNet data. No way to know if it will generalize to other datasets as well.\n\nQUESTIONS:\n* How do different augmentations used by SSLs impact memorization?\n* Is using just one other SSL model enough to give confident results of memorization? Should we include more reference SSL models to increase confidence?\n* How many samples would be needed in the X public dataset? Could be helpful to understand the scale if deployed.\n* When is the generative model actually needed, when is it necessary beyond just looking at the k nearest neighbors?\n\nLIMITATIONS:\n* The lack of a way to estimate the confidence or uncertainty in the memorization score.\n* Although shown for many models, the results are shown only for a single dataset. Using more datasets could have been nice, maybe using CelebA to show background info is memorized to learn person’s identity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows how self-supervised learning (SSL) models can memorize information about their training data in ways that are not intended. SSL is a technique for learning representations of unlabeled data by training a model to solve a \"pretext\" task; for example, we might take training images A, B, and C and crop them in different ways, producing sets of patches (A1,...,An), (B1,...,Bn), and (C1,...,Cn). The algorithm tries to find an embedding of these patches that makes Ai similar to Aj, while minimizing the similarity between Ai and Bj. These embeddings are used for a variety of downstream tasks.\n\nThis paper shows that the representations of training patches can contain surprising information about the training data. In the headlining example, the authors take an image of a black swan from the training data, extract a water-only patch, and produce an embedding of this patch. Then a generative model, which (ideally? see below) has no access to the training data, produces an image from this embedding: it contains a (different?) black swan. The water-only patch contained no information about swan, so the SSL model \"remembered\" what was in the image the patch came from.\n\nThe trick fails when the same patch is run through an SSL model that was trained on different data: it happens only because the patch was seen in the training data.\n\nThe paper has two methods for detecting such memorization. The first predicts the class label given the patch's representation. The second, mentioned above, visually inspects images generated from the representation. In both cases the classification/generation is done by a model trained on a different data set. The label-prediction method demonstrates quantitatively that the representation contains information about the label, while the visual inspection reveals finer details: the paper contains a great example of how SSL models can learn and memorize the distinction between European and American badgers, even though both share an ImageNet label.\n\nSTRENGTHS:\nIf the authors resolve my dataset question below, I am happy to call this a very clean and original demonstration on a significant and relavant topic. It is definitely a paper to generate good discussion.\n\nThe paper performs extensive experiments, showing how the results change as they turn various knobs. The quantitative and qualitative techniques complement each other nicely.\n\nThe work lays out clear directions for future work.\n\nWEAKNESSES:\nAs I understand from Appendices A.2.1 and A.2.3, \"private\" sets A and B come from ImageNet 1k, which is a subset of ImageNet. The \"public\" set X is a face-blurred version of ImageNet. This appears to contradict the experimental setup of Section 3.1 and, if it is the case, means I am not sure how to interpret the results. I could be mistaken and am open to raising my score. EDIT: the rebuttal adequately addressed these concerns.\n\nI was initially confused the use of the word \"reconstruction.\" For example, Figure 1 talks about \"reconstruction of an SSL training image,\" but the generated image looks quite different from the training image. It seems you use the word differently than recent work does [1-5], where it means producing an example that is pixel-by-pixel, or feature-by-feature, similar to the input. I suggest using a different word or clarifying the issue earlier in the paper.\n\nI think synthetic-data experiments might strengthen the paper. They might nail down specific aspects of this phenomenon. One might better probe the types of memorized information. (Of course, easier suggested than done.)\n\nI find the related work section adequate, but I think Section 3, \"Defining Deja Vu Memorization,\" would benefit from clearer connections between prior work and the presented definitions. In particular, the definition at line 111 has similarities to the notion of memorization used in [2], which also attempts to capture the idea of memorization of specific information about individual examples, beyond what one can infer from the data distribution. The definition for testing methodology, at line 130, is similar to the definition of memorization used in [6,7].\n\n[1] Balle, Borja, Giovanni Cherubin, and Jamie Hayes. \"Reconstructing training data with informed adversaries.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[2] Brown, Gavin, et al. \"When is memorization of irrelevant training data necessary for high-accuracy learning?.\" Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing. 2021.\n\n[3] Haim, Niv, et al. \"Reconstructing training data from trained neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 22911-22924.\n\n[4] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[5] Hayes, Jamie, Saeed Mahloujifar, and Borja Balle. \"Bounding Training Data Reconstruction in DP-SGD.\" arXiv preprint arXiv:2302.07225 (2023).\n\n[6] Feldman, Vitaly. \"Does learning require memorization? a short tale about a long tail.\" Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020.\n\n[7] Feldman, Vitaly, and Chiyuan Zhang. \"What neural networks memorize and why: Discovering the long tail via influence estimation.\" Advances in Neural Information Processing Systems 33 (2020): 2881-2891.\n\nQUESTIONS:\nSection 3.1 says the datasets A, B, and B are disjoint. Appendix A.2.1 says that A and B overlap. Furthermore, as mentioned above, my understanding is that X is a face-blurred version of ImageNet, and thus X might contain (possibly blurred) examples from A and B. Can you clarify this?\n\nCould you lay out what the \"adversary,\" the person conducting the privacy attack, has access to?\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSelf-Supervised Learning algorithms provide a label-free alternative for representation learning and is widely used in vision, and language. In this work, the authors try to understand the information content of representations learned by SSL algorithms through the lens of memorization. In particular, they define a \"deja-vu\" memorization notion that measures how much a given sample from the training set can be reconstructed from a pretrained model. \n\nThe authors focus on joint-embedding-based approaches, exploring the extent of such memorization during pretraining by evaluating label-inference accuracy on peripheral crops of a subset of training images ($A$). A \"deja-vu\" score at p% confidence measures the gap in accuracy between a target model (trained on $A$) and a reference model (trained on a different subset $B$). The gap essentially measures the extent to which exposing particular samples during training makes it easier to predict the label from partial background information.\n\nBuilding on these metrics, the authors present interesting results comparing different SSL algorithms under different training configurations. Notably, long pretraining increases the risk of memorization, while the size of the dataset doesn't is as consequential (Fig 4/5). With a sequence of ablations, the authors establish empirical findings and provide design recommendations that minimize deja-vu memorization while maintaining high performance.\n\nSTRENGTHS:\nSome strengths of the work are outlined below:  \n**Originality**: The authors examine the representation quality in SSL from a privacy perspective. The work focuses on train-time characteristics and memorization for non-generative SSL algorithms, in line with similar works in language/generative models. The authors propose a novel leave-one-out mechanism for evaluating the extent of memorization (compared to correlation) of specific training-set examples across SSL algorithms.\n\n**Significance**: Understanding the information content of representations learned by black-box SSL algorithms is important as these algorithms become widely used in safety-critical applications. In particular, the paper makes a solid case for understanding the privacy risk induced by long-pretraining routines in existing SSL algorithms. \n\n**Quality**: The work is well-motivated and grounded in real-world challenging questions related to measuring the quality of representations learned in SSL algorithms.  \n\n**Clarity**: The paper is well written, with definitions and pedagogical examples that motivate the core insights of the work. The authors include clear, well-labeled figures and plots accompanying their observations.\n\n**Reproducibility**: All experiments are performed on public datasets and pretrained models and sufficient details to reproduce the core results of the paper. The authors are encouraged to release the accompanying code at their earliest convenience.\n\nWEAKNESSES:\nSome weaknesses of the work are outlined below:\n\n**Explainability**: The authors have shared some interesting empirical findings about the effectiveness of SSL algorithms in representing data and their ability to recall examples from the training set from partial side information. Although the authors demonstrate that established methods like guillotine regularization reduce the risk of memorization, the current version of the study could benefit from more discussion on the underlying mechanisms behind these mitigation techniques.\n\nQUESTIONS:\n1. Although the authors primarily concentrate on joint-embedding-based techniques, it is probable that reconstruction-based methods are also susceptible to this type of memorization. To enable a fair comparison across the SSL algorithm family, it would be useful to include a baseline such as MAE.nclude a baseline such as MAE.\n2. The focus of the authors is on the linear probe as a potential measure of representation quality. However, they have shown that this metric is inadequate in capturing the full extent of deja-vu memorization. It would be valuable to determine if this inconsistency still exists even when the model is fine-tuned, which involves adapting the backbone as well, in an end-to-end approach.\n\nLIMITATIONS:\nThe paper explores the privacy-risk profile of SSL algorithms through the lens of training-set memorization. The authors are strongly encouraged to discuss their proposed algorithm's limitations (including assumptions and failure modes).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates to what degree self-supervised models exhibit memorization of particular samples in the training set. The authors demonstrate that a simple crop of an image, only comprised of the background, is enough for an adversary to learn the associated class label of the foreground object. Moreover, a conditional generative model can be trained, allowing the adversary to reproduce a very similar-looking image compared to the original. Such memorization capacity can lead to privacy issues and the authors demonstrate that such behaviour occurs in a broad range of models, while at the same time, this effect is not visible in common metrics such as probing accuracy.\n\nSTRENGTHS:\n1. The paper is very well written and builds upon the simple but very interesting idea of memorization in self-supervised models. Such an investigation seems timely, given the similar recent efforts in large language models. \n2. The experiments are very well designed, clearly disentangling the notion of correlation (i.e. actually learning something generalisable about the sample) and pure memorization (i.e. only memorizing the specifics of the sample). The splitting of the training data into two sets and training two separate models is a very neat way to measure this effect, which would otherwise remain very difficult to disentangle.\n3. It is very interesting that supervised and self-supervised methods behave very differently in terms of memorization. In general, studying the difference in the two learning approaches is very worthwile and timely.\n\nWEAKNESSES:\n1. I struggle to really see the privacy concern here. How likely is it really for someone to have access to only a crop of an image, while not being able to access the full image? The authors present the example of an image of a face of a person, which could then be potentially used to infer where that person is. But again, in what situation do I have access to the face of an image without the rest of it? As the authors also demonstrate, such an image has to be present in the training data, it is not enough to simply use another photo of a person and then the model reveals another image of the same person. ﻿I also believe that this scenario is different from the language model case, where simply “guessing” a good query as input for the model to complete seems to be more realistic due to the smaller search space and potential knowledge of similar queries. I hope the authors can comment on this.\n2. This is more minor but while I really like the experimental setup, I’m still not sure if memorization and generalization are completely disentangled. Assume that you have a model that clearly memorizes images but also performs very poorly at the downstream task, i.e. its linear probe is around random guessing. Due to the complete lack of generalization, it will be tough to establish memorization since the nearest neighbours will all share no correlation. So from a pure theoretical perspective, this seems a bit unsatisfying. Of course in practice, no generalization is usually not the case, so from a practical viewpoint this makes complete sense.\n\nQUESTIONS:\nWhat happens if you perform the nearest neighbour class inference on the training set (excluding the image itself)? This would circumwent the issue outlined above, but of course defeats the practicality of the method. Still seems like an interesting experiment though.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}