SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Casey Meehan\*,1, Florian Bordes\*,2,3, Pascal Vincent2,3, Kamalika Chaudhuri†,1,3, Chuan Guo†,3
$^{1}$ UCSD,  $^{2}$ Mila, Université de Montréal,  $^{3}$ FAIR, Meta
*Equal contribution, †Equal direction contribution
cmeehan@eng.ucsd.edu,florian.bordes@umontreal.ca,
{pascal, kamalika, chuanguo}@meta.com
Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models—which we refer to as déjà vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that deja vu memorization is common to different SSL algorithms, is exacerbated by certain hyperparameter choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of deja vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies.
Self-supervised learning (SSL) [11; 12; 33; 2; 9; 20] aims to learn general representations of content-rich data without explicit labels by solving a pretext task. In many recent works, such pretext tasks rely on joint-embedding architectures whereby randomized image augmentations are applied to create multiple views of a training sample, and the model is trained to produce similar representations for those views. When using cropping as random image augmentation, the model learns to associate objects or parts (including the background scenery) that co-occur in an image. However, doing so also arguably exposes the training data to higher privacy risk as objects in training images can be explicitly memorized by the SSL model. For example, if the training data contains the photos of individuals, the SSL model may learn to associate the face of a person with their activity or physical location in the photo. This may allow an adversary to extract such information from the trained model for targeted individuals.
In this work, we aim to evaluate to what extent SSL models memorize the association of specific objects in training images or the association of objects and their specific backgrounds, and whether this memorization signal can be used to reconstruct the model's training samples. Our results demonstrate that SSL models memorize such associations beyond simple correlation. For instance, in Figure 1 (left), we use the SSL representation of a training image crop containing only water and this enables us to reconstruct the object in the foreground with remarkable specificity—in this case a black swan. By contrast, in Figure 1 (right), when using the crop from the background of a test set image that the SSL model has not seen before, its representation only contains enough information to
Figure 1: Left: Reconstruction of an SSL training image from a crop containing only the background. The SSL model memorizes the association of this specific patch of water (pink square) to this specific foreground object (a black swan) in its embedding, which we decode to visualize the full training image. Right: The reconstruction technique fails on a public test image that the SSL model has not seen before.
infer, through correlation, that the foreground object was likely some kind of waterbird — but not the specific one in the image.
Figure 1 shows that SSL models suffer from the unintended memorization of images in their training data—a phenomenon we refer to as déjà vu memorization<sup>1</sup> Beyond visualizing déjà vu memorization through data reconstruction, we also design a series of experiments to quantify the degree of memorization for different SSL algorithms, model architectures, training set size, etc. We observe that deja vu memorization is exacerbated by the atypically large number of training epochs often recommended in SSL training, as well as certain hyperparameters in the SSL training objective. Perhaps surprisingly, we show that deja vu memorization occurs even when the training set is large—as large as half of ImageNet [13]—and can continually worsen even when standard techniques for evaluating learned representation quality (such as linear probing) do not suggest increased overfitting. Our work serves as the first systematic study of unintended memorization in SSL models and motivates future work on understanding and preventing this behavior. Specifically, we:
- Elucidate how SSL representations memorize aspects of individual training images, what we call déjà vu memorization;
- Design a novel training data reconstruction pipeline for non-generative vision models. This is in contrast to many prominent reconstruction algorithms like [7; 8], which rely on the model itself to generate its own memorized samples and is not possible for SSL models or classifiers;
- Propose metrics to quantify the degree of {_dejà vu_} memorization committed by an SSL model. This allows us to observe how {_dejà vu_} changes with training epochs, dataset size, training criteria, model architecture and more.
Self-supervised learning (SSL) is a machine learning paradigm that leverages unlabeled data to learn representations. Many SSL algorithms rely on joint-embedding architectures (e.g., SimCLR [11], Barlow Twins [33], VICReg [2] and Dino [10]), which are trained to associate different augmented views of a given image. For example, in SimCLR, given a set of images  $\mathcal{A} = \{A_1,\dots ,A_n\}$  and a randomized augmentation function aug, the model is trained to maximize the cosine similarity of draws of SSL  $(\mathrm{aug}(A_i))$  with each other and minimize their similarity with SSL  $(\mathrm{aug}(A_j))$  for  $i\neq j$ . The augmentation function aug typically consists of operations such as cropping, horizontal flipping, and color transformations to create different views that preserve an image's semantic properties.
SSL representations. Once an SSL model is trained, its learned representation can be transferred to different downstream tasks. This is often done by extracting the representation of an image from
1The French loanword déjà vu means 'already-seen', just as an image is seen and memorized in training.
the backbone model $^2$  and either training a linear probe on top of this representation or finetuning the backbone model with a task-specific head [3]. It has been shown that SSL representations encode richer visual details about input images than supervised models do [4]. However, from a privacy perspective, this may be a cause for concern as the model also has more potential to overfit and memorize precise details about the training data compared to supervised learning. We show concretely that this privacy risk can indeed be realized by defining and measuring déjà vu memorization.
Privacy risks in ML. When a model is overfit on privacy-sensitive data, it memorizes specific information about its training examples, allowing an adversary with access to the model to learn private information [30; 16]. Privacy attacks in ML range from the simplest and best-studied membership inference attacks [26; 25; 24] to attribute inference [17; 22; 21] and data reconstruction [7; 1; 19] attacks. In the former, the adversary only infers whether an individual participated in the training set. Our study of déja vu memorization is most similar to the latter: we leverage SSL representations of the training image background to infer and reconstruct the foreground object. In another line of work in the NLP domain [6; 7]: when prompted with a context string present in the training data, a large language model is shown to generate the remainder of string at test time, revealing sensitive text like home addresses. This method was recently extended to extract memorized images from Stable Diffusion [8]. We exploit memorization in a similar manner: given partial information about a training sample, the model is prompted to reveal the rest of the sample. In our case, however, since the SSL model is not generative, extraction is significantly harder and requires careful design.
3 Defining Déjà Vu Memorization
What is déjà vu memorization? At a high level, the objective of SSL is to learn general representations of objects that occur in nature. This is often accomplished by associating different parts of an image with one another in the learned embedding. Returning to our example in Figure 1, given an image whose background contains a patch of water, the model may learn that the foreground object is a water animal such as duck, pelican, otter, etc., by observing different images that contain water from the training set. We refer to this type of learning as correlation: the association of objects that tend to co-occur in images from the training data distribution.
A natural question to ask is "Can the reconstruction of the black swan in Figure 1 be reasoned as correlation?" The intuitive answer may be no, since the reconstructed image is qualitatively very similar to the original image. However, this reasoning implicitly assumes that for a random image from the training data distribution containing a patch of water, the foreground object is unlikely to be a black swan. Mathematically, if we denote by  $\mathcal{P}$  the training data distribution and  $A$  the image, then
$$
p _ {\text {c o r r}} := \mathbb {P} _ {A \sim \mathcal {P}} (\text {o b j e c t} (A) = \text {b l a c k s w a n} | \operatorname {c r o p} (A) = \text {w a t e r})
$$
is the probability of inferring that the foreground object is a black swan through correlation. This probability may be naturally high due to biases in the distribution  $\mathcal{P}$ , e.g., if  $\mathcal{P}$  contains no other water animal except for black swans. In fact, such correlations are often exploited to learn a model for image inpainting with great success [32; 27].
Despite this, we argue that reconstruction of the black swan in Figure 1 is not due to correlation, but rather due to unintended memorization: the association of objects unique to a single training image. As we will show in the following sections, the example in Figure 1 is not a rare success case and can be replicated across many training samples. More importantly, failure to reconstruct the foreground object in Figure 1 (right) on test images hints at inferring through correlation is unlikely to succeed—a fact that we verify quantitatively in Section 4.1. Motivated by this discussion, we give a verbal definition of déjà vu memorization below, and design a testing methodology to quantify déjà vu memorization in Section 3.1.
${}^{2}$  SSL methods often use a trick called guillotine regularization [3],which decomposes the model into two parts: a backbone model and a projector consisting of a few fully-connected layers. Such trick is needed to handle the misalignment between the pretext SSL task and the downstream task.
3We recognize that it is easier to find a context string that might have been in the training data of a large language models that finding the exact pixels that constitutes a crop of a training image. However, this paper focuses on revealing a memorization phenomena in SSL and does not aim to provide a complete picture of all the privacy risk that it might entails.
Figure 2: Overview of testing methodology. Left: Data is split into target set  $\mathcal{A}$ , reference set  $\mathcal{B}$  and public set  $\mathcal{X}$  that are pairwise disjoint.  $\mathcal{A}$  and  $\mathcal{B}$  are used to train two SSL models  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  in the same manner.  $\mathcal{X}$  is used for KNN decoding at test time. Right: Given a training image  $A_i \in \mathcal{A}$ , we use  $\mathrm{SSL}_A$  to embed  $\mathrm{crop}(A_i)$  containing only the background, as well as the entire set  $\mathcal{X}$  and find the  $k$ -nearest neighbors of  $\mathrm{crop}(A_i)$  in  $\mathcal{X}$  in the embedding space. These KNN samples can be used directly to infer the foreground object (i.e., class label) in  $A_i$  using a KNN classifier, to be visualized directly or their embeddings can be averaged as input to the trained RCDM to visually reconstruct the image  $A_i$ . For instance, the KNN visualization results in Figure 1 (left) when given  $\mathrm{SSL}_A(\mathrm{crop}(A_i))$  and results in Figure 1 (right) when given  $\mathrm{SSL}_A(\mathrm{crop}(B_i))$  for an image  $B_i \in \mathcal{B}$ .
Definition: A model exhibits déjà vu memorization when it retains information so specific to an individual training image, that it enables recovery of aspects particular to that image given a part that does not contain them. The recovered aspect must be beyond what can be inferred using only correlations in the data distribution.
We intentionally kept the above definition broad enough to encompass different types of information that can be inferred about the training image, including but not restricted to object category, shape, color and position. For example, if one can infer that the foreground object is red given the background patch with accuracy significantly beyond correlation, we consider this an instance of déjà vu memorization as well. We mainly focus on object category to quantify déjà vu memorization in Section 4 since the ground truth label can be easily obtained. We consider other types of information more qualitatively in the visual reconstruction experiments in Section 5.
Distinguishing memorization from correlation. When measuring déjà vu memorization, it is crucial to differentiate what the model associates through memorization and what it associates through correlation. Our testing methodology is based on the following intuitive definition.
Definition: If an SSL model associates two parts in a training image, we say that it is due to correlation if other SSL models trained on a similar dataset from  $\mathcal{P}$  without this image would likely make the same association. Otherwise, we say that it is due to memorization.
Notably, such intuition forms the basis for differential privacy (DP; Dwork et al. [15]; Dwork & Roth [14])—the most widely accepted notion of privacy in ML.
3.1 Testing Methodology for Measuring Déjà Vu Memorization
In this section, we use the above intuition to measure the extent of déjà vu memorization in SSL. Figure 2 gives an overview of our testing methodology.
Dataset splitting. We focus on testing déjà vu memorization for SSL models trained on the ImageNet dataset [13] $^4$ . Our test first splits the ImageNet training set into three independent and disjoint subsets  $\mathcal{A}$ ,  $\mathcal{B}$  and  $\mathcal{X}$ . The dataset  $\mathcal{A}$  is called the target set and  $\mathcal{B}$  is called the reference set. The two datasets are used to train two separate SSL models,  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ , called the target model and the reference model. Finally, the dataset set  $\mathcal{X}$  is used as an auxiliary public dataset to extract information from  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ . Our dataset splitting serves the purpose of
4We used the face-blurred version of ImageNet for privacy purposes.
distinguishing memorization from correlation in the following manner. Given a sample  $A_{i} \in \mathcal{A}$ , if our test returns the same result on  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  then it is likely due to correlation because  $A_{i}$  is not a training sample for  $\mathrm{SSL}_B$ . Otherwise, because  $\mathcal{A}$  and  $\mathcal{B}$  are drawn from the same underlying distribution, our test must have inferred some information unique to  $A_{i}$  due to memorization. Thus, by comparing the difference in the test results for  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ , we can measure the degree of déjà vu memorization<sup>5</sup>.
Extracting foreground and background crops. Our testing methodology aims at measuring what can be inferred about the foreground object in an ImageNet sample given a background crop. This is made possible because ImageNet provides bounding box annotations for a subset of its training images—around 150K out of 1.3M samples. We split these annotated images equally between  $\mathcal{A}$  and  $\mathcal{B}$ . Given an annotated image  $A_{i}$ , we treat everything inside the bounding box as the foreground object associated with the image label, denoted object  $(A_{i})$ . We take the largest possible crop that does not intersect with any bounding box as the background crop (or periphery crop), denoted  $\mathrm{crop}(A_i)^6$
KNN-based test design. Joint-embedding SSL approaches encourage the embeddings of random crops of a training image  $A_{i} \in \mathcal{A}$  to be similar. Intuitively, if the model exhibits déjà vu memorization, it is reasonable to expect that the embedding of  $\mathrm{crop}(A_i)$  is similar to that of object  $(A_{i})$  since both crops are from the same training image. In other words,  $\mathrm{SSL}_A(\mathrm{crop}(A_i))$  encodes information about object  $(A_{i})$  that cannot be inferred through correlation. However, decoding such information is challenging as these approaches do not learn a decoder associated with the encoder  $\mathrm{SSL}_A$ .
Here, we leverage the public set  $\mathcal{X}$  to decode the information contained in  $\operatorname{crop}(A_i)$  about object  $(A_i)$ . More specifically, we map images in  $\mathcal{X}$  to their embeddings using  $\mathrm{SSL}_A$  and extract the  $k$ -nearest-neighbor (KNN) subset of  $\mathrm{SSL}_A(\operatorname{crop}(A_i))$  in  $\mathcal{X}$ . We can then decode the information contained in  $\operatorname{crop}(A_i)$  in one of two ways:
- Label inference: Since  $\mathcal{X}$  is a subset of ImageNet, each embedding in the KNN subset is associated with a class label. If  $\operatorname{crop}(A_i)$  encodes information about the foreground object, its embedding will be close to samples in  $\mathcal{X}$  that have the same class label (i.e., foreground object category). We can then use a KNN classifier to infer the foreground object in  $A_i$  given  $\operatorname{crop}(A_i)$ .
- Visualization: Since we have access to a KNN subset associated to a given  $\text{crop}(A_i)$ , we can visualize directly the images associated to this subset. Then, we can infer through visualizing what is common within this subset, what information can be retrieved for this single crop. In addition, to simplify the visualization pipeline and to map directly a given crop representation to an image, we train an RCDM [4]—a conditional generative model—on  $\mathcal{X}$  to decode  $\text{SSL}_A$  embeddings into images. The RCDM reconstruction can recover qualitative aspects of an image remarkably well, such as recovering object color or spatial orientation using its SSL embedding. Given the KNN subset, we average their SSL embeddings and use the RCDM model to visually reconstruct  $A_i$ .
In Section 4, we focus on quantitatively measuring déjà vu memorization with label inference, and then use the KNN to visualize déjà vu memorization in Section 5.
4 Quantifying Déjà Vu Memorization
We apply our testing methodology to quantify a specific form of {_éjà vu_ memorization: inferring the foreground object (class label) given a crop of the background.
Extracting model embeddings. We test {_j vu memorization on a variety of popular SSL algorithms, with a focus on VICReg [2]. These algorithms produce two embeddings given an input image: a backbone embedding and a projector embedding that is derived by applying a small fully-connected network on top of the backbone embedding. Unless otherwise noted, all SSL embeddings refer to the projector embedding. To understand whether {_j vu memorization is particular to SSL, we also evaluate embeddings produced by a supervised model  $\mathrm{CLF}_A$  trained on  $\mathcal{A}$ . We apply the same set of image augmentations as those used in SSL and train  $\mathrm{CLF}_A$  using the cross-entropy loss to predict ground truth labels.
See Appendix A.2.1 for details on how the dataset splits are generated.
<sup>6</sup>We also present another heuristic in Appendix A.8 which takes a corner crop as the background crop, allowing our test to be run without bounding box annotations.
Identifying the most memorized samples. Prior works have shown that certain training samples can be identified as more prone to memorization than others [16; 28; 29]. Similarly, we provide a heuristic to identify the most memorized samples in our label inference test using confidence of the KNN prediction. Given a periphery crop,  $\mathrm{crop}(A_i)$ , let  $\mathrm{KNN}_A\big(\mathrm{crop}(A_i)\big) \subseteq \mathcal{X}$  denote its  $k$ -nearest neighbors in the embedding space of  $\mathrm{SSL}_A$ . From this KNN subset we can obtain: (1)  $\mathrm{KNN}_A^{\mathrm{prob}}\big(\mathrm{crop}(A_i)\big)$ , the vector of class probabilities (normalized counts) induced by the KNN subset, and (2)  $\mathrm{KNN}_A^{\mathrm{conf}}\big(\mathrm{crop}(A_i)\big)$ , the negative entropy of the probability vector  $\mathrm{KNN}_A^{\mathrm{prob}}\big(\mathrm{crop}(A_i)\big)$ , as confidence of the KNN prediction. When entropy is low, the neighbors agree on the class of  $A_i$  and hence confidence is high. We can sort the confidence score  $\mathrm{KNN}_A^{\mathrm{conf}}\big(\mathrm{crop}(A_i)\big)$  across samples  $A_i$  in decreasing order to identify the most confidently predicted samples, which likely correspond to the most memorized samples when  $A_i \in \mathcal{A}$ .
4.1 Population-level Memorization
Our first measure of déjà vu memorization is population-level label inference accuracy: What is the average label inference accuracy over a subset of SSL training images given their periphery crops? To understand how much of this accuracy is due to  $\mathrm{SSL}_A$ 's déjà vu memorization, we compare with a correlation baseline using the reference model:  $\mathrm{KNN}_B$ 's label inference accuracy on images  $A_i \in \mathcal{A}$ . In principle, this inference accuracy should be significantly above chance level (1/1000 for ImageNet) because the periphery crop may be highly indicative of the foreground object through correlation, e.g., if the periphery crop is a basketball player then the foreground object is likely a basketball. Figure 3 compares the accuracy of  $\mathrm{KNN}_A$  to that of  $\mathrm{KNN}_B$  when inferring the labels of images in  $A_i \in \mathcal{A}^7$  using  $\mathrm{crop}(A_i)$ . Results are shown for VICReg and the supervised model; trends for other models are shown in Appendix A.5. For both VICReg and supervised models, inferring the class of  $\mathrm{crop}(A_i)$  using  $\mathrm{KNN}_B$  (dashed line) through correlation achieves a reasonable accuracy that is significantly above chance level.
However, for VICReg, the inference accuracy using  $\mathrm{KNN}_A$  (solid red line) is significantly higher, and the accuracy gap between  $\mathrm{KNN}_A$  and  $\mathrm{KNN}_B$  indicates the degree of déjà vu memorization. We highlight two observations:
- The accuracy gap of VICReg is significantly larger than that of the supervised model. This is especially notable when accounting for the fact that the supervised model is trained to associate randomly augmented crops of images with their ground truth labels. In contrast, VICReg has no label access during training but the embedding of a periphery crop can still encode the image label.
- For VICReg, inference accuracy on the  $1\%$  most confident examples is nearly  $95\%$ , which shows that our simple confidence heuristic can effectively identify the most memorized samples. This result suggests that an adversary can use this heuristic to identify vulnerable training samples to launch a more focused privacy attack.