SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Accuracy of label inference using the target model (trained on  $\mathcal{A}$ ) vs. the reference model (trained on  $\mathcal{B}$ ) on the top  $\%$  most confident examples  $A_{i} \in \mathcal{A}$  using only  $\mathrm{crop}(A_{i})$ . For VICReg, there is a large accuracy gap between the two models, indicating a significant degree of déjà vu memorization.
The déjà vu score. The curves of Figure 3 show memorization across confidence values for a single training scenario. To study how memorization changes with different hyperparameters, we extract a single value from these curves: the deja vu score at confidence level  $p$ . In Figure 3, this is the gap between the solid red (or gray) and dashed red (or gray) where confidence ( $x$ -axis) equal  $p\%$ . In other words, given the periphery crops of set  $A$ ,  $\mathrm{KNN}_A$  and  $\mathrm{KNN}_B$  separately select and label their top  $p\%$  most confident examples, and we report the difference in their accuracy. The deja vu score captures both the degree of memorization by the accuracy gap and the ability to identify memorized examples by the confidence level. If the score is  $10\%$  for  $p = 33\%$ ,  $\mathrm{KNN}_A$  has  $10\%$
The sets  $\mathcal{A}$  and  $\mathcal{B}$  are exchangeable, and in practice we repeat this test on images from  $\mathcal{B}$  using  $\mathrm{SSL}_B$  as the target model and  $\mathrm{SSL}_A$  as the reference model, and average the two sets of results.
(a) déjà vu vs. epochs
(b) déja vu vs. train set size
Figure 4: Effect of training epochs and train set size on déjà vu score (red) in comparison with linear probe accuracy train-test gap (dark blue) for VICReg. Left: deja vu score increases with training epochs, indicating growing memorization while the linear probe baseline decreases significantly. Right: deja vu score stays roughly constant with training set size while linear probe gap shrinks significantly, suggesting that memorization may be problematic even for large datasets.
(a) déjà vu vs. epochs
Figure 5: Partition of samples  $A_{i}\in \mathcal{A}$  into the four categories: unassociated (not shown), memorized, misrepresented and correlated for VICReg. The memorized samples—those whose labels are predicted by  $\mathrm{KNN}_A$  but not by  $\mathrm{KNN}_B$ —occupy a significantly larger share of the training set than the misrepresented samples—those predicted by  $\mathrm{KNN}_B$  but not  $\mathrm{KNN}_A$  by chance.
(b) déjà vu vs. train set size
higher accuracy on its most confident third of  $\mathcal{A}$  than  $\mathrm{KNN}_B$  does on its most confident third. In the following, we set  $p = 20\%$ , approximately the largest gap for VICReg (red lines) in Figure 3.
Comparison with the linear probe train-test gap. A standard method for measuring SSL performance is to train a linear classifier—what we call a 'linear probe'—on its embeddings and compute its performance on a held out test set. From a learning theory standpoint, one might expect the linear probe's train-test accuracy gap to be indicative of memorization: the more a model overfits, the larger is the difference between train set and test set accuracy. However, as seen in Figure 4, the linear probe gap (dark blue) fails to reveal memorization captured by the déjà vu score (red)  $^{8}$ .
4.2 Sample-level Memorization
The déja vu score shows, on average, how much better an adversary can select and classify images when using the target model trained on them. This average score does not tell us how many individual images have their label successfully recovered by  $\mathrm{KNN}_A$  but not by  $\mathrm{KNN}_B$ . In other words, how many images are exposed by virtue of being in training set  $\mathcal{A}$ : a risk notion foundational to differential privacy. To better quantify what fraction of the dataset is at risk, we perform a sample-level analysis by fixing a sample  $A_i \in \mathcal{A}$  and observing the label inference result of  $\mathrm{KNN}_A$  vs.  $\mathrm{KNN}_B$ . To this end, we partition samples  $A_i \in \mathcal{A}$  based on the result of label inference into four distinct categories: Unassociated - label inferred with neither KNN; Memorized - label inferred only with  $\mathrm{KNN}_A$ ; Misrepresented - label inferred only with  $\mathrm{KNN}_B$ ; Correlated - label inferred with both KNNs.
Intuitively, unassociated samples are ones where the embedding of  $\text{crop}(A_i)$  does not encode information about the label. Correlated samples are ones where the label can be inferred from  $\text{crop}(A_i)$  using correlation, e.g., inferring the foreground object is basketball given a crop showing a basketball player. Ideally, the misrepresented set should be empty but contains a small portion of examples due to chance. Déjà vu memorization occurs for memorized samples where the embedding of  $\text{SSL}_B$  does not encode the label but the embedding of  $\text{SSL}_A$  does. To measure the pervasiveness of déjà vu memorization, we compare the size of the memorized and misrepresented sets. Figure 5 shows how the four categories of examples change with number of training epochs and training set size. The unassociated set is not shown since the total share adds up to one. The misrepresented set remains under 5% and roughly unchanged across all settings, consistent with our explanation that it is due to chance. In comparison, VICReg's memorized set surpasses 15% at 1000 epochs. Considering that up to 5% of these memorized examples could also be due to chance, we conclude that at least 10% of VICReg's training set is déjà vu memorized.
See section 6 for further discussion of the déjà vu score trends of Figure 4.
(a) A correlated dam example
(b) A memorized dam example
Figure 6: Correlated and Memorized examples from the dam class. Both  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  are SimCLR models. Left: The periphery crop (pink square) contains a concrete structure that is often present in images of dams. Consequently, the KNN can infer the foreground object using representations from both  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  through this correlation. Right: The periphery crop only contains a patch of water. The embedding produced by  $\mathrm{SSL}_B$  only contains enough information to infer that the foreground object is related to water, as reflected by its KNN set. In contrast, the embedding produced by  $\mathrm{SSL}_A$  memorizes the association of this patch of water with dam and the KNN select images of dams.
Figure 7: Visualization of déjà vu memorization beyond class label. Both  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  are VICReg models. The four images shown belong to the memorized set of  $\mathrm{SSL}_A$  from the badger class. KNN images using embeddings from  $\mathrm{SSL}_A$  can reveal not only the correct class label, but also the specific badger species: European (left) and American (right). Such information does not appear to be memorized by the reference model  $\mathrm{SSL}_B$ .
5 Visualizing Déjà Vu Memorization
Beyond enabling label inference using a periphery crop, we show that déjà vu memorization allows the SSL model to encode other forms of information about a training image. Namely, we leverage an external public dataset  $\mathcal{X}$  and use it to find the nearest neighborhoods in this public dataset given a training periphery crop. We aim to answer the following two questions: (1) Can we visualize the distinction between correlation and deja vu memorization? (2) What foreground object details can be extracted from the SSL model beyond class label?
Public image retrieval pipeline Following the pipeline in Figure 2, we use the projector embedding to find the KNN subset for the periphery crop,  $\text{crop}(A_i)$ , and visualize the images belonging to this KNN subset.
RCDM pipeline. RCDM is a conditional generative model that is trained on the backbone embedding of images  $X_{i} \in \mathcal{X}$  to generate an image that resembles  $X_{i}$ . At test time, following the pipeline in Figure 2, we first use the projector embedding to find the KNN subset for the periphery crop,  $\operatorname{crop}(A_i)$ , and then average their backbone embeddings as input to the RCDM model. Then, RCDM decodes this representation to visualize its content.
Visualizing Correlation vs. Memorization. Figure 6 shows examples of dams from the correlated set (left) and the memorized set (right) as defined in Section 4.2, along with the associated KNN set. In Figure 6a, the periphery crop is represented by the pink square, which contains concrete structure attached to the dam's main structure. As a result, both  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  produce embeddings of
(a) Loss hyper-parameter
(b) Guillotine regularization
Figure 8: Effect of two kinds of hyper-parameters on VICReg memorization. Left: déjà vu score (red) versus the invariance loss parameter,  $\lambda$ , used in the VICReg criterion (100k dataset). Larger  $\lambda$  significantly reduces déjà vu memorization with minimal effect on linear probe validation performance (green).  $\lambda = 25$  (near maximum deja vu) is recommended in the original paper. Right: deja vu score versus projector layer—guillotine regularization [3]—from projector to backbone. Removing the projector can significantly reduce deja vu. In Appendix A.7 we show that the backbone still can memorize.
(a) déjà vu vs. capacity
(b) déjà vu (DV) vs. Criterion
Figure 9: Effect of model architecture and criterion on déjà vu memorization. Left: deja vu score with VICReg for resnet (purple) and vision transformer (green) architectures versus number of model parameters. As expected, memorization grows with larger model capacity. This trend is more pronounced for convolutional (resnet) than transformer (ViT) architectures. Right: Comparison of deja vu score  $20\%$  conf. and ImageNet linear probe validation accuracy (P: using projector embeddings, B: using backbone embeddings) for various SSL criteria.
Criteria	DV	Acc P/B
Supervised	8.9	55.3/61.1
Byol[18]	8.0	54.3/59.4
SimCLR[11]	10.0	44.2/54.1
Dino[10]	14.5	26.3/55.7
Barlow T[33]	30.5	33.7/54.4
VICReg[2]	33.2	40.3/55.2
$\operatorname{crop}(A_i)$  whose KNN set in  $\mathcal{X}$  consists of dams, i.e., there is a correlation between the concrete structure in  $\operatorname{crop}(A_i)$  and the foreground dam. In Figure 6b, the periphery crop only contains a patch of water, which does not strongly correlate with dams in the ImageNet distribution. Evidently, the reference model  $\mathrm{SSL}_B$  embeds  $\operatorname{crop}(A_i)$  close to that of other objects commonly found in water, such as sea turtle and submarine. In contrast, the KNN set according to  $\mathrm{SSL}_A$  all contain dams despite the vast number of alternative possibilities within the ImageNet classes which highlight memorization in  $\mathrm{SSL}_A$  between this specific patch of water and the dam.
Visualizing Memorization Beyond Class Label. Figure 7 shows four examples of badgers from the memorized set. In all four images, the periphery crop (pink square) does not contain any indication that the foreground object is a badger. Despite this, the KNN set using  $\mathrm{SSL}_A$  consistently produces images of badgers, while the same does not hold for  $\mathrm{SSL}_B$ . More interestingly, the KNN using  $\mathrm{SSL}_A$  in Figure 7a all contain European badgers, while reconstructions in Figure 7b all contain American badgers, accurately reflecting the species of badger present in the respective training images. Since ImageNet-1K does not differentiate between these two species of badgers, our reconstructions show that SSL models can memorize information that is highly specific to a training sample beyond its class label<sup>9</sup>.
6 Mitigation of déjà vu memorization
We cannot yet make claims on why déjà vu occurs so strongly for some SSL training settings and not for others. To gain some intuition for future work, we present additional observations that shed light on which parameters have the most salient impact on déjà vu memorization.
Déjà vu memorization worsens by increasing number of training epochs. Figure 4a shows how déjà vu memorization changes with number of training epochs for VICReg. The training set size is fixed to 300K samples. From 250 to 1000 epochs, the déjà vu score (red curve) grows threefold: from under  $10\%$  to over  $30\%$ . Remarkably, this trend in memorization is not reflected by the linear probe gap (dark blue), which only changes by a few percent beyond 250 epochs.
Training set size has minimal effect on déjà vu memorization. Figure 4b shows how deja vu memorization responds to the model's training set size. The number of training epochs is fixed to 1000. Interestingly, training set size appears to have almost no influence on the deja vu score (red line), indicating that memorization is equally prevalent with a 100K dataset and a 500K dataset. This result suggests that deja vu memorization may be detectable even for large datasets. Meanwhile, the
See Appendix A.1 for additional visualization experiments.
standard linear probe train-test accuracy gap declines by more than half as the dataset size grows, failing to represent the memorization quantified by our test.
Training loss hyper-parameter has a strong effect. Loss hyper-parameters, like VICReg's invariance coefficient (Figure 8a) or SimCLR's temperature parameter (Appendix Figure 14a) significantly impact déjà vu with minimal impact on the linear probe validation accuracy.
Some SSL criteria promote stronger déjà vu memorization. Table 9b demonstrates that the degree of memorization varies widely for different training criteria. VICReg and Barlow Twins have the highest déjà vu scores while SimCLR and Byol have the lowest. With the exception of Byol, all SSL models have more déjà vu memorization than the supervised model. Interestingly, different criteria can lead to similar linear probe validation accuracy and very different degrees of déjà vu as seen with SimCLR and Barlow Twins. Note that low degrees of deja vu can still risk training image reconstruction, as exemplified by the SimCLR reconstructions in Figures 6 and 11.
Larger models have increased déjà vu memorization. Figure 9a validates the common intuition that lower capacity architectures (Resnet18/34) result in less memorization than their high capacity counterparts (Resnet50/101). We see the same trend for vision transformers as well.
Guillotine regularization can help reduce déjà vu memorization. Previous experiments were done using the projector embedding. In Figure 8b, we present how Guillotine regularization[3] (removing final layers in a trained SSL model) impacts déjà vu with VICReg<sup>10</sup>. Using the backbone embedding instead of the projector embedding seems to be the most straightforward way to mitigate déjà vu memorization. However, as demonstrated in Appendix A.7, backbone representation with low deja vu score can still be leveraged to reconstruct some of the training images.
Figure 10: Déjà vu score when fine-tuning a pretrained VICReg model for 20 epochs. We observe that by fine-tuning we significantly increase the classification performances.
A little bit of fine-tuning might help to reduce memorization A common strategy in SSL is to fine-tune the model to solve the downstream task. In Figure 10, we show how the Dejavu score changes when fine-tuning a pretrained VICReg model. This pretrained model was trained on the set  $\mathcal{A}$  for 1000 epochs and fine-tuned on a classification task on the set  $\mathcal{A}$  for 20 epochs (which can be seen in the x axis on the figure). Interestingly, the DejaVu score decreases significantly in the first finetuning epochs while the validation accuracy is increasing. However after 5 epochs, the DejaVu score is increasing and after 20 epochs become almost as high at the original value before fine-tuning. This behavior indicates that even fine-tuning might not help in reducing DejaVu memorization.
We defined and analyzed {_jà vu_ memorization, a notion of unintended memorization of partial information in image data. As shown in Sections 4 and 5, SSL models can largely exhibit {_jà vu_ memorization on their training data, and this memorization signal can be extracted to infer or visualize image-specific information. Since SSL models are becoming increasingly widespread as foundation models for image data, negative consequences of {_jà vu_ memorization can have profound downstream impact and thus deserves further attention. Future work should focus on understanding how {_jà vu_ emerges in the training of SSL models and why methods like Byol are much more robust to {_jà vu_ than VICReg and Barlow Twins. In addition, trying to characterize which data points are the most at risk of {_jà vu_ could be crucial to get a better understanding on this phenomenon.
Further experiments are available in Appendix A.7.
A.1 Additional reconstruction examples
The two reconstruction experiments of Section 5 are each exemplified within one class. However, we see strong reconstructions using  $\mathrm{SSL}_A$  in several classes, and similar experimental results. To demonstrate this, we repeat the experiment of Section 5 using the yellow garden spider class and the experiment of 5 using the aircraft carrier class.
Figure 11: Visualizing the distinction between déjà vu memorization and correlation in the yellow garden spider class. Left, we see the periphery crops of the ten 'most correlated' images: those where both  $\mathrm{KNN}_A$  and  $\mathrm{KNN}_B$  have high confidence. Seven of these crops clearly depict a stabilimentum: the signature zig-zag web pattern sewn by spiders of the argiope genus, thus revealing the concealed spider by correlation. Right, we see the periphery crops of the ten 'most memorized' images: those that have the highest confidence discrepancy between  $\mathrm{KNN}_A$  (high confidence) and  $\mathrm{KNN}_B$  (low confidence). Nearly all of these crops show generic blurred views of the background with no evidence of the foreground spider. Below, we show the public set nearest neighbors of the pink highlighted crop. We see that the target model  $(\mathrm{SSL}_A)$  can be used to find back the yellow garden spider spider in both the memorized and correlated cases. The reference model  $(\mathrm{SSL}_B)$  can only be used to find back this type of spider in the correlated case.
Selection of Memorized and Correlated Images: The images of Figure 6 and 11 were chosen methodically as follows.
Image selection: The 20 images of Figures 6 and 11 are selected deterministically using label inference accuracy and KNN confidence score. The 10 most correlated images are those images in the correlated set (both models infer label correctly) of  $\mathcal{A}$  with the highest confidence agreement between models  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ . To measure confidence agreement we take the minimum confidence of the two models. The 10 most memorized images are those images in the memorized set (only target model infers the label correctly) of  $\mathcal{A}$  with the highest confidence difference between models  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ .
Class selection: To find classes with a high degree of déjà vu, classes were sorted by the label inference accuracy gap between the target and reference model. We selected the class based on a handful of criteria. First, we prioritized classes without images of human faces, thereby removing classes like 'basketball', 'bobsled', 'train station', and even 'tench' which is a fish often depicted in the hands of a fisherman. Second, we prioritized classes that include at least ten images with a high confidence difference between the target and reference models ('most memorized' images described above) and at least ten images with high confidence agreement ('most correlated' images described above). This led us to the dam and yellow garden spider classes.
(a) extracting side of ship from VICReg
Figure 12: Visualization of déjà vu memorization beyond class label. Both  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$  are VICReg models. The four images shown belong to the memorized set of  $\mathrm{SSL}_A$  from the aircraft carrier class. KNN visualization using embeddings from  $\mathrm{SSL}_A$  can reveal not only the correct class label, but also the orientation of the ship: the side of the ship (left) and the front of the ship (right) given only a generic crop of the background sky and/or water. Such information does not appear to be memorized by the reference model  $\mathrm{SSL}_B$ .
(b) extracting front of ship from VICReg
reconstructions
Selection of Beyond-Label-Inference Images: The images of Figure 7 and 12 were chosen methodically as follows.
Image selection: The four images of Figures 7 and 12 are selected using KNN confidence score, and, necessarily, hand picked selection for unlabeled features. Within a given class, we look at the top 40 images with highest target model KNN confidence scores. We then filter through these images to identify a distinguishable feature like different species within the same class or different object positions within the same class. This step is necessary because we are looking for features that are not labeled by ImageNet. We then choose two of these top 40 with one feature (e.g. American badger) and two with the alternative feature (e.g. European badger).
Class selection: To find classes with a high degree of déjà vu, classes were sorted by the target model's top-40 KNN confidence values within each class. As in the memorization vs. correlation experiment, we prioritized classes without images of human faces.
A.2 Details on the experimental setup
A.2.1 Details on dataset splits
There are 1,281,167 training images in the ImageNet dataset. Within these images, only 456,567 of them have bounding boxes annotations (which are needed to compute the Deja Vu score). The private set  $\mathcal{A}$  and  $\mathcal{B}$  are sampled from these 456,567 bounding boxes annotated images in such a way that set  $\mathcal{A}$  and  $\mathcal{B}$  are disjoint.
If we remove the 456,567 bounding boxes annotated images from the 1,281,167 training images, we get 824,600 remaining images without annotations which never overlap with  $\mathcal{A}$  or  $\mathcal{B}$ . From this set of 800K images, we took 500k images as our public set  $X$ . So now, we have three non-overlapping sets  $\mathcal{A}, \mathcal{B}$ , and  $X$ . Then, if we remove the 500K public set images from the 824,600 images without annotations, it leaves us with 324,600 images that are neither in  $\mathcal{A}$ ,  $\mathcal{B}$  or  $X$ . For simplicity, let us call this set of remaining 324,600 images the set  $\mathcal{C}$ . Then, we have split the entire ImageNet training set into four non-overlapping splits called  $\mathcal{A}, \mathcal{B}, \mathcal{C}$  and  $X$ .
When running our experiments with a small number of training images, we only use the set  $\mathcal{A}$  to train  $\mathrm{SSL}_A$  and the set  $\mathcal{B}$  to train  $\mathrm{SSL}_B$  and then use the set  $X$  as a public set for evaluation. However, to run larger scale experiments, we use as additional training data for  $\mathrm{SSL}_A$  and  $\mathrm{SSL}_B$ : the images sampled from the set  $\mathcal{C}$ . Here,  $\mathrm{SSL}_A$  will still be trained on set  $\mathcal{A}$  but it will be augmented with images from set  $\mathcal{C}$ . The same goes for  $\mathrm{SSL}_B$  which will still be trained on the set  $\mathcal{B}$  but augmented with images from the set  $\mathcal{C}$ . As such, some images sampled from  $\mathcal{C}$  to train  $\mathrm{SSL}_A$  or to train  $\mathrm{SSL}_B$  might overlap. However, this is not an issue since the evaluation is done using only images from the bounding boxes annotated set  $\mathcal{A}$  and set  $\mathcal{B}$  which are never overlapping.