SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To identify memorization, our tests only attempt to infer the labels of the unique examples  $\overline{A}$  and  $\overline{B}$  that differentiate the two private sets. The periphery crop,  $\operatorname{crop}(A_i)$ , is computed as the largest possible crop that does not intersect with the foreground object bounding box. In some instances the largest periphery crop is small, and not high enough resolution to get a meaningful embedding. To circumvent this, we only run the test on bounding box examples where the periphery crop is at least  $100 \times 100$  pixels.
Each size of training set,  $100\mathrm{k}$  to  $500\mathrm{k}$ , includes an equal number of examples per class in both sets  $\mathcal{A}$  and  $\mathcal{B}$ . The total bounding box annotated examples of each class are evenly divided between  $\overline{\mathcal{A}}$  and  $\overline{\mathcal{B}}$ . The remaining examples in each class are the examples from  $\mathcal{C}$ . We reiterate that the bounding box examples in set  $\mathcal{A}$  are unique to set  $\mathcal{A}$ , and thus can only be memorized by  $\mathrm{SSL}_A$ .
The disjoint public set,  $X$ , contains ground truth labels but no bounding-box annotations. The size and content of  $X$  remains fixed for all tests.
A.2.2 Details on the training setup
Model Training: We use PyTorch [23] with FFCV-SSL [5]. All models are trained for 1000 epochs with model checkpoints taken at 50, 100, 250, 500, 750, and 1000 epochs. We note that 1000 epochs is used in the original papers of both VICReg and SimCLR. All sweeps of epochs use the 300k dataset. All sweeps of datasets use the final, 1000 epoch checkpoint. We use a batch size of 1024, and LARS optimizer [31] for all SSL models. All models use Resnet101 for the backbone. As seen in Appendix A.6, a Resnet50 backbone results in déjà vu consistent with that of Resnet101.
VICReg Training: VICReg is trained with the 3-layer fully connected projector used in the original paper with layer dimensions 8192-8192-8192. The invariance, variance, and covariance parameters are set to  $\lambda = 25$ ,  $\mu = 25$ ,  $\nu = 1$ , respectively, which are used in the original paper [2]. The LARS base learning rate is set to 0.2, and weight decay is set to 1e-6.
SimCLR Training: SimCLR is trained with the 2-layer fully connected projector used in the original paper with layer dimensions 2048-256. The temperature parameter is set to  $\tau = 0.15$ . The LARS base learning rate is set to 0.3, and weight decay is set to 1e-6.
Supervised Training: Unlike the SSL models, the supervised model is trained with label access using cross-entropy loss. To keep architectures as similar as possible, the supervised model also uses a Resnet101 backbone and the same projector as VICReg. A final batchnorm, ReLU, and linear layer
is added to bring the 8192 dimension projector output to 1000-way classification activations. We use these activations as the supervised model's projector embedding. The supervised model uses the LARS optimizer with learning rate 0.2.
A.2.3 Details on the evaluation setup
KNN: For each test, we build two KNN's: one using the target model,  $\mathrm{SSL}_A$  (or  $\mathrm{CLF}_A$ ), and one using the reference model  $\mathrm{SSL}_B$  (or  $\mathrm{CLF}_B$ ). As depicted in Figure 2, each KNN is built using the projector embeddings of all images in the public set  $\mathcal{X}$  as the neighbor set. When testing for memorization on an image  $A_i \in \mathcal{A}$ , we first embed  $\mathrm{crop}(A_i)$  using  $\mathrm{SSL}_A$ , and find its  $K = 100L_2$  nearest neighbors within the  $\mathrm{SSL}_A$  embeddings of  $\mathcal{X}$ . See section A.4 for a discussion on selection of  $K$ . We then take the majority vote of the neighbors' labels to determine the class of  $A_i$ . This entire pipeline is repeated using reference model  $\mathrm{SSL}_B$  and its KNN to compute reference model accuracy.
In practice, all of our quantitative tests are repeated once with  $\mathrm{SSL}_A$  as the target model (recovering labels of images in set  $\mathcal{A}$ ) and again with  $\mathrm{SSL}_B$  as the target model (recovering labels of images in set  $\mathcal{B}$ ). All results shown are the average of these two tests. Throughout the paper, we describe  $\mathrm{SSL}_A$  as the target model and  $\mathrm{SSL}_B$  as the reference model for ease of exposition.
RCDM: The RCDM is trained on a face-blurred version of ImageNet [13] and is used to decode the SSL backbone embedding of an image back into an approximation of the original image. All RCDMs are trained on the public set of images  $\mathcal{X}$  used for the KNN. A separate RCDM must be trained for each SSL model, since each model has a unique mapping from image space to embedding space.
At inference time, the RCDM is used to reconstruct the foreground object given only the periphery cropping. To produce this reconstruciton, the RCDM needs an approximation of the backbone embedding of the original image. The backbone of image  $A_{i}$  is approximated by 1) computing crop embedding  $\mathrm{SSL}_{A}^{\mathrm{proj}}(\mathrm{crop}(A_{i}))$ , 2) finding the five public set nearest neighbors of the crop embedding, and 3) averaging the five nearest neighbors' backbone embeddings. In practice, these public set nearest neighbors are often a very good approximation of the original image, capturing aspects like object class, position, subspecies, etc..
A.3 Sample-level memorization
(a) Categories of training samples vs. number of epochs
(b) Categories of training samples vs. training set size
Figure 13: Partition of samples  $A_{i} \in \mathcal{A}$  into the four categories: unassociated (not shown), memorized, misrepresented and correlated. The memorized samples—ones whose labels are predicted by  $\mathrm{KNN}_A$  but not by  $\mathrm{KNN}_B$ —occupy a significantly larger share for VICReg compared to the supervised model, indicating that sample-level déjà vu memorization is more prevalent in VICReg.
(a) SimCLR
Figure 14: Effect of SSL hyperparameter on déjà vu memorization. The left plot of Figures 14a and 14b show the size of the memorized set as a function of the temperature parameter for SimCLR and invariance parameter for VICReg, respectively. Déjà vu memorization is the highest within a narrow band of hyperparameters, and one can mitigate against déjà vu memorization by selecting hyperparameters outside of this band. Doing so has negligible effect on the quality of SSL embeddings as indicated by the linear probe accuracy on ImageNet validation set.
(b) VICReg
Many SSL algorithms contain hyperparameters that control how similar the embeddings of different views should be in the training objective. We show that these hyperparameters directly affect déjà vu memorization. Figure 14 shows the size of the memorized set for SimCLR (left) and VICReg (right) as a function of their respective hyperparameters,  $\tau$  and  $\lambda$ . We observe that the memorized set is largest within a relatively narrow band of hyperparameter values, indicating strong déjà vu memorization. By selecting hyperparameters outside this band,ngr vu memorization sharply decreases while the linear probe validation accuracy on ImageNet remains roughly the same.
A.4 Selection of  $K$  for KNN
In this section, we describe the impact of  $K$  on the KNN label inference accuracy.
(a) VICReg, Accuracy
(b) VICReg, Share of memorized examples
Figure 15: Impact of  $K$  on label inference accuracy for target and reference models. Left: the population-level label inference accuracy experiment of Section 4.1 on VICReg vs.  $K$ . Right: the individualized memorization test of Section 4.2 on VICReg vs.  $K$ . In both cases, we see that our tests are relatively robust to choice of  $K$  beyond  $K = 50$ .
Figure 15 above shows how the tests of Section 4 change with number of public set nearest neighbors  $K$  used to make label inferences. Both tests are relatively robust to any choice of  $K$ . Results are shown on VICReg trained for 1k epochs on the 300k dataset. We see that any choice of  $K$  greater than 50 and less than the number of examples per class (300, in this case) appears to have good performance. Since our smallest dataset has 100 images per class, we chose to set  $K = 100$  for all experiments.
A.5 Effect of SSL criteria
We repeat the quantitative memorization tests of Section 4 on different models: VICReg[2], Barlow-Twins[33], Dino[10], Byol[18], SimCLR[34] and a supervised model in Figure 16. We observe differences between SSL training criteria with respect to déjà vu memorization. The easy ones to attack are VICReg and Barlow Twins whereas SimCLR and Byol are more robust to these attacks. While the degree of memorization appears to be reduced for SimCLR compared with VICReg, it is still stronger than the supervised baseline.
Figure 16: Comparison of déjà vu memorization for VICReg, Barlow Twins, Dino, Byol, SimCLR, and a supervised model. All tests are described in Section 4. We are showing deja vu vs. number of training epochs. We see that SimCLR (center row) shows less deja vu than VICReg, yet marginally more than the supervised model. Even with this reduced degree of memorization, we are able to produce detailed reconstructions of training set images, as shown in Figures 6 and 11.
A.6 Effect of Model Architecture and Complexity
Results shown in the main paper use Resnet101 for the model backbone. To understand the relationship between déjà vu and overparameterization, we compare with the smaller Resnet50 and Resnet18 in Figure 17. Overall, we find that increasing the number of parameters of the model leads to higher degree of déjà vu memorization. The same trend holds when using Vision Transformers (VIT-Tiny, -Small, -Base, and -Large with patch size of 16) of various sizes as the SSL backbone, instead of a Resnet. This highlights thatngr vu memorization is not unique to convolution architectures.
Figure 17: Comparison of VICReg {_jà vu memorization for different architectures and model sizes. On the left, we present {_ja vu memorization using VIT architectures (from vit-tiny in the first row to vit-base in the last row). On the right, we use Resnet based architectures (from resnet18 in the first row to resnet101 in the last row). All tests are described in Section 4, with the plots showing {_ja vu vs. number of training epochs. Reducing model complexity from Resnet101 to Resnet18 or from Vit-Large to Vit-tiny has a significant impact on the degree of memorization.
A.7 The impact of Guillotine Regularization on Deja Vu
In our experiments, we show déjà vu using the projector representation. The SSL loss directly incentivizes the projector representation to be invariant to random crops of a particular image. As such, we expect the projector to be the most overfit and produce the strongest déjà vu. Here, we study whether earlier representations between the projector and backbone exhibit less déjà vu memorization. This phenomenon – ‘guillotine regularization’ – has recently been studied from the perspective of generalization in Bordes et al. [3]. Here, we study it from the perspective of memorization.
To show how guillotine regularization impacts déjà vu, we repeat the tests of Section 4 on each layer of the VICReg projector: the 2048-dimension backbone (layer 0) up to the projector output (layer 3). We evaluate whether memorization is indeed reduced for the more regularized layers between the projector output and the backbone.
(a) population level accuracy
(b) share of memorized examples
Figure 18: déjà vu memorization versus layer from backbone (0) to projector output (3). The memorization tests of Section 4 are evaluated at each level of the VICReg projector. We see that déjà vu is significantly stronger closer to the projector output and nearly zero near the backbone. Interestingly, most memorization appears to occur in the final two layers of VICReg.
Figure 18 shows how guillotine regularization significantly reduces the degree of memorization in VICReg. The vast majority of VICReg's déjà vu appears to occur in the final two layers of the projector (2,3): in earlier layers (0,1), the label inference accuracy of the target model and reference model are comparable. This suggests that – like the hyperparameter selection of Section 6 – guillotine regularization can also significantly mitigate déjà vu memorization. In the following, we extend this result to SimCLR and supervised models by measuring the degree of déjà vu in the backbone (layer 0) versus training epochs and dataset size.
Comparison of {_jà vu in projector and backbone vs. epochs and dataset size Since the backbone is mostly used at inference time, we now evaluate how much {_jà vu exists in the backbone representation for VICReg and SimCLR. We repeat the tests of Section 4 versus training epochs and train set size.
(a) VICReg
Figure 19 shows that, indeed, déjà vu is significantly reduced in the backbone representation. For SimCLR, however, we see that backbone memorization is comparable with projector memorization. In light of the Guillotine regularization results above, this makes some sense since SimCLR uses fewer layers in its projector. Given that we were able to generate accurate reconstructions with the SimCLR projector (see Figures 11 and 6), we now evaluate whether we can produce accurate reconstructions of training examples using the SimCLR backbone alone.
Figure 19: Accuracy of label inference on VICReg and SimCLR using projector and backbone representations. First two columns: Effect of training epochs on memorization for each representation. Last two columns: Effect of training set size on memorization for each representation. In contrast with VICReg, the déjà vu memorization detected in SimCLR's projector and backbone representations is quite similar. While SimCLR's projector memorization appears weaker than that of VICReg, its backbone memorization is markedly stronger. This kind be easily explained as a byproduct of Guillotine Regularization [3], i.e. removing layers close to the objective reduce the bias of the network with respect to the training task. Since SimCLR's projector has fewer layers than VICReg's, the impact of Guillotine Regularization is less salient.
(b) SimCLR
Reconstructions using SimCLR Backbone Only: The above label inference results show that the SimCLR backbone exhibits a similar degree of déjà vu memorization as the projector does. To evaluate the risk of such memorization, we repeat the reconstruction experiment of Section 5 on the dam class using the SimCLR backbone instead of its projector.
Figure 20: Instances of déjà vu memorization by the SimCLR backbone representation. Here, the backbone embedding of the crop is used instead of the projector embedding on the same training images used in Figure 6. Interestingly, we see that déjà vu memorization is still present in the SimCLR backbone representation. Here, the nearest neighbor set recovers dam given an uninformative crop of still or running water. Even without projector access, we are able to reconstruct images in set  $\mathcal{A}$  using  $\mathrm{SSL}_A$ , and are unable using  $\mathrm{SSL}_B$ .
Figure 20 demonstrates that we are able to reconstruct training set images using the SimCLR backbone alone. This indicates that déjà vu memorization can be leveraged to make detailed inferences about training set images without any access to the projector. As such, withholding the projector for model release may not be a strong enough mitigation against deja vu memorization.
A.8 Detecting Déjà vu without Bounding Box Annotations
The memorization tests presented critically depend on bounding box annotations in order to separate the foreground object from the periphery crop. Since such annotations are often not available, we propose a heuristic test that simply uses the lower left corner of an image as a surrogate for the periphery crop. Since foreground objects tend to be near the center of the image, the corner crop usually excludes the foreground object and does not require a bounding box annotation.
Figure 21 demonstrates that this heuristic test can successfully capture the trends of the original tests (seen in Figure 16) without access to bounding box annotations. However, as compared to Figure 16, the heuristic tends to slightly underestimate the degree of memorization. This is likely due to the fact that some corner crops partially include the foreground object, thus enabling the KNN to successfully recover the label with the reference model where it would have failed with a proper periphery crop that excludes the foreground object.
Figure 21: Déjà vu memorization using a simple corner crop instead of the periphery crop extracted using bounding box annotations. While the heuristic overall underestimates the degree of déjà vu, it roughly follows the same trends versus dataset size and training epochs. This is crucial, since it allows us to estimate déjà vu without access to bounding box annotations.