Localizing Memorization in SSL Vision Encoders
Authors: Wenhao Wang
Adam Dziedzic
Michael Backes
Franziska Boenisch
Venue: NeurIPS 2024 poster (2024)
Keywords: self-supervised learning
memorization
localization
Primary area: evaluation
TL;DR: We identify where the memorization happens in self-supervised learning encoders.

Abstract: Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap, we propose two metrics for localizing memorization in SSL encoders on a per-layer (LayerMem) and per-unit basis (UnitMem). Our localization methods are independent of the downstream task, do not require any label information, and can be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based) trained on diverse datasets with contrastive and non-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experiences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fully-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies.

Localizing Memorization in SSL Vision Encoders
Wenhao Wang', Adam Dziedzic', Michael Backes' Franziska Boenisch CISPA, Helmholtz Center for Information Security
Abstract
Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained millions of images, they still mem- orize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap. propose two metrics for localizing memorization in SSL encoders per-layer (LayerMem) and per-umit basis (UnitMem). Our localization methods are independent of the downstream task, do require any label information, and be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based trained diverse datasets with contrastive and con-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experi- ences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fulle-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies.
Introduction
Sule-super learning (SSL) ([16 17 4 30 27 29] enables pre-training large encoders unlabeled data to generate feature representations for multitude of downstream tasks. Recently, was found that, even though their training datasets are large, SSL encoders still memorize individual data points [36] 47)). )) While prior work characterizes the memorized data and studies the effect of memorization to improve downstream generalization (47), little is known about where in SSL encoders memorization happens.
The few works on localizing memorization are usually confined to supervised learning (SL) 45 35), ) or operate the language domain (155) 37
14). particular, most results are and/or require labels 3 35
coarse-grained and localize memorization on per-layer basis 3 45
To close the gap, we propose two novel metries for localizing memorization SSL encoders in the vision domain. Our LayerMem localizes memorization of the training data within the SSL encoders on layer- For more fine- fine-grained localization, we turn memorization in individual units (i.e., neurons in fully-connected layers or channels in convolutional layers). We propose UnitMem which measures memorization of individual training data points through the units' sensitivity to these points. Both our metrics can be computed independently of downstream task, in a forward pass without gradient calculation, and without labels, which makes them computationally efficient and well-suited for the large SSL. encoders pretrained on unlabeled data. By performing systematic study on localizing memorization with our two metries on various encoder architectures (convolutional
"Correspondence meenisch@cispa.de
Accepted for publication at: 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

and transformer-basee trained on diverse vision datasets with contrastive and non-contrastive SSL frameworks, we make the following key discoveries:
Memorization happens through the entire SSL encoder. By analyzing our LayerMem scores between subsequent layers, find that the highest memorizing layers in SSL. are not necessarily the last ones, which is in line with findings recently reported for SL [35). While there is a tendency that
higher per-layer memorization can be observed in deeper layers, similar to SL B 45 our analysis of memorization on a per-unit level highlights that highly memorizing units are distributed across the entire SSL encoder, and can also be found in the first layers.
Units in SSL encoders experience high memorization. By analyzing SSL encoders with our UnitMem metric, we find that significant fraction of their units experiences high memorization of individual training data points. This stands in contrast with models trained using SI. for which we observe high class memorization, measured as the unit's sensirivity to any particular class. While these results are in line with the two learning paradigms' objectives where SL. optimizes to separate different classes whereas SSL optimizes foremost for instance discrimination [48), is novel discovery that this yields significantly different memorization patterns between SL and SSL down to the level of individual units.
Atypical data points cause higher memorization in layers and units. While prior work has shown that SSL encoders overall memorize atypical data points more than standard data points [47], our study reveals that the effect is constant throughout all encoder layers. Hence, there are no particular layers responsible for memorizing atypical data points, similarly as observed in SL. (35). Yet, memorization of atypical data points can be attributed on a unit-level where we observe that the highest memorizing units align with the highest memorized (atypical) data points and that overall atypical data points cause higher unit memorization than standard data points.
Memorization in vision transformers happens mainly in the fully-connected layers. The memo- rization transformers [6 was primarily investigated the language domain 26 43 However, the understanding in the vision domain is lacking, and due to the difference in imput and output tokens (language transformers operate on discrete tokens while vision transformers operate on continuous ones), the methods for analysis and the findings to localize memorization, we are the first to show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.
not easily transferable. Yet, with our methods
Finally, investigate future applications that could benefit from localizing memorization and identify more efficient fine-tuning and caanooizzaiionnnnoomeen pruning strategies as promising directions.
In summary, we make the following contributions:
We propose LayerMem and UnitMem, the first practical metrics to localize memorization in SSL encoders on a per-layer basis and down to the granularity of individual units.
We perform an extensive experimental evaluation to localize memorization in various encoder architectures trained on diverse vision datasets with different SSL frameworks.
Through our metrics, we gain new insights into the memorization patterns SSL encoders and can compare them the ones of SL models.
We show that the localization of memorization can yield practical benefits for encoder fine-tuning and pruning.
2 Related Work
SSL. SSL relies on large amounts of unlabeled data to train encoder models that return repre- sentations for multitude of downstream tasks (6). Especially in the vision domain, wide range of SSL frameworks have recently been introduced [16] 17 [3] 4 30 27 29 Some of them rely
on contrastive loss functions [66 29 27 whereas others train with non-contrastive objective func-
tions 41 13 30
In particular, it was shown that it can have detrimental effect on data privacy, since it enables data extraction attacks 19. 10 [T. At the same time, memorization seems to be required for generalization, in
Memorization in SL. Memorization was extensively studied in SL 52 2 15

---

particular for long-tailed data distributions 23 24 It was also shown that harder or more atypical data points 2 45] experience higher memorization. While all these works focus on studying memorization from the data perspective and concerning its impact on the learning algorithm, they do not consider where memorization happens.
Memorization in SSL. Even though SSL rapidly grew in popularity during recent years, work on studying memorization in SSL is limited. Meehan et al. [36] proposed to quantify Déjà Vu memorization of SSL encoders with respect particular data points by comparing the representations of these data points with the representations of a labeled public dataset. Data points whose nearest public neighbors in the representation space highly consistent in labels to be memorized. Since SSL is aimed to train without labels, this approach is limited in practical applicability. More recently, Wang al. [47) proposed SSLMem, a definition of memorization for SSL based on the lcave-one-out definition from SL [23 24 Instead of relying on labels, this definition captures memorization through representation alignment, i.e., measuring the distance between representations of data point's 's multiple augmentations. Since both works rely on the output representations quantify memorization, neither of them is suitable for performing fine- -raigrd localization of memorization. Yet, we use the setup of SSLMeM as building block to design our LayerMem metric which localizes memorization per layer.
considered
Localizing Memorization. In SL, most work focuses localizing memorization on per-layer basis and suggests that memorization happens in the deeper layers [ 45). By analyzing which neurons have the biggest impact on predicting the correct label of a data point, Maini et al. [35] were able to study memorization on a per-unit granularity. They do so by zero-ing out random units until a label flip occurs. Their findings suggest that only a few units are responsible for memorizing outlier data points. Yet, due to the absence of labels in SSL,, this approach is inapplicable to our work. In the language domain, a significant line of work aims at localizing where semantic facts are stored within large language transformers [55] 37 7 44). Chang et al. [14] even proposed benchmarks for localization methods in the language domain. In the injection benchmark (INJ Benchmark), they fine-tune a small number of neurons and then assess whether the localization method detects the memorization in these neurons. The deletion benchmark (DEL Benchmark) first performs localization. followed by the deletion of the responsible neurons, and final assessment of the performance drop on the data points detected as memorized in the identified neurons. Since in SSL, performance drop cannot be measured directly due to the absence of a downstream task, the deletion approach not applicable. Instead, we verify our Uni tMem metric in a similar vein to the INJ Benchmark by fine-tuning single unit on data point and localizing memorization as we describe in Section 5.2
Studying Individual Units in ML Models. Early work in SL 122 already suggested that units at different model layers fulfill different functions: while units in lower layers are responsible for extracting general features, units in higher layers towards the model output are responsible for very specific features [51] In particular, it was found that units represent different concepts required for the primary task [5). where some units focus on single concepts whilst others are responsible for multiple concepts [38, [54]. While these differences have been identified between the units of models trained with SL, we perform corresponding investigation in the SSL domain through the lens of localizing memorization.
3 Backkround and Setup
SSL and Notation. We consider an SSL training framework M. The encoder f : Rd + R is pre-trained, or in short trained, on the unlabeled dataset D to output representations of dimensionality 8. Throughout the training, as the encoder improves, its alignment loss (A( = = between the representations of two random augmentations x' x of any training data point x decreases with respect to a distance metric d' (e.g., Euclidean distance). This effect has also been observed in non-contrastive SSL frameworks (53). We denote by f fL, [l.... L] the 1th layer of encoder f. Data points from the test set 'D are denoted as T.
Memorized Data. Prior work in the SI. domain usually generates outliers for measuring memoriza- tion by flipping the labels of training data points 223 24 35). This turns these points into outliers
3

---

that experience higher level of memorization and leave the strongest possible signal in the model. Yet, such an approach not suitable in SSL where labels are unavailable. Therefore, we rely on the SSLMem metric proposed by [47] to identify the most (least) memorized data points for given encoder. The findings based on the SSLMem metric indicate that the most memorized data points correspond to atypical and outlier samples.
SSLMem for Quantifying Memorization, SSLMem quantifies the memorization of individual data points by SSL encoders. It is, to the best of our knowledge, the only existing method for quantifying memorization SSL without reliance on downstream labels. SSLMem for training data point is defined as
E (~M(D) ot Aug(()) SSLMem f.g (x) (SLg(x) - (s)))))())(())
E d ff(x)), SSLg(x) g~M(D(a) an Aug(a)
E
E d (g(2"))
where f and g are two classes SSL encoders whose training dataset differs in data point x. x' and z" denote two augmentations randomly drawn from the augmentation set Aug that is used during training and is a distance metric, here ly-distance.
Experimental Setup. We localize memorization in encoders trained with different SSL frameworks on five common vision datasets, namely CIFARIO, CIFARI00, SVHN, STL10, and ImageNet. We leverage different model architectures from the ResNet family, including ResNet9, ResNet18, 8, ResNet34, and ResNet50. We also analyze Vision Transformers (ViTs) using their Tiny and Base versions. Results reported over three independent trials. To identify the most memorized training data points, we rely on the SSLMem metric and follow the setup from (47). More details on the experimental setup can be found in Appendix B For the readers' convenience, include glossary with short explanations for all concepts and background relevant to this work in Appendix A
4 Layer-Level Localization of Memorization
In order to localize memorization on per-layer granularity, we propose a new LayerMem metric which relies on the SSLMem metric, building block. Since the SSLMem as defined in Equation (I) not normalized, we introduce the following normalization to the range [0, 1]
SSL, (x)
SSLMem' (x)
SSL, (2) + (5)
such that values close denote no memorization while denotes the highest memorization. This makes the score more interpretable. While SSLMem returns memorization score per data point for given encoder, LayerMem returns a memorization score per encoder layer I, measured on (sub)set D' = {z1, .... }.ae C D of training data D. Similar to SSLMem, LayerMem makes use of a second encoder g as a reference to detect memorization as
LayerMemp (1)
SSLMem' (II).
(3)
f'.g' denote the output of encoders f and after layer I, respectively. Intuitively, our LayerMem metric measures the average per-layer memorization LayerMem build on SSLMem' ', it also inherits the above normalization. Since Equation 33) operates on different layers' outputs which in turn depend all previous layers, LayerMem risks report accumulated memorization up to layer I. Therefore, we also define A LayerMem D' (I) for all layers I 1
training data points 2; D'. As
ALayerMem (I) LayerMem, (1) LayerMem 1).
This reports the increase in memorization of layer / with respect to the previous layer 1.
Our code is attached supplementary material.

---

4.1 Experimental Results and Observations
We present our core results and provide additional ablations on our LayerMem in Appendix C.3
Memorization Increases but not Mono- tonically. We report the LayerMem scores in Table T for the ResNet9- -based SSL encoder trained with SimCLR on CIFARIO (further per-layer breakdown and scores for ResNet18, ResNet34, and ResNe150 are presented in Table 15 Table T6] and Table 17 in Appendix C.3
Table 1: Layer-based Memorization Scores. Res.V denotes residual connection that comes from the pre- vious N-th convolutional layer.
Layer Layerilex
ALM
Layeztien Top50
Top50
Layerfion
0.001
0.225
0.081
0.012
0.154
0.031
0.308
0.083
01002
We report LayerMem across the TO0 randomly chosen training data points, their ALayerMem (denoted as ALM), followed by LayerMem for only the Top 50 memorized data points, their ALayerMem (denoted as ALM Top50), and LayerMem for only the Least 50 memorized data points. The results show that our LayerMem indeed increases with layer depth in SSL, similar to the trend observed for SL [45), i.e., deeper layers experience higher memorization than early layers. However, our ALayerMem presents the memorization from a more accurate perspective, where we discard the accumulated memorization from previous layers, including the residual connections. ALayerMem indicates that the memorization increases in all the layers but is not monotonic.
0.183
0.029
0,402
0.094
0.031
Rea2
0.002
0.41
0.027
0,479
0.076
0.051
0.246
0.034
0.599
0.120
0.061
0.276
0.030
0.697
0,008
0.071
0.120
0.073
Rev6 0.311
0.003
0.817
0.086
We also study the differences in localization of the memorization for most memorized (outliers and atypical examples) vs. least memorized data points (inliers), shown as columns LayerMem Top50 and LayerMem Least50 in Table respectively. While we observe that the absolute memorization for the most memorized data points is significantly higher than for the least memorized data points, they both follow the same trend of increasing memorization in deeper layers. The ALayerMem for the most memorized points (denoted as ALM Top50 in Table I indicates that, following the overall trend, high memorization of the atypical samples is also spread over the entire encoder and not confined to particular layers.
The memorization of Transformers (46 was, so far, primarily investigated in the language domain 126 43), however, its understanding in the vision domain is lacking. The fully-connected layers in lan- guage transformers were shown to act as key-value memories. Still, findings from language transformers cannot be easily transferred to vision transformers (ViTs) (20]: while language transformers operate on the level of discrete and interpretable input and output tokens, ViT's operate on continuous input im- age patches and output representations. Through the analysis of our newly proposed metric for memorization in SSL., in Table (ViT-Tiny trained on CIFAR 10 using MAE (30), we are the first to show that memorization in ViTs occurs more in deeper blocks and that within the blocks, fully-connected layers memorize more than attention layers. We present the full set of results for LayerMem and ALayerMem over all blocks in Table 10
Memorization in Vision Transformers.
Table 2: Memorization in VIT
occurs primarily in the deeper blocks and more in the fully con- nected than attention layers.
VT Block
Layartees
Layar
Atlention Layer
2
0r2g
0100
6
0.114
0.009
0.010
Fallf- Comeeeoo Layer
2
0.039
0.011
0.129
0,015
0,303
0.022
Memorization in Different SSL Frameworks. We also study the differences in memorization behavior between different SSL frameworks. Therefore, we compare the LayerMem score between corresponding layers of a ResNet50 trained on ImageNet with SimCLR [66 and DINO [33, and of ViT- encoder trained on ImageNet with DINO and MAE [30] We ensure by early stopping that the resulting linear probing accuracies of the encoder pairs are similar for better comparability of their memorization. The ImageNet downstream task performance within both encoder pairs is 66.12% for SimCLR and 68.44% for DINO; and 60.43% for MAE and 60. 17% for DINO. Our results in Table show that encoders with the same architecture trained with different SSL frameworks experience similar memorization pattern, namely that memorization occurs primarily the deeper blocks/layers. C.1 we additionally show that memorization patterns between different
In Figure 12 in Appendix
5

---

Table 3: Consistency in 100 most memorized samples according to LayerMem. We report the pairwise overlap between the most memorized samples and the consistency in ranking of most memorized samples using the statistical Kendall's Tau test (r-statistic, p-value). While we observe high overlap and statistical similarity within adjacent layers, especially towards the end of the network, there is low similarity and overlap between early and late layers.
Layan
Lagess
Ovetlip1
7,5
Lagen Ovetap
7.5
Layen
Oratlap
T.P
U1555 Bx
B412 & Dix-7
24
11340 1.Ne-3
16
3.181 60000
17
166. Tode-2
1.8
16
ILUS4 2ie-1
2.3
10
0.902
5.48--19
9.207, 5.536-5
3.4
355
0.112 6036-11
3.6
42
0.350 8.3te-
37
3,50 5.3005
3.
0,300 1.00
45
68
0.57. 1die-1
46
54
5300
6.35--1
2412
0.31, 4.18-114
1.76-42
0664, 4.13677
9
SSL frameworks are similar down to the individual unit level, i.e., the number of highly memorizing units and the magnitude of memorization are roughly the same. We present the full results for all ResNe150 layers and all VIT blocks in Table 29 and Table 31, respectively.
Variability and Consistency of Memorization cross Different Layers. We use LayerMem to analyze the variability and consistency between the samples memorized by different layers in ResNet9 vision encoder trained with CIFAR1O dataset. The results are shown in Table and Figure [3]ure[1]in appendix C.5) The overlap within the 100 most memorized samples between adjacent layers is usually high but decreases the further the layers are separated. Our statistical analysis to compare the similarity of the orderings within different layers' most memorized samples using the Kendall's rank correlation coefficient shows that while for closer layers, we manage to reject the null hypothesis ("no correlation") with high statistical confidence (low p-value) which is not the case for further away layers.
4.2 Verification of Layer-Based Memorization
To analyze whether our LayerMem metric and its A variant indeed localize memo- rization correctly, we first replace differ- ent layers of an encoder and then compute linear probing accuracy on various down- stream tasks. Since prior work shows that memorization in SSL is required for down- stream generalization [47 we expect the highest performance drop when replacing the layers identified as most memorizing.
Table The layer-based memorization is similar across encoders trained with different frameworks. LM-LayerMem, ALy=ALayerMem.
ResNet50 Layer Number
SimCLR
DINO
A1E
AEX
0,040
0.003
0,041
0,302
0.008
0,311
0.007
VIT-Rase
Number
LM
ALM
LR
0,057
0000
0.036
0.011
Our results in Appendix C.7 verify this intu- ition. They show that by replacing the most memorizing layers of an encoder trained on dataset A. e.g., CIFAR1O, with the equivalent layers of another dataset B, e.g., STL10, the linear probing accuracy drop for CIFARIO is significantly larger than when when replacing random or least memorizing layers. Surprisingly, at the same time, the replacement of the most memorizing layers from the CIFAR 10 trained encoder with STL10 layers also causes the highest increase in STL.10 linear probing accuracy (again in comparison to replacing random or least memorizing layers). See full set of results for replacing any combination of 1, 2, and 3 layers in Table 30 Table 32 and Table 33 respectively. These results suggest that we might be able to improve standard encoder fine-tuning by localizing the most memorizing layers and fine-tuning these instead of the last layer(s) the standard practice for fine-tuning in SSL. We verify this assumption in Table and show that fine-tuning the most memorizing layers indeed yields the highest downstream performance the fine-tuning dataset. This shows that localizing memorization might have practical application for more efficient fine-tuning in the future.
0.120
0015
0.116
0.14
0.274
5 Unit-Level Localization of Memorization
Experiments from the previous section highlight that we are able to localize the memorization of data points in particular layers of the SSL encoders. This raises the even more fundamental question

---

Table 5: Fine-tuning most memorizing layers. We train a ResNet9 encoder with SimCLR on CIFARIO and fine-tune different (combinations of) layers on the STL10 dataset, resized to 32x32x3. We train a linear layer trained on top of the encoder (HEAD) and report STL10 test accuracy after fine-tuning, Fine-tuning the most memorizing layer(s), in contrast to the last layer(s), yields higher fine-tuning results.
Fine-tuned Layers
Accuracy (%)
None (HEAD)
48.6% 1.12%
6 (highest ALayerMem) + HEAD
53.0% + 0.86 %
8 (last layer, highest LayerMem) HEAD
52.7% 0.97%
6,8 HEAD
56.7% # 0.84 %
7,8 HEAD
55.3% 0.77%
4,6,8 (highest ALayerMem) + HEAD 6,7,8 + HEAD
57.9% # 0.79% 56.5% 0.95%
whether it is possible to trace down SSL memorization to unit-level. To answer this question, we design Uni tMem. a new metric to localize memorization in individual units of SSL encoders. We use the term unit to refer to both an activation map from a convolutional layer (single-layer output channel) or an individual neuron within a fully connected layer. Our UnitMem metric quantifies for every unit u. in the SSL encoder how much is sensitive to, i.e, memorizes, any particular training data point. Therefore, Un tMem relates the maximum unit activation that occurs for a data point ay in the training data (sub)set D' C D with the mean unit activation all other data points in D' {xx))
The design of Uni tii is inspired by the class selectivity metric defined for SL by selectivity was derived from selectivity indices commonly used in neuroscience 19]
39
Class
25 and
quantifies a unit's discriminability between different classes. It was used as an indicator of good generalization in SL. We provide more background in Appendix D.I To leverage ideas from class selectivity for identifying memorization, we integrate three fundamental changes in our metric in comparison to the class selectivity metric. While class selectivity is calculated on classes of the test set and relies on class labels, our Uni tMem (1) label-agnostic and (2) computed on individual data points from the training dataset to determine their memorization. Additionally, to account for SSL's s strong reliance on augmentations, (3) we calculate Uni tMem over the expectation on the augmentation set used during training. Research from the privacy community [49 34 suggests that those augmentations leave a stronger signal in ML models than the original data point, i.e., relying on the unaugmented point alone might under-report memorization. We verify this effect in Figure S in Appendix C.T We note that through these fundamental changes Uni tMee is able to measure memorization of individual data points within class rather than to solely distinguish between classes or concepts like the original class selectivity. We provide further insights into this difference and perform experimental verification which highlights that Uni tMem captures individual data points' memorization rather than capturing classes or concepts in Appendix C.2
To formalize our Uni tNem, we first define the mean activation μ of unit u on training point r as activation, (x'),
(x(x)
E
(5)
where the activation for convolutions feature maps is averaged across all elements of the feature map and for fully connected layers is an output from single neuron (which is averaged across all patches of x in ViTs). Further, for the unit u, we compute the maximum mean activation Amar,u. across all instances from D', where N [D'), as
Imaaz = max ()x(())
Let be the index of the maximum mean activation Mu (Zx), i.e. the argmar. Then, we calculate the corresponding mean activity p-mar across all the remaining N - 1 instances from D' as
/-mar,u mean( (x:))
Finally, we define the Uni tMem of unit as
Un tMemp (u) Hmax H max.
(8)
Umaaa + 14 man.u
7

---

AM 18.20
SVHN
UnitMom
UnitMam
UnitMem
(a) Different datasets.
(b) UnitMem vs SSLMem.
(c) UnitMem with DP.
Figure I: Insights into UnitMem. We train a ResNet9 encoder with SimCLR: (a) Different datasets, including SVHN, CIFARIO, and STLI0. We report the UnitMem of the last convolutional layer (conv4_2) 22: (b) Comparing alignment between SSLMem and UnitMem on CIFARIO. Data points with higher general memorization (SSLMem) tend to experience higher Unit Mem: (c) Using different strengths of privacy protection according to DP during training on CIFARIO and Vit-Base
The value of the UnitMMm metric bounded between 0 and 1, where indicates that the unit is equally activated by all training data points, while value I denotes exclusive memorization, where only a single data point triggers the activation, while all other points leave the unit inactive.
5.1 Experimental Results and Observations
We present our core results and provide detailed additional ablations on our UnitMem Appendix C.1
Highly Memorizing Units Occur over Entire Encoder. Our analysis highlights that over all encoder architectures and SSL training frameworks, highly memorizing units are spread over the entire encoder. While, on average, earlier layers exhibit lower Uni than deeper layers, even first layer contains highly memorizing units as shown in Figure (first row). Figure [ shows that this trend holds over different datasets. Yet, the SVHN dataset, which visually less complex than the CIFAR 10 or STL10 dataset, has the lowest number of highly memorizing units. This observation motivates us to study the relationship between the highest memorized (atypical or hard to learn) data points and the highest memorizing units.
Most Memorized Samples and Units Align. draw a connection between data points and unit memorization, analyze which data points responsible for the highest jemar scores. This corresponds to a data point which causes the highest activations of a unit, while other points activate the unit only to small degree not at all. We show the results in Figure Tb (also in Table 12 as well as in Figure in Appendix C. D. For each unit in the last convolutional layer of the ResNet9 trained CIFARTO, for the unit's Amaar, and finally measure this point's SSLMeM score. We plot the UnitMem and SSLMem scores for each unit and its corresponding point. Our results highlight that the data points that experience the highest memorization according to the SSLMem score are also the ones memorized in the most memorizing units. Given the strong memorization in individual units, we next look into two methods to reduce it and analyze their impact.
measure its Uni tMem score, then we identify which data point is responsible
Differential Privacy reduces Unit Memorization. The gold standard to guarantee privacy in ML is Differential Privacy (DP) (21). DP formalizes that any training data point should only have negligible influence on the final trained ML model. To implement this, individual data points' gradients during training are clipped to pre-defined norm, and controlled amounts of noise are added M.. This limits the influence that each training data point can have on the final model. Building on the DP framework for SSL encoders [50], we train a ViT-Tiny using MAE on CIFAR1O with three different privacy levels-in in DP usually indicated with E. We train non- non-private (E oc), little private (e 20), and highly private (e = 8) encoders and apply our UnitMem to detect and localize memorization. Our results Figure Te highlight that while with increasing privacy levels, the average MnimMem decreases, there are still individual units that experience high memorization.
Data Point vs Class Memorization. Since stronger training augmentations yield higher class clustering 31 (Lee the fact that data points from the same downstream class are close to each other in representation space but distant to data points from other classes), we also analyze how the SSL encoders differ from the standard class discriminators, namely SL trained models. Therefore,
8

---

Chehlen Chaskeen
layer1
Jayer2
laver
lover5
leyar6
Jayee7
0.5
1.0 0.0
0.3
1.0
0.3
0.3
1.0
0.0
0.5
1.0
0.0
0.3
1.0 0.0
0.55 1.0
0.5
1.0
0.5
1.0
Figure 3: Significantly more (less) units memorize data points rather than classes in SSL (SL). We measure the ClassMem vs UnitMen for 10000 samples from CIFAR 100, with 100 random samples per class. Each i-th column represents the i-th convolutional layer in ResNer9, with 8 convolution layers, where the 1st row is for SSL while the 2nd row for SL. The red diagonal line denotes = x.
go beyond our previous experiments that measure memorization of units with respect to individual data points and additionally study unit memorization at glassgranullrity. Therefore, we adjust the class selectivity metric from [39] perform the training dataset rather than the test data set as the original class selectivity. To avoid confusion between the two versions, refer to our adapted metric ClassMem (sce Appendix D.2 for an explicit definition). Equipped with UnitMem and ClassMem, we study the behavior of SSL encoders and compare between SSL. and SL.. For train encoder with SimCLR and model with SL using the standard cross entropy loss, both on the CIFARTON 100 dataset using ResNer9. We remove the classification layer from the SL trained model to obtain the same architecture for the encoder trained with SimCLR.
comparison,
For comparability, we early stop the SL training once it reaches a comparable performance to the linear probing accuracy on CI- FARI00 obtained by the SSL encoder. Our results in Figure show that overall, in SSL throughout all layers, average memorization of individual data points is higher than class memorization, whereas in SL., in deeper layers, the class memorization increases significantly. We hypothesize that this effect is due earlier layers in SL learning more general features which are independent of the class whereas later layers learn features that are highly class dependent. For SSL, such a difference over the network does not seem to exist; both scores increase slightly, however, probably due to the SSL learning paradigm, memorization of individual data points remains higher.
St. Us than Clasatlee
Sc. Liitterr Classiaa
1.40
UnitMerr
8.2
0.0
Figure 2: ClassMem for SL and SSL.
UnitMem and
To better understand the memorization of units on the micro level, we investigate further the individual units in each layer. In Figure 3, we plot the ClassMem vs UnitMem for each unit and in each of the eight encoder layers of ResNet9. Most units for SSL (row 1) constantly exhibit higher Unit than ClassMem, i.e., they cluster under the diagonal line, which suggests that most units memorize individual data points across the whole network. Contrary, the initial layers for the model trained with SL have a slight tendency to memorize data points over entire classes whereas in later layers, this trend drastically reverses and most units memorize classes. In Appendix C.4 we investigate how this different memorization behavior between SL and SSL affects downstream generalization.
5.2 Verification of Unit-Based Memorization
We verify the unit-based localization of memorization with our Unit by deliberately inserting memorization of particular units and checking if our Uni tMem correctly detects it. Therefore, we first train a SimSiam-basee [17] ResNet18 encoder trained on the CIFARI0 10 dataset. We select SimSiam over SimCLR for this experiment since SimCLR, as a contrastive SSL framework, cannot train on a single data point. Then, using LayerMem, we identity the last convolutional layer in ResNet18 (i.e., layer 4.l.com) as the layer with the highest LayerMem and ALayerMem memorization. We select the unit from the layer with the highest Pmaz and also pick unit with no activation (Amaz = 0) for some test data points. Then, we fine-tune these units using a single test data point and report the change in Un tee for the chosen units Table [3 The results show that our Uni MMem correctly detects the increase in memorization in both units. Additionally, we analyze the impact of zero-ing out the most or least memorizing vs. random units. Again with the argument that memorization required for downstream generalization in SSL 47). expect the highest performance drop when

---

Table 6: Removing the least/most memorized units according to UnitMem preserves most/least linear probing performance. We prune according units with highest or lowest Uni tMem either per layer or for the entire network (total). We also present baselines where we prune randomly selected units. The standard deviation for this baseline is reported over 10 independent trials where different random units were pruned. We train the ResNet9 encoder using CIFAR10 and compute the UnitMem score using 5000 data points from the train set.
Pruning
Selected Units
Accurccy SVINN
Stralegy
CHARIO
STL10
No Pruning
70.44
78.22
62.12
Top lini thees per layer
To
53.4
6:.54
16.92
Random per layer
1.76
67.04 2.44
2.18
layer
72.26
Top lins Mee per layer
20
48
55.88
41.18
Kandom per layer
20
51,34 + 1.21
58.01= 1.34
46.74 + 0.97
Uni tMen layer
62.00
Top Unin tosal
TO
To.
66128
47.30
Random total
10
56.77 1: 2.09
67.09 = 1,56
$3.89 1: 2.33
Low Uni
62.62
72.28
zero-ing out the most memorizing units. Our results Table 0 in and in Appendix C.1 confirm this hypothesis and show that removing the most memorizing units yields the highest loss in linear probing accuracy on various downstream tasks while pruning the least memorized units preserves better downstream performance than removing random units. These results suggest that future work may benefit from using our UnitMem metric for finding which units within a network can be pruned while preserving high performance.
6 Conclusions
We propose the first practical metrics for localizing memorization within SSL encoders on a per-layer and per-unit level. By analyzing different SSL architectures, frameworks, and datasets using our metrics, we find that while memorization SSL increases in deeper layers, a significant fraction of highly memorizing units can be encountered over the entire encoder. Our results also show that SSL encoders significantly differ from SL trained models their memorization patterns, with the former constantly memorizing data points and the latter increasingly memorizing classes. Finally, using metrics for localizing memorization presents itself as an interesting direction towards more efficient encoder fine-tuning and pruning.
Acknowledgments
The project on which this paper is based was funded by the Deutsche Forschungsgemeinnchaft (DFG, German Research Foundation), Project number 550224287. Additional funding came from the Initiative and Networking Fund of the Helmholtz Association in the framework the Helmholtz Al project call under the name ,PAFMIM", funding number 2T-1-PF-5-2277. Responsibility for the content of this publication lies with the authors.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas. David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In International conference on machine learning. pages 233-242. PMLR, 2017.
[3] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10888 10889, 2021.
[4] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianc-iivarianceccvvaaaa regular- ization for self-supervised learning. International Conference on Learning Representations, 2022. URL
10

---

[5] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8): 1798-1828, 2013, doi: 10.1 10110/A0M.2222035500
[7] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URL https://openaipublic blob, core. windows. net/neuron- explainer/paper/mndex (Date accessed: 14.05. 2023), 2023.
[8] K. H. Britten, M. N. Shalden, W. T. Newsome, and J. A. Movshon. The analysis of visual motion: comparison of neuronal and psychophysical performance. Journal of Neuroscience. 12:4745-4765, 1992.
[9] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
[10] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21),
pages 2633-2650, 2021.
[11] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35: 15:13363133 2022.
[12] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from diffusion models. 2023.
In 32nd USENIX Security Symposium (USENIX Security 23), pages
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self supervuee vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 6650-96600 2021.
[14] Ting- Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize memorized data in llms? arXiv preprint 1rXiv:23 11022311..90000 1,09060, 2023.
[15] Satrajit Chatterjee. Learning and memorization. In International conference on machine learning, pages 755-763. PMLR, 2018.
[16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[17] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758, 2021.
[18] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215 223.. JMLR Workshop and Conference Proceedings. 201
[19] Russell L. de Valois, E. William Yund, and Norva Hepler. The orientation and direction selectivity of cells macaque visual cortex. Vision Research, 22:531-544, 1982. URL https://api. http:://app..eennnccccc..ooooooo
orgo :ttp:/Crrui::::33 335066510
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL. https:///penrreiee..o http://wwww...eeereooooro/toooaa

---

[21] Cynthia Dwork. Differential privacy. In International colloquium on automata. languages, and programming, pages 1-12. Springer, 2006.
[22] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. 2009.
[23] Vitaly Feldman. Does learning require memorization? short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954 959, 2020.
[24] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 1881- 2020.
[25] David Freedman and John Assad. seperience-dependenn representation of visual categories in parietal cortex. Nature, 443:85-8, 8, 10 2006. doi: 10. 10.00//ntuure0507..
[26] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ 105533/1/2221.mmmmg .ewnlp-main.44e 446. URL httpss:/aaclanthoooggr. wwwpg///22222 emnlp-main 446
[27] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 15:21271-2123 2020.
[28] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 770-778, 2015. UtLtttss://app hoa.:pppp..mmatiiccccoooo.a... cor..oo07rss
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7729-97335 2020.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[31] Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations, 2023. URL /tttps:/ooperreee wttpw/iiiwinneettrneeeeeee www./eeet/fmimiim http:/orrmmmmdxxxxxxxEE.ttn.ma XDJuuEYHhme)
[32] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[33] Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2081-2095 2021.
[34] Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2081-2095, 2021.
[35] Pratyush Maini, Michael Mozer, Hanie Sedghi, Zachary C Lipton, J 7ico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? arXiv preprint 230iv:330.0094.. 2023.
[36] Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do ssl models have déjà vu? a case of unintended memorization in self-supervised learning. arXiv e-prints, pages arXiv-2304, 2023.
12

---

[37] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. Advances in Neural Information Processing Systems, 35:17359- 2022.
[38] Ari S Morcos, David GT Barrett, Neil Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations. 2018.
[39] Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance single directions for generalization. In International Conference on Learning Representations, 2018. URL S net
[40] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[41] Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 5592-8618 PMLR, 28-30 Mar 2022. URL https:/ https://proceeddnngg mml. wtt../eree//11//ooeee22222 html
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-2523 2015.
[43] Ildus Sadrtdinov, Nadezhda Chirkova, and Ekaterina Lobacheva. On the memorization properties of contrastive learning. arXiv preprint arXiv:2107, 10/43, 2021.
[44] Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. arXiv preprint 1.XXi:23055098663 2023.
[45] Cory Stephenson, Abhinav Ganesh, Yue Hui, Hanlin Tang, Sue Yeon Chung, et al. On the geom- etry of generalization and memorization in deep neural networks. In International Conference on Learning Representations, 2020.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, 2017. URL httpps://procceedinggg neurips. wttpcc//ppaa. http:/ppe__fiiie f fit.ss//ppae//211//1112 Paapee .pdf
[47] Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Boenisch. Memorization in self-supervised learning improves downstream generalization. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[48] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference Learning Representations, 2021.
[49] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tic- Liu. How does data augmentation affect privacy in machine learning? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10746-10753, 2021.
[50] Yaodong Yu, Maziar Sanjabi, Yi Ma, Kamalika Chaudhuri, and Chuan Guo. Vip: A differentially private foundation model for computer vision. arXiv preprint arXiv:2306.08842 2023.
[51] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer VCcV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014.
13

---

[52] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2016.
[53] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances Neural Information Processing Systems, 35:27127-27139, 2022.
[54] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint arXiv: 1806.02881 2018.
[55] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
14

---

A Glossary
For the reader's convenience, we provide glossary with all important terms and concepts related to our work in 7
Table 7: Glossary. We present a concise overview on the concepts relevant to this work.
Concept
Explanation
Atypical examples
Data points that are uncommon in the data distribution and different in terms of their features. Examples: Figure from 47 Sometimes also called "outliers". metric proposed by [39] which quantifies a unit's discriminability between different classes, measured on the test data.
Class Selectivity
ClassMem
Our adaptation of Class Selectivity mcasured on the training data. Expresses how well an encoder is suited to solve some downstream tasks, For classification, it often measured by lincar probing, i.c., training additional classification layer on top the encoder output. Our proposed metric to quantify memorization of any layer in the SSL encoder. A phenomenon where machine learning model stores detailed information on its training data.
Downstream Generalization
LayerMem Memorization
Memorized Data Point Memorization Pattern
data point that experiences high memorization by a machine learning model. A general trend the low-level memorization of an SSL encoder, i.c., which layers or units do memorization localize. Term used refer
Unit
individual neuron in fully connected layers channel in convolutional layers.
UnitMem
Our proposed metric quantify the memorization of any unit the SSL encoder.
B
Experimental Setup
Datasets. We base our experiments on ImageNet ILSVRC-2012 442
CIFARIO
32
CI-
FARI00 32). SVHN 40), and STL10 [8]
Models. We the ResNet family of models (8), including ResNet9, ResNet18, ResNet34, and ResNet50. In T we present the detailed architecture of the ResNet9 model.
Table 8: Architecture of ResNet9, In the Number of Units column, we present the number of activation maps (corresponding to individual filters in the filter bank).
Con-Layer
Layer
Nomber Units
Pommeters
Convl BNI
MaxPooll
32
Coar2-0
MaxPaal2-0
64
Comy2.1
54
36928
BN2-1
MaxPael2.1
Coav2-2
64
36528
BN2-2
64
128
MaxPaol2-2
Com
73856
BN3
128
256
MaxPool3
128
Canu4-D
205168
256
MaxPoale-D
256
Conse.1
255
500080
BM4-1
MasPuolé-1
256
Comu4-2
255
500080
BN4-2
255
512
MuxPuolé-2
SSL Frameworks. SimCLR [16), DINO 13 and SimSiam [17].
We base our experimentation on four state-of-art SSL encoders: MAF 30
15

---

Training Setup. Our experimental setup for training the encoders mainly follows 47 and we rely on their naming conventions and refer to the data points that are used to train encoder f, but not reference encoder ! as candidate data points. In total, we use 50000 data points as training samples for CIFARIO. 10, SVHN, and STL.10 and 100000 for ImageNet with 5000 candidate data points per dataset. The encoders evaluated in the paper are trained with batch size 1024, and trained 600 epochs for CIFARIO, 10, SVHN, and STL10, and 300 epochs for ImageNet. We set the batch size to 1024 for all our experiments and train for 600 epochs CIFARIO, IO, SVHN, and STL10, and for 300 epochs on ImageNet. As distance metric to measure representation alignment, we use the & distance. We repeat all experiments with three independent seeds and report average and standard deviation. For reproducibility, we detail our full setup in Table 9 with the standard parameters that are used throughout the paper if not explicitly specified otherwise.
Training Augmentations. We generate augmentations at random from the following augmentation sets (p indicates augmentation probability):
SL, standard, (referred to as weak augmentations): Coboriiteer(99440---9990000 p=0.44) Ran- donfoorizontadPiipppO...... RandomGraysxcall(po0O.] RandomRRsizzd((nppsiii===)
SSL, standard, (referred to as normal augmentations): ColorJiiter(.88000000--000 p=0.8),
RandommHortzonnuuliippp0000 RandonGGaaysaal((==02)) haaraa/oeeeeedmmmsssssssoosoeea
SSL, stronger, (referred to as strong augmentations): Color/iiter((8--0.---0---22 p-0.0. Ran-
domHooriontallliippp1..0 RundomGrayscalllp==055) RamdomReeszzco/Chggiee--22) Random ticalFlip(p=1.0)
SSL (independent): GaussianBlurkerrll 4), sigma=(0.1, 5.0), p=0.8), RandomIn- vert(p=0.2), RandonkesiieddCoppssse--- Random Vertica/Flip(p=1.0)
Masking (MAE): 75% random masking
Details on Computing Uni. Relying on insights from 33 we calculate Equation 5 for the activations within UnitMem over ten augmentations since this has shown to yield strong signal on the augmented data point. For convolutional feature maps, the activation of the unit is calculated as the average of all elements in the feature map. In ViTs, where we measure activation over fully- connected layers, we compute the activation per neuron and average across all patches of given input. For example, for VIT Tiny encoder pretrained on CIFAR1Ó, the input image of resolution 32x32 is patchified into 64 patches, each of size 4x4. Then, each patch is represented by a 192 dimensional embedding. The classification (CLS) embedding is prepended to the remaining 64 embeddings. Overall, we obtain 65 patches. The last fully connected layer has 192 neurons. For each neuron, we average its activations across the 65 patches. In the case ViT Base, we have 768-dimensional embeddings and 197 patches for the input image of resolution 224x224.
Details on Tine-Tuning with one Test Data Point We provide the exact details of our experiments
to verify our UnitMem through deliberate insertion of memorization in Section 5.2 We train a Sissia [17] ResNet18 encoder the CIFAR10 dataset and use LayerMem identity layer 4. 1.com22, ie.. the last convolutional layer in ResNet18, as the layer with highest accumulated memorization. We select the unit from the layer with the highest Pmaz and also pick unit with no activation (Amoz O) for some test data points. Then, for compatability with pytorch which does support individual unit training,
lock all parameters except for the targeted layer and train the model with single sample from the testing dataset. We choose the sample that achieves the highest activation Pmoz on the unit with the highest UnitMem. We save the checkpoints after each epoch and test the Pmaa for the selected two units. Our results in Table 13 show that the value of Amaz for the selected data point increases in both units and the data point remains the one responsible for the Mmaz.
Details on Hardware resources usage We finish all our experiments on two devices: cloud server with four A 100 GPUs and local workstation with Intel 13700k CPU, Nvidia 4090 graphics card and 64GB of RAM
16

---

Table 9: Training Setup for SSL Frameworks and Hyperparameters. Two numbers denote
ImageNet/ Others.
Model Training
Lincar Probing
MAE
SimCUR
DINO
SimSiam
MAE
SimCLR
DINO
SimSiam
Training Epochs
300 600
600
300 600
200
45/90
45/90
45/90
-/30
Warm-up Epochs
30/ 60
30/ 60
30/60 60
-/24
5/10
5/10
5/10
-/3
Batch Size
2048
4096
1024
128
4096
4096
4096
256
Optimizer
Adam
LARS
AdamW
SGD
LARS
LARS
LARS
SGD
Learning rate
1.2o-3
4.8
20-3
2.5c-2
1.6
4,8
1.6
3o-2
Learning Schecule
Cos. Decay
Decay
Cos. Decay
Decay
Decay
Cos. Decay
Decay
Cos. Decay
C Additional Experiments
C.1
Additional Insights into UnitMem
UnitMem increases over training, First, we assess how UnitMem evolves over training of the SSL encoder. Therefore, we train a ResNet9 encoder using SimCLR on the CIFARIO dataset for 800 epochs, using 120 warm-up epochs. Every five epochs, measure the Uni Our results in Figure@depict the average Uni tMem of the ResNer9's last convolutional layer.
We observe that the unit memorization monotonically increases throughout training and that the increase is particularly high during the first epochs. After the warm-up, in unit memorization stagnates until the level of memorization on the unit level converges. The same trend can be observed over all layers which indicates that SSL encoders increase unit memorization throughout training.
observe that the increase
Measuring UnitMem without using augmentations leads to an under-reporting of memorization.
Figure 4: Average UnitMem of layer 8 over training.
To assess the impact on using augmentation to implement Equa- tion (5) for the calculation of our UnitMem has an impact on the reported results, we train two ResNet9 models on the CIFARI00 dataset, one using SimCLR, the other one using standard SL with cross entropy loss.
During training we rely on the standard augmentations for SL and SSI. reported above. To measure memorization, once ten augmentations from the training augmentation set, and no augmen- tations otherwise and report the results in Figure 5 We find that while the trend of the reported memorization is equal in both settings, the Uni tMem measured without augmentations remains constantly lower than when measured with augmentations. This suggests that when measuring UnitMem, it is important to use augmentations to avoid under-reporting the memorization.
Figure 5: UnitMem & w/o augmentations.
The number of data points used to measure UnitMem does not have a significant impact on the reported memorization. Using the same ResNet9, trained with SimCLR on CIFAR 100, we assess whether the number of data points that we use to calculate UnitMen (the size of D') has impact the reported memorization. Then, we measure UnitMem using 100 random data point chosen one from each class in CIFAR 10. 100 purely randomly chosen data points, and randomly chosen CIFAR 100 data points. We present our find- ings in Figure Our results highlight that all the lines within each other's s standard deviation, indicating that there is no significant difference in the reported Uni UnimMem, dependent the make up of the dataset D'.
Figure 6: Size of D'.
Most memorized data points align with the most memorizing units. We train ResNer9 on CIFARIO IO using SimCLR and measure UMe for the 300 most and 300 least memorized data points identified using SSLMem by [47.. The measurement of the two sets (most vs least memorized
17

---

data points) is performed independently. Our results in Figure 7 show that the Uni Unem calculated on the most memorized data points significantly higher than on the least memorized data points (we verify the significance with a statistical (-test in Table 11
While also some of the least memorized data points lead high activation of the units, highest activation (on average and in partic- ular) can be observed for the most memorized data points. This underlines the trend observed in Table [2 which shows that highly memorized data points align with the highly memorizing units.
Computing UnitMen based on the median yields similar results to using the mean. Our Uni UMem metric is inspired by the class
Figure 7: Least most
memorized data points.
selectivity defined for SL by [39] which quantifies a unit's discrim- inability between different classes, see Appendix [D.]] Yet, we calculate the p-max,u u in Equation (7) using the median on the other individual training data points'
activations while ClassSelectivity computes their equivalent of ft-maz,a using the mean all
other test classes' activations.
We show in Figure 8 over the 300 most and least memorized data points for a ResNet9 trained with SimCLR on CIFARIO 10 that using the median for UnitMem yields very similar results to using the mean.
leand
For SSL, the concrete augmenta- tion set has no strong impact when measuring UnitMem. We addition- ally set out to study the impact of the augmentation set used calculate UnitMem. Therefore, we calculate UnitMen on the ResNet9 trained on CIFAR10 using SimCLR using different augmentations sets. For SSL., we measure once with the standard training augmentations ("Normal"), with an independent set of augmentations of similar strength (" ("Independent"), with a weaker augmentation set for which we rely on the augmentations used to train the SL model ("Weak"), and an independent very strong set of augmentations modeled after MAE training and using masking of 75% of the input image ("Masking"). ") Our results Figure 9 depict the UnitMem over the last convolutional layer of the ResNet9 encoders.
(a) Median
(b) Mean
Figure 8: Mean vs Median.
They highlight that the weak and in- dependent augmentations report ex- tremely similar Uni tMem the orig-
Seading
inal set of training augmentations used. For SL, the impact of using dif- ferent augmentations during training and measuring UnitMem is more ex- pressed. We also measured for weak augmentation set ("Normal"), an in- dependent weak set (" ("Independent", strong augentations for which we relied on the standard SSL augmentations ("Strong"), and the 75% masking ("Masking"), ). We observe that using the augmentations from training to calculate UnitMem yields the highest localization of memorization.
(a) SSL
(b) SL
Figure 9: Different augmentation sets.
Stronger augmentations reduce memorization. We also ana- lyze how the training augmentation strength can impact the fi- nal encoder's Uni MMem. We use ColorJitter, HorizontalFlip, Ran- domGrayscale, and RandomResizedCrop as augmentations. Their strength is determined by the probability of applying them and their level of distortion. Appendix B we present the exact parameters specified for each of them under different strengths. Our results in Figure TO suggest that stronger augmentations yield lower per- unit memorization. These findings are in line with prior theoretical
Strong
Layer
Figure 10: Different augmen- tation sets.
18

---

work on SSL 48] highlighting that SSL performs foremost the task of instance discrimination (i.e. differentiating between individual images), but achieves clustering according classes due to the augmentations: with stronger augmentations, multiple data points" augmented views will look ex- tremely similar (e.g, the wheels of two different images of cars), such that they eventually activate the same unit. Thereby, this unit memorizes individual data points less while units trained with weaker augmentations depend on and memorize individual data points more. Note that we do not observe strong dependency of our reported UnitMem the concrete augmentation set used to calculate the metric (see Equation (5)) as we show in Figure 2 in Appendix C.T Yet, using the original set of training augmentations, as we do for our experiments, yields the strongest signal.
Stronger weight decay reduces memorization. To analyze how training weight decay affects the final encoder's UnitMen we train a ResNet9 using SimCLR on CIFAR10 with three different levels of weight decay. Our results in Figure T that stronger weight decay yields lower memorization, yet also decreases linear probing accuracy.
041
Different SSL frameworks yield similar memorization pattern.
Figure 11: Different weight decay
We compare the UnitMem score between corresponding layers of a ResNet50 pre-trained on ImageNet with SimCLR and DINO, as well as for ViT-Base encoders pre-trained on ImageNet with DINO and MAE. We ensure that the number of epochs. batch sizes, training dataset sizes, and the resulting linear probing accuracies of the encoders are similar for direct comparability. Our results in Figure [2 depict the Uni of the last convolutional layer of the ResNet50, and the final block's s fulle-commected layer in the ViT. The plot indicates that the different SSL frameworks applied the same architecture with the same dataset yield similar memorization pattern.
Table 10: The memorization in VIT occurs primarily in the deeper blocks and more in the fully-comnected than attention layers. We present the results for ViT Tiny pre-trained on CIFARIO using MAE.
ALayerMem ATT
LayerKem ATT - ResBlock PC 11 ALayerMem FC N
LayerMem PC
N
ABlockMemN = ResBlockf ResBlock FC ResBlock ATT
VIT Black
Attention Layer
Fully-Conmertee Layer
Number
LayerHex
ALayeriea
RecBlock
LayorSes
Alayertinn
ResBlock
ABlockiko
0.022
0.039
0011
0.040
0,018
0.846
01.06
0.047
0.000
0013
0.06
0,021
0.067
0.006
0.067
0.063
0.017
0.085
0.024
0.14
0.166
0,021
0.129
0.05
0.131
0.025
0,140
0.009
0.139
0.155
0.16
0.156
0.025
0,164
0.08
0.164
0.182
0018
0.882
0.06
0.000
02111
0,09
0.220
0.230
0.240
0.000
0.211
0.030
11
0.249
0.008
0.249
0.271
0.022
0.271
0,000
12
0,280
0.009
0.280
0.300
0.023
0.364
0,033
Additional Verification of UnitMem. We present the additional verification of the UnitMem metric in Table [4] Therein, we perform two additional experiments to the verification pre- sented in Section 5.2) First, we fine- tune the most memorizing unit and the inactive unit with 300 (instead of 1) data points from the test set (a). We observe that the data points that expe- rienced the highest memorization for the selected unit remains the highest memorized of the 300 data points. Additionally, it experiences the highest memorization in the unit that used to be inactive. Second, we fine-tune the most memorizing unit and the inactive unit with the most memorized data point, but with a batch-size of 300 were we duplicate the data point 300 times (b). We observe that
350
(a) SimCLR vs. DINO
(b) DINO vs. MAE
Figure 12: Different SSL frameworks.
19

---

Table 11: Most vs Least Memorized Data Points. We train a ResNet9 using SimCLR on CIFARIO follwoing the setup by [7]. We then take the 50 most and 50 least memorized data points according to SSLMem and calculate the Uni MMem over for the two sets of points. In the table, we report the average per-layer tMem of the two sets independently. We also perform a statistical (-test to find whether the Uni tMem scores differ among most and least memorized data points. With p << 0.05, we are able to reject the null- sel-hhypthheis and find that the memorization according to UnitMem differs significantly between the most and least memorized data points.
Layer Name
mean thee
meam UnittKen eXen
I-lea
most motrrried (1 10% units)
least menorized (10S) units)
p-talie
couv1
0.507
0.235
25.89/3.00
comv2-0
0,501
(1.231
66.25/0.00
comv3-1
comv3-2
0.512
65.12/0.00
comv5
0.505
0:242
64.13/9.00
comc4-0
0.514
0.346
63,6789.0
comv4-1
64,0900.00
conv4-2
64.15/0.00
Table 12: Highly memorized data points align with most memorizing units. We select 10% of the most memorizing units according to UnitMem in the last layer (conv-4-2) of the ResNet9 encoder pre-trained on CIFARIO. The 1st row represents the number of times given data point was responsible for Pmaz. the 2nd row counts for how many daat points this applics. The last column shows that the highest memorized sample (SSLMem of 0.891) 1) is responsible for the Pmma in the largest number of units (5).
Metric Used-> Frequeccy
Average S0L8en Score
0357
of times Responsible for Fmaz Samples
4
the effect of the fine-tuning this point's memorization is far more expressed than when fine-tuning with 300 different data points.
Additional Verification of Unm. In Table their level of memorization. Our results indicate that by pruning the most memorizing neurons, we cause the highest drop in downstream performance.
B
prune, i.e., zero out neurons according to
C.2 UnitMem Measures Memorization of Individual Data Points
To highlight that Uni tMem reports memorization of individual data points rather than the unit's ability to recognize class-wide concepts, we designed an additional experiment. For the experiment, we rely on the class concept of "wheel" as an example. In the STL10 dataset, three classes have a concept wheel, namely Truck, Plane, Car. If Uni tMem was to report simply unit's 's sensitivity to concepts of different classes (rather than individual data points), we would see a drop in Uni UnitMem as we increase the percentage of data points with the concept wheel in the batch used to compute the metric. This is because then all data points should equally activate the unit, resulting in low memorization according to Equation (8)
We perform the experiment in Table 34 with 1000 data points chosen from different classes, namely 1) all classes (here 30% of the data points have wheels), 2) the classes Truck, Plane, Car (close to 100% of the samples now have the concept of wheels), and 3) purely the class car (close to 100% wheels). In 2) and 3), close to 100% of the samples now have the concept of wheels. Thus, if the units were responsible for the concept wheel, they would have very high activation over all samples and the reported Uni thet should be very low. However, in our results, we see that we do have units with very high Uni MMem. These can, in turn not be the units for the class-concept wheel, but must be units that focus on individual characteristics of the individual training images. This means that there must be unique features from the individual images that are still memorized that go beyond the concepts that are the same within class.
20

---

Table 13: Verification of the UnitMem metric for memorization in individual units. The SSL model based on SimSiam with ResNet18 architecture and trained on CIFARIO is fine-tuned on single data point. We select two units with the highest and lowest Uni scores. The data point used for fine-tuning achieves Mmar in both units. The Uni tMem score increases only for the two selected units while remains unchanged for the remaining units.
Targeted
Numuber of Teeiie---Tnnng Epachs
Unit
10
50
200
500
1000
Hlighstt Umitiles
0.754
0.761
0,792
0.814
0.824
0.826
Lowest Unitken
0.0099
Table 14: The Pmaax and Jmmé after fine-tuning for different number of epochs.
(a) The jmaa and Jmmin after fine-tuning for different numbers epochs. This is tinettuned with 300 data samples from the test dataset. The samples were not seen during the initial training the encoder, thus only single filter is affected by them.
trained nodes
original
10 epoch
50 epoch
200 epoch
500 epoch
1000 epoch
Most Mem
filter 77
0.754
0.766
0.809
0.819
0.826
0.828
Least Mem
filter 459th
0
0.011
0.038
0.046
0.051
(b) The piaex and Jmin after fine-tuning for different number of epochs This is fine-tuned with
only highest praus samples while 300 duplication from training datasets.
trained nodes
original
10 epoch
50 epoch
200 epoch
500 epoch
1000 epoch
Most Mem
0.754
0.798
0.846
0.857
0.861
0.862
filter th
Least Mem
0.039
0.065
0.079
0.081
0.081
filter thg
C.3 Additional Insights into LayerMem
LayerMem is not sensitive to the size and composition the batch. In our ablation study in show that LayerMem is not sensitive to the size and composition of batch it is computed on. The results can be found in Table [9] where report the LayerMem measured for different number of candidate data points. We pre-trained ResNet9 using SimCLR CIFARIO and determined LayerMem on batches of different sizes. For each batch size, we use independent seeds (i.e., different batch compositions) and report the average LayerMem score and its standard deviation. The results show that the reported LayerMem score is, indeed, similar across all setups. This indicates LayerMem 's insensitivity the choice of the batch used compute
Full Results with Memorization Scores over all Layers. We present the LayerMem score for ResNer18 in Table 15 ResNet34 in Table 16, and ResNet50 in Table 17 all trained CIFAR10 1 and using the SimCLR R framework.
We show the further breakdown of the memorization within the layers in Table 18 the batch normalization layers (denoted as BN) together with the MaxPool layers have a negligible impact on memorization and most of the memorization in each layer is due to the convolutional operations. This is due to the much larger number of parameters in the convolutional filters than the batch normalization layers and no additional parameters in the MaxPool layers, shown in Table g However, the memorization reported per convolutional layer is not correlated with the number of parameters of the layer. For instance, our ALayerMem reports the highest memorization for the 6-th layer, while layers and have each twice as many parameters, see Table [
We observe that
21

---

Table 15: Full results ResNet18. We depict our LayerMem of the final trained model (at the end of training with CIFARIO, ResNet18 with SimCLR).
Layer
LayerKon
convl
0.074 +: 0.010
pool
0.002 1 0.007
2onv2
vonx2-2 2
0.110 1: 0.006
2on22-3
0.123 + 0013
rony2 4
0.134 1: 0.010
pony3 2
0.155 + 0.013
vonv3 3
0.166 1 0.011
come3-4
0,183 + 0.007
conv
voned
0,206 0.009
vone6-3
0.220 + 0.011
conet-1 4
0.239 0.010
2omv5
0.257 + 0.007
ro0v5 3
0.272 +: 0.011
vone5-4
0.295 1: 0.012
overge pool
fully connected
0.224 1: 0.010
softmax
0.207 1: 0.007
Table 16: Full results ResNet34, We depict LayerMem of the final trained model (at the end of training with CIFAR10, ResNet34 with SimCLR).
Layer
LayerKon
convt
max pool
0.069 + 0013
0onv2.1
0,788 + 0.08
conv2-3
+ 0012
0on22-4
0.06 + 0.009
conx2-5
0,107 1 0.016
6x003.1
vonv3-2
0,128 1 0.013
0one3.3
0.131 + 007
2onv3-4
sone3-5
0 0.033
0-693-6
0. 149 + 0.15
rone3 7
0.157 1 0.013
oxne4-1
0. + 0.006
vone4 2
0,178 1 0.010
0one6-3 3
0 1 0012
convt-4
cone4-5 5
0 194 1 0.033
0one4-6
0.201 + 0.007
conct 7
0,205
come4-9
0.218 + 0.006
conv5 10
0.227 1: 0.012
convé-1 II
0.235 1 0.010
voue5
0.257 1: 0.011
vonv5 2
0.261 1: 0.014
cony5 3
0.088
20055-4
vonc5-5
0.299 1: 0.011
0.313 + 0.015
0.297 1: 0.009
fully cunnected
softmax
0.233 4: 0.006
Ablation on LayerMem's sensitivity. We perform an additional ablation to show that LayerMem is not sensitive to the number of samples the batch used to compute it or the composition of the batch, iee, which samples are chosen. Our results in Table (99helighh that over different batches with 100, 500, 1000, and 5000 samples, the observed LayerMem scores are alike. This indicates LayerMem insensitivity to the choice of the batch used to compute it.
22

---

Table 17: Full results ResNet50. We depict our LayerMem of the final trained model (at the end of training with CIFARIO, ResNet50 with SimCLR).
Layer
LayerMon
convl
0.016 1: 0.006
pool
0.066 4: 0.012
2onv2-1
vomx2-2
0.068 + 0.013
2on22-3
0.073 + 0012
vony2-1 4
0.079 1 0.015
ronv2 6
0.033 + 0.010
vony2 7
0.068 1 0.007
conv2-8
0.084 + 0.11
conv2
vonx3-1
0.109 1: 0.010
2onv3-2
0.112 + 0.022
2onv3 3
0.118 0.009
co0v5.5
0.127 + 0.010
rony3 6
0.133 + 0,011
vone3-7 7
0.136 1 0.013
comv3 8
pone3 9
0.144 0.005
vonv 10
0, + 0.008
convá-11
0.156
vonet-1
0. 168 + 0.012
conv4.2 2
0.175 + 0.010
conct 3
0,181 1: 0.006
comv4.4
conv4.5
0.192 + 0.008
vony4 6
0.196 1: 0.014
+
convd
conet
0.217 1 0.011
conv4-10
0.225 + 0.005
0.311 + 0.015
conv4.12
comvé-13
0.241 + 0.012
convé 14
0.44 + 0.009
conv4.15
+
conv4.16
0.16
convé-17 17
0.268 + 0.012
c00v4-18 18
0.279 + 0.11
conv5-1
+
00005-2
+
2o0v5:3
1125% + 0.122
200Y5
01308 + 0.010
20065-6
+
200v5:7
01:30 + 0.007
0x0e5-8
(1:37 + 0.111
+
avere-ppool
-: 0.00
fally-eoneetee
11266. + 0.014
softmax
0.245 + 0.010
C.4 Memorization in SL vs. SSL
We conducted an additional experiment where we trained a ResNe19 on CIFARI00 with SSL (SimCLR) and SL (cross-entropy loss). For the SL model, we remove the classification layer to turn into encoder. Then, report linear probing accuracies on multiple downstream tasks in Table 20 Our results highlight that the SI. pretrained encoders exhibit a significantly higher downstream accuracy on their pretraining dataset than the SSL encoder. We assume that this because of the class memorization. In contrast, the SL pretrained encoders perform significantly worse on other datasets than the SSL pretrained encoders since they might overfit the representations to their classes rather than provide more general (instance- we note that prior work has shown that the MAE encoder provides the highest performance when few last layers are fine-tuned. The results in the original MAE paper in Figure 9 [30] indicate that fine-tuning a few last layers/blocks (e.g., 4 or 6 blocks out of 24 ViT-Large) can achieve accuracy close to full fine-tuning (when all 24 blocks are fine-tuned). This is in line with our observation that the difference between UnitMem and ClassMem is the highest in the few last layers/blocks. Thus, fine-tunie fine-tuning only these last layers/blocks suffices for good downstream performance.
representations as the SSL encoders. Additionally,
23

---

Table 18: Layer-based Memorization Scores. We present the layer-wise memorization of an SSL encoder pretrained on CIFARIO using ResNet9 with SimCLR. The 1st column represents the IDs of convolutional layers and the 2nd column shows the name of the layers. Residual denotes that the residual connection comes from the previous N-th convolutional layer. We report LayerMem across the 100 randomly chosen training data points, their ALayerMem (denoted as ALM), followed by LayerMem for only the Top 50 memorized data points, their ALayerMem (denoted as AT)p50), and LayerMem for only the Least 50 memorized data points. The projection head layer (denoted as head) is used only for training.
ID
Name
Layerlen
AIM
Layertien Top00
ATop
LayerHin Leas550
Come1
0.091
0.144
0.003
BNI
MaxPool
0:014
Cams2-
0.123
0.026
0.225
0.067
0.012
BN2-0
0.124
0.001
0.225
(1
0.012
MaxPoel
0.011
Com-2-1
0.072
Cons2-2
0.183
0.029
0,4022
0.094
0031
Residual2
0.185
0.002
0.413
0.01
0041
Comv3
0.076
0.51
BN3
0:001
MaxPooll
0.215
0.004
0.486
0.006
0.000
Cono4-
0.246
0.031
0,599
0,113
0.061
BN4-0
MaxPool
0.003
Conoá_1
0.276
0.029
0.667
0.094
0.071
Com-4-2
0.308
0,032
0.817
0,120
0.073
Residunis
head
0,819
0.002
0.077
MaxPooll
0.118
.0.001
0,819
0
0.86
00.26
0.410
Table 19: LayerMem is not sensitive to the number of samples used for its calculation. We pre-train a ResNet9 using SimCLR on CIFAR1O and determined LayerMem on batches of different sizes. For each batch size, report the average LayerMem score and its standard deviation. The results show that the reported LayerMem score is, indeed, similar across all setups. This indicates LayerMem 's insensitivity the choice of the batch used to compute it.
three independent seeds (i.e., different batch compositions) and
Layer
100 samples
500 samples
1000 samples
5000 samples
0.002 1: Se
0.093
76-4
0.089 1: 76-4
0
:: Re-4
0.122 + 9e
0.124
1e-3
0.122
+: 8e-4
0.121
7
0.150 1: le:3
0.154
9c-4
0.151
8e-4
0.152
6e-
Ic.3
0:182
86-4
0.181
No-4
Res2
0.184 9e-4
0.185
8e-4
0.183
9p-4
0.184
6c
S
0.213 1 Se-1
0.213
8e-4
0.212
1: 7e-4
0.212
Be
0.246
le:3
0.249
76-4
8e-4
0.245
9c-4
7e-4
0
8e-4
0.276
0.309
9c-1
0,314
8e-4
0.310
1 7e-4
0.307
7e-4
Resó
0.310 +: 1e:3
0,316
16-3
0,313
+: 8c-4
0,309
9c-4
C,5 Visualization for Variability and Consistency of Memorization cross Different Layers.
We present the top 10 most memorized samples of each layer for the ResNet9 vision encoder trained with the CIFARIO 10 dataset in Figure 13 The results show that the overlap within the top 10 most memorized samples between adjacent layers is usually high but decreases the further the layers separated. This aligns with the results of overlap rate and Kendall's Tau test reported in Table 3
C.6 Layer-based Memorization Across Different SSL Frameworks and Datasets
We present the full results for the Table which show that the layer-based memorization is similar across encoders trained with different SSL frameworks. The results for the ResNet50 architecture trained with SimCLR and DINO using the ImageNet dataset are presented in Table 29 and the results for the ViT-Base architecture trained with MAE and DINO using the ImageNet dataset are shown in Table 31
24

---

Table 20: Comparing the impact of memorization on downstream generalization between SSL and SL. We train ResNet9 on CIFAR 100 with SSL (pretrained on CIFAR 100 using SimCLR and SL (cross-entropy loss, trained until convergence). For the SI. model, we remove the classification layer to turn it into an encoder. Then, we report linear probing accuracies multiple downstream tasks in
SST.
65.4% 0.900 1.96%%
$7.68 0.87%
48. 7% 989%
50 e 7696%
(trained until cunvergence CIF,R100, last lay removed)
1% 1.12%
56.7% 0.8.3%
46.1% 1.04%
58.65 58.6% 0.82%
Conv. 1
Conv. 2
Conv. 3
Conv. 4
Conv. 5
Conv. 6
Conv. 7
Conv. 8
Figure 13: The most memorized samples per layer according to LayerMem.
C.7 Verification of Layer-Based Memorization
To analyze whether our LayerMem metric and its variant indeed localize memorization correctly, we first replace different layers of encoder and then compute linear probing accuracy various downstream tasks. Since prior work shows that memorization in SSL is required for downstream generalization [47). we expect the highest performance drop when replacing the layers identified as most memorizing. We verify this hypothesis and train a ResNet9 encoder f1 on the CIFAR1O dataset and compute the LayerMem and ALayerMem scores per layer. Then, select the three most memorized, random, and least memorized layers and replace them with the corresponding layers from ResNet9 trained STL.10 (f2). Our results in Table 21 show that the highest linear probing accuracy drop on the CIFARIO 10 test set for f1 is caused by replacing the three most memorized layers from fi according the ALayerMem score. The second biggest drop observed when replacing according to LayerMem, highlighting that indeed our LayerMem metric and its A variant identify the most crucial layers SSL encoders for memorization. Surprisingly, the replacement of the layers in fi with the corresponding layers from fu causes the biggest simultaneous increase in the downstream accuracy for the STL10 dataset. We observe the same trends when f2 is trained on SVHN (Table 22b for replacing single layers in ResNet9 (Appendix C.7) and replacing whole blocks in ResNet50 (Appendix C.7) instead of only individual layers as we present in Appendix C.7 The above analysis
verifies that the LayerMem score and its A variant identify the most crucial layers in SSL encoders.
They further strengthen the claims that memorization is required for generalization 23 24 47
Replacing layers in ResNet9 for SVHN. and least vs random layers of CIFARTO trained ResNet9 on the downstream performance. We replace the layers with the corresponding ones from ResNet9 encoder trained SVHN.
In Table 22b we show the effect of replacing the most
25

---

Table 21: Replacing the most/least memorized layers according to ALayerMem causes the most/least changes in downstream performance. We study the effect of replacing layers of the ResNet9 encoder trained on CIFARTO with layers from another ResNet9 encoder trained on STL10 and report the linear probing accuracy on the CIFAR1O 100 and STL10 test sets. Results for the impact of replacing any combination of 1, 2, and layers on downstream accuracy are shown Appendix C.12
Reglacement Criteria
Reglaced Layeris)
CIPARIO
Nome (Baxeline)
69.08%
Most Memortzed ALayarlian
468
36.5% : 11.3%
32.33%
:: 0..8%
Mood Memorized Lagerteex
678
39,07% + 105%
29,82%
4 -0.91%
Random
41.22%
Least Mottrizzd LayerHon
49.95%
1.21%
Least Memorized Alagerier
235
89.14%
1 0.91%
23.10% 1 1.06%
Table 22: Evaluating the effect of replacing layers of the ResNet9 encoder pre-trained on CIFARIO with layers from ResNet9 pre-trained on STL10. We report the linear probing accuracy of ResNet9 with the replaced layers and tested on the CIFARIO 10, STL test sets.
(a) CIFAR1O & STL10
Replaced Criterium
Replaced Layor(s)
CIFARIO
STLI0
(Baseline)
None
69.08% 1.5%
17.81% +
Must Memorized (della)
468
36.59% 1. 1.13%
32.33%
1. 0.88%
Memorized (absolute)
19.07% 1.05%
29.82%
Random
4.577
43.22% + 1.08%
25.89%
+ 0.93%
Least Memorized (delta)
235
59. 59.14% 14%
0.91%
23.10%
1: 1.06%
Least Memorized (absolute)
234
49.95%
1.21%
24.71% 0.99%
(b) CIFARIO & SVHN
Replaced Criterium
Replaced Layer(s)
CIFAR10
SVHN
None (Raseline)
69.08% 1. 1.05%
19.33%
0.65%
Most Memorized (delta)
468
33.07% + 1.51%
34.05%
1.01%
Most Momorized (absolute)
6
34.97% + 0.84%
31.87%
1.21%
Random
4.57
39.28% + 0.74%
26.04%
0.82%
Least Memorized (delta)
23.5
52.81% 1 1.03%
21.05%
0.89%
Least Memorized (absolute)
234
45.39% 1 1.10%
24.66%
0.57%
Replacing blocks in ResNet50 trained on CIFARIO with SimCLR. We present the results for replacing blocks in ResNet50 trained on CIFAR10 using SimCLR in Appendix C.7
Statistics of Batch-Norm layer for different datasets. Batch-norm layers between different datasets might have different statistics. This could impact the downstream performance. To investigate the changes, measured the cosine similarity between the weights and biases of the batch-norm layers for two encoders (trained on CIFAR10 and STL10, respectively). The results in Table 25 show all 0ayers=0.823). This suggests that the statistics
high per-layer cosine similarity (average similar, hence, no adjustment is required. We hypothesize that the similarity stems from the fact that the data distributions similar and that normalize both input datasets according to the ImageNet normalization parameters.
C.8 LayerMem with Different Distance Metrics
In addition to the & distance, we also used 3 other distance metrics (L,, cosine similarity, and angular distance) to evaluate the stability of LayerMem. Our results in Table 2 highlight that 1) the memorization scores are very similar, independent of the choice of the distance metric, and 2) the most memorizing layers according to and ALayerMem are the same over all metrics. This suggests that our findings are independent of the choice of distance metric.
26

---

Table 23: Evaluating the effect of replacing layers of the ResNet9 encoder pre-trained CIFAR1O with layers from ResNet9 pre-trained on STL10. We report the linear probing accuracy of ResNet9 with the replaced layers and tested on the CIFARIO, 10, STL10 test sets.
(a) CIFARIO STLI0
Replaced Criterium
Replaced Layer(s)
CIFARIO
STL10
None (Raseline)
Nome
69.08% + 1.05%
17.81% + 0.92%
Most Memorized (delta)
59,84% 1.20%
21.98% - 0.41%
Most Memorized (absolute)
8
60.02% 0.94%
21.67% + 0.72%
Random
5
62.98% + 0.57%
20.44% + 0.85%
Least Memorized (delta)
65.52% 0.74%
18.94%
0.63%
Least Memorized (absolute)
64.21% 1: 1.08%
18.89% 1 0.81%
(b) CIFARIO & SVHN
Replaced Criterium
Replaced Layer(s)
CIFARIO
SVHN
None
69.08% + 1.05%
19.33% + 0.68%
Most Memorized (delta)
6
59.22%
0.97%
22.47% + 0.57%
Most Memorized (absolutee)
8
59.69%
1.04%
21.60%
1 0.92%
Random
5
61.07% + 1.12%
21.09%
+ 0.69%
Memorized (dellu)
1.88%
I Least Memorized (absolute)
3
62.35% 4 0.81%
20.18% + 0.98%
Table 24: Evaluating the effect of replacing blocks of ResNet50 pre-trained CIFAR10 with blocks from ResNet50 pre-trained on STL10 and SVHN. The accuracy in the table is the linear probing accuracy of ResNet50 on CIFARIO. We replace block of conv layers, which was selected according to the biggest A LayerMem between two layers (not the absolute value of the LayerMem score of the layers).
(a) CIFAR10 & STL10
Replaced Criterium
Replaced Layor(s)
CIFARI0
STL10
None
77.12%
1.42%
18.22% + 0.88%
Most Memorized
C4 B6 C3 B4
C2 B3
43.66% 1: 1.20%
25.78% 1: 0.95%
Random
C3
C4
B4
CS_B2
51.09%
1.01%
22.55%
1: 1.17%
Least Memorized
C2
B1
C2
B22B220
C3
C3
57.41% + 0.74%
20.10% + 1.11%
(b) CIFARIO & SVHN
Replaced Criterium
Replaced Layer(s)
CIFARI0
SVHN
None
77.12% 1: 1.42%
28.44% 1: 1.23%
Most Memorized
C6 C3 B4
C2 B3
35.21%
0.94%
38.11% + 1.08%
Random
C3 B2
B4
CS B2
44.19%
0.97%
32.44%
1. 1.25%
Least Memorized
BI C2 B2
C3 B3
49.06% + 1.31%
29.98%
+ 0.85%
C.9 LayerMem with Different Augmentation Strength
The results, reported in Table 27 highlight that stronger training augmentations reduce LayerMem.
C.10 LayerMem with Different Initialization of Trainable parameters
We performed an additional experiment where we trained encoders f and g inde- pendently with a different random seed (yielding f' and g') to study how random initialization of trainable parameters can affect the memorization of final vision macoder:The results are reported in Table 28 We compared the overlap in most memorized samples between encoder f (from the paper) and f'. The results (Table 4, attached PDF) show that overlap overall high (min. 69% in layer 2) and increases in the later layers (max. 90%, final layer).
27

---

Table 25: Cosine similarities between batch-norm layer outputs for ResNet9 trained on CIFAR10 and STL10. We normalize the training data according to the ImageNet parameters and train the encoders using SimCLR. We calculate the cosine similarity over the weights (7) and the bias (3) of the respective encoders' trained batch-norm layers.
Layer
Cosine Similarity
0.797 0.844
0.823
0.811
0.779
0.805
0.847
0.881
Table 26: LayerMem (LM) and A-LM under different distance metries. We report for L,, lz (see original submission), cosine similarity (Cos. Sim), and angular distance (Ang. Dist). The results highlight that our memorization measure is independent of the underlying metric. (ResNet9, CIFAR1O, SimCLR).
Cos. Sim.
Ang. Dist.
Layer
ALM
LM
ALM
LM
ALM
ALM
0,099
0.091
0.104
0.096
2
0.128
0.029
0.123
0.032
0.134
0.030
0.128
0.032
3
0.159
0,031
0.154
0,031
0.163
0,029
0.160
0.032
4
0.187
0.028
0.183
0.029
0.190
0.027
0.191
0.031
Res2
0.192
0.005
0.185
0.002
0.193
0.003
0.193
0.002
0,221
0.029
0.212
0.027
0.220
0.027
0.222
0.029
6
0,256
0.035
0,246
0.034
0,256
0.036
0.259
0.037
7
0.289
0,033
0.276
0.030
0.288
0,032
0.293
0.034
8
0.325
0.036
0.308
0.032
0.321
0.033
0.328
0,035
Res6
0.329
0.004
0.311
0.003
0.323
0.002
0.329
0.001
C.11 Impact of Layer Replacement Layer Memorization
According to the definition of SSLMem Equation (2), we let the representations of a given input data point æ pass through the same (replaced) layer in both f and g. We show the LayerMem and ALayerMem scores after replacing a single layer in the ResNet9 encoder pre-trained using SimCLR on the CIFAR10 dataset in Table 23 and Table 30 The LayerMem score of the replaced layer always drops as expected since this layer does not memorize any original training data points. The decrease in LayerMem between the initial and replaced layers is smaller in the earlier layers (e.g., 1st layer) as compared to the later layers (e.g., 6th layer). This might be because, in general, the earlier layers from different models might be more similar as they are responsible for extracting general features instead of specific ones for given dataset. The most important take-away from these experiments is that the ALayerMem is not affected significantly and its values show the same trends after the layer replacement.
C.12 Layer Replacement for Single, Two, and Three Layers at Time
We perform the experiment with the replacement of layer in Table 30 2 layers Table 32 and 3 layers Table 33 The following results confirm our results from Table in the main paper. When only a single layer is replaced, then the 6th (not the last layer) is the most important one. This layer had the highest LayerMem score. Note that the replacement of the 6th layer causes the highest drop in accuracy on the original CIFAR10 10 dataset and the highest gain in accuracy on STL10. Next, when two layers are replaced, then layers 6th and 8th play the most important roles, where
28

---

Table 27: LayerMem (LM) and Á-LM under different augmentation sets used during training. We use the augmentations defined in Appendix during training and metric calculation. The results show that stronger augmentations reduce memorization. (ResNet9, CIFARIO, SimCIR).
weak
normal
strong
Layer
LM
ALM
LM
ALM
LM
ALM
1
0.092
0.091
0.089
2
0.123
0.031
0.123
0.032
0.120
0.031
3
0,154
0.031
0.154
0.031
0.150
0.030
4
0.184
0.030
0.183
0.029
0.178
0,028
Res2
0.187
0.003
0.185
0.002
0.181
0,003
5
0.215
0.028
0.212
0.027
0.208
0.027
6
0.249
0.034
0.246
0.034
0.241
0.033
0,280
0.031
0.276
0.030
0.269
0.028
8 0.313
0.033
0,308
0.032
0.300
0,031
Res6
0.315
0.002
0.311
0.003
0.302
0.002
Table 28: Overlap in 100 most memorized samples according LayerMem between 2 different encoders, We train encoders with different seeds and report the per-layer overlap in their most memorized samples. We observe an overall high overlap, especially in the last layer.
Layer
3
4
6
7
Overlap %
73
69
75
89
85
88
86
90
their replacement with layers from the encoder trained on STL10 causes the highest drop on CIFAR10 and the highest performance increase on STL10. This contrary to the common intuition, which would suggest the replacement of the last two layers instead.
D Additional Setup
D.1 Class Selectivity
We denote the class selectivity metric as ClassSelectivity. It was proposed by [39 to quantify a unit's discriminability between different classes and described more in the main part of the paper in Section 5 We derive the basic metric in more detail here.
To compute the ClassSeecctivity metric per unit u, first the class-conditional mean activity is calculated for the test dataset D. We denote each test data point as Ij E D. We assume . classes Ch each with its corresponding test data points Ie Cj, where с 1, 2, .... |C₁l.
We define the mean activation ju of unit u for class C₁ as
P.. (C,
activation, (T.),
(9)
C
E,EC;
where the activation for convolutional feature maps is averaged across all elements of the feature map. Further, for the unit u, we compute the maximum mean activation Mmarz across all classes C, where M |C as
fmaxx = max (f (f (T: [M
(10)
29

---

Let p be the index location of the maximum mean activation Puu (Tp), i.e., the argmaz. Then, we calculate the corresponding mean activity μ pl-maz,u across all the remaining M - classes as
M
P mamaa,u
Σ (C(C)
Bu
(11)
M
j=1jAP
Finally, the class selectivity is then calculated as follows
ClassSelectivity(u) = Dmaas. 1M-.0.
(12)
fmazz, + ft- 1--maz,u
where Maaz, represents the highest class-conditional mean activity and ft-mar denotes the mean activity across all other classes (for unit u and computed on the test dataset D).
D.2 Class Memorization
We use similar definition as ClassSelectivity for ClassMem, which measures how much a given unit is responsible for the memorization of a class. While ClassSelectivity is calculated on the test set, we compute ClassMem on the training dataset.
To compute the ClassMem metric per unit u, first, the class-conditional mean activity is calculated for the training dataset D'. We denote each train data point x; D'. We assume M classes j=1. each with its corresponding train data points Te E Cj, where 1, 2, |Cyl.
We define the mean activation jt of unit u for class Cj as
(C,
activation, (Ix),
(13)
[c,l Ec
where the activation for convolutional feature maps is averaged across all elements of the feature map. Further, for the unit u, we compute the maximum mean activation Mmarz, across all classes C, where M = |C " as
Pmazz = max
(14)
Let p be the index location of the maximum mean activation (Tp) , i.e., the argmax. Then, we calculate the corresponding mean activity ğ. pi-mar,u across all the remaining M classes as
M
Σ
(,,(C
(15)
H-mo3.a
M
j=li$p
Finally, the class Memorization is then calculated as follows
ClassMem(1) Jimaz, Jo. maz,u
(16)
Amarz + fi-maz,
where Amaaz represents the highest class-conditional mean activity and ğ maz,az denotes the mean activity across all other classes (for unit and computed on the train dataset D
30

---

E Extended Related Work
Localizing Memorization on the Level of Individual Units. In Section 5 we considered memorization from the perspective of individual units and identified that pruning the lcast/most memorized units according to UnitMem preserves the Ieast/most performance (as shown in Table Table 6p. The work by Maini et al. [35] characterized individual examples as mislabeled based on the low number of channels or filters that need to be zeroed out to flip the prediction. They observe that significantly more neurons need to be zeroed out to flip clean examples compared to mislabeled ones.
A similar experiment in the SSL domain could potentially reveal similar trend, where noisy examples are harder to learn and primarily influence a small number of units. However, SSL encoders do not have discrete output changes from zeroing out individual units. One could pre-train the encoder and add linear probing, but this would require labels for the SSL training set, making it inapplicable. Even with labeled data and fine-tuning, identifying noisy SSL examples based on the SSLMem score may not match mislabeled examples in SL. The lack of discrete oracle and the potential mismatch between noisy SSL and mislabeled SL examples makes it difficult to identify individual units responsible for predictions of selected examples using prior methods.
F
Impact & Limitations
The fact that memorization can enable privacy attacks, such as data extraction 9 12), has been established in prior work. Yet, this paper advances the field of machine learning towards novel fundamental understanding on where in SSL encoders memorization happens, and how memorization differs between standard SL models and SSL encoders. Our insights hold the potential to yield societal benefits in the form of the design of novel methods to reduce memorization, improve fine-tuning, and yield better model pruning algorithms.
10
31

---

G
Additional Results
Table 29: All-layer memorization. We train the ResNer50 encoder using SimCLR and DINO SSL frameworks on the ImageNet dataset. We report the full results with the LayerMem and ALayerMem scores for each layer.
KesNet50l Layer
SinCLR
DINO
LayerHen
AlayarMee
Laywrlen
AlayarHens
co0v 1
0.038 1: 0.001
0,010 1: 0.002
pool
01009 +
0.340
0.002
c08V2-1
00ev2-2
0.044 1: 0.002
0.003
0.045 1 0.001
0.002
006V2-3
0.048 + 0.001
0.04
0,048 + 0.002
01033
008V2
0,052 1 0.002
0,052 0.001
000V2-5
+
c06v2-6
0,09 + 0.002
0.04
0,000 + 0.001
0.04
008V2
0,063 1 0.001
0.011
0,065 1 0.001
0.005
c08V2-8
006V2-9
00ev3-1
0,077 1 0.002
0.005
0,073 1 0.002
0,005
008V3-2
0.011 + 0.003
0.044
0,033 + 0.001
0.005
3
0 +
0.002
000v3-4
0A22 + 0001
0,044
c00V3.5
0.0N7 + 0.001
0.005
0.09 + 0.002
0,005
6
0.102 + 0.002
0.006
0.101 + 0.001
0.005
c0nv3.7
0.001
000V3-8
+
copv3-9
0.117 + 00001
0.005
0.120 + 0.001
0,008
000y3-10
0,123 + 0.022
0.066
0.136 + 0.002
0.006
comv3-11
0-113-12
0,154 + 0.02
0.1%6 + 0.002
conv4.1
0.139 + 0.02
0.005
0.142 + 0.002
0.006
008v4-2
0.145 + 0.022
0.006
0.143 + 0.033
0.006
coev4-3
0003
000v4-4
0,156 +
0.003
conv4.5
01161 + 0002
0.005
0.164 + 0.003
0,005
00v4-6
0,166 + 0003
0.05
0,149 + 0,004
0.005
comv4-7
conv4-8
0,178 + 0.03
0.006
0 1 0.003
0.006
00144.
018D 1 0002
0.03
0.186 + 0.002
0.005
c00v4.10
0 + 0.03
0.066
0.192 + 0.003
0.006
comv4-11
008v4-12
0,00 +
+ 0.005
0.005
comvi.13
(1207 + 0.066
0.07
0.210 + 0.003
0.007
0.ev4-14
0213 1 0.02
0.05
0,316 1 0.004
0.006
c0mv4-15
c0nv4-16
0,244 +
0.005
0.007
1o1v4-17
0229 + 0.03
0.005
0.234 + 0.003
0.006
008v4-18
0235 + 0.005
0.06
0.340 + 0.003
0.006
0oev5.1
0.033
00e55.3
0240 +: 0.03
0.254 1 0.005
0.007
00nv5.3
(12T +: 0.022
0.088
0.261 + 0.004
0.007
wov54
0263 1: 0.002
0.008
0,269 1 0.005
0.008
owv5.55
0272 0004
006V566
0272 +
+
0.007
omn55-7
028T 1: 0.003
0.008
0.290 1 0.006
0.009
owy5.8
0,205 1 0.03
0.08
0,300 1 0.004
0.088
0oev5.9
Table 30: Replace a single layer. We follow the settings from the Table 21 (same encoder) and replace a single layer at a time.
Replaced Layers
CIFAR11
$T1.10
None
69.57%6s
20199%
63.07%
18.79%
65. 65.19%10.87%
19.01%
64.47% 15%
18, 99%
60.2916
20.44%5:50
62.74%
19.93%
39.91% 09%
21 21.92%500675 67%
60.77% 20.75 75%
20.97% 52%
60.0145
65 (79
21 90 67%(6)
min( Xa)
50.91 (099)
18. 70% (0.69(2))))
32

---

Table 31: All-layer memorization. We train the ViT-Base encoder using MAE and DINO SSL frameworks on the ImageNet dataset. We report the full results with the LayerMem and ALayerMem scores for each block.
ViT-Base Block Number
DINO
LayerNex
&layerKan
LayorNes
Ataye:Han
0.000
0.019-10.001
0.037-0.001
0.011
0.006-10.002
0.012
0.055
0.013
0.066:0.003
0.012
0.07++000022
0.09::00002
0.012
0.096:10.044
0.013
6
0.118:0.0044
0.016
0.119:20.006
0.15
7
0.139
0.015
0,002:00.044
0,14
0.163:10.05
0.06000000
9
0.088
0.017
0.39-000000
0018
10
0.215
0.018
0.219:10.006
0019
11
0.44=10.005
0.021
0.207
0,020
0.201-00003
02754+0.04
Table 32: Replace two layers. We follow the setting from the Table 21 (same encoder) and replace two layers at time.
Repliced Layars
CIFARIO 68.37% 107%
STI.I0
None
18,45% 0.64%
52.51% 0.83%
20.80% 0.60%
50.77% 0.78%
18% 0.66%
155
50.98% 0.97%
22.02% 0.39%
17
34,87% 66%
L8
45 96% 0.94%
21.66% 0.71%
23
55.44% 0.73%
19.31% 0. 72%
53.61%
24.18%
25
53.30% 1.06%
20.34% 0.57%
26
48,59 0.51%
23.32% O. 70%
27
51.7%
21,91%
22.57%
34
99% 1.01%
21, 1.00%
73%
35
$2.0 67. 0.90%
21.00% 0.81%
36
22% 7999
23.18% 0.62%
22.09%
38
49.07% 0.92%
23%1 0.67%
45
50.09% 0.96%
22. 41% 0.55%
46
24.00%
48
45 09% 0.75%
21.11% 0.66%
46.02% 1.00%
24.29% 0.11%
49.21%
22.99% 0.80%
45.71%
24.33%
6.7
44.76% 0.88%
24.90% 0.54%
68
44.13% 1.01%
25.08% 0.64%
0.94%
24.1%
min (3layerX)
55,44%
25.08% (68)
min
(-laye(X)
44.15%
1.01% (6.8)
19.51% 0.72% (2.3)
33

---

Table 33: Replace three layers. We follow the settings from the Table three layers at a time.
(same encoder) and replace
Replaced Layers
CIFARIO
STLI0
None
60.37%
18.44% 0.64%
49.77%
21.95% 0.42%
48.33% 0.65%
23.28% + 0.77%
2.5
49.31% 0 70%
22,185 a 0.57%
126
45.31%
26.41% 0.539
48.99%
22.54%
128
46, 29% 0.48%
25. 74% à 0.62%
134
47 66% 0 525
24, 37% + 0.425
22.20%
136
45. 19% 72%
26.57% : 0.61%
137
48. 19% 5888
23. 21.76% + 0.82%
138
20% 0.83%
25.67% 0.60% 24, 99%
43.89%
20. 57%
147
44,33% 666
27. 55% 1 0.7 0.71%
148
43.9 3494%- 78%
29. 26 8 0,52%
1
48.03% 0.55%
23, 30% 1 0.63%
158
44,71 0 0.71%
26 90% 8 0.58%
43.30% 0.44%
29, 81% 0.57%
30.71%
178
59%
28 06 0.47%
23 4
48.89% 0.38%
22, 76% 1 0.38%
235
50.48% 067
21. 71% 1 0.60%
25 685
237
49.1%% 0.62%
22.31 1 0. 19%
238
48.07 18.07% 0.93%
23. 23.74% a 0. 70%
48.55% 0.79%
23. 90% 0.82%
247
47.78% 78% 68%
24.87% 2 0.
2
45.41% 0.36%
26, 95% 1 0.51%
255
45.91% 0.44%
26, 17% 0.600
22. 90%
258
47.18% 0.52%
25.57% A 0.81%
267
45.62% 0 0.69%
26. 78% 1 0.48%
0.63%
29 0.45%
25.77%
345
47.90% 56%
24.24% A 084
346
43.32% 0.58%
28, 73% : OLGUNE
347
45.80% 0.57%
26. 26.59% 0.65%
0.55%
356
45.49% 0.71%
26, 89% A 0.90%
357
48. 11% = 0.73%
23 56% & 0.58%
901 0.39%
44.01%
20%
368
42.00%% 0.65%
29, 93% A 0.33%
378 8
15.21% 0.43%
27. 2616 2 0.41%
436
0.70%
20% 0.53%E
27.64%
45 8
42.31% 0.82%
29, 81% E 0.33%
467
40.02% 0.71%
32.55% 1 0.58%E
31.94%
47%
40.96%
30.78% 0.561%
567
42.77% 0.00%%
29, 29.66% A 0.691S
568
40.55% 0 68%
31, 31.02% + 0.47%
0.91%
0.57%
31.59%
BaX (+laya:X)
50.48% + 0.67% (2 5)
31 14.9% + 01.8%% (4 (4)))
min (-layer2X)
37.66% a: 0.49% (4 6
21.71% :: 0.60%6 (2335)
Table 34: UnitMem distinguishes between individual examples within class. We use 1000 samples for each experiment to compute the UnitMem score. All: denotes all classes, TPC: stands for the following classes Truck, Plance, and Car classes, while Car: is simply the car class.
Layer
All
All
All
TPC
TPC
TPC
Car
Car
Car
Number
min
avg
min
avg
min
avg
Layer
010
034410.014
0366100011
0.077le-44
0.00
0.35700005
0e
0.82920.013
0.36020.011
Layer
0.006+96-5
083240.016
0.55240.010
D=O
0.78900015
0.350=0013
0.009-5e-5
0.810+0.011
0.351+0.013
Layer
010
0.1110.017
0.36310.008
0z0
0.800u0
0.355200010
0.010.9e-5
0.62540.000
0.35610.00
Layer
010
0.77100.112
0.377a0.009
0004ele-4
018330000
0,37350012
0.015
0.84420.014
0.371
Layer
0.01012e-4 4
0.5910.016
0.38120.008
0.01612e-4
0.810g0013
0.375g00111
0.013ale-4
0.83720.012
0.38010.010
Layer 6
0.0201:4e-4
090510.018
0.10310.0111
0.02233e-1
0.96
0.380
0.030s55e-4
0.87910.014
0,394100.007
Layer?
0.01943e-4
0.894:0.013
0339810.09
0.21133-44
0.380H0.033
0.019:55-4
0.85 0881100.17
0,387:0.01
Layer
0.01742e-4
0.90520.013
0.10920.010
0.255le-1
0.863200117
0.38520.010
0.021ate-4
0.870=0.013
0.397H0.012
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce our main contributions and key findings from line 5 to line 19 in the abstract and line 73 to line 80 in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss our Limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech- to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algo- rithms and how they scale with dataset size.
35

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All related results are clearly stated and referenced in either the main paper or the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: The detailed experimental setup is introduced in Appendix B and related source code is uploaded to open-review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
36

---

(a) the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: YYs]]
Justification: All related source code is uploaded to open-review and experiments are conducted on open source datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines dattps://nijg
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines for more details.
The authors should provide instructions on data access and preparation, in- cluding how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental Setting/eetails
37

---

Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The detailed experimental setup is introduced in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supple- mental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: For the average results of all multiple experiments in the paper, we report the standard deviation in the tables and draw the error bar used to represent the standard deviation in the figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
38

---

Answer:
[Yes]
Justification:
The hardware usage is introduced in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.com cc/public/
EthicsCuidelines?
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
In this work, we introduce two metrics for locating memorization in SSL vision encoders, as well as some key findings that result from experimenting with our metrics. No potential direct social impact is expected.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness con- siderations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considera- tions.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there
39

---

is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses ad- dition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datascts)?
Answer: [NA]
Justification: All datasets used to train models for this paper are safe public datasets, including ImageNet ILSRCC-011 42), CIFARIO [3], CIFAR100 [32],
SVHN [40]. and STL10
[18] (This is also introduced in Appendix B).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
of the model, for example
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All models and datasets usage is detailed introduced in Appendix B We totally abbey the terms of use of these public sources. All other code works are done by ourselves with no potential risks of licenses.
Guidelines:
40

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswithcoder com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: |Yes]
Justification: We provide all our code for metrics mentioned in our paper to open review and write detailed Readme file for the usage of our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember lo anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
41

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between insti- tutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
42


Reviews:
[{"review_id": "YUcJT88R8P", "reviewer": "Reviewer_BVAM", "summary": "This paper investigates memorization in self-supervised learning encoders. It introduces two metrics, LayerMem and UnitMem, to locate memorization on a layer-wise and unit-wise basis. These metrics provide insights into the distribution of memorization within neural networks. The study reveals that memorization occurs throughout the layers of SSL encoders, not just in the final layers, and that, in vision transformers, memorization primarily takes place in fully connected layers. Furthermore, the authors propose practical applications for their metrics, such as enhancing the efficiency of fine-tuning and pruning through memorization localization.", "questions": "- I am puzzled on why the memorization pattern remains the same between different datasets. For example in Figure 1  the histogram of UnitMem remains unchanged regardless of variations in the dataset or augmentations. Why does this happen? Do the authors have some suggested explanations for this?\n- Besides using these metrics in fine tuning are there other practical benefits to these proposed metrics?", "limitations": "The authors discuss their metrics limitations to some extent.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The empirical evaluations follow a logical flow, utilizing state-of-the-art architectures and datasets, resulting in informative results. Overall the empirical results are quite extensive and thorough.\n- The proposed metrics are computationally efficient and practically convenient, requiring only a forward pass and no labels. Additionally, the UnitMem metric offers an intuitive and insightful definition.\n- The paper is overall very well-written and the topic/motivation is interesting.", "weaknesses": "- There are some missing explanations on some of the observations. In addition to presenting empirical results, the authors should include explanations and insightful remarks to clarify their intuition and the potential implications of these results. This is done to some extent already, but would improve the paper if it is done in all the sections.\n- The presentation of the results could be improved. Tables full of numbers with tiny differences after the decimal point make it hard to read and navigate over. Instead of using tables that explain themselves, the paper relies on text explanations. \n- The proposed metrics, while inspired by previously known metrics, lack complete novelty. The LayerMem metric is primarily based on the existing SSLMem metric, with the only difference being a summation operation. This incremental modification does not constitute significant innovation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1: Adding insights and explanations** \n\nWe thank the reviewer for their suggestion. We went through the paper and identified the following sections for adding explanations and insightful remarks within the additional page that could be added to the paper in case of acceptance:\n- **Section 4, definition of LayerMem:** We will add further insights on how the use of the reference encoder g makes LayerMem independent of the internal representation dimensionality of the encoder. Therefore, we point to our experiments showing that LayerMem is independent of the internal representation dimensionality, since we always set two encoders’ internal layers with same dimensionality in relation.\n- **Section 4.1, line 195**: We will elaborate more why metrics, like LayerMem that measure memorization in the forward pass measure an accumulated effect of memorization. This is happening since the output of the previous layer influences the current one. This motivates the study of other metrics, such as Delta LayerMem, which consider layers more independently and can lead to more fine-grained insights on memorization.\n- **Section 4.1, line 219**: We will relate the observation that “fully-connected layers memorize more than attention layers” to the number of parameters in the layers. The fully connected layers have more storage capacity, which could, potentially have an impact on privacy-preserving architectural choices in neural networks.\n- **Section 5.1, line 314**: We will explain more on the memorization patterns between different datasets and how they relate to the distribution of atypical examples in the dataset.\n- **Section 5.1, line 327**: We will add insights on how our finding–that highest memorizing units memorize the highest memorized data points from prior work–suggests that there is a particular set of neurons responsible for memorization, and that the memorized data is not distributed over all neurons within the network.\n- Following the suggestion by Reviewer 2UD6, we also added additional insights into the consistency of data memorized in different layers (see response to the reviewer and attached PDF).\nIf reviewer see some other parts in the paper where we could add more explanations (as well as intuitions and potential implications), we’d be grateful for the suggestions. \n\n>**W2: Presentation of results** \n\nWe appreciate the reviewer’s suggestion. We wanted the tables to convey the full picture of results. We hope that the added insights (W1) will provide guidance to the reader on how to interpret the trends. Additionally, we performed additional experiments during the rebuttal (see general response, attached PDF, and responses to the other reviewers) and we tried to include more Figures (1 and 2) that would appeal to the readers with visual preference.\n\n**W3: Novelty** \n\nWhile our LayerMem builds heavily on SSLMem, it is only a part of our innovations. In summary, the paper makes the following original contributions:\n1. We introduce UnitMem as the first practical method to localize memorization in SSL encoders down to  the unit level (Section 5).\n2. We derive Delta LayerMem from LayerMem (which is indeed based on SSLMem), inspired by our insight that LayerMem does measure accumulated memorization rather than individual layers’ memorization.\n3. We perform thorough studies on localizing memorization in layers and units and leverage these insights to 1) inform the design on more effective fine-tuning approaches (Table 4) and 2) identify possible future pruning strategies to reduce both memorization and model complexity (Table 5). \n\n> **Q1a: Memorization pattern over different datasets** \n\nWhile the distributions of UnitMem scores (Figure 1a) look similar, the main difference is in the number of highly memorizing units (those that exhibit high values of UnitMem): the SVHN dataset, which is visually less complex than the CIFAR10 or STL10 dataset, has the lowest number of highly memorizing units. Additionally, all encoders considered in this work are trained with self-supervised learning objectives, yielding a similar memorization pattern.\n\n> **Q1b: Memorization pattern over different augmentations** \n\nNote that the different augmentation sets presented in the submission are only used at the inference time only **during the calculation of UnitMem** (see Equation 5), not during training. To study how different augmentations during training affect UnitMem, we performed additional experiments using **different augmentation strengths during training**.\nThe results, reported in Table 2 in the attached PDF highlight that stronger training augmentations lead to higher UnitMem. We hypothesize that this effect is caused by the model having to incorporate more information on the training samples to make up for the scarser input information (through the stronger augmentation). \n\n>**Q2: Practical Benefits** \n\nOur results suggest that UnitMem can also inform memorization-based compressing schemes for neural networks. As shown in Table 5, pruning the least memorized units preserves better downstream performance than just removing random units (while removing memorizing units harms downstream performance most). \nWe further hypothesize that UnitMem could yield a signal to strengthen white-box membership attacks against SSL encoders and that the knowledge of which neuron memorizes which data point from training could be used as a form of model watermark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "800XpMrdsP", "reviewer": "Reviewer_ne6k", "summary": "This paper identifies where memorization occurs in SSLs, noting that memorization increases in deeper network layers, though high-memorizing units are distributed throughout the network. For the first time, they introduce a metric to measure memorization in SSLs and provide justification for its validity. They demonstrate that memorization primarily occurs in the fully-connected layers of vision transformers.", "questions": "* When replacing the two layers of models trained on CIFAR-10 and STL-10, if the replaced layer is at the beginning of the network, don't the activations propagate through the entire network? I don't think I fully understood that experiment.\n\n* Is L2 distance used as the distance metric? If so, what is the justification for using it, and do the results change if a different metric is used?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This paper proposes a metric to define memorization in self-supervised models and identifies the locations where the most memorization occurs.", "weaknesses": "* It is not clear why the defined metric is a good measure of memorization, especially for comparing memorization across different layers. The magnitude of the distance between representations of different augmentations is highly dependent on the activation distribution of the layers and the distance metric used. While the normalization proposed in equations 2 and 4 attempts to address this, I am still unsure if it is sufficient.\n\n* To justify the validity of the LayerMem memorization metric, they argue that effective memorization in SSLs leads to strong downstream performance. They also point out that replacing the layers with the highest memorization scores, as determined by the metric, results in a more performance drop, indicating the metric's reliability. However, I believe this justification alone is insufficient. Are there other pieces of evidence supporting the validity of this memorization metric?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments.\n>**W1: It is not clear why the defined metric is a good measure of memorization, especially for comparing memorization across different layers. The magnitude of the distance between representations of different augmentations is highly dependent on the activation distribution of the layers and the distance metric used. While the normalization proposed in equations 2 and 4 attempts to address this, I am still unsure if it is sufficient.**\n\nTo avoid that our memorization measure overfits to a particular set of augmentations whose activations might indeed vary throughout layers, we measure our LayerMem as an expectation over different pairs of diverse augmentations. Following SSLMem, in our experiments, we calculate LayerMem as an average over 10 different randomly chosen pairs of augmentations from the full augmentation set used during training.\n\nThe only pitfall that could arise would be that layers with higher output dimensionality get a higher LayerMem (simply because there is more mass in the activation). To verify that this is not the case, we report for ResNet50 trained with SimCLR on CIFAR10 the LayerMem and output dimensionality for the last layers of each block. In the Table below, we see that the dimensionality of the representations after the 1st block and the 4th block are the same. However, the memorization score is much higher for the layer after the 4th block highlighting that our LayerMem is independent of the output dimensionality.\n| Output of model block  | LayerMem | Number of Dimensions in the representations | Alignment loss difference for f: d (f (x′), f (x′′)) | Alignment loss difference for g: d (g(x′), g(x′′)) |\n|:-:|:-:|:-:|:-:|:-:|\n| 1 (conv1) | 0.046 ± 0.006 | 64\\*56\\*56= 200704 | 163.25 | 167.56 |\n| 2 (conv2-9) | 0.103 ± 0.014 | 256\\*56\\*56 = 802816 | 304.65 | 315.98 |\n| 3 (conv3-12) | 0.165 ± 0.007 | 512\\*28\\*28 = 401408 | 187.33 | 221.82 |\n| 4 (conv4-18) | 0.279 ± 0.011 | 1024\\*14\\*14 =200704 | 98.21 | 129.6 |\n|5 (conv5-9) | 0.335 ± 0.013  | 2048\\*7\\*7 =100352 | 61.01 | 84.92 |\n\n>**W2: To justify the validity of the LayerMem memorization metric, they argue that effective memorization in SSLs leads to strong downstream performance. They also point out that replacing the layers with the highest memorization scores, as determined by the metric, results in a more performance drop, indicating the metric's reliability. However, I believe this justification alone is insufficient. Are there other pieces of evidence supporting the validity of this memorization metric?**\n\nThe finding that effective memorization leads to strong downstream performance has been proven for supervised learning by [23] and for SSL by [47].  We build on these prior findings and  show that LayerMem can help us to select layers for fine-tuning to achieve better downstream performance. Does the reviewer have any other suggestions on how we could provide additional evidence? We are happy to implement them.\n\n>**Q1: Layer Replacement**\n\nWe agree that swapping affects subsequent layers. If we replace layer N, the input to layer N+1 and **all subsequent layers** is altered. Following this logic of a cascading effect, replacing the earliest layers should result in the highest impact, simply because most layers are affected by the change. However, this is not what we experimentally observe (e.g., in Table 22). Instead we see that swapping out the most memorizing layer(s) causes the largest change in the downstream performance.\n\nWhat is the interesting insight from the results is that the replacement of different layers has a different impact on the test accuracy of STL10. In particular, we observe that replacing most memorized CIFAR10 encoder layers with their STL10 equivalent boosts the STL10 performance significantly more than replacing, for example, the least memorized or random layers. This indicates that the layers with the highest memorization have the highest impact on the downstream performance.\n\n>**Q2: Distance Metric**\n\nFollowing SSLMem, we used the $\\ell_2$ metric for the main paper. To address the reviewer’s questions, we performed an additional experiment reporting LayerMem with different distance metrics ($\\ell_1$ distance, cosine similarity, angular distance). Our results in the table below (see also Table 1 in the attached PDF) highlight that 1) the memorization scores are very similar, independent of the choice of the distance metric, and 2) the most memorizing layers according to LayerMem and Delta Layermem are the same over all metrics. This suggests that our findings are independent of the choice of distance metric.\n\n|  | L2 (from paper) |  | L1 |  | Cosine similarity |  | Angular distance |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Layer | LayerMem | Delta Layermem | LayerMem | Delta Layermem | LayerMem | Delta Layermem | LayerMem | Delta Layermem |\n| 1 | 0.091 | - | 0.099 | - | 0.104 |  | 0.096 | - |\n| 2 | 0.123 | 0.032 | 0.128 | 0.029 | 0.134 | 0.030 | 0.128 | 0.032 |\n| 3 | 0.154 | 0.031 | 0.159 | 0.031 | 0.163 | 0.029 | 0.160 | 0.032 |\n| 4 | 0.183 | 0.029 | 0.187 | 0.028 | 0.190 | 0.027 | 0.191 | 0.031 |\n| Res2 | 0.185 | 0.002 | 0.192 | 0.005 | 0.193 | 0.003 | 0.193 | 0.002 |\n| 5 | 0.212 | 0.027 | 0.221 | 0.029 | 0.220 | 0.027 | 0.222 | 0.029 |\n| 6 | 0.246 | **0.034** | 0.256 | 0.035 | 0.256 | **0.036** | 0.259 | **0.037** |\n| 7 | 0.276 | 0.030 | 0.289 | 0.033 | 0.288 | 0.032 | 0.293 | 0.034 |\n| 8 | 0.308 | 0.032 | 0.325 | **0.036** | 0.321 | 0.033 | 0.328 | 0.035 |\n| Res6 | **0.311** | 0.003 |**0.329** | 0.004 | **0.323** | 0.002 | **0.329** | 0.001 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vECCWg0irq", "reviewer": "Reviewer_GSFs", "summary": "This paper introduces two novel metrics, LayerMem and UnitMem, to measure where memorization occurs within self-supervised neural networks. Through extensive experiments, this paper finds that memorization increases with layer depth, highly memorizing units are distributed throughout the encoder, atypical data points cause higher memorization, and most memorization happens in fully connected layers in ViTs. These findings suggest that localizing memorization can enhance fine-tuning and inform pruning strategies for better model performance and efficiency.", "questions": "- In Figure 2, why does SSL UnitMem start with a non-zero value even at the first layer? I expected it would start near zero. Also, the values appear to be almost the same across all layers.\n- Although high LayerMem values are observed in the later layers, this might be insufficient to conclude that these layers memorize more than the early layers. I anticipated that the effect of memorization would accumulate as the network depth increases.\n- Please refer to the weaknesses section for additional comments.", "limitations": "Please refer to the weaknesses section. No ethical concerns found.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes the first metric to measure memorization in a self-supervised learning context.\n- It presents several interesting and novel findings. For example, it's insightful that the fully connected layers in ViTs memorize the dataset more than the self-attention layers, like in NLP tasks. Also, it is interesting to see that memorization does not significantly depend on the self-supervised learning methods employed.\n- Overall, the paper is well-written and logically organized.", "weaknesses": "- The primary limitations of this paper are the lack of in-depth analysis and practical insights. Although the paper provides a broad range of empirical findings, it could be significantly improved by exploring how and why the neural networks behave as they do. Alternatively, strongly linking to or elaborating on prior work might be a promising direction to enhance the paper. Moreover, it would be beneficial if the paper provided practical methods to improve self-supervised neural networks based on the given takeaways. Even though Section 4.2 was interesting and insightful, it is insufficient for applying these findings in a real-world setting.\n- If I understand correctly, I am not fully convinced by the definitions of `LayerMem` and `UnitMem` as appropriate metrics to measure memorization. For example, according to Equation 1, LayerMem seems to depend on the neural network’s properties. If neural networks are sufficiently equivariant with respect to data augmentation, then SSL and SSLMem should be zero. However, fortunately, the equivariance of ViTs is on par with that of CNNs [1], so I believe the main takeaways can hold in this case. Also, these metrics might depend on the norms of intermediate representations.\n- There is room for improvement in the writing. The paper claims that “memorization increases but not monotonically,” yet I observed that LayerMem increases monotonically and even uniformly, except in some layers, as shown in Tables 1, 14, 15, and 16. Some interesting results, e.g., Lines 247 and 387, are only included in the Appendix. The paper attempts to correlate atypical datasets and memorization, but the evidence does not strongly support the claims. It is a minor thing, but I’m not familiar with ‘SL’ as an abbreviation for supervised learning.\n\nOverall, considering the paper as one of the first attempts to investigate memorization in a self-supervised learning setting, I don’t find major weaknesses. Despite some limitations, I lean toward acceptance since I believe such attempts should be encouraged.\n\n[1] Gruver, Nate, et al. \"The lie derivative for measuring learned equivariance.\" arXiv preprint arXiv:2210.02984 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1a: In depth analysis and practical insights** \n\nWe thank the reviewer for their suggestion. We went through the paper and identified multiple sections for adding practical insights. We summarized them in the [first reply to W1 for Reviewer BVAM](https://openreview.net/forum?id=R46HGlIjcG&noteId=3o0QWN7snN).\n\nIf reviewer see some other parts in the paper where we could add more analyses (as well as intuitions and potential implications), we’d be grateful for the suggestions. \n\n>**W1b: Applying findings in real-world settings**\n\nIn addition to Section 4.2, we also provide a further real-world application of our metrics by showing that UnitMem can effectively be used to inform compression schemes. In particular, it seems to be most helpful to prune the least memorizing units (vs. random units), see Table 5. On the contrary, the most memorizing units should be preserved.\n\n>**W2a: For example, according to Equation 1, LayerMem seems to depend on the neural network’s properties. If neural networks are sufficiently equivariant with respect to data augmentation, then SSL and SSLMem should be zero.**\n\nCould the reviewer further clarify what they mean by SSL (and SSLMem) should be zero? We do agree that a perfectly (over)fitted SSL-trained encoder f can have a zero representation alignment, i.e. the representations of two different augmentations of the same input sample x are identical. However, SSLMem and therefore also our LayerMem rely on an additional reference model g and report the difference in representation alignment between f and g. \n\nIn case g has not seen x but still achieves perfect representation alignment means that the distribution of other data is sufficient to perfectly fit x, which also means that f does not need to memorize x. Then indeed LayerMem will be zero, correctly indicating no memorization. However, if the representation alignment for g is worse than for f, their difference will be non-zero, hence LayerMem will also be non-zero.\n\n>**W2b: Also, these metrics might depend on the norms of intermediate representations.**\n\nTo address the reviewer’s concern, we performed additional measurements on the outputs of each convolution block in ResNet50 trained with SimCLR on CIFAR10. In particular, we measured the alignment loss ($\\ell_2$ distance between the representations of two different augmentations) on the encoder f and g used for LayerMem. This is the signal that the metric operates on (not directly on the representations), and hence, its magnitude (norm) would influence the metric.\n\nIn the Table below, we observe that while the alignment loss first increases and then drops significantly, LayerMem increases continuously, suggesting that LayerMem does not depend on the norm of the representation alignment.\nThe magnitude of the norm could also be influenced by the dimensionality of the representation. Therefore, we also report output dimensionality. We observe that the dimensionality of the representations after the 1st block and the 4th block are the same. However, the memorization score is much higher for the layer after the 4th block highlighting that our LayerMem is also independent of the output dimensionality.\n\n | Output of model block  | LayerMem | Number of Dimensions in the representations | Alignment loss difference for f: d (f (x′), f (x′′)) | Alignment loss difference for g: d (g(x′), g(x′′)) |\n|:-:|:-:|:-:|:-:|:-:|\n|1(conv1)|0.046±0.006|64\\*56\\*56=200704|163.25|167.56|\n|2(conv2-9)|0.103±0.014|256\\*56\\*56=802816|304.65|315.98|\n|3(conv3-12)|0.165±0.007|512\\*28\\*28=401408|187.33|221.82|\n|4(conv4-18)|0.279±0.011|1024\\*14\\*14=200704|98.21|129.6|\n|5(conv5-9)|0.335±0.013|2048\\*7\\*7=100352|61.01|84.92|\n\n\n>**W3a: There is room for improvement in the writing. The paper claims that “memorization increases but not monotonically,” yet I observed that LayerMem increases monotonically and even uniformly [...].**\n\nThe quote used in the review “memorization increases but not monotonically” is not taken from our paper. Our sole statement about monotonicity in layer memorization is (in line 197) “Delta LayerMem indicates that the memorization increases in all the layers but is not monotonic.” We agree that LayerMem would indicate a monotonic increase (Table 1). As we note in the paper (line 195) “However, our Delta LayerMem presents the memorization from a more accurate perspective, where we discard the accumulated memorization from previous layers, including the residual connections.” Looking at the Delta LayerMem column in Table 1, we see that values go from 0.029->0.002->0.027…. which is not a monotonic increase. We changed the above statement to  “Delta LayerMem indicates that the memorization increases **overall with deeper layers** but it is not monotonic.”\n\n>**W3b: Some interesting results, e.g., Lines 247 and 387, are only included in the Appendix.**\n\nWe appreciate the reviewer’s suggestion on which results should be additionally featured in the main body of the paper and would use the additional page in case of acceptance for including the “Verification of Layer-Based Memorization” from Appendix C.6 and the “Additional Verification of UnitMem” experiments from Appendix C.1.\n\n>**W3c: The paper attempts to correlate atypical datasets and memorization, but the evidence does not strongly support the claims.**\n\nRelating atypical examples with memorization is not a contribution from our work. This relation has been established, e.g. by [23] and [47]. We only state their findings.\n\n\n>**Overall, considering the paper as one of the first attempts to investigate memorization in a self-supervised learning setting, I don’t find major weaknesses. Despite some limitations, I lean toward acceptance since I believe such attempts should be encouraged.**\n\nWe thank the reviewer for their encouraging feedback and are happy to provide further insights and clarifications that would strengthen the reviewer’s confidence for an acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2TwxqZLMuQ", "reviewer": "Reviewer_2UD6", "summary": "This work focuses on memorization in self-supervised learning. This paper seems to extent the work from [1] on SSLMem, that leverage the fact that a given training data and its corresponding handcrafted image augmentations should have a lower distance in the SSL embedding space than with a model in which this point was not in the training data. The authors extend SSLMem by using intermediate layers instead of the final representation. By doing so, the authors can localize which layers are the most memorizing training data. They show that indeed the last layer memorize the most, however we can still find significant memorization occurring also in intermediate layers. The author improve then their method by introducing a memorization metric based on individual units instead of a layer basis. \n\n[1] Memorization in self-supervised learning improves downstream generalization, Wang et al, ICLR 2024", "questions": "- When replacing random or most/less memorized layers, how do you deal with normalization layers like batch norm since the statistics might be off?\n- How much time does it take to train both models, compute augmentations and get the LayerMem and UnitMem metric?", "limitations": "The authors provide a limited limitation section. I would have appreciated to see as limitation that the method require training two disjoint model while having access to the original training data augmentation set. Thus, it does not seems that such method might work well on public model for which we do not know well the training details. Also the lack of a sample level analysis of the memorized example is an important limitation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written and the authors provide extensive experiments.  The author provide results on different architecture, SSL methods, training criteria and augmentations. The results are strong.", "weaknesses": "- The paper can be read as just an extension of [1], so this contribution could be seen as lacking novelty. However, the empirical analysis that is provided might still interest the research community.\n- I would be careful with claim such as being \"first metric for localizing memorization\". When we read Meehan et al., we can see that they did an ablation study across different layers, so they seem to be able to analyze which layer memorize the most. So it seems that there should have been a closer analysis and comparison with this paper results. In addition, they also provide experiments with ViT, so the author might also downplay the claim that investigating memorization in ViT is lacking.  So one weakness of this paper is the lack of empirical comparison with respect to the current literature. \n- Even if the authors did a great job in trying different models, they did not analyze on a sample level basis, how the memorized examples might differ depending on random seed or different hyper-parameters.  Like for two models that produces similar LayerMem score, are they memorizing the same examples/data points or not? Because both the overall score could be similar while memorizing different examples. A similar question is, does the memorize examples change across different layers? I would have appreciated reading at least some discussion on the relationship between model memorization with respect to which examples and how much those examples are consistent between models and layers.\n- Concerning the SSLMem metric, I am wondering what is the interplay between both models (training with and without specific points) initialization and hyper-parameters. Like I would suspect that depending on the data shuffling seed, the examples that might be memorized first in the early layers might differ.\n- There is also a lock of discussion on the impact of normalization layer or weight decay on the experimental results. I would suspect that having a stronger weight decay might reduce memorization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1: Novelty vs. extension of [1]**\n\nThe paper targets an orthogonal research question to [1]. While [1] is concerned with the question of **quantifying** memorization and  **identifying memorized samples**, we answer the question of **where inside the SSL encoders** is the information stored. \nWhile doing so, we make multiple original contributions as summarized in [W3 for Reviewer Reviewer BVAM](https://openreview.net/forum?id=R46HGlIjcG&noteId=3o0QWN7snN).\n\n>**W2a: Claims on being first metric for localizing memorization in SSL**\n\nWe wanted to frame our work as being the first metrics *particularly designed* for localizing memorization and the first that allow localization down to the level of individual units. To better express this framing according to the reviewer’s insightful remark, we altered the following sentences (change bolded):\n- **Introduction:** \"We propose LayerMem and UnitMem, **two** practical metrics that allow to localize memorization in SSL encoders on a per-layer basis and, **for the first time, down to the granularity of individual units.**”\n- **Conclusion:** \"We propose **[REMOVED]** practical metrics for localizing memorization within SSL encoders on a per-layer basis, and, **for the first time, down to the individual-unit level.**”\n\nAdditionally, we added in line 107 \"**The same trend was also reported for SSL [36]**\" to give credit to Meehan et al. [36] for their contribution to the field (see their Appendix A7).\n\n>**W2b: Downplaying claim on memorization in ViTs**\n\nWe also changed this claim in the paper (changes in bold):\n- **Introduction:** “Yet, with our methods to measure memorization, we **are able** to show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.”\n- **Line 217:** “The **localization of** memorization **in** transformers was primarily investigated in the language domain **[REMOVED]** **and** the methods for analysis and the findings are not easily transferable. Yet, with our methods to localize memorization, we **[REMOVED]** show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.”\n\n>**W2c: Empirical comparison to prior work**\n\nThe two existing works on memorization in SSL [36,47] (SSLMem and SSL Deja Vu) consider an orthogonal research question, namely **quantifying** memorization and **identifying memorized samples**. We, in contrast, focus on **localizing** memorized content. An empirical comparison does not seem sensible. \n\nYet, we can compare the trends reported by the ablation from Meehan et al. [36]  (their Appendix A7: Deja Vu memorization for 4 different layers of VICReg) to ours. They report that Deja Vu Memorization increases monotonically and occurs mainly in the last two layers. A similar trend for *accumulated* memorization is observed by our LayerMem, while our Delta LayerMem shows that the actual increase is not monotonic.\n\n>**W3a: Difference in memorized samples between two different encoders**\n\nThis interesting research question is orthogonal to our work where we identify **where in the encoder** training data is memorized.\n\n>**W3b: Difference in memorized samples across different layers**\n\nWe performed a new analysis in Figure 2 and Table 3 (attached PDF). The overlap within the 100 most memorized samples between adjacent layers is usually high but decreases the further the layers are separated. Our statistical analysis to compare the similarity of the orderings within different layers’ most memorized samples using the Kendall’s rank correlation coefficient shows that while for closer layers, we manage to reject the null hypothesis (“no correlation”) with high statistical confidence (low p value) which is not the case for further away layers. \n\n\n>**W4: Hyperparameters and variation in memorized samples.**\n\nWe performed an additional experiment where we trained encoders f and g independently with a different random seed (yielding f’ and g’). We compared the overlap in most memorized samples between encoder f (from the paper) and f’.\nThe results (Table 4, attached PDF) show that overlap is overall high (min. 69% in layer 2) and increases in the later layers (max. 90%, final layer).\n\n\n>**W5: Weight decay**\n\nWe performed the experiment suggested by the reviewer and trained ResNet9 using SimCLR on CIFAR10 with three different levels of weight decay. Our results in Figure 1 (attached PDF) show that stronger weight decay yields lower memorization, yet also decreases linear probing accuracy. \n  \n\n>**Q1: BatchNorm**\n\nWe measured the cosine similarity between the weights and biases of the BatchNorm layers for two encoders (trained on CIFAR10 and STL10). The results in Table 5 (attached PDF) show a high per-layer cosine similarity (average over all layers=0.823). This suggests that the statistics are not far off, hence, no adjustment is required. We hypothesize that the similarity stems from the fact that the data distributions are similar and that we normalize both input datasets according to the ImageNet normalization parameters.\n\n>**Q2: Compute times**\n\nFollowing our standard setup from Table 1 and Figure 1 from the main paper, we report the following timing on an Nvidia RTX4090 (24G), 64GB RAM (for ResNet9  trained 600/200 epochs):\n\n |Train a model|Compute UnitMem|Compute LayerMem|\n|-|-|-|\n|195min/63min|5min26sec|5min34sec|\n\n>**Limitation 1: Need of two models**\n\nThis limitation only holds for LayerMem. UnitMem operates directly on the given encoder without the need for a second model. We added the limitation of LayerMem to Appendix F.\n\n>**Limitation 2: Lack of sample-level analysis**\n\nThis is an orthogonal research question to this work. We do not aim to study memorization from the sample but from the encoder perspective, asking the question **where** inside the encoder training data is stored.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their insightful comments and questions. We are happy that the reviewers recognize our work: “presents several interesting and novel findings” (Reviewer GSFs). We are also glad the reviewers appreciate our new metrics to be “computationally efficient and practically convenient” (Reviewer BVAM)”, and our paper to provide thorough and extensive experiments (Reviewer BVAM, Reviewer 2UD6) which are presented in a well-written paper (Reviewer BVAM, Reviewer GSFs).

**Highlights of the rebuttal:**

During the rebuttal, we performed additional experiments and analyses that we present in the individual answers to the reviewers and in summary in the attached PDF.
1. Based on the suggestion by Reviewer ne6k, we show that while in the paper, we follow SSLMem and use the $\ell_2$ distance to compute LayerMem, our LayerMem is insensitive to the choice of the distance metric, and all additional metrics ($\ell_1$ Distance, Cosine Similarity, Angular Distance) yield similar results and trends (see Table 1 in the attached PDF).
2. While previously in the paper, we had only experimented with varying the augmentations used to compute UnitMem *during inference*, based on the suggestion by Reviewer BVAM, we varied the augmentations used *during training* and show that stronger augmentations yield higher memorization in individual units according to UnitMem  (see Table 2 in PDF).
3. Based on the suggestion by Reviewer 2UD6, we experimented additionally with varying the weight decay during training and find that with stronger weight decay, we observe less memorization at the costs of decreased downstream performance (see Figure 1 in PDF).
4. Finally, to provide further insights, in accordance with the suggestion by Reviewer 2UD6, we analyzed the variability and consistency between the samples memorized by different layers in the model. Our results (see Figure 2 and Table 3 in PDF) show the interesting trends that the most memorized samples vary between layers (more the further the layers are away from one another), and tend to be more similar between adjacent layers, in particular the last layers of the model.

Meta-review: This paper investigates memorization in self-supervised learning encoders. It introduces two metrics, LayerMem and UnitMem, to locate memorization on a layer-wise and unit-wise basis. These metrics provide insights into the distribution of memorization within neural networks. The study reveals that memorization occurs throughout the layers of SSL encoders, not just in the final layers, and that, in vision transformers, memorization primarily takes place in fully connected layers. Furthermore, the authors propose practical applications for their metrics, such as enhancing the efficiency of fine-tuning and pruning through memorization localization.

Please update the paper with the clarifications and discussions that took place during rebuttal period.
Decision: Accept (poster)