Localizing Memorization in SSL Vision Encoders

Authors: wenhao  wang
adam  dziedzic
michael  backes
franziska  boenisch
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 13:49:19

Abstract: Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap, we propose two metrics for localizing memorization in SSL encoders on a per-layer (LayerMem) and per-unit basis (UnitMem). Our localization methods are independent of the downstream task, do not require any label information, and can be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based) trained on diverse datasets with contrastive and non-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experiences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fully-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) ( [16, 17, 13, 4, 30, 27, 29] ) enables pre-training large encoders on unlabeled data to generate feature representations for a multitude of downstream tasks. Recently, it was found that, even though their training datasets are large, SSL encoders still memorize individual data points ( [36, 47] ). While prior work characterizes the memorized data and studies the effect of memorization to improve downstream generalization ( [47] ), little is known about where in SSL encoders memorization happens.The few works on localizing memorization are usually confined to supervised learning (SL) ( [3, 45, 35] ), or operate in the language domain ([55, 37, 7, 44, 14] ). In particular, most results are coarse-grained and localize memorization on a per-layer basis [3, 45] and/or require labels [3, 35] .To close the gap, we propose two novel metrics for localizing memorization in SSL encoders in the vision domain. Our LayerMem localizes memorization of the training data within the SSL encoders on a layer-level. For a more fine-grained localization, we turn to memorization in individual units (i.e., neurons in fully-connected layers or channels in convolutional layers). We propose UnitMem which measures memorization of individual training data points through the units' sensitivity to these points. Both our metrics can be computed independently of a downstream task, in a forward pass without gradient calculation, and without labels, which makes them computationally efficient and well-suited for the large SSL encoders pretrained on unlabeled data. By performing a systematic study on localizing memorization with our two metrics on various encoder architectures (convolutional and transformer-based) trained on diverse vision datasets with contrastive and non-contrastive SSL frameworks, we make the following key discoveries:Memorization happens through the entire SSL encoder. By analyzing our LayerMem scores between subsequent layers, we find that the highest memorizing layers in SSL are not necessarily the last ones, which is in line with findings recently reported for SL [35] . While there is a tendency that higher per-layer memorization can be observed in deeper layers, similar to SL [3, 45] , our analysis of memorization on a per-unit level highlights that highly memorizing units are distributed across the entire SSL encoder, and can also be found in the first layers.Units in SSL encoders experience high memorization. By analyzing SSL encoders with our UnitMem metric, we find that a significant fraction of their units experiences high memorization of individual training data points. This stands in contrast with models trained using SL for which we observe high class memorization, measured as the unit's sensitivity to any particular class. While these results are in line with the two learning paradigms' objectives where SL optimizes to separate different classes whereas SSL optimizes foremost for instance discrimination [48] , it is a novel discovery that this yields significantly different memorization patterns between SL and SSL down to the level of individual units.Atypical data points cause higher memorization in layers and units. While prior work has shown that SSL encoders overall memorize atypical data points more than standard data points [47] , our study reveals that the effect is constant throughout all encoder layers. Hence, there are no particular layers responsible for memorizing atypical data points, similarly as observed in SL [35] . Yet, memorization of atypical data points can be attributed on a unit-level where we observe that the highest memorizing units align with the highest memorized (atypical) data points and that overall atypical data points cause higher unit memorization than standard data points.Memorization in vision transformers happens mainly in the fully-connected layers. The memorization of transformers [46] was primarily investigated in the language domain [26, 43] . However, the understanding in the vision domain is lacking, and due to the difference in input and output tokens (language transformers operate on discrete tokens while vision transformers operate on continuous ones), the methods for analysis and the findings are not easily transferable. Yet, with our methods to localize memorization, we are the first to show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.Finally, we investigate future applications that could benefit from localizing memorization and identify more efficient fine-tuning and memorization-informed pruning strategies as promising directions.In summary, we make the following contributions:• We propose LayerMem and UnitMem, the first practical metrics to localize memorization in SSL encoders on a per-layer basis and down to the granularity of individual units. • We perform an extensive experimental evaluation to localize memorization in various encoder architectures trained on diverse vision datasets with different SSL frameworks. • Through our metrics, we gain new insights into the memorization patterns of SSL encoders and can compare them to the ones of SL models. • We show that the localization of memorization can yield practical benefits for encoder fine-tuning and pruning.2 Related Work SSL. SSL relies on large amounts of unlabeled data to train encoder models that return representations for a multitude of downstream tasks [6] . Especially in the vision domain, a wide range of SSL frameworks have recently been introduced [16, 17, 13, 4, 30, 27, 29] . Some of them rely on contrastive loss functions [16, 29, 27] whereas others train with non-contrastive objective functions [41, 17, 13, 30] .Memorization in SL. Memorization was extensively studied in SL [52, 2, 15] . In particular, it was shown that it can have a detrimental effect on data privacy, since it enables data extraction attacks [9, 10, 11] . At the same time, memorization seems to be required for generalization, in particular for long-tailed data distributions [23, 24] . It was also shown that harder or more atypical data points [2, 43] experience higher memorization. While all these works focus on studying memorization from the data perspective and concerning its impact on the learning algorithm, they do not consider where memorization happens.Memorization in SSL. Even though SSL rapidly grew in popularity during recent years, work on studying memorization in SSL is limited. Meehan et al. [36] proposed to quantify Déjà Vu memorization of SSL encoders with respect to particular data points by comparing the representations of these data points with the representations of a labeled public dataset. Data points whose k nearest public neighbors in the representation space are highly consistent in labels are considered to be memorized. Since SSL is aimed to train without labels, this approach is limited in practical applicability. More recently, Wang et al. [47] proposed SSLMem, a definition of memorization for SSL based on the leave-one-out definition from SL [23, 24] . Instead of relying on labels, this definition captures memorization through representation alignment, i.e., measuring the distance between representations of a data point's multiple augmentations. Since both works rely on the output representations to quantify memorization, neither of them is suitable for performing fine-grained localization of memorization. Yet, we use the setup of SSLMem as a building block to design our LayerMem metric which localizes memorization per layer.Localizing Memorization. In SL, most work focuses on localizing memorization on a per-layer basis and suggests that memorization happens in the deeper layers [3, 45] . By analyzing which neurons have the biggest impact on predicting the correct label of a data point, Maini et al. [35] were able to study memorization on a per-unit granularity. They do so by zero-ing out random units until a label flip occurs. Their findings suggest that only a few units are responsible for memorizing outlier data points. Yet, due to the absence of labels in SSL, this approach is inapplicable to our work. In the language domain, a significant line of work aims at localizing where semantic facts are stored within large language transformers [55, 37, 7, 44] . Chang et al. [14] even proposed benchmarks for localization methods in the language domain. In the injection benchmark (INJ Benchmark), they fine-tune a small number of neurons and then assess whether the localization method detects the memorization in these neurons. The deletion benchmark (DEL Benchmark) first performs localization, followed by the deletion of the responsible neurons, and a final assessment of the performance drop on the data points detected as memorized in the identified neurons. Since in SSL, performance drop cannot be measured directly due to the absence of a downstream task, the deletion approach is not applicable. Instead, we verify our UnitMem metric in a similar vein to the INJ Benchmark by fine-tuning a single unit on a data point and localizing memorization as we describe in Section 5.2.Studying Individual Units in ML Models. Early work in SL [22] already suggested that units at different model layers fulfill different functions: while units in lower layers are responsible for extracting general features, units in higher layers towards the model output are responsible for very specific features [51] . In particular, it was found that units represent different concepts required for the primary task [5] , where some units focus on single concepts whilst others are responsible for multiple concepts [38, 54] . While these differences have been identified between the units of models trained with SL, we perform a corresponding investigation in the SSL domain through the lens of localizing memorization.3 Background and Setup SSL and Notation. We consider an SSL training framework M. The encoder f : R n → R s is pre-trained, or in short trained, on the unlabeled dataset D to output representations of dimensionality s. Throughout the training, as the encoder improves, its alignment lossL A (f, x) = d(f (x ′ ), f (x ′′ ))between the representations of two random augmentations x ′ , x ′′ of any training data point x decreases with respect to a distance metric d (e.g., Euclidean distance). This effect has also been observed in non-contrastive SSL frameworks [53] . We denote by f l , l ∈ [1, . . . L] the l th layer of encoder f . Data points from the test set D are denoted as x.Memorized Data. Prior work in the SL domain usually generates outliers for measuring memorization by flipping the labels of training data points [23, 24, 35] . This turns these points into outliers that experience a higher level of memorization and leave the strongest possible signal in the model. Yet, such an approach is not suitable in SSL where labels are unavailable. Therefore, we rely on the SSLMem metric proposed by [47] to identify the most (least) memorized data points for a given encoder. The findings based on the SSLMem metric indicate that the most memorized data points correspond to atypical and outlier samples.SSLMem for Quantifying Memorization. SSLMem quantifies the memorization of individual data points by SSL encoders. It is, to the best of our knowledge, the only existing method for quantifying memorization in SSL without reliance on downstream labels. SSLMem for a training data point x is defined asEQUATIONwhere f and g are two classes of SSL encoders whose training dataset D differs in data point x. x ′ and x ′′ denote two augmentations randomly drawn from the augmentation set Aug that is used during training and d is a distance metric, here ℓ 2 -distance.Experimental Setup. We localize memorization in encoders trained with different SSL frameworks on five common vision datasets, namely CIFAR10, CIFAR100, SVHN, STL10, and ImageNet. We leverage different model architectures from the ResNet family, including ResNet9, ResNet18, ResNet34, and ResNet50. We also analyze Vision Transformers (ViTs) using their Tiny and Base versions. Results are reported over three independent trials. To identify the most memorized training data points, we rely on the SSLMem metric and follow the setup from [47] . More details on the experimental setup can be found in Appendix B. 2 For the readers' convenience, we include a glossary with short explanations for all concepts and background relevant to this work in Appendix A. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss our Limitations in Appendix F Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speechto-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The detailed experimental setup is introduced in Appendix B. 
layer-level localization of memorization: 
 In order to localize memorization on a per-layer granularity, we propose a new LayerMem metric which relies on the SSLMem metric, as a building block. Since the SSLMem as defined in Equation ( 1) is not normalized, we introduce the following normalization to the range [0, 1]EQUATIONsuch that values close to 0 denote no memorization while 1 denotes the highest memorization. This makes the score more interpretable. While SSLMem returns a memorization score per data point for a given encoder, LayerMem returns a memorization score per encoder layer l, measured on a (sub)setD ′ = {x 1 , ..., x |D ′ | } ⊆ D of training data D.Similar to SSLMem, LayerMem makes use of a second encoder g as a reference to detect memorization asEQUATIONf l , g l denote the output of encoders f and g after layer l, respectively. Intuitively, our LayerMem metric measures the average per-layer memorization over training data points x i ∈ D ′ . As our LayerMem build on SSLMem ′ , it also inherits the above normalization. Since Equation (3) operates on different layers' outputs which in turn depend on all previous layers, LayerMem risks to report accumulated memorization up to layer l. Therefore, we also define ∆ LayerMem D ′ (l) for all layers l > 1 asEQUATIONThis reports the increase in memorization of layer l with respect to the previous layer l -1.We present our core results and provide additional ablations on our LayerMem in Appendix C.3. We report LayerMem across the 100 randomly chosen training data points, their ∆LayerMem (denoted as ∆LM), followed by LayerMem for only the Top 50 memorized data points, their ∆LayerMem (denoted as ∆LM Top50), and LayerMem for only the Least 50 memorized data points. The results show that our LayerMem indeed increases with layer depth in SSL, similar to the trend observed for SL [45] , i.e., deeper layers experience higher memorization than early layers. However, our ∆LayerMem presents the memorization from a more accurate perspective, where we discard the accumulated memorization from previous layers, including the residual connections. ∆LayerMem indicates that the memorization increases in all the layers but is not monotonic.We also study the differences in localization of the memorization for most memorized (outliers and atypical examples) vs. least memorized data points (inliers), shown as columns LayerMem Top50 and LayerMem Least50 in Table 1 , respectively. While we observe that the absolute memorization for the most memorized data points is significantly higher than for the least memorized data points, they both follow the same trend of increasing memorization in deeper layers. The ∆LayerMem for the most memorized points (denoted as ∆LM Top50 in Table 1 ) indicates that, following the overall trend, high memorization of the atypical samples is also spread over the entire encoder and not confined to particular layers. Memorization in Vision Transformers. The memorization of Transformers [46] was, so far, primarily investigated in the language domain [26, 43] , however, its understanding in the vision domain is lacking. The fully-connected layers in language transformers were shown to act as key-value memories.Still, findings from language transformers cannot be easily transferred to vision transformers (ViTs) [20] : while language transformers operate on the level of discrete and interpretable input and output tokens, ViTs operate on continuous input image patches and output representations. Through the analysis of our newly proposed metric for memorization in SSL, in Table 2 (ViT-Tiny trained on CIFAR10 using MAE [30] ), we are the first to show that memorization in ViTs occurs more in deeper blocks and that within the blocks, fully-connected layers memorize more than attention layers. We present the full set of results for LayerMem and ∆LayerMem over all blocks in Table 10 .Memorization in Different SSL Frameworks. We also study the differences in memorization behavior between different SSL frameworks. Therefore, we compare the LayerMem score between corresponding layers of a ResNet50 trained on ImageNet with SimCLR [16] and DINO [13] , and of a ViT-Base encoder trained on ImageNet with DINO and MAE [30] . We ensure by early stopping that the resulting linear probing accuracies of the encoder pairs are similar for better comparability of their memorization. The ImageNet downstream task performance within both encoder pairs is 66.12% for SimCLR and 68.44% for DINO; and 60.43% for MAE and 60.17% for DINO. Our results in Table 4 show that encoders with the same architecture trained with different SSL frameworks experience a similar memorization pattern, namely that memorization occurs primarily in the deeper blocks/layers. In Figure 12 in Appendix C.1, we additionally show that memorization patterns between different 3 and Figure 13 in appendix C.5. The overlap within the 100 most memorized samples between adjacent layers is usually high but decreases the further the layers are separated. Our statistical analysis to compare the similarity of the orderings within different layers' most memorized samples using the Kendall's rank correlation coefficient shows that while for closer layers, we manage to reject the null hypothesis ("no correlation") with high statistical confidence (low p-value) which is not the case for further away layers. To analyze whether our LayerMem metric and its ∆ variant indeed localize memorization correctly, we first replace different layers of an encoder and then compute linear probing accuracy on various downstream tasks. Since prior work shows that memorization in SSL is required for downstream generalization [47] , we expect the highest performance drop when replacing the layers identified as most memorizing.Our results in Appendix C.7 verify this intuition. They show that by replacing the most memorizing layers of an encoder trained on a dataset A, e.g., CIFAR10, with the equivalent layers of another dataset B, e.g., STL10, the linear probing accuracy drop for CIFAR10 is significantly larger than when when replacing random or least memorizing layers. Surprisingly, at the same time, the replacement of the most memorizing layers from the CIFAR10 trained encoder with STL10 layers also causes the highest increase in STL10 linear probing accuracy (again in comparison to replacing random or least memorizing layers). See a full set of results for replacing any combination of 1, 2, and 3 layers in Table 30 , Table 32 , and Table 33 , respectively. These results suggest that we might be able to improve standard encoder fine-tuning by localizing the most memorizing layers and fine-tuning these instead of the last layer(s)-currently the standard practice for fine-tuning in SSL. We verify this assumption in Table 5 and show that fine-tuning the most memorizing layers indeed yields the highest downstream performance on the fine-tuning dataset. This shows that localizing memorization might have practical application for more efficient fine-tuning in the future. 
unit-level localization of memorization: 
 Experiments from the previous section highlight that we are able to localize the memorization of data points in particular layers of the SSL encoders. This raises the even more fundamental question on D ′ \ {x k }.The design of UnitMem is inspired by the class selectivity metric defined for SL by [39] . Class selectivity was derived from selectivity indices commonly used in neuroscience [19, 8, 25] and quantifies a unit's discriminability between different classes. It was used as an indicator of good generalization in SL. We provide more background in Appendix D.1. To leverage ideas from class selectivity for identifying memorization, we integrate three fundamental changes in our metric in comparison to the class selectivity metric. While class selectivity is calculated on classes of the test set and relies on class labels, our UnitMem is (1) label-agnostic and (2) computed on individual data points from the training dataset to determine their memorization. Additionally, to account for SSL's strong reliance on augmentations, (3) we calculate UnitMem over the expectation on the augmentation set used during training. Research from the privacy community [49, 34] suggests that those augmentations leave a stronger signal in ML models than the original data point, i.e., relying on the unaugmented point alone might under-report memorization. We verify this effect in Figure 5 in Appendix C.1. We note that through these fundamental changes UnitMem is able to measure memorization of individual data points within a class rather than to solely distinguish between classes or concepts like the original class selectivity. We provide further insights into this difference and perform experimental verification which highlights that UnitMem captures individual data points' memorization rather than capturing classes or concepts in Appendix C.2.To formalize our UnitMem, we first define the mean activation µ of unit u on a training point x asEQUATIONwhere the activation for convolutions feature maps is averaged across all elements of the feature map and for fully connected layers is an output from a single neuron (which is averaged across all patches of x in ViTs). Further, for the unit u, we compute the maximum mean activation µ max,u across all instances from D ′ , where N = |D ′ |, asEQUATIONLet k be the index of the maximum mean activation µ u (x k ), i.e., the argmax. Then, we calculate the corresponding mean activity µ -max across all the remaining N -1 instances from D ′ asEQUATIONFinally, we define the UnitMem of unit u as The value of the UnitMem metric is bounded between 0 and 1, where 0 indicates that the unit is equally activated by all training data points, while value 1 denotes exclusive memorization, where only a single data point triggers the activation, while all other points leave the unit inactive.EQUATIONWe present our core results and provide detailed additional ablations on our UnitMem Appendix C.1.Highly Memorizing Units Occur over Entire Encoder. Our analysis highlights that over all encoder architectures and SSL training frameworks, highly memorizing units are spread over the entire encoder. While, on average, earlier layers exhibit lower UnitMem than deeper layers, even the first layer contains highly memorizing units as shown in Figure 3 (first row). Figure 1a shows that this trend holds over different datasets. Yet, the SVHN dataset, which is visually less complex than the CIFAR10 or STL10 dataset, has the lowest number of highly memorizing units. This observation motivates us to study the relationship between the highest memorized (atypical or hard to learn) data points and the highest memorizing units.Most Memorized Samples and Units Align. To draw a connection between data points and unit memorization, we analyze which data points are responsible for the highest µ max scores. This corresponds to a data point which causes the highest activations of a unit, while other points activate the unit only to a small degree or not at all. We show the results in Figure 1b (also in Table 12 as well as in Figure 7 in Appendix C.1). For each unit u in the last convolutional layer of the ResNet9 trained on CIFAR10, we measure its UnitMem score, then we identify which data point is responsible for the unit's µ max,u , and finally measure this point's SSLMem score. We plot the UnitMem and SSLMem scores for each unit and its corresponding point. Our results highlight that the data points that experience the highest memorization according to the SSLMem score are also the ones memorized in the most memorizing units. Given the strong memorization in individual units, we next look into two methods to reduce it and analyze their impact.Differential Privacy reduces Unit Memorization. The gold standard to guarantee privacy in ML is Differential Privacy (DP) [21] . DP formalizes that any training data point should only have a negligible influence on the final trained ML model. To implement this, individual data points' gradients during training are clipped to a pre-defined norm, and controlled amounts of noise are added [1] . This limits the influence that each training data point can have on the final model. Building on the DP framework for SSL encoders [50] , we train a ViT-Tiny using MAE on CIFAR10 with three different privacy levels-in DP usually indicated with ε. We train non-private (ε = ∞), little private (ε = 20), and highly private (ε = 8) encoders and apply our UnitMem to detect and localize memorization. Our results in Figure 1c highlight that while with increasing privacy levels, the average UnitMem decreases, there are still individual units that experience high memorization.Data Point vs Class Memorization. Since stronger training augmentations yield higher class clustering [31] (i.e., the fact that data points from the same downstream class are close to each other in representation space but distant to data points from other classes), we also analyze how the SSL encoders differ from the standard class discriminators, namely SL trained models. Therefore, we go beyond our previous experiments that measure memorization of units with respect to individual data points and additionally study unit memorization at a class-granularity. Therefore, we adjust the class selectivity metric from [39] to perform on the training dataset rather than on the test data set as the original class selectivity. To avoid confusion between the two versions, refer to our adapted metric as ClassMem (see Appendix D.2 for an explicit definition). Equipped with UnitMem and ClassMem, we study the behavior of SSL encoders and compare between SSL and SL. For our comparison, we train an encoder with SimCLR and a model with SL using the standard cross entropy loss, both on the CIFAR100 dataset using ResNet9. We remove the classification layer from the SL trained model to obtain the same architecture as for the encoder trained with SimCLR. For comparability, we early stop the SL training once it reaches a comparable performance to the linear probing accuracy on CI-FAR100 obtained by the SSL encoder. Our results in Figure 2 show that overall, in SSL throughout all layers, average memorization of individual data points is higher than class memorization, whereas in SL, in deeper layers, the class memorization increases significantly. We hypothesize that this effect is due to earlier layers in SL learning more general features which are independent of the class whereas later layers learn features that are highly class dependent. For SSL, such a difference over the network does not seem to exist; both scores increase slightly, however, probably due to the SSL learning paradigm, memorization of individual data points remains higher.To better understand the memorization of units on the micro level, we investigate further the individual units in each layer. In Figure 3 , we plot the ClassMem vs UnitMem for each unit and in each of the eight encoder layers of ResNet9. Most units for SSL (row 1) constantly exhibit higher UnitMem than ClassMem, i.e., they cluster under the diagonal line, which suggests that most units memorize individual data points across the whole network. Contrary, the initial layers for the model trained with SL have a slight tendency to memorize data points over entire classes whereas in later layers, this trend drastically reverses and most units memorize classes. In Appendix C.4, we investigate how this different memorization behavior between SL and SSL affects downstream generalization.We verify the unit-based localization of memorization with our UnitMem by deliberately inserting memorization of particular units and checking if our UnitMem correctly detects it. Therefore, we first train a SimSiam-based [17] ResNet18 encoder trained on the CIFAR10 dataset. We select SimSiam over SimCLR for this experiment since SimCLR, as a contrastive SSL framework, cannot train on a single data point. Then, using LayerMem, we identity the last convolutional layer in ResNet18 (i.e., layer 4.1.conv2) as the layer with the highest LayerMem and ∆LayerMem memorization. We select the unit from the layer with the highest µ max and also pick a unit with no activation (µ max = 0) for some test data points. Then, we fine-tune these units using a single test data point and report the change in UnitMem for the chosen units in Table 13 . The results show that our UnitMem correctly detects the increase in memorization in both units. Additionally, we analyze the impact of zero-ing out the most or least memorizing vs. random units. Again with the argument that memorization is required for downstream generalization in SSL [47] , we expect the highest performance drop when Table 6 : Removing the least/most memorized units according to UnitMem preserves most/least linear probing performance. We prune according to units with highest or lowest UnitMem either per layer or for the entire network (total). We also present baselines where we prune randomly selected units. The standard deviation for this baseline is reported over 10 independent trials where different random units were pruned. We train the ResNet9 encoder using CIFAR10 and compute the UnitMem score using 5000 data points from the train set. zero-ing out the most memorizing units. Our results in Table 6 in and in Appendix C.1 confirm this hypothesis and show that removing the most memorizing units yields the highest loss in linear probing accuracy on various downstream tasks while pruning the least memorized units preserves better downstream performance than removing random units. These results suggest that future work may benefit from using our UnitMem metric for finding which units within a network can be pruned while preserving high performance. 
conclusions: 
 We propose the first practical metrics for localizing memorization within SSL encoders on a per-layer and per-unit level. By analyzing different SSL architectures, frameworks, and datasets using our metrics, we find that while memorization in SSL increases in deeper layers, a significant fraction of highly memorizing units can be encountered over the entire encoder. Our results also show that SSL encoders significantly differ from SL trained models in their memorization patterns, with the former constantly memorizing data points and the latter increasingly memorizing classes. Finally, using our metrics for localizing memorization presents itself as an interesting direction towards more efficient encoder fine-tuning and pruning. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: For the average results of all multiple experiments in the paper, we report the standard deviation in the tables and draw the error bar used to represent the standard deviation in the figures.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The hardware usage is introduced in Appendix B Guidelines:• The answer NA means that the does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/ EthicsGuidelines? Answer: [Yes]Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: In this work, we introduce two metrics for locating memorization in SSL vision encoders, as well as some key findings that result from experimenting with our metrics. No potential direct social impact is expected.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For the reader's convenience, we provide a glossary with all important terms and concepts related to our work in Table 7 .Table 7 : Glossary. We present a concise overview on the concepts relevant to this work.Atypical examples Data points that are uncommon in the data distribution and different in terms of their features. Examples: Figure 1 from [47] . Sometimes also called "outliers".A metric proposed by [39] which quantifies a unit's discriminability between different classes, measured on the test data.Our adaptation of Class Selectivity measured on the training data. Downstream Generalization Expresses how well an encoder is suited to solve some downstream tasks. For classification, it is often measured by linear probing, i.e., training an additional classification layer on top of the encoder output.Our proposed metric to quantify memorization of any layer in the SSL encoder.A phenomenon where a machine learning model stores detailed information on its training data.A data point that experiences high memorization by a machine learning model.A general trend in the low-level memorization of an SSL encoder, i.e., in which layers or units do memorization localize.Term used to refer to an individual neuron in fully connected layers or a channel in convolutional layers.Our proposed metric to quantify the memorization of any unit in the SSL encoder.Datasets. We base our experiments on ImageNet ILSVRC-2012 [42] , CIFAR10 [32] , CI-FAR100 [32] , SVHN [40] , and STL10 [18] .Models. We use the ResNet family of models [28] , including ResNet9, ResNet18, ResNet34, and ResNet50. In Table 8 , we present the detailed architecture of the ResNet9 model. Training Setup. Our experimental setup for training the encoders mainly follows [47] and we rely on their naming conventions and refer to the data points that are used to train encoder f , but not reference encoder g as candidate data points. In total, we use 50000 data points as training samples for CIFAR10, SVHN, and STL10 and 100000 for ImageNet with 5000 candidate data points per dataset. The encoders evaluated in the paper are trained with batch size 1024, and trained 600 epochs for CIFAR10, SVHN, and STL10, and 300 epochs for ImageNet. We set the batch size to 1024 for all our experiments and train for 600 epochs on CIFAR10, SVHN, and STL10, and for 300 epochs on ImageNet. As a distance metric to measure representation alignment, we use the ℓ 2 distance. We repeat all experiments with three independent seeds and report average and standard deviation. For reproducibility, we detail our full setup in Table 9 with the standard parameters that are used throughout the paper if not explicitly specified otherwise.Training Augmentations. We generate augmentations at random from the following augmentation sets (p indicates augmentation probability):• SL, standard, (referred to as weak augmentations): ColorJitter(0.9-0.9-0.9-0.5, p=0.4), Ran-domHorizontalFlip(p=0.5), RandomGrayscale(p=0.1), RandomResizedCrop(size=32)• SSL, standard, (referred to as normal augmentations): ColorJitter(0.8-0.8-0.8-0.2, p=0.8), RandomHorizontalFlip(p=1.0), RandomGrayscale(p=0.2), RandomResizedCrop(size=32)• SSL, stronger, (referred to as strong augmentations): ColorJitter(0.8-0.8-0.8-0.2, p=0.9), Ran-domHorizontalFlip(p=1.0), RandomGrayscale(p=0.5), RandomResizedCrop(size=32), RandomVer-ticalFlip(p=1.0)• SSL (independent): GaussianBlur(kernel_size=(4, 4), sigma=(0.1, 5.0), p=0.8), RandomInvert(p=0.2), RandomResizedCrop(size=32), RandomVerticalFlip(p=1.0)• Masking (MAE): 75% random masking Details on Computing UnitMem. Relying on insights from [33] , we calculate Equation ( 5) for the activations within UnitMem over ten augmentations since this has shown to yield a strong signal on the augmented data point. For convolutional feature maps, the activation of the unit is calculated as the average of all elements in the feature map. In ViTs, where we measure activation over fully-connected layers, we compute the activation per neuron and average across all patches of a given input. For example, for ViT Tiny encoder pretrained on CIFAR10, the input image of resolution 32x32 is patchified into 64 patches, each of size 4x4. Then, each patch is represented by a 192 dimensional embedding. The classification (CLS) embedding is prepended to the remaining 64 embeddings. Overall, we obtain 65 patches. The last fully connected layer has 192 neurons. For each neuron, we average its activations across the 65 patches. In the case of ViT Base, we have 768-dimensional embeddings and 197 patches for the input image of resolution 224x224.We provide the exact details of our experiments to verify our UnitMem through deliberate insertion of memorization in Section 5.2. We train a SimSiam-based [17] ResNet18 encoder on the CIFAR10 dataset and use LayerMem to identity layer 4.1.conv2, i.e., the last convolutional layer in ResNet18, as the layer with highest accumulated memorization. We select the unit from the layer with the highest µ max and also pick a unit with no activation (µ max = 0) for some test data points. Then, for compatability with pytorch which does not support individual unit training, we lock all parameters except for the targeted layer and train the model with a single sample from the testing dataset. We choose the sample that achieves the highest activation µ max on the unit with the highest UnitMem. We save the checkpoints after each epoch and test the µ max for the selected two units. Our results in Table 13 show that the value of µ max for the selected data point increases in both units and the data point remains the one responsible for the µ max .We finish all our experiments on two devices: a cloud server with four A100 GPUs and a local workstation with Intel 13700k CPU, Nvidia 4090 graphics card and 64GB of RAM We observe that the unit memorization monotonically increases throughout training and that the increase is particularly high during the first epochs. After the warm-up, we observe that the increase in unit memorization stagnates until the level of memorization on the unit level converges. The same trend can be observed over all layers which indicates that SSL encoders increase unit memorization throughout training.Measuring UnitMem without using augmentations leads to an under-reporting of memorization. To assess the impact on using augmentation to implement Equation (5) for the calculation of our UnitMem has an impact on the reported results, we train two ResNet9 models on the CIFAR100 dataset, one using SimCLR, the other one using standard SL with cross entropy loss.During training we rely on the standard augmentations for SL and SSL reported above. To measure memorization, we once use ten augmentations from the training augmentation set, and no augmentations otherwise and report the results in Figure 5 . We find that while the trend of the reported memorization is equal in both settings, the UnitMem measured without augmentations remains constantly lower than when measured with augmentations. This suggests that when measuring UnitMem, it is important to use augmentations to avoid under-reporting of the memorization. The number of data points used to measure UnitMem does not have a significant impact on the reported memorization. Using the same ResNet9, trained with SimCLR on CIFAR100, we assess whether the number of data points that we use to calculate UnitMem (the size of D ′ ) has an impact on the reported memorization. Then, we measure UnitMem using 100 random data point chosen one from each class in CIFAR100, 100 purely randomly chosen data points, and randomly chosen CIFAR100 data points. We present our findings in Figure 6 . Our results highlight that all the lines are within each other's standard deviation, indicating that there is no significant difference in the reported UnitMem, dependent on the make up of the dataset D ′ .Most memorized data points align with the most memorizing units. We train a ResNet9 on CIFAR10 using SimCLR and measure UnitMem for the 300 most and 300 least memorized data points identified using SSLMem by [47] . The measurement of the two sets (most vs least memorized data points) is performed independently. Our results in Figure 7 show that the UnitMem calculated on the most memorized data points is significantly higher than on the least memorized data points (we verify the significance with a statistical t-test in Table 11 .Figure 7 : Least vs most memorized data points.While also some of the least memorized data points lead to a high activation of the units, highest activation (on average and in particular) can be observed for the most memorized data points. This underlines the trend observed in Table 12 which shows that highly memorized data points align with the highly memorizing units.Computing UnitMem based on the median yields similar results to using the mean. Our UnitMem metric is inspired by the class selectivity defined for SL by [39] which quantifies a unit's discriminability between different classes, see Appendix D.1. Yet, we calculate the µ -max,u in Equation ( 7) using the median on the other individual training data points' activations while ClassSelectivity computes their equivalent of µ -max,u using the mean on all other test classes' activations. We show in Figure 8 over the 300 most and least memorized data points for a ResNet9 trained with SimCLR on CIFAR10 that using the median for UnitMem yields very similar results to using the mean.For SSL, the concrete augmentation set has no strong impact when measuring UnitMem. We additionally set out to study the impact of the augmentation set used to calculate UnitMem. Therefore, we calculate UnitMem on the ResNet9 trained on CIFAR10 using SimCLR using different augmentations sets. For SSL, we measure once with the standard training augmentations ("Normal"), with an independent set of augmentations of similar strength ("Independent"), with a weaker augmentation set for which we rely on the augmentations used to train the SL model ("Weak"), and an independent very strong set of augmentations modeled after MAE training and using a masking of 75% of the input image ("Masking"). Our results in Figure 9 depict the UnitMem over the last convolutional layer of the ResNet9 encoders. They highlight that the weak and independent augmentations report extremely similar UnitMem to the original set of training augmentations used. For SL, the impact of using different augmentations during training and measuring UnitMem is more expressed. We also measured for a weak augmentation set ("Normal"), an independent weak set ("Independent"), strong augentations for which we relied on the standard SSL augmentations ("Strong"), and the 75% masking ("Masking"). We observe that using the augmentations from training to calculate UnitMem yields the highest localization of memorization.Figure 10 : Different augmentation sets.Stronger augmentations reduce memorization. We also analyze how the training augmentation strength can impact the final encoder's UnitMem. We use ColorJitter, HorizontalFlip, Ran-domGrayscale, and RandomResizedCrop as augmentations. Their strength is determined by the probability of applying them and their level of distortion. In Appendix B, we present the exact parameters specified for each of them under different strengths. Our results in Figure 10 suggest that stronger augmentations yield lower perunit memorization. These findings are in line with prior theoretical work on SSL [48] highlighting that SSL performs foremost the task of instance discrimination (i.e., differentiating between individual images), but achieves clustering according to classes due to the augmentations: with stronger augmentations, multiple data points' augmented views will look extremely similar (e.g., the wheels of two different images of cars), such that they eventually activate the same unit. Thereby, this unit memorizes individual data points less while units trained with weaker augmentations depend on and memorize individual data points more. Note that we do not observe a strong dependency of our reported UnitMem on the concrete augmentation set used to calculate the metric (see Equation ( 5)) as we show in Figure 9 in Appendix C.1. Yet, using the original set of training augmentations, as we do for our experiments, yields the strongest signal. Table 10 : The memorization in ViT occurs primarily in the deeper blocks and more in the fully-connected than attention layers. We present the results for ViT Tiny pre-trained on CIFAR10 using MAE.∆LayerMem AT T N = LayerMem AT T N -ResBlock F C N -1 , ∆LayerMem F C N = LayerMem F C N - ResBlock AT T N , ∆BlockMem N = ResBlock F C N -ResBlock F C N -1 .ViT Additional Verification of UnitMem.We present the additional verification of the UnitMem metric in Table 14 . Therein, we perform two additional experiments to the verification presented in Section 5.2. First, we finetune the most memorizing unit and the inactive unit with 300 (instead of 1) data points from the test set (a). We observe that the data points that experienced the highest memorization for the selected unit remains the highest memorized of the 300 data points. Additionally, it experiences the highest memorization in the unit that used to be inactive. Second, we fine-tune the most memorizing unit and the inactive unit with the most memorized data point, but with a batch-size of 300 were we duplicate the data point 300 times (b). We observe that Table 11 : Most vs Least Memorized Data Points. We train a ResNet9 using SimCLR on CIFAR10 follwoing the setup by [47] . We then take the 50 most and 50 least memorized data points according to SSLMem and calculate the UnitMem over for the two sets of points. In the table, we report the average per-layer UnitMem of the two sets independently. We also perform a statistical t-test to find whether the UnitMem scores differ among most and least memorized data points. With p << 0.05, we are able to reject the null-hypothesis and find that the memorization according to UnitMem differs significantly between the most and least memorized data points. the effect of the fine-tuning on this point's memorization is far more expressed than when fine-tuning with 300 different data points.Additional Verification of UnitMem. In Table 13 , we prune, i.e., zero out neurons according to their level of memorization. Our results indicate that by pruning the most memorizing neurons, we cause the highest drop in downstream performance.To highlight that UnitMem reports memorization of individual data points rather than the a unit's ability to recognize class-wide concepts, we designed an additional experiment. For the experiment, we rely on the class concept of "wheel" as an example. In the STL10 dataset, three classes have a concept wheel, namely Truck, Plane, Car. If UnitMem was to report simply a unit's sensitivity to concepts of different classes (rather than individual data points), we would see a drop in UnitMem as we increase the percentage of data points with the concept wheel in the batch used to compute the metric. This is because then all data points should equally activate the unit, resulting in low memorization according to Equation (8) .We perform the experiment in Table 34 with 1000 data points chosen from different classes, namely 1) all classes (here 30% of the data points have wheels), 2) the classes Truck, Plane, Car (close to 100% of the samples now have the concept of wheels), and 3) purely the class car (close to 100% wheels). In 2) and 3), close to 100% of the samples now have the concept of wheels. Thus, if the units were responsible for the concept wheel, they would have a very high activation over all samples and the reported UnitMem should be very low. However, in our results, we see that we do have units with very high UnitMem. These can, in turn not be the units for the class-concept wheel, but must be units that focus on individual characteristics of the individual training images. This means that there must be unique features from the individual images that are still memorized that go beyond the concepts that are the same within a class. LayerMem is not sensitive to the size and composition of the batch. In our ablation study in , we show that LayerMem is not sensitive to the size and composition of batch it is computed on. The results can be found in Table 19 , where report the LayerMem measured for different number of candidate data points. We pre-trained a ResNet9 using SimCLR on CIFAR10 and determined LayerMem on batches of different sizes. For each batch size, we use 3 independent seeds (i.e., different batch compositions) and report the average LayerMem score and its standard deviation. The results show that the reported LayerMem score is, indeed, similar across all setups. This indicates LayerMem 's insensitivity to the choice of the batch used to compute it.Full Results with Memorization Scores over all Layers. We present the LayerMem score for ResNet18 in Table 15 , ResNet34 in Table 16 , and ResNet50 in Table 17 , all trained on CIFAR10 and using the SimCLR framework.We show the further breakdown of the memorization within the layers in Table 18 . We observe that the batch normalization layers (denoted as BN) together with the MaxPool layers have a negligible impact on memorization and most of the memorization in each layer is due to the convolutional operations. This is due to the much larger number of parameters in the convolutional filters than in the batch normalization layers and no additional parameters in the MaxPool layers, as shown in Table 8 . However, the memorization reported per convolutional layer is not correlated with the number of parameters of the layer. For instance, our ∆LayerMem reports the highest memorization for the 6-th layer, while layers 7 and 8 have each twice as many parameters, see Table 1 . Ablation on LayerMem's sensitivity. We perform an additional ablation to show that LayerMem is not sensitive to the number of samples in the batch used to compute it or the composition of the batch, i.e., which samples are chosen. Our results in Table 19 highlight that over different batches with 100, 500, 1000, and 5000 samples, the observed LayerMem scores are alike. This indicates LayerMem 's insensitivity to the choice of the batch used to compute it. We conducted an additional experiment where we trained a ResNet9 on CIFAR100 with SSL (SimCLR) and SL (cross-entropy loss). For the SL model, we remove the classification layer to turn it into an encoder. Then, we report linear probing accuracies on multiple downstream tasks in Table 20 .Our results highlight that the SL pretrained encoders exhibit a significantly higher downstream accuracy on their pretraining dataset than the SSL encoder. We assume that this is because of the class memorization. In contrast, the SL pretrained encoders perform significantly worse on other datasets than the SSL pretrained encoders since they might overfit the representations to their classes rather than provide more general (instance-based) representations as the SSL encoders. Additionally, we note that prior work has shown that the MAE encoder provides the highest performance when a few last layers are fine-tuned. The results in the original MAE paper in Figure 9 [30] indicate that fine-tuning a few last layers/blocks (e.g., 4 or 6 blocks out of 24 in ViT-Large) can achieve accuracy close to full fine-tuning (when all 24 blocks are fine-tuned). This is in line with our observation that the difference between UnitMem and ClassMem is the highest in the few last layers/blocks. Thus, fine-tuning only these last layers/blocks suffices for good downstream performance. We present the top 10 most memorized samples of each layer for the ResNet9 vision encoder trained with the CIFAR10 dataset in Figure 13 . The results show that the overlap within the top 10 most memorized samples between adjacent layers is usually high but decreases the further the layers are separated. This aligns with the results of overlap rate and Kendall's Tau test reported in Table 3 .We present the full results for the Table 4 , which show that the layer-based memorization is similar across encoders trained with different SSL frameworks. The results for the ResNet50 architecture trained with SimCLR and DINO using the ImageNet dataset are presented in Table 29 , and the results for the ViT-Base architecture trained with MAE and DINO using the ImageNet dataset are shown in Table 31 .Table 20 : Comparing the impact of memorization on downstream generalization between SSL SL. We train a ResNet9 on CIFAR100 with SSL (pretrained on CIFAR100 using SimCLR and SL (cross-entropy loss, trained until convergence). For the SL model, we remove the classification layer to turn it into an encoder. Then, we report linear probing accuracies on multiple downstream tasks in Encoder CIFAR100 CIFAR10 STL10 SVHN SSL 65.4% ± 0.98% 57.6% ± 0.87% 48.7% ± 0.98% 59.2% ± 0.76% SL (trained until convergence on CIFAR100, last layer removed) 66.1% ± 1.12% 56.7% ± 0.83% 46.1% ± 1.04% 58.6% ± 0.82%Figure 13 : The most memorized samples per layer according to LayerMem.To analyze whether our LayerMem metric and its ∆ variant indeed localize memorization correctly, we first replace different layers of an encoder and then compute linear probing accuracy on various downstream tasks. Since prior work shows that memorization in SSL is required for downstream generalization [47] , we expect the highest performance drop when replacing the layers identified as most memorizing. We verify this hypothesis and train a ResNet9 encoder f 1 on the CIFAR10 dataset and compute the LayerMem and ∆LayerMem scores per layer. Then, we select the three most memorized, random, and least memorized layers and replace them with the corresponding layers from a ResNet9 trained on STL10 (f 2 ). Our results in Table 21 show that the highest linear probing accuracy drop on the CIFAR10 test set for f 1 is caused by replacing the three most memorized layers from f 1 according to the ∆LayerMem score. The second biggest drop is observed when replacing according to LayerMem, highlighting that indeed our LayerMem metric and its ∆ variant identify the most crucial layers in SSL encoders for memorization. Surprisingly, the replacement of the layers in f 1 with the corresponding layers from f 2 causes the biggest simultaneous increase in the downstream accuracy for the STL10 dataset. We observe the same trends when f 2 is trained on SVHN (Table 22b ), for replacing single layers in ResNet9 (Appendix C.7), and replacing whole blocks in ResNet50 (Appendix C.7) instead of only individual layers as we present in Appendix C.7. The above analysis verifies that the LayerMem score and its ∆ variant identify the most crucial layers in SSL encoders. They further strengthen the claims that memorization is required for generalization [23, 24, 47] .Replacing layers in ResNet9 for SVHN. In Table 22b , we show the effect of replacing the most and least vs random layers of a CIFAR10 trained ResNet9 on the downstream performance. We replace the layers with the corresponding ones from a ResNet9 encoder trained on SVHN. Replacing blocks in ResNet50 trained on CIFAR10 with SimCLR. We present the results for replacing blocks in ResNet50 trained on CIFAR10 using SimCLR in Appendix C.7.Statistics of Batch-Norm layer for different datasets. Batch-norm layers between different datasets might have different statistics. This could impact the downstream performance. To investigate the changes, we measured the cosine similarity between the weights and biases of the batch-norm layers for two encoders (trained on CIFAR10 and STL10, respectively). The results in Table 25 show a high per-layer cosine similarity (average over all layers=0.823). This suggests that the statistics are similar, hence, no adjustment is required. We hypothesize that the similarity stems from the fact that the data distributions are similar and that we normalize both input datasets according to the ImageNet normalization parameters.In addition to the ℓ 2 distance, we also used 3 other distance metrics (ℓ 1 , cosine similarity, and angular distance) to evaluate the stability of LayerMem. Our results in Table 26 highlight that 1) the memorization scores are very similar, independent of the choice of the distance metric, and 2) the most memorizing layers according to and ∆LayerMem are the same over all metrics. This suggests that our findings are independent of the choice of distance metric. We performed an additional experiment where we trained encoders f and g independently with a different random seed (yielding f' and g') to study how random initialization of trainable parameters can affect the memorization of final vision encoder.The results are reported in Table 28 . We compared the overlap in most memorized samples between encoder f (from the paper) and f'. The results (Table 4 , attached PDF) show that overlap is overall high (min. 69% in layer 2) and increases in the later layers (max. 90%, final layer). According to the definition of SSLMem Equation (2), we let the representations of a given input data point x pass through the same (replaced) layer in both f and g. We show the LayerMem and ∆LayerMem scores after replacing a single layer in the ResNet9 encoder pre-trained using SimCLR on the CIFAR10 dataset in Table 23 and Table 30 . The LayerMem score of the replaced layer always drops as expected since this layer does not memorize any original training data points. The decrease in LayerMem between the initial and replaced layers is smaller in the earlier layers (e.g., 1st layer) as compared to the later layers (e.g., 6th layer). This might be because, in general, the earlier layers from different models might be more similar as they are responsible for extracting general features instead of specific ones for a given dataset. The most important take-away from these experiments is that the ∆LayerMem is not affected significantly and its values show the same trends after the layer replacement.C.12 Layer Replacement for Single, Two, and Three Layers at a TimeWe perform the experiment with the replacement of 1 layer in Table 30 , 2 layers Table 32 , and 3 layers Table 33 . The following results confirm our results from Table 21 in the main paper. When only a single layer is replaced, then the 6th (not the last layer) is the most important one. This layer had the highest LayerMem score. Note that the replacement of the 6th layer causes the highest drop in accuracy on the original CIFAR10 dataset and the highest gain in accuracy on STL10. Next, when two layers are replaced, then layers 6th and 8th play the most important roles, where their replacement with layers from the encoder trained on STL10 causes the highest drop on CIFAR10 and the highest performance increase on STL10. This is contrary to the common intuition, which would suggest the replacement of the last two layers instead.We denote the class selectivity metric as ClassSelectivity. It was proposed by [39] to quantify a unit's discriminability between different classes and described more in the main part of the paper in Section 5. We derive the basic metric in more detail here.To compute the ClassSelectivity metric per unit u, first the class-conditional mean activity is calculated for the test dataset D. We denote each test data point as xi ∈ D. We assume M classes C M j=1 , each with its corresponding test data points xc ∈ C j , where c = 1, 2, ..., |C j |.We define the mean activation μ of unit u for class C j asEQUATIONwhere the activation for convolutional feature maps is averaged across all elements of the feature map. Further, for the unit u, we compute the maximum mean activation μmax,u across all classes C, where M = |C|, asEQUATIONLet p be the index location of the maximum mean activation μu (x p ), i.e., the argmax. Then, we the corresponding mean activity μ-max,u across all the remaining M -1 classes asEQUATIONFinally, the class selectivity is then calculated as followsEQUATIONwhere μmax,u represents the highest class-conditional mean activity and μ-max,u denotes the mean activity across all other classes (for unit u and computed on the test dataset D).We use a similar definition as ClassSelectivity for ClassMem, which measures how much a given unit is responsible for the memorization of a class. While ClassSelectivity is calculated on the test set, we compute ClassMem on the training dataset.To compute the ClassMem metric per unit u, first, the class-conditional mean activity is calculated for the training dataset D ′ . We denote each train data point as x i ∈ D ′ . We assume M classes C M j=1 , each with its corresponding train data points x c ∈ C j , where c = 1, 2, ..., |C j |.We define the mean activation μ of unit u for class C j asEQUATIONwhere the activation for convolutional feature maps is averaged across all elements of the feature map. Further, for the unit u, we compute the maximum mean activation μmax,u across all classes C, where M = |C|, as μmax,u = max({μ u (x i )} M i=1 ).Let p be the index location of the maximum mean activation μu (x p ), i.e., the argmax. Then, we calculate the corresponding mean activity μ-max,u across all the remaining M -1 classes asEQUATIONFinally, the class Memorization is then calculated as followsEQUATIONwhere μmax,u represents the highest class-conditional mean activity and μ-max,u denotes the mean activity across all other classes (for unit u and computed on the train dataset D ′ ).Localizing Memorization on the Level of Individual Units. In Section 5, we considered memorization the perspective of individual units and identified that pruning the least/most memorized units according to UnitMem preserves the least/most performance (as shown in Table Table 6 ). The work by Maini et al. [35] The fact that memorization can enable privacy attacks, such as data extraction [9, 10, 12] , has been established in prior work. Yet, this paper advances the field of machine learning towards a novel fundamental understanding on where in SSL encoders memorization happens, and how memorization differs between standard SL models and SSL encoders. Our insights hold the potential to yield societal benefits in the form of the design of novel methods to reduce memorization, improve fine-tuning, and yield better model pruning algorithms.Table 29 : All-layer memorization. We train the ResNet50 encoder using SimCLR and DINO SSL frameworks on the dataset. We report the full results with the LayerMem and ∆LayerMem scores for each layer. • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Our code is attached as supplementary material.MaxPool1 


Decision: This paper investigates memorization in self-supervised learning encoders. It introduces two metrics, LayerMem and UnitMem, to locate memorization on a layer-wise and unit-wise basis. These metrics provide insights into the distribution of memorization within neural networks. The study reveals that memorization occurs throughout the layers of SSL encoders, not just in the final layers, and that, in vision transformers, memorization primarily takes place in fully connected layers. Furthermore, the authors propose practical applications for their metrics, such as enhancing the efficiency of fine-tuning and pruning through memorization localization.

Please update the paper with the clarifications and discussions that took place during rebuttal period.

Reviews: review: paper_summary: paper_summary: This paper investigates memorization in self-supervised learning encoders. It introduces two metrics, LayerMem and UnitMem, to locate memorization on a layer-wise and unit-wise basis. These metrics provide insights into the distribution of memorization within neural networks. The study reveals that memorization occurs throughout the layers of SSL encoders, not just in the final layers, and that, in vision transformers, memorization primarily takes place in fully connected layers. Furthermore, the authors propose practical applications for their metrics, such as enhancing the efficiency of fine-tuning and pruning through memorization localization.
strength_weakness: strength_weakness: - There are some missing explanations on some of the observations. In addition to presenting empirical results, the authors should include explanations and insightful remarks to clarify their intuition and the potential implications of these results. This is done to some extent already, but would improve the paper if it is done in all the sections.
- The presentation of the results could be improved. Tables full of numbers with tiny differences after the decimal point make it hard to read and navigate over. Instead of using tables that explain themselves, the paper relies on text explanations. 
- The proposed metrics, while inspired by previously known metrics, lack complete novelty. The LayerMem metric is primarily based on the existing SSLMem metric, with the only difference being a summation operation. This incremental modification does not constitute significant innovation.
questions: questions: - I am puzzled on why the memorization pattern remains the same between different datasets. For example in Figure 1  the histogram of UnitMem remains unchanged regardless of variations in the dataset or augmentations. Why does this happen? Do the authors have some suggested explanations for this?
- Besides using these metrics in fine tuning are there other practical benefits to these proposed metrics?
limitations: limitations: The authors discuss their metrics limitations to some extent.
paper_summary: paper_summary: This paper identifies where memorization occurs in SSLs, noting that memorization increases in deeper network layers, though high-memorizing units are distributed throughout the network. For the first time, they introduce a metric to measure memorization in SSLs and provide justification for its validity. They demonstrate that memorization primarily occurs in the fully-connected layers of vision transformers.
strength_weakness: strength_weakness: * It is not clear why the defined metric is a good measure of memorization, especially for comparing memorization across different layers. The magnitude of the distance between representations of different augmentations is highly dependent on the activation distribution of the layers and the distance metric used. While the normalization proposed in equations 2 and 4 attempts to address this, I am still unsure if it is sufficient.

* To justify the validity of the LayerMem memorization metric, they argue that effective memorization in SSLs leads to strong downstream performance. They also point out that replacing the layers with the highest memorization scores, as determined by the metric, results in a more performance drop, indicating the metric's reliability. However, I believe this justification alone is insufficient. Are there other pieces of evidence supporting the validity of this memorization metric?
questions: questions: * When replacing the two layers of models trained on CIFAR-10 and STL-10, if the replaced layer is at the beginning of the network, don't the activations propagate through the entire network? I don't think I fully understood that experiment.

* Is L2 distance used as the distance metric? If so, what is the justification for using it, and do the results change if a different metric is used?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces two novel metrics, LayerMem and UnitMem, to measure where memorization occurs within self-supervised neural networks. Through extensive experiments, this paper finds that memorization increases with layer depth, highly memorizing units are distributed throughout the encoder, atypical data points cause higher memorization, and most memorization happens in fully connected layers in ViTs. These findings suggest that localizing memorization can enhance fine-tuning and inform pruning strategies for better model performance and efficiency.
strength_weakness: strength_weakness: - The primary limitations of this paper are the lack of in-depth analysis and practical insights. Although the paper provides a broad range of empirical findings, it could be significantly improved by exploring how and why the neural networks behave as they do. Alternatively, strongly linking to or elaborating on prior work might be a promising direction to enhance the paper. Moreover, it would be beneficial if the paper provided practical methods to improve self-supervised neural networks based on the given takeaways. Even though Section 4.2 was interesting and insightful, it is insufficient for applying these findings in a real-world setting.
- If I understand correctly, I am not fully convinced by the definitions of `LayerMem` and `UnitMem` as appropriate metrics to measure memorization. For example, according to Equation 1, LayerMem seems to depend on the neural network’s properties. If neural networks are sufficiently equivariant with respect to data augmentation, then SSL and SSLMem should be zero. However, fortunately, the equivariance of ViTs is on par with that of CNNs [1], so I believe the main takeaways can hold in this case. Also, these metrics might depend on the norms of intermediate representations.
- There is room for improvement in the writing. The paper claims that “memorization increases but not monotonically,” yet I observed that LayerMem increases monotonically and even uniformly, except in some layers, as shown in Tables 1, 14, 15, and 16. Some interesting results, e.g., Lines 247 and 387, are only included in the Appendix. The paper attempts to correlate atypical datasets and memorization, but the evidence does not strongly support the claims. It is a minor thing, but I’m not familiar with ‘SL’ as an abbreviation for supervised learning.

Overall, considering the paper as one of the first attempts to investigate memorization in a self-supervised learning setting, I don’t find major weaknesses. Despite some limitations, I lean toward acceptance since I believe such attempts should be encouraged.

[1] Gruver, Nate, et al. "The lie derivative for measuring learned equivariance." arXiv preprint arXiv:2210.02984 (2022).
questions: questions: - In Figure 2, why does SSL UnitMem start with a non-zero value even at the first layer? I expected it would start near zero. Also, the values appear to be almost the same across all layers.
- Although high LayerMem values are observed in the later layers, this might be insufficient to conclude that these layers memorize more than the early layers. I anticipated that the effect of memorization would accumulate as the network depth increases.
- Please refer to the weaknesses section for additional comments.
limitations: limitations: Please refer to the weaknesses section. No ethical concerns found.
paper_summary: paper_summary: This work focuses on memorization in self-supervised learning. This paper seems to extent the work from [1] on SSLMem, that leverage the fact that a given training data and its corresponding handcrafted image augmentations should have a lower distance in the SSL embedding space than with a model in which this point was not in the training data. The authors extend SSLMem by using intermediate layers instead of the final representation. By doing so, the authors can localize which layers are the most memorizing training data. They show that indeed the last layer memorize the most, however we can still find significant memorization occurring also in intermediate layers. The author improve then their method by introducing a memorization metric based on individual units instead of a layer basis. 

[1] Memorization in self-supervised learning improves downstream generalization, Wang et al, ICLR 2024
strength_weakness: strength_weakness: - The paper can be read as just an extension of [1], so this contribution could be seen as lacking novelty. However, the empirical analysis that is provided might still interest the research community.
- I would be careful with claim such as being "first metric for localizing memorization". When we read Meehan et al., we can see that they did an ablation study across different layers, so they seem to be able to analyze which layer memorize the most. So it seems that there should have been a closer analysis and comparison with this paper results. In addition, they also provide experiments with ViT, so the author might also downplay the claim that investigating memorization in ViT is lacking.  So one weakness of this paper is the lack of empirical comparison with respect to the current literature. 
- Even if the authors did a great job in trying different models, they did not analyze on a sample level basis, how the memorized examples might differ depending on random seed or different hyper-parameters.  Like for two models that produces similar LayerMem score, are they memorizing the same examples/data points or not? Because both the overall score could be similar while memorizing different examples. A similar question is, does the memorize examples change across different layers? I would have appreciated reading at least some discussion on the relationship between model memorization with respect to which examples and how much those examples are consistent between models and layers.
- Concerning the SSLMem metric, I am wondering what is the interplay between both models (training with and without specific points) initialization and hyper-parameters. Like I would suspect that depending on the data shuffling seed, the examples that might be memorized first in the early layers might differ.
- There is also a lock of discussion on the impact of normalization layer or weight decay on the experimental results. I would suspect that having a stronger weight decay might reduce memorization.
questions: questions: - When replacing random or most/less memorized layers, how do you deal with normalization layers like batch norm since the statistics might be off?
- How much time does it take to train both models, compute augmentations and get the LayerMem and UnitMem metric?
limitations: limitations: The authors provide a limited limitation section. I would have appreciated to see as limitation that the method require training two disjoint model while having access to the original training data augmentation set. Thus, it does not seems that such method might work well on public model for which we do not know well the training details. Also the lack of a sample level analysis of the memorized example is an important limitation.

Comments: title: Thank you!
Thank you!
Official Comment of Submission1482 by Reviewer GSFs
Augmentation Strength and Memorization
Higher memorization with stronger data augmentation
Have concerns been addressed?
Have concerns been addressed?
Have concerns been addressed?
Have concerns been addressed?
comment: I would like to thank the authors for addressing my concerns. The rebuttal and new experiments provide interesting insights and will make the paper stronger. Therefore, I increase score.
We are happy that the Reviewer finds interesting takeaways in our paper. Thank you for maintaining the score.
We thank the Reviewer for the response and for increasing the score. We are glad that our answers addressed the concerns. Since the rating is still on the borderline, we are more than happy to provide any additional insights if needed.
Thank you for addressing my concerns. I have increased my score.
Thank you, authors, for addressing my concerns. I still believe the paper offers interesting takeaways. Therefore, I would like to maintain my original score.
We thank the reviewer for engaging in the discussion with us and are happy to answer the questions.

>**Augmentation Strength and Memorization.**

First, we would like to apologize for the confusion regarding the reported metric. Indeed, in the attached PDF, we considered LayerMem (while we wrote UnitMem in the rebuttal). 

Based on the reviewer’s comment, we ran additional experiments to verify our results. This time, for verification with an independent metric, we *actually* computed the average per-layer **UnitMem** for encoders trained with different strengths of augmentations. The results occurred to be the opposite to what we reported on LayerMem in the previous comment. Unfortunately, we made a mistake in the previous experiment with LayerMem due to the heat of the rebuttal and we are terribly sorry for that. When computing the LayerMem score, we did not change the path to the reference model $g$ in the code (and always took the $g$ trained with normal augmentation strength), even when evaluating $f$ trained with weak or strong augmentations. 

As a consequence, when evaluating memorization with the strong augmentations, our $g$--trained with normal augmentations--had an unusually high alignment loss, resulting in the reported memorization being too high. In contrast, when evaluating for weak augmentations, our $g$--again trained with normal augmentations--had unusually small alignment loss, resulting in the reported memorization being too low.  

We corrected the experimental setup and re-ran the experiment. The new results for LayerMem and UnitMem are aligned and demonstrate that memorization is actually smaller for stronger augmentations: 

### **Results for LayerMem:**

| augmentations: | weak |  | normal (from paper) |  | strong |  |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Layer | LayerMem | Delta Layermem | LayerMem | Delta Layermem | LayerMem | Delta Layermem |
| 1 | 0.092 | - | 0.091 | - | 0.089 | - |
| 2 | 0.123 | 0.031 | 0.123 | 0.032 | 0.120 | 0.031 |
| 3 | 0.154 | 0.031 | 0.154 | 0.031 | 0.150 | 0.030 |
| 4 | 0.184 | 0.030 | 0.183 | 0.029 | 0.178 | 0.028 |
| Res2 | 0.187 | 0.003 | 0.185 | 0.002 | 0.181 | 0.003 |
| 5 | 0.215 | 0.028 | 0.212 | 0.027 | 0.208 | 0.027 |
| 6 | 0.249 | 0.034 | 0.246 | 0.034 | 0.241 | 0.033 |
| 7 | 0.280 | 0.031 | 0.276 | 0.030 | 0.269 | 0.028 |
| 8 | 0.313 | 0.033 | 0.308 | 0.032 | 0.300 | 0.031 |
| Res6 | 0.315 | 0.002 | 0.311 | 0.003 | 0.302 | 0.002 |

### **Results for UnitMem:**

| augmentations: | weak | normal (from paper) | strong |
|:---:|:---:|:---:|:---:|
| Layer | Avg. UnitMem | Avg. UnitMem | Avg. UnitMem |
| 1 | 0.362 | 0.360 | 0.355 |
| 2 | 0.357 | 0.354 | 0.352 |
| 3 | 0.361 | 0.357 | 0.351 |
| 4 | 0.365 | 0.362 | 0.355 |
| Res2 | - | - | - |
| 5 | 0.370 | 0.366 | 0.360 |
| 6 | 0.375 | 0.374 | 0.364 |
| 7 | 0.381 | 0.379 | 0.369 |
| 8 | 0.387 | 0.384 | 0.375 |
| Res6 | - | - | - |

Please note that since the residual connections do not contain individual units, it is impossible to compute the UnitMem score for them.

The new results for *augmentations* now align with the effects observed when applying **regularization** through weight decay. As shown in Figure 1 in the attached rebuttal PDF, we observe that with stronger weight decay regularization, memorization also decreases. 

Once again, we sincerely apologize for the confusion and hope that the new results fully address the concerns.
I would like to thank the authors for their response and running additional experiments.

In particular for the data augmentation experiment, I am puzzled by the results. The results show that the stronger the data augmentation during training, the higher memorization happens in a layer level. 
 Why is this the case? Shouldn't it be the opposite? Why the units/layers experience more memorization if during training they have seen more data samples? Data augmentation would in principle decrease memorization. 


(btw, I think there is a typo in your response? I only see the results in the attached PDF for LayerMem but in the response it is written the results are for UnitMem?)
We would like to thank the Reviewer for recognizing that our work enables localizing memorization, down to the granularity of individual units. 
In summary, our rebuttal addressed the following:
1. **Adjusting the claims**: We adjusted the claims according to the reviewer’s suggestion to recognize Meehan et al.’s contributions on analyzing ViTs and performing the ablation study on which layers hold highest accumulated Deja Vu memorization.
2. **Comparing memorization between layers**: Based on the reviewer’s suggestion, we performed additional experiments to study how memorized samples differ between the layers of a single encoder and between the layers of two different encoders. We show that especially adjacent layers memorize similar data points, and that over different encoders, the overlap of most memorized samples is most similar in later layers.
3. **Weights decay**: We performed also additional experiments based on the reviewer’s suggestion to show how UnitMem changes under different strength of weight decay and found that memorization decreases as weight decay increases, yet, also downstream performance drops.

We would like to check if there are any pending concerns that should address to further strengthen the reviewer’s confidence for acceptance of our paper?
We would like to thank the Reviewer for recognizing our work as the first to propose metrics for localization of memorization in SSL encoders and presenting insightful as well as interesting findings. 
In summary, our rebuttal addressed the following:
1. **In depth analysis**: We extended the analysis of the paper and included the study on which samples are memorized in which layers. As requested, we added more explanations to the paper (using the additional 1 page if accepted). 
2. **Validating the metrics**: We provide additional experimental and conceptual insights to validate our metrics, in particular, we show that the design of LayerMem is independent of the norms in the intermediate representations.

We would like to check: are there any pending concerns that we should address to further strengthen the reviewer’s confidence for acceptance of our paper?
We would like to thank the Reviewer for recognizing our work to proposes the first metrics for localization of memorization in SSL. 
In summary, our rebuttal addressed the following:
1. **Validity of the measure**: We present additional conceptual and experimental results validating that our metrics localize memorization.
2. **Clarifications**: We provided clarification on our experimental insights.
3. **Distance metric**: We performed additional experiments highlighting that our LayerMem metric is independent of the underlying distance metric.

We would like to check: are there any other pending concerns that we should address?
We would like to thank the Reviewer for recognizing our work as interesting, practical, and well executed. The paper has definitely improved as a result of the feedback. 
In summary, our rebuttal addressed the following:
1. **Additional Insights**: We improved the writing by adding additional insights, observations, explanations, and insightful remarks.
2. **Augmentations**: We performed additional requested experiments that show that stronger training augmentations lead to higher UnitMem.
3. **Practical Application**: We provided a list of practical applications for our work, where apart from improved fine-tuning, it could also be used for the memorization-based compressing schemes, strengthening the membership inference attacks, and watermarking. 

We would like to check if there are any pending concerns that would further strengthen the reviewer’s confidence for acceptance of our paper?